Packets stop being forwarded after some time with cascade setup with local bridging

Post your questions about SoftEther VPN software here. Please answer questions if you can afford.
Post Reply
pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Wed Jul 20, 2022 11:29 am

See the attached drawing for the following setup we have with the Softether VPN software.
SoftEther bug report.png
We are generating UDP packets at 100 Mbit/s (the physical connection is optional 10 Gbit/s) and while we receive some packets on the other end, it stops working after some seconds and ~3000 Packets (values varying with every run). Afterwards no more packets are received at step VIII in the drawing. Strangely, the counter in the VPN Server Manager still count up, so packets are apparently still received by the VPN server, just not forwarded.

When we put the cascade connection offline and online again then it works again for some seconds until it stops working again.

Notes:
  • We tested it with both the latest beta and the latest stable version. We did not mix versions.
  • When we reduced the size of the packets from 1514 to 500 then it still does not work, but more packets are received until it stops working (our tester also sends more packets per second for the same data rate).
Do you have any idea what could have gone wrong? Seeing that it works and then suddenly stops working seems to imply that it is a bug in the VPN software.
You do not have the required permissions to view the files attached to this post.

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Wed Jul 20, 2022 11:54 am

VI cascades to IV and creates a network loop which kills the connection: IV -> V -> VI -> IV.
Incidentally, bug reports https://github.com/SoftEtherVPN/SoftEtherVPN/issues

eddiewu
Posts: 286
Joined: Wed Nov 25, 2020 9:10 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by eddiewu » Wed Jul 20, 2022 12:46 pm

Check the server log first. Softether has some burst protection.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Wed Jul 20, 2022 12:55 pm

Thanks for the answers!

@solo: Concerning a possible network loop, I doubt it, since the cascade connection only goes into one direction, only from the edge (the second server) to the center (the first server).

@eddiewu: Depending on the traffic sent through (we know also tried it with real-world traffic), there are some message concerning a high volumne of broadcast packets but that's about it.

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Wed Jul 20, 2022 10:18 pm

pkaf wrote:
Wed Jul 20, 2022 12:55 pm
Concerning a possible network loop, I doubt it, since the cascade connection only goes into one direction, only from the edge (the second server) to the center (the first server).
According to your diagram there is a physical connection between Eth8-1 and Eth8-2. If you also cascade from SE-2 to SE-1, you create a loop with broadcast storm. There is no "one direction" on Layer-2 networks.

Dilbert92011
Posts: 3
Joined: Fri Jul 08, 2022 7:34 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by Dilbert92011 » Thu Jul 21, 2022 12:56 pm

solo wrote:
Wed Jul 20, 2022 10:18 pm
According to your diagram there is a physical connection between Eth8-1 and Eth8-2. If you also cascade from SE-2 to SE-1, you create a loop with broadcast storm. There is no "one direction" on Layer-2 networks.
But doesn't the cascade connection *need* the physical connection between eth8-1 and eth8-2 ?
How else could the units communicate?
(I am new to SoftEther and want to learn)

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Thu Jul 21, 2022 1:14 pm

No, you cascade over the internet, not LAN.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Fri Jul 29, 2022 1:53 pm

I have an update on the issue, we managed to simplify the setup where the problem occurs and it now happens:
  • With a simple point-to-point connection.
  • Without the use of an ARM-based device.
  • Even when using a SoftEther VPN Bridge instead of a SoftEther VPN Server.
  • Even when the decrypting VPN Bridge is on a Windows PC, interestingly there the problem already occurs with ~150 instead of ~3000 packets.
So our current setup is made up of:
  • Device 1, is Linux-based, x86-64, it receives traffic on its promiscuous interface 1. The traffic comes from a traffic generator (e.g. tcpreplay on captured traffic or artificial traffic). This interface is configured as a local bridge to the SoftEther VPN Server running there. Interface 2 there has 1.2.3.6/32 configured as its IP and its MAC is 1c:1b:0d:00:eb:a5.
  • Device 2 is a Windows PC, its interface 1 is configured as a local bridge to the SoftEther VPN Bridge running there, it is connected to the traffic receiver appliance. Interface 2 there has 1.2.3.4/32 as its IP and its MAC is 80:ee:73:b0:27:82.
  • A 1G connection between interface 2 of either device, the idea is that the traffic goes into interface 1 of the first device, is then encrypted, sent to interface 2 received on interface 2 of the second device, decrypted and then sent out on interface 1. (The problem apparently occurs between decryption and forwarding, more details later.)
  • The SoftEther VPN Bridge on device 2 is configured to connect to the VPN server on device 1, i.e. IP 1.2.3.6.
Those are commands used on device 1 (the Linux machine), the order is important:

Code: Select all

ip address add 1.2.3.6/32 dev enp7s0
ip route add 1.2.3.4/32 dev enp7s0
arp -s 1.2.3.4 80:ee:73:b0:27:82
And on the Windows machine the configuration is as follows:
  • Configured static IP on interface 1 and with 1.2.3.4 subnet 255.255.255.255
  • For both relevant interfaces all protocol stacks except for IPv4 and SoftEther are disabled (those checkboxes in that one screen)
  • Code: Select all

    route ADD 1.2.3.6 MASK 255.255.255.255 1.2.3.6 IF 39
    arp -s 1.2.3.6 1c-1b-0d-00-eb-a5
    
Like before the connection between the two devices is working perfectly, it’s just after decryption where the problem occurs.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Fri Jul 29, 2022 1:56 pm

Following are some screenshots of the statistics screens with a test case involving 1500 generated packets

Link before the packets are generated:
bef_link.png
Local bridge before:
bef_bridge.png
Link after:
after_link.png
You do not have the required permissions to view the files attached to this post.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Fri Jul 29, 2022 1:59 pm

And last but not leaast, the local bridge afterwards, note how it only menions ~150 packets when we would expect 1500.
after_bridge.png
Notes:
  • Traffic receiver does not answer with ARP packets
  • Occurs with pure L2 packets as well.
  • Occurs in monitor mode as well:
    after_bridge_monitor.png
You do not have the required permissions to view the files attached to this post.

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Sat Jul 30, 2022 10:16 am

Please start the network and when "the problem occurs with ~150 packets" post as code the output of:

Linux
  • ifconfig
  • route -n
  • arp
  • iptables -t nat -L
  • cat /proc/sys/net/ipv4/ip_forward
  • cat /proc/sys/net/ipv4/conf/all/proxy_arp
  • cat /proc/sys/net/ipv4/conf/enp7s0/proxy_arp
  • brctl show
  • vpncmd localhost /server /password:*** /cmd BridgeList
  • vpncmd localhost /server /password:*** /adminhub:@@@ /cmd SecureNatStatusGet
//replace: *** with SE admin password; @@@ with VPN hub's name

Windows
  • ipconfig /all
  • netstat -r
  • arp -a
  • PS> Get-Service RemoteAccess

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Mon Aug 01, 2022 8:01 am

Linux:

Code: Select all

root@ares:~/vpnserver# ifconfig
br-a7a196f16e8d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::42:68ff:fe4f:d07b  prefixlen 64  scopeid 0x20<link>
        ether 02:42:68:4f:d0:7b  txqueuelen 0  (Ethernet)
        RX packets 120869  bytes 9701466 (9.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 74352  bytes 21963198 (21.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:c2:f0:46:8f  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::1e1b:dff:fe00:eba2  prefixlen 64  scopeid 0x20<link>
        ether 1c:1b:0d:00:eb:a2  txqueuelen 1000  (Ethernet)
        RX packets 6984814  bytes 10479005310 (10.4 GB)
        RX errors 82  dropped 578  overruns 0  frame 82
        TX packets 3957817  bytes 348824024 (348.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0f1: flags=4355<UP,BROADCAST,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::1e1b:dff:fe00:eba3  prefixlen 64  scopeid 0x20<link>
        ether 1c:1b:0d:00:eb:a3  txqueuelen 1000  (Ethernet)
        RX packets 17642163  bytes 15491072622 (15.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 761737  bytes 211471711 (211.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp6s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.173  netmask 255.255.252.0  broadcast 192.168.3.255
        inet6 fe80::1e1b:dff:fe00:eba4  prefixlen 64  scopeid 0x20<link>
        ether 1c:1b:0d:00:eb:a4  txqueuelen 1000  (Ethernet)
        RX packets 12302031  bytes 1730490661 (1.7 GB)
        RX errors 0  dropped 1672  overruns 0  frame 0
        TX packets 10234176  bytes 7704553832 (7.7 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfb200000-fb27ffff  

enp7s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 1.2.3.6  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::1e1b:dff:fe00:eba5  prefixlen 64  scopeid 0x20<link>
        ether 1c:1b:0d:00:eb:a5  txqueuelen 1000  (Ethernet)
        RX packets 322363  bytes 56490130 (56.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 343013  bytes 74204655 (74.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfb100000-fb17ffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 33098389  bytes 2351331799 (2.3 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33098389  bytes 2351331799 (2.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth0599c07: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::dc47:84ff:fee0:dd08  prefixlen 64  scopeid 0x20<link>
        ether de:47:84:e0:dd:08  txqueuelen 0  (Ethernet)
        RX packets 69120301  bytes 8206851879 (8.2 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 111360484  bytes 17917330385 (17.9 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth765279a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1875:37ff:feb8:4a7b  prefixlen 64  scopeid 0x20<link>
        ether 1a:75:37:b8:4a:7b  txqueuelen 0  (Ethernet)
        RX packets 4825828  bytes 548338084 (548.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4112762  bytes 446818557 (446.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth7f3a03d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::d831:e5ff:fe38:5462  prefixlen 64  scopeid 0x20<link>
        ether da:31:e5:38:54:62  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 98912  bytes 6201216 (6.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth91cfbac: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::28cf:98ff:fe8e:df78  prefixlen 64  scopeid 0x20<link>
        ether 2a:cf:98:8e:df:78  txqueuelen 0  (Ethernet)
        RX packets 122098  bytes 7965204 (7.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 189453  bytes 10702814 (10.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth9b8f550: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::384d:3aff:fe28:f8ab  prefixlen 64  scopeid 0x20<link>
        ether 3a:4d:3a:28:f8:ab  txqueuelen 0  (Ethernet)
        RX packets 565  bytes 2039303 (2.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99842  bytes 6284194 (6.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth9f9f862: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::b07c:a9ff:fece:ae7c  prefixlen 64  scopeid 0x20<link>
        ether b2:7c:a9:ce:ae:7c  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 98911  bytes 6201146 (6.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vetha4823de: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::831:aeff:fe53:1c39  prefixlen 64  scopeid 0x20<link>
        ether 0a:31:ae:53:1c:39  txqueuelen 0  (Ethernet)
        RX packets 3569978  bytes 742339268 (742.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3944821  bytes 1150999303 (1.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb0d058e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::bc32:32ff:fe83:b198  prefixlen 64  scopeid 0x20<link>
        ether be:32:32:83:b1:98  txqueuelen 0  (Ethernet)
        RX packets 257  bytes 46402 (46.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99132  bytes 6257417 (6.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb42b3a5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::c4b2:37ff:fec2:c671  prefixlen 64  scopeid 0x20<link>
        ether c6:b2:37:c2:c6:71  txqueuelen 0  (Ethernet)
        RX packets 137038985  bytes 23642460830 (23.6 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91880404  bytes 17289748240 (17.2 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethd1addb3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::4094:3bff:fe00:6f18  prefixlen 64  scopeid 0x20<link>
        ether 42:94:3b:00:6f:18  txqueuelen 0  (Ethernet)
        RX packets 86158  bytes 7919781 (7.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 172313  bytes 28079594 (28.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethdd17125: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::c8fb:b2ff:fe8d:f9d6  prefixlen 64  scopeid 0x20<link>
        ether ca:fb:b2:8d:f9:d6  txqueuelen 0  (Ethernet)
        RX packets 29870367  bytes 10628282880 (10.6 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33509858  bytes 6972901347 (6.9 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@ares:~/vpnserver# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 enp6s0
1.2.3.4         0.0.0.0         255.255.255.255 UH    0      0        0 enp7s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-a7a196f16e8d
192.168.0.0     0.0.0.0         255.255.252.0   U     0      0        0 enp6s0
192.168.0.1     0.0.0.0         255.255.255.255 UH    100    0        0 enp6s0
root@ares:~/vpnserver# arp
Address                  HWtype  HWaddress           Flags Mask            Iface
172.18.0.8               ether   02:42:ac:12:00:08   C                     br-a7a196f16e8d
172.18.0.11              ether   02:42:ac:12:00:0b   C                     br-a7a196f16e8d
rp4n1.internal.cubro.or  ether   dc:a6:32:d7:c0:aa   C                     enp6s0
vpn.internal.cubro.org   ether   00:50:56:8a:82:fd   C                     enp6s0
172.18.0.13              ether   02:42:ac:12:00:0d   C                     br-a7a196f16e8d
1.2.3.4                  ether   80:ee:73:b0:27:82   CM                    enp7s0
devo.lan                 ether   00:0c:29:c7:0f:1e   C                     enp6s0
phk.lan                  ether   a0:d3:c1:4f:4b:57   C                     enp6s0
192.168.3.65             ether   b8:97:5a:fe:58:3b   C                     enp6s0
router.internal.cubro.o  ether   ac:1f:6b:1b:1a:5f   C                     enp6s0
192.168.3.128            ether   60:e3:27:13:58:71   C                     enp6s0
192.168.3.204            ether   14:ab:c5:16:ff:da   C                     enp6s0
192.168.0.185            ether   1c:1b:0d:00:eb:92   C                     enp6s0
root@ares:~/vpnserver# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        anywhere            
MASQUERADE  all  --  172.18.0.0/16        anywhere            
MASQUERADE  tcp  --  172.18.0.5           172.18.0.5           tcp dpt:http-alt
MASQUERADE  tcp  --  172.18.0.5           172.18.0.5           tcp dpt:6653
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18321
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18052
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18048
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18032
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18024
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18012
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18006
MASQUERADE  tcp  --  172.18.0.9           172.18.0.9           tcp dpt:18002
MASQUERADE  tcp  --  172.18.0.13          172.18.0.13          tcp dpt:https
MASQUERADE  tcp  --  172.18.0.13          172.18.0.13          tcp dpt:http

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http-alt to:172.18.0.5:8080
DNAT       tcp  --  anywhere             anywhere             tcp dpt:6653 to:172.18.0.5:6653
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18321 to:172.18.0.9:18321
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18052 to:172.18.0.9:18052
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18048 to:172.18.0.9:18048
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18032 to:172.18.0.9:18032
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18024 to:172.18.0.9:18024
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18012 to:172.18.0.9:18012
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18006 to:172.18.0.9:18006
DNAT       tcp  --  anywhere             anywhere             tcp dpt:18002 to:172.18.0.9:18002
DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:172.18.0.13:443
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:172.18.0.13:80
root@ares:~/vpnserver# cat /proc/sys/net/ipv4/ip_forward
1
root@ares:~/vpnserver# cat /proc/sys/net/ipv4/conf/all/proxy_arp
0
root@ares:~/vpnserver# cat /proc/sys/net/ipv4/conf/enp7s0/proxy_arp
0
root@ares:~/vpnserver# brctl show

Command 'brctl' not found, but can be installed with:

apt install bridge-utils

root@ares:~/vpnserver# ./vpncmd localhost:5555 /server /password:*** /cmd BridgeList
vpncmd command - SoftEther VPN Command Line Management Utility
SoftEther VPN Command Line Management Utility (vpncmd command)
Version 4.38 Build 9760   (English)
Compiled 2021/08/17 22:32:49 by buildsan at crosswin
Copyright (c) SoftEther VPN Project. All Rights Reserved.

Connection has been established with VPN Server "localhost" (port 5555).

You have administrator privileges for the entire VPN Server.

VPN Server>BridgeList
BridgeList command - Get List of Local Bridge Connection
Number|Virtual Hub Name|Network Adapter or Tap Device Name|Status
------+----------------+----------------------------------+---------
1     |VPN_01          |enp3s0f0                          |Operating
The command completed successfully.


root@ares:~/vpnserver# ./vpncmd 127.0.0.1:5555 /server /password:*** /adminhub:VPN_01 /cmd SecureNatStatusGet
vpncmd command - SoftEther VPN Command Line Management Utility
SoftEther VPN Command Line Management Utility (vpncmd command)
Version 4.38 Build 9760   (English)
Compiled 2021/08/17 22:32:49 by buildsan at crosswin
Copyright (c) SoftEther VPN Project. All Rights Reserved.

Connection has been established with VPN Server "127.0.0.1" (port 5555).

You have administrator privileges for the entire VPN Server.

The Virtual Hub "VPN_01" has been selected.
VPN Server/VPN_01>SecureNatStatusGet
SecureNatStatusGet command - Get the Operating Status of the Virtual NAT and DHCP Server Function (SecureNat Function)
Item                     |Value
-------------------------+---------
Virtual Hub Name         |VPN_01
NAT TCP/IP Sessions      |0 Session
NAT UDP/IP Sessions      |0 Session
NAT ICMP Sessions        |0 Session
NAT DNS Sessions         |0 Session
Allocated DHCP Clients   |0 Client
Kernel-mode NAT is Active|No
Raw IP mode NAT is Active|No
The command completed successfully.
Windows:

Code: Select all

C:\WINDOWS\system32>ipconfig -all

Windows-IP-Konfiguration

   Hostname  . . . . . . . . . . . . : DESKTOP-TSUFNC2
   Primäres DNS-Suffix . . . . . . . :
   Knotentyp . . . . . . . . . . . . : Hybrid
   IP-Routing aktiviert  . . . . . . : Nein
   WINS-Proxy aktiviert  . . . . . . : Nein

Unbekannter Adapter VPN10 - VPN Client:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : VPN Client Adapter - VPN10
   Physische Adresse . . . . . . . . : 5E-61-B1-14-51-6A
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja

Ethernet-Adapter Ethernet:

   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : Intel(R) Ethernet Connection (3) I218-LM
   Physische Adresse . . . . . . . . : 80-EE-73-B0-27-82
   DHCP aktiviert. . . . . . . . . . : Nein
   Autokonfiguration aktiviert . . . : Ja
   IPv4-Adresse  . . . . . . . . . . : 1.2.3.4(Bevorzugt)
   Subnetzmaske  . . . . . . . . . . : 255.255.255.255
   Standardgateway . . . . . . . . . :
   NetBIOS über TCP/IP . . . . . . . : Aktiviert

Drahtlos-LAN-Adapter WLAN:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : Realtek RTL8188EE Wireless LAN 802.11n PCI-E NIC
   Physische Adresse . . . . . . . . : 4C-BB-58-8F-2C-D2
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja

Drahtlos-LAN-Adapter LAN-Verbindung* 1:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : Microsoft Wi-Fi Direct Virtual Adapter
   Physische Adresse . . . . . . . . : 4E-BB-58-8F-2C-D2
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja

Drahtlos-LAN-Adapter LAN-Verbindung* 2:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : Microsoft Wi-Fi Direct Virtual Adapter #3
   Physische Adresse . . . . . . . . : 4C-BB-58-8F-2C-D2
   DHCP aktiviert. . . . . . . . . . : Nein
   Autokonfiguration aktiviert . . . : Ja

Ethernet-Adapter VMware Network Adapter VMnet1:

   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : VMware Virtual Ethernet Adapter for VMnet1
   Physische Adresse . . . . . . . . : 00-50-56-C0-00-01
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja
   Verbindungslokale IPv6-Adresse  . : fe80::a4c8:91ff:1fcb:8a9d%20(Bevorzugt)
   IPv4-Adresse  . . . . . . . . . . : 192.168.158.1(Bevorzugt)
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Lease erhalten. . . . . . . . . . : Donnerstag, 21. Juli 2022 14:21:02
   Lease läuft ab. . . . . . . . . . : Montag, 1. August 2022 10:16:09
   Standardgateway . . . . . . . . . :
   DHCP-Server . . . . . . . . . . . : 192.168.158.254
   DHCPv6-IAID . . . . . . . . . . . : 1191202902
   DHCPv6-Client-DUID. . . . . . . . : 00-01-00-01-21-44-1F-09-80-EE-73-B0-27-82
   DNS-Server  . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS über TCP/IP . . . . . . . : Aktiviert

Ethernet-Adapter VMware Network Adapter VMnet8:

   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : VMware Virtual Ethernet Adapter for VMnet8
   Physische Adresse . . . . . . . . : 00-50-56-C0-00-08
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja
   Verbindungslokale IPv6-Adresse  . : fe80::6409:fe63:7a5a:7cae%6(Bevorzugt)
   IPv4-Adresse  . . . . . . . . . . : 192.168.117.1(Bevorzugt)
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Lease erhalten. . . . . . . . . . : Donnerstag, 21. Juli 2022 14:20:59
   Lease läuft ab. . . . . . . . . . : Montag, 1. August 2022 10:16:09
   Standardgateway . . . . . . . . . :
   DHCP-Server . . . . . . . . . . . : 192.168.117.254
   DHCPv6-IAID . . . . . . . . . . . : 1207980118
   DHCPv6-Client-DUID. . . . . . . . : 00-01-00-01-21-44-1F-09-80-EE-73-B0-27-82
   DNS-Server  . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   Primärer WINS-Server. . . . . . . : 192.168.117.2
   NetBIOS über TCP/IP . . . . . . . : Aktiviert

Ethernet-Adapter Ethernet 4:

   Medienstatus. . . . . . . . . . . : Medium getrennt
   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : SVN Adapter V1.0
   Physische Adresse . . . . . . . . : 00-FF-F3-C9-05-A2
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja
   
C:\WINDOWS\system32>netstat -r
===========================================================================
Schnittstellenliste
 23...5e 61 b1 14 51 6a ......VPN Client Adapter - VPN10
 39...80 ee 73 b0 27 82 ......Intel(R) Ethernet Connection (3) I218-LM
 33...4c bb 58 8f 2c d2 ......Realtek RTL8188EE Wireless LAN 802.11n PCI-E NIC
  9...4e bb 58 8f 2c d2 ......Microsoft Wi-Fi Direct Virtual Adapter
 27...4c bb 58 8f 2c d2 ......Microsoft Wi-Fi Direct Virtual Adapter #3
 20...00 50 56 c0 00 01 ......VMware Virtual Ethernet Adapter for VMnet1
  6...00 50 56 c0 00 08 ......VMware Virtual Ethernet Adapter for VMnet8
 40...00 ff f3 c9 05 a2 ......SVN Adapter V1.0
  1...........................Software Loopback Interface 1
===========================================================================

IPv4-Routentabelle
===========================================================================
Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          1.2.3.4  255.255.255.255   Auf Verbindung           1.2.3.4    281
          1.2.3.5  255.255.255.255          1.2.3.5          1.2.3.4     26
          1.2.3.6  255.255.255.255          1.2.3.6          1.2.3.4     26
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
    192.168.117.0    255.255.255.0   Auf Verbindung     192.168.117.1    291
    192.168.117.1  255.255.255.255   Auf Verbindung     192.168.117.1    291
  192.168.117.255  255.255.255.255   Auf Verbindung     192.168.117.1    291
    192.168.158.0    255.255.255.0   Auf Verbindung     192.168.158.1    291
    192.168.158.1  255.255.255.255   Auf Verbindung     192.168.158.1    291
  192.168.158.255  255.255.255.255   Auf Verbindung     192.168.158.1    291
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung     192.168.158.1    291
        224.0.0.0        240.0.0.0   Auf Verbindung     192.168.117.1    291
        224.0.0.0        240.0.0.0   Auf Verbindung           1.2.3.4    281
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung     192.168.158.1    291
  255.255.255.255  255.255.255.255   Auf Verbindung     192.168.117.1    291
  255.255.255.255  255.255.255.255   Auf Verbindung           1.2.3.4    281
===========================================================================
Ständige Routen:
  Keine

IPv6-Routentabelle
===========================================================================
Aktive Routen:
 If Metrik Netzwerkziel             Gateway
  1    331 ::1/128                  Auf Verbindung
 20    291 fe80::/64                Auf Verbindung
  6    291 fe80::/64                Auf Verbindung
  6    291 fe80::6409:fe63:7a5a:7cae/128
                                    Auf Verbindung
 20    291 fe80::a4c8:91ff:1fcb:8a9d/128
                                    Auf Verbindung
  1    331 ff00::/8                 Auf Verbindung
 20    291 ff00::/8                 Auf Verbindung
  6    291 ff00::/8                 Auf Verbindung
===========================================================================
Ständige Routen:
  Keine

C:\WINDOWS\system32>arp -a

Schnittstelle: 192.168.117.1 --- 0x6
  Internetadresse       Physische Adresse     Typ
  1.2.3.5               3e-98-b9-51-81-0f     statisch
  1.2.3.6               1c-1b-0d-00-eb-a5     statisch
  192.168.117.254       00-50-56-f2-7d-88     dynamisch
  192.168.117.255       ff-ff-ff-ff-ff-ff     statisch
  224.0.0.22            01-00-5e-00-00-16     statisch
  224.0.0.251           01-00-5e-00-00-fb     statisch
  224.0.0.252           01-00-5e-00-00-fc     statisch
  239.255.255.250       01-00-5e-7f-ff-fa     statisch
  255.255.255.255       ff-ff-ff-ff-ff-ff     statisch

Schnittstelle: 192.168.158.1 --- 0x14
  Internetadresse       Physische Adresse     Typ
  192.168.158.254       00-50-56-ff-42-78     dynamisch
  192.168.158.255       ff-ff-ff-ff-ff-ff     statisch
  224.0.0.22            01-00-5e-00-00-16     statisch
  224.0.0.251           01-00-5e-00-00-fb     statisch
  224.0.0.252           01-00-5e-00-00-fc     statisch
  239.255.255.250       01-00-5e-7f-ff-fa     statisch
  255.255.255.255       ff-ff-ff-ff-ff-ff     statisch

Schnittstelle: 1.2.3.4 --- 0x27
  Internetadresse       Physische Adresse     Typ
  1.2.3.6               1c-1b-0d-00-eb-a5     dynamisch
  172.17.200.10         de-ad-de-ad-de-ad     statisch
  224.0.0.22            01-00-5e-00-00-16     statisch
  224.0.0.251           01-00-5e-00-00-fb     statisch
  224.0.0.252           01-00-5e-00-00-fc     statisch
  239.255.102.18        01-00-5e-7f-66-12     statisch
  239.255.255.250       01-00-5e-7f-ff-fa     statisch
  255.255.255.255       ff-ff-ff-ff-ff-ff     statisch

PS C:\WINDOWS\system32> Get-Service RemoteAccess

Status   Name               DisplayName
------   ----               -----------
Stopped  RemoteAccess       Routing und RAS

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Mon Aug 01, 2022 11:04 am

Thank you, sorry but we need a few more logs:

Linux SE Server:
vpncmd localhost:5555 /server /password:*** /cmd BridgeDeviceList

Windows SE Bridge:
vpncmd localhost:port /server /password:*** /cmd BridgeList
vpncmd localhost:port /server /password:*** /cmd BridgeDeviceList

Also, tell us more about "the traffic comes from a traffic generator". This Linux source "generates" for a Windows destination, or vice-versa, so what are the IP addresses you use for it?

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Mon Aug 01, 2022 2:10 pm

Additional Linux:

Code: Select all

root@ares:~/vpnserver# ./vpncmd localhost:5555 /server /password:*** /cmd BridgeDeviceList
vpncmd command - SoftEther VPN Command Line Management Utility
SoftEther VPN Command Line Management Utility (vpncmd command)
Version 4.38 Build 9760   (English)
Compiled 2021/08/17 22:32:49 by buildsan at crosswin
Copyright (c) SoftEther VPN Project. All Rights Reserved.

Connection has been established with VPN Server "localhost" (port 5555).

You have administrator privileges for the entire VPN Server.

VPN Server>BridgeDeviceList
BridgeDeviceList command - Get List of Network Adapters Usable as Local Bridge
br-a7a196f16e8d
docker0
enp3s0f0
enp3s0f1
enp6s0
enp7s0
vboxnet0
veth0599c07
veth765279a
veth7f3a03d
veth91cfbac
veth9b8f550
veth9f9f862
vetha4823de
vethb0d058e
vethb42b3a5
vethd1addb3
vethdd17125
The command completed successfully.
Additional Windows:

Code: Select all

C:\WINDOWS\system32>vpncmd localhost /server /password:*** /cmd BridgeList
vpncmd command - SoftEther VPN Command Line Management Utility
SoftEther VPN Command Line Management Utility (vpncmd command)
Version 4.39 Build 9772   (English)
Compiled 2022/04/26 18:00:50 by buildsan at crosswin
Copyright (c) SoftEther VPN Project. All Rights Reserved.

Connection has been established with VPN Server "localhost" (port 443).

You have administrator privileges for the entire VPN Server.

VPN Server>BridgeList
BridgeList command - Get List of Local Bridge Connection
Number|Virtual Hub Name|Network Adapter or Tap Device Name                      |Status
------+----------------+--------------------------------------------------------+---------
1     |BRIDGE          |Intel(R) I211 Gigabit Network Connection (ID=2174020970)|Operating
The command completed successfully.


C:\WINDOWS\system32>vpncmd localhost /server /password:*** /cmd BridgeDeviceList
vpncmd command - SoftEther VPN Command Line Management Utility
SoftEther VPN Command Line Management Utility (vpncmd command)
Version 4.39 Build 9772   (English)
Compiled 2022/04/26 18:00:50 by buildsan at crosswin
Copyright (c) SoftEther VPN Project. All Rights Reserved.

Connection has been established with VPN Server "localhost" (port 443).

You have administrator privileges for the entire VPN Server.

VPN Server>BridgeDeviceList
BridgeDeviceList command - Get List of Network Adapters Usable as Local Bridge
Intel(R) Ethernet Connection (3) I218-LM (ID=3065593586)
Intel(R) I211 Gigabit Network Connection (ID=2174020970)
SVN Adapter V1.0 (ID=2616286990)
VMware Virtual Ethernet Adapter (ID=2964995231)
VMware Virtual Ethernet Adapter (2) (ID=2952033193)
The command completed successfully.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Mon Aug 01, 2022 2:17 pm

Concerning the generated traffic, it comes from a specialized device which can generate any kind of network packets, the IP addresses (if there is even L3) can be assumed to be arbitrary. This device sends the traffic to one of the bridged interfaces, in this case we sent it to the bridged interface of the Linux device.

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Mon Aug 01, 2022 3:21 pm

pkaf wrote:
Mon Aug 01, 2022 2:17 pm
...This device sends the traffic to one of the bridged interfaces, in this case we sent it to the bridged interface of the Linux device.
The bridged Linux NIC is enp3s0f0 which has no IPv4 assigned. The corresponding Windows bridge is "Intel(R) I211 Gigabit..." and has no IP at all, and is not even listed in ipconfig (physically disconnected maybe). Then no wonder that the specialized device can manage to send merely ~150 packets from nothing to nowhere over nonexistent bridge.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Mon Aug 01, 2022 3:50 pm

Odd that it's missing in "ipconfig" but it's true that the bridged interfaces which are in promiscuous mode anyway have no IP. They don't have to talk to anyone and the traffic should be L2 transparent anyway was my thinking.

Will have a more detailed look tomorrow.

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Mon Aug 01, 2022 3:51 pm

Oh, and sending all the 1500 to the bridged IP-less interface works without any problems, it's sending out of the decrypted traffic where the problem occurs.

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Tue Aug 02, 2022 1:46 am

While waiting for the third reconfiguration of your network experiment, I propose a diagnostic modification. Do assign IPs to the bridged interfaces and when cross-ping is successful then you could resume stress-testing with tcpreplay, or the like. What do you say?

pkaf
Posts: 14
Joined: Wed Jul 20, 2022 8:50 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by pkaf » Tue Aug 09, 2022 8:12 am

We do not assign IPs for the exact reason that we want it to be totally transparent.

And this seems to be where the problem is, since there are no L2 (ARP) responses something in the network stack is apparently stalling, no matter that the interface is in promiscious mode.

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: Packets stop being forwarded after some time with cascade setup with local bridging

Post by solo » Tue Aug 09, 2022 9:16 am

Again: "The corresponding Windows bridge is "Intel(R) I211 Gigabit..." ...and is not even listed in ipconfig", why?

Post Reply