Bridging Ethernet to ZeroTier Virtual Networks on Linux

Follow

Comments

25 comments

  • Avatar
    modulusx

    Thanks for the write-up!

    Sadly I must be doing something incorrectly. Should zt0 receive an IP?

    Can you provide the output of ifconfig and route on your bridge after its all configured and working properly?

  • Avatar
    dafyre

    What IP address range should the remote ZT Clients bet getting? Should they be on the same subnet as the network behind the ZT Gateway?

  • Avatar
    Adam Ierymenko

    @dafyre IP devices on a LAN that share an address space must know that other devices share that same address space in order to speak.


    So if you have devices on a LAN in 10.0.1.0/24 and 10.0.2.0/24, 1.0.1.0/24 devices have to have a special route that says that 10.0.2.0/24 devices are also "local" -- as in on the same LAN -- otherwise they will try to send 10.0.2.0/24 traffic to their default gateway.


    Of course the simplest solution is to put everyone on the same LAN in the same IP range. That way everyone knows where everyone else is located and everyone has the same routing table.

  • Avatar
    dafyre

    Thanks for the pointers...

    So I could do like a 10.0.0.0/23 and have my LAN devices be on 10.0.0.1-254, and my ZT Devices be 10.0.1.1-254 ?


    Edit: I don't think I ever got the email notification about this.

  • Avatar
    dafyre

    I have gone back and built out my home network where everything is a /23.... 192.168.10.0/23...


    My LAN DHCP hands out IP addresses in 192.168.10.100 - 192.168.10.200.

    My ZT IP range is configure for 192.168.11.100 - 192.168.11.200...


    (before setting up bridging)

    ZTBridge eth0 (LAN): 192.168.10.100/23

    ZTBridge eth1 (br0): No IP

    ZTBridge zt0 : 192.168.11.119/23


    OffSite Host zt1: 192.168.11.143


    ======================


    Before activating the bridge settings via /etc/network/interfaces,

    I am unable to ping 192.168.11.119 from the off-site host.

    I am also unable to ping 192.168.11.143 from the ZTBridge host.


    After Activating the br0 interface via /etc/network/interfaces..

    I am unable to ping 192.168.11.119 from the off-site host.

    I am unable to ping 192.168.10.112 (windows box) from the off-site host.

    I am unable to ping 192.168.11.143 from any LAN device.


    My internet currently works fine for all LAN devices, but I am unable to ping any device that is connected on the ZT Segment... I can ping my Windows box from the LAN.


    Any ideas?

  • Avatar
    Adam Ierymenko

    @dafyre When you activate br0, the IP should now be assigned to br0. As far as I know Linux bridging "enslaves" the member interfaces and they can no longer have IPs independent of the new bridge device. If you didn't do that, that could be the problem.

  • Avatar
    dafyre

    I'm also checking through the configuration on the system... In order to allow bridging from your web interface , all I need to do is check the Allowed & Bridging box on the device that I want to actually do the bridging, right?

  • Avatar
    Adam Ierymenko

    @dafyre Yes. I do think something's up here with Linux bridging. Another thing I see is that on the bridge box both zt0 and eth0 are in 192.168.10.0/23 -- which is going to cause confusion. Which interface should an outbound packet use? Instead, br0 should have a single IP (maybe 192.168.10.100?) so that packets enter and leave via br0.


    Bridging is confusing without network virtualization. There are also sysctl settings around things like whether bridged packets traverse iptables and these can have an effect.

  • Avatar
    dafyre

    @adam.ierymenko said:



    @dafyre Yes. I do think something's up here with Linux bridging. Another thing I see is that on the bridge box both zt0 and eth0 are in 192.168.10.0/23 -- which is going to cause confusion. Which interface should an outbound packet use? Instead, br0 should have a single IP (maybe 192.168.10.100?) so that packets enter and leave via br0.


    Bridging is confusing without network virtualization. There are also sysctl settings around things like whether bridged packets traverse iptables and these can have an effect.



    I was actually following the guide above, and he had eth1 and zt0 as the bridge, with eth0 having an LAN... but I bet his management LAN is different than the normal LAN.


    Yet another tip! 8-)


    I just got my system wiped and reloaded... ready to go at it again. :-)


    I'll post back later with any notes I can come up with.

  • Avatar
    dafyre

    So... I'm getting closer.


    I can ping to the LAN and the Remote IP addresses from my bridge now (configured as 192.168.10.100/23)...


    I can also ping the 192.168.10.100 from my remote ZT client (192.168.11.143)... So I've got the remote ZT client and the bridge talking... I can't, however, seem to get past the bridge to any of my internal devices.


    I have enabled ip_forwarding, on the off chance that could have been the issue, but that doesn't seem to be it either.


    Any other ideas?

  • Avatar
    dafyre

    Hey Adam,


    I think I have it working now. The problem in mysituation was Hyper-V not allowing a VM to communicate on the Network using a Mac Address different than the one assigned to it by Hyper-V....


    The Fix from PowerShell (typed on one line)... This is done per VM.


    get-vmnetworkadapter -VMName MYVMNAME|where {$_.SwitchName -eq "MY_HYPERV_SWITCH"}|
    
    set-vmnetworkadapter -MacAddressSpoofing on

    Edit: In VMware, you will need to enable Forged Transmits and Promiscuous Mode on the VM that you run things like this on. I don't have access to a VMware system to chek this.

  • Avatar
    Adam Ierymenko

    @dafyre Ahh yes! Forgot to mention this. Many hypervisors have a setting to allow or not allow bridging for security reasons, since bridging means allowing promiscuous Ethernet I/O. Normally you don't want VM hosts to be able to spoof MACs for security.

  • Avatar
    Adam Ierymenko

    @dafyre This is one reason we have a setting for allowing bridging in ZT -- it allows MAC spoofing (though not of ZT-domain MACs). The other reason is that bridges must receive multicast traffic more liberally.

  • Avatar
    dafyre

    @adam.ierymenko said:



    @dafyre Ahh yes! Forgot to mention this. Many hypervisors have a setting to allow or not allow bridging for security reasons, since bridging means allowing promiscuous Ethernet I/O. Normally you don't want VM hosts to be able to spoof MACs for security.



    The only thing that made me think about is is that when I originally tried to set up a VDI deployment, the guys I work with wanted me to do it by nesting Hyper-V inside of VMware... I ran into that problem there too, and they didn'tw ant to allow me to spoof MACs, so I got some real hardware for it, lol.

  • Avatar
    broemi

    @modulusx I have the same problem... When I put the configuration for the bridge in /etc/network/interfaces zt0 don't receive an IP when the Ethernet Bridge checkmark is on.

    Can anyone help me please?

  • Avatar
    dafyre

    @broemi said in Bridging Ethernet to ZeroTier Virtual Networks on Linux:



    @modulusx I have the same problem... When I put the configuration for the bridge in /etc/network/interfaces zt0 don't receive an IP when the Ethernet Bridge checkmark is on.

    Can anyone help me please?



    The way I got around this was simply to add a second NIC to my VM... the zt0/eth0 bridge does not get an IP address... eth1 is my management interface and gets an IP address in my LAN.

  • Avatar
    fkooman

    There is an additional problem with bridgin mode on Linux. I managed to get it work according to the above instructions, however, it does not work on boot.


    It seems zt0 is not available by the time the bridge is created on Debian (defined in /etc/network/interfaces).


    I need to do a ifdown br0; ifup br0 once ZT is online to make it work.


    Is there a way to make this work @ boot without requiring manually to stop/start the bridge?

  • Avatar
    Adam Ierymenko

    There's a service for Linux that can run scripts when network devices come up/down. Maybe that would help. I think it's called ifplugd but I'm not sure.

  • Avatar
    fkooman

    I was looking how this works with OpenVPN in bridge mode, there is some more documentation available on that. It seems OpenVPN calls a script when OpenVPN starts that adds it tap interface to the bridge. I guess something similar could be done with ZT? Another option would be, probably a clear thing, I think, to have a persistent tun/tap interface that is then used by ZT on boot, so it already exists when the network interfaces come up and can be added to the bridge and ZT will use that tun/tap device when it starts up.


    I was playing with /etc/network/interfaces with bridge_waitport and bridge_maxwait, but that doesn't help. It seems that ifplugd (as documented here: https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_ifplugd_package) is only meant for physical devices (and is also considered legacy).


    Another solution may be to run ZT in 'preup' from interfaces file, which is also kinda ugly.


    It is strange that it is not easy to find more documentation on only starting a bridge when all its ports become available without resulting in hacking around...


    OpenVPN: https://help.ubuntu.com/community/OpenVPN

  • Avatar
    undefined

    A mechanism to spawn a shell script on zerotier interface up/down actions would be great. Even better if that script was able to configure the interface (e.g. set IP).


    Say a script that took parameters of up/down, network ID, network name, interface name (e.g. zt0), IP address, netmask. If it's present, zerotier wouldn't configure the interface at all -- it'd be left to the script. This would be enough to allow users to integrate zerotier with whatever wacky system they wanted, even use ifrename.

  • Avatar
    richardc

    I am pretty sure that Zertotier doesn't attach to a bridge on boot because it's systemd config say to start the Zerotier daemon after the network is up, i.e. after the bridge is configured.


    I tried messing with the systemd configuration order but gave up. Instead to make the ZeroTier interface attach to the bridge on boot I use 'allow-hotplug' in /etc/network/interfaces. That way the interface is configured when the Zerotier interface comes up, even if it is much later - apparently this functionality is to support USB netwrok interfaces and the like. Then I use 'up' and 'down' options to add and remove the Zerotier interface from the bridge. Relevant parts of my /etc/network/interfaces:


    auto br0

    iface br0 inet static

    bridge_ports none

    address 10.8.0.9

    netmask 255.255.255.0


    auto zt0

    allow-hotplug zt0

    iface zt0 inet manual

    up brctl addif br0 zt0

    down brctl delif br0 zt0

  • Avatar
    dafyre

    I actually had worked around this by using a script that does the same thing with crontab & @reboot.

  • Avatar
    naterr

    Trying to wrap my head around this,

    Do you think it would be possible to add zt0 to a OVS bridge on ubuntu 16.04 on boot?

    I would actually like 2 - ZT interfaces, one management to the host, and the other tied into the OVS bridge that my KVM vms would access.

  • Avatar
    wlaws

    @adam.ierymenko I use this to solve this


    /etc/rc.local


    systemctl restart zerotier-one.service

    #this sleep can be shorter but i like to be safe

    sleep 60

    brctl addif br0 zt0

    exit 0

  • Avatar
    Tamil Vanan

    Nice post, if i have only single physical NIC, What are the changes i suppose to make.Thanks in advance

     

Please sign in to leave a comment.