General System Design
How does ZeroTier work?
ZeroTier endpoint nodes form a peer to peer network and use a set of pre-configured nodes called root servers (currently run by us, federation is planned) as stable anchor points for near-instantaneous zero-configuration peer location and connection setup. In older documentation we called these supernodes, but we started calling them roots because their role is more limited than the "super" term implies and is closer to the DNS root name servers. They run the same open source code as regular ZeroTier nodes. They're just "blessed" as such and run at big data centers with fast static Internet connections.
We call this peer to peer network virtual layer one (VL1). In the OSI model of network architecture this is the "wire." Since ZeroTier is about network virtualization, its wires are imaginary ones made of math (cryptography and addressing) instead of physical cables or radio links.
Over this virtual wire the ZeroTier software creates VL2: a virtual Ethernet layer similar to VXLAN. VL2 LANs are what your devices, containers, VMs, and apps see as a "network." Since VL2 looks and behaves just like Ethernet, all the standard protocols we use every day (IPv4, IPv6, TCP, etc.) work just fine. ZeroTier enables peer to peer communication over flat networks with standard network protocols.
ZeroTier's 10-digit device addresses are VL1 addresses. The 16-digit hex numbers that identify networks are VL2 network IDs. Network IDs are longer than device addresses because they contain them: for each network there exists a primary network controller responsible for issuing network configuration and membership certificates. The network 8056c2e21c000001 that we suggested for testing on the home page is controlled by 8056c2e21c, a network controller we operate.
There are 1,099,511,627,776 (2^40) possible VL1 addresses. A single VL1 device could control up to 16,777,216 networks (2^24 from the extra 6 hex digits in a network ID) and can join an unlimited number of them. ZeroTier can provide network virtualization for about 125 devices for every human being alive today, and with a protocol extension this could in theory be extended up to 2^256. Addresses are computed from a 256-bit public key that serves as the true unique identifier of a device.
We chose a 40-bit identifier for devices because it can be used to compute unique 48-bit MAC addresses on virtual Ethernet networks. It's also quite nice that a 40-bit device ID and a 64-bit network ID can fit into a 128-bit IPv6 address with room to spare. The space of all ZeroTier devices and virtual networks in the world can be mapped back to IPv6.
Once the entire Internet is V6, this provides an elegant disruption-free path from the complexity of VL1 back down to the simplicity of straight IP. We have many long-term ideas that revolve around this.
What network ports does it use?
By default ZeroTier uses UDP port 9993, though remote peers often appear on different ports due to NAT at their end.
If UDP is impossible ZeroTier will also try outbound TCP to port 443 (https) as a fallback. We provide last-resort TCP relays as a free service so that ZeroTier will work almost anywhere, but if you can fix your network to eliminate the need for it you'll see much better performance. TCP relay fallback is the slowest mode of operation.
How does traffic flow on the ZeroTier network?
Whenever possible, traffic flows directly from device to device. ZeroTier uses a mechanism called transport triggered link provisioning to establish direct links, as well as techniques like UDP hole punching to make this work behind standard NAT devices. These are similar to the techniques used by voice over IP phones, video chat applications, and other software that makes use of direct peer to peer networking.
If direct connectivity is impossible, ZeroTier provides free traffic relaying. This ensures almost universal service availability, though performance becomes poor if direct connectivity can't be established. (See troubleshooting section for steps that can be taken to ensure that direct connectivity is working.)
How reliable is ZeroTier? What happens if its servers go down?
ZeroTier's root servers are distributed across three cloud providers in North America, South America, Europe, Africa, Asia (and Japan), and Australia. Only one must be reachable for ZeroTier to set up new links. If all root servers went offline at once, devices that are already communicating would continue to do so but new connections would not be able to be formed until at least one root server became reachable. In the future we plan to offer some form of root server federation so you can further augment this infrastructure with your own.
Network controllers are fault tolerant. If a controller goes down, existing members of the network can still communicate. It's simply not possible to authorize new members, de-authorize existing members, or change network configuration until the controller is back online. Since controllers have VL1 addresses that are mobile, fast fail-over is trivial to implement.
Most of the connectivity problems we see in the field revolve around instabilities, bugs, and other issues that are close to the endpoints and typically involve broken or peer-to-peer hostile NAT gateways. Fixing these problems is an ongoing project. It's one of the things we do.
How is ZeroTier licensed?
ZeroTier is open source and is licensed under the GNU GPL v3. It's free to use and free to include with other open source software. If you want to embed it in a proprietary product, contact us for commercial licensing.
Some of our services such as our hosted network controller are paid or "freemium." Our business model revolves around SaaS and enterprise services and support.
Do I need an account to use it?
No. You only need an account on our site if you want to use our hosted network controllers or other services.
How are device IDs assigned?
Every ZeroTier endpoint device possesses a unique public/private key pair that is generated the first time the device is initialized. From the public portion of this key pair, a 40-bit/10-digit number is derived via a proof-of-work function. Device addresses are then registered with our root servers, where they are allocated on a first-claim basis. There are 1,099,511,627,776 possible addresses so collisions are incredibly unlikely, and if one occurs your device will simply repeat the process by generating a new key pair.
How are network IDs assigned?
A network ID is actually the device ID of the network's primary controller plus a 24-bit/6-digit network number on that controller. For example, 8056c2e21c000001 refers to network 000001on controller 8056c2e21c.
How many networks can I join?
The limit depends on your operating system. On Mac the limit is 32. Linux, FreeBSD, and Windows have no defined limit, though we doubt the ability of the Windows networking stack to handle many more than 8-16 networks.
When I join networks does all my traffic go through them?
No, only traffic to/from the IP address range of the network. We do plan to add a feature for gateways on networks in the future, since some users want this behavior.
ZeroTier can't connect. What's wrong?
Assuming your Internet connection is up, this usually means that you're behind a very restrictive firewall.
Sometimes that's a firewall at your network boundary. If you have access to it, check there. Otherwise there might be nothing you can do outside of contacting your IT department or network support.
If you have administration access to your computer, local firewalls can be changed. ZeroTier takes care of opening the default Windows firewall, but you may have to manually configure third party firewall products. It works behind the Mac firewall, but some users have reported improved performance by adding a rule to permit traffic to/from UDP port 9993. Linux firewall configuration varies by distribution, so refer to your distribution's documentation. Adding a rule permitting UDP traffic to/from local port 9993 is usually easy.
It says I'm on a virtual network, but I can't reach anything.
Check your local firewall. Virtual network ports are just like real ones, so local firewall rules apply.
ZeroTier seems slow. Is there anything I can do?
The first thing to check is your connection status. If it says 'TUNNELED', you are using TCP fallback mode. This is the slowest mode of operation, and is available only to ensure operation in restrictive network environments. If you have access to your router or firewall, add a rule to allow UDP traffic to/from port 9993. Also check to make sure your local system firewall is not blocking ZeroTier's UDP traffic.
Even if UDP traffic is allowed, some users are behind "symmetric" NAT devices. These make direct connectivity difficult, forcing the use of relaying. If you have access to your NAT/firewall router device, see if it has a setting for NAT type. If it does, select "normal" or "full cone." This will also improve the performance of SIP-based VoIP phones, video chat applications, and games. About 90-95% of NATs are "full cone," but symmetric NATs are common in some corporate and university environments.
I get PORT_ERROR when joining networks.
This indicates a problem with the virtual network port driver. On Macs this can occur if the software is not installed properly (tap.kext is not available) or if you attempt to join more than 32 networks. Windows users may also see this error if there is an installation problem. Linux users might see if if the 'tun/tap' kernel module is not available on the system. Try reinstalling the software or (for Linux) check to ensure that kernel modules matching your running kernel version are available.
I'm on two networks with the same IP address scheme. How can I make this work?
In the future we plan to add an automatic IP address remapping feature to handle this case. In the meantime, the best option is to change the IP address scheme of one network (if you control it) or delve into your system's network filtering support (e.g. 'iptables' on Linux) and find a way to accomplish address remapping yourself.
How is my traffic protected?
All traffic is encrypted and authenticated end-to-end. This means that private keys that are kept only on your device are used to negotiate shared keys and encrypt content. Nobody else along the wire can obtain these keys without breaking into your or the other party's systems.
What encryption algorithms are used?
ZeroTier currently uses 256-bit Curve25519 elliptic curve Diffie-Hellman for shared key agreement and Ed25519 for elliptic curve signatures. 256-bit Salsa20 with Poly1305 authentication is used to encrypt traffic in transit. The construction and use of these algorithms is identical to the well-regarded NaCl cryptographic library.
How hard is it to impersonate a device?
Traffic is both authenticated and encrypted, so "man in the middle" attacks aren't possible. Counterfeiting an identity for an existing address is extremely difficult.
Addresses are derived from public/private key pairs via a proof of work function, requiring about one second per new address generation. To find an intentional collision would require about 2^39 trials, or about 17,000 years on a single fast CPU core. This is not beyond the reach of someone with a large supercomputing cluster (or botnet) or a lot of money to rent one, but it provides a first line of defense against casual attempts.
Root servers provide a second line of defense, caching address mappings to public/private key pairs. After spending considerable computing power to generate a collision, an attacker would have to then compromise all root servers to force them to serve its "impostor" identity for an already-claimed address. Clients cache identities too, so even after all this effort the impostor might still be unable to communicate.
How is network access controlled?
Access to a private network is governed by certificates issued and signed by its network controller. When peers wish to communicate, they "push" these certificates to each other. Peers will only communicate if their certificates agree. Certificates are re-issued periodically and on demand, so de-authorizing a device is accomplished by simply issuing new certificates to all the others. The de-authorized device then "falls over the horizon" and is unable to communicate. (This typically takes 30-60 seconds.)
Certificates are used instead of access control lists to permit huge networks. An access control list for a network with millions of devices would become unmanageably large.
If I join a network, can people use my Internet connection or access other networks I'm connected to?
Normally, no. This would only be possible if you've configured your system to forward traffic. As far as we know no operating systems ship this way, so it's something you'd have to explicitly set up.
Does ZeroTier provide anonymity?
ZeroTier encrypts all your traffic, but it does not hide the underlying location or IP address of your computer. If you want anonymity online we recommend using Tor.
Does ZeroTier track me?
Through our root servers we are able to see the IP addresses of running devices. This information is kept private and will not be shared with anyone without a court order compelling us to do so. Since all traffic is encrypted, we can't see much else. Our business model is built on freemium services and (in the future) enterprise management products, not user tracking.
Does ZeroTier support perfect forward secrecy (PFS)?
No, and this feature has been shelved for now as potentially out of scope. We decided to avoid PFS because the primary goal of ZeroTier is reliable connectivity and adding PFS requires adding shared state that must be negotiated. A deep discussion occurred in our GitHub issues section on this subject and the rationale is explained in some detail there.
Has ZeroTier's protocol or code base been professionally audited?
Not yet, though an informal audit has been performed by our security partner PKC Security. This is also planned for the near future (6-12 months). We plan to commission two audits: one thorough code and protocol audit from PKC Security, followed by a second from a well-regarded firm or consultant with whom we have no pre-existing relationship.
ZeroTier was released as open source in 2013 and is used by thousands of users globally. So far no serious security bugs (such as remote vulnerabilities or attacks to compromise encrypted traffic) have been reported.
Is ZeroTier FIPS compliant?
No. We only plan to support FIPS modes if there is significant customer demand.
How does ZeroTier manage its own infrastructure security?
Access to our servers is exclusively via two-factor authentication using a physical access token with a hardware-enforced PIN. All root servers and network controllers are independent of one another, so compromise of one would not immediately compromise others. Secret key information for our infrastructure is stored ephemerally or on single-tenant bare metal servers located in physically secured data centers.
What happens if ZeroTier's servers are compromised?
Consequences depend on the server's role. Despite their importance, root servers are actually less critical from a security point of view. The compromise of a root server would only permit denial of service, and could be mitigated by simply destroying the compromised server and replacing it. Network controller compromise would allow an attacker to gain access to the customer networks it manages. As a result, we do not recommend relying on the virtual network perimeter alone to secure your systems. (Relying on a physical network perimeter alone is also a bad idea. We recommend defense in depth.)
ZeroTier desktop clients for Macintosh and Windows auto-update. How is this secured?
While all our Windows and Mac binaries are signed by appropriate certificate authorities, we do not trust this mechanism exclusively to protect software updates. In addition to ordinary code signing we also include our own set of 256-bit elliptic curve signing keys. These are used to validate the download prior to installation. The private portions of these keys are kept on encrypted storage secured by physical access tokens.