I have started to test out ZeroTier as an alternative to traditional VPN for my home lab. I’m not unhappy about how VPN works with my Ubiquity USG setup, but I need something more intrinsic and independant of whereever my workloads would run and not depend on a connection to homelap or me connecting to it for access to these. In my opinion VPN is great for high performance site-to-site tunnels or for roadworkers out and about needing to connect to the “office/data center” i.e some aggregate of workloads in close proximity. They’re not well suited for scattered ephemeral workloads and I do not want to maintain scripting handeling a cumpersome setup processes for each new VM/contrainer or VPN site-to-site tunnels between my homelab, worklab and a couple of public cloud providers (those guys even charge for VPN connections). For my requirements a simple to setup and encrypted overlay solution is a far better option as I tend to spin up short lived workloads on everything between my own raspberry pi cluster, worklab and clouds like AWS and Google.
There are great VPN alternatives available like WireGuard or Tinc with powerfull features not found elswhere. Though they’re still more comparable to traditional VPN - plus they’re still more complex to setup than ZeroTier.

Anyway, let dig into ZeroTier!

What is ZeroTier?

ZeroTier is a virtual ethernet switch for global area networking built on the principle of de-perimeterisation. It’s a distributed network hypervisor built atop a cryptographically secured peer-to-peer network that spans any physical LAN/WAN boundary. It makes devices of any type and any location believe it is conneced to the same global switch. All traffic is end-to-end encrypted by the network who will try to always use the most direct path available for best latency and performance.

As for security I believe ZeroTier have that quite well covered as it uses one of the fastest elliptic curve cryptos for public key encryption, a trusted 256-bit stream cipher for end-to-end packet encryption and a strong message authentication code (MAC) as well. The recent inclution of WireGuard into the mainline Linux kernel (5.6) actually benefits ZeroTier as well. Both use the Poly1305 for MAC and the people behind WireGuard have been hard at work optimizing the code and made really great performance improvements.

The ZeroTier encrypted peer-to-peer operate in a DNS-like “world topology”. There is only one planet (Earth) and ZeroTier operate the top-level root servers. Users can add their own user-defiend root servers (moons) to improve latency or to build an on-premise solution that can keep the local ZeroTier network alive despite loosing internet connectivity. Nodes are the members/devices joining a network and they can “orbit” any number of moons (i.e. use them as root servers). In typical peer-to-peer fasion roots/moons handles only direct connection route discovery and initial peer relay. They will only relay packets untill peers know a direct route between them, but as all packets are end-to-end encrypted they can’t be read by anyone else.
In the upcomming ZeroTier 2.0 release the planet/moon terminology for root servers goes away.

As users there is only two types of cryptographic identifiers in ZeroTier we need to know of:

  • Node ID (a 10-digit hex address known by the user)
  • Network IDs (16-digit hex address known by the network owner or exixting members)

A Node ID address identifies a single device/member (i.e laptop, server, VM, phone) while a Network ID identifies the ZeroTier network whom devices/members can join.
Another way of thinking about it is that a Node IDs are switch-ports on a giant earth-sized virtual switch and network IDs are the VLANs these ports can be assigned.

Network controllers are ZeroTier nodes that act as access control, certificate authorities and configuration managers of the virtual ethernet layer ontop of the peer-to-peer network. It’s very common to mistake them for root servers, but as they’re are connection facilitators, network controllers are the managers of the virtual overlay. The first 10 digits of a Network ID is the global ZeroTier address of its controller. The rest of the ID is the unique network number of the controller. You can create networks with the controller hosted by ZeroTier for free - if you can keep under 100 devices per network (unlimited networks, though). If you want to roll your own network controller then Key Networks have made a pretty great web UI and improved the installaion process compared to what ZeroTier made available.

Unlike a typical VPN setup a ZeroTier network operate pretty much as an isolated virtual switch where every authorized device can communicate with each other. Configured as a Private network only an admin can authorize new members, but a in a Public network every new device member gets immediate access. Routes and switch flow rules can be configured per network. If a more “VPN-like experience” is needed a ZeroTier admin can configure a member to become a bridge device and via routes have straffic for specific IP ranges send to this device member. After configuring IP forwarding in the OS of the bridge it is possible to reach devices on a LAN outside the ZeroTier network. With some concurrent multipath and QoS support build into ZeroTier, you can even use it as a sort of “SD-WAN” solution by only joining linux software routers as devices and via dynamic routing securely connect office brances together over ZeroTier.

Something that is completely missing from ZeroTier is DNS lookup. IP adresses are assigned in a DHCP-like fasion, but there is no DNS feature in ZeroTier at the moment. In my lab I run DNSMASQ and created a script that goes through all my ZeroTier networks and updates my DNS server with all the members it finds. In a later blog article I’ll go through the scipt I created for this and other things for my ZeroTier usage.

Installing the client

ZeroTier have packages for almost everything of interest and they provide a universal install script you can run with just a one-liner: 

1
$ curl -s https://install.zerotier.com | sudo bash

If you’re not keen on just running the script without GPG verification they have you covered as well:

1
$ curl -s 'https://raw.githubusercontent.com/zerotier/ZeroTierOne/master/doc/contact%40zerotier.com.gpg' | gpg --import && if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi

My installation was painless thanks to the diffrent communities who made ZeroTier available as a Snap, via Homebrew for macOS and via Chocolatey for Windows. Only macOS and Windows have a desktop app with UI, and the rest have only zerotier-cli available. I did not use the ZeroTier one-liner install script as it only lets Linux machines handle future updates via a local packemanager (DEB/RPM). I wanted the same experience on all platforms, so that is why I went with Homebrew (already using it extensively) and Chocolatey.

Ansible
If you’re familiar with Ansible then Marcus Meurs have created a excellent role for ZeroTier you can install from Ansible Galaxy with just ansible-galaxy install m4rcu5nl.zerotier-one. Besides installing ZeroTier the role it can also join the new device to a specified Network ID (though not leaving a network).

Playbook example:

1
2
3
4
5
6
7
8
- hosts: zerotier-servers
    vars:
        zerotier_network_id: 9999c2e54cc6aa7b
        zerotier_accesstoken: "{{ vault_zerotier_accesstoken }}"
        zerotier_register_short_hostname: true

    roles:
        - { role: m4rcu5nl.zerotier, become: true }

Inventory example (YAML):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
servers:
  hosts:
    fileserver1.domain.com:
        zerotier_member_description: "Filerserver 1"
    fileserver2.domain.com:
        zerotier_member_description: "Filerserver 2"
        zerotier_member_ip_assignments='["192.168.244.10"]'
desktops:
    hosts:
        macos.desktop.com
        linux.desktop.com
        windows.desktop.com
      vars:
        zerotier_member_description "Just another desktop"

Ansible is almost straightforward to use and this role is quite easy to use as well. Though Ansible is not always the answer to all prayers as its build arround a push architecture and the expectation that you can reach the server via the LAN/internet and are allowed to ssh into the server. It just not all servers that can be reaced from your Ansible management node or that you are the admin. Personally I ended up writing a set of compresensive scripts for joining/authorizing members to a network. I’ll share more about these in another blogpost coming soon.

In later sections for joining or leaving a network I won’t cover it from a Ansible perspective as the ansible role can handle all that. Instead I’ll try to dig into the API for these tasks.

Docker
On Linux ZeroTier require access to the /dev/net/tun interface on the host to create the virtual zt* interface. This is a challenge because running it in a container will then require priveledged access to the host. For now I can only get ZeroTier to work in Docker by running it with a few runtime capabilities (SYS_ADMIN and NET_ADMIN) so it can access and modify the host network interfaces. It is still not full priveledged access, but close and it is definately not best practice… not at all…
I doubt I will be possible to circumvent this with Docker alone, so later in the future I’ll try to see if is possible mitigate the risc via a micro-vm based container solution like Firecracker / Ignite or Nautilus / Project pasific. It all depends on any of these micro-vm have the TUN interface exposed/impemented in their kernels.

To prevent a container from generating a new node ID (device member ID) at reach run, then mount /var/lib/zerotier-one from the Docker host for persistant conifg.

For container images I prefer Alpine over my usual choice of Ubuntu/Debian for everything not macOS. I really want this to be able to run on any CPU architecture you can thow at Docker (yes, even a Raspberry Pi), but Apline only have community packages for x86_64 and ppc. Thankfully ZeroTier have their own multi-platform debian reposotory, so combined with Docker buildx I will use multistage-build (for each platform) to install the correct binaries in a temporary Debian image and them copy them over to the final Alpine image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
FROM --platform=$BUILDPLATFORM debian:buster-slim AS buildstage

ENV ZEROTIER_VERSION=1.4.6
RUN apt-get update && apt-get install -y curl gnupg && \
    apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys 0x1657198823e52a61  && \
    echo "deb http://download.zerotier.com/debian/buster buster main" > /etc/apt/sources.list.d/zerotier.list && \
    apt-get update && apt-get install -y zerotier-one=$ZEROTIER_VERSION
COPY main.sh /var/lib/zerotier-one/main.sh

FROM alpine:latest
LABEL VERSION="1.4.6"
LABEL MAINTAINER="[email protected]"
LABEL DISCRIPTION="Containerized ZeroTier for Docker Linux hosts. NOTE: Needs to run with priviledged network access to work :("
##   docker run --name zerotier-one \
##              --device=/dev/net/tun --net=host --cap-add=NET_ADMIN --cap-add=SYS_ADMIN \
##              --volume=/var/lib/zerotier-one:/var/lib/zerotier-one kimtholstorf/zerotier

RUN apk add --update --no-cache libgcc libc6-compat libstdc++
RUN mkdir -p /var/lib/zerotier-one
COPY --from=buildstage /usr/sbin/zerotier-one /usr/sbin/zerotier-one
COPY --from=buildstage /usr/sbin/zerotier-cli /usr/sbin/zerotier-cli
COPY --from=buildstage /usr/sbin/zerotier-idtool /usr/sbin/zerotier-idtool
COPY --from=buildstage /var/lib/zerotier-one/main.sh /main.sh
RUN chmod 0755 /main.sh

EXPOSE 9993/udp

ENTRYPOINT ["/main.sh"]
CMD ["zerotier-one"]

This will soon be available on Docker Hub at kimtholstorf/docker-zerotier. Though be carefull untill there is a stable release.

Create a network

First you’ll need to create an account on my.zerotier.com. If you’re expecting to use the API for scripting then go to the Account page and click '+ Create Access Token'. The API documentation is available here.

There’s two kind of networks:

  • Public network can anyone join without authorization and get access.
  • Private networks can anyone join, but they need to be authorized by the network owner or delegated users to be able to communicate with other members.

WebIU

On the Networks page on my.zerotier.com click 'Create a Network'. A new private network with a unique Network ID and random generated name will be spun up. Under the network you can change name, description and advanced settings such as IP assignments, routes or share the network with other users and assign permissions. This is also here members will show up after they have joined and where indivudual member settings are configured.

API/CLI

To create a new private network just run the below curl command against the Zerotier API URL with your <API-ACCESS-TOKEN> and your desired <NETWORK-NAME>. Change the private value to false to create a public network. Just posting empty JSON data '{}' will create a new public network, but with a blank name field. 

1
$ curl -H "Authorization: Bearer <API-ACCESS-TOKEN>" -X POST -d '{"config":{"name":"<NETWORK-NAME>","private":true}}' https://my.zerotier.com/api/network

Join a network

WebIU

To “join” a client to a network from my.zerotier.com you need to know the 10-hex-digit Node ID of the client. Windows and macOS users can see this ID in the ZeroTier-One desktop app. Linux users or cli advocates can get their ID via the zerotier-cli commandline tool (also available on Windows and macOS).

1
$ zerotier-cli info | cut -d " " -f 3

In the settings section under each network (nearly at the botton) there’s a 'Manually Add Member' field. And, yes there’s actually 2 fields to add a member but they do the same. Put in the Node ID of the member and click 'Submit / Add New Member'.
NOTE: This wil not actually join the member but will pre-authorize it so that when the member joins they’re already authorized and able to use the network.
It’s also possible to send a user a e-mail invitation with join instructions and it makes sense to manually add (i.e pre-authorize) the member, so when they follow the join instructions they’re already authendicated.

API/CLI

To actually join a network from a client you need to know the 16-hex-digit Network ID of the network. If you been invited via e-mail the Network ID is included along with join instructions. For existing members or admins the ID is available from the ZeroTier WebUI under Networks or via the zerotier-cli commandline tool.

1
$ zerotier-cli listnetworks

To join run:

1
$ zerotier-cli join <NETWORK-ID>

If you’re an admin you can authorize a member via this curl command:

1
$ curl -H "Authorization: Bearer <API-ACCESS-TOKEN>" -X POST -d '{"name":"short-name","description":"long-description","config":{"authorized":true}}' https://my.zerotier.com/api/network/<NETWORK-ID>/member/<NODE-ID>

This only works if the member is already joined. In the example above I also configure a name and description for the new member. Just posting '{"config":{"authorized":true}}' as JSON data will just authorize a member.

In a later post I’ll share a script I wrote that’ll join a member, then wait for the succes responce and finally authorize.

Leave a network

A network member can via the zerotier-cli commandline tool or desktop UI leave a network of their own volition.

If you’re a network owner/admin you have three options to remove access for a member:

  • Deauthorize a member
  • Hide a member
  • Ban a member

Any of the options will take a couple of minutes to take effect on the client side.
Strangely enough I haven’t found any API reference to banning a member, though it is available as an option in the WebUI.

WebIU

Deauthorizing a member will basically revert it back to the initial joined state without a assigned IP or ability to communicate with other members.
Hiding a member will deauthorize it and hide it from view. Hidden members can be found by selecting the hidden checkmark in the member list and here can they be unhidden and authorized again (2-click process).
Banning a member will deauthorize it and permantly hide it from view. The member have to be manually added to the network to regain access.

API/CLI

For a member to leave a network run:

1
$ zerotier-cli leave <NETWORK-ID>

If you’re an admin you can hide a member:

1
$ curl -H "Authorization: Bearer <API-ACCESS-TOKEN>" -X POST -d '{"hidden":true}' https://my.zerotier.com/api/network/<NETWORK-ID>/member/<NODE-ID>

To unhide a member and authorize them again:

1
$ curl -H "Authorization: Bearer <API-ACCESS-TOKEN>" -X POST -d '{"hidden":false,"config":{"authorized":true}}' https://my.zerotier.com/api/network/<NETWORK-ID>/member/<NODE-ID>

I think that is enough for now :) In my next blog article I’ll go through the scripts I created to automatically join a network and create DNS records in DNSMASQ.