Joining nodes to a Space

A Space runs on machines you own. Joining one is three steps: get a ticket, prepare the machine, run the join. It takes a few minutes and works the same whether the box is a cloud VM or a server in your own rack.

What a machine needs

• Linux, 64-bit, with systemd. Any modern distribution.
• 2 GB RAM minimum, 4 GB if you plan to run anything real. Two cores is a sensible floor.
• No Kubernetes packages. The join script installs containerd and a kubelet that matches your control plane’s version. A machine that already carries Kubernetes state must be reset first.
• Outbound access to your Space’s control plane on TCP 6443 and 8088, plus DNS, NTP and HTTPS for image pulls. Nothing needs to be reachable from the internet: the node dials out and keeps the connection open.
• Swap off, or kubelet configured to tolerate it.
• A stable name. The name you choose is pinned into the join ticket and cannot be changed afterwards without re-joining.

You do not need a public IP for the control plane to reach you

kubehz never connects to your node. Everything (including kubectl logs and kubectl exec) travels back through the tunnel your node opened. That is why the kubelet port stays closed in the firewall below.

1. Mint a join ticket

In the dashboard, open your Space and choose Add node. Give the machine the name it will carry in the cluster, and you get a ticket.

With the CLI:

lo kubehz join worker-1

The CLI mints the ticket for the Space that belongs to the active cluster directory. It needs four things: a cluster.lok8s.yaml with kind: Kubehz and spec.kubehz.hosting: shared, a completed lo provision for that cluster (it creates the Space), and KUBEHZ_TOKEN exported in your shell (an API token minted in the dashboard under Access → API Tokens). Without them, use the dashboard.

Three things about the ticket, all deliberate:

• It is shown once. Nothing stores it: not our database, not the dashboard. If you lose it, mint another; there is no “show me that ticket again”.
• It works for one node, once, and it is bound to the exact name you chose. A ticket for worker-1 cannot enroll worker-2.
• It expires in about ten minutes. Mint it when you are at the machine, not the evening before.

2. Prepare the machine

The node firewall

The node’s perimeter is a service to you, not a control over you: you have root and can change or remove it. The platform’s actual controls live cluster-side. What this recipe does is keep the machine’s own attack surface small while leaving the cluster fabric intact.

Save as /etc/nftables.d/kubehz-node.nft and apply with nft -f /etc/nftables.d/kubehz-node.nft. Re-running it replaces the previous table, so it is safe to apply repeatedly, and safe over SSH, because established connections are accepted first.

table inet kubehz_node {}
flush table inet kubehz_node

table inet kubehz_node {
  chain input {
    type filter hook input priority filter; policy drop;

    ct state established,related accept
    ct state invalid drop
    iif "lo" accept

    # Cluster fabric interfaces. Traffic on cilium_wg0 has already proved
    # itself to WireGuard; dropping post-decryption flows here is how you
    # break kubectl logs/exec without ever seeing why.
    iifname "cilium_wg0" accept
    iifname "cilium_vxlan" accept
    iifname "cilium_host" accept
    iifname "cilium_net" accept
    iifname "lxc*" accept

    # ICMP carries path-MTU discovery. Pod traffic is double-encapsulated, so
    # the usable MTU is smaller than the wire's — block these and you get
    # silent stalls instead of errors.
    icmp type {
      echo-request, echo-reply, destination-unreachable,
      time-exceeded, parameter-problem
    } accept
    icmpv6 type {
      echo-request, echo-reply, destination-unreachable,
      time-exceeded, parameter-problem, packet-too-big,
      nd-neighbor-solicit, nd-neighbor-advert,
      nd-router-solicit, nd-router-advert
    } accept

    # WireGuard: every node-to-node path rides this. The peers are other
    # machines in the cluster with arbitrary addresses, so the source cannot
    # be pinned — WireGuard's own keys are the authentication.
    udp dport 51871 accept

    # VXLAN (pod traffic, normally inside WireGuard).
    udp dport 8472 accept

    # Your SSH. Pin it to your own addresses if you can:
    #   tcp dport 22 ip saddr { 203.0.113.0/24 } accept
    tcp dport 22 accept

    # NodePorts: closed. Uncomment only if you deliberately want to expose
    # them on this machine's public address.
    # tcp dport 30000-32767 accept
    # udp dport 30000-32767 accept

    counter
  }

  # Cilium's datapath owns pod forwarding — the perimeter must not fight the
  # CNI. Pod-level rules belong in your NetworkPolicies.
  chain forward { type filter hook forward priority filter; policy accept; }
  chain output  { type filter hook output  priority filter; policy accept; }
}

Two ports are deliberately not open:

• 10250 (kubelet). kubectl logs and kubectl exec arrive through the tunnel your node opened, so the kubelet API never has to face the network. Opening it adds risk and buys nothing.
• NodePorts. Commented out above; opt in only if you mean it.

If your machine sits behind a cloud firewall (Hetzner Cloud Firewalls, a security group, an appliance), apply the same shape there. Two layers that agree are fine; a cloud firewall that silently disagrees with the host is how a node joins and then goes unreachable an hour later.

Lane: a Hetzner Cloud VM

Nothing special. A cx22 or larger with a standard image; the join script brings its own containerd and kubelet. If you attach a Hetzner Cloud Firewall, allow inbound UDP 51871 and 8472 and your SSH; outbound can stay open.

Lane: bare metal, or a machine in your own rack

Also nothing special, with two things worth checking:

• NAT is fine, a symmetric NAT is not always. The node dials out, so a plain home or office NAT works. WireGuard between your node and other nodes needs UDP 51871 to survive the NAT in both directions. Most do; some carrier-grade NATs do not.
• MTU. If your link is already reduced (PPPoE, a tunnel), pod traffic is encapsulated twice on top of that. Keep ICMP open (the recipe above does) and the path-MTU machinery sorts it out.

3. Run the join

Your ticket comes with a script. The dashboard shows it right under the ticket, as a copy field; lo kubehz join writes it to $TMPDIR/kubehz-join-<node>.sh (owner-only) and prints the scp/ssh line; the API returns it as the script field of the ticket (POST /api/spaces/{id}/join-token). Read it, then run it as root on the machine; it carries the ticket, so delete it once the node has joined. It installs containerd and a kubelet matching your control plane’s version, reads the cluster’s CA from the control plane itself and verifies it against your ticket (a machine can only ever join the control plane that minted its ticket, so there is no CA file to download and nothing to compare by hand), writes a bootstrap configuration pointed at your Space’s control plane (join.endpoint on the Space in the API), and starts the kubelet. On a machine that runs systemd-resolved, the script points the kubelet at /run/systemd/resolve/resolv.conf, so pods get the real upstream resolvers instead of the 127.0.0.53 stub. The node then:

1. presents the ticket and gets its own client certificate,
2. registers under the name the ticket pinned,
3. is labelled and tainted for your tenant: this is what keeps other customers’ pods off your machine, and it is applied by the platform, not by the node,
4. gets the CNI, and goes Ready.

The first join takes a few minutes, most of it pulling images.

Afterwards

• The node appears on your Space’s page with its status. kubectl get nodes from your own kubeconfig shows your nodes.
• Removing a node is Deregister in the dashboard. It withdraws the node’s credentials and takes it out of the cluster; your workloads reschedule onto your remaining nodes. The machine keeps running: shutting it down, and stopping whatever it costs you, is yours to do.
• Re-joining under the same name works: deregister, fix the machine, mint a fresh ticket for the same name, join again.

When a node will not join

Symptom	Usual cause
The join command reports the ticket is invalid	It expired (ten minutes), was already used, or was minted for a different node name. Mint a new one.
kubelet retries and never registers	Outbound 6443 blocked, or the control-plane hostname does not resolve on the machine. Check with curl -v from the node.
Pods cannot resolve names, the node itself can	The kubelet reads a stub resolver. The script handles systemd-resolved; with another local resolver, point resolvConf in the kubelet config at a file with real upstream servers.
The node registers but stays NotReady	The CNI is still starting, or the fabric ports (UDP 51871/8472) are blocked inbound. Check the firewall on both layers.
kubectl logs/exec time out, everything else works	The tunnel is up but return traffic is filtered, usually a rule dropping the cilium_wg0 interface.
Pods stay Pending on a Ready node	The pod does not tolerate your tenant taint. Deploy into your Space’s namespace, where tolerations are applied for you, rather than a namespace you created some other way.

Doc status

Aspect	Detail
State	Live: join flow as shipped (ticket → prepare → join); firewall recipe validated against nft
Last reviewed	5 September 2026