Skip to content

CNI Compatibility

WireKube is designed to work alongside any CNI plugin without interference. Most of the specific compatibility considerations involve Cilium.

Route Isolation

WireKube places peer and gateway AllowedIPs in routing table 22347, selected by an IP rule at priority 200:

10.0.0.2/32 dev wire_kube table 22347  <- WireKube
10.244.0.0/24 dev cilium_host           <- CNI (lower metric = higher priority)

Operators should keep mesh and gateway AllowedIPs from overlapping CNI-managed routes unless that routing behavior is intentional.

Cilium BPF cgroup Hooks

The Problem

Cilium attaches eBPF programs to cgroup socket operations:

Hook BPF Program Trigger
BPF_CGROUP_UDP4_SENDMSG cil_sock4_sendmsg sendto() / sendmsg() with msg_name
BPF_CGROUP_UDP4_RECVMSG cil_sock4_recvmsg recvfrom() / recvmsg()

These hooks implement Cilium's socket-level load balancing for services. The current userspace Bind delivers relay packets directly and normally avoids the legacy localhost UDP proxy. The behavior in this section applies to the retained UDPProxy fallback path and older deployments.

Trigger Conditions

The BPF hook triggers when:

  1. The agent runs in a container (DaemonSet pod)
  2. The container is in a cgroup that Cilium monitors
  3. The agent calls sendto() or sendmsg() with a destination address (msg_name set)

The hook does not trigger when:

  1. Using write(2) on a connected UDP socket (no msg_name)
  2. The container runs in the host cgroup (some configurations)

Legacy UDPProxy Solution

The fallback relay UDP proxy uses net.DialUDP to create a connected UDP socket:

conn, _ := net.DialUDP("udp4", localAddr, remoteAddr)

On a connected socket, Go's conn.Write() translates to the write(2) syscall (not sendto(2)). The write(2) syscall does not set msg_name in the kernel's msghdr, so BPF_CGROUP_UDP4_SENDMSG is never triggered.

sendto(fd, buf, len, flags, dest_addr, addrlen)
  -> kernel: msghdr.msg_name = dest_addr
  -> triggers BPF_CGROUP_UDP4_SENDMSG
  -> Cilium cil_sock4_sendmsg runs -> may return EPERM

write(fd, buf, len)  [on connected socket]
  -> kernel: msghdr.msg_name = NULL (uses socket's connected address)
  -> does NOT trigger BPF_CGROUP_UDP4_SENDMSG
  -> bypasses Cilium entirely

Adaptive Fallback

As an additional safety net, the proxy detects EPERM errors and switches to raw syscall.Write() on a duplicated file descriptor:

// Standard path
_, err := conn.Write(payload)
if errors.Is(err, syscall.EPERM) {
    // Switch to raw syscall.Write for all future writes
    rawMode.Store(true)
    syscall.Write(dupFD, payload)
}

This ensures compatibility even if future kernel or Cilium versions change the BPF hook trigger conditions.

Verification

In testing with Cilium (kube-proxy replacement mode) on Linux 5.15+:

  • net.DialUDP + conn.Write() → No EPERM (confirmed: write(2) bypasses BPF)
  • net.ListenUDP + conn.WriteToUDP() → EPERM (confirmed: sendto(2) triggers BPF)

Alternative: Cilium Configuration

If you control the Cilium deployment, you can disable socket-level load balancing for host namespace pods:

# Cilium Helm values
socketLB:
  hostNamespaceOnly: true

This prevents cil_sock4_sendmsg from being attached to hostNetwork: true pods, eliminating the issue at the source.

fwmark Routing

WireKube uses fwmark 0x574B to prevent WireGuard packet loops:

# WireGuard marks its own UDP packets with fwmark 0x574B
# This rule ensures marked packets use the main routing table
# (bypassing the wire_kube interface)
ip rule add fwmark 0x574B lookup main priority 110

The exact fwmark priority is dynamic: WireKube chooses at least 110, places it after the current local-table rule, and keeps it below the WireKube table rule at priority 200. Without this rule, a WireGuard UDP packet destined for a peer AllowedIP could re-enter the tunnel recursively.

Tested CNI Plugins

CNI Status Notes
Cilium (kube-proxy replacement) Verified Connected socket bypass works
Cilium (with kube-proxy) Expected to work Same BPF hooks apply
Calico Expected to work No known socket BPF hooks
AWS VPC CNI Expected to work No known conflicts
Flannel Expected to work No known conflicts