boxkite
← All posts
Security

Closing the last plaintext hop: manager-to-sidecar TLS

SECURITY.md had disclosed this since early in the project: src/boxkite/manager.py's httpx.AsyncClient calls to a pod's sidecar were plain HTTP, not TLS. The X-Sidecar-Auth-Token header and every exec command or file body crossed the pod network in cleartext — readable by a compromised co-located pod, a misconfigured or widened NetworkPolicy, or a compromised node. This closes that gap (GitHub issue #51).

Three options, one constraint that decided it

We scoped three options before writing any code. The deciding constraint was boxkite's promise that it runs on any cluster with just RBAC and NetworkPolicy — no mesh, no cert-manager required to be secure.

OptionVerdictWhy
CA-backed mTLSRejected as defaultWould require a service mesh or cert-manager just to be secure — breaks the RBAC-and-NetworkPolicy-only baseline.
NetworkPolicy onlyRejectedDescribes today's mitigation, not a fix — no protection against a compromised node or a widened policy (a bug class this repo has already had to correct once).
Per-pod pinned certChosenA fresh, self-signed cert per pod, pinned by the manager — reusing the exact per-pod Secret that already carries the auth token. No new infrastructure, no operator setup.
The three transport options we scoped for issue #51, and the verdict on each.

How the pinned cert flows through one pod

At the same point in _create_pod() where generate_sidecar_auth_token()already runs, we now also generate a cert/key pair and store both in that same per-pod Secret. The sidecar serves HTTPS when a cert and key are mounted; the manager pins its trust to that pod's exact certificate rather than validating against a public CA.

Per-pod TLS, minted at creation and pinned

In _create_pod(): mint per podTLS cert + key · X-Sidecar-Auth-Token
per-pod Kubernetes Secretcert · key · auth token — created before create_namespaced_pod
SandboxManagermanager process · pins this pod's exact cert

one Kubernetes pod

sidecar containerserves HTTPS · terminates TLS
The manager mints the cert itself in _create_pod(), keeps it to pin against, and bakes it into the per-pod Secret the sidecar mounts — so the same call that creates the pod also establishes the trust anchor. No shared CA is ever involved.

The one deviation from the design doc

One thing didn't go exactly as the design doc scoped it, and we documented the deviation rather than silently diverging. The design called for an IP-SAN cert matching the pod's IP, generated at cert-gen time. But a pod's IP is only assigned once the CNI schedules it — strictly after the Secret (which must exist before create_namespaced_podis even called) already needs the cert baked in. There's no way to know the IP at generation time without delaying pod creation or restarting the sidecar mid-boot.

What this closes, and what it deliberately doesn't

What this closes: passive sniffing of the pod network by a compromised co-located pod, a misconfigured or widened NetworkPolicy, or a compromised node can no longer read the auth token or command/file contents in transit. What it deliberately does notclose: a compromise of the sidecar process itself still sees the same plaintext data it always did, since the sidecar terminates TLS — this is a wire-transport fix, not a change to the sidecar's own trust level. The same is true symmetrically for a compromised manager or control-plane process.

There's a fallback for operators who already run a service mesh with its own mTLS: the sidecar falls back to HTTP under an explicit SIDECAR_TLS_DISABLED escape hatch (default false). And the escape hatch is exactly that — an operator who sets SIDECAR_TLS_DISABLED=truecarelessly, rather than because they run their own mesh's mTLS, reopens the gap this change closes.