Building a Multi-Tenant Control-Plane for Ephemeral Agent Workloads

"Give every team a way to let their agents run code safely" sounds like an infrastructure request. In practice it decomposes into three separate, reusable problems — and building the wrong one first is the expensive mistake.

Three problems that get conflated into one ticket

A platform team asked to support agent code execution across the org usually gets a single request that's actually three distinct concerns wearing a trenchcoat: who's allowed to run what and how much of it (accounts, auth, fair-use limits), how a pod actually gets created and torn down fast enough that a user isn't staring at a ten-second cold start (lifecycle and warm-pool management), and what a single running pod is actually allowed to do (the isolation boundary itself). Building all three as one tangled service is how you end up rewriting the whole thing when the second team asks for a slightly different quota policy.

Layer one: the control-plane doesn't know what a pod is

The control-plane's entire job is deciding whether a request should be allowed to proceed — API key validation, per-account fair-use enforcement (a token-bucket limiter, not a hard wall that fails oddly at the boundary), and routing. It has no opinion about Kubernetes at all. That separation is what lets fair-use policy change — a new tier, a different sandbox-hours cap — without anyone touching pod-provisioning code, and it's what lets the same control-plane sit in front of a self-hosted cluster or a hosted one without a rewrite.

Layer two: the warm pool is what makes "fast" not fight "isolated"

The naive version of pod-per-session isolation is genuinely correct but slow: schedule a new pod, wait for image pull and container start, then hand it to the user. A warm-pool manager breaks that tradeoff by keeping a small number of pre-provisioned, not-yet-assigned pods on hand, so "create a session" usually means "claim an already-running pod" instead of "wait for the scheduler." This is the layer worth actually load-testing before rollout — pool size versus average session duration versus how fast the pool refills determines whether your p99 session-start latency is fine or embarrassing, and that number is specific to your own workload shape, not something to copy from someone else's benchmark.

Layer three: isolation is a property of the pod spec, not the application code

Non-root, dropped capabilities, read-only root filesystem, a default-deny NetworkPolicy— none of this lives in a service's business logic. It lives in the pod template and the cluster's RBAC/NetworkPolicy manifests, which means a platform team can audit and change the isolation posture for every tenant at once, in one place, instead of trusting every product team to configure their own pod specs correctly.

The cleanup problem that gets skipped in the first design

Ephemeral workloads create an ongoing garbage-collection problem that's easy to miss in a first design: a session whose caller crashed, disconnected, or simply forgot to call "destroy" still has a pod consuming cluster capacity indefinitely. A reaper process that periodically finds and destroys sessions past a configured max-lifetime is the unglamorous but load-bearing piece that keeps a cluster's actual capacity matching what the control-plane thinks is in use — worth designing in from the start, not bolted on after the first capacity incident.