We Built Our Own Sandboxes

We recently released our in-app CloudX Agent to help our customers more easily understand their monetization performance, find opportunities for improvement, prepare experiments, and make changes to grow their business. The CloudX Agent can access the same dashboard context as a customer and use CloudX tools to investigate performance, identify opportunities, and make approved changes automatically.

For example, a customer might ask the CloudX Agent to investigate why their eCPM has increased recently. The agent can access the customer's apps, ad units, line items, revenue data, and configuration history. The customer doesn't need to specify what data to pull — the agent investigates automatically, discovers that a recent A/B test was promoted to production, and confirms that the new setup drove the revenue increase.

It may feel like magic, but like so many things it's really just engineering. In this blog post, we'll explain how we built the system that makes the CloudX Agent work. Making "magic" real — quickly, reliably, and with appropriately scoped access to customer data — requires dedicated sandbox environments that are fast, available, and securely isolated from each other. We run these sandboxes in dedicated sandbox clusters to control startup times, capacity planning, lifecycle operations, and security controls.

Agents need sandboxes

To complete work for customers, an agent needs a place to use tools, inspect context, and maintain working files — without risking interference with other systems or agents. These environments are commonly called sandboxes, isolated workspaces that give an agent enough access to complete a task without exposing other systems or customers. In our sandboxes, each agent gets an isolated filesystem, information about the customer and the task it's trying to solve, a specific set of tools that it can use, and explicit resource limits.

As software engineers, we often think in terms of 0, 1, and many. Running one agent in a sandbox is hard enough, but for the CloudX Agent to work reliably for all of our customers, the real problem is running many agents in many sandboxes.

We're not the only team building agent tools, and a number of startups provide on-demand sandboxes for agents. Our first MVP used a hosted sandbox service, which let us quickly iterate and validate the approach. But even during internal testing, we saw compute throttling, high and unpredictable response latency, frustrating rate limits, and limited telemetry. From a security perspective, it also meant sharing sensitive financial data with a third party. Not ideal!

So we built our own

So, as part of releasing to production, we built our own sandbox platform on top of our existing secure processing environment. Today, every CloudX Agent runs in a dedicated sandbox cluster on compute that we provision. This gives us control over:

• Startup speed: We keep complete application-level sandboxes warm, including the agent process and its session, rather than only warming compute. The agent responds quickly, and even more importantly, consistently quickly.
• Resource usage: Concurrency and burst capacity are bounded by cluster resources and product policy instead of provider quotas. We can autoscale our sandboxes using the same battle-tested tools that we use to scale our auction infrastructure.
• Network traffic: Sandboxes run beside the services and data they use, avoiding unnecessary network or regional hops. We can observe, rate-limit, firewall, and inspect sandbox network traffic.
• Sandbox lifecycle: Sandbox pods are automatically cleaned up when their sessions complete, fail, or expire.
• Operational stability: Sandbox logs, Kubernetes events, and live pod state use the same Datadog and Kubernetes tooling as the rest of the platform. New images, resource shapes, tools, and warm-pool policies are deployment changes rather than provider integrations.
• Security posture: Credentials, network access, runtime isolation, and retention are controlled as one system that we own and operate.

How our sandbox platform works

Unlike companies that provide dedicated sandboxes and need to roll their own schedulers, we've built our system on top of a dedicated sandbox cluster running Kubernetes. Kubernetes is familiar to our team, observable in production, and already the foundation for our auction infrastructure. We use Amazon's managed EKS and apply the same operational practices we use across the rest of the platform.

We have a dedicated control-plane component, the sandbox scheduler, that uses the Kubernetes API to schedule CloudX Agent instances onto sandbox pods.

When a customer starts a CloudX Agent session, the scheduler claims an application-ready warm sandbox pod or creates a new one. Each agent session gets one sandbox pod, which is later cleaned up according to specific lifecycle policies. Warm sandbox pods already contain the sandbox image, agent process, and initialized session, reducing both startup time and variance.

Each sandbox pod runs without cloud credentials, service-account tokens, mounted secrets, or provider API keys. LLM requests use a short-lived virtual key. The LLM Relay validates the key, adds the provider credential, records usage, and ultimately revokes the key.

Pods run as non-root with restricted system calls, no elevated kernel permissions, default-deny network policies, and gVisor for additional isolation, on dedicated nodes in the sandbox cluster.

All network traffic from the sandbox is proxied through the LLM Relay, which streams agent sessions and events between our agents and upstream model providers. It also enforces firewall rules that prevent unexpected ingress or egress. Because we own the system end-to-end, we can observe, monitor, and secure this traffic using the same tooling we use to secure our auction processes.

What this enables

Customers get an Agent that can investigate harder questions and complete more involved work without sacrificing speed or isolation. Internally, we now have a secure, observable platform that we can extend to other agent-powered CloudX features. Which is good, because there's always more to build!