Why we run every code submission in a Firecracker microVM

Running arbitrary user-submitted code is one of those things that sounds fine in a design doc and terrifying the moment you look at your production traffic. Every online judge, every notebook host, every AI coding assistant that emits an executable snippet, every CI runner — they all face the same question: how do you run someone else's program without letting it eat your box?

There are five common answers. Four of them are compromises. We picked the fifth, and this post is why.

The five options, briefly

1. Nothing. Run the code in the same process as your server. Fine for a hackathon demo, catastrophic for anyone else.

2. OS-level namespacing. chroot, cgroups, seccomp, nsjail. The kernel becomes your isolation boundary. When the kernel has a bug — and it does, regularly — the isolation is gone.

3. Containers. Docker, Podman. Namespacing plus filesystem layering plus a nice API. Same fundamental weakness as (2): the host kernel is shared. A CVE in the kernel gives root on the host from inside the container. Google's security team has been publicly clear about this: containers are a convenience boundary, not a security boundary.

4. User-mode kernel. gVisor intercepts every syscall from the guest and re-executes it in a Go-based user-mode kernel. Real isolation, but the syscall interception is expensive (Netflix measured 10-50% overhead on I/O-heavy workloads) and the compatibility matrix has holes.

5. Hardware virtualisation. A real hypervisor spinning up a real (tiny) VM per submission. The guest has its own kernel; the isolation is enforced by Intel VT-x / AMD-V hardware. This is what AWS Lambda does, and what we do. The primitive is Firecracker — AWS's open-source hypervisor, purpose-built for this exact shape of workload.

What Firecracker actually gives you

Firecracker is not a general-purpose hypervisor. It intentionally supports one device model — virtio-net, virtio-block, virtio-vsock, that's it — and nothing else. No PCI passthrough, no video, no USB. That constrained surface makes the binary small (~5 MB), the attack surface tiny, and the boot time fast enough to be interactive: measured cold boot on our fleet is 125-150 ms, warm-snapshot restore is under 20 ms.

Firecracker also has a jailer binary that runs each microVM inside its own chroot, its own PID namespace, and its own resource cgroup. So even if a Firecracker vulnerability let a guest escape the microVM (there have been zero reported in the wild since launch), the escaped code lands in an unprivileged chroot with no network route and a resource ceiling. Belt and braces.

The trade-off we accepted

Firecracker costs more than a container per run. Not much — a warm-snapshot restore is ~15 ms and a fresh boot is ~135 ms — but it's not zero. On a typical run our overhead is 40 ms of Firecracker + jailer + kernel work per submission, versus ~2 ms for a container start.

For a batch job that runs for 30 seconds, that's invisible. For a latency-sensitive per-keystroke evaluator (which we don't offer, but some judges do), it would matter. We priced the trade-off consciously: the average user waits 400 ms for hot-lane runtimes and 1-2 s for cold, and 40 ms of isolation setup is a rounding error inside that envelope.

How we run 92 runtimes on one host

A separate Firecracker microVM per submission would be too much cold-boot cost if we did it naively. Two tricks make it work at our scale:

Snapshot pools. Popular runtimes (Python, Java, C++, Go, Node, Rust) live in a warm pool: N pre-booted microVMs per language, sitting at the post-init snapshot with the interpreter/compiler loaded. A submission attaches to one, injects source, runs. Restore is 15 ms. The pool refills asynchronously.

Cold lane for the long tail. Ada, COBOL, BQN, Brainfuck — most runtimes are cold-lane. First submission cold-boots a fresh microVM (135 ms), runs, tears down. No pool because the traffic doesn't justify it. This keeps the memory footprint on the host bounded — the warm pool at 8 slots × 6 languages costs ~1.5 GB of resident RAM; putting Zig and Crystal into the same pool would double that for no user benefit.

What we'd do differently

Nothing structural — we'd pick Firecracker again. Two small things:

Per-language kernel images. Right now every runtime shares one minimal kernel image. In hindsight, cutting language-specific images that only include the syscalls that runtime actually uses would let us tighten seccomp filters per runtime rather than run a superset filter. Small win, painful to maintain.

Networking. We currently disable network in every microVM. If we re-did it, we'd support a scoped egress option (proxy DNS + HTTPS only, allowlist-based) so runtimes that legitimately need to fetch packages (Rust with cargo, Node with npm) can be opt-in — with per-user rate limits and a whitelist of hosts.

Try it

Every submission on this site — every one — runs in its own Firecracker microVM. Every language listed on the languages page. Every problem on the problems page. Every anonymous cURL to POST /execute. You can watch it happen live per-test over WebSocket in the playground.

Firecracker was built specifically for serverless computing. It provides the security and workload isolation of traditional VMs while enabling the speed and resource efficiency of containers.

If you have questions about how we run untrusted code, or you spot something in this post that's wrong or naive, we'd genuinely like to hear it. Email us or open an issue on the source repo.