Skip to main content

​
Results

​
Cold Start Latency

Sequential — runtimes created one at a time:
Batch SizeSamplesMeanp50p95p99
1515.0 ms14.9 ms15.3 ms—
105014.6 ms14.4 ms15.9 ms—
5025014.6 ms14.3 ms16.6 ms18.1 ms
10050014.6 ms14.4 ms16.2 ms17.9 ms
200100014.6 ms14.3 ms16.1 ms19.6 ms
Concurrent — up to os.availableParallelism() - 4 runtimes created in parallel (16 on this machine):
Batch SizeSamplesMeanp50p95p99
1518.4 ms15.8 ms28.0 ms—
105024.4 ms23.0 ms32.3 ms—
5025035.0 ms35.0 ms44.5 ms47.2 ms
10050035.1 ms35.5 ms44.9 ms48.7 ms
200100035.2 ms35.1 ms44.6 ms51.0 ms
p99 is omitted (—) where sample count is below 100, as the percentile is not statistically meaningful at that size. Key takeaway: Sequential cold start is stable at ~14.3 ms p50 regardless of batch size — no degradation over time. Concurrent cold start scales from 15 ms to ~35 ms at 200 instances due to CPU contention, with p95 staying under 45 ms and p99 under 51 ms.

​
Warm Start Latency

Sequential:
Batch SizeSamplesMeanp50p95p99
153.0 ms3.0 ms3.3 ms—
10503.1 ms3.0 ms3.6 ms—
502503.1 ms3.0 ms3.7 ms5.0 ms
1005003.2 ms3.0 ms3.9 ms5.7 ms
20010003.1 ms3.0 ms3.7 ms4.6 ms
Concurrent:
Batch SizeSamplesMeanp50p95p99
153.6 ms3.3 ms5.3 ms—
10505.8 ms5.8 ms7.8 ms—
5025010.2 ms10.2 ms13.3 ms16.0 ms
10050010.4 ms10.2 ms15.9 ms20.7 ms
200100010.7 ms10.4 ms15.1 ms23.2 ms
p99 is omitted (—) where sample count is below 100. Key takeaway: Warm start is ~3 ms sequential — roughly 5× faster than cold start. The difference (~11 ms) is the cost of V8 isolate creation, which only happens once. Concurrent warm start tops out around 11 ms, dominated by thread contention rather than runtime overhead.

​
Memory Overhead

Batch SizeIterationsTotal RSS DeltaPer-Runtime RSSPer-Runtime HeapTeardown Reclaimed
156.1 MB6.1 MB0.25 MB2.3 MB
10541.9 MB4.2 MB~0 MB24.0 MB
505170.8 MB3.4 MB~0 MB120.8 MB
1005303.9 MB3.0 MB~0 MB241.4 MB
2005609.7 MB3.1 MB~0 MB483.0 MB
Key takeaway: Per-runtime RSS converges to ≤3.1 MB at scale. The batch=1 figure (6.1 MB) is inflated by fixed per-process overhead that amortizes away. JS heap delta is ~0, indicating the cost is dominated by native memory (V8 isolate, thread stacks, OS-mapped pages). RSS is an upper bound — true per-isolate memory is likely lower. Teardown reclaims 38–79% of RSS (higher at larger batch sizes where fixed overhead is a smaller fraction).

​
Methodology

​
Cold Start

Time from new NodeRuntime() through the first runtime.run() completing. This captures V8 isolate creation, context setup, bridge installation, module compilation, and initial execution. A trivial ESM module (export const x = 1) is used so the measurement reflects pure runtime overhead without workload noise. Each configuration runs 5 iterations (× batch size samples each) with 1 warmup iteration discarded. Tail percentiles at small batch sizes (≤10) have low sample counts and should be interpreted with caution. Sandbox provider comparison uses the p95 TTI (time-to-interactive) from ComputeSDK benchmarks. As of March 2026, e2b is the best-performing sandbox provider at 0.95s p95 TTI.

​
Warm Start

Time for a second runtime.run() on an already-initialized runtime. The V8 isolate is reused, but a fresh V8 context is created and all bridge globals (console, require, import, process) are re-installed. Module caches are cleared between runs. The difference between cold and warm start (~12 ms) isolates the cost of new ivm.Isolate().

​
Memory Per Instance

RSS (Resident Set Size) delta per live runtime, measured via process.memoryUsage().rss before and after spinning up N runtimes. Testing in batch averages out per-process fixed costs. Each batch size runs 5 iterations. GC is forced (two passes) between measurements (--expose-gc). RSS is a process-wide metric that includes JS-side wrappers, thread stacks, and OS-mapped pages beyond the isolate itself — the reported per-runtime figure is an upper bound on the true per-isolate cost. Sandbox provider comparison uses the minimum allocatable memory across popular providers (e2b, Daytona, Modal, Cloudflare) as of March 2026. The minimum is 256 MB (Modal and Cloudflare).

​
Cost Per Second

See the cost evaluation for full methodology and multi-provider comparison.

​
Test Environment

ComponentDetails
CPU12th Gen Intel i7-12700KF, 12 cores / 20 threads @ 3.7 GHz, 25 MB cache
Node.jsv24.13.0
RAM2× 32 GB Kingston FURY Beast DDR4 (KF3200C16D4/32GX)
RAM rated3200 MHz CL16, dual-rank
RAM actual2400 MT/s
OSLinux (kernel 6.x)
Timing mitigation"freeze" (default) — Date.now() and performance.now() are frozen inside the isolate; host-side performance.now() used for measurement is unaffected

​
Reproducing

# Clone and install
git clone https://github.com/rivet-dev/secure-exec
cd secure-exec && pnpm install

# Run both benchmarks (saves timestamped results to benchmarks/results/)
cd packages/secure-exec
./benchmarks/run-benchmarks.sh

# Or run individually
npx tsx benchmarks/coldstart.bench.ts        # cold + warm start
node --expose-gc --import tsx/esm benchmarks/memory.bench.ts  # memory
Results will vary by hardware. The numbers above are from the test environment described above.