Sizing and measured performance

The first question after "is it secure" is "how many developers will it hold". This document answers it with numbers from a run anyone can repeat, together with the caveats that make those numbers honest. Where the measurement does not support a claim, the claim is not made.

Everything below was measured, not modelled. Reproduce it with hack/load.sh.

The measurement

One command, on a kind cluster, using the Helm chart as a customer would install it:

ENVS=30 SOAK=5m hack/load.sh

It creates N environments through the API and waits for each to report Ready, then drives the read path with concurrent clients, then soaks with several writers appending to the audit chain, and finally verifies the chain. Results land in .load-results/.

Results

Measured 2026-09-12 on an Apple M4 Pro, 14 cores, 24 GiB RAM, macOS 26.5.2, Docker Desktop with 14 CPUs, single-node kind, one API replica and one operator replica.

Environment creation

Measured from the API accepting the create to the Environment reporting Ready — the reconcile, the pod scheduling and the routing update, which is what a developer actually waits for.

These environments deploy a prebuilt image. Building from source adds the build, which depends on your repository and registry and is not included here.

API read load

This is the endpoint a dashboard polls, with 30 environments in the workspace.

Soak, with sustained audit appends

Every write appends to the hash chain, which cannot be parallelised inside the database — each entry hashes onto the previous one. Four concurrent writers is the shape that would expose that as a bottleneck, and at this rate it is not one: write latency is lower than read latency, because a read serialises 30 environments and a write appends one row.

What this supports

Tested to 30 concurrent environments on a 14-core laptop, with the API answering 290 reads/second at a p99 of 78 ms and sustaining 42 audited writes/second with the chain intact.

For a rough sense of scale: a team of 200 developers, each creating a couple of environments a day and having a browser poll the dashboard, is nowhere near any of these figures. The single-replica decision in ADR 0005 assumed "a few hundred developers doing occasional writes"; nothing measured here contradicts that.

What this does not support

Read this part before quoting the numbers above.

• kind is not your cluster. Every node is a container on one machine, with local storage and no real network. A real cluster has slower disks, slower networking and other tenants. Treat these as a floor for the software, not a capacity plan for your hardware.
• 30 environments is what this machine held, not a product limit. It is the largest run that fit comfortably; the numbers were flat across it, with no sign of a knee. What happens at 300 is unmeasured, and we will not guess.
• No build was involved. Every environment ran a prebuilt image.
• Memory over the soak is unmeasured. kind runs no metrics-server, so the before/after figures came back n/a. A leak over five minutes would not necessarily show anyway; a longer soak on a cluster with metrics is the honest way to answer this, and it has not been done.
• One workspace, one application, two services per environment. A wider application changes the reconcile cost.
• No terminal sessions, no SIEM forwarding, no evidence upload were running concurrently.

Rules of thumb

Until you have measured your own cluster:

• API: 1 replica, 500m CPU / 512 MiB is comfortable at this load. It is single-replica by design (ADR 0005); do not scale it horizontally, and do not need to.
• Operator: 1 replica. Reconciles are short and the work is per-environment.
• Data volume: the audit log only grows. At roughly 42 writes/second sustained — far above any real installation — the soak wrote 12,640 entries in five minutes. A real install of 200 developers writes a few thousand entries a day. Size the volume for years, watch nezal_data_volume_free_bytes, and archive old ranges to object storage.
• Per environment: whatever the services you override request. The shared baseline is paid once per application, not once per developer, which is the entire point.

Repeating this

ENVS=60 PARALLEL=8 SOAK=30m hack/load.sh    # bigger, longer
KEEP=1 hack/load.sh                          # keep the cluster afterwards

The script fails if the audit chain does not verify at the end. That is not decoration: this measurement is how the canonicalisation bug in phase 10 item 15 was found, where the chain reported tampering on a log nobody had touched.