Nezalنزال
← Home الرئيسية

Nezal security whitepaper (v0.1, draft for the buyer's security team)

Deployment model

Nezal runs entirely inside the customer's Kubernetes cluster: an operator (with a small per-application reverse proxy), an API with an embedded database on a persistent volume, and a static dashboard served by that API. No component calls the internet. Outbound connections are limited to the cluster API server, the customer's git server (operator), the customer's identity provider and SIEM collector (API).

Isolation

Each developer environment is a Kubernetes namespace with a ResourceQuota and a LimitRange derived from the team Policy. Copies of changed services run there; every other service is an ExternalName alias to the shared baseline. The router sits behind the baseline Service names and routes by the X-Nezal-Env header. At the cluster edge the header is derived only from the Host and any client-supplied routing header is discarded and counted. Unknown environments fall back to the baseline and are reported as Events.

Identity and access

Authentication is delegated to the customer's IdP via OIDC (authorization code with PKCE) or SAML 2.0. Session tokens are EdDSA-signed; roles are re-read from the store on every request. Three roles are enforced in the API, never only in the UI. A bootstrap token exists for the first run and is ignored once SSO is configured.

Rate limiting. Every endpoint that accepts a secret from someone not signed in carries two budgets at once: ten wrong passwords locks one account, and sixty failed attempts in fifteen minutes exhausts one client address across all of them. Counters live in SQLite, so restarting the API cannot clear a lockout and an auditor can see what happened. A forwarded address is honoured only behind a configured trusted proxy — otherwise any caller could invent a new address per request and side-step the budget.

The per-PR webhook endpoint is unauthenticated by design (a forge posts to it with an HMAC signature) and carries its own budget, charged only on requests that fail to authenticate. A forge posting correctly signed events is never limited however busy the repository; an address guessing webhook secrets runs out. It has its own bucket so that traffic cannot exhaust the budget protecting sign-in.

Policy enforcement

A validating admission webhook rejects Environment writes that exceed per-user or per-team quotas, TTL limits, override shared dependencies, use images outside the allowed registries, or request terminals where disabled. The reconciler re-checks TTL caps and override validity. The webhook's certificate is issued and renewed by the operator, with cert-manager optional.

Secret access

The operator holds no cluster-wide Secret permission of any kind. Not read, not write. Two independent limits enforce that, so a mistake in either is not sufficient:

  1. Permission is granted per Secret, by name. Connecting a repository makes the API create a namespaced Role naming exactly that Secret in resourceNames, bound to the operator's service account. Disconnecting revokes it. The full list of what the operator may read is kubectl get rolebindings -A.
  2. The informer cache watches only labelled Secrets (nezal.dev/git-credentials=true), so an unrelated application Secret is never fetched and never held in memory, even where permission would allow it.

There is deliberately no kubebuilder:rbac marker for Secrets in the operator's source, and a unit test reads the generated ClusterRole and fails if one reappears: editing the YAML alone would be undone by the next make manifests.

The operator also cannot write Secrets outside its own namespace. Distributing the preview signing key to application namespaces goes through the API rather than the operator, for exactly this reason. See ADR 0006.

Credential values never appear in logs, status or audit entries, and error messages from git operations are scrubbed of credential material, by value and by shape.

Audit

Every user action is appended to a hash-chained log in a database that refuses UPDATE and DELETE. Evidence packages are signed with a per-installation ed25519 key. Entries can be forwarded to a SIEM. Terminal sessions are recorded (asciicast) with input and output and audited at open and close. Secrets never appear in log lines or audit details; error messages from git operations are scrubbed of credential material.

Where evidence is held. By default everything Nezal keeps — the database, the audit log, terminal recordings, backups — is on one volume inside the cluster. A customer may instead configure their own S3-compatible object storage, and recordings, signed audit archives and scheduled backups are then copied there as they are produced.

That endpoint is the customer's own: MinIO or Ceph on-premises, a storage appliance, or a sovereign cloud's object storage inside the Kingdom. It is off until an administrator enters it, it joins the short list of things the product talks to at all, and nothing about it assumes or reaches a public cloud. The credential lives in a Kubernetes Secret rather than the database, because the database is what gets backed up and shipped around, and a credential for the place the backups go should not travel inside the backups.

Archives and backups are never deleted by Nezal once uploaded: their lifecycle belongs to the customer, enforced by bucket rules. Where the bucket has object lock enabled, those archives cannot be altered or deleted by anyone — including a Nezal administrator — until their retention expires, which is the write-once control an auditor asks for. The settings screen detects and states whether the bucket has it, because Nezal cannot turn it on. Terminal recordings are the exception: they expire from the bucket with the team's retention policy, because the policy is about the recording and not about which disk it sits on.

An upload failure never fails the thing that produced the file. The file is written locally first, exactly as it would be with no object storage configured, and the upload is retried; nezal_evidence_pending_uploads and the dashboard say plainly when the queue is not draining, because evidence that never arrives is evidence the customer does not have.

Licensing

Licences are ed25519-signed payloads verified offline against vendor public keys built into the binaries. There is no call home, so verification works in a sealed room. The development signing key is behind a build tag and is not compiled into release binaries.

The tiers, and what no key means. An install with no licence key is not refused: it runs as Community, one seat and one workspace, with none of the optional features. A free registered Community key raises that to ten seats. Paid plans (team, enterprise, sovereign) carry their seat count and feature set in the signed payload.

Licence state What happens
missing (no key installed) Community: 1 seat, 1 workspace, no optional features
valid The plan's seats and features
grace Everything continues; the dashboard states the expiry
expired (after grace) Only the left-hand column below is refused; nothing stops
invalid (bad signature, or a development key in a release build) Refused, in the API and in the admission webhook

missing and invalid are deliberately different. The first is somebody evaluating the product; the second is somebody who installed something wrong, and quietly treating it as the free tier would hide that.

Where it is enforced, exactly. The licence is a commercial mechanism, not a security boundary, and it is worth being precise about what it can and cannot stop, because a security team will ask.

Checked Not checked
Creating an environment Waking a sleeping environment
Extending an environment's TTL Opening a preview URL
Syncing an environment Reading or exporting the audit log
Creating a per-PR environment from a webhook Signing in, reading the dashboard
Building a service from source Anything already running
The optional features: terminal, siem, saml, per-pr, build, exec-reports Reading a build's status or logs

Everything in the right-hand column is deliberate. A developer opening their own preview must never meet a billing error, an auditor must never be told the evidence is unavailable for commercial reasons, and nothing is ever deleted or stopped because of licence state.

Expiry enters a grace period (30 days, 7 for a trial) in which everything continues to work and the dashboard says so. After grace, only the left-hand column is refused. Administrators are warned at 30, 14, 7, 3 and 1 days, on the expiry day, weekly through grace and once at the freeze — by email, a Prometheus gauge, a Kubernetes Event, a dashboard banner and a CLI line, so an air-gapped site with no mail relay is still told. Installing a renewed key lifts the freeze on the next request. See ADR 0009.

Seat counts are soft: exceeding them warns, a 10% burst is tolerated, and beyond that new invitations are refused while everyone already working continues. Enforcement runs in the API, which is also where the audit record is written, so a licence refusal is auditable. The operator does not enforce licensing; an environment already reconciling is never interrupted by it. See ADR 0004.

Building from source

This is the one thing in Nezal that deliberately executes code the customer's developers wrote, on the customer's cluster, and it is worth being exact about what that costs and what contains it.

What runs. A Dockerfile from the repository, whatever it RUNs, and every dependency it fetches. All of it is treated as untrusted input.

Where. One namespace, nezal-build, which holds builder Jobs and nothing else. It is the only namespace Nezal runs at the privileged Pod Security Standard. Rootless BuildKit maps its own user namespace, which needs an unconfined seccomp profile, and the "baseline" standard forbids that outright. That is the price of building container images inside a cluster without a Docker daemon, and it is not hidden in a values file.

What contains it:

Control Effect
No service account token (automountServiceAccountToken: false) A build cannot talk to the Kubernetes API at all.
Runs as uid 1000, never root Nothing in the build is root, inside the container or on the node.
All capabilities dropped except SETUID and SETGID The two rootless BuildKit needs to map a user namespace; both apply within the container's own namespace.
No Docker socket, no host path, no privileged container A mounted Docker socket is root on the node. There is none.
NetworkPolicy allow-list A build reaches DNS, the git server, the registry and the team's declared dependency mirrors. Nothing else. A fetch from the public internet fails rather than quietly succeeding.
Link-local carved out of every allowed range Cloud instance metadata at 169.254.169.254 hands credentials to anything that can reach it.
activeDeadlineSeconds and CPU/memory limits, per team A build cannot hang forever or take the cluster with a make -j.
backoffLimit: 0 A failed build is reported and looked at, never silently retried under a new pod whose predecessor's logs are gone.
Separate checkout container The git credential is used by an init container and is never visible to the build container, where repository code runs.
Build arguments screened at admission They are recorded in the image history and readable by anyone who can pull it, so anything named like a credential is refused.
Context and Dockerfile paths screened at admission A path climbing out of the checkout would read the Job's own filesystem.

What it produces. An image pushed to the customer's own registry, tagged by commit, and deployed by digest: a tag can be moved after the fact, a digest cannot. Every build is audited with the commit and the digest, and a build is always of one resolved commit, never of a branch, so two builds of "main" cannot produce two different images under one name.

Switching it off. build.enabled: false in the chart removes the namespace entirely. The build licence feature gates it commercially, and a team's Policy can disable it independently: a team may be allowed personal environments without being allowed to execute arbitrary build steps.

Scanning what a build produces. A preview image can be scanned for known vulnerabilities before the environment uses it, set per team in Policy as off (the default), warn (record the findings, allow the build) or block (refuse it above a severity threshold). Every build records what the scan concluded — clean, findings, or unscanned — because "clean" and "never looked" are different answers and a customer whose policy requires scanning needs to see which they have.

The scanner image and its vulnerability database are the customer's, supplied at install. Nothing is fetched: the scan runs with every "do not update the database" flag the scanner has, mounts the database read-only from a volume the customer populates, and simply does not run if either is missing — an air-gapped cluster that has not mirrored a scanner should not have every build start failing because somebody ticked a box.

It is available in every edition that can build. A security control is never a paid feature (ADR 0004).

Preview images are still not signed. The release pipeline signs what ships; a preview image is built from a developer's branch, used by one environment, and deleted with it, so a signature would attest to something nobody promotes. A customer whose policy requires every image in the cluster to be signed should build into their own registry and pipeline instead — registry per workspace exists for exactly that.

Supply chain

All binaries are static Go builds on distroless images, running as a non-root user with a read-only root filesystem and all capabilities dropped. Third-party dependencies are the Kubernetes client libraries, Helm SDK, kustomize, compose-go, go-git, go-oidc, crewjam/saml, golang-jwt, coder/websocket, modernc SQLite, cobra and Prometheus client.

Each release builds multi-architecture images (amd64 and arm64), produces an SBOM per image in both SPDX and CycloneDX, scans for critical vulnerabilities and fails the release on any, and signs each image by digest with cosign, attaching the SBOM as an attestation. Signing by digest rather than by tag matters: a tag can be moved to a different image after signing, and a signature over the name would still verify.

The air-gap bundle carries those SBOMs and signatures alongside the images, so a customer with no route to the internet can still verify what they are installing.

Known gaps

Stated plainly, because a whitepaper that lists none is not believed.

Closed since v0.1: preview URLs are now authenticated (ADR 0003); probes, mounted ConfigMaps and Secrets, scratch volumes, working directory and uid are now carried over from the source; the optional bundled registry serves TLS from a certificate the operator issues and renews from its own CA, rather than plain HTTP with verification disabled on every node.