Administrator guide

First run

1. Sign in with the bootstrap token (api.bootstrap.* in the chart). Everything you do is audited under the actor bootstrap-admin. The token stops working once SSO is configured and the API restarts, which is the intended end of its life.
2. Licence: paste the key from Nezal. It is verified locally, with no call to us, so it works air-gapped. State valid shows customer, seats, expiry and features. After expiry there are 30 days of grace (7 for a trial) during which everything still works and the dashboard says so. After grace, no new environments or users are admitted. Nothing is ever deleted because of the licence, and nothing existing stops running.
3. SMTP, before you invite anyone. Host, port, security (starttls, tls or none), credentials and the From address, under Security & email. Send test proves it before an invitation depends on it. The password is write-only: it comes back redacted and the dashboard sends the redaction back untouched, so saving the form cannot overwrite a working password with an empty one.
4. SSO, or local accounts, or both. OIDC (issuer, client id and secret, groups claim) or SAML (IdP metadata XML, SP key pair). Map IdP groups to the admin role and to workspaces as developer or team lead.
5. MFA policy under Security & email. With enforced on, every local account must enrol a TOTP authenticator before it can do anything. SSO accounts are unaffected: the second factor there belongs to your identity provider.
6. Workspaces: one per team. A workspace maps to an application namespace; the Baselines, Environments and Policy of that team live there.
7. Policy per workspace: environments per user and per team, CPU and memory quota, default and maximum TTL, idle sleep, grace period after expiry, allowed registries, terminal on or off, terminal idle and maximum session length, and how long session recordings are kept. The operator's admission webhook enforces it on every write, so it holds for kubectl too, not only for the dashboard.
8. Trusted proxies, if the API sits behind an ingress controller or load balancer. Set api.trustedProxies to that proxy's pod network. Until you do, X-Forwarded-For is ignored entirely and the audit log records the proxy's address rather than the user's. That default is deliberate: believing the header unconditionally would let any caller write an arbitrary address into the tamper-evident record an investigator relies on.
9. Check the install answered: helm test nezal -n nezal-system.

People: local accounts, invitations and second factors

Users come from your identity provider, or are invited directly, or both.

Inviting: Users & teams > Invite, with an email, an optional workspace and role, and whether they are an admin. They receive a link to set a password. An invited account holds a seat from the moment it is created, not from first sign-in, because otherwise an administrator could invite an unlimited number of people and have them all working before the seat count noticed.

Second factor: with MFA enforced, a new local account is walked through enrolment on first sign-in and shown ten single-use recovery codes, once. Reset MFA on a user clears their enrolment so they can enrol again; it is the answer to a lost phone, and it is audited.

Rate limiting is on and needs no configuration: ten failed sign-ins for one account within fifteen minutes locks that account for fifteen minutes, and sixty failures from one address in the same window blocks the address. An account that has been invited but has not yet set a password is rate-limited too.

Sessions and tokens: each person can see their own active sessions under Account, with the device and the last time each was used, and revoke any of them. "Sign out everywhere" spares the session doing the asking, so nobody signs themselves out by accident. An administrator can revoke every session of another user, which takes effect on that user's next request rather than whenever their token happens to expire. CLI tokens are minted from the dashboard and appear in the same list.

Removing someone: deleting a user revokes their sessions and tokens and frees the seat. A user who still owns environments is refused, with the count, rather than deleted: removing them silently would leave environments nobody can reach and quota nobody is accountable for. Delete or hand over their environments first, then delete the account.

Roles

Terminal sessions are always owner-only, never on the baseline, and can be switched off per team by an admin or by the Policy.

Scanning built images

If your policy says every image in the cluster is scanned, set imageScan in the team's Policy:

imageScanSeverity defaults to HIGH,CRITICAL. A preview environment full of MEDIUM findings nobody will act on is how a scan gets switched off again.

This needs two things at install: imageScan.image (a scanner — Trivy — mirrored into your registry) and imageScan.dbClaim (a PersistentVolumeClaim in the build namespace holding its vulnerability database). Nothing is downloaded. The scan runs with every "do not update" flag the scanner has and mounts the database read-only, so a build can never alter the database it is judged against. With either value unset, no scan runs whatever a Policy asks for — an air-gapped cluster that has not mirrored a scanner should not have every build start failing.

Refresh the database on your own schedule, the same way you refresh any other offline feed. A stale database is a scan that misses recent findings, and nothing here will tell you it has gone stale.

The result appears on the environment's Builds tab against each service.

Terminal limits and recording retention

A web shell is the highest-privilege thing a developer can do here, and the only action that would otherwise leave no detailed trace: every other action is an audit entry naming what was done, while a shell is one entry saying it was opened. That is why the recording is not optional — terminal on means recorded — and why the Policy bounds the session rather than the recording:

A session closed by a limit says so: the sessions list and the audit entry both carry the reason (idle, max, cap or user), because "the shell exited" and "we closed it after an hour idle" are different facts to whoever reads the log later.

A recording is also capped at 20 MiB. Reaching the cap ends the session rather than silently stopping the recording — an unrecorded shell is worse than a closed one — and a runaway process reaches it in seconds, which is the case the cap exists for.

Retention deletes the recording, never the audit entry. What happened stays in the log permanently; only the replayable file ages out, and the sessions list then says "Recording expired" rather than offering a download that cannot work. Each expiry is itself audited, as actor system.

Setting retention to 0 is supported and is sometimes what a regulator asks for, but recordings share a volume with the database: a team that keeps them forever eventually fills the volume, and when it fills the database becomes read-only and the audit log stops with it. Watch nezal_data_volume_free_bytes, and see Monitoring below.

Repositories and applications

Connect the repository first, under Repositories: its URL, the default ref, and how to authenticate (an HTTPS token, an SSH key, or none for a public mirror). Test clones it before anything depends on it. Credentials go into a Kubernetes Secret in the application namespace labelled nezal.dev/git-credentials=true, and are never readable back through the API. The operator is granted access to that one Secret by name and holds no cluster-wide Secret permission at all (see ADR 0006).

Optionally set a webhook secret on the repository. That is what GitHub's X-Hub-Signature-256 or GitLab's X-Gitlab-Token is checked against on POST /webhooks/{workspace}/{repository}, which is how a pull request gets its own environment. Like the git credential, it is write-only.

Then connect the application, pointing at that repository and a path (Helm, Kustomize, plain manifests or docker-compose). The operator renders it, lists the services, and reports in status.warnings everything it could not carry over. Read those warnings on the application page: they are where "this service has no health check" and "this bind mount cannot exist in a cluster" are said out loud. Mark databases and queues as shared so environments never copy them.

Promoting a release means changing the source ref. Every promotion is recorded in the application's release history with the commit, the time and who did it, and rolling back is promoting an older ref from that list. Environments keep sharing the baseline; only their overridden services are their own.

Per-PR environments (licensed feature per-pr): set prEnvironments to all, or to label with a label name, and a pull request opening creates pr-<number>-<application>, a push syncs it, and closing the pull request deletes it. They are owned by webhook, not by a person, so they do not consume anyone's personal environment quota.

Posting the preview URL back. Tick Post the preview URL to the pull request on the repository and Nezal sets a commit status (nezal/preview) and writes one comment, editing that same comment on every push rather than repeating it — a dozen identical bot comments is why people mute these integrations.

Off by default, because writing to your pull requests is not something to assume. It needs a token that can write to the repository: a read-only deploy key connects and clones perfectly well but cannot comment, and Nezal skips posting quietly rather than filling the log. GitHub (including Enterprise) and GitLab are recognised from the repository URL; any other git server has no pull requests to post to and nothing is attempted.

Each post is in the audit log as forge.post. Nothing new is contacted: this is the same git server the repository already points at.

Preview URLs

Every environment publishes <environment>--<application>.<domain>. With previewAuth.enabled (the default) a visitor must have signed in to Nezal and have read access to that environment; the cookie is scoped to one environment and lasts eight hours.

Leave it on. With it off, a preview host is reachable by anyone who can reach the ingress, which on a bank's internal network is every employee, every contractor's laptop and every other workload in the cluster. A developer testing against production-shaped data would be publishing it to all of them without being told. See ADR 0003.

Audit and evidence

Every action of every user is one entry with a sha256 chained to the previous entry. Audit > Verify walks the chain; Export downloads a package of JSON lines plus a manifest signed with the installation's key (public key at /api/v1/audit/public-key). Forward entries live to your SIEM over syslog (TCP or UDP, RFC 5424) or HTTP JSON under Settings > SIEM. Terminal recordings (asciicast) are listed under Terminal sessions.

Retention

The audit table refuses UPDATE and DELETE at the database level, which is the property that makes it worth anything to an auditor. Nezal therefore has no way to delete an old entry, and will not grow one.

What it has instead is archiving: export everything older than a cut into a signed evidence package you keep wherever you keep evidence, and record that the range is accounted for.

curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://nezal.bank.internal/api/v1/audit/archive?olderThanDays=365&location=vault://audit/2025" \
  -o nezal-audit-archive.multipart

olderThanDays defaults to 365. location is recorded exactly as you give it and is never read back — an archive Nezal could reach is an archive Nezal could alter. Archiving a range that holds nothing is not an error and records nothing, so a scheduled call can run on a quiet week.

GET /api/v1/audit/archives lists what has been archived: the range each one covers, its head hash, and the head hash of the archive before it. Consecutive archives join up with no gap, so an auditor can show the sequence is complete — a gap would be indistinguishable from a deletion.

Nothing is deleted by any of this. After archiving, the live log still holds every entry and Audit > Verify still walks the whole chain. Reclaiming space is a separate decision, taken with the evidence already in hand rather than instead of it; the supported way is to restore an older backup into a separate installation for reference and start the live one fresh, which is deliberately more effort than deleting rows.

Each archive carries a manifest with a sha256 of the entries and an ed25519 signature over the manifest. An auditor can check both years later without this installation being available: the manifest carries the public key it was signed with.

Lifecycle

Environments expire at their TTL: copies scale to zero, the preview URL shows an "expired" page, and after the grace period the environment is deleted. Idle environments sleep; the first request wakes them. Each transition is a Kubernetes Event and an audit entry.

Licence and seats

Licence shows the customer, the term, the features and the seat count against active users. A seat is someone who has signed in or holds a pending invitation.

Seats are soft. Passing the count raises a warning, a 10% burst is tolerated outright, and beyond that new invitations are refused while everyone already working carries on. A bank does not want its developers stopped mid-sprint by a counter; over-deployment is a renewal conversation, not an outage.

Usage report downloads a signed report of monthly active seats over the term, which is what a renewal is negotiated against. It needs no network and reveals nothing outside the cluster.

Being warned before it expires

Nezal warns about expiry through four channels at once, because in a regulated site at least one of them is usually unavailable. It starts 30 days ahead, then again at 14, 7, 3 and 1 days, on the expiry day itself, weekly through the grace period, and once when the grace period ends. A trial warns at 7, 3 and 1 days instead — its whole term is shorter than a paid licence's first notice.

Each notice is sent once. Restarting the API does not resend one, and installing a renewed key starts the sequence again for the new term.

The email recipients are every enabled administrator, plus whatever is set at PUT /settings/licence-notify. That list exists because the people who approve a renewal are usually in procurement and rarely hold an admin account. A disabled administrator is not notified.

If there is no SMTP relay the other four channels still work, and the audit log records licence.notice with smtp: false — so "we were never told" has an answer either way.

Nothing about any of this leaves the cluster. The mail goes through the customer's own relay; Nezal does not contact its vendor at any point.

Monitoring

The API and the operator both serve Prometheus metrics on /metrics, unauthenticated on the pod network and not exposed through the ingress. Nothing is sent anywhere: your Prometheus scrapes them, the same as anything else in the cluster.

No metric carries a user, a customer, an environment or a workspace name. Labels end up in dashboards and in long-term storage, so a label is a place data goes and stays — and one series per environment would be unbounded cardinality besides.

Alert rules worth having

These are the five that matter. Everything else is a dashboard.

groups:
  - name: nezal
    rules:
      # The licence stops new work after its grace period. Thirty days is enough for a
      # purchase order to go through; the product also emails, but not every site has SMTP.
      - alert: NezalLicenceExpiringSoon
        expr: nezal_licence_seconds_until_expiry < 30 * 86400
        for: 1h
        annotations: {summary: "Nezal licence expires in under 30 days"}

      # The volume holds the database and every recording. When it fills, the database
      # becomes read-only and the audit log stops with it.
      - alert: NezalDataVolumeLow
        expr: nezal_data_volume_free_bytes / nezal_data_volume_total_bytes < 0.15
        for: 15m
        annotations: {summary: "Nezal data volume below 15% free"}

      - alert: NezalAPIErrorRate
        expr: |
          sum(rate(nezal_api_requests_total{status=~"5.."}[5m]))
            / sum(rate(nezal_api_requests_total[5m])) > 0.05
        for: 10m
        annotations: {summary: "Over 5% of Nezal API requests are failing"}

      # A stalled audit head is either a very quiet install or a broken one, and you need to
      # know which. Pair it with the error-rate alert: quiet plus errors is not quiet.
      - alert: NezalAuditLogStalled
        expr: increase(nezal_audit_head_seq[6h]) == 0
        for: 30m
        annotations: {summary: "No Nezal audit entries in six hours"}

      # Evidence that never reaches your object storage is evidence you do not have.
      - alert: NezalEvidenceUploadsPending
        expr: nezal_evidence_pending_uploads > 0
        for: 1h
        annotations: {summary: "Nezal evidence files are not reaching object storage"}

The operator publishes its own series (nezal_environments_by_phase, render counts and durations). A disagreement between the operator's view and the API's is itself a signal: they read the same objects by different paths.

Getting support

Nezal sends no telemetry, so there is nothing for us to look at when something goes wrong. The support bundle is the substitute: one file with what a support engineer would otherwise spend a day asking for.

Admin → Installation → Download support bundle, or from the command line when the dashboard itself is the problem:

kubectl exec -n nezal-system deploy/nezal-api -- \
  /nezal-api support-bundle > nezal-support.json

It holds: component versions; the licence state, plan and seat counts (never the key); every setting with credentials redacted; the last known result of each git, registry, SMTP and evidence-store check; pod status and recent warning Events in the release namespace; the audit head and whether the chain verifies; and the last 2,000 API log lines.

Nothing is sent anywhere by producing it. The first line of the file says so. Read it, then attach it to your support request yourself — that is the whole point of the format being one readable JSON file rather than an archive.

Credentials, tokens, keys and Authorization headers are redacted wherever they appear, including in log lines, which are redacted as they are recorded rather than as the bundle is written. A test plants realistic credentials through the API and fails the build if any of them reaches a bundle. Even so: read the file before you send it. It describes your installation in detail, and only you can judge whether that detail is something you are happy to share.

The command-line version reads the database directly, so it works when the API is not serving. It cannot include the running process's recent logs; the dashboard download can.

Backup and restore

The API's PVC holds the SQLite database, the session and audit-export keys, and terminal recordings. It carries helm.sh/resource-policy: keep, so helm uninstall leaves it alone; removing it is a deliberate kubectl delete pvc.

Back up with the command, not a file copy. Copying the database file while it is open can produce an unusable backup:

kubectl exec -n nezal-system deploy/nezal-api -- \
  /nezal-api backup --out - > "nezal-$(date +%F).db.enc"

The backup is encrypted. It holds every password hash, every stored second-factor secret and the whole audit log, and once it reaches object storage its custodian is not necessarily the person administering Nezal. AES-256-GCM, with a 32-byte key in the nezal-api-keys Secret alongside the signing keys.

Lose that Secret and every backup becomes unreadable. There is no recovery path and there is deliberately no escrow. Back the Secret up separately, somewhere your database backup tooling does not reach, before you need it.

--plaintext writes an unencrypted backup for a customer whose own key management requires it. The command says plainly what that file contains.

--out - streams the backup to stdout. That is the retrieval path, not a convenience: kubectl cp shells out to tar inside the container, and the API image is distroless — no shell, no tar — so kubectl cp cannot work against it at all. Everything the command says about itself goes to stderr, so the redirected file holds nothing but the database.

Check it before trusting it. verify-backup opens the file with SQLite, so it needs a real file rather than a pipe — run it wherever the backup now lives:

nezal-api verify-backup --in "nezal-$(date +%F).db.enc" --key-file ./backup.key

Inside the cluster the key is found automatically on the data volume, so kubectl exec -n nezal-system deploy/nezal-api -- /nezal-api verify-backup --in … needs no --key-file. verify-backup names the key each file was made with, so an unlabelled backup found later can be matched to the right one.

The API image is distroless — /nezal-api is the only executable in it — so the check runs outside the cluster, on the copy you have just taken. The scheduled CronJob verifies each backup it writes, which is the path to prefer.

A backup nobody has opened is a hope rather than a backup.

backup uses SQLite's VACUUM INTO, which is safe while the database is being written to. The chart ships a CronJob that does all of this on a schedule, writing to its own claim; see backup in values.yaml. That is the path to prefer — the manual commands above are for a backup taken before a deliberate change.

Restoring is deliberately a documented sequence rather than a subcommand, because a restore command that can be run against a live database is a way to destroy one:

1. Scale the API to zero: kubectl scale -n nezal-system deploy/nezal-api --replicas=0.

2. Copy the backup over nezal.db in the data directory, from a pod mounting the same claim. With the API scaled to zero nothing else holds the claim, so a short-lived pod mounting nezal-api-data is enough:

   kubectl cp ./nezal-2026-09-11.db nezal-system/<restore-pod>:/var/lib/nezal/nezal.db
   

   Remove any nezal.db-wal and nezal.db-shm beside it first: they belong to the database you are replacing, and SQLite will otherwise try to replay them over the restored file.

3. Scale back to one and check the audit chain verifies in the dashboard.

Baselines, Environments and Policies are Kubernetes objects; back them up with whatever your cluster already uses. They are also rebuilt from the repository, so the database is the part that matters.