How-to guides

Deploy to production

Goal: ninho serving an audited fleet, with sign-in required, sessions that survive a restart, and a database the ingest path cannot destroy.

The deliverable of make build is a single static binary with the UI inside it. There is nothing to install alongside it but PostgreSQL.

1. Prepare the database

Create the database and the owner, then let the server do the rest. Migrations and partitions are applied at startup.

For the split roles that keep the trail append-only, follow Split the database roles. It is worth the ten minutes: without it, the identity that writes the trail can also erase it.

2. Write the environment

NINHO_DSN=postgres://ninho_owner:...@db.internal:5432/ninho?sslmode=require
NINHO_INGEST_DSN=postgres://ninho_app:...@db.internal:5432/ninho?sslmode=require
NINHO_READ_DSN=postgres://ninho_web:...@db.internal:5432/ninho?sslmode=require

NINHO_ADDR=:8080
NINHO_INGEST_TOKEN=<the same token as in corvo.yaml>

NINHO_SESSION_KEY=<openssl rand -base64 32>
NINHO_SESSION_HOURS=12

NINHO_LDAP_URL=ldaps://dir.example.com:636
NINHO_LDAP_BASE_DN=ou=people,dc=example,dc=com
NINHO_LDAP_BIND_DN=cn=readonly,dc=example,dc=com
NINHO_LDAP_BIND_PASSWORD=<the service account's password>
NINHO_LDAP_GROUP_DN=cn=ninho,ou=groups,dc=example,dc=com

Export them for real: a systemd EnvironmentFile=, an orchestrator secret, a vault. .env is a development convenience and is not read here.

Three of these decide whether the deployment is sound:

• NINHO_SESSION_KEY. Without it, each process invents a random key at startup. Sessions then die on every restart, and two instances never agree, so behind a load balancer the sign-in succeeds and every screen after it answers 401. Set the same value on every instance.
• NINHO_LDAP_URL. Without it the UI and the read API are open. An audit trail readable by anyone who can reach the port is worse than no trail, because it is trusted.
• NINHO_INGEST_TOKEN. Without it any host can write into the trail.

Startup warns about each of the three. Read the first lines of the log after every deploy.

3. Run it

Under systemd, ninho needs no special treatment: it is one process, it logs to stderr, and it shuts down cleanly on SIGTERM with a 15 second grace period for in-flight requests.

[Service]
EnvironmentFile=/etc/ninho/env
ExecStart=/usr/local/bin/ninho
User=ninho
Restart=always

The container image is scratch plus the binary, the CA bundle, and a non-root user (10001:10001), exposing 8080. It takes the same environment variables and nothing else.

4. Put TLS in front of it

ninho speaks plain HTTP and expects to sit behind a reverse proxy that terminates TLS. Two things to get right:

Do not buffer /api/stream. It is Server-Sent Events. A proxy that buffers responses will hold the live tail until the buffer fills. In nginx, proxy_buffering off for that location. The 20 second heartbeat handles idle timeouts on its own.

Do not impose a read timeout on that same route. The stream is meant to stay open. ninho deliberately sets no write timeout for this reason.

Note that the session cookie is marked Secure only when ninho itself terminated the TLS. Behind a proxy the server sees plain HTTP and omits the flag, so the cookie will also be sent over a plain HTTP request to the same host. If you serve the site on both schemes, redirect HTTP to HTTPS at the proxy and enable HSTS.

5. Point the agents at it

In each host's corvo.yaml:

endpoint:
  url: https://ninho.example.com/api/ingest
  token_file: /etc/corvo/token
  gzip: true

Check it with corvo check, which sends a synthetic probe. It gets a 2xx and is not persisted, so a successful check leaves no trace in the trail.

6. Verify the deployment

# The three warnings must be absent from the startup log.
journalctl -u ninho | grep WARN

# Sign-in works and the session survives a second request.
curl -s -c jar -X POST https://ninho.example.com/api/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"you","password":"..."}'
curl -s -b jar -o /dev/null -w '%{http_code}\n' \
  'https://ninho.example.com/api/events?limit=1'   # expect 200

# An agent has checked in.
curl -s -b jar https://ninho.example.com/api/agents

A 401 on the second call, after a 200 on the first, is almost always NINHO_SESSION_KEY. See Troubleshooting.

Running more than one instance

ninho keeps no state in memory that matters across requests, so it scales horizontally with two conditions:

1. NINHO_SESSION_KEY identical on every instance. This is not optional.
2. Every instance polls for the live tail every 2 seconds when at least one client is connected to it. That cost is per instance, not per tab.

Migrations are safe with several instances starting at once: an advisory lock serialises them.

Configure LDAP sign-in

Goal: only people the directory knows can read the trail, and optionally only those in one group.

There is no user registry in ninho. The directory decides, and nothing about who may sign in is stored here.

The minimum

NINHO_LDAP_URL=ldaps://dir.example.com:636
NINHO_LDAP_BASE_DN=ou=people,dc=example,dc=com
NINHO_LDAP_BIND_DN=cn=readonly,dc=example,dc=com
NINHO_LDAP_BIND_PASSWORD=<its password>

The service account is required. The lookup never falls back to an anonymous search, and the server refuses to start without those two variables. A directory that answers anonymous reads today can stop answering them after a hardening pass, and the failure would reach users as "wrong password", all at once, with nothing in the log to say otherwise.

The account needs one privilege: reading the entries under NINHO_LDAP_BASE_DN, and the group object when you configure one. It never writes.

Restart, and check the log:

level=INFO msg="LDAP authentication enabled" url=ldaps://... base=ou=people,... group=

Choosing what people type

NINHO_LDAP_FILTER defaults to (uid=%s). %s receives the typed username, escaped, so a username containing (, ), or * cannot rewrite the filter.

To let people sign in with their email:

NINHO_LDAP_FILTER=(mail=%s)

To accept either:

NINHO_LDAP_FILTER=(|(uid=%s)(mail=%s))

The search must match exactly one entry. Zero or more than one is refused as invalid credentials, which is the correct answer for an ambiguous filter.

Restricting sign-in to one group

NINHO_LDAP_GROUP_DN=cn=ninho,ou=groups,dc=example,dc=com

After the password is verified, the group object itself is asked whether it lists the user. It covers both conventions:

The uid compared against memberUid comes from the directory entry, not from what was typed, so the check keeps working with a filter that matches on mail or cn.

The group search runs as the service account, not as the person who just signed in, who is not guaranteed to be allowed to read the group. If it ran as them, a restrictive ACL would answer "no entries", which is indistinguishable from not being a member.

A posixGroup that is only the user's primary group does not count. If someone's gidNumber points at the group but there is no memberUid entry for them, they are refused. Add them to memberUid explicitly.

Verify with someone inside the group and someone outside it. Both get the same 401 and the same message. The log says which happened.

A directory without TLS

NINHO_LDAP_URL=ldap://dir.example.com:389

With an ldap:// URL, the connection is upgraded with StartTLS, and a failure to upgrade fails the sign-in. To speak plain LDAP anyway:

NINHO_LDAP_INSECURE=true

The signing-in password then crosses the network in the clear. Development and trusted networks only.

Letting a script in without a session

NINHO_READ_TOKEN=<a long random string>

See Query the API from a script. This bypasses the directory entirely, so treat it as a credential of the same weight.

When sign-in fails

The response is always the same 401 and the same text, on purpose: an unknown user, a wrong password, and a non-member must not be distinguishable to whoever is guessing. The server log carries the real reason:

level=WARN msg="sign-in refused" username=ana error="..."

A sign-in that hangs and then fails after ten seconds is the deadline that covers the whole exchange, including the dial. It means the directory is not answering.

Split the database roles

Goal: the identity that writes the trail cannot erase it, and the identity that serves the UI cannot write to it.

This is the scenario the separation exists for: the endpoint token leaks, an attacker can now POST anything into /api/ingest, and the trail still cannot be deleted or rewritten, because the connection behind that path has no privilege to do so.

1. The roles already exist

002_roles.sql created them when the server first started:

They are born NOLOGIN and passwordless, because a password in versioned SQL is a leaked password. The UPDATE on agent is exactly what the per-batch upsert needs, and nothing more.

If the database owner could not create roles, the migration logged a notice and carried on. Create them with a superuser:

CREATE ROLE ninho_ingest NOLOGIN;
CREATE ROLE ninho_reader NOLOGIN;
GRANT INSERT         ON audit_event TO ninho_ingest;
GRANT INSERT, UPDATE ON agent       TO ninho_ingest;
GRANT SELECT ON audit_event, agent, v_command TO ninho_reader;

2. Create users and grant the roles

CREATE USER ninho_app LOGIN PASSWORD '...';  GRANT ninho_ingest TO ninho_app;
CREATE USER ninho_web LOGIN PASSWORD '...';  GRANT ninho_reader TO ninho_web;

3. Point the connections at them

NINHO_DSN=postgres://ninho_owner:...@db:5432/ninho     # owner, migrations, partitions
NINHO_INGEST_DSN=postgres://ninho_app:...@db:5432/ninho
NINHO_READ_DSN=postgres://ninho_web:...@db:5432/ninho

NINHO_DSN stays the owner. It is used for migrations and for creating partitions, both of which need DDL, and for nothing else once the server is up. When the other two are unset they fall back to it, which is the development default and is exactly what you are moving away from here.

Restart.

4. Verify

The check that matters is the negative one:

psql "postgres://ninho_app:...@db:5432/ninho" \
  -c "DELETE FROM audit_event WHERE true;"
# ERROR:  permission denied for table audit_event

psql "postgres://ninho_web:...@db:5432/ninho" \
  -c "INSERT INTO agent (host, last_seen) VALUES ('x', now());"
# ERROR:  permission denied for table agent

Then confirm the normal paths still work: send a batch and load a screen.

A note on new partitions

Privileges granted on a partitioned table propagate to partitions created afterwards, so the monthly partitions the server creates need no extra grants. If you create a partition by hand as a different owner, check the grants on it.

Query the API from a script

Goal: read the trail from cron, a notebook, or an incident response script, without a browser and without a person's credentials.

Set a read token

NINHO_READ_TOKEN=$(openssl rand -base64 32)

Restart. This token bypasses the directory entirely and grants read access to every route the UI uses. Treat it with the same care as a password, and give it its own value per consumer if you need to revoke one without disturbing the others. Requests made with it are attributed to token, not to a person.

Use it

NINHO=https://ninho.example.com
AUTH="Authorization: Bearer $NINHO_READ_TOKEN"

curl -s -H "$AUTH" "$NINHO/api/agents" | jq

Alternatively, sign in as a person and keep the cookie:

curl -s -c jar -X POST "$NINHO/api/login" \
  -H 'Content-Type: application/json' \
  -d '{"username":"ana","password":"..."}'
curl -s -b jar "$NINHO/api/agents" | jq

The cookie expires after NINHO_SESSION_HOURS, so a long-running script wants the token.

Recipes

Everything one login session did

curl -s -H "$AUTH" "$NINHO/api/sessions/web-01/4711" | jq -r \
  '.events[] | "\(.occurred_at)  \(.actor // .auid)  \(.args_text)"'

Every use of a command in a window

curl -s -H "$AUTH" --get "$NINHO/api/events" \
  --data-urlencode 'q=userdel' \
  --data-urlencode 'from=2026-01-01T00:00:00Z' \
  --data-urlencode 'to=2026-02-01T00:00:00Z' | jq

q is a case-insensitive substring of args_text, which is process.args joined by a space. It never matches process.title.

Paging through a large result

The API uses keyset pagination: a full page comes back with next_cursor_at and next_cursor_id, and you hand both back. They travel together, because without the id there is no tiebreaker between two events in the same instant.

at=""; id=""
while :; do
  page=$(curl -s -H "$AUTH" --get "$NINHO/api/events" \
    --data-urlencode "limit=500" \
    ${at:+--data-urlencode "cursor_at=$at"} \
    ${id:+--data-urlencode "cursor_id=$id"})
  echo "$page" | jq -c '.events[]'
  at=$(echo "$page" | jq -r '.next_cursor_at // empty')
  id=$(echo "$page" | jq -r '.next_cursor_id // empty')
  [ -n "$at" ] || break
done

One event, with its full raw JSON

occurred_at is required: the table is partitioned by it and the key is the pair.

curl -s -H "$AUTH" \
  "$NINHO/api/events/tutorial-0001?occurred_at=2026-01-15T10:30:00Z" | jq .event

Agents that have gone quiet

The alert worth wiring into monitoring. Anything with a silence_seconds beyond your threshold is a machine that stopped reporting.

curl -s -H "$AUTH" "$NINHO/api/agents" \
  | jq -r '.agents[] | select(.silence_seconds > 3600)
           | "\(.host) silent for \(.silence_seconds)s"'

Read Reading the trail before setting a threshold: on an idle machine, silence is legitimate.

Following the live tail

curl -sN -H "$AUTH" "$NINHO/api/stream"

Server-sent events, one data: line per event, plus a : ping comment every 20 seconds. The stream starts at the moment you connect and does not replay history.

Status codes to handle

Retry on 503, that one is always our side. Fix the request on a 4xx. See The contract and the status codes.

Build and release

Build a binary

make build

Runs Vite, then go build, and leaves ./ninho: one static file, with the UI embedded, no CGO. Copy it to the server. Building needs Go 1.23 or newer and Node 20 or newer. Running needs neither.

The version is stamped in at link time:

• On an exact git tag, the version is that tag.
• Off a tag, it is the contents of .version plus git describe, for example 0.1.0+8db552f-dirty.

It appears in the startup log, in /api/me, and in the UI footer. A footer reading dev means the binary was built without the ldflags, which is what a bare go build ./cmd/ninho does.

Cut a release

1. Bump .version (it holds a bare 0.1.0, no v).
2. Commit that.
3. Tag it as v<.version>, matching the existing v0.1.0.
4. Push the commit and the tag.

Everything built from that commit now carries the tag as its version, because git describe --exact-match finds it.

Build the image

make docker

Nothing is built locally. This calls the Quay trigger's webhook with the current commit sha, and Quay fetches and builds it. Two variables are needed in .env, both from the trigger's "View Credentials" page, where the webhook URL looks like https://$token:PASSWORD@quay.io/webhooks/push/trigger/UUID:

NINHO_QUAY_TOKEN=PASSWORD
NINHO_QUAY_TRIGGER=UUID

The basic-auth user is the literal string $token. Only the password and the UUID go in .env.

The target refuses to fire when the commit is not on a remote branch. Quay builds from what it can fetch, so triggering on an unpushed commit would build something else entirely and label it with your sha. Push first.

When HEAD is on a tag, the ref sent is refs/tags/<tag> and the image is built and labelled for the release.

What the image contains

A three-stage build ending in scratch:

1. Node builds the UI into web/dist.
2. Go builds the static binary with web/dist already in place for the //go:embed.
3. The final layer holds the binary, the CA bundle (needed for LDAPS and for PostgreSQL over TLS), and nothing else.

It runs as 10001:10001, exposes 8080, and takes the same environment variables as the binary. There is no shell in it, so debugging is done through logs and the API, not by exec'ing in.

Regenerate the database code

make generate

The SQL is hand-written in internal/store/query/, and sqlc generates internal/store/db/ from it. The generated code is committed, so whoever builds the binary does not need sqlc installed. Run this after touching any .sql file in that directory, and commit the result with it.

Troubleshooting

Start with the first ten lines of the startup log. Three warnings there describe the three ways a deployment is commonly wrong, and they are printed without being asked for.

level=WARN msg="NINHO_SESSION_KEY is empty, sessions die on restart and are not shared between instances"
level=WARN msg="NINHO_INGEST_TOKEN is empty, ingest accepts any agent"
level=WARN msg="NINHO_LDAP_URL is empty, the UI and the read API are open"

Sign-in works, then every screen says "not authenticated"

The session cookie is stateless and signed with HMAC-SHA256 using NINHO_SESSION_KEY. The sign-in itself needs no key, so it succeeds even when the key situation is broken. The next request is where it shows.

The first one is the usual answer, and it produces exactly this symptom: the sign-in lands on one replica and the next request lands on another, whose key is different, so the signature does not verify.

The server refuses to start

Failing at boot is deliberate for the LDAP case: the alternative is a server that runs and refuses every sign-in with "wrong password".

Nobody can sign in

See the table at the end of Configure LDAP sign-in. The response is always the same 401. The log line msg="sign-in refused" carries the real reason.

A sign-in that takes ten seconds and then fails is the deadline that covers the whole LDAP exchange: the directory is not answering.

One person cannot sign in, others can

Almost always NINHO_LDAP_GROUP_DN. Check that they are listed in the group, by memberUid for a posixGroup or by member or uniqueMember for the DN conventions. Being in the group only as its gidNumber primary group does not count.

The other possibility is a search filter matching more than one entry for them, which is refused: (uid=%s) on a directory with duplicate uids, for instance.

Events are not arriving

Work from the agent inwards.

# 1. Does the endpoint answer at all?
curl -i -X POST https://ninho.example.com/api/ingest \
  -H 'Authorization: Bearer <token>' -d '[]'
# expect 204

Then check the server log for msg="batch received", which reports how many events and probes each batch carried. A batch of nothing but probes is corvo check doing its job.

If batches arrive but nothing appears on screen, look at the times: the Events screen filters by occurred_at, so a host with a badly wrong clock lands its events far from now. The Agents screen and the live tail both use received_at and will still show it.

The live tail shows nothing

• It only shows what arrives after you connect. It never replays history.
• Behind a reverse proxy, response buffering holds the stream. Turn it off for /api/stream.
• Polling is every 2 seconds, so expect up to that much delay.
• A client that cannot keep up drops messages rather than blocking the others. A tab left in the background on a busy fleet will have gaps.

"partition not created" at startup

level=WARN msg="partition not created" name=audit_event_2026_03 error=...

The default partition already holds rows in that month's range, usually because events arrived from a host with a wrong clock before the monthly partition existed. Startup carries on: the rows are safe in audit_event_default and queries still find them.

Moving them is manual, and worth doing only if the default partition has grown large enough to hurt query times. Nothing is lost either way.

Queries are slow

Check what is actually being filtered. Substring search on q uses the trigram index on args_text, which is fast for a substring of three characters or more and much less so below that. Filters on host, auid, record_type, and a time range all have indexes behind them.

Pagination is keyset, never OFFSET, so page 500 costs what page 1 costs. If a client is paging by throwing away rows, fix the client.

The footer says "dev"

The binary was built without the version ldflags. make build and make dev both inject it. A bare go build ./cmd/ninho does not. See Build and release.