The whole path first, then the five stages between the kernel and PostgreSQL. Each stage exists because of a known failure mode, and each one has a declared cost. What the pair does not do is at the bottom, in the same words we would use after an incident.
Diagram description: the kernel hands the audit records to auditd, which writes each line to corvo stdin. corvo reassembles the fragments of one command, resolves the numeric identifiers into names and writes the event to the queue on disk. The queue delivers over HTTPS to ninho, which acknowledges with 2xx, persists to PostgreSQL and serves the web UI. Only after that acknowledgement does the offset move.
The reassembler groups the lines by sequence number, resolves uid, gid and auid into names with a 60 s TTL, and closes the event when the final record arrives. A 500 ms ticker closes whatever stayed incomplete, so that no event is stuck in there forever.
stdin (auditd) → line parser (1 MiB buffer)
→ reassembler keyed by sequence
├─ final record arrived → close the event
└─ 500 ms ticker fired → close what it has
→ resolve uid→name (cache, 60 s TTL)
→ queue on diskbufio.Scanner fails silently when it overflows, and what overflows first is exactly the long EXECVE, the most interesting record in an audit. corvo reads with 1 MiB.16 MiB JSONL segments plus one offset file. It survives a process restart and a machine reboot, and repairs itself on open if a crash cut a line in half. During an incident you read the queue with the tools you already have.
$ ls /var/lib/corvo/queue
dead/ offset seg-00000040.jsonl seg-00000041.jsonl seg-00000042.jsonl
$ tail -1 seg-00000042.jsonl | jq -r '.event.process.args | join(" ")'
sudo systemctl restart auditd
$ cat offset
seg-00000042.jsonl 11534336When the queue fills, it deletes the oldest segment. It loses the past and preserves the present.
It stops consuming stdin. No event is dropped by corvo, but any eventual loss becomes the kernel's loss and shows up in the metric. Strict compliance is worth that trade, and the cost is real.
Exponential backoff with jitter, with no attempt limit by default, because giving up means losing audit. The jitter keeps a whole fleet from hitting the endpoint together when it comes back.
| response | what corvo does | why |
|---|---|---|
| 2xx | acknowledges and advances the offset | the only way a batch leaves the queue |
| 429 · 503 | retry honoring Retry-After | infrastructure busy or down, the data is fine |
| 401 · 403 | retry for 15 min, then dead-letter | usually an expired token, not bad data |
| 400 · 404 · 413 · 422 | immediate dead-letter | does not improve on repeat and would freeze the queue |
| no response | infinite retry with jitter | the network comes back, and giving up means losing audit |
A malformed batch in infinite retry would freeze the queue forever. It moves to queue/dead/ with the server response beside it, and comes back once the cause is fixed.
$ ls /var/lib/corvo/queue/dead batch-20260904T123010Z.jsonl batch-20260904T123010Z.response.txt $ cat batch-20260904T123010Z.response.txt HTTP 422 · field "event.sequence" missing in 3 of 128 events $ corvo replay /var/lib/corvo/queue/dead/batch-20260904T123010Z.jsonl
The offset can lag behind, and then the same batch is sent twice. Every event carries a stable id, and the receiver has to treat a repeated id as idempotent. The contract requires it.
srv-web-01-2841-1756938610123000000 └── host ──┘ └seq┘ └── timestamp in ns ──┘
corvo only POSTs when it has an event to deliver. On an idle machine the silence is legitimate, and the server cannot tell legitimate silence from a killed agent.
The queue is a second copy on disk, so the written volume roughly doubles. It keeps uncompressed JSONL even with gzip: true. That tradeoff is deliberate, and the disk cost is real.
corvo does not terminate TLS, and neither does ninho. The reverse proxy in front does it. Out of scope in ninho: user accounts, multi-tenancy, caching and an alerting system.