Explanation

Why a plugin instead of netlink

The Linux kernel delivers audit events to exactly one registered process, identified by PID. That single fact shapes corvo's whole design.

If corvo registered itself on the audit netlink socket in unicast mode, it would take that registration away from auditd. auditd would stop receiving events. Every other tool that expects auditd to be writing /var/log/audit/audit.log would quietly get nothing. You would have traded one collector for another instead of adding one.

There is a multicast mode that does coexist with auditd. Several readers can attach to it at once. The catch is that it has no backpressure. When the reader falls behind, the kernel drops events on that socket and does not tell anyone. For a debugging session that is an acceptable trade, which is why it exists in corvo as corvo tail, marked lossy in the help text. For an audit trail, where a missing event is indistinguishable from "nothing happened", it is not acceptable at all.

The plugin interface is the way in that does not require any of those compromises. auditd stays the one registered process. It owns the audit rules and the kernel buffer. It writes every event, as text, to corvo's stdin. If corvo is slow, auditd buffers. If corvo dies, auditd restarts it. corvo becomes one more consumer of a stream auditd already manages well, rather than a replacement for the part of the system that manages it.

The cost of this choice is that corvo cannot outlive a decision auditd makes. When auditd reloads its plugins it sends SIGHUP, and corvo has to reload its config in place. When auditd restarts, it re-execs the plugin, and there is a brief gap. The durable on-disk queue is what makes that gap free: whatever was captured is on disk, and delivery picks up where it left off. See How an event survives.

How an event survives

corvo's job is to not lose an audit event. Everything about its structure follows from that. There are five places a fragment could vanish between the kernel and the endpoint, and corvo puts a barrier at each one.

The five barriers

auditd owns the buffer, not corvo. If corvo slows down or crashes, auditd holds events in its own queue and restarts the plugin. corvo never has to be the thing standing between a fast kernel and a slow network.

A 1 MiB line buffer. The kernel emits long EXECVE records for commands with many arguments. Go's bufio.Scanner defaults to a 64 KiB line limit and fails silently past it, which would drop exactly the long command lines that matter most in a trail. corvo raises the limit to 1 MiB.

A Maintain() ticker every 500 ms. Regrouping five records into one event means holding partial groups in memory until they complete. If the last record of a group never arrives, that group would sit there forever. The ticker flushes stale groups so a half-assembled event still gets delivered.

The on-disk queue sits before the network. Capture writes to disk and returns. It never blocks waiting for the endpoint to answer. This is the barrier that turns an endpoint outage from an incident into a non-event.

Shutdown follows the data flow. On SIGTERM, corvo stops reading stdin first, then lets the pipeline drain whatever was half assembled, then gives the sink a deadline to empty the queue, then closes and syncs the queue. Anything that does not fit in the deadline stays on disk for the next start.

The queue

/var/lib/corvo/queue/
  seg-00000042.jsonl   active segment, rotates at 16 MiB
  seg-00000043.jsonl
  offset               "seg-00000042 1048576"
  dead/                permanently rejected batches

The format is one JSON object per line. That choice buys three things.

A write interrupted by a crash is repaired on open. Whatever comes after the last newline is a partial record, so corvo truncates it and moves on. No separate journal, no checksum, no recovery mode.

The offset only advances after a 2xx. It is written to a temp file and renamed, which is atomic on every filesystem corvo targets. The offset can lag behind reality, which causes a resend, but it can never get ahead, which would skip an event.

You can read the queue during an incident. tail -1 seg-00000042.jsonl | jq shows you the next event waiting to go out, in the same shape it will reach the endpoint.

At-least-once, and what that asks of the server

Because the offset advances only after confirmation, a crash between the endpoint's 2xx and the offset write means the whole batch is resent on the next start. That is deliberate. Resending an audit event is harmless. Losing one is not.

So every event carries a stable id, built as <host>-<sequence>-<timestamp>, and the server has to treat a repeated id as idempotent. An INSERT ... ON CONFLICT (id) DO NOTHING covers it. This is not advice, it is part of the delivery contract. Without it, every ungraceful restart duplicates data.

When the queue fills

An outage makes the queue grow. What happens when it hits queue.max_bytes depends on queue.on_full:

• drop_oldest deletes the oldest segment whole and stays alive. You lose the oldest events, you keep collecting the newest, and corvo_events_dropped_total records the loss. This is the default because a live agent missing an hour of history beats a dead agent.
• block stops consuming stdin. auditd buffers, and if the outage outlasts auditd's own buffer the kernel drops at the source, which shows up in corvo_events_lost_total. corvo itself drops nothing. This is the right choice for strict compliance, and it has a real cost: back-pressure all the way to the kernel.

When the endpoint is down

Capture is untouched. The POST fails, the batch is not confirmed, the offset does not move, nothing is erased. corvo retries with exponential backoff and jitter, from 1s up to 5 minutes, spread plus or minus 30 percent so a whole fleet does not stampede the endpoint the moment it recovers. There is no attempt limit by default, because giving up means losing a trail. 429 and 503 honour Retry-After in both formats, capped by retry.max so an absurd header cannot freeze the agent.

The queue grows on disk while this happens. corvo_queue_depth_bytes and corvo_send_failures_total climb. That is the alert that fires before any loss. Once the endpoint is back, the backlog drains in full and in order. A machine reboot in the middle changes nothing, the segments and the offset are on disk.

What retry cannot fix

A 400, 404, 413, or 422 will not improve on retry, and a malformed batch retried forever would freeze the queue behind it. Those go to queue/dead/, with the response body alongside in a .err file, corvo_events_dead_lettered_total climbs, and the pipeline moves on. Nothing is erased, and you can replay them.

401 and 403 are the exception inside the exception. They are usually an expired token, not bad data, so they stay in retry for retry.auth_retry_window (15 minutes by default), giving a SIGHUP with a fresh token a chance to fix it, and only then become a dead letter.

auid, uid, and the chain of accountability

An audit trail answers one question above all others: who did this. On Linux the answer lives in two numbers that corvo carries in every event's user.ids, and the difference between them is the whole point.

auid  the audit login uid. Who logged in. Survives every su and sudo.
uid   the effective user. Under which identity the command actually ran.

auid=1000, uid=0 means "user 1000, acting as root". Keep only uid and that record says "root ran X", which on a machine with five administrators holds nobody accountable. The auid is set once, by pam_loginuid, at login, and the kernel carries it through every privilege change after that. It is the thread that ties a root shell back to a person.

corvo puts the resolved names in summary.actor: primary is auid turned into a name, secondary is uid. The raw numbers stay in user.ids so the server can key on them.

unset means a system process

An auid of unset, or the raw form 4294967295, means there is no login behind this process. systemd and everything it starts look like this. The audit rules use it as the line between people and daemons: -F auid>=1000 and -F auid=0 both exclude unset, so a cron job or a service restart does not land in the command trail.

unset on an interactive session is a problem

If a real login shows up with auid=unset, something skipped pam_loginuid. This happens on a hypervisor console, on agetty with no PAM, inside docker exec. No audit rule with an auid filter will match that session, not even the root rule. The natural conclusion is that corvo is broken, and it is not: there is simply no event to deliver.

The fix is in PAM, not in a rule. Adding a rule with no auid filter to "catch" these sessions would also catch every daemon on the box. corvo check reads /proc/self/loginuid and tells you the current session's auid and which rule it matches, so you can spot this before it confuses an investigation.

Names resolve only as far as the host allows

corvo's binary is static, so its own lookups read /etc/passwd. A user from SSSD, LDAP, or AD gets no name from corvo, and summary.actor stays numeric, unless auditd runs with log_format = ENRICHED, in which case auditd resolves the name (it links libc) and corvo passes it through. The server should treat a missing name as normal and always key on ids. See Resolve names for domain users.

sudo records the intent, syscall records the act

When sudo runs a command you get two events. user_cmd is what the person asked sudo to do, recorded as typed. The syscall event is what actually executed. Keep both. The gap between "what was requested" and "what ran" is exactly where a sudoers misconfiguration or abuse shows up.

What corvo costs

The numbers here are measured, not estimated. A replay of 50 thousand execve events (300 thousand lines, 49 MB of audit.log) on an i7-11800H, from parse to queue write:

On a modest ARM VM, expect two to three times the CPU per event. Reproduce with go test ./internal/pipeline -bench Pipeline.

In practice an interactive session generates 5 to 50 execve per second, well under 1 percent of a core. A burst from dnf update or make -j hits hundreds per second for a few seconds, and the on-disk queue absorbs that, not memory.

Where the time actually goes

Writing a line to corvo's stdin is a 1 to 2 microsecond memcpy. It is not the bottleneck. The real costs, in order, and three of the four are not corvo's:

1. The audit rules. -S execve makes the kernel generate a record on every exec on the machine. Anyone who writes -S all tanks performance with no plugin involved at all.
2. auditd writing audit.log. Every event is written to disk by auditd before corvo ever sees it.
3. Parsing. ParseLogLine is the largest cost inside corvo. About 27 percent of the agent's CPU is in regexp, inside go-libaudit's parser. Nothing corvo does works around that short of a different parser.
4. uid-to-name resolution, cached with a 60 second TTL. Without the cache this would be a getpwuid per event, and on a machine with SSSD or LDAP that is a directory query per command. With log_format = ENRICHED corvo reuses the name auditd already resolved and only falls back to its own /etc/passwd lookup for what is missing.
5. JSON and gzip, once per batch, not per event.

Disk

The default queue.fsync: none is the cheapest I/O policy: a sequential append flushed in batches by the kernel. It survives a process crash, kill -9, and an auditd restart. A power loss loses the window that had not been flushed yet. batch and always exist for when the requirement changes.

Under normal conditions the queue holds almost nothing. Segments are consumed and deleted within seconds. It only grows during an outage, at 1.5 to 2.5 KB per event:

Two honest caveats. The queue is a second copy: auditd already wrote the event to audit.log, so the bytes written to disk roughly double. And the queue stores uncompressed JSONL even with gzip: true, because the gzip happens on the POST body. That is a deliberate trade. Being able to run tail | jq on a segment during an incident is worth the space, and the space only exists during an outage anyway.