How-to guides

Install from a package

The RPM and the .deb come from a single deploy/nfpm.yaml built with nfpm, which does not need rpmbuild or dpkg-buildpackage on the build machine. Both architectures come out of an x86_64 host with no emulation, because the build is static (CGO_ENABLED=0).

Build the package

make rpm
#   dist/corvo-0.1.2-1.x86_64.rpm    x86_64 servers
#   dist/corvo-0.1.2-1.aarch64.rpm   Raspberry Pi 3/4/5, 64-bit

make deb
#   dist/corvo_0.1.2_amd64.deb
#   dist/corvo_0.1.2_arm64.deb

Install it

On AlmaLinux, RHEL, or Fedora:

sudo dnf install ./dist/corvo-*.$(uname -m).rpm

On Debian or Ubuntu:

sudo apt install ./dist/corvo_*_$(dpkg --print-architecture).deb

Both put the binary at /usr/bin/corvo, drop the plugin file at /etc/audit/plugins.d/corvo.conf, drop the rules at /etc/audit/rules.d/corvo.rules, and create /var/lib/corvo/queue. The three config files are marked so an upgrade never overwrites what you edited. The RPM applies the SELinux labels in its %post and prints the steps that are left.

The .deb ships no AppArmor profile. auditd on Debian and Ubuntu does not confine its plugins, so the process runs as root with no LSM policy in the way. On AlmaLinux the SELinux module is doing real work and a denial there is the usual failure.

Finish the setup

The package cannot know your endpoint or your token. Do these by hand:

# 1. Token, readable only by root. The plugin runs as auditd's child, so as root.
printf '%s' "$TOKEN" | sudo tee /etc/corvo/token >/dev/null
sudo chmod 0600 /etc/corvo/token

# 2. Endpoint URL in /etc/corvo/corvo.yaml. See:
#    docs/how-to/point-corvo-at-your-endpoint.md

# 3. Validate before touching auditd. Nonzero exit on any problem.
sudo corvo check

# 4. Load the rules and restart auditd.
sudo augenrules --load
sudo service auditd restart

If the machine has users from SSSD, LDAP, or AD, also set log_format = ENRICHED in /etc/audit/auditd.conf. Without it those users reach the endpoint as a bare uid with no name. corvo check warns when it is missing. Full explanation in Resolve names for domain users.

Use service auditd restart, not systemctl restart auditd. Some RHEL and AlmaLinux builds refuse the systemctl form.

Confirm it is running

sudo auditctl -l | grep cmds          # the rules are loaded
pgrep -af '/usr/bin/corvo'            # corvo shows up as auditd's child
sudo journalctl -u auditd -n 50       # corvo's startup line lands here

Upgrades and removal

An upgrade runs service auditd restart so the new binary replaces the running plugin. auditd only re-execs a plugin on a restart, and the durable queue means the gap costs nothing.

Uninstalling takes the plugin down before the binary disappears, so auditd does not burn through its restart budget on a missing executable. It keeps /var/lib/corvo/queue, because there may be an undelivered event in there. Check with corvo check before you erase it.

One thing removal does not fix: the package deletes corvo.rules, but the rule stays loaded in the kernel until the next augenrules --load or reboot. Until then auditd keeps generating those events with nobody to deliver them. Reloading the full rule set from an uninstall script would be worse, because augenrules --load starts by flushing every rule on the machine.

Install from source

Use this when you cannot install a package, or you want corvo somewhere other than /usr/bin. You need Go 1.22 or later. Nothing else, the build pulls its two dependencies and cross-compiles without a C toolchain.

Install the files

sudo make install                 # PREFIX=/usr/local by default

This copies the binary to $PREFIX/bin/corvo, installs the example corvo.yaml at /etc/corvo/, the plugin file at /etc/audit/plugins.d/corvo.conf, the rules at /etc/audit/rules.d/corvo.rules, and creates /var/lib/corvo/queue.

The install target rewrites the path line in corvo.conf to match your PREFIX, so auditd starts the binary from where you actually put it.

To install to a staging root for packaging, pass DESTDIR:

make install DESTDIR=/tmp/stage PREFIX=/usr

Finish the setup

# 1. Token, 0600 root.
printf '%s' "$TOKEN" | sudo tee /etc/corvo/token >/dev/null
sudo chmod 0600 /etc/corvo/token

# 2. Endpoint in /etc/corvo/corvo.yaml.
#    See docs/how-to/point-corvo-at-your-endpoint.md

# 3. SELinux, on AlmaLinux and RHEL. The package does this itself; a source
#    install does not. See docs/how-to/diagnose-selinux-denials.md
sudo semanage fcontext -a -t auditd_log_t '/var/lib/corvo(/.*)?'
sudo restorecon -Rv /var/lib/corvo

# 4. Validate.
sudo corvo check

# 5. Load rules, restart auditd.
sudo augenrules --load
sudo service auditd restart

If the machine has users from SSSD, LDAP, or AD, also set log_format = ENRICHED in /etc/audit/auditd.conf, or those users reach the endpoint as a bare uid with no name. corvo check warns when it is missing. See Resolve names for domain users.

semanage lives in policycoreutils-python-utils, which is not on a minimal AlmaLinux 9 install. Without it, sudo chcon -R -t auditd_log_t /var/lib/corvo works right away but does not survive a full filesystem relabel.

Confirm it is running

sudo auditctl -l | grep cmds
pgrep -af 'corvo'
sudo journalctl -t corvo -n 50

Remove it

sudo make uninstall

This pulls the plugin file, the rules file, and the binary. It leaves /var/lib/corvo/queue alone. As with the package, the -k cmds rule stays loaded in the kernel until augenrules --load or a reboot.

Point corvo at your endpoint

corvo reads its settings from three layers, each overriding the one before: the YAML file, then CORVO_* environment variables, then command-line flags. The commented example is at deploy/corvo.yaml.

The minimum

Edit /etc/corvo/corvo.yaml:

endpoint:
  url: https://logs.example.com/v1/audit
  token_file: /etc/corvo/token   # 0600 root, reread on every SIGHUP
queue:
  dir: /var/lib/corvo/queue

Everything else has a working default. See the configuration reference for the full list.

Add the token

The token never goes inline in the YAML. It lives in its own file, readable only by root:

printf '%s' "$TOKEN" | sudo tee /etc/corvo/token >/dev/null
sudo chmod 0600 /etc/corvo/token

corvo sends it as Authorization: Bearer <token>. The config file's own headers cannot override Authorization, corvo sets that one last.

Use mTLS instead of a token

Drop the token file and give a client certificate:

endpoint:
  url: https://logs.example.com/v1/audit
  tls:
    cert_file: /etc/corvo/client.pem
    key_file: /etc/corvo/client.key

cert_file and key_file go together or not at all. With a client certificate configured, corvo does not send the Authorization header.

Private CA

If the endpoint's certificate is signed by a CA that is not in the system trust store:

endpoint:
  tls:
    ca_file: /etc/pki/ca-trust/source/anchors/my-ca.pem

Leave ca_file empty for a certificate from a public CA. Never set insecure_skip_verify: true outside a throwaway test.

Extra headers

endpoint:
  headers:
    X-Tenant: production

Containers with a useless hostname

If os.Hostname() returns something like a random container id, override the host field in every event:

endpoint:
  host_override: prod-web-01

Check it

sudo corvo check

check validates the YAML, checks the token and its permissions, resolves the endpoint's DNS, negotiates TLS, and sends one synthetic event to confirm the body contract. Add --offline to skip the network part.

Apply a change without a restart

Most of endpoint reloads on SIGHUP. After editing the file:

sudo corvo reload
sudo journalctl -t corvo -n 5      # look for "configuration reloaded"

corvo reload signals every running corvo process. systemctl reload auditd does not work, the auditd unit refuses it. What reloads and what needs a restart is in the configuration reference.

Pick which events ship

auditd hands the plugin every audit event on the machine, not just what corvo's -k cmds rule generated. SELinux AVC records, CRYPTO_KEY_USER from every SSH handshake, SERVICE_START from every systemd unit: all of it arrives at corvo. On a busy server most of that volume is not a command anyone typed. The filter section cuts it.

See what your machine generates

Before you write a list, find out what is actually there:

corvo tail -json | jq -r .event.record_type | sort | uniq -c | sort -rn

tail ignores filter on purpose. It is how you decide what to filter.

Exclude known noise (recommended)

filter:
  exclude:
    - crypto_key_user
    - crypto_session
    - service_start
    - service_stop
    - netfilter_cfg

exclude beats include. An empty include lets through everything not named in exclude, including record types that did not exist when you wrote the list. That is why this form ages well.

Or name exactly what you want

filter:
  include:
    - syscall
    - login
    - user_login
    - user_auth
    - user_acct
    - user_cmd
    - user_start
    - user_end
    - avc

This form has a trap. In a composite event the record_type comes from the first record, not from SYSCALL. An execve that trips a SELinux denial arrives as avc, carrying the full process and paths. An include without avc drops that command. For the same reason, do not put avc in exclude. If AVC volume is the problem, the fix is SELinux policy, not the filter.

Use one list or the other. Filling both only makes sense to say "this whole family, except this one type".

What the filter does not touch

• Kernel-side cuts are cheaper. The -F auid>=1000 filters in the audit rule run inside the kernel, before the record exists. filter runs after corvo has already built the event, so it costs CPU. Use the rule for what the rule can express (a specific user, a whole syscall) and filter for what it cannot (record type).
• A nonexistent record_type stops the process at startup. A misspelled name would otherwise filter silently forever.
• The whole event is dropped, never half. The filter runs after regrouping, so a filtered event never lands in the queue as a fragment.

What the filter cut shows up in corvo_events_filtered_total.

A starting set for command auditing

For the common case, "I want to know what people ran and where they logged in from", the minimal set is syscall, user_cmd, and the session trail around them: login, user_start, user_end, user_auth, user_acct. Without the session records, an execve with auid=1000 does not tell you where that person connected from.

It is worth adding config_change and the account family (add_user, del_user, add_group, and so on). Those are a handful of events per day and exactly the ones an attacker would rather not generate.

What each record type carries is in the event JSON reference and, in full, in design/api-contract.md.

Rotate a token with no downtime

corvo rereads token_file on every SIGHUP. You can swap the token without stopping collection and without restarting auditd.

Steps

# 1. Write the new token over the old file, keeping 0600 root.
printf '%s' "$NEW_TOKEN" | sudo tee /etc/corvo/token >/dev/null
sudo chmod 0600 /etc/corvo/token

# 2. Tell the running plugin to reread its config.
sudo corvo reload

# 3. Confirm.
sudo journalctl -t corvo -n 5      # "configuration reloaded"

That is the whole procedure. The next batch uses the new token.

If the old token expires before you rotate

corvo does not throw away audit events over an auth failure. A 401 or 403 from the endpoint stays in retry for retry.auth_retry_window, 15 minutes by default, on the assumption that it is an expired token and not bad data. During that window the queue grows on disk and nothing is lost. A SIGHUP with a fresh token inside the window fixes it with no dead-letter.

If the window runs out first, the batch goes to queue/dead/<timestamp>-401.jsonl and corvo_events_dead_lettered_total climbs. You can still recover it with corvo replay once auth works again.

To give yourself more room, raise the window:

retry:
  auth_retry_window: 1h

This value reloads on SIGHUP.

Resolve names for SSSD, LDAP, and AD users

corvo's binary is static (CGO_ENABLED=0), so its own uid-to-name lookups read /etc/passwd and nothing else. On a machine where users come from SSSD, LDAP, or Active Directory, a domain user's command reaches the endpoint with a bare number in user.ids.uid and no entry in user.names. The summary.actor fields stay numeric too.

The fix: turn on ENRICHED

Set log_format in /etc/audit/auditd.conf:

log_format = ENRICHED

Then restart auditd:

sudo service auditd restart

auditd links libc, so it can resolve names through NSS the way the rest of the system does. In ENRICHED mode it adds the resolved names to each record, and corvo passes them straight through. corvo still falls back to its own /etc/passwd lookup for anything auditd did not resolve.

Confirm

sudo corvo check

Look for the enriched line. check warns when log_format is anything other than ENRICHED, or when the line is missing entirely (the default, RAW, has the same problem).

Cost

auditd does one cached getpwuid per new id, and audit.log grows a little. On a machine with SSSD this is far cheaper than the alternative, which would be corvo hitting the directory server once per command.

On the endpoint side

Treat a missing name as normal and key your queries on user.ids, not user.names. A machine with SELinux disabled can also send values like unlabeled or kernel in the SELinux fields. Those are real values, not errors.

Recover rejected batches

Some HTTP failures do not improve on retry. A 400, 404, 413, or 422 means the endpoint will reject this batch every time, and retrying it forever would freeze the queue behind it. corvo moves those batches to queue/dead/ and keeps going. Nothing is erased.

A 401 or 403 lands here too, but only after retry.auth_retry_window (15 minutes by default) has passed without the auth being fixed.

Find them

corvo check

The dead-letter line reports how many rejected batches are waiting. corvo_events_dead_lettered_total counts them for your alerting.

Each rejection writes two files:

/var/lib/corvo/queue/dead/20260904T120000.000Z-400.jsonl   the batch
/var/lib/corvo/queue/dead/20260904T120000.000Z-400.err     the response body

Inspect

cat /var/lib/corvo/queue/dead/20260904T120000.000Z-400.err
jq . /var/lib/corvo/queue/dead/20260904T120000.000Z-400.jsonl

The .err file holds the first 8 KiB of the endpoint's response, which is usually enough to say why it was rejected. Common causes: the batch was larger than the endpoint accepts (413), the endpoint's schema validation is stricter than the contract allows (400 or 422), or the URL is wrong (404).

Fix the cause, then resend

Once the endpoint accepts that shape of data again:

corvo replay /var/lib/corvo/queue/dead/20260904T120000.000Z-400.jsonl

replay runs the batch back through the same pipeline, queue, and retry policy as live delivery. What arrives at the endpoint is byte for byte what would have arrived in production.

After a clean resend, delete the two files by hand. corvo never removes anything from dead/ on its own.

If the cause was a 413

Lower the batch size so the next batches fit:

batch:
  max_events: 200
  max_bytes: 524288

Both reload on SIGHUP.

Backfill after a gap

If collection was off for a while, or you are onboarding a machine that has been running without corvo, the events are still in /var/log/audit/audit.log and its rotated copies. corvo replay reads those files and sends them.

Resend one file

corvo replay /var/log/audit/audit.log

corvo parses the log, regroups the fragments, and delivers the events through the normal queue and retry path. Every event carries the same stable id it would have had in real time, so an endpoint that treats a repeated id as idempotent (which the contract requires) absorbs any overlap with what was already delivered.

Resend several files in order

Rotated logs are audit.log.1, audit.log.2, and so on, oldest last. Feed them oldest first:

for f in $(ls -1r /var/log/audit/audit.log.*) /var/log/audit/audit.log; do
  corvo replay "$f"
done

Read from stdin

zcat /var/log/audit/audit.log.2.gz | corvo replay -

Test a new endpoint with real data

replay is also the way to point real traffic at an endpoint before you trust the plugin to it:

corvo replay --endpoint https://new-collector.example.com/v1/audit /var/log/audit/audit.log

What the new endpoint receives is exactly what production would send.

Notes

• replay uses the same config as the plugin, including the queue directory. Run it on the machine whose logs you are sending, or point --queue-dir somewhere scratch.
• The sequence number in old logs resets at each boot, which is why id carries a timestamp. Do not rely on sequence to order across reboots.

Diagnose an SELinux denial

This is where an AlmaLinux or RHEL install usually fails. The plugin runs in auditd's SELinux domain (auditd_t), not your shell's. So corvo check run by hand passes, and the same binary started by auditd hits permission denied. The same split affects networking: your shell can dial 443, the plugin might get connect: permission denied on the identical endpoint.

corvo check reports which context it is running in, so you can see the mismatch. It does not replace reading journalctl -t corvo after the plugin comes up.

The queue needs a label

Give /var/lib/corvo/queue the auditd_log_t type, the same one auditd uses for the files it writes under /var/log/audit:

sudo semanage fcontext -a -t auditd_log_t '/var/lib/corvo(/.*)?'
sudo restorecon -Rv /var/lib/corvo

The RPM does this in its %post. A source install does not.

semanage is in policycoreutils-python-utils, absent from a minimal AlmaLinux 9. Without it:

sudo chcon -R -t auditd_log_t /var/lib/corvo

That works immediately but is lost on a full filesystem relabel, so add the semanage rule when you can.

Do not use audit_spool_t

The name of the spool for undelivered events looks like it describes corvo's queue. It does not. audit_spool_t belongs to the audisp_remote_t domain, and with it applied auditd_t is denied even search on the directory.

The binary label

The binary keeps the bin_t it inherits from /usr/bin. With no domain transition, the plugin runs as auditd_t, the domain of whoever executed it. Moving it to audisp_exec_t would change the plugin's domain and its whole permission set. That is a decision to make with the machine's policy in hand, not a default.

Read an actual denial

sudo ausearch -m avc -ts recent

The line names scontext (the plugin's domain), tcontext (what it touched), and the denied permission. That is the answer. As a last resort, build a local module:

sudo ausearch -m avc -ts recent | audit2allow -M corvo-local
sudo semodule -i corvo-local.pp

Version the .pp. What you do not do is setenforce 0.

Turn on metrics

Metrics are off by default. An audit agent should not open a port in production without someone asking it to.

Enable

metrics:
  enabled: true
  listen: 127.0.0.1:9909

This needs an auditd restart, not a SIGHUP. The HTTP server is built once at startup.

sudo service auditd restart

What you get

• GET /metrics, Prometheus text format
• GET /healthz, returns ok as long as the process is alive and serving

/healthz does not check the endpoint on purpose. corvo stays healthy with the endpoint down, that is what the queue is for.

Scrape it

curl -s localhost:9909/metrics

Point Prometheus at the same address. Bind to 127.0.0.1 unless a collector on another host needs it, in which case use a real interface address and a firewall rule.

The alerts worth setting

The full list is in the metrics reference.