corvo.rda.run
git.rda.run/corvo/corvo
Install
the facts, for looking up

Reference

Every command, config key, environment variable, and metric.

Commands

corvo <command> [options]

With no command, corvo prints usage and exits nonzero. corvo version prints the version and commit. corvo help prints usage.

Common options

Every command that loads config accepts these. They override the YAML file and the CORVO_* environment.

FlagDefaultMeaning
--config/etc/corvo/corvo.yamlPath to the YAML file. An explicit path that does not exist is an error. The default path missing is not.
--endpointfrom configEndpoint URL.
--log-levelfrom configdebug, info, warn, or error.
--queue-dirfrom configOn-disk queue directory.

corvo run

corvo run

Plugin mode. Reads audit lines from stdin, regroups them, writes the queue, and POSTs to the endpoint. This is what auditd calls, via /etc/audit/plugins.d/corvo.conf. It is the default command in the plugin file's args.

Logs go to stderr and, when available, to syslog. Never to stdout, which belongs to the auditd plugin protocol.

Responds to two signals:

  • SIGHUP: reload config. auditd sends this when it reloads plugins. See what reloads.
  • SIGTERM or SIGINT: clean shutdown. Stops reading stdin, drains the reassembler, tries to empty the queue within shutdown.drain_timeout, syncs. Whatever does not fit stays on disk.

corvo tail

corvo tail [-json] [-log-level LEVEL]

Reads audit events over the multicast netlink socket and prints them. Coexists with auditd. Lossy: the kernel drops events on this socket under load with no backpressure and no warning. Debugging only. Needs CAP_AUDIT_READ or root.

-json prints the full event object instead of a one-line summary.

tail ignores corvo.yaml's filter, on purpose. It is how you find out what record types your machine generates before writing the filter.

corvo replay

corvo replay <file>
corvo replay -            # read from stdin

Reads an audit.log from disk and sends it through the same pipeline, queue, and retry policy as plugin mode. Uses for this:

  • Backfill after a gap in collection.
  • Resend a batch from queue/dead/.
  • Test a new endpoint with real data before pointing the plugin at it.

corvo check

corvo check [--offline]

Validates config and environment before production. Exits nonzero on any failure, so it works in ExecStartPre and in CI. Checks:

  • The YAML parses and validates.
  • The token file exists, is readable, and is not world-readable.
  • The current session's auid and which rule it matches.
  • Whether auditd is in ENRICHED mode.
  • Which SELinux context check itself runs in.
  • TLS file paths.
  • Queue depth and the contents of dead/.
  • Endpoint DNS, TLS, auth, and the body contract, by sending one synthetic event. Skipped with --offline.

Example:

$ corvo check
  ok      config         https://logs.example.com/v1/audit
  ok      token          /etc/corvo/token (0600)
  ok      queue          /var/lib/corvo/queue: 0 pending event(s), 0 B on disk, 1 segment(s)
  ok      dead-letter    empty
  ok      dns            logs.example.com -> 10.0.0.5
  ok      endpoint       HTTP 200 in 42ms

All good.

corvo reload

corvo reload

Sends SIGHUP to every running corvo process and reports how many it reached. Same effect as pkill -HUP -x corvo, without needing pkill installed. systemctl reload auditd does not work, the auditd unit refuses it.

Configuration keys

corvo builds its config from three layers, each overriding the last:

  1. /etc/corvo/corvo.yaml (or --config)
  2. CORVO_* environment variables (reference)
  3. command-line flags

A missing default config file is tolerated and corvo runs on defaults plus environment. An explicit --config pointing nowhere is an error. An unknown key in the YAML, or an unknown CORVO_* variable, stops the process at startup. A misspelled CORVO_ENDPIONT_URL has to fail loudly, not quietly keep sending to the old endpoint.

Durations are strings like 10s, 5m, 1h.

endpoint

KeyTypeDefaultMeaning
urlstringnone, requiredWhere to POST. Must be http or https with a host.
timeoutduration10sPer-request timeout. Must be positive.
token_filestringnoneFile holding the bearer token. Reread on every SIGHUP. Must be readable if set.
headersmapnoneExtra request headers. Cannot set Authorization.
gzipbooltruegzip the POST body. The on-disk queue stays uncompressed.
host_overridestringnoneValue for each event's host field, instead of os.Hostname().

endpoint.tls

KeyTypeDefaultMeaning
ca_filestringnoneCA bundle for a private CA. Empty uses the system trust store.
cert_filestringnoneClient certificate for mTLS. Goes with key_file.
key_filestringnoneClient key for mTLS. Goes with cert_file.
insecure_skip_verifyboolfalseSkip certificate verification. Test only.

With cert_file and key_file set, corvo does not send Authorization.

batch

Sending triggers on the first limit to overflow.

KeyTypeDefaultMeaning
max_eventsint500Events per batch. Positive.
max_bytesint641048576 (1 MiB)Batch size before gzip. Positive. Cannot exceed queue.segment_bytes.
flush_intervalduration5sSend after this long even if under the size limits. Positive.

retry

KeyTypeDefaultMeaning
initialduration1sFirst backoff step. Positive.
maxduration5mBackoff ceiling. Cannot be less than initial.
multiplierfloat2.0Backoff growth per attempt. Must be at least 1.
jitterfloat0.3Random spread on each delay, plus or minus this fraction. Between 0 and 1.
max_attemptsint0Attempts before giving up. 0 is infinite. Giving up loses an audit trail.
auth_retry_windowduration15mHow long a 401 or 403 stays in retry before going to dead-letter.

queue

KeyTypeDefaultMeaning
dirstring/var/lib/corvo/queueQueue directory. Required.
max_bytesint641073741824 (1 GiB)Queue size ceiling. Positive.
max_eventsint641000000Queue event-count ceiling. Positive.
segment_bytesint6416777216 (16 MiB)Rotation size for one segment. Positive. Cannot exceed max_bytes.
on_fullstringdrop_oldestdrop_oldest or block. See below.
fsyncstringnonenone, batch, or always. See below.

queue.on_full

ValueBehavior
drop_oldestDelete the oldest segment whole (os.Remove, disk freed at once), count corvo_events_dropped_total, stay alive. Loses the past, keeps the present.
blockStop consuming stdin. auditd buffers, and if the outage lasts the kernel drops at the source, visible in corvo_events_lost_total. corvo drops nothing itself. Correct for strict compliance, and the cost is real. Pair it with a larger q_depth in auditd.conf.

queue.fsync

ValueBehavior
noneNo fsync. Survives a process crash, kill -9, and an auditd restart. A power loss loses the not-yet-flushed window. Cheapest I/O.
batchfsync on every written batch.
alwaysfsync on every event. Caps throughput at hundreds of events per second on rotational disk. Only for an explicit compliance requirement.

filter

Picks which events ship, by record_type. See Pick which events ship.

KeyTypeDefaultMeaning
includelistnoneOnly these record types pass. Empty means "everything not excluded".
excludelistnoneThese record types are dropped. Beats include.

A record type that auparse does not recognize stops the process at startup. An empty list item does too.

shutdown

KeyTypeDefaultMeaning
drain_timeoutduration30sOn SIGTERM, how long to keep trying to empty the queue before exiting. Cannot be negative.

metrics

KeyTypeDefaultMeaning
enabledboolfalseServe /metrics and /healthz.
listenstring127.0.0.1:9909Bind address. Validated as host:port when enabled.

log

KeyTypeDefaultMeaning
levelstringinfodebug, info, warn, or error. Always to stderr, never stdout.

What reloads on SIGHUP

auditd sends SIGHUP when it reloads its plugins. corvo reload sends it directly. An invalid new config is rejected whole and the previous one stays, because a plugin that dies over a typo is worse than one with a stale config.

SectionReloads
endpoint (url, token, headers, tls, timeout, gzip)yes
batchyes
retryyes
filteryes, from the next event on
log.levelyes
queueno, needs an auditd restart
metricsno, needs an auditd restart
shutdownno, needs an auditd restart

For the three that do not reload, corvo logs a warning that the new value was not applied, instead of letting you believe it was. The queue is excluded because swapping its directory live would abandon events already written in the old one. Metrics and shutdown are excluded because the server and the deadline are assembled once, at startup.

Environment variables

Every CORVO_* variable maps to one config key. They override the YAML file and are overridden by command-line flags. A CORVO_* variable that matches no key stops the process at startup, with the list of valid names in the error.

The name is the key's path, uppercased, joined by underscores. List values are comma-separated (syscall,user_login); an empty string clears the list, which lets the environment turn off a filter set in the YAML.

endpoint.headers has no variable. Arbitrary header names have no fixed mapping, so headers come only from the YAML.

VariableKeyType
CORVO_ENDPOINT_URLendpoint.urlstring
CORVO_ENDPOINT_TIMEOUTendpoint.timeoutduration
CORVO_ENDPOINT_TOKEN_FILEendpoint.token_filestring
CORVO_ENDPOINT_GZIPendpoint.gzipbool
CORVO_ENDPOINT_HOST_OVERRIDEendpoint.host_overridestring
CORVO_ENDPOINT_TLS_CA_FILEendpoint.tls.ca_filestring
CORVO_ENDPOINT_TLS_CERT_FILEendpoint.tls.cert_filestring
CORVO_ENDPOINT_TLS_KEY_FILEendpoint.tls.key_filestring
CORVO_ENDPOINT_TLS_INSECURE_SKIP_VERIFYendpoint.tls.insecure_skip_verifybool
CORVO_BATCH_MAX_EVENTSbatch.max_eventsint
CORVO_BATCH_MAX_BYTESbatch.max_bytesint64
CORVO_BATCH_FLUSH_INTERVALbatch.flush_intervalduration
CORVO_RETRY_INITIALretry.initialduration
CORVO_RETRY_MAXretry.maxduration
CORVO_RETRY_MULTIPLIERretry.multiplierfloat
CORVO_RETRY_JITTERretry.jitterfloat
CORVO_RETRY_MAX_ATTEMPTSretry.max_attemptsint
CORVO_RETRY_AUTH_RETRY_WINDOWretry.auth_retry_windowduration
CORVO_QUEUE_DIRqueue.dirstring
CORVO_QUEUE_MAX_BYTESqueue.max_bytesint64
CORVO_QUEUE_MAX_EVENTSqueue.max_eventsint64
CORVO_QUEUE_ON_FULLqueue.on_fullstring
CORVO_QUEUE_SEGMENT_BYTESqueue.segment_bytesint64
CORVO_QUEUE_FSYNCqueue.fsyncstring
CORVO_FILTER_INCLUDEfilter.includelist
CORVO_FILTER_EXCLUDEfilter.excludelist
CORVO_SHUTDOWN_DRAIN_TIMEOUTshutdown.drain_timeoutduration
CORVO_METRICS_ENABLEDmetrics.enabledbool
CORVO_METRICS_LISTENmetrics.listenstring
CORVO_LOG_LEVELlog.levelstring

Setting them for the plugin

auditd's plugin file (corvo.conf) has a two-argument limit and no way to pass environment. To set a variable for the plugin, use a systemd drop-in on the auditd unit:

# /etc/systemd/system/auditd.service.d/corvo.conf
[Service]
Environment=CORVO_LOG_LEVEL=debug

Then systemctl daemon-reload and restart auditd. For most settings the YAML file is the better place.

Metrics

Off by default. With metrics.enabled: true, corvo serves GET /metrics in Prometheus text format and GET /healthz on metrics.listen (127.0.0.1:9909 by default). Enabling needs an auditd restart, not a SIGHUP. See Turn on metrics.

/healthz returns ok while the process is alive and serving. It does not probe the endpoint. corvo is healthy with the endpoint down, that is the queue's job.

Pipeline

MetricTypeMeaning
corvo_lines_received_totalcounterAudit lines read from the input.
corvo_parse_errors_totalcounterLines that could not be parsed.
corvo_events_emitted_totalcounterCoherent events produced by regrouping.
corvo_coalesce_failures_totalcounterMessage groups that could not be coalesced into an event.
corvo_events_filtered_totalcounterEvents dropped by corvo.yaml's filter before the queue.
corvo_events_lost_totalcounterEvents the kernel dropped before corvo saw them. Above 0, raise -b in the audit rules.

Queue

MetricTypeMeaning
corvo_queue_depth_itemsgaugeEvents written and not yet confirmed by the endpoint.
corvo_queue_depth_bytesgaugeBytes not yet confirmed. Sustained growth means the endpoint is not keeping up.
corvo_queue_disk_bytesgaugeDisk used by the queue's segments.
corvo_queue_segmentsgaugeSegment count on disk.
corvo_events_appended_totalcounterEvents written to the queue.
corvo_events_dropped_totalcounterEvents erased by drop_oldest when the queue filled. Real loss.
corvo_events_dead_lettered_totalcounterEvents moved to dead/ after a rejection no retry fixes.

Sink

MetricTypeMeaning
corvo_batches_sent_totalcounterBatches confirmed with a 2xx.
corvo_events_sent_totalcounterEvents confirmed by the endpoint.
corvo_send_failures_totalcounterSend attempts that failed. Climbs before any loss.
corvo_send_retries_totalcounterResends after a failure.
corvo_post_duration_secondshistogramLatency of the POST.

Build

MetricTypeMeaning
corvo_build_infogaugeAlways 1, with a version label.

What to alert on

  • corvo_events_lost_total > 0: kernel-side loss, the audit buffer is too small.
  • rate(corvo_queue_depth_bytes) positive for several minutes: the endpoint is slow or down. This fires before anything is lost.
  • corvo_events_dropped_total > 0: the queue filled and drop_oldest ran.
  • corvo_events_dead_lettered_total > 0: a batch was permanently rejected, inspect dead/.

Audit rules and the plugin file

The plugin file

Installed at /etc/audit/plugins.d/corvo.conf (0640). auditd reads it on start and on reload.

active = yes
direction = out
path = /usr/bin/corvo
type = always
args = run
format = string
  • path points at the binary. A source install rewrites this to $PREFIX/bin/corvo.
  • args = run is the whole command line. man audit-plugins warns of a two-argument limit, so the config path comes from the default or from CORVO_*, not from here.
  • format = string: auditd converts each event to text before writing it to the pipe, which is what corvo's parser expects. binary would hand over the raw netlink record and add work for nothing.

On AlmaLinux 9 and other audit 3.x systems, audispd was merged into auditd and plugins live at this path. RHEL 7 and 8 tutorials show /etc/audisp/plugins.d/, which is wrong here.

The rules

Installed at /etc/audit/rules.d/corvo.rules, loaded with augenrules --load.

-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset -k cmds
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=unset -k cmds

-a always,exit -F arch=b64 -S execve -F auid=0 -k cmds-root
-a always,exit -F arch=b32 -S execve -F auid=0 -k cmds-root

#-a always,exit -F arch=b64 -S execve -k cmds-all

The -F filters run inside the kernel, before the record exists. That is why they matter for cost: they cut at the source instead of generating a record and discarding it.

KeyCatches
cmdsCommands from real users, auid >= 1000.
cmds-rootCommands from whoever logged in as root, console or direct SSH. Auditing root is the point of a trail, root is exactly who can erase their own tracks.
cmds-allCommented out. Every execve on the machine, including every process systemd starts. Heavy volume. Uncomment only to validate an install end to end, then comment it back.

System daemons are not swept in. systemd and what it starts have auid=unset, which does not match auid=0.

The auid=unset case

What sets auid is pam_loginuid, present in AlmaLinux 9's /etc/pam.d/login and /etc/pam.d/sshd. An interactive session that shows up with auid=unset (hypervisor console, agetty with no PAM, docker exec) is not fixed by a rule. With no auid the event holds nobody accountable. Fix it in PAM.

To check the current session:

corvo check                 # shows the session's auid and which rule matches
cat /proc/self/loginuid     # the raw value

auditd.conf settings that matter

SettingWhy
log_format = ENRICHEDauditd resolves user and group names for ids outside /etc/passwd (SSSD, LDAP, AD). Without it those users reach the endpoint as bare numbers. See Resolve names for domain users.
q_depthThe in-memory event queue auditd keeps for its plugins. Raise it if you run queue.on_full: block.
overflow_actionWhat auditd does when q_depth fills. Decide this deliberately with block.

The event JSON

This page orients you. The full specification for whoever writes the receiving server is design/api-contract.md, kept honest by a golden file that a test compares byte for byte on every build.

The envelope

corvo POSTs a JSON array. Each element has four fields:

{
  "v": 1,
  "id": "srv-web-01-2841-1756938610123000000",
  "host": "srv-web-01",
  "event": { }
}
FieldTypeMeaning
vintegerEnvelope version. 1 today.
idstringDeduplication key, <host>-<sequence>-<nanoseconds>. Treat as opaque, the hostname can contain hyphens.
hoststringThe capturing host. os.Hostname() or endpoint.host_override.
eventobjectThe coalesced audit event.

The event object

This is go-libaudit's aucoalesce.Event, serialized as-is. Fields you can count on: @timestamp, sequence, category, record_type, session, summary, user, ecs. Fields that appear only when populated: result, tags, file, source, destination, network, data, paths.

summary, user, process, and ecs are Go structs, so they show up in the JSON even when empty ({}). Test process.exe for being populated, not "process" in event.

A full worked example, id -u run as root, is in testdata/execve-completo.json.

Record types

record_type is the record that originated the event. category is the family go-libaudit derives from it. The full catalogue, with what each type carries, is in design/api-contract.md.

Two rules to know up front:

  • Some types never arrive alone. execve, path, cwd, proctitle, sockaddr, bprm_fcaps are fragments. corvo merges them into the main record, where they become process.args, paths, process.cwd, process.title, source, destination. Seeing one as an event's record_type means the rest never arrived.
  • In a composite event the type comes from the first record, not from SYSCALL. An execve that trips a SELinux denial arrives as avc, with the syscall's paths and data attached. So filtering by syscall alone misses those commands.

Traps for the receiver

  • process.title is cut at 128 bytes by the kernel, mid-word, no marker. process.args carries the full argv. Index and search on args, use title for display only.
  • @timestamp has variable fractional precision, zero to nine decimal places within one batch. Use a real RFC 3339 parser, not a fixed format string.
  • Almost everything is a string, including pid, uid, inode, mode, port. audit delivers those in mixed formats and the library does not risk converting them. Only v and sequence are integers.
  • data and paths are open maps whose keys vary by syscall. On a store with strict mapping, use flattened or dynamic: false or a jsonb column.
  • Delivery is at-least-once. A repeated id must be idempotent on the server, this is a hard requirement of the contract.
auditd → https · nothing gets lost corvo (Portuguese for raven) is the agent, ninho (nest) is the server. Both speak the same versioned contract.
docs how it works corvo (agent) ninho (server)
ninho docs tutorials how-to guides reference explanation
corvo docs tutorials how-to guides reference explanation
reference API contract install metrics and queue
code git.rda.run/corvo/corvo git.rda.run/corvo/ninho MIT license