Linux Audit Framework: Minimal auditd Design, Trust Boundaries, and Failure Modes

Problem and takeaway

Many teams enable auditd with broad watches on /etc or entire directory trees, then discover increased syscall latency, dropped audit events, or silent logging gaps when the local disk fills. The useful takeaway is to run auditd with a small, purpose-built rule set, write to the local append-only audit log, forward records to a collector over a protected transport, and monitor daemon status, backlog depth, and free space before an outage forces the issue.

Requirements

Linux Audit (auditd) produces kernel-generated records for system calls, file access, and privilege changes. Unlike application logs, which an application can skip or forge, audit records are generated by the kernel as events occur. This makes auditd suitable for compliance evidence and intrusion detection, but only if the rule set is small enough not to interfere with normal system operation.

Smallest suitable design

Start with a minimal rule set rather than auditing everything. A practical starting point watches a few critical files and privileged execution:

• /etc/sudoers and /etc/passwd for write and attribute changes
• Execve auditing for a small set of privileged users or sensitive binaries
• Local storage in /var/log/audit/audit.log with rotation
• Forwarding to a remote collector over TLS or authenticated syslog

Load a test rule as root on the audited host:

# Requires root or CAP_AUDIT_CONTROL
auditctl -w /etc/sudoers -p wa -k sudoers_write

The -w flag inserts a file watch, -p wa audits write and attribute-change syscalls, and -k sudoers_write attaches a search key. Trigger an event and verify it appears:

# Run as root
chmod 640 /etc/sudoers
ausearch -k sudoers_write -i

Expected result: an audit record containing the watched path, the calling process, and the key sudoers_write. Confirm the rule is loaded with auditctl -l. Do not watch entire filesystems during initial rollout; broad rules such as -w /etc -r generate high volume and can overflow the audit backlog, causing dropped events.

Reversing a test rule is not granular: auditctl -D clears every loaded rule and removes production auditing as well. Use it only if you intend to stop all kernel auditing on that host.

Trust and data boundaries

The trust boundary starts inside the kernel, where audit records are generated, and extends to the log collector. The local audit log is append-only while auditd runs, but root on the audited host can stop auditd, clear rules with auditctl -D, or alter local log files if permissions allow. Therefore local audit alone is not a trust anchor.

Protect the path from host to collector:

• Forward audit records over TLS or authenticated syslog, not plain UDP
• Verify host integrity before trusting its audit records
• Store forwarded records in a write-once or signed store if tamper resistance is required

Operational checks

Run these checks on the audited host. Most require root:

Check	Command	Expected result
Daemon state	systemctl is-active auditd	active
Audit status	auditctl -s	Enabled, no failed status, backlog below limit
Backlog wait time	cat /proc/sys/kernel/audit_backlog_wait_time	Matches operational threshold
Disk space	df -h /var/log/audit	Free space above alert threshold
Rule loaded	auditctl -l	Test rule appears with correct key
Event present	ausearch -k sudoers_write -i --start today	Record returned for the triggered event

Check dmesg and the audit log for overflow messages if auditctl -s shows a rising backlog. Log rotation and remote forwarding must be tested; lost connectivity can fill the local disk and trigger audit suspension.

Failure modes

• Daemon crash or stop: the kernel may still generate records, but without auditd reading the queue the backlog fills and events are dropped or logging suspends.
• Disk full: when /var/log/audit fills, auditd stops writing. Depending on disk_full_action in /etc/audit/auditd.conf, logging may suspend or the system may take a configured emergency action.
• Excessive rules: each audited syscall adds latency; too many rules slow interactive workloads and increase the chance of backlog overflow.
• Backlog overflow: the kernel drops audit records when the queue cannot keep up. Lost events are not recoverable from the local log.

Conditions that change the design

• High event rates or ephemeral cloud hosts: consider eBPF-based auditing or agentless cloud logging instead of persistent auditd rules.
• Strict tamper resistance: local audit is insufficient; use signed logs, write-once storage, or a remote collector the host cannot rewrite.
• Compliance mandates: kernel audit records are generally preferred over application logs because they are harder for a compromised application to suppress.

Limitations and verification

Root on the audited host can disable or reconfigure auditd, so treat local records as evidence, not proof of integrity. Verify the result after any change:

1. Confirm systemctl is-active auditd returns active.
2. Confirm auditctl -s shows enabled: 1 and no failure flag.
3. Trigger a known event and confirm it appears in ausearch with the expected key.

If any check fails, investigate before assuming the host is producing audit evidence.