Linux audit logging records kernel and user-space events that ordinary application logs may miss. With auditd, you can answer focused questions about a file change, failed login, process identity, or system call and then retrieve the related records as one event.
After validating those records locally, you can copy complete audit events to Fluxtail over TLS Syslog for centralized search without replacing the host's authoritative audit file.
The useful goal is not “log everything.” It is to keep a small rule set that matches real investigation needs, prove the rules are active, watch the kernel backlog and lost-event counter, and preserve the evidence through rotation or forwarding. An audit trail supports an investigation; it does not prove that a host meets a compliance standard or that no activity was missed.
Install and check auditd
First check whether the tools and service already exist:
command -v auditctl ausearch aureport
sudo auditctl -s
sudo auditctl -l
sudo systemctl status auditd --no-pager
auditctl -s reads the kernel audit status. auditctl -l lists the rules currently loaded in the kernel. A running daemon with an empty rule list can still record some events from components such as PAM, SELinux, or AppArmor, but it does not provide the file and syscall coverage you may expect.
Package names vary. Use the command for the host's supported distribution and confirm it against that release's package repository:
# RHEL, Rocky Linux, AlmaLinux, Fedora
sudo dnf install audit
# Debian and Ubuntu
sudo apt-get install auditd
# SUSE Linux Enterprise and openSUSE
sudo zypper install audit
On Debian, Ubuntu, and SUSE systems with the standard systemd unit, sudo systemctl enable --now auditd is the usual activation path. Red Hat's current audit service guidance documents a deliberate exception: use sudo systemctl enable auditd, then sudo service auditd start. RHEL reserves systemctl for enable and status so service-control actions receive the correct Audit user ID (auid). Follow the host distribution here rather than applying one command everywhere.
The default audit file is commonly /var/log/audit/audit.log, but log_file in /etc/audit/auditd.conf is authoritative. Containers normally rely on the host's kernel audit subsystem; installing auditd inside an ordinary unprivileged container does not create an independent audit boundary.
Understand events, records, and identities
One audit event can contain several records. They share the timestamp and serial inside msg=audit(timestamp:serial). A file access might include SYSCALL, CWD, PATH, and PROCTITLE records. Searching by any matching record returns the records that make up the event.
Representative raw records look like this:
type=SYSCALL msg=audit(1789482107.512:1842): arch=c000003e syscall=257 success=yes exit=3 items=1 ppid=2140 pid=2188 auid=1000 uid=0 euid=0 comm="vim" exe="/usr/bin/vim" key="identity_files"
type=CWD msg=audit(1789482107.512:1842): cwd="/etc"
type=PATH msg=audit(1789482107.512:1842): item=0 name="/etc/passwd" inode=131221 nametype=NORMAL
type=PROCTITLE msg=audit(1789482107.512:1842): proctitle=76696D002F6574632F706173737764
type=EOE msg=audit(1789482107.512:1842):
Values vary by kernel, architecture, distribution, and action. The relationships matter more than the sample numbers:
auidis the login identity and normally follows the session throughsudoor another effective-identity change;uidandeuiddescribe the process's real and effective user IDs;pid,ppid,comm, andexeidentify the process and executable;successandexitdescribe the syscall result;keycomes from the matching audit rule;- the serial after the colon,
1842above, joins the event's records.
ausearch -i interprets numeric values such as IDs and syscall numbers. The upstream ausearch manual warns that interpretation on unenriched logs uses the current machine's account and system data. If accounts changed or logs moved to another host, keep the raw values available and treat interpreted names carefully.
Add one focused persistent rule
Start with a file or operation tied to a specific investigation question. The legacy -w watch syntax remains accepted, but the upstream audit.rules documentation recommends syscall-style file rules for current configurations.
For example, this fragment records writes and attribute changes to /etc/passwd on a 64-bit ABI and assigns the key identity_files:
# /etc/audit/rules.d/40-identity-files.rules
-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -k identity_files
The path must exist when the rule is loaded. w means write and a means an attribute change. This is intentionally narrower than logging every execve call or recursively watching all of /etc.
Architecture is part of the rule contract. On a bi-architecture host that executes both 64-bit and 32-bit programs, an equivalent arch=b32 rule may also be required. Do not add it blindly: syscall tables differ, and some hosts do not support the 32-bit ABI. Put arch before -S whenever a rule names syscalls, and validate rules on each architecture you deploy.
Also verify assumptions such as the first interactive-user ID in /etc/login.defs. When filtering human sessions, the common pattern -F auid>=1000 -F auid!=unset is correct only if UID_MIN is actually 1000 on that host.
awk '$1 == "UID_MIN" { print $2 }' /etc/login.defs
If the investigation question is “which commands did logged-in users execute?”, an execve rule can use that threshold. On a bi-architecture x86_64 host that actually supports 32-bit programs, the pair would be:
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset -k user_exec
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=unset -k user_exec
Replace 1000 with the host's verified UID_MIN. Omit the b32 rule when that ABI is unsupported, and use only the rule appropriate to a 32-bit-only system. Command-execution auditing can generate substantial volume and does not record shell built-ins that do not call execve, so measure it on a representative host before broad rollout.
Check, load, and confirm the rule
augenrules merges files ending in .rules from /etc/audit/rules.d/ into the active rules file. Review the complete directory before loading because ordering matters and directives such as -D or a final immutable setting affect other fragments. Also look for a never,task rule: on systems that ship one, its first-match behavior can prevent syscall rules from matching newly created processes.
sudo augenrules --check
sudo augenrules --load
sudo auditctl -l
sudo ausearch -k identity_files -ts recent -i
augenrules --check reports whether the merged rules need updating; it is not a substitute for loading and inspecting errors. After --load, confirm the exact rule in auditctl -l. A key is an investigation handle, not an access-control mechanism.
If auditctl -s reports enabled 2, the rule set is immutable until reboot. Do not add an immutable final rule casually. First prove that the configuration-management, maintenance, and recovery paths work, because a typo cannot be corrected live after the lock is active.
Audit 3.x installations commonly load persistent fragments with augenrules, while audit 4.x introduced audit-rules.service. Check the installed version and unit files before choosing the loader; then confirm the resulting kernel rules rather than assuming a successful service command loaded them:
sudo auditctl -v
systemctl list-unit-files 'audit-rules*' 'auditd*'
sudo auditctl -l
To verify that a runtime rule loads, fires, is searchable, and can be removed exactly, use a disposable file instead of changing a protected identity file:
audit_guide_file="$(sudo mktemp /var/tmp/audit-guide-test.XXXXXX)"
sudo auditctl -a always,exit -F arch=b64 \
-F path="$audit_guide_file" -F perm=wa \
-k audit_guide_test
printf '%s\n' test | sudo tee -a "$audit_guide_file" >/dev/null
sudo ausearch -k audit_guide_test -ts recent -i
sudo auditctl -d always,exit -F arch=b64 \
-F path="$audit_guide_file" -F perm=wa \
-k audit_guide_test
sudo rm -- "$audit_guide_file"
Use the host's actual architecture rather than copying b64 blindly. Validate persistent rules on a test host or during an approved maintenance change; do not edit identity or authentication files only to make a test event.
Enabling audit only after the system has started can miss activity from processes created earlier in boot. Early-process coverage normally requires a tested audit=1 kernel boot parameter. Treat that, immutable mode, and failure-mode changes as availability-sensitive system policy, not casual tutorial steps.
Search with ausearch and summarize with aureport
Begin with a time window and one other criterion. Every different ausearch option is combined as an AND condition, while a comma-separated list passed to -m matches any listed record type.
# Events produced by one rule key in the last ten minutes
sudo ausearch -k identity_files -ts recent -i
# Failed login events during the current week
sudo ausearch -m USER_LOGIN -sv no -ts this-week -i
# Activity tied to login UID 1000, regardless of later identity changes
sudo ausearch --loginuid 1000 -ts today -i
# Compare searches by real and effective process identity
sudo ausearch --uid 0 -ts today -i
sudo ausearch --uid-effective 0 -ts today -i
# Every record belonging to audit event serial 1842
sudo ausearch -a 1842 -i
# Summary reports
sudo aureport --auth --summary -i -ts this-week
sudo aureport --key --summary -i -ts this-week
sudo aureport --comm --summary -i -ts today
recent means the last ten minutes in current ausearch. The event serial is the number after the colon in msg=audit(...). aureport shows totals and event numbers; use an event number with ausearch -a when you need the underlying records.
Use -if when examining a copied file or directory rather than the configured live logs:
sudo ausearch -if /var/log/audit/audit.log.1 -k identity_files -i
Check ausearch --help on the target host before automating queries across mixed audit package versions. Compressed rotations may need to be restored to a readable file first.
Watch backlog and lost records
Service health alone does not show whether the kernel queue overflowed. The upstream auditctl manual defines the status fields. Inspect both the daemon and kernel audit state:
sudo auditctl -s
sudo auditctl -l
sudo systemctl status auditd --no-pager
sudo journalctl -u auditd --since today
A representative status includes these fields:
enabled 1
failure 1
pid 742
rate_limit 0
backlog_limit 8192
lost 0
backlog 0
pid 0 means no audit daemon is registered. backlog is the number of records currently waiting for auditd; a brief nonzero value is not automatically a failure, but sustained growth needs investigation. lost counts records discarded after the kernel queue overflowed. If it increases, the audit trail for that period is incomplete.
Do not immediately raise backlog_limit or reset the lost counter. Preserve the evidence, then inspect rate_limit, failure, backlog_limit, daemon state, dispatcher queues, and audit-disk latency and capacity together. A noisy rule, explicit rate limit, slow storage, plugin congestion, or daemon outage can produce different failure modes. Test any bounded backlog, rate, failure-action, storage, or rule-volume change under representative load. Combining syscall names into fewer otherwise-identical rules can reduce rule-evaluation work.
Plan rotation and disk-failure behavior
Audit rotation is controlled in /etc/audit/auditd.conf. Review the active values and the filesystem that actually contains log_file:
sudo grep -E '^(log_file|max_log_file|max_log_file_action|num_logs|space_left|space_left_action|admin_space_left|admin_space_left_action|disk_full_action|disk_error_action|q_depth|overflow_action|max_restarts)' /etc/audit/auditd.conf
sudo du -sh /var/log/audit
df -h /var/log/audit
max_log_file controls a file-size threshold measured in MiB, while max_log_file_action decides what happens at that threshold. num_logs matters when rotation is used. q_depth, overflow_action, and max_restarts govern the dispatcher path and its failure behavior. The space_left and admin_space_left thresholds should leave enough time and space for an operator to respond. Derive them from measured event volume, retention needs, and the size of the audit filesystem rather than copying fixed numbers from another host.
Disk actions are operational policy. rotate can discard the oldest retained file. suspend leaves the daemon alive but stops writing audit records. single can change the host to single-user mode, and some versions expose a halt action. Those choices can affect availability and must be reviewed with the service owner before deployment. The current upstream auditd.conf manual documents the exact behavior for the installed audit version.
After any policy change, use the distribution-supported auditd reload or restart procedure and confirm that logging resumes. On RHEL, that means service auditd reload or another documented service auditd action rather than systemctl restart auditd.
Forward audit records over TLS with rsyslog
Keep the local audit file even when you centralize a copy. The supported chain is:
kernel audit -> auditd -> audisp-syslog -> local rsyslog -> TLS Syslog receiver
The audisp-syslog plugin is often packaged separately. On Debian and Ubuntu it is available in audispd-plugins; package layout varies elsewhere. Locate the installed file instead of assuming one path:
sudo find /etc/audit /etc/audisp -path '*/plugins.d/syslog.conf' -print 2>/dev/null
Modern RHEL uses /etc/audit/plugins.d/; older packages may use /etc/audisp/plugins.d/. Preserve the packaged path, type, direction, and format settings. Enable the plugin and assign a dedicated local facility only if that installed version supports it:
# Edit the installed syslog plugin file; preserve its other settings.
active = yes
args = LOG_INFO LOG_LOCAL6
The upstream audisp-syslog plugin sends every audit record it receives to Syslog; it does not select by audit rule key. Control volume with focused audit rules, and remember that one audit event may become several Syslog messages. LOG_INFO is the Syslog priority for the forwarded records, not the success or severity of the audited action.
Keep the plugin's raw string output unless you have tested its version-specific interpreted mode and parser behavior. The upstream manual warns that interpreted output can be ambiguous when an attacker controls file names, process names, or other decoded values. Preserve the audit serial so related Syslog messages can still be correlated.
Next, create a dedicated Fluxtail Syslog receiver, allow the sender's public IPv4 or CIDR, wait for the endpoint to become active, and copy its exact hostname and assigned port. Fluxtail accepts Syslog on TLS TCP, not UDP or plain TCP. Configure rsyslog with the host's real CA bundle path:
# /etc/rsyslog.d/98-fluxtail-audit.conf
global(
DefaultNetstreamDriverCAFile="YOUR_CA_BUNDLE_PATH"
WorkDirectory="/var/spool/rsyslog"
)
if ($syslogfacility-text == "local6") then {
action(
type="omfwd"
target="YOUR_POOL_HOSTNAME"
port="YOUR_ASSIGNED_PORT"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="YOUR_POOL_HOSTNAME"
TCP_Framing="octet-counted"
template="RSYSLOG_SyslogProtocol23Format"
action.resumeRetryCount="-1"
queue.type="LinkedList"
queue.filename="fluxtail-audit"
queue.maxdiskspace="1g"
queue.saveOnShutdown="on"
)
}
The uppercase values are placeholders. Debian-family CA bundles commonly use /etc/ssl/certs/ca-certificates.crt; RHEL-family systems commonly use /etc/pki/tls/certs/ca-bundle.crt. Confirm the file locally, and verify that the rsyslog service user can write to /var/spool/rsyslog. The 1 GiB queue limit is an example safety bound; size it from measured event rate, available disk, and the outage window you intend to absorb.
Validate rsyslog before restarting it:
sudo rsyslogd -N1
openssl s_client \
-connect YOUR_POOL_HOSTNAME:YOUR_ASSIGNED_PORT \
-servername YOUR_POOL_HOSTNAME \
-verify_hostname YOUR_POOL_HOSTNAME
sudo systemctl restart rsyslog
logger -p local6.info -t audit-forward-test "fluxtail tls route check"
sudo auditctl -m "fluxtail_audit_pipeline_check"
The logger event verifies the facility-to-TLS route without involving auditd. The auditctl -m event then verifies the whole audit plugin path. Restart or reload auditd with the distribution-supported command after changing its plugin, preferably in a maintenance window, and search for both markers in Fluxtail Live Tail.
The public Fluxtail Syslog documentation provides the current TLS settings and certificate-verification test. The official omfwd documentation explains the TLS and queue parameters, while Fluxtail's rsyslog centralized logging guide covers the broader sender workflow.
Compare a forwarded event with its local counterpart, including the audit serial, host, record type, and key. Fluxtail can filter parsed Syslog host, application, severity, and message fields, but audit keys and identities may remain inside the message unless the deployed parser maps them. Verify the received shape before building alerts or claiming field-level search.
Treat the audit trail as evidence, not proof
A healthy audit setup has a known rule inventory, expected test events, lost 0 during the observed period, enough disk headroom, reviewed failure actions, and a retrieval test across rotations. Central forwarding adds another copy and faster cross-host search; it does not repair events that the kernel or daemon already lost.
For compliance work, map the exact control to rules, retention, access restrictions, time synchronization, review procedures, and change records with the responsible assessor. auditd data can support that evidence, but installing the package or forwarding its output is not compliance by itself.
The practical operating loop is short: load focused rules, confirm them with auditctl -l, query them by key with ausearch, summarize trends with aureport, watch backlog and disk state, and verify the same event after TLS forwarding.
When that local evidence path is working, create a Fluxtail account and configure a dedicated Syslog receiver with the assigned hostname and port.