Fluxtail
Syslog

Rsyslog Configuration: Inputs, Rule Order and Missing Messages

Understand rsyslog configuration, include order, stop and rulesets. Use tested local and TCP examples to trace missing messages before checking transport.

By Fluxtail Engineering

Rsyslog is a logging daemon that receives messages from local applications and network connections, applies rules, and writes the results to files or remote receivers. Linux operators use it to keep local logs and collect messages from several machines in one place.

A valid rsyslog configuration can still send a message to the wrong destination. Validation checks configuration errors; it does not demonstrate where a real message goes. An earlier rule can stop processing before forwarding, or an input can send its messages to a different ruleset—a named sequence of processing rules.

The distinction matters when a local file contains a message but the central receiver does not. Before changing certificates or network settings, establish whether the message reached the forwarding action. Rsyslog’s official documentation explains both sequential rule processing and input-to-ruleset binding.

Understand the input, filter and action

An input accepts messages. A filter selects messages by their properties. An action writes or forwards the selected messages. A ruleset groups the rules that an input will run.

This small RainerScript example accepts TCP messages on loopback and writes messages from the routeprobe program to a file:

module(load="imtcp")

ruleset(name="probe") {
    if $programname == "routeprobe" then {
        action(type="omfile" file="/var/log/routeprobe.log")
    }
}

input(type="imtcp" address="127.0.0.1" port="18514" ruleset="probe")

The listener is local to the machine. It does not accept connections addressed to an external interface. The file action also needs permission to write its destination. This is a configuration example, not a replacement for the machine’s existing logging configuration.

An input without an explicit ruleset binding normally uses the default ruleset. A bound input processes messages in its selected ruleset; it does not automatically run every rule elsewhere in the configuration. The TCP input reference documents the input parameters.

Trace a missing message through rsyslog

These checks apply when rsyslog accepts the configuration but an expected message is missing or appears in the wrong file. They help establish which rules process the message. They do not, by themselves, prove remote delivery or explain every runtime failure.

Check includes and rule order

Start with the configuration used by the running service. /etc/rsyslog.conf and /etc/rsyslog.d/ are common locations, but distribution packages and service arguments can change them. A directory is not loaded merely because it exists: the configuration must include its files.

For example:

include(file="/etc/rsyslog.d/*.conf")

Matching files are processed in lexical filename order at the position of the include. Thus 10-local.conf precedes 20-forward.conf. Rules before and after that include also matter.

Within the processing sequence, matching one rule does not normally prevent later rules from running. A message can be written locally and forwarded. An explicit stop ends further processing for that message. It does not undo an action already performed. Rsyslog’s order tutorial demonstrates this distinction.

Do not rename production files just to make forwarding run first. Identify which earlier filters and stops are intentional, then change the narrow rule responsible for the unexpected route.

Check the input’s ruleset

Find the input that actually receives the event. A message sent to a local Unix socket and a message received over TCP can enter different rulesets even on the same host.

Inspect the input’s ruleset parameter, then follow that ruleset’s filters, actions and includes. A forwarding action in the default ruleset will not automatically receive messages bound to another ruleset.

Also inspect the parsed properties used by filters. A condition on $programname needs the parsed program name to match; finding the same word somewhere in the message body is insufficient. Local socket and network messages can have different header formats.

Compare local and destination markers

Use a harmless marker such as routeprobe-20261115-001, with a new suffix for each test. Send it through the same input as the missing application message. Testing a local socket does not establish that a network listener has the same route.

Compare the marker at a local action and at the destination. A local copy establishes that the local action processed it. A destination copy establishes receipt there. Neither observation alone proves every other action ran.

If the marker reaches the local file but not the destination, inspect the rules between those actions before investigating transport. If it reaches neither, check the input, binding, filter and local write errors. Keep the search window narrow and account for the message timestamp.

A routing example with measured results

The following isolated test used rsyslog 8.2504.0. It sent synthetic messages through a private Unix socket and a loopback TCP listener. A second rsyslog process received forwarded messages on another loopback port. No production service configuration was changed.

To reproduce it, create a writable lab directory. Replace every /ABS/LAB below with that directory’s absolute path. Keep the two TCP ports unused. These examples require the imuxsock and imtcp modules supplied by your installation.

Create the sender and two include files

Save this as main.conf:

module(load="imuxsock" SysSock.Use="off")
module(load="imtcp")
template(name="marker" type="string" string="%msg%\n")

ruleset(name="routed") {
    include(file="/ABS/LAB/[12]0-*.conf")
}

ruleset(name="isolated") {
    action(type="omfile" file="/ABS/LAB/isolated.log" template="marker")
}

input(type="imuxsock" Socket="/ABS/LAB/input.sock" ruleset="routed")
input(type="imtcp" address="127.0.0.1" port="18514" ruleset="routed")

SysSock.Use="off" disables the normal system socket in this test process. The explicit input creates a separate socket inside the lab directory. The Unix socket input reference documents these settings.

Save this as 10-local.conf:

if $programname == "routeprobe" then {
    action(type="omfile" file="/ABS/LAB/local.log" template="marker")
}

Save this as 20-forward.conf:

if $programname == "routeprobe" then {
    action(type="omfwd" target="127.0.0.1" port="18515"
           protocol="tcp" template="RSYSLOG_SyslogProtocol23Format")
}

The forwarding connection is plaintext TCP restricted to loopback for this experiment. It is not a production TLS configuration.

Create the separate destination

Save this as sink.conf:

module(load="imtcp")
template(name="marker" type="string" string="%msg%\n")

ruleset(name="sink") {
    action(type="omfile" file="/ABS/LAB/remote.log" template="marker")
}

input(type="imtcp" address="127.0.0.1" port="18515" ruleset="sink")

Validate both configurations, using the actual lab path:

rsyslogd -v
rsyslogd -N1 -f /ABS/LAB/main.conf
rsyslogd -N1 -f /ABS/LAB/sink.conf

Both validations exited successfully in all three test variants. The validation output ended with:

rsyslogd: End of config validation run. Bye.

Validation does not start the listeners or prove that the output files will receive messages. See rsyslog’s troubleshooting reference.

Start the destination first, then the sender, in separate terminals:

rsyslogd -n -i /ABS/LAB/sink.pid -f /ABS/LAB/sink.conf
rsyslogd -n -i /ABS/LAB/main.pid -f /ABS/LAB/main.conf

These commands start additional foreground processes, create the lab socket and listen on the two loopback ports. Check their startup errors before sending messages. They do not reload the system service. Stop each foreground process with Ctrl-C after the test.

Send two synthetic messages

With both listeners running, execute this Python 3 example after replacing the socket path:

import socket

with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as local:
    local.connect("/ABS/LAB/input.sock")
    local.send(b"<14>Nov 15 12:00:00 routeprobe: route-001")

with socket.create_connection(("127.0.0.1", 18514), timeout=3) as remote:
    remote.sendall(b"<14>Nov 15 12:00:00 lab routeprobe: route-002\n")

The local socket message omits the hostname; the network message includes it. These are synthetic Syslog fixtures, not captured application logs.

For the baseline, both local.log and remote.log contained these message bodies. The template preserves the leading space in each body:

 route-001
 route-002

Change one routing condition at a time

Stop both lab processes between variants. Use a fresh lab directory, update its absolute paths, and validate again so old rows cannot be mistaken for new arrivals.

For the early-stop variant, add stop immediately after the file action inside 10-local.conf. Leave both inputs bound to routed.

For the binding variant, remove that stop and change only the sender’s TCP input to ruleset="isolated". Leave the Unix socket bound to routed.

Variant Expected local.log Observed local.log Expected remote.log Observed remote.log Observed isolated.log
Baseline 001, 002 001, 002 001, 002 001, 002 No file
Stop after local action 001, 002 001, 002 No markers No file No file
TCP bound to isolated 001 001 001 001 002

The numbers abbreviate route-001 and route-002. The isolated ruleset was expected to receive only 002 in the binding variant, which matched the observation.

All three configurations validated. Their destinations differed because the processing path changed. The early stop retained the local copies while preventing forwarding. The binding change sent only the TCP message to isolated.log.

These results establish routing for the two fixtures on the tested version. They do not demonstrate behavior during an outage, queue recovery, sustained load or delivery to an external receiver.

Check the forwarded marker in Fluxtail

Once the intended ruleset reaches its forwarding action, use the existing Syslog forwarding guide for Fluxtail’s dedicated TLS receiver configuration. Use the assigned hostname and port, certificate verification and allowed source addresses. Dedicated ports require a paid Starter or Pro plan; the trial uses shared receivers.

Send a new harmless marker through the source input you just traced. Open the intended stream in Live Tail and filter by the marker’s message text. A matching row establishes that Fluxtail received that marker. An empty search does not identify the failed stage: check the stream and time range, then work backward through receiver settings, forwarding errors and the input’s routing rules.

For the wider collection arrangement, see rsyslog centralized logging. The Syslog server guide explains the receiving side. Keep routing tests separate from transport tests so each result answers a specific question.