Centralized Logging on Kubernetes: Architecture and Fluent Bit Setup

Centralized logging on Kubernetes needs to survive pod restarts, node replacement, log rotation, and temporary destination failures. A practical design runs one collector on each eligible node, reads container stdout and stderr, attaches Kubernetes metadata, and forwards bounded batches to a store outside the cluster.

This guide builds that path with Fluent Bit 5.1.2 and an explicit Fluxtail HTTP JSON receiver. It focuses on the contracts that determine whether an incident is searchable later: node coverage, file offsets, multiline parsing, metadata mapping, finite buffering, receiver credentials, and end-to-end verification.

Use one collector per node

Kubernetes does not provide native cluster-level log storage. Its documented approaches include a node-level agent, sidecar collectors, and direct application delivery. For the common node-level pattern, Kubernetes recommends running the logging agent as a DaemonSet so each eligible node gets a collector pod. See the official Kubernetes logging architecture and DaemonSet documentation.

The data path is:

1. The application writes to stdout or stderr.
2. The container runtime records each stream using the CRI logging format.
3. The kubelet places container logs under /var/log/pods; /var/log/containers normally contains convenient symlinks to those files.
4. Fluent Bit tails the files and saves its offsets in a SQLite database.
5. The Kubernetes filter adds pod and workload context.
6. Fluent Bit buffers and sends records to the exact Fluxtail receiver URL.
7. Fluxtail stores the records in the receiver's named stream for Live Tail, search, and filters.

This article covers that end-to-end architecture. The broader Kubernetes logging overview explains the product capabilities, while the Kubernetes log aggregation guide covers the wider collection and retention problem.

Mount the container logs and collector state

The DaemonSet needs read-only access to /var/log, including the pod files and container symlinks. It also needs a writable directory for its tail database and filesystem chunks. Keep this state separate from the read-only log mount.

The important DaemonSet fragment looks like this:

apiVersion: v1
kind: Namespace
metadata:
  name: observability
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-pod-metadata
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-pod-metadata
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-pod-metadata
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: observability
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: observability
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:5.1.2
          env:
            - name: FLUXTAIL_RECEIVER_HOST
              value: ingest.fluxtail.io
            - name: FLUXTAIL_RECEIVER_PATH
              valueFrom:
                secretKeyRef:
                  name: fluxtail-receiver
                  key: path
            - name: FLUXTAIL_TOKEN
              valueFrom:
                secretKeyRef:
                  name: fluxtail-receiver
                  key: token
            - name: FLUXTAIL_CLUSTER_NAME
              value: production-cluster
          volumeMounts:
            - name: container-logs
              mountPath: /var/log
              readOnly: true
            - name: collector-state
              mountPath: /var/lib/fluent-bit
            - name: config
              mountPath: /fluent-bit/etc
      volumes:
        - name: container-logs
          hostPath:
            path: /var/log
        - name: collector-state
          hostPath:
            path: /var/lib/fluent-bit
            type: DirectoryOrCreate
        - name: config
          configMap:
            name: fluent-bit

The example pins the current Fluent Bit 5.1.2 release. Validate that version in a non-production cluster, then pin an image digest for production. Change production-cluster to the stable name used in your searches.

A DaemonSet only covers nodes that can schedule it. Add tolerations for control-plane or specially tainted nodes when their logs belong in the same pipeline. Confirm coverage by comparing eligible nodes with ready Fluent Bit pods:

kubectl get nodes
kubectl get pods -n observability -l app=fluent-bit -o wide

The pod count alone is insufficient. The node names in the second command must cover the intended nodes in the first.

Tail CRI logs without replaying every file

Fluent Bit's tail database stores the inode and last committed offset for each file. Without it, a collector restart can replay old records or resume from the wrong position, depending on the input settings and file state.

This configuration tails the standard container path, preserves runtime stream information, and keeps the offset database on the writable state mount:

[SERVICE]
    Flush                     2
    Log_Level                 info
    Storage.Path              /var/lib/fluent-bit/storage
    Storage.Sync              normal
    Storage.Checksum          on
    Storage.Max_Chunks_Up     64
    Storage.Backlog.Mem_Limit 64M
    Multiline_Buffer_Limit    2M

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    DB                /var/lib/fluent-bit/tail.db
    DB.Sync           normal
    Read_From_Head    off
    Refresh_Interval  10
    Rotate_Wait       30
    Skip_Long_Lines   on
    Multiline.Parser  docker, cri
    Storage.Type      filesystem

Read_From_Head off is appropriate when introducing collection without importing every existing file from its beginning. Change that decision deliberately for a migration or controlled backfill.

The docker, cri multiline setting repairs records split by the container runtime. It also preserves the parsed stream value, allowing stdout and stderr to remain distinguishable. Fluent Bit documents both the offset database and container multiline modes in its tail input reference.

An unexpected shutdown can cause a small replay from the last committed offset. Consumers should therefore tolerate occasional duplicate events instead of assuming exactly-once delivery.

Add Kubernetes metadata before routing

A filename can identify a pod instance, but useful searches need stable context such as namespace, pod, container, node, and labels. Fluent Bit's Kubernetes filter derives the basic identity from the expanded tail tag, then queries the Kubernetes API server for additional metadata.

[FILTER]
    Name                    multiline
    Match                   kube.*
    Multiline.Key_Content   log
    Multiline.Parser        go, python, java
    Buffer                  on
    Emitter_Storage.Type    filesystem
    Flush_MS                2000

[FILTER]
    Name             kubernetes
    Match            kube.*
    Kube_URL         https://kubernetes.default.svc:443
    Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix  kube.var.log.containers.
    Labels           on
    Annotations      off
    Merge_Log        on
    Keep_Log         on

[FILTER]
    Name    lua
    Match   kube.*
    Script  /fluent-bit/etc/map_fluxtail.lua
    Call    map_fluxtail

[FILTER]
    Name   grep
    Match  kube.*
    Regex  message .

The multiline filter is first because buffered records are re-emitted at the start of the filter chain. The built-in go, python, and java parsers are starting points; the Java parser targets the Google Cloud Java stack-trace format. Replace or extend them when sanitized samples from your application do not match.

The service account above can read pod metadata. Add namespace permissions only when the chosen configuration enables namespace metadata. Use the narrowest RBAC rules that still satisfy the filter.

Merge_Log on attempts to parse a JSON object found in the log field. Applications that already emit one JSON object per line usually produce cleaner results than applications that mix JSON, prose, and stack traces on the same stream.

Save this small Lua filter as map_fluxtail.lua. It supplies Fluxtail's required top-level message field, maps collector metadata to the documented Kubernetes filter fields, and retains pod labels as exact-match labels:

function map_fluxtail(tag, timestamp, record)
  local kubernetes = record["kubernetes"] or {}
  local message = record["message"] or record["log"]

  if type(message) ~= "string" or message == "" then
    return -1, timestamp, record
  end

  record["message"] = message
  record["log"] = nil
  record["k8s_namespace"] = kubernetes["namespace_name"] or ""
  record["k8s_pod_name"] = kubernetes["pod_name"] or ""
  record["k8s_container_name"] = kubernetes["container_name"] or ""
  record["k8s_node_name"] = kubernetes["host"] or ""
  record["k8s_cluster_name"] = os.getenv("FLUXTAIL_CLUSTER_NAME") or ""
  record["service_name"] = record["service_name"] or kubernetes["container_name"]
  record["host"] = record["host"] or kubernetes["host"]

  local labels = record["labels"]
  if type(labels) ~= "table" then labels = {} end
  if type(kubernetes["labels"]) == "table" then
    for key, value in pairs(kubernetes["labels"]) do
      labels[key] = tostring(value)
    end
  end
  if record["stream"] ~= nil then
    labels["container_stream"] = tostring(record["stream"])
  end
  if record["multiline_truncated"] ~= nil then
    labels["multiline_truncated"] = tostring(record["multiline_truncated"])
  end
  record["labels"] = labels
  record["stream"] = nil
  record["multiline_truncated"] = nil

  return 1, timestamp, record
end

The exact record sent by Fluent Bit remains collector-dependent. A nested field such as kubernetes.namespace_name does not automatically prove that the destination exposes it as a Kubernetes namespace filter. Confirm the received shape, then map any additional deployment, service, and severity fields to the destination's documented field contract where necessary.

The current Fluent Bit Kubernetes filter reference describes its tag parsing, metadata cache, label behavior, and API access.

Handle application stack traces separately

Container-runtime multiline parsing and application multiline parsing solve different problems.

The cri parser joins payload fragments marked partial by the runtime. It does not automatically know that several complete CRI records belong to one Java exception, Python traceback, or Go panic. Fluent Bit provides built-in parsers for common language stack traces and configurable state-based parsers for application-specific formats.

Prefer one-line structured JSON events when the application can produce them. If multiline output is unavoidable:

• select a parser for the actual language and format;
• test its first-line and continuation rules against real sanitized samples;
• test stack traces emitted on stdout and stderr separately;
• set a finite multiline buffer limit;
• verify that two consecutive exceptions are not combined;
• verify that an incomplete exception eventually flushes.

Fluent Bit's multiline parsing documentation notes that its multiline buffer can exceed ordinary tail line limits. Multiline_Buffer_Limit bounds that memory; when the configured limit is exceeded, current Fluent Bit releases emit a truncated record with metadata indicating truncation.

A representative result may look like this:

{
  "message": "java.lang.IllegalStateException: inventory unavailable\n    at checkout.Inventory.reserve(Inventory.java:84)",
  "service_name": "checkout-api",
  "k8s_namespace": "production",
  "k8s_pod_name": "checkout-api-7d8f6c9d9b-2rjxm",
  "k8s_container_name": "checkout-api",
  "k8s_node_name": "worker-3",
  "k8s_cluster_name": "production-cluster",
  "labels": {"container_stream": "stderr"}
}

Field order, timestamps, pod names, and any downstream normalization can differ at runtime.

Bound filesystem buffering and backpressure

Filesystem buffering protects the collector when the receiver is briefly unavailable or slower than the incoming log rate. It must still have a finite capacity.

The earlier configuration enables filesystem chunks under /var/lib/fluent-bit/storage. Add a limit to the Fluxtail output queue:

[OUTPUT]
    Name                     http
    Match                    kube.*
    Host                     ${FLUXTAIL_RECEIVER_HOST}
    Port                     443
    URI                      ${FLUXTAIL_RECEIVER_PATH}
    Format                   json
    TLS                      on
    TLS.Verify               on
    Header                   Authorization Bearer ${FLUXTAIL_TOKEN}
    Header                   Content-Type application/json
    Storage.Total_Limit_Size 2G
    Retry_Limit              False

Create an active Fluxtail HTTP JSON receiver, then copy its host, path, and receiver-bound token exactly. The path follows the receiver-specific form /v1/receivers/RECEIVER_ID/logs; it must come from the same receiver as the token. Keep the token in a Kubernetes Secret, not a ConfigMap or image.

After saving the combined Fluent Bit sections as fluent-bit.conf and the mapping function as map_fluxtail.lua, create the receiver Secret and configuration, then apply the manifest:

kubectl create namespace observability --dry-run=client -o yaml \
  | kubectl apply -f -

kubectl create secret generic fluxtail-receiver \
  --namespace observability \
  --from-literal=path='/v1/receivers/RECEIVER_ID/logs' \
  --from-literal=token='YOUR_RECEIVER_TOKEN'

kubectl create configmap fluent-bit \
  --namespace observability \
  --from-file=fluent-bit.conf \
  --from-file=map_fluxtail.lua

kubectl apply -f fluent-bit-daemonset.yaml

Fluxtail's shared HTTP receiver accepts one JSON object or an array over TLS port 443. Fluent Bit's HTTP Format json sends batches that fit this receiver contract. Retry_Limit False keeps retrying failed chunks; the finite output queue remains the hard disk bound. A dedicated Fluent Forward receiver is another option, but it has a different assigned endpoint and source-IP authentication model. Do not combine settings from the two receiver types. The Fluxtail receiver matrix explains those boundaries.

A 2G queue is an example, not a universal recommendation. Size it from the peak input rate and the outage interval you intend to absorb, then ensure the node volume has extra headroom. When an output reaches Storage.Total_Limit_Size, Fluent Bit discards the oldest queued chunk to accept newer data. The buffer is therefore a bounded failure window, not permanent storage.

Monitor filesystem use, output retries, dropped chunks, paused inputs, and collector restarts. Fluent Bit's backpressure documentation explains how memory and filesystem limits interact.

Verify every boundary with one known event

A healthy DaemonSet does not prove that records reach the correct stream. Test the full path with a unique marker from a known namespace and keep the pod alive long enough for metadata enrichment:

CHECK_ID="fluxtail-k8s-check-$(date +%s)"

kubectl run fluxtail-log-check \
  --namespace default \
  --image=busybox:1.36.1 \
  --restart=Never \
  --labels=app=fluxtail-log-check \
  --command -- sh -c \
  "echo \"$CHECK_ID stdout\"; echo \"$CHECK_ID stderr\" >&2; sleep 600"

Then verify:

1. The pod is running on a node covered by the DaemonSet.
2. The matching Fluent Bit pod can read /var/log/containers.
3. Fluent Bit reports no tail database, RBAC, TLS, or HTTP errors.
4. The Fluxtail receiver shows accepted traffic.
5. The marker appears in the intended named stream.
6. The container_stream label distinguishes the stdout and stderr records.
7. Namespace, pod, container, node, and labels are present where the configured mapping supports them.
8. A tested stack trace arrives as one event when application multiline handling is enabled.

In Fluxtail Live Tail, begin with the stream and marker text. Add Kubernetes field filters only after confirming those fields arrived with the expected mapping. An empty result with several filters selected only means that no retained record matches all of them. The Fluxtail search and filter guide lists the available fields.

Diagnose the common failure modes

Keep the architecture observable after the first successful test. Alert on missing collectors, sustained retries, queue growth, disk pressure, and ingestion gaps. Re-run the known-event test after collector upgrades, receiver-token rotation, parser changes, and cluster-runtime changes.

For a narrower configuration path, see Fluent Bit with Kubernetes. The key outcome is not merely that Fluent Bit is running. It is that a record from any covered node reaches the intended Fluxtail stream with enough verified context to find it during an incident.