Kubelet Logs: Find Node Diagnostics Beyond kubectl logs

Kubelet logs explain what Kubernetes’ node agent reported while managing pods. They are different from application logs: kubectl logs retrieves a container’s standard output and standard error, rather than the kubelet service’s own diagnostics.

Kubernetes runs containerized applications across machines called nodes. Each node’s kubelet manages its assigned pods and asks a container runtime, such as containerd, to run their containers. A runtime failure can stop a container from starting before the application has any output to show. Kubernetes documents these separate logging paths in its logging architecture.

The commands below apply to Linux nodes using systemd, where kubelet and runtime diagnostics commonly go to the journal. Other installations can use files, different service names or provider-managed logging. Windows nodes need their configured Windows logging path. A failed log request or an empty journal selection does not establish that the node is healthy.

Find kubelet logs on the affected node

With authorized node access, these read-only checks help locate retained service records and compare them with runtime failures. They do not restart services, recover deleted history or prove that a workload has recovered.

Identify the node and container runtime

Start from the affected pod. Replace the namespace and pod name with your own:

kubectl get pod checkout-api-7c8d9 -n shop -o wide

Read the NODE column. If it is empty, the pod has not been assigned to a node; there is no assigned node’s kubelet journal to inspect yet. For an assigned pod, inspect that node’s reported runtime:

kubectl get node node-a \
  -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}{"\n"}'

A value beginning with containerd:// identifies containerd. Do not assume every cluster uses it. These API observations can also be stale when a node cannot report its status. Kubernetes’ cluster troubleshooting guide explains inspecting node status and events.

Connect to the selected node through your established access method. Running journalctl on your laptop reads your laptop’s journal. For managed clusters, use the provider’s supported access or retained-log service; for example, Microsoft documents AKS kubelet-log access.

Select the service, boot and time window

On the node, check matching installed service files and loaded service units before selecting logs:

systemctl list-unit-files --type=service 'kubelet*' 'containerd*'
systemctl list-units --all --type=service 'kubelet*' 'containerd*'
journalctl --list-boots --no-pager

list-unit-files lists matching service files installed on the system and their enablement state. list-units --all lists matching units currently loaded in systemd’s memory, including inactive units. It does not list every installed service. Neither an installed file nor an enabled state proves that a service is running. The systemctl manual explains these separate views.

These name patterns can miss custom units. A distribution may run kubelet under another service. Follow that installation’s service configuration rather than treating a missing kubelet.service as proof that kubelet is absent.

For a node with kubelet.service, select a short interval around the failure:

journalctl -u kubelet.service -b \
  --since '2026-11-13 10:00:00 UTC' \
  --until '2026-11-13 10:10:00 UTC' \
  --utc --no-pager

Replace both timestamps. -b selects the current boot. If the failure preceded a reboot, use -b BOOT_ID with the identifier shown by --list-boots. Only retained boots are available. Both time boundaries are inclusive, so adjoining windows can repeat entries at their shared boundary. These selection rules are documented in the journalctl manual.

Journal access depends on the account’s permissions. Use an existing authorized account or approved elevation method. Do not change journal ownership or broaden access just to make a query work. The Linux system logging guide covers the wider relationship between journald and syslog.

Compare the matching runtime records

If containerd is the actual runtime and its local unit is containerd.service, query the same boot and interval:

journalctl -u containerd.service -b \
  --since '2026-11-13 10:00:00 UTC' \
  --until '2026-11-13 10:10:00 UTC' \
  --utc --no-pager

Use the same explicit boot ID in both commands when investigating an earlier boot. Keep successful records around the failure too; filtering only for the word error can hide useful context.

Compare pod names, pod UIDs, container IDs and request details where available. Nearby timestamps alone do not prove two messages describe the same operation. For sandbox-specific failures, continue with FailedCreatePodSandBox troubleshooting. For application output, use the separate kubectl logs guide.

Explain empty or unavailable results

An empty result means no accessible retained entries matched the selection. Check the node, service name, boot and timestamps before widening the interval.

A permission error means the account could not read the requested evidence. A connection failure means the access path failed. Neither establishes that the service emitted no logs. A reboot, journal rotation or volatile storage may have removed earlier records. An inaccessible node may leave only previously collected logs and API observations available.

Record the limitation directly: “The incident boot is no longer retained” is more useful than “No kubelet errors found.”

Read the failure sequence without assuming recovery

The following timeline is synthetic. Its messages are simplified teaching examples, not universal kubelet or containerd signatures. boot-1 and boot-2 are readable labels rather than actual journal boot IDs.

10:01:50Z  API event   pod=shop/checkout-api-7c8d9 assigned to node-a
10:02:00Z  node-b boot-2 containerd  runtime request timed out
10:02:00Z  node-a boot-1 containerd  runtime request timed out
10:02:00Z  node-a boot-2 containerd  runtime request timed out
10:02:01Z  node-a boot-2 kubelet     pod start failed: runtime request timed out
10:08:00Z  API observation         node-a Ready=True
10:08:05Z  API observation         checkout-api-7c8d9 Ready=True

Selecting node-a, boot-2 and the incident interval excludes the other node and earlier boot. The remaining runtime and kubelet records describe a reported failure sequence. They do not explain why the runtime timed out.

Check current status separately:

kubectl get node node-a
kubectl get pod checkout-api-7c8d9 -n shop -o wide

Inspect the relevant conditions and their times when the summary needs explanation. Node readiness does not guarantee every workload is ready. Pod readiness does not prove that an important user operation succeeds. Confirm the affected application through its normal health checks and a safe functional check.

Retain node-service logs in Fluxtail

Central collection helps when a node becomes unreachable or its local journal expires. It must explicitly include node-service logs. A collector that tails pod container files does not thereby collect kubelet or containerd journals; the centralized Kubernetes logging guide describes that separate container-file path.

Select journal sources explicitly

Fluent Bit’s Systemd input reads journald records. This input fragment selects either of two verified local units:

[INPUT]
    Name                 systemd
    Tag                  node.services
    Systemd_Filter       _SYSTEMD_UNIT=kubelet.service
    Systemd_Filter       _SYSTEMD_UNIT=containerd.service
    Systemd_Filter_Type  Or
    DB                   /var/lib/fluent-bit/node-services.db

This is an input fragment, not a complete deployment. The collector needs authorized access to the correct node journal and a persistent, writable location for its cursor database. The cursor records reading progress; it is not a backup of the journal. With no saved cursor, the default Read_From_Tail Off permits reading retained history. Plan that initial volume before enabling collection.

Map fields before relying on filters

For HTTP JSON delivery, deliberately map journal _HOSTNAME to host, _SYSTEMD_UNIT to service, and MESSAGE to message. Preserve the original event time and boot ID too. Inspect real records first: fields can be absent, and the destination does not automatically infer every journal mapping.

A representative synthetic destination record is:

{
  "host": "node-a",
  "service": "kubelet.service",
  "message": "Synthetic node diagnostic marker kubelet-check-20261113-a7"
}

Send verification records only to your intended receiver. Fluxtail’s documented HTTP JSON path uses an HTTPS POST to the exact receiver URL, https://ingest.fluxtail.io/v1/receivers/RECEIVER_ID/logs, with Content-Type: application/json and a receiver-bound Bearer token. Follow the single-event ingestion example for a command that keeps the token out of process arguments.

Verify the source and destination separately

A synthetic HTTP record checks the receiver path, not journal collection. For that path, HTTP 202 with accepted:1 means accepted for processing. Locate the same unique message in the intended stream to verify arrival.

Then choose one recent, harmless record from each actual service journal. Find those same records in Fluxtail, comparing time, host, service and message. This checks both source selections and their field mapping. A test message under an unrelated journal tag does not test a kubelet unit filter.

In Live Tail, narrow the intended stream by host and service, then inspect message text within the incident window. Keep a record of any collection gaps. Retained rows can show what the node services reported; current node status and application checks still determine whether service has recovered. The broader Kubernetes logging overview explains how these node records complement pod logs.