FailedCreatePodSandBox: Find the Error Behind ContainerCreating

Kubernetes runs containerized applications across worker machines called nodes. A pod groups containers that share resources, including networking. Before application containers can start, the node's kubelet asks its container runtime to prepare a pod sandbox: the environment in which those containers will run. A Container Network Interface (CNI) plugin handles pod networking where required. Kubernetes documents this networking boundary.

FailedCreatePodSandBox means that preparation failed. It names a failed operation, not its cause. The complete error may point to the runtime, its sandbox image, or a network-plugin operation. If the application container never started, it has no application output for kubectl logs to retrieve.

You may see ContainerCreating in the STATUS column while the Pod phase remains Pending. These describe different things: ContainerCreating is a displayed status associated with container startup, while the phase summarizes the pod's lifecycle. The warning alone cannot identify the broken component. The Pod lifecycle reference explains these distinctions.

Why the final timeout may hide an earlier failure

Cilium is a Kubernetes networking implementation. In Cilium issue #48778, opened on September 16, 2026, alexalbr reported sandbox networking failures with Cilium 1.20.2 on a kernel without BTF, contrasting the behavior with 1.20.1. The report included endpoint API errors and timeouts alongside agent initialization errors involving missing kernel BTF.

BTF supplies kernel type information used when loading networking programs. In September 18–21 replies, maintainers emphasized that BTF was already a requirement. The discussion also questioned whether older releases consistently avoided that dependency.

The diagnostic lesson is to inspect the agent behind the failing endpoint operation. An API timeout can be the visible result of an earlier initialization problem. This report does not establish a universal cause for sandbox failures or a safe downgrade for other clusters. A matching warning is a reason to compare component versions, kernel requirements and earlier diagnostics—not to assume the same fault.

Trace the sandbox failure on the affected node

These checks apply when a pod has an assigned node and reports sandbox creation failures. They help locate the failed operation and collect evidence for a targeted correction. They do not repair the node or prove a root cause by themselves. Preserve the relevant events and logs before restarting components or replacing pods.

Capture the pod identity and full event

Replace the namespace and pod name with the affected pod. These commands read its state:

ns=checkout
pod=checkout-api-7c9d8f6b5f-m4r2x

kubectl -n "$ns" get pod "$pod" -o wide
kubectl -n "$ns" get pod "$pod" \
  -o jsonpath='{.metadata.uid}{"\n"}{.spec.nodeName}{"\n"}{.status.phase}{"\n"}'
kubectl -n "$ns" describe pod "$pod"

Record the UID, assigned node and failure window. The UID distinguishes this pod from a later replacement with a similar name. Read the complete Events message, including the nested error after the runtime's rpc error text. Kubernetes recommends inspecting pod details and events when debugging startup.

For events tied to that exact pod identity:

uid=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.metadata.uid}')
kubectl -n "$ns" get events \
  --field-selector "involvedObject.uid=$uid" -o yaml

Inspect the event message, count and available timestamps. Repeated events can summarize several attempts; do not treat them as a complete, precisely ordered execution trace. Events also have limited retention.

Follow the operation named in the message:

If no node is assigned, investigate scheduling first. If an application container has already run, its output may also help; the kubectl logs guide covers current and previous container logs.

Inspect the node runtime and network agent

Get the assigned node directly from the affected pod:

node=$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.spec.nodeName}')
kubectl get node "$node" -o wide

For Cilium installations using its documented label, select the agent on that node:

kubectl -n kube-system get pods \
  -l k8s-app=cilium \
  --field-selector "spec.nodeName=$node" -o wide

Use your network plugin's namespace and labels for other implementations. Inspect the selected pod's containers before choosing which log to read:

agent=cilium-example
kubectl -n kube-system get pod "$agent" \
  -o jsonpath='{.spec.containers[*].name}{"\n"}{.spec.initContainers[*].name}{"\n"}'

kubectl -n kube-system logs "$agent" -c cilium-agent \
  --since-time=2026-11-05T09:14:00Z \
  --timestamps=true --tail=300

Replace the example agent name and timestamp. This snapshot starts at the chosen time and is capped at 300 lines; it has no upper time boundary. Heavy output may hide the beginning of the window. Use retained logs with explicit start and end times when needed. If the selected container restarted, inspect its previous instance separately with --previous; that does not retrieve every older instance. These options are documented in the kubectl logs reference.

An agent displaying Running does not establish that networking works. Read initialization failures and use the plugin's own health checks. Cilium's troubleshooting documentation describes its component checks.

On systemd Linux nodes, kubelet and runtime diagnostics normally go to journald. Through your approved node-access method, inspect the actual service units. For a node using kubelet.service and containerd.service:

journalctl -u kubelet.service -u containerd.service \
  --since='2026-11-05 09:14:00 UTC' \
  --until='2026-11-05 09:16:00 UTC' \
  --utc --no-pager -o short-iso-precise

Confirm unit names locally; distributions and runtimes differ. Managed Kubernetes services may expose these records through provider diagnostics instead of node access. Application stdout collection alone does not collect kubelet journals. Kubernetes logging architecture distinguishes these sources.

Compare a bounded timeline

The following is an original, synthetic teaching fixture inspired by the diagnostic pattern in the Cilium report. It is not captured output or a reproduction. Names, timestamps, UID and messages are invented; messages are simplified.

{"time":"2026-11-05T09:14:02Z","node":"worker-7","component":"cilium-agent","message":"Datapath initialization failed: kernel BTF unavailable"}
{"time":"2026-11-05T09:14:20Z","node":"worker-7","component":"kubelet","pod_uid":"11111111-2222-4333-8444-555555555555","message":"FailedCreatePodSandBox: network ADD failed: endpoint API timeout"}
{"time":"2026-11-05T09:14:21Z","node":"worker-8","component":"kubelet","message":"Unrelated node record"}

The first row identifies only a node and component. It has no pod UID. The second identifies the synthetic pod and its failed network operation. The third belongs to another node.

Save those rows as timeline.jsonl. This Python 3 example selects the affected node within a start-inclusive, end-exclusive UTC window:

import json

with open('timeline.jsonl') as source:
    for line in source:
        row = json.loads(line)
        if (row['node'] == 'worker-7'
                and '2026-11-05T09:14:00Z' <= row['time'] < '2026-11-05T09:16:00Z'):
            print(line.rstrip())

It returns the first two rows. String comparison works here because every timestamp uses the same fixed-width UTC format; parse timestamps before comparing mixed formats or offsets.

This selection preserves the preceding agent failure instead of filtering it away for lacking a pod UID. The sequence supports investigating agent initialization. It does not prove that the first event caused the second. Check component health, runtime records and clock alignment before making that conclusion.

Verify the correction

Choose a correction supported by the full error and the documentation for your installed runtime or network plugin. An image-access failure, exhausted address pool and incompatible kernel require different changes. Do not use blanket restarts, pod deletion or network-directory cleanup as a substitute for diagnosis.

After the approved change, inspect the pod again. Confirm that sandbox preparation succeeds, expected containers start, and readiness checks pass. For a serving application, verify an appropriate read-only health request through its normal access path. A missing warning alone does not establish recovery.

If the workload creates a replacement pod, record its new UID and node. Success on another node does not show that the original node has recovered. Compare new attempts and component diagnostics on the original node before closing that part of the investigation.

Search retained node diagnostics in Fluxtail

When node and network-agent logs already reach Fluxtail, select their stream and the failure's start and end times. Search a stable message fragment from the full event, such as endpoint, then inspect preceding agent records. Narrow by host or service only after checking the values stored in an actual row. Fluxtail documents stream, time, message and field filters.

Inspect available pod identifiers manually. A node-agent record may have no pod UID, and filtering every row by that UID can remove the earlier failure you need. Fluxtail searches retained records; it does not inspect Kubernetes state or automatically join events. An empty search means no retained row matched the active filters.

Collection must cover the right source. Fluent Bit's systemd input can read selected journal units. A collector can normalize those records for Fluxtail's documented HTTP JSON ingestion, but journal collection and field mapping must be configured and verified separately. Container stdout collection cannot recover a kubelet journal it never read.

The Kubernetes logging guide explains source coverage, and Kubernetes log aggregation covers bringing those records together. The live log viewer can help watch fresh attempts once collection is working. Keep the bounded historical window as evidence of what happened before the correction.