Incident investigation is the evidence-backed process of establishing what a service disruption affected, reconstructing what happened, testing possible explanations, and deciding what must change. It starts while responders are still reducing customer impact and continues after recovery. A useful investigation does not stop at “the error cleared,” and it does not label the first correlated event a root cause.
For a live service incident, keep two priorities visible: mitigate harm now and preserve enough evidence to learn why it happened. Google's effective troubleshooting guidance makes the same distinction: restore useful service during triage without abandoning observations needed for subsequent analysis. Investigation is the method; the monitoring, log-search, coordination, and status tools are supporting instruments.
Start with the impact, not a suspected cause
Confirm that there is an incident and describe its observable boundary. Which user journey or API operation is failing? When was it first observed? Is the impact limited to one region, tenant class, release, or dependency? What proportion of eligible requests or jobs is affected, and from which measurement source? Record what is known before discussing why it happened.
An alert is a starting signal, not a complete incident description. Check an external or user-facing signal where available, then compare it with internal service metrics and a small sample of failed requests. An increase in ERROR logs can reflect a changed log level rather than changed user impact; an apparently healthy service check may miss one broken route. Keep counts and denominators from the same observation boundary, and annotate low-traffic cases rather than treating one failed request as a stable rate.
Declare an incident according to the service's own severity and coordination rules. Google's incident-management chapter emphasizes clear responsibilities, a live incident record, and explicit handoffs. A small response can combine roles, but someone should still own the current impact statement and decisions. As more responders join, name who coordinates, who performs operational changes, who investigates evidence, and who communicates updates. Without that separation, unrecorded changes can destroy a useful comparison or worsen the outage.
Build a timestamped incident record
Use one shared record for the current service state, working hypotheses, owners, actions, and next update. A concise first entry can contain:
- The affected service and user journey, current impact, first observed time in UTC, and confidence or unknowns.
- The current mitigation state and person authorized to change production.
- Links to bounded, access-controlled evidence: an alert window, a representative request ID, deployment history, and relevant log or metric views.
- Each action with its time, operator, reason, expected effect, and observed result.
- The next evidence check and communication update.
Keep three kinds of statement separate. Observation: “The edge recorded 504 responses for checkout requests from 14:04 UTC.” Hypothesis: “A dependency wait may have consumed the gateway deadline.” Decision: “The authorized operator will shift traffic from the unhealthy target and verify the checkout path.” This prevents a plausible theory from turning into an unquestioned fact in later handoffs.
Use source timestamps and record the timezone. Where a collector adds an observed or ingest time, distinguish it from the event time; clocks can be skewed and delivery delayed. A sequence inferred from two independent systems may need a margin of uncertainty. Preserve links or stable IDs to original evidence, not just pasted screenshots or a model's summary. Do not dump whole production configurations, full request bodies, credentials, or broad customer logs into a shared incident channel.
Collect evidence from each relevant boundary
Choose evidence that could confirm or reject the current question. Start with the user-visible symptom and its entry point, then move through the actual request or job path: edge or load balancer, application, queue, dependency, and storage where applicable. Useful sources include response status and duration, application events, service and host metrics, separately instrumented traces, deployment/configuration changes, and authoritative business state. No one signal is a complete reconstruction.
For each source, ask what it observes and what it cannot. A gateway 504 proves that a gateway timed out waiting for its upstream response; it does not prove which dependency stalled. An application exception identifies a failing code path but may not show how many customers were affected. A missing log line may mean the operation did not run, or it may mean the event was never emitted, was dropped, was delayed, or fell outside retention. How to read logs expands on these completeness and ordering limits.
Limit queries to an incident window and exact service, environment, version, or request ID. Compare a failed request with a successful peer on the same route and release when possible. Look for the first divergence rather than collecting every warning that occurred afterward. Protect access to raw events: they can contain personal data or secrets even when the incident note is sanitized.
A log clue is not a conclusion
This single record is synthetic and sanitized; its fields illustrate an investigation, not a Fluxtail payload guarantee or a real customer event:
{"timestamp":"2026-09-16T14:04:18Z","severity":"ERROR","service_name":"checkout-api","environment":"prod","release":"r42","request_id":"req-example-42","message":"payment authorization call timed out"}
The event supports a narrow observation: one checkout-api execution logged a timeout at its source timestamp. It does not prove the payment failed, that every checkout request failed, that release r42 caused the timeout, or that the payment provider was at fault. First locate the edge request and application events for req-example-42. Then check whether the operation committed in the payment system, whether comparable successful requests took a different path, and whether timeout frequency changed at a consistent measurement boundary before and after the reported onset.
If request_id or release was never delivered into the log backend, search only the fields actually available. Do not invent correlation from timestamps alone. A trace ID is useful only when the application actually propagated trace context; a log viewer does not manufacture a trace. The record may also arrive late or be duplicated, so compare event time and observed time and inspect the collection path before asserting exact order or totals.
Test competing explanations safely
Write each hypothesis as a falsifiable question. “The deployment caused checkout failures” is too broad. “Requests on r42 have a higher timeout rate than requests on the prior release at the same edge and route, after accounting for traffic” defines a comparison. Another candidate may be a dependency outage, queue saturation, or network change. Test the most consequential plausible explanations with the least risky evidence first.
Google's troubleshooting model describes iterating between observations and hypotheses, and warns against mistaking correlation for causation. Use a known-good peer, a pre-incident window, or an unaffected region where the comparison is valid. Note confounders: traffic may have shifted, sampling may differ, one deployment may carry several changes, and an alert may lag the first failure. Record evidence that contradicts the favored theory, not only supporting rows.
Read-only inspection is usually the first test. A production change or replay is a separate, authorized decision. Replaying a checkout, payment, upload, or delete request can create a second effect even if the first response was lost. Establish the endpoint's idempotency and test path before reproduction. If a controlled change is needed to protect users, log its owner, scope, time, and expected result before applying it. One changed variable is easier to evaluate than several simultaneous edits.
Mitigate, then verify recovery
Do not wait for a polished root-cause story while users are losing service. A safe, preapproved mitigation may route around a demonstrably failing target, disable a broken path, or restore a dependency. The operational owner must judge side effects and authorization; a log search result alone is not approval to restart, roll back, or modify a production resource.
After an action, check the original user journey and the relevant failure signal. Did the affected requests complete? Did the error rate and latency recover at the same observation boundary? Are there unfinished jobs, duplicates, or partial writes? Watch for the failure moving to a different target or region. Record user-visible recovery time separately from later resolution or closure. The service may be restored by a workaround while the defect and long-term correction remain open.
Give stakeholders a factual update with impact, current mitigation, uncertainty, and next check. Do not turn an unverified hypothesis into a public cause. Continue to preserve enough context for a handoff. A reliable incident record includes the unsuccessful actions too; those results narrow the search and keep another responder from repeating the same risky step.
Turn findings into a useful follow-up
Once the service is stable, reconstruct a timeline from the original records and mark uncertain intervals. Describe the trigger, the conditions that allowed it to affect users, detection and response gaps, mitigation, and recovery evidence. There may be several contributing factors rather than one person or one “root cause.” Google's postmortem guidance focuses on learning without blame and on specific actions that reduce recurrence or impact.
Give each corrective action an owner, a due date, and a way to verify it worked: a test, a restored dependency check, a narrower alert, a capacity limit, or a runbook exercise. Separate immediate containment from durable prevention. Revisit actions after the relevant change ships; a completed ticket is not proof that the failure mode is gone. For suspected compromise or data exposure, follow the organization's security-response and evidence-preservation process as well. NIST's SP 800-61 Rev. 3 addresses cybersecurity incident response, which has obligations beyond ordinary service restoration.
The process does not depend on buying one product. An incident manager tracks ownership and updates; monitoring detects impact; logs, metrics, and traces provide different evidence; deployment and business systems verify state. For a separate comparison of those categories, see incident investigation tools. This guide remains a way to investigate with whatever authorized evidence the service actually has.
Search delivered log evidence in Fluxtail
Fluxtail is a paid Starter/Pro, logs-focused service—not an incident-command, paging, metrics, tracing, or automatic root-cause platform. Once a supported collector or receiver delivers logs, its documented search and filters can narrow retained events by stream, time, message, service, severity, labels, and Kubernetes fields when present. Live Tail lets the responder inspect a raw message and public log ID during the active incident. Field availability depends on the source and collector mapping; zero matching rows do not establish that no event occurred.
An authorized external agent can use Fluxtail's hosted MCP to query and summarize retained logs through one OAuth/PKCE-bound account. Verify the account with whoami, keep the query window bounded, and check any summary against raw rows. MCP operator changes to streams or receivers are separate from investigation: the documented flow proposes a mutation and requires a short-lived confirmation token before applying it. Neither an AI summary nor a missing-log diagnostic should silently become a production change.
Use those log results alongside the service's user-impact measurements and authoritative outcome checks. The goal is a defensible sequence from observation to tested explanation, safe mitigation, verified recovery, and owned follow-up—not a fast answer unsupported by the evidence.