Root cause analysis (RCA) is a structured investigation into why an unwanted event happened and what system changes could reduce its recurrence or impact. Its goal is not to find one person to blame or to label the last visible error as “the cause.” In SRE, the useful outcome is an evidence-supported explanation of the trigger and contributing conditions, followed by actions that can be owned and verified. Google describes this as a central purpose of a blameless postmortem.
RCA usually follows restoration. During an outage, responders first protect users and restore service with an approved mitigation. A restart, rollback, or traffic shift can stop the symptom without explaining why the system needed it. Preserve the facts gathered during response, then investigate the mechanism and the controls that failed or were missing. Do not delay safe mitigation merely to complete a causal diagram.
Separate symptom, trigger, and contributing conditions
A useful analysis names several kinds of fact rather than collapsing them into one “root cause” sentence:
- The symptom is the observed user or system effect: requests failed, jobs stopped completing, or data was unavailable.
- The trigger is the event that started this occurrence: a particular deployment, configuration change, dependency failure, or traffic shift.
- A contributing condition made the trigger harmful or allowed the failure to spread: inadequate validation, a missing limit, a stale runbook, insufficient isolation, or an alert that did not fire.
- A cause is a supported mechanism linking conditions and events to the observed impact. A complex incident can have several causes; the aim is to identify changeable parts of the system, not force one label.
Google's postmortem-analysis guidance separates triggers from root-cause categories so repeated patterns can be studied. That distinction prevents a superficial conclusion such as “the deploy caused it” from ending the review. The deployment may have been the trigger; the question is why a flawed change reached users, why it produced that particular failure, and why detection or containment did not limit the effect.
Treat these categories as a working model, not a rigid taxonomy. A factor can be a trigger in one event and a standing condition in another. State uncertainty where evidence is incomplete.
Write the problem statement before exploring causes
Start with the affected service and user journey, the observable failure, the start and end of impact, and the population affected. Keep the statement factual: “Between these times, eligible checkout requests in this region had elevated failures,” not “the database migration broke checkout.” The latter is already a hypothesis.
Define the measurement boundary. A client-visible error rate may differ from an application exception count; retries may hide a dependency failure from users or multiply backend requests. If a batch job exits successfully but produces stale output, technical success and business success diverge. Use the same route, service, population, and time zone when comparing before and after. Record whether an estimated impact count is complete, sampled, or inferred.
Capture what the team knows, what it believes, and what remains unknown as separate entries. That small discipline makes later corrections possible without rewriting the history. It also keeps the review from treating a plausible story as established fact.
Reconstruct a timeline from independent evidence
Build an ordered sequence of the first known symptom, alerts, recent changes, internal observations, mitigation attempts, and verified recovery. For each point, record its source and timestamp. Useful sources include monitoring metrics, deployment and configuration history, application logs, traces if instrumented, support reports, and operator notes. Google’s postmortem practice emphasizes accurate incident detail, supporting data, and clear ownership of the resulting review.
Do not assume that every timestamp marks the original event. A log may have both event time and ingest time; clocks can skew, buffers can delay records, and some events may be filtered or lost. A trace may cover only sampled requests. A deployment marker may indicate start of a rollout rather than the moment traffic reached a new version. Use these limits to widen an uncertain interval rather than manufacture a precise order.
Compare an affected path with an unaffected peer where possible: another route, region, release, or instance handling the same kind of work. This is often more informative than searching only ERROR lines. Preserve the raw record or source link behind each important assertion, with appropriate access controls and without copying secrets or customer data into the postmortem.
A compact hypothetical example
Suppose a configuration rollout is followed by a rise in checkout failures. The team shifts traffic back and verifies that the customer-visible error rate returns to its prior range. That establishes a useful mitigation and a temporal relationship, not yet the full cause.
The review might then find that the new configuration rejected one valid request shape, the pre-release test set lacked that shape, and rollout checks aggregated all routes together so the affected route was obscured. The configuration change is the trigger; the incompatible rule is the failure mechanism, while the test gap and coarse monitoring are contributing prevention and detection gaps if evidence supports them. Each claim needs evidence: a bounded comparison of requests, configuration versions, test coverage, and route-level outcomes. If the evidence supports only the incompatibility, leave the other proposed factors as hypotheses until tested. This example is illustrative, not an account of a real incident.
Test causal explanations instead of stopping at a plausible story
For each proposed link, ask: what observation would we expect if it were true, what observation would argue against it, and can the change be reproduced safely? A controlled test, a comparison with a healthy peer, or a narrow code/configuration review can strengthen a hypothesis. A temporal match alone does not show that the change caused the effect. The fact that a rollback worked narrows the search but may not distinguish code, config, cache state, or a dependency interaction.
The familiar 5 Whys technique asks repeatedly why a symptom occurred. It is useful for keeping a review from ending at “the server crashed.” It is not a proof method or a requirement to stop after exactly five questions. A linear chain can conceal simultaneous factors and can drift into speculation or blame. Stop when evidence runs out; branch when several mechanisms contributed. For a complex system, a cause-and-effect diagram or fault tree can help enumerate and test branches. NASA's Goddard investigation handbook describes flowing logic from an event to its causes; the method is a structure for inquiry, not a substitute for facts.
Counterfactual questions can help: if this control had existed, would the trigger still have caused user impact? If yes, the proposed action may be too weak. But counterfactuals remain models; confirm them with tests or operational evidence where feasible. Do not claim every contributing factor has an exact quantitative share of blame.
Keep the review blameless and specific
Blameless does not mean vague. Name the action, decision, interface, and system behavior that mattered without describing a person as the cause. Ask what information and constraints were available at the time. A responder following a runbook that omitted a dependency check reveals a design or documentation gap; “operator error” hides it. Google's postmortem-culture guidance explains why blame discourages disclosure of facts needed for prevention.
Invite people who saw different parts of the event to review the timeline and causal claims. Note disagreements and unresolved questions. Reviewers should challenge an unsupported conclusion as readily as an omitted cause. If a security, privacy, or compliance concern was involved, restrict the evidence appropriately while keeping the explanation useful to those who must act on it.
Turn findings into verified changes
RCA is unfinished if it ends with “be more careful.” Group follow-up work by what it changes:
- Prevention: reject invalid configuration, add a contract test for the missed request shape, or reduce the blast radius of a rollout.
- Detection: measure the affected user journey or route with an actionable owner and a deliberate no-data state.
- Mitigation: make a safe rollback, traffic shift, or isolation step easier to execute and verify.
- Learning: update a runbook, training exercise, or shared design review where it changes future decisions.
Each action needs one accountable owner, a due date or priority, a tracking location, and an observable completion test. “Improve monitoring” is not verifiable; “add a route-level failure-rate alert, test its firing and delivery, and link its runbook” is. The SRE Workbook's postmortem critique calls out vague actions, missing ownership, and plans that only mitigate rather than prevent recurrence. It does not imply every review needs an enormous project: choose changes proportional to impact and recurrence risk.
After implementation, verify the control under a safe test or later production observation. Close the action only when the behavior is shown, not merely when a ticket is marked done. Revisit repeated incident patterns to see whether the action changed the failure mode; a new alert that no one can act on is not a successful fix.
Use logs as one source, not an automatic answer
Structured logs can help reconstruct what a service did, which release handled work, and where a request or job diverged. Metrics establish population-level symptoms; traces can show a sampled, instrumented path; deployment and configuration records establish change history. None is complete by default. The strongest explanation often combines sources and states the gaps.
Fluxtail is a paid Starter/Pro, logs-focused service. Its search and filters and Live Tail can help inspect application records that a supported source has delivered and mapped. Its built-in AI log chat can help ask bounded questions about retained logs, but a generated explanation is a hypothesis to check against raw rows and independent evidence. Fluxtail does not perform automatic RCA or provide native metric-series storage, metrics dashboards, traces, deployment history, or an incident-management process. Prometheus and StatsD samples can be accepted as log events, not native time series.
For a repeatable investigation, retain a safe event timestamp, service, environment, version, event name, and correlation value where the source genuinely provides them. Query a bounded window and verify the original rows. If no event appears, first check whether instrumentation, collection, filtering, or retention could explain the gap. Do not turn absence of a log into proof that an operation never happened. See how to read logs for a practical evidence sequence and mean time to resolution for distinguishing recovery clocks from later closure.
The goal of root cause analysis is a better-controlled system, not a satisfying one-line story. A useful review can say what happened, how the conditions combined, what remains uncertain, who owns the changes, and how the team will know those changes worked.