Why Most Post-Mortems Fail
The post-mortem meeting ends. Someone writes a doc. It sits in Confluence, unread, until the same incident happens six months later.
That's not a documentation problem — it's a process problem. A post-mortem only has value if it surfaces the real contributing factors and produces action items that get completed. Here's how to write one that does both.
Start Before the Meeting: Collect a Timeline First
Don't walk into a review session with blank slides. Before anyone speaks, assemble a factual, timestamped timeline from:
- Monitoring alerts (when did the system first detect the issue?)
- On-call notifications and acknowledgements
- Chat logs (Slack, PagerDuty threads)
- Deployment and config change logs
- Status page updates
If your uptime monitoring tool records check history across regions, pull that data now. Knowing whether an outage started in us-east-1 before spreading elsewhere often reveals whether a deployment or a network event was the trigger — details that memory alone won't give you accurately.
A concrete timeline depersonalises the narrative. It shifts the room from "who did what" to "what happened when."
The Five Sections Every Post-Mortem Needs
1. Impact
State what users or systems were affected and for how long. Be specific: "checkout was unavailable for 23 minutes affecting paid users on the EU cluster" is useful. "Some users experienced issues" is not.
2. Timeline
Chronological, factual, written in past tense. Include detection time, escalation steps, mitigation actions, and resolution. Note gaps — if there's a 40-minute window where nothing was logged, say so.
3. Root Cause Analysis
Use the Five Whys or a simple cause-and-effect chain. Keep asking why until you reach a systemic factor, not a person's decision. Example:
- The API returned 503s. Why?
- The database connection pool was exhausted. Why?
- A background job held connections open during a bulk import. Why?
- The job had no timeout configured. Why?
- Timeout configuration wasn't part of the service's deployment checklist.
That last answer is actionable. "An engineer forgot to set a timeout" is not.
4. Contributing Factors
List conditions that made the incident worse or harder to detect. These are not causes — they're amplifiers:
- Alerts were configured with a 10-minute evaluation window, delaying detection
- Runbook for this service hadn't been updated after a refactor
- On-call engineer was covering two rotations simultaneously
5. Action Items
Every action item needs an owner and a due date. If it doesn't have both, it won't ship. Group them by type:
- Prevent recurrence – fix the specific failure mode
- Improve detection – close monitoring gaps
- Improve response – update runbooks, add playbook steps
- Reduce impact – add circuit breakers, fallbacks, or feature flags
Running a Blameless Review Meeting
Blameless doesn't mean consequence-free — it means the review focuses on systems, not individuals. A few ground rules that help:
- Assume good intent. Everyone made the best decisions they could with the information available at the time.
- Use "the system" language. Say "the alert didn't fire" not "you didn't set up the alert correctly."
- Invite the people who were actually involved. The engineer who was paged at 2 a.m. has context no manager does.
- Timebox the retrospective. Sixty minutes is usually enough. More time often means more speculation.
- Read the timeline aloud at the start. It anchors the group in facts before discussion begins.
A Pre-Publish Checklist
Before you share the post-mortem internally, run through this:
- Timeline is sourced from logs, not memory
- Impact statement includes duration and affected scope
- Root cause traces to a systemic factor
- No individual is named as a cause
- Every action item has an owner and a deadline
- The document has been reviewed by at least one person who was on-call
- A follow-up date is scheduled to review action item completion
On Detection Lag
One pattern that shows up repeatedly in post-mortem timelines: the incident started well before anyone was paged. A deployment goes out at 14:00; users start seeing errors at 14:03; the alert fires at 14:18.
That 15-minute gap is worth its own action item. Multi-region monitoring with short check intervals — Pingy runs checks as frequently as every 30 seconds from multiple locations — can surface regional degradation before it becomes a full outage. If your timeline consistently shows a lag between incident start and detection, that's the gap to close first.
Key Takeaways
- Build the timeline from logs before the meeting, not during it
- Root cause analysis ends at a systemic factor, not a person
- Every action item needs an owner and a due date — no exceptions
- Blameless means focusing on systems, not absolving bad processes
- Detection lag is often the highest-leverage thing to fix after an incident
- Schedule a follow-up to confirm action items are actually completed