Why Most Post-Mortems Fail
Post-mortems get written, reviewed once, and then rot in a wiki. The usual culprits: they focus on who made a mistake instead of what made the mistake possible, and they produce action items nobody owns or prioritizes.
A blameless post-mortem isn't just a morale exercise. It's an engineering tool. When people aren't afraid of being singled out, they share the full picture — including the awkward decisions, the unclear runbooks, and the monitoring gaps they'd otherwise stay quiet about.
The Core Principle: Assume Rational Actors
Everyone involved in an incident made the best decision they could given the information, tools, and time pressure available at that moment. Your job in a post-mortem is to understand why that information was incomplete, not to second-guess the person who had it.
This framing changes the questions you ask. Instead of "Why did the engineer deploy without testing?", you ask "What made it reasonable to believe that deployment was safe?"
Before You Write Anything: Gather the Timeline
A good post-mortem lives or dies on an accurate, granular timeline. Start here before drafting a single sentence of analysis.
Sources to pull from:
- Monitoring and alerting logs (timestamps matter — when did the alert fire vs. when did the incident actually start?)
- On-call chat history (Slack, PagerDuty, etc.)
- Deployment and CI/CD logs
- External uptime data — if you're using multi-region monitoring like Pingy, you can see exactly when and where degradation was first detected, which often predates internal alerts
- Customer support tickets and status page updates
Get the timeline into a shared doc as raw facts first. No interpretation yet.
The Post-Mortem Structure That Works
1. Incident Summary
Two to four sentences. What happened, when, how long, what was affected. Write this last so it accurately reflects the full document.
2. Timeline
Chronological. Use UTC. Include detection, escalation, mitigation, and resolution. Note gaps — if there's a 40-minute window where nothing was logged, that itself is a finding.
3. Root Cause Analysis
Avoid stopping at the proximate cause. Use the "five whys" or a simple cause-and-effect chain to get to systemic factors. A config change breaking prod isn't a root cause — the absence of a staging environment that mirrors prod, or a deploy process without automated rollback, often is.
Be specific. "Insufficient monitoring" is not actionable. "No alert existed for database connection pool exhaustion" is.
4. Contributing Factors
List conditions that made the incident worse or harder to detect. These often include:
- Alerts that fired too late or not at all
- Runbooks that were out of date
- On-call handoffs that lost context
- Time of day (reduced staffing)
- A system that failed silently instead of loudly
5. Impact
Quantify what you can: duration, affected services, error rates, number of users or requests affected. Avoid vague language like "some users experienced slowness."
6. Action Items
This section is where post-mortems usually collapse. Make every item:
- Specific: "Add connection pool exhaustion alert" not "improve monitoring"
- Owned: one named person or team, not "the team"
- Time-bound: a target date or sprint
- Tracked: linked to a ticket in your issue tracker before the meeting ends
If an action item isn't worth filing a ticket for, cut it.
Running the Review Meeting
Keep it under an hour. Invite the people who were directly involved, plus one or two stakeholders. A facilitator who wasn't in the incident helps keep things on track and neutral.
Ground rules worth stating explicitly at the start:
- We're here to improve the system, not evaluate individuals
- Hindsight is not a valid criticism
- All findings stay in the doc; nothing leaves as ammunition
Let the person who wrote the timeline walk through it. Pause at decision points and ask what information was available at that moment. That's where the real learning is.
What Good Looks Like
A post-mortem is working when:
- Engineers volunteer information they'd normally hide
- The action items close within the quarter
- The next similar incident is caught earlier or handled faster
- Patterns across multiple post-mortems start becoming visible
That last point is underrated. Reviewing post-mortems in aggregate — quarterly, for example — often reveals that the same underlying system weakness keeps appearing in different costumes.
Key Takeaways
- Blame prevents learning; assume rational actors working with incomplete information
- Build the timeline from raw data sources before any analysis
- Stop root cause analysis at systemic factors, not human error
- Every action item needs an owner, a ticket, and a deadline
- A post-mortem with no closed action items is just documentation of a bad day