Why Game Days Get Skipped (And Why That's a Problem)
Most teams write a disaster-recovery plan, store it in Confluence, and never touch it again. The problem surfaces at 2 a.m. when a region goes down and nobody has actually practiced the runbook. Game days fix that by turning your DR plan into something the team has done, not just read.
This guide walks through running a focused, low-chaos game day that produces real signal without burning out your team.
1. Define the Scope Before You Do Anything Else
A game day with a vague objective wastes everyone's time. Pick one failure scenario per session. Good candidates:
- Primary database becomes unreachable (failover to replica)
- A cloud region goes offline (traffic must shift to secondary region)
- Object storage bucket is corrupted or deleted (restore from backup)
- Third-party payment API is down (fallback or graceful degradation)
Write a one-sentence hypothesis: "If our primary Postgres instance fails, we can promote the replica and restore service within 20 minutes with no data loss beyond the last replication lag interval." That sentence becomes your success criterion.
2. Assign Roles
Clear ownership prevents people from standing around waiting for someone else to act.
| Role | Responsibility |
|---|---|
| Incident Commander | Runs the drill, tracks time, calls Go/No-Go |
| Responders | Execute the runbook steps |
| Observer | Notes what's skipped, unclear, or wrong in the runbook |
| Comms lead | Drafts the internal status update as if it were real |
Keep the observer separate from the responders. Their job is specifically to watch the process, not fix the problem.
3. Prepare the Environment
Never run a game day against production without explicit sign-off and a rollback plan. A staging environment that mirrors production closely enough is the right starting point. Before the drill:
- Snapshot or back up any data you'll be touching
- Confirm your monitoring and alerting is active so you can see what the system "sees" during the drill
- Brief stakeholders — support, sales, and your manager should know a drill is happening and roughly when
- Set a hard stop time so the drill doesn't bleed into the workday indefinitely
If you're testing regional failover, having uptime checks running from multiple geographic probes (not just one) gives you an independent view of what's reachable and from where. This matters because your internal dashboards might still show green while external traffic is failing.
4. Execute the Drill
Inject the failure
The incident commander introduces the failure condition — kill the database process, block the subnet, delete the bucket, whatever the scenario demands. Note the exact time.
Follow the runbook, literally
Responders should execute only what the runbook says, in the order it says it. If a step is missing, they say so out loud and the observer logs it. This is how you find the gaps.
Measure against your hypothesis
Track:
- Time from failure injection to first alert firing
- Time from alert to responder action
- Time from action start to service restored
- Any data loss or inconsistency
If your uptime monitoring is configured correctly, you'll see the outage window reflected in your status timeline automatically — useful for the post-drill review.
5. Run the Retrospective the Same Day
Don't schedule the retro for next week. Do it within a few hours while context is fresh. Structure it around four questions:
- What did we expect to happen?
- What actually happened?
- What was missing, wrong, or unclear in the runbook?
- What's the one change we'll make before the next drill?
Assign a specific owner and due date to every action item. Without that, retrospectives produce notes that nobody reads.
6. Iterate and Increase Complexity
First game days should be narrow and controlled. Once your team can handle a single clean failure scenario, layer in complexity:
- Run the drill without announcing which failure will be injected
- Combine two failure modes (database down and on-call engineer unavailable)
- Introduce a time constraint or a degraded-comms scenario (Slack is down, use backup channel)
Cadence matters more than sophistication. A simple drill every quarter beats an elaborate one every two years.
Key Takeaways
- Pick one specific failure scenario per game day and write a testable hypothesis before you start
- Separate the observer role from responders so someone is explicitly watching the process
- Execute the runbook literally — gaps and ambiguities are the point, not a failure
- Measure four things: time to alert, time to action, time to recovery, data loss
- Run the retrospective the same day and assign owners to every action item
- Increase complexity only after your team can handle clean, single-failure scenarios reliably