A disaster-recovery game day tests whether your team can restore a useful service within its agreed downtime and data-loss limits. Pick one recovery path, run it under controlled conditions, and keep evidence of what worked. A running database or a green homepage is only part of that evidence.
Start with a restore into an isolated environment if the recovery procedure is untested. A production failover can follow once the team has rehearsed the steps, checked the escape route and agreed on the customer impact it will allow.
Choose the failure you can measure
Name a user operation, such as signing in and saving an order. Then choose the failure that threatens it: an unavailable primary database, a damaged dataset, or loss of access to a site. An Availability Zone failure and a whole Region failure have different boundaries. Write down which one the exercise covers.
A game day can include a chaos experiment. Chaos engineering also uses bounded experiments, hypotheses and stop conditions; it is not defined by surprise or random breakage. AWS describes both controlled fault experiments and game days that exercise team response.
For a first restore drill, use an isolated copy of the application and a selected backup. Block outbound customer email, payment calls and production queue access. Record those differences from production. This exercise can establish whether that backup restores and whether the recovered application works. It cannot establish production failover time if DNS changes, live traffic and on-call detection were excluded.
Agree on the clocks before starting
The recovery time objective (RTO) is the maximum acceptable interval between service interruption and restoration. The recovery point objective (RPO) sets the acceptable data-loss window before the interruption. They are targets. The drill produces measurements to compare with them. AWS defines these objectives in terms of downtime and data loss.
Start the outage clock when the chosen user operation becomes unavailable. Record the fault injection time separately if the two differ. Include detection, approval and routing time in the service recovery interval. Timing only the restore command hides work users still have to wait for.
Agree on the finish line too. For an order service, it might require a successful login, a durable new order, a correct read of that order and a completed background job at the agreed test load. Record the first successful transaction and the end of a defined observation period separately.
The following timeline is a hypothetical worked example, not a Pingy incident or a performance promise. Its targets are 30 minutes for recovery and 5 minutes of data loss.
| UTC time | Evidence recorded |
|---|---|
| 09:58 | Latest point through which the recovered dataset is verified complete |
| 10:00 | Order operation fails; outage clock starts |
| 10:02 | Alert reaches the responder |
| 10:05 | Responder approves recovery |
| 10:18 | Restore finishes; application validation starts |
| 10:22 | First successful order through the intended client route |
| 10:27 | Agreed five-minute observation period ends without a failed test |
Under that acceptance rule, recovery took 27 minutes, including a two-minute detection delay. The verified recovery point is two minutes before the interruption. Both meet this example's targets. A team that uses the first successful order as its finish line would record 22 minutes instead; choose the rule before the drill and report it with the result.
A recent row alone does not establish that the dataset is complete through that time. For the exercise, generate identifiable test transactions and keep their acknowledged results outside the failed system. Compare the recovered records against that ledger, including missing or duplicate operations. A timestamp on the backup file tells you when the file was written, not which transactions survived.
Write a brief the operator can use
The brief needs enough detail for someone to decide whether to start and when to stop. Use this as a template; fill in actual identifiers and thresholds before the exercise.
| Field | What to record |
|---|---|
| Scope | User operation, environment, exact resources and excluded dependencies |
| Prediction | Expected detection and recovery behavior, plus RTO and RPO targets |
| People | Exercise lead, fault operator, recovery decision maker and observer |
| Fault | Reviewed action, target list, duration and permission required |
| Stop rule | Measured impact limit, maximum duration and who can call stop |
| Recovery | Tested undo steps, backup location, required access and validation procedure |
| Evidence | UTC event log, transaction ledger, alert receipt and recovery results |
Have the service owner and affected support staff review the plan. Give the operator a confirmed target list, not a broad tag selector that might include a newly created production resource. Keep the runbook and access instructions available if the primary site or identity service fails.
AWS recommends involving the teams that would respond to the real event and tracking findings through completion. Rehearse the mechanics in a production-like environment before expanding the exercise. Its game-day guidance covers preparation and follow-up.
Stopping the experiment takes its own procedure
The stop rule must lead to an action somebody can perform while the fault is active. If a firewall change cuts off the operator's only access, the undo command on their laptop is insufficient. Verify an independent management path or a tested automatic expiry before starting.
With AWS Fault Injection Service, stop conditions use CloudWatch alarms. Define the acceptable baseline first and check that the alarm receives current data. An alarm cannot protect the exercise if its input disappeared unnoticed. AWS documents how FIS stop conditions work.
A stopped experiment does not mean the application has recovered. FIS completes applicable post-actions when stopped, and recovery behavior depends on the action. For example, restarting instances after aws:ec2:stop-instances requires the optional startInstancesAfterDuration setting; terminating an instance is a different action. Read the selected action's behavior and verify the resulting resource state. See stopping an experiment and the EC2 action reference.
If telemetry goes stale, impact exceeds the agreed boundary, or a real incident starts, stop the exercise and run its recovery procedure. Record the interruption. Keeping the drill going to obtain a cleaner result is the wrong trade.
Run the restore and prove the application works
For the isolated restore exercise, use the backup format and recovery procedure documented for your database version. The order below is a proposed drill procedure, not a vendor-specific restore command.
- Save the baseline results and the test transaction ledger. Confirm the recovery operator can read and decrypt the chosen backup without relying on the failed environment.
- Start the scenario and record the user-visible interruption. Let the designated responder follow the runbook, including the usual approval path.
- Restore into the named isolated target. Record the selected recovery point and every manual correction needed to start the application.
- Compare recovered data with the ledger. Run the agreed user operations through the recovery endpoint and check background work as well as HTTP responses.
- Apply the agreed test load and observation period. Save the result, the measured times and any skipped steps before cleanup.
A small restore proves less than a full-size restore. Record dataset size, load and infrastructure differences so nobody uses a staging result as an unsupported production recovery guarantee. AWS specifically calls for exercising recovery paths and checking whether standby capacity still supports the workload. See its disaster-recovery testing guidance.
An external HTTP monitor adds evidence about the URL it checks. It does not establish that an order was durably saved or a queue consumer completed its work. Compare the external result with application checks and internal telemetry. Google's SRE guidance distinguishes user-visible behavior from internal signals and explains why both matter. See monitoring distributed systems.
For an isolated target, record alert delivery as untested unless the drill deliberately includes it. A healthy public production URL says nothing about that target's recovery.
Failback is another state change
A production database failover needs a single accepted writer. Prevent the old primary from accepting writes before the replacement becomes writable. PostgreSQL's failover documentation warns that an old primary returning as a second primary can cause data loss. It also explains that a new standby must be established after failover to restore redundancy. See PostgreSQL 16 failover behavior.
Once the recovery site accepts new writes, the original site's data can be stale. Follow the database or managed service's documented resynchronization and failback procedure. Pointing traffic back at the old endpoint is not enough. Give failback its own decision point, data checks and abort rule.
Keep the service recovery time separate from the time needed to restore redundancy. The application may be usable while it is still running with fewer failure protections. Leave the incident or exercise record explicit about that remaining work.
Close the finding with a repeat test
Attach the timeline and data comparison to the runbook revision. Turn each failure into a task with an owner, a due date and a repeatable acceptance test. "Fix recovery access" is vague. "The recovery role can decrypt the selected backup while primary-site access is blocked" can be tested.
Schedule the next drill around the unresolved failure or a material architecture change. Keep the previous evidence with its environment and date. Close a recovery finding when the corrected procedure passes the same check that failed.