A warm-standby recovery site keeps a working copy of an application running at reduced capacity. During a failure, the team promotes its data stores, adds capacity where needed and moves users to it. The difficult part is proving that those steps still work when the primary site is unavailable.
This guide uses a web application with PostgreSQL as a design example. The database details refer to PostgreSQL 18; managed database services can have different promotion and recovery procedures. It is a planning guide, not a command sequence to run against production.
Decide what the standby must restore
AWS describes warm standby as a scaled-down, fully functional workload. A pilot-light design keeps core resources running but still needs additional application resources before serving requests. Neither label proves a recovery time. Test your own path from service interruption to the agreed level of service. AWS recovery strategies
Write down the failure you are preparing for: loss of one site, loss of a region, or loss of access to a shared dependency. Specify which user operations must return, the load they must support and how you will judge success. A readable catalogue and a working checkout are different recovery outcomes.
Set the recovery time objective (RTO) and recovery point objective (RPO) for that outcome. Include detection and decision time in the recovery clock. Define acceptable data loss separately for each durable store. Our RTO and RPO guide explains how to record targets and compare them with a drill.
Remove dependencies on the failed site
A second application server is little help if it retrieves secrets, deployment images or uploaded files from the site you have lost. Trace a request through the recovery environment and record every dependency it touches. AWS's multi-region guidance calls for each region to operate independently and warns that third-party services can share your primary region. Workload dependencies
For a hypothetical checkout service, use a dependency record like this. These are suggested checks, not results from a tested deployment.
| Component | Evidence to collect |
|---|---|
| Application | The approved release starts from an image available at the recovery site. |
| Access | Responders can sign in and obtain recovery credentials without the primary site. |
| Data | Database records and uploaded objects recover to a documented, usable state. |
| Payments | The provider accepts recovery-site egress and sends callbacks to the active endpoint. |
| Workers | Queue consumers and scheduled jobs have one active owner after cutover. |
Include TLS certificates, encryption keys and session handling in that review. Decide whether users must sign in again. Check whether an order record can refer to an object that has not arrived at the standby. Database replication alone cannot establish consistency across those stores.
Keep deployment versions and schema compatibility under review. Test credential rotation and application releases in the recovery environment too. A standby that passed a drill before a major schema change needs fresh evidence.
Choose capacity from a load test
There is no useful universal percentage of production servers to keep warm. Measure the capacity needed for your recovery service level, including the database, connection pools and queue backlog. A smaller application fleet may be enough for a deliberately reduced service; record what you will disable and how users will see that restriction.
Exercise the complete scale-up while representative requests run. Record when instances become usable, how long startup takes and when the service meets its latency and error limits. Check quotas, subnet address space and available instance types before relying on more compute. A quota increase permits a request; it does not reserve physical capacity.
AWS warns that Auto Scaling depends on control-plane operations and says recovery-region quotas must support the intended production load. Its recovery guidance also recommends considering capacity reservations. If scale-up is required to meet your target, include failure to obtain that capacity in the drill. Keep more capacity running when the tested scale-up path cannot meet the requirement. Recovery options
Know which data can survive promotion
PostgreSQL streaming replication is asynchronous by default. A primary can acknowledge a commit before the standby receives it, so promotion can lose acknowledged transactions. Measure replication under peak writes and after a link interruption. Check receipt and replay separately: received WAL may still need to be applied. WAL retention also needs monitoring; replication slots can retain enough WAL to fill the primary's disk. PostgreSQL standby operation
Synchronous replication changes the acknowledgement rule. With synchronous standbys configured, PostgreSQL's synchronous_commit=on waits for the selected standby acknowledgement of durable WAL storage. remote_write has a weaker guarantee against a standby operating-system crash; remote_apply waits for replay. Transaction-level settings can change the protection. Verify the chosen promotion candidate contains the required commits and survives the failure you are planning for. Waiting for a standby also adds commit latency and can block writes if the required acknowledgement is unavailable. Commit durability settings
Record the age and origin of each replication measurement. PostgreSQL's lag fields describe recent write, flush and replay delays; they do not predict catch-up time. An idle, caught-up standby can eventually show null lag values. Missing or stale monitoring therefore needs investigation before anyone signs off an RPO claim. Replication statistics
Keep retained backups and test restoration independently. A replicated deletion can reach the standby too. If the incident involves corrupted or malicious changes, a clean historical restore may be the appropriate recovery path. See backup and replication limits.
Separate promotion from traffic cutover
External checks help establish user impact, but an unreachable endpoint does not prove that its database has stopped accepting writes. A network partition can leave both sites running. The failover procedure must prevent the old primary from remaining a writer when the standby is promoted. PostgreSQL explicitly requires a mechanism to prevent both servers acting as primary and leaves failure detection and orchestration to external tooling. PostgreSQL failover
Choose a fencing mechanism appropriate to the platform, such as verified power isolation or a tested cluster mechanism that revokes write ownership. DNS changes alone do not fence a database. Existing application connections, workers and direct clients may still reach the old writer. If ownership cannot be established, stop the promotion path and escalate under the incident plan.
Use explicit gates in the runbook. Capacity work can proceed alongside data checks, but public traffic must wait for the conditions that protect it.
| Gate | Required evidence |
|---|---|
| Decision | Named owner authorizes recovery using current impact and data evidence. |
| Writer | Old writer is fenced; the promotion candidate and potential data loss are recorded. |
| Service | Promoted database, application and workers pass controlled checks at the required load. |
| Traffic | Routing changes reach clients; retries and reconnections behave as expected. |
| Recovery | User operations and data checks meet the agreed acceptance criteria. |
For the checkout example, a controlled acceptance test could create a test order, read it back and verify that its background job executes once. Use test accounts and a payment sandbox; keep duplicate-charge prevention in the acceptance criteria. Passing a homepage health check would leave those paths untested.
Configure traffic steering before an incident and test it with the standby's readiness rules. AWS recommends Route 53 health checks or ARC routing controls for recovery because they use data-plane functions instead of relying on DNS control-plane changes. DNS TTL affects resolver caching, with shorter values trading more queries for quicker refresh. It does not define the full recovery duration. Route 53 DNS guidance
Measure client reconnection as part of cutover. Include long-lived connections and application DNS caching in the test, and verify that traffic is not sent to a standby that cannot yet handle writes or load.
Rebuild protection before returning traffic
After promotion, the new primary needs its own backup and replication coverage. Keep the former primary isolated until you have established the authoritative data set and prepared it to rejoin. Plan failback as another controlled transition with capacity, write ownership and client checks.
For PostgreSQL, pg_rewind can help a former primary follow the new primary after their histories diverge. It has prerequisites, including the required WAL history and either data checksums or wal_log_hints, plus full_page_writes. Review its warnings and preserve needed evidence before using it. A fresh base backup may be required. Rewind is not a merge of conflicting writes. PostgreSQL pg_rewind
Drill the path your responders will use
Start in an isolated environment with controlled data, an abort condition and a named operator. Block the primary dependencies included in the scenario, then execute the recovery runbook through failback. Record the start of disruption, promotion, usable capacity, client recovery and the last verified recovered transaction. Leave uncertain data loss marked as unknown.
Repeat after changes that affect the recovery path and on a schedule justified by your risk. AWS recommends regular failover tests to validate RTO, RPO and recovery-site capacity. A staging drill has limits; any production exercise needs its own approved scope and safeguards. Test disaster recovery
I would start with one critical user operation and require it to pass from the recovery site while its primary-site dependencies are unavailable. Save the evidence beside the runbook, with an owner for each failure found.