Building a Warm-Standby Disaster Recovery Site

What Is a Warm Standby?

A warm-standby DR site sits between a cold standby (infrastructure provisioned but idle) and a hot standby (fully active, serving traffic). Your secondary environment is running and continuously synchronized, but it isn't handling production requests. When failure hits your primary region, you promote the standby and redirect traffic—typically within 2–15 minutes depending on your stack.

This guide focuses on the practical mechanics: data replication, infrastructure parity, traffic failover, and testing.

1. Define Your Recovery Objectives First

Before touching infrastructure, nail down two numbers:

• RTO (Recovery Time Objective): How long can your service be down? Warm standby typically targets 5–30 minutes.
• RPO (Recovery Point Objective): How much data loss is acceptable? With synchronous replication, RPO approaches zero. Async replication introduces lag you need to measure and accept.

These numbers drive every architectural decision that follows.

2. Choose Your Secondary Region

Pick a region that is geographically separated but not so distant that replication lag becomes painful. For AWS users, us-east-1 → us-west-2 is a common pairing. For GCP, you'd typically use a different multi-region. Avoid picking a region that shares the same physical availability zone infrastructure or the same upstream network paths.

3. Replicate Your Data Layer

This is the hardest part. Get it wrong and your standby site either can't serve traffic or serves stale, inconsistent data.

Relational Databases

• PostgreSQL: Use streaming replication with a standby promoted via pg_ctl promote or tools like Patroni or repmgr. Monitor replication lag with pg_stat_replication.
• MySQL/MariaDB: GTID-based replication across regions. Consider Orchestrator for automated topology management.
• RDS/Aurora: Aurora Global Database provides sub-second replication lag and one-click regional promotion through the console or API.

Other Data Stores

• Redis: Use Redis replication with a replica in the secondary region, or Redis Enterprise's Active-Passive geo-replication.
• Object storage (S3, GCS): Enable cross-region replication. Replication is async, so factor in the lag for your RPO.
• Elasticsearch/OpenSearch: Cross-cluster replication (CCR) for read-only replicas in the secondary region.

4. Mirror Your Application Infrastructure

Your standby region should be a near-identical mirror of production—same instance types, same autoscaling policies, same configuration. Differences will bite you during a real failover.

Use infrastructure-as-code (Terraform, Pulumi, CloudFormation) with environment variables or workspace separation to deploy the same modules to both regions. Keep the secondary at reduced capacity (e.g., 25–50% of primary) to save cost, but ensure autoscaling can ramp quickly.

Checklist for infrastructure parity:

• Same AMI or container image versions deployed
• Identical security group and firewall rules
• Secrets and configuration synced (via Secrets Manager replication or Vault DR replication)
• TLS certificates valid and deployed
• Service dependencies (queues, caches, third-party APIs) mapped and accessible from secondary region
• IAM roles and service accounts configured

5. Traffic Failover Mechanism

You need a way to shift production traffic from primary to standby. Common approaches:

1. DNS failover: Update your DNS record (low TTL, 30–60 seconds) to point to the secondary region's load balancer. Route 53 health checks can automate this. Simple but DNS propagation adds latency.
2. Global load balancer: AWS Global Accelerator, GCP Cloud Load Balancing, or Cloudflare Load Balancing can detect unhealthy origins and reroute within seconds without DNS propagation delays.
3. Anycast + BGP: More complex but used for very low RTO requirements.

For most teams, a global load balancer with health checks is the right tradeoff between complexity and speed.

6. Monitor Both Sites Continuously

You can't failover to a standby you don't trust. Run synthetic checks against both your primary and secondary endpoints at all times—not just during drills. This tells you whether the standby is actually healthy before you need it.

Monitoring from multiple geographic vantage points matters here: a check from a single location can miss regional network issues. External uptime monitoring tools that probe from several regions give you an honest, outside-in view of both sites simultaneously.

7. Test Failover on a Schedule

A DR plan you haven't exercised is fiction. Run a full failover drill at minimum quarterly:

1. Announce a maintenance window.
2. Promote the database replica in the secondary region.
3. Cut traffic via your global load balancer or DNS.
4. Validate application health with smoke tests.
5. Measure actual RTO and RPO against your targets.
6. Document gaps and fix them before the next drill.

Game-day exercises where you fail over without warning (to a staging environment first) reveal assumptions that scheduled drills miss.

Key Takeaways

• Define RTO and RPO before designing anything—they constrain every other decision.
• Data replication is the critical path; choose sync vs. async based on your RPO.
• Use IaC to enforce infrastructure parity between regions.
• A global load balancer with health checks gives faster failover than DNS alone.
• Monitor both regions continuously so you trust your standby when it counts.
• Failover drills are non-negotiable—run them on a schedule and fix what breaks.