Building a Warm-Standby Disaster Recovery Site

What Is a Warm Standby?

Disaster recovery strategies exist on a spectrum. A cold standby keeps infrastructure defined but powered off — cheap, but slow to activate. A hot standby runs a full duplicate that serves live traffic at all times — fast, but expensive. A warm standby sits in between: infrastructure is running and data is reasonably fresh, but the environment isn't actively serving production traffic until a failover is triggered.

For most teams, warm standby hits the right balance between cost and recovery time. You're targeting an RTO (Recovery Time Objective) in the range of minutes to a low number of hours, not days.

Core Components You Need to Replicate

Before writing any infrastructure code, map out what actually needs to exist in your DR region:

• Compute — Auto-scaling groups, container clusters, or equivalent. They should be sized to handle a meaningful portion of peak traffic immediately, with room to scale.
• Database — A read replica in the DR region, promotable to primary. PostgreSQL, MySQL, and most managed services (RDS, Cloud SQL, AlloyDB) support cross-region replicas natively.
• Object storage — Enable cross-region replication on your buckets. For most providers this is a checkbox, but verify replication lag SLAs.
• Secrets and configuration — Your DR environment must have access to the same secrets. Use a secrets manager that operates across regions, or replicate explicitly.
• DNS — You'll be changing DNS records during failover, so low TTLs matter. Set your failover records to 60 seconds or less well before you need them.
• TLS certificates — Make sure certs are provisioned in the DR region in advance, not on demand during an incident.
• CDN and load balancer config — Export and version-control these. Recreating them under pressure is miserable.

Setting Up Cross-Region Database Replication

This is usually the hardest part. The general steps for a PostgreSQL setup on AWS RDS:

1. Enable automated backups on your primary instance (required for read replicas).
2. Create a cross-region read replica in your DR region via the console or CLI.
3. Monitor replication lag — aws rds describe-db-instances returns ReplicaLag in seconds. Alert if it exceeds your RPO threshold.
4. Test promotion in a staging context. aws rds promote-read-replica is the command; know what it does to your replication slot and downstream consumers before you need to run it live.
5. After promotion, update your application's database connection string — either via environment config or a DNS CNAME that you control.

Replication lag is your RPO (Recovery Point Objective) in practice. If your replica is 4 minutes behind and your primary dies, you lose 4 minutes of writes. Make sure your stakeholders understand this number.

Infrastructure as Code Is Non-Negotiable

Your DR environment should be defined entirely in Terraform, Pulumi, CloudFormation, or equivalent. There are two good patterns:

• Single multi-region stack — One set of modules parameterized by region. Easier to keep in sync, slightly more complex to manage state.
• Mirrored stack — A separate workspace or stack for DR, kept in sync via shared modules. Clearer separation, but discipline is required to avoid drift.

Whichever you choose, DR infrastructure drift is a silent killer. Run terraform plan against your DR stack regularly as part of CI, and alert on any diff that isn't expected.

Failover Runbook

Document this before you need it. A basic failover sequence:

1. Confirm the primary region is genuinely impaired (not a monitoring fluke).
2. Stop writes to the primary database if possible — prevents split-brain.
3. Promote the DR read replica to primary.
4. Update application config to point at the new database endpoint.
5. Scale up compute in the DR region to full capacity.
6. Update DNS to route traffic to DR load balancers.
7. Verify health checks pass and error rates normalize.
8. Communicate status to stakeholders.

Print this out. Literally. People make mistakes under pressure and a laminated checklist is underrated.

Monitoring Across Regions

Your primary monitoring should not live exclusively in your primary region. If your US-East infrastructure goes down and your uptime checks are also running from US-East, you'll get a noisy or delayed signal.

Using a monitoring service that probes from multiple independent geographic locations — like Pingy — means your alerting infrastructure is inherently separate from what it's watching. Configure checks against both your primary and DR endpoints so you can verify the DR site is reachable and responding correctly before and after a failover.

Test It or It Doesn't Exist

A DR plan that hasn't been exercised is a hypothesis. Schedule a quarterly failover drill:

• Promote the replica in staging first
• Do a full failover in production during a low-traffic window at least once a year
• Time every step against your RTO targets
• Document what broke and fix it before the next drill

Key Takeaways

• Warm standby targets minutes-to-hours RTO at a fraction of hot standby cost.
• Cross-region database replication lag is your real-world RPO — measure and alert on it.
• Infrastructure as Code prevents DR environment drift; run plan checks in CI.
• Write and test your failover runbook before you're in an incident.
• Monitor from outside your primary region so your alerting survives a regional outage.
• Test your DR failover end-to-end at least annually. Untested recovery plans fail when it counts.