Survive an AWS zone failure with capacity and tested failover

A multi-AZ application can still fail when one Availability Zone disappears. The surviving servers may be too small, their outbound route may cross the failed zone, or their database connections may never recover. Design around the full request path, then test it under load.

An AWS Availability Zone contains one or more data centers with separate power and networking infrastructure. AWS designs zones to limit correlated infrastructure failures. That isolation does not remove shared application dependencies or bad deployments. AWS describes these isolation boundaries here.

This guide uses a hypothetical web application with an Application Load Balancer, EC2 application servers and an RDS Multi-AZ DB instance. The numbers are worked planning examples, not measurements of AWS or Pingy.

Budget for the capacity that survives

Start with the load you must serve during a failure and the capacity each zone has already running. Use capacity measured at an acceptable error rate and latency, with the expected request mix. Instance count alone says little about how much work the application can finish.

For equal zones, surviving capacity is the number of remaining zones multiplied by usable capacity per zone. Consider a hypothetical peak load of 1,200 requests per second:

The third row has three-zone placement and a capacity shortfall. In the second row, each surviving zone moves from 400 to 600 req/s if traffic splits evenly. That consumes the full assumed capacity; allow further margin for load variation and recovery work.

This is the purpose of static stability: keep enough capacity available to absorb a failure without waiting for replacement instances. Auto Scaling can replenish the fleet, but a launch request is not usable capacity yet.

Check the group's actual subnet placement and distribution strategy. AWS calls the strategies balanced-best-effort and balanced-only. The former can try another healthy zone when launches fail; the latter continues trying the failing zone to preserve balance. Choose deliberately for the workload. Neither setting proves the surviving fleet can carry production traffic. See the Auto Scaling distribution behavior.

Follow both inbound and outbound traffic

Enable the load balancer in the zones containing its targets. An ALB requires at least two zones; an NLB can be deployed with only one. Check the configuration instead of assuming the product name guarantees multiple zones.

Cross-zone routing lets a load balancer node select targets in other enabled zones. ALB enables it at the load-balancer level, while individual target groups can disable it. NLB disables it by default. AWS documents these routing rules.

Cross-zone routing helps share target capacity. A design that deliberately keeps requests within a zone makes a different tradeoff: it needs balanced capacity and a way to withdraw traffic from an impaired zone. AWS's zone-independence pattern describes that approach. Record the effective setting for each target group before testing.

Now follow an application server's outbound call to a payment sandbox or another required internet service. With zonal NAT gateways, private subnets should use the gateway in their own zone. Routing every subnet through a gateway in zone A makes A a dependency for otherwise healthy servers. AWS documents that shared-NAT failure. Audit route tables as well as gateway inventory.

The diagram shows the intended paths after zone A is withdrawn. It assumes surviving load-balancer nodes can send traffic to healthy targets and each surviving zone has its own egress. It does not claim that withdrawal is instant. Database traffic follows the database endpoint and can cross zone boundaries.

Database promotion is part of recovery

For an RDS Multi-AZ DB instance deployment, AWS synchronously replicates the primary to a standby in another zone. That standby does not serve reads. A Multi-AZ DB cluster is a different deployment type, so record which one you operate before copying a failover procedure. The DB instance documentation explains the distinction.

For the DB instance deployment, AWS gives a typical failover time of 60 to 120 seconds and warns that database activity and recovery can extend it. This is not an application recovery deadline. Failover changes the database endpoint's DNS record, and existing connections must be re-established. Verify the application's resolver caching and connection-pool behavior against the RDS failover guidance.

A connection loss can leave a client uncertain whether a write committed. Use an operation identifier and application-level duplicate prevention where a retry could create a second order or payment. Set bounded timeouts and retry limits, with backoff and jitter, so reconnecting clients do not overwhelm the remaining capacity. AWS's retry guidance explains both the load amplification and the need for idempotent operations.

Check other state separately. Session data, queued work and uploaded files each need an explicit failure policy. For S3, inspect the storage class: S3 One Zone-IA and S3 Express One Zone store data in a single zone. A blanket statement that every S3 object is spread across zones is wrong. Use the S3 storage-class reference when reviewing the application.

Health checks can miss the failing operation

A successful health endpoint only proves what that endpoint checks. A server may answer a local health request while checkout stalls on a remote dependency. Conversely, tying every target's readiness to one shared dependency can mark the whole fleet unhealthy.

ALB also has fail-open behavior: if all registered targets are unhealthy across all enabled zones, it routes to those targets anyway. Do not use an unhealthy status as a security boundary or assume it always blocks traffic. Check the ALB health-check rules for your target group.

During a drill, observe both the target health decision and a representative application operation. External probes add evidence about the paths and requests they exercise. They cannot establish that every user, background job or write recovered. Pair them with application results and per-zone capacity metrics; the health-check tutorial covers the measurement choices.

Run a bounded zone-loss drill

Use an isolated test environment with the same dependency layout and routing as production. Use synthetic data and sandbox integrations. Define the acceptable error and latency limits, the maximum experiment duration, and who can stop it before injecting a fault.

Test one mechanism at a time first: remove application capacity from one zone, exercise database failover, and interrupt a required network path. Then combine the relevant faults. Terminating one instance only tests that instance's loss. It does not reproduce an entire zone impairment.

AWS FIS has an AZ Availability: Power Interruption scenario. Review its target selectors and service limitations before use. Actions can be skipped when no matching targets exist, so a completed experiment can leave an intended dependency untested. The scenario does not include stop conditions; add your own.

FIS stop conditions use CloudWatch alarms to stop an experiment when a defined threshold is reached. Also document the cleanup for every selected action. Stopping fault injection does not establish that the application or its data has recovered.

Use this record for the drill. Fill it with observations from the test, including failures and missing evidence.

Measure recovery from the first affected operation until the application meets the agreed criteria for a sustained observation period. Record a database promotion separately from the first successful application write. Use the RTO and RPO guide to define the clock and data-loss boundary.

Before returning the zone to normal traffic, check its capacity and dependency health. Keep the test record with the deployed configuration and load profile. A later change to instance size, routing or connection handling needs another test of the affected path.