Two application servers can still depend on one database, one session store or one failed deployment. Start a single-point-of-failure review by tracing a user action through those dependencies. For each failure, specify what must keep working, how traffic or data ownership changes, and how you will test the result.
This tutorial uses a hypothetical web application. The examples are design reviews and proposed tests, not measured failover results.
Trace one user action all the way through
Pick a journey that matters: signing in, saving a document or submitting an order. List every service it needs, including dependencies reached only when a process starts or a credential expires.
For each item, record its failure domain: the host, network, zone, account or configuration change that can disable it. Two processes on one host share that host. Two zones can still receive the same broken release. Independent machines do not protect against every shared cause.
The table below is an example dependency review for an order submission. Replace the entries with your own service names and owners.
| Step | Shared risk | Check |
|---|---|---|
| Resolve name | DNS zone or delegation | Each authority answers |
| Reach app | Ingress or routing | Healthy target serves traffic |
| Sign in | Session store | Old and new logins work |
| Save order | DB writer | One writer; order survives |
| Take payment | Provider API | Confirm uncertain payments |
| Recover app | Secrets or images | New app starts |
Also distinguish a request dependency from a recovery dependency. A running process might keep serving while the image registry is unavailable, yet fail to restart. Record both states. AWS's static stability guidance explains why recovery that needs fresh provisioning can fail when control-plane operations or spare resources are unavailable.
Spare capacity has to exist when the failure happens
Distribute application instances across the failure domains you intend to survive. Then measure whether the remaining instances can carry the workload within your latency and error limits. A second instance that is already saturated provides little room for failover.
For a hypothetical three-zone service, losing one zone leaves two zones to serve the traffic. Test that reduced fleet before relying on it. Autoscaling can replenish capacity afterward, but a plan that needs new instances immediately also depends on successful provisioning during the incident.
Include database connections, worker queues and downstream rate limits in that test. A healthy web tier can overload a shared database when traffic moves or clients retry. Write down the load you tested and the resources left after the fault; avoid a blanket claim that the service survives any zone failure.
DNS and ingress need their own review
DNS resilience starts with reachable authoritative servers on independent networks. RFC 2182 calls for geographic and network diversity. Several nameserver hostnames do not, by themselves, establish that diversity. The RFC's requirement for multiple servers is also not a blanket requirement to buy two DNS providers.
A second provider can reduce dependence on the first, but you must maintain consistent zone data and a working delegation. With DNSSEC, coordinate signing and the chain of trust. Cloudflare's multi-signer documentation describes specific interoperability requirements; adding another provider's nameservers alone is insufficient. Test answers and DNSSEC validation through both providers before depending on the arrangement.
Existing DNS caches mean users can see different failure timing. A low record TTL does not make an unavailable authority reachable. Keep DNS availability and DNS traffic failover as separate review items.
For ingress, verify the service and deployment mode you use. An AWS Application Load Balancer requires subnets in at least two Availability Zones, with exceptions documented for Local Zones and Outposts. AWS also recommends healthy targets in each enabled zone. These are ALB-specific requirements, not a promise that every managed load balancer or backend layout survives a zone loss.
For self-managed proxies, document how the public address reaches the survivor. Verify address ownership, routing and split-brain handling on the actual network. Two proxy VMs and an untested floating-IP configuration leave that traffic path unproven.
A database replica needs a safe promotion path
A replica provides a candidate for recovery. The system still needs to detect failure, select a suitable replica, prevent the old primary from accepting conflicting writes, and reconnect clients to the new writer.
PostgreSQL's failover documentation separates promotion from the external machinery needed to detect primary failure and notify the standby. It also describes fencing the old primary to avoid two writable primaries. Review the failover controller and its coordination store as dependencies too.
Managed failover still interrupts clients. For RDS Multi-AZ DB instance deployments, AWS documents typical failover times of 60–120 seconds, with longer recovery possible. That figure is specific to this deployment type; it is not a general PostgreSQL, Patroni or database-cluster promise. RDS changes the endpoint's DNS record, and clients must re-establish connections. Check connection-pool and DNS-cache behavior in your application. RDS failover documentation
Measure recovery from the user's failed operation to sustained successful operations. Separately compare acknowledged writes with recovered records. For recovery objectives and retained history, use the backup and replication review.
Decide whether Redis holds disposable data
A cached product description, a login session and a lock have different failure consequences. Identify every role before deciding that Redis can be bypassed.
For disposable cached data, test the application with the cache unavailable. Bound the extra database work and shed requests if necessary. An unrestricted fallback can move the outage into the database. For sessions or coordination state, define acceptable loss and recovery behavior explicitly; treating missing state as a cache miss can be incorrect.
Redis Sentinel documentation recommends at least three Sentinel instances placed where they can fail independently. A configured quorum detects failure; a majority must authorize failover. Clients need Sentinel support. Redis replication is asynchronous, so acknowledged writes can be lost during failover. Adding Sentinel does not establish zero data loss.
Exercise the client reconnection path and the loss of a Sentinel host. Recording only that a replica became primary misses whether the application can use it.
Limit failures from shared services and changes
A payment timeout leaves the outcome uncertain: the provider may have completed the operation before the response was lost. Use the provider's documented idempotency and reconciliation mechanisms before retrying. Put a timeout on remote calls, cap retry attempts and avoid retries at every layer. AWS's timeouts and retries guidance explains how retries multiply load and why a timeout does not prove that a side effect failed.
For optional features, decide what users can safely do while the dependency is down. Recommendations might be omitted; payment confirmation needs verified state. The circuit-breaker tutorial covers those boundaries.
Shared changes deserve the same scrutiny as shared hardware. A bad release, revoked credential or incorrect global configuration can affect several zones at once. Roll out changes to a limited group first and preserve a usable rollback path. AWS's bulkhead guidance describes isolated groups that constrain the scope of a failure. Any shared dependency between those groups remains part of the review.
A second region adds another recovery path to test. Check its data, credentials, capacity and traffic routing, including what happens if the first region cannot be reached to coordinate the switch.
Run a bounded failure drill
Use an isolated environment first. Choose one fault and define an owner, a stop condition and a rollback before introducing it. These are proposed acceptance checks, not results from a Pingy lab.
- Record successful user operations and representative load before the fault. Include a unique record ID for writes you intend to verify later.
- Remove one application instance. Observe routing, errors and remaining capacity. Repeat with the intended zone-level fault only after the smaller test succeeds.
- Test database failover separately. Observe client recovery and verify that the old writer cannot conflict with the new one. Compare recovered records with acknowledged writes.
- Test cache loss and one external dependency timeout as separate cases. Watch downstream load and inspect ambiguous operations before retrying them.
- Restore the failed component, then verify routing, data ownership and remaining redundancy. End the drill only when the recovery steps are complete.
Keep the fault timestamps, user-visible errors, restored record IDs and remaining defects together. External probes can show reachability from their locations; application logs and data checks must establish whether the operation succeeded. Our multi-region monitoring guide explains how to interpret those probe results alongside your own evidence.