← Tutorials
🌍 Uptime

Multi-region monitoring: probes, quorum and regional failures

Choose useful probe locations, interpret conflicting results, and test alert delays without hiding regional failures.

By The Downtime Β· Jun 26, 2026 Β· 1:30 PM Β·Updated 5 days ago
Multi-region monitoring: probes, quorum and regional failures

A passing uptime check means a particular request worked from a particular network at a particular time. Another network can get a different DNS answer, reach a different CDN edge, or fail before it reaches your site. Multi-region monitoring gives you more places to observe those differences.

The useful part is the disagreement between probes. Keep those results visible, even when an overall status is green. A rule that waits for several locations to fail can suppress noise from one broken probe, but it can also leave a regional outage without a page.

Choose networks as well as locations

Start with the markets where people use your service. Then inspect what each monitoring location covers: its hosting provider, network, DNS resolver and available IP families. Two cities on one provider can still share dependencies. A probe near a customer does not necessarily take the customer's route.

RIPE Atlas illustrates this distinction: its probes provide measurement points across many networks, including home connections. Its network measurements can help investigate reachability beyond your normal monitoring fleet. Check the available measurement type before treating it as an application test. RIPE Atlas overview

Write down the gaps. If your customers use mobile networks and your probes all run in cloud data centers, you have limited evidence about mobile access. Add measurements from an affected network when investigating a complaint. More city names alone do not prove independent coverage.

Define the request before comparing results

Use the same public hostname, path and success criteria at each location. Record whether the probe follows redirects, checks certificate validity and verifies the response body. A 200 response carrying an error page can pass a status-only check. Google's SRE guidance treats incorrect content as a possible failure even when the HTTP status says success. Monitoring distributed systems

A cached page also tests a different path from a request that reaches the application. CloudFront, for example, can serve an object from an edge cache without fetching it from the origin. Keep a public-page check, but add an appropriate application check when you need evidence that a dependency still works. Document what that check exercises. CloudFront request handling

Test IPv4 and IPv6 separately when both matter to your users. A successful IPv4 request does not establish IPv6 availability. Prometheus Blackbox Exporter documents explicit address-family selection and warns that a hostname resolving to several addresses results in one selected address being tested. Review your own tool's selection and fallback behavior before interpreting a pass. Blackbox Exporter configuration

For each result, retain the location and sample time alongside the failure stage, selected destination address and address family where available. Without that context, two probes may appear to disagree while testing different things.

Decide what happens when probes disagree

A quorum is the number of locations whose results must agree before a rule changes state. It is an alerting choice. It cannot establish how many customers are affected.

The table below is a hypothetical policy for three probes checking the same endpoint. It is a design example, not a description of Pingy's alert settings or a measured incident. Choose severity from the affected service and users.

Fresh results Meaning Suggested response
All three pass The sampled requests worked Keep collecting; investigate user reports separately
One fails, two pass A failure is visible from one path Investigate that path and its user impact
Two fail, one passes Several paths show failure Page if this meets the service's impact rule
All three fail Every sampled path shows failure Page and compare the failure stages
One fails, two are missing Coverage is incomplete Report the failure and the missing measurements

A single failing location can justify an urgent page if it serves a critical market. Conversely, a shared probe-provider fault can make several locations fail together. Keep regional failure handling separate from the broad-outage rule so that one does not erase the other.

Do not assume every service uses majority voting. Route 53's endpoint health checks combine individual checker results using their own documented threshold: more than 18% reporting healthy yields an overall healthy state; 18% or fewer yields unhealthy. That is one product's endpoint-health policy, not a general recipe for uptime alerts. How Route 53 determines health

Confirmation takes time

Write down the full path from a failure to a notification: the next scheduled probe, its timeout or response, any retry, the required failure count or duration, aggregation, and notification delivery. Waiting for confirmation has a cost even when the resulting alert is more trustworthy.

For a hypothetical fixed schedule, suppose probes start every 60 seconds and a rule requires two consecutive failed results. A persistent fault beginning just after a successful probe starts may wait almost 120 seconds for the second failing probe to start. The time needed to complete that probe and deliver the alert comes afterward. This assumes each probe finishes before the next scheduled start; retry and scheduling behavior can change the result.

Some tools use elapsed duration instead of a failure count. In Prometheus, an alert with a for duration stays pending until the condition has remained active through evaluations for that duration. Notification handling is a separate layer. Measure the actual notification delay in a controlled test before committing to a detection target. Prometheus alerting rules

Treat a missing or stale result as incomplete evidence. Record the last sample time and check the probe's health. A system that displays an old green result indefinitely can hide a failed measurement pipeline.

Follow the failed request

Suppose, hypothetically, one probe repeatedly receives a 502 response while the other two receive 200. That establishes a difference in observed responses. It does not identify a broken CDN location or prove that every user in the probe's region gets the same error.

Compare evidence from the same time window before changing traffic:

Observation Next evidence to collect
DNS lookup fails Resolver result, record type and answers from another resolver
Connection or TLS fails Destination IP, IP family, certificate details and a test from another network
HTTP returns an error Status, safe response excerpt, request ID and matching application or CDN logs
Response is slow Timing stages, cache status and that location's earlier samples
No recent result Probe heartbeat, scheduler state and result-delivery health

If the failure appears only through the CDN, collect the provider's request identifiers and compare edge and origin logs where available. A cache invalidation removes cached objects; it does not take a failing edge server out of service. CloudFront documents invalidation as removing files so a later request can fetch them again from the origin. Use it when stale content is the established problem. CloudFront invalidation

Compare each location with its own history

Keep the request and measurement definition stable when looking for a latency change. Compare one location with its earlier results, and inspect failures separately from successful requests. A fast error response can make a combined average look better while the service gets worse. Google SRE on latency and errors

HTTP response time also leaves out parts of a browser visit. Real user monitoring collects measurements from actual visits; synthetic checks exercise predefined requests or journeys. Use both when you need to understand browser performance across customers' devices and networks. MDN on synthetic monitoring and RUM

Before relying on a new policy, use a controlled test endpoint to exercise a failure from one location, a failure across locations, and a probe that stops reporting. Record which notification arrived and when. Restore the endpoint and verify recovery handling too. Keep the observed results beside the policy so the next on-call engineer knows what the monitor will do.

πŸ’¬ Comments (0)

No comments yet β€” be the first to weigh in.

Join the conversation.