Liveness, readiness and deep health checks

A health check should have a defined consequence. A failed liveness probe can restart a container. Failed readiness can stop new traffic reaching it. A diagnostic check should tell an operator what broke. Wiring all three to an endpoint that queries every dependency can turn one database outage into a fleet-wide disruption.

This tutorial designs those checks for an HTTP service on Kubernetes, then shows how to test their behavior. The endpoint names are examples you implement in your application, not built-in Kubernetes routes.

Decide what each result controls

Start with a small contract for each endpoint. This is a proposed design, not a requirement to create four separate HTTP handlers. Example paths are /healthz/startup, /healthz/live, /healthz/ready and /healthz/deep.

Kubernetes has startup, liveness and readiness probes. "Deep check" is an application convention. A startup probe suppresses liveness and readiness until it succeeds; readiness failure does not itself restart the container. Use startup probes when initialization needs more time than your normal liveness tolerance. Kubernetes documents these distinct actions.

Make liveness detect a failure a restart can fix

A responsive HTTP handler proves that handler responded. It can still return success while a separate worker pool is stuck. Choose a signal tied to the work this process must perform, such as whether its request-processing loop is making progress. Document what the signal misses.

Keep shared dependency availability out of this restart decision. A database outage usually needs database recovery; restarting every client adds connection churn. A process that exits reliably on fatal errors may not need an extra liveness probe. Kubernetes warns that badly designed liveness checks can cause cascading failures under load. See its liveness guidance.

Timeouts need measurements. Capture probe latency during cold starts and representative peak load before choosing a limit. A universal 50-100 millisecond limit can reject healthy instances under load.

Keep readiness specific to the traffic you can serve

Readiness can reject an instance whose configuration has not loaded or whose local connection pool is broken. A shared dependency failure needs a separate decision: can any useful requests still succeed, and would moving traffic help?

For a hypothetical catalog service, cached product reads might remain usable when the database is unavailable. Making database availability a blanket readiness condition would remove that remaining service. A write-only API with no useful fallback may reasonably become unready. Decide per workload and test both one-instance and fleet-wide failures. Amazon describes how dependency checks can turn an optional dependency into a mandatory one and expand an outage. Its health-check design discussion explains that tradeoff.

Do not treat temporary overload as an automatic reason to remove every replica. The surviving replicas would inherit the load. Bound queued work and consider rejecting excess requests or serving reduced responses. Google's overload guidance covers those options. Measure whether your chosen readiness behavior improves successful requests during a load test.

Check the routing layer's all-unhealthy behavior too. AWS Application Load Balancer can route to all unhealthy registered targets when the target group has no healthy targets. A red health status therefore does not guarantee that requests stopped arriving. ALB documents this fail-open behavior.

Configure probe timing and verify the response

For each Kubernetes probe, set the path and port, periodSeconds, timeoutSeconds and failureThreshold. Readiness also supports a successThreshold greater than one; startup and liveness require it to be one. Choose startup tolerance from measured initialization time, including slow starts.

Kubernetes HTTP probes accept status codes from 200 through 399. Use a direct success response for a healthy endpoint and a failure status such as 503 for an unhealthy one. Keep probe routes out of login redirects. A JSON body saying "status": "failed" inside a 200 response does not make an HTTP probe fail. The probe configuration documentation describes status handling.

Treat interval multiplied by failure threshold as a rough planning budget. Probe scheduling, timeouts and propagation to the traffic-routing layer affect when users see a change. Record the actual time between fault injection and the last new request to that replica. Also measure when it returns to service. Avoid reporting either as a guaranteed failover time.

Separate dependency diagnostics from user-path checks

A diagnostic endpoint can report database connectivity and dependency errors without controlling restarts. Bound each operation and the total request duration. If it returns a cached diagnostic result, include when the check ran and treat an expired result as unknown. Restrict detailed responses to authorized callers; keep credentials, internal addresses and exception traces out of public responses.

A successful SELECT 1 only proves that simple query worked through the tested connection. It does not verify the application's schema, write permissions or a completed order. Likewise, reaching an API's homepage proves little about the operation your application uses. Write down the assertion each check makes and the failures it cannot detect.

An external synthetic check should exercise a chosen public path, including its expected response. A separate controlled transaction can test a critical workflow using dedicated test data and cleanup. Google distinguishes black-box checks of visible behavior from internal monitoring. Keep both: dependency diagnostics help explain a failure, while a public-path check measures what a client could do from that probe.

Regional disagreement is evidence to investigate. It does not establish a routing fault by itself. Compare resolved addresses, TLS failures, response content and timestamps before assigning a cause. Set alert rules from the failure you need to detect and test their delivery delay; a check interval alone does not promise a notification deadline.

Test shutdown as well as failure

Readiness is not a complete connection-draining protocol. During Pod termination, Kubernetes marks terminating endpoints as not ready for regular traffic, while the application still needs to handle shutdown and finish appropriate in-flight work. The termination grace period also includes a preStop hook's execution. When that period expires, remaining processes can be forcibly killed. The Pod lifecycle documentation explains the sequence.

In staging, send representative requests continuously while terminating a replica normally. Record interrupted requests and how long connections remain open. Set the shutdown budget from that evidence and verify the ingress or load balancer's draining behavior. Long-lived sessions need their own reconnection policy.

Run a failure drill before enabling automatic action

Use an isolated staging workload with test data. These are proposed acceptance checks, not measured results. Adapt the expected behavior to the readiness contract you wrote above.

For each case, save probe responses, application errors, restart counts and the timestamps of routing changes. Restore the dependency and repeat the same user request to verify recovery. Reject a design that reports green while the required operation fails, or that creates more failed requests than the injected fault.