The problem: don't route to a dead instance#
TL;DRthe 30-second version
- A health check answers one question about an instance: should traffic go to it right now? The load balancer (or orchestrator) uses the answer to route around broken instances.
- Split the question in two. Liveness asks 'is the process alive, or should it be killed and restarted?' Readiness asks 'is it ready to take traffic this second?' A warming-up or overloaded instance is alive but not ready β conflating the two causes restart loops or blackholed traffic.
- Active checks poll a /health endpoint on a schedule. Passive checks infer health from real request success and failure (this is what a circuit breaker does). Most systems run both.
- At scale, one machine can't poll everyone. Nodes send heartbeats β a periodic 'I'm alive' β and a missed-heartbeat timeout marks a node dead. Too short a timeout gives false positives from a GC pause; too long is slow to notice a real death.
- The danger: a check that's too aggressive marks healthy instances dead under load, removes them, piles their traffic onto the survivors, and cascades into a full outage.
Start with the simplest thing that works. You register your three instances with the load balancer, and it spreads requests across all three. This is fine right up until one instance breaks. Say order-2 hits a bug and its process is still running but wedged β it accepts the TCP connection and then never replies. The load balancer has no reason to suspect anything, so it keeps handing order-2 its share of traffic. One in three requests now hangs until it times out. The instance is broken, but nothing is watching for that, so nobody routes around it.
The obvious fix is to watch. Have something ask each instance, over and over, 'are you okay?' β and stop sending traffic to any instance that stops saying yes. That single idea is the health check. Everything else on this page is the consequence of one hard question hiding inside it: what exactly does 'okay' mean, and how sure do you have to be before you act on a 'no'?
Two questions, not one: alive vs ready#
The first instinct is to ask each instance one yes-or-no question: are you healthy? But 'healthy' bundles together two things that need different responses, and pulling them apart is the single most useful idea here.
The first question is liveness: is the process alive, or is it wedged and beyond saving? A wedged process β deadlocked, out of memory, stuck in a bad state β won't recover on its own. The right response is to kill it and start a fresh copy. In Kubernetes this is the liveness probe: if it fails enough times in a row, the kubelet (the agent running on each machine) restarts the container.
The second question is readiness: is this instance ready to serve a request this second? An instance can be perfectly alive and still not want traffic. It might be warming up β loading a large model into memory, filling a cache, opening database connections. It might be briefly overloaded and shedding load on purpose. Restarting it would be exactly wrong; it's not broken, it's just busy. The right response is to leave it running but route around it until it says it's ready again. In Kubernetes this is the readiness probe: fail it and the pod is pulled out of the load balancer's rotation, but the container is not restarted.
How does the checker ask? The instance exposes a health endpoint β a plain HTTP route, conventionally /healthz or /ready, that returns 200 when things are good and a 5xx (or nothing) when they're not. The checker makes a request to that endpoint on a schedule and reads the answer. The load balancer only routes to instances whose latest readiness check passed β you can watch exactly that routing decision in the load balancer simulator (/loadbalancer/sim), which sends requests only to backends currently marked healthy.
PredictYour service takes 40 seconds to load its data on startup. You give it a liveness probe on /healthz that starts checking at 10 seconds and fails the container after 3 misses. What happens when you deploy?
Hint: The check fires before the service has finished starting. What does a failed liveness check do?
It never starts. At 10 seconds the instance is still loading, so /healthz fails; three misses later the kubelet decides the container is wedged and restarts it β right back to a fresh, un-warmed process that will fail the same check again. That's a restart loop: the container dies every ~15 seconds and never once reaches the ready state. The fix is to keep startup out of liveness: use a separate startup probe (or a generous initial delay) that gives the process its 40 seconds before any liveness check counts against it, and use readiness β not liveness β to keep it out of rotation while it warms. Liveness is only for 'wedged, needs a restart', never for 'still starting'.
Active checks vs passive checks#
So far the checker actively asks each instance how it's doing. That's an active check: a dedicated probe hits /health on a schedule, independent of real traffic. It's precise and predictable, but it has a blind spot β the probe can pass while real requests fail, because the health endpoint runs a different, cheaper code path than the request that's actually breaking.
The other approach uses the traffic you already have. A passive check infers health from the outcome of real requests: if calls to order-2 start timing out or returning 500s, order-2 is probably sick β no separate probe needed. This is exactly what a circuit breaker does. It watches the live success and failure rate of calls to a dependency, and when failures cross a threshold it 'trips' and stops sending requests there for a while. You can watch a breaker trip on a rising failure rate in the circuit breaker simulator (/circuitbreaker/sim). Passive checks cost nothing extra and see the real failure, but they only learn about a problem once real users have hit it.
The two are complementary, so most production systems run both. Here's how they line up:
| Active check | Passive check | |
|---|---|---|
| How it learns | Polls a /health endpoint on a schedule | Infers from real request success/failure |
| Extra load | Probe traffic, even when idle | None β reuses real traffic |
| Sees the real failure? | Only what the endpoint tests | Yes β the actual request path |
| Reacts before users hit it? | Yes β probes independently of traffic | No β needs failing requests first |
| Typical home | Load balancer / orchestrator probe | Circuit breaker in the client |
Heartbeats: when the instance reports in#
Polling has a direction problem at scale. If one monitor has to actively poll every instance, its work grows with the fleet, and it becomes the single thing whose failure blinds you to everyone. So flip the direction: instead of a monitor asking each node 'are you alive?', each node periodically tells the monitor 'I'm alive'. That periodic signal is a heartbeat β a small message sent on a fixed interval, say once a second.
Now health is measured by silence. As long as heartbeats keep arriving, the node is considered alive. When they stop, a timer runs down, and after a chosen timeout with no heartbeat the monitor declares the node dead and acts β removes it from rotation, triggers a failover, reassigns its work. The whole failure detector is this: a clock per node, reset by each heartbeat, and a threshold that converts 'hasn't been heard from in a while' into 'dead'.
One monitor collecting every node's heartbeat still has the single-point problem. Large clusters solve it by making failure detection everyone's job: nodes heartbeat to a few peers, and those peers spread what they've learned about who's alive and who's dead through gossip β the same rumor-spreading protocol used for membership. The gossip simulator (/gossip/sim) shows a cluster converging on a shared view of which members are up as that information ripples outward, so no single monitor has to watch the whole fleet.
Go deeperGo deeper: SWIM's indirect probe and the phi-accrual detector
Two refinements make large-cluster failure detection both cheaper and less trigger-happy. The first is SWIM (Scalable Weakly-consistent Infection-style process group Membership), the protocol behind many gossip systems. Instead of every node pinging every other node, each node periodically pings one random member and waits for an ack. If the ack doesn't come, SWIM doesn't immediately declare that member dead β the ping or the ack might have been the only thing lost. Instead it asks k other members (often around 3) to each try pinging the same target on its behalf. Only if all those indirect probes also fail does the node suspect the target. That indirect probe is what tells a genuine death apart from a single dropped packet between just two nodes.
The second is the phi-accrual failure detector (Hayashibara et al.), used by Cassandra and Akka. A plain timeout gives a hard yes/no on a fixed threshold, which is fragile: pick 3 seconds and a 3.1-second GC pause is a false positive while a network that's grown slow gets no slack. Phi-accrual instead outputs a continuous suspicion value, phi, computed from the recent statistical distribution of that node's heartbeat inter-arrival times. Phi rises the longer a heartbeat is overdue relative to how punctual that node usually is; the higher it climbs, the lower the probability the node is actually still alive. You act when phi crosses a configured threshold β Cassandra's default phi_convict_threshold is 8, meaning roughly a one-in-a-hundred-million chance you're wrong to convict. The win is that the detector adapts to real network conditions instead of trusting one hand-picked number.
The timeout is a trade you can compute#
The timeout β how long a silence has to last before you call a node dead β is the one dial that decides everything about a heartbeat detector, and it's a genuine trade with two failure modes pulling in opposite directions.
- Too short, and you get false positives. Set the timeout below the pauses that happen in normal operation β a stop-the-world GC pause of a second or two, a brief network hiccup β and you'll declare a perfectly healthy node dead every time it stutters. Now you've evicted a working instance, moved its traffic and work for nothing, and you'll flap it right back when the next heartbeat arrives.
- Too long, and you're slow to react. Set the timeout to 30 seconds to be safe, and a node that genuinely crashes keeps its share of traffic β and any work assigned to it β for a full 30 seconds before anyone notices. That's 30 seconds of errors, or 30 seconds before a failover starts.
The detection window is roughly the heartbeat interval times how many missed beats you tolerate. Heartbeat every 1 second and convict after 3 misses, and a dead node is spotted in about 3 seconds β but any single pause longer than 3 seconds falsely convicts a live one. The interval and the threshold are the knobs; the product is how fast you see a real death, and the same product is how much normal jitter you can absorb without a false alarm. There's no setting that's fast and forgiving at once, which is the whole reason adaptive detectors like phi-accrual exist β they widen the window automatically when the network is behaving badly and tighten it when it's calm.
PredictHeartbeat interval is 1 second and you convict a node after 5 consecutive misses. How long can a genuinely crashed node keep receiving traffic β and would a 4-second GC pause falsely convict a healthy node?
Hint: Detection time β interval Γ misses tolerated. Compare the pause length to that window.
A crashed node goes unnoticed for up to about 5 seconds (5 misses Γ 1 second), so it keeps its share of traffic and work for that long before it's pulled. And no β a 4-second GC pause produces at most 4 missed beats, one short of the 5-miss threshold, so the node survives it (as long as heartbeats resume before the 5th). That's the tuning in one example: 5 misses buys you tolerance for pauses up to ~5 seconds at the cost of ~5 seconds to detect a real death. Want faster detection? Drop the threshold β and accept that shorter pauses now trip it. This is exactly the trade phi-accrual automates by scoring the pause against the node's own history instead of a fixed count.
Tuning the check β and the cascade if you get it wrong#
It's tempting to make checks aggressive β probe often, fail fast, pull anything that so much as flinches. Under load that instinct is dangerous, because a health check that's too eager doesn't just remove one instance; it can remove the whole fleet.
- Traffic rises. Every instance slows down a little under the extra load, and response times creep up β including the response time of the /health endpoint.
- The probe times out. Its 1-second deadline is now shorter than how long a loaded-but-healthy instance takes to answer, so the check fails on instances that are working, just slow.
- The load balancer pulls them. It marks the 'failing' instances unhealthy and takes them out of rotation, believing it's routing around broken machines.
- The survivors get more load. The same total traffic now lands on fewer instances, so they slow down further and start failing the same probe β and the removal spreads until nothing is left in rotation.
There's a matching trap in what the check tests, which pulls the other way. A shallow check ('is the process answering?') can pass while the service is useless because its database is down. A deep check ('can I actually reach my dependencies?') catches that β but it also means that when the shared database has a hiccup, every instance's deep check fails at once, the load balancer pulls the entire fleet, and a recoverable dependency blip becomes a total outage. The judgment call is how deep to check: deep enough to catch a truly broken instance, shallow enough that a shared dependency's wobble doesn't convict every instance simultaneously.
In the wild
- Kubernetes runs three probe types per container: a liveness probe (fail β the kubelet restarts the container), a readiness probe (fail β the pod is removed from Service endpoints but not restarted), and a startup probe that holds off the other two until a slow-starting app has finished booting. Defaults are a 1-second timeout and a failureThreshold of 3.
- AWS Elastic Load Balancing actively health-checks each target on an interval (default 30 seconds), requiring a configurable number of consecutive successes (HealthyThresholdCount) to put a target back in service and consecutive failures (UnhealthyThresholdCount) to take it out β the consecutive-count rule that stops a single blip from flapping a target.
- Cassandra uses a phi-accrual failure detector for node liveness, with phi_convict_threshold (default 8) deciding when a peer is marked down β an adaptive detector chosen precisely because a fixed timeout misbehaves across variable cluster networks.
- Consul and other service meshes combine active checks (HTTP/TCP/script probes) with a SWIM-style gossip layer (Serf) so membership and failure detection scale across a large cluster without a central monitor polling everyone.
Common questions & gotchas
Why not just use one health check for everything?
Because 'restart it' and 'route around it' are opposite responses to different states. A warming-up or briefly overloaded instance is alive but not ready β restarting it (a liveness action) is wrong; you want to pull it from rotation (a readiness action) until it recovers. A wedged process is the reverse: pulling it from rotation leaves it hung forever, while what it needs is a restart. One check can't trigger two opposite actions, which is why liveness and readiness are separate.
My pods restart in a loop right after deploy. What's the usual cause?
A liveness probe that fires before a slow-starting app has finished booting. The app needs, say, 40 seconds to warm up; liveness checks at 10 seconds, fails, and the kubelet restarts the container into another cold start that fails the same way. Fix it by moving startup out of liveness β use a startup probe (or a longer initial delay) to grant the boot time, and gate traffic with readiness, not liveness.
How can a bad health check take down a whole healthy fleet?
Under rising load, every instance's /health endpoint slows down too. If the probe timeout is tight, it starts failing on instances that are healthy but slow. The load balancer removes them, their traffic piles onto the survivors, those slow down and fail the same probe, and the eviction cascades until the fleet is empty. The defenses are generous probe timeouts, a cheap health endpoint, requiring several consecutive failures, and failing open when almost everything looks unhealthy at once.
Should my health check test the database and other dependencies?
Carefully. A deep check that pings dependencies catches an instance that's up but can't do its job β but it also means a shared dependency's blip fails every instance's check simultaneously, and the load balancer pulls the entire fleet over a problem that would have recovered on its own. Test enough to catch a genuinely broken instance, but avoid making a shared dependency a single point that can convict every instance at once.
Why heartbeats instead of just having the monitor poll everyone?
Polling makes the monitor's work grow with the fleet and makes that one monitor a single point of failure β if it dies, you're blind to everyone. Heartbeats flip the direction (nodes report in), and large clusters go further by gossiping liveness among peers so no single machine has to watch the whole fleet. The cost is that health is now inferred from silence, so you must tune the missed-heartbeat timeout.
QuizDuring a traffic spike, your load balancer suddenly marks almost every backend unhealthy at once and requests start failing everywhere. What is the most likely cause and the right guardrail?
- Every backend crashed simultaneously β add more replicas.
- The health-probe timeout is too tight for loaded response times, so it's failing healthy-but-slow instances; the load balancer should fail open when nearly all targets look unhealthy.
- The readiness probe is testing the wrong endpoint β point it at /.
- Heartbeats are being sent too often β increase the interval.
Show answer
The health-probe timeout is too tight for loaded response times, so it's failing healthy-but-slow instances; the load balancer should fail open when nearly all targets look unhealthy. β A whole fleet going unhealthy at the same instant is almost never a simultaneous crash; it's a correlated slowdown fooling an over-tight probe. Under load the /health endpoint answers slower, a tight timeout fails healthy-but-slow instances, they're evicted, their traffic crushes the survivors, and the removal cascades. The guardrail is to fail open β when nearly everything looks unhealthy at once, keep routing to all of it rather than route to nothing β plus generous timeouts and requiring consecutive failures so one slow beat can't evict a node.
In an interview
Health checks come up whenever you draw a load balancer in front of a fleet, or a cluster that has to survive a node dying. The tell of a strong answer is that you don't treat 'health' as one boolean β you name the distinctions and the traps before you're asked.
- Lead with liveness vs readiness: liveness = 'is it wedged, should we restart it'; readiness = 'is it ready for traffic right now'. Name the failure of merging them (restart loops on slow starts, blackholed traffic on real crashes).
- Distinguish active from passive: an active probe polls /health on a schedule; a passive check infers health from real request outcomes (a circuit breaker). Say you'd run both, and why each covers the other's blind spot.
- Explain heartbeats and the timeout trade as the core mechanism: health is inferred from silence, and the timeout is a bet β too short false-positives on a GC pause, too long is slow to detect a real death. Detection time β interval Γ missed-beats tolerated.
- Name the cascade as the senior-level trap: an over-aggressive check under load evicts healthy-but-slow instances, overloads the survivors, and takes down the fleet. The guardrails β fail open, generous timeouts, consecutive-failure thresholds, cheap and not-too-deep endpoints β are what you actually get graded on.
- For cluster scale, reach for gossip-based detection (SWIM's indirect probing, phi-accrual's adaptive suspicion) instead of one central monitor β and ground it: Kubernetes probes, AWS ELB thresholds, Cassandra's phi detector.
PredictAn interviewer says: 'Your health checks are green, but users are still getting errors from that service. How is that possible?' What's the strong answer?
Hint: What does a shallow probe actually test, versus what a real request exercises?
The probe is passing on a code path that isn't the one failing. A shallow /health that just returns 200 tests that the process is up and answering β not that the actual request path works. So the service can be 'healthy' while every real request throws, because the failing dependency or code branch is never touched by the probe. That's the exact gap passive checks close: a circuit breaker watching real request outcomes would already see the failures the probe can't. The strong answer names the shallow-vs-deep-check trade (make the check exercise enough of the real path to catch this, without going so deep that a shared dependency's blip convicts the whole fleet) and pairs the active probe with passive detection so real failures are caught even when the probe is fooled.
References & further reading
- Kubernetes β Configure Liveness, Readiness and Startup Probes β the canonical spec for the three probe types, thresholds, and timeouts
- Kubernetes β Pod Lifecycle: Container probes β what each probe outcome does (restart vs remove from endpoints)
- Das, Gupta & Motivala β SWIM (2002) β scalable failure detection with random and indirect probing
- Hayashibara et al. β The Ο Accrual Failure Detector β suspicion as a continuous score instead of a fixed timeout
- AWS β Health checks for Application Load Balancer target groups β interval, healthy/unhealthy thresholds, and the consecutive-count rule
- Google SRE Book β Addressing Cascading Failures β why over-aggressive health checks and correlated slowdowns take down a fleet
- Cassandra β Failure detection (phi-accrual) β phi_convict_threshold and gossip-based node liveness in production