Start here: the address you hardcoded is already wrong#
TL;DRthe 30-second version
- In a cluster, instances come and go every minute — deploys, autoscaling, and crashes all hand out fresh addresses. A hardcoded address for another service is stale almost immediately.
- The fix is a service registry: a live directory of name → the addresses of that service's healthy instances. Instances register themselves on startup and deregister on shutdown.
- A crash skips the deregister step, so the registry would keep a dead address. Health checks fix that — the registry keeps probing each instance and drops the ones that stop answering.
- Turning a name into a connection happens one of two ways. Client-side discovery: the caller asks the registry for the list and picks an instance itself. Server-side discovery: the caller sends to a load balancer or gateway, which asks the registry and forwards.
- The registry can be plain DNS (simple, but caches poorly and carries no health) or a dedicated system like Consul, etcd, ZooKeeper, or Eureka. Kubernetes builds it in: a Service name, resolved by CoreDNS, backed by an endpoint list the control plane keeps current.
You're running two services in a cluster. The `orders` service takes a checkout and needs to call the `payments` service to charge the card. On your laptop this was one line: `payments` lives at `10.2.1.7:8080`, so `orders` connects there. In production that line is a time bomb.
Here's why. The `payments` service isn't one process — it's a pool of identical instances behind that name, and the pool is never still. Deploy a new version and the platform rolls the instances one at a time: each old instance is killed and a replacement starts, and the replacement comes up on a brand-new address. Traffic spikes at lunch and autoscaling doubles the pool from four instances to eight — the four new ones have addresses that didn't exist a second ago. A machine fails and its instance vanishes with no notice at all. Over a single busy hour, the set of addresses behind `payments` can turn over completely.
So `10.2.1.7:8080` isn't wrong because someone typed it badly. It's wrong because it describes a world that no longer exists. Point `orders` at a fixed address and it will, sooner rather than later, be dialing an instance that has been deployed away, scaled down, or killed — while healthy instances sit idle because nothing told `orders` they exist. The caller needs to ask its question at the moment it calls, not read an answer someone wrote down last week.
The registry: a live directory of who's up#
If a fixed list won't do, the caller needs to look the answer up somewhere that stays current. That somewhere is a service registry: a directory that maps each service name to the addresses of its currently running instances. Ask it for `payments` and it hands back the live set — `10.2.1.7:8080`, `10.2.3.4:8080`, and so on — as they are at this moment.
The trick is keeping that directory honest as the pool churns. The instances themselves do most of the work. When a `payments` instance finishes starting up, its first act is to register: it tells the registry "I'm an instance of `payments`, reach me at `10.2.3.4:8080`." Now the registry knows it exists. When the instance is asked to shut down — say, during a deploy — it deregisters before it exits, telling the registry to drop that address. Register on the way in, deregister on the way out: that alone keeps the directory tracking the deploys and the scale-ups.
But deregister-on-shutdown assumes a polite exit, and crashes aren't polite. A machine that loses power or a process that segfaults never gets to say goodbye — its address just stops answering, while the registry still lists it as up. A caller handed that address connects to nothing. So registration can't be a one-time announcement; the registry has to keep checking that each instance is still alive.
That check is the health check, and it comes in two shapes. In the pull shape, the registry (or an agent next to the instance) periodically calls a lightweight endpoint on the instance — an HTTP `GET /health` that returns 200, or a TCP connection that succeeds. Miss a few in a row and the instance is marked unhealthy and pulled from the list. In the push shape, the instance sends the registry a small heartbeat on a timer — "still here" — and each heartbeat resets a countdown called a TTL (time-to-live). If a full TTL passes with no heartbeat, the countdown hits zero and the registry evicts the instance on the assumption it died. Netflix's Eureka registry uses exactly this: an instance renews its registration (its lease) every 30 seconds, and Eureka evicts it if no renewal arrives for 90 seconds.
With registration, deregistration, and health checks in place, the registry now holds a set of names, each mapped to just the healthy instances behind it, updated within seconds of any change. The only step left is turning that into an actual connection — and there are two ways to do it.
Two ways to turn a name into a connection#
The registry knows the healthy addresses. The remaining choice is who reads that list and picks an instance: the caller itself, or something in the middle. That single question splits service discovery into its two models.
In client-side discovery, the caller does the work. When `orders` wants `payments`, it asks the registry directly for the list of healthy instances, then chooses one itself — round-robin, least-busy, or any load-balancing rule you like — and connects straight to it. The caller is its own load balancer. Netflix ran this at scale: services queried the Eureka registry and used a client library called Ribbon to pick and call an instance. The upside is one fewer hop and full control over how the caller spreads its own traffic. The cost is that every service, in every language you use, now needs the discovery-and-balancing logic baked into it — a Go service and a Java service each need their own working copy.
In server-side discovery, the caller stays dumb. `orders` sends its request to a single stable address — a load balancer or gateway sitting in front of `payments`. That router is the one that consults the registry, picks a healthy instance, and forwards the request. The caller never sees the registry or the instance list at all; it just talks to the router. The upside is that the discovery logic lives in one place instead of inside every service, so a new service in any language needs no special code. The cost is an extra network hop through the router, and the router itself is now a component you run and keep alive. If the idea of a component that fronts a pool and spreads requests across it sounds familiar, it's the load balancer — you can watch one distribute traffic across a changing backend pool in the load balancer simulator (/loadbalancer/sim).
Both models lean on the same registry underneath. They differ only in where the lookup-and-pick happens — inside the caller, or inside a router the caller sends to. That's the whole fork, and most real setups land on server-side because keeping the discovery logic out of every service is worth one extra hop.
DNS, or a dedicated registry?#
One more decision: what actually stores the directory? There's a tempting free option. DNS already maps names to addresses — it's how `example.com` becomes an IP before any request goes out (see the DNS topic, and watch a name resolve in the DNS simulator, /dns/sim). So you could register each instance as a DNS record under `payments.internal` and let callers resolve the name the normal way. It works, and for simple setups it's genuinely enough.
DNS starts to strain in exactly the conditions service discovery cares about. The first problem is staleness. DNS is built to be cached hard: every record carries a TTL, and resolvers and clients hold onto the answer until it expires. When your instance set turns over in seconds, a record cached for even 30 seconds means callers keep dialing an address that's already gone. It gets worse: many client libraries ignore the TTL and cache a resolved address for the life of the process, so they never see the change at all. The second problem is that plain DNS returns addresses and nothing else: it can't tell a caller which instances are healthy, how loaded each one is, or anything beyond a flat list. Health lives outside DNS, so a dead instance can linger in the records until its TTL lapses.
A dedicated registry — Consul, etcd, ZooKeeper, or Eureka — is a system built for this churn instead of adapted to it. It holds richer entries than a name-to-address record (health status, metadata, tags), it health-checks instances and drops the dead ones in seconds, and it can push changes to callers rather than waiting for a cache to expire. You run and operate it, which is the price, but you get a directory that keeps up with a pool that never sits still. The rule of thumb: reach for DNS when the instance set is fairly stable and you want zero extra infrastructure; reach for a dedicated registry when instances churn fast and you need health and freshness DNS can't give.
PredictYour registry health-checks each `payments` instance every 5 seconds and evicts one after 3 consecutive misses. A `payments` instance crashes the instant after passing a check. For roughly how long can a caller still be handed its dead address — and what does the caller need to do about it?
Hint: Count the worst-case time from the crash to the third missed probe. Then ask what's still true about an address the registry hasn't caught up on yet.
Up to about 15 seconds. The crash happened just after a passing check, so the next probe is ~5s away; it takes 3 misses in a row to evict, and the probes are 5s apart, so ~15s elapses before the dead address is dropped from the registry. During that window the registry is honestly wrong — it's showing an instance it hasn't yet noticed is gone. That gap is unavoidable: any health check that isn't instantaneous has one. So the caller can't assume every address it gets is live. It must handle a failed connection gracefully — time out fast and retry the request against a different instance from the list. Discovery narrows the pool to probably-healthy instances; the retry covers the ones that died inside the detection window. Tighten the probe interval or the miss threshold and you shrink the window, but you never close it, and you pay in more health-check traffic.
When to reach for which model
The two discovery models aren't ranked — they fit different situations. A few rules of thumb that hold up in an interview:
- One language, latency-sensitive, you control every caller: client-side discovery is fine and saves a hop. Netflix ran this way (Eureka + Ribbon) because they owned their whole JVM stack and could ship the discovery client everywhere.
- Many languages, or you'd rather not touch every service: server-side discovery. The discovery logic lives in the router (load balancer, gateway, or the platform), so a new service in any language needs no special code to be reachable.
- You're on Kubernetes (or a similar platform): the decision is largely made for you — the platform provides server-side discovery as a built-in, so callers just use a stable service name and the platform routes it.
- You want the simplest possible thing and instances are fairly stable: DNS-based discovery, no extra system to run — accepting DNS's caching staleness and lack of health awareness.
Client-side and server-side sit side by side below; introduce the table with the trade each column encodes.
| Client-side discovery | Server-side discovery | |
|---|---|---|
| Who picks the instance | The caller itself | A load balancer / gateway in front |
| Discovery logic lives | In every service, in every language | In one place (the router / platform) |
| Network hops | One — caller connects directly | Two — caller → router → instance |
| New service in a new language | Needs its own discovery client | Works with no special code |
| Example | Netflix Eureka + Ribbon | Kubernetes Service, AWS ELB, an API gateway |
Under the hood: is the registry AP or CP?
The registry is shared state that every caller depends on, so it's usually replicated across several nodes for fault tolerance. That raises the classic question for any replicated store: when the network splits the registry's own nodes apart, does it stay available and risk serving slightly stale data (an AP choice), or does it refuse reads it can't prove are current to stay consistent (a CP choice)? For a service registry the answer is often the opposite of your instinct.
A slightly stale instance list is survivable — callers already retry dead addresses, as the health-check window showed. But a registry that refuses to answer during a partition is a catastrophe: if no one can discover anyone, every service-to-service call fails at once. That logic pushes a discovery-focused registry toward AP — stay available, serve the last known list, accept some staleness. Eureka is the clearest example: it's deliberately AP and eventually consistent, and under a partition each node keeps serving from its last snapshot rather than going dark. Clients even cache the registry locally, so they can keep resolving names if the registry is briefly unreachable.
The CP registries — etcd, Consul, ZooKeeper — make the other call: they run a consensus protocol (etcd and Consul use Raft) so that a majority of nodes agree on every change, and during a partition the minority side stops serving writes rather than diverge. You can watch that majority-agreement machinery run in the Raft simulator (/raft/sim).
Why pick CP for discovery, given the argument above? Because these systems are rarely just a service registry — they're also the store for leader election, distributed locks, and configuration, where a stale answer is genuinely dangerous. Consul layers both: it uses a gossip protocol to spread instance liveness cheaply across many nodes (see the gossip simulator, /gossip/sim) and Raft for its strongly-consistent key-value store. The takeaway for an interview: a registry used purely for discovery leans AP; one that doubles as a coordination store leans CP, and naming that split shows you understand what the registry is really for.
Under the hood: how Kubernetes does it
Kubernetes is the most common place engineers meet service discovery, and it hides almost all of the above behind one object. You declare a Service named `payments`, and Kubernetes gives it a single stable virtual address — a ClusterIP — that never changes for the life of the Service, even as the instances (pods) behind it come and go.
- You label your `payments` pods and create a Service that selects them. The Service gets a fixed ClusterIP and a DNS name, `payments.<namespace>.svc.cluster.local`.
- The control plane continuously reconciles the set of pods matching that label into an endpoint list — the current healthy pod addresses — and keeps it current as pods start, stop, or fail their readiness probes.
- CoreDNS, the cluster's DNS server, answers a lookup of the `payments` name with the Service's stable ClusterIP. So `orders` just resolves a name, exactly like normal DNS (/dns/sim).
- kube-proxy programs packet-forwarding rules on every node so that traffic sent to the ClusterIP is rewritten to one of the real, healthy pod addresses from the endpoint list — spreading requests across them.
Read against the two models, this is server-side discovery built into the platform: `orders` sends to a stable name that resolves to a stable virtual IP, and the platform (the endpoint controller plus kube-proxy) does the registry lookup and the instance pick for it. The instances register implicitly by matching the Service's label selector and passing their readiness probe; they're evicted implicitly when the probe fails or the pod dies. The caller writes no discovery code at all — which is exactly why the platform-provided model won.
In the wild
- Consul (HashiCorp): a dedicated registry with an agent on every node that runs health checks (HTTP, TCP, or TTL heartbeats), gossip for membership, and a Raft-backed consistent key-value store. Also serves DNS, so callers can discover services either by DNS name or via its API.
- etcd: a strongly-consistent (CP) key-value store on Raft, used as the backing store for Kubernetes' own state — including the Service and endpoint objects that drive its discovery. Chosen where a stale answer would be dangerous.
- ZooKeeper: the elder of the group, a CP coordination store long used for service registration and discovery in the Hadoop/Kafka world, before lighter-weight registries appeared.
- Netflix Eureka: the canonical AP registry, paired with the Ribbon client for client-side discovery. Deliberately favors availability and eventual consistency so discovery keeps working through partitions.
- Kubernetes: discovery as a platform built-in — a Service name resolved by CoreDNS to a stable ClusterIP, with kube-proxy routing to healthy pods. Most engineers today meet service discovery here first.
Pitfalls & gotchas
If the registry knows the healthy instances, why do callers still get connection errors?
Because the registry is always a little behind reality. An instance can die in the gap between health checks, so its address stays in the list until the next few probes miss. Discovery gives you a probably-healthy address, not a guaranteed-live one. Callers must still time out fast and retry against another instance — discovery narrows the pool, the retry covers the detection window.
Isn't the registry a single point of failure — if it's down, nothing can find anything?
It would be if you ran one copy, which is why registries are replicated across several nodes. Beyond that, clients usually cache the last instance list locally, so a brief registry outage doesn't immediately break every call — they keep using the last known set. And AP registries like Eureka deliberately keep serving from their last snapshot during a partition rather than going dark. The registry is designed so its own failure degrades discovery, not halts it.
Why is DNS-based discovery so often a trap?
Because DNS is built to be cached, and service instances change faster than the cache expires. Even a short TTL means callers dial addresses that are already gone, and many client libraries cache a resolved address for the whole process lifetime and never re-resolve — so they never see a scale-up or a moved instance at all. DNS also carries no health information, so a dead instance lingers in the records until its TTL lapses. Fine for stable pools; dangerous for churning ones.
What's a stale-cache problem in client-side discovery specifically?
In client-side discovery each caller caches the instance list to avoid asking the registry on every request. If that cache updates slowly, a caller keeps sending to an instance that was removed, or ignores a fresh instance and leaves it idle. The fix is short cache lifetimes plus the registry pushing changes, but there's always some lag — which is the same reason callers must retry on failure.
QuizA team moves from client-side discovery (each service embeds a discovery client) to server-side discovery behind a load balancer, mainly because they're adding services in three new languages. What's the concrete win, and the cost they take on?
- Win: discovery is now instant and never stale. Cost: none worth mentioning.
- Win: the discovery-and-balancing logic lives in one router instead of in every service and every language. Cost: an extra network hop per call, and a router they must run and keep alive.
- Win: they no longer need a registry at all. Cost: higher latency.
- Win: the registry becomes strongly consistent. Cost: the caller must embed more logic.
Show answer
Win: the discovery-and-balancing logic lives in one router instead of in every service and every language. Cost: an extra network hop per call, and a router they must run and keep alive. — Server-side discovery pulls the discovery-and-balancing logic out of every service and into a single router (load balancer or gateway), so a new service in any language is reachable with no special client code — exactly what a polyglot team wants. The costs are real but modest: every call now goes caller → router → instance, one hop more than connecting directly, and the router is a component they operate and must keep highly available. Discovery is not now instant or never-stale (the registry still lags reality), and they still need a registry — the router is what reads it. The registry's consistency model (AP vs CP) is a separate axis this change doesn't touch.
In an interview
Service discovery shows up the moment you draw more than one service on the board. The strong answer isn't reciting Consul's feature list — it's showing you know why fixed addresses fail and what the registry actually guarantees.
- Lead with the problem: instances come and go (deploys, autoscaling, crashes), so addresses are never stable — a hardcoded address is stale almost immediately. Everything follows from that.
- Describe the registry as three parts: register on startup, deregister on shutdown, and health-check to catch the crash that skipped the deregister. Missing the third part is the common gap.
- Name the two models and their trade: client-side (caller queries the registry and load-balances itself — one hop, but discovery logic in every service) vs server-side (a router does it — one place, but an extra hop). Say which most teams pick and why.
- Name the trap: the registry is always slightly stale, so discovery does not remove the need to retry a failed connection. A candidate who says 'the registry guarantees the address is live' has missed it.
- For depth, raise AP vs CP: a discovery-only registry leans AP (stay available, tolerate staleness — Eureka), while one that also does locks or leader election leans CP (etcd, Consul on Raft). And note Kubernetes gives you server-side discovery for free via Services + CoreDNS + kube-proxy.
PredictAn interviewer asks: "Your registry itself gets partitioned — half its nodes can't see the other half. Should it keep answering discovery queries or stop? Defend your choice."
Hint: Compare the blast radius of 'serve slightly stale' against 'serve nothing.' Then ask what else the registry might be responsible for.
Argue for staying available (the AP choice) for a discovery-focused registry, and defend it with the failure math. If the registry stops answering during the partition, every service-to-service call that needs a lookup fails at once — you've turned a registry problem into a total outage. If instead it keeps serving each side's last known instance list, the worst case is some staleness: a caller might get an address that's since moved or died. But callers already handle that — they retry failed connections against another instance — so staleness degrades gracefully while unavailability does not. That's why Eureka is deliberately AP and clients cache the list locally. The honest caveat: this reasoning holds when the registry's only job is discovery. If the same system also backs leader election, distributed locks, or configuration, a stale read there can cause split-brain (two nodes both acting as the leader) or corruption, and you want CP (etcd, Consul on Raft) even at the cost of refusing to answer in the minority partition. So the real answer is 'AP for pure discovery, CP when the registry is also a coordination store' — naming that split is the strong finish.
References
- microservices.io — Service Registry pattern — Chris Richardson's canonical description of the registry and how instances register/deregister.
- microservices.io — Client-side discovery — The caller queries the registry and load-balances itself.
- microservices.io — Server-side discovery — A router in front of the service does the registry lookup and forwarding.
- Consul — Health checks — HTTP, TCP, and TTL (heartbeat) check types, and how a failed check drops an instance.
- Netflix Eureka wiki — The AP registry: heartbeat renewal, self-preservation, and client-side caching.
- Kubernetes — Service — ClusterIP, endpoint reconciliation, and DNS-based discovery via CoreDNS + kube-proxy.
- etcd — Raft consensus — The CP key-value store on Raft that backs Kubernetes' own service/endpoint state.