Start here: three equal machines, one job#
TL;DRthe 30-second version
- Leader election: a group of equal machines agree that exactly one of them is 'the leader' and does the work that must happen on one node only, while the rest stand by ready to take over.
- You want one leader for the same reason a database wants one writer — a single decision-maker means a single order, with no two machines making conflicting changes at once.
- The election itself is a majority vote: candidates ask the others to vote, and a candidate that collects more than half the votes wins. This is the mechanism Raft and ZooKeeper already give you — this page treats it as a building block, not something to rebuild.
- A leader doesn't hold the job forever. It holds a lease — leadership good for a bounded time — and must renew it before it expires. Renew in time and you stay leader; miss the renewal and you must step down, and a standby takes over.
- The danger is split-brain: two machines each believing they lead. A majority vote stops two from being elected at once, and the lease stops a cut-off old leader from ruling forever — but a leader that freezes can wake up still thinking it leads, so the work it guards still needs a fencing token (see Distributed Locks).
Run one copy of a service and it's a single point of failure: when it dies, the job it was doing stops until someone restarts it. So you run three copies. Now the job is safe from any one machine dying — but a new problem appears. Some of what the service does must happen on exactly one machine at a time. Think of Kubernetes: the controller that watches your deployments and scales them up and down. You run three copies of that controller for safety, but only one may actually act. If all three scale the same deployment at once, they fight — one adds two pods, another removes them, a third adds them back. The work is not safe to do three times in parallel.
The naive fix is to pick one machine ahead of time and call it the leader forever. That just moves the single point of failure: when the chosen one dies, nothing promotes a replacement, and you're back to a stopped job and a human paged at 3am. What you actually need is for the machines to choose a leader among themselves, notice on their own when that leader dies, and choose a new one — with no human in the loop. That self-run choosing is leader election.
Why insist on a single leader at all, rather than letting everyone act carefully? For the same reason a single-leader database has one machine that accepts writes: one decision-maker gives you one order of events. The moment two machines both act, you have to reconcile who did what first, and there is often no correct way to merge two conflicting decisions after the fact. Funneling the work through one leader turns 'whose change wins?' into 'whatever the leader decided' — the hard coordination problem collapses into a simple one. That is the whole payoff, and it is why the rest of the page is about protecting the 'exactly one' part.
Build it: a vote to choose, a lease to keep#
Choosing the leader is the part most people picture, and it's the part you should reach for a library to do, not build yourself. The machines hold an election. A machine that thinks there's no leader becomes a candidate and asks every other machine to vote for it. Each machine votes for at most one candidate, and a candidate that collects a majority — more than half the machines — wins and becomes the leader. That majority is called a quorum. You can watch this exact vote play out in the Raft simulator (/raft/sim), and watch a majority form in the quorum simulator (/quorum/sim).
Winning the vote is the easy half. Here's the half that makes leader election hard, and it's the reason this topic is worth its own page. A machine wins an election at some instant — and then time keeps passing. A moment later it might have been cut off from the others by a network fault, or frozen by its own runtime, and a new leader elected in its place, all without it noticing. So a leader cannot just win once and assume it rules forever. It has to keep checking that it is still the leader. The question 'am I the leader?' has an expiry date on it.
The answer is a lease: leadership granted for a bounded stretch of time, not permanently. When a machine wins, it doesn't get 'leader' forever — it gets 'leader for the next 15 seconds'. To stay leader, it must renew the lease before those seconds run out, by sending a small 'I'm still here and still working' message. Renew in time, and the clock resets and you keep the job. Fail to renew — because you crashed, or got cut off, or froze — and the lease expires on its own, and you are no longer the leader. A lease is how leadership cleans up after a dead holder without anyone having to notice the death and intervene.
Now the two sides fit together. The standbys don't poll the leader directly; they watch the shared lease. Walk one failover through, slowly, because the timing is the whole point:
- The three controllers hold an election. Controller A wins the majority and takes a lease on the 'leader' record, good for 15 seconds. A is now the active leader; B and C stand by.
- A does its work and renews the lease every couple of seconds, well inside the 15-second window. Each renewal resets the countdown. B and C watch the lease and see it stay fresh, so they stay put.
- A's machine dies. The renewals stop. The lease is no longer being reset.
- B and C keep watching. Once the lease has gone its full duration with no renewal, they conclude A is gone. The lease is now free to be taken.
- B campaigns, wins the vote, and takes a fresh lease — B is the new leader. It starts renewing, and C stands by watching B's lease. The job continues, no human involved.
That's the whole pattern: a majority vote decides who leads, and a renewable lease decides how long they get to keep believing it. It's worth naming the connection out loud — a leader election is really a distributed lock whose protected resource is the role of leader itself. The same lease you'd use to let one machine hold a lock is here used to let one machine hold the title. Everything the Distributed Locks page says about leases applies, and the interesting differences are all about timing, which the next section takes on.
The timing trade: fail over fast, or don't cry wolf#
The lease duration looks like a small config value. It's actually the single most important decision in the whole design, because it sets two things you can't both have. A machine can't tell the difference between a leader that has died and a leader that is merely slow — a brief network blip or a garbage-collection pause (the runtime freezing a program to clean up memory) looks exactly like death from the outside. All the standbys can see is: the renewals stopped. The lease duration decides how long they wait before calling it.
Pull the lease short and failover gets fast: a dead leader is replaced within a couple of seconds, so the job barely pauses. Push it long and failover gets slow: the job sits idle while everyone waits out the lease. So shorter is better — until you look at the other cost. Every routine blip that lasts longer than the lease now looks like a death, and the standbys fail over needlessly, yanking the job away from a leader that was perfectly healthy and just paused for a moment. That's a false positive — crying wolf — and it causes real churn: the old leader wakes to find it's been deposed, work thrashes between machines, and nothing gets done smoothly. So the two costs pull opposite ways: short lease = fast failover but frequent false alarms; long lease = calm and stable but slow to recover from a real death. There is no setting that is good at both.
Put numbers on it with the Kubernetes defaults. The lease is 15 seconds. That's the worst-case failover time: if the leader dies right after its last renewal, a standby waits roughly 15 seconds before taking over. Fifteen seconds of a stopped controller is usually fine — Kubernetes isn't serving user requests on that path — so they picked stability over speed. Now suppose you wanted sub-second failover and cut the lease to 1 second. Look at what else that changes. A garbage-collection pause of even a second or two is completely routine on a busy node, and it now outlasts the lease. So a healthy leader gets deposed every time its runtime hiccups. You'd trade a rare 15-second gap for constant, pointless churn. That's why the lease is long: it's sized to comfortably swallow the ordinary pauses, so a failover means something really failed.
PredictYour service uses a 10-second leadership lease and fails over cleanly when a leader crashes. But every few hours the whole job freezes and thrashes between machines for a minute, even though no machine actually died. What's likely happening, and does shortening the lease help?
Hint: A pause that's longer than the lease looks identical to a death from the outside. Which direction should the lease move, and what protects the work during the overlap?
Something is pausing the leader for longer than 10 seconds without killing it — most commonly a long garbage-collection pause, but a slow disk, an overloaded CPU, or a network blip does the same. From the standbys' side the renewals simply stop, so they can't tell this from a death: after 10 seconds they declare the leader gone and fail over. Then the paused leader wakes up, still thinks it leads, and re-campaigns — and the job thrashes between the two. This is a false positive, not a real failure. Shortening the lease makes it strictly worse: a shorter lease is crossed by even smaller pauses, so you'd cry wolf more often, not less. The right move is the other direction — lengthen the lease past your worst realistic pause so an ordinary freeze no longer looks like death — and, separately, make sure the work the leader does is fenced (a fencing token) or idempotent, so a brief two-leader overlap can't corrupt anything. Sizing the lease to swallow routine pauses is the fix; shrinking it chases the wrong failure.
Split-brain: when two machines both think they lead
The failure the whole design exists to prevent is split-brain: two machines each convinced they are the one true leader, both acting, both making conflicting changes. It's the exact thing a single leader was supposed to rule out. Two defenses stop it, and it's worth seeing what each one covers — and the gap they don't.
The first defense is the majority vote, and it stops two leaders from being elected at the same time. Winning requires more than half the machines, and each machine votes once, so two candidates can't both collect a majority of the same group — the two majorities would have to share a machine, and that machine only voted once. This is why a network partition (the cluster splitting into groups that can't reach each other) can't produce two leaders: at most one side has a majority, so the minority side simply can't elect anyone. This overlap argument is Raft's and quorum's; you can watch a partition play out in the Raft simulator (/raft/sim).
The second defense is the lease, and it handles the leader that was legitimately elected but has since been cut off. That old leader can't reach a majority to renew, so its lease expires, and it's required to step down on its own the moment it notices it hasn't renewed in time. Meanwhile the majority side, seeing no renewals, elects a fresh leader. The lease guarantees the old leader gives up the title instead of clinging to it forever.
Pitfalls & gotchas
If leader election already picks one leader, why do I still need fencing tokens?
Because 'one leader' is only true when everyone can measure time. A leader that freezes past its lease can wake up and act while a new leader is already running — a brief two-leader overlap that the vote and the lease can't prevent, since the frozen machine never got to step down. Leader election gives you 'one leader almost always'; a fencing token (or making the work idempotent) is what makes the rare overlap harmless. Treat election as availability — keeping a leader around — and fencing as safety — keeping the work correct.
Why not just make the lease very short so failover is instant?
Because a short lease is crossed by smaller and smaller pauses, and every crossing looks like a death. A one-second lease means a routine one-second garbage-collection pause deposes a healthy leader, so the job thrashes constantly. The lease has to be longer than your worst realistic pause, which puts a floor on how fast failover can be. Fast failover and few false alarms are in direct tension; the lease length is where you pick your point on that trade.
Can't the standbys just ping the leader to check it's alive, instead of watching a lease?
A direct health check has the same blind spot: a leader that's frozen or partitioned won't answer, and one that's slow will answer late — you still can't tell 'dead' from 'slow', and you've added a second thing that can fail. Worse, if two standbys reach different conclusions you can get two would-be leaders. Routing the decision through one shared lease (backed by a majority) gives everyone the same answer at the same time. The lease isn't just a timer; it's the single source of truth for who leads.
Does electing a leader hurt my availability — isn't it a single point of failure again?
It reintroduces a brief unavailability, not a permanent one. While a leader is up, one machine is a bottleneck for the leader-only work; when it dies, there's a failover gap (the lease duration) where no one is doing that work. That's the price of 'exactly one'. It's not a single point of failure like a hardcoded leader, because a replacement is elected automatically — but it does mean the leader-only path is at most as available as your failover is fast, which is another reason the timing trade matters.
QuizA team runs 3 replicas with a 15-second leadership lease. Their leader-only job is 'move money between accounts', and it must never run twice. They've tuned the lease and failover works. Is the leader election enough to keep the money safe?
- Yes — one leader at a time means the transfer only ever runs once.
- No — a paused leader can overlap with a new one, so the transfer itself needs a fencing token or must be idempotent.
- Yes, as long as the lease is longer than any network delay.
- No — they need to shorten the lease so the old leader is deposed faster.
Show answer
No — a paused leader can overlap with a new one, so the transfer itself needs a fencing token or must be idempotent. — Leader election gives you 'one leader almost always', not 'one leader with certainty'. A leader frozen by a garbage-collection pause can wake up after its lease expired — while a new leader is already running — and do one more transfer before it notices, so two machines briefly act. For money, that brief overlap is a double-spend. The vote and the lease can't close this window, because the frozen machine never got to step down. Safety has to be enforced at the resource: a fencing token the account store checks to reject the stale leader's late write, or making the transfer idempotent so running it twice is harmless. Shortening the lease makes false failovers more frequent, not the money safer; a lease longer than network delay still can't cover an arbitrary pause.
When to elect a leader — and when not to
A leader is a coordination tool with a real cost: a bottleneck while it's up, a failover gap when it dies, and a dependency on the vote-and-lease machinery. Before reaching for one, it's worth asking whether you need a single leader at all.
- If the work is idempotent — safe to run more than once with the same result — you may not need a leader. Let every replica do it; running twice just converges to the same answer. This is often cheaper and more available than electing anyone.
- If the work is naturally partitioned — each replica owns a distinct slice of the data — you don't need one global leader, you need one leader per slice, which spreads the load instead of funneling it. This is the per-shard-leader model behind systems like CockroachDB.
- If you genuinely need exactly-one across the fleet (one scheduler, one compactor, one writer to a shared resource), elect a leader — but size the lease for your real pause profile and budget for fencing on anything that must stay correct.
- If the leader-only work is just an optimization — dedup effort, avoid two machines doing the same harmless task — a plain lease with no fencing is fine, and an occasional overlap costs you nothing but wasted work.
Three ways to run the election
The vote-and-lease pattern is one shape, but there's a real choice in what holds the lease and runs the vote. The table lines up the three common approaches on what matters: what decides the winner, and how failover happens.
| Approach | How a leader is chosen | How failover happens |
|---|---|---|
| Lease on a consensus store (etcd, Consul) | Contend for one key tied to a lease; the winner keeps it alive with a keep-alive heartbeat | Lease expires when renewals stop; a waiting candidate campaigns and takes it |
| ZooKeeper ephemeral sequential nodes | Each client creates a numbered node; the lowest number leads | The leader's node vanishes when its session drops; the next-lowest is notified and takes over |
| Full consensus protocol (Raft built in) | A majority vote per term, with a check that only a fully-caught-up node can win | Followers time out on missing heartbeats and elect a new leader — sub-second |
They're less different than they look: all three come down to a lease or session that must be kept alive, and a majority underneath deciding the winner. The consensus store and ZooKeeper are the same idea with a friendlier front door — you get leader election as a service and don't implement the vote yourself. Rolling your own Raft group is for when the leader and the replicated data are the same system (a database that elects its own write leader), so the election and the data share one protocol.
Under the hood: how the real systems do it
The open path treats the vote and the lease as building blocks. Here's what's actually happening inside the three systems you'd reach for, one deep-dive each.
Go deeperZooKeeper: ephemeral sequential nodes and the herd it avoids
To run an election in ZooKeeper, every candidate creates a node under a shared path, and ZooKeeper stamps each with an ever-increasing number ('sequential') and ties it to the client's session so it vanishes automatically if that client drops ('ephemeral'). The candidate with the lowest number is the leader. That vanishing-on-disconnect is a cleaner death signal than a timeout: the session ending is the lease expiring.
The clever part is how standbys wait. The obvious design has everyone watch the leader's node, so when it disappears all of them wake and re-campaign at once — a 'thundering herd' of redundant work. Instead, each candidate watches only the node just below its own number. When the leader leaves, only the single next-in-line is notified, promotes itself, and everyone else keeps waiting on their own predecessor. One notification instead of a stampede. That increasing number also doubles as a ready-made fencing token.
Go deeperetcd: campaign, lease, and keep-alive
etcd's election API makes the lease explicit. A candidate first creates a lease with a time-to-live, then calls Campaign, which writes a key under the election's name attached to that lease. etcd stamps every write with a revision — an ever-increasing sequence number — and Campaign blocks until this candidate holds the lowest revision under that name, at which point it is the leader. To stay leader, the client sends periodic keep-alive messages that reset the lease's countdown. Stop sending them — crash, freeze, partition — and the lease expires, etcd deletes the key, and leadership passes to the next campaigner in line.
The nice property is that the revision is monotonic, so the revision on your leader key is a fencing token you get for free — read it when you win, pass it to whatever you're protecting, and the resource can reject any write carrying an older revision. This is the same monotonic-number idea the Distributed Locks page derives, handed to you by the store.
Go deeperRaft and Chubby: the election timeout, and the master lease
Raft's failover is fast because it doesn't wait out a long lease — followers each keep a short randomized election timeout, typically 150 to 300 milliseconds. Miss that many milliseconds of heartbeats and a follower starts an election. The randomization matters: if every follower used the same timeout they'd all campaign together and split the vote, so each rolls a different value and the first to fire usually wins outright before the others wake. This is the sub-second-failover end of the trade — Raft can afford it because heartbeats are cheap and frequent.
Google's Chubby, the lock service that inspired most of these, ran leadership on a master lease: the elected master holds leadership for a lease interval and renews it, and crucially the replicas promise not to elect a new master until the old lease is sure to have expired. That mutual promise is what stops two masters overlapping in the common case — the same lease logic as everywhere on this page, named and shipped in 2006.
In the wild
- Kubernetes runs its control plane on lease-based leader election: the controller-manager and scheduler each run several replicas that contend for a single Lease object, one active and the rest hot standbys, with the 15s / 10s / 2s dials tuned for stability over speed because a short controller gap is harmless.
- etcd and Consul offer leader election as a service — a lease plus keep-alive plus a campaign call — so applications get 'exactly one worker' without implementing a vote. Kubernetes' own Lease is built on etcd underneath.
- ZooKeeper is the classic election coordinator via ephemeral sequential nodes; HBase, Kafka (before it moved to its own Raft-based KRaft), and countless systems used it to elect an active master and fail over to a standby.
- HDFS elects an active NameNode with a standby via ZooKeeper's ZKFC (failover controller) — a textbook case where two active NameNodes writing metadata would be catastrophic, so the election is paired with fencing to guarantee the old one is truly stopped before the new one starts.
- Databases that elect their own write leader — CockroachDB, TiKV — run a Raft group per data shard, so leadership and the replicated log are the same protocol, and 'who leads' is decided thousands of times over, once per shard, rather than one leader for the whole cluster.
In an interview
Leader election is a favorite because the naive answer — 'just pick one and have the others watch it' — sounds complete and misses every hard part. The interviewer is usually probing whether you know that detecting a dead leader is guesswork, and that 'one leader' isn't a guarantee. Lead with why one leader, then the lease, then the trap.
- Say why a single leader at all: one decision-maker means one order, no conflicting writes to reconcile — the same reason a database has one writer. That framing shows you know what the leader buys you.
- Describe the mechanism in two moves: a majority vote picks the leader, and a renewable lease keeps it — renew the heartbeat in time to stay leader, miss it and step down while a standby takes over. Note you'd use Raft, ZooKeeper, or etcd rather than build the vote.
- Name the timing trade before you're asked: you can't tell a dead leader from a slow one, so the lease length trades failover speed against false failovers. A short lease fails over fast but cries wolf on every GC pause; a long lease is stable but slow to recover. Cite real dials (Kubernetes 15s) if you can.
- Name the split-brain defenses and the gap: a majority vote stops two elections at once, a lease evicts a cut-off leader — but a frozen leader can wake and overlap, so correctness-critical work needs a fencing token, not just the election. This is the point that separates a strong answer.
- Close with the escape hatch: sometimes the best leader election is none — make the work idempotent, or partition it so each shard has its own leader, and you avoid the bottleneck and the failover gap entirely.
PredictAn interviewer says: 'You've got leader election working with a lease, and your leader-only job is sending a daily billing email to each customer. A leader occasionally pauses and you get a brief second leader. Do you need fencing tokens here?' What's the strong answer?
Hint: Leader election gives 'almost always one leader'. What must be true of the billing effect so a brief overlap can't double-bill — and can a fencing token even be checked by an email gateway?
Start by tying the answer to cost, not mechanism. A billing email sent twice is a correctness problem — the customer is notified twice, or double-billed if the email triggers a charge — so the brief overlap is not harmless, and yes, you must defend against it. Then make the sharper point: a fencing token only works if the resource can check it, and an email gateway you don't control can't inspect one. So the defense here isn't a token on the send; it's making the operation idempotent — attach a stable key like (customer + billing date) and dedup on it, so a second send for the same key is dropped. The general rule to state: leader election promises almost-always-one, never exactly-one, so any correctness-critical effect must be made safe against a double-run at the effect itself — a fencing token where the resource can enforce order, an idempotency key where it can't. Naming that boundary between the election and the work it guards is the strong answer.
References
- In Search of an Understandable Consensus Algorithm (Raft) — Ongaro & Ousterhout, 2014 — The election mechanics this page borrows: terms, RequestVote, majority, and the randomized election timeout.
- ZooKeeper Recipes — Leader Election — Ephemeral sequential nodes, the lowest-number leader, and watching your predecessor to avoid the herd.
- etcd — Concurrency (Election) API reference — Campaign, lease, keep-alive, and the revision that doubles as a fencing token.
- Kubernetes client-go — leaderelection package — The LeaseDuration / RenewDeadline / RetryPeriod dials and the Lease-object election used across the control plane.
- The Chubby lock service — Burrows, Google, 2006 — Master leases and the replicas' promise not to re-elect until the old lease expires.
- How to do distributed locking — Martin Kleppmann, 2016 — Why a lease alone can't stop a paused leader, and why fencing tokens are the fix.