Start here: memory is fast, but a crash erases it#
TL;DRthe 30-second version
- A database must be durable: once it acknowledges a write, that write has to survive a crash. But it keeps its working data in volatile memory (RAM), which is wiped the moment the process dies — so acknowledging a write that lives only in memory is a lie waiting to be exposed by a power cut.
- Writing every change straight to its final place in the on-disk data file instead is durable but slow: those writes land at scattered locations (random I/O), and a crash partway through updating a page can leave it half-written and corrupt.
- The write-ahead log resolves the tension. Every change is first appended as a record to a log file on disk — a fast, sequential write — and only after that record is safely on disk is the change applied to memory and acknowledged. Log before apply.
- After a crash, recovery replays the log to redo every logged change, rebuilding the exact state the database had acknowledged. A periodic checkpoint folds memory into the data file and truncates the log, so it can't grow forever and recovery stays short.
Durability is one of the core promises a database makes — the D in ACID (atomicity, consistency, isolation, durability). It means that once the database has told a client "committed," that data will still be there after a crash, a restart, or a power failure. Nothing acknowledged is ever silently lost. The problem is that the place a database does its fast work — memory — makes exactly the opposite promise.
Memory (RAM) is volatile: it holds data only while the machine is powered. Pull the plug and every value in memory vanishes. That's fine for scratch work, but it means a write that exists only in memory is not durable — if the process crashes right after you acknowledged it, it's gone, and you've broken your promise. Disk, by contrast, is durable: an SSD or hard drive keeps its contents across a crash. So durability has to end up on disk somehow.
The obvious fix — write each change straight to its home in the on-disk data file before acknowledging — is safe but painfully slow, for two reasons. First, records live at scattered positions in the data file, so updating them means the disk jumps around: random I/O, the slowest thing a disk does. Second, and worse, updating data in place is not crash-safe. Disks write in fixed-size blocks called pages, and if the machine dies halfway through overwriting a page, that page is left half-old, half-new — a torn write. Now the data file itself is corrupt, and you've traded losing one write for potentially losing the whole file.
The fix: write to a log before you touch anything#
The write-ahead log is an append-only file on disk — you only ever add to the end, never go back and overwrite. When a change comes in, the database does three things in a strict order: (1) append a record describing the change to the end of the log; (2) make that record durable on disk; (3) only then apply the change to the in-memory state and acknowledge it to the client. The name says the rule: the log is written ahead of the change it describes.
- A log record is a compact description of one change — enough to redo it — for example "set key user:1 to alice". Records are framed with their length so recovery can read them back one at a time.
- Making it durable means calling fsync: an operating-system request that tells the disk "don't just buffer this, actually persist it," and doesn't return until the bytes are physically on the drive. Until fsync returns, the write isn't truly safe; after it returns, it survives a crash.
- Applying the change means updating the in-memory table that queries actually read from. This is the fast, mutable working copy — and the part a crash wipes.
Why is logging first any faster than writing the data file directly? Because appending to a log is sequential I/O — every write lands at the very end of one file, so the disk head (or the SSD's controller) never has to seek to scattered locations. Sequential writes are dramatically faster than the random writes an in-place update needs, and many changes can be flushed with a single fsync. You still pay one durable write per change, but you pay it in the cheapest possible form.
Here's the payoff. Suppose the crash happens at the worst possible moment: after the log record is durable but before the change reaches the in-memory table. The in-memory change was never made, or was made and then wiped. But the log record is on disk. So on restart, the database reads the log and redoes the change: this is replay. Because the record was durable before the client was ever told "committed," replay can always reconstruct every acknowledged write. The in-memory state is disposable; the log is the source of truth for anything not yet safely in the data file.
Checkpoints: keeping the log from growing forever#
If every change appends a record and nothing is ever removed, the log grows without bound — and recovery, which replays the log, would take longer and longer. Something has to let old records be thrown away. That something is a checkpoint.
A checkpoint flushes the current in-memory state out to the durable data file — the permanent home of the data, laid out for efficient lookup. Once that's done, every log record describing a change already reflected in the data file is redundant: recovery would never need it, because the data file already has that change. So the log can be truncated up to the checkpoint. From then on, recovery only has to replay records written after the last checkpoint.
- Write the current in-memory state to the on-disk data file (its durable home).
- Record that a checkpoint happened at this point in the log.
- Truncate the log — discard every record up to the checkpoint, since the data file now holds those changes.
- Recovery after a future crash = reload the data file, then replay only the log records since this checkpoint.
Checkpoint frequency is a tuning knob. Checkpoint often and the log stays tiny and recovery is fast — but you pay for flushing the data file more often. Checkpoint rarely and you do less background I/O — but the log grows and a crash means a longer replay. Every real WAL implementation exposes some version of this dial. The key property either way: the log is bounded, so recovery time is bounded.
Walkthrough: a crash, and coming back#
Follow one sequence end to end. Write user:1=alice and user:2=bob; each is logged, then applied. Checkpoint — both land in the data file and the log is truncated. Now write user:3=carol and overwrite user:1=alice2; these two are logged but not yet checkpointed. Then the machine loses power.
The two writes made after the checkpoint were never in the data file — memory held them, and memory is gone. They come back only because the log recorded them. Turn the log off and repeat the same crash, and those two writes are lost for good: recovery can restore the checkpoint and nothing more. That contrast — zero loss with the log, real loss without it — is the entire value of a write-ahead log, and it's the thing to watch in the simulator.
What it costs: writing twice#
A WAL isn't free. Every change is now written to disk twice: once as a log record, and later again when a checkpoint writes the in-memory state into the data file. That doubling is called write amplification, and here it's roughly 2× the logical data volume. The design bets that two cheap writes beat one expensive one — and it usually wins, because both writes are the good kind: the log write is sequential, and the data-file write is deferred and batched, so many changes to the same page collapse into one physical write at checkpoint time instead of one per change.
The other cost is the fsync on the write path. Forcing bytes to disk is slow relative to memory, and doing one fsync per write would cap throughput hard. The standard escape is group commit: the database gathers the log records of many concurrent transactions and flushes them to disk with a single fsync, so the fixed cost of durability is shared across a whole batch. One durable write, many transactions made safe.
| Approach | Speed | Crash safety |
|---|---|---|
| Write to memory only | Fastest | None — a crash loses everything not in the data file |
| Update the data file in place per write | Slow (random I/O) | Poor — a torn write can corrupt a page |
| Write-ahead log + checkpoint | Fast (sequential log, batched fsync) | Full — replay restores every acknowledged write |
The durability knob#
"Log before apply" tells you the order, but not when the client is told "committed" relative to the fsync. That timing is a knob, and it trades latency against how much you can lose in a crash.
| Setting | When it acknowledges | Crash loses | Cost |
|---|---|---|---|
| Synchronous commit | After the log fsync completes | Nothing acknowledged | Higher write latency |
| Group commit | After a shared batch fsync | Nothing acknowledged | Tiny added latency, much higher throughput |
| Asynchronous commit | Before the fsync, trusting it will land | The last few milliseconds of "committed" writes | Lowest latency, a small loss window |
Synchronous commit is the strict reading of durability: don't say "committed" until the record is truly on disk. Asynchronous commit relaxes it deliberately — acknowledge first and let the fsync happen a moment later — which is much faster but opens a small window where a crash can lose writes the client thinks are safe. That can be a fine trade for data you can regenerate, and a terrible one for money. The point is that it's a conscious choice, exposed by the same WAL machinery; the log is what makes even the strict setting fast enough to be the default.
Where the log shows up in other systems#
Once you see the pattern — record the intent durably, then apply it — you see it everywhere in storage, under different names.
| System | Its write-ahead log | What it protects |
|---|---|---|
| LSM tree | A WAL guarding the in-memory memtable | Buffered writes not yet flushed to a sorted file on disk |
| PostgreSQL | The WAL segments in pg_wal | Every change, and the basis for replication |
| MySQL / InnoDB | The redo log | Committed changes not yet written to the table pages |
| Redis | The append-only file (AOF) | Commands, replayed on restart to rebuild memory |
| Bitcask | The data file is itself an append-only log | Every write — the log is the database |
The LSM (log-structured merge) tree is the clearest relative: its whole design is to buffer writes in an in-memory memtable and flush them to disk in batches — which would be catastrophic on a crash, except that a write-ahead log sits in front of the memtable doing exactly what this topic describes. Bitcask goes further and erases the distinction entirely: its append-only data file is the log, and an in-memory index points into it. The WAL isn't a niche feature; it's the shared foundation these engines are built on.
In the wild
- PostgreSQL writes all changes to the WAL first; the same log stream is shipped to replicas for replication and used for point-in-time recovery, so the durability log doubles as the change feed.
- SQLite's WAL mode (a switch from its older rollback-journal design) lets readers keep reading the data file while a writer appends to the WAL, improving concurrency — a direct, everyday use of this exact mechanism.
- MySQL's InnoDB engine calls its WAL the redo log and pairs it with checkpoints that run without pausing writes ("fuzzy checkpointing") to bound recovery time — the classic textbook implementation.
- Redis, though an in-memory store, offers the append-only file (AOF): it logs every write command and replays the file on startup to rebuild memory, with a fsync policy (always / every second / OS-decides) that is precisely the durability knob.
- Kafka takes the idea to its logical end: the durable, append-only log isn't a supporting file, it's the primary abstraction the whole system exposes to producers and consumers.
- Raft and other consensus systems keep a replicated log and apply entries only once they're durably logged on a majority — the same log-before-apply rule, extended across machines.
Pitfalls & gotchas
Isn't writing to the log just as slow as writing to the data file? It's still a disk write.
It's a disk write, but the cheap kind. The log is append-only, so every write lands sequentially at the end of one file — no seeking to scattered locations, which is what makes data-file updates slow. Sequential writes are far faster than random ones on both spinning disks and SSDs, and group commit lets many transactions share a single fsync. Same number of durable writes, a fraction of the cost.
Does a WAL guarantee zero data loss, always?
Only if you acknowledge writes after the log fsync completes (synchronous commit). With asynchronous commit, the database acknowledges first and flushes a moment later, so a crash in that window can lose writes the client thought were durable. The WAL makes zero-loss achievable and cheap; whether you actually get it depends on the commit setting you choose.
What if the crash happens while the log record itself is only half-written?
That partial record is at the very end of the log and was never acknowledged to any client — the client never got "committed" because the fsync never returned. Recovery detects the torn tail (records are length-framed, often with a checksum) and discards it. You lose only a write that was never promised, which is allowed. The already-durable records before it are untouched.
How is this different from a journaling filesystem?
It's the same idea at a different layer. A journaling filesystem write-ahead-logs its own metadata (and optionally data) so the filesystem structure survives a crash. A database WAL logs the database's changes so the database survives a crash. A database generally can't rely on the filesystem's journal for its own durability, so it keeps its own log — sometimes on top of a journaling filesystem, logging at both layers.
QuizA database appends a write to its log, calls fsync, and after fsync returns it acknowledges the client — but it crashes before applying the change to its in-memory table. What happens to that write after recovery?
- It's lost — it never reached the in-memory table or the data file.
- It's recovered — replay reads the durable log record and re-applies the change.
- It's undefined — the database can't know whether it was acknowledged.
- It's rolled back — un-applied changes are discarded on recovery.
Show answer
It's recovered — replay reads the durable log record and re-applies the change. — This is the exact case the write-ahead log exists for. The log record was made durable (fsync returned) before the client was acknowledged, so it's on disk even though the in-memory apply never happened. On recovery the database replays the log and re-applies the change, reconstructing the acknowledged state. That the change never touched memory or the data file before the crash doesn't matter — the durable log is the source of truth, and replay is how it's used.
In an interview
The strong version of this answer is a single sentence you can defend: durability is achieved by ordering, not by where the data lands — log the change durably before applying it, and a crash can never lose an acknowledged write. Everything else hangs off that.
- State the rule precisely: append the change to an append-only log and fsync it before applying it to memory and acknowledging. "Log before apply."
- Explain why it's fast, not just safe: the log is sequential I/O (append to the end of one file), versus the random I/O of updating scattered pages in the data file — and group commit shares one fsync across many transactions.
- Explain checkpoints: they flush memory to the data file so old log records can be truncated, which bounds both log size and recovery time. Without them the log grows forever.
- Name the durability knob: synchronous commit (fsync before ack, zero loss) vs asynchronous commit (ack first, a small crash-loss window). Know that it's a deliberate latency-vs-loss trade.
- Connect it: an LSM tree's memtable is protected by a WAL; Postgres's WAL doubles as its replication stream; Redis's AOF and InnoDB's redo log are the same mechanism. It's a foundation, not a feature.
PredictAn interviewer asks: "Your service writes to an in-memory store for speed, but occasionally loses recent data on restart. How would you keep the speed and stop the loss?" What's the strong answer?
Hint: What has to be true on disk before you acknowledge a write? And how do you keep that disk write cheap?
Add a write-ahead log. Before applying each write to the in-memory store, append it to an append-only log on disk and fsync it, then apply and acknowledge — so any acknowledged write is durable before the client hears "ok," and a restart replays the log to rebuild memory. Keep the speed by making the log sequential (append-only, one file) and using group commit to batch fsyncs, and add periodic checkpoints that flush memory to a data file and truncate the log so recovery stays fast. If some latency is unacceptable, expose the durability knob — asynchronous commit trades a small crash-loss window for lower latency. The key framing: you don't have to choose between the fast in-memory path and durability; the WAL is exactly the design that gives both, by changing the order of writes rather than abandoning memory.
References
- PostgreSQL — Write-Ahead Logging (WAL) — The canonical, readable explanation of WAL, checkpoints, and commit settings.
- SQLite — Write-Ahead Logging — How SQLite's WAL mode works and why it improves concurrency over the rollback journal.
- MySQL — InnoDB redo log — InnoDB's WAL, checkpointing, and how it bounds recovery.
- Redis — Persistence (AOF) — The append-only file and its fsync policies — the durability knob in practice.
- ARIES (Mohan et al., 1992) — The foundational paper on write-ahead logging, checkpoints, and crash recovery.
- Designing Data-Intensive Applications, Ch. 3 — Kleppmann's treatment of logs, LSM engines, and durability.
Ready to try it?
The simulator is a real, deterministic implementation — pick a scenario and step through it, scrubbing the timeline forward and backward through every change.