DEV Community

gxqwb
gxqwb

Posted on

Raft vs Multi-Paxos: What You Pay for Understandability

Consensus protocols all do the same thing: get a majority of machines to agree on an ordered sequence of values while some of them are failing. The interesting difference is not what they guarantee but what they force the implementer to think about. Multi-Paxos leaves the leader election, the log structure and the membership change as design decisions. Raft fixes all three, adds a strong leader and a log-matching property, and in exchange gives up a little theoretical generality.

Raft and multi-Paxos both commit a write in one round trip once a stable leader exists; the difference is that Raft spends one extra round trip on election and a lot of specification on state-machine safety, which is why it ships in more systems.

Part One: What is being compared

Paxos, in its classic single-decree form, gets one value chosen. Multi-Paxos runs it repeatedly, one instance per log slot, and the obvious optimisation is to elect a distinguished proposer once so that subsequent slots skip the prepare phase. That leader is not part of the original protocol; it is a performance layer that every real implementation adds and every real implementation has to get right on its own.

Raft writes the leader into the protocol. A candidate increments its term, requests votes, and wins with a majority. The winner appends entries and replicates them; a follower accepts an entry only if its log matches the leader's at the preceding index and term. The election restriction — a candidate may not win unless its log is at least as up to date as a majority — is what makes the log-matching property hold, and it is stated as an invariant rather than left as an exercise.

Both tolerate the same failures: a minority of crashes, message loss, reordering and duplication, and network partitions. Both need a majority to make progress. Both are typically deployed with three or five replicas, because two failures out of five is the point where the cost of another copy starts to look reasonable.

Part Two: The arithmetic

The steady-state cost is the same. A leader receives a write, appends it locally, sends it to followers, waits for a majority to acknowledge, then commits and replies. On a local network that is one round trip, so with 0.5 milliseconds between racks the commit latency is about half a millisecond plus disk time. Group commit amortises the fsync across many writes: a single fsync at 1 millisecond can cover a hundred entries, so throughput is limited by network and batching rather than by the disk.

The election cost differs. Raft uses randomised timeouts, typically 150 to 300 milliseconds, so a failed leader is replaced within roughly one timeout plus one vote round trip. Multi-Paxos with an elected leader has the same shape, but the election mechanism is implementation-defined, and many systems use an external lock service or a lease, which adds a dependency and its own failure modes.

The interesting number is the cost of a leader change under load. During the election window the cluster accepts no writes. A 300-millisecond window at 50,000 writes per second is 15,000 writes queued or rejected, and the clients that retry them add load precisely when the system is least able to absorb it. Both protocols pay this; the question is how predictable the window is and how quickly the new leader's log is brought up to date.

The theoretical difference that gets cited most is that multi-Paxos can commit with fewer round trips in some pipelined configurations. In practice, both systems batch and pipeline, and the measured difference at the same replication factor is in the low single-digit percent. The protocol is not where the latency goes.

Part Three: Where the model breaks

Raft's simplicity is real but narrower than the papers suggest. The core protocol fits in a few hundred lines; the production system around it — snapshots, log compaction, membership changes, leader leases for linearizable reads, client session tracking — is tens of thousands. Joint consensus for membership change is subtle enough that the original design was replaced in several implementations with a single-server-at-a-time approach.

Multi-Paxos's flexibility is also its hazard. Because the leader election is not specified, two implementations can both be "multi-Paxos" and disagree about what happens when a leader is partitioned and a new one is elected. The classic failure — a leader that believes it is still authoritative and serves stale reads — is a configuration bug rather than a protocol bug, which makes it harder to find.

Both protocols assume fail-stop behaviour: a node either works or stops. A node that pauses for a long garbage collection, or whose clock jumps, can violate timing assumptions that leases depend on. Raft's election timer makes a paused node stand for election when it wakes up, which is safe but disruptive; a lease-based system may have already granted reads elsewhere.

Part Four: The later cost

The operational bill for Raft is well understood because so many systems implement it: etcd, Consul, CockroachDB, TiKV, RabbitMQ's quorum queues. Tooling, documentation and hiring all exist. The failure modes are documented, and a team can reason about them without a distributed-systems specialist.

The bill for multi-Paxos is concentrated in a few very large systems — Google's Chubby and Spanner are the canonical examples — where the protocol is embedded in a stack with TrueTime or a lock service that handles the hard parts. Adopting multi-Paxos without that surrounding infrastructure means reimplementing the parts the paper leaves out, and that is where most of the risk sits.

There is a third cost that rarely appears in comparisons: the cost of a wrong default. Raft implementations usually default to a safe configuration — fsync on, majority reads, no stale reads. A custom multi-Paxos deployment may default to whatever the first author found fast, and the difference surfaces during an incident.

Part Five: Outside the team

For a user, consensus is invisible until it is not. The visible symptoms are a checkout that hangs for a second during a leader change, or a configuration change that appears to apply and then reverts. The protocol determines how long that window is and whether the system fails closed or open.

For the business, the choice determines who can operate the system. A Raft-based store can be run by a general infrastructure team; a bespoke multi-Paxos implementation usually needs the people who wrote it, which is a hiring and bus-factor decision disguised as a technical one.

A fair comparison fixes the replication factor, the network round-trip time, the write size and the failure injection schedule, then measures commit latency, election recovery time and the number of writes lost or duplicated. Comparing a three-node Raft cluster on local SSD against a five-node Paxos cluster across regions measures the topology, not the protocol.

Market sizing check

If distributed databases and coordination services continue to absorb the workloads currently on single-node systems, the segment built directly on a consensus protocol is plausibly $15B to $30B a year by 2030. The swing factor is how much of that growth goes to managed services where the protocol is hidden behind an API.

These figures synthesise published database and infrastructure market ranges under a stated adoption model.

The comparison that matters

The comparison that matters is between a protocol that specifies the hard parts and one that leaves them to the implementer. Raft wins on implementability, on the availability of reference implementations and on the ability of an ordinary team to operate it correctly. Multi-Paxos wins on flexibility and on the handful of systems where a tightly integrated stack can exploit it.

The ranking assumes the team is building a general-purpose replicated log. If the workload needs a custom quorum scheme or a leaderless path, Paxos-derived designs leave room that Raft does not.

Choose Raft unless you have a specific reason and a specialist: its extra specification is not overhead, it is the part of the design that keeps a sleepy node from corrupting the log.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
If this saved you time, a tip is appreciated.

Top comments (0)