A runtime for partitioned stateful services

https://zrz.io/zbp/5-partitioned-stateful-services/v1.md

TL;DR

This specifies how to run a service whose state for a given key is held in memory by exactly one instance at a time — and how to move that ownership without dropping the state on the floor.

The shape of the answer: the key space is divided into a fixed number of slots; a versioned partition map published by a control plane assigns slot ranges to partitions and partitions to nodes; ownership is a lease carrying a monotonically increasing epoch that fences every side effect; state is recovered from a checkpoint, then a journal, then catch-up on the input stream; and a rollout, a rebalance and a scale event are all the same operation — a handoff — differing only in what moves where.

The controlling decision is that the partition map is data, not a function. A consistent-hash ring computes an assignment so that no coordination is needed. Here coordination exists and is wanted, so the map is published instead of computed — which is what makes it possible to pin one customer to dedicated hardware, split a hot range at a chosen boundary, run different node sizes for different parts of the key space, and drain a node on a schedule. None of those is expressible as a hash function.

The second decision is that scaling is a plan a tool applies, not a reaction an autoscaler has. Terminating a stateful node is not a capacity adjustment; it is a handoff that happens to end with one fewer node. An autoscaling policy cannot know which node is drainable, so it must not be the component that chooses.

Applicability

Use this when a service holds per-key state in memory — a counter, a rate-limit window, a session, a model, a materialised aggregate — that must be authoritative in one place at a time, and where rebuilding it from scratch on every deploy is too slow, too expensive, or too visible.

Do not use this when:

• The service is stateless. A load balancer and an autoscaling group solve this problem for free. Every requirement here is overhead you are paying for nothing.
• There is no natural partition key. This document specifies how to move ownership of a partition between nodes. It does not specify how to replicate a single logical state machine that cannot be divided; that is a consensus problem, and you want a consensus library.
• The state is the system of record. Everything here assumes state is recoverable by replaying a durable log. If losing the in-memory state means losing customer data, you need a database, and this runtime in front of it.
• The workload is batch. If work can be scheduled, shuffled and retried without a live owner, a scheduler is simpler and cheaper.

Scope and non-goals

In scope: the slot key space and the partition map; ownership leases, epochs and fencing; checkpoint and journal formats and the recovery algorithm; catch-up from an ordered input stream; standby replication and promotion; handoff, rollout, split and merge; the client routing contract and its redirect semantics; the placement controller, node classes and placement constraints; scheduled scale-down and scale-up; and the observability that makes the whole thing debuggable.

Out of scope: the application’s own data model and applier logic; consensus for a non-partitionable singleton; transactions spanning partitions; cross-partition queries and joins; the durable system of record behind the runtime; and authentication and authorisation of the callers, which are assumed to be solved above this layer.

Vocabulary

• Key — the application-level string that identifies a unit of state. Frequently a tenant id.
• Slot — the atom of the key space. A key belongs to exactly one slot, computed by hash, for the life of the system. Slots are numbered 0 .. 2^SLOT_BITS - 1.
• Partition — the unit of ownership, state, recovery and movement: a set of slot ranges, the state belonging to their keys, a journal and a checkpoint lineage.
• Generation — a partition’s lineage counter, incremented when it is created by a split or a merge. Identifies which stored state belongs to which shape of the partition.
• Epoch — a partition’s ownership counter, incremented on every change of primary. The fencing token. Never decreases, never reused.
• Node — a process that can host partitions. Has a class, a capacity budget and a heartbeat.
• Class — the shape of node a partition requires: size, local storage, and whether the node may be shared. A class marked exclusive hosts exactly one partition.
• Primary — the node currently authoritative for a partition. Exactly one, per epoch.
• Standby — a node holding a replica of a partition’s state, applying behind the primary, eligible for promotion. Serves no writes.
• Partition map — the versioned document assigning slot ranges to partitions and partitions to nodes. Published by the placement controller, cached by everyone, authoritative.
• Map version — a monotonically increasing integer identifying one published map. Never reused.
• Placement controller — the control plane. Decides the map; does not sit on the serving path.
• Lease store — the strongly consistent store holding ownership records. Separate from the placement controller, and reachable by nodes without it.
• Journal — the ordered, durable, per-partition log of state changes, addressed by LSN.
• LSN — the journal’s sequence number. Dense, gapless, per partition and generation.
• Checkpoint — an immutable snapshot of a partition’s state at a stated LSN and set of input positions.
• Input stream — the external ordered, replayable log the partition folds into state.
• Input position — the per-shard cursor into the input stream from which replay resumes.
• Derived state — state that is a deterministic fold over the input stream alone.
• Originated state — state created by requests made directly to this service, recoverable only from its own journal.
• Handoff — moving primary ownership of a partition from one node to another.
• Seal — the point in a handoff after which the outgoing primary can never accept another write.
• Lock shard — an intra-process striping of a map across several mutexes to keep a hot path off a single lock. It has nothing to do with a partition, and is named here only because the same codebase will call both of them shards.

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as in RFC 2119 and RFC 8174.

Architecture

Four properties do the work, and all four are structural rather than behavioural.

The map is published, not computed. A consistent-hash ring exists so that independent parties can agree on an assignment without talking to each other. That is the right trade when there is no control plane. Here there is one, and it is wanted — so the assignment becomes a document the controller writes and everyone reads. Everything an operator wants from this system is a property of a document and impossible in a function: pinning a slot to dedicated hardware, choosing where a hot range splits, running one class of node for most of the key space and a larger class for part of it, draining a named node before a maintenance window.

The slot is the atom, and it is a hash prefix. Taking the top bits of the hash rather than a modulus makes the key space refinable in place: doubling the number of slots splits every slot in two and moves no key across a partition boundary. A modulus would move nearly all of them. It also means a partition’s keys occupy one contiguous range of the hash space, which is what lets the input stream’s shards line up with partitions for free.

The epoch fences the storage, not the lock. A lease tells a node it may serve. It cannot tell a node it has stopped serving, because the node may be paused, partitioned, or garbage-collecting at the moment it stops being true. So safety does not come from the lease; it comes from every durable side effect carrying the epoch it was produced under, and from the store rejecting a lower one.

The control plane is not on the serving path. Nodes talk to the lease store directly, so the placement controller can be down, redeployed, or broken for an hour without a request failing. What stops is change: no rebalance, no rollout, no split. That is the correct thing to lose.

How it works

Everything below is explanation, not specification. It carries no normative keywords and defines no requirement; where it appears to disagree with Normative requirements, the numbered requirement is right and this is loose. It exists because the requirement list that follows is a wall, and a reader who meets the wall without the model will build the letter of it and miss the shape.

Read it once, in order. Each part assumes the one before it.

The key space, and why a slot is a hash prefix

A key — usually a tenant id — is not assigned to a node. It is assigned to a slot, and slots are assigned to partitions, and partitions to nodes. Two indirections, and both earn their keep.

The slot comes from the top bits of a 128-bit hash of the key (ZBP-5-R1). With SLOT_BITS = 12 there are 4096 slots and the slot is, quite literally, the first three hex digits of the digest:

t_whale     md5 = 55b0c21b24e83694bdafcb69a8da3391
                  ^^^
                  0x55b = 1371     → slot 1371

The choice of prefix rather than modulus is the single decision the rest of the key-space design rests on, and it is invisible until you need it. Consider raising the slot count from 4096 to 8192.

With a prefix, slot 1371 becomes slots 2742 and 2743, and every key that was in 1371 is now in one of those two — because taking one more bit off the front of a number cannot move it outside the range its first twelve bits already described. A range [0, 1370] becomes [0, 2741] mechanically. No key changes partition, no state moves, and the whole operation is a rewrite of range bounds in a document (ZBP-5-R3).

With md5 % 4096, raising the count to 8192 rehashes the entire key space. Nearly every key lands somewhere new, which means nearly every key’s state has to move. One is a config change; the other is a migration. They differ by which end of the number you took.

The second consequence is that a slot is a contiguous interval of hash space, and so is a partition, being a set of slot ranges. This is what makes the input stream line up for free — a stream that places records by hashing the same key into the same 128-bit space puts a partition’s records in a computable, contiguous set of shards, with nothing to configure. It is also why there is no key-level pin (ZBP-5-R4): a partition that is not a hash range breaks stream alignment, split, and merge simultaneously, and it does so quietly. A very large tenant gets its own slot, not its own key exception.

Why the map is published rather than computed

A consistent-hash ring answers “who owns this key?” with arithmetic. That is exactly right when there is no coordinator — every party computes the same answer without talking to anyone.

Here there is a coordinator, and it is wanted. So the assignment is a document the controller writes and everyone else reads (ZBP-5-R5). The document can express things arithmetic cannot:

• this one slot lives alone on a large node, because that tenant is worth more than the disruption;
• this hot range splits here, at a boundary someone chose after looking at the traffic;
• most of the key space runs on small nodes and this part runs on big ones;
• this named node is draining before a maintenance window and holds nothing by 02:00.

Every one of those is a property of a document. None is expressible as a hash function, and the usual attempt — weighted virtual nodes — is a worse map with no reader.

The cost of publishing is that the document is now a trust boundary and a blast radius. It is signed by the controller and verified before it is applied, from any source (ZBP-5-R10, ZBP-5-R11), which is what makes it safe for a peer to hand a client a delta in a redirect. And because a map that is perfectly well-formed can still name the wrong endpoints — coverage validation proves shape, not correctness — publication is rate-limited by a movement cap and staged through a canary (ZBP-5-R119, ZBP-5-R120). It is the highest-blast-radius write in the system: one document, the whole fleet, immediately.

A partition, and the three numbers it carries

A partition is the unit of ownership, state, recovery and movement. It carries three numbers, and confusing them is a common early bug because all three look like versions.

Generation is lineage. A split produces children at a higher generation whose stored state lives under their own prefix, and whose recovery walks back to the parent’s. Epoch is the fencing token, and increases on every ownership change including a split (ZBP-5-R16). LSN is position within one (partition, generation), dense and gapless, and it restarts for a child — which is exactly why a position token is not comparable across a lineage boundary.

Ownership, and the thing a lease cannot tell you

Ownership is a lease: a record naming an owner, with an expiry, renewed on a timer.

A lease is very good at telling a node it may serve. It is incapable of telling a node it has stopped serving, and this is not an implementation quality issue — it is structural. The moment a node stops being the owner is precisely the moment it is most likely to be unable to find out: paused by a long garbage collection, partitioned from the network, descheduled, or simply slow. Any design where safety depends on the old owner noticing in time is a design that has assumed away its own hardest case.

So safety does not come from the lease. It comes from two things underneath it.

The node fences itself, locally, on the monotonic clock. When a renewal is acknowledged, the node records monotonic_now() + ttl − margin and refuses to admit work past that instant (ZBP-5-R17, ZBP-5-R18). This needs no network, which is the point: it still works during the pause that caused the problem. It uses the monotonic clock because wall time jumps, slews and runs backwards, and a lease decision made on wall time is a bug waiting for an NTP correction.

The storage rejects the stale writer. Every durable write carries the epoch it was produced under, and the store refuses anything below the record’s current epoch (ZBP-5-R20). This is the part that actually holds. A zombie primary that missed both its renewal and its own self-check is still harmless, because the thing it is trying to corrupt is what turns it away:

n_7a2c  FENCE   at epoch 73   →  ACCEPT              (still the owner)
n_1f90  ACQUIRE at epoch 74   →  ACCEPT              (ownership moves)
n_7a2c  FENCE   at epoch 73   →  REJECT_STALE_EPOCH  (harmless, and it need not know)
n_7a2c  ACQUIRE at epoch 74   →  REJECT_STALE_EPOCH  (cannot climb back)

Object storage plays the same game differently, because it has no conditional writes worth relying on. Objects are immutable and named by partition, generation, epoch and LSN (ZBP-5-R21), so a zombie’s write does not overwrite anything — it lands at a key nothing references and is eventually collected. What makes a checkpoint current is the pointer in the lease store, and moving that pointer is the epoch-fenced operation (ZBP-5-R22). A checkpoint becomes real when the pointer moves, never when the bytes land.

Two kinds of state, and why the difference decides recovery

State comes in two flavours, and a partition declares which it has (ZBP-5-R29).

Derived state is a deterministic fold over the input stream. A counter, a rolling window, an aggregate. Given the stream and a starting point, it can always be rebuilt, because the stream is the truth and the state is just a cached opinion about it. Such a partition needs no journal at all (ZBP-5-R30): a checkpoint records the input positions it is consistent with, and recovery is restore plus replay from those positions.

Originated state is created by requests made to this service. Nothing upstream knows it exists, so nothing upstream can replay it. It has to be written to a journal before it is applied, and that journal is the only record of it (ZBP-5-R31).

Most real services have both, which is where the trap is. If the two sets are genuinely disjoint and neither applier reads the other, they can recover independently. The moment an originated write reads derived state — a write whose effect depends on the current counter — they are one state machine, and recovering them on two independent paths produces two different answers. That case has to be fully journalled (ZBP-5-R32), and the honest version of this rule is: if you have to think about whether they interact, they interact.

Underneath both sits the requirement that makes replay mean anything: the applier is deterministic (ZBP-5-R33). No wall-clock reads, no random values, no iteration over an unordered map where order changes the result, no calls out to another service. Anything that cannot be recomputed is captured into the record when it is written and read back on replay (ZBP-5-R34), never recomputed.

Get this wrong and there is no error, ever. The service is correct until the day it recovers, and then it is quietly, plausibly different — a counter off by a little, a window that starts somewhere else, a decision that would not have been made. That is why replay equivalence is a continuous test rather than a design note.

Recovery, in three layers

Three layers, because each covers a window the next cannot. The checkpoint is cheap to load and always behind. The journal covers checkpoint-to-seal. The stream covers everything after, up to live.

The trap sits between layers two and three, and it is the most commonly implemented bug in this whole document. A checkpoint records the input positions it is consistent with (ZBP-5-R36). The journal then advances past it, and every journal record carries the positions consumed at that point. After replaying the journal to its tail, the place to resume the stream is the tail’s marks, not the checkpoint’s. Take the checkpoint’s, and every input record between the checkpoint and the journal tail is applied a second time. Nothing errors. The counters are simply too high, by an amount proportional to how long it had been since the last checkpoint — which is to say, worst exactly when the incident was worst.

Two details make the layers work. The checkpoint is ordered by slot (ZBP-5-R38), so a child of a split can fetch only the parts overlapping its own ranges instead of loading the parent whole. And a class declares whether its checkpoints are full or a base plus a bounded delta chain (ZBP-5-R39, ZBP-5-R40) — incremental makes frequent checkpointing affordable for a large partition, at the price of a second lineage to walk before anything is deleted.

The stream side has its own two. Positions are tracked per contributing shard, never as one cursor (ZBP-5-R53), because a stream reshard leaves a partition reading several. And a parent shard is drained to its end before either child is read (ZBP-5-R54) — read a child early and records inside a single key arrive out of order, which is the one ordering guarantee everything else assumes.

Finally, the quiet failure that no other signal catches: a checkpoint older than the stream’s retention means the partition cannot be recovered at all, and nothing about a healthy running service reveals it. It is found at the exact moment it must not be found. Hence an alarm on the margin between checkpoint age and configured retention (ZBP-5-R47), and a restore exercised continuously against real checkpoints rather than tested once.

Standbys, and what a standby actually buys

A standby holds a replica and is eligible for promotion. In a journalled partition it consumes the primary’s journal; in a derived one it needs no link to the primary at all and simply reads the same input stream (ZBP-5-R63).

What a standby buys depends entirely on the commit tier the class declares, and the tiers are exact (ZBP-5-R67):

• LOCAL — acknowledged when applied in memory on the primary. Fastest. A crash loses the tail.
• STANDBY — acknowledged when the required standbys have applied it. Survives one node dying.
• DURABLE — acknowledged when the record is in a store that survives losing the whole fleet.

There is deliberately no default (ZBP-5-R68). The three differ in what a caller loses, and a default decides that for a workload nobody examined. Below DURABLE there is a window — acknowledged, not yet in a durable segment — exposed to correlated loss, and that window is monitored rather than left as an accident of the batching interval.

The related trap is that “a standby exists” reads as durability and often is not. A standby with unbounded lag is a second copy of an old state, which is why lag is a metric rather than an inference from liveness. And a standby in its primary’s own failure domain satisfies every count-based rule while surviving nothing at all — so diversity is declared per class and enforced by placement (ZBP-5-R61), with “none required” being a thing a class says out loud rather than a thing it does by accident.

Handoff, and why the seal is the last step

Handoff is how ownership moves, and rollout, rebalance and scale-down are all just handoff with a different reason attached. It has exactly one point of no return — the seal (ZBP-5-R74).

Before the seal, abandoning costs nothing: the outgoing primary never stopped serving and the map was never changed (ZBP-5-R75). After it, the partition can only go forward — either the incoming node finishes, or some node recovers the partition from storage under a new epoch. The outgoing primary does not resume, ever, including when the handoff fails (ZBP-5-R76).

That asymmetry is the whole design. Everything expensive — restore, replay, catch-up, which is seconds to minutes — happens on the safe side of the line (ZBP-5-R78). The seal itself is one conditional write. The partition is unavailable only from the seal to the moment the new map names its primary (ZBP-5-R79), and that window is the number the design exists to make small, so it is measured on every handoff rather than estimated once.

Two things follow that are easy to get wrong. A rollout is a bounded sequence of these, one partition at a time, never a fleet restart and never delegated to a platform rolling deployment that picks its own order and starts replacements before anything drains (ZBP-5-R80). And the graceful path is the optimisation, not the guarantee — a node has to survive being killed with no warning (ZBP-5-R87), which is why cold recovery is the path that gets exercised most. Relatedly, the shutdown phase deadlines are summed and checked against the platform’s kill budget at boot (ZBP-5-R85): if they overrun, the last phase never runs, and the last phase is the one that accounts for what was lost — so the loss becomes not merely unrecovered but unrecorded, and the shutdown looks clean.

Split and merge as a lineage walk

A split divides a partition’s slot ranges at slot boundaries; a merge combines adjacent ones. Both produce children at a higher generation with new ids, and both work the same way: children restore from their parents’ state filtered to their own ranges, every parent seals, then the children start (ZBP-5-R92, ZBP-5-R93).

The property that makes this possible at all is narrow and worth stating plainly: ordering is guaranteed per key, never across keys (ZBP-5-R94). If cross-key ordering were promised, no partition could ever be divided, because dividing it means two independent appliers with no shared sequence. Everything in this document that looks like flexibility is bought with that one restriction.

The floor is one slot (ZBP-5-R96). A key hotter than a single node cannot be relieved by splitting, because a key is one point in the hash space — it is one slot, in one partition, on one node, and on the input side it is one point in the stream’s hash space and therefore one shard’s worth of ingest. That ceiling is a key-design problem, answered by composing the key with a sub-key and explicitly giving up ordering between sub-keys. It is not a placement problem, which is why people look for it in the placement layer and do not find it.

Routing, and the five things a server can say

The client computes the slot itself and resolves it against a cached map (ZBP-5-R97). There is no router and no per-request lookup, because a lookup service on the hot path is the coupling the whole control-plane separation exists to avoid.

Requests carry the key, the slot, and the map version the client resolved against — and that last field is what makes the answers unambiguous:

The last two rows are the interesting ones. Collapse them into one “you’re wrong, try elsewhere” and two nodes with disagreeing maps will redirect a client back and forth indefinitely, at full speed, with no errors logged anywhere — the failure looks like a latency problem. The rule that prevents it is that a server never issues a redirect derived from a map older than the client’s (ZBP-5-R102).

MISROUTED deserves its own code because it is not transient. It means client and server do not compute the same slot from the same key — a different hash, a different SLOT_BITS, a different string encoding. Retrying cannot fix a disagreement about arithmetic, so it is alarmed rather than retried (ZBP-5-R105).

Finally, every mutating response carries the position it was applied at (ZBP-5-R107), so a caller that needs to read its own write can demand that position on the read rather than guessing at replication lag.

Placement, and what the controller is not

The controller decides the map. That is all it does. It is never called on the serving path, by clients or by nodes (ZBP-5-R109), and the data plane keeps serving from the last published map while it is down (ZBP-5-R110). A controller outage freezes change — no rebalance, no rollout, no split — and changes nothing about serving. That is the correct thing to lose, and it is only true because the lease store is a separate component that nodes reach directly.

Placement is a constrained optimisation, and the order of the objectives matters more than the optimisation does (ZBP-5-R115):

1. satisfy constraints — class, exclusivity, failure domains. Not negotiable.
2. minimise movement — every move is a handoff, and a handoff is an unavailability window.
3. balance load — last.

Put balance above movement and the controller thrashes: whenever load is near-even, tiny differences justify moves, each move disturbs load, and it never settles. A controller that optimises balance continuously serves worse than one that does nothing.

Decisions come out as plans — renderable, dry-runnable, applied as a separate step, with policy deciding which kinds apply automatically and which wait for a human. This is deliberately the shape of an infrastructure tool rather than a control loop, because the interesting operations here are rare, consequential, and worth looking at.

Scaling on a clock, not on a signal

Autoscaling a stateful fleet does not work, and not because the arithmetic is hard. A scale-in policy picks a victim by CPU, and CPU is uncorrelated with drainability — it will choose the node holding the largest partition about as often as any other. Termination here is not a capacity adjustment; it is a handoff that happens to end with one fewer node, so the component that knows about handoffs has to be the component that chooses (ZBP-5-R126).

What replaces it is scheduled, and shaped around the fact that state takes time to move:

Evening, into a known trough. Merge under-utilised partitions, rewrite checkpoints to drop expired state, retire the drained nodes. Compaction is abandonable at any point — an interrupted one leaves the previous checkpoint live and nothing else changed.

Morning, ahead of a known ramp. Start at T_ramp − lead, where lead comes from measured restore and catch-up times, not from a guess (ZBP-5-R128). Add nodes, split partitions, and let the children warm up while the parents are still serving. The map flips last, when every child is synced (ZBP-5-R124) — and if warming is not finished by T_ramp, it does not flip at all, because serving a ramp from a warm parent beats serving it from a cold child.

That ordering — warm everything, then flip one document — is what turns a scale event from a thirty second outage into a few milliseconds.

The life of one write

Pulling it together, for a journalled partition at the STANDBY commit tier:

1. The client hashes the key, takes the top twelve bits, looks up slot 1789 in its cached map, and sends the request to p_0002’s primary with map_version: 1487.
2. The node recomputes the slot and agrees. It checks it holds the lease at its believed epoch and that its monotonic deadline has not passed. It checks the partition is SERVING and inside its capacity budget.
3. The applier produces a record. It is appended to the journal at LSN 918408 — sequence from the journal, never from a clock — carrying the epoch and the input positions consumed.
4. The record streams to the standby. The standby applies it and acknowledges.
5. Only now is the write acknowledged to the client, with X-Partition-Position: p_0002:74:918408.
6. In the background, journal records batch into a segment in object storage; periodically a checkpoint is written and the pointer flipped under the epoch.
7. Some hours later the partition is handed to a new node during a rollout. The new node loads the checkpoint, replays the journal to the seal, resumes the stream from the tail’s marks, and reaches the same state — because the applier is deterministic and everything that was not was captured into the records at step 3.

Step 7 is the one that has to be true, and every requirement in this document is in service of it.

Normative requirements

The key space and the partition map

• ZBP-5-R1 The key space MUST be divided into a fixed number of slots, 2^SLOT_BITS. A key’s slot MUST be the most significant SLOT_BITS bits of a 128-bit hash of the key’s UTF-8 encoding, interpreted big-endian.
• ZBP-5-R2 The hash function MUST be fixed for the life of the system and named in the map. It MUST NOT be a hash whose output varies between processes, runs, or library versions — a seeded or runtime-randomised hash is disqualified regardless of its statistical quality.
• ZBP-5-R3 SLOT_BITS MAY be increased and MUST NOT be decreased. An increase is a pure refinement: slot s at b bits becomes the range [s << d, (s << d) | (2^d − 1)] at b + d bits, and no key changes partition. Implementations MUST perform the increase by rewriting slot ranges in the map, and MUST NOT move state to do it.
• ZBP-5-R4 The slot MUST be the atom of assignment. A partition’s key set MUST be expressible entirely as a set of slot ranges. There MUST NOT be a key-level pin, override, or exception, because a partition that is not a hash range breaks stream alignment, split, and merge at once.
• ZBP-5-R5 The partition map MUST be a single versioned document that assigns every slot in 0 .. 2^SLOT_BITS − 1 to exactly one partition. Total coverage and non-overlap MUST be validated before a map is published, and a map failing validation MUST NOT be published.
• ZBP-5-R6 The map version MUST be a monotonically increasing integer, MUST NOT be reused, and MUST change whenever any assignment, endpoint, epoch, or partition state in the document changes.
• ZBP-5-R7 Every partition entry MUST carry its id, generation, epoch, class, slot ranges, primary endpoint and state, and each standby’s endpoint and state. Slot ranges MUST be inclusive [low, high] pairs, sorted and non-overlapping.
• ZBP-5-R8 The map MUST be readable by nodes and by clients without calling the placement controller.
• ZBP-5-R9 Every published map version MUST remain addressable for at least the longest client cache lifetime, so that a client presenting an old version can be answered with a delta rather than a full document.
• ZBP-5-R10 The placement controller MUST sign every published map version with a key held only by the controller. The signature MUST cover the map version, every slot assignment, and every endpoint in the document, and a delta MUST be signed as well as a full map.
• ZBP-5-R11 Clients and nodes MUST verify that signature before applying a map or a delta, from whatever source it arrived. A delta relayed by a peer is acceptable precisely because authenticity rests on the signature rather than on the transport or on trusting the peer that sent it.
• ZBP-5-R12 A map or delta that fails verification MUST be rejected, counted, and alarmed as a correctness fault, and MUST NOT be applied even when its version is newer than the cached one. The signing key MUST be rotatable with overlapping validity, and verifiers MUST accept any key currently within its validity window.
• ZBP-5-R13 Partition ids MUST be opaque and MUST NOT encode the slot range they currently hold. The range changes; the id must not.

Ownership, leases, and fencing

• ZBP-5-R14 Primary ownership MUST be a lease with a stated expiry, a named owner, and a renewal interval. An ownership record without an expiry MUST NOT exist.

• ZBP-5-R15 The lease record MUST carry the partition’s epoch, and acquisition MUST increment that epoch in the same conditional write that claims the lease.

• ZBP-5-R16 An epoch MUST increase monotonically per partition, MUST NOT be reused, and MUST NOT reset when a partition is created by a split or a merge.

• ZBP-5-R17 A node MUST determine that its own lease has expired from a monotonic clock reading taken when its last renewal was acknowledged. It MUST NOT compare local wall time to the expiry recorded in the store.

• ZBP-5-R18 A node MUST stop accepting work for a partition before that lease can have expired at the store, allowing a margin of at least the maximum renewal round-trip plus the maximum tolerated clock error, and MUST be able to do so without contacting any other component.

• ZBP-5-R19 Every write to shared storage on behalf of a partition MUST carry the epoch under which it was produced.

• ZBP-5-R20 The lease store MUST reject any conditional write carrying an epoch lower than the record’s current epoch. This rejection is the fence: safety comes from the resource refusing a stale token, not from the lock being held correctly.

• ZBP-5-R21 Objects in the object store MUST be immutable and uniquely named by at least partition, generation, epoch, and LSN, so that a write by a stale primary lands at an unreferenced key rather than overwriting a live one.

• ZBP-5-R22 The pointer naming a partition’s current checkpoint MUST live in the lease store and MUST be advanced by a conditional write fenced on the epoch. A checkpoint becomes current when the pointer moves, never when the object lands.

• ZBP-5-R23 A node MUST NOT be recorded as ready for a partition it has not finished restoring. Holding the lease and being ready to serve MUST be distinct states.

• ZBP-5-R24 Losing the lease MUST cause the node to discard the partition’s in-memory state without attempting to flush, checkpoint, or replicate it. A stale primary’s state is not a contribution.

• ZBP-5-R25 The lease store MUST NOT be the placement controller, and MUST remain reachable by nodes when the placement controller is unavailable.

• ZBP-5-R26 An operator force-takeover MUST exist that acquires a partition at a higher epoch without waiting for the current lease to expire. Waiting out a TTL is not an acceptable only-option when a node is known to be gone.

• ZBP-5-R27 A force-takeover MUST advance the epoch through the same conditional write as an ordinary acquisition. There MUST NOT be any path that takes ownership without advancing the epoch, because the epoch is the only thing that makes the displaced node harmless.

• ZBP-5-R28 A force-takeover MUST record the operator, the time, and a stated reason, and MUST NOT be reachable by automated placement. It is a human override of a safety property, and an automated caller that can invoke it has removed the property.

State, checkpoints, and the journal

• ZBP-5-R29 Each partition MUST declare a recovery mode of DERIVED, JOURNALED, or MIXED, and the mode MUST be recorded in the map.
• ZBP-5-R30 In DERIVED mode all state MUST be a deterministic fold over the input stream, there MUST be no journal, and recovery MUST be a checkpoint restore followed by input replay from the positions the checkpoint records.
• ZBP-5-R31 In JOURNALED mode every state change — originated and input-derived alike — MUST be appended to the journal before it is applied, and the input positions consumed MUST be recorded on the journal record that consumed them. There MUST be exactly one write path into state.
• ZBP-5-R32 MIXED mode MUST be used only where the derived and originated state sets are disjoint and neither applier reads the other’s state. Where they interact, JOURNALED MUST be used.
• ZBP-5-R33 The applier MUST be deterministic given a state and a record. It MUST NOT read wall time, generate random values, depend on the iteration order of an unordered collection, or call an external service.
• ZBP-5-R34 Any value that cannot be recomputed deterministically MUST be captured into the journal record when the record is written and read back on replay. It MUST NOT be recomputed during recovery.
• ZBP-5-R35 The LSN MUST be dense and gapless within a (partition, generation), and MUST be assigned by the journal rather than derived from any clock.
• ZBP-5-R36 A checkpoint MUST record the LSN and the complete set of input positions it is consistent with, its format version, its byte length, and a content digest.
• ZBP-5-R37 A checkpoint MUST be verified against its digest on restore, and a mismatch MUST be treated as the checkpoint not existing rather than as an error to surface and stop on.
• ZBP-5-R38 The checkpoint format MUST be ordered by slot, or otherwise permit extracting the state for a slot range without materialising the whole checkpoint. Split cannot be implemented without this property.
• ZBP-5-R39 A class MUST declare its checkpoint kind as full or incremental, and the kind MUST be recorded in every checkpoint manifest. It MUST NOT be inferred from the presence or absence of deltas.
• ZBP-5-R40 An incremental checkpoint MUST consist of one full base plus an ordered chain of deltas, each naming the base and the delta immediately preceding it. Restore MUST apply the base and then every delta in order, and MUST verify the digest of each; a mismatch anywhere in the chain MUST be treated as the entire chain being absent.
• ZBP-5-R41 A delta chain MUST have a stated maximum length and MUST be compacted into a new base before that bound is reached, so that restore time has a ceiling rather than growing until somebody notices. The bound MUST be derived from the shutdown budget of ZBP-5-R85.
• ZBP-5-R42 A base MUST NOT be deleted while any delta depending on it is still reachable by a live partition’s restore path, under the same lineage rule as ordinary checkpoints. An incremental chain adds a second lineage, and both MUST be walked before anything is deleted.
• ZBP-5-R43 Checkpoint and journal formats MUST be versioned, and a node MUST refuse to restore a format version it does not fully understand rather than interpreting it partially.
• ZBP-5-R44 Adjacent releases MUST be able to read each other’s checkpoint and journal formats. A format change MUST be shipped as expand, migrate, contract across at least three releases, because a rollout and a rollback both run two versions at once.
• ZBP-5-R45 Journal segments and checkpoints MUST NOT be deleted on age alone. Deletion MUST be driven by the map’s lineage records and MUST retain everything reachable by the recovery path of any live partition, including a child that has not yet taken a checkpoint of its own.
• ZBP-5-R46 Deletion MUST actually run, on a schedule, and the volume of retained-but-unreachable objects MUST be monitored. A rule that forbids deleting the wrong thing is not a retention policy; without a collector that runs, storage grows without bound and the prohibition reads as one.
• ZBP-5-R47 The age of the newest usable checkpoint MUST be monitored against the input stream’s retention window, and an alarm MUST fire while the remaining margin is still large enough to act on. A checkpoint older than retention means the partition cannot be recovered at all, and nothing else in the system reports that.
• ZBP-5-R48 Restoring a checkpoint into a running process MUST be exercised continuously against real production checkpoints, not only in tests.

The input stream and catch-up

• ZBP-5-R49 A partition MUST publish the earliest instant its retained state and journal can answer about. A query whose window reaches before that instant MUST be reported as incomplete rather than answered with a number that silently omits the missing history.
• ZBP-5-R50 The input stream MUST be ordered per key and replayable from a stated position. Ordering across keys MUST NOT be relied upon anywhere in the system.
• ZBP-5-R51 The stream’s shard boundaries MUST be ranges over the same hash space as the slots, and producers MUST use the service key as the stream’s partition key, so that a partition’s records occupy a contiguous and computable set of shards.
• ZBP-5-R52 Where the stream’s partition key cannot be the service key, the producer MUST set an explicit hash key equal to the low hash of the record’s slot. This is an escape hatch for a pre-existing producer, not the default arrangement.
• ZBP-5-R53 A partition MUST track an input position per contributing shard. A single scalar cursor MUST NOT be used, because a partition reads from several shards after a stream reshard.
• ZBP-5-R54 During catch-up a reader MUST drain a parent shard to its end before reading either child shard. Reading a child while its parent still has records reorders records within a key.
• ZBP-5-R55 Every input record MUST carry a stable id, and ingest MUST deduplicate on it. During recovery the same record arrives on more than one feed — the live path, the stream tail, and an archive seed — and a record that cannot be deduplicated double-counts the moment a second feed delivers it.
• ZBP-5-R56 The deduplication horizon MUST be exactly the retention horizon, so that a duplicate is recognised for precisely as long as the original is retained. A separate, shorter horizon is a second number to tune and a second way for the two to disagree.
• ZBP-5-R57 A partition MUST NOT admit traffic until catch-up has reached a stated freshness bound, expressed both as position lag and as record age. The bound MUST be configurable per class and MUST be reported, so that “ready” names something an operator chose.
• ZBP-5-R58 Records for slots the partition does not own MUST be discarded on read and counted. They MUST NOT be applied, and MUST NOT be treated as an error.
• ZBP-5-R59 A record that causes the applier to fail MUST NOT be retried indefinitely. After a bounded number of attempts across a bounded number of nodes, the partition MUST enter a quarantined state and cease to be placed. A poison record whose only limit is the restart loop will crash every node the partition is assigned to, in turn, until the fleet is gone.

Standbys and replication

• ZBP-5-R60 A class MUST be able to require a minimum number of synced standbys, and a partition MUST NOT be reported healthy while below it.
• ZBP-5-R61 A class MUST declare the failure-domain diversity required between a primary and each synced standby, and a class that requires none MUST say so explicitly. A standby placed in its primary’s failure domain satisfies every count-based requirement while surviving nothing, and the only difference between that and a deliberate choice is whether it was written down.
• ZBP-5-R62 A partition whose standby placement does not meet its class’s declared diversity MUST NOT be reported healthy, by the same rule as one below its standby count.
• ZBP-5-R63 In JOURNALED mode a standby MUST consume the primary’s journal. In DERIVED mode a standby MUST read the input stream independently, and no replication link is required.
• ZBP-5-R64 A standby MUST report its applied position, and the lag of every standby MUST be exposed as a metric rather than inferred from its liveness.
• ZBP-5-R65 A standby MUST NOT accept writes. It MUST NOT serve reads unless the request explicitly opts in to a stated staleness bound, and every such read MUST return the position at which it was served.
• ZBP-5-R66 The commit policy — whether a write is acknowledged on local apply, on standby acknowledgement, or on durable persistence — MUST be an explicit per-class setting.
• ZBP-5-R67 The commit tiers MUST be defined exactly as: LOCAL, acknowledged once applied in memory on the primary; STANDBY, acknowledged once at least the class’s required number of standbys have applied it; and DURABLE, acknowledged once the record is held by a store that survives the loss of the entire node fleet. A tier that is not one of these three MUST NOT be offered.
• ZBP-5-R68 A class MUST declare its commit tier and MUST NOT start without one. There is deliberately no default: the three tiers differ in what a caller loses, and a default would decide that for a workload nobody examined.
• ZBP-5-R69 The tier a write actually achieved MUST be reported on the response. A caller that asked for DURABLE and got LOCAL because the class was misconfigured MUST be able to tell.
• ZBP-5-R70 Below DURABLE, the journal tail — records acknowledged but not yet in a durable segment — is exposed to correlated loss of the acknowledging nodes. The count and the age of the oldest such record MUST be monitored, and the exposure MUST be stated in the class rather than left as a property of the batching interval.
• ZBP-5-R71 Where the commit policy requires standby acknowledgement, the behaviour when no standby remains MUST be configured explicitly as either continue-and-alarm or refuse-writes. It MUST NOT be whatever the code happens to do.
• ZBP-5-R72 A standby MUST verify that the replication stream it consumes carries the epoch it believes the primary holds, and MUST stop consuming and re-resolve ownership if it does not.
• ZBP-5-R73 Promotion MUST increment the epoch through the lease store before the promoted node accepts its first write.

Handoff and rollout

• ZBP-5-R74 A handoff MUST have a single point of no return — the seal — after which the outgoing primary MUST NOT accept another write for that partition under any circumstance, including the handoff subsequently failing.
• ZBP-5-R75 Before the seal, an aborted handoff MUST leave the outgoing primary serving and the published map unchanged.
• ZBP-5-R76 After the seal, the partition MUST only move forward: either the incoming primary completes promotion, or some node recovers the partition from storage under a new epoch. The outgoing primary MUST NOT resume.
• ZBP-5-R77 Requests in flight at the outgoing primary that have not committed by the seal MUST be rejected with a retryable outcome carrying the new assignment. They MUST NOT be reported as succeeded, and MUST NOT be failed permanently.
• ZBP-5-R78 The incoming primary MUST have restored and caught up to its class’s freshness bound before the seal. The seal MUST be the last and cheapest step of a handoff, never the first.
• ZBP-5-R79 A map version naming a new primary MUST NOT be published before that node has reported ready under its new epoch.
• ZBP-5-R80 A rollout MUST be expressed as a sequence of handoffs with bounded concurrency. It MUST NOT be expressed as a fleet restart or delegated to a platform-driven rolling deployment that chooses its own order and its own timing.
• ZBP-5-R81 The number of partitions in handoff simultaneously MUST be bounded, both per node and globally.
• ZBP-5-R82 Every handoff MUST carry a deadline. On expiry it MUST abort if the seal has not happened, or complete by recovery from storage if it has. It MUST NOT wait indefinitely.
• ZBP-5-R83 Receiving a termination signal MUST initiate handoff of every partition the node hosts, and the platform’s shutdown grace period MUST exceed the worst-case handoff time for a fully loaded node.
• ZBP-5-R84 Shutdown MUST proceed in order: stop being chosen, end unbounded streams with a stated reason, stop accepting new calls, then settle bounded background work. Run out of order the phases undo each other — a listener stopped before readiness is withdrawn drops work that was already routed here.
• ZBP-5-R85 The sum of every shutdown phase deadline MUST be validated at process start against the platform’s termination budget, and a process whose phases cannot fit MUST refuse to boot. If the phases overrun, the final phase never runs, and the final phase is the one that accounts for what was lost — so the loss becomes not merely unrecovered but unrecorded, and the shutdown looks clean.
• ZBP-5-R86 Work that does not complete within the bounded final phase MUST be written to durable object storage in the same record shape the partition’s durable sink already uses, so that the reader which consumes that sink’s archive consumes the spilled object unchanged. A recovery path that needs its own parser is a recovery path nobody runs.
• ZBP-5-R87 A node MUST survive being killed without notice. Graceful handoff is the optimisation; recovery from checkpoint, journal, and stream is the requirement, and MUST be the path exercised most often in testing.
• ZBP-5-R88 A rollback MUST be a rollout in the opposite direction under the same format-compatibility rule, and MUST NOT require an older binary to read a checkpoint written in a format version it does not understand.

Split, merge, and redistribution

• ZBP-5-R89 A split MUST divide a partition’s slot ranges at slot boundaries into two or more children, each with a new partition id and a generation greater than the parent’s. A child covering zero slots MUST NOT be produced.
• ZBP-5-R90 A merge MUST combine partitions whose slot ranges are adjacent into one child with a new id and a generation greater than every parent’s.
• ZBP-5-R91 A child MUST record its parents in the map, and that lineage MUST be retained for as long as the child depends on any parent’s stored state.
• ZBP-5-R92 A child MUST recover by restoring its parents’ checkpoints filtered to its own slot ranges, replaying its parents’ journals from those checkpoints filtered to its own slot ranges, and then resuming the input stream from the parents’ sealed positions.
• ZBP-5-R93 A split or merge MUST seal every parent before any child accepts a write, and no parent MUST resume after any child has started.
• ZBP-5-R94 Cross-key ordering MUST NOT be relied upon across a split or a merge. Per-key ordering MUST be preserved, and it is the narrowness of that guarantee that makes redistribution possible at all.
• ZBP-5-R95 A child MUST take its own checkpoint before its parents’ stored state becomes eligible for deletion.
• ZBP-5-R96 One slot MUST be the smallest unit that can be split off. A key whose load exceeds a single node MUST be addressed in the application’s key design — by composing the key with a sub-key and explicitly forfeiting ordering between sub-keys — and MUST NOT be treated as a defect in the placement layer.

Routing and the client contract

• ZBP-5-R97 A client MUST compute the slot itself and resolve it against a cached map. It MUST NOT consult a router or the placement controller per request.

• ZBP-5-R98 Every request MUST carry the key, the resolved slot, and the map version the client resolved against.

• ZBP-5-R99 A server receiving a request for a slot it owns and is ready to serve MUST serve it, regardless of the client’s map version.

• ZBP-5-R100 A server receiving a request for a slot it owns but is not ready to serve MUST return NOT_READY with a retry delay, and MUST NOT redirect.

• ZBP-5-R101 A server receiving a request for a slot it does not own MUST distinguish three cases by map version: its own map is newer, and it MUST return MOVED carrying the current assignment; the versions are equal, and it MUST return MISROUTED with no assignment; its own map is older, and it MUST return STALE_MAP with a retry delay.

• ZBP-5-R102 A server MUST NOT issue a redirect derived from a map older than the client’s. This is what prevents two disagreeing nodes redirecting a client to each other indefinitely.

• ZBP-5-R103 A MOVED response MUST carry enough of the map for the client to route the retry, and the client MUST verify its signature per ZBP-5-R11 and apply it before retrying. A response whose delta asserts assignments beyond the requested slot is legitimate only because it is signed; an unsigned one MUST be discarded rather than narrowed.

• ZBP-5-R104 A client MUST bound the number of redirects for one logical request and MUST fail the request rather than follow an unbounded chain.

• ZBP-5-R105 MISROUTED responses MUST be counted and alarmed. They mean client and server disagree about the hash or the slot arithmetic, which is a correctness fault rather than a transient condition, and no amount of retrying will resolve it.

• ZBP-5-R106 Clients MUST refresh a cached map on a jittered interval as well as on redirect, so that a map change does not produce a synchronised refresh from the entire client population.

• ZBP-5-R107 Every mutating response MUST carry the position at which it was applied, as an (epoch, LSN) pair ordered lexicographically, so that a caller needing read-your-writes can require that position on a later read.

• ZBP-5-R108 A position token MUST name the partition it was issued by, and a caller presenting a token from a partition that no longer exists MUST be told so explicitly rather than made to wait for a position that can never arrive. A split or merge resets the LSN under a new partition id, so (epoch, LSN) is comparable only within one partition id and MUST NOT be compared across a lineage boundary.

Placement and the control plane

• ZBP-5-R109 The placement controller MUST NOT be called on the serving path, by clients or by nodes.
• ZBP-5-R110 The data plane MUST continue serving from the last published map while the placement controller is unavailable. A controller outage MUST freeze change, never serving.
• ZBP-5-R111 The placement controller MUST NOT store its own state inside the service it places.
• ZBP-5-R112 Every partition MUST declare a required class and a capacity estimate; every node MUST declare its class and its capacity budget. Placement MUST satisfy class constraints before optimising anything else.
• ZBP-5-R113 A class marked exclusive MUST result in its node hosting exactly one partition, and the controller MUST NOT co-locate another partition there for any reason, including transiently during a rebalance.
• ZBP-5-R114 Placement MUST enforce the declared failure-domain diversity as a hard constraint, ranked with the class constraints and above any balance objective. Where it cannot be satisfied the partition MUST be placed and reported undiversified rather than left unplaced — an unowned partition is worse than an undiversified one, and the health signal is what makes the difference visible.
• ZBP-5-R115 The placement objective MUST be ordered: satisfy constraints, then minimise movement, then balance load. Movement MUST rank above balance, or the controller will thrash whenever load is close to even.
• ZBP-5-R116 The controller MUST bound concurrent moves globally and per node, and MUST NOT start a move for a partition that moved more recently than a stated hysteresis interval.
• ZBP-5-R117 Placement decisions MUST be emitted as an explicit plan that can be rendered, reviewed, and applied as separate steps, and the plan MUST be dry-runnable against live state.
• ZBP-5-R118 Whether a plan applies automatically or waits for a human MUST be a policy per plan kind, and every applied plan MUST be recorded with what it changed and what triggered it.
• ZBP-5-R119 A new map version MUST NOT move more than a configured fraction of slots relative to the current one unless an override is explicitly recorded with a reason. Coverage validation proves a map is well formed, not that it is right, and a well-formed wrong map moves the entire fleet in one write.
• ZBP-5-R120 Map publication MUST be staged: a canary population of clients and nodes MUST adopt the new version and be observed against a stated outcome before it is promoted to everyone.
• ZBP-5-R121 Promotion MUST be gated on the canary’s observed outcome, and a canary that regresses MUST cause the version to be withdrawn rather than promoted. A stage nobody can fail is not a stage.
• ZBP-5-R122 The staged rollout MUST NOT delay a map version that completes a handoff already past its seal. That partition is unavailable until the map names its new primary, so the brake applies to planned movement and never to the completion of movement already committed.
• ZBP-5-R123 A global freeze MUST exist that halts all NEW movement while leaving serving untouched, and it MUST take effect without a deploy or a restart. It MUST NOT stop a handoff already past its seal, which ZBP-5-R74 forbids from stopping: freezing there would strand the partition with no primary rather than protecting anything.

Scheduled scaling and load shedding

• ZBP-5-R124 Scale-out MUST be pre-warmed: incoming owners restore and catch up while the current owners continue to serve, and the map flips only once they are ready.
• ZBP-5-R125 Scale-in MUST be expressed as merges and handoffs followed by node removal, and the node removed MUST be one the controller selected.
• ZBP-5-R126 The platform MUST NOT terminate a node the controller did not select. Where the platform can terminate on its own — an autoscaling policy, a spot reclamation, a rolling deployment — that capability MUST be disabled, or held behind a hook the controller completes after draining.
• ZBP-5-R127 Scheduled plans MUST be expressible against a calendar, so that state can be compacted and the fleet reduced through a known trough and split again ahead of a known ramp.
• ZBP-5-R128 A scheduled scale-up MUST begin far enough ahead of the expected ramp for pre-warming to complete before load arrives, and the lead time MUST be derived from measured restore times rather than assumed.
• ZBP-5-R129 Compaction — rewriting a checkpoint to drop expired or unreachable state — MUST be safe to abandon at any point and MUST NOT be a precondition for serving.
• ZBP-5-R130 A partition over its capacity budget MUST shed load at admission with a retryable outcome. It MUST NOT absorb the excess into an unbounded in-memory backlog, which converts a throughput problem into a state-loss problem.

Observability and assurance

• ZBP-5-R131 Every partition MUST expose its id, generation, epoch, state, class, current node, lease expiry, checkpoint age, journal LSN, input lag, and standby lag.
• ZBP-5-R132 Every request outcome MUST be attributable to a partition and an epoch.
• ZBP-5-R133 A slot with no ready owner MUST be alarmed within a bounded interval. Unowned is a distinct condition from unhealthy, and MUST NOT be reported as either healthy or as a node failure.
• ZBP-5-R134 Two nodes reporting the primary role for one partition at the same epoch MUST be alarmed as a correctness fault, not as a transient inconsistency.
• ZBP-5-R135 Handoff duration, seal-to-ready duration, and the number of requests rejected during a handoff MUST be measured per handoff and retained per partition.
• ZBP-5-R136 The recovery path MUST be exercised in production on a schedule by deliberately moving a partition, and the measured restore time MUST feed the pre-warm lead time required by ZBP-5-R128.
• ZBP-5-R137 Replay equivalence MUST be tested continuously on real partitions: take a checkpoint at LSN n, restore from an earlier checkpoint at LSN m < n, replay forward to n, and compare the two states. This is the only check that detects a non-deterministic applier, whose entire symptom is that recovered state differs from state that was never recovered.
• ZBP-5-R138 A silent partition MUST be distinguishable from an idle one. Absence of work MUST be reported as a positive statement of liveness, because a partition that has stopped consuming and one with nothing to consume produce identical graphs.
• ZBP-5-R139 The age of the oldest map version still in use by any client or node MUST be visible, so that convergence after a map change is a measured fact rather than an assumption.

Wire formats and interfaces

The partition map

Published by the placement controller, cached by every client and node, and the single answer to “where does this key live right now.”

{
  "map_version": 1487,
  "generated_at": "2026-08-21T09:14:02Z",
  "slot_bits": 12,
  "hash": "md5-128-be",
  "partitions": [
    {
      "id": "p_0002",
      "generation": 5,
      "epoch": 73,
      "class": "std.2x",
      "mode": "JOURNALED",
      "slots": [[1372, 2047]],
      "parents": [{ "id": "p_0001", "generation": 4 }],
      "primary": { "node": "n_7a2c", "endpoint": "10.0.4.11:7443", "state": "SERVING" },
      "standbys": [
        { "node": "n_1f90", "endpoint": "10.0.5.23:7443", "state": "SYNCED", "lag_records": 0 }
      ]
    },
    {
      "id": "p_whale",
      "generation": 1,
      "epoch": 9,
      "class": "ded.8x",
      "mode": "JOURNALED",
      "slots": [[1371, 1371]],
      "parents": [],
      "primary": { "node": "n_c41d", "endpoint": "10.0.6.4:7443", "state": "SERVING" },
      "standbys": [
        { "node": "n_9b02", "endpoint": "10.0.7.9:7443", "state": "SYNCED", "lag_records": 0 }
      ]
    }
  ],
  "signature": {
    "key_id": "map-2026-08",
    "alg": "ed25519",
    "value": "MEUCIQD…"
  }
}

The signature covers every field above it, and is what makes a delta safe to relay. A client applies a delta handed to it by a peer in a MOVED response — see ZBP-5-R11 — so authenticity has to travel with the document rather than with the connection it arrived on. key_id names the signing key so rotation can overlap: a verifier accepts any key inside its validity window and rejects everything else, including a map that is newer than the one it holds.

Primary states are RESTORING, CATCHING_UP, SERVING, SEALED, QUARANTINED, and UNASSIGNED. Standby states are RESTORING, SYNCING, SYNCED, and LOST. UNASSIGNED and QUARANTINED are distinct on purpose: the first says the controller has not placed it, the second says it must not be placed, and conflating them is how a poison partition gets retried forever.

A delta carries only what changed, and names the base it applies to:

{
  "map_version": 1488,
  "base_version": 1487,
  "changed": [ { "id": "p_0002", "…": "…" } ],
  "signature": { "key_id": "map-2026-08", "alg": "ed25519", "value": "MEQCIF…" }
}

The lease record

One record per partition in the lease store, holding ownership, the epoch, and the checkpoint pointer together — because they must move atomically with respect to each other.

partition            p_0002
epoch                74
owner_node           n_1f90
owner_endpoint       10.0.5.23:7443
state                SERVING
lease_ttl_ms         15000
renew_interval_ms    3000
expires_at           2026-08-21T09:14:19.482Z
checkpoint_pointer   {prefix}/p_0002/g5/e73/ckpt-000000918233.manifest.json
sealed_lsn           null
taken_by             null            (operator, on a force-takeover)
taken_reason         null
taken_at             null

Three conditional writes, and no others, may touch it:

ACQUIRE   condition:  attribute_not_exists(partition)
                      OR (epoch < :new_epoch AND expires_at < :now)
          set:        epoch=:new_epoch, owner_node=:me, state=RESTORING, expires_at=:exp

RENEW     condition:  epoch = :my_epoch AND owner_node = :me
          set:        expires_at = :exp

FENCE     condition:  epoch = :my_epoch AND owner_node = :me
          set:        checkpoint_pointer = :ptr        (or state, or sealed_lsn)

FORCE     condition:  epoch < :new_epoch                    -- no expires_at clause
          set:        epoch=:new_epoch, owner_node=:me, state=RESTORING,
                      taken_by=:operator, taken_reason=:reason, taken_at=:now

FORCE is ACQUIRE without the expires_at test, and nothing else (ZBP-5-R26). It is the whole break-glass: it cannot skip the epoch increment, so the displaced node is fenced by exactly the mechanism that would have fenced it anyway — which is why a human may safely be given it and automated placement may not.

:new_epoch is the epoch read a moment earlier plus one. The epoch < :new_epoch clause is what makes two racing acquirers safe: the first write leaves epoch = :new_epoch, so the second’s condition is false and it must re-read and try again at a higher epoch.

Object layout

Immutable, uniquely named, and never overwritten (ZBP-5-R21), so a write from a stale primary lands at a key nothing points at.

{prefix}/{partition}/g{generation}/e{epoch}/ckpt-{lsn:012}.manifest.json
{prefix}/{partition}/g{generation}/e{epoch}/ckpt-{lsn:012}/part-{n:04}
{prefix}/{partition}/g{generation}/e{epoch}/journal/seg-{first_lsn:012}-{last_lsn:012}
{prefix}/{partition}/g{generation}/e{epoch}/spill/{iso8601}-{n:04}.ndjson

The class declaration

The class is where every per-workload choice is declared, and nothing in it defaults. A partition names a class; the class answers what shape of node it needs, what it costs to lose a record, and what it must survive.

{
  "name": "ded.8x",
  "exclusive": true,
  "node": { "cpu": 8, "memory_gb": 64, "local_ssd_gb": 500 },
  "min_synced_standbys": 1,
  "failure_domains": "distinct",
  "commit_tier": "DURABLE",
  "checkpoint": { "kind": "incremental", "max_delta_chain": 16 },
  "freshness_bound": { "max_lag_records": 5000, "max_record_age_ms": 2000 },
  "capacity": { "state_bytes": 34359738368, "requests_per_second": 4000 }
}

commit_tier (ZBP-5-R67), failure_domains (ZBP-5-R61) and checkpoint.kind (ZBP-5-R39) have no defaults on purpose. A class that omits one does not start, which is the only way an omission gets noticed before the incident that depended on it.

The checkpoint manifest

{
  "partition": "p_0002",
  "generation": 5,
  "epoch": 73,
  "format_version": 3,
  "kind": "incremental",
  "base": "…/ckpt-000000901004.manifest.json",
  "deltas": [
    { "key": "…/delta-000000909117", "after": null, "sha256": "77ae…" },
    { "key": "…/delta-000000918233", "after": "…/delta-000000909117", "sha256": "b104…" }
  ],
  "journal_lsn": 918233,
  "input_positions": {
    "shard-0004": "49590000000000000000000012",
    "shard-0005": "49590000000000000000000031"
  },
  "coverage_start": "2026-08-14T00:00:00Z",
  "slots": [[1372, 2047]],
  "created_at": "2026-08-21T09:02:11Z",
  "size_bytes": 412998144,
  "sha256": "9f2c…",
  "parts": [
    { "slot_low": 1372, "slot_high": 1599, "key": "…/part-0000", "sha256": "3a17…" },
    { "slot_low": 1600, "slot_high": 2047, "key": "…/part-0001", "sha256": "c8d4…" }
  ]
}

kind is declared, never inferred (ZBP-5-R39), and on an incremental manifest each delta names the one before it so the chain has exactly one order (ZBP-5-R40). On a full manifest base and deltas are absent and parts alone carries the state.

parts is what makes ZBP-5-R38 real: a child of a split fetches only the parts overlapping its own ranges, so splitting a 400 MB partition four ways reads 400 MB in total rather than 1.6 GB.

Journal record framing

Fixed header, opaque payload, trailing checksum. Sizes in bytes, all integers big-endian.

off   len  field
  0     4  frame length — every byte after this field, including the CRC
  4     1  record type — 1 APPLY, 2 MARK, 3 SEAL
  5     8  lsn
 13     8  epoch
 21     4  slot
 25     4  marks length (L)
 29     L  marks — input positions consumed at and before this record
29+L    P  payload — opaque to the runtime, interpreted only by the applier
 tail   4  CRC32C over bytes 4 .. end-4

A MARK record carries positions and no payload; it is how a partition advances its resume point during a quiet period without writing state. A SEAL record is the last record of a generation and carries the final positions.

The routing envelope

Returned on every non-served outcome, and readable without knowing the application’s own error shape.

{
  "code": "MOVED",
  "slot": 1789,
  "map_version": 1488,
  "retry_after_ms": null,
  "assignment": {
    "partition": "p_0002",
    "epoch": 74,
    "primary": { "node": "n_1f90", "endpoint": "10.0.5.23:7443" }
  },
  "map_delta": {
    "map_version": 1488,
    "base_version": 1487,
    "changed": [ "…" ],
    "signature": { "key_id": "map-2026-08", "alg": "ed25519", "value": "MEQCIF…" }
  }
}

HTTP 421 is the correct status for MOVED — it means precisely that the request reached a server unable to produce a response for it — and unlike a 3xx it is not followed automatically by intermediaries that know nothing about slots.

Position tokens

Every mutating response carries the position it was applied at; a read may demand a floor.

X-Partition-Position: p_0002:74:918407        (partition : epoch : lsn)
X-Min-Position:       p_0002:74:918407        (on a read, the floor the caller requires)

Positions compare as the ordered pair (epoch, lsn), numerically, never as strings — 74:9 precedes 74:10, which a lexicographic comparison gets backwards.

Handoff control interface

PrepareHandoff(partition, generation, target, deadline) → { accepted, incoming_epoch }
StreamState(partition, from_lsn)                        → stream of journal records
ReportReady(partition, incoming_epoch, position)        → { ok }
Seal(partition, epoch)                                  → { sealed_lsn, input_positions }
Promote(partition, expected_epoch)                      → { new_epoch }
AbortHandoff(partition, reason)                         → { ok }

Algorithms

Each step names its failure outcome. An implementation that reaches a state not named here has found a gap in this document, and should be treated as such rather than improvised around.

A.1 — Resolve a key (client)

1. slot ← hash(utf8(key)) >> (128 − slot_bits).
2. Look up slot in the cached map. Not found → NO_MAP: fetch a full map and restart; if that fails, fail the request.
3. Read the partition’s primary endpoint. State is not SERVING → NOT_READY: wait the class’s retry delay and restart, subject to the caller’s deadline.
4. Send the request carrying key, slot, and the cached map_version.
5. On MOVED, apply map_delta, increment the redirect counter, and restart from step 2. Counter past its bound → REDIRECT_LOOP: fail, and alarm.
6. On MISROUTED → HASH_DISAGREEMENT: fail immediately without retrying, and alarm. Retrying cannot help; the client and the server do not compute the same slot.
7. On NOT_READY or STALE_MAP, wait retry_after_ms and restart from step 2.

A.2 — Admit a request (server)

1. Recompute slot from the key. Disagrees with the client’s slot → MISROUTED.
2. Check the lease: is this node the primary for the partition owning slot, at the epoch it believes it holds, with the lease not near expiry per ZBP-5-R18? No → go to step 5.
3. Is the partition SERVING? No → NOT_READY with the class’s retry delay. Never a redirect: a partition restoring here is not somewhere else.
4. Is the partition within its capacity budget? No → OVERLOADED, a retryable shed, per ZBP-5-R130. Serve otherwise.
5. Not the owner. Compare map versions per ZBP-5-R101: mine newer → MOVED with a delta; equal → MISROUTED; mine older → STALE_MAP.

A.3 — Acquire and restore a partition (cold path)

This is the path that must work; A.4 is the optimisation.

1. ACQUIRE the lease at epoch + 1. Condition failed → LOST_RACE: re-read and retry with backoff. Set state RESTORING.
2. Start lease renewal (A.7) immediately, before any expensive work. A restore that outlives its own lease has been wasted.
3. Read checkpoint_pointer from the lease record — never by listing the object store, which is eventually consistent about its own contents and cannot distinguish a live checkpoint from a zombie’s.
4. Fetch the manifest. Fetch only the parts overlapping this partition’s slot ranges. Digest mismatch on any part → treat the checkpoint as absent per ZBP-5-R37 and fall back to the previous pointer; none left → NO_CHECKPOINT, restore from the beginning of retained input.
5. Refuse any format_version this binary does not fully understand → FORMAT_UNKNOWN: release the lease and report. Do not restore partially.
6. Restore state. In DERIVED mode go to step 8 with the manifest’s input_positions.
7. JOURNALED or MIXED: replay journal segments from journal_lsn + 1 forward, discarding records for slots not owned. A gap in the LSN sequence → JOURNAL_GAP: stop, quarantine, alarm. Never skip a gap. On reaching the tail, take the resume positions from the tail record’s marks — not from the checkpoint, which is behind the journal and would re-apply the window between them.
8. Set state CATCHING_UP. Open an iterator per contributing shard at its resume position. Drain parent shards fully before their children per ZBP-5-R54.
9. Apply records for owned slots; count and discard the rest. An applier failure → retry bounded, then POISON: quarantine per ZBP-5-R59, release the lease, and do not re-place.
10. When lag is inside the class’s freshness bound, FENCE state to SERVING, report ready, and let the controller publish a map naming this node.

A.4 — Live handoff with a synced standby

The steps before the seal are reversible and cost nothing if abandoned. The seal is one conditional write. Everything expensive happens before it.

1. Controller calls PrepareHandoff on the incoming node with a deadline. Refused → abort; nothing has changed.
2. Incoming node restores per A.3 steps 3–9, but stops short of acquiring the primary lease. It runs as a standby, consuming the primary’s journal (JOURNALED) or the input stream (DERIVED).
3. Incoming node reports SYNCED when its lag is inside the class’s bound. Deadline passes first → PREPARE_TIMEOUT: abort, outgoing primary keeps serving, map unchanged.
4. Seal. Outgoing primary stops admitting, appends a SEAL record, and FENCEs sealed_lsn and state SEALED into the lease record under its own epoch. From here it may never accept another write, whatever happens next (ZBP-5-R74).
5. Outgoing primary rejects every uncommitted in-flight request with MOVED and no assignment yet — the client will get NOT_READY from the incoming node until step 7, which is correct and bounded.
6. Incoming node consumes to sealed_lsn. Outgoing node unreachable → recover the remaining segments from the object store; if the tail was never durable, replay from the last durable mark and the input stream. It cannot roll back, because the seal already happened.
7. Incoming node ACQUIREs at epoch + 1, sets SERVING, and reports ready.
8. Controller publishes the new map version. Only now do clients learn.
9. Outgoing node discards its state without flushing (ZBP-5-R24).

Between steps 4 and 8 the partition is unavailable. That window — seal to ready — is the number this whole design exists to make small, and it must be measured per handoff (ZBP-5-R135).

A.5 — Split

1. Controller chooses a split point at a slot boundary and creates child entries with a generation above the parent’s, state RESTORING, and parents recording the parent’s id and generation.
2. Each child restores from the parent’s checkpoint, fetching only the parts overlapping its own ranges, then replays the parent’s journal filtered to its ranges.
3. Each child reports SYNCED. A child that cannot → abort: the parent has not sealed, so nothing has moved.
4. Parent seals as in A.4 step 4. Every child then consumes to sealed_lsn.
5. Children acquire their own leases at epoch 1 for the new partition ids, and open input iterators at the parent’s sealed positions.
6. Controller publishes a map in which the parent no longer appears and the children cover its slots exactly. Coverage validation (ZBP-5-R5) is what catches an arithmetic error here, before anyone routes against it.
7. Parent state is retained until every child has taken its own checkpoint (ZBP-5-R95).

A.6 — Merge

Identical to A.5 with the arrows reversed: one child restores from every parent’s checkpoint and journal, all parents seal before the child accepts a write, and no parent resumes afterwards. Because the parents’ slot ranges are disjoint, per-key order survives; cross-key order does not, and nothing may depend on it (ZBP-5-R94).

A.7 — Lease renewal and self-fencing

Runs continuously on every primary, in its own goroutine, independent of request handling.

1. On a successful RENEW, record deadline ← monotonic_now() + lease_ttl − margin, where margin covers the worst renewal round-trip plus tolerated clock error.
2. Every renewal interval, attempt RENEW. Condition failed → LEASE_LOST: go to step 4 immediately.
3. Before admitting any request, and on a timer regardless of traffic, check monotonic_now() < deadline. False → LEASE_EXPIRED: go to step 4. This check uses no network and no wall clock, which is exactly why it still runs correctly after a long process pause.
4. Stop admitting, fail in-flight work as retryable, discard state without flushing, and report. Do not attempt a final checkpoint: this node may already have been superseded, and its write would be fenced anyway (ZBP-5-R20).

A.8 — Reconciliation

1. Read observed state: node heartbeats with class and capacity, lease records, partition health.
2. Compute the desired placement: satisfy class constraints, then minimise movement, then balance (ZBP-5-R115).
3. Diff against the current map to produce a plan of moves, splits, and merges.
4. Drop moves that violate the hysteresis interval or the concurrency bound (ZBP-5-R116).
5. Emit the plan. Auto-apply or hold for a human per policy (ZBP-5-R118).
6. Execute one step at a time, re-reading observed state between steps. A step that fails aborts the plan; it does not roll back the steps already applied, because each completed handoff is already a consistent state.

A.9 — Scheduled compaction and pre-split

The evening pass, at a known trough:

1. Merge adjacent under-utilised partitions per A.6.
2. Rewrite each surviving checkpoint, dropping expired and unreachable state, and flip the pointer. Abandonable at any point (ZBP-5-R129) — an interrupted compaction leaves the previous checkpoint live and nothing else changed.
3. Retire the drained nodes, each selected by the controller (ZBP-5-R125).

The morning pass, ahead of a known ramp:

1. Start from T_ramp − lead, where lead is measured restore time plus catch-up time plus a margin, taken from the production restore exercises required by ZBP-5-R136 — never assumed.
2. Add nodes; split partitions per A.5, with children pre-warmed while the parents still serve.
3. Flip the map only once every child is SYNCED. If pre-warming is not finished by T_ramp, do not flip: serving a ramp from a warm parent beats serving it from a cold child.

AWS binding

The requirements above are cloud-neutral. This section is the concrete mapping, and names three limits that are properties of the platform rather than of the design — which means no amount of care in the runtime removes them.

Kinesis alignment

A stream maps a record to a shard by taking the MD5 of the partition key as a 128-bit integer, and each shard owns a contiguous [StartingHashKey, EndingHashKey] range of that space. The slot is defined here as the top SLOT_BITS bits of the same MD5 of the same key — so a slot is a sub-range of the shard hash space, and a partition, being a set of slot ranges, is a set of hash ranges. Nothing needs configuring: use the service key as the stream partition key and alignment follows.

Choose SLOT_BITS so that 2^SLOT_BITS comfortably exceeds the largest shard count the stream will ever have — at 12 bits, 4096 slots against a few hundred shards leaves every shard boundary landing inside a slot rather than splitting one. Where a slot does straddle two shards, the partition simply reads both; ZBP-5-R53’s per-shard position vector already accommodates it.

For the escape hatch of ZBP-5-R52, the explicit hash key for slot s is s × 2^(128 − SLOT_BITS), rendered as a decimal string. On a reshard, ListShards reports ParentShardId and AdjacentParentShardId; a parent is drained when GetRecords returns a null NextShardIterator, and only then may its children be read (ZBP-5-R54).

Three platform limits

One key cannot exceed one shard. A key is a single point in the hash space, so every record for it lands in one shard and is bounded by that shard’s ingest quota. Pinning a large tenant to its own slot and its own node buys dedicated compute; it buys no additional ingest at all. Past the shard quota the key must become a composite of tenant and sub-key, explicitly forfeiting ordering between sub-keys (ZBP-5-R96). This is the first ceiling a very large tenant meets, and it is not in the placement layer — which is why teams look for it there and do not find it.

The container stop timeout bounds partition size. The grace period between the termination signal and the kill is capped by the platform — on the serverless launch type, at two minutes. Every partition on a node must complete handoff inside it, and ZBP-5-R85 checks the arithmetic at boot. So the maximum state a partition may hold is not a memory question but a drain-rate question: whatever can be sealed, streamed, and promoted inside the budget, divided by the partitions per node. A partition that has outgrown the budget must be split, and the alarm for it is the measured handoff duration of ZBP-5-R135 approaching the budget — not an out-of-memory event, which will never come.

Checkpoint age must be read against the configured retention, not the default. A stream retains 24 hours unless configured otherwise, extensible to a year. ZBP-5-R47’s alarm compares the newest usable checkpoint against whatever is configured now, so that lowering retention is caught as the recovery regression it is. Raising retention is also the cheapest immediate mitigation when the alarm does fire, and is worth having in the runbook.

Node management

Do not use a platform rolling deployment. It starts a replacement before the outgoing task has drained, chooses its own ordering, and has no concept of which partitions are mid-handoff — every one of which contradicts ZBP-5-R80. The controller starts tasks, drains them, and stops them.

Where an autoscaling group is used for capacity, enable instance scale-in protection and a termination lifecycle hook, so the group cannot choose a victim and the controller completes the hook after draining (ZBP-5-R126). Spot capacity MUST NOT host primaries unless the interruption notice is wired to a handoff, and even then the notice is a courtesy rather than a guarantee — a standby on on-demand capacity is what makes spot survivable.

Threat model and failure modes

Three of these deserve more than a row.

The stale checkpoint is the one that does not announce itself. Every other failure here produces a signal at the moment it happens. This one produces nothing: the partition serves normally, the checkpoint pointer is valid, the input stream is healthy, and the system is simply no longer recoverable. It becomes visible at the exact moment you need it not to be — during a recovery, when the stream has already aged past the checkpoint and there is no path from one to the other. It is why ZBP-5-R47 alarms on margin rather than on failure, and why ZBP-5-R48 insists the restore path is exercised against real checkpoints continuously rather than tested once.

A non-deterministic applier is a correctness bug that no test of the happy path can see. The service is right until it recovers, and then it is quietly wrong — different counters, a different window, a decision that would not have been made. The test that finds it is a replay-equivalence check: take a checkpoint at LSN n, restore from the checkpoint at LSN m < n, replay to n, and compare the two states byte for byte. Run it continuously, on real partitions. Wall-clock reads, map iteration order, and a stray call to an external service are the three sources, in that order of frequency.

Quarantine is a containment, and also an attack surface. ZBP-5-R59 exists because an applier failure that is merely retried will crash every node the partition is placed on, in turn. But the same mechanism means that anyone who can get a record into the input stream that reliably fails the applier can take that partition offline for as long as the record is retained, using one input. The two readings are the same requirement seen from opposite sides, and the trade is not avoidable at this layer — it is chosen. Where the application can tolerate skipping a record, skipping and counting it is the better default and quarantine the fallback; where it cannot, quarantine is correct and the exposure has to be answered upstream, by validating what may enter the stream at all. What is not acceptable is leaving it unstated, so that the first crafted record is discovered as a mystery outage rather than as the known cost of a known control.

The poison record turns a partition failure into a fleet failure. Without ZBP-5-R59 the sequence is mechanical: the partition is placed, the applier fails, the node dies, the controller places it elsewhere, that node dies. There is no natural stopping point, and the blast radius grows to every node in the class. Quarantine is not a nicety; it is the difference between one unavailable partition and no service at all.

Anti-patterns

If you find yourself doing one of these, you have misread the document.

• Using a consistent-hash ring as the assignment. The moment you add weighted virtual nodes to express “this customer needs its own hardware”, you are reimplementing a published map, badly, in a form nobody can read or diff. Publish the map.
• A key-level pin. It seems harmless and it breaks stream alignment, split, and merge simultaneously. Give the key its own slot instead (ZBP-5-R4).
• Asking the controller who owns a key. Any per-request call to the control plane makes it a serving dependency, which is the exact coupling ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Why it's cited here: The reason the lease store is separate from the placement controller: a control-plane outage must freeze decisions, not stop serving.Open ZFN-16 → exists to prevent.
• A sticky load balancer in place of a map. Stickiness is a hint with no notion of epoch, so it cannot tell a client its assignment is stale and will route happily to a node that has been fenced.
• Autoscaling the fleet. A scale-in policy chooses a victim by CPU, which is uncorrelated with drainability. It will pick the node holding the largest partition roughly as often as any other.
• A platform rolling deployment. It starts the replacement before the outgoing task drains and picks its own order. A rollout here is a sequence of handoffs (ZBP-5-R80).
• Treating “a standby exists” as durability. A standby with unbounded lag is a second copy of an old state. Measure the lag or do not claim the property (ZBP-5-R64).
• Rebalancing on every metric tick. Without hysteresis and a movement penalty, a controller optimising balance will move partitions continuously and serve worse than one that does nothing.
• Taking a final checkpoint when the lease is lost. The instinct is to save what you have. It is precisely wrong: this node may already have been superseded, its state is a divergent branch, and the write is fenced anyway (ZBP-5-R24).
• Retrying MISROUTED. It means the client and the server do not compute the same slot. No number of retries fixes a disagreement about arithmetic; it needs an alarm and a human.
• Deleting journal segments by age. A split child that has not yet checkpointed depends on its parent’s segments, and age says nothing about that (ZBP-5-R45).
• Recomputing on replay what was computed at write time. Anything not captured into the record is a source of divergence, and divergence here is silent (ZBP-5-R34).
• Calling lock shards partitions. Striping a map across 64 mutexes is a concurrency technique with no relationship to ownership, and sharing the word costs somebody a day.

Test vectors

Every value below was produced by the reference implementation at the end of this section and can be reproduced by running it. SLOT_BITS = 12, so SLOT_COUNT = 4096 and the hash is MD5 of the key’s UTF-8 bytes read big-endian.

V1 — Key to slot

At 12 bits the slot is exactly the first three hex digits of the MD5, which makes these checkable by eye as well as by code.

V2 — Refinement from 12 to 13 bits

Raising SLOT_BITS must move no key across a partition boundary (ZBP-5-R3). Each slot becomes exactly two, and each key lands in one of its own slot’s two children.

Ranges rewrite mechanically: [1371, 1371] at 12 bits becomes [2742, 2743] at 13 bits, and [0, 1370] becomes [0, 2741].

V3 — Map resolution

Against the map in Wire formats at map_version 1487, where p_whale holds the single slot 1371 on an exclusive class:

V4 — Routing decision

The complete outcome table for ZBP-5-R99 through ZBP-5-R102. Note rows 4 and 6: identical from the client’s side, opposite in cause, and a system that collapses them redirects in circles.

V5 — Recovery positions

The trap this vector exists for is taking the resume positions from the checkpoint rather than from the journal tail, which silently re-applies every input record between them.

Given a checkpoint at journal_lsn 918233 with positions {shard-0004: …012, shard-0005: …031}, and two journal segments after it — 918234–940100 and 940101–961887, the latter marking {shard-0004: …901, shard-0005: …877}:

V6 — Epoch fencing

Partition p_0002 during a handoff from n_7a2c to n_1f90. The record starts at epoch 73 owned by n_7a2c; ACQUIRE and FENCE are the conditions from Wire formats, and the sequence runs against the reference implementation below.

Steps 4 and 5 are the whole safety argument. The outgoing node need not know it has been superseded, and need not be reachable, for its write to be harmless — and it cannot recover by re-claiming at the epoch it thinks it holds, because ACQUIRE requires a strictly higher one.

V7 — Slot to explicit hash key

For ZBP-5-R52’s escape hatch, explicit_hash_key(s) = s × 2^116, as a decimal string:

The top of the space is 2^128 − 1 = 340282366920938463463374607431768211455, so slot 4095 spans from its low hash key to there.

Reference implementation

import hashlib

SLOT_BITS = 12

def slot(key: str, bits: int = SLOT_BITS) -> int:
    h = int.from_bytes(hashlib.md5(key.encode("utf-8")).digest(), "big")
    return h >> (128 - bits)

def explicit_hash_key(s: int, bits: int = SLOT_BITS) -> str:
    return str(s << (128 - bits))

def refine(lo: int, hi: int, d: int) -> tuple[int, int]:
    return (lo << d, (hi << d) | ((1 << d) - 1))

def resolve(key: str, partitions: list[dict], bits: int = SLOT_BITS) -> str | None:
    s = slot(key, bits)
    for p in partitions:
        if any(lo <= s <= hi for lo, hi in p["slots"]):
            return p["id"]
    return None

def route(owns: bool, ready: bool, server_v: int, client_v: int) -> str:
    if owns:
        return "SERVE" if ready else "NOT_READY"
    if server_v > client_v:
        return "MOVED"
    return "MISROUTED" if server_v == client_v else "STALE_MAP"

def resume_positions(checkpoint: dict, segments: list[dict], mode: str) -> dict:
    if mode == "DERIVED" or not segments:
        return checkpoint["input_positions"]
    return segments[-1]["marks"]          # the TAIL's marks, never the checkpoint's

def acquire(record: dict, new_epoch: int, claimant: str) -> str:
    """ACQUIRE: lands only at a strictly higher epoch, which is what makes two
    racing claimants safe — the loser must re-read and retry higher."""
    if new_epoch > record["epoch"]:
        record.update(epoch=new_epoch, owner_node=claimant)
        return "ACCEPT"
    return "REJECT_STALE_EPOCH"

def fence(record: dict, write_epoch: int, writer: str) -> str:
    """FENCE: a write lands only if it names the record's CURRENT epoch and its
    current owner. This, not the lease, is what makes a stale primary harmless."""
    if write_epoch == record["epoch"] and writer == record["owner_node"]:
        return "ACCEPT"
    return "REJECT_STALE_EPOCH"

Conformance checklist

Every MUST above, grouped. An implementation grades itself here; a box that cannot be ticked is a citable gap rather than an oversight.

Key space and map

• Slot is the top SLOT_BITS bits of a fixed, named, process-stable 128-bit hash of the UTF-8 key (R1, R2)
• SLOT_BITS can be raised by rewriting ranges only, never lowered, and raising moves no state (R3)
• No key-level pin exists; every partition is a set of slot ranges (R4)
• Publication validates total coverage and non-overlap and refuses otherwise (R5)
• Map version is monotonic, never reused, and changes on any content change (R6)
• Entries carry id, generation, epoch, class, sorted ranges, primary and standby endpoints and states (R7)
• Map is readable without the controller, and old versions stay addressable for the longest cache lifetime (R8, R9)
• Partition ids are opaque and encode no range (R13)
• Every published map version and delta is signed by the controller, over versions, assignments and endpoints (R10)
• Clients and nodes verify that signature before applying, whatever the source, including a delta relayed by a peer (R11)
• An unverifiable map is rejected, counted and alarmed, and is never applied for being newer; signing keys rotate with overlap (R12)

Ownership and fencing

• Every ownership record has an expiry, a named owner, and a renewal interval (R14)
• Acquisition increments the epoch in the same conditional write that claims the lease (R15)
• Epochs are monotonic per partition, never reused, and survive split and merge (R16)
• Expiry is judged from a monotonic reading with a margin, needing no network (R17, R18)
• Every shared-storage write carries its epoch, and the store rejects a lower one (R19, R20)
• Objects are immutable and uniquely named; the checkpoint pointer lives in the lease store and moves under an epoch-fenced write (R21, R22)
• Holding the lease and being ready are distinct states (R23)
• Lease loss discards state without flushing, checkpointing, or replicating (R24)
• The lease store is separate from the controller and reachable without it (R25)
• A force-takeover exists that acquires at a higher epoch without waiting out the lease (R26)
• Force-takeover advances the epoch through the same conditional write; no path takes ownership without advancing it (R27)
• Force-takeover records operator, time and reason, and is unreachable by automated placement (R28)

State, checkpoints, journal

• Recovery mode is declared per partition and recorded in the map (R29)
• DERIVED has no journal and recovers by checkpoint plus input replay (R30)
• JOURNALED appends before applying, records consumed positions on the record, has one write path (R31)
• MIXED is used only for disjoint, non-interacting state sets (R32)
• The applier is deterministic; non-deterministic values are captured at write time, never recomputed (R33, R34)
• LSNs are dense, gapless per generation, and journal-assigned (R35)
• Checkpoints record LSN, all positions, format version, length, digest; digest is verified and a mismatch means absent (R36, R37)
• Checkpoints are slot-ordered so a range extracts without a full read (R38)
• Checkpoint kind is declared per class and recorded in every manifest, never inferred (R39)
• An incremental checkpoint is a base plus an ordered delta chain, each naming its predecessor, every digest verified, a mismatch meaning the whole chain is absent (R40)
• Delta chains have a stated maximum length and are compacted before it, so restore time has a ceiling (R41)
• A base outlives every delta that depends on it, and both lineages are walked before any deletion (R42)
• Formats are versioned, an unknown version is refused not partially read, and adjacent releases interoperate (R43, R44)
• Deletion is lineage-driven, never age-driven (R45)
• Deletion actually runs on a schedule, and retained-but-unreachable volume is monitored (R46)
• Checkpoint age alarms against configured retention with an actionable margin (R47)
• Restore is exercised continuously against real production checkpoints (R48)

Input and catch-up

• Coverage start is published; an under-covered query answers incomplete rather than partial (R49)
• The stream is ordered per key and replayable; nothing depends on cross-key order (R50)
• Shards and slots share a hash space and producers key by the service key; explicit hash keys are the exception (R51, R52)
• Positions are per contributing shard, never one cursor (R53)
• Parents drain fully before children on every reshard (R54)
• Records carry a stable id, ingest deduplicates, and the dedup horizon equals the retention horizon (R55, R56)
• Readiness requires a stated, per-class, reported freshness bound in both lag and age (R57)
• Unowned slots’ records are discarded and counted, never applied and never an error (R58)
• Applier failures are bounded then quarantined, not retried across the fleet (R59)

Standbys

• A class can require a minimum synced standby count, and below it the partition is not healthy (R60)
• Every class declares required failure-domain diversity between primary and standby, and ‘none’ is said explicitly (R61)
• A partition not meeting its declared diversity is not reported healthy (R62)
• Standbys consume the journal in JOURNALED mode and the stream in DERIVED mode (R63)
• Applied position is reported and lag is a metric, not an inference (R64)
• Standbys never write, and read only under an explicit staleness bound returning the served position (R65)
• Commit policy is an explicit per-class setting (R66)
• The three commit tiers are defined exactly as LOCAL, STANDBY, DURABLE, and nothing else is offered (R67)
• Every class declares its tier and refuses to start without one; there is no default (R68)
• The tier a write actually achieved is reported on the response (R69)
• Below DURABLE, acked-but-not-durable count and oldest age are monitored and the exposure is stated in the class (R70)
• The no-standby-remaining behaviour is explicitly configured, not emergent (R71)
• Standbys verify the epoch of the stream they consume and stop on a mismatch (R72)
• Promotion increments the epoch before the first write (R73)

Handoff and rollout

• There is exactly one seal, after which the outgoing primary never writes again (R74)
• Pre-seal abort leaves the primary serving and the map unchanged (R75)
• Post-seal the partition only moves forward; the outgoing primary never resumes (R76)
• Uncommitted in-flight requests are rejected retryably, never as success and never permanently (R77)
• The incoming primary is caught up before the seal; the seal is the last step (R78)
• No map names a new primary before it reports ready under its new epoch (R79)
• Rollout is a bounded sequence of handoffs, not a fleet restart or a platform rolling deploy (R80)
• Concurrent handoffs are bounded per node and globally (R81)
• Every handoff has a deadline and never waits indefinitely (R82)
• Termination initiates handoff of every hosted partition (R83)
• Shutdown phases run in order and do not undo each other (R84)
• Phase deadlines are summed and validated against the platform budget at boot; a bad budget refuses to start (R85)
• Unfinished work spills to object storage in the sink’s own record shape (R86)
• Unannounced kill is survivable, and that path is the one most exercised (R87)
• Rollback obeys the same format-compatibility rule (R88)

Split and merge

• Splits divide at slot boundaries into children with higher generations and no empty child (R89)
• Merges combine adjacent ranges into one higher-generation child (R90)
• Lineage is recorded and retained while any child depends on it (R91)
• Children restore from parents’ checkpoints and journals filtered to their own ranges, then resume from sealed positions (R92)
• Every parent seals before any child writes, and no parent resumes (R93)
• Nothing depends on cross-key order across a split or merge (R94)
• A child checkpoints before its parents’ state becomes deletable (R95)
• One slot is the split floor, and a hotter-than-a-node key is treated as a key-design problem (R96)

Routing

• Clients compute the slot and resolve against a cached map, never per-request lookup (R97)
• Requests carry key, slot, and the client’s map version (R98)
• An owned, ready slot is served regardless of the client’s map version (R99)
• An owned, not-ready slot returns NOT_READY and never redirects (R100)
• A non-owned slot resolves to MOVED, MISROUTED, or STALE_MAP by map-version comparison (R101)
• No redirect is ever derived from a map older than the client’s (R102)
• MOVED carries enough map to route the retry, and the client applies it first (R103)
• Redirects per logical request are bounded and the request fails rather than looping (R104)
• MISROUTED is counted and alarmed as a correctness fault (R105)
• Map refresh is jittered as well as redirect-driven (R106)
• Mutating responses carry an (epoch, LSN) position compared numerically as a pair (R107)
• Position tokens name their partition, are never compared across a lineage boundary, and a token from a vanished partition is answered explicitly (R108)

Placement and control plane

• The controller is never called on the serving path (R109)
• The data plane serves from the last map through a controller outage (R110)
• The controller stores no state inside the service it places (R111)
• Partitions declare class and capacity; nodes declare class and budget; constraints bind first (R112)
• An exclusive class means exactly one partition per node, including transiently (R113)
• Placement enforces diversity as a hard constraint, and an unsatisfiable one places-and-reports rather than leaving the partition unowned (R114)
• The objective is constraints, then movement, then balance — in that order (R115)
• Concurrent moves are bounded and hysteresis is enforced per partition (R116)
• Decisions are explicit, renderable, dry-runnable plans (R117)
• Auto-apply versus hold is a per-plan-kind policy, and applied plans are recorded (R118)
• A map moving more than the configured fraction of slots is refused without a recorded override (R119)
• Publication is staged through a canary population observed against a stated outcome (R120)
• Promotion is gated on the canary, and a regressing canary withdraws the version (R121)
• The stage never delays a map completing a handoff already past its seal (R122)
• A global freeze halts movement without a deploy and without touching serving (R123)

Scaling and shedding

• Scale-out pre-warms, and the map flips last (R124)
• Scale-in is merges and handoffs, and the controller picks the node (R125)
• The platform cannot terminate a node the controller did not select (R126)
• Plans can be scheduled against a calendar (R127)
• Scale-up lead time comes from measured restore time, not assumption (R128)
• Compaction is abandonable and never a precondition for serving (R129)
• Over-budget partitions shed at admission rather than buffering (R130)

Observability

• Every partition exposes id, generation, epoch, state, class, node, lease expiry, checkpoint age, LSN, input lag, standby lag (R131)
• Every outcome is attributable to a partition and an epoch (R132)
• An unowned slot is alarmed, and unowned is distinct from unhealthy (R133)
• Duplicate primaries at one epoch alarm as a correctness fault (R134)
• Handoff duration, seal-to-ready, and rejected-request counts are measured per handoff (R135)
• Recovery is exercised in production on a schedule and feeds the lead time (R136)
• Replay equivalence is tested continuously on real partitions: restore earlier, replay forward, compare (R137)
• Silence is a positive liveness statement, not an absent signal (R138)
• The oldest map version in use is visible (R139)

Rationale

The arguments behind these requirements are not made here; they are made in the Field Notes this document rests on, and repeating them would double its length without improving an implementation.

Ownership is a lease with an epoch because ZFN-37Field Note · currentZFN-37 — Every lock is a leaseA lock that can outlive its holder is a deadlock scheduled for later. Give every lock — including informal ones like claimed_by columns — a TTL, a named owner, and a heartbeat; make expiry automatic and server-side; fence the side effects so a stale holder can't corrupt anything.Why it's cited here: Ownership here is a lease with a TTL, an owner and a heartbeat — and the epoch is the fencing token that makes a stale holder harmless.Open ZFN-37 → is right that a lock which can outlive its holder is a deadlock scheduled for later, and because the fencing token is the only part of that arrangement which is actually safe. The lease is judged on the monotonic clock and recorded in wall time for the two-clock reasons in ZFN-59Field Note · currentZFN-59 — Two clocks: monotonic for durations, wall time for recordsYou have two clocks. Wall time names moments — and it jumps, slews, and runs backwards. The monotonic clock measures elapsed time — and means nothing across machines or reboots. Every timeout, lease, and cross-machine ordering bug is one clock doing the other's job.Why it's cited here: Why a node decides its own lease has expired on the monotonic clock while the lease record's expiry is wall time.Open ZFN-59 → — a process pause is exactly the case where the two disagree and exactly the case that matters.

The write path is append-then-apply because of ZFN-65Field Note · currentZFN-65 — Journal the write, apply it in micro-batchesIf nothing the caller does next depends on a write, it does not belong in the request path: append it to a durable ordered journal and apply it in micro-batches. The bill is ordering. Sequence comes from the journal, never the clock, and never two paths to one row.Why it's cited here: The write path here is that pattern: append to the partition journal, apply in micro-batches, take sequence from the journal and never from the clock.Open ZFN-65 →: sequence comes from the journal and never from the clock, and there is never a second path to one row. The input stream and the partition journal are both journals in the sense ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Why it's cited here: The distinction this document builds on: the input stream is a journal, the per-partition write log is a journal, and neither is a queue.Open ZFN-12 → draws, and neither is a queue — which is why replay is a first-class operation here rather than a recovery hack.

The lease store sits apart from the placement controller because ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Why it's cited here: The reason the lease store is separate from the placement controller: a control-plane outage must freeze decisions, not stop serving.Open ZFN-16 → is the difference between a control-plane outage that freezes change and one that stops serving, and the controller keeps no state inside the service it places because ZFN-4Field Note · currentZFN-4 — Incident tooling must not depend on what it recoversAnything you need to respond to an incident — deploy/rollback, kill switches, observability, break-glass access — must not depend, directly or transitively, on the systems likely to be down during it. Never gate incident tooling behind a system it might need to recover.Why it's cited here: Why the placement controller must not keep its own state inside the service it places — the recovery tool cannot depend on what it recovers.Open ZFN-4 → is about exactly the moment you need it not to.

Handoff is a protocol rather than a stop because ZFN-60Field Note · currentZFN-60 — Drain before you die: graceful shutdown is a protocolSIGTERM isn't an emergency — it's every deploy and scale-in. Shutdown is a protocol: stop attracting work, drain while the balancer catches up, hand back in-flight work, release leases, exit before SIGKILL. But graceful is only the optimisation — crash-safe is the requirement.Why it's cited here: Handoff is that protocol made explicit: stop attracting work, drain, hand back what is in flight, release the lease, exit.Open ZFN-60 → sets out the four phases and, more usefully, says that graceful is the optimisation and crash-safe is the requirement. Checkpoint and journal formats evolve by expand, migrate, contract per ZFN-62Field Note · currentZFN-62 — Expand, migrate, contract: schema changes in three movesEvery deploy runs two code versions against one database — and rollback runs yesterday's code on today's schema. No schema change may break either. So every migration is three shippable moves: expand (additive), migrate (backfill, verify), contract (remove, later, deliberately).Why it's cited here: How checkpoint and journal formats evolve across a rollout, given that two adjacent versions must read each other's state.Open ZFN-62 →, because a rollout and its rollback both run two versions at once. Over-budget partitions shed rather than buffer for the flow-control reasons in ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Why it's cited here: Where shedding belongs when a partition is over budget — push the failure back to the caller rather than absorbing it into a growing in-memory backlog.Open ZFN-13 →. Position tokens on mutating responses are ZFN-25Field Note · currentZFN-25 — Track the version a client has seen for read-your-writesFor read-your-writes across backends, track the latest version a client has seen — a token or vector clock. Return it on write; reads then go to a backend at least that fresh. Hold it client-side (a token they present) or server-side (a gateway tracks the session and routes).Why it's cited here: The position token returned on every write is that idea applied to a partition — a client that needs read-your-writes carries the position it saw.Open ZFN-25 → applied to a partition. And the restore canary exists because ZFN-36Field Note · currentZFN-36 — An untested backup is not a backup — test it by restoringAn untested backup is a hope, not a backup — the only thing that counts is a restore. Rehearse restores regularly (game days), measure and meet your RTO/RPO, automate them, and cover the whole recovery path — data, schema, config, secrets, cutover — not just the dump.Why it's cited here: Why a checkpoint that has never been restored is not a checkpoint, and the restore canary is a requirement rather than a nicety.Open ZFN-36 → is unambiguous that a backup nobody has restored is not a backup — which is doubly true of a checkpoint, since the thing that expires underneath it is a retention window nobody is watching.

Where the partition key is a tenant id, this document is also how ZBP-3Blueprint · v1.0.0ZBP-3 — Tenant isolation in a multi-tenant systemOne system stores or serves data belonging to more than one customer, and one customer seeing another's data would be a serious incident.Why it's cited here: Partitioning by tenant is the common case, and moving one tenant to dedicated hardware is how that document's isolation-tier requirement is satisfied for a stateful service.Open ZBP-3 → satisfies its own requirement that a tenant be movable to a stronger isolation tier without application changes: an exclusive class and a pinned slot are what “dedicated” means for a stateful service.

Changelog

• 2026-08-21 (1.0.0): Adds How it works, a detailed non-normative walkthrough between the architecture and the requirements — the key space and why a slot is a hash prefix rather than a modulus, the three numbers a partition carries, what a lease cannot tell you and where safety actually comes from, the two kinds of state and why the difference decides recovery, the three recovery layers and the resume-position trap between two of them, what a standby buys at each commit tier, why the seal is the last step of a handoff, split and merge as a lineage walk, the five things a server can say and why two of them must stay distinct, what the controller is not, why scaling runs on a clock, and the life of one write end to end. It carries no normative keywords and defines no requirement: it cites the numbered ones, 53 of them, and where the two disagree the requirement is right. Adds a handoff sequence diagram and a recovery-layer diagram. No requirement changed.
• 2026-08-21 (1.0.0): Draft revised after an expert-panel review, while still draft and with no consumers, so the requirement set moves without a major bump — which is what draft is for, and a door that closes the moment this goes stable. Adds signed partition maps verified before apply, so a delta relayed in a MOVED response cannot be a routing-hijack primitive; a movement cap and a canary stage on map publication, previously the highest-blast-radius operation with no brake; an operator force-takeover, fenced by the same epoch increment as an ordinary acquisition; declared checkpoint kinds with a normative incremental restore path and a bounded delta chain; a requirement that deletion actually runs; position tokens that do not compare across a lineage boundary; and replay equivalence promoted from prose to a requirement. Reconciles the global freeze with the post-seal rule they contradicted, and states the poison-record quarantine as an attack surface as well as a containment. Also makes the class the single declaration surface: the three commit tiers are defined normatively and every class must choose one with no default, since the tail between an acknowledgement and a durable segment is exposed to correlated loss and a default would decide that for a workload nobody examined; and failure-domain diversity between a primary and its standby is declared per class and enforced by placement, because a standby in its primary’s own domain satisfies every count-based requirement while surviving nothing. 139 requirements.
• 2026-08-21 (1.0.0): Initial publication as a draft. 115 requirements covering the slot key space and partition map, epoch-fenced ownership leases, checkpoint and journal recovery, input catch-up, standby replication, handoff and rollout, split and merge, the client routing contract, operator-governed placement, scheduled scaling, and observability.