--- id: 5 title: "A runtime for partitioned stateful services" version: "1.0.0" status: draft date: 2026-08-21 authors: - "Theo Zourzouvillys" tags: [architecture, reliability, scalability, infra, aws, consistency] summary: "A specification for services whose per-key state lives in memory on one instance at a time: a published partition map rather than a hash function, epoch-fenced ownership leases, checkpoint-journal-stream recovery, live handoff on rollout, and operator-governed placement." use_when: "Your service holds per-key state in memory on exactly one instance at a time, and rollout, scaling and placement must move that ownership deliberately rather than restart it." not_for: - "Stateless services, where a load balancer and an autoscaling group are the right answer and all of this is pure cost." - "One global state machine with no natural partition key — this moves ownership of a partition, it does not replicate a singleton." - "Durable transactional storage; this is a runtime for state recoverable by replay, not a database." - "Batch or analytics workloads, where a scheduler and a shuffle beat live ownership." requires: [] provides: - "partitioned-state" - "partition-ownership" - "live-state-handoff" - "operator-governed-placement" implements_notes: [4, 12, 13, 16, 25, 36, 37, 59, 60, 62, 65] supersedes: null superseded_by: null aliases: [] crossrefs: ZFN-4: "Why the placement controller must not keep its own state inside the service it places — the recovery tool cannot depend on what it recovers." ZFN-12: "The distinction this document builds on: the input stream is a journal, the per-partition write log is a journal, and neither is a queue." ZFN-13: "Where shedding belongs when a partition is over budget — push the failure back to the caller rather than absorbing it into a growing in-memory backlog." ZFN-16: "The reason the lease store is separate from the placement controller: a control-plane outage must freeze decisions, not stop serving." ZFN-25: "The position token returned on every write is that idea applied to a partition — a client that needs read-your-writes carries the position it saw." ZFN-36: "Why a checkpoint that has never been restored is not a checkpoint, and the restore canary is a requirement rather than a nicety." ZFN-37: "Ownership here is a lease with a TTL, an owner and a heartbeat — and the epoch is the fencing token that makes a stale holder harmless." ZFN-59: "Why a node decides its own lease has expired on the monotonic clock while the lease record's expiry is wall time." ZFN-60: "Handoff is that protocol made explicit: stop attracting work, drain, hand back what is in flight, release the lease, exit." ZFN-62: "How checkpoint and journal formats evolve across a rollout, given that two adjacent versions must read each other's state." ZFN-65: "The write path here is that pattern: append to the partition journal, apply in micro-batches, take sequence from the journal and never from the clock." ZBP-3: "Partitioning by tenant is the common case, and moving one tenant to dedicated hardware is how that document's isolation-tier requirement is satisfied for a stateful service." references: - id: redis-cluster title: "Redis Cluster Specification" url: https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/ abstract: "Specifies a fixed 16384-slot key space, slot-to-node assignment published to clients, and the MOVED/ASK redirection protocol by which a client discovers it addressed the wrong node. The closest widely deployed precedent for the routing contract in this document." - id: kinesis-resharding title: "Amazon Kinesis Data Streams — resharding, hash key ranges and explicit hash keys" url: https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-resharding.html abstract: "Describes how a stream maps a record to a shard by taking the MD5 of the partition key as a 128-bit integer, how shards own contiguous hash key ranges, and how splits and merges change that mapping while parent shards drain." - id: chubby title: "The Chubby lock service for loosely-coupled distributed systems" url: https://research.google/pubs/pub27897/ abstract: "Burrows, OSDI 2006. Introduces leases held by clients with a defined expiry and sequencers passed to downstream services so that a holder whose lease has silently expired cannot complete an operation. The origin of the fencing pattern used here." - id: fencing-tokens title: "How to do distributed locking" url: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html abstract: "Kleppmann, 2016. Shows why a lock without a monotonically increasing token is unsafe in the presence of process pauses, and why the resource being protected — not the lock service — must be the component that rejects a stale token." - id: flink-snapshots title: "Lightweight Asynchronous Snapshots for Distributed Dataflows" url: https://arxiv.org/abs/1506.08603 abstract: "Carbone et al., 2015. Describes taking consistent snapshots of operator state in a streaming dataflow without stopping the flow, and recovering by restoring a snapshot and replaying the source from the offsets it recorded." - id: asg-scale-in-protection title: "Amazon EC2 Auto Scaling — instance scale-in protection and lifecycle hooks" url: https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-lifecycle-hooks.html abstract: "Documents the mechanisms by which an instance can be excluded from an autoscaling group's own termination choices and by which termination can be held pending an external signal, which is what allows a placement controller rather than a scaling policy to decide which node dies." --- ## TL;DR This specifies how to run a service whose state for a given key is held in memory by exactly one instance at a time — and how to move that ownership without dropping the state on the floor. The shape of the answer: **the key space is divided into a fixed number of slots; a versioned partition map published by a control plane assigns slot ranges to partitions and partitions to nodes; ownership is a lease carrying a monotonically increasing epoch that fences every side effect; state is recovered from a checkpoint, then a journal, then catch-up on the input stream; and a rollout, a rebalance and a scale event are all the same operation — a handoff — differing only in what moves where.** The controlling decision is that **the partition map is data, not a function.** A consistent-hash ring computes an assignment so that no coordination is needed. Here coordination exists and is wanted, so the map is published instead of computed — which is what makes it possible to pin one customer to dedicated hardware, split a hot range at a chosen boundary, run different node sizes for different parts of the key space, and drain a node on a schedule. None of those is expressible as a hash function. The second decision is that **scaling is a plan a tool applies, not a reaction an autoscaler has.** Terminating a stateful node is not a capacity adjustment; it is a handoff that happens to end with one fewer node. An autoscaling policy cannot know which node is drainable, so it must not be the component that chooses. ## Applicability **Use this when** a service holds per-key state in memory — a counter, a rate-limit window, a session, a model, a materialised aggregate — that must be authoritative in one place at a time, and where rebuilding it from scratch on every deploy is too slow, too expensive, or too visible. **Do not use this when:** - **The service is stateless.** A load balancer and an autoscaling group solve this problem for free. Every requirement here is overhead you are paying for nothing. - **There is no natural partition key.** This document specifies how to move ownership of a partition between nodes. It does not specify how to replicate a single logical state machine that cannot be divided; that is a consensus problem, and you want a consensus library. - **The state is the system of record.** Everything here assumes state is recoverable by replaying a durable log. If losing the in-memory state means losing customer data, you need a database, and this runtime in front of it. - **The workload is batch.** If work can be scheduled, shuffled and retried without a live owner, a scheduler is simpler and cheaper. ## Scope and non-goals In scope: the slot key space and the partition map; ownership leases, epochs and fencing; checkpoint and journal formats and the recovery algorithm; catch-up from an ordered input stream; standby replication and promotion; handoff, rollout, split and merge; the client routing contract and its redirect semantics; the placement controller, node classes and placement constraints; scheduled scale-down and scale-up; and the observability that makes the whole thing debuggable. Out of scope: the application's own data model and applier logic; consensus for a non-partitionable singleton; transactions spanning partitions; cross-partition queries and joins; the durable system of record behind the runtime; and authentication and authorisation of the callers, which are assumed to be solved above this layer. ## Vocabulary - **Key** — the application-level string that identifies a unit of state. Frequently a tenant id. - **Slot** — the atom of the key space. A key belongs to exactly one slot, computed by hash, for the life of the system. Slots are numbered `0 .. 2^SLOT_BITS - 1`. - **Partition** — the unit of ownership, state, recovery and movement: a set of slot ranges, the state belonging to their keys, a journal and a checkpoint lineage. - **Generation** — a partition's lineage counter, incremented when it is created by a split or a merge. Identifies which stored state belongs to which shape of the partition. - **Epoch** — a partition's ownership counter, incremented on every change of primary. The fencing token. Never decreases, never reused. - **Node** — a process that can host partitions. Has a class, a capacity budget and a heartbeat. - **Class** — the shape of node a partition requires: size, local storage, and whether the node may be shared. A class marked exclusive hosts exactly one partition. - **Primary** — the node currently authoritative for a partition. Exactly one, per epoch. - **Standby** — a node holding a replica of a partition's state, applying behind the primary, eligible for promotion. Serves no writes. - **Partition map** — the versioned document assigning slot ranges to partitions and partitions to nodes. Published by the placement controller, cached by everyone, authoritative. - **Map version** — a monotonically increasing integer identifying one published map. Never reused. - **Placement controller** — the control plane. Decides the map; does not sit on the serving path. - **Lease store** — the strongly consistent store holding ownership records. Separate from the placement controller, and reachable by nodes without it. - **Journal** — the ordered, durable, per-partition log of state changes, addressed by LSN. - **LSN** — the journal's sequence number. Dense, gapless, per partition and generation. - **Checkpoint** — an immutable snapshot of a partition's state at a stated LSN and set of input positions. - **Input stream** — the external ordered, replayable log the partition folds into state. - **Input position** — the per-shard cursor into the input stream from which replay resumes. - **Derived state** — state that is a deterministic fold over the input stream alone. - **Originated state** — state created by requests made directly to this service, recoverable only from its own journal. - **Handoff** — moving primary ownership of a partition from one node to another. - **Seal** — the point in a handoff after which the outgoing primary can never accept another write. - **Lock shard** — an intra-process striping of a map across several mutexes to keep a hot path off a single lock. It has nothing to do with a partition, and is named here only because the same codebase will call both of them shards. The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as in RFC 2119 and RFC 8174. ## Architecture ```mermaid flowchart TB CL(["`client computes slot · caches map`"]) PC["`**Placement controller** decides the map · emits plans never on the serving path`"] LS[("`**Lease store** strongly consistent leases · epochs · map versions checkpoint pointers`")] N1["`**Node n_7a2c** · std.2x p_0002 PRIMARY epoch 73`"] N2["`**Node n_1f90** · std.2x p_0002 STANDBY`"] ST[("`**Object store** checkpoints · journal segments immutable · named by epoch and LSN`")] IN[/"`**Input stream** ordered per key · replayable shard = contiguous hash range`"/] CL -- "key → slot → partition" --> N1 CL -. "reads map, on change only" .-> LS PC -- "publishes map" --> LS N1 -- "acquire · renew · fence" --> LS N2 -- "acquire · renew · fence" --> LS N1 -- "writes" --> ST N2 -. "restores" .-> ST IN --> N1 N1 == "replication" ==> N2 N1 -. "heartbeat · capacity" .-> PC ``` Four properties do the work, and all four are structural rather than behavioural. **The map is published, not computed.** A consistent-hash ring exists so that independent parties can agree on an assignment without talking to each other. That is the right trade when there is no control plane. Here there is one, and it is wanted — so the assignment becomes a document the controller writes and everyone reads. Everything an operator wants from this system is a property of a document and impossible in a function: pinning a slot to dedicated hardware, choosing where a hot range splits, running one class of node for most of the key space and a larger class for part of it, draining a named node before a maintenance window. **The slot is the atom, and it is a hash prefix.** Taking the top bits of the hash rather than a modulus makes the key space refinable in place: doubling the number of slots splits every slot in two and moves no key across a partition boundary. A modulus would move nearly all of them. It also means a partition's keys occupy one contiguous range of the hash space, which is what lets the input stream's shards line up with partitions for free. **The epoch fences the storage, not the lock.** A lease tells a node it may serve. It cannot tell a node it has *stopped* serving, because the node may be paused, partitioned, or garbage-collecting at the moment it stops being true. So safety does not come from the lease; it comes from every durable side effect carrying the epoch it was produced under, and from the store rejecting a lower one. **The control plane is not on the serving path.** Nodes talk to the lease store directly, so the placement controller can be down, redeployed, or broken for an hour without a request failing. What stops is change: no rebalance, no rollout, no split. That is the correct thing to lose. ## How it works Everything below is explanation, not specification. It carries no normative keywords and defines no requirement; where it appears to disagree with *Normative requirements*, the numbered requirement is right and this is loose. It exists because the requirement list that follows is a wall, and a reader who meets the wall without the model will build the letter of it and miss the shape. Read it once, in order. Each part assumes the one before it. ### The key space, and why a slot is a hash prefix A key — usually a tenant id — is not assigned to a node. It is assigned to a **slot**, and slots are assigned to partitions, and partitions to nodes. Two indirections, and both earn their keep. The slot comes from the top bits of a 128-bit hash of the key (ZBP-5-R1). With `SLOT_BITS = 12` there are 4096 slots and the slot is, quite literally, the first three hex digits of the digest: ``` t_whale md5 = 55b0c21b24e83694bdafcb69a8da3391 ^^^ 0x55b = 1371 → slot 1371 ``` The choice of *prefix* rather than *modulus* is the single decision the rest of the key-space design rests on, and it is invisible until you need it. Consider raising the slot count from 4096 to 8192. With a prefix, slot 1371 becomes slots 2742 and 2743, and every key that was in 1371 is now in one of those two — because taking one more bit off the front of a number cannot move it outside the range its first twelve bits already described. A range `[0, 1370]` becomes `[0, 2741]` mechanically. No key changes partition, no state moves, and the whole operation is a rewrite of range bounds in a document (ZBP-5-R3). With `md5 % 4096`, raising the count to 8192 rehashes the entire key space. Nearly every key lands somewhere new, which means nearly every key's state has to move. One is a config change; the other is a migration. They differ by which end of the number you took. The second consequence is that a slot is a contiguous interval of hash space, and so is a partition, being a set of slot ranges. This is what makes the input stream line up for free — a stream that places records by hashing the same key into the same 128-bit space puts a partition's records in a computable, contiguous set of shards, with nothing to configure. It is also why there is no key-level pin (ZBP-5-R4): a partition that is not a hash range breaks stream alignment, split, and merge simultaneously, and it does so quietly. A very large tenant gets its own *slot*, not its own key exception. ### Why the map is published rather than computed A consistent-hash ring answers "who owns this key?" with arithmetic. That is exactly right when there is no coordinator — every party computes the same answer without talking to anyone. Here there is a coordinator, and it is wanted. So the assignment is a **document** the controller writes and everyone else reads (ZBP-5-R5). The document can express things arithmetic cannot: - this one slot lives alone on a large node, because that tenant is worth more than the disruption; - this hot range splits *here*, at a boundary someone chose after looking at the traffic; - most of the key space runs on small nodes and this part runs on big ones; - this named node is draining before a maintenance window and holds nothing by 02:00. Every one of those is a property of a document. None is expressible as a hash function, and the usual attempt — weighted virtual nodes — is a worse map with no reader. The cost of publishing is that the document is now a trust boundary and a blast radius. It is signed by the controller and verified before it is applied, from any source (ZBP-5-R10, ZBP-5-R11), which is what makes it safe for a peer to hand a client a delta in a redirect. And because a map that is perfectly well-formed can still name the wrong endpoints — coverage validation proves shape, not correctness — publication is rate-limited by a movement cap and staged through a canary (ZBP-5-R119, ZBP-5-R120). It is the highest-blast-radius write in the system: one document, the whole fleet, immediately. ### A partition, and the three numbers it carries A partition is the unit of ownership, state, recovery and movement. It carries three numbers, and confusing them is a common early bug because all three look like versions. | number | changes when | answers | |---|---|---| | **generation** | the partition is created by a split or a merge | *which shape of the key space does this state belong to?* | | **epoch** | ownership moves to a different primary | *is the writer still the owner?* | | **LSN** | every journal record | *how far through the log is this?* | Generation is lineage. A split produces children at a higher generation whose stored state lives under their own prefix, and whose recovery walks back to the parent's. Epoch is the fencing token, and increases on every ownership change including a split (ZBP-5-R16). LSN is position within one `(partition, generation)`, dense and gapless, and it restarts for a child — which is exactly why a position token is not comparable across a lineage boundary. ### Ownership, and the thing a lease cannot tell you Ownership is a lease: a record naming an owner, with an expiry, renewed on a timer. A lease is very good at telling a node it *may* serve. It is incapable of telling a node it has *stopped* serving, and this is not an implementation quality issue — it is structural. The moment a node stops being the owner is precisely the moment it is most likely to be unable to find out: paused by a long garbage collection, partitioned from the network, descheduled, or simply slow. Any design where safety depends on the old owner noticing in time is a design that has assumed away its own hardest case. So safety does not come from the lease. It comes from two things underneath it. **The node fences itself, locally, on the monotonic clock.** When a renewal is acknowledged, the node records `monotonic_now() + ttl − margin` and refuses to admit work past that instant (ZBP-5-R17, ZBP-5-R18). This needs no network, which is the point: it still works during the pause that caused the problem. It uses the monotonic clock because wall time jumps, slews and runs backwards, and a lease decision made on wall time is a bug waiting for an NTP correction. **The storage rejects the stale writer.** Every durable write carries the epoch it was produced under, and the store refuses anything below the record's current epoch (ZBP-5-R20). This is the part that actually holds. A zombie primary that missed both its renewal and its own self-check is still harmless, because the thing it is trying to corrupt is what turns it away: ``` n_7a2c FENCE at epoch 73 → ACCEPT (still the owner) n_1f90 ACQUIRE at epoch 74 → ACCEPT (ownership moves) n_7a2c FENCE at epoch 73 → REJECT_STALE_EPOCH (harmless, and it need not know) n_7a2c ACQUIRE at epoch 74 → REJECT_STALE_EPOCH (cannot climb back) ``` Object storage plays the same game differently, because it has no conditional writes worth relying on. Objects are immutable and named by partition, generation, epoch and LSN (ZBP-5-R21), so a zombie's write does not overwrite anything — it lands at a key nothing references and is eventually collected. What makes a checkpoint *current* is the pointer in the lease store, and moving that pointer is the epoch-fenced operation (ZBP-5-R22). A checkpoint becomes real when the pointer moves, never when the bytes land. ### Two kinds of state, and why the difference decides recovery State comes in two flavours, and a partition declares which it has (ZBP-5-R29). **Derived state** is a deterministic fold over the input stream. A counter, a rolling window, an aggregate. Given the stream and a starting point, it can always be rebuilt, because the stream is the truth and the state is just a cached opinion about it. Such a partition needs no journal at all (ZBP-5-R30): a checkpoint records the input positions it is consistent with, and recovery is restore plus replay from those positions. **Originated state** is created by requests made to this service. Nothing upstream knows it exists, so nothing upstream can replay it. It has to be written to a journal before it is applied, and that journal is the only record of it (ZBP-5-R31). Most real services have both, which is where the trap is. If the two sets are genuinely disjoint and neither applier reads the other, they can recover independently. The moment an originated write reads derived state — a write whose effect depends on the current counter — they are one state machine, and recovering them on two independent paths produces two different answers. That case has to be fully journalled (ZBP-5-R32), and the honest version of this rule is: if you have to think about whether they interact, they interact. Underneath both sits the requirement that makes replay mean anything: **the applier is deterministic** (ZBP-5-R33). No wall-clock reads, no random values, no iteration over an unordered map where order changes the result, no calls out to another service. Anything that cannot be recomputed is captured into the record when it is written and read back on replay (ZBP-5-R34), never recomputed. Get this wrong and there is no error, ever. The service is correct until the day it recovers, and then it is quietly, plausibly different — a counter off by a little, a window that starts somewhere else, a decision that would not have been made. That is why replay equivalence is a continuous test rather than a design note. ### Recovery, in three layers ```mermaid flowchart TB CK["`**1 · checkpoint** state as of LSN 918233 plus the input positions it is consistent with`"] J["`**2 · journal** replay 918234 → 961887 the TAIL carries the marks`"] S["`**3 · input stream** resume AFTER the tail's marks drain parent shards first`"] R(["ready — lag inside the class bound"]) CK --> J --> S --> R ``` Three layers, because each covers a window the next cannot. The checkpoint is cheap to load and always behind. The journal covers checkpoint-to-seal. The stream covers everything after, up to live. The trap sits between layers two and three, and it is the most commonly implemented bug in this whole document. A checkpoint records the input positions it is consistent with (ZBP-5-R36). The journal then advances past it, and every journal record carries the positions consumed at that point. After replaying the journal to its tail, the place to resume the stream is **the tail's marks, not the checkpoint's**. Take the checkpoint's, and every input record between the checkpoint and the journal tail is applied a second time. Nothing errors. The counters are simply too high, by an amount proportional to how long it had been since the last checkpoint — which is to say, worst exactly when the incident was worst. Two details make the layers work. The checkpoint is ordered by slot (ZBP-5-R38), so a child of a split can fetch only the parts overlapping its own ranges instead of loading the parent whole. And a class declares whether its checkpoints are full or a base plus a bounded delta chain (ZBP-5-R39, ZBP-5-R40) — incremental makes frequent checkpointing affordable for a large partition, at the price of a second lineage to walk before anything is deleted. The stream side has its own two. Positions are tracked per contributing shard, never as one cursor (ZBP-5-R53), because a stream reshard leaves a partition reading several. And a parent shard is drained to its end before either child is read (ZBP-5-R54) — read a child early and records inside a single key arrive out of order, which is the one ordering guarantee everything else assumes. Finally, the quiet failure that no other signal catches: a checkpoint older than the stream's retention means the partition cannot be recovered *at all*, and nothing about a healthy running service reveals it. It is found at the exact moment it must not be found. Hence an alarm on the **margin** between checkpoint age and configured retention (ZBP-5-R47), and a restore exercised continuously against real checkpoints rather than tested once. ### Standbys, and what a standby actually buys A standby holds a replica and is eligible for promotion. In a journalled partition it consumes the primary's journal; in a derived one it needs no link to the primary at all and simply reads the same input stream (ZBP-5-R63). What a standby buys depends entirely on the commit tier the class declares, and the tiers are exact (ZBP-5-R67): - **LOCAL** — acknowledged when applied in memory on the primary. Fastest. A crash loses the tail. - **STANDBY** — acknowledged when the required standbys have applied it. Survives one node dying. - **DURABLE** — acknowledged when the record is in a store that survives losing the whole fleet. There is deliberately no default (ZBP-5-R68). The three differ in what a caller loses, and a default decides that for a workload nobody examined. Below DURABLE there is a window — acknowledged, not yet in a durable segment — exposed to correlated loss, and that window is monitored rather than left as an accident of the batching interval. The related trap is that "a standby exists" reads as durability and often is not. A standby with unbounded lag is a second copy of an old state, which is why lag is a metric rather than an inference from liveness. And a standby in its primary's own failure domain satisfies every count-based rule while surviving nothing at all — so diversity is declared per class and enforced by placement (ZBP-5-R61), with "none required" being a thing a class says out loud rather than a thing it does by accident. ### Handoff, and why the seal is the last step Handoff is how ownership moves, and rollout, rebalance and scale-down are all just handoff with a different reason attached. It has exactly one point of no return — the **seal** (ZBP-5-R74). ```mermaid sequenceDiagram autonumber participant C as controller participant O as outgoing primary participant I as incoming node participant L as lease store C->>I: PrepareHandoff(deadline) I->>L: read checkpoint pointer I->>I: restore, replay, catch up O-->>I: replication I->>C: SYNCED Note over C,L: everything above is free to abandon O->>O: stop admitting, append SEAL O->>L: FENCE sealed_lsn Note over O,L: THE SEAL — forward only I->>I: consume to sealed_lsn I->>L: ACQUIRE at epoch+1 I->>C: ready C->>L: publish new map O->>O: discard state, no flush ``` Before the seal, abandoning costs nothing: the outgoing primary never stopped serving and the map was never changed (ZBP-5-R75). After it, the partition can only go forward — either the incoming node finishes, or some node recovers the partition from storage under a new epoch. The outgoing primary does not resume, ever, including when the handoff fails (ZBP-5-R76). That asymmetry is the whole design. Everything expensive — restore, replay, catch-up, which is seconds to minutes — happens on the safe side of the line (ZBP-5-R78). The seal itself is one conditional write. The partition is unavailable only from the seal to the moment the new map names its primary (ZBP-5-R79), and that window is the number the design exists to make small, so it is measured on every handoff rather than estimated once. Two things follow that are easy to get wrong. A rollout is a bounded sequence of these, one partition at a time, never a fleet restart and never delegated to a platform rolling deployment that picks its own order and starts replacements before anything drains (ZBP-5-R80). And the graceful path is the optimisation, not the guarantee — a node has to survive being killed with no warning (ZBP-5-R87), which is why cold recovery is the path that gets exercised most. Relatedly, the shutdown phase deadlines are summed and checked against the platform's kill budget *at boot* (ZBP-5-R85): if they overrun, the last phase never runs, and the last phase is the one that accounts for what was lost — so the loss becomes not merely unrecovered but unrecorded, and the shutdown looks clean. ### Split and merge as a lineage walk A split divides a partition's slot ranges at slot boundaries; a merge combines adjacent ones. Both produce children at a higher generation with new ids, and both work the same way: children restore from their parents' state filtered to their own ranges, every parent seals, then the children start (ZBP-5-R92, ZBP-5-R93). The property that makes this possible at all is narrow and worth stating plainly: ordering is guaranteed **per key**, never across keys (ZBP-5-R94). If cross-key ordering were promised, no partition could ever be divided, because dividing it means two independent appliers with no shared sequence. Everything in this document that looks like flexibility is bought with that one restriction. The floor is one slot (ZBP-5-R96). A key hotter than a single node cannot be relieved by splitting, because a key is one point in the hash space — it is one slot, in one partition, on one node, and on the input side it is one point in the stream's hash space and therefore one shard's worth of ingest. That ceiling is a key-design problem, answered by composing the key with a sub-key and explicitly giving up ordering between sub-keys. It is not a placement problem, which is why people look for it in the placement layer and do not find it. ### Routing, and the five things a server can say The client computes the slot itself and resolves it against a cached map (ZBP-5-R97). There is no router and no per-request lookup, because a lookup service on the hot path is the coupling the whole control-plane separation exists to avoid. Requests carry the key, the slot, and the map version the client resolved against — and that last field is what makes the answers unambiguous: | the server… | versions | answer | why | |---|---|---|---| | owns it, ready | any | `SERVE` | a correct answer is correct regardless of what map produced it | | owns it, not ready | any | `NOT_READY` | it is restoring *here*; it is not somewhere else | | does not own it | server newer | `MOVED` | the client is behind — here is the assignment | | does not own it | equal | `MISROUTED` | same map, different answer: the *arithmetic* disagrees | | does not own it | server older | `STALE_MAP` | the server is behind; the client should wait, not be redirected | The last two rows are the interesting ones. Collapse them into one "you're wrong, try elsewhere" and two nodes with disagreeing maps will redirect a client back and forth indefinitely, at full speed, with no errors logged anywhere — the failure looks like a latency problem. The rule that prevents it is that a server never issues a redirect derived from a map older than the client's (ZBP-5-R102). `MISROUTED` deserves its own code because it is not transient. It means client and server do not compute the same slot from the same key — a different hash, a different `SLOT_BITS`, a different string encoding. Retrying cannot fix a disagreement about arithmetic, so it is alarmed rather than retried (ZBP-5-R105). Finally, every mutating response carries the position it was applied at (ZBP-5-R107), so a caller that needs to read its own write can demand that position on the read rather than guessing at replication lag. ### Placement, and what the controller is not The controller decides the map. That is all it does. It is never called on the serving path, by clients or by nodes (ZBP-5-R109), and the data plane keeps serving from the last published map while it is down (ZBP-5-R110). A controller outage freezes *change* — no rebalance, no rollout, no split — and changes nothing about serving. That is the correct thing to lose, and it is only true because the lease store is a separate component that nodes reach directly. Placement is a constrained optimisation, and the order of the objectives matters more than the optimisation does (ZBP-5-R115): 1. **satisfy constraints** — class, exclusivity, failure domains. Not negotiable. 2. **minimise movement** — every move is a handoff, and a handoff is an unavailability window. 3. **balance load** — last. Put balance above movement and the controller thrashes: whenever load is near-even, tiny differences justify moves, each move disturbs load, and it never settles. A controller that optimises balance continuously serves worse than one that does nothing. Decisions come out as **plans** — renderable, dry-runnable, applied as a separate step, with policy deciding which kinds apply automatically and which wait for a human. This is deliberately the shape of an infrastructure tool rather than a control loop, because the interesting operations here are rare, consequential, and worth looking at. ### Scaling on a clock, not on a signal Autoscaling a stateful fleet does not work, and not because the arithmetic is hard. A scale-in policy picks a victim by CPU, and CPU is uncorrelated with drainability — it will choose the node holding the largest partition about as often as any other. Termination here is not a capacity adjustment; it is a handoff that happens to end with one fewer node, so the component that knows about handoffs has to be the component that chooses (ZBP-5-R126). What replaces it is scheduled, and shaped around the fact that state takes time to move: **Evening, into a known trough.** Merge under-utilised partitions, rewrite checkpoints to drop expired state, retire the drained nodes. Compaction is abandonable at any point — an interrupted one leaves the previous checkpoint live and nothing else changed. **Morning, ahead of a known ramp.** Start at `T_ramp − lead`, where `lead` comes from *measured* restore and catch-up times, not from a guess (ZBP-5-R128). Add nodes, split partitions, and let the children warm up while the parents are still serving. The map flips last, when every child is synced (ZBP-5-R124) — and if warming is not finished by `T_ramp`, it does not flip at all, because serving a ramp from a warm parent beats serving it from a cold child. That ordering — warm everything, then flip one document — is what turns a scale event from a thirty second outage into a few milliseconds. ### The life of one write Pulling it together, for a journalled partition at the `STANDBY` commit tier: 1. The client hashes the key, takes the top twelve bits, looks up slot 1789 in its cached map, and sends the request to `p_0002`'s primary with `map_version: 1487`. 2. The node recomputes the slot and agrees. It checks it holds the lease at its believed epoch and that its monotonic deadline has not passed. It checks the partition is `SERVING` and inside its capacity budget. 3. The applier produces a record. It is appended to the journal at LSN 918408 — sequence from the journal, never from a clock — carrying the epoch and the input positions consumed. 4. The record streams to the standby. The standby applies it and acknowledges. 5. Only now is the write acknowledged to the client, with `X-Partition-Position: p_0002:74:918408`. 6. In the background, journal records batch into a segment in object storage; periodically a checkpoint is written and the pointer flipped under the epoch. 7. Some hours later the partition is handed to a new node during a rollout. The new node loads the checkpoint, replays the journal to the seal, resumes the stream from the tail's marks, and reaches the same state — because the applier is deterministic and everything that was not was captured into the records at step 3. Step 7 is the one that has to be true, and every requirement in this document is in service of it. ## Normative requirements ### The key space and the partition map - **ZBP-5-R1** The key space MUST be divided into a fixed number of slots, `2^SLOT_BITS`. A key's slot MUST be the most significant `SLOT_BITS` bits of a 128-bit hash of the key's UTF-8 encoding, interpreted big-endian. - **ZBP-5-R2** The hash function MUST be fixed for the life of the system and named in the map. It MUST NOT be a hash whose output varies between processes, runs, or library versions — a seeded or runtime-randomised hash is disqualified regardless of its statistical quality. - **ZBP-5-R3** `SLOT_BITS` MAY be increased and MUST NOT be decreased. An increase is a pure refinement: slot `s` at `b` bits becomes the range `[s << d, (s << d) | (2^d − 1)]` at `b + d` bits, and no key changes partition. Implementations MUST perform the increase by rewriting slot ranges in the map, and MUST NOT move state to do it. - **ZBP-5-R4** The slot MUST be the atom of assignment. A partition's key set MUST be expressible entirely as a set of slot ranges. There MUST NOT be a key-level pin, override, or exception, because a partition that is not a hash range breaks stream alignment, split, and merge at once. - **ZBP-5-R5** The partition map MUST be a single versioned document that assigns every slot in `0 .. 2^SLOT_BITS − 1` to exactly one partition. Total coverage and non-overlap MUST be validated before a map is published, and a map failing validation MUST NOT be published. - **ZBP-5-R6** The map version MUST be a monotonically increasing integer, MUST NOT be reused, and MUST change whenever any assignment, endpoint, epoch, or partition state in the document changes. - **ZBP-5-R7** Every partition entry MUST carry its id, generation, epoch, class, slot ranges, primary endpoint and state, and each standby's endpoint and state. Slot ranges MUST be inclusive `[low, high]` pairs, sorted and non-overlapping. - **ZBP-5-R8** The map MUST be readable by nodes and by clients without calling the placement controller. - **ZBP-5-R9** Every published map version MUST remain addressable for at least the longest client cache lifetime, so that a client presenting an old version can be answered with a delta rather than a full document. - **ZBP-5-R10** The placement controller MUST sign every published map version with a key held only by the controller. The signature MUST cover the map version, every slot assignment, and every endpoint in the document, and a delta MUST be signed as well as a full map. - **ZBP-5-R11** Clients and nodes MUST verify that signature before applying a map or a delta, from whatever source it arrived. A delta relayed by a peer is acceptable precisely because authenticity rests on the signature rather than on the transport or on trusting the peer that sent it. - **ZBP-5-R12** A map or delta that fails verification MUST be rejected, counted, and alarmed as a correctness fault, and MUST NOT be applied even when its version is newer than the cached one. The signing key MUST be rotatable with overlapping validity, and verifiers MUST accept any key currently within its validity window. - **ZBP-5-R13** Partition ids MUST be opaque and MUST NOT encode the slot range they currently hold. The range changes; the id must not. ### Ownership, leases, and fencing - **ZBP-5-R14** Primary ownership MUST be a lease with a stated expiry, a named owner, and a renewal interval. An ownership record without an expiry MUST NOT exist. - **ZBP-5-R15** The lease record MUST carry the partition's epoch, and acquisition MUST increment that epoch in the same conditional write that claims the lease. - **ZBP-5-R16** An epoch MUST increase monotonically per partition, MUST NOT be reused, and MUST NOT reset when a partition is created by a split or a merge. - **ZBP-5-R17** A node MUST determine that its own lease has expired from a monotonic clock reading taken when its last renewal was acknowledged. It MUST NOT compare local wall time to the expiry recorded in the store. - **ZBP-5-R18** A node MUST stop accepting work for a partition before that lease can have expired at the store, allowing a margin of at least the maximum renewal round-trip plus the maximum tolerated clock error, and MUST be able to do so without contacting any other component. - **ZBP-5-R19** Every write to shared storage on behalf of a partition MUST carry the epoch under which it was produced. - **ZBP-5-R20** The lease store MUST reject any conditional write carrying an epoch lower than the record's current epoch. This rejection is the fence: safety comes from the resource refusing a stale token, not from the lock being held correctly. - **ZBP-5-R21** Objects in the object store MUST be immutable and uniquely named by at least partition, generation, epoch, and LSN, so that a write by a stale primary lands at an unreferenced key rather than overwriting a live one. - **ZBP-5-R22** The pointer naming a partition's current checkpoint MUST live in the lease store and MUST be advanced by a conditional write fenced on the epoch. A checkpoint becomes current when the pointer moves, never when the object lands. - **ZBP-5-R23** A node MUST NOT be recorded as ready for a partition it has not finished restoring. Holding the lease and being ready to serve MUST be distinct states. - **ZBP-5-R24** Losing the lease MUST cause the node to discard the partition's in-memory state without attempting to flush, checkpoint, or replicate it. A stale primary's state is not a contribution. - **ZBP-5-R25** The lease store MUST NOT be the placement controller, and MUST remain reachable by nodes when the placement controller is unavailable. - **ZBP-5-R26** An operator force-takeover MUST exist that acquires a partition at a higher epoch without waiting for the current lease to expire. Waiting out a TTL is not an acceptable only-option when a node is known to be gone. - **ZBP-5-R27** A force-takeover MUST advance the epoch through the same conditional write as an ordinary acquisition. There MUST NOT be any path that takes ownership without advancing the epoch, because the epoch is the only thing that makes the displaced node harmless. - **ZBP-5-R28** A force-takeover MUST record the operator, the time, and a stated reason, and MUST NOT be reachable by automated placement. It is a human override of a safety property, and an automated caller that can invoke it has removed the property. ### State, checkpoints, and the journal - **ZBP-5-R29** Each partition MUST declare a recovery mode of DERIVED, JOURNALED, or MIXED, and the mode MUST be recorded in the map. - **ZBP-5-R30** In DERIVED mode all state MUST be a deterministic fold over the input stream, there MUST be no journal, and recovery MUST be a checkpoint restore followed by input replay from the positions the checkpoint records. - **ZBP-5-R31** In JOURNALED mode every state change — originated and input-derived alike — MUST be appended to the journal before it is applied, and the input positions consumed MUST be recorded on the journal record that consumed them. There MUST be exactly one write path into state. - **ZBP-5-R32** MIXED mode MUST be used only where the derived and originated state sets are disjoint and neither applier reads the other's state. Where they interact, JOURNALED MUST be used. - **ZBP-5-R33** The applier MUST be deterministic given a state and a record. It MUST NOT read wall time, generate random values, depend on the iteration order of an unordered collection, or call an external service. - **ZBP-5-R34** Any value that cannot be recomputed deterministically MUST be captured into the journal record when the record is written and read back on replay. It MUST NOT be recomputed during recovery. - **ZBP-5-R35** The LSN MUST be dense and gapless within a `(partition, generation)`, and MUST be assigned by the journal rather than derived from any clock. - **ZBP-5-R36** A checkpoint MUST record the LSN and the complete set of input positions it is consistent with, its format version, its byte length, and a content digest. - **ZBP-5-R37** A checkpoint MUST be verified against its digest on restore, and a mismatch MUST be treated as the checkpoint not existing rather than as an error to surface and stop on. - **ZBP-5-R38** The checkpoint format MUST be ordered by slot, or otherwise permit extracting the state for a slot range without materialising the whole checkpoint. Split cannot be implemented without this property. - **ZBP-5-R39** A class MUST declare its checkpoint kind as full or incremental, and the kind MUST be recorded in every checkpoint manifest. It MUST NOT be inferred from the presence or absence of deltas. - **ZBP-5-R40** An incremental checkpoint MUST consist of one full base plus an ordered chain of deltas, each naming the base and the delta immediately preceding it. Restore MUST apply the base and then every delta in order, and MUST verify the digest of each; a mismatch anywhere in the chain MUST be treated as the entire chain being absent. - **ZBP-5-R41** A delta chain MUST have a stated maximum length and MUST be compacted into a new base before that bound is reached, so that restore time has a ceiling rather than growing until somebody notices. The bound MUST be derived from the shutdown budget of ZBP-5-R85. - **ZBP-5-R42** A base MUST NOT be deleted while any delta depending on it is still reachable by a live partition's restore path, under the same lineage rule as ordinary checkpoints. An incremental chain adds a second lineage, and both MUST be walked before anything is deleted. - **ZBP-5-R43** Checkpoint and journal formats MUST be versioned, and a node MUST refuse to restore a format version it does not fully understand rather than interpreting it partially. - **ZBP-5-R44** Adjacent releases MUST be able to read each other's checkpoint and journal formats. A format change MUST be shipped as expand, migrate, contract across at least three releases, because a rollout and a rollback both run two versions at once. - **ZBP-5-R45** Journal segments and checkpoints MUST NOT be deleted on age alone. Deletion MUST be driven by the map's lineage records and MUST retain everything reachable by the recovery path of any live partition, including a child that has not yet taken a checkpoint of its own. - **ZBP-5-R46** Deletion MUST actually run, on a schedule, and the volume of retained-but-unreachable objects MUST be monitored. A rule that forbids deleting the wrong thing is not a retention policy; without a collector that runs, storage grows without bound and the prohibition reads as one. - **ZBP-5-R47** The age of the newest usable checkpoint MUST be monitored against the input stream's retention window, and an alarm MUST fire while the remaining margin is still large enough to act on. A checkpoint older than retention means the partition cannot be recovered at all, and nothing else in the system reports that. - **ZBP-5-R48** Restoring a checkpoint into a running process MUST be exercised continuously against real production checkpoints, not only in tests. ### The input stream and catch-up - **ZBP-5-R49** A partition MUST publish the earliest instant its retained state and journal can answer about. A query whose window reaches before that instant MUST be reported as incomplete rather than answered with a number that silently omits the missing history. - **ZBP-5-R50** The input stream MUST be ordered per key and replayable from a stated position. Ordering across keys MUST NOT be relied upon anywhere in the system. - **ZBP-5-R51** The stream's shard boundaries MUST be ranges over the same hash space as the slots, and producers MUST use the service key as the stream's partition key, so that a partition's records occupy a contiguous and computable set of shards. - **ZBP-5-R52** Where the stream's partition key cannot be the service key, the producer MUST set an explicit hash key equal to the low hash of the record's slot. This is an escape hatch for a pre-existing producer, not the default arrangement. - **ZBP-5-R53** A partition MUST track an input position per contributing shard. A single scalar cursor MUST NOT be used, because a partition reads from several shards after a stream reshard. - **ZBP-5-R54** During catch-up a reader MUST drain a parent shard to its end before reading either child shard. Reading a child while its parent still has records reorders records within a key. - **ZBP-5-R55** Every input record MUST carry a stable id, and ingest MUST deduplicate on it. During recovery the same record arrives on more than one feed — the live path, the stream tail, and an archive seed — and a record that cannot be deduplicated double-counts the moment a second feed delivers it. - **ZBP-5-R56** The deduplication horizon MUST be exactly the retention horizon, so that a duplicate is recognised for precisely as long as the original is retained. A separate, shorter horizon is a second number to tune and a second way for the two to disagree. - **ZBP-5-R57** A partition MUST NOT admit traffic until catch-up has reached a stated freshness bound, expressed both as position lag and as record age. The bound MUST be configurable per class and MUST be reported, so that "ready" names something an operator chose. - **ZBP-5-R58** Records for slots the partition does not own MUST be discarded on read and counted. They MUST NOT be applied, and MUST NOT be treated as an error. - **ZBP-5-R59** A record that causes the applier to fail MUST NOT be retried indefinitely. After a bounded number of attempts across a bounded number of nodes, the partition MUST enter a quarantined state and cease to be placed. A poison record whose only limit is the restart loop will crash every node the partition is assigned to, in turn, until the fleet is gone. ### Standbys and replication - **ZBP-5-R60** A class MUST be able to require a minimum number of synced standbys, and a partition MUST NOT be reported healthy while below it. - **ZBP-5-R61** A class MUST declare the failure-domain diversity required between a primary and each synced standby, and a class that requires none MUST say so explicitly. A standby placed in its primary's failure domain satisfies every count-based requirement while surviving nothing, and the only difference between that and a deliberate choice is whether it was written down. - **ZBP-5-R62** A partition whose standby placement does not meet its class's declared diversity MUST NOT be reported healthy, by the same rule as one below its standby count. - **ZBP-5-R63** In JOURNALED mode a standby MUST consume the primary's journal. In DERIVED mode a standby MUST read the input stream independently, and no replication link is required. - **ZBP-5-R64** A standby MUST report its applied position, and the lag of every standby MUST be exposed as a metric rather than inferred from its liveness. - **ZBP-5-R65** A standby MUST NOT accept writes. It MUST NOT serve reads unless the request explicitly opts in to a stated staleness bound, and every such read MUST return the position at which it was served. - **ZBP-5-R66** The commit policy — whether a write is acknowledged on local apply, on standby acknowledgement, or on durable persistence — MUST be an explicit per-class setting. - **ZBP-5-R67** The commit tiers MUST be defined exactly as: `LOCAL`, acknowledged once applied in memory on the primary; `STANDBY`, acknowledged once at least the class's required number of standbys have applied it; and `DURABLE`, acknowledged once the record is held by a store that survives the loss of the entire node fleet. A tier that is not one of these three MUST NOT be offered. - **ZBP-5-R68** A class MUST declare its commit tier and MUST NOT start without one. There is deliberately no default: the three tiers differ in what a caller loses, and a default would decide that for a workload nobody examined. - **ZBP-5-R69** The tier a write actually achieved MUST be reported on the response. A caller that asked for `DURABLE` and got `LOCAL` because the class was misconfigured MUST be able to tell. - **ZBP-5-R70** Below `DURABLE`, the journal tail — records acknowledged but not yet in a durable segment — is exposed to correlated loss of the acknowledging nodes. The count and the age of the oldest such record MUST be monitored, and the exposure MUST be stated in the class rather than left as a property of the batching interval. - **ZBP-5-R71** Where the commit policy requires standby acknowledgement, the behaviour when no standby remains MUST be configured explicitly as either continue-and-alarm or refuse-writes. It MUST NOT be whatever the code happens to do. - **ZBP-5-R72** A standby MUST verify that the replication stream it consumes carries the epoch it believes the primary holds, and MUST stop consuming and re-resolve ownership if it does not. - **ZBP-5-R73** Promotion MUST increment the epoch through the lease store before the promoted node accepts its first write. ### Handoff and rollout - **ZBP-5-R74** A handoff MUST have a single point of no return — the seal — after which the outgoing primary MUST NOT accept another write for that partition under any circumstance, including the handoff subsequently failing. - **ZBP-5-R75** Before the seal, an aborted handoff MUST leave the outgoing primary serving and the published map unchanged. - **ZBP-5-R76** After the seal, the partition MUST only move forward: either the incoming primary completes promotion, or some node recovers the partition from storage under a new epoch. The outgoing primary MUST NOT resume. - **ZBP-5-R77** Requests in flight at the outgoing primary that have not committed by the seal MUST be rejected with a retryable outcome carrying the new assignment. They MUST NOT be reported as succeeded, and MUST NOT be failed permanently. - **ZBP-5-R78** The incoming primary MUST have restored and caught up to its class's freshness bound before the seal. The seal MUST be the last and cheapest step of a handoff, never the first. - **ZBP-5-R79** A map version naming a new primary MUST NOT be published before that node has reported ready under its new epoch. - **ZBP-5-R80** A rollout MUST be expressed as a sequence of handoffs with bounded concurrency. It MUST NOT be expressed as a fleet restart or delegated to a platform-driven rolling deployment that chooses its own order and its own timing. - **ZBP-5-R81** The number of partitions in handoff simultaneously MUST be bounded, both per node and globally. - **ZBP-5-R82** Every handoff MUST carry a deadline. On expiry it MUST abort if the seal has not happened, or complete by recovery from storage if it has. It MUST NOT wait indefinitely. - **ZBP-5-R83** Receiving a termination signal MUST initiate handoff of every partition the node hosts, and the platform's shutdown grace period MUST exceed the worst-case handoff time for a fully loaded node. - **ZBP-5-R84** Shutdown MUST proceed in order: stop being chosen, end unbounded streams with a stated reason, stop accepting new calls, then settle bounded background work. Run out of order the phases undo each other — a listener stopped before readiness is withdrawn drops work that was already routed here. - **ZBP-5-R85** The sum of every shutdown phase deadline MUST be validated at process start against the platform's termination budget, and a process whose phases cannot fit MUST refuse to boot. If the phases overrun, the final phase never runs, and the final phase is the one that accounts for what was lost — so the loss becomes not merely unrecovered but unrecorded, and the shutdown looks clean. - **ZBP-5-R86** Work that does not complete within the bounded final phase MUST be written to durable object storage in the same record shape the partition's durable sink already uses, so that the reader which consumes that sink's archive consumes the spilled object unchanged. A recovery path that needs its own parser is a recovery path nobody runs. - **ZBP-5-R87** A node MUST survive being killed without notice. Graceful handoff is the optimisation; recovery from checkpoint, journal, and stream is the requirement, and MUST be the path exercised most often in testing. - **ZBP-5-R88** A rollback MUST be a rollout in the opposite direction under the same format-compatibility rule, and MUST NOT require an older binary to read a checkpoint written in a format version it does not understand. ### Split, merge, and redistribution - **ZBP-5-R89** A split MUST divide a partition's slot ranges at slot boundaries into two or more children, each with a new partition id and a generation greater than the parent's. A child covering zero slots MUST NOT be produced. - **ZBP-5-R90** A merge MUST combine partitions whose slot ranges are adjacent into one child with a new id and a generation greater than every parent's. - **ZBP-5-R91** A child MUST record its parents in the map, and that lineage MUST be retained for as long as the child depends on any parent's stored state. - **ZBP-5-R92** A child MUST recover by restoring its parents' checkpoints filtered to its own slot ranges, replaying its parents' journals from those checkpoints filtered to its own slot ranges, and then resuming the input stream from the parents' sealed positions. - **ZBP-5-R93** A split or merge MUST seal every parent before any child accepts a write, and no parent MUST resume after any child has started. - **ZBP-5-R94** Cross-key ordering MUST NOT be relied upon across a split or a merge. Per-key ordering MUST be preserved, and it is the narrowness of that guarantee that makes redistribution possible at all. - **ZBP-5-R95** A child MUST take its own checkpoint before its parents' stored state becomes eligible for deletion. - **ZBP-5-R96** One slot MUST be the smallest unit that can be split off. A key whose load exceeds a single node MUST be addressed in the application's key design — by composing the key with a sub-key and explicitly forfeiting ordering between sub-keys — and MUST NOT be treated as a defect in the placement layer. ### Routing and the client contract - **ZBP-5-R97** A client MUST compute the slot itself and resolve it against a cached map. It MUST NOT consult a router or the placement controller per request. - **ZBP-5-R98** Every request MUST carry the key, the resolved slot, and the map version the client resolved against. - **ZBP-5-R99** A server receiving a request for a slot it owns and is ready to serve MUST serve it, regardless of the client's map version. - **ZBP-5-R100** A server receiving a request for a slot it owns but is not ready to serve MUST return NOT_READY with a retry delay, and MUST NOT redirect. - **ZBP-5-R101** A server receiving a request for a slot it does not own MUST distinguish three cases by map version: its own map is newer, and it MUST return MOVED carrying the current assignment; the versions are equal, and it MUST return MISROUTED with no assignment; its own map is older, and it MUST return STALE_MAP with a retry delay. - **ZBP-5-R102** A server MUST NOT issue a redirect derived from a map older than the client's. This is what prevents two disagreeing nodes redirecting a client to each other indefinitely. - **ZBP-5-R103** A MOVED response MUST carry enough of the map for the client to route the retry, and the client MUST verify its signature per ZBP-5-R11 and apply it before retrying. A response whose delta asserts assignments beyond the requested slot is legitimate only because it is signed; an unsigned one MUST be discarded rather than narrowed. - **ZBP-5-R104** A client MUST bound the number of redirects for one logical request and MUST fail the request rather than follow an unbounded chain. - **ZBP-5-R105** MISROUTED responses MUST be counted and alarmed. They mean client and server disagree about the hash or the slot arithmetic, which is a correctness fault rather than a transient condition, and no amount of retrying will resolve it. - **ZBP-5-R106** Clients MUST refresh a cached map on a jittered interval as well as on redirect, so that a map change does not produce a synchronised refresh from the entire client population. - **ZBP-5-R107** Every mutating response MUST carry the position at which it was applied, as an `(epoch, LSN)` pair ordered lexicographically, so that a caller needing read-your-writes can require that position on a later read. - **ZBP-5-R108** A position token MUST name the partition it was issued by, and a caller presenting a token from a partition that no longer exists MUST be told so explicitly rather than made to wait for a position that can never arrive. A split or merge resets the LSN under a new partition id, so `(epoch, LSN)` is comparable only within one partition id and MUST NOT be compared across a lineage boundary. ### Placement and the control plane - **ZBP-5-R109** The placement controller MUST NOT be called on the serving path, by clients or by nodes. - **ZBP-5-R110** The data plane MUST continue serving from the last published map while the placement controller is unavailable. A controller outage MUST freeze change, never serving. - **ZBP-5-R111** The placement controller MUST NOT store its own state inside the service it places. - **ZBP-5-R112** Every partition MUST declare a required class and a capacity estimate; every node MUST declare its class and its capacity budget. Placement MUST satisfy class constraints before optimising anything else. - **ZBP-5-R113** A class marked exclusive MUST result in its node hosting exactly one partition, and the controller MUST NOT co-locate another partition there for any reason, including transiently during a rebalance. - **ZBP-5-R114** Placement MUST enforce the declared failure-domain diversity as a hard constraint, ranked with the class constraints and above any balance objective. Where it cannot be satisfied the partition MUST be placed and reported undiversified rather than left unplaced — an unowned partition is worse than an undiversified one, and the health signal is what makes the difference visible. - **ZBP-5-R115** The placement objective MUST be ordered: satisfy constraints, then minimise movement, then balance load. Movement MUST rank above balance, or the controller will thrash whenever load is close to even. - **ZBP-5-R116** The controller MUST bound concurrent moves globally and per node, and MUST NOT start a move for a partition that moved more recently than a stated hysteresis interval. - **ZBP-5-R117** Placement decisions MUST be emitted as an explicit plan that can be rendered, reviewed, and applied as separate steps, and the plan MUST be dry-runnable against live state. - **ZBP-5-R118** Whether a plan applies automatically or waits for a human MUST be a policy per plan kind, and every applied plan MUST be recorded with what it changed and what triggered it. - **ZBP-5-R119** A new map version MUST NOT move more than a configured fraction of slots relative to the current one unless an override is explicitly recorded with a reason. Coverage validation proves a map is well formed, not that it is right, and a well-formed wrong map moves the entire fleet in one write. - **ZBP-5-R120** Map publication MUST be staged: a canary population of clients and nodes MUST adopt the new version and be observed against a stated outcome before it is promoted to everyone. - **ZBP-5-R121** Promotion MUST be gated on the canary's observed outcome, and a canary that regresses MUST cause the version to be withdrawn rather than promoted. A stage nobody can fail is not a stage. - **ZBP-5-R122** The staged rollout MUST NOT delay a map version that completes a handoff already past its seal. That partition is unavailable until the map names its new primary, so the brake applies to planned movement and never to the completion of movement already committed. - **ZBP-5-R123** A global freeze MUST exist that halts all NEW movement while leaving serving untouched, and it MUST take effect without a deploy or a restart. It MUST NOT stop a handoff already past its seal, which ZBP-5-R74 forbids from stopping: freezing there would strand the partition with no primary rather than protecting anything. ### Scheduled scaling and load shedding - **ZBP-5-R124** Scale-out MUST be pre-warmed: incoming owners restore and catch up while the current owners continue to serve, and the map flips only once they are ready. - **ZBP-5-R125** Scale-in MUST be expressed as merges and handoffs followed by node removal, and the node removed MUST be one the controller selected. - **ZBP-5-R126** The platform MUST NOT terminate a node the controller did not select. Where the platform can terminate on its own — an autoscaling policy, a spot reclamation, a rolling deployment — that capability MUST be disabled, or held behind a hook the controller completes after draining. - **ZBP-5-R127** Scheduled plans MUST be expressible against a calendar, so that state can be compacted and the fleet reduced through a known trough and split again ahead of a known ramp. - **ZBP-5-R128** A scheduled scale-up MUST begin far enough ahead of the expected ramp for pre-warming to complete before load arrives, and the lead time MUST be derived from measured restore times rather than assumed. - **ZBP-5-R129** Compaction — rewriting a checkpoint to drop expired or unreachable state — MUST be safe to abandon at any point and MUST NOT be a precondition for serving. - **ZBP-5-R130** A partition over its capacity budget MUST shed load at admission with a retryable outcome. It MUST NOT absorb the excess into an unbounded in-memory backlog, which converts a throughput problem into a state-loss problem. ### Observability and assurance - **ZBP-5-R131** Every partition MUST expose its id, generation, epoch, state, class, current node, lease expiry, checkpoint age, journal LSN, input lag, and standby lag. - **ZBP-5-R132** Every request outcome MUST be attributable to a partition and an epoch. - **ZBP-5-R133** A slot with no ready owner MUST be alarmed within a bounded interval. Unowned is a distinct condition from unhealthy, and MUST NOT be reported as either healthy or as a node failure. - **ZBP-5-R134** Two nodes reporting the primary role for one partition at the same epoch MUST be alarmed as a correctness fault, not as a transient inconsistency. - **ZBP-5-R135** Handoff duration, seal-to-ready duration, and the number of requests rejected during a handoff MUST be measured per handoff and retained per partition. - **ZBP-5-R136** The recovery path MUST be exercised in production on a schedule by deliberately moving a partition, and the measured restore time MUST feed the pre-warm lead time required by ZBP-5-R128. - **ZBP-5-R137** Replay equivalence MUST be tested continuously on real partitions: take a checkpoint at LSN *n*, restore from an earlier checkpoint at LSN *m* < *n*, replay forward to *n*, and compare the two states. This is the only check that detects a non-deterministic applier, whose entire symptom is that recovered state differs from state that was never recovered. - **ZBP-5-R138** A silent partition MUST be distinguishable from an idle one. Absence of work MUST be reported as a positive statement of liveness, because a partition that has stopped consuming and one with nothing to consume produce identical graphs. - **ZBP-5-R139** The age of the oldest map version still in use by any client or node MUST be visible, so that convergence after a map change is a measured fact rather than an assumption. ## Wire formats and interfaces ### The partition map Published by the placement controller, cached by every client and node, and the single answer to "where does this key live right now." ```json { "map_version": 1487, "generated_at": "2026-08-21T09:14:02Z", "slot_bits": 12, "hash": "md5-128-be", "partitions": [ { "id": "p_0002", "generation": 5, "epoch": 73, "class": "std.2x", "mode": "JOURNALED", "slots": [[1372, 2047]], "parents": [{ "id": "p_0001", "generation": 4 }], "primary": { "node": "n_7a2c", "endpoint": "10.0.4.11:7443", "state": "SERVING" }, "standbys": [ { "node": "n_1f90", "endpoint": "10.0.5.23:7443", "state": "SYNCED", "lag_records": 0 } ] }, { "id": "p_whale", "generation": 1, "epoch": 9, "class": "ded.8x", "mode": "JOURNALED", "slots": [[1371, 1371]], "parents": [], "primary": { "node": "n_c41d", "endpoint": "10.0.6.4:7443", "state": "SERVING" }, "standbys": [ { "node": "n_9b02", "endpoint": "10.0.7.9:7443", "state": "SYNCED", "lag_records": 0 } ] } ], "signature": { "key_id": "map-2026-08", "alg": "ed25519", "value": "MEUCIQD…" } } ``` The `signature` covers every field above it, and is what makes a delta safe to relay. A client applies a delta handed to it by a peer in a `MOVED` response — see ZBP-5-R11 — so authenticity has to travel with the document rather than with the connection it arrived on. `key_id` names the signing key so rotation can overlap: a verifier accepts any key inside its validity window and rejects everything else, including a map that is newer than the one it holds. Primary states are `RESTORING`, `CATCHING_UP`, `SERVING`, `SEALED`, `QUARANTINED`, and `UNASSIGNED`. Standby states are `RESTORING`, `SYNCING`, `SYNCED`, and `LOST`. `UNASSIGNED` and `QUARANTINED` are distinct on purpose: the first says the controller has not placed it, the second says it must not be placed, and conflating them is how a poison partition gets retried forever. A delta carries only what changed, and names the base it applies to: ```json { "map_version": 1488, "base_version": 1487, "changed": [ { "id": "p_0002", "…": "…" } ], "signature": { "key_id": "map-2026-08", "alg": "ed25519", "value": "MEQCIF…" } } ``` ### The lease record One record per partition in the lease store, holding ownership, the epoch, and the checkpoint pointer together — because they must move atomically with respect to each other. ``` partition p_0002 epoch 74 owner_node n_1f90 owner_endpoint 10.0.5.23:7443 state SERVING lease_ttl_ms 15000 renew_interval_ms 3000 expires_at 2026-08-21T09:14:19.482Z checkpoint_pointer {prefix}/p_0002/g5/e73/ckpt-000000918233.manifest.json sealed_lsn null taken_by null (operator, on a force-takeover) taken_reason null taken_at null ``` Three conditional writes, and no others, may touch it: ``` ACQUIRE condition: attribute_not_exists(partition) OR (epoch < :new_epoch AND expires_at < :now) set: epoch=:new_epoch, owner_node=:me, state=RESTORING, expires_at=:exp RENEW condition: epoch = :my_epoch AND owner_node = :me set: expires_at = :exp FENCE condition: epoch = :my_epoch AND owner_node = :me set: checkpoint_pointer = :ptr (or state, or sealed_lsn) FORCE condition: epoch < :new_epoch -- no expires_at clause set: epoch=:new_epoch, owner_node=:me, state=RESTORING, taken_by=:operator, taken_reason=:reason, taken_at=:now ``` `FORCE` is ACQUIRE without the `expires_at` test, and nothing else (ZBP-5-R26). It is the whole break-glass: it cannot skip the epoch increment, so the displaced node is fenced by exactly the mechanism that would have fenced it anyway — which is why a human may safely be given it and automated placement may not. `:new_epoch` is the epoch read a moment earlier plus one. The `epoch < :new_epoch` clause is what makes two racing acquirers safe: the first write leaves `epoch = :new_epoch`, so the second's condition is false and it must re-read and try again at a higher epoch. ### Object layout Immutable, uniquely named, and never overwritten (ZBP-5-R21), so a write from a stale primary lands at a key nothing points at. ``` {prefix}/{partition}/g{generation}/e{epoch}/ckpt-{lsn:012}.manifest.json {prefix}/{partition}/g{generation}/e{epoch}/ckpt-{lsn:012}/part-{n:04} {prefix}/{partition}/g{generation}/e{epoch}/journal/seg-{first_lsn:012}-{last_lsn:012} {prefix}/{partition}/g{generation}/e{epoch}/spill/{iso8601}-{n:04}.ndjson ``` ### The class declaration The class is where every per-workload choice is declared, and nothing in it defaults. A partition names a class; the class answers what shape of node it needs, what it costs to lose a record, and what it must survive. ```json { "name": "ded.8x", "exclusive": true, "node": { "cpu": 8, "memory_gb": 64, "local_ssd_gb": 500 }, "min_synced_standbys": 1, "failure_domains": "distinct", "commit_tier": "DURABLE", "checkpoint": { "kind": "incremental", "max_delta_chain": 16 }, "freshness_bound": { "max_lag_records": 5000, "max_record_age_ms": 2000 }, "capacity": { "state_bytes": 34359738368, "requests_per_second": 4000 } } ``` `commit_tier` (ZBP-5-R67), `failure_domains` (ZBP-5-R61) and `checkpoint.kind` (ZBP-5-R39) have no defaults on purpose. A class that omits one does not start, which is the only way an omission gets noticed before the incident that depended on it. ### The checkpoint manifest ```json { "partition": "p_0002", "generation": 5, "epoch": 73, "format_version": 3, "kind": "incremental", "base": "…/ckpt-000000901004.manifest.json", "deltas": [ { "key": "…/delta-000000909117", "after": null, "sha256": "77ae…" }, { "key": "…/delta-000000918233", "after": "…/delta-000000909117", "sha256": "b104…" } ], "journal_lsn": 918233, "input_positions": { "shard-0004": "49590000000000000000000012", "shard-0005": "49590000000000000000000031" }, "coverage_start": "2026-08-14T00:00:00Z", "slots": [[1372, 2047]], "created_at": "2026-08-21T09:02:11Z", "size_bytes": 412998144, "sha256": "9f2c…", "parts": [ { "slot_low": 1372, "slot_high": 1599, "key": "…/part-0000", "sha256": "3a17…" }, { "slot_low": 1600, "slot_high": 2047, "key": "…/part-0001", "sha256": "c8d4…" } ] } ``` `kind` is declared, never inferred (ZBP-5-R39), and on an incremental manifest each delta names the one before it so the chain has exactly one order (ZBP-5-R40). On a `full` manifest `base` and `deltas` are absent and `parts` alone carries the state. `parts` is what makes ZBP-5-R38 real: a child of a split fetches only the parts overlapping its own ranges, so splitting a 400 MB partition four ways reads 400 MB in total rather than 1.6 GB. ### Journal record framing Fixed header, opaque payload, trailing checksum. Sizes in bytes, all integers big-endian. ``` off len field 0 4 frame length — every byte after this field, including the CRC 4 1 record type — 1 APPLY, 2 MARK, 3 SEAL 5 8 lsn 13 8 epoch 21 4 slot 25 4 marks length (L) 29 L marks — input positions consumed at and before this record 29+L P payload — opaque to the runtime, interpreted only by the applier tail 4 CRC32C over bytes 4 .. end-4 ``` A `MARK` record carries positions and no payload; it is how a partition advances its resume point during a quiet period without writing state. A `SEAL` record is the last record of a generation and carries the final positions. ### The routing envelope Returned on every non-served outcome, and readable without knowing the application's own error shape. ```json { "code": "MOVED", "slot": 1789, "map_version": 1488, "retry_after_ms": null, "assignment": { "partition": "p_0002", "epoch": 74, "primary": { "node": "n_1f90", "endpoint": "10.0.5.23:7443" } }, "map_delta": { "map_version": 1488, "base_version": 1487, "changed": [ "…" ], "signature": { "key_id": "map-2026-08", "alg": "ed25519", "value": "MEQCIF…" } } } ``` | code | HTTP | gRPC | client action | carries assignment | |---|---|---|---|---| | `SERVE` | 200 | `OK` | — | — | | `NOT_READY` | 503 | `UNAVAILABLE` | retry same endpoint after `retry_after_ms` | no | | `MOVED` | 421 | `FAILED_PRECONDITION` | apply delta, retry named endpoint | yes | | `MISROUTED` | 400 | `INVALID_ARGUMENT` | do not retry; alarm | no | | `STALE_MAP` | 503 | `UNAVAILABLE` | retry after `retry_after_ms`, refresh map | no | HTTP 421 is the correct status for `MOVED` — it means precisely that the request reached a server unable to produce a response for it — and unlike a 3xx it is not followed automatically by intermediaries that know nothing about slots. ### Position tokens Every mutating response carries the position it was applied at; a read may demand a floor. ``` X-Partition-Position: p_0002:74:918407 (partition : epoch : lsn) X-Min-Position: p_0002:74:918407 (on a read, the floor the caller requires) ``` Positions compare as the ordered pair `(epoch, lsn)`, numerically, never as strings — `74:9` precedes `74:10`, which a lexicographic comparison gets backwards. ### Handoff control interface ``` PrepareHandoff(partition, generation, target, deadline) → { accepted, incoming_epoch } StreamState(partition, from_lsn) → stream of journal records ReportReady(partition, incoming_epoch, position) → { ok } Seal(partition, epoch) → { sealed_lsn, input_positions } Promote(partition, expected_epoch) → { new_epoch } AbortHandoff(partition, reason) → { ok } ``` ## Algorithms Each step names its failure outcome. An implementation that reaches a state not named here has found a gap in this document, and should be treated as such rather than improvised around. ### A.1 — Resolve a key (client) 1. `slot ← hash(utf8(key)) >> (128 − slot_bits)`. 2. Look up `slot` in the cached map. Not found → **`NO_MAP`**: fetch a full map and restart; if that fails, fail the request. 3. Read the partition's primary endpoint. State is not `SERVING` → **`NOT_READY`**: wait the class's retry delay and restart, subject to the caller's deadline. 4. Send the request carrying `key`, `slot`, and the cached `map_version`. 5. On `MOVED`, apply `map_delta`, increment the redirect counter, and restart from step 2. Counter past its bound → **`REDIRECT_LOOP`**: fail, and alarm. 6. On `MISROUTED` → **`HASH_DISAGREEMENT`**: fail immediately without retrying, and alarm. Retrying cannot help; the client and the server do not compute the same slot. 7. On `NOT_READY` or `STALE_MAP`, wait `retry_after_ms` and restart from step 2. ### A.2 — Admit a request (server) 1. Recompute `slot` from the key. Disagrees with the client's `slot` → **`MISROUTED`**. 2. Check the lease: is this node the primary for the partition owning `slot`, at the epoch it believes it holds, with the lease not near expiry per ZBP-5-R18? No → go to step 5. 3. Is the partition `SERVING`? No → **`NOT_READY`** with the class's retry delay. Never a redirect: a partition restoring here is not somewhere else. 4. Is the partition within its capacity budget? No → **`OVERLOADED`**, a retryable shed, per ZBP-5-R130. Serve otherwise. 5. Not the owner. Compare map versions per ZBP-5-R101: mine newer → **`MOVED`** with a delta; equal → **`MISROUTED`**; mine older → **`STALE_MAP`**. ### A.3 — Acquire and restore a partition (cold path) This is the path that must work; A.4 is the optimisation. 1. `ACQUIRE` the lease at `epoch + 1`. Condition failed → **`LOST_RACE`**: re-read and retry with backoff. Set state `RESTORING`. 2. Start lease renewal (A.7) immediately, before any expensive work. A restore that outlives its own lease has been wasted. 3. Read `checkpoint_pointer` from the lease record — never by listing the object store, which is eventually consistent about its own contents and cannot distinguish a live checkpoint from a zombie's. 4. Fetch the manifest. Fetch only the `parts` overlapping this partition's slot ranges. Digest mismatch on any part → treat the checkpoint as absent per ZBP-5-R37 and fall back to the previous pointer; none left → **`NO_CHECKPOINT`**, restore from the beginning of retained input. 5. Refuse any `format_version` this binary does not fully understand → **`FORMAT_UNKNOWN`**: release the lease and report. Do not restore partially. 6. Restore state. In `DERIVED` mode go to step 8 with the manifest's `input_positions`. 7. `JOURNALED` or `MIXED`: replay journal segments from `journal_lsn + 1` forward, discarding records for slots not owned. A gap in the LSN sequence → **`JOURNAL_GAP`**: stop, quarantine, alarm. Never skip a gap. On reaching the tail, take the resume positions from the tail record's marks — *not* from the checkpoint, which is behind the journal and would re-apply the window between them. 8. Set state `CATCHING_UP`. Open an iterator per contributing shard at its resume position. Drain parent shards fully before their children per ZBP-5-R54. 9. Apply records for owned slots; count and discard the rest. An applier failure → retry bounded, then **`POISON`**: quarantine per ZBP-5-R59, release the lease, and do not re-place. 10. When lag is inside the class's freshness bound, `FENCE` state to `SERVING`, report ready, and let the controller publish a map naming this node. ### A.4 — Live handoff with a synced standby The steps before the seal are reversible and cost nothing if abandoned. The seal is one conditional write. Everything expensive happens before it. 1. Controller calls `PrepareHandoff` on the incoming node with a deadline. Refused → abort; nothing has changed. 2. Incoming node restores per A.3 steps 3–9, but stops short of acquiring the primary lease. It runs as a standby, consuming the primary's journal (`JOURNALED`) or the input stream (`DERIVED`). 3. Incoming node reports `SYNCED` when its lag is inside the class's bound. Deadline passes first → **`PREPARE_TIMEOUT`**: abort, outgoing primary keeps serving, map unchanged. 4. **Seal.** Outgoing primary stops admitting, appends a `SEAL` record, and `FENCE`s `sealed_lsn` and state `SEALED` into the lease record under its own epoch. From here it may never accept another write, whatever happens next (ZBP-5-R74). 5. Outgoing primary rejects every uncommitted in-flight request with `MOVED` and no assignment yet — the client will get `NOT_READY` from the incoming node until step 7, which is correct and bounded. 6. Incoming node consumes to `sealed_lsn`. Outgoing node unreachable → recover the remaining segments from the object store; if the tail was never durable, replay from the last durable mark and the input stream. It cannot roll back, because the seal already happened. 7. Incoming node `ACQUIRE`s at `epoch + 1`, sets `SERVING`, and reports ready. 8. Controller publishes the new map version. Only now do clients learn. 9. Outgoing node discards its state without flushing (ZBP-5-R24). Between steps 4 and 8 the partition is unavailable. That window — seal to ready — is the number this whole design exists to make small, and it must be measured per handoff (ZBP-5-R135). ### A.5 — Split 1. Controller chooses a split point at a slot boundary and creates child entries with a generation above the parent's, state `RESTORING`, and `parents` recording the parent's id and generation. 2. Each child restores from the parent's checkpoint, fetching only the `parts` overlapping its own ranges, then replays the parent's journal filtered to its ranges. 3. Each child reports `SYNCED`. A child that cannot → abort: the parent has not sealed, so nothing has moved. 4. Parent seals as in A.4 step 4. Every child then consumes to `sealed_lsn`. 5. Children acquire their own leases at epoch 1 for the new partition ids, and open input iterators at the parent's sealed positions. 6. Controller publishes a map in which the parent no longer appears and the children cover its slots exactly. Coverage validation (ZBP-5-R5) is what catches an arithmetic error here, before anyone routes against it. 7. Parent state is retained until every child has taken its own checkpoint (ZBP-5-R95). ### A.6 — Merge Identical to A.5 with the arrows reversed: one child restores from *every* parent's checkpoint and journal, all parents seal before the child accepts a write, and no parent resumes afterwards. Because the parents' slot ranges are disjoint, per-key order survives; cross-key order does not, and nothing may depend on it (ZBP-5-R94). ### A.7 — Lease renewal and self-fencing Runs continuously on every primary, in its own goroutine, independent of request handling. 1. On a successful `RENEW`, record `deadline ← monotonic_now() + lease_ttl − margin`, where `margin` covers the worst renewal round-trip plus tolerated clock error. 2. Every renewal interval, attempt `RENEW`. Condition failed → **`LEASE_LOST`**: go to step 4 immediately. 3. Before admitting any request, and on a timer regardless of traffic, check `monotonic_now() < deadline`. False → **`LEASE_EXPIRED`**: go to step 4. This check uses no network and no wall clock, which is exactly why it still runs correctly after a long process pause. 4. Stop admitting, fail in-flight work as retryable, discard state without flushing, and report. Do not attempt a final checkpoint: this node may already have been superseded, and its write would be fenced anyway (ZBP-5-R20). ### A.8 — Reconciliation 1. Read observed state: node heartbeats with class and capacity, lease records, partition health. 2. Compute the desired placement: satisfy class constraints, then minimise movement, then balance (ZBP-5-R115). 3. Diff against the current map to produce a plan of moves, splits, and merges. 4. Drop moves that violate the hysteresis interval or the concurrency bound (ZBP-5-R116). 5. Emit the plan. Auto-apply or hold for a human per policy (ZBP-5-R118). 6. Execute one step at a time, re-reading observed state between steps. A step that fails aborts the plan; it does not roll back the steps already applied, because each completed handoff is already a consistent state. ### A.9 — Scheduled compaction and pre-split The evening pass, at a known trough: 1. Merge adjacent under-utilised partitions per A.6. 2. Rewrite each surviving checkpoint, dropping expired and unreachable state, and flip the pointer. Abandonable at any point (ZBP-5-R129) — an interrupted compaction leaves the previous checkpoint live and nothing else changed. 3. Retire the drained nodes, each selected by the controller (ZBP-5-R125). The morning pass, ahead of a known ramp: 1. Start from `T_ramp − lead`, where `lead` is measured restore time plus catch-up time plus a margin, taken from the production restore exercises required by ZBP-5-R136 — never assumed. 2. Add nodes; split partitions per A.5, with children pre-warmed while the parents still serve. 3. Flip the map only once every child is `SYNCED`. If pre-warming is not finished by `T_ramp`, do not flip: serving a ramp from a warm parent beats serving it from a cold child. ## AWS binding The requirements above are cloud-neutral. This section is the concrete mapping, and names three limits that are properties of the platform rather than of the design — which means no amount of care in the runtime removes them. | Concern | Service | Notes | |---|---|---| | Lease store | DynamoDB, one item per partition | The three condition expressions map directly onto `ConditionExpression`. On-demand capacity; the write rate is a renewal per partition per interval, not per request. | | Map publication | DynamoDB item, mirrored to S3 | Nodes read the item; clients read the object through a CDN. Delta objects keyed by `base_version`. | | Checkpoints, journal segments, spill | S3 | Immutable keys per ZBP-5-R21. The pointer flip is the DynamoDB write, never an S3 overwrite. | | Input stream | Kinesis Data Streams | Its MD5 hashing is what makes slot alignment free. | | Nodes | ECS tasks, or EC2 instances | Started and stopped by the controller, never by a scaling policy. | | Controller | A separate service | Deployed apart from the nodes it places, per ZBP-5-R111 and ZFN-4. | ### Kinesis alignment A stream maps a record to a shard by taking the MD5 of the partition key as a 128-bit integer, and each shard owns a contiguous `[StartingHashKey, EndingHashKey]` range of that space. The slot is defined here as the top `SLOT_BITS` bits of the same MD5 of the same key — so a slot *is* a sub-range of the shard hash space, and a partition, being a set of slot ranges, is a set of hash ranges. Nothing needs configuring: use the service key as the stream partition key and alignment follows. Choose `SLOT_BITS` so that `2^SLOT_BITS` comfortably exceeds the largest shard count the stream will ever have — at 12 bits, 4096 slots against a few hundred shards leaves every shard boundary landing inside a slot rather than splitting one. Where a slot does straddle two shards, the partition simply reads both; ZBP-5-R53's per-shard position vector already accommodates it. For the escape hatch of ZBP-5-R52, the explicit hash key for slot `s` is `s × 2^(128 − SLOT_BITS)`, rendered as a decimal string. On a reshard, `ListShards` reports `ParentShardId` and `AdjacentParentShardId`; a parent is drained when `GetRecords` returns a null `NextShardIterator`, and only then may its children be read (ZBP-5-R54). ### Three platform limits **One key cannot exceed one shard.** A key is a single point in the hash space, so every record for it lands in one shard and is bounded by that shard's ingest quota. Pinning a large tenant to its own slot and its own node buys dedicated *compute*; it buys no additional ingest at all. Past the shard quota the key must become a composite of tenant and sub-key, explicitly forfeiting ordering between sub-keys (ZBP-5-R96). This is the first ceiling a very large tenant meets, and it is not in the placement layer — which is why teams look for it there and do not find it. **The container stop timeout bounds partition size.** The grace period between the termination signal and the kill is capped by the platform — on the serverless launch type, at two minutes. Every partition on a node must complete handoff inside it, and ZBP-5-R85 checks the arithmetic at boot. So the maximum state a partition may hold is not a memory question but a *drain-rate* question: whatever can be sealed, streamed, and promoted inside the budget, divided by the partitions per node. A partition that has outgrown the budget must be split, and the alarm for it is the measured handoff duration of ZBP-5-R135 approaching the budget — not an out-of-memory event, which will never come. **Checkpoint age must be read against the configured retention, not the default.** A stream retains 24 hours unless configured otherwise, extensible to a year. ZBP-5-R47's alarm compares the newest usable checkpoint against whatever is configured *now*, so that lowering retention is caught as the recovery regression it is. Raising retention is also the cheapest immediate mitigation when the alarm does fire, and is worth having in the runbook. ### Node management Do not use a platform rolling deployment. It starts a replacement before the outgoing task has drained, chooses its own ordering, and has no concept of which partitions are mid-handoff — every one of which contradicts ZBP-5-R80. The controller starts tasks, drains them, and stops them. Where an autoscaling group is used for capacity, enable instance scale-in protection and a termination lifecycle hook, so the group cannot choose a victim and the controller completes the hook after draining (ZBP-5-R126). Spot capacity MUST NOT host primaries unless the interruption notice is wired to a handoff, and even then the notice is a courtesy rather than a guarantee — a standby on on-demand capacity is what makes spot survivable. ## Threat model and failure modes | Failure | What it looks like | What catches it | |---|---|---| | Zombie primary after a pause | Two nodes believe they are primary; one's writes vanish | Epoch fence at the store (ZBP-5-R20), duplicate-primary alarm (ZBP-5-R134) | | Zombie serving stale *reads* | Correct-looking answers from expired state | Self-fence before admitting **any** request, reads included (ZBP-5-R18) | | Checkpoint older than retention | Nothing at all, until a recovery is attempted | Margin alarm against configured retention (ZBP-5-R47) | | Poison record | Every node crashes in turn; the fleet drains itself | Bounded attempts then quarantine (ZBP-5-R59) | | Non-deterministic applier | No error; recovered state simply differs | Replay-equivalence test (below) | | Redirect loop | Latency spike, no errors, clients pinned at 100% CPU | Version-ordered redirect rule (ZBP-5-R102), redirect bound (ZBP-5-R104) | | Synchronised map refresh | A load spike on the map store at every change | Jittered refresh (ZBP-5-R106) | | Lease lost to a GC pause | A partition briefly owned by nobody | Monotonic self-fence with margin (ZBP-5-R17, ZBP-5-R18) | | Silent loss of the last standby | Durability quietly degraded to one copy | Explicit no-standby policy (ZBP-5-R71), standby lag metric (ZBP-5-R64) | | Hot key beyond one node | Latency on one partition that splitting will not fix | Named as a key-design problem (ZBP-5-R96) | | Segment deleted that a child still needs | Recovery fails months later, for one partition | Lineage-driven deletion (ZBP-5-R45) | | Partition stalled | Graphs identical to a quiet partition | Positive liveness statement (ZBP-5-R138) | | Answer from a partial window | A confident number that omits history | Coverage start, incompleteness reported (ZBP-5-R49) | | Forged or relayed map | A tenant's traffic routed to an endpoint of the attacker's choosing | Signed maps, verified before apply from any source (ZBP-5-R10, ZBP-5-R11) | | Valid map, wrong endpoints | Total outage from one write, with no invalid input anywhere | Movement cap and canary stage (ZBP-5-R119, ZBP-5-R120, ZBP-5-R121) | | Crafted poison record | One input takes a partition offline for as long as it is retained | Quarantine bounds it to one partition — and is itself the denial | | Unbounded delta chain | Restore time drifts past the handoff budget, invisibly | Chain length bound with forced compaction (ZBP-5-R41) | Three of these deserve more than a row. **The stale checkpoint is the one that does not announce itself.** Every other failure here produces a signal at the moment it happens. This one produces nothing: the partition serves normally, the checkpoint pointer is valid, the input stream is healthy, and the system is simply no longer recoverable. It becomes visible at the exact moment you need it not to be — during a recovery, when the stream has already aged past the checkpoint and there is no path from one to the other. It is why ZBP-5-R47 alarms on *margin* rather than on failure, and why ZBP-5-R48 insists the restore path is exercised against real checkpoints continuously rather than tested once. **A non-deterministic applier is a correctness bug that no test of the happy path can see.** The service is right until it recovers, and then it is quietly wrong — different counters, a different window, a decision that would not have been made. The test that finds it is a replay-equivalence check: take a checkpoint at LSN *n*, restore from the checkpoint at LSN *m* < *n*, replay to *n*, and compare the two states byte for byte. Run it continuously, on real partitions. Wall-clock reads, map iteration order, and a stray call to an external service are the three sources, in that order of frequency. **Quarantine is a containment, and also an attack surface.** ZBP-5-R59 exists because an applier failure that is merely retried will crash every node the partition is placed on, in turn. But the same mechanism means that anyone who can get a record into the input stream that reliably fails the applier can take that partition offline for as long as the record is retained, using one input. The two readings are the same requirement seen from opposite sides, and the trade is not avoidable at this layer — it is chosen. Where the application can tolerate skipping a record, skipping and counting it is the better default and quarantine the fallback; where it cannot, quarantine is correct and the exposure has to be answered upstream, by validating what may enter the stream at all. What is not acceptable is leaving it unstated, so that the first crafted record is discovered as a mystery outage rather than as the known cost of a known control. **The poison record turns a partition failure into a fleet failure.** Without ZBP-5-R59 the sequence is mechanical: the partition is placed, the applier fails, the node dies, the controller places it elsewhere, that node dies. There is no natural stopping point, and the blast radius grows to every node in the class. Quarantine is not a nicety; it is the difference between one unavailable partition and no service at all. ## Anti-patterns If you find yourself doing one of these, you have misread the document. - **Using a consistent-hash ring as the assignment.** The moment you add weighted virtual nodes to express "this customer needs its own hardware", you are reimplementing a published map, badly, in a form nobody can read or diff. Publish the map. - **A key-level pin.** It seems harmless and it breaks stream alignment, split, and merge simultaneously. Give the key its own slot instead (ZBP-5-R4). - **Asking the controller who owns a key.** Any per-request call to the control plane makes it a serving dependency, which is the exact coupling ZFN-16 exists to prevent. - **A sticky load balancer in place of a map.** Stickiness is a hint with no notion of epoch, so it cannot tell a client its assignment is stale and will route happily to a node that has been fenced. - **Autoscaling the fleet.** A scale-in policy chooses a victim by CPU, which is uncorrelated with drainability. It will pick the node holding the largest partition roughly as often as any other. - **A platform rolling deployment.** It starts the replacement before the outgoing task drains and picks its own order. A rollout here is a sequence of handoffs (ZBP-5-R80). - **Treating "a standby exists" as durability.** A standby with unbounded lag is a second copy of an old state. Measure the lag or do not claim the property (ZBP-5-R64). - **Rebalancing on every metric tick.** Without hysteresis and a movement penalty, a controller optimising balance will move partitions continuously and serve worse than one that does nothing. - **Taking a final checkpoint when the lease is lost.** The instinct is to save what you have. It is precisely wrong: this node may already have been superseded, its state is a divergent branch, and the write is fenced anyway (ZBP-5-R24). - **Retrying `MISROUTED`.** It means the client and the server do not compute the same slot. No number of retries fixes a disagreement about arithmetic; it needs an alarm and a human. - **Deleting journal segments by age.** A split child that has not yet checkpointed depends on its parent's segments, and age says nothing about that (ZBP-5-R45). - **Recomputing on replay what was computed at write time.** Anything not captured into the record is a source of divergence, and divergence here is silent (ZBP-5-R34). - **Calling lock shards partitions.** Striping a map across 64 mutexes is a concurrency technique with no relationship to ownership, and sharing the word costs somebody a day. ## Test vectors Every value below was produced by the reference implementation at the end of this section and can be reproduced by running it. `SLOT_BITS = 12`, so `SLOT_COUNT = 4096` and the hash is MD5 of the key's UTF-8 bytes read big-endian. ### V1 — Key to slot At 12 bits the slot is exactly the first three hex digits of the MD5, which makes these checkable by eye as well as by code. | key | `md5(utf8(key))` | slot | |---|---|---| | `t_0a1b2c3d` | `f46c623bb62cbfaccb5bda238027048f` | 3910 | | `t_4e5f6a7b` | `dc26ce93477593944417a2fbedc0a796` | 3522 | | `t_8c9d0e1f` | `e0eb5ec32f3dee754ddba9a57eb1e1f4` | 3598 | | `t_2a3b4c5d` | `b37217dd05eca994256faa1c52b83808` | 2871 | | `t_6e7f8a9b` | `6fdc12fabcee63d79703d23879cca844` | 1789 | | `t_c0d1e2f3` | `772bfae2d19867e9963da50827e1b120` | 1906 | | `t_whale` | `55b0c21b24e83694bdafcb69a8da3391` | 1371 | | `t_a4b5c6d7` | `f5f977e8fde5c2e1091e22c029b7fc1f` | 3935 | ### V2 — Refinement from 12 to 13 bits Raising `SLOT_BITS` must move no key across a partition boundary (ZBP-5-R3). Each slot becomes exactly two, and each key lands in one of its own slot's two children. | key | slot @12 | slot @13 | in `[s·2, s·2+1]` | |---|---|---|---| | `t_0a1b2c3d` | 3910 | 7821 | yes | | `t_whale` | 1371 | 2742 | yes | | `t_6e7f8a9b` | 1789 | 3579 | yes | Ranges rewrite mechanically: `[1371, 1371]` at 12 bits becomes `[2742, 2743]` at 13 bits, and `[0, 1370]` becomes `[0, 2741]`. ### V3 — Map resolution Against the map in *Wire formats* at `map_version` 1487, where `p_whale` holds the single slot 1371 on an exclusive class: | key | slot | partition | |---|---|---| | `t_0a1b2c3d` | 3910 | `p_0004` | | `t_4e5f6a7b` | 3522 | `p_0004` | | `t_8c9d0e1f` | 3598 | `p_0004` | | `t_2a3b4c5d` | 2871 | `p_0003` | | `t_6e7f8a9b` | 1789 | `p_0002` | | `t_c0d1e2f3` | 1906 | `p_0002` | | `t_whale` | 1371 | `p_whale` | | `t_a4b5c6d7` | 3935 | `p_0004` | ### V4 — Routing decision The complete outcome table for ZBP-5-R99 through ZBP-5-R102. Note rows 4 and 6: identical from the client's side, opposite in cause, and a system that collapses them redirects in circles. | owns slot | ready | server map | client map | outcome | |---|---|---|---|---| | yes | yes | 1487 | 1487 | `SERVE` | | yes | yes | 1487 | 1486 | `SERVE` | | yes | no | 1487 | 1486 | `NOT_READY` | | no | — | 1487 | 1486 | `MOVED` | | no | — | 1487 | 1487 | `MISROUTED` | | no | — | 1486 | 1487 | `STALE_MAP` | ### V5 — Recovery positions The trap this vector exists for is taking the resume positions from the checkpoint rather than from the journal tail, which silently re-applies every input record between them. Given a checkpoint at `journal_lsn` 918233 with positions `{shard-0004: …012, shard-0005: …031}`, and two journal segments after it — `918234–940100` and `940101–961887`, the latter marking `{shard-0004: …901, shard-0005: …877}`: | mode | replay journal | resume input after | |---|---|---| | `JOURNALED` | 918234 → 961887 | `{shard-0004: …901, shard-0005: …877}` — the **tail's** marks | | `DERIVED` | not applicable | `{shard-0004: …012, shard-0005: …031}` — the checkpoint's | | `JOURNALED`, no segments after the checkpoint | 918234 → 918233, i.e. nothing | `{shard-0004: …012, shard-0005: …031}` | ### V6 — Epoch fencing Partition `p_0002` during a handoff from `n_7a2c` to `n_1f90`. The record starts at epoch 73 owned by `n_7a2c`; `ACQUIRE` and `FENCE` are the conditions from *Wire formats*, and the sequence runs against the reference implementation below. | step | actor | operation | result | record after | |---|---|---|---|---| | 1 | `n_7a2c` | `FENCE` pointer at epoch 73 | `ACCEPT` | epoch 73, owner `n_7a2c` | | 2 | `n_1f90` | `ACQUIRE` at epoch 74 | `ACCEPT` | epoch 74, owner `n_1f90` | | 3 | `n_1f90` | `FENCE` pointer at epoch 74 | `ACCEPT` | epoch 74, owner `n_1f90` | | 4 | `n_7a2c` | `FENCE` pointer at epoch 73 | `REJECT_STALE_EPOCH` | epoch 74, owner `n_1f90` | | 5 | `n_7a2c` | `ACQUIRE` at epoch 74 | `REJECT_STALE_EPOCH` | epoch 74, owner `n_1f90` | Steps 4 and 5 are the whole safety argument. The outgoing node need not know it has been superseded, and need not be reachable, for its write to be harmless — and it cannot recover by re-claiming at the epoch it thinks it holds, because `ACQUIRE` requires a strictly higher one. ### V7 — Slot to explicit hash key For ZBP-5-R52's escape hatch, `explicit_hash_key(s) = s × 2^116`, as a decimal string: | slot | explicit hash key | |---|---| | 0 | `0` | | 1371 | `113898223888819978859444967477772025856` | | 1789 | `148624305278700906039056926927596027904` | | 4095 | `340199290171201906221318119490500689920` | The top of the space is `2^128 − 1 = 340282366920938463463374607431768211455`, so slot 4095 spans from its low hash key to there. ### Reference implementation ```python import hashlib SLOT_BITS = 12 def slot(key: str, bits: int = SLOT_BITS) -> int: h = int.from_bytes(hashlib.md5(key.encode("utf-8")).digest(), "big") return h >> (128 - bits) def explicit_hash_key(s: int, bits: int = SLOT_BITS) -> str: return str(s << (128 - bits)) def refine(lo: int, hi: int, d: int) -> tuple[int, int]: return (lo << d, (hi << d) | ((1 << d) - 1)) def resolve(key: str, partitions: list[dict], bits: int = SLOT_BITS) -> str | None: s = slot(key, bits) for p in partitions: if any(lo <= s <= hi for lo, hi in p["slots"]): return p["id"] return None def route(owns: bool, ready: bool, server_v: int, client_v: int) -> str: if owns: return "SERVE" if ready else "NOT_READY" if server_v > client_v: return "MOVED" return "MISROUTED" if server_v == client_v else "STALE_MAP" def resume_positions(checkpoint: dict, segments: list[dict], mode: str) -> dict: if mode == "DERIVED" or not segments: return checkpoint["input_positions"] return segments[-1]["marks"] # the TAIL's marks, never the checkpoint's def acquire(record: dict, new_epoch: int, claimant: str) -> str: """ACQUIRE: lands only at a strictly higher epoch, which is what makes two racing claimants safe — the loser must re-read and retry higher.""" if new_epoch > record["epoch"]: record.update(epoch=new_epoch, owner_node=claimant) return "ACCEPT" return "REJECT_STALE_EPOCH" def fence(record: dict, write_epoch: int, writer: str) -> str: """FENCE: a write lands only if it names the record's CURRENT epoch and its current owner. This, not the lease, is what makes a stale primary harmless.""" if write_epoch == record["epoch"] and writer == record["owner_node"]: return "ACCEPT" return "REJECT_STALE_EPOCH" ``` ## Conformance checklist Every MUST above, grouped. An implementation grades itself here; a box that cannot be ticked is a citable gap rather than an oversight. **Key space and map** - [ ] Slot is the top `SLOT_BITS` bits of a fixed, named, process-stable 128-bit hash of the UTF-8 key (R1, R2) - [ ] `SLOT_BITS` can be raised by rewriting ranges only, never lowered, and raising moves no state (R3) - [ ] No key-level pin exists; every partition is a set of slot ranges (R4) - [ ] Publication validates total coverage and non-overlap and refuses otherwise (R5) - [ ] Map version is monotonic, never reused, and changes on any content change (R6) - [ ] Entries carry id, generation, epoch, class, sorted ranges, primary and standby endpoints and states (R7) - [ ] Map is readable without the controller, and old versions stay addressable for the longest cache lifetime (R8, R9) - [ ] Partition ids are opaque and encode no range (R13) - [ ] Every published map version and delta is signed by the controller, over versions, assignments and endpoints (R10) - [ ] Clients and nodes verify that signature before applying, whatever the source, including a delta relayed by a peer (R11) - [ ] An unverifiable map is rejected, counted and alarmed, and is never applied for being newer; signing keys rotate with overlap (R12) **Ownership and fencing** - [ ] Every ownership record has an expiry, a named owner, and a renewal interval (R14) - [ ] Acquisition increments the epoch in the same conditional write that claims the lease (R15) - [ ] Epochs are monotonic per partition, never reused, and survive split and merge (R16) - [ ] Expiry is judged from a monotonic reading with a margin, needing no network (R17, R18) - [ ] Every shared-storage write carries its epoch, and the store rejects a lower one (R19, R20) - [ ] Objects are immutable and uniquely named; the checkpoint pointer lives in the lease store and moves under an epoch-fenced write (R21, R22) - [ ] Holding the lease and being ready are distinct states (R23) - [ ] Lease loss discards state without flushing, checkpointing, or replicating (R24) - [ ] The lease store is separate from the controller and reachable without it (R25) - [ ] A force-takeover exists that acquires at a higher epoch without waiting out the lease (R26) - [ ] Force-takeover advances the epoch through the same conditional write; no path takes ownership without advancing it (R27) - [ ] Force-takeover records operator, time and reason, and is unreachable by automated placement (R28) **State, checkpoints, journal** - [ ] Recovery mode is declared per partition and recorded in the map (R29) - [ ] DERIVED has no journal and recovers by checkpoint plus input replay (R30) - [ ] JOURNALED appends before applying, records consumed positions on the record, has one write path (R31) - [ ] MIXED is used only for disjoint, non-interacting state sets (R32) - [ ] The applier is deterministic; non-deterministic values are captured at write time, never recomputed (R33, R34) - [ ] LSNs are dense, gapless per generation, and journal-assigned (R35) - [ ] Checkpoints record LSN, all positions, format version, length, digest; digest is verified and a mismatch means absent (R36, R37) - [ ] Checkpoints are slot-ordered so a range extracts without a full read (R38) - [ ] Checkpoint kind is declared per class and recorded in every manifest, never inferred (R39) - [ ] An incremental checkpoint is a base plus an ordered delta chain, each naming its predecessor, every digest verified, a mismatch meaning the whole chain is absent (R40) - [ ] Delta chains have a stated maximum length and are compacted before it, so restore time has a ceiling (R41) - [ ] A base outlives every delta that depends on it, and both lineages are walked before any deletion (R42) - [ ] Formats are versioned, an unknown version is refused not partially read, and adjacent releases interoperate (R43, R44) - [ ] Deletion is lineage-driven, never age-driven (R45) - [ ] Deletion actually runs on a schedule, and retained-but-unreachable volume is monitored (R46) - [ ] Checkpoint age alarms against *configured* retention with an actionable margin (R47) - [ ] Restore is exercised continuously against real production checkpoints (R48) **Input and catch-up** - [ ] Coverage start is published; an under-covered query answers incomplete rather than partial (R49) - [ ] The stream is ordered per key and replayable; nothing depends on cross-key order (R50) - [ ] Shards and slots share a hash space and producers key by the service key; explicit hash keys are the exception (R51, R52) - [ ] Positions are per contributing shard, never one cursor (R53) - [ ] Parents drain fully before children on every reshard (R54) - [ ] Records carry a stable id, ingest deduplicates, and the dedup horizon equals the retention horizon (R55, R56) - [ ] Readiness requires a stated, per-class, reported freshness bound in both lag and age (R57) - [ ] Unowned slots' records are discarded and counted, never applied and never an error (R58) - [ ] Applier failures are bounded then quarantined, not retried across the fleet (R59) **Standbys** - [ ] A class can require a minimum synced standby count, and below it the partition is not healthy (R60) - [ ] Every class declares required failure-domain diversity between primary and standby, and 'none' is said explicitly (R61) - [ ] A partition not meeting its declared diversity is not reported healthy (R62) - [ ] Standbys consume the journal in JOURNALED mode and the stream in DERIVED mode (R63) - [ ] Applied position is reported and lag is a metric, not an inference (R64) - [ ] Standbys never write, and read only under an explicit staleness bound returning the served position (R65) - [ ] Commit policy is an explicit per-class setting (R66) - [ ] The three commit tiers are defined exactly as LOCAL, STANDBY, DURABLE, and nothing else is offered (R67) - [ ] Every class declares its tier and refuses to start without one; there is no default (R68) - [ ] The tier a write actually achieved is reported on the response (R69) - [ ] Below DURABLE, acked-but-not-durable count and oldest age are monitored and the exposure is stated in the class (R70) - [ ] The no-standby-remaining behaviour is explicitly configured, not emergent (R71) - [ ] Standbys verify the epoch of the stream they consume and stop on a mismatch (R72) - [ ] Promotion increments the epoch before the first write (R73) **Handoff and rollout** - [ ] There is exactly one seal, after which the outgoing primary never writes again (R74) - [ ] Pre-seal abort leaves the primary serving and the map unchanged (R75) - [ ] Post-seal the partition only moves forward; the outgoing primary never resumes (R76) - [ ] Uncommitted in-flight requests are rejected retryably, never as success and never permanently (R77) - [ ] The incoming primary is caught up before the seal; the seal is the last step (R78) - [ ] No map names a new primary before it reports ready under its new epoch (R79) - [ ] Rollout is a bounded sequence of handoffs, not a fleet restart or a platform rolling deploy (R80) - [ ] Concurrent handoffs are bounded per node and globally (R81) - [ ] Every handoff has a deadline and never waits indefinitely (R82) - [ ] Termination initiates handoff of every hosted partition (R83) - [ ] Shutdown phases run in order and do not undo each other (R84) - [ ] Phase deadlines are summed and validated against the platform budget at boot; a bad budget refuses to start (R85) - [ ] Unfinished work spills to object storage in the sink's own record shape (R86) - [ ] Unannounced kill is survivable, and that path is the one most exercised (R87) - [ ] Rollback obeys the same format-compatibility rule (R88) **Split and merge** - [ ] Splits divide at slot boundaries into children with higher generations and no empty child (R89) - [ ] Merges combine adjacent ranges into one higher-generation child (R90) - [ ] Lineage is recorded and retained while any child depends on it (R91) - [ ] Children restore from parents' checkpoints and journals filtered to their own ranges, then resume from sealed positions (R92) - [ ] Every parent seals before any child writes, and no parent resumes (R93) - [ ] Nothing depends on cross-key order across a split or merge (R94) - [ ] A child checkpoints before its parents' state becomes deletable (R95) - [ ] One slot is the split floor, and a hotter-than-a-node key is treated as a key-design problem (R96) **Routing** - [ ] Clients compute the slot and resolve against a cached map, never per-request lookup (R97) - [ ] Requests carry key, slot, and the client's map version (R98) - [ ] An owned, ready slot is served regardless of the client's map version (R99) - [ ] An owned, not-ready slot returns NOT_READY and never redirects (R100) - [ ] A non-owned slot resolves to MOVED, MISROUTED, or STALE_MAP by map-version comparison (R101) - [ ] No redirect is ever derived from a map older than the client's (R102) - [ ] MOVED carries enough map to route the retry, and the client applies it first (R103) - [ ] Redirects per logical request are bounded and the request fails rather than looping (R104) - [ ] MISROUTED is counted and alarmed as a correctness fault (R105) - [ ] Map refresh is jittered as well as redirect-driven (R106) - [ ] Mutating responses carry an `(epoch, LSN)` position compared numerically as a pair (R107) - [ ] Position tokens name their partition, are never compared across a lineage boundary, and a token from a vanished partition is answered explicitly (R108) **Placement and control plane** - [ ] The controller is never called on the serving path (R109) - [ ] The data plane serves from the last map through a controller outage (R110) - [ ] The controller stores no state inside the service it places (R111) - [ ] Partitions declare class and capacity; nodes declare class and budget; constraints bind first (R112) - [ ] An exclusive class means exactly one partition per node, including transiently (R113) - [ ] Placement enforces diversity as a hard constraint, and an unsatisfiable one places-and-reports rather than leaving the partition unowned (R114) - [ ] The objective is constraints, then movement, then balance — in that order (R115) - [ ] Concurrent moves are bounded and hysteresis is enforced per partition (R116) - [ ] Decisions are explicit, renderable, dry-runnable plans (R117) - [ ] Auto-apply versus hold is a per-plan-kind policy, and applied plans are recorded (R118) - [ ] A map moving more than the configured fraction of slots is refused without a recorded override (R119) - [ ] Publication is staged through a canary population observed against a stated outcome (R120) - [ ] Promotion is gated on the canary, and a regressing canary withdraws the version (R121) - [ ] The stage never delays a map completing a handoff already past its seal (R122) - [ ] A global freeze halts movement without a deploy and without touching serving (R123) **Scaling and shedding** - [ ] Scale-out pre-warms, and the map flips last (R124) - [ ] Scale-in is merges and handoffs, and the controller picks the node (R125) - [ ] The platform cannot terminate a node the controller did not select (R126) - [ ] Plans can be scheduled against a calendar (R127) - [ ] Scale-up lead time comes from measured restore time, not assumption (R128) - [ ] Compaction is abandonable and never a precondition for serving (R129) - [ ] Over-budget partitions shed at admission rather than buffering (R130) **Observability** - [ ] Every partition exposes id, generation, epoch, state, class, node, lease expiry, checkpoint age, LSN, input lag, standby lag (R131) - [ ] Every outcome is attributable to a partition and an epoch (R132) - [ ] An unowned slot is alarmed, and unowned is distinct from unhealthy (R133) - [ ] Duplicate primaries at one epoch alarm as a correctness fault (R134) - [ ] Handoff duration, seal-to-ready, and rejected-request counts are measured per handoff (R135) - [ ] Recovery is exercised in production on a schedule and feeds the lead time (R136) - [ ] Replay equivalence is tested continuously on real partitions: restore earlier, replay forward, compare (R137) - [ ] Silence is a positive liveness statement, not an absent signal (R138) - [ ] The oldest map version in use is visible (R139) ## Rationale The arguments behind these requirements are not made here; they are made in the Field Notes this document rests on, and repeating them would double its length without improving an implementation. Ownership is a lease with an epoch because [ZFN-37](/zfn/37-every-lock-is-a-lease/) is right that a lock which can outlive its holder is a deadlock scheduled for later, and because the fencing token is the only part of that arrangement which is actually safe. The lease is judged on the monotonic clock and recorded in wall time for the two-clock reasons in [ZFN-59](/zfn/59-two-clocks/) — a process pause is exactly the case where the two disagree and exactly the case that matters. The write path is append-then-apply because of [ZFN-65](/zfn/65-journal-writes-micro-batches/): sequence comes from the journal and never from the clock, and there is never a second path to one row. The input stream and the partition journal are both journals in the sense [ZFN-12](/zfn/12-queues-topics-journals/) draws, and neither is a queue — which is why replay is a first-class operation here rather than a recovery hack. The lease store sits apart from the placement controller because [ZFN-16](/zfn/16-separate-data-plane-control-plane/) is the difference between a control-plane outage that freezes change and one that stops serving, and the controller keeps no state inside the service it places because [ZFN-4](/zfn/4-incident-tooling-independence/) is about exactly the moment you need it not to. Handoff is a protocol rather than a stop because [ZFN-60](/zfn/60-graceful-shutdown-is-a-protocol/) sets out the four phases and, more usefully, says that graceful is the optimisation and crash-safe is the requirement. Checkpoint and journal formats evolve by expand, migrate, contract per [ZFN-62](/zfn/62-expand-migrate-contract/), because a rollout and its rollback both run two versions at once. Over-budget partitions shed rather than buffer for the flow-control reasons in [ZFN-13](/zfn/13-load-shedding-and-flow-control/). Position tokens on mutating responses are [ZFN-25](/zfn/25-read-your-writes-version-token/) applied to a partition. And the restore canary exists because [ZFN-36](/zfn/36-test-backups-by-restoring/) is unambiguous that a backup nobody has restored is not a backup — which is doubly true of a checkpoint, since the thing that expires underneath it is a retention window nobody is watching. Where the partition key is a tenant id, this document is also how [ZBP-3](/zbp/3-tenant-isolation/) satisfies its own requirement that a tenant be movable to a stronger isolation tier without application changes: an exclusive class and a pinned slot are what "dedicated" means for a stateful service. ## Changelog - **2026-08-21** (1.0.0): Adds *How it works*, a detailed non-normative walkthrough between the architecture and the requirements — the key space and why a slot is a hash prefix rather than a modulus, the three numbers a partition carries, what a lease cannot tell you and where safety actually comes from, the two kinds of state and why the difference decides recovery, the three recovery layers and the resume-position trap between two of them, what a standby buys at each commit tier, why the seal is the last step of a handoff, split and merge as a lineage walk, the five things a server can say and why two of them must stay distinct, what the controller is not, why scaling runs on a clock, and the life of one write end to end. It carries no normative keywords and defines no requirement: it cites the numbered ones, 53 of them, and where the two disagree the requirement is right. Adds a handoff sequence diagram and a recovery-layer diagram. No requirement changed. - **2026-08-21** (1.0.0): Draft revised after an expert-panel review, while still `draft` and with no consumers, so the requirement set moves without a major bump — which is what `draft` is for, and a door that closes the moment this goes `stable`. Adds signed partition maps verified before apply, so a delta relayed in a `MOVED` response cannot be a routing-hijack primitive; a movement cap and a canary stage on map publication, previously the highest-blast-radius operation with no brake; an operator force-takeover, fenced by the same epoch increment as an ordinary acquisition; declared checkpoint kinds with a normative incremental restore path and a bounded delta chain; a requirement that deletion actually runs; position tokens that do not compare across a lineage boundary; and replay equivalence promoted from prose to a requirement. Reconciles the global freeze with the post-seal rule they contradicted, and states the poison-record quarantine as an attack surface as well as a containment. Also makes the class the single declaration surface: the three commit tiers are defined normatively and every class must choose one with no default, since the tail between an acknowledgement and a durable segment is exposed to correlated loss and a default would decide that for a workload nobody examined; and failure-domain diversity between a primary and its standby is declared per class and enforced by placement, because a standby in its primary's own domain satisfies every count-based requirement while surviving nothing. 139 requirements. - **2026-08-21** (1.0.0): Initial publication as a draft. 115 requirements covering the slot key space and partition map, epoch-fenced ownership leases, checkpoint and journal recovery, input catch-up, standby replication, handoff and rollout, split and merge, the client routing contract, operator-governed placement, scheduled scaling, and observability.