Annotate read-only and idempotent endpoints; make every mutation idempotent

TL;DR

Every endpoint declares its semantics, ideally in the schema (ZFN-14Field Note · currentZFN-14 — Define every API with a schema, and generate the clientsDefine every API with a machine-readable schema (OpenAPI, Protobuf, GraphQL) as the source of truth, and generate clients and server stubs from it — never hand-roll request-building and JSON parsing. Hand-written clients drift and break silently; check schema compatibility in CI.Open ZFN-14 →): is it read-only (safe — no side effects), idempotent (repeating it leaves the same state), or neither? Make those properties machine-readable so infrastructure can act on them: safely retry an idempotent call, route a read to a replica, cache a safe response, let automated tooling call read-only endpoints freely.

And the rule that makes the system resilient: every endpoint that changes state must be idempotent. Retries are inevitable — networks time out, clients re-send, queues redeliver — and a retry of a non-idempotent mutation double-charges, double-creates, double-sends. Use natural idempotency where you can (set-this-state operations) and idempotency keys where you can’t (create, charge, send). Idempotency is the precondition that “retry from the startField Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 →” quietly depends on.

Context

Two facts about distributed systems make this non-optional:

• You can never tell a lost request from a lost response. When a call fails or times out, the client doesn’t know whether the server did the work. Its only safe move is to retry — which is exactly what ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 → says to build in from the start. But a retry of an operation that isn’t idempotent applies it twice: two charges, two orders, two emails. “Exactly once” delivery is a fiction; what you actually get is at-least-once, plus idempotent processing = effectively once. The same is true for messages off a queue (ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 →), which redeliver by design and demand idempotent consumers.
• Infrastructure wants to know what’s safe, and currently has to guess. A gateway could retry a failed call, a router could send a read to a replica, a cache could serve a response — but only if it knows the call is safe/idempotent. Without a declared annotation, each client hand-codes that knowledge, inconsistently, and the safe optimisations either don’t happen or happen unsafely.

HTTP encodes some of this in method semantics (GET/HEAD safe; PUT/DELETE idempotent; POST neither), but methods alone are too coarse and easy to misuse, and non-HTTP APIs have nothing. The property needs to be declared explicitly, per endpoint, and trustworthy.

Recommendation

Declare endpoint semantics in the contract, and make every mutation idempotent.

• Annotate every endpoint, in the schema. Mark each as read-only/safe, idempotent, or idempotency-key-required, as part of the schema/IDL (ZFN-14Field Note · currentZFN-14 — Define every API with a schema, and generate the clientsDefine every API with a machine-readable schema (OpenAPI, Protobuf, GraphQL) as the source of truth, and generate clients and server stubs from it — never hand-roll request-building and JSON parsing. Hand-written clients drift and break silently; check schema compatibility in CI.Open ZFN-14 →) so it’s machine-readable and generated into clients and gateways — not a comment a human has to find.
• Don’t lie about it. A read-only annotation is a promise of no side effects — no writes, no state-changing downstream calls, no “read that also logs a mutation.” Tooling, routing, and retries will trust it; if it’s wrong, the bug is subtle and nasty.
• Make every state-changing endpoint idempotent. Design mutations so applying them twice equals applying them once:
  • Natural idempotency — prefer operations that set a state rather than increment it (PUT-style “make it so”), keyed by a client-known identifier, so a repeat is a no-op.
  • Idempotency keys — for operations with no natural key (create, charge, send), require a client-supplied unique idempotency key; the server records it and returns the original result on a repeat instead of doing the work again. Dedup within a sensible window and persist the outcome.
  • Conditional writes — use optimistic concurrency (ETags / version numbers / compare-and-set) so a replayed or racing update can’t silently double-apply.
• Let infrastructure use the annotations. Once semantics are declared and honest: gateways/clients retry idempotent calls automatically (closing the loop with ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 →), read-only traffic routes to replicas and caches, and automated callers (test harnesses, agents) can exercise read-only endpoints without fear of side effects.

Scope. “Idempotent” here means effect idempotence — the observable state ends up the same. It doesn’t forbid an endpoint from returning different representations over time (a GET reflects current state); it forbids a repeated mutation from applying more than once.

Consequences

Easier:

• Retries become safe, which is what makes the whole resilience story (ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 →) actually work without double-applying effects.
• Infrastructure can optimise safely on declared semantics — automatic retries, replica routing, caching — instead of guessing.
• Queue consumers can be at-least-once and still correct, because processing is idempotent (ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 →).
• Automated and AI tooling can call read-only endpoints freely, which is increasingly valuable.

Harder:

• Idempotency keys are real machinery: storing keys and outcomes, choosing a dedup window, handling the in-flight-duplicate race, and deciding how long to remember. Worth it; not free.
• Some operations resist natural idempotency and need careful key or conditional-write design.
• Honest annotations require discipline — a read-only endpoint that quietly writes (an audit log, a last-seen timestamp) is a trap, so even “harmless” side effects have to be accounted for.
• Declaring and maintaining the semantics is one more thing the contract must keep accurate.

References

• ZFN-13Field Note · currentZFN-13 — Fail fast and push back: retries, load shedding, and flow controlBuild client retries (backoff, jitter, Retry-After) from day one. Under overload, shed fast and push the failure back to the source to retry — don't retry internally and amplify it. Flow-control everywhere, bound every queue, and don't take more work than you can finish in time.Open ZFN-13 → — retries from the start are only safe if mutations are idempotent; this note is that precondition.
• ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 → — queues redeliver (at-least-once); idempotent consumers are how you stay correct.
• ZFN-14Field Note · currentZFN-14 — Define every API with a schema, and generate the clientsDefine every API with a machine-readable schema (OpenAPI, Protobuf, GraphQL) as the source of truth, and generate clients and server stubs from it — never hand-roll request-building and JSON parsing. Hand-written clients drift and break silently; check schema compatibility in CI.Open ZFN-14 → — declare read-only/idempotent in the schema so it’s machine-readable and generated, not folklore.
• RFC 9110 §9.2 — Safe and idempotent methods; idempotency-key patterns (e.g. Stripe’s) as a worked design.

Changelog

• 2026-06-12: First published as a Field Note.