---
id: 19
title: "Annotate read-only and idempotent endpoints; make every mutation idempotent"
status: current
date: 2026-06-12
authors:
  - "Theo Zourzouvillys"
tags: [api, reliability, architecture, correctness]
summary: "Annotate every endpoint as read-only (safe) or idempotent, in the schema, so infrastructure can retry, route to replicas, and cache safely. Make every state-changing endpoint idempotent (idempotency keys for create/charge/send); a non-idempotent retry double-applies."
supersedes: null
superseded_by: null
aliases: []
---

## TL;DR

Every endpoint declares its **semantics**, ideally in the schema ([ZFN-14](/zfn/14-schema-first-apis-generate-clients/)):
is it **read-only** (safe — no side effects), **idempotent** (repeating it leaves the same state), or
neither? Make those properties **machine-readable** so infrastructure can act on them: safely retry an
idempotent call, route a read to a replica, cache a safe response, let automated tooling call read-only
endpoints freely.

And the rule that makes the system resilient: **every endpoint that changes state must be idempotent.**
Retries are inevitable — networks time out, clients re-send, queues redeliver — and a retry of a
*non*-idempotent mutation double-charges, double-creates, double-sends. Use natural idempotency where
you can (set-this-state operations) and **idempotency keys** where you can't (create, charge, send).
Idempotency is the precondition that "[retry from the start](/zfn/13-load-shedding-and-flow-control/)"
quietly depends on.

## Context

Two facts about distributed systems make this non-optional:

- **You can never tell a lost request from a lost response.** When a call fails or times out, the client
  doesn't know whether the server did the work. Its only safe move is to **retry** — which is exactly
  what [ZFN-13](/zfn/13-load-shedding-and-flow-control/) says to build in from the start. But a retry of
  an operation that *isn't* idempotent applies it twice: two charges, two orders, two emails. "Exactly
  once" delivery is a fiction; what you actually get is **at-least-once, plus idempotent processing =
  effectively once.** The same is true for messages off a queue ([ZFN-12](/zfn/12-queues-topics-journals/)),
  which redeliver by design and demand idempotent consumers.
- **Infrastructure wants to know what's safe, and currently has to guess.** A gateway could retry a
  failed call, a router could send a read to a replica, a cache could serve a response — but only if it
  knows the call is safe/idempotent. Without a declared annotation, each client hand-codes that
  knowledge, inconsistently, and the safe optimisations either don't happen or happen unsafely.

HTTP encodes some of this in method semantics (`GET`/`HEAD` safe; `PUT`/`DELETE` idempotent; `POST`
neither), but methods alone are too coarse and easy to misuse, and non-HTTP APIs have nothing. The
property needs to be **declared explicitly**, per endpoint, and trustworthy.

## Recommendation

**Declare endpoint semantics in the contract, and make every mutation idempotent.**

- **Annotate every endpoint, in the schema.** Mark each as **read-only/safe**, **idempotent**, or
  **idempotency-key-required**, as part of the schema/IDL ([ZFN-14](/zfn/14-schema-first-apis-generate-clients/))
  so it's machine-readable and generated into clients and gateways — not a comment a human has to find.
- **Don't lie about it.** A read-only annotation is a promise of **no side effects** — no writes, no
  state-changing downstream calls, no "read that also logs a mutation." Tooling, routing, and retries
  will trust it; if it's wrong, the bug is subtle and nasty.
- **Make every state-changing endpoint idempotent.** Design mutations so applying them twice equals
  applying them once:
  - **Natural idempotency** — prefer operations that *set* a state rather than *increment* it
    (`PUT`-style "make it so"), keyed by a client-known identifier, so a repeat is a no-op.
  - **Idempotency keys** — for operations with no natural key (create, charge, send), require a
    client-supplied unique **idempotency key**; the server records it and returns the original result on
    a repeat instead of doing the work again. Dedup within a sensible window and persist the outcome.
  - **Conditional writes** — use optimistic concurrency (ETags / version numbers / compare-and-set) so a
    replayed or racing update can't silently double-apply.
- **Let infrastructure use the annotations.** Once semantics are declared and honest: gateways/clients
  retry idempotent calls automatically (closing the loop with [ZFN-13](/zfn/13-load-shedding-and-flow-control/)),
  read-only traffic routes to replicas and caches, and automated callers (test harnesses, agents) can
  exercise read-only endpoints without fear of side effects.

**Scope.** "Idempotent" here means *effect* idempotence — the observable state ends up the same. It
doesn't forbid an endpoint from returning different *representations* over time (a GET reflects current
state); it forbids a repeated *mutation* from applying more than once.

## Consequences

**Easier:**

- Retries become safe, which is what makes the whole resilience story
  ([ZFN-13](/zfn/13-load-shedding-and-flow-control/)) actually work without double-applying effects.
- Infrastructure can optimise safely on declared semantics — automatic retries, replica routing,
  caching — instead of guessing.
- Queue consumers can be at-least-once and still correct, because processing is idempotent
  ([ZFN-12](/zfn/12-queues-topics-journals/)).
- Automated and AI tooling can call read-only endpoints freely, which is increasingly valuable.

**Harder:**

- Idempotency keys are real machinery: storing keys and outcomes, choosing a dedup window, handling the
  in-flight-duplicate race, and deciding how long to remember. Worth it; not free.
- Some operations resist natural idempotency and need careful key or conditional-write design.
- Honest annotations require discipline — a read-only endpoint that quietly writes (an audit log, a
  last-seen timestamp) is a trap, so even "harmless" side effects have to be accounted for.
- Declaring and maintaining the semantics is one more thing the contract must keep accurate.

## References

- [ZFN-13](/zfn/13-load-shedding-and-flow-control/) — retries from the start are only *safe* if
  mutations are idempotent; this note is that precondition.
- [ZFN-12](/zfn/12-queues-topics-journals/) — queues redeliver (at-least-once); idempotent consumers are
  how you stay correct.
- [ZFN-14](/zfn/14-schema-first-apis-generate-clients/) — declare read-only/idempotent in the schema so
  it's machine-readable and generated, not folklore.
- [RFC 9110 §9.2 — Safe and idempotent methods](https://www.rfc-editor.org/rfc/rfc9110#section-9.2);
  idempotency-key patterns (e.g. Stripe's) as a worked design.

## Changelog

- **2026-06-12**: First published as a Field Note.