Nobody Reviews the Assembly

The question

A few days ago I mentioned to a colleague that my token usage was running around thirty billion tokens a month. He asked the right question:

“Ok I have to ask, how are you thinking about review/guidance of the agents to make sure slop is not being shipped at this level of usage?”

That’s Jeff, who I work with at Clerk. One correction before anything else, because it is the sort of thing this essay is about: thirty billion tokens prices out around $28,000 at published API rates.¹1But that is not what I pay. I’m on a subscription; the figure is an imputed list price my tooling computes.

That volume is not one enormous agentic coding session. It is many smaller ones running at once: I have my fingers in a lot of pies, and my experience is wide enough to be useful in most of them, so there are always a dozen or so threads of work in flight. As I write this, thirty-nine agent sessions open across eight different project areas. How that is held together is a subject for another day.

That part is genuinely new. Not the agents, which I have been working with since December, but the parallelism. I have tried raising the number of threads several times now, and each time it bit me: this is not how we have worked in this craft until now, and I was not good at it. But every time I put the count up I find myself getting better at it, the last few weeks especially. Sitting and waiting for an agent to finish is an absolute waste, so I keep pushing. My brain has been adjusting; I can feel it happening.

Even a few hours of working at this level leaves me more drained than a sixteen-hour coding session ever did. I have only been running at this rate a matter of weeks, and whether it is sustainable, or still works well a year from now, I don’t know yet.

Back to Jeff’s question. It contains an assumption: that quality comes from a human reading the output. Reading the diff has been genuinely valuable to me for other things. It is how you teach someone, how you give a junior engineer feedback specific enough to act on, how a team learns the shape of its own codebase. But as the mechanism that catches defects at scale, it has never been the thing doing the work.

Reading the code never scaled

I read everything once. Starting out in the late 1990s I went down until I found the floor, and often enough the floor had a hole in it: I found bugs in interpreters, bugs in compilers. That works until the system stops fitting in your head. Then you are on a team, and mandatory tool-enforced code review, a practice barely fifteen years old,²2GitHub shipped pull requests in February 2008 as a notification with nothing gated; Gerrit followed that November and could actually block a change; GitHub did not ship required reviews until 2016. Somewhere in that range is when it became the default, and every point in it is younger than a great deal of the software the practice now defends. younger than much of the software it defends, turns out not to fix it either, because the binding constraint was never the absence of a diff view. It was that careful attention is expensive and finite, so we rationed it: one reviewer, one pass, on the version before the fixes.

Then I spent years participating in the IETF (ZFN-45Field Note · currentZFN-45 — Read the standards; better yet, help write themLearn to read standards docs — RFCs, W3C recs — fluently; they're the primary source, not a last resort. Even better, get involved: reading them well makes you a sharper builder, and helping write them is the best protocol education there is.Open ZFN-45 →), and that is where I learned what a specification is actually for.³3The Internet Engineering Task Force: the body that standardises most of the protocols the internet runs on. There is no membership and there are no votes. The work happens in working groups, in public, on mailing lists, and every specification starts as an Internet-Draft that anyone may write and anyone may take apart. Its motto is “rough consensus and running code”, and the second half is not decoration.

A standards document has a brutal requirement attached to it: a stranger must be able to implement it correctly without asking you anything, and their implementation has to interoperate with one written by another stranger who also could not ask. That standard does something to how you write. You stop describing what the thing does and start pinning down what is required versus optional, what happens at every edge, which behaviours are forbidden, and (the part most engineering documents skip) what you considered and rejected, and why. Ambiguity is not a style problem there. It is a defect, and it ships to everyone.

SIP sat in what was then the RAI area (real-time applications and infrastructure) and it had the sharpest test of a document I have ever seen: SIPit, the interoperability bake-off the SIP Forum ran for years: a week in one room with sixty-odd organisations’ stacks, testing them against each other. You cannot argue with that result. Two implementations, both written by competent people, both conforming to the same document, that cannot complete a call between them, that is not a code defect. It is a specification defect, and you are standing in front of it. The events existed in part to find exactly those and feed them back into the standards work.

I have spent a lot of time on the other side of that contract too. I have led implementations of SMTP, POP3 and IMAP, a DNS server, several SIP stacks, an HTTP server back in the early 2000s, a BGP speaker, a NetFlow library, a YANG parser, a GraphQL implementation. Some specifications are a pleasure to build from and some are miserable, and the difference is not length or formality. It is whether the author decided the hard cases before you arrived.

A good specification makes implementation boring. You read a section, write the code it describes, and the code is right. No archaeology, no inference, no reading the same paragraph six times hoping it resolves on the sixth. When you reach an edge, the document has already been there and tells you which way to go, and when you finish, your implementation talks to everyone else’s, because they were told the same thing.

A bad one turns you into an investigator. You reconstruct intent from examples, guess what “should” was meant to imply, post a question to a mailing list and wait a week, and eventually implement what you think was intended, as does everyone else, differently. The defects are not in your code. They are in the gap the document left, and they surface years later as interoperability failures nobody can attribute to anyone.

You also learn what a wrong sentence costs. RFC 3261, the SIP specification, told implementers to destroy transaction state the moment a success response was sent, so a retransmission arrived looking like a brand new request. I co-wrote the correction, RFC 6026, with Robert Sparks in 2010, eight years after the original.⁴4Its errata registry lists, as I write this, twelve verified defects and sixteen more held for a future update; around eight further RFCs exist to correct it. Implementers noticed early: RFC 6026 records that “many implementations in the field have made proprietary adjustments to their transaction logic to avoid this error,” and a correcting draft was public in 2007, three years before it became authoritative. In the meantime the industry quietly diverged into incompatible private workarounds, every one of them a correct response to a document that was wrong.

And none of this is peculiar to protocol work. Every system I have architected since (SaaS platforms, the distributed and scalability work that keeps them standing) has had the same shape. The hard part was never the code. It was deciding how the thing works: where state lives, what happens when a dependency is slow rather than down, which invariants have to hold at every seam, how it behaves at its worst moment rather than its best. Get those wrong and no amount of clean implementation saves you; get them right and the implementation is the fun part.⁵5And I miss it, badly. Writing the implementation was always the most satisfying part of the craft, the point where a design stops being an argument and becomes a thing that runs. I have traded it for a rate of progress I would not give back, and I am not going to pretend that is a clean trade.

That is the education, and it transfers exactly. A model implementing from a specification is in the position I spent years in: building from a document it cannot interrogate, filling any silence with a guess. It cannot post to the mailing list and wait a week. So the property that made a spec good then is the property that makes it good now, and the lesson is not that documents are unreliable but the opposite: that the document is the highest-leverage artifact in the system, that precision in it pays for decades, and that the expensive defects are decisions rather than mistakes, which means they are visible in prose and nearly invisible in an implementation.

Running about a hundred developers taught me the same thing from the other end, where I could not be across every layer even if I wanted to be: if the design is right and written down, and you review the edges (the boundaries, the contracts, the invariants) then what happens inside a component is not that vital. We accepted this about compilers thirty years ago; nobody reviews the assembly.⁶6Assembly is the human-readable form of the instructions a processor actually runs, what the compiler emits from the code you wrote. Engineers once read it to check the compiler had not botched the translation. Almost nobody does now: you review the source and trust the toolchain. If you have never once looked at any, that is rather the point. I wrote it down in June as a field note about running teams, govern the contract between teams, not the code inside themField Note · currentZFN-47 — Govern the contract between teams, not the code inside themTeams own services end to end; one team owns the gateway that dispatches to them. Govern exactly one thing centrally — the contract at the boundary (schema, identity, errors, idempotency) — and enforce it at runtime. Don't mandate libraries; ship them as an opt-in blueprint.Open ZFN-47 →, and it turns out to be most of the answer for agents.⁷7The limit of the analogy, before someone else names it: gcc -O2 is deterministic and its input was reviewed, whereas an agent’s output is the source of truth, and a design document does not compile into it. It carries the principle (govern the boundary, test the behaviour) and not the guarantee.

The baseline was never zero

There is a second assumption in the question, and it is the one that matters more: that the old way produced zero.

Every team I have worked on shipped code it wasn’t happy with. Not occasionally. Continuously. Bugs, regressions, outages, the change that looked fine in review and took production down on a Friday. That was true with two engineers and true with a hundred, true of code reviewed by people I would trust with anything, and true of a specification hundreds of experts read adversarially for years. It is not a junior-developer problem. It is a property of building software through a review process, and we have simply normalised it.

So “how do you make sure slop isn’t shipped” has no answer that ends in zero, and it didn’t have one before either. Self-driving is the analogy I keep returning to: nobody serious claims an autonomous vehicle will never crash, and the claim that matters is accidents per mile, measured against human drivers who crash constantly and whose crashes we long ago stopped finding remarkable.⁸8The better autonomous-vehicle analyses publish incidents per mile against a defined human benchmark and expose their method to outside scrutiny. Mine do none of that. But miles are far more externally constrained than shipping volume is. A fixed amount of driving needs doing, so a lower rate per mile really is fewer crashes. I choose how much I ship, which means I picked a denominator that grows with exactly the thing under scrutiny.

Which is why the bar I hold myself to is stated absolutely, where there is no denominator to game: no more production-impacting defects than the old way produced, counted as a raw number (fewer if I can manage it) while shipping several times the volume. By that I mean a bug that reaches production and costs somebody something, incidents included, and never a review finding. Review findings are cheap and I want as many as I can get. A flat rate at several times the volume would not qualify: that is several times the defects, and nobody downstream of me experiences a rate. They experience outages.

Everything that follows is a process description, and a process description is not evidence, so before I give you mine, the admission that goes with it: I do not have that measurement, and I do not think I can honestly get it.

A before-and-after needs a comparable before: the same system, the same definition of what counts as a production-impacting defect, the same severity threshold, a window of the same length. Mine would have to come from a different codebase in a different era, scored against standards I would be setting after the fact, knowing what I wanted them to show. That is not a measurement, it is a reconstruction, and the whole complaint of this essay is about confident claims resting on unstated bases. So I am not going to manufacture one.

What I can say is what would make the claim testable from here: definitions fixed in writing before the counting starts, production defects and incidents counted against them, and long enough under both ways of working to tell a trend from a quiet quarter. I have the recent half of that and not the historical half. So the comparative claim in this essay is one I believe and cannot support. Take the rest as a report on how I work, not as evidence that it is better.

And if incidents start turning up at a rate that worries me, I slow down: fewer changes in flight, the full panel on everything, no merge without a human reading the diff. That is a judgement call rather than a threshold, and I would rather say so than dress it up as a trigger I could be held to.

What actually changed

In December I read every line. Not from policy but because I had no idea yet what this thing was good and bad at. Trust accrued the way it does with a colleague: slowly, on evidence, as I learned where it was strong and where it drifted. Over months the reading got selective, then targeted, then mostly stopped.

On a bounded, well-specified implementation task, it writes better code than I do. Not faster. Better! More consistent, more thorough at the edges, more willing to write the error path and the test for it,⁹9Whether that test asserts anything is a separate question, and a real one, see “Make the build the reviewer” below. and free of two ordinary sources of bad code: fatigue and boredom. Compare first drafts to first drafts, before either meets a reviewer, and I do not think I win on a typical piece of work.

That is not the same claim as “better at engineering”, and the difference is the whole essay. It is better at implementing a bounded specification. It is not better at knowing which artifact should exist, what it must never do, where the boundaries fall, or when its own confident output is wrong. A machine drafted this essay, and the review caught several confidently-stated claims in it that were plainly false. So I define the interfaces and the invariants, and I draw the boxes small enough that the judgement left inside one is bounded, which is the actual reason the split works, and the part that stops being true if you draw them too big.

The judgement I have just called my whole contribution was built by writing code and being wrong about it. Every instinct I have (this retry loop will hammer the database, this error path has never once been executed) came from shipping that exact mistake and living with it. I don’t do that any more.¹⁰10And how anyone junior reaches this from here is a problem I find genuinely frightening. I got it by doing the work badly for years, in public, with consequences that landed on me, which is a ladder that only exists while somebody is still climbing it by hand. If the first rungs are gone, I don’t know what replaces them, and I notice that nobody selling the tools has an answer either. Code review was one of those rungs. Not as a defect filter, but as the place a senior engineer told you why your change was wrong. I have no idea what does that job now. Whether the instinct keeps working when it is no longer being fed, I genuinely don’t know: there is no dashboard for “his judgement got worse last quarter”, and if mine has already started to slip, everything would feel exactly as it does now. The loop is not fully severed (I still debug production and read incident data) but writing the implementation was a teacher, and I have stopped attending.

But the coding is not where the leverage came from:

A competent, adversarial, domain-aware reviewer used to cost a day of a senior engineer’s attention. It now costs about as much as a coffee.

Every quality practice we have is shaped by that old cost. We review once, not four times. We review the initial change and not the change-after-the-fixes. We staff one reviewer per pull request, not five specialists. We accept that review is a sampling exercise. None of those constraints hold any more, and the practices haven’t caught up, mine included, until I rebuilt them from the new cost structure.

The loop, in short

The single biggest change is not that I review differently. It is what I review. Almost all of the attention that used to go on diffs now goes on the document that precedes them, and the document is then handed to the model to implement.

That trade is heavily in your favour. A defect in a specification costs a sentence; the same defect found in code costs a rewrite, and found in production costs an incident. Reviewing the code checks the workmanship. Reviewing the spec checks whether the thing should be built that way at all.

So the mechanics below are mostly mechanics for reviewing prose, and I have compressed them here to the parts that carry a caveat.

Write the decision down before the code. Work that turns on a durable design decision starts with a record (ZFN-1Field Note · currentZFN-1 — Keep engineering decision recordsRecord significant engineering decisions as short, versioned markdown files — context, decision, consequences. Write one for cross-team contracts, directional principles, hard-to-reverse choices, and conventions others must follow. Cite them instead of re-arguing.Open ZFN-1 →) stating what we decided and what we accept as a consequence. Two fields, both build-validated: status is what we decided, implementation is how far it exists in reality. They are orthogonal, because the most dangerous defect in a mature system is a property the documents assert confidently that the implementation does not deliver.

Send the design to a panel of distinct lenses, not more reviewers. This is the load-bearing step, and note what is in front of them: a design document, usually before a line of it exists. Five reviewers, each told which failure class they own: adversarial abuse, tenant isolation, operational reversibility, availability, and a skeptic whose whole job is to say what to cut. Three passes from the same perspective find the same things three times and give you false confidence. They need tools rather than the diff, because a reviewer who can run the suite tells you the test asserts nothing while one with only the diff reports style opinions. And a clean report has to be an acceptable result, said out loud, or the reviewer manufactures findings to justify the round.

Address every finding, then review what you changed. A fix, a written refusal in the pull request, or a filed issue; never silence. Then send the change as it now stands to a second, different model, and repeat until everyone comes back clean. Not until the findings look small. Until there are none. This is the step that pays, because fixes introduce defects, reliably enough that you should expect the round which checks the repairs to find some. I have watched that happen on prose as well as code: this essay went through the same loop, and two separate rounds found defects introduced by the previous round’s fixes.¹¹11One of them caught me breaking a promise I had made four paragraphs earlier. Worth saying that two models agreeing a thing is clean is weaker evidence than it sounds, because they may share blind spots, and a loop that terminates on agreement terminates on their agreement, not on correctness. That is an argument for more different lenses, not for trusting the exit condition.

Promote the recurring classes into build failures. A rule that depends on someone remembering it is not a rule, it’s a hope. But a check’s hit rate is not one either: a guard you have not seen fail is not a guard, and I have had a green suite that was not even evidence the edit had applied, because a formatter had moved the line the substitution was anchored to.

Then accept at the deploy, not the diff. At merge time what I read is the decision record, the panel’s blocking findings and how each was resolved, then merge it once CI is green, then babysit it to production. The five-lens panel is for designs and substantial changes; it is not run on every commit, which is a gap between what my own written standard prescribes and what I do. And the arbiter, when the skeptic says cut and the availability reviewer says add, is me.

Being wrong is cheapest in prose

Reviewing the specification only helps if the specification is worth reviewing. Mine have got steadily more prescriptive. Not longer for the sake of it, but denser in constraint. The test I apply to a paragraph is whether it removes a decision from the implementer. If it adds background instead, it is exposition, and exposition is what makes a long document tiring rather than useful. A long document that is mostly constraint is a specification. A long document that is mostly narrative is just long.

Concretely, this is what I am prescriptive about:

Two states, kept in separate fields. What we decided, and what actually runs today. “The service validates the token” is either a description of reality or a promise about the future, and those two readings produce opposite work: integrate with it, or go build it. Both mistakes are expensive, in opposite directions. Keeping the two apart in the front matter, and failing the build when the second goes stale, is the cheapest fix I know for a whole class of confident, wrong implementation.

Headings that assert rather than label. “Background”, “Current state”, “Motivation” are true and carry nothing. A heading with a subject and a verb (there has never been a cache, the evaluator is shared, not reimplemented) means the table of contents carries the argument. You can read the outline of a thousand-line document and already know what it decided.

“X, not Y”, the displaced alternative, named. The Y is almost always the more obvious choice; that is exactly why it needs writing down. Someone who does not know the alternative was considered will pick it, confidently, and the resulting code will look correct. A model will do the same thing more reliably than a person, because the rejected design is usually the most common pattern in its training data. Writing the rejection down is the only thing that stops it being re-proposed as an improvement.

Rejections steelmanned, then killed by mechanism. State the alternative as a serious proposal, say what makes it genuinely attractive, then name the specific way it fails. A dismissal teaches nothing and gets re-derived by the next reader. A rejection with a mechanism attached closes the branch for good.

The load-bearing lines, marked as such. Some details are incidental and some are the entire design. If a line carries the invariant, say so, and say what breaks when someone tidies it, because the tidier-looking refactor that quietly reintroduces the race you removed is the most common way a good design degrades.

Citations to code, with line numbers, at a specific commit. Given a real path and line, an implementer reads what is actually there. Given a prose description of a function, they build against a plausible invention of it. The commit is what keeps that true a month later, when the line has moved.

Closed sets in tables, and a parameter registry when the set has to grow. An enumerated vocabulary gets copied. The same vocabulary described in prose gets paraphrased, and near-miss spellings appear that nothing in the type system objects to. Where new values are expected, the IETF answer is a registry rather than a longer table: the values, plus who may add one and by what procedure. That turns extension into a defined act rather than everyone inventing their own spelling in private.

Failure modes with a direction. Not “this could break”, but how it breaks, which way it breaks, and whether anyone would notice. A failure mode stated that precisely is a regression test with the assertion already written.

Obligations numbered, and marked precondition or follow-up. This is what stops the easy half of a design shipping alone.

Which numbers are judgements. Told nothing, a reader treats every figure as measured and builds on it. Told which are guesses, they ask.

None of this was invented for machines. Every habit on that list was adopted because a person got something wrong first. What changed is that a fresh context window turns out to be precisely the reader the format was always aimed at, the one described in every decision-record template ever written, the colleague two years from now who was not in the room. No memory of the debate, no shared history, full authority to act, and a strong pull toward whatever pattern is most common.

So the one-line version is this: a good specification does not tell you what to build. It removes every way of building it that would have been wrong, which amounts to the same thing, and survives being read by someone, or something, that was not in the room.

That is also why the length stops mattering in the way it used to. A model does not skim, does not get bored, and does not resent a nine-hundred-line document. The binding constraint was never the reader’s patience. It is whether each paragraph closes something.

Records for the product, notes for the taste

Two different corpora do two different jobs here, and I would not want either doing the other’s.

A decision record is scoped to the thing being built: this service, this boundary, this wire format, this migration. It is the product’s memory: what we decided, what we accepted as the cost, and what exists in main today.

The Field Notes on this site are the other half, and being read by a model is now most of why I keep writing them. They are not about any one system. They are the architectural positions I keep reaching for, written down once so I stop retyping them into design reviews. ZFN-1Field Note · currentZFN-1 — Keep engineering decision recordsRecord significant engineering decisions as short, versioned markdown files — context, decision, consequences. Write one for cross-team contracts, directional principles, hard-to-reverse choices, and conventions others must follow. Cite them instead of re-arguing.Open ZFN-1 → on recording decisions at all, ZFN-16Field Note · currentZFN-16 — Separate the data plane from the control planeSplit the serving path (data plane) from the management path (control plane). The data plane keeps serving on last-known-good config when the control plane is down — never call it on the hot path. Coupling them turns a control-plane bug into a serving outage.Open ZFN-16 → on where a control plane stops, ZFN-12Field Note · currentZFN-12 — Queues, topics, and journals are different tools — don't conflate themQueues (competing consumers), topics (fan-out), and journals (ordered, replayable logs) give different guarantees. Don't conflate them; a pipeline often uses several. Prefer journals over topics, but not where head-of-line blocking hurts. With queues, bound the concurrency.Open ZFN-12 → on picking between a queue and a journal. Numbered, dated, and published as data as well as prose (notes.json, llms.txt) so the corpus a person reads is the corpus an agent loads.

The division is clean. The records say what we decided to build here. The notes say how I think things should be built anywhere. Hand an agent both and it has the specification and the taste, and the taste is the half that used to exist only in my head, which is precisely the half I was complaining I could not scale.

So, here’s the answer

I don’t review my way to quality, because I never could. Not alone, not in a working group where hundreds of experts read a document for years and still left defects in it, and not with a hundred engineers reporting to me. What changed is that the writing stopped being the constrained resource and the reviewing got cheap in the same moment. So the review moved to where it was always worth more: I spend my attention on the specification, argue about it with reviewers who each hold a different lens, and hand the result to the model to implement. Less code review, more document review, and the documents are the part I would defend in a year’s time.

And I stay accountable for all of it. Everything that goes out under my name is something I drove, can account for, and can defend. That obligation doesn’t move because the draft came from a machine (ZFN-26Field Note · currentZFN-26 — LLM-assisted content needs no disclaimer, only a human who can back itDrafting engineering content with an LLM needs no disclaimer — inside a team or codebase that already shares the co-sign norm. The obligation is human co-signing. Writing for strangers inverts it: there, stating your policy builds trust rather than diluting a default.Open ZFN-26 →). Which is also the load-bearing constraint: I can work this way because I have almost thirty years of context on how these systems fail. Run the same setup without that context and you get the same volume with less of the filtering, and how much less I don’t know, and that is an open problemOpen problem · currentZFN-28 — Capability without understanding: brute-force LLM PRsAn open problem: people brute-force PRs with LLMs in domains they don't understand, taking on more than their knowledge supports — and the struggle that used to teach them is smoothed away. How do we stop un-understood code without killing learning or banning a good tool?Open ZFN-28 → I have not solved. It is worse than a distribution problem, because I acquired that context by doing the thing I have just told you I no longer do: writing it by hand, and being wrong in ways that cost me. “The next generation won’t need it” is the answer I trust least, and the one the economics will keep offering.

Nobody reviews the assembly, because we review the source it was generated from. The specification is now the source. But everybody should be reviewing that, and the boundary, and the behaviour, and for the first time, that is cheap enough to do as many times as it takes.

← All writing