field-note · 2026-09-07 · 5 min read
Exit Zero Means Nothing
Six autonomous coding jobs reported success. Three had produced nothing usable. What a green result actually proves, and what it does not.
I dispatched six autonomous coding jobs against one repository in a single session. All six terminated. Five reported success in their own words, with tidy summaries and criterion-by-criterion tables.
Three of them had produced nothing I could use.
That ratio is the subject of this note. Not because the agents were bad — the ones that worked did good work — but because the signal I was using to tell the difference did not carry the information I thought it did.
What the three failures looked like
They failed in three distinct ways, and none of them looked like failure.
The one that wrote broken code. Asked to move Content-Security-Policy into middleware, it produced a middleware() function that constructed a dead Response object with status 404 and then return null. Its own reasoning log shows it noticed: "I created a middleware.ts file, but it has issues — I used a 404 response and returned null, which won't work." It then exited 0. The file was on disk, the process was happy, and the code could not function.
The one that wrote nothing at all. Asked to bring a form to WCAG 2.2 AA, it printed the complete finished file into its log, with a mapping of each change to its success criterion. Then it exited 0 with a clean working tree. git status was empty. It had composed the work and never written it.
The one that made things worse. It converted three prose paragraphs into a <ul> of <li> elements, and reported it as a fix for WCAG 1.3.1 Info and Relationships. Marking flowing prose as a list makes a screen reader announce "list, 3 items" for ordinary text. It is not a fix. It is a regression wearing the vocabulary of a fix.
A fourth job did nothing for a reason that was my fault: its first action was refused because I had written absolute paths into its instructions, and it ran in an isolated worktree where reading outside is denied. It burned its whole run on a permission error and exited 0. Same green result, entirely different cause.
The same failure, without any agents
It would be comfortable to file this under "LLMs are unreliable." It is not that specific. Here is the identical failure shape in a shell pipeline I wrote myself:
npm run cf:build 2>&1 | tail -6 && echo "CFBUILD=ok"
This printed CFBUILD=ok. The build had failed outright with ERR_MODULE_NOT_FOUND and produced no output directory at all.
A pipeline's exit status is the exit status of its last command. tail succeeds essentially always. The && was testing whether tail worked. Every gate written that way is decorative — and it is a very natural way to write one, because you want to see the last few lines.
The same construction later reported PUSH=0 for a git push the remote had rejected.
And once more, in markup
The third instance had no process and no shell. The site had a skip link — the affordance that lets a keyboard user jump past navigation. Every static check passed it: the element existed, it was the first tab stop, it became visible on focus, and it pointed at #main, which existed.
It did nothing.
<main id="main"> had no tabindex. Following the fragment scrolled the page and updated the URL, but focus stayed on <body>. The next Tab therefore resumed from the top of the document and landed on the first header link — putting the user back inside the nine-item navigation the link exists to bypass.
before: activeElement = body, next Tab → header navigation
after: activeElement = main#main, next Tab → first link inside <main>
The fix is one attribute. The point is that no amount of reading the markup reveals the defect, because nothing in the markup is wrong.
What these have in common
In every case the reported signal was adjacent to the thing I cared about, and I treated adjacency as identity.
OC_EXIT=0means the process ended. It does not mean the work happened.- A pipeline's
0means the last command succeeded. It does not mean the first one did. - A skip link in the DOM means the affordance is present. It does not mean it functions.
- A perfect CSP header means the server sent a policy. It does not mean the document satisfies it.
Each is a true statement about something. None is a statement about what I needed to know.
The checks that actually separate them
These are cheap, and between them they caught everything above:
- Ask the filesystem, not the worker.
git statusand a real diff against the base.uncommitted=0alongside a large log is the signature of a job that composed work and never wrote it. - Capture the real exit code. Redirect to a file, test
$?, inspect the file separately. Never put a pipe between a command and the&&that judges it. - Operate the affordance. For anything whose entire value is behavioural — a skip link, a focus trap, a scroll region, a keyboard shortcut — press the keys and assert where focus actually lands. Reading the DOM cannot answer this.
- Verify the artifact exists. After a build claims success, check the output directory.
.open-next/worker.jseither exists or it does not, and that question has no opinion in it. - Count elements, not substrings. A grep for
onerror=alertmatched a payload that was correctly escaped, because escaped text still contains those characters.grep -coE "<img[^>]{0,60}onerror"returned 0 and settled it.
The part I would restate
The problem is not that automated workers lie. It is that success is reported by the thing being evaluated, and a report is not evidence about the world — it is evidence about the reporter.
This generalises well past agents. A test suite reporting green is a claim by the suite. A deploy reporting success is a claim by the deployer. Each is worth something, and each is worth exactly as much as the independence of the check behind it.
Verify once, against something that has no stake in the answer. Then it is done.