Tools claiming to identify machine-written text are deployed in schools, universities and hiring, on the assumption that they are approximately right. The reasons they cannot be reliable are structural rather than temporary, and the errors fall hardest on people least able to contest them.
What the tools claim
A passage goes in and a percentage comes out. The number is described as a probability and consumed as a verdict, and the distance between those two readings is where every problem in this piece lives.
A probability invites interpretation. A verdict ends a conversation. What is produced is the first and what happens next treats it as the second.
Why it is structurally hard
The systems producing this text were trained, explicitly, to be indistinguishable from human writing. A detector is therefore asking whether a system achieved its stated objective — and the better the generation gets, the worse the detection must do. The two are not independent.
Worse, the signals available are not distinctive. Unusually even phrasing, limited surprise, regular sentence construction: these are properties of machine output and equally of careful editing, of formal registers, of technical writing, and of anybody taught to write plainly and doing it well.
And any signal that becomes known becomes a target. Publishing what a detector looks for tells the next generation what to avoid. The target moves, and it moves in the direction of the detector's failure, by construction.
Whom the false positives land on
This is the part that makes it more than a technical disappointment. The errors are not scattered randomly, so they do not average out across a population.
Writers working in a second language often produce careful, regular, deliberately plain prose — the exact profile these tools flag. People whose writing is unusually consistent, including many neurodivergent writers, produce the same signature. Students drilled in a rigid essay structure produce it by instruction. Anybody using ordinary grammar assistance produces it as a side effect of the tool doing its job.
Each of those groups is also, on average, less equipped to contest an accusation: less familiar with the appeals process, less confident challenging an institution, more likely to be believed to have cheated in the first place. The errors and the inability to contest them land on the same people.
Why an unreliable tool still gets deployed
Because it answers a real need, and the need is not accuracy.
An institution facing a dispute requires something it can put in a file and point to. A number does that. A considered judgement by an experienced marker, who is probably right, does not — it looks like an opinion, and opinions are contestable in a way a score is not.
The demand is for defensibility. The tool supplies defensibility. Whether it supplies accuracy is a separate question that the procurement process was never really asking.
What cannot be recovered by the accused
The accusation is generated in a second. The response takes days.
Version histories must be produced. Drafts must be found. A person is asked to explain their own sentences, which is both difficult to do convincingly and humiliating to be asked. Much of the evidence that would help was never created, because nobody keeps a forensic record of writing an essay.
That asymmetry is the mechanism. It is not that people cannot defend themselves. It is that defending yourself costs vastly more than accusing you did, and the cost is paid in the currency of somebody who is already losing.
What would be defensible instead
Assessment designed so the question does not arise, which is more work and actually works.
Writing produced under observation. Oral defence of submitted work, where understanding is demonstrated rather than inferred. Assignments requiring specific personal, local or recent material that a general-purpose system has no access to. And process evidence collected routinely from everybody, as a normal part of the work, rather than demanded from one person after suspicion has already formed.
The last is the important one. Evidence gathered as standard is evidence; evidence demanded under suspicion is a test the accused is already failing.
Close
Deploying an unreliable classifier to make consequential decisions about people is not a technical choice. It is a decision about who absorbs the error, and it has already been made in favour of the institution.