Assistants are routinely asked how sure they are, why they answered as they did, and whether they can do a thing. Each answer is generated by the same process that produced the original output, which means it is a plausible continuation rather than an observation. What this rules out, and what to measure instead.
The request that feels reasonable
Ask an experienced person how sure they are and they consult something — a sense of how often they have been wrong about this sort of thing, a memory of the last time it went badly. The answer is a report on a state.
Ask a model the same and it produces the most plausible continuation of a conversation that contains a question about confidence. The output has the grammar of a report and the provenance of a prediction.
That disanalogy is the whole of this piece. Both answers arrive in the same shape, which is why the distinction is so easy to lose.
Three self-reports, each failing differently
Confidence. Generated as text, so it tracks how confident answers of this kind usually sound, and how the question was phrased. It is not reading a gauge, because there is no gauge being read.
Reasoning. A chain of explanation is produced alongside or after the answer. It is not a recording of how the answer arose. It can be coherent, persuasive, internally consistent and unrelated to the computation it purports to describe — which makes it worse than nothing when it is used as evidence, because it is convincing.
Capability. Can you do X is answered by what claims about capability usually look like. This is why systems assert abilities they do not have and disclaim abilities they do, without any inconsistency from their own side.
Why fluent wrongness is the hard case
The dangerous output is not the obviously broken one. It is the confident, well-organised, entirely wrong one.
Every human heuristic for spotting unreliability keys on surface features: hesitation, contradiction, disorganisation, hedging in the wrong places. Those features are informative when the speaker is a person, because production difficulty correlates with uncertainty.
Here the fluency is uniform. Correct and incorrect outputs are equally well-formed, equally composed, equally confident in register. The signal the heuristic depends on is simply absent, and the heuristic does not announce that it has stopped working.
What can actually be measured
Agreement across repeated samples. Detects instability. Does not detect a stable error, and stable errors are the ones that matter.
Agreement across rephrased prompts. Detects sensitivity to wording, which is often where fragility lives.
Verification against something outside the model. The only check that catches a confident consensus that happens to be wrong, and therefore the only one that addresses the failure mode this piece is about.
Task-level outcome measurement. Expensive, slow, and the only thing that measures what you actually care about. Everything cheaper is a proxy, and proxies are chosen for their cost.
Calibration is a property of a system, not of an answer
Confidence means something once it has been checked against outcomes across many cases. A number attached to a single answer, with no such history behind it, is decoration in the shape of a measurement.
And the aggregate does not rescue the instance. A system can be well calibrated over ten thousand cases and badly wrong about the one in front of you, which is the one you are about to act on.
What this means for building
Never route on self-reported confidence. It is not a weak signal to be used carefully; it is a generated artefact that resembles one.
Put verification outside the model wherever the stakes justify the cost, and decide that threshold deliberately rather than by default.
Prefer architectures that can cite something that exists over architectures that assert. A citation is checkable by a third party; an assertion is checkable only by asking again.
Treat any statement the system makes about its own behaviour as an input to verify, never as instrumentation. It is a participant in the evaluation, not an observer of it.
The honest version of "I do not know"
A system can abstain usefully. It cannot get there by being asked to be honest, because a request for honesty produces text about honesty.
The shape that works is mechanical: check whether support for an answer exists — a retrieved passage, a matching record, a computation that succeeded — and let the absence of support trigger the refusal. The model is not consulted about whether it knows. Something else establishes whether there is anything to know from, and the silence follows from that.
Then an abstention is a fact about the available evidence rather than a performance of modesty.
Close
The limitation is not a defect awaiting a better model. It is what the architecture is: a system that produces plausible text, including when the subject of the text is itself.
Build so that the things you rely on are established elsewhere, and the fluency becomes an asset rather than a hazard.