Evaluation scores cannot see a targeted change, because a targeted change is designed not to move a mean. Why steering a model is a more useful attack than degrading one, and why the defence is provenance rather than a better benchmark.
Why the metric stays flat
An average over ten thousand cases does not move when forty of them change.
That is not a weakness in the evaluation. It is what an average is for — it is supposed to be insensitive to small subsets, and it succeeds. The problem is that we then read a flat number as evidence that nothing happened.
A targeted change is built to live inside that insensitivity. If it moved the headline figure, it would be a worse attack.
Degrading against steering
Two different goals, and only one of them is interesting.
Degrading makes the model worse generally. It is loud, it shows up immediately in the numbers somebody already watches, and it achieves very little — a model that stops working gets rolled back within the hour.
Steering makes the model behave a particular way on a particular kind of input, and normally everywhere else. Nobody rolls it back, because nothing looks wrong.
The second is what anybody competent would want. Which means the failure mode worth designing against is the one that produces no signal at all.
Who can put an example in
The list is longer than most teams have written down.
A scraped source that somebody else controls. A labelling vendor, and the individual contractors behind them. An internal contributor. A public dataset that was clean when it was surveyed and has since been edited. A synthetic augmentation step generating from a model that was itself trained on something.
None of these requires breaking in. Each is a supported route by which text enters the process, working exactly as designed.
Provenance is the control
Not a better benchmark. Knowing where each example came from.
Record the origin of every example, at the granularity of the source rather than the batch. Without this, discovering a bad input tells you nothing about what else that source contributed.
Hold out adversarially chosen slices, not random ones. A random held-out set samples the same distribution the poison was hidden inside, so it inherits the same blindness. Slices built around the cases you would most mind being wrong about will not.
Compare each checkpoint against the previous one on those slices, not against an absolute threshold. You are looking for a change, and a change is only visible against the thing it changed from.
Make ingestion of a new source a decision somebody makes, rather than an event that happens when a scheduled job runs.
What this does not do
It does not detect somebody who knows your held-out slices.
If the slices are in the repository, then anybody who can add training data can usually also read which cases you check, and craft examples that behave on exactly those. The control is meaningfully weaker against an insider than against an outsider, and pretending otherwise would be the same error as trusting the flat metric.
What it does is raise the cost and narrow the field. The attacker now needs access to two things rather than one, and the smaller set of people who hold both is a set you can name.
That is a real improvement. It is not a solution, and the difference is worth stating plainly rather than leaving for somebody to discover later.
Close
The reassuring number was measuring the wrong thing, and measuring it correctly.
What you needed to know was never the average. It was where the data came from — and that is a question about process, which no amount of evaluation will answer for you.