Two attacks share a name but not a mechanism: corrupting what a model learns from, and corrupting the model itself once training has finished. Why the second is the easier attack in practice, and what actually mitigates it.
Two attacks wearing one name
A checkpoint sits in a bucket, is copied to a cache, and is pulled onto a serving machine. Nobody in that chain retrains anything.
The model answering requests can still be a different model from the one that was evaluated, without a single training example having been touched.
That is the case for keeping the two apart. Data poisoning corrupts the material a model learns from and needs a foothold in the training pipeline. Model poisoning skips training entirely — it edits the weights after the fact, and needs only access to wherever the file is sitting.
Two doors. A team watching one of them has left the other open, without ever deciding to.
Why the artefact is the easier target
Teaching a model a poisoned behaviour through its training data is difficult. You have to guess which examples will move the weights the way you want, run enough of the pipeline to know whether it worked, and hope the effect survives the cleaning and regularisation downstream. It is slow, uncertain, and it leaves traces.
Editing a finished checkpoint is a file operation.
Copied between buckets during a migration, under permissions scoped for convenience rather than for this artefact.
Cached on a build machine that ten other projects also write to, none of them treating that cache as adversarial ground.
Mirrored to a second registry for redundancy, which doubles the number of places a swap can go unnoticed.
Passed between teams over object storage or a file share that was never hardened the way the cluster was.
The training environment usually has the tightest controls in the stack, because everybody remembers to lock the room where the expensive thing happens. The paths the finished file travels afterwards get none of that attention, precisely because the interesting work looks finished by then.
The object does not diff
A change to source code arrives as a diff, and somebody reads it before it merges. That discipline depends entirely on the object being readable.
A weight tensor is not readable in that sense. Two checkpoints can differ in ways that change behaviour on a narrow set of inputs while being, to any inspection a person could actually perform, identical.
There is no line to approve and no reviewer who could catch a targeted alteration by looking at the numbers. So provenance carries the whole load by itself — not because that is a good arrangement, but because review of the artefact is not on offer at any price.
Why a clean evaluation proves nothing
It still scores well on our suite sounds like reassurance.
Targeted poisoning is built to survive an aggregate, whether the corruption happened during training or after it. A benchmark averages across many inputs, and a change aimed at a narrow trigger barely moves an average.
Here it is sharper still. The attacker edited the finished object rather than hoping an optimisation run would carry their intent through. Nothing random is working against them. The edit does exactly what it was built to do, and the evaluation was never built to notice it.
What actually holds
None of this is exotic. The order matters more than the sophistication.
Verify signatures at load time, not at download time. A file that was clean on arrival can still be replaced before it is read. Checking at the wrong end of that gap is the most common way this control is implemented and defeated.
Record the checksum where the model was trained, and check it where it is used. Not somewhere in between, where both ends could have been compromised separately.
Let production pull from one registry. So that where did this come from has a single answer rather than several plausible ones.
Refuse to load anything unsigned, including when that is inconvenient. The exception granted under deadline is the breach, and it will not look like one at the time.
Close
None of this helps if the signing key lives in the same environment as the artefact it signs, which is the arrangement most teams actually have.
And a signature proves origin only. It has never had anything to say about whether the thing signed deserved to be trusted.