Risk
Data and Model Poisoning
Training data, fine-tuning datasets, or embeddings are deliberately manipulated to introduce vulnerabilities, backdoors, or biases that compromise model security and behaviour.
Exposure
Data poisoning occurs when an attacker interferes with the data used to train, fine-tune, or embed a model. The contamination may happen at any stage — during initial pre-training, vendor-supplied fine-tuning, third-party dataset ingestion, or user-provided feedback loops — and the effects persist silently until the model is queried.
The risk is compounded when organisations rely on pre-trained models from external sources without visibility into the training pipeline, or when fine-tuning datasets include public contributions, scraped content, or unverified third-party data.
Signals
- Unexplained degradation in model accuracy, safety behaviour, or refusal rates after a dataset update.
- Model responses diverge sharply from expected outputs for specific inputs or demographic groups.
- Training or fine-tuning data sources include user-contributed content without integrity verification.
- Vendor-supplied model updates lack a verifiable provenance chain or cryptographic signature.
Failure pattern
An attacker inserts malicious examples into a dataset that the target model will later consume. The model internalises the poison during training. At inference time, a trigger input — a specific phrase, image pattern, or data structure — activates the backdoor, causing the model to produce attacker-chosen outputs, bypass safety filters, or leak training data.
Related controls
- Verify provenance and integrity of all training and fine-tuning data sources.
- Maintain a controlled, versioned, and auditable data catalogue for each model version.
- Apply cryptographic signing to datasets and model artefacts.
- Monitor model behaviour after every data update for anomalous output patterns.
- Isolate user feedback loops from direct model retraining without human review.