Secure AI Atlas mark Secure AI Atlas SECURITY & GOVERNANCE
EN / ES

Risk

Data and Model Poisoning

Training data, fine-tuning datasets, or embeddings are deliberately manipulated to introduce vulnerabilities, backdoors, or biases that compromise model security and behaviour.

Training DataSupply ChainModel Integrity
Domain
Data
Model
Mechanism
Integrity
Boundary
Between training, fine-tuning or embedding data and model weights
Trigger
Manipulated training or fine-tuning data is introduced
Consequence
Model integrity and behavior are compromised; backdoors or bias
Required evidence
  • Dataset provenance and hashes
  • Training and fine-tuning audit records
  • Behavioral backdoor testing

Exposure

Data poisoning occurs when an attacker interferes with the data used to train, fine-tune, or embed a model. The contamination may happen at any stage — during initial pre-training, vendor-supplied fine-tuning, third-party dataset ingestion, or user-provided feedback loops — and the effects persist silently until the model is queried.

The risk is compounded when organisations rely on pre-trained models from external sources without visibility into the training pipeline, or when fine-tuning datasets include public contributions, scraped content, or unverified third-party data.

Signals

  • Unexplained degradation in model accuracy, safety behaviour, or refusal rates after a dataset update.
  • Model responses diverge sharply from expected outputs for specific inputs or demographic groups.
  • Training or fine-tuning data sources include user-contributed content without integrity verification.
  • Vendor-supplied model updates lack a verifiable provenance chain or cryptographic signature.

Failure pattern

An attacker inserts malicious examples into a dataset that the target model will later consume. The model internalises the poison during training. At inference time, a trigger input — a specific phrase, image pattern, or data structure — activates the backdoor, causing the model to produce attacker-chosen outputs, bypass safety filters, or leak training data.

  • Verify provenance and integrity of all training and fine-tuning data sources.
  • Maintain a controlled, versioned, and auditable data catalogue for each model version.
  • Apply cryptographic signing to datasets and model artefacts.
  • Monitor model behaviour after every data update for anomalous output patterns.
  • Isolate user feedback loops from direct model retraining without human review.

Provenance

SOURCE
MAPPED
Not recorded
ANALYZED
Not recorded
UNRESOLVED
Not recorded

Last reviewed: Sep 20, 2026