Misinformation

Exposure

Misinformation from language models is not merely an accuracy problem — it becomes a security concern when users or downstream systems act on fabricated information. The model’s output style — confident, articulate, well-structured — lends credibility to content that may be entirely false.

The risk compounds when model output feeds into automated decision pipelines, customer-facing applications, medical or legal advisory systems, or public communication channels where the cost of correction is high and the window for verification is narrow.

Signals

• Model generates citations, statistics, or technical details that cannot be verified against source material.
• Users or downstream systems treat model output as authoritative without independent verification.
• Automated workflows execute decisions based on unverified model-generated conclusions.
• Content produced by the model is published or distributed before human review.

Failure pattern

A user queries the model for factual information. The model, lacking grounding in verifiable sources, generates a plausible-sounding but fabricated response complete with invented citations, statistics, and technical terminology. The user accepts the output as accurate. In a security context, this could lead to incorrect risk assessments, flawed incident response decisions, or compliance submissions based on non-existent evidence.

Related controls

• Ground model responses in verifiable, authoritative sources where accuracy is critical.
• Implement retrieval-augmented generation with auditable source attribution.
• Require human review for model output that informs high-stakes decisions.
• Display confidence indicators and source links alongside generated content.
• Log and monitor factuality complaints to identify systematic hallucination patterns.