Hallucinations

Hallucination is generated content that lacks support from the available evidence. It can be false, or merely impossible to verify from the supplied context. Fluency hides the failure: a language model predicts plausible tokens rather than checking each claim against reality.

Several mechanisms can produce the same symptom. Sparse or stale training data leaves the model with weak evidence. Preference tuning may reward a confident, agreeable answer. Sampling can then select an invented detail from several plausible continuations. None of these causes can be diagnosed from polished prose alone.

flowchart TD
    A[Query] --> B[Model generates claim]
    B --> C{Claim supported by context}
    C -->|Yes| D[Grounded]
    C -->|No| E[Hallucination]

If retrieved context says Austen wrote Pride and Prejudice and the answer names Dickens, the contradiction is visible. Many production failures are less obvious because the model adds a plausible date or citation that the source never mentioned. Generation explains how sampling and output constraints shape these continuations.

Intrinsic and Extrinsic Hallucination

Ji et al. separate two cases. An intrinsic hallucination contradicts the source, such as naming Dickens when the passage names Austen. An extrinsic hallucination adds a claim the source does not contain. That extra claim may happen to be true, but the response has no evidence for it. Intrinsic failures can often be found by comparing answer and context. Extrinsic claims need another source or an explicit abstention policy.

Detection

Detection starts by splitting an answer into claims. Each technique answers a different question about those claims.

• NLI-based checking scores a claim against source context as entailed, neutral, or contradicted. It works best when the required evidence is already present and the relationship is stated clearly.
• Self-consistency (SelfCheckGPT) compares several samples from the same prompt. Contradictory or unstable details are warning signals. But stable repetition still does not prove truth, even when the method needs no external knowledge base.
• LLM-as-judge estimates answer faithfulness against supplied context. It handles semantic variation better than exact matching, but the evaluator is another fallible model and needs calibration against reviewed examples.
• Atomic fact verification (FActScore) breaks a response into small claims, retrieves evidence for each one, and scores support separately. This makes the failing claim visible instead of hiding it inside an answer-level score.

For a RAG system, RAG Evaluation must measure retrieval and generation separately. A faithful answer cannot recover evidence that retrieval never supplied.

Mitigation

Grounding is the usual starting point. More expensive checks belong on claims whose failure has a real cost.

• Retrieval grounding (RAG) supplies passages that the answer can cite and check against, turning many recall tasks into source-based synthesis. It reduces reliance on parametric recall without guaranteeing correctness. The legal-system study in the references still found hallucinations above 17% across evaluated tools. See RAG.
• Chain-of-Verification (CoVe) drafts an answer, creates verification questions, answers them independently, then revises the draft. The separation matters because verification should not treat the draft’s own claims as evidence.
• Constrained output enforces a schema and allowed values. It prevents structural invention, which protects downstream automation, but a valid field can still contain a false claim.
• Abstention returns a defined fallback when evidence is missing. The threshold must be calibrated because an overly cautious system becomes useless.
• Tool-backed generation sends factual subproblems to authoritative databases or calculators and synthesizes their results. The tool response still needs provenance and error handling.

Guardrails turns these techniques into enforced runtime behavior: citations can be checked, unsupported answers can abstain, and invalid tool requests can be rejected.

Pitfalls

RAG Does Not Eliminate Hallucinations

• Failure. RAG is treated as proof that an answer is grounded.
• Cause. The corpus may lack the fact, retrieval may miss it, or generation may add a claim beyond the returned passages.
• Control. Track retrieval recall separately from faithfulness. Then verify material claims against the passages actually used.

Preference Tuning Can Reward the Wrong Signal

• Failure. An answer becomes more agreeable or polished without becoming better supported.
• Cause. Preference data can reward agreement with the user even when that agreement is wrong. The cited sycophancy work demonstrates this failure mode. It does not imply that every RLHF model is less factual.
• Control. Evaluate factual precision and calibration alongside preference scores. Reviewed counterexamples should include prompts with a false premise so agreement is not mistaken for quality.

Over-Aggressive Mitigation Causes Over-Refusal

• Failure. The system refuses answerable questions or returns fragments despite sufficient evidence.
• Cause. A strict abstention threshold trades fabrication risk for under-answering.
• Control. Measure answer coverage beside faithfulness and set thresholds by consequence. Medical advice and an internal search summary should not share the same operating point.

Tradeoffs

Approach	What it covers	Runtime cost	Main limitation
RAG grounding	Supplies external evidence	Retrieval and indexing	Bad retrieval silently caps answer quality
Self-consistency	Finds unstable claims	Several generations	Repeated agreement is not proof
NLI fact checking	Finds contradiction or missing support in context	One or more checks per claim	The checker has its own error rate
LLM-as-judge	Handles semantic claim-to-context comparison	Evaluator-model calls	Requires calibration and can reproduce model bias
Constrained output	Prevents structural fabrication	Usually low	Does not establish factual truth
Abstention policy	Stops unsupported answers	Low at runtime	Poor calibration causes over-refusal

For evidence-backed answers, start with retrieval and claim-to-context checks. Self-consistency is worth its extra calls when an unstable answer would be costly. An LLM judge is usually easier to calibrate offline before it is trusted as a live gate.

Questions

Why can an LLM still hallucinate when using RAG?

RAG gives the model context, but it does not prove that the answer follows from it. The source may not contain the fact, retrieval may miss the right passage, or the model may ignore the passage and add unsupported details. Retrieval quality and answer faithfulness should be measured separately so the failing stage is clear.

Why can preference tuning make an answer sound better without making it more factual?

Preference data may reward agreement, confidence, or style without checking whether a claim is supported. The model can then produce a polished answer that accepts a false premise. Factual accuracy and calibration still need separate evaluation from preference scores.

References

• Survey of hallucination in natural language generation: intrinsic and extrinsic taxonomy (Ji et al., ACM Computing Surveys 2022)
• FActScore: atomic evaluation of factual precision (Min et al., EMNLP 2023)
• SelfCheckGPT: black-box hallucination detection (Manakul et al., EMNLP 2023)
• Reduce hallucinations (Anthropic Docs)
• Chain-of-Verification reduces hallucination in LLMs (Dhuliawala et al., Meta AI 2023)
• Hallucination in RAG-based legal AI tools (Magesh et al., JELS 2025)