Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors
Quick Answer
This study investigates hallucination in large language models, attributing it to inference misalignment rather than missing knowledge.
Quick Take
This study investigates hallucination in large language models, attributing it to inference misalignment rather than missing knowledge. The authors introduce TrapQA, a diagnostic testbed revealing that biased latent inference can lead to hallucinations, impacting tasks like entity disambiguation and action choice.
Key Points
- Hallucination arises from inference misalignment, not just absent knowledge.
- TrapQA includes ScientistQA and Real-Life Constrained QA for testing.
- Entity disambiguation and action choice are particularly affected by biases.
- The framework formalizes how pretraining-frequency imbalance leads to inference loss.
- Two failure modes are identified: task-retrieval bias and key-selection bias.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2607. 00447v1 Announce Type: new Abstract: Large language models often produce hallucinated answers that violate prompt-level constraints. A key diagnostic question is whether these failures reflect missing knowledge, or whether the model has the relevant information but follows the wrong inference path. We study this phenomenon as inference misalignment: a mismatch between the answer supported by the prompt and the answer favored by statistically salient latent associations.
We formalize this view with a latent key-task model, in which pretraining-frequency imbalance can cause a shortcut path to dominate the constraint-sensitive path and induce positive inference loss. The framework predicts two failure modes: task-retrieval bias in entity disambiguation and key-selection bias in action choice. We introduce TrapQA, a controlled diagnostic testbed with two components.
ScientistQA tests disambiguation among similar scientists with supplementary factual probes, while Real-Life Constrained QA tests everyday constraint following under salient shortcuts. Our results show that hallucination can arise from biased latent inference rather than absent knowledge alone.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.