Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution
Quick Answer
This paper shows that PRIG, a new gradient attribution method, effectively localizes prompt ambiguity in large language models, achieving 0.840 AUROC on synthetic benchmarks and 0.891 AUROC on human-written gold sets.
Quick Take
PRIG, a new gradient attribution method, effectively localizes prompt ambiguity in large language models, achieving 0.840 AUROC on synthetic benchmarks and 0.891 AUROC on human-written gold sets. It outperforms GPT-5.4 in sentence-level ambiguity identification, demonstrating that latent properties can be localized through intermediate representations.
Key Points
- PRIG attributes latent ambiguity to token positions using a probe logit.
- Achieved 0.840 AUROC on synthetic ambiguity datasets and 0.891 AUROC on gold benchmarks.
- Outperformed GPT-5.4 in identifying sentence-level ambiguities.
- Constructed synthetic datasets across coding, math, and writing tasks.
- Demonstrates that intermediate representations can reveal latent prompt properties.
Article Content
From source RSS / original summaryarXiv:2606. 05486v1 Announce Type: new Abstract: Prompt ambiguity is a common source of failure in large language models, but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens. We introduce PRIG, a gradient attribution method that uses a probe logit to attribute latent ambiguity to token positions.
Specifically, PRIG trains a linear probe to distinguish clear prompts from ambiguous prompts and attributes the probe score to earlier token representations in the residual stream. To enable token-level evaluation, we construct synthetic ambiguity datasets across coding, math, and writing by rewriting one task-critical sentence per prompt, and complement them with a human-written gold benchmark. In this setting, PRIG localizes ambiguous spans substantially better than gradient attribution baselines, achieving 0.
840 AUROC on the combined synthetic benchmark and 0. 891 AUROC on the gold set. It also outperforms GPT-5. 4 on sentence-level ambiguity identification and retains useful signal out-of-domain. These results establish PRIG as a practical tool for identifying which parts of a prompt are ambiguous. More broadly, they suggest that latent prompt properties can be localized through intermediate representations, rather than through output-level attribution.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.