Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution
Quick Answer
This paper shows that PRIG, a new gradient attribution method, effectively localizes prompt ambiguity in large language models, achieving 0.840 AUROC on synthetic benchmarks and 0.891 AUROC on human-written gold sets.
Quick Take
It outperforms GPT-5.4 in sentence-level ambiguity identification, demonstrating that latent properties can be localized through intermediate representations.
Key Points
- PRIG attributes latent ambiguity to token positions using a probe logit.
- Achieved 0.840 AUROC on synthetic ambiguity datasets and 0.891 AUROC on gold benchmarks.
- Outperformed GPT-5.4 in identifying sentence-level ambiguities.
- Constructed synthetic datasets across coding, math, and writing tasks.
- Demonstrates that intermediate representations can reveal latent prompt properties.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 05486v1 Announce Type: new Abstract: Prompt ambiguity is a common source of failure in , but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens. We introduce PRIG, a gradient attribution method that uses a probe logit to attribute latent ambiguity to token positions.
Specifically, PRIG trains a linear probe to distinguish clear prompts from ambiguous prompts and attributes the probe score to earlier token representations in the residual stream. To enable token-level evaluation, we construct synthetic ambiguity datasets across coding, math, and writing by rewriting one task-critical sentence per prompt, and complement them with a human-written gold benchmark. In this setting, PRIG localizes ambiguous spans substantially better than gradient attribution baselines, achieving 0.
840 AUROC on the combined synthetic benchmark and 0. 891 AUROC on the gold set. It also outperforms GPT-5. 4 on sentence-level ambiguity identification and retains useful signal out-of-domain. These results establish PRIG as a practical tool for identifying which parts of a prompt are ambiguous. More broadly, they suggest that latent prompt properties can be localized through intermediate representations, rather than through output-level attribution.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.