Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution
Quick Answer
This paper shows that PRIG, a new gradient attribution method, effectively localizes prompt ambiguity in large language models, achieving 0.840 AUROC on synthetic benchmarks and 0.891 AUROC on human-written gold sets.
Quick Take
It outperforms GPT-5.4 in sentence-level ambiguity identification, demonstrating that latent properties can be localized through intermediate representations.
Key Points
- PRIG attributes latent ambiguity to token positions using a probe logit.
- Achieved 0.840 AUROC on synthetic ambiguity datasets and 0.891 AUROC on gold benchmarks.
- Outperformed GPT-5.4 in identifying sentence-level ambiguities.
- Constructed synthetic datasets across coding, math, and writing tasks.
- Demonstrates that intermediate representations can reveal latent prompt properties.
Paper Resources
Source Excerpt
arXiv:2606. 05486v1 Announce Type: new Abstract: Prompt ambiguity is a common source of failure in , but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens. We introduce PRIG, a gradient attribution method that uses a probe logit to attribute latent ambiguity to token positions.
Specifically, PRIG trains a linear probe to distinguish clear prompts from ambiguous prompts and attributes the probe score to earlier token representations in the residual stream. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →RF-Agent: A Practical Framework for Building Language Agents for RFIC Design
RF-Agent introduces a novel framework for RF circuit design using , creating a unique RF-domain reasoning dataset with over 11,000 samples. The study reveals that domain-specific supervised fine-tuning and semantic retrieval strategies significantly enhance RF reasoning performance, particularly for smaller models.