Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics
Quick Answer
This paper reveals that query position is a critical variable in diffusion large language models (dLLMs), impacting generation quality significantly.
Quick Take
It introduces Average Confidence ($\overline{C}$) as a new metric for iterative decoding and proposes Auto-ICL, an adaptive routing strategy that optimizes query placement, achieving near-oracle performance across various tasks.
Key Points
- Positional variance in dLLMs affects generation quality comparably to semantic quality.
- Traditional single-step confidence metrics are ineffective in dLLMs.
- Average Confidence ($\overline{C}$) tracks iterative decoding for better performance.
- Auto-ICL dynamically optimizes query placement without requiring training.
- The study highlights the importance of bidirectional attention in dLLMs.
Paper Resources
Source Excerpt
While In-Context Learning (ICL) is extensively studied in Autoregressive (AR) , its mechanism within Diffusion Large Language Models (dLLMs) remains largely unexplored. Unlike AR models restricted by unidirectional causal masking, dLLMs intrinsically utilize bidirectional attention, offering extensive spatial flexibility for query placement. Unfortunately, current practices conventionally inherit AR-style trailing-query templates, often overlooking the structural paradigm shift. This paper p
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.