Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics
Quick Answer
This paper reveals that query position is a critical variable in diffusion large language models (dLLMs), impacting generation quality significantly.
Quick Take
It introduces Average Confidence ($\overline{C}$) as a new metric for iterative decoding and proposes Auto-ICL, an adaptive routing strategy that optimizes query placement, achieving near-oracle performance across various tasks.
Key Points
- Positional variance in dLLMs affects generation quality comparably to semantic quality.
- Traditional single-step confidence metrics are ineffective in dLLMs.
- Average Confidence ($\overline{C}$) tracks iterative decoding for better performance.
- Auto-ICL dynamically optimizes query placement without requiring training.
- The study highlights the importance of bidirectional attention in dLLMs.
Paper Resources
📖 Reader Mode
~2 min readAbstract:While In-Context Learning (ICL) is extensively studied in Autoregressive (AR) LLMs, its mechanism within Diffusion Large Language Models (dLLMs) remains largely unexplored. Unlike AR models restricted by unidirectional causal masking, dLLMs intrinsically utilize bidirectional attention, offering extensive spatial flexibility for query placement. Unfortunately, current practices conventionally inherit AR-style trailing-query templates, often overlooking the structural paradigm shift. This paper presents a comprehensive analysis unveiling that query position is actually a first-order variable in dLLMs. Through empirical decoupling, we demonstrate that positional variance impacts generation quality on par with example semantic quality. Internally, this positional sensitivity stems from a spatial ``Recency Effect'' in attention flow and task-dependent shifts in decoding trajectories. To mitigate this instability without ground-truth labels, we reveal that traditional single-step confidence ($C_{decoded}$) fails in dLLMs. Instead, we propose Average Confidence ($\overline{C}$), a novel metric tracking the iterative decoding process. By establishing the foundational spatial ICL baselines, we introduce Auto-ICL, a training-free adaptive routing strategy that dynamically optimizes query placement, robustly approaching oracle performance across heterogeneous reasoning and perception tasks.
| Comments: | 9 figures, 4 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.19349 [cs.CL] |
| (or arXiv:2606.19349v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.19349 arXiv-issued DOI via DataCite |
Submission history
From: Zhengheng Li [view email]
[v1]
Sun, 26 Apr 2026 20:22:47 UTC (4,349 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.