Token-weighted Direct Preference Optimization with Attention
Quick Answer
The paper introduces Token-weighted Direct Preference Optimization (TwDPO) and its implementation AttentionPO, which enhances Large Language Models' alignment with human preferences by using token weights derived from the model's attention.
Quick Take
This method outperforms existing Preference Optimization techniques on benchmarks like AlpacaEval and MT-Bench, requiring only two additional forward passes per example, thus improving efficiency and robustness.
Key Points
- TwDPO uses token weights to enhance model alignment with human preferences.
- AttentionPO leverages 's attention for token weight estimation.
- Significant performance improvements on AlpacaEval, MT-Bench, and ArenaHard.
- Requires only two additional forward passes per example for efficiency.
- Outperforms existing token-level Preference Optimization methods.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.21883 [cs.CL] |
| (or arXiv:2605.21883v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21883 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chengyu Huang [view email]
[v1]
Thu, 21 May 2026 01:43:09 UTC (364 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.