Multi-Objective Exploration and Preference Optimization via Mutual Information
Quick Answer
The paper introduces MI-EPO, an information-theoretic framework that enhances multi-objective alignment in large language models by maximizing mutual information among responses and preferences.
Quick Take
This method significantly improves alignment and controllability in generated outputs, outperforming existing techniques in safe alignment and helpful assistant tasks.
Key Points
- MI-EPO maximizes joint conditional mutual information for better alignment.
- The framework decomposes objective alignment and preference-aware exploration.
- Experiments show significant improvements in response alignment with preference vectors.
- Outputs generated are more controllable and distinguishable across preferences.
- MI-EPO addresses exploration uncertainty in multi-objective alignment.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.
| Comments: | Accepted at ECML/PKDD 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.01392 [cs.CL] |
| (or arXiv:2607.01392v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01392 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongyan Xie [view email]
[v1]
Wed, 1 Jul 2026 18:50:14 UTC (973 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.