Steerable Cultural Preference Optimization of Reward Models
Quick Answer
The paper introduces a novel reward model training algorithm, SCPO, which enhances large language models' alignment with diverse cultural preferences, achieving up to 7-point performance improvements on minority models across two datasets, while being 280% more data-efficient than traditional fine-tuning methods.
Key Points
- SCPO incorporates diverse cultural preferences in a balanced manner.
- Performance increases of up to 7 points on minority reward models were observed.
- The method is 280% more data-efficient than full-data fine-tuning.
- Bias analysis shows reduced excessive bias through a weighting method.
- Code for the model is publicly available on GitHub.
Paper Resources
📖 Reader Mode
~2 min readAbstract:It is essential for large language model (LLM) technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at this https URL
| Comments: | Accepted to Pluralistic Alignment @ ICML 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.18606 [cs.CL] |
| (or arXiv:2606.18606v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.18606 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minsik Oh [view email]
[v1]
Wed, 17 Jun 2026 02:10:07 UTC (290 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.