From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
Quick Take
The Preference Delta Aggregation (PDA) framework enhances large language models (LLMs) like Qwen3 8B by effectively aggregating weak signals from weaker model pairs, achieving performance improvements of 6.8 and 7.3 points on knowledge reasoning and agentic search benchmarks, respectively. This method outperforms all single and multi-delta baselines, demonstrating the value of combining complementary capabilities across distinct preference deltas.
Key Points
- PDA is the first framework to aggregate weak signals from weaker model pairs.
- Introduces Geometric Alignment Merging (GAM) to mitigate directional interference.
- Achieves 6.8 and 7.3 average performance gains on knowledge reasoning and agentic search.
- Outperforms the best single-delta baseline by 2.1 and 4.3 points.
- Demonstrates effective composition of complementary capabilities across preference deltas.
Article Content
From source RSS / original summaryarXiv:2606. 00357v1 Announce Type: new Abstract: Training strong large language models (LLMs) requires high-quality supervision, which is often scarce. Recent work shows that paired preference data from weak-weaker model pairs (e. g. , Qwen3 4B over 1. 7B), despite the limited quality of individual responses, can provide an effective supervision signal through relative quality deltas, which we term a "weak" signal.
This motivates a key research question: can multiple "weak" signals be constructively aggregated for improving strong models (e. g. , Qwen3 8B)? To this end, we propose Preference Delta Aggregation (PDA), the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging.
To further mitigate directional interference during LoRA merging, we introduce Geometric Alignment Merging (GAM), a geometry-aware merging method that aligns adapter subspaces before aggregation, enabling more robust composition of diverse deltas. Evaluations on knowledge reasoning and agentic search benchmarks show that aggregating multiple "weak" signals pushes performance beyond any single signal, with further gains as additional signals are incorporated.
Correspondingly, PDA with GAM improves the strong model by 6. 8 and 7. 3 points on average for knowledge reasoning and agentic search, respectively. It outperforms all single-delta and multi-delta baselines, exceeding the best single-delta baseline by 2. 1 and 4. 3 points. Further analysis attributes these gains to the effective composition of complementary capabilities encoded across distinct preference deltas.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →MindGames Arena Generalization Track: In2AI Solution with Delayed Per-Step Reward Attribution
The In2AI solution introduces delayed per-step reward attribution for training language model agents in multi-agent environments, achieving top performance on the MindGames Arena benchmark at NeurIPS 2025. An 8-billion-parameter model outperformed larger proprietary systems, including GPT-5, in competitive play, demonstrating enhanced stability and sample efficiency in reinforcement learning.