Beyond Consensus: Trace-Level Synthesis in Mixture of Agents
Quick Take
The study introduces Self-Consistent Mixture of Agents, which leverages trace-level synthesis for improved problem-solving in LLMs, outperforming traditional majority voting methods. By utilizing semantic-preserving input perturbations, this approach consistently generates correct solutions through diverse reasoning traces, enhancing performance across various complex tasks like structured reasoning and competitive programming.
Key Points
- Majority voting is shown to be lossy compared to trace-level aggregation.
- The new model generates trace diversity through semantic-preserving perturbations.
- Self-Consistent Mixture of Agents consistently outperforms heterogeneous model pools.
- Correct intermediate steps are recovered from minority chains that voting discards.
- The focus shifts from answers to reasoning traces for better aggregation.
Article Content
From source RSS / original summaryarXiv:2605. 29116v1 Announce Type: new Abstract: When multiple LLM agents solve the same problem, standard practice compresses each agent's reasoning into a majority vote or layered synthesis, treating agreement as the finish line. We show this is unnecessarily lossy: an LLM aggregator that reads complete reasoning traces recovers correct solutions even when agents unanimously agree, with beneficial corrections consistently outweighing harmful ones -- the \emph{aggregation paradox}.
Majority voting has a ceiling that perturbation diversity does not raise (error correlations are identical); the aggregator's gain comes from trace-level complementarity, assembling correct intermediate steps from minority chains that voting discards.
These findings motivate Self-Consistent Mixture of Agents which generates trace diversity through semantic-preserving input perturbations, safeguards the majority via anchored refinement with provable non-degradation guarantees, and always synthesizes -- never gates on consensus. A single model with perturbation-induced trace variation outperforms heterogeneous model pools across structured reasoning, PhD-level science, competition mathematics, and competitive programming.
The unit of aggregation should be the reasoning trace, not the answer.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.