Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models
Quick Answer
This paper shows that The Mixture-of-Translators (MoT) framework enables the translation of key-value caches across heterogeneous large language models (LLMs) like Qwen2.5, GPT-2, and OPT, achieving 51.0% average closed-set QA accuracy.
Quick Take
By employing multiple translator modules and a Context Correction Loss, MoT preserves downstream QA performance and retains 96.3% context quality in long-context generation, facilitating scalable memory reuse.
Key Points
- MoT translates KV caches between models, enhancing multi-model reasoning.
- Achieves 51.0% average closed-set QA accuracy with Qwen2.5-7B-scale translation.
- Introduces Context Correction Loss to align target trajectories effectively.
- Retains 96.3% context quality in long-context cache-augmented generation.
- Addresses cache translation failures through translator mixtures and corrections.
DeepSignal Analysis
What happened
The Mixture-of-Translators (MoT) framework has been introduced to translate key-value caches across different large language models (LLMs) such as Qwen2.5, GPT-2, and OPT. This approach aims to enhance the scalability of multi-model reasoning and long-context generation by enabling cache reuse.
Key evidence
- MoT achieves an average closed-set QA accuracy of 51.0% when translating caches among Qwen2.5, GPT-2, and OPT models.
- The framework employs multiple translator modules and a Context Correction Loss to maintain 96.3% context quality in long-context generation.
- MoT addresses two failure modes in cache translation: propagated translation shift and last-state shift, improving the accuracy of the translated caches.
Why it matters
The inability to reuse key-value caches across different LLMs has limited the efficiency of multi-model systems. MoT's approach to translating these caches could significantly reduce redundancy and improve performance in applications requiring long-context reasoning. This advancement may lead to more effective use of resources and better integration of various LLMs in practical scenarios.
What to watch
Paper Resources
Source Excerpt
Heterogeneous (LLM) systems increasingly rely on shared contexts, retrieved evidence, and dialogue histories, yet their internal key-value (KV) caches remain model-specific and cannot be reused across architectures. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi-model reasoning and long-context generation. We propose Mixture-of-Translators(MoT), a cache translation framework that maps conte
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.