Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models
Quick Answer
This paper shows that The Mixture-of-Translators (MoT) framework enables the translation of key-value caches across heterogeneous large language models (LLMs) like Qwen2.5, GPT-2, and OPT, achieving 51.0% average closed-set QA accuracy.
Quick Take
By employing multiple translator modules and a Context Correction Loss, MoT preserves downstream QA performance and retains 96.3% context quality in long-context generation, facilitating scalable memory reuse.
Key Points
- MoT translates KV caches between models, enhancing multi-model reasoning.
- Achieves 51.0% average closed-set QA accuracy with Qwen2.5-7B-scale translation.
- Introduces Context Correction Loss to align target trajectories effectively.
- Retains 96.3% context quality in long-context cache-augmented generation.
- Addresses cache translation failures through translator mixtures and corrections.
DeepSignal Analysis
What happened
The Mixture-of-Translators (MoT) framework has been introduced to translate key-value caches across different large language models (LLMs) such as Qwen2.5, GPT-2, and OPT. This approach aims to enhance the scalability of multi-model reasoning and long-context generation by enabling cache reuse.
Key evidence
- MoT achieves an average closed-set QA accuracy of 51.0% when translating caches among Qwen2.5, GPT-2, and OPT models.
- The framework employs multiple translator modules and a Context Correction Loss to maintain 96.3% context quality in long-context generation.
- MoT addresses two failure modes in cache translation: propagated translation shift and last-state shift, improving the accuracy of the translated caches.
Why it matters
The inability to reuse key-value caches across different LLMs has limited the efficiency of multi-model systems. MoT's approach to translating these caches could significantly reduce redundancy and improve performance in applications requiring long-context reasoning. This advancement may lead to more effective use of resources and better integration of various LLMs in practical scenarios.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Heterogeneous Large Language Model (LLM) systems increasingly rely on shared contexts, retrieved evidence, and multi-agent dialogue histories, yet their internal key-value (KV) caches remain model-specific and cannot be reused across architectures. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi-model reasoning and long-context generation. We propose Mixture-of-Translators(MoT), a cache translation framework that maps context KV caches from a source LLM into the cache space of a target LLM. Unlike prior approaches that depend on a single projection path or global shared latent space, MoT uses multiple translator modules to capture diverse source--target mappings. To further reduce residual translation error, we introduce a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory. We reveal two competing failure modes in cache translation: propagated translation shift from early injection and last-state shift from late injection. MoT addresses them through translator mixtures and target-side correction. Across homogeneous and heterogeneous translations among Qwen2.5, GPT-2, and OPT models, MoT preserves downstream QA performance, including Qwen2.5-7B-scale translation with 51.0% average closed-set QA accuracy and 0.43 average extractive QA F1. In practical case studies, MoT enables quality-preserving memory reuse for multi-agent reasoning and retains 96.3% of direct-context quality in long-context cache-augmented generation, demonstrating scalable KV cache reuse across heterogeneous LLMs.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.28979 [cs.CL] |
| (or arXiv:2607.28979v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28979 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jin-Woo Lee [view email]
[v1]
Fri, 31 Jul 2026 03:07:31 UTC (2,053 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.