ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding
Quick Answer
ReLoop-UME introduces a recurrent model for universal multimodal embedding, achieving 44.9x faster retrieval than UME-R1 and 1.5x faster than PLUME.
Quick Take
By utilizing Learnable Retrieval Registers, it enhances retrieval performance on benchmarks MMEB-V2 and MRMR, while maintaining a fixed token workspace.
Key Points
- ReLoop-UME improves retrieval efficiency while keeping token workspace fixed.
- The model executes early layers once and reuses retrieval blocks recurrently.
- Learnable Retrieval Registers accumulate evidence across loops for better performance.
- Achieves consistent improvement across different backbone architectures.
- Significantly faster than existing UME models, enhancing practical applications.
DeepSignal Analysis
What happened
ReLoop-UME is a new recurrent model designed for universal multimodal embedding, which significantly enhances retrieval speed and performance. It achieves a retrieval speed that is 44.9 times faster than UME-R1 and 1.5 times faster than PLUME. The model employs Learnable Retrieval Registers to improve retrieval performance on specific benchmarks.
Key evidence
- ReLoop-UME executes early layers once and reuses a parameter-shared retrieval-forming block, optimizing computation while maintaining a fixed token workspace.
- The model shows consistent improvements in retrieval performance on benchmarks MMEB-V2 and MRMR across different backbones.
- ReLoop-UME's design allows it to achieve a retrieval speed of 44.9x faster than UME-R1 and 1.5x faster than PLUME.
Why it matters
The advancements in ReLoop-UME could lead to more efficient processing of multimodal data, which is crucial in applications like computer vision and natural language processing. Faster retrieval times can enhance user experience and enable real-time applications, making this model relevant for industries relying on multimodal data integration.
What to watch
Paper Resources
Source Excerpt
Universal multimodal embedding (UME) maps heterogeneous multimodal inputs into a shared embedding space. Existing UME models either form embeddings through single forward encoding or add computation through explicit rationale tokens and latent autoregressive states. Although token expansion can improve complex matching, serial generation increases retrieval latency and makes the final embedding depend on generated intermediate states. This raises a different question: can useful computation be e
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.