Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learning
Quick Answer
This paper shows that The Conditional Retrieval Alignment (CoRA) framework enables effective on-device in-context learning by using a gradient-free method for task-conditioned retrieval, demonstrated across ten textual datasets and four multimodal benchmarks with models like Llama-3.2-1B and MobileLLM-Pro.
Quick Take
CoRA achieves optimal low-rank compression without requiring retriever fine-tuning or backpropagation, making it suitable for resource-constrained environments such as Raspberry Pi 5 deployments.
Key Points
- CoRA converts a frozen encoder into a task-conditioned retriever using paired inputs and outputs.
- It employs closed-form ridge regression for aligning candidate input representations.
- The framework supports multimodal exemplar retrieval by integrating visual representations.
- Experiments show CoRA's effectiveness across various datasets without retriever fine-tuning.
- Deployment on Raspberry Pi 5 demonstrates its efficiency in resource-constrained settings.
DeepSignal Analysis
What happened
The Conditional Retrieval Alignment (CoRA) framework has been introduced for on-device in-context learning, utilizing a gradient-free method for task-conditioned retrieval. It has been tested on ten textual datasets and four multimodal benchmarks, showing effectiveness with models like Llama-3.2-1B and MobileLLM-Pro.
Key evidence
- CoRA transforms a frozen encoder into a task-conditioned retriever using paired candidate inputs and outputs, enabling efficient retrieval without fine-tuning.
- The framework achieves optimal low-rank compression through closed-form ridge regression and avoids the need for backpropagation or target-model calls.
- Experiments demonstrate CoRA's performance across various datasets and its suitability for resource-constrained environments, such as deployments on Raspberry Pi 5.
Why it matters
CoRA's gradient-free approach to task-conditioned retrieval is significant for on-device learning, particularly in environments with limited computational resources. By eliminating the need for retriever fine-tuning and backpropagation, it simplifies deployment and enhances accessibility for developers working with edge devices. This could lead to broader adoption of advanced machine learning techniques in practical applications.
Paper Resources
📖 Reader Mode
~2 min readAbstract:On-device in-context learning (ICL) relies on pre-inference retrieval to select demonstrations for useful context before downstream model inference. This retrieval must exploit task-specific information while operating over local memories under limited computation, memory, and data-exposure budgets. We propose Conditional Retrieval Alignment (CoRA), a gradient-free framework that converts a frozen encoder into a task-conditioned retriever using paired candidate inputs and outputs. CoRA selects complementary encoder layers, constructs an output-derived conditioning space from candidate memory, and aligns candidate input representations to this space through closed-form ridge regression. Low-rank factorization then produces a compact retrieval basis where candidate outputs are used only during offline index construction, whereas query-time retrieval requires only the query input and precomputed index. We show that CoRA's rank-constrained basis is the optimal low-rank compression of the output-conditioned fitted representation, and derive an exact two-pass streaming construction that avoids materializing the full fitted matrix. We further extend the framework to multimodal exemplar retrieval by incorporating visual representations into the conditioning and retrieval spaces. Experiments across ten textual datasets and four multimodal benchmarks with Llama-3.2-1B, MobileLLM-Pro, OpenFlamingo-3B, and Qwen3.5-2B, as well as end-to-end Raspberry Pi~5 deployment demonstrate that CoRA supports effective task-conditioned retrieval without retriever fine-tuning, backpropagation, or target-model calls.
| Comments: | Under review |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.27766 [cs.CL] |
| (or arXiv:2607.27766v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.27766 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xinyu Luo [view email]
[v1]
Thu, 30 Jul 2026 07:03:18 UTC (535 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.