In-Context Reinforcement Learning under Non-Stationarity: A Survey
Quick Answer
This paper shows that This survey explores non-stationary in-context reinforcement learning (ICRL), focusing on how fixed policy parameters can adapt to changing environments without test-time updates.
Quick Take
This survey explores non-stationary in-context reinforcement learning (ICRL), focusing on how fixed policy parameters can adapt to changing environments without test-time updates. It emphasizes the need for decision models to infer current rules and assess the relevance of accumulated context, addressing gaps in existing ICRL literature.
Key Points
- Non-stationary ICRL adapts to changing environments with fixed policy parameters.
- Accumulated context may become stale or misleading in dynamic settings.
- The survey categorizes literature based on changes, their unfolding, and observability.
- Key areas include meta-RL, decision sequence modeling, and retrieval-augmented RL.
- Existing ICRL surveys largely overlook the non-stationary aspect.
Paper Resources
📖 Reader Mode
~2 min readAbstract:The development of decision-pretrained transformers, algorithm distillation, long-context meta-RL, and retrieval-augmented agents has renewed interest in in-context reinforcement learning (ICRL): the ability of a pretrained or fine-tuned decision model to infer latent task rules and improve future behavior from interaction context, without test-time parameter updates. This line of work asks when trial-and-error evidence, rewards, transitions, demonstrations, feedback, or retrieved experience can make learning-like computation happen inside the context window. However, existing surveys of ICRL mainly organize the field around pretraining objectives, architectures, context formats, evaluation protocols, and theoretical mechanisms, while the non-stationary setting remains comparatively underexamined. In changing environments, accumulated context is not merely more evidence about a fixed task: the reward specification, transition kernel, observation channel, action interface, constraint model, or demonstration and memory distribution can fall out of alignment with the current regime. Previously useful context can therefore become stale, misleading, or useful again when an old regime returns. We survey non-stationary ICRL as the problem of adapting through context while deployed policy parameters remain fixed: the policy must infer both the current decision rule and which parts of its accumulated evidence still support that rule. We define non-stationary ICRL, relate it to meta-RL, decision sequence modeling, retrieval-augmented RL, value- and model-aware ICRL, and reward-feedback agents, and organize the literature along three questions: what changes, how the change unfolds, and how observable the change is to the agent.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.11906 [cs.AI] |
| (or arXiv:2607.11906v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.11906 arXiv-issued DOI via DataCite |
Submission history
From: Ziluo Ding [view email]
[v1]
Wed, 1 Jul 2026 12:55:17 UTC (672 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Emerging Paradigm of Geospatial Foundation Models: From Pre-Training to Agentic Reasoning
The paper introduces Geospatial Foundation Models (GeoFMs), AI/ML models pre-trained on extensive geospatial datasets, enabling domain experts to fine-tune them for specific tasks. This paradigm shift democratizes access to advanced AI/ML while ensuring security, and proposes a framework for cost-effective adaptation strategies. The vision of Agentic Geospatial Reasoning is also presented, where Large Language Models orchestrate GeoFMs to automate complex analytical workflows.