One of the more persistent frustrations with deploying large language models in production is that the world does not hold still. Data distributions shift, new domains emerge, and the model you fine-tuned last quarter may quietly degrade on inputs that would have seemed unremarkable a year ago. Retraining from scratch is expensive; standard fine-tuning risks catastrophic forgetting; and most parameter-efficient methods like LoRA, while computationally tractable, are static solutions applied once and left unchanged. This paper, SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptation, proposes a different framing: rather than asking how to fine-tune a model efficiently, ask how a model can learn to adapt itself, autonomously and continuously, by treating its own weight space as an environment to explore.
Key Contributions
The central claim of SOLAR is that LLM weight spaces contain navigable meta-knowledge, and that a reinforcement learning agent can learn to discover, validate, and accumulate weight-modification strategies without requiring labelled data or human-curated adaptation recipes at deployment time. The specific contributions are:
- A three-level RL training curriculum that progresses from single valid self-edits, to chained edit sequences, to fully open-ended hypothesis generation beyond human-crafted strategies.
- A hyper-convolutional decoder architecture for weight-space initialisation, conditioned on task embeddings from Sentence-BERT, that samples plausible LoRA parameter distributions rather than committing to a single starting point.
- A persistent, evolving knowledge base of validated weight-modification strategies that functions as an episodic memory buffer, explicitly balancing plasticity and stability.
- A meta-level weight regularisation scheme to mitigate catastrophic forgetting at the strategy-retention layer, not just the task layer.
The framing draws explicitly on cognitive science analogies: a student who adapts their study strategy for a new subject without forgetting how to study in general. This is more than rhetorical dressing; it motivates the architectural separation between rapid test-time adaptation and long-term strategy retention, which is the structural innovation the paper is built around.
Methodology
SOLAR operates on LoRA parameters rather than full model weights, keeping the search space to roughly 1% of total parameters. The weight-space exploration initialiser uses a convolution-based decoder conditioned on prompt embeddings to generate starting LoRA adapters. Parameter tokenisation preserves both layer-wise distributions and cross-layer correlations through normalisation, uniform slicing, and 2D positional embeddings encoding layer index and within-layer position. The decoder itself applies sequences of width, height, and layer-wise 2D convolutions, transforming prompt embeddings into LoRA-compatible adapter tensors.
The RL agent, built on Qwen2.5-0.5B-Instruct as the base model, then proposes hypotheses about how to modify these initialised weights. Hypothesis validation uses either held-out evaluation splits or, when those are unavailable, adversarially generated synthetic evaluation data produced by two LLM instances in a proposer-solver configuration. Validated strategies are added back to the knowledge base, expanding the action space for future iterations. This closed loop, propose, test, retain, is the mechanism by which SOLAR claims to accumulate meta-knowledge rather than merely adapting to individual tasks.
The seed knowledge base, curated by hand at the outset, covers a range of adaptation families: test-time training (TTT) variants, label-smoothing (LS) approaches, LoRA mixing strategies, and test-time scaling (TTS) ensemble methods. The RL training then learns which strategy family and configuration to select given a new task context. At the current published stage, only Level I (single self-edit selection) has been fully trained and evaluated.
Results
Experiments use Qwen2.5-0.5B-Instruct evaluated against four baselines: task-specific LoRA, Test-Time Learning (TTL), Decoupled and Orthogonal Merging (DOM), and DnD (a prompt-conditioned hyper-convolutional parameter generator). Benchmarks span common-sense reasoning (ARC-e, ARC-c, BoolQ, HellaSwag, PIQA) and out-of-domain tasks covering mathematics (GSM-MC, MATH-MC), logical reasoning (DivLogicEval), social reasoning (SocialIQA), and code (CodeMMLU).
SOLAR Level I achieves the following average accuracy improvements over the next-best baseline (DnD) across task groups:
- In-domain average: 59.5% vs DnD's 49.0%, a gain of 10.4 percentage points.
- Out-of-domain average: 34.1% vs DnD's 25.7%, a gain of 8.4 percentage points.
- HellaSwag specifically shows a 21.8 point improvement over DnD, the largest single-benchmark gain.
- CodeMMLU shows a 21.5 point improvement over DnD, suggesting the approach generalises well to structured reasoning domains.
The strategy selection that emerges from Level I training is itself informative. TTT-family strategies dominated for ARC-e and PIQA; label-smoothing worked best for ARC-c and SocialIQA; LoRA mixing with task-specific blending suited BoolQ and the mathematical benchmarks; and ensemble-based TTS approaches were selected for HellaSwag, DivLogicEval, and CodeMMLU. The fact that different task types pull toward different adaptation families is consistent with the intuition motivating the whole framework.
Limitations and Implications
Several caveats deserve attention. The seed knowledge base is hand-curated, which means the system's initial action space reflects human assumptions about what constitutes a plausible weight-modification strategy. Level III, where the agent is meant to go beyond these human-crafted approaches entirely, has not yet been trained or evaluated. The paper is candid about this, but it means the most ambitious claim, that SOLAR can discover genuinely novel adaptation strategies, remains unvalidated.
Compute requirements are substantial. Level I training alone took approximately four days on two A6000 GPUs. Scaling to Level III, or to larger base models, is an open question. The current experiments also use a 0.5B parameter model, which limits the generalisability of the results to the frontier models where concept drift is arguably most consequential.
The adversarial synthetic evaluation data generation, used when no labelled evaluation split exists, introduces a potential circularity: the same model family generating hypotheses is also generating the evaluation signal. The paper cites prior work (SQLM, R-Zero) as precedent, but the quality and calibration of this synthetic signal under distribution shift is not analysed in depth.
The broader significance of this work sits at an interesting intersection. Parameter generation research (DnD, T2L, RPG, COND P-DIFF) has produced strong static adapters; RL-based continual learning (SEAL, Memento) has shown promise for low-cost adaptation; and self-referential systems that modify their own weights have a long theoretical history. SOLAR's contribution is to connect these threads into a single framework where the adaptation strategy itself is the learned object, not just the adapted parameters. If Levels II and III can be validated, the implications for deploying models in genuinely non-stationary environments would be significant. As the authors note, once web-scale pretraining data is exhausted, progress may increasingly depend on a model's ability to generate its own high-utility training signal. SOLAR is an early, incomplete, but technically serious attempt to operationalise that idea.
The full paper is available at arXiv:2605.20189.