How Dream-RSI lets an AI agent improve how it searches

AI systems like AlphaEvolve find better code by trial and error. A language model writes a candidate, a program scores it, and the loop repeats thousands of times. Deciding which attempt to build on next is usually a hand-written rule that never learns. Dream-RSI, a new paper from researchers at the University of Maryland, Google DeepMind and the University of Virginia, trains that rule by replaying old searches instead of paying for new ones. On GPU kernels it reached the same speed with up to 2.4 times fewer calls. We'll place it in the bigger picture first, walk through how it works, then look at what the results really show and what you can reuse in your own agent systems.

Dream-RSI improves the search strategy around a fixed coding agent, not the model. It tests new strategies on recorded search history, the same idea behind experience replay in reinforcement learning and backtesting in finance. This post explains the method, checks what the results really show, and picks out the parts you can use in your own agent loops.

Where Dream-RSI fits in the bigger picture

The full build is on GitHub, Dream-RSI Explorer. It runs the three-stage loop on the live Gemini API, using one of the paper's own benchmarks, packing 26 circles in a square.

You can improve an AI system that solves hard problems in three places.

• Improve the model: Train a bigger or better LLM. This is the most expensive layer, and it's what most headlines are about.
• Improve the answer: Keep the model fixed and wrap it in a search loop. The model proposes a solution, an evaluator scores it automatically, and the loop keeps building on the best ideas. FunSearch and AlphaEvolve from Google DeepMind work this way.
• Improve the search: Keep the model and the loop fixed. Improve the strategy that decides where to search next. This is the layer Dream-RSI works on.

The middle layer already has faster algorithms and new math results. The weak spot is how the search is guided. That is usually a hand-written rule that does not learn from past searches.

Dream-RSI makes that rule learn. The phrase recursive self-improvement often suggests a model rewriting its own weights. Here it means something narrower and more practical. The system rewrites the code that steers its own search, using its own search history.

With the same LLM, two search strategies can reach very different results at very different costs. Dream-RSI treats the search strategy as something worth training, just like the model.

Why the search strategy is hard to improve

A discovery agent is an LLM coding agent that writes a candidate, runs it, and gets a score. Then it tries again, building on earlier attempts.

The exploration policy decides which earlier attempt to build on, how many to try in parallel, and when to stop. Testing a new policy normally means running a full search with it. A bad policy wastes the whole budget, and there are many policies worth trying.

The three-stage loop

The discovery agent, the evaluator and the tools stay fixed the whole time. Only the exploration policy changes. Each round runs three stages.

• Explore for real: The current exploration policy guides the discovery agent. Every attempt becomes a node in a discovery tree.
• Build replay simulators: Finished trees go into a pool. Each one can now act as a replay simulator.
• Dream: A policy-development agent reads the replay results and rewrites the policy code. The highest-scoring version runs the next real round.

How dreaming works

Every attempt in a discovery tree keeps its code, diagnostics and score, linked to the attempt it grew from.

To test a new policy, let it walk a recorded tree. When it picks a node to expand, replay returns the children the agent actually produced. If nothing was recorded there, it gets nothing, so it can't invent results.

Each replay is scored on the best result found. It loses points for every node it opens, since each one stands for a real agent call. It gains a little for running attempts in parallel.

Where this idea comes from

Reinforcement learning has used this idea for years. Experience replay, used in DeepMind's Atari-playing DQN, stores past experience so the learner can reuse it without playing again. Recommender systems use offline evaluation to test a new ranking policy on logged clicks before any user sees it.

Think of dreaming as backtesting a trading strategy on past market data. It's cheap, and it carries the same two risks.

• Coverage: Replay can only show outcomes that were recorded. A policy that would explore a new direction gets no credit for it in replay.
• Fitting the past: A policy can learn the quirks of the recorded trees rather than a better way to search. Overfitting to history is the classic backtesting failure.

The online rounds are the check on both. Each real round tests the chosen policy for real and adds fresh history for the next replay.

Reading the results carefully

The paper uses Gemini-3.1-Pro and Gemini-3.7-Flash as the discovery agent. The clearest test is a solver for the Lasso regularisation path. Lasso is a linear regression method that pushes many weights to exactly zero. The solver is scored by average runtime on six held-out datasets, and lower is better.

This table supports two different comparisons, and they say different things.

• Dream-RSI against SimpleTES: The headline number, 317 agent calls against 51,200 with a faster solver. Most of that gap exists before any dreaming, since Recursive Fixed (Pro) already needs only 550 calls. SimpleTES also runs a different model.
• Dream-RSI against Recursive Fixed: The fair test of dreaming, with the same model and the same loop. With Pro, runtime drops from 3,587.1 ms to 2,931.0 ms using 317 calls instead of 550. With Flash, it drops from 2,516.7 ms to 2,350.6 ms using 1,879 calls instead of 3,200.

Flash, the smaller model, built faster solvers than Pro while making several times more calls. How a system spends its attempts can matter as much as which model it uses.

The other two domains show a similar pattern, with the clearest gains in cost.

• GPU kernels: On VGG16 and LayerNorm, GPU kernels reached the same speed with 2.43x and 1.79x fewer generations. On ConvDiv and ConvMax, they were 2.09x and 1.44x faster on the same budget.
• Maths problems: Mostly level with the best systems, and slightly ahead on the sum-difference problem. The win is budget. It used fewer than 1,000 generations, which the authors put at over 50x less than SimpleTES.

What the learned policy does, and a warning about hints

On the ConvDiv kernel task, the policy first cut its attempts from 110 to 50. When progress stalled, it raised the effort again, and scores improved. A fixed rule spends the same way whether the search is going well or stuck.

The authors also tried giving the agent past history as hints about which direction to search. Under the same budget, that consistently did worse than no hints. Strong hints narrowed the search and cut the variety of ideas.

Many agent setups feed lessons learned from past runs straight back into the prompt. For open-ended search, this result points the other way. Use history to evaluate the strategy, and keep it out of the agent's instructions.

Where this approach works, and where it doesn't

Every domain in the paper has an evaluator that returns a number, such as runtime, kernel speed or a maths score. Replay depends on those stored scores.

• Good fit: Code and algorithm optimisation, compiler and kernel tuning, and any search where a program can grade the result.
• Poor fit: Tasks graded by people, such as writing quality or product design. An LLM judge could stand in, but its scoring errors would be copied into every replay. The LLM judge lifecycle post covers how those errors creep in.
• Early rounds: The first rounds have little recorded history, so dreaming has less to learn from.

The name also sounds bigger than the method. The model, the evaluator and the tools stay fixed. What improves is the code that steers the search, inside limits a person set.

What you can reuse in your own agent systems

Any agent loop that retries, branches or decides when to stop has an exploration policy, even if it's a few if-statements. You can apply the core idea without a research setup.

• Log attempts as a tree: Record each attempt with its parent, its output and its score. A flat log loses which attempt grew from which, and replay needs that link.
• Replay before you spend: Before changing retry limits, branching or stop rules, run the new rules over logged trees. Compare the best score and the number of calls.
• Compare against your own fixed loop: Judge a change against the same model and loop with the old rules, the way the paper uses Recursive Fixed Exploration.
• Confirm with a live run: Replay can only confirm what was already tried, so a real run is still the final check.

The loop engineering post covers the retry and validation loop this sits inside. The Agentic AI and Multi-Agent Systems track in this app covers planning, tool use and reasoning patterns.

Sources

• Zheng et al., 2026: Dream-RSI: Recursive Self-Improvement through Evolving Worlds, arXiv:2609.14858. All Dream-RSI numbers in this post come from this paper.
• Romera-Paredes et al., 2023: Mathematical discoveries from program search with large language models, Nature. The FunSearch paper.
• Google DeepMind, 2025: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms.
• Mnih et al., 2015: Human-level control through deep reinforcement learning, Nature. The DQN paper that popularised experience replay.