Discover, don't look up.
Novel, sometimes counterintuitive rules create unfamiliar situations. Agents see actions and feedback—not game source code or solutions.
Learning from experience in unfamiliar environments
How effectively can LLM agents learn from experience?
20 game templates with hidden rules to discover. Different seeds generate different game instances. Agents learn each game’s hidden rules by playing repeatedly and use what they learn to improve in later episodes, without updating their model weights.
Research snapshot · October 2026 · Game portal requires sign-in
Game examples
Choose a game and scroll through its opening situation, sample actions, and feedback. These are recorded demonstrations—not a live game or a recommended strategy.
Evaluation protocol
For each game template, we use one selected rule seed and run 3 independent trials. The leaderboard does not average over all possible seeds.
We report 5 episodes for normal games and 10 for challenge games. Scores are normalized using each game's reference ceiling, then averaged with weight 3 for challenge games and weight 1 for normal games.
Average peak score across trials.
Average score across episodes.
Improvement from early to late episodes.
Fitted rate of score change across episodes.
Results
Compare backbone models with human references, self-evolving methods, and agent harnesses. Default ranking uses Max.
| Rank | Model / Human reference | Max ↑ | Mean ↑ | LG ↑ | LS ↑ |
|---|
Human references: Top-1, Top-3, and Top-5 select the highest-scoring eligible players separately for each game. Human Mean includes all eligible participants. These references cover the same 20 games, scoring windows, normalization, and challenge weights; Top-k describes selected performance, not an average person. Humans played without AI tools or code.
Source: manuscript backbone/method, human, and harness comparison tables · Updated October 8, 2026. Sorted highest first; equal displayed values share a rank. Highlighting marks the best value in each column. These are aggregate point estimates, not significance rankings. This leaderboard is a curated snapshot, not automatically updated.
OpenCode + Claude Opus 5.5 completed three trials on each of the 20 selected game instances. Of the 60 trials, 50 completed in their original sessions and seven resumed after API interruptions. Three incomplete trials were replaced with independent runs that added a generic reminder to finish each episode before advancing. No earlier trial results or game-specific hints were supplied to replacements. All retained terminal scores were verified. The same normalization, game weights, and scoring windows apply.
GPT-6 Astra ranks second among the eleven OpenCode backbones on all four metrics: Max 76.25, Mean 56.07, LG +23.79, and LS +5.88. Twenty fixed game–seed instances each have three isolated trials of ten episodes: 600 verified terminal replays and 375 episodes in the manuscript’s scoring window. OpenCode uses the native Responses adapter with medium reasoning effort. These descriptive comparisons do not establish statistical significance.
Forty retained trials needed no continuation, eleven resumed their original sessions, and six needed only missing quit markers with unchanged game decisions and verified scores. Three protocol-invalid or interrupted trials were independently replaced; selection preceded replacement scores, and replacements received no prior trial experience. Original attempts remain archived. Seven retained trials precommitted multiple episodes before receiving feedback, reducing opportunities for sequential adaptation. All retained histories and terminal scores were independently verified. This cohort is separate from Codex; no matched Astra harness comparison is claimed.
Learning through repeated interaction
A peak score does not show the path to that score. These curves track normalized performance over the evaluation window across all 20 games.
Human Top-k curves use the manuscript’s selected players across all 20 games. Unobserved positions carry that player’s prior best score forward for plotting; reported table metrics are unchanged.
Efficiency
Compare Max scores with mean cost per completed episode. Astra uses a confirmed bill; other configurations use standard-price estimates.
Hover, tap, or focus a point to see its full name and reported values.
GPT-6 Astra: $150 total billed · $0.25 per episode. Bill confirmed by the experiment owner; total divided by 600 completed episodes.
Benchmark design
Strong task performance can reflect prior knowledge. Novel game rules let us examine how agents acquire and reuse knowledge through interaction.
Novel, sometimes counterintuitive rules create unfamiliar situations. Agents see actions and feedback—not game source code or solutions.
Agents play each game repeatedly and can use what they learned in earlier attempts. We measure whether their scores improve.
In some games, the situation changes between attempts, but the rules stay the same. We test whether agents can still use what they learned.