Learning from experience in unfamiliar environments

Learn2Play Bench

How effectively can LLM agents learn from experience?

20 game templates with hidden rules to discover. Different seeds generate different game instances. Agents learn each game’s hidden rules by playing repeatedly and use what they learn to improve in later episodes, without updating their model weights.

Research snapshot · October 2026 · Game portal requires sign-in

Game examples

See how each game is played.

Choose a game and scroll through its opening situation, sample actions, and feedback. These are recorded demonstrations—not a live game or a recommended strategy.

Gameplay preview

How to play

Loading recorded examples…

Play in the game portal
Gameplay transcriptScroll to read actions and feedback
Loading…

Evaluation protocol

How we evaluate agents

For each game template, we use one selected rule seed and run 3 independent trials. The leaderboard does not average over all possible seeds.

We report 5 episodes for normal games and 10 for challenge games. Scores are normalized using each game's reference ceiling, then averaged with weight 3 for challenge games and weight 1 for normal games.

Max

Average peak score across trials.

Mean

Average score across episodes.

LG Learning Gain

Improvement from early to late episodes.

LS Learning Slope

Fitted rate of score change across episodes.

Results

Leaderboard

Compare backbone models with human references, self-evolving methods, and agent harnesses. Default ranking uses Max.

Main evaluation leaderboard
RankModel / Human referenceMax ↑Mean ↑LG ↑LS ↑

Human references: Top-1, Top-3, and Top-5 select the highest-scoring eligible players separately for each game. Human Mean includes all eligible participants. These references cover the same 20 games, scoring windows, normalization, and challenge weights; Top-k describes selected performance, not an average person. Humans played without AI tools or code.

Source: manuscript backbone/method, human, and harness comparison tables · Updated October 8, 2026. Sorted highest first; equal displayed values share a rank. Highlighting marks the best value in each column. These are aggregate point estimates, not significance rankings. This leaderboard is a curated snapshot, not automatically updated.

Opus 5.5 evaluation details

OpenCode + Claude Opus 5.5 completed three trials on each of the 20 selected game instances. Of the 60 trials, 50 completed in their original sessions and seven resumed after API interruptions. Three incomplete trials were replaced with independent runs that added a generic reminder to finish each episode before advancing. No earlier trial results or game-specific hints were supplied to replacements. All retained terminal scores were verified. The same normalization, game weights, and scoring windows apply.

GPT-6 Astra evaluation details

GPT-6 Astra ranks second among the eleven OpenCode backbones on all four metrics: Max 76.25, Mean 56.07, LG +23.79, and LS +5.88. Twenty fixed game–seed instances each have three isolated trials of ten episodes: 600 verified terminal replays and 375 episodes in the manuscript’s scoring window. OpenCode uses the native Responses adapter with medium reasoning effort. These descriptive comparisons do not establish statistical significance.

Forty retained trials needed no continuation, eleven resumed their original sessions, and six needed only missing quit markers with unchanged game decisions and verified scores. Three protocol-invalid or interrupted trials were independently replaced; selection preceded replacement scores, and replacements received no prior trial experience. Original attempts remain archived. Seven retained trials precommitted multiple episodes before receiving feedback, reducing opportunities for sequential adaptation. All retained histories and terminal scores were independently verified. This cohort is separate from Codex; no matched Astra harness comparison is claimed.

Learning through repeated interaction

How does performance change with experience?

A peak score does not show the path to that score. These curves track normalized performance over the evaluation window across all 20 games.

Learning curves for eleven OpenCode backbones, including GPT-6 Astra and Claude Opus 5.5, showing normalized score against progress through the evaluation window.
The horizontal axis shows progress from the first to the last evaluated episode, with linear interpolation to align curves. Lines show aggregate mean scores, not uncertainty intervals.

Efficiency

Performance and inference cost

Compare Max scores with mean cost per completed episode. Astra uses a confirmed bill; other configurations use standard-price estimates.

Open chart ↗
OpenCodeMethodsClaude CodeCodexFrontier · standard-price estimates31 configurations
31 agent configurations by normalized Max score and US dollars per episode. Astra uses billed cost; other points use standard-price estimates. Higher and farther left is better. Claude Code with Opus 5 has the highest Max at 81.8%.

Hover, tap, or focus a point to see its full name and reported values.

GPT-6 Astra: $150 total billed · $0.25 per episode. Bill confirmed by the experiment owner; total divided by 600 completed episodes.

Max is normalized to each game’s reference ceiling. The dashed frontier compares standard-price estimates and excludes Astra’s billed point because the cost bases differ. Astra’s total bill is confirmed by the experiment owner; individual charges for retries and preflight are not separately itemized here. Other costs use recorded token usage and the August 30, 2026 pricing snapshot, except Opus 5.5. Opus 5.5 uses Anthropic’s standard prices accessed October 7, 2026. Costs average all collected episodes; performance uses the scoring windows above. Only Astra uses a billed amount; the other estimates are not billing records or current price quotes.

Benchmark design

Key ideas behind Learn2Play Bench

Strong task performance can reflect prior knowledge. Novel game rules let us examine how agents acquire and reuse knowledge through interaction.

01

Discover, don't look up.

Novel, sometimes counterintuitive rules create unfamiliar situations. Agents see actions and feedback—not game source code or solutions.

02

Do agents improve with practice?

Agents play each game repeatedly and can use what they learned in earlier attempts. We measure whether their scores improve.

03

Can agents use what they learn in new situations?

In some games, the situation changes between attempts, but the rules stay the same. We test whether agents can still use what they learned.