Continual Learning Exam: Do AI Agents Actually Learn From Experience?

Most agent benchmarks ask whether a model can finish one task. The Continual Learning Exam (CLE) asks something harder: does an agent get better at the next task because of what it did on the last one? The paper was posted on October 24, 2026, trended on Hugging Face with 27 upvotes, and ships with a public repository and project site. Its headline number is sobering: the best system, Claude Code running Claude Opus 5, completes 47.67% of levels even when it gets one round of feedback and a revision.

This post explains how the exam is built, what the results say, where the caveats are, and how to try the public subset.

TL;DR: the questions people will ask

How the exam works

Each of the 120 tasks is a ladder of five levels. The agent must pass level one to unlock level two, and so on. Later levels deliberately reuse things from earlier ones, so an agent that carries forward the right information should do better than one that starts fresh each time.

According to the paper's abstract and the repository README, the benchmark includes:

• 120 expert-written tasks across 89 professional roles and 12 occupational domains, built by more than 50 domain experts
• 23,981 real-world input files across 77 file extensions
• 19 stateful mock applications exposing 1,024 APIs, so actions in one level change the state seen in the next
• 16,125 human-authored rubric criteria, scored by a multi-turn evaluation agent
• Five kinds of transferable knowledge linking levels: tools and skills, data dependency, strategy, user context, and domain knowledge

Quality control used an automated review agent followed by two randomly assigned human reviewers per task. The authors report 96.86% agreement between their evaluator and adjudicated human labels at the rubric-item level. That figure is the paper's own claim and we have not reproduced it.

A level counts as passed only when every rubric criterion is met, per the README, which makes the scoring strict.

Two settings: one shot vs. one revision

The paper compares two conditions. In K=1, the agent submits once per level. In K=2, it receives diagnostic feedback after a failure and gets one revision. The repository calls these Trial 1 and Trial 2, and also offers a "Direct" mode where all five levels are attempted before grading, versus the "CLE" mode where each level is graded before the agent advances.

This design isolates two things that are often blurred: raw capability, and the ability to use feedback. It also tests whether an agent's behavior, not just its answer, improves over a sequence.

The results

All figures below are from the paper as summarized on its Hugging Face page, and come from the authors' own evaluation.

Two patterns stand out. First, the ladder gets steeper: even with feedback, the median agent passes only about one in twelve final-level attempts. Second, the paper says Hermes-Agent beat OpenClaw on all six shared backbone models, which suggests the harness around a model matters as much as the model. If you use either harness, our Hermes-Agent release notes show where that project is heading.

The leaderboard in the repository covers 17 configurations across 11 model variants and four harnesses: Claude Code, Codex, Hermes-Agent and OpenClaw. Note that the Claude and GPT model names above are as given in the paper; we have not independently checked them against vendor documentation.

Why this matters for people building agents

The common pitch for agents is that they get better with use. Memory files, skills, and persistent state all make that promise. CLE puts a number on how far that promise is from reality on professional work: under half the levels completed by the best configuration.

That connects to a debate we covered in Jack Clark's recursive self-improvement forecast. Benchmarks like CLE are one way to measure the early, in-context version of that idea. Note the distinction: CLE does not test agents rewriting their own weights. It tests whether an agent reuses tools, data, strategy and context it has already seen, which is the practical question for anyone running a long-lived assistant, as in our guide to always-on persistent agents.

It also reinforces a finding from other harness work. In Google's Harness2 recursive agent harnessing project, the scaffolding around a model drove results as much as the weights. CLE's harness comparison points the same way.

What the five kinds of transferable knowledge mean in practice

The paper names five channels through which a level can depend on an earlier one. Thinking about them separately is useful even if you never run the benchmark, because each maps to a design choice in a real agent.

Tools and skills. An agent that worked out how to call one of the mock applications' APIs in level two should not rediscover it in level four. In production this is the case for saved skills and tool notes. A harness that persists them has an edge over one that starts every session cold.

Data dependency. Later levels reuse files and outputs created earlier. This tests whether an agent knows where its own earlier work lives and whether it trusts it. Agents that overwrite or ignore prior artifacts lose the ladder quickly.

Strategy. Some tasks reward a plan discovered earlier, such as the order in which to process inputs. This is the closest thing to "learning how to do the job" rather than "remembering a fact."

User context. Preferences and constraints stated in early levels carry forward. Anyone who has watched an assistant forget a formatting rule halfway through a project will recognize the failure.

Domain knowledge. Facts about the professional field, learned from the inputs, that apply later.

The reason this decomposition matters is diagnostic. A single aggregate score of 47.67% does not tell you why an agent failed a level. If the failures cluster on data dependency, the fix is better state handling in the harness. If they cluster on strategy, the problem is closer to the model. The authors' rubric structure, with thousands of criteria, is designed to support that kind of breakdown, though we have not seen a per-channel table in the material we read.

Reading the feedback result carefully

A median gain of 5.33 points from feedback and one revision can be read two ways. Optimistically, it shows agents can use a diagnostic message to fix mistakes. Skeptically, a revision is simply a second attempt, and any stochastic system improves when given two tries. The paper's framing, per its abstract, is that feedback and revision raise completion for most configurations but the best system still falls short, which fits both readings.

The Sonnet result, from 27.00% to 40.17%, is the large outlier. A 13-point jump suggests that for some model and harness pairings the first attempt fails for fixable reasons, such as misread instructions, rather than lack of ability. That is encouraging for teams that can put a verifier or reviewer in the loop, and it echoes the case for review steps in agent workflows generally.

For practitioners, the takeaway is concrete: if your agent runs long, multi-step professional work, budget for a feedback pass. Measure it against a plain retry so you know whether the feedback content is doing the work.

Caveats before you cite the numbers

• Self-reported. The results come from the authors' own runs and evaluator, with no independent replication yet.
• Judge dependence. Scoring relies on an evaluation agent. The README says the judge is a model such as DeepSeek V4 Flash via OpenRouter or GPT-6.1-Sol via Codex, so results could shift with the judge.
• Mostly closed data. Only CLE Lite is public. The full 120-task benchmark is described in the paper, but the README does not say it is downloadable, and rubrics need approval.
• Provenance. The project site credits a "CLE Team"; the paper's appendix lists university authors, led by corresponding author Jun Wang of UCL.
• No license shown. The repository page did not display a license, and third-party input files keep their original terms per the package notices.
• Small community so far. At the time we looked, the repository had 5 stars and no forks.

How to try CLE Lite

According to the README:

1. Use Linux x86-64 with Docker Compose, Python 3.10+, uv and Node.js 22+. Budget at least 30 GB of disk for a single task.
2. Get the public task cards and input workspaces for the 20 CLE Lite tasks from Hugging Face. Request approval for the rubrics and full task packages.
3. Set up a judge model and API key, as described in docs/evaluation.md in the repository.
4. Run a single task with the run command, or an ordered sequence with stream --setting cle.
5. Record the pinned runtime and task-data revisions when you report results, as the guide asks.

If you only want the idea, you can copy the structure into your own evaluations: chain tasks so later ones depend on files and state from earlier ones, and compare a one-shot run with a feedback-and-revise run.

What to watch

• Independent reproductions and the full leaderboard page
• Whether labs report CLE scores in model launch notes, as with the new Claude generation covered in our Claude Haiku 5.5 launch post
• Release of the full benchmark and rubrics
• Training experiments: the authors suggest the dataset could be used to post-train models for continual learning

Related reading

• Jack Clark's RSI 2027 forecast and the taste problem
• Hermes-Agent v0.22.0 release
• Building an always-on persistent AI agent
• Google Harness2: recursive agent harnessing
• Claude Haiku 5.5 launch, pricing and benchmarks
• AI assistance and persistence: Berkeley and CMU study
• Primary sources: the CLE paper on arXiv and the GitHub repository

Figures are as reported in the paper and repository on October 27, 2026 and may change as the leaderboard updates.