IBM Research says it can lift an AI agent's score on a hard customer-service benchmark without touching the model's weights: let the agent attempt training tasks, have an LLM distill the successes and failures into short rules, and paste those rules into the system prompt. The approach is called ALTK-Evolve, and the write-up, ALTK-Evolve: how learned memory tops the tau2-bench leaderboard without a frontier model, reports a Pass^4 jump from 57.5 percent to 67.5 percent for GPT-4.1 on retail tasks. This post explains how it works, what the numbers do and do not show, and what builders can take from it.
All figures are IBM's own, and leaderboard entries were still pending review when the post was published.
TL;DR: ALTK-Evolve in one table
| Question | Answer |
|---|---|
| What is it? | Guidelines learned from an agent's own past attempts, appended to the system prompt |
| Does it change weights? | No; no fine-tuning and no human annotation |
| Benchmark | tau2-bench retail and airline customer-service domains, stock ReAct scaffold, 4 trials per task |
| Best retail result (GPT-4.1) | Pass^4 57.5% to 67.5%; mean accuracy 78.1% to 82.5% |
| Airline (GPT-4.1) | Pass^4 35% to 45% (small high-support set); accuracy 57.5% to 65% (full set) |
| Airline (GPT-5.5) | Pass^4 55% to 70% on test tasks |
| Cost | About 65% more per retail conversation with the full set |
| Status | Leaderboard PRs #542 (retail) and #551 (airline) pending |
How does ALTK-Evolve work?
The loop has three steps. First, the agent attempts a set of training tasks in tau2-bench, which simulates customer-service conversations where the agent must follow policy and call tools correctly. Second, an LLM analyzes those attempts and writes guidelines from them, for example how to authenticate a customer or when to confirm a transaction before acting. Third, the guidelines are appended to the system prompt and applied unchanged to held-out test tasks.
Each guideline carries a support count, the number of distinct training tasks it came from. That gives a natural dial: use only high-support guidelines for a short, conservative prompt, or use the full set for broader coverage. IBM's results show the best size differs by domain.
The idea belongs to a growing family of agent-memory work that avoids retraining. If you want the surrounding landscape, see our coverage of Hindsight agent memory, Cognition's Devin dreaming memory, and a provenance benchmark for agent memory. What distinguishes ALTK-Evolve is the evaluation: held-out tasks, a stock scaffold, and a public leaderboard rather than a self-graded test.
Agent beside a notebook with a bookmark, representing ALTK-Evolve learned guidelines for an AI agent
What are the results?
On held-out test tasks with GPT-4.1:
- Retail: Pass^4 rose from 57.5% to 67.5% with the full guideline set, and mean accuracy from 78.1% to 82.5%.
- Airline: Pass^4 rose from 35% to 45% with the small high-support set, and accuracy from 57.5% to 65% with the full set.
With GPT-5.5 on airline, IBM reports Pass^4 rising from 55% to 70%. Here GPT-5.5 served as both agent and guideline extractor; in the main runs GPT-4.1 was the agent and extractor while GPT-5.5 played the simulated user.
Pass^4 is a strict metric: it counts a task as solved only if all four independent trials succeed, so it rewards consistency, not luck. That is the right measure for customer service, where one bad conversation in four is a failure. For how this benchmark family is being used by new models, see Celeris-1 Magnus on tau3-bench.
The leaderboard claim
On the full base split, IBM reports 67.5% Pass^4 in retail, which it says is #1 on the tau2-bench snapshot dated October 12, 2026. In airline it reports 82.0% with GPT-5.5 (#1) and 44% with GPT-4.1 (#8). Two details matter. The submissions were pending as pull requests (#542 and #551) to the tau2-bench repository, so the official board may not yet show them. And the test-split numbers quoted earlier differ from full-split numbers, so compare like with like.
The headline "without a frontier model" refers to the retail result using GPT-4.1, an older and cheaper model than the newest flagships. That is the interesting part for practitioners: learned guidelines may close part of the gap between a mid-tier model and a stronger one on a narrow, repetitive task.
What does it cost?
Nothing is free. With the full set, retail cost per conversation rose about 65 percent with GPT-4.1, mostly because the prompt got larger. Conversations were slightly shorter, not longer, so the extra spend is input tokens rather than extra turns. IBM says caching would offset input costs in production, since the guidelines are a fixed prefix, but it did not measure that saving. If you adopt the idea, budget for it and measure.
Why learned guidelines instead of fine-tuning?
There are three common ways to make an agent better at a narrow job. You can fine-tune the model, which changes weights, needs data pipelines and compute, and ties you to one checkpoint. You can hand-write a long system prompt, which is cheap but depends on someone knowing what the failure modes are. Or you can let the agent's own history write the prompt, which is the ALTK-Evolve bet.
The appeal is operational. Guidelines are plain text, so they can be read, edited, audited and removed. If a rule is wrong, a person can delete it, which is far harder with a weight update. They are portable across models in principle, since the same text can be given to a different agent, and IBM's use of GPT-4.1 for the main runs and GPT-5.5 for a second experiment suggests the idea is not tied to one model. And the support count gives a built-in confidence signal: a rule that came from many distinct tasks deserves more trust than one that came from a single lucky or unlucky episode.
The trade-off is that prompt-based learning does not make the model more capable, it makes the model better informed about a specific environment. Rules about how a particular retailer authenticates customers will not help on a different company's policy. That is why the benchmark choice matters: tau2-bench environments are fixed and policy-heavy, which is exactly where accumulated rules pay off.
What people are likely to ask
Is this the same as agent memory? It is a form of it, but narrower. Many memory systems store facts or episodes to retrieve later. ALTK-Evolve compresses experience into a small, always-present set of behavioral rules.
Could the rules leak the test set? IBM evaluates on held-out tasks that the guideline extraction never saw, which is the standard guard. The small size of those test sets is the weaker point.
Does a bigger rule set always help? No. IBM reports that the best size varies, and a short high-support set won the airline Pass^4 comparison for GPT-4.1 while the full set won accuracy. More rules means more tokens and more chances for conflicting instructions.
Why Pass^4 and not plain accuracy? Because customer-facing agents are judged on reliability. A system that solves a task three times out of four will still frustrate one customer in four, and Pass^4 punishes exactly that inconsistency, which makes IBM's retail jump from 57.5% to 67.5% a more meaningful signal than a similar change in average accuracy.
Should I ship this in production? Treat it as a method to evaluate, not a drop-in product. Reproduce the held-out gains on your own tasks, track cost per conversation, and keep a human review step for the rules themselves, since a rule extracted from a failure can encode the wrong lesson.
Caveats worth taking seriously
- Small test sets. Forty retail and 20 airline held-out tasks means a few-point difference could be noise.
- Two domains. Retail and airline customer service with a strict binary reward do not stand in for every agent workload.
- Tuning needed. The best number of guidelines depends on the domain.
- Leaderboard status. Entries were pending review.
- Prompt bloat. Longer prompts raise cost and could crowd out other context in a real deployment.
How to try something similar
IBM released the ALTK-Evolve library and links its prior posts on the approach, though the article gives no step-by-step setup. A practical recipe in the same spirit:
- Run your agent on a training set of real or simulated tasks and log outcomes.
- Ask a strong model to extract short, imperative rules from failures and successes, and record how many distinct tasks support each rule.
- Test a small high-support set against the full set on held-out tasks, and track accuracy and cost.
- Re-run the evaluation with multiple trials, as IBM did with four, to measure consistency.
If you build with skills and reusable instructions, the pattern will feel familiar; our guide to agent skills explains the packaging side, and IBM's own open models are covered in the Granite 4.2 reasoning release. For a comparison of self-reported agent-memory claims against independent evaluation, see Garry Tan's GBrain evals.
Numbers are from IBM Research's October 2026 post; leaderboard positions may change once the pull requests are merged.
Related reading
- Hindsight agent memory
- Devin dreaming memory
- Agent memory provenance benchmark
- Celeris-1 Magnus and tau3-bench
- IBM Granite 4.2
- What are agent skills?
- GBrain evals and self-graded memory benchmarks
Official sources: IBM Research post on Hugging Face, ALTK-Evolve on GitHub and tau2-bench.