Ai2 StateBench: Frontier Models Fall to 13% on Hard State Tracking, Hybrid Models Do Not

Ai2 has released StateBench, a benchmark that asks a simple question: when a long document keeps changing the facts about things in it, can a language model still say what is true at the end? According to Ai2's Hugging Face post, published November 3, 2026, four frontier models do well on easy versions and fall to 13% accuracy or lower on the hardest ones as the input grows. Small recurrent and hybrid models trained directly on the task kept going far longer.

The results come from Ai2's own tech report, dataset and code, which the post links. They are a lab's measurements on synthetic tasks, not an independent audit. Still, StateBench is released for others to use, which makes it a useful tool for anyone comparing architectures.

TL;DR: StateBench in one table

The four levels, explained as variable assignments

Ai2 frames the difficulty levels in terms any programmer knows. All four use the same number of variables and distinct values, with assignments at the same points. Only the kind of assignment changes.

1. R-trivial: immutable assignment. A value is set once, like let x = 4;. The model only has to recover the first value. Everyday analogue: a birth date.
2. Aperiodic: reassignment by value. x = 4 then later x = 2. The model must find the latest value and can ignore earlier ones. Analogue: a thermostat setting.
3. Soluble: self-referential updates. x = x + 5. Each update builds on the current value, so changes must be carried forward. Analogue: a game score.
4. General finite: assignment by reference. x = y or swapping two variables. This creates dependencies across updates. Analogue: a magician swapping cups, which is the S5 problem of tracking five objects through swaps.

The levels are grounded in Krohn-Rhodes theory from algebra, which breaks state-tracking problems into primitives. That is why Ai2 argues the ladder is not arbitrary: each level needs a different set of building blocks.

What the frontier models did

Ai2 asked Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna to read each sequence of assignments and output the final value of every variable. All four did well on R-trivial and aperiodic problems. On soluble and general finite problems, accuracy "fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems."

Ai2 offers one likely factor, which it calls sensitivity. At the harder levels, more steps can change the final answer, so missing one update is more likely to ruin the result. The post does not give per-model numbers in the text, so we cannot rank the four. See our coverage of Claude Opus 5.5 and Gemini 3.8 Flash for how these models are positioned.

Open pretrained models

Ai2 also tested open models with documented training: two 7B models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), plus smaller 275M and 2.7B models. All saw the same pretraining data, and none had seen contexts past 64,000 tokens.

Almost all did well on R-trivial, lost accuracy on aperiodic as inputs grew, and sat at chance on general finite. Format mattered a great deal. The 275M hybrid and Gated DeltaNet models were near the ceiling on RULER-style R-trivial problems but near chance on the same problems written as spreadsheet edits. That gap is one reason StateBench tests the same problem in several formats. If you rely on a long-context score from a single format, this is a warning.

Training directly on the task

Is the weakness in the architecture, or in what pretraining teaches? To separate the two, Ai2 trained six 275M models from scratch on StateBench: two transformers, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. They trained on sequences of up to 256 assignments and tested up to 64,000.

Results, as Ai2 describes them:

• Gated DeltaNet and hybrids generalized almost perfectly on R-trivial, aperiodic and soluble problems out to tens of thousands of assignments.
• Transformers learned the easier levels, degraded on longer sequences, and never learned general finite problems.
• On the hardest level, the Gated DeltaNet variant with a more flexible memory update stayed near perfect to roughly 10 times its longest training sequence.

Ai2's reading: for Gated DeltaNet and hybrids, the pretrained models' trouble points to training rather than architecture, since direct training unlocked more. For transformers, even direct training did not reach the hardest level.

Why developers should care

State tracking sounds abstract but it shows up in daily work:

• Reading code. Variables are assigned and reassigned, and a coding agent must know the current value after many edits.
• Agent logs and traces. Judging what an agent did requires tracking what changed. Compare Sarvam's Prism results on judging agent runs.
• Long documents and chats. Contracts, tickets and stories alter facts as they go.
• Architecture choices. As hybrid and recurrent models spread, you need tests that show which kinds of tracking they handle. Ai2 also stresses the simpler cases, which may be more common.

What people are asking

Is a transformer simply worse at this? Ai2 is careful. Theory predicts transformers fail on the general level, and the direct-training runs agree for small models. But frontier transformers are far larger and use tricks such as reasoning tokens, so the 13% figure describes the tested prompting setup, not a hard ceiling. The post does not say how many reasoning tokens each frontier model used.

Why did hybrids look bad when pretrained and good when trained directly? Ai2's answer is that pretraining data rarely contains long chains of dependent updates. A hybrid has the capacity but never gets the practice. That is encouraging for builders, because fine-tuning or synthetic data may close part of the gap. It is a hypothesis supported by the 275M runs, not yet shown at 7B scale.

What does "250 times the training length" mean? The models saw sequences of at most 256 assignments in training and were then scored on sequences of up to 64,000. Length generalization of that size is rare. It says the learned procedure is a real update rule rather than a memorized pattern, at least for these tasks.

Does a long context window solve it? No. A window lets the model see all the updates. StateBench asks whether it can combine them correctly. The RULER-versus-spreadsheet gap shows that seeing the text and using it are different skills.

What this means for model choice

If your workload is mostly lookup, such as finding the latest value of a setting, today's frontier models are fine at the lengths tested. If it involves running totals, balances or swaps across many steps, treat any single-pass answer from a long prompt as risky. Cheaper options exist: ask the model to write code that replays the updates, keep a structured state table, or break the document into stages and carry a verified summary. These are engineering choices, not Ai2 recommendations, but they follow from the failure pattern the benchmark exposes.

How to use it

Ai2 links the tech report, the dataset on Hugging Face and the code. A practical plan:

1. Run your model at several input lengths on each level, not only the hardest.
2. Test more than one format, such as RULER style and spreadsheet edits.
3. Plot accuracy against length. A cliff between levels two and three tells you where the model needs help.
4. If your product depends on long chains of updates, add tools that store the state outside the model.

Related Ai2 work from the past weeks includes NoveltyBench on research novelty judging and LitPivot.

Limits and open questions

• Synthetic tasks. Variable assignment is not natural language, so real-world transfer is unproven.
• Small trained models. The architecture comparison used 275M models, not frontier scale.
• Fixed variable count. StateBench assumes a fixed number of variables. Real documents add new entities over time. Ai2 says its framework does not cover that yet.
• Open theory. Ai2's proof shows that a Gated DeltaNet with non-negative eigenvalues can solve all aperiodic tasks and some periodic ones, such as tracking even or odd counts. Its exact place in the hierarchy, including whether it solves all soluble tasks, remains open.
• No per-model table in the post text. Check the tech report before citing exact frontier figures.

Ai2 calls StateBench a possible "North Star" for architectures with more robust state tracking.

Related reading

• Claude Opus 5.5 launch, benchmarks and pricing
• Claude Fable 5.1 and Gemini 3.8 Flash for coding agent routing
• Ai2 NoveltyBench: LLM judges and research novelty
• Ai2 LitPivot research ideas
• Ai2 AstaBrief 8B cited research reports
• Sarvam Prism decision model

Details reflect Ai2's November 3, 2026 post and may be updated by the authors.