The model was not as stuck as the leaderboard made it look — the harness kept wiping its mind.
On July 30, 2026, OpenAI’s Tibo (@thsottiaux — Codex & ChatGPT) put the claim bluntly: GPT-5.6 Sol is SoTA on ARC-AGI-3 — “just two setting changes… allow it to reason and work over multiple context windows with the help of our canonical compaction implementation,” linking How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. Punchline in the post: GPT-5.6 Sol can solve open math and late-cutoff games, yet the official ARC-AGI-3 harness made it look weak because it discarded private reasoning after every move and truncated older actions. Flip on retained reasoning + compaction (Responses API — what ChatGPT and Codex already use) and the public-set score moves 13.3% → 38.3% with ~6× fewer output tokens.
That lands one week after Opus 5’s verified ~30.2% ARC Prize row and Anthropic’s “three times the next best model” framing. The internet’s correct instinct is not “one lab lied.” It is: a benchmark score is model × harness × settings.
TL;DR
| Question | Answer |
|---|---|
| What changed? | Responses API harness with retained reasoning + compaction |
| Public-set official harness | 13.3% RHAE |
| Public-set OpenAI harness | 38.3% RHAE (~188% relative lift / ~3×) |
| Token effect | ~6× fewer output tokens |
| Human baseline (OpenAI est.) | ~48% RHAE |
| Official board Sol Max (Jul 24) | ~7.8% (ARC Prize) |
| Official board Opus 5 High | ~30.2% — our Opus 5 read |
| Root cause of “dumb Sol” | Reasoning dropped each turn + rolling truncation |
| Builder fix | Responses API · retain reasoning · compact — not Chat Completions defaults |
What people are asking
“Did OpenAI secretly train a new model?”
No. Same GPT-5.6 Sol (max) story, different serving stack. OpenAI’s claim is that eval runners were measuring a generic harness that fights how reasoning models are trained — private thoughts retained across turns, then summarized when the window gets long.
If your agent harness throws away CoT every tool call, you are not measuring Sol-in-Codex. You are measuring Sol-with-amnesia.
“Is Sol now SoTA over Opus 5?”
OpenAI’s Tibo says yes — on the Responses harness. On July 30 he posted that Sol is SoTA after “two setting changes”: let it reason across turns and work over multiple context windows via OpenAI’s canonical compaction. That is advocacy for the public-set 38.3% curve below, not ARC Prize rubber-stamping a new official row.
Not on the official ARC Prize board. That board still shows Sol Max ~7.8% (Jul 24 snapshot) vs Opus 5 High ~30.2% under ARC’s protocol. Fair statement:
| Claim | Status |
|---|---|
| Tibo / OpenAI: Sol is SoTA after retained reasoning + compaction | Employee claim on OpenAI’s public-set harness |
| Harness settings tripled Sol’s public-set score (13.3% → 38.3%) | OpenAI-reported |
| Official Sol Max ~7.8% under ARC protocol | ARC-verified (snapshot) |
| Opus 5 High ~30.2% under ARC protocol | ARC-verified |
| Sol 38.3% “beats” Opus 5 30.2% head-to-head | Apples ≠ oranges until ARC re-runs with documented setting parity |
Treat the post as a harness autopsy plus a SoTA marketing frame, not a silent leaderboard rewrite. Same lesson as our AI benchmarks guide: always ask who ran it and with what wrapper.
“Why did Anthropic’s victory tweet look absurd?”
Because the gap was sold as pure model IQ. Claude’s launch-week line — Opus 5 three times the next best on ARC-AGI-3 — was directionally true on the official board (30.2% vs Sol Max ~7.8%). Peter Steinberger’s jab (linking OpenAI’s post) asks the right follow-up: did anyone ask whether the “next best” was handcuffed by a harness that deletes thoughts?
Both can be true:
- Opus 5 is strong on interactive adaptation under ARC’s protocol.
- Sol’s reported official-board weakness partly reflected API/harness defaults, not a hard ceiling.
That is model vs effort / trying harder energy applied to memory policy.
What broke in the standard harness
OpenAI’s investigation (credited to Ilan Bigio and Ted Sanders in Tibo’s follow-up) found two concrete failure modes:
1. Discarded reasoning every action
After each game action, private reasoning messages were dropped. The model still saw a record of past moves and short notes — but not the plans and hypotheses that produced them. Every turn forced a cold start on “how does this game work?”
For a benchmark whose whole point is learning unfamiliar 2D games without instructions, that is sabotage dressed as fairness.
2. Rolling truncation of history
When context exceeded ~175,000 characters, the oldest messages were cut. Sol lost earlier observations and actions and spent long stretches with a stuffed window — which OpenAI says can slightly hurt performance.
Together: no durable thoughts + disappearing past ≈ “keeps starting over,” which matches OpenAI’s public framing on X.
What OpenAI turned on
| Setting | What it does | Why ARC-AGI-3 cares |
|---|---|---|
| Responses API | Modern multi-turn / tool surface; easier previous-response chaining | Matches ChatGPT / Codex training+deploy loop |
| Retained reasoning | Keep opaque reasoning items across turns (e.g. via previous_response_id) | Plans survive move→move |
| Compaction | Summarize / compress state instead of hard-dropping oldest turns | Long games keep what was learned |
OpenAI: with reasoning retained, Sol thought less per action (no full re-parse) and learned coherent strategies over time. Compaction then beat truncation on long runs — higher score, fewer output tokens.
Docs: Reasoning models · Compaction · Responses API.
Concrete demo claim
On ARC task cd82, OpenAI says no frontier model solved beyond level one on the official harness leaderboard for that game — while Sol with the Responses harness cleared all six levels. That single-game clip is illustrative, not a full-board replacement.
The numbers, carefully
| Metric | Value | Caveat |
|---|---|---|
| Sol Max on official ARC-AGI-3 board (Jul 24) | ~7.8% | ARC protocol / snapshot |
| GPT-5.5 on ARC-AGI-3 (OpenAI) | 0.4% | Nearly unplayable |
| Public set + official harness | 13.3% | OpenAI’s re-run baseline |
| Public set + retained reasoning + compaction | 38.3% | OpenAI harness |
| Relative score change | +188% | Matches “tripled” marketing |
| Output tokens | ~6× fewer | Efficiency, not just accuracy |
| Human tester estimate | ~48% RHAE | From OpenAI citing human gameplay logs |
OpenAI chart: dashed green = official harness (~13% at ~2.9M output tokens/game); solid blue = retained reasoning + compaction (~38% near ~0.5M tokens/game). Same model, different memory policy — higher score and far lower token spend.
RHAE (Relative Human Action Efficiency) scores models against human baselines — agents are not shown their score mid-run; they only get frame text and level info (ARC methodology).
ARC intentionally ships a generic harness without tools so model weaknesses stay visible and comparisons stay “fair.” OpenAI’s counter: fairness that drops reasoning is fairness that mis-measures reasoning models. Commercial products optimize the wrapper; generic evals often do not.
That tension is the whole harness engineering debate — also why HN argued “include harnesses” vs “exclude harnesses” on the Opus 5 thread.
How this fits the Opus 5 story
| Snapshot | Read |
|---|---|
| Opus 5 High 30.2% (ARC-verified) | Real interactive-adaptation jump under ARC rules |
| Sol Max ~7.8% (ARC board) | Understates Sol if production memory is disabled |
| Sol public 38.3% (OpenAI harness) | Shows ceiling when thoughts persist |
| Anthropic “3× next best” | True on that board; incomplete as a product claim |
| OpenAI “perf = model + product” | Correct engineering slogan; self-serving on “SoTA” |
explainx.ai’s posture stays the same as our Opus 5 ARC writeup: celebrate verified rows, refuse both AGI cosplay and “all benches are fake,” and run private interactive toys that never hit social media.
Pair this with Claude 5 context engineering and Claude Cookbook compaction recipes — every lab is discovering that memory policy is product, not an afterthought.
What builders should do this week
1. Stop eval-ing GPT-5.6 Sol on Chat Completions defaults
If your suite is still Completions-only and drops reasoning items, you are reproducing OpenAI’s bug report on purpose.
2. Enable the production triad
OpenAI’s explicit recommendation for API developers:
- Responses API, not legacy Chat Completions
- Retain reasoning across turns / tool calls
- Use compaction instead of silent rolling truncation
Sketch (Python-shaped; confirm against live docs):
# Long agent loop pattern (conceptual)
# 1) Create responses with previous_response_id so reasoning carries
# 2) When the window grows, compact instead of slicing oldest turns
resp = client.responses.create(
model="gpt-5.6-sol", # or your routed Sol SKU
input=turn_input,
previous_response_id=prev_id, # retains relevant reasoning items
# enable compaction / context_management per current API docs
)
# Later:
compacted = client.responses.compact(
model="gpt-5.6-sol",
input=long_input_items,
)
# next turn input = compacted.output + new user/tool messages
Exact parameter names (reasoning.context, context_management.compact_threshold, etc.) move — pin the reasoning and compaction pages on the day you ship.
3. Split your scoreboard
| Track | What it answers |
|---|---|
| Protocol-fair | Cross-lab comparison under published harness rules |
| Product-real | Same model with your production memory / compaction / tools |
| Private novel env | Your moat — never publish the game |
Do not let marketing pick only one track.
4. Instrument “thoughts dropped” as a bug class
Log whether each step includes prior reasoning items. If your harness strips them for “clean logs,” you just recreated ARC’s failure mode. Same class of footgun as Codex full-access disasters: the wrapper decides the outcome.
5. Play the public games yourself
OpenAI points builders at arcprize.org/tasks (25 demo games). Feel the learning curve humans face before you trust a percentage.
Honest limitations
- 38.3% is OpenAI-reported on the public set with their harness — not an ARC Prize leaderboard swap.
- Effort / SKU labels (max vs high) still muddy cross-lab charts.
- Compaction is opaque; you trade inspectability for continuity (context engineering tension).
- Token savings on ARC grids (often ~1:1 tokenized) may not equal savings on code agents.
- “SoTA” tweets from employees are advocacy, not ARC verification.
- ARC’s generic-harness philosophy still has a point: product wrappers can hide model flaws.
- Independent replication on the semi-private set under documented setting parity is the missing piece.
Closing
OpenAI’s ARC-AGI-3 post is less “Sol secretly crushed Opus” and more a public autopsy of amnesia evals. If your agent forgets its own plan every tool call, you will under-score every reasoning model — then ship a misleading chart. Turn on the settings your products already use in ChatGPT/Codex, keep a separate protocol-fair board, and update your private suite before the next victory tweet.
Follow @explainx_ai when ARC republishes rows with explicit harness notes — that is the scoreboard upgrade this story still needs.
Related on explainx.ai
- Opus 5 hits 30.2% on ARC-AGI-3 — leaderboard & HN
- What is an agent harness?
- Agent harness engineering — seven planes
- Claude 5 context engineering (Thariq)
- Claude Cookbook — PTC & compaction
- Claude Code model vs effort
- AI benchmarks complete guide 2026
- Claude Opus 5 launch
- GPT-5.6 Sol / Terra / Luna preview
- Recursive reasoning — HRM / TRM
Sources
- OpenAI — How enabling two settings tripled our scores on ARC-AGI-3
- Tibo (@thsottiaux) on X — “Sol is actually SoTA… two setting changes”
- OpenAI — Reasoning models
- OpenAI — Compaction
- ARC Prize leaderboard
- ARC methodology (RHAE)
- Human gameplay logs (HF)
- ARC public tasks
Figures from OpenAI’s July 29–30, 2026 post and ARC Prize snapshots around late July 2026. Re-verify live API parameters, public-set numbers, and leaderboard rows before citing in investor or procurement docs.