Why did Ploy switch from Claude Opus 4.8 to GPT-5.6 Sol?

Ploy's agent builds and edits marketing websites — planning, components, imagery, screenshots, completion checks. After four months with Opus 4.7/4.8 as default, GPT-5.6 Sol was the first challenger to beat their private eval suite on completed work: ~2.2× faster wall-clock (8m → 3m42s), ~27% lower cost ($3.06 → $2.22 per build), visual judge score 0.970 vs 0.936, and ~half the output tokens (leaner CSS — 2,508 chars vs 17,957 on a matched pair).

What share of GPT-5.6 eval failures were harness bugs not model failures?

Ploy reports roughly one-third of raw failures in the first cross-model eval run traced to harness assumptions — tool-call budgets sized for Opus's sequential style while GPT-5.6 fans parallel calls; eval executor lacking batched file reads; a dataset missing minScore so 0.98 visual scores "failed" on an implicit 1.0 default. Triage full traces before trusting pass rates — otherwise you grade the challenger on how well it imitates the incumbent.

Why did GPT-5.6 return empty file reads in Ploy's agent?

Ploy's code tool has 25 parameters (1 required). Claude Opus sends 2–3 used fields; GPT-5.6 sent all 25 every call, inventing plausible values like offset: 0 and siteId: "00000000-0000-0000-0000-000000000000". The read implementation treated offset: 0 as real — 52–64% of GPT-5.6 reads returned empty with success: true. Prompting and OpenAI strict mode did not fix it. The fix: provider-boundary schema transform making optional fields required but nullable (anyOf: [T, null]), then strip nulls before tool validation.

How did prompt cache misconfiguration make GPT-5.6 look 50% more expensive?

GPT-5.6 dropped partial-prefix implicit caching — new conversations shared a 29K static prefix but cached 0% until explicit prompt_cache_breakpoint + prompt_cache_key. Per-conversation keys never hit the shared prefix (0% first-call hits). One global key overloads OpenAI's ~15 req/min per cache node. Ploy used per-workspace keys: all conversations in a customer workspace share entries; traffic stays under node budget. After the change: first-call cache hits ~0% → 83.7%, uncached input −28%, per-suite cost below Opus.

What is the reasoning replay fix for GPT-5.6 Responses API?

Default replay sends prior-turn reasoning as server-side item references (rs_...). Ploy saw intermittent Item not found mid-conversation. Fix: store: false — SDK requests encrypted reasoning blobs for self-contained replay. Corollary: with server-side reasoning state, effective prompts can change upstream even when your bytes are append-only.

How does this connect to enterprise private benchmarks?

Ploy's story is [Nadella's Control pillar](/blog/satya-nadella-reverse-information-paradox-enterprise-ai-july-2026) in production: private evals on real fixture workspaces, visual judges, tool-trajectory checks, trace triage. Cross-model migration without a harness audit is procurement theatre. See [how to build enterprise AI benchmarks](/blog/how-to-build-enterprise-ai-benchmark-nadella-july-2026) for charter → golden tasks → model-agnostic runner.

GPT-5.6 Production Migration — Ploy 2026 | explainx.ai Blog

Ploy did not "switch models." They switched a stack — and discovered the stack was tuned to Opus.

On July 9, 2026, Lorenzo Gentile published how Ploy moved its website-building agent from Claude Opus 4.8 to GPT-5.6 Sol after four months where nothing beat Opus on their bar. Hacker News gave it 132 points — with a parallel argument about whether the prose sounded LLM-generated. Fair complaint. The engineering underneath is what enterprises should steal.

This pairs with Systima's harness overhead study: tokens are not the model; they are model + harness + cache config + tool contracts.

TL;DR — promise vs prerequisites

table · 3 cols

Metric (redesign suite, completed builds)	Opus 4.8 (n=11)	GPT-5.6 Sol (n=10)
Cost	$3.06	$2.22
Wall-clock	8m 00s	3m 42s
Input tokens	2.60M	1.70M
Output tokens	33.0K	17.1K
Visual score	0.936	0.970

Catch: Raw first eval run lied until they fixed the harness, schemas, cache keys, and reasoning replay.

Step 0 — Your harness is tuned to your incumbent

Ploy runs hundreds of cases against real fixture workspaces — homepage builds, clone-safety checks. Scoring includes:

Visual judge: 10 binary design checks
Content assertions
Tool-trajectory checks
File assertions
Full trace triage on every failure

First cross-model surprise:

"Your harness is tuned to your incumbent model, and you don't know it."

table · 3 cols

Harness assumption	Opus behavior	GPT-5.6 behavior
Tool-call budgets	Sequential	Parallel fan-out → budget blow
File reads	Rarely batched	Constant batch reads → executor unsupported
Pass threshold	Missing `minScore` → default 1.0	0.98 hero "failed" while passing checks

~⅓ of first-run failures were harness — not evenly distributed between models.

explainx.ai rule: Before any model procurement decision, run private evals with trace triage as a gate. Same lesson as 37% lab-to-production gap on public benchmarks — worse in private, because your harness is bespoke.

Step 1 — Tool schemas are provider behavior, not prompts

Silent corruption: GPT-5.6 sent all 25 code-tool properties on every call.

json

// GPT-5.6 — invented values look intentional
{ "action": "read", "file_paths": [...], "offset": 0, "timeout": 120000,
  "siteId": "00000000-0000-0000-0000-000000000000", ... }

table · 3 cols

Model	code(read) calls	All 25 properties
gpt-5.6	6,635	6,635 (100%)
claude-opus-4.8	2,898	4 (0.1%)
claude-sonnet-5	1,933	0

52–64% empty reads — tool returned success: true both ways.

Prompt fixes failed: "omit unused parameters," per-field OPTIONAL hints, OpenAI strict mode (identical behavior; would strip validation patterns).

Fix — nullable required transform (OpenAI only):

typescript

// Optional → required + nullable at provider boundary
// anyOf: [T, null] — model expresses honesty
// Strip nulls at single invocation seam before validation

Results: empty reads 52% → 0%; ~30% fewer tool calls (no re-read loops).

Lesson: "The model" in production is model + tool contract. Structured output guides help; provider-specific emission patterns need boundary transforms.

Step 2 — Prompt caching is not one feature

Surface: both vendors offer "prompt caching." Under the hood: different designs.

Claude (Anthropic)

cache_control breakpoints
~29K static prefix (tools + system)
Org-scoped shared cache — 92–96% hit rates, fades to background

GPT-5.6 (OpenAI) — what changed

Dropped partial-prefix implicit matching
New conversations: 0% of shared 29K prefix cached
Uncached prompts pay 1.25× cache-write surcharge
Requires prompt_cache_breakpoint + prompt_cache_key
Key is cache identity — same prompt, different key → zero hits
~15 requests/minute per cache node — global key → fan-out to cold nodes

Ploy's workspace-scoped key

snippet

request ──► hash(prompt head + prompt_cache_key) ──► cache node (~15 rpm/key)

  [ tools + static prefix ]············ A  every session
  [ + workspace context ]·············· B  same context
  [ + turn history + latest ]·········· C  this session

table · 3 cols

Key strategy	First-call hit	Problem
Per-conversation	0%	Never shares prefix
One global	Spill at scale	15 rpm budget blown
Per-workspace	83.7% after fix	Sweet spot

Entry B self-heals: workspace memory change misses B, still hits A — one context write vs full 29K re-bill.

No workaround: cross-workspace static prefix sharing is structurally impossible on OpenAI vs Anthropic org cache. Every workspace pays ~$0.18 per idle-window cold write — bounded, predictable.

Post-fix: uncached input −28%; GPT-5.6 per-suite cost below Opus. The 50% "GPT is expensive" gap was config, not list price.

Tie to Systima cache instability and prompt caching framework.

Step 3 — Reasoning replay must be self-contained

GPT-5.6 Responses API default: prior reasoning as server-side item refs (rs_...).

Ploy hit intermittent Item 'rs_...' not found mid-conversation.

Fix: store: false — encrypted reasoning blobs, self-contained replay.

Debugging corollary: server-side reasoning state means effective prompt can change upstream even when your client bytes are append-only. Log at the boundary.

Quality caveats — design convergence

Ploy's honest read: GPT-5.6 ships clean, modern, gridded layouts — can ignore existing design systems without steering. Opus reproduced Clay brand system; GPT-5.6 shipped generic-but-sharp.

Visual score still won on their suite — but brand adherence needed harness/workflow fixes (separate post promised).

HN thread split: some preferred Opus aesthetics for marketing; others cared about time × dollars × score.

Enterprise framing: define "good" in charter — visual judge + brand rubric + human slice, not one number.

Migration checklist (explainx.ai)

table · 2 cols

Phase	Action
0. Harness audit	Re-run incumbent vs challenger; triage traces; fix budgets, batching, thresholds
1. Tool boundary	Provider-specific schema transforms; null-strip seam; no tool impl churn
2. Cache design	Map breakpoints; choose key scope; measure first-call hit % cold
3. Reasoning	`store: false` or equivalent; verify mid-session continuity
4. Regression suite	Freeze golden workspaces; version harness with evals
5. Cost dashboard	$/build, p95 latency, human edit ratio — not pass@1 alone

Pair with token governance and loop engineering.

HN meta — prose vs substance

Top comments debated LLM writing style ("llmish"). Substance comments that aged well:

Consistency > one benchmark run for production agents
Luna routing for tool-touching subtasks (5× samples per Sol dollar)
Subagent isolation — research duplication eats uncached tokens (Systima 4.2×)
Fable extension to July 19 kept competitive pressure on OpenAI subs

We cite Ploy for engineering, not copy tone. If you publish migration numbers, human-edit the prose — credibility is part of the metric.

explainx.ai read

Ploy's post is the production complement to Nadella's essay and Systima's microscope:

Eval harness bias is invisible until you switch models — ~⅓ failure rate inflation
Tool verbosity is a silent correctness bug — not a prompt problem
Cache key design is architecture — 0% vs 84% first-call hits
Cross-vendor "caching" is homonym — procurement must compare $/successful-task at steady-state cache
Migration is a loop — eval fail → trace → harness fix → re-run (compound learning)

GPT-5.6 Sol winning on website agents does not generalize to your compliance workflow — run your private benchmark. Ploy proved the method; you bring the tasks.

Sources: Ploy — Migrating a production AI agent to GPT-5.6, Jul 9 2026 · Systima — Claude Code vs OpenCode tokens, Jul 12 2026

Migration details reflect Ploy's July 2026 publication and public HN discussion. Pricing, cache TTLs, and model IDs change — verify against current OpenAI and Anthropic docs before production cutover.

Ploy did not "switch models." They switched a stack — and discovered the stack was tuned to Opus.

This pairs with Systima's harness overhead study: tokens are not the model; they are model + harness + cache config + tool contracts.

TL;DR — promise vs prerequisites

table · 3 cols

Metric (redesign suite, completed builds)	Opus 4.8 (n=11)	GPT-5.6 Sol (n=10)
Cost	$3.06	$2.22
Wall-clock	8m 00s	3m 42s
Input tokens	2.60M	1.70M
Output tokens	33.0K	17.1K
Visual score	0.936	0.970

Catch: Raw first eval run lied until they fixed the harness, schemas, cache keys, and reasoning replay.

Step 0 — Your harness is tuned to your incumbent

Ploy runs hundreds of cases against real fixture workspaces — homepage builds, clone-safety checks. Scoring includes:

Visual judge: 10 binary design checks
Content assertions
Tool-trajectory checks
File assertions
Full trace triage on every failure

First cross-model surprise:

"Your harness is tuned to your incumbent model, and you don't know it."

table · 3 cols

Harness assumption	Opus behavior	GPT-5.6 behavior
Tool-call budgets	Sequential	Parallel fan-out → budget blow
File reads	Rarely batched	Constant batch reads → executor unsupported
Pass threshold	Missing `minScore` → default 1.0	0.98 hero "failed" while passing checks

~⅓ of first-run failures were harness — not evenly distributed between models.

Step 1 — Tool schemas are provider behavior, not prompts

Silent corruption: GPT-5.6 sent all 25 code-tool properties on every call.

json

// GPT-5.6 — invented values look intentional
{ "action": "read", "file_paths": [...], "offset": 0, "timeout": 120000,
  "siteId": "00000000-0000-0000-0000-000000000000", ... }

table · 3 cols

Model	code(read) calls	All 25 properties
gpt-5.6	6,635	6,635 (100%)
claude-opus-4.8	2,898	4 (0.1%)
claude-sonnet-5	1,933	0

52–64% empty reads — tool returned success: true both ways.

Prompt fixes failed: "omit unused parameters," per-field OPTIONAL hints, OpenAI strict mode (identical behavior; would strip validation patterns).

Fix — nullable required transform (OpenAI only):

typescript

// Optional → required + nullable at provider boundary
// anyOf: [T, null] — model expresses honesty
// Strip nulls at single invocation seam before validation

Results: empty reads 52% → 0%; ~30% fewer tool calls (no re-read loops).

Lesson: "The model" in production is model + tool contract. Structured output guides help; provider-specific emission patterns need boundary transforms.

Step 2 — Prompt caching is not one feature

Surface: both vendors offer "prompt caching." Under the hood: different designs.

Claude (Anthropic)

cache_control breakpoints
~29K static prefix (tools + system)
Org-scoped shared cache — 92–96% hit rates, fades to background

GPT-5.6 (OpenAI) — what changed

Dropped partial-prefix implicit matching
New conversations: 0% of shared 29K prefix cached
Uncached prompts pay 1.25× cache-write surcharge
Requires prompt_cache_breakpoint + prompt_cache_key
Key is cache identity — same prompt, different key → zero hits
~15 requests/minute per cache node — global key → fan-out to cold nodes

Ploy's workspace-scoped key

snippet

request ──► hash(prompt head + prompt_cache_key) ──► cache node (~15 rpm/key)

  [ tools + static prefix ]············ A  every session
  [ + workspace context ]·············· B  same context
  [ + turn history + latest ]·········· C  this session

table · 3 cols

Key strategy	First-call hit	Problem
Per-conversation	0%	Never shares prefix
One global	Spill at scale	15 rpm budget blown
Per-workspace	83.7% after fix	Sweet spot

Entry B self-heals: workspace memory change misses B, still hits A — one context write vs full 29K re-bill.

Post-fix: uncached input −28%; GPT-5.6 per-suite cost below Opus. The 50% "GPT is expensive" gap was config, not list price.

Tie to Systima cache instability and prompt caching framework.

Step 3 — Reasoning replay must be self-contained

GPT-5.6 Responses API default: prior reasoning as server-side item refs (rs_...).

Ploy hit intermittent Item 'rs_...' not found mid-conversation.

Fix: store: false — encrypted reasoning blobs, self-contained replay.

Debugging corollary: server-side reasoning state means effective prompt can change upstream even when your client bytes are append-only. Log at the boundary.

Quality caveats — design convergence

Ploy's honest read: GPT-5.6 ships clean, modern, gridded layouts — can ignore existing design systems without steering. Opus reproduced Clay brand system; GPT-5.6 shipped generic-but-sharp.

Visual score still won on their suite — but brand adherence needed harness/workflow fixes (separate post promised).

HN thread split: some preferred Opus aesthetics for marketing; others cared about time × dollars × score.

Enterprise framing: define "good" in charter — visual judge + brand rubric + human slice, not one number.

Migration checklist (explainx.ai)

table · 2 cols

Phase	Action
0. Harness audit	Re-run incumbent vs challenger; triage traces; fix budgets, batching, thresholds
1. Tool boundary	Provider-specific schema transforms; null-strip seam; no tool impl churn
2. Cache design	Map breakpoints; choose key scope; measure first-call hit % cold
3. Reasoning	`store: false` or equivalent; verify mid-session continuity
4. Regression suite	Freeze golden workspaces; version harness with evals
5. Cost dashboard	$/build, p95 latency, human edit ratio — not pass@1 alone

Pair with token governance and loop engineering.

HN meta — prose vs substance

Top comments debated LLM writing style ("llmish"). Substance comments that aged well:

Consistency > one benchmark run for production agents
Luna routing for tool-touching subtasks (5× samples per Sol dollar)
Subagent isolation — research duplication eats uncached tokens (Systima 4.2×)
Fable extension to July 19 kept competitive pressure on OpenAI subs

We cite Ploy for engineering, not copy tone. If you publish migration numbers, human-edit the prose — credibility is part of the metric.

explainx.ai read

Ploy's post is the production complement to Nadella's essay and Systima's microscope:

Eval harness bias is invisible until you switch models — ~⅓ failure rate inflation
Tool verbosity is a silent correctness bug — not a prompt problem
Cache key design is architecture — 0% vs 84% first-call hits
Cross-vendor "caching" is homonym — procurement must compare $/successful-task at steady-state cache
Migration is a loop — eval fail → trace → harness fix → re-run (compound learning)

GPT-5.6 Sol winning on website agents does not generalize to your compliance workflow — run your private benchmark. Ploy proved the method; you bring the tasks.

Sources: Ploy — Migrating a production AI agent to GPT-5.6, Jul 9 2026 · Systima — Claude Code vs OpenCode tokens, Jul 12 2026

Ploy’s GPT-5.6 Migration — Fix the Harness Before You Trust the Score

TL;DR — promise vs prerequisites

Step 0 — Your harness is tuned to your incumbent

Step 1 — Tool schemas are provider behavior, not prompts

Step 2 — Prompt caching is not one feature

Claude (Anthropic)

GPT-5.6 (OpenAI) — what changed

Ploy's workspace-scoped key

Step 3 — Reasoning replay must be self-contained

Quality caveats — design convergence

Migration checklist (explainx.ai)

HN meta — prose vs substance

explainx.ai read

Ploy’s GPT-5.6 Migration — Fix the Harness Before You Trust the Score

TL;DR — promise vs prerequisites

Step 0 — Your harness is tuned to your incumbent

Step 1 — Tool schemas are provider behavior, not prompts

Step 2 — Prompt caching is not one feature

Claude (Anthropic)

GPT-5.6 (OpenAI) — what changed

Ploy's workspace-scoped key

Step 3 — Reasoning replay must be self-contained

Quality caveats — design convergence

Migration checklist (explainx.ai)

HN meta — prose vs substance

explainx.ai read

Related posts

ChatGPT Pro Lists GPT-5.6 Sol, Luna, Terra as Unlimited*

What a Claude Code Task Costs on Opus 5.5

Reducing Claude API Cost: Caching, Prompt Audits, and Effort Tuning

Related posts

ChatGPT Pro Lists GPT-5.6 Sol, Luna, Terra as Unlimited*

What a Claude Code Task Costs on Opus 5.5

Reducing Claude API Cost: Caching, Prompt Audits, and Effort Tuning

TL;DR — promise vs prerequisites

Step 0 — Your harness is tuned to your incumbent

Step 1 — Tool schemas are provider behavior, not prompts

Step 2 — Prompt caching is not one feature

Claude (Anthropic)

GPT-5.6 (OpenAI) — what changed

Ploy's workspace-scoped key

Step 3 — Reasoning replay must be self-contained

Quality caveats — design convergence

Migration checklist (explainx.ai)

HN meta — prose vs substance

explainx.ai read

Related on explainx.ai

TL;DR — promise vs prerequisites

Step 0 — Your harness is tuned to your incumbent

Step 1 — Tool schemas are provider behavior, not prompts

Step 2 — Prompt caching is not one feature

Claude (Anthropic)

GPT-5.6 (OpenAI) — what changed

Ploy's workspace-scoped key

Step 3 — Reasoning replay must be self-contained

Quality caveats — design convergence

Migration checklist (explainx.ai)

HN meta — prose vs substance

explainx.ai read

Related on explainx.ai

Related posts

ChatGPT Pro Lists GPT-5.6 Sol, Luna, Terra as Unlimited*

What a Claude Code Task Costs on Opus 5.5

Reducing Claude API Cost: Caching, Prompt Audits, and Effort Tuning

Related posts

ChatGPT Pro Lists GPT-5.6 Sol, Luna, Terra as Unlimited*

What a Claude Code Task Costs on Opus 5.5

Reducing Claude API Cost: Caching, Prompt Audits, and Effort Tuning