explainx / blog
Fireworks AI ran Kimi K3 against Fable 5 on 1,030 agentic tasks — near-tied on average, wildly different underneath, and routing between them beat both models alone at up to 50x lower cost. Here's the methodology and the catch.
Jul 21, 2026
On July 20, 2026, The Bugged Dev gave Kimi K3 the same 3D football stadium challenge previously run on Claude Fable 5. Fable finished under an hour with one big HTML file; Kimi took nearly three hours — running E2E tests, screenshots, and responsive checks. explainx.ai unpacks the anecdote, the "vibe coding vs vibe engineering" frame, and why harness + verification behavior matter more than wall-clock time.
Jul 13, 2026
@EntelligenceAI claims Gemini 3.5 Pro tops Fable 5 and GPT-5.6 in internal tests with a July 17 launch window. X replies say wait for real evals — explainx.ai separates leak hype from what Google must prove.
Jul 7, 2026
Polymarket and @kimmonismus amplified an r/LocalLLaMA projection: frontier models like Claude Fable reach laptop-runnable open-weight parity in ~24.8 months. explainx.ai unpacks the historical lag math, why you will not run Fable weights locally, and what builders should plan for by mid-2028.
A company that sells inference hosting for open-weight models just published a benchmark arguing you shouldn't pick one model at all. Fireworks AI ran Kimi K3 (Moonshot's open-weight model) against Claude Fable 5 (Anthropic's closed frontier model) on 1,030 agentic tasks across five categories, then measured what happens if you route each task to whichever model handles it best. The headline result: routing beat either model running alone, at up to 50x lower cost than defaulting to Fable 5 for everything.
The methodology and the incentive behind it both matter here — Fireworks profits directly from Kimi K3 traffic, and the study's most eye-catching number relies on a routing method that can't actually be deployed as described. Here's what the data shows once you separate the real findings from the framing.
| Question | Answer |
|---|---|
| Who ran the study? | Fireworks AI, published July 21, 2026 |
| What was tested? | Kimi K3 vs. Claude Fable 5, 1,030 tasks, 5 categories |
| Overall accuracy? | Near-tied — 92.4% (K3) vs. 92.6% (Fable) on SWE tasks |
| Cost difference? | Up to 50x cheaper for K3 on some task types |
| What's "oracle routing"? | Best-case ceiling using hindsight — not a real deployable router |
| How often did the oracle pick K3? | 72-96% of tasks depending on category |
| Is Fireworks neutral? | No — it's an inference host that profits from K3 traffic |
| Is K3 fully open yet? | Weights/technical report expected ~July 27, 2026 |
Fireworks built its evaluation around five distinct task families, each stressing a different kind of agentic work rather than static code-completion:
| Category | What it tests | Tasks |
|---|---|---|
| SWE | Real repo bug-fixes, SWE-bench style | 460 |
| Terminal | Long agentic ops — security, crypto, reverse-engineering, sysadmin | 89 |
| Algorithmic | LeetCode/AtCoder-style problems | 100 |
| Multi-Language | Implementation across six languages | 225 |
| Legal | Lawyer-graded legal-agent tasks | 120 |
Running both models through the same harness on identical tasks is the methodologically sound part of this study — it isolates the model as the variable rather than conflating model quality with harness differences, the same distinction explainx.ai's agent harness guide covers as the reason two different harnesses running the same model can produce wildly different results.
At the aggregate level, Kimi K3 and Fable 5 land close enough to call it a wash — 92.4% vs. 92.6% on the SWE category specifically. But Fireworks' domain-level breakdown tells a different story: Kimi K3 was sharpest on symbolic math and dev tooling, while Fable 5 won on web and data visualization work. The same specialization pattern held in the multi-language set, where Fable's breadth carried Java, Python, and C++, while K3 drew even specifically on JavaScript and Rust.
The terminal category showed the widest gap. Of 89 long-horizon terminal tasks — the kind involving dozens of turns operating a real shell — Kimi K3 cleared several that Fable 5 never solved at all: a 7z hash crack, FEAL cryptanalysis, leaked-secret detection, a live vulnerability, and a runaway async job. K3 took 11 solo wins on that category versus Fable's 7, with K3 specifically dominating the security and cryptography cluster.
The 50x cost figure comes from a combination of per-token pricing and prompt-caching behavior, not just a flat "K3 is cheaper" story. On SWE tasks specifically, Kimi K3 actually used more effort than Fable 5 — roughly 55 turns and 1.3 million tokens per task, versus Fable's 21 turns and 130,000 tokens. Despite that heavier workload, K3's cost still came in lower because the bulk of that context was served from cache rather than processed fresh each time.
The pattern flips on long terminal tasks: there, Fable 5 is the one that spirals, running up to 64 turns and 1.5 million tokens per task — sometimes hitting a timeout outright. Neither model is uniformly "leaner" — the extra effort just falls on different task types for each one, which is exactly why a single aggregate cost number obscures more than it reveals.
This is the part of the study that drew the sharpest Hacker News pushback. Oracle routing works by running a task through both models, checking which one actually got the correct answer, and then reporting the cheaper of the two correct results — a theoretical ceiling on what routing could achieve, computed with the benefit of already knowing the outcome. Fireworks reported the oracle router selected Kimi K3 for 72% to 96% of task traffic depending on category, with cost savings reaching 50x in the best case.
Commenter platinumrad put the core objection bluntly: "Oracle routing is by definition not possible." A real production router has to predict which model will succeed before running the task — it doesn't get to peek at both outcomes first and pick the winner in hindsight. Commenter robthompson2018 pushed further: "The way they published this is baffling to me... surely you can try to implement some router and then see how well it does. Using an Oracle makes the whole writeup so much less interesting." Fireworks' own framing acknowledges this gap, describing oracle routing explicitly as "the cost/performance ceiling" rather than claiming it as an achieved result — but the headline "up to 50x cheaper" figure is the ceiling number, not a demonstrated production outcome, a distinction easy to lose in a skim of the post.
Several commenters raised the obvious point directly: Fireworks is an inference hosting company whose business depends on serving open-weight models like Kimi K3, giving it a clear commercial incentive to publish a study favorable to routing traffic toward open models it can host at a markup. Commenter jrflo put it simply: "a company that hosts open models is telling us how good open models are." Others pushed back that disclosure of a commercial angle doesn't automatically invalidate the underlying methodology — commenter mrinterweb noted Fireworks hosts more than just open models, and that publishing a result doesn't make it false — but the incentive is real and worth weighing when reading the framing (which model "wins" the story) separately from the per-category task data (which is independently reproducible against the same benchmark suite).
Fireworks' core argument — that per-task routing beats any single static model choice — isn't a new idea in isolation. explainx.ai has covered the same underlying pattern from multiple angles this year: OmniRoute's auto-fallback combos exist precisely because no single provider reliably wins every task at every price point, and OpenRouter's own auto routing mode — mentioned repeatedly in the Hacker News discussion of this study — already ships a live version of task-to-model routing based on aggregate usage patterns rather than a hand-tuned oracle. What Fireworks adds isn't the concept, it's a controlled, task-labeled dataset that quantifies how much is actually on the table if routing were done well — even if the 50x figure represents the ceiling rather than what any shipped router currently achieves.
The open question raised directly in the thread — commenter johnhess asked whether this was "an out of sample test of the router or was it trained on these specific use cases" — is exactly the right skepticism to apply before trusting any routing benchmark's generalization claim. A router tuned on the same 1,030 tasks it's evaluated against will look far more impressive than one facing genuinely novel task distributions in production, and Fireworks' post doesn't fully clarify which scenario its oracle numbers represent.
Regardless of how the oracle-routing number should be discounted, the underlying specialization pattern is the more durable finding: Kimi K3 and Fable 5 are not interchangeable substitutes that happen to score similarly — they're differently-shaped tools that win on different task types for different reasons (terminal/security work vs. web/multi-language breadth). That argues less for "switch entirely to K3" and more for the routing-and-fallback pattern already covered in explainx.ai's OmniRoute gateway guide — running a cost-optimized default model for most traffic, with a fallback or escalation path to a different model for the task types where it demonstrably wins, rather than a single static model choice for every workload.
Benchmark figures, quotes, and methodology reflect Fireworks AI's July 21, 2026 publication and Hacker News discussion from the same day. Kimi K3's full weights and technical report were not yet published at the time of this post — verify current release status before citing licensing terms.
