Every coding-agent benchmark has the same structural weakness: if the tasks come from public GitHub repositories, there's a real chance the model being tested has already seen that exact code — and the fix — somewhere in its training data. On September 10, 2026, Specific Labs co-founder janak (@janaksunil) launched Real-SWE, a coding benchmark built specifically to close that gap: every task comes from a private, out-of-distribution company codebase no model could have memorized.
TL;DR
| Question | Direct answer |
|---|---|
| What is it? | A coding-agent benchmark built entirely from private, real company codebases rather than public repos. |
| Why does that matter? | Public-repo benchmarks like SWE-bench risk memorization — frontier models may have seen the exact code and fix during training. Private codebases can't be memorized. |
| What kind of companies? | An app with 200,000+ users, a fintech platform processing 100,000+ bank statements, and enterprise sales tools, among the examples given. |
| What's a sample task? | Fix invoice billing so each business charges the right tax and exempt customers aren't taxed — requiring the agent to infer the business's existing tax-handling logic from context. |
| What's the standout result? | GLM-5.3 scored surprisingly high relative to frontier closed models — notable for an open-weight model on a benchmark designed to resist memorization. |
| Where do I see it? | realswe.withspecific.com and Y Combinator's Launches page. |
The problem Real-SWE is built to solve
SWE-bench and its many descendants — Senior SWE-Bench, Terminal-Bench 2.0, and others explainx.ai has covered this year — have done real work pushing coding-agent evaluation beyond leetcode-style toy problems toward realistic software engineering tasks. But nearly all of them draw from public repositories. A frontier model trained on a large slice of GitHub has plausibly seen the exact bug, the exact pull request that fixed it, or at minimum the coding conventions and architecture of that specific project.
janak's framing cuts straight at that gap: "Can an agent figure out how a company handles billing, permissions, or customer data from the code and context it's given? These are the systems companies would actually deploy agents into." Real-SWE's answer is to source every task from a codebase the evaluated model has never seen and never could have seen — because it's private, and belongs to a real operating company.
What's actually in the benchmark
Real-SWE draws its tasks from real companies operating at real scale, per janak's launch thread:
- An app with 200,000+ users
- A fintech platform processing 100,000+ bank statements
- Enterprise sales tools
The representative task example is concrete and specifically chosen to require system-level inference, not just code-pattern matching: "Fix invoice billing so each business charges the right tax and exempt customers aren't taxed." To solve it, an agent has to:
- Work out how the business currently handles tax logic — by reading the existing code, not by being told
- Connect the right tax provider integration
- Keep invoices consistent across the change, without breaking cases already handled correctly
None of that is answerable by having memorized a similar GitHub issue. It requires the same kind of contextual system-reading a new engineer does in their first week at a company — a distinction Specific Labs is betting matters more than raw benchmark familiarity for real enterprise deployment.
Why "out-of-distribution" is the load-bearing phrase
The core thesis, stated directly in janak's thread: "For coding agents to be useful inside companies, they need to solve problems in systems they haven't encountered before." That's a different bar than most public benchmarks test. A model can score well on SWE-bench by being extremely good at recognizing patterns common across open-source Python and JavaScript projects — patterns it has seen thousands of times in pretraining. Real-SWE is designed so that pattern-recognition shortcut doesn't work; the agent has to actually reason about an unfamiliar system's architecture and business logic from the artifacts in front of it.
This lines up with a critique explainx.ai has tracked across 2026's benchmark landscape: as leaderboards saturate, the benchmarks that keep differentiating models are the ones structurally resistant to memorization, not just harder versions of the same task shape. How to Read AI Benchmarks covers this pattern in more depth — Real-SWE is a fresh, concrete instance of the "make the eval un-memorizable" strategy applied specifically to enterprise coding.
The GLM-5.3 result people are reacting to
Early reaction to Real-SWE's leaderboard has centered on one number: GLM-5.3 scored surprisingly high, prompting visible surprise even from commenters expecting frontier closed models to dominate a benchmark this hard to game. explainx.ai covered the real mechanics behind GLM-5.3's "50% coding boost" claim — a specific internal Z.ai benchmark score, not a blanket capability jump — and separately covered its independent CyberGym validation and its third-place Terminal-Bench 4 finish. A strong Real-SWE showing is a different kind of evidence than any of those: it suggests GLM-5.3's coding gains generalize to genuinely unfamiliar, private enterprise code — not just benchmarks or task shapes closer to its own training distribution.
That's a meaningfully higher bar to clear than most open-weight coding claims get held to in 2026, which is exactly why the result is drawing attention.
What this means for teams evaluating coding agents
If you're choosing a coding agent for internal deployment — not a green-field open-source contribution, but changes to your own company's private, idiosyncratic codebase — Real-SWE's approach is closer to your actual use case than a public-repo benchmark:
- Public-repo scores overstate real-world readiness for private codebases the model has never seen, because familiarity with open-source patterns doesn't transfer cleanly to a company's specific conventions and business logic
- A model's Real-SWE score is a better proxy for "can this agent onboard onto our system the way a new hire would" than its SWE-bench score
- Open-weight models closing the gap on out-of-distribution tasks (per GLM-5.3's showing) matters more for cost-sensitive internal tooling than closing the gap on public leaderboards that are increasingly saturated at the top anyway
The pattern this fits: harder-to-game evaluation
Real-SWE isn't an isolated idea — it's part of a broader shift in how the field is trying to keep coding-agent evaluation meaningful as models get better at recognizing familiar benchmark shapes. The same instinct shows up across several 2026 benchmarks explainx.ai has covered: Senior SWE-Bench replaced over-specified prompts with Slack-style, under-specified instructions closer to how a real engineer gets handed a task; Terminal-Bench 2.0 moved evaluation into full terminal environments rather than isolated code snippets. Real-SWE's specific contribution to that trend is sourcing data — using genuinely unseen, private codebases rather than restructuring the task format on top of public data that's still potentially memorized.
The three approaches are complementary, not competing: a benchmark can be both under-specified in its prompts (Senior SWE-Bench's contribution) and sourced from private, unseen code (Real-SWE's contribution). Expect future benchmarks to combine both properties as the field converges on what actually predicts real deployment success.
Honest limitations
- Benchmark is brand new (launched September 10, 2026) — no independent replication or long-run leaderboard stability yet; treat early rankings as a first data point, not settled fact.
- Task sourcing methodology isn't fully public in the launch thread. How companies are recruited, how tasks are selected and verified as solvable, and how data leakage is prevented going forward aren't detailed in the tweets — check realswe.withspecific.com for methodology before citing scores authoritatively.
- A private-codebase benchmark can't be fully open-sourced the way SWE-bench can, by design — that's the point (it resists memorization), but it also means outside researchers can't independently audit every task the way they can with a public dataset.
- Specific Labs is a young company (YC F25) launching its own benchmark — a legitimate methodology doesn't require third-party origin, but as with any lab-run leaderboard, watch for how it's maintained and whether new company codebases keep rotating in over time to stay resistant to overfitting.
The takeaway
Real-SWE is a direct answer to a criticism that's been building against public-repo coding benchmarks all year: they're increasingly measuring memorization and familiarity with open-source conventions, not the kind of contextual system-reading that actually determines whether a coding agent is safe to deploy inside a real company's codebase. By sourcing every task from a private, out-of-distribution business — and posting results like GLM-5.3's surprising showing — Specific Labs is pushing the coding-agent evaluation conversation toward the question that actually matters for enterprise adoption: can the agent figure out how your system works, not just recognize one it's seen before.
Related on explainx.ai:
- GLM-5.3's "50% Coding Boost" Explained — the real benchmark behind Z.ai's headline claim
- GLM-5.3 CyberGym: 84.5% Independent Validation — third-party validation of GLM-5.3's security-relevant coding skill
- Senior SWE-Bench: Snorkel AI's Benchmark for Tasteful Code — another public benchmark pushing past over-specified prompts
- Terminal-Bench 2.0: AI Agent Benchmark Evaluation — agentic terminal-task evaluation methodology
- How to Read AI Benchmarks — a framework for judging what a benchmark score actually tells you
- Top 10 Open/Closed Source Agent Harnesses 2026 — choosing the harness that runs whichever model you land on
Details in this post reflect Specific Labs' September 10, 2026 launch thread on X and its Y Combinator Launches listing. Benchmark methodology and leaderboard rankings may be updated post-launch — check realswe.withspecific.com for the current state before citing specific scores.