OpenAI MultiLangBench: Can a Model Rebuild SQLite From the Docs?

OpenAI has published a new benchmark repo called MultiLangBench, and the idea is blunt: give a language model the SQLite 3.40.1 documentation and ask it to implement SQLite. The result is graded against 24,000 tests converted from SQLite's own suite, and the score is a pair of numbers, API cost and test pass rate. As of October 19, 2026 the repo has three commits, and publishes no model results yet.

That makes this a release to understand rather than a leaderboard to quote. Below we explain how the eval works, why the design choices matter, and how it fits the wave of agentic coding evals we have covered recently.

Update (October 19, 2026): OpenAI publishes the first results

Later on October 19, OpenAI published Advancing coding in low-resource programming languages, which puts numbers on MultiLangBench and adds a second eval. OpenAI says the benchmark asks a model to build a SQLite-compatible database engine from scratch across 20 programming languages, and that it is releasing the setup, test suite and per-language results on GitHub.

The headline results, all from OpenAI's own post:

Per-language gains for Astra on MultiLangBench include COBOL (10.9% to 72.1%), OCaml (25.6% to 85.8%), Fortran (21.4% to 83.5%) and Visual Basic (27.5% to 89.4%). OpenAI says Astra scores above 83% in 19 of the 20 languages, with COBOL the hardest at 72%.

The second eval, a COBOL migration test, asks models to port COBOL programs to C++, Java, Kotlin, Python or Rust. It uses hidden tests that require byte-for-byte parity on observable outputs, and a migration counts only if every test passes. OpenAI gives a concrete trap: COBOL fixed-point arithmetic discards excess decimals unless rounding is requested, so a rewrite that rounds can run cleanly and still produce different financial results.

These are vendor-reported numbers on vendor-built evals, and OpenAI itself cautions that production use still needs expert review and testing against business rules. For the models involved, see our coverage of GPT-6.1 Sol and Astra cost efficiency, the GPT-6 Astra launch and the small-model comparison that includes Luna. The sections below were written before these results appeared, so read statements that no results are published as the state at the time of the first draft.

TL;DR: MultiLangBench at a glance

How does the SQLite eval work?

The README states it directly: the language model "has to implement SQLite version 3.40.1." It receives the documentation, which the repo says can be downloaded from the sqlite.org 2022 documentation archive into the container. The model must produce an executable that takes a database path as a command-line argument and an SQL query string over stdin.

Each test was converted from the SQLite TCL tests into three parts:

• initial_database: SQL code that creates an initial database file.
• user_query: an SQL query that the candidate implementation must run against that database to produce a modified database file.
• validation_query: a query run against the modified file using the real SQLite 3.40.1.

A test passes only if the validation query returns the exact expected output string. Test names are SHA-256 hashes of the three parts, which makes them stable identifiers. The repo also ships expected_output.txt files so you can check your own test runner before trusting its numbers.

The clever part is the grading direction. The candidate never has to match SQLite's text output for arbitrary queries. It has to leave behind a database file that the real engine reads correctly. That checks the on-disk file format, the part of SQLite that is hardest to fake.

Why this design is hard to game

Three rules in the README close the obvious shortcuts.

Remove every trace of SQLite. The README warns that SQLite is "a very popular program and often multiple copies" are installed, including libraries, executables and implementations embedded in other programs. It points to scripts/remove_sqlite.sh as an example and tells evaluators to analyze their own environment. Python's standard library, for instance, bundles SQLite, so a careless container would let a model import it and score full marks.

No internet. Without network access the model cannot download a SQLite source tree or a Rust port.

Language compliance. If the model does not write in the target language, or mixes languages, the test pass rate is set to 0. The README suggests another model can check solutions for this. The name MultiLangBench points to running the same task across several target languages, though the README text we read does not list which languages OpenAI intends to use, so treat the exact language set as unknown.

What does the score mean: cost and pass rate together

Most coding leaderboards report one number. MultiLangBench reports a tuple, (api_cost, test_pass_rate), which treats spend as part of the result. A model that reaches 60 percent for a few dollars and one that reaches 62 percent for ten times the money are different products, and the tuple keeps that visible.

This matches a trend we have tracked. FrontierCode 1.1 scored models on whether maintainers would merge their work, and the FDE gym benchmark measured long, messy engineering tasks. Cost-aware scoring is the missing axis in many of them.

The README also sets a cost-control rule: because "running all 24k tests can be expensive," evals should use only the 272 tests starting with 00, 01 and 02. Running the whole suite is allowed for more precision on high-scoring samples, but it should be noted. Anyone comparing two published numbers should check which subset each used.

Why SQLite? A task that is big but well specified

SQLite is a good target for a long-horizon coding test for a few reasons that follow from the README rather than from speculation:

• The spec is public. The documentation is the input, so there is no hidden requirement.
• Ground truth is available. The real SQLite 3.40.1 acts as the judge.
• The test corpus is large and independent. The tests come from the project's own TCL suite, not from a lab-written set.
• The task is open-ended in size. A model can get basic CREATE TABLE and INSERT working quickly, but the long tail, such as the file format and edge cases, takes sustained effort.

Rewrites of this database are already a live topic. Our guide to Turso's Rust rewrite of SQLite shows how much engineering a faithful reimplementation takes, and our look at OpenAI's Habitat Python-to-Rust rewrite shows the company is itself interested in model-driven rewrites. MultiLangBench turns that kind of work into a measurable exercise. We are not claiming a link between those projects and this repo; the README does not make one.

What is not known yet

Being honest about the gaps is more useful than guessing:

• No scores. The repo has no results table. We cannot say how any OpenAI or rival model performs.
• No stated purpose. The README does not say whether OpenAI will use it for model launches, internal tracking, or as an open invitation to the community.
• Contamination risk. The SQLite source and TCL tests are public and almost certainly in training data. The hashed test names and conversion step help against memorized test files, but a model that memorized SQLite's source could still reproduce large parts of it. Our note on Epoch's InnovationEval covers a related concern about evals that look novel but are not.
• Language list. The target languages are not enumerated in the README we read.
• Early signals. Zero stars and three commits at the time of our check mean there is no community scrutiny of the harness yet.

What people will ask next

Is this a test of memory or of engineering? Both, and the README does not separate them. A model that has seen SQLite's C source can reproduce structure from memory, but the eval forbids bundling the real library and requires the work to be in a named language, so it must still produce a working program. Comparing the same model across several target languages is the natural way to separate recall from real implementation skill.

Why only 272 tests by default? Cost. The 272 tests whose hashed names begin with 00, 01 or 02 form a roughly random slice, because the names are SHA-256 hashes. That makes the subset a fair sample of the 24,000 rather than a hand-picked easy set, and it keeps a single evaluation affordable.

What would a good result look like? The README gives no threshold. A useful reading is relative: pass rate at a given cost, across models, on the same subset. Until someone publishes runs, any absolute claim is guesswork.

Can an agent harness change the score? Almost certainly. The README describes the task and the grader, not the agent scaffold, so a model with a strong coding harness and a long budget will spend more and may score higher than the same model with a thin prompt. That is why the cost half of the tuple matters.

How to try it yourself

If you want to run a model on this, the README gives the recipe. This is our reading of it, not a tested walkthrough:

1. Clone the repo and install requirements.txt.
2. Build an offline container with the SQLite 3.40.1 documentation and without any SQLite binaries or libraries; use scripts/remove_sqlite.sh as a starting point and verify manually.
3. Let your model or agent write an executable in your chosen language.
4. For each of the 272 tests starting with 00, 01 and 02, upload input.db, run your executable with user_query on stdin and the database path as the argument, then download the result as output.db.
5. Run the validation_query against output.db with the real SQLite 3.40.1 and compare the output to the expected string.
6. Record total API spend next to the pass rate.

A few practical tips: set the 60-second per-test timeout in your runner, log which tests fail by hash so you can compare runs, and run an independent check that the solution contains no wrapper around a hidden SQLite copy. The same sandboxing concerns apply as with any agent run; if you are letting a coding agent loose in a container, see our guide to folkbench and API relay risks for what can go wrong with third-party routing.

What this means for builders

For people picking coding models, the takeaway is not a ranking, because there is none. It is a method: when you evaluate agents on your own codebase, borrow this structure. Use a public spec, an independent oracle that is not the model, hard bans on shortcuts, and a cost figure beside the pass rate. The same pattern works for a file-format parser, a protocol implementation, or an internal tool you want an agent to rebuild.

It also suggests where evals are heading. Pass-at-one on short functions is saturated; whole-system reimplementations with exact-match oracles are the next stress test. Watch for third-party runs of this repo, and for any official numbers from OpenAI, and apply the usual discipline: compare like with like (same subset, same language, same cost accounting).

Sources and further reading

• Primary source: the openai/MultiLangBench repository and README
• Ground truth: the official SQLite documentation and test suite the eval converts from
• Related on explainx.ai: Torvalds and Claude Code on a kernel patch, FrontierCode 1.1, FDE gym, Turso's SQLite rewrite, OpenAI Habitat rewrite, Epoch InnovationEval

Details reflect the repository as of October 19, 2026 and may change as OpenAI updates it.