Google Cloud AI Research has released a benchmark that asks AI agents to do what a working engineer would bill for, and the best result so far is about half marks. The paper, PLE: Physical-Engineering's Last Exam (arXiv 2610.30537, submitted October 21, 2026), introduces 85 long-horizon tasks across structural, civil, electrical and mechanical engineering. The strongest configuration, Claude Opus 5.5 in Claude Code, reaches a mean score of 50.6%, spending 27 million tokens and 63 minutes per task on average, at roughly $17.18 per task according to the paper's cost chart.
This post explains what PLE tests, how it was built, and what the numbers say.
TL;DR: PLE at a glance
| Question | Answer |
|---|---|
| Who made it? | Google Cloud AI Research (13 authors, including Chen-Yu Lee and Tomas Pfister) |
| What does it test? | Multi-hour professional engineering deliverables, not textbook questions |
| How many tasks? | 85, in structural, civil, electrical and mechanical engineering |
| Best result | Claude Opus 5.5 in Claude Code: 50.6% mean, about $17.18 per task |
| Models evaluated | 14 frontier models across seven public agent harnesses |
| What is public? | 29 tasks with graders and harness; 56 stay private for a leaderboard |
| Repo and license | google-research/ple, Apache-2.0 |
| Not yet independently verified | The full leaderboard; 56 tasks are private |
What does an engineering task look like?
The authors describe tasks as realistic project folders. Each places an agent in a sandbox with a mean of 61 files: PDFs, spreadsheets, CAD and BIM drawings in DXF, DWG or IFC formats, correspondence and logs. The agent gets tools for documents, spreadsheets and code, and must produce cited, production-format deliverables such as engineering memos, calculation workbooks, equipment schedules or 2D and 3D drawings.
Figure 1 of the paper gives one example per discipline:
- Structural: size steel roof decking, open-web joists and lateral diaphragm framing under gravity, snow-drift and shear loads, keeping member demand inside code capacity.
- Civil: lay out a phased campus, access roads and utility corridors around wetland buffers, delivered as a 3D IFC model.
- Electrical: size low-voltage switchboards, cable trays and 32 feeder and branch circuits under ampacity and voltage-drop limits.
- Mechanical: select heat-rejection equipment, pumps and piping to meet a thermal load while minimizing 25-year life-cycle cost.
The paper's argument is that such work is hard for agents but unusually measurable, because deliverables can be checked against governing documents and physics. It also argues that rubric-only grading cannot separate a code-compliant design from an economically good one: two wire schedules can both pass every check while one costs more than 20% more to install.
How the benchmark was built
PLE uses two routes, and both end with expert validation:
| Route | How tasks are made | How they are scored |
|---|---|---|
| Expert-authored | Practising engineers from the pool of the authors' data vendor (Mercor) write the task end to end as a client or colleague request | Weighted rubric of binary, evidence-bound criteria |
| Co-authored | Engineers write the brief; an adversarial Creator-Solver agent pipeline builds the environment, an executable grader and reference deliverables from licensed engineering documents | Rubric constraints plus an economic or performance objective |
For co-authored tasks the authors say each is redesigned until a frontier solver agent no longer saturates it, passes an independent ground-truth audit, and is then validated or made harder by experts. This is the same adversarial idea behind other agent evaluations we have covered, such as TasteVal on research taste and Build-a-Chip Bench for chip agents.
The results
The paper evaluates 14 frontier models under seven publicly available harnesses. Figure 2 plots score against cost per task at list prices, averaged over three runs, with a median cost of $6.63 and a median score of 25.7%. Three pairs are highlighted off the performance-cost frontier: Claude Fable 5 in Claude Code, Grok 4.7 in Grok Build and Gemini 3.8 Flash in Antigravity. The best pair is Claude Opus 5.5 in Claude Code at 50.6% and $17.18.
Model-level averages across harnesses are circulating on social media, but we did not find them in the paper text we could read, so we do not repeat them here. The 50.6% is a single model-harness pair, not a model-wide average.
What the paper does say plainly: even the best configuration is "only 50.6%," which the authors read as "the substantial gap between current agents and autonomous physical engineering design." Compare that with benchmarks the authors call saturated: frontier models "score above 90% on popular knowledge benchmarks."
How PLE differs from GDPval and APEX-Agents
The authors name GDPval and APEX-Agents as the closest in spirit, since they test professional work. Their critique is about grading: those use expert pairwise preferences or expert rubrics, while physical engineering needs both a compliance check and an objective measure such as cost. PLE adds executable graders and economic objectives on the co-authored tasks. See our coverage of Claude Opus 5.5 on the Epoch Capabilities Index for how general capability scores relate, and the APEX-Harm safety benchmark for a different APEX evaluation.
How to try it
The repository google-research/ple is Apache-2.0 licensed. Its README describes a competition: you submit a GitHub commit containing your harness, the model it calls and the artifacts/ produced on the 29 public dev tasks. In practice that means committing your harness, running the kit check command until it reports the entry is ready to submit, pushing the repository to a GitHub account you own, registering at ple-bench.com and submitting on the competition site. Each task runs in an isolated container and produces a deliverable, such as a calculation package or technical memo, scored from 0 to 1. Organizers then grade your artifacts, re-run your harness on a sample of dev tasks to check the score reproduces, and run it three times on the 56 hidden test tasks. The leaderboard ranks by best test score.
The quick start uses the Harbor runner, copies a starter submission (the sample is named pi-opus), starts a local model proxy billed to your Google Cloud project, then runs kit check and the dev-set evaluator. Budget accordingly: at the best configuration's $17.18 per task, one pass over the 29 public tasks is roughly $500 in model fees by our arithmetic, and the hidden-test protocol runs three passes over 56 more tasks.
What the task format teaches builders
Even if you never enter the competition, the task design is a useful template for evaluating your own agents on professional work. Four choices stand out:
- Messy inputs. The project folder mixes formats and averages 61 files, so retrieval and file handling are part of the test, not a preamble to it.
- Deliverables, not answers. Agents must output memos, workbooks, schedules and drawings in production formats, with citations back to the source documents. That makes grading about evidence, not fluency.
- Objective plus constraints. Co-authored tasks pair code-compliance checks with a number to optimize, such as 25-year life-cycle cost, which stops an agent from winning by over-designing.
- Tasks that resist saturation. The Creator-Solver loop keeps redesigning each task until a frontier solver no longer maxes it out, then has experts raise the difficulty again.
If you run agents in other regulated fields, such as accounting or healthcare operations, the same pattern of realistic folders, checkable deliverables and an economic objective is worth copying.
Why it matters, and the caveats
- It measures work, not trivia. Where knowledge benchmarks sit above 90%, PLE gives a headroom number that tracks economic value.
- Cost is part of the result. A $17 average per task is far below an engineer's billing rate, but the agent spends an hour of wall-clock time and tens of millions of tokens, and a licensed reviewer still has to check the output.
- Vendor and hosting effects. Results mix model and harness, so "Opus 5.5 in Claude Code" is not the same as Opus 5.5 everywhere. The authors are at Google, and the top score belongs to a rival lab's model, which lends the paper some credibility, but independent reruns are what will settle it.
- Hidden test set. 56 of 85 tasks are private, which protects against contamination but means outsiders cannot audit the full score.
- Not a sign-off. Automated checks and an LLM rubric judge are not a licensed engineer's stamp. The paper's own framing is that errors in these designs "are costly and potentially catastrophic."
What to watch
The leaderboard will show whether the open competition moves the 50.6% ceiling quickly, as similar agent benchmarks have. Watch the breakdown by discipline and the run-to-run variance the paper analyzes, and whether Google reports its own Gemini harness configurations at the top. For practitioners in construction and infrastructure software, the practical question is narrower: which sub-tasks, such as cable sizing or code lookups, already score high enough to automate with human review.
Details are accurate as of October 23, 2026, based on the arXiv paper and the google-research/ple repository. Leaderboard standings will change.