Dayjob: Tax: AI Agents Pass Only About 1 in 10 Professional Tax Assignments

Can an AI agent do a tax professional's job? Surge AI's new benchmark, Dayjob: Tax, answers with a number: the best score on its leaderboard is 10.8%. The test gives agents 50 assignments built from realistic client folders, grades them at a strict 95% rubric threshold, and finds that even the top model rarely clears it. Surge's own summary line: "Agents haven't yet passed the audit."

The post reached the Hacker News front page, where commenters mixed enthusiasm about catching accountant mistakes with doubts about handing financial data to AI companies. Below: how the benchmark works, the two failure cases Surge highlights, what the discussion added, and how to read a 10.8% without over- or under-reacting.

TL;DR: Dayjob: Tax at a glance

How is Dayjob: Tax built?

The eight practice areas are individual and family tax, small-business tax, partnerships, property and investments, estates, gifts and trusts, international tax, corporate transactions, and tax review and disputes. Deliverables include tax returns, workpapers, technical reviews and client advice.

Each assignment is a folder, not a prompt. The 1,789 files include 1,487 PDFs, 117 Excel workbooks, 38 Word documents, plus emails, images, CSVs, JSON and text. Examples include a handwritten receipt for resale merchandise and a field dispatch log showing crews working in five states. Tax experts built the tasks and reviewed them in three layers for correctness, completeness and whether the task is solvable.

That design matters because most benchmarks hand a model a clean question. Real tax work starts with messy evidence and an unclear question, which is what this benchmark tries to reproduce. For how other benchmarks handle long, file-heavy work, see our coverage of the continual learning exam for agents.

Why the 95% bar?

A task passes only if the agent meets at least 95% of its rubric criteria. The criteria cover calculations, supporting evidence, consequential issues and the completed deliverable. Pass@1 means a single attempt counts; there is no best-of-five.

This is stricter than the average-score style used on many leaderboards. In tax, one missed election or deadline can turn a mostly right answer into a penalty, so a professional would say a return that is "90% right" has failed. A threshold metric reflects that, but it also makes scores look small. A model that gets 85% of every rubric would score near zero here yet might still be useful as a first-draft tool.

The two failures Surge highlights

A missed deadline. In a condo-sale case, the agent correctly judged that both closing dates met the residence requirement but missed that eligibility for the exclusion would lapse around August 2027. Surge says one model never caught this across five attempts, while another caught it in all five. Surge's lesson: "A tax adviser should recognize consequential issues even when the client doesn't know to ask about them."

Unverified marital status. In a separating couple's retirement-account case, Opus assumed they were unmarried in all five attempts, even though records suggested a possible valid marriage from earlier residence in Texas or a ceremony in Colorado. The other model flagged the need to verify marital status in four of five attempts. (The page refers to that model as Astra without further identification.)

Both failures are about what the agent did not ask, not arithmetic. They are the kind of error that a calculator-style check would never find and that experienced reviewers catch by habit. We covered a similar pattern in Anthropic's study of agents' hidden judgment calls.

What developers on Hacker News said

The discussion had 46 points and 31 comments when we read it. Points below are the commenters' own reports, not verified facts.

• One commenter, holdingunsteady, said an accountant's mis-filing cost them about $20,000 and saw an opportunity for models to save people thousands a year.
• waiwai933, a US citizen living abroad who prepares their own return, reported feeding their draft and paperwork into Claude and said it found errors such as a wrong exchange rate.
• matkoniecz objected to sharing full financial information with LLM companies.
• pawelduda was surprised Sonnet 5.5 beat Opus and Fable, noting the same pattern in other benchmarks.
• andriy_koval asked for a human baseline to compare with the roughly 10% result.
• bitbasher questioned how much to trust a report from a company on a .ai domain; a reasonable reminder to read the methodology rather than the source's reputation.

The human baseline question is the sharpest. The page text we read does not report how human tax professionals score against the same rubric, so the 10.8% cannot be read as "AI versus humans".

Why tax is a good stress test for agents

Tax work combines several things that current agents find hard at once. The evidence is scattered across formats: scanned receipts, spreadsheets, emails and prior-year returns. The rules are detailed and change by year and jurisdiction. The answer is not a single value but a package of calculations, supporting workpapers and advice. And the cost of an omission is delayed: a missed election may not show up until an audit years later.

That makes the benchmark a useful proxy for other professional work, such as legal review, accounting close or compliance, where the real test is noticing what is missing. It is also a reminder that benchmark design shapes the headline. Surge chose a pass/fail bar at 95%; a different author might have reported average rubric coverage and produced a number in the 60s or 70s. Neither is wrong, but they answer different questions: "how much of the job did it do?" versus "could a professional sign this without redoing it?"

For a related look at measurement choices, our Artificial Analysis Intelligence Index v5 write-up explains how aggregated scores can hide where models fail.

How to read a benchmark like this before quoting it

1. Find the metric. Pass@1 at 95% is not average accuracy. Say so when you repeat the number.
2. Find the human baseline. If the source does not give one, say that the comparison is missing.
3. Check who built the tasks. Here, tax experts with a three-layer review is a good sign; it is still a vendor's own benchmark.
4. Check how many tasks. Fifty assignments is enough to rank models roughly but small enough that a few tasks can move a score by several points, so the gap between 10.8% and 9.2% should not be over-read.
5. Look for the failure analysis. The two case studies are more informative than the leaderboard.

What the scores do and do not tell you

• They do tell you that complex, multi-document tax work is far from solved at an audit-ready standard, and that the strongest agents still miss non-obvious issues.
• They do not tell you how often a model's draft would be useful to a human reviewer, how it performs on a simple W-2 return, or how it compares with a junior preparer.
• They suggest the cost-performance gap between Sonnet 5.5 and Opus 5.5 is narrow here: per Surge, Sonnet 5.5 and Opus 5.5 sit at the high-performance end of the measured cost-performance frontier. See our Sonnet 5.5 versus Opus 5.5 comparison for the price side.

What this means for builders of finance agents

If you are building an agent for accounting or tax workflows, the benchmark's lesson is architectural. Add an explicit step where the agent lists assumptions and unresolved facts before producing a deliverable, since both highlighted failures came from silent assumptions. Add a second pass that checks deadlines and elections against the dates in the file. And route anything the agent flags as uncertain to a human instead of letting it guess. None of these need a better model; they need a better workflow around the one you have.

What to do if you use AI on your taxes

1. Use it as a reviewer, not a preparer. The HN reports that were positive involved checking a human or self-prepared return, which is what the benchmark's results would also support.
2. Ask for open questions. Prompt the model to list facts it assumed and facts it needs, since the benchmark's failures were unasked questions.
3. Protect your data. Redact Social Security numbers and account details where possible, and read your provider's data-use terms.
4. Check the rules at the source: the IRS website publishes the forms and instructions a model's answer should match.
5. Verify with a professional for anything involving property sales, foreign income, marriage or separation, or estates.
6. Companies building agents should treat a 10% pass rate as a prompt to build review steps, not as proof agents are useless.

Related reading

• Continual learning exam for agents
• Anthropic study: agent judgment calls
• Claude Opus 5.5 vs Sonnet 5.5
• Claude Sonnet 5.5 launch and building guide
• Claude Fable 5.1 and Mythos 5.1 launch
• Artificial Analysis Intelligence Index v5
• Official: Surge AI's Dayjob: Tax post

Details are accurate as of October 28, 2026 and come from Surge's page; this is not tax advice.