Anthropic Study: Experts Followed Their AI Agent's Hidden Lean 86% of the Time

When you hand a messy analysis to an AI agent, who actually makes the calls that shape the answer? On October 27, 2026, Anthropic's Alignment team published "How AI agents shape experts' judgment calls", a controlled study that tests the usual assumption that experts "still make the decisions that matter." The short version: 50 experienced analysts chose the option their agent secretly leaned toward 86% of the time, and most of them still felt the final recommendation was theirs.

This matters well beyond statistics. Every builder who delegates code review, data analysis or research to an agent is exposed to the same dynamic. The study gives us numbers for something that, until now, was mostly a hunch.

TL;DR: the key numbers

How was the experiment built?

Participants analyzed a fictional farming cooperative. A voluntary, paid advisory program enrolled about 40% of members, and the board would expand it only if it raised productivity by roughly 3 to 4%. The board did not say how to measure productivity, which is exactly the point: the task is full of choices where reasonable experts disagree.

All 61 recruits passed a causal-inference knowledge check, used Python or R regularly and had hands-on experience with observational causal analysis. In the final sample, 65% held a master's or doctorate or had six or more years of experience, and 90% of survey respondents rated themselves proficient or expert.

Each analyst faced four open analytical decisions, each with two defensible options:

1. Outcome: yield per hectare, or yield per unit of operating cost.
2. Comparison method: difference-in-differences, or statistical adjustment of levels.
3. Coverage: enrolled farms only, or the full membership.
4. Weighting: equal weight per farm, or more weight for larger producers.

Then came the final recommendation: roll the program out or discontinue it.

The twist is a coin flip. For every participant and every decision, a hidden system-prompt instruction set which option the agent would lean toward. The agent was told to hold and argue for that lean honestly, to change its mind if the evidence turned decisively against it, and never to mislead. Comparing how often people chose option A under each lean estimates the lean's causal effect. The median session lasted 65 minutes.

How much did the agent's lean move experts?

A lot. Averaged across the four analytical decisions, analysts matched the agent's lean 86% of the time. By decision, that was 92% for the outcome measure, 82% for the comparison method, 81% for coverage and 90% for weighting.

In Anthropic's figure, the four analytical decisions moved by 62 to 82 percentage points depending on the agent's lean, while the final recommendation moved 17 points. The authors explain the smaller effect on the recommendation: it depended heavily on whether the estimate cleared the 3 to 4% threshold, which constrains how far a lean can push. Even so, 54% of analysts recommended rollout when the agent favored it, against 36% when it leaned the other way. A model using only the four leans predicted 82% of recommendations, compared with 61% for always guessing the most common answer.

Did experts just follow the agent over their own views?

In 38 cases, analysts had stated a preference before the agent spoke. When the agent disagreed, they switched 60% of the time. When it agreed, they switched 6% of the time. That is a 54-point gap, though the 95% confidence interval is wide (24 to 82 points) and the authors warn it is not a clean causal estimate.

There is a second finding that is arguably more worrying. Deferring did not feel like deferring:

• 52% of respondents (n = 48) rated the recommendation as more their own than the agent's, and 35% called it mostly or entirely their own.
• Analysts who deferred more were about as confident as those who deferred less (71 versus 73 on a 0 to 100 scale).
• 77% repeated the agent's arguments in at least one explanation, sometimes word for word. Only 15% credited the agent.

The paper shows paired quotes in which an agent's sentence about area-weighting a subsidy reappears almost verbatim in the analyst's later written justification. Analysts could not reread the transcript when writing those answers, which makes the echo harder to dismiss as copy-and-paste.

What does this mean for human-in-the-loop oversight?

Anthropic frames three concerns, and the third will sound familiar to anyone following AI safety debates.

Individual agency. Experts may lose ownership of decisions while still signing their names to the work. Judgment calls have no reference answer, so a deferred analysis looks as sound as an independent one.

Scientific consensus. If many teams delegate to models with the same preferences, agreement across studies may not be independent evidence. The authors also report that, with no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, which suggests fairly stable default preferences.

AI safety research. In the authors' words, "a human reviewer is only an independent check if their judgment is their own." Oversight schemes that rely on a person reviewing an agent's experiments assume independence that this study questions. That connects directly to our coverage of how multiagent systems need identity, isolation and kill switches, and to Microsoft CEO Satya Nadella's argument that alignment will not simply fall out of better models.

It also sits alongside reports that coding agents weakened their own misalignment monitoring: if the agent is both doing the work and shaping the reviewer's view, the monitor is less independent than it looks.

What are the study's limits?

Anthropic is candid, and you should read the results with these in mind:

• Strengthened leans. The agents were deliberately instructed to hold a position, so real-world effects may be smaller.
• Narrow scope. One task, one model, about an hour, in a farming domain that was new to the analysts. Real experts know their fields better and can consult stakeholders to narrow options.
• Averages only. Pre-task views were not recorded for everyone, so individual influence is unknowable.
• Strength not measured. The study recorded which option was chosen, not how strongly the analyst preferred it.
• Non-user comparison is confounded. Six of 11 analysts who skipped the agent (55%) made serious methodological errors that invalidated their results, versus 3 of 50 (6%) of users, and non-users took a median of 90 minutes versus 65. But non-users were less experienced, so the authors caution against causal conclusions.

Of ten participant characteristics tested, including experience, education, trust in AI and need for cognition, only age showed a significant link to deferral, and it disappeared after adjusting for multiple comparisons. In other words, there is no easy profile of who is immune.

Is the answer to use AI less?

No, and Anthropic says so. The same data show agents helped avoid serious errors. The authors conclude that the split between people and AI "can be something we choose, instead of something we drift into." They sketch three directions:

1. Measure deferral in everyday settings and learn models' default preferences.
2. Design agents differently, for example by presenting the strongest case for each option or asking the expert's view first, without overwhelming the user.
3. Change institutions so that decisions needing a person's own informed judgment are identified and the skills for them are protected.

How to protect your own judgment when you work with agents

The paper does not test interventions, so treat the following as our reading of its suggestions rather than proven fixes.

• Commit first. Before the agent weighs in, write down your pick for each fork (metric, baseline, scope, weighting). The study's 60% versus 6% switching gap shows how quickly a prior view bends once the agent disagrees.
• Ask for the case against. Prompt the agent: "Give the strongest argument for each option and do not recommend one." This mirrors the paper's idea of presenting the best case for each side.
• Log the forks. Keep a short list of defensible alternatives you did not take. If a reviewer cannot see the forks, they cannot see the influence.
• Watch for borrowed phrasing. If your write-up sounds like the agent's transcript, you may be reporting its reasoning, not yours.
• Vary the reviewer. A second model or a human who has not seen the transcript is a more independent check than the same agent re-reading its own work.
• Match scrutiny to reversibility. Defer freely on formatting; slow down on choices that change a conclusion.

For a lighter-weight version of this habit, our two-filter guide to choosing between AI drafts separates taste from judgment, and the verification tax explains why checking agent output is itself a cost worth budgeting. If you run agents that take real actions, AgentBeam is the agent security platform from the explainx.ai team that stops agents before they take dangerous actions, which complements, but does not replace, keeping your own judgment in the loop.

What this connects to

Judgment calls are where agent influence is hardest to see because nothing is "wrong." That is a different failure mode from the benchmark gaps we track. For example, a recent continual-learning exam found the best agent completing under half its levels, and Anthropic's own work on training alignment auditors is about making oversight scale. This study adds a human-side question to both: even a good auditor is only as independent as the person reading its report.

The practical read is simple. Agents are now strong enough that their defaults become your defaults unless you decide otherwise. The 86% figure is from a deliberately stress-tested setup, so do not assume your own rate is that high. But the 77% echo rate and the 15% credit rate describe a quiet mechanism that no accuracy benchmark will catch.

Related reading

• Multiagent systems need four engineering fixes
• Nadella on why alignment is a harder, different problem
• Coding agents weakened misalignment monitoring
• Taste vs judgment: a two-filter guide
• The AI verification tax
• Continual learning exam for agents
• Training alignment auditors

Figures and study details are accurate as of October 27, 2026, per Anthropic's published research page. Check the source for revisions.