Dartmouth's Phosphor Study: What an AI Tutor With a 0.71–1.30 SD Effect Actually Did

A new paper from Dartmouth, presented at the iTextbooks '26 workshop in Seoul on June 28, 2026, reports numbers that education researchers rarely see together: 90.2% voluntary adoption of an optional, ungraded study platform, and a 0.71–1.30 standard deviation association between full engagement and final exam performance. The platform, Phosphor, embeds LLM-graded quizzes directly into course readings for Introductory Statistics (151 students, three sections, Spring 2026).

The headline invites a "AI tutors work!" reading. The paper is more interesting than that — and the Hacker News discussion did a genuinely good job separating what the study shows from what it doesn't. Since teaching AI to 350,000+ students is literally our business, here's our close read — and what it means if you're using AI to learn, a theme we cover from kids' roadmaps to adult upskilling.

Update — July 19, 2026: New product coverage — Academa.ai LLM-integrated multivariable calculus puts contextual lecture chat on reviewed animated content; compare to Phosphor's finding that unscoped chat saw almost no use.

TL;DR: What the Study Does and Doesn't Show

The Setup: Students Don't Read the Textbook

The motivating problem is blunt. Reading compliance in the course was estimated at 10–15%; student descriptions ranged from "literally no one does that" to "is this being recorded?" Meanwhile, a 2026 HEPI survey found 94% of university students use generative AI on assessed work, up from 53% two years earlier — and the best randomized evidence, Bastani et al. (2025), shows that unrestricted GPT-4 access harmed subsequent performance by 17% once the tool was removed. Students use raw chatbots as a crutch.

Phosphor's bet: don't bolt AI onto studying as a chat window — build it into the reading as assessment. Each lesson carries a quiz drawing from a 15–20 question bank, 40% constructed-response (CRQ) and 60% multiple choice. The CRQs are graded by Claude Sonnet 4.6 against instructor-written, question-specific rubrics, with unlimited retries and a 75% pass threshold. Cumulative module reviews (10 questions, 90% threshold) sit on top.

The Accidental Experiment: Written Answers Beat Multiple Choice

The most useful finding came from an unplanned design change. Students complained the CRQ auto-grader was rigid, so Module 2 went multiple-choice only. The result:

Engagement in Module 2 was comparable or higher — students liked the easier quizzes — but the learning signal vanished. When the format demanded active generation, engagement translated into exam points; when it didn't, it was busywork. This matches the classic testing-effect literature (Kang et al. 2007: short-answer quizzing beat multiple choice, d = 0.41) and is the practical takeaway for anyone building or choosing study tools — including the house rules we recommend for AI and homework: make the human produce the answer, let the AI grade it.

The second strong lever was cumulative module reviews: students who passed all three scored 7.1 points higher on the final (d = 0.66), and two-thirds of review attempts involved spaced retries a day or more apart — interleaved, spaced retrieval practice doing exactly what the cognitive-science literature says it should.

And the chatbot? The RAG assistant drew 72 queries total, with only 14 students asking more than one. Khan Academy reports a similar pattern — only 15% of users regularly engage its supplementary chatbot. The AI that mattered was invisible: rubric-based grading at scale, not conversation.

The Skeptic's Case: What Hacker News Got Right

The HN thread's top comments were unusually sharp, and the criticisms are fair:

1. Selection bias is the elephant. Only ~11% of students hit "full engagement." Students who voluntarily complete every quiz are the motivated ones. The authors control for midterm scores — which halves the estimate to 0.71 SD — but a midterm control is not a randomized trial. One former TA recounted the classic version of this trap: students who self-organized study groups dramatically outperformed, until everyone was assigned to study groups and the effect vanished.
2. The design was shaped by its own results. Quiz formats changed mid-course in response to engagement and exam data, which makes the module comparison partly circular. The authors acknowledge this; the statistics can't fully rescue it.
3. "Engaged students score better" is the boring explanation. The 0.71 SD lower bound may still be over-generous, since the cumulative final re-tests midterm content.
4. It's not a tutor. The accurate description is a practice-quiz platform with an AI autograder. That's less romantic than Bloom's vision of a tutor for every student — the two-sigma problem — but it's also more scalable and cheaper.

The authors, to their credit, flag nearly all of this in the limitations section and plan randomized format assignment next. This is a promising pilot with honest error bars, not a proven intervention.

Reading the Statistics Like a Reviewer, Not a Headline Writer

The paper's own numbers reward a closer look than "0.71 to 1.30 SD" gets in a headline, because the gap between those two figures is the finding.

The 1.30 SD number comes from an engagement-only Tobit model: compare students who did everything (24/24 lessons, 3/3 reviews) against students who did nothing. That comparison is dominated by who chooses to do everything — the most conscientious, most prepared, most likely-to-succeed-anyway students in the class. It's the same selection effect a former teaching assistant described in the Hacker News thread: self-organized study groups looked like magic until everyone was assigned to one and the effect disappeared.

The 0.71 SD number adds midterm scores as a control, which absorbs most of that selection bias — students who were already scoring well on midterms don't get extra credit in the model for also being the type of student who finishes every quiz. But the authors flag their own overcorrection risk: because the cumulative final exam re-tests material the midterms already assessed, controlling for midterms partially controls for the very thing Phosphor may have caused earlier in the term. Statistically, this creates a band, not a point estimate — the true causal effect plausibly sits somewhere in [0.71, 1.30], and neither endpoint is more "correct" than the other; they're two different assumptions about how much of midterm performance to attribute to the tool itself.

For comparison, published effect sizes for interventions education researchers already trust:

Even at the conservative 0.71 SD end, that places an optional, ungraded, zero-marginal-cost tool in the same range as effective formative-assessment programs that normally require dedicated instructor time to run — which is the actual headline, once you strip out the hype.

What This Means If You're Learning (or Teaching) With AI

Our takeaways, consistent with what we see across our own courses and workshops:

• Use AI as an examiner, not an oracle. The gains came from students writing answers and getting rubric-graded feedback. Prompting a chatbot to explain things to you is the 17%-worse-when-removed failure mode.
• Prefer tools that force generation. Multiple-choice drilling feels productive and measured ~zero here. This applies to picking AI learning platforms as much as to course design.
• LLM rubric grading is the quietly big capability. Grading free-text answers at scale used to be the binding constraint on formative assessment. Claude-graded CRQs made it free. Expect this pattern — AI embedded in the workflow rather than a chat sidebar — to win broadly, as it did here.
• For educators: the 90% adoption number is the one to steal. Students overwhelmingly chose interactive, quizzed readings over a textbook. The format was the intervention.

Related on explainx.ai

• Estonia’s AI Leap: Teach Students to Think With AI, Not Instead of It
• AI Homework House Rules: Making AI Help, Not Replace, Learning
• How to Learn AI: Beginner to Expert Roadmap
• Top 10 Free AI Learning Platforms
• AI Curriculum for College Students
• Age-by-Age AI Roadmap for Kids 5–14
• AI and Parenting: A Practical ChatGPT Guide
• AI Benchmarks: Complete Guide
• Academa.ai: LLM-Integrated Multivariable Calculus

Source: Jonah Bard, "Balancing Efficacy and Engagement in Interactive Texts," iTextbooks '26 (Seventh Workshop on Intelligent Textbooks), June 28, 2026. See also Bastani et al., PNAS 2025.

Study figures reflect the workshop paper as of July 6, 2026. Phosphor (formerly Spongium) is an early pilot; a randomized follow-up is planned.