Dartmouth's Phosphor study — 90.2% voluntary adoption of an AI-graded quiz platform, a 0.71-1.30 SD exam gain — reads like an outlier when it lands in your feed as a single headline. It isn't one. Author Jonah Bard presented it at the iTextbooks workshop, part of a research series that's been quietly accumulating evidence since 2024 on exactly this question: what happens to learning when LLMs get embedded into the reading itself, not bolted on as a chatbot.
We pulled together the 20 studies most directly related to Phosphor's design choices — LLM-graded formative practice, embedded quizzing, and the fight over whether generative AI helps or crutches learning — and mapped where they agree, where they don't, and what the disagreement is actually about. Since teaching AI to 350,000+ students is our business, this is the literature we check our own product decisions against, including Melo, our quiz-and-rubric learning copilot.
TL;DR: What the Research Cluster Shows
| Question | Answer |
|---|---|
| Nearest research to Phosphor | VitalSource's doer-effect replication, Google's Learn Your Way, and the iTextbooks'25 intelligent-text papers — same mechanism, same vocabulary |
| Best-matched venues | iTextbooks workshop (AIED) and ACM Learning@Scale — both publish doer-effect, AI-grading, and intelligent-text work |
| What reliably helps | AI as formative practice + grading: doer effect ~6x more effective than reading alone; retrieval practice raised quiz accuracy 73% to 89% |
| What reliably hurts | AI as an answer-crutch: 17% score drop once access removed (Bastani); d=0.68 retention loss (Barcaui) |
| LLM grading accuracy | QWK 0.585-0.94 depending on task structure and rubric quality — approaching but not matching human inter-rater reliability |
| The dividing line | Whether the student still has to generate the answer, or the model does |
Tier 1: The Papers Phosphor Is Directly Building On
VitalSource's doer-effect replication is the closest methodological relative. Van Campenhout, Autry, Clark & Johnson analyzed 15.2 million data events from 18,546 students across seven courses at Learning@Scale 2025 to test whether AI-generated formative questions — not human-authored ones — still trigger the doer effect: the finding that doing practice while reading beats reading alone, "an average of six times more effective for learning," and that the relationship is causal. A companion paper describes CoachMe, VitalSource's AI-generated practice feature inside Bookshelf.
Google's Learn Your Way (arXiv:2509.13348, September 2025) is the most prominent industry entrant. The LearnLM-built system personalizes textbook text by reading level and interest, renders it as narrated slides, audio, mind maps, and timelines, and — the part relevant here — dynamically generates embedded quizzes and formative assessments. Its 60-student RCT found 78% vs 67% on a retention test taken 3-5 days later (p=0.03) and 100% vs 70% of students reporting comfort with the material, against a standard digital reader.
Crossley et al., iTextbooks'25 — the exact predecessor workshop to Bard's iTextbooks'26 presentation — reports the iTELL platform's propensity-score-matched analysis: intelligent-text users showed ~5% pre-to-post-test gains, attributed to interactivity and engagement rather than content alone.
A cautionary counterpoint: Fecko et al.'s four-year comparative study of a GenAI interactive textbook in an electrotechnics course (Computers and Education: AI, 2026; N=736) found a significant association with mid-term (formative) success (χ²=24.38, p<0.001) but no significant effect on final (summative) assessment (χ²=0.61, p=0.433) — a reminder that formative gains don't automatically transfer to high-stakes outcomes, and that this is a quasi-experimental design, not a randomized one.
Tier 2: Can LLMs Actually Grade Written Answers?
Phosphor's core mechanism — Claude Sonnet grading constructed-response questions against instructor rubrics — depends on a specific capability the field has been benchmarking hard since 2025.
| Study | Finding |
|---|---|
| Mello et al., LAK'25 | GPT-4 + prompt engineering reached QWK ~0.94, rivaling fine-tuned BERT; outperformed SVM (0.80) |
| AutoSCORE (AAAI'26) | Adding an explicit rubric-recognition step before scoring fixes single-prompt inconsistency |
| Technology, Knowledge and Learning, 2025 | Human-LLM alignment gap shrinks substantially when high-quality rubrics are supplied |
| Emirtekin et al., JCAL 2026 | GPT-4o, Gemini 2.0 Flash, DeepSeek V3, LLaMA 3.3 reached moderate agreement (QWK 0.585-0.640) on Bloom's-taxonomy rubrics — below human inter-rater reliability (ICC 0.667-0.800) |
| Flodén, 2025 | ~70% of ChatGPT's Master's-exam scores fell within 10% of human scores |
The spread between Mello's 0.94 and Emirtekin's 0.585 isn't noise — it's a design variable. Task structure and rubric quality drive the gap, which is exactly why Phosphor's approach (question-specific, instructor-written rubrics with unlimited retries) sits on the favorable end rather than the sobering one. It's also why the Dartmouth deep-dive found that when the course briefly switched to multiple-choice-only quizzes, the learning signal vanished entirely (R² dropped to 0.001) — automated grading only pays off if the question format still demands the student generate something worth grading.
Tier 3: The Efficacy-vs-Crutch Fight
This is the tension in Bard's paper title, and it splits the literature into two poles with real evidence on each side.
The positive pole: scaffolded AI helps
- Kestin et al., Scientific Reports 2025 — 194 Harvard students, RCT crossover design. A pedagogically scaffolded AI tutor that held answer keys and released hints one step at a time produced 0.7-1.3 SD gains over the instructor's own active-learning class, students learning more than twice as much in less time.
- World Bank, Nigeria (Policy Research Working Paper 11125) — ~800 students, six-week RCT with Copilot as an after-school English tutor. 0.23-0.31 SD gains, equivalent to 1.5-2 years of business-as-usual schooling.
- An et al., arXiv:2507.05629 — LLM-generated retrieval-practice MCQs raised weekly quiz accuracy from 73% to 89% in data science courses.
- Tutor CoPilot (Stanford) — 700+ tutors, ~1,000 K-12 students. Students whose tutors used the AI copilot were 4 points more likely to master math topics, rising to +9 points for lower-rated tutors.
The negative pole: unguarded AI harms
- Bastani et al., PNAS 2025 — the canonical counterweight. Nearly 1,000 Turkish high schoolers with GPT-4 access scored up sharply during practice (+48% to +127%), but once access was removed, unrestricted-GPT students scored 17% worse than students who'd never had access. Teacher-designed hint guardrails largely mitigated the harm.
- Barcaui, Social Sciences & Humanities Open 2025 — 120 undergraduates, RCT. On a surprise retention test 45 days later, ChatGPT users scored 57.5% vs 68.5% for traditional studiers (d=0.68).
- Kosmyna et al. (MIT Media Lab), arXiv:2506.08872 — EEG study of 54 participants writing essays under LLM, search, or brain-only conditions found LLM users showed the weakest neural connectivity and lower recall. It's an unreviewed preprint on essay writing, not formative assessment, but it's the most-cited "cognitive debt" evidence going.
The pattern across both poles is consistent, not contradictory: AI that makes the student produce an answer before revealing anything helps; AI that produces the answer for the student hurts. Bastani's own data shows this inside a single study — the guardrailed GPT Tutor avoided most of the harm the unrestricted GPT Base version caused. It's the same design principle explainx.ai leans on in AI homework house rules and the one that shows up again in the 26,811-student CEPR study: homework scores rose 18% with AI help, then exam scores fell 20% within six months — concentrated in students who used AI to skip practice rather than to check it.
What This Means If You're Building an AI Learning Tool
- Embed formative practice in content, don't bolt on a chatbot. The doer-effect literature is unambiguous: doing beats reading by roughly 6x, and the effect survives AI-generated questions. A sidebar chat window doesn't reproduce this — Phosphor's own RAG assistant drew only 72 queries all term while the quiz engine drove the entire effect.
- Require generation before grading, not before answering. Constructed-response formats plus rubric grading is the throughline from VitalSource to Phosphor to the retrieval-practice literature. Multiple-choice-only formats measured ~zero dosage effect in the Dartmouth data.
- Guardrail the chat surface if you have one. Bastani's split-condition data is the cleanest evidence available: identical model, different scaffolding, opposite outcomes. Hint-first, answer-last design is the actionable lesson, not "avoid chatbots entirely."
- Validate your grading pipeline against a QWK benchmark before trusting it at scale. The 0.585-0.94 spread in the literature means "we used GPT-4 to grade" is not a claim by itself — report agreement against human raters, and treat sub-0.6 QWK grading as formative-only, not high-stakes.
- Watch for the formative/summative gap. Fecko et al.'s null result on final exams is a live risk, not a corner case — measure delayed and unassisted performance, not just in-tool completion.
This is the same design logic behind Melo's Quiz, Practice, and Explain Back modes on explainx.ai: the model grades against a rubric after the learner produces an answer, rather than answering first. It's also why we keep flagging the same guardrails-matter pattern across Estonia's national AI curriculum and our own AI curriculum guidance — the research cluster around Phosphor isn't an isolated finding, it's converging evidence.
Caveats Worth Keeping in Mind
- Recency vs. peer review. Learn Your Way, the MIT cognitive-debt study, and the retrieval-practice paper are all preprints. The strongest peer-reviewed causal evidence here is Kestin (Scientific Reports), Bastani (PNAS), and the World Bank working paper.
- Effect sizes aren't directly comparable. Kestin's 0.7-1.3 SD used a tightly scaffolded custom tutor against active learning; the World Bank's 0.23-0.31 SD used teacher-facilitated Copilot. Scaffolding design, not "AI" as a category, drives the outcome — the same lesson the Dartmouth study's statistics reward on close reading.
- Grading accuracy figures don't transfer freely across language, domain, or rubric. A grading-accuracy figure above 95% from Spanish short-answer grading or a QWK of 0.94 from one task structure shouldn't be assumed to hold for a different subject without separate validation.
Related on explainx.ai
- Dartmouth's Phosphor Study: What a 0.71-1.30 SD Effect Actually Did
- The Generative AI Learning Penalty: 26,811-Student Study
- Introducing Melo, the explainx.ai Learning Copilot
- AI Homework House Rules: Making AI Help, Not Replace, Learning
- Estonia's AI Leap: Teach Students to Think With AI
- What Should Schools Teach in the AI Era?
- AI Curriculum for High School Students
- How to Learn AI: Beginner to Expert Roadmap
Sources: Jonah Bard, "Balancing Efficacy and Engagement in Interactive Texts," iTextbooks '26. Van Campenhout et al., L@S'25 (DOI 10.1145/3698205.3729545). Google LearnLM, "Towards an AI-Augmented Textbook," arXiv:2509.13348. Mello et al., LAK'25. Kestin et al., Scientific Reports 15:17458. Bastani et al., PNAS 122(26):e2422633122. Barcaui, SSHO 12:102287. Kosmyna et al., arXiv:2506.08872.
Figures reflect the cited papers and preprints as of publication date; several remain unpublished preprints and may be revised on peer review.