explainx / blog
A Dartmouth statistics course got 90% voluntary adoption of an AI-quizzing platform and a 0.71–1.30 SD exam gain. What worked, what's overstated, and what HN got right.
Jun 29, 2026
A 2024 paper from Sun Yat-sen University and Alibaba introduced Proxy-KD — using a white-box proxy to distill black-box teachers like GPT-4. Hacker News resurfaced it amid Anthropic's Fable 5 distillation allegations. Here's what the method does and why policymakers care.
Jun 23, 2026
A 0.22B model matching an 11.9B industrial giant on inpainting benchmarks is not a rounding error — it is a structural claim about what task-specific specialist models can do. Moebius achieves this via a novel attention block and latent-space distillation from PixelHacker. 26ms per step. Consumer hardware. Worth understanding.
Jun 22, 2026
Shortlist of 20 explainx.ai prompt generators for Research, spanning audio, text, video modalities and 10 high-level categories.
A new paper from Dartmouth, presented at the iTextbooks '26 workshop in Seoul on June 28, 2026, reports numbers that education researchers rarely see together: 90.2% voluntary adoption of an optional, ungraded study platform, and a 0.71–1.30 standard deviation association between full engagement and final exam performance. The platform, Phosphor, embeds LLM-graded quizzes directly into course readings for Introductory Statistics (151 students, three sections, Spring 2026).
The headline invites a "AI tutors work!" reading. The paper is more interesting than that — and the Hacker News discussion did a genuinely good job separating what the study shows from what it doesn't. Since teaching AI to 350,000+ students is literally our business, here's our close read — and what it means if you're using AI to learn, a theme we cover from kids' roadmaps to adult upskilling.
| Question | Answer |
|---|---|
| What is Phosphor? | A web platform embedding quizzes into course readings; written answers graded by Claude Sonnet 4.6 against instructor rubrics |
| Is it a chatbot tutor? | No — its RAG chat assistant got only 72 queries all term; the gains came from AI-graded practice |
| Headline effect | 0.71 SD (controlling for midterms) to 1.30 SD (unadjusted) for full vs. zero engagement on the final exam |
| Adoption | 90.2% used it at least once, vs. a 10–15% reading-compliance baseline |
| What actually worked | Written-response questions and cumulative module reviews; multiple-choice-only quizzes showed no dosage effect |
| Randomized trial? | No — observational, single institution, real selection-bias concerns |
The motivating problem is blunt. Reading compliance in the course was estimated at 10–15%; student descriptions ranged from "literally no one does that" to "is this being recorded?" Meanwhile, a 2026 HEPI survey found 94% of university students use generative AI on assessed work, up from 53% two years earlier — and the best randomized evidence, Bastani et al. (2025), shows that unrestricted GPT-4 access harmed subsequent performance by 17% once the tool was removed. Students use raw chatbots as a crutch.
Phosphor's bet: don't bolt AI onto studying as a chat window — build it into the reading as assessment. Each lesson carries a quiz drawing from a 15–20 question bank, 40% constructed-response (CRQ) and 60% multiple choice. The CRQs are graded by Claude Sonnet 4.6 against instructor-written, question-specific rubrics, with unlimited retries and a 75% pass threshold. Cumulative module reviews (10 questions, 90% threshold) sit on top.
The most useful finding came from an unplanned design change. Students complained the CRQ auto-grader was rigid, so Module 2 went multiple-choice only. The result:
| Module | Quiz format | Dosage → exam relationship |
|---|---|---|
| Module 1 | MCQ + written (CRQ) | +1.6 points per lesson completed (R² = 0.123) |
| Module 2 | MCQ only | None among users (R² = 0.001) |
| Module 3 | CRQ restored | Cumulative signal returns on the final (R² = 0.091) |
Engagement in Module 2 was comparable or higher — students liked the easier quizzes — but the learning signal vanished. When the format demanded active generation, engagement translated into exam points; when it didn't, it was busywork. This matches the classic testing-effect literature (Kang et al. 2007: short-answer quizzing beat multiple choice, d = 0.41) and is the practical takeaway for anyone building or choosing study tools — including the house rules we recommend for AI and homework: make the human produce the answer, let the AI grade it.
The second strong lever was cumulative module reviews: students who passed all three scored 7.1 points higher on the final (d = 0.66), and two-thirds of review attempts involved spaced retries a day or more apart — interleaved, spaced retrieval practice doing exactly what the cognitive-science literature says it should.
And the chatbot? The RAG assistant drew 72 queries total, with only 14 students asking more than one. Khan Academy reports a similar pattern — only 15% of users regularly engage its supplementary chatbot. The AI that mattered was invisible: rubric-based grading at scale, not conversation.
The HN thread's top comments were unusually sharp, and the criticisms are fair:
The authors, to their credit, flag nearly all of this in the limitations section and plan randomized format assignment next. This is a promising pilot with honest error bars, not a proven intervention.
The paper's own numbers reward a closer look than "0.71 to 1.30 SD" gets in a headline, because the gap between those two figures is the finding.
The 1.30 SD number comes from an engagement-only Tobit model: compare students who did everything (24/24 lessons, 3/3 reviews) against students who did nothing. That comparison is dominated by who chooses to do everything — the most conscientious, most prepared, most likely-to-succeed-anyway students in the class. It's the same selection effect a former teaching assistant described in the Hacker News thread: self-organized study groups looked like magic until everyone was assigned to one and the effect disappeared.
The 0.71 SD number adds midterm scores as a control, which absorbs most of that selection bias — students who were already scoring well on midterms don't get extra credit in the model for also being the type of student who finishes every quiz. But the authors flag their own overcorrection risk: because the cumulative final exam re-tests material the midterms already assessed, controlling for midterms partially controls for the very thing Phosphor may have caused earlier in the term. Statistically, this creates a band, not a point estimate — the true causal effect plausibly sits somewhere in [0.71, 1.30], and neither endpoint is more "correct" than the other; they're two different assumptions about how much of midterm performance to attribute to the tool itself.
For comparison, published effect sizes for interventions education researchers already trust:
| Intervention | Typical effect size (Cohen's d) |
|---|---|
| Reducing class size (Tennessee STAR) | ~0.15–0.20 |
| Formative assessment / feedback (Black & Wiliam) | ~0.4–0.7 |
| Bloom's one-on-one tutoring (1984) | ~2.0 |
| Short-answer vs. multiple-choice quizzing (Kang et al. 2007) | 0.41 |
| Phosphor, full engagement (this study) | 0.71–1.30 |
Even at the conservative 0.71 SD end, that places an optional, ungraded, zero-marginal-cost tool in the same range as effective formative-assessment programs that normally require dedicated instructor time to run — which is the actual headline, once you strip out the hype.
Our takeaways, consistent with what we see across our own courses and workshops:
Source: Jonah Bard, "Balancing Efficacy and Engagement in Interactive Texts," iTextbooks '26 (Seventh Workshop on Intelligent Textbooks), June 28, 2026. See also Bastani et al., PNAS 2025.
Study figures reflect the workshop paper as of July 6, 2026. Phosphor (formerly Spongium) is an early pilot; a randomized follow-up is planned.
