A dermatologist has put two of the most capable AI models available in October 2026 against a convolutional neural network (CNN) first published in 2015, and the old network won. In a test published on Lynch Lab, Dr Magnus Lynch showed 200 biopsy-proven skin lesions to GPT-6 Astra and Claude Fable 5.1 and to a fine-tuned Inception v3. The CNN named the right diagnosis for 125 lesions (62%). The two frontier models named it for 84 and 85 (42% each).
The result is not a verdict on medical AI, and the Hacker News thread was quick to say so. But it is a clean, reproducible illustration of a point builders keep relearning: a small model trained on the right data for one narrow job can beat a frontier generalist that has never been taught that job. This post walks through the setup, the numbers, the caveats the author states himself, and what to take from it.
Quick answers
| Question | Answer |
|---|---|
| Who ran it? | Dr Magnus Lynch, consultant dermatologist and Mohs surgeon at Guy's Hospital and honorary reader at King's College London |
| What was tested? | GPT-6 Astra, Claude Fable 5.1, and a fine-tuned Inception v3 CNN |
| Test set | 200 random, histology-confirmed lesions from the public MILK10k dataset |
| Headline result | CNN 125/200 (62%); GPT-6 Astra 84/200 (42%); Fable 5.1 85/200 (42%) |
| Biggest caveat | Small sample (30 melanomas), one run per LLM, every lesion was biopsied |
| Can I try the CNN? | Yes, the author links an in-browser demo that runs locally via WebGPU |
| Is this clinical evidence? | No. It is a single-author blog test, not peer-reviewed |
How was the test built?
The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025, in which each lesion is photographed twice: a clinical close-up and a dermoscopic image taken through the illuminated magnifying lens dermatologists use. Lynch drew 200 lesions at random, all confirmed by histopathology:
- 100 lesions that should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas (including Bowen's disease) and 20 actinic keratoses.
- 100 lesions safe to discharge: 50 benign naevi (moles) and 50 seborrhoeic keratoses.
Every lesion was biopsied, including the benign ones. That matters: a mole that someone decided to cut out is a mole that worried a clinician, so the benign cases are harder than a random sample of moles would be.
The two LLMs received both images and nothing else. They were asked, as a clinician might use them, for the most likely of six diagnoses (or "unsure"), a confidence, a second choice, and whether the lesion could be safely discharged with no treatment, biopsy or follow-up. Both got the same prompt at default settings, and neither refused a case.
What is the CNN, and why was it deliberately weak?
The comparison network is Inception v3, an architecture Google introduced in a 2015 paper. Lynch pretrained it on everyday photographs, then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive using a single H100 GPU. Training took about twenty minutes.
Two details protect the comparison. First, the 200 test lesions and every other photograph of the same lesion or patient were removed from the CNN's data, including its validation set, so it had never seen them. Second, they were not used to choose the model or set its threshold. The CNN scored both images of each lesion, weighted toward the dermoscopic view, and chose from the same six diagnoses.
Lynch is explicit that this is a baseline, not the state of the art. Modern skin cancer AI uses ensembles of image transformers trained on more data, and the regulated products built on them perform considerably better. The point of the experiment is that even a deliberately old, cheap model beat the generalists.
What did the numbers show?
| Metric | GPT-6 Astra | Claude Fable 5.1 | CNN (Inception v3) |
|---|---|---|---|
| Correct diagnosis (of 200) | 84 (42%) | 85 (42%) | 125 (62%) |
| Correct in first or second choice | 134 | 138 | 173 |
| Melanoma as first choice (of 30) | 8 | 11 | 13 |
| Unsafe lesions kept (of 100) | 97 | 96 | 77 at preset threshold; 98 or 99 when matched |
| Safe lesions discharged (of 100) | 32 | 22 | 76 at preset threshold |
| Melanomas discharged (of 30) | 3 | 4 | 18 at preset threshold; 2 or 1 when matched |
Source: Lynch Lab. Figures are the author's; we have not re-run the test.
Three patterns stand out.
The CNN was right more often overall. Allowing a second choice, it reached 173 of 200 against 134 and 138. The biggest gap was on harmless lesions: it recognised 37 of 50 moles and 26 of 50 seborrhoeic keratoses, against 16 and 13 for GPT-6 Astra and 33 and 6 for Claude Fable 5.1. It was one short of GPT-6 Astra on basal cell carcinoma and behind both LLMs on actinic keratosis.
The models failed differently. Each LLM had a favourite answer. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, including 14 of the 30 melanomas. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. The CNN's melanoma errors went the other way: of the 17 it got wrong, it called 8 a naevus and 6 a seborrhoeic keratosis.
Safety trade-offs depend on the threshold. At its preset threshold the CNN discharged more benign lesions correctly (76 of 100) but also discharged 18 of 30 melanomas, which would be unacceptable in practice. When its threshold is matched to the LLMs' sensitivity, it keeps 98 or 99 of the 100 unsafe lesions and discharges only 1 or 2 melanomas. The LLMs were conservative: they kept almost everything, which is why they discharged so few safe lesions (32 and 22).
Where the example cases make it concrete
Lynch's write-up shows individual cases. In one, an invasive melanoma on the ear under 0.8 mm thick, both LLMs called it a seborrhoeic keratosis, a harmless warty growth, and said the patient could be discharged. The CNN called it a melanoma. In another thin, low-pigment melanoma, GPT-6 Astra called it a basal cell carcinoma with 95% confidence while Claude Fable 5.1 and the CNN both called it a melanoma.
The confident wrong answer is the part worth noticing. A model that sounds sure while missing a melanoma is more dangerous than one that hedges, which is why calibration, not only accuracy, matters when you build with these models. For the launch-day claims behind the two models tested, see our coverage of Claude Fable 5.1 and GPT-6 Astra.
What the author flags as limits
Lynch lists the caveats himself, and they deserve equal weight with the headline:
- Small numbers. Two hundred lesions show a pattern but cannot rank close results, and 30 melanomas give wide uncertainty on the figure that matters most.
- One run per LLM. Answers vary between runs.
- Hard cases. Every lesion was biopsied. On ordinary archive images the CNN separated should-discharge from should-not-discharge far better (area under the ROC curve 0.96, against 0.86 in this set).
- Possible contamination. MILK10k has been public since 2025, so the LLMs could have seen it in training. If anything that would have helped them. The CNN had not.
- Disclosure. The analysis and first drafts were produced with Claude Code, which runs on a Claude model, and one of the two LLMs tested is also Anthropic's. Scoring was done by script against the histology.
When the question was reframed the way a patient might ask it, with the clinical photograph alone and the question "do I need to see a doctor?", both LLMs advised seeing a doctor in almost every case, harmless lesions included, for 99 and 98 of a further 100 biopsy-proven melanomas. That is the safe behaviour for a consumer chatbot, but it is also a reminder that "go see a doctor" is not a diagnosis.
What developers are saying
The Hacker News thread (about 57 points when we checked) was sceptical of the framing rather than the data. One commenter, MitPitt, compared it to a calculator outperforming LLMs at arithmetic. Another, podgietaru, argued the outcome is unsurprising because one model is trained on a simple narrow task while the other carries a huge general corpus. A third, howunfortunate, who identified as a machine learning engineer, said a fine-tuned CNN against general-purpose models on a tiny dataset does not belong on the front page.
These are the commenters' own views, not verified findings. The fair reading is that the experiment confirms an expectation many practitioners already hold, with real numbers from a clinician who can judge the clinical stakes. The notable part is less that a CNN won and more how large the gap was against models marketed as the most capable available.
What this means if you build with AI
Match the model to the task. If you have labelled data for a narrow perception task, a small fine-tuned model is cheap, fast and can run on-device. Lynch notes his CNN is small enough to run entirely in a phone browser with WebGPU, with nothing uploaded. That privacy property is hard to get from a frontier API.
Measure, don't assume. General-purpose benchmarks say little about your domain. A 200-case test with ground truth took one clinician a short time to build, and it changed the conclusion about which model to use. The same logic applies to the benchmark claims we track and to head-to-head comparisons of the frontier models.
Check the threshold, not just accuracy. In medicine the cost of a missed melanoma is not the cost of a missed benign mole. Report sensitivity and specificity at the operating point you will actually ship.
Keep a human in the loop. The industry is already moving AI toward clinical roles, such as Utah's AI initial prescriptions pilot. Evidence like this is the reason oversight and domain-specific validation matter. And mind the hidden cost of checking AI output, which surveys call the verification tax.
How to reproduce or extend it
Lynch says the cases, prompts, every LLM reply and the scoring code are in one download on his page, and the CNN is linked there too. If you want to extend the test, sensible next steps are:
- Run each LLM several times per case to measure variance.
- Add other frontier models and a vision-specialised medical model.
- Evaluate at matched sensitivity rather than at default thresholds.
- Test on a dataset released after the models' training cutoffs to rule out contamination.
- Compare with the dermatologist-level CNN work published in Nature in 2017, which first showed deep networks classifying skin lesions at the level of dermatologists.
Bottom line
A 2015 architecture, fine-tuned for twenty minutes on public images, beat two frontier models at a narrow medical vision task: 62% versus 42%. The sample is small and the author is candid about it. The durable lesson is practical. For well-defined perception tasks with data, specialised models still earn their place, and for any high-stakes use you should test on your own cases before trusting a general model, however capable its launch benchmarks look.
Details reflect the source article as of October 25, 2026. This is not medical advice; see a clinician about any changing or worrying skin lesion.