Small Language Models in the Browser Fail at RAG: Faithful on 3 of 30 Questions Under 350 MB

Can a tiny language model that runs entirely in your browser answer questions about a website without making things up? One developer measured it, and the answer is mostly no. In a post on his blog, Bart de Goede describes "ask my blog": static-embedding search finds the four most relevant passages, then a small model in a Web Worker summarizes them in two sentences, with nothing leaving the visitor's machine. He then evaluated seven models on how often the summaries were faithful to the retrieved text. Models that fit a 350 MB download were faithful on at most 3 of 30 answerable questions. This post walks through the eval, the numbers, and the lessons for anyone considering on-device RAG.

(The author's blog is the source for all figures; the model-side background is in the Hugging Face docs for transformers.js, the library that runs the models.)

Quick reference

How does the system work?

The pipeline has four steps, all client-side:

1. Retrieve. The question goes through the same static-embedding search as the blog homepage, a roughly 4 MB lookup table, but keeps the four best passages instead of rolling up to posts.
2. Gate. If the best passage scores below a threshold, the model is not called and the page says it could not find an answer.
3. Prompt. The numbered passages plus a system prompt: answer in at most two sentences, use only these passages, say so if they do not answer the question.
4. Generate. A Web Worker runs the model and streams the answer, stopping after two complete sentences.

Loading a model is an explicit click because it downloads a few hundred megabytes from Hugging Face, cached afterward. If you want the basics of RAG, our guides to RAG versus agentic RAG and RAG context injection pipeline design cover the pattern, and the WebGPU guide explains the browser runtime.

Which models were tested?

The author wanted a reasonable click for a reader: a few hundred megabytes, not gigabytes. Candidates with browser-ready ONNX builds were SmolLM2 (135M and 360M), Gemma 3 270M, LFM2.5-350M, Qwen2.5-0.5B and Qwen3-0.6B. For reference he added Gemma 4 E2B (the smallest Gemma 4 model, a 3.1 GB download) and Claude Haiku 5.5 through the API. A practical surprise: for several models the 4-bit q4 build is larger than the q4f16 build, because q4 keeps the embedding table in 32-bit precision. SmolLM2-360M is 390 MB as q4 and 275 MB as q4f16 with identical output; all results use q4f16.

What were the results?

The eval reused 30 questions from his earlier semantic-search post, rephrased as questions, and added 15 questions the blog does not answer, mostly near-misses (for example which GPU to buy to run Llama 70B locally). Every model got the same four passages and prompt.

"Invented" means the model answered a question the passages do not cover instead of declining. The author's conclusion: nothing off the shelf under 350 MB is usable for this job. They are not slightly worse; they are faithful at most three times in thirty and "happily answer almost any question."

The GPU example is instructive. Given four passages about embeddings and a coding agent, the small models recommended specific GPUs with fabricated reasoning, including one that cited "6.4 million documents required by Llama 70B." Gemma 4 E2B correctly said the passages did not contain the information.

Grading by Claude Opus 5.5 means the judge is a model. The author hand-labeled 20 and agreed on 16, and notes that most disagreement was about the kind of bad an answer was; on the faithful-or-not question the judge was stricter twice and more lenient twice. Claude also generated the question set, which he reviewed. Treat the numbers as a careful single-developer eval rather than a benchmark.

Why do small models loop and invent?

Two failure modes stand out.

Looping. With greedy decoding, tiny models fall into repetition: SmolLM2-135M repeated "Search from your URL bar" and Gemma 3 270M produced strings of fives until tokens ran out. A repetition penalty of 1.15 helped, but it did not fix faithfulness; the models just wrote longer, more varied nonsense. All reported numbers use that penalty.

Never declining. The most consequential finding is that small models almost never say the passages do not cover the question. That behavior, not raw knowledge, is why size matters here: Gemma 4 and Haiku decline appropriately, the small ones do not.

A garbled answer is not always the model's fault. In one example the retriever did not hand LFM2.5 the section that explained how Bloom filters work, so the summary was wrong. "A summary can only be as good as the passages it reads" is a point that applies to every RAG system, including our own RAG pipeline guidance.

Does a relevance gate fix it?

A gate seems like the cheap fix: if search finds nothing relevant, skip the model. It works for far-off questions; "How do I bake sourdough bread?" scored 0.25 and a 0.45 threshold blocked it. But near-misses scored 0.54 to 0.63, higher than several questions the blog actually answers, because they land on posts about the same topic. To block them the threshold would have rejected 13 of the 30 real questions. So the gate only catches obviously off-topic queries, and the rest depends on the model.

Does a longer answer help?

No. In a quick experiment allowing two, four or six sentences, Qwen3-0.6B fell from 15 to 9 to 5 faithful answers, and at six sentences it stopped declining the GPU question and summarized every passage instead. Gemma 4 E2B slipped from 28 to 26 to 24 but barely used the extra room, about 2.8 sentences when allowed six. Every extra sentence is another chance to hallucinate, so keep summaries short.

Is hosting a model cheaper than an API?

The author's thought experiment: run Gemma 4 E2B for readers instead of making them download it. The cheapest sensible cloud GPU, an NVIDIA L4 on Google Cloud at about $0.71 an hour (prices he cites from computeprices.com on October 12, 2026), is about $516 a month always on. At his traffic that is over a dollar per question. Haiku answered all 45 eval questions for $0.006, about seven cents at 500 questions a month. His advice: with a real use case, start with Haiku, unless compliance, privacy or air-gapped deployment rules out an API. He also warns against putting an API key in a static site. For Haiku pricing and benchmarks see our Claude Haiku 5.5 launch post.

What this means for what you build

• Do not ship a sub-400 MB model for grounded Q and A without testing refusal. Measure the unanswerable questions, not just the answerable ones.
• Qwen3-0.6B is the floor, and it is a coin flip. At 579 MB the author uses it as default, with the warning to check passages before believing the summary.
• Show the sources first. His page displays the retrieved passages as cards before any summary. For small models, the retrieval result is the product.
• Fine-tuning is the open path. A 300 MB model does not need to know everything; it needs to read four passages and summarize or decline. The author flags that as a possible follow-up, and Gemma-family models are good candidates for tuning, as in our Gemma 4 open duck robot and Gemma 4 updates coverage.
• Be honest about privacy. In-browser inference means prompts never leave the device, which is a real benefit, but it is not worth wrong answers if your use case needs accuracy.

Caveats

• One developer's blog corpus, 45 questions, one passage-selection method.
• LLM-judged, with a small hand-label check.
• Single-vendor judge and generator of questions (both Claude).
• Browser results vary by WebGPU support; the author notes CPU caveats for some models.
• Prices and model availability change quickly.

What we are watching next

• A fine-tuned sub-300 MB model for the read-and-decline task.
• Whether newer small models close the refusal gap.
• Browser runtime improvements for larger on-device models.

Related reading

• RAG vs agentic RAG
• RAG context injection pipeline design
• WebGPU complete guide
• Claude Haiku 5.5 launch and pricing
• Gemma 4 on an open duck mini robot
• Gemma 4 updates

Details are accurate as of October 13, 2026. All eval figures come from the author's blog post; we have not reproduced them.