"ask my blog": Qwen3-0.6B faithful 15/30, Gemma 4 E2B 28/30

The tool sits on top of an existing static-embedding search that already ran in the browser for the blog's homepage, so retrieval needed no server. What changed with "ask my blog" is that a model now reads the four top-scoring passages instead of just ranking posts, and that forced a measurement question: whether a tiny model can stick to what those passages say. The test pitted download size against faithfulness across models with browser-ready ONNX builds and two references, and the split was sharp. Models up to a few hundred megabytes were faithful at most three times out of thirty, while Gemma 4 E2B, at 3.1 GB, reached 28/30 and Haiku 5.5 through the API hit 27/30.

The failures were often not wrong so much as stuck. SmolLM2-135M repeated "Search from your URL bar" four times, and Gemma 3 270M emitted a long string of digits until it ran out of tokens, which a repetition penalty of 1.15 addressed without improving faithfulness. On the GPU question, asked against passages about embeddings and a coding agent on a MacBook, the small models named cards from GTX 960s to A100s and TensorRT, while Gemma 4 E2B correctly said the passages did not contain the answer. Even answerable questions suffered, partly because retrieval did not always hand over the explaining section: search gave a Bloom filter post's use cases and demo, and LFM2.5-350M described a Bloom filter as a hash table.

The cost question also got a concrete look. Running Gemma 4 E2B on an NVIDIA L4 in Google Cloud is about $0.71 an hour, or $516 a month to keep running, which for the blog's monthly readership would be over a dollar per question even if every reader asked one. Haiku 5.5 answered all 45 eval questions for $0.006, or about seven cents at 500 questions a month. On the eval itself, the question set was generated by Claude and reviewed by the author, and 15 of the 45 questions were ones the blog does not answer, most of them near-misses designed to tempt a plausible-sounding wrong answer.