Jev tuned 6 times, still loses to Claude (Opus) grep by 2x

The author's starting point was a hook that injected excerpts of Obsidian notes into every Claude Code request. Over the most recent 14 days it fired 2,333 times, averaging 4,840 characters per insertion, yet Claude read only 0.9% of the vault on its own and just 3 of 154 notes it read had been among the hook's candidates. That made context selection the problem to solve, and Jev — a model that returns typed judgments and probabilities instead of text — looked like a cheap, fast classifier for the job.

The tuning history shows why the final comparison mattered. Early attempts asked for yes/no on hundreds of candidates and missed 46% of the time. A version that classified only the request text was fast but caught the right domain just 55% of the time. The third round, using a choice over all candidates plus recent conversation, produced a 0% wrong-proposal rate on 20 requests — a result the author later called too good, since the threshold was picked on the same 20. When the test grew to 100 requests and 1,331 candidates, that method's error rate rose to 40%.

Against that backdrop, the Opus grep baseline was never measured until the author asked for it. On the same 100 requests, Opus with grep and conversation context found 52 correct notes versus Jev's 24, and combining the two only moved the count from 52 to 54. Jev did have a lower wrong-proposal rate (20% versus 25% for Opus's top pick) and fewer false proposals on requests needing no note. The author's conclusion is that Jev fits narrow, high-volume classification or gating tasks, while a context-aware model already in the conversation is hard to beat on note selection.