Jev scores 60% vs home AI's 83%, but errs only below 74% confidence

The test began with a suspicion rather than a benchmark. In an earlier piece, the author had tried to replicate Jev's approach on a home PC and watched it declare, with 96% confidence, that an email saying a transaction had completed meant no money moved. That kind of confident error is harder to catch than a plain mistake, because a system that filters only low-confidence answers will wave it straight through. Jev's developer claimed special training aligned the confidence numbers with reality; this experiment was the check.

The 602-email comparison produced a split verdict. On raw accuracy over the 35 disputed cases, the home-built AI won at 83% against Jev's 60%, and the prior production AI trailed at 49%. But the error patterns told a different story: Jev never made a mistake above 74% confidence, while the home AI's errors arrived stamped with 96%, 89%, and similar figures and persisted even when the review threshold was raised to 95%. The home AI also labeled 191 of 543 emails as 99%-or-more confident, leaving almost no room to separate good answers from bad.

A second lesson came from the labeling work itself. Deciding whether an order confirmation counts as "money moved" changed the ranking of all three AIs, suggesting much of the disagreement reflected an undefined question rather than model weakness. And rewriting the prior AI's prompt with more detailed rules backfired: false positives multiplied twelvefold while misses fell, with nine of the new false positives landing on examples explicitly marked as non-matching. The author's takeaway is to split instructions into small questions rather than piling on conditions.