Jeff, a 0.8B model, beats keyword rules on tricky email sorting

Jeff is built on Qwen3.5-0.8B, fine-tuned so that instead of generating an answer it returns a probability for each option. The weights sit on Hugging Face as mstrasser/Jeff-Qwen3.5-0.8B, and the project's README frames the model as a first stop in front of a 27B model: Jeff handles the decisions it is confident about, and only the uncertain ones get passed on. That framing is the reason the author set up the comparison at all.

The test pitted the model against a deliberately naive baseline: a keyword classifier that scans a sentence from the start and adopts the category of the first keyword it hits. On ten easy emails both scored 100%, so the difference only showed up on eight adversarial emails built with mixed keywords, negations and sarcasm. There the rule-based classifier dropped to 37.5%, missing five cases by latching onto words like "cancel" or "delay", while Jeff held 87.5%. Jeff's single miss came with a low-confidence probability spread, which is the kind of signal a 0/1 classifier cannot produce.

Speed is where the pitch did not hold up in this setup. The README cites 463 ms per decision on CPU with 32 threads, but on a GPU-less 4-core cloud environment the author measured roughly 1,500 ms per decision, likely because requests were sent one at a time with no batching and few cores were available. The honest takeaway seems to be that small models can beat keyword rules on messy language, but whether that is worth the latency depends on the use case and on running on GPU or Apple Silicon hardware.