Jeff-0.8B hits 100.0% on long contracts, 0.56s per query

The test set out to answer a practical question: how much of a cloud decision model's behavior a local setup can reproduce. The author had already run earlier experiments with Kev-4B and Qwen3.5-4B, but those used per-model Python scripts, so prompt format, parallelism and HTTP overhead varied between runs. This round put all six models behind the same REST protocol, the TypeSafe ADR-010 /v1/systemone wire format, and measured them through levbench, an open evaluation harness published by Interfaze AI. That change alone shifted results: Jev's measured accuracy moved from 76.67% to 77.33% and its wall time halved to 6.4 seconds, while Qwen3.5-4B dropped from 67.00% to 60.00% because the harness sends one request at a time instead of using SGLang's dynamic batching.

The long-document phase tested something different from the 300-question S1Bench suite. The 15 questions asked whether models could locate specific clauses inside 2,000 to 4,500+ token agreements, such as a 14-day written-confirmation rule for oral disclosures or a three-year survival period. The results split cleanly by architecture. Cross-Attention designs (Jev, Lev, Jeff-0.8B) and Qwen3.5-4B's RadixAttention prefix cache all reached 100.0%. Kev-4B's Gated DeltaNet, which runs in linear time but compresses history into a fixed-size state, managed only 60.0%, and CLM-8B's pooled document embedding collapsed to 40.0% even with Action Caching, with 13 of 15 answers biased to False.

The contrast between these results and the HelpSteer2 scoring task is the study's most useful signal. Jeff-0.8B scored 100.0% on contract retrieval but just 18.0% on 5-point response-quality judgments, where the model must check answers against its own stored knowledge rather than text in front of it. The author reads this as a deployment rule rather than a flaw: when judging facts a model has not memorized, or rules it has never seen, feed the premises directly in the prompt and let the model only compare. Whether that rule holds outside these 15 documents is the open question, and it is likely to decide how much of this local setup reaches production.