Jev first, LLM second: task-grouping cost down about 90%

The problem the author set out to solve is bookkeeping: if you want to know how much AI usage went into each piece of work, you have to decide, request by request, whether a new request continues an existing task or starts a new one. Doing that with an LLM for every request means paying an LLM call per request. The proposed split is to let Jev classify everything and escalate only the doubtful cases, with a design shaped by an observed bias: in this data, all 12 of Jev's errors on the development set were "continued task misread as new task," and none went the other way. That is why the "new task" condition escalates whenever Jev says "new task" or its confidence falls below 0.5.

The cost comparison needs care. The gap between about $0.15 for the all-LLM method and about $0.016〜$0.02 for the Jev-first method includes not just Jev's screening but the dropping of a per-request summarization step that the all-LLM method performed. The author estimates that judging all 31 requests directly with the LLM would cost about $0.05〜$0.06, so the portion attributable to Jev is roughly a third of the original figure.

Results also varied by dataset: cost fell about 90% on the held-out data, about 70% on the development data, and about 60% on the paraphrase data, where vaguer wording lowered Jev's confidence and pushed the re-check rate from 41% to 62%. The author notes the held-out data may simply have been easier, since Jev alone scored 0.89 there versus 0.71 on the development data. On language, keeping requests in Japanese did not notably hurt task-continuation judgment, but the separate "is this request complete?" judgment was much weaker in Japanese (63〜70%) than when translated to English (90%).