Tokyo dev validates typesafe/jev job-listing checks at 97.8% accuracy

The test grew out of a specific compliance need: Japanese care-worker job postings must disclose 14 labor conditions under the Employment Security Act, and the developer built a checker that classifies each condition as stated, ambiguous, or missing. Rather than using one AI to grade another, the developer hand-labeled 30 sample postings — ten from Hello Work-style formats, ten from private job sites, and ten deliberately corrupted — and treated the most damaging error type separately. This mattered because a job listing that says 'not stated' when something is stated is the costliest mistake for this use case.

The other practical findings point to how such models should sit inside a real pipeline. Jev abstains when confidence falls below a threshold, and because the calibrated confidence behaves differently across states, the cutoffs were set per state and exposed in a config file so operations staff can adjust them. In production, the developer also pins the model version and caps calls per run, reflecting a concern that an unversioned third-party model could shift behavior without notice.

The broader pattern emerges from a second project: when classifying event announcements, Jev could reliably tell whether a text was a tournament notice, but not distinguish a new event from an alias or a moved one, because that requires matching against a historical ledger. That suggests judgement-only AI is likely best suited to narrow checks decided by a single document, with a rule-based system and human review handling anything that needs external records.