TruthHive rebuilds LLM research pipeline: AI judges, code decides

The account is a post-mortem of building an asynchronous research pipeline for TruthHive, a web service where users deliberate posted information. The initial design followed a textbook-looking pattern: extract claims from a starting statement, manage source mappings in a matrix, run multiple AI validation gates, and have a parent hub summarize everything. In practice the gates could not explain why a post was blocked, the support judgment deleted correct sentences as unsupported, and a rule stripping hedged expressions removed the feature's stated purpose of honestly disclosing what remains unconfirmed.

The author identifies a less obvious culprit behind missing information: analyzing execution logs showed the main cause was not model performance but the complex intermediate steps themselves, since each summarization or extraction pass shed a little content. The replacement pipeline limits the stages that rewrite content to memo creation, direct translation, and summary writing, while newly added stages are checks that do not alter the material.

Other decisions point the same way. Trustworthiness badges are computed by a pure function in code from source metadata, with AI used only where meaning judgment is needed, such as deciding whether a date or figure mismatch is real; source numbering skips canonical-URL normalization after it wrongly merged documents from different years, and sessions are explicitly closed as having no scheduled work when the task chain ends. The author notes the repair rules and the upstream structural change were not measured separately, leaving that comparison as open work.