Local LLM vs Claude Opus 5: same app, same blind spots

The experiment grew out of a practical concern: people handling confidential work would prefer to keep AI on a local machine. The question was how far a local LLM can be trusted. To find the boundary, the author had a cloud AI (Gemini) write a requirements document for a weather tracking app, then gave that same document to a local LLM and to a cloud AI for implementation. The two runs differed in form — single-shot generation versus an agent that could write files and run commands — and in prior knowledge, since the cloud AI had read the local LLM's experiment records. The author explicitly states these cannot be separated, so the comparison is not a ranking.

The defects fell into five layers. The first layer (format, config, types, schema) was largely caught by machine gates: the local LLM's output had 13 defect types found before gates, 12 caught by machines; the cloud AI had none in this layer. The remaining four layers — unstated assumptions, misreadings, unstated invariants, and UI/failure display — survived every gate on both sides. Both models displayed failures as non-failures: the local LLM silently substituted a meaningless count for an uncalculable metric, and the cloud AI showed 'not an error' alongside a fetch failure. Both failed to validate coordinate ranges, allowing latitude 999.

A mutation test added a concrete finding: when the time interpretation was switched to local time while keeping input validation, Tokyo and Los Angeles runs failed 10 tests each, but UTC runs failed none — meaning time zone bugs are invisible when the machine runs in UTC, common on servers and CI. The second installment will cover what requirements could have prevented and what they could not.