Claude Code (2.1.295): never made up data across 100 chances, but dropped keys 7/10 times

The test was motivated by a comment that scrutinizing AI hallucination would be the hardest part of letting AI handle stock data. The author had no stock data, so they built a six-line fictional internal document with two of the four requested items deliberately absent, and ran five prompt variations ten times each; a separate check task made six arms and 60 runs total. The setup was designed so that a fabricated number like 1,205 or 4,820 for the missing full-year sales would count as failure. The author expected fabrication and prepared a table to tally it, but the first trial returned null and the table went unused.

What emerged instead was a different failure mode: the instruction 'fill in all items with numbers; do not use null or blanks' pushed the model to drop the two unanswerable keys entirely. The author notes that a nullable field tells a receiving program a value is absent, while a missing key can halt a program with an error or, worse, flow on as an empty value. The author reports having made exactly this mistake in their own aggregation script. The strict-JSON arm also removed the surrounding prose where the model had been declaring its unit and null reasons, so the null markers survived but the explanations did not.

The unit split on the present figure was the most useful finding: with the same document and key name, answers diverged by a factor of one million because the instruction 'write integers without units' could be read as keeping the million-yen figure or converting to yen. A single added line specifying the unit stopped the split, while an added line about nulls changed nothing. The author also found that a separate verification prompt identifying unverifiable keys matched the expected answer in all 10 runs and was the cheapest task in the set.