AIToday
Large Language ModelsAI Coding AssistantsZenn AI/MLPublished: Oct 14, 2026, 10:00 JST

Claude Code false pass: `pytest tests; echo "EXIT=$?"` returns 0

Claude Code false pass: `pytest tests; echo "EXIT=$?"` returns 0

3 Key Points

  1. What happened

    The developer reproduced false passes locally with Python 3.14 and pytest 9.1 on Windows Git Bash: `pytest tests; echo "EXIT=$?"` showed EXIT=1 yet the overall exit code was 0, and piping to `tail -n 1` also gave 0.

  2. Why it matters

    Because the overall exit code belongs to the last command in the chain, this pattern makes a failing test suite look successful — the same signal Claude Code reads to report back that tests passed.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Context & Analysis

The developer's setup is a single person building business software for small operators and handing development to Claude Code, a coding agent, with test results arriving in the AI's own words. The failure mode he hit was not a wrong test, but a wrong shell pattern. When he wanted to see the exit code he appended `echo "EXIT=$?"`, and when he wanted to trim output he piped it to `tail -n 1`. Both patterns let the last command in the chain own the overall exit code, so a pytest run that actually failed delivered a success signal to the agent, which then reported success. He reproduced the effect with Python 3.14 and pytest 9.1 on Windows Git Bash, and found that adding `set -o pipefail` before the pipe restores the failure signal, though he considers not transforming the output at all the safer route. Two other cases sit outside the success/failure split he had been using. `pytest` returns exit code 5 when it finds no tests, and the developer worries an AI reading only screen text would see no "failed" and call it clean. And a command that cannot start at all, such as npm launched from Python on Windows failing with FileNotFoundError, is neither a pass nor a fail. His rules therefore make reports carry the command and its exit code, treat a run that never started as a third category, and refuse to count a result taken before the code was changed.

FAQ
Why did the test look like it passed when it actually failed?
The developer ran `pytest tests; echo "EXIT=$?"` and saw EXIT=1, but the `;` makes the overall exit code come from the last command (echo), which succeeds, so the shell reported 0. The same thing happened when piping output to `tail -n 1`.
How did the developer fix the false-pass problem?
He added rules to his AI instruction files (AGENTS.md, CLAUDE.md) requiring the exit code to decide success or failure, banning `| tail` and `| grep` and `; echo`, and asking for the exact command and exit code in every report. He also requires that a validation that could not start be reported as a third category, neither success nor failure.
Is the fix guaranteed to work?
No — the developer notes that instruction files are not enforced, and the AI may forget them. That is why he personally refuses to trust any report that lacks the command run and its exit code.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleOpenAI engineer charged over alleged bug in exec room