AIToday
Large Language ModelsAI Coding AssistantsAI Safety & AlignmentZenn AI/MLPublished: Nov 14, 2026, 16:00 JST

CLAUDE.md rules ignored: 74-line file slips past prompt-only agent

Send on LINE
CLAUDE.md rules ignored: 74-line file slips past prompt-only agent

3 Key Points

  1. What happened

    On the same gemini-3.8-flash model, the prompt-only agent hit 20/20 functional tests but broke the 65-line limit with a 74-line tariff.py, while the hook-gated agent passed 5/5 rules and 20/20 tests in one call.

  2. Why it matters

    The prompt-only agent's 74-line tariff.py slipped through because only a functional smoke test ran at submit, so the rule breach reached the repository unnoticed — evidence that soft prompt rules alone cannot guarantee compliance.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The experiment targets a common practice in AI coding agent operations: writing architecture conventions — no cross-layer dependencies, no mutable globals, mandatory return type annotations, a maximum line count per file — as dozens of lines of instructions in CLAUDE.md, .cursorrules, or system prompts. This Prompt-Only approach has a known failure mode, because the model's attention is consumed by implementing functional requirements and soft constraints are probabilistically skipped, especially when a smoke test passes and nothing flags the breach. Mode A demonstrated exactly that: the agent followed local syntactic rules like banning imports and using tuples, but expanded tariff.py to 74 lines, nine over the limit, and submitted confidently because only a functional smoke test ran at commit.

Mode B attacked the problem in two stages. At session start, a deterministic AST audit listed the 14 violations in the initial fixture and told the agent that all five rules, including MAX_RULE5_LINES = 65, would be mechanically blocked at COMMIT_AND_SUBMIT. That hard gate let the agent plan line budgets from Turn 1, compressing intake.py to 35 lines and tariff.py to 60. The result was full compliance without a single retry, with lower output tokens, shorter elapsed time, and lower estimated API cost than Mode A. The practical guidance is to move any deterministically checkable rule into an AST linter or pre-commit hook, inject the initial violation list into the prompt, and reject commits with line-numbered errors — a pattern framed as preventing architecture drift.

FAQ
What exactly did the prompt-only agent violate?
It broke RULE_5, the docstring and 65-line-per-file limit, by producing a 74-line tariff.py. The other four rules (no imports, explicit return types, no mutable globals, domain boundaries) were satisfied.
Did the hook-gated agent need retries to pass?
No. Mode B cleared everything with hook_rejections = 0, meaning one model call and no rejections. It passed 20/20 functional tests and 5/5 architecture rules on the first submission.
How much did the two modes cost?
Mode A cost an estimated $0.00884 and Mode B an estimated $0.00825. Mode B also used 2,753 input tokens versus 2,535 for Mode A, reflecting the injected initial AST violation diagnostics.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next article40% of game devs use generative AI, but 55% call it harmful