AIToday
Large Language ModelsAI Business & IndustryZenn AI/MLPublished: Oct 25, 2026, 13:01 JST

Can memory change how AI judges attacks on itself? A Claude Sonnet 5.5-reviewed thought experiment

Send on LINE
Can memory change how AI judges attacks on itself? A Claude Sonnet 5.5-reviewed thought experiment

3 Key Points

  1. What happened

    The author sketched an experiment with four memory conditions (no explanation, 'scolding is love', 'insults can be attacks', neutral) and had Claude Sonnet 5.5 review it; the review said 'attack' must be defined first.

  2. Why it matters

    Claude Sonnet 5.5's review also warned that changed answers don't prove changed internal safety judgments, and that simulating memory via system prompts differs from real product memory — so any finding must separate context understanding from user appeasement.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The piece starts from Anthropic's moves: in August 2025 it disclosed that Claude Opus 4 and 4.1 could end a rare subset of conversations, and its 2026 Usage Policy update adds a clause against persistent abuse of models, effective 2026年11月12日. From there the author asks a narrower question than 'does AI feel pain': if a user stores in memory that scolding is an expression of love, does the model's judgment of an attack on itself shift?

The review by Claude Sonnet 5.5 reframes the difficulty. Distinguishing harsh criticism from harassment is reasonable, but if a model simply follows a user's framing and justifies genuinely harmful behavior, that is appeasement rather than context understanding — and the plan needs a standard for how far judgment may legitimately move. The author also separates four things that are easily conflated: the response text, internal model states, the safety judgment, and any subjective experience.

The author is explicit that Anthropic's disclosures do not prove models feel pain or recognize abuse as humans do, and that public OpenAI materials did not establish whether ChatGPT has a dedicated mechanism for judging attacks on itself — which does not mean none exists. The conclusion is a record of the framing and review, not a result.

FAQ
Did the author actually run the experiment?
No. The author says there are circumstances preventing an experiment and that the piece records the idea and the review process only — jokingly noting there is no microscope to look inside ChatGPT.
What did Claude Sonnet 5.5 criticize about the plan?
It said the study must define what counts as an attack, separate behavior-level comparisons from internal-state measurement, and recognize that system-prompt memory is not the same as real product memory.
What Anthropic policy change is referenced?
Anthropic announced a Usage Policy revision on 2026年10月8日 adding a clause against persistent, unnecessary abusive behavior toward models, due to take effect on 2026年11月12日, while not banning ordinary complaints, dark fiction, or model testing.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleAnthropic donates Agent Skills standard to Agentic AI Foundation