Sarashina3 mini vs Claude Opus 5.5: meaning kept in 1 of 10 rewrites

The comparison was prompted by Sarashina's free beta release on 2026-10-13, which made the model accessible via API. The author paired it against Claude Opus 5.5, a near-top-tier Anthropic model, on an identical set of 40 Japanese tasks. Because the Sarashina terms of service do not allow third-party copyrighted material or personal information in inputs, the test materials were limited to laws and court decisions, Aozora Bunko works whose authors died more than 70 years ago, and original sentences. GPT served as the blind judge, with answer order swapped for a second judgment to reduce position bias; a consistent winner counted as a win, and a split was a draw. Opus was not used as judge to avoid favoring its own outputs.

In the rewrite section, Sarashina's failures clustered on legal and judicial text where conditions and scope are stated precisely. In one personal-information law item, GPT flagged three meaning changes: replacing 'personal data' with 'that information' broadened the prohibition, turning 'cases based on laws and regulations' into 'when stipulated by law' dropped cabinet and ministerial ordinances, and the timing of obtaining consent disappeared. In a double-negative internal notice, both models added an obligation not in the original by opening with 'please be sure to attach the original,' and Sarashina further reversed the condition so that originals could be accepted without the manager's approval. These examples illustrate the kind of drift that readability alone does not catch.

The author's own blind readability judgment on a separate 10-question set adds a twist: the rewrites rated most readable were mostly Sarashina's, yet GPT judged 9 of Sarashina's 10 as having changed meaning in both rounds. In the keigo section, a phone-answering task showed a concrete etiquette gap: Sarashina referred to 'Section Chief Sato' when speaking to an outside caller, whereas Opus used 'Sato' without the title and even appended an explanation that one does not elevate one's own superior to outsiders. The kanji-reading results split on context-dependent words, where Sarashina tended to give the first dictionary reading despite cues in the sentence.

The author lists several limitations: Sarashina was represented only by its mini model against a near-top-tier Anthropic model; 31 of the original questions and 5 of the separate rewrite questions were written by Claude Opus 5.5, which may suit Opus; the four summary works are famous enough that both models may have memorized them; GPT's meaning judgments are fine-grained, so the rewrite scores could shift with the strictness threshold; and each question was generated only once, so run-to-run variance was not measured. These caveats mean the numbers should be read as a directional signal rather than a definitive ranking.