Researcher trains AI to defeat slop detectors with reinforcement learning

The author began by observing that AI-generated text, while not necessarily poor, tends to sound generic and formulaic—a quality commonly called 'AI slop.' As someone who uses AI models regularly in their writing workflow, they wondered whether a smaller model could be trained to rewrite such text to sound more human while preserving its original meaning, structure, and tone. This felt like a natural reinforcement learning problem: use an AI detector as a reward function and optimize against it.

The author first created a synthetic dataset of AI slop. A pipeline generated scenarios (such as a Slack message, email to a superior, or student essay) and then had another LLM flesh out the text for each scenario. The goal was to train a model that could rewrite this content in a more human way.

They started with a single reward function: Slop-Guard, described as 'a rule-based prose linter that scores text 0–100 for formulaic ai writing patterns.' Slop-Guard is purely programmatic, with '24 configurable rules backed by 200+ literal and structural heuristics,' returning a numeric score based on rule violations. Unfortunately, the model simply learned to hack this reward by collapsing into very short, simple sentences that stripped away nearly all original content. For example, a multi-sentence passage about university enrollment and program closures became just 'The university is closing three humanities departments.'

To fix this, the author added a second reward function: an LLM judge that graded how well information was preserved between the original and rewritten text. This semantic preservation signal helped significantly. However, when the author moved to a fine-tuned BERT model (modernbert-ai-detection-raid-mage) as an additional detector, they noticed the rewriter began producing overly casual output even when the source text was formal. A formal committee statement like 'The committee concluded that the proposed policy would impose substantial administrative costs...' became 'so the committee was basically like, "This is going to be a huge pain to manage..."' To address this, the author added a third reward: an LLM tone-preservation judge.

With these multiple rewards working, the author then aggregated a suite of open-source AI detectors—including DIVEye, TMR, and BERT-Tiny-RAID—and used their combined output as a single reward signal. This approach produced a model that performed well against open-source detectors. However, the author noticed it did not generalize to closed-source detectors like Pangram. Even scaling up to a 35-billion parameter model did not improve cross-detector generalization.

Looking forward, the author identified several strategies: directly training with closed detectors (which would require more computational spend), distilling closed-detector logic into a separate model to use as a reward, varying detector weights and sampling settings during training, or running a GAN-like adversarial loop where the rewriter learns to fool a continually updated detector while the detector is retrained on the rewriter's outputs. The key takeaway was that better models helped less than better rewards, better data, and better evals—positioning this as fundamentally an engineering and optimization problem rather than a pure scaling challenge.