What happened
A tutorial explains how to build an AI text detector from scratch, inspired by Substack's recently launched AI detector feature in its UI. The project involves fine-tuning a DistilBERT classifier to return a 0–100 score indicating how likely a text is AI-generated.
Why it matters
AI detectors have practical uses beyond flagging spam—they can help writers improve grammar and clarity without inadvertently making their own writing sound AI-generated and triggering detection. The tutorial also demonstrates how to build a verifier-based LLM application, a broader technique beyond reasoning models trained on math and code.
What to watch
The tutorial acknowledges a fundamental limitation: AI detection is a cat-and-mouse game. As LLMs evolve to avoid detection patterns, detectors must be updated in turn. False positives (human text flagged as AI) are also likely to occur.
Summaries like this, in your inbox every morning.
Substack's recent launch of a built-in AI detector feature triggered interest in understanding how such detectors work. The tutorial responds to this interest by offering a hands-on educational project that breaks down the mechanics of detection and applies them in a real-world context. The author references earlier work on AI-text detection approaches—supervised classifiers, perturbation-based probability tests, perplexity measures, and watermarking—and positions the DistilBERT classifier as a practical middle ground, similar to the method behind Substack's own feature (which the author understands to be based on Pangram models).
The project frames AI detection not merely as a content-moderation tool but as a utility for writers themselves. This reframes the detector as a collaborative rather than adversarial tool: writers using grammar-checking or AI-assisted editing can query the detector to ensure their personal voice is preserved. This dual-use perspective reflects a real tension in the writing workflow—the temptation to over-polish text with AI can inadvertently make human writing sound machine-generated.
The author is transparent about the fundamental fragility of the approach: detection will always lag behind generative capability. As LLMs evolve, intentionally or incidentally, to avoid detection patterns, the detector becomes outdated and must be retrained. Combined with the risk of false positives, this suggests that no AI detector can be a foolproof solution—only a tool in an ongoing arms race.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
