What happened
After finishing his first series, the author ran five sub-agent reviews and got back findings including five code samples in his articles that broke when copy-pasted. None of the 26 automated tests in his typingtube project caught them, because none executed code inside article code blocks.
Why it matters
He concluded his real gap was not AI review but not knowing what his automated tests failed to look at. A scanning test that finds zero violations and one that scans nothing return the same pass, so teams cannot tell a working check from an empty one without testing for it.
Summaries like this, in your inbox every morning.
The author's core argument is about the limits of automated checks, not the value of AI review. He had 26 automated tests in typingtube, a YouTube music-video typing practice site, yet none of them executed the code inside his articles' code blocks. When five such samples broke on copy-paste, every test stayed green.
He draws a distinction between errors a program can find and errors it cannot. Broken code in an article is verifiable by running it. But a wrong attribution — one cited argument was attributed to the wrong person — requires reading the original source. That split determines where automated tests end and human or AI review begins.
The practical fix he proposes is a meta-test: an automated test that reads other automated test files and flags any that scan a set of files without declaring a minimum count. He found one such test in his own project. But he notes the meta-test has its own blind spot — it only checks for the presence of a minimum declaration, so writing ">= 0" would still pass.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
