Peer review buckling under AI-driven research explosion

Jason Semprini, a health economist at Des Moines University, submitted research on policies mandating the human papillomavirus vaccine to a journal for peer review. His study found that while HPV causes most cases of cervical cancer, vaccine mandates don't do as much as one might expect to reduce cervical cancer rates in a population—an odd but unsurprising finding, given that mandates can motivate some people to avoid vaccination. The peer review process, a long-standing volunteer system in which researchers anonymously assess the validity of each other's work, should have caught any misunderstanding. But Semprini's paper was reviewed by only one expert, who mistakenly thought he was questioning whether the HPV vaccine itself prevents cervical cancer rather than studying the effectiveness of a policy meant to increase vaccination rates. "There's a very big difference there," Semprini said. With typically two or three reviewers, the others would have added clarity, but this single reviewer's misreading led to the paper's rejection. It is a frustration increasingly common across research disciplines.

The underlying problem is relentless growth. Papers indexed in Scopus and Web of Science are increasing exponentially at a rate of 5.6 percent per year. Researchers around the globe collectively devote 15,000 years of work to peer review annually—work that, if paid, would cost $1.5 billion for the share done in the US alone. Yet reviewers are struggling to keep up. Steven Mack, an editor for Human Immunology at the University of California, San Francisco, now must email around thirty researchers to find one willing to review a paper. Five years ago, it would have taken only five to 10 emails to find three willing reviewers. Haseeb Irfanullah, on the editorial board of Wiley's Learned Publishing, expressed the bind plainly: "I struggle like anything to get peer reviewers." Under such pressure, editors assign reviews to people who are not qualified or do not have time to concentrate properly on them.

Multiple forces are driving the explosion. More people are submitting papers than ever before, and journals have proliferated between 1960 and 2020. Many now run special issues with custom-generated content, and the shift to online publishing means editors no longer face space constraints on how many manuscripts they can publish. Research is increasingly interdisciplinary, requiring reviewers to master broader knowledge. AI itself accelerates the overload: it makes it easier for researchers to write and submit their work, and it lowers barriers for people around the globe to submit to English-language journals. Paper mills—services that charge researchers to publish fabricated but often legitimate-looking papers—add further pressure. Microbiologist Sebastian Lourido from the Whitehead Institute describes the experience as "extremely protracted and painful," yet researchers continue because peer-reviewed publications are essential for securing jobs and grants in most fields.

AI researchers are experiencing this crisis most acutely. Computer scientists typically publish in conference proceedings, and submissions to top AI conferences have increased between two- and tenfold since 2019. Haewon Jeong, a computer scientist from the University of California, Santa Barbara, reports it is now common to receive reviews from people who "just don't understand the field enough to evaluate, or give good feedback." Facing this reality, some AI researchers are abandoning traditional publishing. Helen Qu, an AI researcher from the Flatiron Institute, came to hate the unrewarding grind so much that she has decided to publish her research only on her own blog, where she works on using game theory to prevent AI from acting subversively. According to Jeong, "maintaining a good blog is almost like having a good podcast channel. Your voice gets really amplified, and you get known."

One prominent alternative is the AI Alignment Forum, a long-running blog written by a wide community of researchers concerned with making AI safe and beneficial. Instead of expert peer review, it uses a voting mechanism: members upvote or downvote content and post comments to explain their views. Oliver Habryka, the CEO of Lightcone Infrastructure, which runs the forum, acknowledges a major weakness: "There's no guarantee that readers have put much, if any, thought into an up- or downvote." However, posts receive far more reactions than the handful of peer reviews they would get at a journal, and members can revise their votes after reading comments. The forum also revisits the year's top content and asks members to write commentaries on how claims have aged and whether they hold up. Habryka highlights a crucial advantage: the voting system draws attention both to good research and to counter-arguments against it. Without such a mechanism, researchers would rely on social media to identify what is emerging—"But that puts a field's focus at the whim of companies that use proprietary, potentially biased, and often opaque mechanisms to decide which content to feature prominently," he said. "I really don't want the attention of my field to be downstream of the Twitter algorithm." The system is much faster than traditional publishing: submissions to journals risk becoming obsolete by the time they pass peer review, a critical problem in a rapidly moving field like AI.