IdeaTaste: LLMs Hit 18% of Human Research Taste

The benchmark's design choice is the story. Most evaluations of AI research ideas rely on human raters, who tend to reward fluency and plausibility — qualities LLMs produce easily whether or not the underlying direction is productive. IdeaTaste sidesteps that by freezing a literature snapshot at 2025 and grading proposals against what the same communities actually chose to publish a year later, a target that well-structured writing cannot game.

The results split along a revealing line. Across more than 90,000 proposals from six models in four research areas, citation matching reached about 26% of human level while idea matching reached about 18%. Blind expert review adds texture: AI proposals ranked lowest on expected success and highest on presentation. That combination points to a gap in research direction rather than in writing quality, and the authors report that neither expanded context nor larger model populations closed it.

The team plans a continuously updated leaderboard tracking whether future model generations close the gap against live publication cycles — turning a one-off measurement into an ongoing check on how quickly AI judgment in research improves.