DeepMind Kaggle Prize Criticized for Rewarding 'Nonsensical' Benchmark Work

A Google DeepMind-sponsored Kaggle competition inviting novel cognitive-science-based benchmarks for measuring progress toward artificial general intelligence (AGI) has just announced its results. The grand prize winner received $25K. However, a critic with expertise in machine learning has publicly questioned the award, alleging that the winning submission is fundamentally unsound. According to the critic's analysis, the original intent of the winning work was legitimate: to test whether a large language model would modify its own assessment of five claims in a complex scenario if first presented with alternative viewpoints on those same claims from other LLMs. This is described as an 'interesting question' from a cognitive-science perspective. In practice, however, the critic argues the submission deteriorated into what they describe as 'a vibed pile of spaghetti 10 times the size of the requested submission format.' The work reportedly contains unfounded claims and unclear methodology throughout. The critic further contends that neither the authors nor the DeepMind and Kaggle judges appear to have conducted a careful review of the material, or else did not see fit to reject it despite its apparent deficiencies. The critic has documented their concerns across two posts in the competition forum, offering what they characterize as 'AI research slop detective work' in support of their argument that a submission of questionable rigor was nonetheless awarded the competition's top prize.

The Kaggle competition 'Measuring Progress Toward AGI - Cognitive Abilities' was designed to crowdsource new benchmarks for assessing how far AI systems have progressed toward artificial general intelligence. A submitted entry won the $25K grand prize, but a researcher has now publicly challenged the decision, claiming the winning work is fundamentally flawed. The intended research question — whether language models shift their conclusions when exposed to alternative LLM perspectives on the same claims — is described as 'interesting,' but the critic argues execution fell far short. The submission apparently ballooned to roughly ten times the requested size and consists largely of unclear reasoning and unsupported assertions, yet appears to have passed DeepMind and Kaggle's review process without substantial scrutiny. This incident may reflect a broader tension in automated AI benchmark design: the challenge of ensuring that novel benchmarks themselves meet rigorous standards, particularly when competition volume or submission complexity makes thorough review difficult.

No comments yet. Be the first to share your thoughts!