AI benchmark v4.2 adds private tests to curb gaming

Artificial Analysis's update reflects a broader challenge in AI benchmarking: as models improve, public test sets become less useful because developers can optimize for them. By increasing the share of private data to 40%, the company aims to keep its index meaningful for comparing frontier models like Claude Fable 5.1 and GPT-6 Astra, which now top the ranking. The addition of more realistic tasks, such as knowledge work simulations and PDF reading, also aligns with the goal of measuring practical capabilities rather than test-specific tricks.

The decision to exclude GPQA Diamond, where scores have plateaued, is a pragmatic response to a test that no longer discriminates between top models. The planned further increase in private data share for v5 suggests that this is an ongoing effort. The index's credibility depends on whether the private data actually prevents gaming, which is hard to verify from the outside. For businesses using benchmarks to choose AI vendors, this update is a step toward more trustworthy comparisons, but it remains to be seen if it fully solves the problem.