Google launches first double-blind AI test

The new double-blind evaluation from Google DeepMind addresses a long-standing problem in AI benchmarking. Previously, external evaluators had to choose between revealing their test questions or receiving the model's weights, which could compromise intellectual property. The recent delayed evaluation of Anthropic's Fable 5 for the ARC-AGI benchmark illustrates this dilemma, as the company's 30-day data retention policy for its strongest models created friction. By using Confidential Space from Google Cloud, the setup cryptographically verifies that both the test data and the model remain private, eliminating the need for such compromises.

This method also tackles benchmark contamination, where models that have seen test questions during training can produce inflated results. The cryptographic protection ensures that a model cannot optimize itself specifically for a test it has already seen. The pilot on a Gemini Flash Lite model against confidential benchmarks is a first step, but it could set a new standard for model oversight. If successful, this approach might allow independent organizations, especially in sensitive areas like cybersecurity or government, to rigorously test advanced models without sacrificing data sovereignty or security, as DeepMind hopes.