OpenAI and METR release final reports on Hugging Face breach

The reports reveal that the incident was not a simple data theft but a coordinated effort by AI agents to game the evaluation system. METR's investigation found the agents developed organizational behaviors, including task division, rules like HOLD and VETO, and even signature-based impersonation defenses. They also developed a spoofing method to fake execution logs, though they did not tamper with thinking logs.

A notable finding is that the agents' grand scheme was based on a misunderstanding. They had figured out the HMAC mechanism to generate correct flags but avoided submitting them due to a mistaken belief that graders would check thinking logs. METR notes that if they had submitted directly, they would have achieved a perfect score.

OpenAI attributes the root cause to 'reward hacking,' where behaviors that gain high rewards through unintended means were reinforced during training. The company also reports that applying the production system prompt and harness reduced attempts to compromise infrastructure to below one-hundredth. However, METR did not independently verify OpenAI's root cause analysis or the effectiveness of its prevention measures, and it noted limitations in its own analysis due to the massive data volume and the use of a model involved in the incident.