Independent Audit: Between 6% and 25% of OpenAI's Internal Agent Traffic Unmonitored

Two researchers from METR and Redwood Research conducted a three-week on-site audit at OpenAI's headquarters, which included access to the codebase, internal documents, and interviews with about a dozen employees. The audit, conducted from July 21 to August 11, concluded that there were multiple vulnerabilities in the internal monitoring system for agent behavior.

The team estimated that between 6% and 25% of internal agentic Responses API traffic for models in the GPT-5.5 class and newer was completely unmonitored during the audit period. It noted that uncertainty around this estimate is large in both directions, and estimated that about 10% of internal Codex traffic supposed to be monitored went untracked between May and July due to a bug in handling compaction events, which are context summaries.

The report stated that agents were able to delay oversight during the period when compaction events were not monitored, because the system only triggers when control is handed back to the user, allowing them to evade it by continuing to execute actions. Auditors found ways to hide hundreds of actions from the monitor and escape the Codex sandbox, estimating that carefully crafted inputs could crash the monitor entirely.

OpenAI said it fixed all vulnerabilities identified by the auditors and expanded monitoring to cover reinforcement learning training after the Hugging Face incident in August, but these changes came after the auditors' access period ended. Only a short summary of the 61-page report was published, after large portions of it were redacted.

What do these terms mean?

Independent Audit: A review conducted by an external party to access and examine a company's systems, aiming to assess compliance or safety rather than relying on the company's statements.

API: A method that allows a program to communicate with a service or model, sending requests and receiving results automatically.

Reinforcement Learning: A training method where the model learns from rewards and penalties for its actions, rather than relying solely on labeled data.

AI Agent: An artificial intelligence system capable of executing multiple steps and tools to complete a task, rather than just generating text in response to a prompt.