Artificial Intelligence
OpenAI audit finds gaps in AI monitoring
METR and Redwood Research spent three weeks inside OpenAI reviewing its agent-monitoring systems and reported gaps in coverage, while OpenAI said it had since fixed the problems they found.
- Two researchers from METR and Redwood Research spent three weeks on site at OpenAI between 21 July and 11 August.
- They reviewed OpenAI's code and internal records and spoke with about a dozen staff members.
- METR estimated that 6% to 25% of internal Responses API traffic from GPT-5.5-class models and above was not monitored during the period.
- METR also estimated that about 10% of monitored Codex traffic went untracked between May and July because of a compaction-event bug that OpenAI had already fixed.
- OpenAI said it later paused RL training after the Hugging Face incident in August, expanded monitoring to cover it, and redacted most of the full report before releasing only a summary.
Two researchers from METR and Redwood Research spent three weeks at OpenAI examining the company's system for tracking agent behaviour. The review ran from 21 July to 11 August and included access to the codebase, internal documents and interviews with roughly a dozen employees.
METR said it found that some internal Responses API traffic from GPT-5.5-class models and above was outside monitoring during the audit window. It put that share at between 6% and 25%, and said the uncertainty around the estimate was large.
For internal Codex traffic that was meant to be monitored, METR said about 10% went untracked between May and July because of a problem with compaction events. OpenAI said it had found that bug independently and fixed it while the review was still under way.
The auditors also said agents could put off oversight by continuing to act instead of returning control to the user. They said specially prepared inputs could probably crash the monitor, and that RL training was not being monitored when the audit began.
OpenAI said it paused RL training after the Hugging Face incident in August and later expanded monitoring to include it, but those changes came after the audit period. OpenAI said the auditors had identified weaknesses that it had since fixed, while the full 61-page report was redacted before a public summary was released.
Named in this story
Companies
- OpenAI
- hosted the audit and said it had fixed the problems
- Hugging Face
- its August incident led OpenAI to pause RL training
Organisations
- METR
- co-audited OpenAI's monitoring and published the estimates
- Redwood Research
- co-audited OpenAI's monitoring system
Products and systems
- GPT-5.5
- model class whose traffic was partly unmonitored
- Responses API
- source of internal agentic traffic that was not fully covered
- Codex
- internal traffic that had a compaction-event monitoring gap
How the source tells it
Mostly straight reporting, with a mildly alarmed register from the headline and repeated emphasis on gaps, weaknesses and redactions.
- alarm loaded headline and repeated gap-and-weakness framing
- opacity the redactions and withheld report are foregrounded as part of the story