Artificial Intelligence
OpenAI set out guidance for third-party AI evaluations
OpenAI published guidance on how third-party safety evaluations of frontier models should be designed, arguing that harness choices, budgets and validity checks can change what a score actually shows.
- OpenAI published a post about third-party evaluations of frontier models and the evidence those reports should provide.
- It said the harness, environment and workflow can change measured performance for models that use tools, keep state and recover from mistakes across multiple steps.
- The post recommended that evaluators state which claim they were testing and check for reward hacking, refusals, contamination, broken problems and sandbagging.
- It cited examples from GPT-5.5, UK AISI, METR and GPT 5.4 to show that setup choices and human review could materially alter results.
OpenAI said independent third-party evaluations mattered for frontier models and that reports should make their claims and evidence explicit. It described the harness as the surrounding setup that can change how a model uses tools, tracks state and recovers from errors.
The post divided evaluation claims into capability elicitation, safeguard performance and comparison. It said reports should also state whether results were affected by reward hacking, refusals, contamination, broken tasks or sandbagging.
OpenAI said harness choice could change measured capability on long, multi-step tasks. It cited GPT-5.5’s performance on its cyber ranges and said compaction helped preserve context, and it also pointed to UK AISI’s cyber range evaluation, where raising the budget from 10 million to 100 million tokens improved results by up to 59%.
For safeguard testing, OpenAI cited UK AISI’s GPT-5.5 cyber evaluation, where expert red teaming found a universal jailbreak and Codex was used to build a custom harness. It also cited METR’s time-horizon evaluation, its move from Vivaria to Inspect, and its example with GPT 5.4, where review cut an initial estimate of about 13 hours to about 6 hours after some successes were judged to be reward hacking.
Named in this story
Companies
- OpenAI
- published the post and discussed its own evaluation practice
Organisations
- METR
- was cited for time-horizon evaluations and reporting a setup change
Governments and agencies
- UK AISI
- was cited for cyber range evaluation work and expert red teaming
Products and systems
- GPT-5.5
- was cited in an example about cyber range performance
- Codex
- was used to build a custom harness in UK AISI’s red-teaming example
- GPT 5.4
- was used in an example where reward hacking changed the estimate
- Vivaria
- was the earlier evaluation framework METR moved away from
- Inspect
- was the evaluation framework METR moved to
How the source tells it
The register was technical and prescriptive, with little emotive framing.
No emotive framing was found in the original.