Artificial Intelligence
Research finds agent teams cost more than they improve results
Vals AI found that teams of AI agents scored only one statistically significant win over solo runs across four tests, while token costs rose to as much as 5.1 times.
- Vals AI tested GPT-6 Sol and Claude Opus 5.5 on Vibe Code Bench both alone and in teams at medium and maximum reasoning effort.
- The team setups cost between 1.8 and 5.1 times as much as single-agent runs.
- Across four solo-versus-team comparisons, only GPT-6 Sol at medium reasoning produced a statistically significant gain, which was 7.3 points.
- Anthropic said its own Opus 5.5 tests showed gains flattening as agent counts increased, while token use kept rising in separate ProgramBench runs.
Vals AI compared GPT-6 Sol and Claude Opus 5.5 on Vibe Code Bench in solo and team configurations at two reasoning levels. The team versions cost between 1.8 and 5.1 times more than the single-agent runs.
Only one of the four comparisons showed a statistically significant improvement: GPT-6 Sol at medium reasoning, where the team scored 7.3 points higher. At maximum reasoning, neither model gained much from the team setup.
Anthropic reported that its own Opus 5.5 tests showed quality gains tapering as more agents were added. It said teams of ten and 100 reached similar performance only slightly farther apart after 24 hours, and that faster runs in separate ProgramBench tests used more tokens.
OpenAI researcher Noam Brown said on the Dwarkesh Podcast that multi-agent systems mainly improved speed rather than quality. He said four agents solved tasks in half the time at twice the cost, and Eric Provencher warned that coordination problems made agent swarms poor value.
Named in this story
People
- Noam Brown
- said multi-agent systems mainly improved speed rather than quality
- Eric Provencher
- warned against agent swarms because coordination breaks down
Companies
- Vals AI
- tested GPT-6 Sol and Claude Opus 5.5 on the benchmark
- Anthropic
- reported internal tests showing gains flattening as agent numbers rose
- OpenAI
- employed the two named speakers quoted in the article
Products and systems
- GPT-6 Sol
- was tested alone and in teams
- Claude Opus 5.5
- was tested alone and in teams
- Vibe Code Bench
- was the benchmark used for the comparisons
- ProgramBench
- was used in Anthropic's separate speed tests
- Fable 5.1
- was compared on theorem-proving and knowledge-base tasks
How the source tells it
Mostly straight reporting, with a repeated waste-and-efficiency frame that makes the findings read as a warning.
- alarm the cost-versus-gain framing turns the benchmark results into a warning about waste