Artificial Intelligence
OpenAI reports coding gains in low-resource languages
OpenAI said its models scored far better on a company-run coding benchmark and on a COBOL migration test, but the article said the results had not been independently replicated.
- OpenAI reported that GPT-6 Astra averaged 87.4% on MultiLangBench across 20 languages, compared with 26.8% for GPT-5.6 Sol.
- The company also reported gains in COBOL, Fortran and Visual Basic on a separate migration evaluation.
- The article said the evaluations were designed and run by OpenAI, which also sells the models, and that there was no independent replication.
- It said the benchmark repository documented 24,000 converted SQLite tests and recommended a 272-test subset, but did not identify which set produced the headline results.
OpenAI said GPT-6 Astra averaged 87.4% on MultiLangBench across 20 programming languages. The article compared that with 26.8% for GPT-5.6 Sol, and said the benchmark asked models to build a SQLite-compatible database engine from scratch.
The source also reported gains in COBOL, Fortran and Visual Basic. In the COBOL evaluation, models were asked to translate programs into C++, Java, Kotlin, Python or Rust while keeping observable behaviour the same.
The article said the released repository converted 24,000 SQLite tests and also recommended a 272-test subset for regular use. It said the report did not state which tests underpinned the headline scores, how many full runs were made, or whether uncertainty was calculated.
The source said OpenAI designed and ran the evaluations and sold the models being tested. It added that there was no independent replication and that the tests did not measure production defects, security incidents, maintenance burden or worker outcomes.
Named in this story
Companies
- OpenAI
- reported the benchmark results and ran the evaluations
Products and systems
- GPT-6 Astra
- was the model reported to score 87.4% on MultiLangBench
- GPT-5.6 Sol
- was the comparison model in the reported benchmark results
- GPT-6.1 Sol
- was the model reported in the COBOL migration test
- MultiLangBench
- was the coding benchmark used for the SQLite-style task
- SQLite
- provided the database target and test suite used in the benchmark
- COBOL
- was one of the lower-resource languages assessed for migration
- Fortran
- was one of the languages where OpenAI reported a gain
- Visual Basic
- was one of the languages where OpenAI reported a gain
How the source tells it
The piece is measured and cautionary, with repeated methodological caveats tempering the performance claims.
- cautious optimism improved benchmark scores are presented alongside explicit limits on what the tests do not prove
- caution the article repeatedly highlights missing methodology, absent replication and unmeasured operational effects