What benchmarks did OpenAI use to measure progress?

OpenAI developed two evaluation tools: MultiLangBench, which tests whether a model can build a SQLite-compatible database engine from scratch in 20 programming languages, and a COBOL migration evaluation that tests whether a model can migrate existing programs to another language while preserving their behavior.

Is the benchmark available for others to use?

Yes, the MultiLangBench evaluation setup and test suite are published on GitHub.

Which model performed better on the tests?

GPT-6 Luna outperformed GPT-5.6 Sol on the SQLite test in every language tested and offers a lower-cost option, according to OpenAI.

AI Business & IndustryAI Watch (Impress)2m agoOct 20, 2026, 16:00 JST

OpenAI's GPT-6 Luna beats GPT-5.6 Sol on legacy-language coding at lower cost

Send on LINE

3 Key Points

What happened
OpenAI said its latest model, GPT-6 Luna, outperformed GPT-5.6 Sol on SQLite-compatible database engine tests in every language tested, at lower cost.
Why it matters
A newer model that delivers higher accuracy at lower cost gives companies a cheaper option for building and migrating software in poorly documented languages, according to OpenAI.

WHO IT HITSEnterprise IT teams maintaining or migrating legacy systems, such as banking and procurement software written in COBOL, could get a cheaper, more accurate AI-assisted option; OpenAI's claims are based on its own benchmarks and have not been independently verified in the body.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

OpenAI's announcement focuses on a practical problem: many critical systems still run on older languages like COBOL and Fortran, but there is little training data for AI models in those languages, making migration difficult. To measure progress, OpenAI built two complementary evaluations. MultiLangBench checks whether a model can construct a SQLite-compatible database engine from scratch across 20 programming languages. The COBOL migration evaluation tests whether a model can move an existing program to another language while keeping it working. OpenAI says its latest model, GPT-6 Luna, performed better than GPT-5.6 Sol on the SQLite test in every language tested and does so at lower cost. The company has published the MultiLangBench setup and test suite on GitHub, which may allow outside teams to examine the methodology. The claims are OpenAI's own and have not been independently verified in the body.

FAQ

What benchmarks did OpenAI use to measure progress?: OpenAI developed two evaluation tools: MultiLangBench, which tests whether a model can build a SQLite-compatible database engine from scratch in 20 programming languages, and a COBOL migration evaluation that tests whether a model can migrate existing programs to another language while preserving their behavior.
Is the benchmark available for others to use?: Yes, the MultiLangBench evaluation setup and test suite are published on GitHub.
Which model performed better on the tests?: GPT-6 Luna outperformed GPT-5.6 Sol on the SQLite test in every language tested and offers a lower-cost option, according to OpenAI.

AI Watch (Impress)Read Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.