Recent model updates deliver substantially stronger capabilities for building and migrating software in low‑resource programming languages such as COBOL, Fortran and Visual Basic. These languages offer less publicly available code and documentation than widely used languages like Python or JavaScript, yet they support critical infrastructure and essential services. For example, COBOL underpins more than 40% of online banking systems, 80% of in‑person credit card transactions and 95% of ATM transactions, while Fortran is used in weather forecasting and nuclear reactor safety analysis.
The work was guided by feedback from developers and enterprise customers reporting that coding models often struggle with these languages. Maintaining and modernizing such software is frequently challenging: documentation can be incomplete, original developers may have left, and important business rules may only exist in code. Whether teams are fixing bugs or migrating systems to another language, they need to preserve the behavior the systems depend on.
Evaluations: MultiLangBench and COBOL migration
To measure progress, two evaluations were developed. MultiLangBench tests whether a model can build a SQLite‑compatible database engine from scratch across 20 programming languages. The COBOL migration evaluation tests whether a model can migrate existing COBOL programs to another language while preserving their observable behavior. Together, these evaluations cover two complementary skills: building new software and preserving what existing software does.
The latest models show substantial improvements on both evaluations, while also lowering cost.
MultiLangBench results
On MultiLangBench, GPT‑6 Astra achieves an average test pass rate of 87.4% across the 20 languages, up from 26.8% for GPT‑5.6 Sol. Every tested language improved; examples include COBOL (10.9% → 72.1%), OCaml (25.6% → 85.8%), Fortran (21.4% → 83.5%) and Visual Basic (27.5% → 89.4%). Astra scored above 83% in 19 of the 20 languages, with COBOL remaining the hardest at 72%.
GPT‑6 Luna provides a lower‑cost option and also improves on GPT‑5.6 Sol in every language. At maximum reasoning effort it reaches an average test pass rate of 40.3%. Improvements include COBOL (10.9% → 17.1%), OCaml (25.6% → 34.1%), Fortran (21.4% → 31.4%) and Visual Basic (27.5% → 41.6%). The estimated average API cost for GPT‑6 Luna was $0.77 per sample versus $3.48 for GPT‑5.6 Sol, about 78% lower.
MultiLangBench asks a model to implement a SQLite‑compatible engine in a specified language: organizing a codebase, managing state, handling errors and adding features while preserving existing functionality. To compare implementations across languages, database files serve as a shared interface: the candidate program receives an initial database and a SQL query and must produce a modified database that is compared against the reference SQLite implementation. Tests were adapted from SQLite’s existing test suite, and the evaluation reports both test pass rate and API cost to measure how much working functionality a model can build during an extended coding session.
The MultiLangBench setup and test suite have been released on GitHub together with per‑language results, allowing developers to inspect the tests, review performance for languages they use, and identify gaps relevant to their work.
COBOL migration evaluation
The second evaluation asks models to migrate COBOL programs to C++, Java, Kotlin, Python or Rust while preserving program behavior. Successful migration requires understanding both business logic and COBOL’s language‑specific behavior. COBOL programs often process financial transactions and business records where details like rounding rules or missing‑record handling encode business requirements. For instance, COBOL’s fixed‑point arithmetic discards excess decimal digits unless rounding is explicitly requested; a rewrite that implicitly rounds those values can run without errors but produce different financial results.
To verify behavior preservation, the evaluation uses hidden tests that require byte‑for‑byte parity on observable outputs; a migration is considered successful only if all tests pass. These behavior‑preserving capabilities are useful for teams modernizing legacy systems, although applying such migrations in production still requires expert review and testing against a system’s business rules and dependencies.
On this evaluation, GPT‑6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared with GPT‑5.6 Sol’s 5.4% at the same setting, while costing 61% of the API cost. At higher reasoning effort GPT‑6.1 Sol reached 97.9% accuracy, bringing the evaluation close to saturation. These results indicate progress toward more accurate and affordable code migration for teams updating critical software.
Goals and next steps
These evaluations, together with direct feedback from developers and enterprise customers, help identify where models are improving and where shortcomings remain. The stated goal is to enable models to apply software engineering skills across more languages: understanding unfamiliar repositories, planning changes, using build tools, running tests and debugging failures.
The latest models are available through the API for coding tasks in the languages teams use, and feedback is requested on which languages and workflows should receive further focus.
