Two recent projects run on Department of Energy (DOE) systems demonstrate how multiple national‑lab supercomputers can be combined to train and fine‑tune large language models across sites. The efforts show practical approaches to federated or cross‑facility training while highlighting both operational challenges and remaining scale gaps compared with industry.
Tri‑laboratory pretraining on NNSA systems
In one project involving National Nuclear Security Administration (NNSA) systems, researchers used compute at Sandia National Laboratories, Lawrence Livermore National Laboratory (LLNL) and Los Alamos National Laboratory (LANL) to pretrain a LLaMA‑based model. The setup included LLNL’s El Capitan system (AMD MI300A) alongside NVIDIA H100 GPUs at SNL and LANL.
A critical constraint for this work was that access‑restricted data — for example, NNSA‑related or nuclear‑weapons‑associated data — could not leave its originating site. The team performed backend training using torchtitan and evaluated multiple federated training frameworks, including NVFlare, Flower, and APPFL. They pretrained LLaMA 3.x models on post‑training‑cutoff arXiv data at scales of 1 billion, 8 billion and 70 billion parameters.
According to the authors, “This workflow has been made production‑ready and deployed on larger classified machines including El Capitan, and subsequently used to train models on proprietary data spread across the three NNSA labs.” In other words, the pipeline was hardened for operational use and applied to proprietary datasets distributed across the three NNSA sites.
Cross‑facility fine‑tuning on ASCR systems
A second effort used Office of Advanced Scientific Computing Research (ASCR) systems to perform federated fine‑tuning. The team fine‑tuned LLaMA‑2 7B on SMolInstruct, a chemistry instruction‑tuning dataset, leveraging roughly 1,700 GPUs from Intel, NVIDIA and AMD.
Those GPUs were distributed across three labs and four large supercomputers: Argonne National Laboratory’s Aurora and Polaris, Oak Ridge National Laboratory’s Frontier, and Berkeley Lab’s Perlmutter.
Who conducted the work
Researchers on these projects were affiliated with Oak Ridge National Laboratory, Sandia National Laboratories, Argonne National Laboratory, Flower Labs, Arizona State University and the Ontario Brain Institute.
Why this matters
The papers illustrate how governments can pool existing high‑performance computing assets to perform non‑trivial AI work, including on sensitive data that must remain site‑local. That capability could be relevant for national security, scientific missions, and other government needs.
At the same time, the authors note the current scale is far smaller than the industry frontier — roughly two orders of magnitude lower by one estimate — meaning that to match leading commercial models, governments would need to interconnect far larger numbers of chips (on the order of ~100,000 distinct chips or more) and continue scaling in the future.
Nevertheless, these pilots represent an important operational step: they have produced production‑ready workflows and practical experience in cross‑institutional federated training.
Further reading
See “From Pilots to Production: Lessons in Cross‑Institutional Federated Training and Artificial Intelligence for Science” (arXiv) for full technical details.
