Core Automation, an AI startup founded by senior staff from OpenAI and Google, published a post describing distributed training and shared results from its own experiments. Distributed training means training a model across multiple data centers, sometimes with different network connections and accelerator types. As the technique matures, it could allow more organizations to develop large models without operating single, massive hyperscale data centers.
What they did
According to the company, “a mixture-of-experts model with roughly 32B total parameters, 32x sparsity was trained across two clusters about 9,000 miles apart.” Notably, the clusters used different accelerators and setups: one site used GB300 accelerators and the other used B300, with different parallelism schemes and different total batch sizes.
Concretely, they used 160 B300 GPUs in the United States and 160 GB300 GPUs in Asia. Training synchronization was performed every 50 steps, but not by syncing the whole model at once: “rather than syncing the whole model at once, we synced it fragment by fragment, staggered across each 50-step window.” Each site used a variant of Distributed Shampoo for optimization.
This run is not the single largest model trained with distributed techniques (for comparison, Covenant reported a 72B model), but it is notable that Core Automation—previously relatively quiet—chose to publish these experiments. That signals the perceived importance of distributed training for future large-scale runs and highlights the limited availability of large contiguous compute capacity for newer startups.
Toward orbital compute
Part of the authors’ interest comes from a long-term vision of training at orbital scale: they foresee scenarios where large-scale AI development could occur in space using similar techniques. They suggest an ambitious first step would be to pretrain a small model across a few satellites and compare it to a ground baseline, while noting that a trillion-parameter orbital run is much further off.
Why this matters
The company argues the main bottlenecks for deploying accelerated compute are not just chips but infrastructure: “The biggest bottleneck in deployment of accelerated compute is not the chips themselves but power, grid equipment and land.” They add that many organizations are held back not by raw compute quantity but by the ability to turn separate islands of compute into a single, valuable frontier training run.
Conclusions
Core Automation’s experiments show it is technically feasible to link heterogeneous, geographically distant clusters into a single training job, which may help address access constraints to large-scale compute. Their public disclosure underscores growing interest in distributed training approaches and even speculative ideas about orbital compute as potential paths to scaling future AI systems.
(Source: Core Automation — The Hitchhiker’s Guide to Training on Compute Islands.)
