Google DeepMind's DiLoCo lets AI training survive hardware failures across data centers — maintains 90% training speed vs. 40% for older methods

Google DeepMind unveiled DiLoCo (decoupled distributed low-communication), a new architecture that trains large AI models across multiple data centers by splitting the work into independent 'islands' that communicate asynchronously instead of waiting for every part to sync perfectly. The system was tested on Google's Gemma 4 models, training a 12-billion parameter model across four U.S. regions using only 2-5 Gb/s of internet bandwidth.

When hardware fails or networks slow down, DiLoCo keeps training at 90% speed—the time spent on actual learning work—versus traditional elastic methods that drop to 40%. It does this by letting fast learners and slow learners work at their own pace, then merging their results intelligently without forcing everyone to wait for the slowest machine. Training also runs 20× faster than conventional synchronization because computation happens in longer blocks instead of frequent stop-and-wait cycles.

For companies building large language models (AI systems that understand and generate text), this means they can use older generations of Google's TPU chips (specialized AI processors) mixed with newer ones in the same training job, squeezing more value from existing hardware before replacing it. Stranded compute resources in regional data centers—chips sitting idle—can now contribute to training jobs, turning expensive unused capacity into productive work.

The technology is open in principle (published in a research paper), and Google is already embedding similar ideas into Virgo, a networking system it announced this week designed to connect up to 134,000 AI chips for its AI Hypercomputer platform.