Decentralized-Training Doctrine
From the AI Research Compass wiki
Decentralized-Training Doctrine: Large models can and should be trained across many loosely connected machines owned by many parties, not only inside centralized datacenters.
Beliefs
Decentralized-Training Doctrine challenges the assumption that frontier-scale training requires a single datacenter with ultra-fast interconnect. Its adherents work on algorithms that tolerate slow, unreliable links between workers: infrequent synchronization, compressed gradients, local steps followed by occasional averaging, and fault tolerance for nodes that come and go. If these work at scale, training could be pooled from many independent clusters, labs or even volunteers, loosening the grip that compute concentration has on who gets to build large models.
It sits just right of the middle of the Language-World axis, since the techniques are modality-agnostic, and in the lower half toward Harness, because the substance is the distributed system that produces the model rather than any particular architecture. Its paradigm level is neutral: it trains mainstream architectures, so its revolution is in infrastructure rather than in the research paradigm.
Arguments
- Low-communication training methods that synchronize only every many local steps have trained language models with far less bandwidth than standard data parallelism, with modest quality loss.
- Gradient compression research showed that most gradient communication can be removed without harming convergence.
- Volunteer computing has a long track record in science, suggesting that pooled, heterogeneous hardware can deliver serious compute.
Critiques
- Decentralized runs still trail centralized ones in scale and efficiency, and the gap may widen as frontier training grows.
- Heterogeneous, untrusted participants raise problems of verification, poisoning and coordination that datacenters never face.
- Even decentralized training needs organizers, data and capital, so power may concentrate again at a different layer.
Representative works
- DiLoCo: Distributed Low-Communication Training of Language Models2023
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training2017
Relationships
Friends
Frenemies
How to get it
Land on this cell in the full or pro test: x ≈ +0.00, y ≈ −0.69, paradigm ≈ +0.00.