SeedFlood: Toward Scalable Decentralized LLM Fine-tuning via Zeroth Order Optimization

This work presents a new approach to decentralized LLM finetuning---SeedFlood---designed to scale for large models across complex network topologies and achieve global consensus with negligible communication cost. Traditional methods suffer from high communication costs that grow with model size, while information decay over network hops renders global consensus inefficient. SeedFlood takes a significant departure from these practices by exploiting the seed-reconstructible structure of zeroth-order updates and effectively making the messages near-zero in size, allowing them to be flooded to every client in the network, and thereby, enhancing scalability of decentralized training. Consequently, SeedFlood enables training in regimes previously considered impractical, such as billion-parameter models distributed across hundreds of clients. Our experiments on decentralized LLM fine-tuning demonstrate that SeedFlood consistently outperforms standard baselines in both communication efficiency and generalization performance, and even achieves results comparable to first-order methods in large-scale settings.