Skip to yearly menu bar Skip to main content

Poster - Recorded Presentation
Workshop: Machine Learning for Systems

Automatic Discovery of Composite SPMD Partitioning Strategies in PartIR

Sami Alabed · Dominik Grewe · Juliana Franco · Bart Chrzaszcz · Tom Natan · Tamara Norman · Norman Rink · Dimitrios Vytiniotis · Michael Schaarschmidt


Large neural network models are commonly trained through a combination of advanced parallelism strategies in a single program, multiple data (SPMD) paradigm. For example, training large transformer models requires combining data, model, and pipeline partitioning; and optimizer sharding techniques. However, identifying efficient combinations for many model architectures and accelerator systems requires significant manual analysis. In this work, we present an automatic partitioner that identifies these combinations through a goal-oriented search. Our key findings are that a Monte Carlo Tree Search-based partitioner leveraging partition-specific compiler analysis directly into the search and guided goals matches expert-level strategies for various models.

Chat is not available.