Galvatron: An Automatic Distributed Training System for Efficient Large-Scale Transformer Training - Xinyi Liu & Fangcheng Fu, Peking University
Galvatron is a PyTorch-native, open-source framework for the efficient distributed training of large-scale Transformer models, with specialized optimizations for automatic hybrid parallelism strategies. Given a Transformer model, Galvatron first employs PyTorch profiler to analyze the model execution workload characteristics, creating a precise cost model. Then, Galvatron uses decision trees and dynamic programming to automatically deduce the best combination of parallelism dimensions for each model layer, covering data, tensor, pipeline, sharded data, sequence parallelism, and recomputation. Finally, Galvatron leverages PyTorch features like FSDP and checkpointing—enjoying its seamless integration with various accelerators like NVIDIA GPUs and Ascend NPUs—to deploy and train the model. As an open-source project with comprehensive documentation, Galvatron is designed to be user-friendly, enabling easy integration with minimal code changes. Collaborations from both academia and industry, such as BAAI, Huawei, and ByteDance, highlight its practical applications and superior efficiency compared to existing frameworks. Discover more at https://github.com/PKU-DAIR/Hetu-Galvatron.