DeepSpeed Is Not Just ZeRO: Tensor, Sequence, and Expert Parallelism with ZeRO-Level Usability // TRAIN BRAIN

DeepSpeed Is Not Just ZeRO: Tensor, Sequence, and Expert Parallelism with ZeRO-Level Usability

Masahiro Tanaka will present “DeepSpeed Is Not Just ZeRO: Tensor, Sequence, and Expert Parallelism with ZeRO-Level Usability” at PyTorch Conference North America 2026.
The session will cover DeepSpeed’s AutoTP, AutoSP, and AutoEP capabilities for tensor, sequence, and expert parallelism, how these capabilities can be combined with ZeRO, and where each parallelism strategy fits. Masahiro will also walk through benchmark results from representative use cases.
Register for PyTorch Conference North America 2026: https://hubs.la/Q04v4SL60
#PyTorchCon

PyTorch

Welcome to the official PyTorch YouTube Channel. Learn about the latest PyTorch tutorials, new, and more. PyTorch is an open source machine learning framework that is used by both researchers and developers to build, train, and deploy ML systems that so...

Involved in the AI ecosystem? There's something for everyone at PyTorch Conference North America

DeepSpeed Is Not Just ZeRO: Tensor, Sequence, and Expert Parallelism with ZeRO-Level Usability

From torch.profiler to Hardware Cycles: A Practical Profiling Playbook

Extending TorchInductor with FlyDSL: A New MLIR-Native Backend for High-Performance GEMMs

What You Cannot Profile, You Cannot Optimize: Learning to Read PyTorch Traces

Build a neural network from scratch, train it, and optimize it for real-world AI infrastructure

Hardware-Aware AI: Building Agentic Systems from Cloud to Edge with PyTorch, ExecuTorch

Make Enterprise Agentic Inference Production-Ready with PyTorch and vLLM & the PyTorch Landscape

Scaling ExecuTorch & Disaggregating LLM Serving

A Tiered Cross-Repository CI Relay for Out-of-Tree Backends

Retrieval as Inference: Building a Production-Scale Torch-Native Retrieval Engine with Dhritiman Das

Federated Learning for Medical Imaging at PyTorch Conference 2026

PyTorch Con: Where Open Innovation in AI Happens

Empowering developers through collaboration, choice and open contributions

PyTorch can extend language modeling beyond classification to practical time-to-event prediction

Optimizing LLM Inference: Disaggregated Serving, PD Protocol, & KV Pinning

AI Agents for LLM Inference Runtimes on Edge Hardware

Optimizing Multi-Stage AI Pipelines: Prefix Caching for Faster Autoregressive Inference

Why PyTorch Foundation Ambassador Arun Bhandari thinks you should attend PyTorch Conference NA 2026

Building high-performance recommendation inference systems using PyTorch

Learn how PyreFly enables end-to-end static shape coverage for PyTorch models.

Learn how to grow a local PyTorch community at PyTorch Conference North America 2026.

Richard Liaw (Ray) PyTorch Conference North America 2025

Primus Tuning Hybrid Projection and Agentic Search for Distributed Training

PyTorch Conferences are where the open source AI community gathers to decide what's next

PyTorch Conference Europe 2026, Niles Burbank (AMD)

Meet the people behind the open source tools you use every day at PyTorchCon North America

Efficient Pretraining of LLMs in NVFP4 at PyTorchCon

Tackle the systems challenges of Agentic RL at PyTorchCon North America

PyTorch Conference 2025: Christian Jacobi (IBM)

PyTorch Conference 2025: Simon Mo (vLLM, Inferact)

PyTorch Conference 2025 Jana van Greunen (Meta)

Learn how to train more with less at PyTorch Conference North America 2026 this October

PyTorch Conference Europe 2026 – Official Highlights | Paris

PyTorch Conference brings together so many of the smartest people in the world

PyTorch Conference 2025 Tristan Rice (Meta)

PyTorch Conference North America 2025

What PyTorch Conference Europe 2026 Was Really Like – Official PyTorchCon EU Highlights | Paris

Why WideEP Inference Needs Data-Parallel-Aware Scheduling - Maroon Ayoub & Tyler Michael Smith

Lightning Talk: How DeepInverse Is Solving Imaging in Science and H... Andrew Wang & Minh Hai Nguyen

Write Once, Run Everywhere with Pytorch Transformers - Pedro Cuenca, Hugging Face

The Science and Practice of Open and Scalable LLM Evaluations - Grzegorz Chlebus, NVIDIA

Lightning Talk: Deep Learning in the Wild: Embedded PyTorch for... Taraqur Rahman & Owen O'Donnell

The Token Slice: Implementing Preemptive Scheduling Via Chunked Decod... Maroon Ayoub & Kellen Swain

Enabling State-of-the-art Asynchronous Execution in Torch.compile With CUDA Streams - Michael Lazos

Lightning Talk: ExecuTorch on Microcontrollers: Deploying PyTorch To... RJ Ascani & Matthias Cremon

Lightning Talk: TerraKit: Standardising AI-Ready Geospatial Data... Rosie Lickorish & Romeo Kienzler

Lightning Talk: Bayesian Neural Networks With Variational Inference in PyTorch - Lars Heyen

Lightning Talk: Why Your Forecasting Transformer Isn’t Working (And How To Fix It... Rosheen Naeem

Build PyTorch to Understand PyTorch - Vijay Janapa Reddi & Andrea Mattia Garavagno

Lightning Talk: Jigsaw: Domain and Tensor Parallelism for High-Resolution Inp... Deifilia Kieckhefen

Optimizing PyTorch on CPU-GPU Coherent Platforms - Matthias Jouanneaux, Nvidia

torch.compile and Diffusers: A Hands-On Guide to Peak Performance - Sayak Paul, Hugging Face

PyTorch Foundation Spotlight: Matt White

PyTorch Foundation Spotlight: Prabha Ganapathy (AWS)

PyTorch Foundation Spotlight: Christian Jacobi

PyTorch Foundation Spotlight: Simon Mo

PyTorch Foundation Spotlight: Richard Liaw

PyTorch Foundation Spotlight: Zhipeng Wang

PyTorch Foundation Spotlight: Joe Spisak

PyTorch Foundation Spotlight: Ankit Patel

PyTorch Foundation Spotlight: Red Hat

PyTorch Foundation Spotlight: Zesheng Zong PyTorch Ambassador

PyTorch Foundation Spotlight: Niles Burbank, AMD

PyTorch Foundation Welcomes Snowflake as a Premier Member

PyTorch Foundation Spotlight: Ankita Guha, PyTorch Ambassador

PyTorch Foundation Spotlight - Luca Antiga, Lightning AI

PyTorch Conference 2025 Recap

Behind Every AI SBOM Lies a Minefield: Here's How to Not Step... - Matthew Crawford & Jimmy Ahlberg

SPDX: Tackling System Risk in Modern Supply Chains - Kate Stewart & Gary O'Neall

The OpenMDW License Agreement: Simple, Permissive Terms for AI Model Materials - Steve Winslow

Sponsored Session: Lightning Talk: Accelerating Experimentation... - Niall Lyons & Luca Antiga

Using LlamaIndex to Build an Open-Source NotebookLM Alternative - Tuana Çelik, LlamaIndex

Our Journey With TorchTitan - Linsong Chu & Garrett Goon, IBM Research

Panel Discussion: The Future of Agents and Human-Agent Collaboration

PyTorch-Native Stack for Agents - Allen Wang & Davide Testuggine, Meta

Sponsored Session: Architecting AI: Efficient Models, Everywhere - Sharbani Roy, Arm

Sponsored Session: Lightning Talk: Optimizing Model Inference with PyTorch 2.0 - Devansh Ghatak

Keynote: Building an Open Agentic Future - Samuel Colvin, Founder, Pydantic

Open Source Model Performance Optimization With SGLang - Yineng Zhang, Together AI