SGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving - Liangsheng Yin, Shanghai Jiao Tong University / LMSYS
SGLang is an open-source Large Language Model (LLM) inference system that is highly efficient and widely adopted by many companies like xAI, Nvidia and AMD. In this session, I will introduce some key features of SGLang, including the design and implementation of PD disaggregation, large-scale expert parallelism and data parallelism for DeepSeek models, hierarchical KV cache offloading, and highly efficient speculative decoding. I will also share some insights into the future development of the SGLang community.