LLM Distillation and Compression // Guanhua Wang // #278

MLOps.community

Guanhua Wang is a Senior Researcher in DeepSpeed Team at Microsoft. Before Microsoft, Guanhua earned his Computer Science PhD from UC Berkeley. Domino: Communication-Free LLM Training Engine

// MLOps Podcast #278 with Guanhua "Alex" Wang, Senior Researcher at Microsoft.

// Abstract

Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs to parallelize and accelerate the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking the data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces of training and provides a generic strategy of fine-grained communication and computation overlapping. Extensive results show that compared with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.

// Bio

Guanhua Wang is a Senior Researcher in the DeepSpeed team at Microsoft. His research focuses on large-scale LLM training and serving. Previously, he led the ZeRO++ project at Microsoft, which helped reduce over half of model training time inside Microsoft and LinkedIn. He also led and was a major contributor to Microsoft Phi-3 model training. He holds a CS PhD from UC Berkeley, advised by Prof Ion Stoica.

// MLOps Swag/Merchhttps://shop.mlops.community/

// Related Links

Website: https://guanhuawang.github.io/

DeepSpeed hiring: https://www.microsoft.com/en-us/research/project/deepspeed/opportunities/

Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference: https://youtu.be/cntxC3g22oU

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Guanhua on LinkedIn: https://www.linkedin.com/in/guanhua-wang/

Timestamps:

[00:00] Guanhua's preferred coffee

[00:17] Takeaways

[01:36] Please like, share, leave a review, and subscribe to our MLOps channels!

[01:47] Phi model explanation

[06:29] Small Language Models optimization challenges

[07:29] DeepSpeed overview and benefits

[10:58] Crazy unimplemented crazy AI ideas

[17:15] Post-training vs QAT

[19:44] Quantization over distillation

[24:15] Using Lauras

[27:04] LLM scaling sweet spot

[28:28] Quantization techniques

[32:38] Domino overview

[38:02] Training performance benchmark

[42:44] Data dependency-breaking strategies

[49:14] Wrap up