Which Models and Frameworks Have Adopted DeepSpeed?

DeepSpeed has been used to train large language models including Megatron-Turing NLG (530B), Jurassic-1 (178B), BLOOM (176B), and GLM (130B), and it is integrated into popular open-source frameworks such as Transformers, Accelerate, Lightning, and MosaicML. These adoption examples are the site's own list of models it is aware of, so treat them as evidence of real-world use at scale rather than an exhaustive registry.

Large models trained with DeepSpeed

The DeepSpeed site lists the following models as examples of large-scale training with the library:

The spread matters more than any single entry: it covers dense autoregressive language models, a 530B-parameter model, and models in the 5B–20B range. That range is the practical signal — DeepSpeed is presented as usable from mid-size training runs up to the largest published models of their time. The site notes these were the most powerful language models "at the time of this writing," so read the list as historical evidence of capability, not a current leaderboard.

Framework integrations

DeepSpeed is integrated with several open-source deep learning frameworks, each with its own documentation path on the site:

• Transformers with DeepSpeed — Hugging Face Transformers
• Accelerate with DeepSpeed — Hugging Face Accelerate
• Lightning with DeepSpeed — PyTorch Lightning
• MosaicML with DeepSpeed — MosaicML

If you already train inside one of these frameworks, the integration means you can adopt DeepSpeed's optimization features without rewriting your training loop around a separate launcher. The documentation links are the place to check the exact configuration surface for your framework version.

Why these adoptions exist: the underlying techniques

The adoption list is backed by a set of system innovations the site describes as a "confluence" that made large-scale training effective, efficient, and easier to use:

• ZeRO — memory optimization for distributed training
• 3D-Parallelism — combining data, pipeline, and tensor parallelism
• DeepSpeed-MoE — mixture-of-experts training
• ZeRO-Infinity — extending memory beyond a single device

Recent news items show the library is still evolving along the same axes: a 2026/05 note on using the Muon optimizer with DeepSpeed, a 2026/05 system DMA (SDMA) feature for ZeRO-3 that offloads collectives off compute units on AMD GPUs, 2025/12 Core API updates adding PyTorch-style backward and low-precision master states, and 2025/10 SuperOffload for large-scale LLM training on superchips. ZenFlow (2025/08) and Arctic Long Sequence Training (2025/06) address offloading stalls and multi-million-token sequences respectively.

Connection to Microsoft's AI at Scale

DeepSpeed is described as an integral part of Microsoft's AI at Scale initiative, which aims to enable next-generation AI capabilities at scale. This explains the project's direction: it is developed as infrastructure for very large model training, and the model and framework adoptions above are the public evidence that the infrastructure works in practice.

How to use this list when deciding

• If you are choosing a training stack for a model in the billions of parameters, the presence of 100B–530B models in the list is the relevant signal; check the specific technique (ZeRO stage, offload, parallelism mode) against your hardware rather than the model names alone.
• If you already use Transformers, Accelerate, Lightning, or MosaicML, start from that framework's DeepSpeed documentation page — the integration is the shortest path.
• If you want to add your model to the list, the site invites submissions via a pull request.
• If you plan to contribute code, note that most contributions require agreeing to a Developer Certificate of Origin (DCO), and DCOs are per-commit — each commit needs its own sign-off.