Fully sharded data parallel is a distributed deep learning training method that divides a model's parameters, gradients, and optimizer states across multiple GPUs. Each GPU stores only a shard instead of a full model copy, allowing larger models to train within available memory. It trades some additional GPU-to-GPU communication for substantially lower per-device memory requirements.
Fully Sharded Data Parallel
What Is Fully Sharded Data Parallel?
Fully sharded data parallel, or FSDP, is a distributed deep learning training method that splits a model's training state across GPUs. Rather than storing a complete model copy on every GPU, each data-parallel worker stores a shard, reducing memory use per device.
Training state includes parameters, gradients, and optimizer states used to calculate updates. FSDP helps when a model cannot fit in one GPU's memory.
How FSDP Works During Training
FSDP rebuilds only the model block needed for the next calculation instead of keeping the full model in memory throughout training.
- FSDP divides parameters, gradients, and optimizer states into shards across GPUs.
- Before a model block runs, GPUs use all-gather communication to assemble its parameters temporarily.
- Each GPU processes a different portion of training data during forward and backward passes.
- GPUs use reduce-scatter to combine gradients and return each GPU's gradient shard.
- Each GPU updates its local optimizer shard and releases temporary full parameters when no longer needed.
FSDP vs. Distributed Data Parallel
Both are data-parallel approaches. FSDP is not pipeline parallelism, which assigns successive model stages to different GPUs.
| Area | FSDP | Distributed Data Parallel |
|---|---|---|
| Model-state storage | Sharded across GPUs | Full copy on every GPU |
| Memory per GPU | Lower for model states | Higher |
| Communication | All-gather and reduce-scatter | Gradient all-reduce |
| Best fit | Models that strain GPU memory | Models that already fit comfortably |
Benefits of Fully Sharded Data Parallel
FSDP makes model-state memory a shared resource across workers.
- It lowers per-GPU memory use for parameters, gradients, and optimizer states.
- It can train or fine-tune models too large for one device.
- It supports mixed precision and optional CPU offloading where hardware and workloads allow.
- Model-state storage scales as more workers are added.
Practical Limits of FSDP
FSDP saves memory but does not automatically make training faster.
- All-gather and reduce-scatter add communication overhead, especially on slow interconnects.
- Activation memory and batch size can still exceed GPU memory.
- Uneven layer sizes, checkpoint saving, and distributed debugging need careful configuration.
When to Use FSDP and How to Choose a Sharding Strategy
Choose FSDP when model state, rather than data volume, is the memory bottleneck.
- Use ordinary DDP when the model fits on each GPU and simplicity matters most.
- Test FSDP when memory is tight and the cluster has fast GPU-to-GPU networking.
- Pair it with activation checkpointing or mixed precision when activations remain the constraint.
- Plan checkpoint formats before long runs because sharded and full checkpoints support different recovery needs.
FSDP in AI Application Development
FSDP is usually used to train or fine-tune a foundation model, not to run an application day to day. Teams commonly use the resulting model through an API or inference service and connect it to application data with a vector database and a data pipeline. With an AI app builder or AI web app builder, Emergent helps teams turn trained or hosted models into products.
Start Building
on Emergent today
on Emergent today
Start Building
