Skip to content

YaFSDP

Distributed data parallelism tuned for transformers

Visit official site

What YaFSDP is

YaFSDP is described as a distributed data parallel framework designed for transformer-style neural networks, reducing communication and memory overhead. The material states it performs better under high memory pressure.

What you can do with it

  • Pretrain large language models efficiently
  • Follow examples for supervised fine-tuning
  • Receive custom event notifications

Application scenarios

  • Pretrain models from 7B to 70B parameters
  • Train across 64 to 256 devices
  • Train with 2048 to 8192 token sequences

Pros & cons

✓ What we like

  • Memory and communication overhead is reduced
  • Custom event notifications are supported

! What to watch out for

  • Setup follows a Docker environment from the examples
  • It is built on an NVIDIA PyTorch image with patches

FAQ

How much faster is pretraining?

The material states pretraining is 20% faster than conventional FSDP.

What sequence lengths are named?

The material lists training with 2048 to 8192 token sequences.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review