Skip to content

Zero Bubble Pipeline Parallelism

A scheduling strategy for zero pipeline bubbles

Visit official site

What Zero Bubble Pipeline Parallelism is

The method is described as a scheduling strategy that achieves zero pipeline bubbles for large-scale distributed training under synchronous semantics.

What you can do with it

  • Reach zero pipeline bubbles in synchronous training
  • Use hand-designed novel pipeline schedules
  • Find a schedule automatically from model config and memory limits

Model notes

  • Backward computation is split into input and parameter gradient parts
  • A technique bypasses synchronization during the optimizer step
  • The material reports up to 23 percent higher throughput than 1F1B under similar memory limits

Pros & cons

✓ What we like

  • A technique bypasses synchronization during the optimizer step
  • The algorithm accounts for a specific model configuration and memory limit

! What to watch out for

  • Backward computation is split into input-gradient and parameter-gradient parts
  • Reported throughput figures are tied to stated memory constraints

FAQ

What is the key idea?

The material says backward computation is split into two parts, one for input gradients and one for parameter gradients.

What throughput is reported?

The material states up to 23 percent over 1F1B under similar memory limits, and up to 31 percent when constraints are relaxed.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review