Skip to content

LatentSync

Lip sync from audio with a latent diffusion model

Visit official site

What LatentSync is

The framework is described as a lip-sync system from ByteDance that models audio-visual links with a latent diffusion model. It uses Stable Diffusion directly and avoids an intermediate motion representation.

What you can do with it

  • Model audio-visual correspondence with diffusion
  • Improve temporal consistency with TREPA

Application scenarios

  • A virtual host gets lip movements from speech
  • An animation studio matches lip animation to dubbing
  • An effects team repairs lip sync in a shot

Pros & cons

✓ What we like

  • A full data pipeline covers restoration, resampling and face alignment
  • U-Net and SyncNet training scripts are open

! What to watch out for

  • Environment setup and checkpoints come first
  • Inference is run through a script with tunable parameters

FAQ

What models does it build on?

The material names Stable Diffusion and an audio-conditioned latent diffusion approach.

Is the code available?

The material states training and inference scripts and model checkpoints are open.

Last reviewed: 2026-11-15

More AI video generation tools

View all →

How we review