Skip to content

Diffusion as Shader

Control video generation with 3D tracking input

Visit official site

What diffusion-as-shader is

The model is described as a video generation control model that uses 3D tracking video to drive several control tasks within one architecture.

What you can do with it

  • Prepare a text prompt, image or animated mesh
  • Preprocess inputs such as depth maps and object masks
  • Generate a video from the 3D tracking input

What to watch out for

  • Depth Pro and SAM are named preprocessing tools
  • Camera paths are specified by the user
  • Temporal consistency is the stated goal

Pros & cons

✓ What we like

  • 3D track videos improve temporal consistency
  • Fine-tuning with little data is described

! What to watch out for

  • Input prompts, images or meshes are prepared by the user
  • Preprocessing such as depth maps comes first

FAQ

What acts as control input?

The material names 3D tracking videos as the control input.

Who developed it?

The material credits research teams at several universities.

Last reviewed: 2026-11-15

More AI video generation tools

View all →

How we review