Skip to content

VSP-LLM

A framework joining visual speech processing with large language models

Visit official site

What VSP-LLM is

The framework is described as combining visual speech processing with large language models (LLMs) to make the most of their context modelling ability, using a self-supervised visual speech model to map input video into the LLM input latent space for visual speech recognition and translation.

What you can do with it

  • Perform visual speech recognition
  • Perform visual speech translation
  • Use a self-supervised visual speech model
  • Train efficiently with a deduplication method and a low-rank adapter (LoRA)

Application areas

  • Multilingual speech recognition
  • Cross-language video content understanding
  • Real-time speech translation

Example uses

  • Use it for real-time speech translation in a multilingual setting
  • Analyse video content, extract key information and produce a summary
  • Support language learning in education with speech recognition

Pros & cons

✓ What we like

  • Designed for both visual speech recognition and translation
  • Efficient training through deduplication and LoRA

! What to watch out for

  • Training relies on a self-supervised visual speech model and a LoRA adapter
  • Input video is mapped into the LLM input latent space

FAQ

What tasks does it target?

The material names visual speech recognition and visual speech translation as multi-tasks.

How is it trained?

The framework proposes a deduplication method and a low-rank adapter (LoRA) for efficient training.

Last reviewed: 2026-11-15

More AI audio tools tools

View all →

How we review