What VSP-LLM is
The framework is described as combining visual speech processing with large language models (LLMs) to make the most of their context modelling ability, using a self-supervised visual speech model to map input video into the LLM input latent space for visual speech recognition and translation.
What you can do with it
- Perform visual speech recognition
- Perform visual speech translation
- Use a self-supervised visual speech model
- Train efficiently with a deduplication method and a low-rank adapter (LoRA)
Application areas
- Multilingual speech recognition
- Cross-language video content understanding
- Real-time speech translation
Example uses
- Use it for real-time speech translation in a multilingual setting
- Analyse video content, extract key information and produce a summary
- Support language learning in education with speech recognition
Pros & cons
✓ What we like
- Designed for both visual speech recognition and translation
- Efficient training through deduplication and LoRA
! What to watch out for
- Training relies on a self-supervised visual speech model and a LoRA adapter
- Input video is mapped into the LLM input latent space
FAQ
What tasks does it target?
The material names visual speech recognition and visual speech translation as multi-tasks.
How is it trained?
The framework proposes a deduplication method and a low-rank adapter (LoRA) for efficient training.
Last reviewed: 2026-11-15
More AI audio tools tools
View all →-
Accent Guesser Accent recognition from a recording Free tools Creator tools Audio Speech to text Education tools -
AccurateScribe.ai Transcription for audio and video Free tools Creator tools Video to text Audio Speech to text -
Adobe Podcast Browser podcast recording and cleanup Free tools Creator tools Audio Speech to text Audio editing -
Agilotext French-first transcription and summaries Audio Speech to text Podcast Meeting transcription Meeting notes -
AI Dubbing Short video dubbing without signup Free tools Video translation Audio Voiceover Content localization -
AI Mastering Automatic online mastering Free tools Freemium tools Audio Podcast