Skip to content

SpeechGPT

A multimodal language model with cross-modal conversation

Visit official site

What SpeechGPT is

The model is described as a multimodal language model with intrinsic cross-modal conversational ability. It perceives and generates multimodal content and follows multimodal human instructions.

What you can do with it

  • Perceive and generate multimodal content
  • Generate speech through an information chain
  • Simulate human communication with a multimodal multi-agent system
  • Apply unified speech tokenisation

Model notes

  • SpeechGPT handles multimodal dialogue generation
  • SpeechGPT-Gen extends an information chain for speech generation
  • SpeechAgents models human communication with multiple agents
  • SpeechTokenizer is a unified tokenizer for speech language models

Pros & cons

✓ What we like

  • Multimodal perception and generation in one model
  • A unified speech tokenizer for speech language models

! What to watch out for

  • Use cases centre on the named model variants
  • The listed abilities are carried by separate named components

FAQ

What kind of model is it?

The material describes a multimodal language model with intrinsic cross-modal conversational ability.

Which components are named?

SpeechGPT-Gen, SpeechAgents and SpeechTokenizer are named alongside SpeechGPT.

Last reviewed: 2026-11-15

More AI audio tools tools

View all →

How we review