Skip to content

SpeechGPT2

An end-to-end speech dialogue model that senses and expresses emotion

Visit official site

What SpeechGPT2 is

The model is described as an end-to-end speech dialogue language model developed at the School of Computer Science, Fudan University, able to sense and express emotion and give speech responses in multiple styles. The supplementary note calls it a fully end-to-end human-like speech dialogue model built on a 750bps ultra-low-bitrate speech codec and initialized from MIMO-LM.

What you can do with it

  • Sense and express emotion in spoken dialogue
  • Respond in styles such as rap, drama, robot, funny and whisper
  • Model semantic and acoustic information with a 750bps speech codec
  • Generate one second of speech in 25 autoregressive decoding steps

Getting started

  • Visit the GitHub page for the technical report and code
  • Read the technical report to understand the architecture and functions
  • Download and install the required software dependencies
  • Configure model parameters and training data from the documentation

What to watch out for

  • The current system is turn-based while a full-duplex real-time version is in development
  • The material notes limits in computing and data resources
  • Noise robustness in speech understanding and stability of generated voice quality are stated as weak points
  • The technical report, code and model weights are planned for future open source release

Pros & cons

✓ What we like

  • Multiple speaking styles follow context and human instructions
  • One second of speech takes 25 autoregressive decoding steps

! What to watch out for

  • It is still a turn-based dialogue system
  • Noise robustness and voice quality stability are noted as weak points

FAQ

Who developed it?

The material names the School of Computer Science, Fudan University.

Is everything released?

The material states a plan to open source the technical report, code and model weights in the future.

Last reviewed: 2026-11-15

More AI audio tools tools

View all →

How we review