Skip to content

LookOnceToHear

Pick a speaker by looking, then hear only that voice

Visit official site

What LookOnceToHear is

The system is described as a smart headset interaction system that lets users pick a target speaker by looking at the person for a few seconds. It performs real-time speech extraction with synthesized audio mixtures, head-related transfer functions and binaural room impulse responses, and received an honorable mention at CHI 2024.

What you can do with it

  • Select a target speaker by looking at them for a few seconds
  • Generate audio mixtures with the Scaper toolkit
  • Use the self-contained dataset and .jams specification files for training
  • Evaluate real-time speech extraction with the provided checkpoints

Application scenarios

  • Choose a specific speaker to hear during a meeting
  • Help hearing-impaired people focus on conversation in noisy public places
  • Distinguish and extract multiple sound sources in audio analysis research

Getting started

  • Download and extract the provided .zip file into the data/ directory
  • Run the command to start the training process
  • Use Scaper generate_from_jams on the .jams specification files to build audio mixtures
  • Download and load the target speech hearing model checkpoints for evaluation

Pros & cons

✓ What we like

  • Model checkpoints are provided for training and evaluation
  • Real-time speech extraction works in noisy environments

! What to watch out for

  • Training starts from the provided .zip data and .jams specification files
  • The workflow relies on Scaper to generate audio mixtures

FAQ

How is a speaker selected?

The material states a user looks at the target speaker for a few seconds to choose the voice to hear.

What is supplied for training?

A self-contained dataset, .jams specification files, Scaper-based mixture generation and model checkpoints are listed.

Last reviewed: 2026-11-15

More AI audio tools tools

View all →

How we review