Auditory Scene Analysis: How We Hear Separate Sounds

One mixture, many sources

Whatever is actually happening in a room — a violin, a fridge hum, two people talking — the eardrum only ever receives one thing: a single pressure waveform, the sum of every source added together in the air. There is no channel separation at the cochlea. Yet listeners effortlessly hear a violin, a hum and two voices as four distinct objects. The psychologist Albert Bregman named this decomposition problem auditory scene analysis (ASA) in his 1990 book of the same title, drawing a deliberate parallel to the Gestalt grouping laws that explain how the eye organizes a jumble of lines into separate shapes — except the ear has to do it in time as well as in frequency.

Bregman split the machinery into two kinds of process. Primitive grouping is fast, automatic and present from infancy — it relies on statistical regularities that almost always hold for a single real-world sound source. Schema-based grouping is learned and knowledge-driven — it lets a fluent listener track a familiar melody or a known voice through a mixture even when the primitive cues are ambiguous. Most of what makes ASA tractable to study, and to model, is the primitive layer.

Simultaneous grouping: what belongs together right now

At a single instant, the cochlea reports a spectrum of many frequency components, and the first job is deciding which ones came from the same source. The strongest cue is harmonicity: the partials of a periodic sound sit at exact integer multiples of a fundamental frequency, and the auditory system has dedicated machinery for detecting that pattern and fusing all of the matching partials into one pitch, one timbre, one object. A close second is onset and offset synchrony — components that start and stop within a few milliseconds of each other are assumed to share a cause, because two unrelated sources almost never begin at exactly the same instant. Common amplitude and frequency modulation (components that wobble together) and, more weakly, shared spatial location from interaural time and level differences round out the set.

Sequential grouping and the van Noorden trade-off

Grouping across time is what turns isolated notes into a melody or a stream of phonemes into a sentence, and it is governed largely by frequency proximity and speed. The classic demonstration alternates two tones, high (A) and low (B), in a repeating ABAB pattern. Leo van Noorden mapped the perceptual outcome in 1975 as a trade-off between the frequency separation of A and B and the tempo of the sequence:

ΔF small, tempo slow   → one coherent stream, heard as a "galloping" ABAB rhythm
ΔF large, tempo fast   → two segregated isochronous streams (A-A-A... / B-B-B...)
                          the relative timing of A and B becomes hard to judge
"build-up": streaming strength grows over roughly 4-10 s of continuous listening,
            even though the stimulus itself never changes

The build-up effect is the strangest part of the phenomenon: two constant, unchanging tones alternating at a fixed rate sound more and more segregated the longer you listen, an effect widely attributed to adaptation in frequency-selective neurons or to the accumulation of attentional bias toward one of the two streams. It resets almost completely after a short silence or a switch of attention.

Why the cocktail party works

Colin Cherry coined the term "cocktail party effect" in 1953 for the ability to follow one conversation in a room full of others, and ASA supplies the mechanism underneath the label: primitive grouping segregates the mixture into candidate streams — one per talker, roughly — before attention ever gets involved, and selective attention then simply chooses which already-segregated stream to follow. The same grouping machinery explains phonemic restoration, where a word with a syllable masked by a cough is heard as complete because the auditory system infers the missing energy belongs to the ongoing speech stream, and it explains why a Bach cello suite can imply two independent melodic lines from a single instrument playing one note at a time — the wide, alternating leaps between a high and low register are, acoustically, exactly the van Noorden stimulus that produces stream segregation.

Modelling it computationally

Computational auditory scene analysis (CASA) systems try to reproduce these cues algorithmically: a cochleagram or short-time Fourier front end estimates the time-frequency energy, autocorrelation or cepstral analysis estimates periodicity and harmonicity, onset detectors flag synchronous energy, and interaural cues estimate direction. Oscillatory correlation networks, notably the work of DeLiang Wang and Guy Brown, used coupled neural oscillators whose phase-locking mimicked perceptual grouping directly. These hand-engineered pipelines were the standard approach in hearing aids and robust speech recognition for two decades. Modern source separation has largely moved to deep networks trained end-to-end (deep clustering, permutation-invariant training, conv-TasNet), which learn their own grouping statistics from data — but the target they are implicitly learning to reproduce is still the same set of cues Bregman catalogued, and ASA experiments remain the standard way to check whether a separation system's errors look human-like or merely numerically small.

Frequently asked questions

Try it live

Everything above runs in your browser — open Auditory Scene Analysis and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.