An ASR front end (Whisper and similar models) turns a raw waveform into a mel spectrogram via the short-time Fourier transform, framing the signal and taking the magnitude spectrum of each windowed segment:
X(m,k) = Σ_n x[n]·w[n−mH]·e^(−j2πkn/N)
The spectrogram frames are then fed as tokens into a transformer encoder, where self-attention lets every time step weigh every other time step when building its representation:
Attention(Q,K,V) = softmax(QK^T / √d_k)·V
- ASR (Whisper) — encoder output is decoded autoregressively into text tokens; noisier audio and faster speech both push down decoding confidence.
- TTS (ElevenLabs, Bark) — the same encoder-decoder shape runs in reverse: text/codec tokens condition a decoder that predicts a mel spectrogram, which a vocoder turns back into a waveform.
- Neural codec LMs — voice cloning and real-time translation models treat compressed audio codes as just another token stream, so one transformer backbone can read speech, write speech, or translate between languages.
- RTF — real-time factor: processing time ÷ audio duration. RTF < 1 means the model transcribes/synthesizes faster than the audio plays, which is what makes live captioning and voice assistants possible.