An AI assistant's reply travels through several distinct processing stages before it ever reaches you. Text is tokenized into sub-word units, a natural-language-understanding stage extracts intent and entities, a dialogue manager folds the result into a running conversation state, and a generative model then produces the reply one token at a time before it is rendered on whichever channel — text, speech or an image — the interface uses.
P(token_t | token_<t, state) — next-token prediction, conditioned on dialogue state
state_new = f(state_old, input) — dialogue-state update each turn
L = T_tok / (1 − ρ), ρ = λ · T_tok · N_tok — latency under queueing load
- Message rate (λ) — how often new messages arrive at the pipeline; a higher rate raises channel utilization ρ.
- Temperature — sampling temperature: higher values widen the token-probability distribution, producing longer, more varied replies (more tokens per message) at the cost of more visible routing jitter.
- Output channel — text, speech and vision share the same generator but diverge at the final hop, matching how a single LLM backend can serve a chat bubble, a synthesized voice or a captioned image.
- Latency follows a simple queueing identity: service time per message T_tok·N_tok stretches out by 1/(1−ρ) as the channel approaches saturation (ρ→1) — the same reason chat apps feel sluggish under heavy load.
Real-world relevance: production LLM serving stacks batch and schedule exactly these token streams, and the latency-vs-load curve here is the same shape engineers watch on their dashboards.