Each token gets an embedding vector, then passes through a stack of Transformer encoder layers. Inside every layer, every token attends to every other token in the same layer: it scores how relevant each other token's Key is to its own Query, turns those scores into a probability distribution with softmax, and mixes the Values accordingly.
Attention(Q,K,V) = softmax(QK^T / √d_k) · V
y = softmax(W · pool(H_L) + b)
- Sentence picker — swaps the input tokens (and the toy ground-truth labels behind the demo).
- Task head — the same encoder output
H_L is pooled and fed into a different softmax classifier: 3-way Sentiment, per-token Named-Entity tags, or 4-way Intent (NLU).
- Encoder layers — how many self-attention + feed-forward blocks the tokens pass through before pooling; deeper stacks mean more mixing between tokens and higher simulated latency.
- Attention temperature — scales the softmax logits everywhere (attention and the task head): low temperature sharpens onto a few connections and one confident label, high temperature spreads attention and confidence out.
This is the same computational shape behind BERT (encoder-only, used for NLU tasks like sentiment and NER) and GPT (decoder-only, used for NLG/generation): stacked self-attention layers over sub-word tokens, ending in a task-specific classifier head.