A multimodal LLM fuses two separate token streams: an image encoder splits the picture into a grid of patches and projects each into a feature vector (a "visual token"), while a text encoder turns each word into a language token in the same-dimensional space. Cross-attention lets each text token query every visual token: it computes a similarity score against every patch, turns those scores into a probability distribution, and blends the patches together weighted by that distribution — pulling the relevant part of the image into the token's representation before the language model reasons over it.
score(t,p) = (q_t · k_p) / √d
attn(t,·) = softmax( score(t,·) / T )
fused_t = Σ_p attn(t,p) · v_p
- Image patches — grid resolution the image encoder splits the picture into; more patches give finer-grained visual tokens for the text to attend over.
- Softmax temperature (T) — scales the attention logits; low T makes attention sharp and confident (near one-hot), high T spreads it evenly across many patches.
- Embedding noise — corrupts how well each token's vector matches its true concept, simulating an under-trained alignment between the two modalities; push it up and attention starts landing on the wrong patches.
- Text tokens — how many language tokens are attending into the shared visual space at once.
This is the mechanism behind models like CLIP-aligned vision-language transformers: a shared embedding space where visual and language tokens are compared directly, rather than the single-modality self-attention used when a transformer only reads text.