The image is split into a grid of patches, each with its own embedding vector (like ViT/CLIP patch tokens). A query token β a question ("where is the cat?") in VQA mode, or a generic caption token in Captioning mode β scores its similarity against every patch, then a softmax turns those scores into cross-attention weights:
Ξ±_i = softmax(s_i / T) = exp(s_i / T) / Ξ£_j exp(s_j / T)
s_i β similarity between the query and patch i
T β temperature: larger models learn sharper, lower-T attention
Ξ±_i β fraction of "attention mass" placed on patch i
- Task β VQA scores patches against one target object; Captioning scores every patch against each object class's intrinsic salience and reports what's most attended overall.
- Query object (VQA only) β which region the model is asked to locate.
- Model scale β bigger models sharpen attention (lower T, tighter grounding radius), the way GPT-4V/Gemini Vision/LLaVA outperform smaller CLIP-style encoders at precise grounding.
- Noise / occlusion β randomly zeroes or jitters patch similarity, mimicking blur, glare or partial occlusion; watch grounding accuracy fall as it rises.
- Entropy βΞ£ Ξ±_iΒ·logβΞ±_i measures how spread out the attention is β near 0 bits means one patch dominates, higher means the model is "unsure" where to look.
Real systems: CLIP aligns image/text embeddings for retrieval, Florence-2 and LLaVA run this attention inside a full vision-language transformer, and GPT-4V/Gemini Vision use it to answer open-ended questions or write captions grounded in the actual pixels.