Instead of only looking at nearby pixels like a CNN filter, a transformer lets every patch directly weigh the relevance of every other patch.
attn(i,j) = softmax(Q_i . K_j / sqrt(d))
output_i = sum_j attn(i,j) * V_j
- Image patches — fixed-size image tiles treated as tokens the transformer attends across.
- Attention connectivity — fraction of all patch pairs actively exchanging attention weight.
- Attention sharpness — how peaked (versus diffuse) the softmax attention distribution is.
- Update threshold — attention weight needed before a connection visibly lights up.
Vision transformers now match or beat CNNs on large-scale image recognition benchmarks by letting every patch directly attend to every other, unlike a CNN's local filters.