Capsule Networks: Preserving Spatial Hierarchies

The Problem: Max-Pooling Throws Away Where Things Are

Convolutional neural networks build up recognition through layers of filters followed by pooling, most commonly max-pooling, which slides a small window over a feature map and keeps only the strongest activation in that window. This operation is what gives CNNs their prized translation invariance: a filter that fires on an eye will still fire even if the eye shifts a few pixels, because pooling simply reports that the eye-detector fired somewhere nearby. The cost is that pooling discards almost all information about exactly where within that window the feature occurred, and more importantly it discards the precise spatial relationship between different detected features. A CNN can look at a scrambled picture with eyes, a nose and a mouth arranged in the wrong places, or even a nose sitting above an eyebrow, and still confidently call it a face, because all it verified was that face-like parts were present somewhere in the image, not that they were correctly arranged relative to one another. Hinton argued that this is a fundamental architectural flaw rather than a minor quirk, since human vision is exquisitely sensitive to exactly this kind of part-whole spatial consistency.

What a Capsule Actually Is

A capsule replaces the single scalar output of a traditional neuron with a small group of neurons packaged as a vector. The length of that vector represents the probability that the entity the capsule detects is present, while the vector's orientation in its multi-dimensional space encodes the entity's instantiation parameters, things like precise pose, rotation, thickness, skew, or size. A traditional neuron can only say 'yes, an eye is roughly here, with this confidence.' A capsule can say 'an eye is here, tilted at this angle, at this scale, viewed from this perspective.' This matters because it lets the network reason about geometry explicitly. If a lower-level capsule detects an eye at a certain pose, and another detects a nose at a compatible pose, a face capsule can check whether those poses are mutually consistent with what a real face should look like, rather than just checking that both parts exist somewhere in the frame. This is a shift from asking 'what features are present' to asking 'do these features agree on a coherent object hypothesis,' which is the seed of everything dynamic routing builds on.

Dynamic Routing-by-Agreement

Instead of pooling, capsule networks connect layers through dynamic routing-by-agreement. Each lower-level capsule uses a learned transformation matrix to produce a prediction for what each higher-level capsule's pose vector should look like, essentially asking 'if I am really part of you, what would your output vector be?' If several lower-level capsules independently produce predictions that cluster tightly together, that agreement is strong evidence a coherent higher-level entity is present, and those capsules are routed to send most of their output to that higher-level capsule. This routing is not a fixed, learned weight the way a normal layer's connections are; it is recomputed dynamically for every single input through an iterative process. Over a handful of iterations, the coupling coefficients between a lower capsule and each candidate parent are adjusted upward when their predictions agree and downward when they conflict, so information flows preferentially along paths of consensus. In effect, lower-level capsules 'vote' for which higher-level entity they belong to, and the network settles into an interpretation where the votes cohere, rather than blindly forwarding the single strongest signal the way max-pooling does.

Why This Should Improve Robustness to Viewpoint Changes

Because capsules encode pose explicitly and routing depends on geometric agreement between predictions, capsule networks were designed to generalize across viewpoints far more gracefully than CNNs. A CNN trained mostly on upright, front-facing objects often struggles when the same object appears rotated, tilted, or viewed from an unfamiliar angle, because it has to relearn the pattern rather than simply account for a geometric transformation. A capsule network, in principle, can recognize a familiar object under a novel viewpoint because the individual part-capsules still detect the parts, just with different pose vectors, and the routing mechanism can still find consistent agreement among the transformed predictions, since the transformation matrices are built to model exactly this kind of geometric relationship. Hinton described this property as achieving something closer to true equivariance rather than mere invariance: the network's internal representation changes in a predictable, structured way as the viewpoint changes, instead of collapsing all variation into a single blurred-out signal the way pooling does.

Practical Tradeoffs That Have Limited Adoption

Despite the elegant theory, capsule networks have not displaced CNNs as the default architecture for large-scale vision tasks, and the reasons are mostly practical. The iterative routing procedure is significantly more computationally expensive than a simple pooling or convolution operation, and it scales poorly as the number of capsules and layers grows, making capsule networks slow and memory-hungry on the large, high-resolution datasets where modern vision systems are typically trained. Training dynamics can also be finicky, with routing iterations sometimes converging to poor agreements or requiring careful tuning of hyperparameters like the number of routing iterations. Perhaps most importantly, on large benchmark datasets like ImageNet, capsule networks have generally failed to match the accuracy of well-tuned CNNs or, more recently, transformer-based vision models, both of which have received far more engineering investment and hardware-level optimization. Capsule networks remain an active and influential research direction, and their core insight about explicitly modeling part-whole geometric relationships continues to inspire newer architectures, but as of today they occupy a niche role rather than the mainstream one Hinton originally envisioned for them.

Frequently asked questions

Try it live

Everything above runs in your browser — open Capsule Networks: Preserving Spatial Hierarchies and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.