How Recommendation Engines Decide What You See

Collaborative filtering, embeddings, and the exploration-exploitation trade-off that actually drive personalised feeds and product recommendations.

▶ Open the simulation

Collaborative filtering: people like you liked this

The founding idea behind recommendation systems, popularised at scale by early Amazon and the Netflix Prize competition in the mid-2000s, is collaborative filtering: rather than trying to understand why an item is good (which would require deep content analysis of every product), the system looks at patterns of agreement across many users' behaviour and lets those patterns do the work. User-based collaborative filtering finds other users whose rating or purchase history correlates closely with yours, and recommends items they liked that you haven't seen yet, on the logic that people who agreed on twenty things are likely to agree on a twenty-first. Item-based collaborative filtering, which turned out to scale better for large catalogues and became the more common production approach, flips this around: it computes similarity between items based on how similarly users rated or interacted with them (a shirt and a matching pair of trousers that get purchased together frequently look similar in this sense, even though nothing about their pixels or descriptions was ever compared), and recommends items similar to ones you've already engaged with.

The core mathematical technique underlying most collaborative filtering at scale is matrix factorisation: imagine a giant, extremely sparse matrix with one row per user and one column per item, where most entries are empty because any individual user has only rated or interacted with a tiny fraction of the catalogue. Matrix factorisation approximates that huge sparse matrix as the product of two much smaller, dense matrices — a user-factor matrix and an item-factor matrix, each with, say, 50 or 100 latent dimensions — trained so that multiplying a user's row by an item's column reproduces the known ratings as closely as possible. Those latent dimensions aren't hand-labelled (dimension 17 isn't explicitly "loves period dramas"), but they end up capturing genuine underlying taste structure purely from the pattern of who-liked-what, and the trained factorisation can then predict a rating for any user-item pair that was never actually observed, which is exactly what a recommendation is.

Embeddings: turning items and users into points in a shared space

Modern recommendation systems generalise this latent-dimension idea explicitly through embeddings, learned vector representations that place users, items, and often contextual signals (time of day, device, recent search query) all as points in the same continuous high-dimensional space, positioned so that things that are behaviourally similar end up close together geometrically. An embedding for a specific film isn't hand-designed; it's learned end-to-end as part of training a neural network on the actual task (predicting whether a user will click, watch, or purchase), and the network discovers, purely from data, that certain dimensions of that space usefully separate action films from documentaries, or discovers subtler groupings a human curator would never have thought to label, like a cluster of visually slow, dialogue-heavy films that a particular demographic tends to watch late at night.

Once users and items live in the same space, recommendation partly reduces to a geometric nearest-neighbour search: given a user's current embedding (itself often computed as a function of their recent interaction history, updated continuously as they browse), find the items whose embeddings sit closest to it, using efficient approximate nearest-neighbour algorithms (like HNSW or product quantisation) because doing an exact distance calculation against millions or billions of item embeddings for every single recommendation request would be far too slow for a real-time system. This embedding-space framing is also what lets modern systems blend collaborative signal (behavioural co-occurrence) with content signal (an item's actual attributes, like a product's category, or a video's transcript) into a single unified representation, addressing collaborative filtering's classic weakness — the cold-start problem, where a brand-new item with no interaction history yet has nothing for the collaborative signal to work with, but can still be placed sensibly in embedding space using its content attributes alone until real behavioural data accumulates.

The exploration-exploitation trade-off behind every feed

A recommendation system that only ever showed you the single item it currently predicts you're most likely to engage with would actually perform worse over time than one that occasionally shows you something more uncertain, and the reason is a structural problem borrowed directly from reinforcement learning: exploitation (showing what the model is currently confident you'll like) maximises today's expected engagement, but exploration (showing something the model is less certain about) is what generates the new data needed to correct the model's mistakes and discover that you actually like something outside your established pattern. A system that purely exploits converges towards a narrow, self-reinforcing loop — it recommends what it already believes you like, you engage with that because it's genuinely relevant, and that engagement reinforces the model's existing belief, meaning the system never gets a chance to learn that you'd have loved something it never showed you, a well-documented dynamic often blamed for filter bubbles and stagnating recommendation diversity over time.

Production systems handle this trade-off with techniques adapted from the multi-armed bandit literature. Thompson sampling, one common approach, maintains a probability distribution over how good each candidate item is likely to be for a given user rather than a single point estimate, and picks which item to show by literally sampling from those distributions — items the model is very confident about get a tight distribution clustered around a high value and get shown often, while items the model is uncertain about have a wide distribution that occasionally samples a high value purely by chance, giving them an occasional, principled opportunity to be shown and for the system to learn from the result. This is a fundamentally different mechanism from simply injecting random noise or random recommendations, because the exploration is targeted specifically at the items and users where the model's uncertainty is highest and the potential information gain is largest, rather than being uniformly wasteful across the whole catalogue.

Ranking, re-ranking and the layers most users never see

What actually happens between a request landing on a recommendation service and a list of items appearing on screen is rarely a single model making a single pass over the whole catalogue — for a catalogue of tens of millions of items, scoring every single one for every request would be far too slow. Production systems almost universally use a two-stage (sometimes three-stage) funnel: a fast, relatively simple candidate generation stage (often exactly the embedding nearest-neighbour lookup described above) narrows the full catalogue down to perhaps a few hundred or a few thousand plausible candidates in milliseconds, and then a much more expensive, more accurate ranking model — often a gradient-boosted tree ensemble or a deeper neural network that can afford to consider dozens of detailed features per candidate, like recency, diversity relative to what's already been shown, and predicted watch-time rather than just predicted click — re-scores and orders just that shortlist.

A final re-ranking pass frequently applies business logic that has nothing to do with predicted relevance at all: diversity constraints (don't show five nearly-identical items in a row even if they all score highly, because that produces a worse experience despite each individual prediction being accurate), freshness boosts for new content that hasn't accumulated enough interaction data to rank well on pure relevance yet, and explicit business rules (promoting certain categories, respecting content policy exclusions). This layered structure is why the phrase "the algorithm" oversimplifies what's actually a pipeline of several distinct models and rule sets working at different scales and speeds, each optimising a slightly different objective, stacked together to produce the single list of items a user actually sees.

Frequently Asked Questions

What's the difference between user-based and item-based collaborative filtering?

User-based filtering finds other users with similar behaviour and recommends what they liked; item-based filtering computes similarity directly between items based on shared interaction patterns and recommends items similar to ones you've already engaged with, which scales better for large catalogues and became the more common production approach.

How does an embedding-based system handle a brand-new item with no interaction history?

This is the cold-start problem; systems that blend collaborative signal with content attributes can place a new item sensibly in the shared embedding space using its own features (category, description, metadata) even before any behavioural data has accumulated for it.

Why don't recommendation systems just always show the single highest-predicted item?

Doing so would create a self-reinforcing loop where the system only ever learns from the narrow set of items it already believes you like, contributing to filter bubbles; deliberately exploring uncertain recommendations, for instance via Thompson sampling, generates the data needed to correct the model's blind spots over time.

Why do large-scale recommendation systems use a multi-stage pipeline instead of one model?

Scoring every item in a catalogue of millions with an expensive, highly accurate model for every request would be too slow; a fast candidate generation stage first narrows the catalogue to a shortlist, which a more expensive ranking model then scores and orders in detail.