How a Large Online Retailer Built a Recommendation Engine: A Machine Learning Case Study
A detailed case study of how a major e-commerce marketplace used collaborative filtering, content-based recommendations and personalisation to lift average order value and conversion.
The business problem behind the project
By the late 2010s a large Eastern European online marketplace selling more than five million SKUs faced a familiar e-commerce problem: growth in average order value had stalled even as traffic kept climbing. Site-wide conversion sat at roughly 1.8%, well below what the merchandising team believed was achievable, and about 30% of completed orders contained only a single item.
Internal research pointed to a specific cause. With millions of listings on the site, shoppers were spending too long searching for what they wanted and leaving before finding related items they might also have bought. The existing "customers also bought" widgets were static, hand-curated lists that did not adapt to an individual shopper's behaviour, so they under-performed against what a learned, personalised system could do.
This is a useful pattern to recognise in your own business: the underlying pain was not "we lack AI" but a measurable, specific symptom (low items-per-order, flat average order value) with a plausible mechanical cause (poor product discovery at scale). That specificity is what made the eventual project fundable and measurable.
What the retailer built: a layered recommendation system
Rather than a single "recommendation algorithm," the team shipped four distinct surfaces, rolled out incrementally over about a year:
- Personalised homepage — the homepage layout adapted per visitor based on browsing history, seasonality and current promotions, rather than showing the same curated grid to everyone.
- "Similar items" recommendations — a hybrid of collaborative filtering (customers who bought A also bought B) and content-based similarity (matching products on shared attributes such as category, brand and specifications), combined to cover both popular and long-tail products.
- Basket completion prompts — at checkout, the system suggested complementary items (a laptop purchase triggering suggestions for a case, mouse or antivirus subscription), which is a much narrower and higher-intent recommendation problem than homepage discovery.
- Behaviour-triggered email — "you viewed this but didn't buy" campaigns with personalised product picks, timed per recipient using a model of each user's typical engagement window rather than a single blanket send time.
The stack was built on Python and TensorFlow for model training, Apache Spark for processing catalogue-scale data, and cloud infrastructure for serving recommendations at low latency. Every change was validated with A/B testing rather than shipped on faith — a discipline that turns out to be one of the strongest predictors of whether a recommendation project actually moves revenue.
The team and the investment
The core build team was intentionally small: two ML engineers, one data scientist, one backend developer and a product owner. Development cost was estimated at roughly $200,000, with an additional $15,000 per month for infrastructure and ongoing support once live — a useful benchmark for a mid-sized retailer scoping a similar project, though your own numbers will depend heavily on catalogue size and traffic.
Two technical obstacles dominated the build. First, serving relevant recommendations across five million SKUs in real time required real engineering investment in indexing and caching — naive nearest-neighbour lookups do not scale to that catalogue size. Second, the team had to solve the classic cold-start problem: a first-time visitor has no browsing history to personalise against. Their solution combined demographic signals available at first touch with fallback recommendations drawn from currently popular items, gradually replacing that fallback with personalised results as the visitor generated more signal.
There was also an organisational obstacle that is easy to underestimate: the marketing team initially resisted, worried that an automated system would replace their hand-curated merchandising. Retrospectively, the technical lead said involving marketing earlier in the design process — rather than presenting the system as a finished replacement — would have reduced friction significantly.
Measured results
Comparing the twelve months before rollout with the first full year after, the retailer reported the following headline metrics:
| Metric | Before | After | Change |
|---|---|---|---|
| Average order value | Baseline | +30% | Higher basket value per order |
| Site-wide conversion rate | 1.8% | 2.4% | +33% |
| Items per order | 1.3 | 1.9 | +46% |
| Returning-visitor rate | 22% | 31% | +41% |
| Average session duration | 4.2 min | 6.8 min | +62% |
Company estimates attributed roughly $45 million in incremental annual revenue to the recommendation system, against a first-year cost in the low hundreds of thousands of dollars — implying a payback period measured in weeks to a couple of months rather than years. Even allowing for optimistic internal attribution (isolating "revenue caused by recommendations" from organic growth is genuinely hard, a caveat the team itself raised), the scale of the lift is consistent with published industry benchmarks for mature recommendation systems, which commonly report double-digit percentage gains in conversion and basket size.
Adapting the system to sudden demand shifts
One instructive part of this case study is how the system handled a period of abrupt, unpredictable change in customer demand — a scenario every business eventually faces in some form, whether from a supply shock, a regulatory change, a competitor exit or a macroeconomic disruption. When the retailer's usual demand patterns shifted rapidly (a sudden swing away from discretionary electronics toward essential goods), the existing recommendation models, trained on historical purchase patterns, began producing stale, less relevant suggestions almost overnight.
The team's response is a useful template for building resilience into any personalisation or forecasting system: they retrained models on a much shorter, more recent data window rather than the usual multi-year history, prioritising responsiveness over long-run stability during the disruption. Because part of their infrastructure had also become unreliable, they accelerated a planned migration to fully cloud-hosted infrastructure, trading some cost efficiency for resilience. The broader lesson — build models and infrastructure that can be retrained quickly on short notice, and don't assume last year's demand patterns will hold — is directly transferable to demand forecasting, inventory planning and marketing personalisation in any sector exposed to sudden disruption.
What worked, what the team would change
What worked: starting with the simplest recommendation surface ("similar items") and adding complexity only once it was proven; investing in cloud infrastructure before it was strictly necessary, rather than after a scaling crisis; and treating every UI change as an A/B test rather than a judgement call.
What the team would do differently: involve the marketing and merchandising teams earlier, since much of the early internal resistance came from feeling bypassed rather than from any technical objection; and invest more heavily in mobile experience sooner, given that roughly 60% of traffic was mobile — a segment the first release under-optimised for.
The retailer's chief technology officer summarised the underlying lesson well: recommendation systems succeed or fail less on model sophistication and more on whether they make each shopper feel the store was built specifically for them — personalisation at scale is the real product, and the algorithm is just the mechanism.
Frequently Asked Questions
Do I need five million products to justify a recommendation engine?
No. The core techniques — collaborative filtering, content-based similarity and basket-completion prompts — pay off with catalogues of a few hundred SKUs and up, though the engineering complexity of "similar items at scale" only becomes a genuine bottleneck once you have hundreds of thousands of items. Smaller retailers can start with simpler, rules-based or off-the-shelf recommendation tools before building anything custom.
What is the cold-start problem and why does it matter?
Cold start refers to the difficulty of personalising recommendations for a new visitor or a new product that has no interaction history yet. It matters because a large share of e-commerce traffic is first-time visitors; a common solution is to fall back on popularity- or demographic-based recommendations until enough behavioural data accumulates.
How is collaborative filtering different from content-based recommendations?
Collaborative filtering recommends items based on the behaviour of similar users ("customers who bought A also bought B"), while content-based recommendations match items by shared attributes (category, brand, specifications). Hybrid systems combine both because collaborative filtering struggles with new or low-traffic products, and content-based matching alone misses cross-category buying patterns.
How should a business measure whether a recommendation system is actually working?
Track business metrics — conversion rate, items per order, average order value, returning-visitor rate — not just model accuracy metrics. A/B testing each change against a control group is essential, since it is otherwise very difficult to separate the effect of recommendations from underlying organic growth or seasonality.
What should a business do when a demand shock makes its models stop working?
Shorten the training window to prioritise recent data over long historical averages, retrain more frequently than usual, and make sure infrastructure can be redeployed quickly if part of it becomes unreliable. Treating rapid retraining as a standing operational capability, rather than a one-off emergency response, is what separates businesses that adapt quickly from those that do not.