Learning to Rank (LTR) uses machine learning to order search results by relevance. Unlike classification or regression, LTR optimizes the ordering of items directly. It powers search engines, recommendation systems, and any application where showing the best items first matters.

Where It Is Used

LTR is essential wherever result ordering affects user experience:

Web search: Google, Bing, DuckDuckGo
E-commerce: Product search and recommendations
Job platforms: Job and candidate matching
Streaming: Netflix, Spotify content ordering
Social media: Feed ranking, content discovery
Enterprise: Document search, knowledge bases

Products: Elasticsearch LTR, Solr LTR, TensorFlow Ranking, XGBoost rankers.

Typical Data You Will See

Aspect	Typical Values
Queries per day	10K to 1B
Candidates per query	100 to 10,000
Features per item	50 to 500
Relevance labels	Binary, graded (0-4), or click data
Training set size	100K to 100M query-item pairs

Data sources: - Query logs with clicks and conversions - Human relevance judgments - Implicit feedback (dwell time, scrolls) - Item features (text, categories, popularity) - Query features (terms, intent, context)

What You Do in Practice

Define success - What makes a good ranking? CTR? Conversions?
Collect training data - Click logs, human labels, or both
Generate candidates - Initial retrieval (BM25, embedding search)
Extract features - Query-item, query-only, item-only features
Choose approach - Pointwise, pairwise, or listwise
Train ranker - Optimize ranking metric
Evaluate offline - NDCG, MAP, MRR on held-out data
A/B test online - Compare to baseline on real traffic
Monitor and iterate - Track metrics, refresh models

Baseline First

Start with BM25 or simple scoring before ML ranking:

# BM25 baseline
from rank_bm25 import BM25Okapi

tokenized_docs = [doc.split() for doc in documents]
bm25 = BM25Okapi(tokenized_docs)
scores = bm25.get_scores(query.split())
ranked_docs = sorted(zip(documents, scores), key=lambda x: -x[1])

Why baseline first: - Establishes performance floor - Fast to implement - Often surprisingly competitive - Helps identify where ML helps most

Move to ML ranking when baseline limitations are clear (personalization, complex relevance).

Ranking Approaches

Approach	Loss Type	Pros	Cons
Pointwise	Regression/classification on relevance	Simple, fast	Ignores list structure
Pairwise	Compares pairs (which is better?)	Natural for clicks	Doesn't optimize list metric
Listwise	Optimizes list-level metric directly	Best metric alignment	Complex, slower

Common models:

Model	Type	Notes
LambdaMART	Pairwise/listwise	Industry standard, gradient boosting
LambdaRank	Pairwise	Neural network based
ListNet	Listwise	Probability distribution over permutations
RankNet	Pairwise	Neural pairwise comparison
BERT-based	Various	Cross-encoder for reranking

LambdaMART example with LightGBM:

import lightgbm as lgb

train_data = lgb.Dataset(
    X_train, y_train,
    group=query_groups,  # Number of items per query
)

params = {
    'objective': 'lambdarank',
    'metric': 'ndcg',
    'ndcg_eval_at': [5, 10],
    'num_leaves': 31,
    'learning_rate': 0.05
}

model = lgb.train(params, train_data, num_boost_round=100)

Feature Engineering

Ranking features fall into three categories:

Category	Examples
Query features	Query length, query intent, query frequency
Item features	Item popularity, recency, category, price
Query-item features	BM25 score, embedding similarity, click history

Common features: - Text match scores (BM25, TF-IDF) - Semantic similarity (embedding dot product) - Popularity signals (views, clicks, purchases) - Freshness (age of content) - Quality signals (ratings, reviews) - Personalization (user history with item/category) - Position bias features (for debiasing)

Feature engineering tips: - Combine text and behavioral features - Include position-aware features if using click data - Log-transform count features - Create query-specific aggregates (avg price clicked)

Evaluation and Metrics

Metric	What It Measures	Range
NDCG@k	Ranking quality with graded relevance	0 to 1
MAP	Precision at each relevant item	0 to 1
MRR	Position of first relevant result	0 to 1
Precision@k	Fraction relevant in top k	0 to 1
Recall@k	Fraction of relevant items in top k	0 to 1

NDCG (Normalized Discounted Cumulative Gain):

DCG@k = Σ (2^rel_i - 1) / log2(i + 1)  for i in 1..k
NDCG@k = DCG@k / IDCG@k  (where IDCG is ideal DCG)

Offline vs online evaluation: - Offline: Test on held-out labeled data - Online: A/B test on real users - Interleaving: Mix results from two rankers

Pitfalls: - Position bias in click data - Offline metrics may not correlate with online - Need stratified train/test split by query

Failure Modes and Debugging Checklist

[ ] Position bias: Clicks favor top positions regardless of relevance
[ ] Cold start: New items have no behavioral features
[ ] Query drift: User queries change over time
[ ] Feature leakage: Using future information in features
[ ] Overfitting to clicks: Model learns bias, not relevance
[ ] Diversity collapse: All top results are too similar
[ ] Latency: Complex models too slow for real-time

Production Checklist

Before deployment: - [ ] Offline metrics beat baseline significantly - [ ] Model latency acceptable (p99 < SLA) - [ ] Feature pipeline tested end-to-end - [ ] A/B test framework ready - [ ] Fallback to baseline if model fails

After deployment: - [ ] Online metrics tracked (CTR, conversions) - [ ] Model freshness maintained (retraining schedule) - [ ] Feature drift monitored - [ ] Edge cases logged (zero results, slow queries) - [ ] Position bias correction applied if using clicks

Scaling considerations: - [ ] Two-stage ranking (retrieval → reranking) - [ ] Feature caching for hot items - [ ] Model quantization for latency - [ ] Query segmentation for different models

Two-Stage Ranking Architecture

Most production systems use two stages:

Query → Candidate Retrieval (1000s) → Feature Extraction → 
ML Reranker (top 100) → Business Rules → Final Results (10-20)

Stage 1 - Retrieval: - Fast, approximate (BM25, ANN) - High recall, lower precision - Returns hundreds to thousands

Stage 2 - Reranking: - Slower, precise ML model - Takes top-k from retrieval - Optimizes for ranking quality

Tools and Stack

Category	Tools
Gradient boosting	LightGBM, XGBoost, CatBoost
Deep learning	TensorFlow Ranking, PyTorch
Search integration	Elasticsearch LTR, Solr LTR
Evaluation	RankLib, custom metrics
Feature stores	Feast, Tecton

Example Mini-Case

Scenario: E-commerce site wants to improve product search results beyond keyword matching.

Approach: 1. Baseline: Elasticsearch BM25 - NDCG@10: 0.45 2. Collect click data: 2M search sessions over 3 months 3. Position debias: Apply inverse propensity weighting 4. Features: BM25, embedding similarity, popularity, price, reviews 5. Model: LambdaMART via LightGBM 6. Results: NDCG@10: 0.62 (+38% vs baseline)

Production architecture:

Query → Elasticsearch (top 500) → Feature service → 
LightGBM reranker → Diversity filter → Top 50

A/B test results: - CTR: +8% - Add-to-cart: +5% - Revenue per search: +3%

Key learnings: - Click debiasing critical for model quality - Popularity features dominated initially (needed regularization) - Real-time features (session context) added +2% CTR

Index: ML Field Guide
Semantic Search Systems
Collaborative Filtering
A/B Testing for ML

Try it live

Everything above runs in your browser — open Retrieval Evaluation and Reranking Lab and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Retrieval Evaluation and Reranking Lab simulation

Learning to Rank