Query Understanding

Users type messy queries. They misspell, use ambiguous terms, and rarely express exactly what they want. Query understanding transforms raw user input into structured intent that search systems can act on. It's the bridge between what users type and what they mean.

Where It Is Used

Query understanding powers search everywhere:

• Web search: Google, Bing query analysis
• E-commerce: Product search, faceted navigation
• Enterprise search: Document retrieval, knowledge bases
• Voice assistants: Natural language understanding
• Customer support: Intent routing, FAQ matching
• Legal/medical: Specialized terminology handling

Query Understanding Pipeline

What You Do in Practice

1. Analyse query logs - What do users actually search for?
2. Build taxonomy - Intents, entities, attributes
3. Collect training data - Annotate queries with labels
4. Implement pipeline - Modular, debuggable stages
5. Handle edge cases - Empty queries, very long queries
6. Evaluate end-to-end - Search quality metrics
7. Monitor in production - Track failures and drift
8. Iterate continuously - User feedback loop

Baseline First

Start with rules and dictionaries before ML:

import re
from collections import defaultdict

class QueryUnderstandingBaseline:
    """Rule-based query understanding."""

    def __init__(self):
        self.brand_dict = {'apple', 'samsung', 'google', 'sony'}
        self.category_dict = {'phone', 'laptop', 'tablet', 'headphones'}
        self.attribute_patterns = {
            'color': r'\b(red|blue|black|white|green|gold|silver)\b',
            'size': r'\b(\d+)\s*(gb|tb|inch|")\b',
            'price': r'\b(cheap|expensive|budget|premium)\b'
        }
        self.spelling_corrections = {
            'ipohne': 'iphone',
            'samsugn': 'samsung',
            'labtop': 'laptop'
        }

    def process(self, query):
        """Full query understanding pipeline."""
        # Normalize
        query = self.normalize(query)

        # Spell correct
        query = self.spell_correct(query)

        # Extract entities
        entities = self.extract_entities(query)

        # Classify intent
        intent = self.classify_intent(query, entities)

        return {
            'original_query': query,
            'normalized_query': query,
            'entities': entities,
            'intent': intent
        }

    def normalize(self, query):
        """Normalize query text."""
        query = query.lower().strip()
        query = re.sub(r'\s+', ' ', query)
        query = re.sub(r'[^\w\s]', ' ', query)
        return query

    def spell_correct(self, query):
        """Simple dictionary-based spelling correction."""
        words = query.split()
        corrected = []

        for word in words:
            if word in self.spelling_corrections:
                corrected.append(self.spelling_corrections[word])
            else:
                corrected.append(word)

        return ' '.join(corrected)

    def extract_entities(self, query):
        """Extract known entities from query."""
        entities = defaultdict(list)
        words = set(query.split())

        # Brand detection
        for brand in self.brand_dict:
            if brand in words or brand in query:
                entities['brand'].append(brand)

        # Category detection
        for category in self.category_dict:
            if category in words:
                entities['category'].append(category)

        # Attribute patterns
        for attr_type, pattern in self.attribute_patterns.items():
            matches = re.findall(pattern, query)
            if matches:
                entities[attr_type].extend(matches)

        return dict(entities)

    def classify_intent(self, query, entities):
        """Classify query intent."""
        # Navigational: looking for specific brand/product
        if 'brand' in entities and len(query.split()) <= 3:
            return 'navigational'

        # Transactional: buying signals
        buy_signals = ['buy', 'price', 'cheap', 'order', 'purchase']
        if any(signal in query for signal in buy_signals):
            return 'transactional'

        # Informational: questions, comparisons
        info_signals = ['how', 'what', 'which', 'best', 'vs', 'compare', 'review']
        if any(signal in query for signal in info_signals):
            return 'informational'

        return 'unknown'

Intent Classification with ML

import torch
from transformers import BertTokenizer, BertForSequenceClassification
from sklearn.model_selection import train_test_split

class IntentClassifier:
    """BERT-based intent classifier."""

    def __init__(self, model_name='bert-base-uncased', num_intents=5):
        self.tokenizer = BertTokenizer.from_pretrained(model_name)
        self.model = BertForSequenceClassification.from_pretrained(
            model_name,
            num_labels=num_intents
        )
        self.intent_labels = [
            'navigational', 'informational', 'transactional',
            'local', 'other'
        ]

    def predict(self, queries):
        """Predict intents for queries."""
        inputs = self.tokenizer(
            queries,
            padding=True,
            truncation=True,
            max_length=64,
            return_tensors='pt'
        )

        with torch.no_grad():
            outputs = self.model(**inputs)
            predictions = torch.argmax(outputs.logits, dim=1)

        return [self.intent_labels[p] for p in predictions]

    def train(self, queries, labels, epochs=3):
        """Fine-tune on query-intent pairs."""
        # Convert labels to indices
        label_indices = [self.intent_labels.index(l) for l in labels]

        # Tokenize
        inputs = self.tokenizer(
            queries,
            padding=True,
            truncation=True,
            max_length=64,
            return_tensors='pt'
        )
        inputs['labels'] = torch.tensor(label_indices)

        # Training
        optimizer = torch.optim.AdamW(self.model.parameters(), lr=2e-5)

        self.model.train()
        for epoch in range(epochs):
            optimizer.zero_grad()
            outputs = self.model(**inputs)
            loss = outputs.loss
            loss.backward()
            optimizer.step()
            print(f'Epoch {epoch+1}: Loss {loss.item():.4f}')

class QueryEntityExtractor:
    """NER-based entity extraction from queries."""

    def __init__(self):
        from transformers import pipeline
        self.ner = pipeline('ner', aggregation_strategy='simple')

        # Custom entity mapping for search
        self.entity_mapping = {
            'ORG': 'brand',
            'PRODUCT': 'product',
            'LOC': 'location',
            'CARDINAL': 'quantity'
        }

    def extract(self, query):
        """Extract entities from query."""
        entities = self.ner(query)

        result = defaultdict(list)
        for entity in entities:
            entity_type = self.entity_mapping.get(
                entity['entity_group'], 
                entity['entity_group'].lower()
            )
            result[entity_type].append({
                'text': entity['word'],
                'score': entity['score']
            })

        return dict(result)

Query Expansion

Improve recall by expanding queries:

from gensim.models import KeyedVectors
import numpy as np

class QueryExpander:
    """Expand queries with related terms."""

    def __init__(self, embeddings_path=None):
        if embeddings_path:
            self.embeddings = KeyedVectors.load(embeddings_path)
        else:
            self.embeddings = None

        # Manual synonyms for critical terms
        self.synonyms = {
            'laptop': ['notebook', 'computer', 'pc'],
            'phone': ['mobile', 'smartphone', 'cell'],
            'cheap': ['budget', 'affordable', 'inexpensive'],
            'buy': ['purchase', 'order', 'get']
        }

    def expand(self, query, max_expansions=3):
        """Expand query with related terms."""
        words = query.lower().split()
        expansions = set()

        for word in words:
            # Check manual synonyms
            if word in self.synonyms:
                expansions.update(self.synonyms[word][:max_expansions])

            # Use embeddings for semantic expansion
            if self.embeddings and word in self.embeddings:
                similar = self.embeddings.most_similar(word, topn=max_expansions)
                for term, score in similar:
                    if score > 0.7:  # Similarity threshold
                        expansions.add(term)

        return list(expansions - set(words))

    def build_expanded_query(self, query, strategy='or'):
        """Build expanded search query."""
        original_terms = query.split()
        expansions = self.expand(query)

        if strategy == 'or':
            all_terms = original_terms + expansions
            return ' OR '.join(all_terms)
        elif strategy == 'boost':
            # Original terms boosted higher
            boosted = [f'{t}^2.0' for t in original_terms]
            return ' '.join(boosted + expansions)
        else:
            return query

class SpellingCorrector:
    """Context-aware spelling correction."""

    def __init__(self, vocabulary):
        self.vocabulary = set(vocabulary)
        self.word_freq = defaultdict(int)
        for word in vocabulary:
            self.word_freq[word] += 1

    def correct(self, query):
        """Correct spelling errors in query."""
        words = query.split()
        corrected = []

        for word in words:
            if word in self.vocabulary:
                corrected.append(word)
            else:
                # Find closest match
                candidates = self.get_candidates(word)
                if candidates:
                    best = max(candidates, key=lambda w: self.word_freq[w])
                    corrected.append(best)
                else:
                    corrected.append(word)

        return ' '.join(corrected)

    def get_candidates(self, word, max_distance=2):
        """Get candidate corrections within edit distance."""
        candidates = []

        for vocab_word in self.vocabulary:
            if abs(len(vocab_word) - len(word)) <= max_distance:
                distance = self.edit_distance(word, vocab_word)
                if distance <= max_distance:
                    candidates.append(vocab_word)

        return candidates

    def edit_distance(self, s1, s2):
        """Calculate Levenshtein distance."""
        if len(s1) < len(s2):
            return self.edit_distance(s2, s1)

        if len(s2) == 0:
            return len(s1)

        previous_row = range(len(s2) + 1)
        for i, c1 in enumerate(s1):
            current_row = [i + 1]
            for j, c2 in enumerate(s2):
                insertions = previous_row[j + 1] + 1
                deletions = current_row[j] + 1
                substitutions = previous_row[j] + (c1 != c2)
                current_row.append(min(insertions, deletions, substitutions))
            previous_row = current_row

        return previous_row[-1]

Query Rewriting

Transform queries for better matching:

class QueryRewriter:
    """Rewrite queries for improved search."""

    def __init__(self):
        self.abbreviations = {
            'nyc': 'new york city',
            'la': 'los angeles',
            'uk': 'united kingdom',
            'ml': 'machine learning'
        }

        self.rewrite_rules = [
            (r'^what is (.+)$', r'\1 definition'),
            (r'^how to (.+)$', r'\1 guide tutorial'),
            (r'^best (.+) (\d{4})$', r'\1 top rated \2'),
            (r'^(.+) vs (.+)$', r'\1 \2 comparison'),
        ]

    def rewrite(self, query):
        """Apply rewriting rules to query."""
        query = query.lower().strip()

        # Expand abbreviations
        for abbrev, expansion in self.abbreviations.items():
            query = re.sub(rf'\b{abbrev}\b', expansion, query)

        # Apply rewrite rules
        for pattern, replacement in self.rewrite_rules:
            new_query = re.sub(pattern, replacement, query)
            if new_query != query:
                return new_query

        return query

    def segment_query(self, query):
        """Segment query into meaningful parts."""
        # Identify potential n-grams
        words = query.split()
        segments = []

        i = 0
        while i < len(words):
            # Try longest match first
            matched = False
            for length in range(min(4, len(words) - i), 0, -1):
                ngram = ' '.join(words[i:i+length])
                if self.is_known_entity(ngram):
                    segments.append(('entity', ngram))
                    i += length
                    matched = True
                    break

            if not matched:
                segments.append(('word', words[i]))
                i += 1

        return segments

    def is_known_entity(self, text):
        """Check if text is a known entity."""
        # Would check against entity dictionary
        known_entities = {'new york city', 'machine learning', 'iphone 13'}
        return text in known_entities

Evaluation Metrics

Failure Modes and Debugging Checklist

• [ ] Over-correction: Fixing words that aren't misspelled
• [ ] Wrong intent: Ambiguous queries misclassified
• [ ] Entity confusion: Similar entity names mixed up
• [ ] Expansion noise: Adding irrelevant terms
• [ ] Long-tail queries: Rare queries not handled
• [ ] Latency: Pipeline too slow
• [ ] Language mixing: Multi-lingual queries

Production Checklist

Before deployment: - [ ] Coverage analysis on query logs - [ ] A/B test framework ready - [ ] Latency budget established - [ ] Fallback behavior defined - [ ] Logging for debugging

After deployment: - [ ] Monitor intent distribution - [ ] Track correction rates - [ ] Measure search quality impact - [ ] Review failure cases - [ ] Continuous model updates

Tools and Stack

Example Mini-Case

Scenario: E-commerce product search query understanding.

Pipeline: 1. Normalize: lowercase, remove special chars 2. Spell correct: "ipone 14 blak" → "iphone 14 black" 3. Extract entities: {brand: apple, product: iphone, model: 14, color: black} 4. Classify intent: transactional 5. Expand: add "smartphone", "mobile"

Results: - 15% improvement in search NDCG - 23% reduction in null results - Spelling correction catches 8% of queries - Average latency: 12ms

Related Pages

• Index: ML Field Guide
• Semantic Search
• Learning to Rank
• Named Entity Recognition

Further Reading

• Query Understanding for Search Engines
• Understanding Searches Better Than Ever Before
• Elasticsearch Query DSL

Try it live

Everything above runs in your browser — open Word Embeddings Explorer - NLP Vector Space Visualization and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.