Text Classification at Scale

Text classification assigns labels to documents - spam detection, sentiment analysis, topic categorization, intent recognition. It is one of the most common NLP tasks in production, powering everything from email filters to customer support routing.

Where It Is Used

Text classification drives critical business workflows:

• Customer support: Ticket routing, priority classification, intent detection
• Content moderation: Spam detection, toxicity filtering, policy violations
• E-commerce: Product categorization, review sentiment, search relevance
• Finance: News sentiment, document classification, compliance screening
• Healthcare: Clinical note coding, symptom classification
• Marketing: Lead qualification, feedback analysis, brand monitoring

Products: Gmail spam filter, Twitter content moderation, Amazon product categories, Zendesk ticket routing.

Typical Data You Will See

Common table structure:

doc_id | text | label | created_at | annotator_id | confidence

Data sources: - Historical labeled data from production systems - Human annotation campaigns - Weak supervision (rules, heuristics) - Semi-supervised labeling with active learning

What You Do in Practice

1. Define the taxonomy - What labels, how many, hierarchical or flat?
2. Audit existing labels - Check quality, consistency, coverage
3. Clean and preprocess - Normalize text, handle special characters
4. Split data properly - Stratified split, temporal split if relevant
5. Establish baseline - TF-IDF + logistic regression
6. Try pretrained models - Fine-tune BERT or similar
7. Handle class imbalance - Resampling, class weights, focal loss
8. Evaluate comprehensively - Per-class metrics, confusion analysis
9. Calibrate confidence - Threshold tuning, rejection option
10. Deploy and monitor - Track prediction distribution, drift

Baseline First

TF-IDF + Logistic Regression is the baseline for text classification:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

vectorizer = TfidfVectorizer(max_features=10000, ngram_range=(1, 2))
X_train = vectorizer.fit_transform(train_texts)
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, train_labels)

Why this baseline: - Trains in seconds on 100K examples - Interpretable (inspect feature weights) - Often gets 80%+ of final performance - Works well for domain-specific vocabulary

If baseline is within 5% of transformer performance, keep the baseline.

Common Models and When to Choose

Industry default: DistilBERT for most use cases. Full BERT when accuracy is critical.

Feature Engineering Patterns

Text preprocessing: - Lowercasing (usually yes) - Punctuation removal (domain-dependent) - Stopword removal (skip for transformers) - Lemmatization (skip for transformers)

Additional features: - Document length - Capitalization patterns - Special character presence - Language detection - Named entity counts

Label encoding: - Multi-class: Single label per document - Multi-label: Multiple labels possible - Hierarchical: Labels have parent-child relationships

Data augmentation: - Back-translation (translate to other language and back) - Synonym replacement - Random insertion/deletion - Paraphrasing with LLMs

Evaluation and Metrics

Primary metrics:

Per-class analysis: - Precision and recall for each class - Confusion matrix patterns - Error analysis on misclassified examples

Pitfalls: - Evaluating only on majority class performance - Not stratifying test set - Ignoring label noise in evaluation - Not testing on out-of-domain examples

Failure Modes and Debugging Checklist

• [ ] Label noise: Annotator disagreement corrupts training
• [ ] Class imbalance: Model predicts majority class
• [ ] Domain shift: Model trained on formal text, deployed on informal
• [ ] Vocabulary mismatch: New terms, slang, abbreviations
• [ ] Length bias: Model overfits to document length
• [ ] Spurious correlations: Model uses shortcuts (author style, timestamps)
• [ ] Multilingual gaps: Poor performance on non-English
• [ ] Adversarial inputs: Intentionally crafted to fool classifier

Production Checklist

Before deployment: - [ ] Per-class performance acceptable - [ ] Evaluated on held-out production distribution - [ ] Latency requirements met - [ ] Confidence calibration checked - [ ] Edge cases documented (empty text, very long text) - [ ] Rejection threshold defined for low-confidence predictions

After deployment: - [ ] Prediction distribution monitored - [ ] Confidence distribution tracked - [ ] Human review sample for quality assurance - [ ] Retraining schedule established - [ ] Feedback loop for label corrections - [ ] A/B testing framework for model updates

Tools and Stack

Example Mini-Case

Scenario: Route customer support tickets to correct team (5 teams: billing, technical, account, shipping, general).

Data: 50K historical tickets with team assignments, class imbalance (40% general, 5% shipping).

Approach: 1. Baseline: TF-IDF + LogReg - Weighted F1: 0.72 2. Error analysis: Billing vs Account confusion, short tickets misclassified 3. Fine-tune DistilBERT - Weighted F1: 0.84 4. Add rejection: If confidence < 0.6, route to human review 5. Result: 85% automated routing, 12% human review, 3% escalation

Production setup: - Model served via FastAPI - 50ms p95 latency requirement met - Daily prediction distribution check - Weekly sample review (50 tickets) - Monthly retraining with new labels

Impact: 40% reduction in misrouting, faster resolution times.

Related Pages

• Index: ML Field Guide
• Named Entity Recognition
• LLM Integration Patterns
• Binary Classification in Production

Further Reading

• Hugging Face Transformers Documentation
• spaCy Documentation
• scikit-learn Text Feature Extraction
• SetFit: Efficient Few-shot Learning

Try it live

Everything above runs in your browser — open Bag of Words vs TF-IDF: How Machines Turn Sentences into Numbers and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.