Named Entity Recognition in Production

Named Entity Recognition (NER) identifies and classifies named entities in text into predefined categories like person names, organizations, locations, dates, and domain-specific entities. Production NER systems must handle noisy input, domain-specific vocabulary, and high-throughput requirements.

Where It Is Used

NER powers information extraction across industries:

• Finance: Extracting companies, amounts, dates from financial documents
• Healthcare: Identifying drugs, diseases, symptoms from clinical notes
• Legal: Extracting parties, dates, clauses from contracts
• E-commerce: Product attribute extraction, brand recognition
• Customer support: Extracting order numbers, product names, issues
• Media: Person, organization, location extraction for news analysis

Products: Google Search (entity cards), Bloomberg Terminal, clinical NLP systems, Salesforce Einstein.

Typical Data You Will See

Common annotation format (BIO scheme):

John    B-PER
Smith   I-PER
works   O
at      O
Google  B-ORG
in      O
London  B-LOC
.       O

Data challenges: - Nested entities ("Bank of America" - ORG contains LOC) - Ambiguous entities ("Apple" - company or fruit?) - Domain-specific entities (medical, legal terminology) - Inconsistent annotation guidelines - Class imbalance (many O tags, few entity tags)

What You Do in Practice

1. Define entity types - What entities matter for your use case?
2. Create annotation guidelines - Clear rules for annotators
3. Collect and annotate data - Quality over quantity
4. Choose annotation scheme - BIO, BIOES, or span-based
5. Select model architecture - Transformer + CRF or span extraction
6. Train baseline - Pre-trained model with fine-tuning
7. Handle domain adaptation - Continue pre-training or few-shot
8. Evaluate thoroughly - Per-entity metrics, error analysis
9. Deploy with monitoring - Track entity distributions
10. Iterate on guidelines - Refine based on errors

Baseline First

Start with a pre-trained transformer fine-tuned on your data:

from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

# Load pre-trained NER model
model_name = "dslim/bert-base-NER"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Create pipeline
ner_pipeline = pipeline("ner", model=model, tokenizer=tokenizer)

# Run inference
results = ner_pipeline("John Smith works at Google in London.")

Why pre-trained NER baseline: - Models trained on CoNLL, OntoNotes cover common entities - Fine-tuning on small domain data is effective - Immediate working system to validate requirements - Easy to replace with custom training later

Common Models and When to Choose

Decision guide: - General English NER? → BERT-base or spaCy - Domain-specific? → Domain pre-trained model - Nested entities? → Span-based approach - Production speed critical? → spaCy or distilled models - Low data? → Few-shot with LLMs or pattern matching

Feature Engineering Patterns

Linguistic features (for traditional models): - Part-of-speech tags - Chunk tags - Capitalization patterns - Word shape (Xxxx, XXXX, xxxx) - Gazetteer membership

Contextual features (for transformers): - Subword tokenization alignment - Positional encoding - Attention patterns

Domain adaptation: - Continue pre-training on domain corpus - Add domain-specific tokens to vocabulary - Gazetteer-augmented attention - Entity type embeddings

Data augmentation: - Entity replacement (swap "Google" with "Microsoft") - Synonym replacement for context words - Back-translation - Mention dropout

Evaluation and Metrics

Primary metrics:

Per-entity analysis: - F1 per entity type (identify weak categories) - Confusion matrix for entity types - Error analysis by entity length

Evaluation framework:

from seqeval.metrics import classification_report

# y_true and y_pred are lists of BIO tags
print(classification_report(y_true, y_pred))

Pitfalls: - Token-level metrics inflate scores (many O tags) - Exact match is strict (off-by-one fails) - Rare entities have high variance - Cross-domain evaluation essential

Failure Modes and Debugging Checklist

• [ ] Boundary errors: Entity spans off by one token
• [ ] Type confusion: Person vs Organization mix-ups
• [ ] Rare entities missed: Insufficient training examples
• [ ] Domain shift: Model trained on news, deployed on medical
• [ ] Tokenization mismatch: Subword splits entity incorrectly
• [ ] Nested entities: Inner entity overwritten by outer
• [ ] Inconsistent annotations: Labelers disagree on guidelines
• [ ] Context dependence: Same string, different entity type

Production Checklist

Before deployment: - [ ] Entity types match business requirements - [ ] Per-entity F1 above threshold - [ ] Model tested on production-like data - [ ] Inference latency acceptable - [ ] Confidence thresholds tuned - [ ] Unknown entity handling defined

After deployment: - [ ] Entity distribution monitored - [ ] Low-confidence extractions flagged - [ ] Human review sampling - [ ] Feedback loop to training - [ ] Model version tracking - [ ] A/B testing for updates

Scaling considerations: - [ ] Batch inference optimized - [ ] GPU/CPU trade-offs evaluated - [ ] Caching for repeated documents - [ ] Async processing for large volumes

Tools and Stack

Example Mini-Case

Scenario: Financial services firm needs to extract entities from earnings reports - companies, financial metrics, dates, and monetary amounts.

Data: 5K annotated documents, 8 entity types, mix of tables and prose.

Approach: 1. Baseline: bert-base-NER fine-tuned - Entity F1: 0.68 2. Domain adaptation: Continue pre-training on financial corpus - Entity F1: 0.76 3. Custom entities: Add financial entity types, retrain - Entity F1: 0.82 4. Rule augmentation: Regex for amounts/dates + model - Entity F1: 0.88

Production architecture:

Document → PDF extraction → Sentence splitting →
NER model → Post-processing → Entity linking →
Structured output → Database

Metrics achieved: - Overall entity F1: 88% - Company names: 92% F1 - Financial amounts: 95% F1 (regex + model) - Dates: 94% F1 - Inference: 50 docs/sec on GPU

Key learnings: - Domain pre-training gave biggest lift - Hybrid rules + ML best for structured entities - Table extraction needed special handling - Quarterly retraining for new company names

Related Pages

• Index: ML Field Guide
• Text Classification at Scale
• LLM Integration Patterns
• RAG Systems

Further Reading

• Hugging Face NER Tutorial
• spaCy NER Documentation
• CoNLL-2003 Shared Task
• OntoNotes Dataset

Try it live

Everything above runs in your browser — open Named Entity Recognition (NER) - Interactive NLP Demo and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.