OCR and Document Processing

Document processing extracts structured information from unstructured documents - PDFs, scanned images, forms, invoices. Modern systems combine OCR (Optical Character Recognition) with layout analysis, table extraction, and NLP to understand document semantics, not just recognize text.

Where It Is Used

Document processing powers digitization across industries:

• Finance: Invoice processing, bank statement parsing, check reading
• Insurance: Claims processing, policy document extraction
• Healthcare: Medical records digitization, prescription reading
• Legal: Contract analysis, court document processing
• Government: Form processing, ID verification, tax documents
• Logistics: Shipping label reading, customs document processing

Products: Adobe Acrobat, ABBYY, Google Document AI, AWS Textract, Azure Form Recognizer.

Typical Data You Will See

Document categories: - Structured: Forms with fixed fields (tax forms, applications) - Semi-structured: Variable layout but consistent elements (invoices, receipts) - Unstructured: Free-form text (contracts, letters, reports)

Challenges: - Skewed or rotated pages - Poor scan quality, noise, artifacts - Handwritten text mixed with printed - Complex tables spanning pages - Multi-column layouts - Watermarks and stamps

What You Do in Practice

1. Understand document types - What documents, what fields to extract?
2. Preprocess images - Deskew, denoise, binarize
3. Run OCR - Text recognition with confidence scores
4. Analyze layout - Detect regions, columns, tables, figures
5. Extract tables - Structure detection and cell recognition
6. Post-process text - Spell correction, entity normalization
7. Apply NLP - Named entity extraction, classification
8. Validate output - Business rules, format checks
9. Human review - Low confidence items flagged
10. Continuous improvement - Feedback loop to training

Baseline First

Start with a cloud OCR API before building custom:

# Google Cloud Vision API
from google.cloud import vision

client = vision.ImageAnnotatorClient()

with open('document.pdf', 'rb') as f:
    content = f.read()

image = vision.Image(content=content)
response = client.document_text_detection(image=image)

# Extract full text
full_text = response.full_text_annotation.text

# Extract with bounding boxes
for page in response.full_text_annotation.pages:
    for block in page.blocks:
        for paragraph in block.paragraphs:
            for word in paragraph.words:
                word_text = ''.join([s.text for s in word.symbols])
                bbox = word.bounding_box

Why cloud API baseline: - State-of-the-art accuracy out of the box - Handles multiple languages - No training required - Fast to validate use case

Move to custom models only when APIs don't meet accuracy or cost requirements.

Common Tools and When to Choose

Decision guide: - Quick start, good quality? → Cloud APIs - Cost-sensitive, simple docs? → Tesseract - Need customization? → PaddleOCR, DocTR - Document understanding? → LayoutLM, Donut

Document Processing Pipeline

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Input     │────▶│  Preprocess │────▶│    OCR      │
│  Document   │     │  (deskew,   │     │  (text +    │
│             │     │   denoise)  │     │   boxes)    │
└─────────────┘     └─────────────┘     └─────────────┘
                                               │
                                               ▼
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Output    │◀────│  Extract    │◀────│   Layout    │
│  Structured │     │  Fields     │     │  Analysis   │
│   Data      │     │  + Tables   │     │  (regions)  │
└─────────────┘     └─────────────┘     └─────────────┘

Preprocessing techniques: - Deskewing: Correct rotation angle - Binarization: Convert to black/white - Denoising: Remove artifacts - Resolution normalization: Scale to optimal DPI

Layout analysis: - Region segmentation (text, table, figure, header) - Reading order detection - Column detection - Table structure recognition

Table Extraction

Tables are particularly challenging. Approaches:

Rule-based: - Detect lines and intersections - Works for bordered tables - Fails on borderless tables

Deep learning: - Table detection (object detection) - Cell detection and association - Works on varied table styles

# Using img2table library
from img2table.document import PDF
from img2table.ocr import TesseractOCR

ocr = TesseractOCR(lang="eng")
pdf = PDF(src="document.pdf")

# Extract tables
tables = pdf.extract_tables(ocr=ocr)

for table in tables:
    df = table.df  # Get as pandas DataFrame

Evaluation and Metrics

Evaluation approach: 1. Create ground truth dataset (100-500 documents) 2. Measure OCR accuracy (CER, WER) 3. Measure extraction accuracy per field 4. Track end-to-end accuracy 5. Analyze errors by document type/quality

Failure Modes and Debugging Checklist

• [ ] Poor image quality: Low resolution, blur, noise
• [ ] Skewed pages: Rotation not corrected
• [ ] Wrong language: OCR using incorrect language model
• [ ] Table detection miss: Tables not recognized
• [ ] Column confusion: Multi-column text merged
• [ ] Handwriting fails: Printed OCR on handwritten
• [ ] Field mismatch: Extracted value in wrong field
• [ ] Character confusion: Similar characters (0/O, 1/l)

Production Checklist

Before deployment: - [ ] Document types catalogued - [ ] Quality thresholds defined - [ ] OCR accuracy validated on test set - [ ] Table extraction tested - [ ] Field extraction accuracy measured - [ ] Human review workflow designed - [ ] Throughput requirements met

After deployment: - [ ] Processing success rate monitored - [ ] Confidence scores tracked - [ ] Low-confidence items flagged - [ ] Human review queue monitored - [ ] Accuracy sampled periodically - [ ] Model updates tested before deployment

Compliance: - [ ] PII handling defined - [ ] Data retention policies - [ ] Audit trail for extractions - [ ] Error correction process

Tools and Stack

Example Mini-Case

Scenario: Insurance company needs to automate claims processing from scanned documents - 50K documents/month.

Document types: Claim forms, medical bills, receipts, ID documents.

Approach: 1. Baseline: AWS Textract API - Field accuracy: 78% 2. Preprocessing: Deskew + denoise pipeline - Field accuracy: 82% 3. Custom extraction: Fine-tuned LayoutLM for claim forms - Field accuracy: 89% 4. Validation: Business rules + confidence thresholds - End-to-end: 85% 5. Human review: Low confidence (15%) routed to reviewers

Architecture:

S3 upload → Lambda trigger → Textract OCR →
LayoutLM extraction → Validation → DynamoDB →
Human review queue (if needed) → Claims system

Metrics achieved: - Processing time: 30 sec/document - Field accuracy: 89% - Straight-through processing: 85% - Cost: $0.15/document - Human review reduced by 70%

Key learnings: - Preprocessing critical for scanned documents - Custom model needed for domain-specific forms - Confidence thresholds balance automation vs accuracy - Human review essential for edge cases

Related Pages

• Index: ML Field Guide
• Object Detection
• Named Entity Recognition
• Image Classification

Further Reading

• Tesseract Documentation
• Google Document AI
• AWS Textract
• LayoutLM Paper
• PaddleOCR

Try it live

Everything above runs in your browser — open AI Document Processing Lab — An Invoice Extraction Case Study and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.