Object Detection in Production

Object detection locates and classifies multiple objects within images or video frames. Unlike classification that assigns one label per image, detection outputs bounding boxes with class labels and confidence scores. Production systems require balancing accuracy, speed, and resource constraints.

Where It Is Used

Object detection powers visual understanding across industries:

• Autonomous vehicles: Pedestrian, vehicle, sign, and obstacle detection
• Retail: Shelf monitoring, checkout-free stores, inventory tracking
• Manufacturing: Defect detection, quality control, safety monitoring
• Security: Surveillance, intrusion detection, crowd monitoring
• Healthcare: Medical imaging, cell counting, tumor localization
• Agriculture: Crop monitoring, pest detection, yield estimation

Products: Tesla Autopilot, Amazon Go, Google Lens, Facebook photo tagging, warehouse robotics.

Typical Data You Will See

Common annotation structure:

{
  "image_id": "img_001.jpg",
  "annotations": [
    {"bbox": [x, y, width, height], "class": "person", "confidence": 0.95},
    {"bbox": [x, y, width, height], "class": "car", "confidence": 0.88}
  ]
}

Data challenges: - Annotation cost (bounding boxes are expensive) - Class imbalance (rare objects) - Small objects (hard to detect) - Occlusion and overlap - Varying lighting and weather conditions

What You Do in Practice

1. Define detection requirements - What objects, what accuracy, what speed?
2. Collect and annotate data - Quality annotations are critical
3. Choose architecture - One-stage vs two-stage based on speed/accuracy needs
4. Select backbone - ResNet, EfficientNet, or custom based on constraints
5. Train baseline - Pre-trained model with minimal fine-tuning
6. Augment aggressively - Flips, crops, color jitter, mosaic
7. Tune hyperparameters - Learning rate, anchor boxes, NMS thresholds
8. Optimize for deployment - Quantization, pruning, TensorRT
9. Deploy with monitoring - Track inference speed, detection quality
10. Handle edge cases - Fallback behavior for unusual inputs

Baseline First

Start with a pre-trained YOLOv8 model fine-tuned on your data:

from ultralytics import YOLO

# Load pre-trained model
model = YOLO('yolov8n.pt')  # nano version for speed

# Fine-tune on your dataset
model.train(
    data='dataset.yaml',
    epochs=100,
    imgsz=640,
    batch=16
)

Why YOLOv8 baseline: - Pre-trained on COCO (80 classes, millions of images) - Single-stage, fast inference - Easy to fine-tune - Good documentation and community

Move to larger models or two-stage detectors only if accuracy requirements demand it.

Common Models and When to Choose

Decision guide: - Need real-time (>30 FPS)? → YOLO family - Small objects critical? → Faster R-CNN or larger YOLO - Limited compute (edge)? → YOLOv8-nano, MobileNet-SSD - Research/flexibility? → DETR, custom architectures

Feature Engineering Patterns

Data augmentation (critical for detection): - Horizontal flip (preserve bbox coordinates) - Random crop with rescaling - Mosaic augmentation (4 images combined) - MixUp (blend images) - Color jitter, blur, noise - Cutout and grid mask

Anchor box optimization: - Run k-means on your bbox sizes - Match anchors to your data distribution - Consider anchor-free methods (FCOS, CenterNet)

Multi-scale handling: - Feature pyramid networks (FPN) - Train on multiple resolutions - Test-time augmentation (TTA)

Domain-specific: - For vehicles: orientation, 3D bboxes - For medical: pixel-level precision - For aerial: rotated bounding boxes

Evaluation and Metrics

Primary metrics:

Per-class analysis: - AP per class (identify weak classes) - Confusion matrix for misclassifications - Size-stratified metrics (small/medium/large objects)

Pitfalls: - mAP hides per-class performance - IoU threshold matters significantly - Test set distribution must match production - FPS varies by hardware and batch size

Failure Modes and Debugging Checklist

• [ ] Small objects missed: Need higher resolution or FPN
• [ ] Overlapping objects merged: NMS threshold too aggressive
• [ ] False positives on background: Need hard negative mining
• [ ] Class confusion: Similar classes need more data
• [ ] Poor generalization: Augmentation insufficient, domain shift
• [ ] Slow inference: Model too large, need optimization
• [ ] Inconsistent detection: Need tracking for video
• [ ] Edge case failures: Unusual lighting, angles, occlusion

Production Checklist

Before deployment: - [ ] Model tested on representative data - [ ] Inference speed meets requirements - [ ] Model optimized (quantization, TensorRT) - [ ] Confidence threshold tuned for use case - [ ] NMS parameters validated - [ ] Fallback behavior defined - [ ] Memory footprint acceptable

After deployment: - [ ] Inference latency monitored (p50, p99) - [ ] Detection confidence distributions tracked - [ ] False positive rate monitored - [ ] Input image quality checked - [ ] Model version tracking - [ ] A/B testing for model updates

Edge deployment: - [ ] Model converted (ONNX, TensorRT, CoreML) - [ ] Quantization tested (INT8) - [ ] Hardware acceleration enabled - [ ] Thermal constraints considered - [ ] Power consumption measured

Tools and Stack

Example Mini-Case

Scenario: Retail chain needs shelf monitoring to detect out-of-stock products across 500 stores.

Data: 50K labeled images, 200 product classes, varying shelf layouts and lighting.

Approach: 1. Baseline: YOLOv8-small pre-trained, fine-tuned - mAP@0.5: 0.72 2. Augmentation: Aggressive augmentation + synthetic data - mAP@0.5: 0.81 3. Architecture: Custom anchors + larger model (YOLOv8-medium) - mAP@0.5: 0.86 4. Optimization: TensorRT INT8 quantization - 45 FPS on edge devices

Production architecture:

Camera → Edge device (inference) → Detection results →
Cloud aggregation → Store dashboard → Restock alerts

Metrics achieved: - mAP@0.5: 86% - Inference: 45 FPS on Jetson Xavier - Out-of-stock detection accuracy: 92% - False alert rate: < 3%

Key learnings: - Lighting normalization critical for consistency - Product rotation/updates require continuous training - Edge deployment reduced latency from 2s to 50ms

Related Pages

• Index: ML Field Guide
• Image Classification
• Model Compression
• MLOps Foundations

Further Reading

• Ultralytics YOLOv8 Documentation
• COCO Dataset and Evaluation
• Papers with Code - Object Detection
• TensorRT Optimization Guide

Try it live

Everything above runs in your browser — open Object Detection Playground and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.