🌊 Optical Flow Visualizer

Understanding Optical Flow

Optical flow is the pattern of apparent motion of objects in a visual scene caused by relative motion between the observer and the scene. It's a fundamental technique in computer vision for understanding motion, tracking objects, and analyzing video sequences.

What is Optical Flow?

Optical flow represents the distribution of velocities of pixels in an image. For each pixel, it estimates:

• Displacement Vector: (dx, dy) showing where the pixel moved
• Velocity: Speed and direction of motion
• Motion Field: Dense field of vectors across entire image

Fundamental Assumption: Brightness Constancy

Optical flow algorithms assume that pixel intensities remain constant between frames:

I(x, y, t) = I(x + dx, y + dy, t + dt)

Linearizing (Taylor expansion):
I_x · u + I_y · v + I_t = 0

Where:

• I_x, I_y = Spatial gradients (image derivatives)
• I_t = Temporal gradient (change over time)
• u, v = Optical flow components (what we're solving for)

This is the Optical Flow Constraint Equation - one equation with two unknowns. This is the fundamental challenge requiring additional constraints.

Lucas-Kanade Method

Lucas-Kanade (1981) is a sparse optical flow method:

• Key Idea: Assume constant flow in local neighborhood
• Approach:
  • Select features to track (corners, strong gradients)
  • For each feature, solve over small window (e.g., 5×5)
  • Solve least-squares problem: min Σ(I_x·u + I_y·v + I_t)²
• Advantages:
  • Fast and efficient
  • Works well for small motions
  • Robust at corners and textured regions
• Limitations:
  • Only sparse flow (at selected points)
  • Fails with large motions
  • Aperture problem in low-texture regions

Horn-Schunck Method

Horn-Schunck (1981) computes dense optical flow:

• Key Idea: Add smoothness constraint - neighboring pixels have similar motion
• Energy Minimization:
  • Data term: (I_x·u + I_y·v + I_t)²
  • Smoothness term: (∇u)² + (∇v)²
  • Total: E = E_data + λ·E_smooth
• Solution: Iterative global optimization
• Advantages:
  • Dense flow everywhere
  • Smooth, continuous fields
  • Handles low-texture regions
• Limitations:
  • Slow (many iterations)
  • Over-smooths motion boundaries
  • Still limited to small motions

Multi-Scale (Pyramidal) Approaches

To handle large motions:

• Coarse-to-Fine Strategy:
  • Build image pyramid (multiple scales)
  • Estimate flow at coarsest level
  • Propagate to finer levels
  • Refine estimates at each level
• Benefits:
  • Large motions become small at coarse scales
  • Faster convergence
  • More robust

Farnebäck Method

Farnebäck (2003) is a modern dense flow method:

• Polynomial expansion of image neighborhoods
• Estimates flow from polynomial coefficients
• Dense, accurate results
• Handles larger motions
• Available in OpenCV

Deep Learning Methods

• FlowNet (2015):
  • First deep learning approach
  • End-to-end supervised learning
  • Fast but required synthetic training data
• FlowNet 2.0 (2017):
  • Stacked architecture
  • Improved accuracy
  • Real-time capable
• PWC-Net (2018):
  • Pyramid, Warping, Cost volume
  • Compact and efficient
  • State-of-the-art accuracy
• RAFT (2020):
  • Recurrent All-Pairs Field Transforms
  • Current state-of-the-art
  • Strong generalization

Applications

• Video Stabilization:
  • Estimate camera motion
  • Compensate for shaking
  • Smooth video playback
• Object Tracking:
  • Track objects across frames
  • Motion prediction
  • Video surveillance
• Action Recognition:
  • Motion features for classification
  • Two-stream CNNs (RGB + flow)
  • Sports analysis
• Video Compression:
  • Motion-compensated prediction
  • Inter-frame compression
  • H.264, H.265 codecs
• Autonomous Driving:
  • Ego-motion estimation
  • Obstacle detection
  • Time-to-collision
• Medical Imaging:
  • Heart wall motion analysis
  • Organ tracking during surgery
  • Respiratory motion compensation
• Visual Effects:
  • Motion blur synthesis
  • Frame interpolation
  • Rotoscoping assistance

Challenges

• Aperture Problem:
  • Can't determine motion perpendicular to edges
  • Ambiguous in regions with uniform texture
  • Solution: Use neighborhood constraints
• Occlusions:
  • Objects appearing/disappearing
  • Violates brightness constancy
  • Need explicit occlusion handling
• Large Motions:
  • Linearization breaks down
  • Solution: Multi-scale pyramid
• Lighting Changes:
  • Violates brightness constancy
  • Shadows, reflections
  • Solution: Robust cost functions
• Textureless Regions:
  • Weak gradients
  • Unreliable estimates
  • Solution: Smoothness priors

Evaluation Metrics

• End-Point Error (EPE): Euclidean distance between estimated and ground truth
• Angular Error: Angle between flow vectors
• Percentage of Bad Pixels: Pixels with EPE > threshold

Benchmarks

• Middlebury: Classic optical flow benchmark
• KITTI: Autonomous driving scenes
• Sintel: Animated movie frames with ground truth
• Flying Chairs/Things: Synthetic data for training

Visualization Techniques

• Vector Field: Draw arrows showing motion
• Color Coding:
  • Hue encodes direction
  • Saturation/brightness encodes magnitude
  • HSV color wheel standard
• Motion History Images: Temporal templates
• Streamlines: Follow flow paths

Implementation Tips

• Pre-filter images (Gaussian blur) to reduce noise
• Use multi-scale pyramid for large motions
• Choose appropriate window size (Lucas-Kanade)
• Tune smoothness parameter (Horn-Schunck)
• Handle boundaries carefully
• Consider bi-directional consistency check
• Post-process to remove outliers

Related Techniques

• Scene Flow: 3D motion in 3D space
• Dense Trajectories: Long-term point tracking
• Motion Segmentation: Grouping by motion
• Structure from Motion: 3D reconstruction from motion

Future Directions

• Self-supervised learning (no ground truth needed)
• Unsupervised domain adaptation
• Real-time on mobile devices
• Integration with depth sensors
• Long-term temporal modeling

Experiment with the Demo

Use the interactive tool above to:

• Visualize different types of motion
• Compare Lucas-Kanade vs Horn-Schunck
• See how flow vectors represent motion
• Understand color-coded flow visualization
• Experiment with different motion scenarios

Optical flow is fundamental to understanding motion in videos. From stabilizing shaky footage to enabling self-driving cars, motion estimation powers countless computer vision applications!