Two-Stream Action Recognition: Spatial vs Temporal Fusion

Action Being Performed

Stream Fusion

Fusion weight w (spatial ↔ temporal) 0.50

Sensor noise 0.10

Playback speed 1.00×

Live Recognition

Spatial stream (pose shape) —

—

Temporal stream (joint velocity) —

—

Fused prediction —

—

Fused accuracy (last 180 frames)

—

How it works

This models a two-stream action-recognition network (Simonyan & Zisserman, 2014 — the ancestor of I3D and SlowFast): one stream classifies from a single frame's appearance, the other from the motion between frames, and their softmax outputs are late-fused.

spatial stream:  s_i = cos(J(t) + noise, template_spatial[i])
temporal stream: v_i = cos(dJ/dt + 1.8·noise, template_temporal[i])
P_spatial  = softmax(τ·s)
P_temporal = softmax(τ·v)
P_fused    = w·P_spatial + (1−w)·P_temporal

J(t) is the 9-dimensional joint-angle vector of the skeleton (shoulders, elbows, hips, knees, hip-height bob) driven by real forward kinematics. Each action's template is the time-averaged |J| (a shape descriptor — a low, bent stance vs an upright wave) and time-averaged |dJ/dt| (a motion descriptor — legs alternating vs both legs moving together), sampled once over a full gait cycle at start-up.

Fusion weight w — 1 trusts only appearance, 0 trusts only motion. Watch Squat vs a paused Jump: their instantaneous shapes can look alike, so the spatial stream sometimes hesitates between them while the temporal stream — which sees the synchronized bounce vs the slow crouch — stays confident.
Sensor noise — Gaussian jitter added before classification (not to the rendered skeleton, only to what the "network" measures). Motion estimates are numerically differentiated, so they amplify noise faster than the raw pose does — exactly why real optical-flow streams degrade first under compression or motion blur.
The trailing ghost dots at each hand/foot are the finite temporal window a flow-based stream actually needs — a single frame has no velocity to read.

Real systems swap the hand-built templates here for learned 3D-CNN or Transformer features (I3D, SlowFast, TimeSformer), but the late-fusion arithmetic — weighted combination of a spatial and a temporal score — is unchanged.

Skeleton (spatial input) Motion trail (temporal input)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.