Two-Stream Action Recognition in 2D: Spatial vs Temporal Fusion

Action Being Performed

Stream Fusion

Fusion weight w (spatial ↔ temporal) 0.50

Sensor noise 0.10

Playback speed 1.00×

Live Recognition

Spatial stream (pose shape) —

—

Temporal stream (joint velocity) —

—

Fused prediction —

—

Fused accuracy (last 180 frames)

—

How it works

This models a two-stream action-recognition network (Simonyan & Zisserman, 2014 — the ancestor of I3D and SlowFast): one stream classifies from a single frame's appearance, the other from the motion between frames, and their softmax outputs are late-fused.

spatial stream:  s_i = cos(J(t) + noise, template_spatial[i])
temporal stream: v_i = cos(dJ/dt + 1.8·noise, template_temporal[i])
P_spatial  = softmax(τ·s)
P_temporal = softmax(τ·v)
P_fused    = w·P_spatial + (1−w)·P_temporal

J(t) is the 9-dimensional joint-angle vector of the skeleton (shoulders, elbows, hips, knees, hip-height bob) driven by real forward kinematics — identical formulas to the 3D version of this simulation. Each action's template is the time-averaged |J| (a shape descriptor) and time-averaged |dJ/dt| (a motion descriptor), sampled once over a full gait cycle at start-up.

Fusion weight w — 1 trusts only appearance, 0 trusts only motion. Watch Squat vs a paused Jump: their instantaneous shapes can look alike, so the spatial stream sometimes hesitates between them while the temporal stream — which sees the synchronized bounce vs the slow crouch — stays confident.
Sensor noise — Gaussian jitter added before classification (not to the rendered skeleton, only to what the "network" measures). Motion estimates are numerically differentiated, so they amplify noise faster than the raw pose does — the live signal strip below the skeleton makes this visible directly: the purple (temporal) trace gets noticeably jagged before the teal (spatial) one does.
The trailing ghost dots at each hand/foot are the finite temporal window a flow-based stream actually needs — a single frame has no velocity to read.

Real systems swap the hand-built templates here for learned 3D-CNN or Transformer features (I3D, SlowFast, TimeSformer), but the late-fusion arithmetic — weighted combination of a spatial and a temporal score — is unchanged. This 2D version reprojects the exact same forward-kinematics angles used by the 3D original: every joint's swing/bend rotation happens about a single axis, so it collapses onto one plane without approximation — only the camera changed, not the mechanism.

Skeleton (spatial input) Motion trail (temporal input)

drag to pan · scroll to zoom