🦿 3D Tele-Rehabilitation Pose Fusion

Why a 3D pose-fusion view

A monocular camera stream, however good the pose-estimation model, always flattens the patient's body into a single viewing angle: motion toward or away from the lens (depth) is the hardest axis to judge, both for the model and for the therapist watching the video feed. This 3D companion reconstructs the same joint skeleton as genuine geometry in space — orbit the scene and the shoulder-abduction arc, the elbow's forward/back sway and the offset between the camera-only estimate and the wearable-fused correction become visible in a way a flat 2D projection cannot show.

What's rendered, and how it maps to the 2D mechanic

The joint-angle math is carried over unchanged from the original simulator: the target trajectory follows target(t) = 90° + sin(t·0.018)·80°, technique noise scales with (5 − difficulty)·4, and the live camera-tracked angle deviates from that reference by the same formula. Whenever the deviation exceeds the adjustable clinical threshold, the live skeleton turns red — exactly the "bad form" trigger from the 2D version, now visible as a real 3D pose rather than a 2D silhouette.

Toggling Sensor fusion swaps the raw camera skeleton for a Kalman-corrected reconstruction that converges toward the reference trajectory, echoing the camera+IMU fusion step described in the original: the camera supplies absolute position, the wearable IMU nodes (shown as glowing octahedra at the wrist, shin and waist) supply high-frequency orientation that smooths out camera jitter and tracks the limb even when framing is imperfect.

Reading the scene

The blue skeleton is the live camera estimate; the translucent green ghost is the therapist's reference trajectory recorded at the clinic; the purple skeleton (when fusion is enabled) is the corrected reconstruction. A pulsing purple bar above the shoulder stands in for surface EMG amplitude — its height tracks simulated deltoid activation, the same muscle-activity signal the original uses to infer fatigue. Small glowing packets travel along the network link between the patient's and therapist's positions at a rate set by the bandwidth slider, mirroring the original's WebRTC round-trip-time model (RTT = 190 − bandwidth·8 ms).