WebXR: 3D Simulations in AR/VR

1. The WebXR session model

Everything starts by requesting a session of a given mode: immersive-vr (fully replaces the view, headset only), immersive-ar (overlays 3D content on the real world via passthrough camera or optical see-through), or inline (renders stereo-capable content directly in the page without taking over the display — useful for a "look around" preview before entering VR).

async function enterVR() {
  if (!navigator.xr) return;  // no WebXR support
  const supported = await navigator.xr.isSessionSupported('immersive-vr');
  if (!supported) return;

  const session = await navigator.xr.requestSession('immersive-vr', {
    requiredFeatures: ['local-floor'],
    optionalFeatures: ['hand-tracking', 'bounded-floor'],
  });

  const gl = canvas.getContext('webgl2', { xrCompatible: true });
  const xrLayer = new XRWebGLLayer(session, gl);
  session.updateRenderState({ baseLayer: xrLayer });

  const refSpace = await session.requestReferenceSpace('local-floor');
  session.requestAnimationFrame(function onXRFrame(time, frame) {
    session.requestAnimationFrame(onXRFrame);
    const pose = frame.getViewerPose(refSpace);
    if (pose) renderStereo(gl, xrLayer, pose);
  });
}

Sessions are ephemeral and user-initiated by design — WebXR requires a user gesture (a button click) to enter, and the browser owns the headset's display buffer via XRWebGLLayer / XRGPUBinding for WebGPU.

2. Reference spaces: local, bounded, unbounded

A reference space defines the coordinate system poses are reported in. Getting this wrong is the single most common cause of a simulation that "teleports" or floats at the wrong height in a headset.

local-floor is the right default for almost every educational simulation: it puts y=0 at the real floor, so a life-size N-body model or a solar system scaled to walking distance feels physically grounded rather than floating at head height.

3. The XR render loop and stereo views

Unlike a normal requestAnimationFrame loop rendering one camera, an XR frame provides a pose with one view per eye (two for stereo headsets, sometimes more for exotic multi-view hardware), each with its own projection and view matrix reflecting the physical inter-pupillary distance.

function renderStereo(gl, xrLayer, pose) {
  gl.bindFramebuffer(gl.FRAMEBUFFER, xrLayer.framebuffer);

  for (const view of pose.views) {
    const vp = xrLayer.getViewport(view);
    gl.viewport(vp.x, vp.y, vp.width, vp.height);
    drawScene(view.projectionMatrix, view.transform.inverse.matrix);
  }
}

Each eye's viewport occupies half the shared framebuffer side by side; the browser compositor handles lens distortion correction and chromatic aberration compensation, so the simulation only needs to render the scene twice with the correct matrices — physics state itself is computed once per frame, shared between both eyes.

4. Input: controllers, hand tracking, gaze

session.inputSources exposes every tracked controller or hand, each with a targetRaySpace (where it's pointing) and, for hand tracking, a full 25-joint skeleton per hand via frame.getJointPose() — enough to detect a pinch gesture for "grab this particle" interactions without any controller hardware at all.

5. AR specifics: hit-testing and anchors

immersive-ar sessions add real-world understanding on top of the same rendering model. The hit-test feature casts a ray from a controller or screen tap into the device's understanding of real surfaces (via its depth sensors/SLAM), and returns candidate poses where a virtual object could be placed — this is how you tap a table and drop a simulated solar system onto it. Anchors then keep that placed object spatially fixed even as the device refines its tracking over time.

6. Performance budget for comfort

VR is far less forgiving of dropped frames than a flat-screen demo: most headsets require 72–120fps depending on hardware, and missing that budget causes visible judder that can trigger motion sickness within seconds, not just look sluggish. The rendering budget is effectively halved compared to a non-XR scene at the same resolution, since both eyes must be drawn every frame.

Practical levers: reduce simulation particle counts for the XR path specifically, use fixed foveated rendering where supported (render periphery at lower resolution), and skip expensive post-processing (heavy AO, bloom passes) that a flat-screen demo could afford.

7. Progressive enhancement and fallback

WebXR support is broad on Android/Quest browsers but essentially absent on iOS Safari as of this writing, so every simulation needs a fully functional non-XR fallback: the same scene, same physics, mouse/touch-orbit camera instead of headset pose. Feature-detect with navigator.xr?.isSessionSupported(...) and only show an "Enter VR" button when the check resolves true — never assume the API exists just because the browser is Chromium-based.