Surgeons rehearse the operation on a patient-specific holographic 3D model before entering the OR
Mixed-reality preoperative rehearsal begins long before a headset is put on. A patient's CT or MRI series — hundreds of 2D axial slices — must first be segmented into discrete anatomical structures (bone, vasculature, tumor, nerve, viscera) and converted into a lightweight, manipulable 3D mesh that can be rendered in real time on a holographic display. The result is not a static model but an interactive, patient-specific hologram that a surgeon can walk around, resize, peel back layer by layer, and slice open at will — all before the patient is ever wheeled into the operating room.
The pipeline that produces a surgical hologram runs through several distinct computational stages:
• DICOM import: CT/MRI series (often multi-phase contrast CT or T1/T2 MRI) loaded from PACS, anonymized, and resampled to isotropic voxel spacing.
• Segmentation: each anatomical structure is isolated by intensity thresholding, region growing, or (increasingly) a deep convolutional network such as a 3D U-Net trained on organ-specific atlases. Vasculature is segmented from contrast-enhanced phases; tumor margins are drawn or auto-contoured and reviewed by a radiologist; bone is trivial by threshold, soft tissue is not.
• Mesh extraction: each segmented volume is converted to a surface mesh via marching cubes, then smoothed and decimated (typically from several million triangles down to 1–3 million) so it can be rendered at 60 fps on a standalone headset GPU.
• Color and material assignment: vessels rendered translucent red/blue by arterial or venous phase, tumor rendered in a distinct warm hue, bone opaque white — mirroring conventions radiologists already use so the hologram reads instantly.
• Export to the MR runtime: the finished asset is packaged (glTF/FBX) and loaded into a surgical planning application — commercial platforms include Surgical Theater's Precision VR/AR, EchoPixel True 3D, Novarad OpenSight, and custom Unity/HoloLens pipelines built by academic centers such as Stanford's Lucile Packard Children's Hospital for congenital heart disease planning.
Unlike a printed 3D model, the hologram remains fully interactive after generation: a surgeon can toggle structures on and off, cut cross-sectional planes anywhere in space, and measure distances directly on the model — capabilities that make the holographic format strictly more useful than either 2D PACS review or a static physical model, at a fraction of the turnaround time and cost of 3D printing.
Radiologists and surgeons have reviewed 3D reconstructions on flat 2D monitors for two decades — so what does true stereoscopic holography add? The key is binocular depth perception combined with real-world scale and hands-free manipulation. On a 2D monitor, a 3D rendering is still a 2D projection: depth is inferred from shading, occlusion, and rotation, and the viewer cannot naturally reach "into" the model. A hologram viewed through a headset presents a genuinely stereoscopic image locked to real-world coordinates — the surgeon can lean in, walk around the back of the tumor, or bring their own hands into the same space to gesture at a vessel, exactly as they would examine a physical specimen.
This matters most for spatially complex cases: a horseshoe kidney with anomalous vasculature, a skull base tumor abutting the carotid, or a single-ventricle congenital heart defect where the three-dimensional relationship between chambers and great vessels is the entire surgical problem. Multiple published case series in congenital cardiac surgery (holographic review of hypoplastic left heart and double-outlet right ventricle anatomy) and in craniofacial/skull-base surgery report that surgeons identify spatial relationships in the hologram that were not appreciated on 2D CT/MRI review alone, and that holographic review changes the stated operative plan in a meaningful minority of complex cases even before the rehearsal-with-instruments stage begins.
A holographic model becomes far more powerful once it can be viewed by an entire surgical team at once. Shared mixed-reality sessions let the primary surgeon, assisting surgeon, anesthesiologist, and trainees each wear a headset and see the identical hologram, correctly registered in the same physical room, from their own vantage point — turning the traditional flat-screen case conference into something closer to gathering around a real patient on the table.
Rendering the same hologram identically for multiple simultaneous viewers requires solving a distributed spatial-anchoring problem:
• Spatial anchors: each headset independently maps its surrounding room using SLAM (simultaneous localization and mapping); a shared anchor point is established and broadcast so every device agrees on where the hologram sits in physical space, typically to within a few millimeters.
• State synchronization: a session server (often Azure Spatial Anchors or an equivalent networked backend) keeps hologram transform state — position, scale, active cross-section plane, visible layers — consistent across all connected headsets in near real time, so when one surgeon rotates or slices the model, every other viewer sees the same change instantly.
• Per-user gaze and gesture avatars: each participant is represented to the others as a small headset/name marker so the team can see where a colleague is looking or pointing, replicating the "look where I'm pointing" dynamic of an in-person huddle around an actual patient.
• Voice + spatial annotation: participants speak naturally (spatial audio anchored to their avatar position) while dropping persistent 3D annotation markers directly onto the hologram — "clamp here," "watch this branch" — that remain attached to the anatomy for the rest of the session and can be reviewed later.
The practical benefit is that case conferences move from a radiologist narrating a 2D scan on a shared screen to every team member independently exploring the same 3D anatomy, asking their own questions, and reaching consensus on the approach faster and with fewer misunderstandings about spatial relationships — particularly valuable for trainees, who benefit from being able to walk around anatomy rather than infer depth from a flat image.
This is the rehearsal proper: using tracked hand controllers or hand-tracking gestures, the surgeon manipulates virtual instruments — a scalpel, a needle, a drill trajectory, a catheter path — directly against the patient's hologram. Multiple candidate approaches are tested in minutes, something impossible to do safely on the actual patient, letting the surgeon converge on the entry angle, retraction plane, or resection margin that best avoids critical structures.
Unlike passive review, trajectory rehearsal is an active, iterative simulation loop:
1. The surgeon selects a candidate entry point and approach vector on the hologram using a tracked virtual instrument. 2. The system renders the projected trajectory as a line or cone through the 3D anatomy, immediately showing which structures it would pass near or through — a vessel, a nerve bundle, healthy parenchyma. 3. The surgeon adjusts angle, depth, or entry point and re-tests, iterating through several candidate trajectories in the same session — each rehearsal taking seconds rather than the operative minutes a real attempt would cost. 4. For resection cases, the surgeon can rehearse the resection plane itself, checking margin clearance around a tumor from multiple viewing angles simultaneously, something not achievable by rotating a single 2D image. 5. The final selected approach is saved back into the plan and can be exported as an overlay for intraoperative navigation systems, effectively transferring the rehearsed trajectory into the real OR.
This mirrors, in miniature, the flight-simulator logic long used in aviation and increasingly in complex neurosurgical and orthopedic navigation: mistakes made in rehearsal cost nothing, while the same mistake made for the first time on the actual patient can be catastrophic. Neurosurgical and orthopedic literature on MR-assisted approach planning (e.g., pedicle screw trajectory rehearsal, ventriculostomy catheter placement, skull-base corridor selection) consistently reports that rehearsal shortens the number of adjustments needed once the real instrument is in the patient, and that surgeons rate their confidence in the chosen approach measurably higher after rehearsal than after 2D-imaging review alone.
The single highest-value moment in mixed-reality rehearsal is when it fails to confirm the existing plan. An aberrant hepatic artery, a tumor that wraps further around a vessel than the radiology report suggested, an unexpectedly short pedicle — these variants are sometimes genuinely difficult to appreciate on 2D slices but become obvious the moment a surgeon rotates a 3D hologram and tries to rehearse an approach through it. Catching the variant here, days before surgery, converts an intraoperative surprise into a planned adjustment.
2D axial, sagittal, and coronal slices require the reviewer to mentally reconstruct a 3D structure one plane at a time — a skill radiologists and surgeons are highly trained in, but one that is cognitively demanding and error-prone for genuinely unusual anatomy, especially branching vascular variants that snake obliquely through multiple slices. A hologram removes that reconstruction step: the vessel is already assembled in 3D, and an aberrant branch that would require careful cross-referencing across a dozen 2D slices to notice is immediately visible as a shape that does not match the expected pattern.
Rehearsal amplifies this further. A variant that is subtle when the hologram sits passively in space becomes unmistakable the moment the surgeon tries to rehearse the planned approach through it and the virtual instrument path collides with a structure that was not accounted for. This is functionally similar to a "near miss" caught in a simulator: the system does not need to explicitly flag the anomaly — the rehearsal itself forces the discovery, because the plan simply does not work as originally conceived once tested against the true 3D anatomy.
When a variant is confirmed, the standard workflow is to convene the team (often the same shared MR session used for initial review) to walk through the revised approach together, update the documented operative plan, and — where relevant — flag the finding back to the imaging/radiology team so the reporting protocol for similar future cases can be refined. Because this happens preoperatively, the revised plan can be discussed calmly with the full team rather than decided under time pressure with the patient already open.
The ultimate test of any preoperative rehearsal technology is what happens on surgery day. Across published case series in cardiac, craniofacial, orthopedic, and oncologic surgery, mixed-reality-rehearsed cases are consistently reported to run faster, encounter fewer unplanned intraoperative findings, and leave trainees measurably more confident than teams that planned from 2D imaging and static 3D renderings alone.
Mixed-reality preoperative rehearsal is still a relatively young clinical practice, so the evidence base is built primarily from single-center case series, comparative cohort studies, and a growing number of small randomized or matched-cohort trials rather than large multicenter RCTs. Within that literature, several consistent findings recur:
• Congenital cardiac surgery: teams using holographic 3D reconstructions for complex single-ventricle and conotruncal anomaly planning (pioneered clinically at centers including Stanford's Lucile Packard Children's Hospital, using HoloLens-based platforms) report that the hologram changed the surgical strategy in a meaningful subset of cases compared to the plan derived from 2D imaging and echo alone, and that surgeons found the spatial relationships of great-vessel anomalies substantially clearer.
• Craniofacial and skull-base surgery: holographic rehearsal of tumor resection and reconstruction plans has been reported to shorten operative time and improve margin planning around critical neurovascular structures, by allowing surgeons to test multiple resection corridors before the first incision.
• Orthopedic and spine navigation: MR-assisted rehearsal of screw trajectories and osteotomy planes has been associated with fewer intraoperative trajectory adjustments and improved accuracy versus fluoroscopy-guided freehand technique in comparative series.
• Surgical education: multiple studies using pre/post confidence surveys report that trainees who rehearse a case holographically before assisting rate their preparedness and spatial understanding of the anatomy significantly higher than trainees who reviewed only 2D imaging — an effect that also benefits attending surgeons planning genuinely novel anatomy they have not encountered before.
The overall pattern across this literature is directionally consistent even though effect sizes vary by specialty, case complexity, and rehearsal platform: adding a stereoscopic, interactive, rehearsable model on top of standard imaging review measurably improves both the efficiency of surgery and the preparedness of the team performing it, with the largest gains concentrated in the most anatomically complex cases — exactly the cases where 2D imaging leaves the most room for surprise.
In a widely cited 2021 case series from Stanford's Lucile Packard Children's Hospital on holographic 3D reconstruction for congenital heart surgery planning, surgeons reported that reviewing the holographic model changed their stated operative approach in roughly 1 in 4 to 1 in 5 complex cases compared with the plan formed from cross-sectional imaging and echocardiography alone — a striking demonstration that stereoscopic, rehearsable 3D review surfaces clinically actionable information that flat imaging review does not.