Articles

Toward AR Navigation in Laparoscopic Liver Surgery: Multimodal Tissue Marker-Based CT-NIRF Registration with Machine-Learning Object Detection

Koki FUJIMAGARI, Sayaka KAMANO, Natsuki MORITA, Kouta AITAKE, Toshiya NAKAGUCHI, Hideki HAYASHI, Nahoko NAGANO
JSCAS Special Issue, Advance Publication

Modern hepatectomies are planned along anatomical segments of the liver, and intraoperative injection of indocyanine green enables visualization of these segments using near-infrared fluorescence (NIRF) laparoscopy. However, subsegmental resections that extend partially beyond boundaries of the segments are increasingly attempted in patients with liver insufficiency, requiring additional anatomical understanding of the lesions. Recently, augmented reality (AR) has gained attention as a technique for overlaying preoperative anatomical information directly onto the surgical field. Nonetheless, achieving real-time and precise alignment of preoperative data onto surgical objects remains a challenge. To address this issue, we propose an AR system using a multimodal tissue marker detectable by both X-ray and NIRF, and an object-detection machine-learning model (YOLOv9). The multimodal AR marker was administered to excised porcine liver samples, and their NIRF laparoscopic and X-ray computed tomography (CT) images were acquired. Marker coordinates in the NIRF laparoscopic images were obtained using the YOLOv9 model, and the correspondence between markers in the CT and NIRF views were automatically determined using our original algorithm. CT marker images were reprojected onto the laparoscopic images based on their coordinates. The system achieved an 80.2% marker detection success rate in NIRF images, a 78.4% registration success rate, and a mean reprojection error of 3.9 pixels, with an average processing speed of 28.9 fps. These results demonstrate the potential of our proposed system to realize real-time AR navigation during laparoscopic liver surgeries. Although clinical application would require a hybrid operating room equipped with CT at the time of marker administration, this technology has the potential to superimpose internal structures of the liver onto the organ surface in the laparoscopic view on video monitors.

READ FULL ARTICLE ON J-STAGE