'Google Maps' for Surgeons to Help Perform Complex Robot-Assisted Esophagectomy

Robot-assisted surgery, especially for complex procedures such as removing parts of the stomach and esophagus in esophageal cancer patients, allows for greater surgical precision and quicker recovery with fewer complications. However, despite these advantages, training new surgeons to use robotic systems like the Da Vinci robot remains a lengthy and challenging process, even for experienced surgeons. These procedures require the surgeon to operate from a distance, relying solely on a 3D screen to visualize the surgical field, which can lead to disorientation and fatigue during long and intricate operations. Now, researchers have developed a system that adds real-time guidance to the surgeon’s view—helping them know exactly where they are in the body and what steps remain during surgery.

The solution, developed by researchers at Eindhoven University of Technology (TU/e, Eindhoven, Netherlands) in collaboration with UMC Utrecht (Utrecht, Netherlands), is part of the IntraSurge project. It involves creating an augmented reality navigation and orientation system often described as a "Google Maps for the surgeon." The system layers vital anatomical and procedural information onto the real-time video feed used in robotic surgery. By recognizing key anatomical structures such as the pericardium, lungs, large blood vessels, and nerve bundles, the platform helps orient the surgeon throughout the operation.

The researchers are now moving from 2D recognition to integration with 3D images, providing depth perception and allowing surgeons to “see” beyond obstructions like fat or blood using preoperative CT scans. This visual enhancement not only improves spatial awareness but also allows visualization of deeper, hidden organs and tissues, adding critical guidance during surgery. The final stage will incorporate step-by-step procedural instructions, enabling surgeons to track their progress and anticipate upcoming actions during the operation.

Early testing of the IntraSurge system has shown promising results. Surgeons in training who were involved in testing expressed enthusiasm about the enhanced spatial awareness and clarity the system offers. The first-stage results, such as successful 2D recognition and automatic material tagging, have already led to published research and awards, including first prize at the SPIE conference on medical imaging. The researchers now aim to transition their work from the lab to the operating room, collaborating with industrial partners to ensure their work does not end up on a shelf. Their vision is to improve training efficiency, reduce surgeon fatigue, and lower the risk of errors during long, complex procedures.

“Just as it can be nice for a driver to turn on the sat nav on a known route, it can be equally valuable for experienced surgeons to have this type of information while they work. You know where you are, can anticipate the next turn, and know how much of the journey remains,” said Yasmina Al Khalil, a Postdoc Researcher in the Department of Biomedical Engineering at TU/e." "That's why Jelle likens it to Google Maps. It will provide the same reassurance when you are on the road, but also provide tips and information you need to arrive safely and on time. This will make long operations less exhausting and minimize the risk of mistakes due to fatigue."

Related Links:
TU/e
UMC Utrecht