Augmented Reality App Increases Biopsy Accuracy

Combining simultaneous localization and mapping (SLAM) and augmented reality (AR) technologies could enable rapid reconstruction of three-dimensional (3D) body sections using a smartphone.

Developed at the University of Twente (UT; Enschede, The Netherlands), the AR visualization and guidance smartphone application will display internal layers of the body on the skin surface in order to visualize invisible information. Using information collected from magnetic resonance imaging (MRI), vein-scanning devices that use laser speckle, and other sources, doctors can see the inner body segments, tumors, veins, and the status of diabetic ulcers with the use of AR. The information can be visually layered as procedure progress.


SLAM systems construct or update a map of an unknown environment while simultaneously keeping track of an agent's location within the same environment. According to the researchers, SLAM localization and 3D AR reconstruction software can be used with a multitude of imaging sensors, and not only with smartphones and their cameras, as the algorithms they developed can quickly understand 3D models of surrounding objects, regardless of the type of sensors used.

“Even though I didn't have a medical background, our common focus on SLAM and AR provides a great platform. Visual support enables doctors to oversee the situation better and make better decisions,” said senior researcher Beril Sirmacek, PhD, of the UT Robotics and Mechatronics (RAM) research group. “In a biopsy situation, this visual support can help with guiding the robot arms to reach the tumor for biopsy at the first attempt, instead of taking the off-chance and reach for the correct location by working for a whole day and making unsuccessful biopsy holes on the patient's body.”

“SLAM is a core technology in robotics, but its universal usage will only be possible if we have methods to easily interface many sensors in a robotic system,” said Professor Stefano Stramigioli, PhD, chairman of the RAM research group. “We intend to create a modular SLAM box which will automatically reconfigure itself if extra sensors are attached in a ‘Plug and Play’ fashion. Then it would be possible to connect this SLAM Box to a complete robotic system, giving it powerful perception capabilities.”

An example of functioning SLAM technology are self-driving cars, which make extensive use of highly detailed map data collected in advance. This can include map annotations such marking locations of individual white line segments and curbs on the road. Essentially, such systems simplify the SLAM problem to a simpler localization only task, perhaps allowing for moving objects such as cars and people only to be updated in the map at runtime.

Related Links:
University of Twente