How AR and Computer Mapping are Leading to Safer Medical Procedures
When performing an operation, there are always risks. Part of the problems arise from not having a clear view. MRIs and x-rays help give doctor a better understanding of a patient’s internal organs, but not until they start to operate does the picture come clear.
At the University of Twente in the Netherlands, a research team is developing an economical smartphone technology based on the Simultaneous Localization and Mapping (SLAM) technology, along with augmented reality (AR). The device allows doctors to reconstruct 3D body sections by directing the smartphone to the areas of interest. The smartphone will display AR layers of the 3D skin surface so that it can help visualize information that would otherwise be invisible to the doctors.
“We would like to layer the information from the MRI and other vein scans, derived from sensor devices using laser speckle, and make them visible with augmented reality,” said Dr. Eng. Beril Sirmacek, a member of Twente’s Computer Science department and the leader of the project. “This approach allows doctors to view the inner body segments, including features such as tumors, veins, and the status of diabetic ulcers with the use of augmented reality.
By combining SLAM technology with AR, Srmacek and her team are trying to construct a smartphone device that could virtually map the human body.
“This visual support enables doctors to oversee the situation better and make better decisions,” Sirmacek added. “In a biopsy situation, this visual support can help with guiding the robot arms to reach the tumor for biopsy at the first attempt, instead of taking the off-chance and reach for the correct location by working with a patient for a whole day and making unsuccessful biopsy holes on the patient’s body.”
According to Søren Riisgaard and Morten Rufus Blas, researchers from MIT, “SLAM consists of multiple parts: Landmark extraction, data association, state estimation, state update and landmark update.” By using the data from the body scans, the computer outputs a continuous update mapping by constantly reorienting the scans and interpreting the new data into updated maps.
The new smartphone device would enable viewing of inner body segments, including organs, tumors, and veins.
“We want our SLAM toolbox software solution to support different cameras like a plug-and-play system,” Srmacek added. “So, this allows users to use higher resolution cameras when needed. After medical approvals are obtained for our SLAM toolbox, we would like it to be commercialized as soon as possible.”
“SLAM is a core technology in robotics, but its universal usage will only be possible if we have methods to easily interface many sensors in a robotic system,” said Prof. Dr. Ir. Stefano Stramigioli, chairman of the Robotics and Mechatronics (RAM) Group, Robotics and Mechatronics. “At RAM, we intend to create a modular SLAM box, which will automatically reconfigure itself if extra sensors are attached in a ‘plug-and-play’ fashion. Then it would be possible to connect this ‘SLAM Box’ to a complete robotic system, giving it powerful perception capabilities.”
