Augmented Reality For Time-Delayed Telsurgical Robotics

Background

Teleoperation brings the advantage of remote control and manipulation to distant locations or harsh or constrained environments. The system allows operators to send commands from a remote console, traditionally called a master device, to a robot, traditionally called a slave device, and offers synchronization of movements. This allows the remote user to operate as if on-site, making teleoperational systems an ideal and often only solution to a wide range of applications such as underwater exploration, space robotics, mobile robots, and telesurgery.

The main technical challenge in realizing remote telesurgery (and similarly, all remote teleoperation) is the latency from the communication distance between the master and slave. This delay causes overshoot and oscillations in the commanded positions, and are observable and statistically significant in as little as 50msec of round trip communication delay.

Predictive displays are virtual reality renderings, generally designed for space operations, that show a prediction of the events to follow in a short amount of time. It can be used to overcome the negative effects of delay by giving the operator immediate feedback from a predicted environment. Furthermore, it does not suffer stability issues that arise with delayed haptic feedback. Early predictive displays included manipulation of the Engineering Test Satellite 7 from ground control where the round trip delay can be up to 7sec and Augmented Reality (AR) rendering where the prediction is overlaid on raw image data. These strategies can be applied to telesurgery, but require overcoming the unique challenges in calculating and tracking the 3D environment for a full environment prediction, which includes non-rigid material such as tissue. Furthermore, prior work in the surgical robotics community highlights the need for active tracking rather than only relying on kinematic calibrations to localize the slave due to the millimeter scale of a surgical operation and the often utilized cable driven actuation.

Technology Description

Researchers from UC San Diego present a complete system, and show through a user study that the result is an efficient AR rendering architecture for streaming stereoscopic displays. They call this a Stereoscopic AR Predictive Display (SARPD). We show that over 30fps for the stereoscopic AR rendering, including lens matching, and 24Hz for the slave-tool tracking from the EKF can be achieved while running simultaneously on a commodity GPU and using the robot operating system (ROS). A user study is carried out to demonstrate its ability to improve the speed of procedures without affecting error rates. Beyond telesurgery, this solution can be deployed to any teleoperated robot with visual feedback once calibrated. Furthermore, it does not require stereo cameras or displays since the real-time slave-tool tracking is done on a single monocular camera data stream.

Applications

There are many potential applications for augmented reality in robotic surgery, both in 'traditional' robotic surgery (non-remote telesurgery), and in remote telesurgery (tele robotics). This technology specifically applies to the potential for remote tele robotic surgery which could open up a wide field of remote telerobotic surgery in the military and civilian arenas.

Advantages

Both prior works and the conducted user study on this system highlight the advantages when utilizing a predictive display for teleoperating under delay. The current findings show SARPD is decreases the time to complete tasks when operating under delay. Reduction of time for operations is critical for patient safety. Furthermore, the AR rendering allows for additional overlays to provide the surgeon additional information during an operation such as highlighting a region and critical patient information.

State Of Development

The architecture for SARPD is complete, and the AR rendering procedure has been optimized by utilizing an OpenGL rendering procedure on GPU. Additional overlay renderings can be easily implemented into the current framework. In addition to using different 3D models to render the prediction, the tool tracking portion of the system can be interchanged with other methods to support different teleoperated robotic systems. While the user study that was conducted shows the importance of SARPD, further studies need to be done for both ex vivo and in vivo.

Intellectual Property Info

This technology is patent pending and available for licensing and/or research sponsorship.

Related Materials

• Florian Richter, Yifei Zhang, Yuheng Zhi, Ryan K. Orosco, Michael C. Yip. Augmented Reality Predictive Displays to Help Mitigate the Effects of Delayed Telesurgery. arXiv.org 21 Feb 2019 - 02/21/2019