Surgery, Virtual Reality, and the Future

Abstract

MMVR has provided the leading forum for the multidisciplinary interaction and development of the use of Virtual Reality (VR) techniques in medicine, particularly in surgical practice. Here we look back at the foundations of our field, focusing on the use of VR in Surgery and similar interventional procedures, sum up the current status, and describe the challenges and opportunities going forward.

Introduction

Richard Satava, MD has been an articulate and effective spokesman for the vision of high tech interventional medicine:

“..most (of the) information that a health care provider needs can be acquired in electronic form (images, scans, vital signs, the medical record). And with the emergence of teleoperation, we can leverage the power of the advanced information tools of software (AI, 3D visualization and decision support), hardware (high performance computing) and networking (the information superhighway). All this will enhance the skills of the health care provider beyond mere physical limitations to enable a quality of care previously considered unachievable. Better access will be provided by remote telemedicine. Lower cost will be achieved through flexible manufacturing, just in time inventory, and best-in-class business management.”

(Proceedings, MMVR III, 1995)

In this paper, we look back at these foundations, focusing on the use of virtual reality (VR) in Surgery and similar interventional procedures, sum up the current status, and describe the challenges and opportunities going forward.

1. The Promise in the Early 1990s

From the beginning it was clear that progress would involve intense collaboration between physicians, scientists, and engineers. Military medicine and the manned space programs were often used as a model for this type of focused effort; some of the early MMVR papers even seemed to adopt the goals of the medical device designers for Star Trek.[1]

Very quickly, efforts were made to organize the information to be presented. For example, the Visible Human Project[2] adopted the terminology and generative syntax of anatomy. This fit well with the vision of displaying a 3D model of the patient's body in an augmented reality view…the simplest concept being to “see beneath the skin.” The anatomic descriptors also matched the essential task of surgery: to remove (or denature) diseased tissue with minimal damage to healthy tissue and its functions. Likewise, the creation of modeling software which corrected for (or accommodated) artifacts and noise in the raw data enabled differentiation and labeling of anatomic structures to simplify communication. As a culmination of these developments, atlases of key organs and organ systems were made to support teaching, identify anatomic variations in a specific patient, and to serve as “strong priors” for image interpretation and cohort analysis.

The development of patient-specific digital models provided a foundation for procedure simulation, and ongoing studies of approaches to adapt these models so that they accurately present conditions during a specific intervention. The two primary goals of VR (or “immersive”) simulators are 1) to enable training in a realistic and consistent environment, and 2) to rehearse an actual procedure, so that the physician does not “practice on the patient.” Real time adapted models ere focused on providing augmented reality data support intraprocedurally.

In 1992, the surgeon was surrounded by opportunity (Figure 1). We were at the peak of a golden age of medical imaging technology established by CT, MRI, and the ongoing conversion to digital image data; the following decade would see an explosion of advanced systems and concepts in several modalities, and a wide range of new diagnostic instruments and approaches. This was matched by comparable potential in telemedicine, robotics, surgical simulation, and minimally invasive surgery:

Figure 1. The opportunities surrounding the surgeon in the early 1990s.

Master-slave Robotics

The development of surgical robotics and surgical telepresence proceeded in parallel, since optimal operation of the instruments was best done sitting at a console connected only electronically to the activity at the surgical site. In 1990, the ROBODOC system for hip replacement had been developed to the prototype stage, Hap Paul, DVM, did the first animal model studies, and IBM infused $3M into the concept to create Integrated Surgical Systems.[3] At the same time, Philip Green and colleagues at the Stanford Research Institute (now SRI International) were completing the configuration of underlying technology for the DaVinci surgical robot. Other early experiments, particularly in neurosurgery, were occurring around the world.[4]

Surgical Simulation and Training

At the beginning of the MMVR era, the development of increasingly accurate and sophisticated digital models and real time interfaces to interact with them was envisioned to provide capabilities for improving training, the capability to evaluate operator performance in a standardized fashion, and the ability to do surgical planning and, eventually, rehearsal.

Minimally Invasive Surgery

In 1990, flexible endoscopy and rigid endoscopes (laparoscopes) were in clinical use, but large incisions still characterized surgical practice. Flexible scopes were used (as they are today) primarily for diagnostic studies; the use of laparoscopes in surgery, pioneered by Semm, was generally restricted to gynecology. Improvements in instrumentation, which would stimulate broader use by general surgeons[5] and other surgical specialties were just being introduced. Of particular importance for visualization was the introduction of video cameras positioned independently from the scopes (generally attributed to C. Nezhat), enabling “operating off the monitor.”

Also, 1990 marked the end of a period when intracranial surgery was a dangerous and unpredictable intervention. Imaging support (Figure 2) to guide procedures was limited, and difficult to utilize in the OR.

Figure 2. Neurosurgical Guidance Display in the early 1990s.

2. Twenty Years of Accomplishment

Although there have been no significant new imaging modalities for surgery guidance (with the possible exception of Cone-beam CT), imaging technology has improved across the board: Resolution is higher, and contrast has been improved by new intralumenal and particularly “molecular targeted” parenchymal contrast agents. In addition to the well established optical tracking devices, small footprint electromagnetic tracking systems, both stand-alone (Ascension Technologies, Northern Digital), and integrated into guidance systems (St. Jude, Biosense-Webster, Brainlab, GE Logiq E9) have made it possible to track the motion of instruments and anatomic targets with high accuracy. Magnetic actuation has recently demonstrated potential for complimenting or conducting therapeutic procedures in workspaces such as the peritoneal cavity.

As impressive as hardware development has been, the advancement of image based software has arguably had greater impact. Imaging system vendors all supply capable workstations for image analysis and display. Several commercial systems bring specialized and ever-more expanding capabilities, while software packages originating in academic labs, such as Richard Robb's team at the Mayo Clinic, provide support to research as well as clinical applications.[6] An open-source system for image analysis and presentation, 3DSlicer,[7] has thousands of worldwide users. Due to its flexibility and large base of active users, the 3D Slicer community is in the vanguard of converting off-line analytic tools into real time actions for direct surgical support.[8,9]

Over these two decades, surgical practice evolved through the broad acceptance of minimally invasive techniques, particularly laparoscopy. This change was not driven by improvement in long term clinical outcomes, but rather by less traditional criteria such as shorter recovery time and, to some extent, cosmesis, primarily through pressure from patients. The process was markedly unsystematic, as noted by Cuschieri,[10] but it spawned a significant effort to implement training protocols and certification criteria to ensure patient safety, which is still ongoing through organizations such as the Society of American Gastrointestinal and Endoscopic Surgeons and the American College of Surgeons.[11]

Attempts to build on the success of laparoscopic surgery have has less spectacular results. Following a logical train that, if a few small skin incisions are better than open surgery, then only one incision is better yet, so “single port” systems have proliferated. It is not yet clear that the cosmetic advantages will outweigh the difficulties of accomplishing both surgery and imaging through a single aperture. Likewise, the extension of the image display from 2D to 3D has moved slowly, at best.

Orthopedic surgery, and particularly hip replacement, was an early target for robotic techniques because it appeared that higher precision would lead to better long term functionality (fewer joint displacements) and less traumatic procedures, leading to faster recovery. Both ROBODOC,[3] and HipNav[12] were developed extensively, but have not displaced existing, less refined approaches. The “lesson learned,” in attempts to automate hip (and knee) prosthetic surgery, is that the body adapts remarkably well to changes in joint structure, thus limiting the benefits of these more accurate procedures

By contrast, the DaVinci surgical system has had significant commercial success. While the current market for this massive system is driven by patient demand for prostate cancer resection, it may be that DaVinci's success is due its configuration as a surgical platform with many potential applications, so it could be adapted to serve a market need that was identified well after product introduction.[13]

For both minimally invasive and robotic approaches, the trend toward minimal access implies that VR will be even more helpful, because the surgeon's natural view is more constricted. The past decade has seen a steady improvement in the realism of simulated anatomic features for training and surgical rehearsal and all of us now access YouTube videos routinely to better understand surgical procedures and techniques.

3. The Status Today

The benefits of VR to clinicians today include 1) Higher quality simulation, planning, and patient staging, 2) Improved capabilities for interventional radiologists and coupling of diagnostic radiology expertise, nuclear medicine, and radiation therapy. In particular, Neurosurgery has become a routine elective procedure with most patients leaving the hospital in a few days, usually in better or an equivalent neurologic condition than they were preoperatively. These astounding advances are due to technical innovation in imaging, visualization, and operative techniques that allow the surgeon to have a much better understanding of the anatomy and pathology that are the targets of the intervention.

However, integrating these capabilities has gone slowly; procedures are getting longer, and, the physician's job has become harder. Current technology levels do not provide the sensing, computational, and display technology to provide up to date, lag-free augmented reality displays that are helpful to the surgeon.

For example, much attention has been paid to the deformation of the liver. Several commercial and academic prototype systems (the MeVis effort,[14] for example) have addressed liver deformation, particularly using real time ultrasound updates, but these are still struggling to achieve broad use.[15]

Except for hi-tech microsurgery, which was already established in the early 1990s for cataract and retinal surgery, the accuracy and precision of robotic approaches haven't made them the standard of care. For example, as noted above, high precision robotics orthopedics does not appear to provide better functional results for hip joint replacement (for the typical patient) than more traditional approaches.

With notable exceptions, such as robotic prostate surgery, it has been difficult to study large numbers of patients. Technology prove-out is often stuck in the prototype stage, so the benefits of new approaches are difficult to measure convincingly.

In general surgery, the evolution of procedures from laparoscopic to single port to “natural orifice translumenal endoscopic surgery” (NOTES) is not getting traction. Challenges include difficulty with closing the translumenal entry, but more significantly, the lack of an attractive application to motivate the next stage of instrument engineering and guidance development.

In neurosurgery, there still remain many times when neurosurgeons find themselves uncertain of how to proceed due to a lack of information. Both pre-operative planning and intra-operative decision-making need to consider the functional organization of the brain tissue around the lesion to avoid causing a new neurologic deficit. However, differentiating critical functional areas from areas that can be resected is not possible either on conventional imaging or by visual inspection at the time of surgery. To decide whether surgery is feasible for a patient with a given lesion, the surgeon requires a complete and accurate map of the complex and critical functional anatomy of that individual's brain. New imaging tools give surgeons information about the relationship of functional brain areas using advanced imaging such as fMRI and DTI. (Figure 3). Intra-operative imaging allows visualization in the operating room of features not visible to the naked eye, such as the presence of residual tumor. These capabilities have yet to be effectively integrated into procedures.

Figure 3. Neurosurgery Guidance in 2012. Anatomic presentation (left), Processed Diffusion Tensor Imaging (right).

4. Future Challenges and Approach

We are now faced with parallel challenges: first to adapt the rapidly evolving materials, electronics, and information technologies to overcome the remaining barriers to their practical, daily use in the operating and procedure rooms. We seem well motivated and resourced to do this. As important, however, is to return to the basic task of making “useful tools” for surgeons. For many years to come, effective surgical therapy will depend on the skills and performance of these individuals; we should not only provide the best environment for them to work in, but also seek technology-based approaches to make their tasks easier as well as more effective, perhaps as follows:

1. Address the problems surgeons really have, not what engineers think they have. Implement technology that adapts to and faithfully represents the patient's anatomic, functional, physiologic status

2. Supply only the information that is needed, when it's needed.

3. Augment rather than redefine the workflow.

4. Focus on more natural interfaces. Move toward a systems architecture that supports continuously updated, easily displayed information. Use the information available to update patient models, etc.

5. Extend the procedure time only if there is major benefit, such as a significant reduction in repeat procedures

6. Aim for “zero overhead” by eliminating time requirements for setup and calibration, so the surgeon is never kept waiting by slow technology performance.

5. Summary

As we have moved from the eras of hand-built graphics cards and CRTs, through refrigerator-sized SGI Reality Engines and LCD headsets, to iPads and GPU computing, the intersection of graphical computing and medicine has continued to promise significant improvements to human health. As we look ahead to lightweight commodity displays like Google Goggles connected to supercomputing clouds applying Big Data to show the optimum disease management path and the best treatment for individual patients, it is hard not to be optimistic that the founding dreams of the field are becoming attainable, particularly as they evolve to incorporate the collected experience of the practitioners.

At the same time, this history makes clear that our hardware and software systems will always be made obsolete within a few years, but that the human body in health and disease is far more complex than our systems will be able to fully model anytime soon. The physician's gift has always been to find the relevant aspects of the patient's condition that can be captured efficiently while providing useful information for his or her care.

We are, perhaps, halfway to our goals. Defining success as more efficient and effective procedures and environments to care for patients, the greatest impact will come from following two paths: 1) Improve technology so that it supports enhanced surgeon performance seamlessly. 2) Re-engineer the technical systems supporting the surgeon (and all caregivers) to make each caregiver optimally effective.