Virtual reality in medicine

Computer modelling and simulation have become increasingly important in many scientific and technological disciplines owing to the wealth of computational power. Calculation of the behaviour of these computational models is increasingly replacing experiments on real world objects—for example, in the car industry (tests based on simulated crashes) or in the development of nuclear weapons—and is becoming an indispensable tool in the development of new products and procedures.

Likewise, the development of techniques for acquiring data (for example, medical imaging) has enabled the easy generation of high resolution copies of real world objects from the computer's memory. The development of imaging technologies, such as magnetic resonance imaging, computed tomography, and ultrasound, has made the acquisition of highly detailed anatomical and partially functional models of three dimensional human anatomy a routine component of daily clinical practice.

Summary points

• The principal aim of virtual reality technology is to present virtual objects or complete scenes to all human senses in a way identical to their natural counterpart

• Simulated three dimensional reconstruction of organs from radiological cross sections is an important diagnostic tool by providing clinicians with a more naturalistic view of a patient's anatomy

• Preoperative planning with the use of a computer including realistic prediction of the outcome has become an integral part of the intervention in certain disciplines, such as radiation therapy, craniofacial surgery, or neurosurgery

• Computerised three dimensional atlases of human anatomy, physiology, and pathology are about to revolutionise the teaching of these subjects

• Several virtual reality systems have been developed and tested for the physical or mental rehabilitation of patients and for supporting mental health therapy

• Virtual reality technology plays an important role in telemedicine from remote diagnosis to complex teleinterventions

Technical data for virtual reality devices

Display screens

Usual screen resolution: 1024×768 or 1280×1024 pixels

In stereoscopic mode the vertical resolution is halved: 1024×384 or 1280×512

For head mounted displays resolution is usually lower: starts at 400×200 pixels

Necessary update rate for smooth visual appearance is miniumum 25 Hz (monitor refresh rate for flicker free viewing is much higher—72-100 Hz—which also doubles in stereoscopic mode)

Force feedback

Maximum exertable force: 5-25 N

Continuous exertable force: 1-3 N

Usually 3-6 degrees of freedom force driven

Active working space usually from 15 cm×15 cm×15 cm up to 50 cm×50 cm×100 cm

Necessary force update rate for vibration free force sensation: 500-1000 Hz

Tracking devices

Position accuracy: 0.1-2 mm, 0.1-1°

Working range: 1-10 m

Update rate 20-500 Hz

For a long time the examination of, and interaction with, these computational models was reserved for specialists who were able to understand the limited representation of data offered by computer programs. In the case of medical imaging the mental reconstruction of three dimensional anatomical objects from slices of images in cross section, as usually presented on a light box or a computer screen, is not a natural cognitive ability of humans. Radiologists need intensive training and extensive experience to cope with this task.

Virtual reality technology^1–5 aims at closing the gap between the capability of present technology to acquire images and properties and then to calculate the behaviour of virtual objects, and the ability to observe and interact with them. The ultimate goal is to allow the presentation of virtual objects to all of the human senses in a way identical to their natural counterpart.⁶ In some applications real and virtual objects need to be integrated making it necessary to present and manipulate them simultaneously in a single scene, leading to the development of hybrid systems referred to as augmented reality systems.⁷

The technology

Computer graphics

Three dimensional computer graphics initiated the development of virtual reality technology several decades ago.⁸ The principal aim is to generate images with virtual objects or complete scenes in a near photorealistic manner. Two major components are needed to reach this goal: appropriate algorithms to calculate the visual appearance of the virtual scene to be visualised (rendering); and physical devices (in most cases graphical displays) to present the resulting images to the user.

A vast collection of methods has been developed during the past decades for rendering. They basically simulate the interaction of light with the geometry of the virtual objects, which can be represented as a collection of surfaces (surface rendering) or as volumetric models (volume rendering). In order to achieve real time performance for large objects different specialised accelerator units have been developed, which are readily available today—even for personal computers. The level of realism can be considerably improved by the use of texture mapping techniques, allowing the coloration of the represented surfaces by detailed photographic images.

The results of these rendering algorithms are usually presented on a video monitor. Since stereoscopic vision is a fundamental condition of realistic immersion into a virtual scene substantial effort has been made in the development of stereoscopic display systems. The basic technique of these devices is to present the scene from a slightly different point of view for each eye. The simplest way to achieve this separation is the use of polarised glasses, which selectively filter the images appearing on the screen. Another solution is provided by head mounted displays, with a completely separate screen for each eye. New technological developments based on the construction of special display screens or holography promise to eliminate the need to wear special equipment while maintaining fully stereoscopic scene presentation. The usual resolution of commercial stereoscopic displays is around 1200×500 pixels, whereas cheaper head mounted displays provide a much worse resolution of around 400×200 pixels.

Interaction devices and trackers

In the early years of computer graphics the results of the rendering algorithms were presented as static images or precalculated animated sequences (movies). The graphical rendering power of today's computers allows the real time calculation of scene appearance—which requires a minimum of 25 images per second—making the visual exploration of, and interaction with, virtual scenes possible. Besides classic computer devices, such as the keyboard and mouse, a rapidly growing group of devices—for example, three dimensional pointers or data gloves—allows increasingly realistic interactions with virtual objects.

Owing to the rapid development of new paradigms of interaction the user is becoming increasingly an integral part of the virtual scene. To facilitate this integration and the resulting immersion it is indispensable to provide information about the actual physical status of the observer including his or her position, gaze direction, and gestures. Such information is acquired by tracking devices, which follow certain anatomical structures of the user or external landmarks attached to the display or to interaction devices. A large number of commercial systems are available for the determination of three dimensional landmark position, based on visual and optical tracking technology or magnetic tracking technology. These units allow position determination up to submillimetre accuracy within a working distance of several metres.

Force and tactile feedback

Tactile and haptic (force) sensing is an important source of information when interacting with the virtual environment. Accordingly, these forces have to be transmitted to the user if immersion in a virtual scene with a reasonable level of realism is desired. Although research into the development of devices with haptic or tactile feedback (often called haptic or tactile displays)⁹ has been considerably less intensive than with their visual counterparts a comparatively large number of devices has already been developed and some are commercially available—even specifically for medical applications.¹⁰

Tactile feedback is usually provided by actuators using thermal, vibrational, or mechanical stimulators. Especially popular are the tactile displays based on a two dimensional array of individually actuated miniature rods, similar to devices used to present the braille alphabet for the blind.

Force feedback is usually generated by “inverse” robot units, where the electric motors of the device do not generate active motion but are used to counteract the movement of the user. Currently available instruments range from simple three dimensional pointers to large complex exoskeletal devices, which have a large number of degrees of freedom for human joints coupled with force feedback. Commercial devices allow the generation of a peak force in the range of 10 Newtons. The requirement for update rates of tactile or force feedback for realistic force sensation are much higher than for visualisation, usually up to 1 kHz.

Other sensory feedback

Realistic representation of auditory information is technologically less demanding than visual or tactile simulation. Comparatively inexpensive devices used in consumer electronics have been readily available for a long time and are easily integrated into virtual reality systems.¹¹ Interestingly, sound is seldom used in medical applications and is very rarely part of the simulation of a medical virtual environment. In few applications auditory signals are used selectively to augment information during interaction with the scene, such as indicating the distance of a device from an object.

Currently the integration of the sense of smell into virtual environments is exclusively a research issue.¹² Olfactory displays are under development for the computer controlled generation of a wide range of possible odours. Most currently envisioned applications are in training—for example, combat and fire fighting.

Medical applications

In the past decade medical applications of virtual reality technology have been rapidly developing, and the technology has changed from a research curiosity to a commercially and clinically important area of medical informatics technology.²,¹³ Research and development activity is well summarised by the yearly “Medicine Meets Virtual Reality” meetings,^14–17 and the commercialisation of the technology is already at an advanced stage.¹³

Diagnostics

Initially, algorithms for graphical rendering of anatomy have been used to provide support for three dimensional organ reconstruction from radiological cross sections. For the clinician this method of visualisation provided a more natural view of a patient's anatomy without losing the see through capability of the radiologist.

Virtual endoscopy techniques (such as virtual colonoscopy or bronchoscopy) based on the virtual reconstruction and visualisation of individual patient anatomy are rapidly developing. Owing to the potential benefits of patient comfort and cost effectiveness virtual endoscopic procedures could replace real endoscopic investigations in the foreseeable future in some areas of diagnosis. The most impressive development has been demonstrated in virtual colonoscopy as a screening tool for colon polyps and cancer and which is currently in the clinical validation phase.

Preoperative planning

In many areas today the use of computer models to plan and optimise surgical interventions preoperatively is part of daily clinical practice. In some areas, such as conformal radiotherapy and stereotactic neurosurgery, treatment is not possible without preoperative planning with the aid of a computer. In other areas, such as craniofacial neurosurgery and open neurosurgery, the possibility of planning surgery on a computer screen, trying out different surgical approaches with realistic prediction of the outcome (for example, postoperative appearance of the patient), and planning individualised custom made implants have substantial impact on the success and safety of the intervention.

Education and training systems

Education and training is one of the most promising application areas for virtual reality technologies. Computerised three dimensional atlases presenting different aspects of the anatomy, physiology, and pathology as a unified teaching atlas are about to revolutionise the teaching of anatomy to medical students and the general public.

Systems based on virtual reality offer a unique opportunity for the training of professional surgical skills on a wide scale and in a repeatable manner, in a way similar to the routine training of pilots. Contrary to the preoperative planning systems, which require an extreme level of accurate registration and alignment of tissue (data fusion), medical and surgical education and training rely more on high fidelity visualisation and realistic immersion into the virtual scene than on the precise data fusion of the applied models with the specific anatomy of a patient.

The rapid adoption of minimally invasive surgical techniques is one of the major driving forces in the development of surgical trainers. The extreme limitations placed on visual and manipulative freedom, including the loss of tactile feedback and the unusual hand-eye coordination, makes extensive specialised training for such interventions necessary. Virtual reality is the technology of choice with the greatest potential for future development, and a rapidly growing number of commercial units is becoming available.

Image guided surgery

Even the best preoperative planning is of limited use if its implementation in the operating room is not guaranteed. Whereas traditionally these plans are transformed mentally by the surgeon during the intervention, computer assistance and virtual reality technology can substantially contribute to the precise execution of preoperative plans.¹⁸

Image guided surgery is the typical application area where virtual objects (data from the preoperative image and the anatomical objects extracted from them) and real objects (the patient and the surgical tools) must be merged into a single unified scene, calling for augmented reality techniques. The major technical issue to be solved is the registration of the real and virtual objects—that is, to make the preoperative data coincide with the actual patient anatomy—and the tracking of the movement of real objects such as the surgical instruments.

Although still needing substantial research image guided surgery is one of the major development areas today, with several systems in routine clinical practice, especially in orthopaedics and neurosurgery.

Other application areas

Virtual reality offers promising solutions in many other areas of medical care, where the immersion into a virtual world can help the patient, the physician, and the developer of the technology. Several systems have been developed and tested for physical or mental rehabilitation and for supporting mental health therapy by exposing the patient to appropriate experience or illusion. Finally, virtual reality based technology plays a major role in telemedicine, ranging from remote diagnosis to complex teleinterventions.

Conclusion

Virtual reality based technology is a new but rapidly growing area in medicine, which will revolutionise health care in the foreseeable future. The impact of this technology is just beginning to be recognised owing to methodological, technical, and manufacturing breakthroughs in the past few years. It must, however, be emphasised that the technology is simply a tool and that the other critical areas of content development and physician-patient relationship must be incorporated into the new systems.

Figure.

NLM/VISIBLE HUMAN INDEX (FEMALE)

Part of the vertebral column

Figure.

NLM/VISIBLE HUMAN INDEX (FEMALE)

Complex model containing the most important organs of the abdomen

Figure.

NLM/VISIBLE HUMAN INDEX (FEMALE)

NLM/VISIBLE HUMAN INDEX (FEMALE)

a) Liver: inter-operative photo (left). b) Liver: virtual organ coloured by synthetic texture (right)

Figure.

NLM/VISIBLE HUMAN INDEX (FEMALE)

The colon related to the original crysectional slices