Instructive head-mounted display system: pointing device using a vision-based finger tracking technique applied to surgical education

Abstract

Introduction

The head-mounted display (HMD) system is a novel personalized imaging monitoring system for use in a medical setting.

Aim

To support interactive intraoperative communication among HMD wearers, we integrated vision-based finger tracking into our system as a pointing device.

Material and methods

Our vision-based finger tracking system is composed of a commercially available real-time video camera, which is mounted on the modern HMD, and computer software that enables tracking of the tip of the operator's index finger and superimposing the marker on the endoscopic view.

Results

We used this system in an experimental demonstration. The operator used the finger-tracking pointer to explain the intraoperative findings of transurethral resection for bladder cancer to medical students.

Conclusions

This finger tracking system-based pointing device can function as a supportive tool for the HMD system, enabling interactive instruction and communication between the operator and other attending physicians or medical students.

Introduction

Recent technological advancements have led to progress in operative procedures, especially minimally invasive surgery. The da Vinci Surgical System has dramatically transformed endoscopic procedures, which were previously carried out with operators observing a flat panel display [1–3]. The da Vinci Surgical System provides the operator with a three-dimensional view through the immersive viewer at the surgeon console. With this system, the operator sits on the console with his or her head tilted forward, so that the operator is secluded in the console and isolated from the other participants attending the surgery. Another novel image monitoring system, the personal head-mounted display (HMD) system, has also emerged, providing spectacular improvement in terms of resolution, wearability, and weight. The application of the HMD system has been reported in ultrasound scanning, anesthesia management, endourological surgery, laparoscopic surgery, and gasless single-port urologic surgery [4–10]. The modern HMD, designed for medical use, is now commercially available; the HMM-3000MT (Sony Corporation, Tokyo, Japan) is a 490-g binocular HMD composed of 0.7-inch organic light-emitting diode screens (resolution: 1,280 × 720 pixels) for each eye.

The HMD-based monitoring system is distinct from that of the da Vinci Surgical System. The HMD system enables image sharing among all HMD wearers including the operator, assistants, anesthesiologist, and even students and patients. Using the imaging splitter, the images can be delivered to multiple HMDs, as many as needed, depending on the number of participants in the procedure. However, owing to the sense of immersion, communication between participants can be reduced with this system. Based on the idea of providing perceptual support to the HMD wearer, we reported on the use of a video camera mounted on the HMD to support the operator's spatial perception by supplying a comprehensive panoramic view [8]. We further think that the ability of the operator wearing the HMD to use a pointer on the endoscopic view without touching any pointing device would benefit communication.

Aim

To support interactive intraoperative communication among HMD wearers, we integrated vision-based finger tracking into our system as a pointing device. In the current paper, we report a case in which an operator communicated with medical students using this system during transurethral resection for bladder cancer.

Material and methods

Our vision-based finger tracking system is composed of a commercially available real-time video camera, which is mounted on the HMM-3000MT, and computer software that enables tracking of the tip of the index finger and superimposing the marker on the endoscopic view. Photo 1 shows an overview of the system. The software converts an HMD-mounted video image into a binary image based on the green color in order to detect and track the green surgical globe in real time. A sequence of points contouring the operator's hand is created based on the tracked data. The degree between three adjacent points of the contour is calculated, and the position of the tip of the index finger showing a low angle (less than 60°) is marked as a circle. The position of the circle is superimposed on the endoscopic view in real time. The time gap between the endoscopic view and the superimposed pointer is around 0.2 s.

Photo 1.

During software processing, a circle tracer of the tip of the index finger is superimposed on the input image

We used this system in an experimental demonstration on a 54-year-old man who underwent transurethral resection for bladder cancer. The operator and the medical students each wore an HMD in which the endoscopic view was displayed, and the video camera was mounted on the operator's HMD (Photo 2). The position of the marker was superimposed on the endoscopic view. The superimposed image and the video camera view were displayed in the HMD using the picture-in-picture technique, which was simultaneously shared with the operator and the medical students. During the operation, the operator used this finger-tracking pointer to explain the intraoperative findings to the medical students.

Photo 2.

A scene from transurethral resection for bladder cancer in which an operator employs a pointing device using the vision-based finger tracking technique. The operator wore a head-mounted display. A video camera (framed arrow head) was mounted to track the operator's index finger (arrow head)

Discussion

As a novel personalized imaging monitoring system, the HMD system provides the HMD wearer with a sharp high-contrast view in an ergonomically natural position. In contrast to the da Vinci Surgical System, the HMD system is compact, easily introduced, and affordable. Importantly, various types of devices can be connected to the HMD system. The HMD system can be easily customized and applied to many types of medical procedures including ultra-scanning, esophagogastroduodenoscopy, and bronchoscopy, even outside the operating room. The HMD has enormous potential to dramatically alter medical practice.

The potential of the HMD system in the clinical setting has been further evaluated. The HMD system can be used not only as a personalized image monitoring device, but also as a wearable device for multisensory visualization. In the current paper, we experimentally used a finger tracking system-based pointing device integrated into the HMD system. Using this system, the operator can communicate visually while maintaining hand sterility. This system can be potentially useful for interactive instruction and communication with other attending physicians and medical students.

Many motion-capture tracking technologies are used for 3D design and animation [11–13]. Here, we selected the vision-based finger tracking technique because of its low cost and the compact size of the system. Furthermore, this technology does not require initial registration. However, the time gap between the endoscopic image and the movement of the superimposed pointer may represent a drawback of this system. Further studies are needed to improve and widen the application of the HMD system for medical use.