Comparative Efficacy of New Interfaces for Intra-procedural Imaging Review: the Microsoft Kinect, Hillcrest Labs Loop Pointer, and the Apple iPad

Abstract

We adapted and evaluated the Microsoft Kinect (touchless interface), Hillcrest Labs Loop Pointer (gyroscopic mouse), and the Apple iPad (multi-touch tablet) for intra-procedural imaging review efficacy in a simulation using MIM Software DICOM viewers. Using each device, 29 radiologists executed five basic interactions to complete the overall task of measuring an 8.1-cm hepatic lesion: scroll, window, zoom, pan, and measure. For each interaction, participants assessed the devices on a 3-point subjective scale (3 = highest usability score). The five individual scores were summed to calculate a subjective composite usability score (max 15 points). Overall task time to completion was recorded. Each user also assessed each device for its potential to jeopardize a sterile field. The composite usability scores were as follows: Kinect 9.9 (out of 15.0; SD = 2.8), Loop Pointer 12.9 (SD = 13.5), and iPad 13.5 (SD = 1.8). Mean task completion times were as follows: Kinect 156.7 s (SD = 86.5), Loop Pointer 51.5 s (SD = 30.6), and iPad 41.1 s (SD = 25.3). The mean hepatic lesion measurements were as follows: Kinect was 7.3 cm (SD = 0.9), Loop Pointer 7.8 cm (SD = 1.1), and iPad 8.2 cm (SD = 1.2). The mean deviations from true hepatic lesion measurement were as follows: Kinect 1.0 cm and for both the Loop Pointer and iPad, 0.9 cm (SD = 0.7). The Kinect had the least and iPad had the most subjective concern for compromising the sterile field. A new intra-operative imaging review interface may be near. Most surveyed foresee these devices as useful in procedures, and most do not anticipate problems with a sterile field. An ideal device would combine iPad’s usability and accuracy with the Kinect’s touchless aspect.

Electronic supplementary material

The online version of this article (doi:10.1007/s10278-014-9687-y) contains supplementary material, which is available to authorized users.

Background

The power of computing systems has rapidly advanced. However, the original computer mouse from the 1980s has remained the de facto method of human computer interaction. New devices have been introduced such as the Hillcrest Labs Loop Pointer (Rockville, MD), a gyroscopic mouse, the Apple iPad (Cupertino, CA), a multi-touch tablet, and the Microsoft Kinect (Seattle, WA), a touchless gesture-based input device. These new devices, respectively, free the user from a table for the mouse, from the mouse itself, and from physical contact with a device. By changing the mode of interaction, these new tools may potentially enable a more effective means of manipulating radiological images [1–9], particularly during real-time imaging.

Real-time imaging review is important in radiology, particularly during procedures. Procedures are adapted to intra-operative findings which oftentimes necessitate additional imaging evaluation or change in technique. The standard paradigm of viewing images on a computer workstation sitting atop a table with a mouse and keyboard is ill-suited for intra-procedural use. A tabletop and mouse can occupy substantial space in a limited-sized procedural room and can impede interaction required between the physician and the patient. Moreover, a handheld or hands-free device may be more convenient, freeing a hand during procedures which require re-positioning the patient or operating other devices such as the fluoroscope. Procedural difficulty can be compounded when sterile technique is required such as with endovascular or intra-abdominal procedures. Computer keyboards and mice have been shown to harbor high infectious colonization rates [10]. These new interfaces, especially the Kinect, limit physical contact, which may reduce the risk of contamination and limit transmission of infection [11].

Gesture-based systems have been described in the past [3–6, 12], but their efficacy was limited by then-existing technology. The Kinect brings a new generation of motion tracking with far greater accuracy and response time. Some have described using the iPad and a gyroscopic mouse for surgery or radiology; however, these studies do not provide a comparison across devices [13–15]. The suitability of each of these devices has not been well evaluated for the radiologist, particularly for intra-procedural use, and comparative head-to-head analysis has not been performed [11]. Using qualitative and quantitative survey data, we evaluate the comparative efficacy of each of these devices by 29 radiologists at our institution.

Methods

Interface Setup

We adapted three new computer interfaces—the Microsoft Kinect (Microsoft, Seattle, WA), Hillcrest Labs Loop Pointer (Hillcrest Labs, Rockville, MD), and the Apple iPad (Apple, Cupertino, CA)—for intra-procedural imaging review and compared their efficacy. The Kinect utilizes a camera and depth sensor to track and respond to gestures without a requiring handheld device. The Loop Pointer uses gyroscopic sensors and accelerometers to enable wireless image manipulation without a flat surface. The iPad is a tablet with a multi-touch interface to register distinct positions of inputs to manipulate images without a separate pointing device or keyboard. Figure 1a, b, and c illustrates how to operate each device.

Fig. 1.

a, b, c Microsoft Kinect, a touchless interface, Hillcrest Labs Loop Pointer, a gyroscopic mouse, and Apple iPad, a multi-touch device. a The Microsoft Kinect (arrow) and a laptop utilizing the MIM Software (arrowhead) are shown in relationship to the operator in the IR suite. A magnified view of the laptop monitor shows the CT image being reviewed along with the operator toolbar at the top of the screen. b The Loop Pointer which is enclosed in a plastic bag is held in either hand and can be operated by moving in any direction (up, down, left, or right) as indicated (by the arrowheads). The engage button is indicated by the word “back.” c The iPad is loaded with the MIM software application. Using the indicated finger movements against the tablet screen, the program can pinch-to-zoom on the image (red arrowheads) or pan the image (yellow arrowheads). The measurement function requires use of a toolbar (as shown)

To adapt the Kinect for use, publicly available device drivers from Primesense (Tel-Aviv, Israel) were installed onto a laptop computer with an Intel Core i7-2620M CPU (Santa Clara, CA) and 4 gigabytes of RAM, running Windows 7 Professional 64-bit (Microsoft, Redmond, WA). Custom software was written in the C# programming language, utilizing the OpenNI software (OpenNI.org) framework and skeletal tracking middle-ware module NITE (Primesense). The software enabled interpretation of motion data captured by the Kinect and conversion into mouse and keyboard commands to control the MIM Cloud DICOM viewer (Cleveland, OH). The software also included a large tool bar to enable selection of different tools such as scroll, window level, pan, zoom, and measure.

The Loop Pointer wireless receiver self-installs its own drivers when plugged into a computer’s universal serial bus (USB) and is immediately ready for use. The iPad is a second-generation iPad powered by an Apple A5 processor. The iPad was readily configured for the study by installing and loading the Mobile MIM DICOM viewer app (Cleveland, OH). The Loop Pointer and iPad were placed in plastic bags to simulate sterile covers for intra-procedural use.

The MIM Cloud Viewer was installed on the computer for use with the Kinect and Loop Pointer. MIM Software Mobile MIM was installed as an application on the iPad. Although similar, the MIM Cloud Viewer and MIM Software Mobile MIM have slightly different methods to engage each function, as each is optimized for each device. Both versions are freely available for use, and the latter was the first application approved by the FDA for viewing of DICOM images on an Apple iPad. An included video (Electronic Supplementary Material 1) demonstrates a few of the functions on all three devices.

Participants and Study Design

We recruited 29 participants consisting of radiology residents, fellows, and attendings at our institution to evaluate all three devices. An exemption from Institutional Review Board was approved. Demographic characteristics and technology experience were assessed utilizing a multiple-choice survey that each participant completed.

A routine contrast enhanced axial CT of the abdomen and pelvis consisting of 181 images and containing a single 8.1-cm hepatic lesion (as initially measured by PACS viewer measurement function using standard mouse and keyboard) was selected as a representative test case. All patient identifiers were removed from the images. Approximately 2 to 5 minutes of guided training was provided to each participant on the use of each device until each participant reported that they were ready to proceed to the simulation. For each device, each participant was asked to perform the following five common basic interactions to complete the overall task of measuring the 8.1-cm hepatic lesion: scroll, window, zoom, pan, and measure. The scroll, window, zoom, and pan interactions were used to accomplish a navigation subtask of placing the lesion in the field of view, with a subsequent measurement subtask performed using the PACS viewer measurement function. The users performed the measurement interaction by selecting the measurement function from the toolbar, manually navigating to a point on the perimeter of the lesion (long axis), clicking to place a caliper, then selecting and placing a second caliper on the opposing perimeter. The PACS viewer then reports the measurement above the resultant drawn line. For each user with each device, the authors recorded the time to complete the task and their respective measurements. Each radiologist then anonymously completed a survey of their impressions. For each of the five basic interactions, participants assessed the devices on a 3-point subjective scale, where 1 is lowest score and 3 is the highest score. Subjects were informed of the scale valence by a 3-point-labeled scale of task difficulty on the survey form (1 = difficult, 2 = moderately difficult, 3 = easy). The five individual scores were summed to calculate a subjective composite usability score with a maximum of 15 points. Each user also assessed each device for its potential to jeopardize a sterile field. The authors also identified the number of surfaces that each user touched during the simulation through real-time observation to try to gauge the risk of contamination associated with each device.

Statistics

Frequencies and percentages are used to describe participant demographics and general technology and device experiences. Frequencies and percentages or means and standard deviations (SD) are reported for basic interaction measures. A repeated-measures ANOVA was used to assess differences between basic interaction specific measures. When the repeated-measures ANOVA results indicated significance at p = 0.05, pairwise comparisons were made between the devices and reported as mean differences and associated 95 % confidence intervals (CIs) with associated p values. Because of multiple comparisons, a Bonferroni adjustment was used to define statistical significance (p = 0.017). Comparisons between video game and device experience and basic interaction time were compared with a t test. All statistical analyses were conducted using the IBM SPSS Statistic 19.0 software package (SPSS, Inc., Chicago, IL).

Results

A total of 29 radiologists participated in the study including 11 residents, 4 fellows, and 14 attendings, including 5 interventional radiology attendings. Demographic characteristic are reported in Tables 1 and 2. Technology experience is shown in Table 3. No one had previous experience with the Loop Pointer. Only 4 of those surveyed had experience with the Kinect. A total of 9 (31.0 %) of participants had some experience with an iPad or iPhone, and 12 (41.4 %) had extensive experience with these devices. Experience with technology is reported in Table 3. The majority of participants rarely or never play video games, and almost all reported being comfortable or proficient with computers.

Table 1.

Demographic characteristics among 29 participants: participant title, age, and years in radiology

Demographic	Frequency	Percentage
Title
Attending	14	48.3
Fellow	4	13.8
Resident	11	37.9
Age group
20–40	19	65.5
40–60	10	34.5
Years in radiology
<10	17	58.6
10+	12	41.4

Table 2.

Demographic characteristics among 29 participants: level of experience with devices

Experience	iPad/iPhone		Kinect		Gyroscopic mouse
	Frequency	%	Frequency	%	Frequency	%
Little/none	8	27.6	25	86.2	29	100.0
Some	9	31.0	4	13.8	0	0.0
Extensive	12	41.4	0	0.0	0	0.0

Table 3.

Technology experience among 29 participants

	Frequency	Percentage
Video games
Never	8	27.6
Rarely	6	20.7
In past, not now	10	34.5
Sometimes	2	6.9
Often	3	10.3
Computer comfort
Web/email only	2	6.9
Comfortable	16	55.2
Proficient	11	37.9

For each device, Table 4 reports the means and standard deviations for the completion times, measurement, deviation from the true measurement, and subjective composite usability score. There was a significant difference between devices for all measures except deviation from the true measurement (p = <0.001 for time and composite score, p = 0.002 for measurement, and p = 0.920 for deviation).

Table 4.

Performance measures for the devices among the 29 participants. The score is the sum of each of the five measures on a 3-point, summing to a maximum of 15.0 points

Measure	Kinect		Loop Pointer		iPad
	Mean	SD	Mean	SD	Mean	SD
Usability score*	9.9	2.8	12.9	2.9	13.5	1.8
Completion time (s)*	156.7	86.5	51.5	30.6	41.1	25.3
Measurement (cm)*a	7.3	0.9	7.8	1.1	8.2	1.2
Deviation (cm)	1.0	0.7	0.9	0.7	0.9	0.7

*p < 0.05, repeated-measures ANOVA

^aHepatic lesion measured 8.1 cm

The Kinect had a mean usability score of 9.9 (SD = 2.8), the Loop Pointer 12.9 (SD = 2.9), and iPad 13.5 (SD = 1.8). The differences in mean score between Loop Pointer and iPad were not statistically significant (Kinect vs. Loop Pointer, diff = 3.0, 95 % CI = 1.7, 4.4, p < 0.001; Kinect vs. iPad, diff = 3.6, 95 % CI = 2.1, 5.2, p < 0.001; and Loop Pointer vs. iPad, diff = 0.6, 95 % CI = −0.7, 1.9, p = 0.358). Specific components of the usability scores are reported in Fig. 2.

Fig. 2.

Mean rating on a 3-point scale of each device for each of the five basic interactions

For completion times, there were significant differences between the Kinect and the other two devices. The Kinect had the longest mean time with 156.7 s (SD = 86.5), and the iPad has the shortest at 41.1 s (SD = 25.3) (Kinect vs. Loop Pointer, diff = 105.2, 95 % CI = 72.8, 137.7, p < 0.001; Kinect vs. iPad, diff = 115.7, 95 % CI = 85.5, 145.9, p < 0.001; and Loop Pointer vs. iPad, diff = 10.5, 95 % CI = −0.3, 21.2, p = 0.055). Figure 3 shows the minimum, mean, and maximum completion times with each device. Figure 4 shows the relationship of video game experience with completion times for each device. There was a significant difference in completion times by video game experience for the Loop Pointer (mean difference = 29.52; 95 % CI = 0.34, 58.70; p = 0.048), but not for the iPad or Kinect (p > 0.05 for both). Figure 5 shows the relationship of a prior experience with each respective device and completion times; the Loop Pointer was excluded since no user had prior experience with this device. Differences in completion times based on experience with either the Kinect or iPad (p > 0.05 for both) were not statistically significant. Figure 6 shows the minimum, mean, and maximum completion times on the Microsoft Kinect based on participant’s previous experience with the device.

Fig. 3.

Mean, maximum, and minimum completion times of each device

Fig. 4.

Mean completion times of each device by video game experience. Rare video game experience refers to participants who reported no or rare video game experience or experience with video games in the past. Some video game experience refers to users who stated that they often or sometimes play video games

Fig. 5.

Mean completion times of the Microsoft Kinect and Apple iPad by level of prior experience with respective device. Little/no prior use refers to participants who stated no prior or little prior experience with the respective device. Prior use refers to those reported some or extensive prior experience with the respective device

Fig. 6.

Minimum, mean, and maximum completion times for the Microsoft Kinect based on prior experience with the device. Little/no prior use refers to participants who claimed no prior or little prior experience with the respective device. Prior use refers to those reported some or extensive prior experience with the respective device

The mean hepatic lesion measurement of the Kinect was 7.3 cm (SD = 0.9), and the iPad was 8.2 cm (SD = 1.2), which was the only significant difference identified for this measure (Kinect vs. Loop Pointer , diff = 0.5, 95 % CI = 0.1, 1.0, p = 0.033, NS with Bonferroni correction; Kinect vs. iPad, diff = 0.9, 95 % CI = 0.4, 1.4, p = 0.001; and Loop Pointer vs. iPad, diff = 0.3, 95 % CI = −0.1, 0.8, p = 0.167). The authors observed the following number of surfaces contacted by each user during the simulation with each device: (1) Kinect, none; (2) Loop Pointer, two surfaces (the device and the table upon which it rested) with a single hand; and (3) Apple iPad, three surfaces (both sides of the device and the table) using both hands, one hand cradling the device, and another interacting with it. Most thought each device could be useful in interventional radiology; 55 % for the Kinect, 59 % for the Loop Pointer, and 62 % for the iPad.

Discussion

To our knowledge, this is the first study to perform a head-to-head comparison of these devices, though prior studies analyzing the iPad and Kinect individually have reported a generally positive response towards their utility in the operative environment [11, 13, 14, 16–19]. Most users favored the usability of the iPad (mean score of 13.5) in almost every basic interaction over the Loop Pointer (12.9) and Kinect (9.9; Fig. 2).

The mean usability scores mirrored mean completion times; the iPad had fastest and Kinect slowest mean times (Fig. 7). The mean time with the iPad was almost four times lower than the Kinect (41 vs. 157 s). However, completions times widely varied (Fig. 3). Several users were faster using the Kinect than others were with the other devices. We suspected that a larger number of users with prior experience with the iPad (20 users) versus the Kinect (4) might explain the difference. Though the Kinect had the longest mean basic interaction completion time, prior studies have demonstrated reasonable moving, zooming, and windowing basic interaction completion times with the Kinect, requiring 1.4 times more duration as compared to a mouse/keyboard setup [11]. Although not reaching statistical significance, users with prior experience with the respective devices had decreased completion times (Fig. 5). Users with prior experience with the Kinect also decreased variance in completion times (Fig. 6).

Fig. 7.

Relationship of the completion times versus composite usability scores versus percentage of users finding a particular device useful

The effect of additional training on performance was not evaluated, but prior experience with the devices may serve as a proxy. Training may be more effective than prior experience since it targeted. Video game experience may also serve as a proxy for training. Individuals with video game experience had decreased completion times for all devices (Fig. 4). This reached statistically significance only for the Loop Pointer. Interestingly, computer experience was not associated with decreased completion times.

Users were more accurate with the iPad. The actual liver lesion measured 8.1 cm. As shown in Table 4, the iPad’s mean measurement was closest and the Kinect farthest. However, precision was comparable across the devices. Differences in deviation from measurement were not statistically significant, and standard deviations were similar. Users tended to under-measure with the Kinect, decreasing accuracy, but still maintaining precision.

The most difficult basic interactions were those requiring fine movements such as measuring lesions and setting the exact window level (Fig. 2). However, the average difficulty level for each of the five basic interactions for almost all the devices was rated as moderate. It is also important to note that we utilized a generic image review interface across all devices, which likely shows the base level of capability of each device. The usability and accuracy of the individual interactions are likely to improve if future studies employ device-specific interfaces leveraging each device’s native abilities (e.g., an iPad app developed specifically for medical imaging review and supporting window and level operations by measuring motions on a screen).

A potential limitation of the composite usability score utilized in our study is that each basic interaction does not necessarily contribute equally in importance to the navigation and measurement subtasks. Further studies could employ an individual assessment on the navigation subtask overall instead of summing the interactions.

Most did not anticipate that the devices would have problems with preserving a sterile field; only 1 individual anticipated a problem with the Kinect; on the other hand, 7 individuals anticipated a problem for the Loop pointer and 10 for the iPad. One person thought our two-handed implementation of the Kinect could affect the sterile field, but use of two hands was a design choice that can be modified.

Assessing risk of infection complications would be difficult, probably requiring a large-scale study with follow-up on infectious complications. Instead, we relied upon subjective evaluation by experienced physicians. In addition, the authors identified the number of surfaces that each user touched during the simulation as a gauge for the risk of contamination. Both measures corresponded; namely, the Kinect has least concern and the iPad has the most concern in jeopardizing the sterile field. Notably, prior studies have employed varying techniques to allow for sterility, including sealing the unit between two pieces of Tegaderm by a sterile operator [13], use of a sterile-wrapped iPad touch-display system mounted on an operation microscope positioned just above the surgeon for neurosurgical application [14], and sterile-bagged iPad intra-operative CT image review during lung segmentectomy.

Comparison was not made to the current standards of practice, either scrubbing out of a procedure and using a mouse or verbally conveying commands to a second party. Such a comparison would depend on factors outside of the devices, such as time required to scrub out and back in or inter-communication abilities. Since nearly everyone is familiar with those scenarios, most probably used that frame of reference to evaluate the devices. Moreover, since multiple factors are involved in determining whether the devices are an improvement over existing technology, an overall subjective evaluation is probably more insightful.

Conclusion

A new intra-operative imaging review interface may be near. Most of the surveyed radiologists could foresee the utility of these devices for procedural imaging review and believe that these devices can preserve a sterile field. A new device may be driven by hospital demands to reduce infectious complication and more complicated procedures. The studied characteristics of usability, speed, accuracy, and preservation of the sterile field are probably important components for that interface. An ideal device would combine the higher usability and accuracy of the iPad with the touchless aspect of the Kinect. Implementations such as 3Gear Systems’ (San Francisco, CA) setup using two Kinects appear to substantially improve the usability and accuracy of the system while preserving the sterile field. Alternative devices like Leap Motion’s (San Francisco, CA) Leap which uses two different cameras for touch-free motion sensing may embody the ideal combination. Future research can be directed to optimizing the user interfaces, improving and standardizing the training, and studying infectious complications arising from intra-procedural use of these devices.