Usability and Reliability of Smart Glasses for Secondary Triage During Mass Casualty Incidents

Abstract

Wearable smart glasses like Google Glass provide real-time video and image transmission to remote viewers. The use of Google Glass and other Augmented Reality (AR) platforms in mass casualty incidents (MCIs) can provide incident commanders and physicians at receiving hospitals real-time data regarding injuries sustained by victims at the scene. This real-time data is critical to allocation of hospital resources prior to receiving victims of a MCI. Remote physician participation in real-time MCI care prior to victims’ hospital arrival may improve triage, and direct emergency and critical care services to those most in need. We report the use of Google Glass among first responders to transmit real-time data from a simulated MCI to allow remote physicians to complete augmented secondary triage.

1. Introduction

Mass Casualty Incidents (MCIs) occur in disaster situations in which the number of casualties exceeds the resources available to care for them.[1] On-scene management at MCIs is accomplished through a rapid MCI triage to determine quickly who will benefit most from the limited transport and treatment resources available at the scene.[2,3] MCI triage is conducted by first responders whose level of training may vary from volunteer Emergency Medical Services (EMS) crews to experienced paramedics. In order to standardize MCI triage, most pre-hospital agencies in the United States use the Simple Triage and Rapid Treatment (START) protocol, to assign initial triage categories of “Immediate” designated by the color red, “Delayed” designated by the color yellow, “Minor” designated by the color green, and dead or unlikely to survive designated by the color black.[4,5] This process is known as primary triage and is the first layer in an ongoing process which ultimately attempts to assign treatment and transport resources first to those patients who are most likely to benefit from them in order to reduce “critical mortality” or preventable deaths. [6,7]

In very large MCIs, even severely injured patients may need to wait for transport as ambulances and other forms of transportation become available. When victims await transport from the scene and as they arrive in Emergency Departments (EDs), secondary triage occurs. In this process, START triage and other algorithms are applied with reassessment of victim condition in order to detect any further clinical deterioration and to begin to assign treatment resources.

The ultimate goal of primary and secondary triage (and further levels and re-evaluations as the incident progresses) is to assign resources to the most critical of MCI victims. While primary field triage is necessarily a rapid process designed to broadly separate those who can and cannot wait for treatment, secondary triage is a more complex and often more difficult process. One particularly difficult aspect of the process is that hospital-based personnel are asked to make judgements about assignment of limited resources without knowledge of the totality of the event or the total number of patients who will eventually present to the hospital. In this way, an intensive care unit (ICU) bed, operating room (OR) time, blood product, or piece of equipment may be assigned to a person who arrives first even though a patient in greater need or who cannot be stabilized by other means might be arriving later. Alternatively, if resources are withheld until all casualties arrive in order to make the most informed decisions possible, victims are likely to deteriorate while awaiting definitive treatment. An urgent need therefore exists to provide augmented secondary triage that is efficient, rapid, and yet provides greater MCI scene information about potential victims.

Telemedicine solutions that incorporate live video and physical examination tools have long been proposed as a way to project knowledge and expertise to remote locations to assist in medical care.[8,9,10] Although successful in remote hospitals, telemedicine support systems for EMS have specific challenges such as durability and portability of hardware, reliability of software packages, and the ability of users to rapidly gain proficiency in technology. AR has begun to find its role in medicine in the hospital, but has not been expanded into the setting of Disaster Medicine.[11] The expansion of telemedicine into MCI has been previously proposed but has yet to be piloted and rigorously tested. [12,13]

Smart glasses (e.g. head-mounted computers that can project first person, point-of-view data to a remote viewer) can serve as an unobtrusive and simple technological conduit between first responders at the scene of a MCI and receiving physicians in the ED. Smart glass platforms that provide augmented reality have been proposed and studied in a variety of medical and training applications and, in addition to being useful, it has been shown that secure, Health Insurance Portability and Accountability Act (HIPAA) compliant communications can occur through this type of platform.[14] The more integrated nature of AR and the “hands free” capability of many platforms make this technology of interest to potential EMS and disaster medicine applications. [12,15] When applied to primary triage, AR platforms appear to slow the process of triage without a gain in accuracy. [16] However, the use of smart glasses to provide AR enhanced secondary triage where decisions must integrate MCI scene data with the realities of tertiary care receiving centers has yet to be studied.

Media Richness Theory (MRT), defines the ideal media choice for communicating a given piece of information or task. [17, 18, 19] MRT stresses the importance of verbal and non-verbal cues to determine how best to convey information. For example, while a voice call may be able to convey critical information, the receiver of the call lacks the ability to convey subtle facial cues that may contribute to the message. Similarly, a video call conveys more richness compared to a voice call as the context of the message delivered is immediately apparent to the viewer. MRT therefore demonstrates that the medium used to convey information has important effects upon the task at hand based on the interactivity of the medium, the ability to communicate in both directions, and the ability to include non-verbal cues. These studies have also shown that the correct selection of media can improve the performance of tasks including negotiation.[19] Given the increased richness of AR to communicate scene information to a receiving hospital, these theories suggest that adding this modality to MCI field triage can improve the speed and fidelity of communications between the scene and the receiving facility.

The promise of this technology is to give both scene and hospital providers increased awareness of the others’ sphere of operation and to improve resource allocation. [20,21] The most basic task in disaster management is optimal matching of needs and resources. In current practice, scene providers have accurate information about the needs with little ability to access information about resources available at the hospital. Hospital personnel, on the other hand, have very detailed knowledge of the resources available in the hospital but almost no real time information about the extent of the casualties and ultimate need for those resources. In this study, we used Google Glass, a novel smart glass platform, to perform secondary triage by experienced emergency medicine physicians in a simulated MCI.

2. Materials and Methods

This study was conducted in two parts, broadly referred to here as the usability analysis and the reliability analysis. The usability portion sought to understand whether smart glass technology would be perceived as easily usable and acceptable to EMS personnel. The reliability portion was designed to determine if the technology could be used to make accurate judgements about secondary triage when used collaboratively between EMS personnel and physicians.

2.1. Usability of Google Glass for MCI Triage

For the usability analysis, paramedics and EMT-Basics were recruited between patient transports at the University of Massachusetts Medical Center Emergency Department and asked to trial Google Glass with an integrated software package allowing real time transmission of live video feeds to a remote viewer. (Figure 1).

Figure 1.

Paramedic (first responder) wearing Google Glass

Training on each device consisted of the investigators giving a brief demonstration of use and then allowing each participant to ask questions. The training for each device lasted less than 5 minutes and was provided just prior to use.

The survey consisted of 10 questions, both qualitative and quantitative, and focused on two subjects; ease of use of each technology including what training might be required and the applicability of each technology to a real world MCI. Usability Questions were conceived by two members of the study team (JB, PC). A draft version of questions was reviewed with authors AH, EWB to ensure readability. A refined version of the questionnaire was reviewed during a discussion with all authors, and a final version of the usability questionnaire was agreed upon and used. Participants were asked to rate the ease of use of the technology using a 5 point Likert scale with the following response possibilities: 1- very difficult to use, 2- difficult to use, 3- neither hard nor easy to use, 4-easy to use, 5- very easy to use. The survey was administered online using SurveyMonkey®. This research protocol was submitted to, and considered exempt by, our institutional review board.

2.2. Reliability of Google Glass

The reliability portion of this study was completed during a Full Scale Exercise (FSE) conducted by the University of Massachusetts Medical Center in collaboration with numerous local first response and emergency preparedness collaborators in Worcester, MA in September 2016. Community volunteers were recruited, consented and moulaged to reflect various injuries including gunshot wounds, abrasions, and lacerations. Forty-four “victims” were assigned to this research protocol and victim injury patterns were chosen to simulate a mass shooter incident at a concert and represented various levels of injury (Figure 2).

Figure 2.

overhead image of paramedic (A), triaging simulated victim (B).

START triage categories were assigned by the authors and validated by 2 independent, blinded emergency physicians specializing in EMS practice and board certified in EMS (Table 1).

Table 1.

Victim distribution by START triage categories.

Triage Category	Number of Victims
Red	20
Yellow	13
Green	11
Black	0

All victims then had secondary triage performed by two groups of physicians. Two Emergency Medicine physicians walked throughout the simulated incident and made triage decisions after examining the patients in person. As physicians are not usually on scene at MCIs in an official capacity in the U.S., this condition was meant to simulate EM physicians triaging patients as they arrived in the ED. Two other EM physicians simultaneously evaluated the same group of patients via real-time point-of-view video stream from a paramedic wearing Google Glass loaded with a HIPAA compliant secure video streaming package (Pristine IO, Austin, TX) (Figure 1). The Pristine IO video package streams real-time video from Google Glass to a laptop computer through a secure portal on a web browser. Viewers of the video feed are able to send short text messages to the Glass wearer that are projected onto a prism and visible to the Glass wearer. These physicians made triage decisions based upon the video feed and communication with the paramedic. All four physicians made independent triage decisions for each patient.

This process was meant to simulate secondary triage so the physicians categorized each patient into categories that reflect the immediate decision making of ED providers (Table 2).

Table 2.

Secondary triage categories.

Secondary Triage Categories
Patient should go immediately to operating room Patient should go immediately to trauma by for evaluation by a trauma team Patient should go to the emergency department for delayed evaluation by trauma team when resources available Patient should go to the emergency department waiting room for nurse triage as resources are available.

For the purposes of this study, we were most interested in whether the raters agreed on the need for immediate trauma team evaluation (category 1 or 2) or delayed trauma team evaluation (category 3 or 4). The primary outcome of the reliability study was agreement within and between groups of physicians on the need for immediate trauma evaluation. This study was approved by our institutional review board (IRB).

3. Results

3.1. Perceived usability of Google Glass

Fifteen responders were recruited and completed the training and trial phase of the study and 14 completed all survey questions for a response rate of 93%.

Google Glass received an average score of 3.79 with 64% of respondents rating the technology either “easy” or “very easy” to use.

To assess the utility of augmented reality technology, for medical control we asked respondents: “How useful do you feel that medical control provided through augmented reality headsets would be as compared to traditional over the phone or via radio communication with a physician?” 13/14 respondents (93%) replied that the technology would be either “very useful” or “extremely useful.”

One potential barrier to adoption of this technology would be if the devices interfered with a provider’s usual duties and practice. To assess this we asked participants:

“To what extent do you feel that augmented reality headsets would impede your ability to perform your usual duties?” 11/14 respondents (78%) replied that they felt that such technology would only “minimally impede their duties” while only 3/14 stated that the technology would either moderately or severely impede their practice.

Finally, participants were also asked whether they felt that “just in time training” of the type they received prior to using each technology would be sufficient to give responders command of the technology and enable them to use them effectively in a real world scenario. 13/14 or 93% replied that this would be sufficient to learn to use the technology.

3.2. Reliability of Google Glass for secondary triage

For analysis, we coded each physician according to a letter, MDA and MDB were the “Google Glass group” and MDC and MDD were the “in person group”. Pairwise comparisons were made of all possible groups (AB, AC, AD, BC, BD, CD) with AB and CD representing comparisons between physicians in the same condition and AC, AD, BC, and BD representing comparisons between physicians in different groups.

There is a certain degree of expected inter-rater variability based upon individual practitioners’ style, experience, and judgment. The goal of this study was to determine if additional and statistically significant difference could be related to doing triage in person vs. via Google Glass. For this reason, we compared the inter-rater reliability from the “like condition comparison” (i.e. MDA compared to MDB) and the “different condition comparison” (i.e. MDA compared to MDC).

To calculate the percentage agreement among the comparisons, we coded each patient based upon whether the MD scored them in either the emergent (category 1 and 2 group) or the non-emergent (category 3 and 4 group). We recorded for each comparison the percentage of times that the MDs agreed and the number of observations in that comparison. Although there were 44 patients in the sample, not all MDs scored all patients, therefore in some cases a pairwise comparison was not possible and this comparison was eliminated from the data.

Once the pairwise percentage agreement calculations were complete, a mean percentage agreement was calculated by averaging the percent agreement scores for the comparisons AB and CD and the average of the comparisons AC, AD, BC and BD (Table 2). A Student’s t-test was then performed to determine whether there was a significant difference between these percentages. Equal variances were assumed after a Levene’s test for the data indicated a significance of 0.06 with an F statistic of 3.63. The p value for the comparison of means using a 2-tailed t-test was .414 indicating no significant difference (Microsoft Excel 2013).

We also used a Kappa analysis to do individual between group comparisons to determine if there was any significant difference between the levels of agreement in any individual comparison. The agreement levels, Kappa statistics, and confidence intervals for each comparison are presented in Table 3 (Stata, Release 13.1 (StataCorp LP, College Station, TX)).

Table 3.

Inter-rater reliability

	AB	AC	AD	BC	BD	CD
N-comparisons	42	35	41	34	40	34
IRR	0.95	0.94	0.85	0.97	0.92	1.00
Same condition comparison average agreement			0.976
Different condition comparison average agreement			0.923
p-value for difference between mean % agreement			0.414

4. Discussion

Our data demonstrate that first responders are accepting and positively perceive the use of smart glasses for augmented triage and decision support. Additionally, first responders using Google Glass during a simulated MCI can project first person, real-time video of MCI victims to remote physicians who are able to accurately perform secondary triage. Our data are important because they demonstrate first, that smart glass technology is accepted among first responders who may have to use these devices in MCI settings, and second, that physicians can accurately perform secondary triage using real-time video streaming provided through smart glasses worn by a first responder.

Current practice in management of MCIs includes field triage of victims by EMS personnel who then perform treatment in the field and prioritize patients for transport to an emergency department. Other than brief and infrequent radio updates that EMS personnel provide to the hospital, there is often very little, if any, direct communication or collaboration in terms of triage and treatment of patients in the field by hospital personnel. As patients arrive, ED physicians make decisions about resource allocation for the incident without detailed knowledge of patients still pending transport from the scene. In addition, only limited preparations can be made prior to patient arrival as the current system does not allow ED and Trauma Surgery physicians detailed knowledge of individual cases prior to patient arrival.[15] Improved communication and knowledge sharing is a priority for both EMS and hospital-based care givers.[15] Our data demonstrate that smart glasses such as Google Glass can function to provide a collaborative environment that does not obstruct a first responder’s immediate task of performing triage and treatment, yet provides physicians in a tertiary care setting with the ability to better prepare for the receipt of MCI victims prior to their arrival. Most practitioners in our study felt that the technology was easy to use and that brief, just in time training would be adequate to operationalize the platforms. Most respondents also agreed that wearing the AR headsets would not be a major impediment to their work in the field. Although a small sample size, these results indicate that AR platforms would likely find acceptance with pre-hospital providers and would be relatively easy to use.

This study demonstrated a very high degree of agreement between secondary triage decisions made by physicians examining simulated patients in person and the triage decisions made by different physicians performing a “virtual exam” using Google Glass. There was no statistically different degree of inter-rater reliability when pairwise comparisons were made between physicians in the two conditions, and physicians in both conditions had a very high and consistent degree of agreement when it came to deciding the triage categories of the victims. In fact, in all such comparisons, the degree of agreement was greater than 85%, and in five out of the six cases, the level of agreement was greater than 92%.

The Kappa statistics for all pairwise comparisons similarly indicated a high degree of inter-rater reliability and confirm that no pair-wise comparison was significantly different from any other comparison.

Overall, these results indicate that remote physician triage achieves the same level of inter-rater reliability as in person physician triage and suggest that using such a platform to begin secondary triage remotely, prior to patient arrival in the ED, would have a similar reliability compared to in person secondary triage upon patient arrival.

Another significant concern that this technology may have utility in addressing is the triage of patients from an MCI scene to hospitals with different capabilities. Previous work has shown that patients are often distributed in less than ideal ways to hospitals surrounding an MCI scene.[22] Especially concerning is the problem of the closest hospital to the scene being overwhelmed with casualties while facilities with greater capability and only slightly longer transport distances are underutilized. Direct communication between physicians and EMS personnel and the ability to virtually evaluate patients could provide improved patient distribution.

5. Limitations

This study has a variety of limitations which are important to note. The usability analysis had a small sample size and is best understood as a pilot study which, although encouraging, has not fully explored the issues surrounding acceptance and applicability of AR technology in the prehospital and disaster medicine spheres. In the reliability analysis, while the number of patients triaged (44) is a reasonable approximation of a large scale MCI, there were only four physicians that participated in the triage process for this study. While there are numerous pair-wise comparisons for analysis, the number of raters is low and may affect the generalizability of the results.

Our physician raters were also chosen as a convenience sample and we did not analyze their responses with respect to years of experience or EMS knowledge. This was intentional, as we wanted to simulate using the technology with any ED physician who might be on duty during the time of an MCI (whether or not that person had specific Disaster Medicine or EMS training). However, there may be important differences in decision making that we did not understand as a result of not including this type of analysis.

Finally, any investigation that incorporates simulation may suffer from the artificiality of the situation and lack of realism. Although we went to great lengths to provide realistic moulage and a setting and patient distribution that would reflect an actual incident, simulated incidents will never fully approximate a real world scenario.

6. Conclusions

There are two important conclusions to be drawn from the current investigation. The first is that the results are encouraging with respect to the likelihood of acceptance of AR technology by EMS providers and the ease of use of these platforms. The second is that the AR technology used provided sufficient fidelity and technical ease of use to enable reliable secondary triage decision making.

The authors’ hope is that the process of secondary triage, if done collaboratively in this manner, would allow both EMS and physician personnel to make decisions with knowledge of both the total demand (number and severity of injuries) as well as the supply of resources. Optimal matching of resources to need necessitates this kind of situational awareness. Second, knowledge of individual patients and the ability to examine them in real time may provide physicians with the ability to prepare specific resources that might be needed for that individual, even reserving OR space, making blood products or equipment ready, etc.

This study only addresses whether the fidelity of AR-enabled interaction is of high enough quality to allow reliable decision making, not whether this application of AR technology would have the desired effect of better resource matching. Further investigation is needed to determine if the technical and fidelity results demonstrated here can be translated into improvements in MCI patient care.

Table 4.

Percent agreement, Kappa scores, and CI for inter-group comparisons

Comparison	Agreement (%)	Kappa	CI
A to B	95.24	0.9050	0.775–1.00
C to D	91.89	0.8252	0.636–1.00
A to C	89.19	0.7784	0.578–0.978
A to D	87.80	0.7568	0.559–0.954
B to C	91.67	0.8280	0.644–1.00
B to D	90.00	0.8000	0.615–0.985