Abstract
Objective
We sought to identify a scale or components of a scale that optimize detection of older adult TBI patients who require transport to a trauma center, regardless of mechanism.
Methods
We assembled a consensus panel consisting of nine experts in geriatric emergency medicine, prehospital medicine, trauma surgery, geriatric medicine, and TBI, as well as prehospital providers, to evaluate the existing scales used to identify TBI. We reviewed the relevant literature and solicited group feedback to create a list of candidate scales and criteria for evaluation. Using the nominal group technique, scales were evaluated by the expert panel through an iterative process until consensus was achieved.
Results
We identified 15 scales for evaluation. The panel’s criteria for rating the scales included: ease of administration, prehospital familiarity with scale components, feasibility of use with older adults, time to administer, and strength of evidence for their performance in the prehospital setting. After review and discussion of aggregated ratings, the panel identified the Simplified Motor Scale, GCS–Motor Component, and AVPU (alert, voice, pain, unresponsive) as the strongest scales but determined that none meet all EMS provider and patient needs due to poor usability and lack of supportive evidence. The panel proposed that a dichotomized decision scheme that includes domains of the top-rated scales — level of alertness (alert vs. not alert) and motor function (obeys commands vs. does not obey) — may be more effective in identifying older adult TBI patients who require transport to a trauma center in the prehospital setting.
Conclusions
Existing scales to identify TBI are inadequate to detect older adult TBI patients who require transport to a trauma center. A new algorithm, derived from elements of previously established scales, has potential to guide prehospital providers in improving the triage of older adult TBI patients, but needs further evaluation prior to use.
Keywords: Brain Injuries, Triage, Emergency Medical Services, Geriatrics
INTRODUCTION
Traumatic brain injury (TBI) is common among older adults (age≥55 years), resulting in nearly 200,000 emergency department (ED) visits annually.1 Older adults have a higher incidence of TBI-related hospitalization and worse TBI-related outcomes than younger adults.2–8 Part of this increased TBI-related mortality may be due to failure to recognize risk factors and clinical indicators of life-threatening intracranial injuries, or “high-risk TBIs,” such as subdural hematoma, epidural hematoma, cerebral contusions, and cerebral edema.
Older adults can often return to baseline function if provided proper evaluation and treatment.9,10 This includes prehospital identification of these high-risk TBIs and rapid transport to a trauma center, which can perform Computed Tomography (CT)-based diagnosis of these lesions and neurosurgical intervention. Trauma centers can also more easily provide interventions that quickly reverse anticoagulants, which are commonly used among older adults.11–16 Unfortunately, some research has shown that adults aged 65–74 have a lower incidence of admission to trauma centers for TBI than adults under age 65.2
The presence of clinically significant TBI can be difficult to identify in the prehospital setting, especially among older adults, making it difficult for EMS providers to appropriately triage these patients to trauma centers. The most commonly used triage algorithm for the acutely injured patient is the Field Triage Decision Scheme (FTDS), developed by the American College of Surgeons Committee on Trauma and the Centers for Disease Control and Prevention.17 One component of the FTDS defines TBI patients who need a trauma center as those with Glasgow Coma Scale (GCS) score ≤13; however, studies have shown that older adult patients with a GCS of 14 or 15 may still have significant brain injuries.18–20 Further, prehospital providers may not be accurate in their calculation of the GCS.21 Several other tools are available to potentially help prehospital providers detect indicators of these TBIs that require transport to a trauma center; however these tools were derived among younger adult cohorts and may not function well among older adults who may have cognitive impairment.22–24 Additionally, older adults with normal presenting vital signs and absence of other significant injuries may still have a brain injury that warrants transport to a trauma center.7 The majority of TBI-related ED visits among older adults are due to falls (61%), many from standing height, which are low-energy mechanism injuries that alone do not require transport to a trauma center according to the FTDS.1,17 Finally, Emergency Medical Services (EMS) providers also face challenges in determining injury circumstances, past medical and medication history, and whether findings such as altered mental status are chronic or due to the acute injury.25
These factors suggest that current triage algorithms may be suboptimal for the identification of older adults with brain injuries. A new evaluation process may better identify older adults with TBI who would benefit from transport to a trauma center.19 The goal of this investigation was to identify a neurologic scale or components of scales that optimize detection of high-risk older adult TBI patients – head-injured patients who need transport to a trauma center – and are feasible for use in the prehospital setting.
METHODS
This study used a consensus panel of purposefully selected experts and was conducted using the nominal group technique.26,27 This technique uses both qualitative and quantitative components in the context of a highly structured meeting such that the influence of any particular member is minimized and ideally involves less than 10 participants. It is comprised of two rounds in which members of the panel rate, discuss, and re-rate the candidate scales. The first round was completed electronically, and the second was completed at an in-person meeting with a credible non-content expert facilitator with experience in the nominal group technique.
The assembled panel included nine experts from the fields of geriatric emergency medicine, prehospital medicine, trauma surgery, geriatric medicine, and TBI, as well as active EMS providers at the advanced and basic life support levels. The participants practice in Rochester, New York, and Columbus, Ohio. All potential members who were invited to join the panel agreed to do so.
Identification of scales
To identify the scales for consideration by the expert panel, we conducted a targeted literature review including scales from four sources: 1) suggestions from the CDC; 2) the panel members; 3) a PubMed search using combinations of “prehospital,” “traumatic brain injury,” and “scale” for all articles published in English from any date up to October 1, 2012; 4) references from the first three methods. These same sources were used to obtain the scales’ validity and reliability. We sent a comprehensive list of potential scales to the panel members for feedback.
Criteria for evaluation
Before the face to face meeting, we contacted the panel members via individual face to face and electronic communication to identify criteria that the panel members thought would be best to evaluate the strength of each scale. Via group electronic communication, the panel agreed upon 10 criteria (Table 2). In addition to rating the scales (described below), the panel members rated the importance of each of the evaluation criterion on a scale from 1 (not at all important) to 5 (very important). At the consensus panel meeting, two changes were made to the criteria. “Current EMS familiarity” was modified to “current EMS familiarity with the scale components” because EMS providers may not know the scale, but they may still be able to easily apply it if they are familiar with the components. Additionally, “ability of scale to perform trauma triage beyond the current guidelines” was removed because panel members chose to consider each scale independent of the FTDS (considered the “current guidelines” in this criterion). After discussion at the meeting, the panel members re-rated the importance of each criterion (Table 2).
Table 2.
Ratings of Importance of Criteria for Evaluation of Scales
| Criterion | Mean Initial Ratinga |
Key Discussion Points | Mean Final Ratinga |
|---|---|---|---|
| Ease of administration | 4.9 | No discussion | 4.9 |
| Time to administer | 4.6 | No discussion | 4.6 |
|
Feasibility of use with older adults |
4.5 | No discussion | 4.4 |
| Ease of interpretation | 4.3 | No discussion | 4.3 |
|
Feasibility of training EMS |
4.1 | Much more important than current familiarity |
4.2 |
|
Fits into current flow of assessment |
4.0 | No discussion | 4.0 |
|
Strength of evidence among older adults |
3.8 | No discussion | 3.8 |
|
Strength of evidence in the prehospital setting |
3.6 | No discussion | 3.6 |
| Current EMS familiarity | 2.6 | Panel members interpreted differently – some panels thought this was referring to a complete scale and others thought it was referring to components of the scale, so this criterion was changed to “Current EMS familiarity with the scale components.” Additionally, the panel noted that EMS may be familiar with a scale but use it poorly. |
2.9 |
|
Ability of the scale to perform trauma triage beyond the current guidelines |
3.5 | Very subjective criterion; the panel thought that this should be kept separate from FTDS |
Eliminated |
Criteria were rated on a scale from 1 (not important at all) to 5 (very important).
EMS = Emergency Medical Services; FTDS = Field Triage Decision Scheme
Evaluation of scales
The panel completed the first round of evaluations electronically by reviewing a summary packet–compiled by the study staff and the primary investigator – that contained an overview of each of the scales (modeled after McDowell’s Measuring Health).28 Materials included information about scales’ components, the purpose of the scales, settings and populations in which the scales were tested, and a summary of evidence for the scales’ validity and reliability. This included sensitivity and specificity for various published outcomes (e.g. in-hospital mortality or neurosurgical intervention), correlation with other published scales, and inter-rater reliability. The packet also included a summary of results, strengths, and weaknesses of all the published literature for each scale. Based on the material presented in the packet and previous knowledge of the scales, each panel member independently rated how well he or she thought each scale met the criteria for evaluation (described above) on a scale from 1 to 5 (scale anchors varied based on the criterion, e.g. “ease of administration by EMTs” had the anchors of 1 – difficult to 5 – easy).
We assigned a weight to each criterion based on the panel’s rating of the criterion’s importance to create an overall criteria matrix for scale evaluation. Criteria the panel deemed more important were given higher weights. To calculate the score for an individual scale, we multiplied the mean score in a particular criterion by the mean importance rating for that criterion and summed these values. Scores are presented as percentage of total possible points because the total possible points differed at each stage of evaluation. We determined the highest possible score by multiplying the average importance score for a criterion by 5 (the maximum points a scale could receive in any particular criterion) and summing these values.
We calculated the scores from the first round and summarized them prior to the in-person consensus panel meeting such that each individual member’s ratings remained anonymous.
The second round of evaluations was completed in-person. Panel members discussed the results of the first rating and further clarified their viewpoints regarding these scales and the criteria for evaluation. They anonymously re-scored the scales and criteria for evaluation based on the discussion to yield a final rating of the scales. The panel then discussed the final ratings and reflected on discussion to reach its final recommendations. Following the meeting, each panel member verified a written summary of the conclusions to assure agreement.
RESULTS
Literature Review
The panel did not suggest any additions or deletions to the proposed list of scales; thus, the final list contained 15 scales. In Table 1, these are described by their components, which were extracted from the identified scales.
Table 1.
Candidate Scales and their Componentsa
| Component | GCS | GCS- motor component |
SMS | SVS | AVPU | ACDU | Pupil Size & Reactivity |
GCS- Extended |
NOS-TBI | SAC | MACE | ACEP | NICE | RAPS | REMS |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Verbal Response | X | X | X | X | X | X | X | ||||||||
| Eye Opening Response | X | X | X | X | |||||||||||
| Motor Response | X | X | X | X | X | X | X | X | X | ||||||
| Alertness | X | X | X | X | |||||||||||
| Pupil Size | X | ||||||||||||||
| Pupil Reactivity | X | X | X | ||||||||||||
| Pupil Asymmetry | X | ||||||||||||||
| Loss of Consciousness | X | ||||||||||||||
| Amnesia | X | X | X | X | X | ||||||||||
| Altered Mental Status | X | X | X | ||||||||||||
| Headache | X | X | |||||||||||||
| Dizziness | X | ||||||||||||||
| Response to Pain | X | ||||||||||||||
| Orientation | X | X | X | ||||||||||||
| Memory, Immediate | X | X | |||||||||||||
| Memory, Delayed | X | X | |||||||||||||
| Concentration | X | X | |||||||||||||
| Nausea/Vomiting | X | X | X | ||||||||||||
| Neurologic Exam | X | X | X | X | X | ||||||||||
| Visual Fields | X | ||||||||||||||
| Hearing | X | X | |||||||||||||
| Facial Paresis | X | ||||||||||||||
| Unsteady on Feet | X | X | X | ||||||||||||
| Injury Mechanism | X | X | X | ||||||||||||
| Hit Head | X | ||||||||||||||
| Smell ability | X | ||||||||||||||
| Drug/Alcohol Intoxication | X | ||||||||||||||
| Trauma above clavicles | X | ||||||||||||||
| Post-traumatic seizures | X | X | |||||||||||||
| Suspected skull fracture | X | X | |||||||||||||
| Coagulopathy | X | X | |||||||||||||
| Age | X | X | X | ||||||||||||
| Mean Arterial Pressure | X | X | |||||||||||||
| Heart Rate | X | X | |||||||||||||
| Respiratory Rate | X | X | |||||||||||||
| Oxygen Saturation | X |
GCS=Glasgow Coma Scale; SMS=Simplified Motor Score; SVS=Simplified Verbal Score; AVPU=Alert, Voice, Pain, Unresponsive; ACDU=Alert, Confused, Drowsy, Unresponsive; NOS-TBI=Neurological Outcome Scale for Traumatic Brain Injury; SAC=Standardized Assessment of Concussion; MACE=Military Acute Concussion Evaluation; ACEP=American College of Emergency Physicians Guidelines; NICE=British National Institute for Health and Clinical Excellence; RAPS=Rapid Acute Physiology score; REMS=Rapid Emergency Medicine score
The GCS has the largest related body of literature, but evidence for its use in the prehospital setting is limited. The GCS received a weak recommendation for use in assessment of TBI and neurological deterioration in the prehospital setting from the Brain Trauma Foundation.29 Most studies of prehospital use of GCS only examine its utility in the severely brain injured patient (GCS ≤ 8).30–33 Some studies have shown that GCS has lower inter-rater reliability and prehospital scores only moderately correlate with ED scores, leading to further concerns regarding its use, though one study did show nearly 98% agreement for mild TBI patients.21,34,35
The expert panel considered “descendants” of the GCS, including the GCS Motor Component (GCS-motor), the Simplified Motor Score (SMS) and the Simplified Verbal Score (SVS). One study found that GCS-motor was just as predictive of Abbreviated Injury Scale score as the full GCS.36 In the prehospital setting and in the ED, GCS-motor is not significantly different from the GCS in predicting ED intubation, neurosurgical intervention, brain injury, admission to the ICU or mortality.37–41 SMS, a further reduced version of the GCS-motor, and SVS, a simplified version of the GCS Verbal Component, have been validated in the prehospital and ED settings, showing little difference from GCS in predictive ability of emergency tracheal intubation, TBI, neurosurgical intervention, and in-hospital mortality.37,38,42–44 Additionally, SMS has a much higher inter-rater reliability among emergency physicians than the GCS.34 The panel also considered the Glasgow Coma Scale – Extended (GCS-E), which was designed to better discern concussion.45
The panel also considered scales of alertness, specifically AVPU (Alert, responds to Voice, responds to Pain, Unresponsive) and ACDU (Alert, Confused, Drowsy, Unresponsive).46,47 Unlike other scales considered, these were designed primarily for the assessment of trauma patients as part of Advanced Trauma Life Support and have higher inter-rater reliability than GCS among adults with altered levels of consciousness in the ED.34
Based on Brain trauma Foundation prehospital recommendations, the panel considered pupil size, reactivity, and asymmetry; however, pupil abnormality is better used as predictor of mortality than of other TBI outcomes.29,48–52
In addition to the GCS-E, panel members considered others tools specific to mild TBI identification, specifically the Standardized Assessment of Concussion (SAC) and the Military Acute Concussion Evaluation (MACE), which includes the SAC.53,54 The SAC is widely used in populations of athletes and has demonstrated reliability, validity, sensitivity and specificity in this population, but neither the SAC nor the MACE has been evaluated in the prehospital setting or among older adults, as they were designed for younger populations.55 The Neurologic Outcome Scale for Traumatic Brain Injury (NOS-TBI), which is used by non-physicians to measure brain injury severity in the post-acute rehabilitation setting56,57, and two sets of guidelines for brain injury triage, imaging and treatment – the American College of Emergency Physicians (ACEP) guidelines and the British National Institute for Health and Clinical Excellence (NICE) guidelines were also considered; however, none were designed for, nor have been evaluated for use in prehospital settings.58,59 Finally, the panel considered the rapid acute physiology score (RAPS) and the rapid emergency medicine score (REMS), which were both designed to be used in the prehospital setting to predict in-hospital mortality.60,61
Scale Evaluation
The panel rated “ease of administration” and “time to administer” as the most important criteria for evaluation of scales (Table 2). The least important were “EMS familiarity with the scale components” and the “strength of evidence” criteria. While seemingly counterintuitive, the strength of evidence criteria were rated as least important because nearly all scales considered lacked quality evidence for use in prehospital triage of older adult populations (the average scores for each scale in the strength of evidence categories were 2.0). The panel members’ ratings of the criteria were consistent. The most variation was for the “EMS familiarity with the scale components” criterion, which received scores from 1 to 4.
The panel first independently rated the scales on the criteria described in Table 2. The results of these initial ratings are summarized in Table 3. Simplified “descendants” of GCS (GCS-Motor, SMS, SVS) scored highest, whereas the scales designed for use in mild TBI, the guidelines, and the longer scales received the lowest scores (NICE, MACE, ACEP, NOS-TBI, SAC). Most scales had similar ratings among panel members; scales with highly variable scores were further discussed among panel members to work toward achieving consensus.
Table 3.
Scale Evaluation Results
| Scalea | Initial Rating (out of 199 possible points) |
Key Discussion Points | Final Rating (out of 201 possible points) |
|---|---|---|---|
| SMS | 166.7 (83.6%) |
Most existing data on this scale is from trauma registries, which do not include low mechanism injuries. Most studies also just used GCS from EMS to calculate SMS in analysis, so proper EMS use of scale unknown. SMS is closer to the “normal vs. abnormal” that was discussed with GCS-Motor |
175.5 (87.5%) |
|
GCS-Motor Component |
176.1 (88.3%) |
EMS has the most difficulty with the motor component – likely “guess” rather than know the 6 scores. Thus, feasibility and ease of interpretation are questionable. Discussion continued on whether it matters what the exact score is vs. normal/abnormal. Agreement that normal vs. abnormal is sufficient for the GCS and subscales. |
161.4 (80.5%) |
| SVS | 156.5 (78.5%) |
Question of whether this scoring could be followed exactly – oriented vs. confused conversation vs. inappropriate words. How to account for confusion if it is underlying – especially a problem in the older population. |
155.7 (77.6%) |
| AVPU | 154.7 (77.6%) |
Discussion about exact meaning of different states and if EMS could easily differentiate – alert vs. not alert probably more practical. |
156.5 (78.0%) |
| GCS | 150.3 (75.4%) |
Discussion focused on individual components and “descendants.” |
148.4 (74.0%) |
| ACDU | 140.1 (70.3%) |
Limited discussion, as most was covered with AVPU. |
139.5 (69.5%) |
|
Pupil Size & Reactivity |
143.5 (72.0%) |
A valuable component, but not sufficient by itself. | Not rescored |
| RAPS | 133.8 (67.1%) |
Complex scoring system difficult in prehospital setting; not specific to TBI. |
Not rescored |
| REMS | 127.4 (63.9%) |
Complex scoring system difficult in prehospital setting; not specific to TBI. |
Not rescored |
| GCS-Extended | 106.4 (53.4%) |
The marginal value of the scale is minimal; not feasible in the prehospital setting due to extended time period included in mental status component. |
Not rescored |
| NICE | 101.3 (50.8%) |
Focused on aspect of guideline related to older adult hospital destination; panel thought it was excessive and would lead to overtriage. |
Not rescored |
| MACE | 75.5 (37.9%) |
Too long and not practical in the prehospital setting; EMS providers may already be asking many of the questions; amnesia should be considered as a valuable component. |
Not rescored |
| ACEP | 72.3 (36.3%) |
Targeted to ED physicians to detect ICH; not designed for EMS setting; pieces on anticoagulants, GCS<15, and skull fracture should be considered as valuable components. |
Not rescored |
| NOS-TBI | 69.4 (34.8%) |
Outside the realm of the charge of the group; not good for triage; questions dealing with motor function and paresis should be considered as valuable components. |
Not rescored |
| SAC | 57.1 (28.6%) |
Familiarity and feasibility in prehospital setting questioned. |
Not rescored |
SMS=Simplified Motor Score; GCS=Glasgow Coma Scale; SVS=Simplified Verbal Score; AVPU=Alert, Voice, Pain, Unresponsive; ACDU=Alert, Confused, Drowsy, Unresponsive; RAPS=Rapid Acute Physiology score; REMS=Rapid Emergency Medicine score; NICE=British National Institute for Health and Clinical Excellence; MACE=Military Acute Concussion Evaluation; ACEP=American College of Emergency Physicians Guidelines; NOS-TBI=Neurological Outcome Scale for Traumatic Brain Injury; SAC=Standardized Assessment of Concussion
In addition to the ratings, the panel discussed a few overarching themes related to identifying older adult patients with TBIs that require transport to a trauma center. First, the panel considered whether a numerical score is necessary for prehospital decision-making, or if a dichotomous consideration of normal vs. abnormal is sufficient. The panel preferred dichotomous scales, as the members viewed dichotomous scales as easier to use in the prehospital setting. Second, the panel recognized that older patients often have small hemorrhages, particularly due to the high prevalence of antiplatelet and anticoagulant agents, and delayed presentation of intracerebral hemorrhage symptoms, and it is not currently feasible for EMS personnel to identify these patients. Third, the panel noted that the risk “overtriage” (i.e., sending patients who do not need trauma center care to a trauma center) was an important consideration for evaluation and care of older adults. Triage of all older adults to trauma centers was thought to be an unacceptable approach. Finally, the panel thought that patients with an unknown baseline mental status who present with altered mental status should be treated under the assumption that they were normal at baseline. Members suggested that the number of patients with an unknown baseline would be relatively small, and it would be best to potentially overtriage these patients rather than to potentially miss an opportunity for a critical, emergently-needed intervention.
The panel also identified two fundamental concepts relevant to any triage decision tool. First, the group noted the importance of careful terminology. The EMS provider panel members noted that they consider “trauma” as a severe injury or severe mechanism. Thus, the panel determined that the context needs to be reframed more broadly as “injury,” rather than “trauma.” Thus, older adults with lower-energy mechanisms of injury, such as a fall from standing height, or less impressive physical findings would still be considered for TBI assessment. Second, the group agreed that any scale implemented in the prehospital setting must be usable by EMS providers of all certification levels, with the inherent limitations of time, environment, and multiple competing demands. For instance, while the instruments designed to identify mild TBI may better identify older adults with brain injury, their completion burden is such that they are not practical for use in the prehospital setting.
Although GCS-motor initially scored the highest (88% of total possible points), the panel members were concerned that EMS providers have the most difficulty assessing the motor component of the full GCS, thus limiting its feasibility of use and ease of interpretation. The SMS is more similar to the dichotomous decision-making approach preferred by the panel; however, the panel was concerned that the existing evidence supporting the validity and reliability of this scale was based primarily from trauma registries, which often do not include patients with the lower severity injuries that commonly occur in older adults.
SVS and AVPU also scored highly (79% and 78% of total possible points, respectively), which the panel partially attributed to the scales’ simplicity. Despite SVS’s simplicity compared to the GCS verbal component, the panel members questioned whether the scoring could be followed exactly and were unsure of how to account for underlying confusion in the older adult population. Similarly, with AVPU, the panel members were unsure if the four different states could be easily discerned and suggested that a distinction of “alert” vs. “not alert” may be more appropriate.
The remaining scales all received less than 75% of total possible points. The panel thought that the relevant components of the GCS and ACDU scales were covered by previously discussed scales. Similarly the panel viewed the RAPS, REMS and GCS-Extended as not feasible for use in the prehospital setting; further, they noted that RAPS and REMS were not TBI-specific. The NICE, MACE, ACEP, NOS-TBI, and SAC were removed from further consideration as they did not address the goals of the study. Any scales that the panel members deemed as not feasible or removed from further consideration were not rescored in an effort to make the consensus process more efficient. Pupil size and reactivity were thought to be valuable components but not sufficient alone. The panel viewed other components as similarly valuable but insufficient: paresis, motor function, coagulopathy, GCS <15, physical/basilar skull fracture, and amnesia.
Consensus Panel Conclusions
The panel agreed that none of the six highest rated scales (SMS, GCS-Motor, SVS, AVPU, GCS, and ACDU) adequately met all EMS provider and patient needs. The panel did note that all of the highly-rated scales fit into two domains: motor function and alertness. Ultimately, the panel concluded that the domains could be simplified to a dichotomous motor decision of “obeys commands” vs. “does not obey commands” and a dichotomous alertness decision of “alert” vs. “not alert.” The EMS provider panel members noted that these decisions were clearer and easier to implement than current scale components.
Given the absence of an existing scale to reliably identify older adults with TBI, the expert panel offered a novel decision scheme depicted in Figure 1: if a patient is ≥55 years old and there is a possibility of injury, the EMS provider should identify the patient as a TBI patient who requires transport to a trauma center if he or she is not alert or at baseline alertness and/or does not obey motor commands and transport him or her to the nearest designated trauma center. The panel did recognize that research is needed to evaluate the classification accuracy of the new scheme before it should be adopted.
Figure 1.
Panel-devised decision scheme for detecting older adult TBI patients who require transport to a trauma center.
DISCUSSION
To our knowledge, this is the first attempt to identify a neurologic scale that optimizes prehospital identification of older adults with high risk TBI to facilitate appropriate triage to a trauma center. Our panel determined that existing scales, including the GCS, are inadequate to optimize detection in older adults. Reasons for this decision included the poor usability of the scales, especially those with complicated scoring systems, and the paucity of evidence that demonstrate the efficacy of these scales for older adults in the prehospital setting. Instead, the panel recommended the development and validation of a new scale that assesses alertness and ability to follow motor commands.
Triage of significantly injured patients to trauma centers results in improved survival.12 Given that older adults with high GCS scores (13–15) have worse outcomes than younger adults with the same GCS score the current FTDS may not appropriately triage high-risk older adults with TBI with normal presenting vital signs and absence of other significant injuries to trauma centers.7 Certain patients (e.g., older adults, on anticoagulants) require a head CT after injury, which most non-trauma center hospitals can provide.62,63 Although most hospitals can diagnose and stabilize these patients, time to definitive care is important in clinically significant brain injury and would represent undertriage. Alternatively, triaging all patients with a suspected head injury to trauma centers would lead to excessive crowding and costs and would represent overtriage. Balancing these considerations is challenging, and given the current research in this field, the panel thought that EMS providers should take high-risk patients directly to trauma centers for advanced care that is available at any time.62,63
The challenges of delayed presentation of clinical symptoms from TBI factored into the discussion and final recommendation. Some patients with TBI from low-risk mechanisms may present to EMS with minimal symptoms and develop more troublesome symptoms hours or days later.64,65 The scales currently available cannot eliminate this problem.
The issue of instrument usability by EMS providers also played a large role in the panel’s deliberations. The decision to use the term “injury” rather than “trauma,” consistent with the terminology of the FTDS, should ensure that patients with lower-energy mechanisms (e.g., fall from standing vs. fall from a rooftop) are not ignored. The decision to endorse instruments with only binary choices (present/not present) and without complex algorithms ensures that the scale is easily usable at all EMS provider levels and in many different environments. Although usability should be considered for all scales used in medicine, it often is not, which impairs uptake.66,67
Although the panel thought that existing scales, including the GCS, are inadequate for assessment of older adults with TBI that requires transport to a trauma center, the members leveraged core components of these scales that it viewed as key domains in TBI assessment. The panel thought that the new algorithm was usable and could help EMS providers triage injured older patients between trauma and non-trauma centers. Further, the panel noted that existing data cannot determine the rate of overtriage based on this newly-designed scale. Future studies are required to determine the sensitivity, specificity, and inter-rater reliability of the proposed scale among injured older adults in the prehospital setting.
Additionally, although it was not the focus of this expert panel, many of the limitations of the existing scales are applicable to the younger adult population in addition to the older adult population. The newly-developed scale may also be applicable to identify younger adults who have a TBI that requires trauma center care, but this needs to be empirically tested.
As with any study, ours has limitations that must be considered. First, the panel’s deliberations and conclusions were limited by the lack of empirical evidence for many of these scales, especially among older adults and in the prehospital setting. Therefore, future research must collect data that can be used for validation of the newly developed scheme, as well as existing neurologic scales, to detect TBI that requires transport to a trauma center in injured older adults. Second, the literature review of the scales was a targeted literature review to seek out scales. As such, we do not have information on the number of articles reviewed and did not use a systematic screening process; however, all panel members were given the list of articles, and these content experts with extensive knowledge related to the goal of the study did not identify any missing literature. Third, our panel members were primarily drawn from a single region in the United States and the EMS providers were from one city. These individuals are, however, national leaders in TBI, EMS, trauma, geriatric medicine, and geriatric emergency medicine with extensive grant funding and publications in these areas, as well as participation as a content expert in ACEP, the Institute of Medicine, and the CDC. Fourth, while the panel’s findings about specific scales may not apply to all EMS systems, the overarching clinical concepts should be applicable across geographic areas. Furthermore, our use of the nominal group technique strengthens our findings, as this technique has been shown to produce greater numbers of high quality suggestions and solutions.27 Finally, we did not test the usability or validity of any instrument or the overall scheme, but depended upon the opinions of experienced EMS providers and clinicians. Future work must explore the acceptability, usability and validity of the various scales and schemes.
CONCLUSION
This expert panel determined that existing scales to identify TBI were inadequate for prehospital detection of older adult TBI patients who require transport to a trauma center. The panel proposed a new, dichotomized pathway to assist EMS providers in their selection of an appropriate destination hospital for high-risk older adults with TBI. Future study is needed to determine the validity of this new scale, as well as its undertriage and overtriage rates as applied in the prehospital setting, prior to widely implementing the scale in the prehospital setting.
REFERENCES
- 1.Faul M, Xu L, Wald MM, Coronado VG Centers for Disease Control and Prevention. Traumatic Brain Injury in the United States: Emergency Department Visits, Hospitalizations and Deaths 2002–2006. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. 2010 [Google Scholar]
- 2.Dams-O'Connor K, Cuthbert JP, Whyte J, Corrigan JD, Faul M, Harrison-Felix C. Traumatic brain injury among older adults at Level I and II trauma centers. Journal of neurotrauma. 2013;30:2001–2013. doi: 10.1089/neu.2013.3047. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Thompson HJ, McCormick WC, Kagan SH. Traumatic brain injury in older adults: epidemiology, outcomes, and future implications. J Am Geriatr Soc. 2006;54:1590–1595. doi: 10.1111/j.1532-5415.2006.00894.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Coronado VG, Thomas KE, Sattin RW, Johnson RL. The CDC traumatic brain injury surveillance system: characteristics of persons aged 65 years and older hospitalized with a TBI. J Head Trauma Rehabil. 2005;20:215–228. doi: 10.1097/00001199-200505000-00005. [DOI] [PubMed] [Google Scholar]
- 5.Coronado VG, Xu L, Basavaraju SV, McGuire LC, Wald MM, Faul MD, Guzman BR, Hemphill JD. Surveillance for traumatic brain injury-related deaths--United States, 1997–2007. MMWR Surveill Summ. 2011;60:1–32. [PubMed] [Google Scholar]
- 6.Hukkelhoven CW, Steyerberg EW, Rampen AJ, Farace E, Habbema JD, Marshall LF, Murray GD, Maas AI. Patient age and outcome following severe traumatic brain injury: an analysis of 5600 patients. J Neurosurg. 2003;99:666–673. doi: 10.3171/jns.2003.99.4.0666. [DOI] [PubMed] [Google Scholar]
- 7.Susman M, DiRusso SM, Sullivan T, Risucci D, Nealon P, Cuff S, Haider A, Benzil D. Traumatic brain injury in the elderly: increased mortality and worse functional outcome at discharge despite lower injury severity. J Trauma. 2002;53:219–223. doi: 10.1097/00005373-200208000-00004. discussion 23-4. [DOI] [PubMed] [Google Scholar]
- 8.Mosenthal AC, Lavery RF, Addis M, Kaul S, Ross S, Marburger R, Deitch EA, Livingston DH. Isolated traumatic brain injury: age is an independent predictor of mortality and early outcome. J Trauma. 2002;52:907–911. doi: 10.1097/00005373-200205000-00015. [DOI] [PubMed] [Google Scholar]
- 9.Battistella FD, Din AM, Perez L. Trauma patients 75 years and older: long-term follow-up results justify aggressive management. J Trauma. 1998;44:618–623. doi: 10.1097/00005373-199804000-00010. discussion 23. [DOI] [PubMed] [Google Scholar]
- 10.Meldon SW, Reilly M, Drew BL, Mancuso C, Fallon W., Jr Trauma in the very elderly: a community-based study of outcomes at trauma and nontrauma centers. J Trauma. 2002;52:79–84. doi: 10.1097/00005373-200201000-00014. [DOI] [PubMed] [Google Scholar]
- 11.Roush WR. Principles of EMS systems. 2nd. Washington, DC: American College of Emergency Physicians; 1994. [Google Scholar]
- 12.MacKenzie EJ, Rivara FP, Jurkovich GJ, Nathens AB, Frey KP, Egleston BL, Salkever DS, Scharfstein DO. A national evaluation of the effect of trauma-center care on mortality. N Engl J Med. 2006;354:366–378. doi: 10.1056/NEJMsa052049. [DOI] [PubMed] [Google Scholar]
- 13.Clesham K, Mason S, Gray J, Walters S, Cooke V. Can emergency medical service staff predict the disposition of patients they are transporting? Emerg Med J. 2008;25:691–694. doi: 10.1136/emj.2007.054924. [DOI] [PubMed] [Google Scholar]
- 14.Levine SD, Colwell CB, Pons PT, Gravitz C, Haukoos JS, McVaney KE. How well do paramedics predict admission to the hospital? A prospective study. J Emerg Med. 2006;31:1–5. doi: 10.1016/j.jemermed.2005.08.007. [DOI] [PubMed] [Google Scholar]
- 15.Price TG, Hooker EA, Neubauer J. Prehospital provider prediction of emergency department disposition: implications for selective diversion. Prehosp Emerg Care. 2005;9:322–325. doi: 10.1080/10903120590962012. [DOI] [PubMed] [Google Scholar]
- 16.Mulholland SA, Gabbe BJ, Cameron P. Is paramedic judgement useful in prehospital trauma triage? Injury. 2005;36:1298–1305. doi: 10.1016/j.injury.2005.07.010. [DOI] [PubMed] [Google Scholar]
- 17.2011 Guidelines for the Field Triage of Injured Patients. National Center for Injury Prevention and Control, Office of Statistics and Programming. 2012 at http://www.cdc.gov/fieldtriage/pdf/decisionscheme_poster_a.pdf.
- 18.Sasser SM, Hunt RC, Faul M, Sugerman D, Pearson WS, Dulski T, Wald MM, MLS M, Jurkovich GJ, Newgard CD. Guidelines for field triage of injured patients. Recommendations of the national expert panel on field triage. 2012 [PubMed] [Google Scholar]
- 19.Caterino JM, Raubenolt A, Cudnik MT. Modification of Glasgow Coma Scale criteria for injured elders. Acad Emerg Med. 2011;18:1014–1021. doi: 10.1111/j.1553-2712.2011.01164.x. [DOI] [PubMed] [Google Scholar]
- 20.Reynolds FD, Dietz PA, Higgins D, Whitaker TS. Time to deterioration of the elderly, anticoagulated, minor head injury patient who presents without evidence of neurologic abnormality. J Trauma. 2003;54:492–496. doi: 10.1097/01.TA.0000051601.60556.FC. [DOI] [PubMed] [Google Scholar]
- 21.Bazarian JJ, Eirich MA, Salhanick SD. The relationship between pre-hospital and emergency department Glasgow coma scale scores. Brain Inj. 2003;17:553–560. doi: 10.1080/0269905031000070260. [DOI] [PubMed] [Google Scholar]
- 22.Plassman BL, Langa KM, Fisher GG, Heeringa SG, Weir DR, Ofstedal MB, Burke JR, Hurd MD, Potter GG, Rodgers WL, Steffens DC, Willis RJ, Wallace RB. Prevalence of dementia in the United States: the aging, demographics, and memory study. Neuroepidemiology. 2007;29:125–132. doi: 10.1159/000109998. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Graham JE, Rockwood K, Beattie BL, Eastwood R, Gauthier S, Tuokko H, McDowell I. Prevalence and severity of cognitive impairment with and without dementia in an elderly population. Lancet. 1997;349:1793–1796. doi: 10.1016/S0140-6736(97)01007-6. [DOI] [PubMed] [Google Scholar]
- 24.Boustani M, Peterson B, Hanson L, Harris R, Lohr KN. Screening for dementia in primary care: a summary of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med. 2003;138:927–937. doi: 10.7326/0003-4819-138-11-200306030-00015. [DOI] [PubMed] [Google Scholar]
- 25.Sporer KA, Solares M, Durant EJ, Wang W, Wu AH, Rodriguez RM. Accuracy of the initial diagnosis among patients with an acutely altered mental status. Emerg Med J. 2012 doi: 10.1136/emermed-2011-200452. [DOI] [PubMed] [Google Scholar]
- 26.Lloyd-Jones G, Fowell S, Bligh JG. The use of the nominal group technique as an evaluative tool in medical undergraduate education. Med Educ. 1999;33:8–13. doi: 10.1046/j.1365-2923.1999.00288.x. [DOI] [PubMed] [Google Scholar]
- 27.Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–380. doi: 10.1136/bmj.311.7001.376. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.McDowell I, Newell C, McDowell I. Measuring health: a guide to rating scales and questionnaires. New York: Oxford University Press; 2006. [Google Scholar]
- 29.Guidlines for Prehospital Management of Traumatic Brain Injury. 2011 at https://www.braintrauma.org/coma-guidelines/ [Google Scholar]
- 30.Baxt WG, Moody P. The impact of advanced prehospital emergency care on the mortality of severely brain-injured patients. J Trauma. 1987;27:365–369. doi: 10.1097/00005373-198704000-00004. [DOI] [PubMed] [Google Scholar]
- 31.Servadei F, Nasi MT, Cremonini AM, Giuliani G, Cenni P, Nanni A. Importance of a reliable admission Glasgow Coma Scale score for determining the need for evacuation of posttraumatic subdural hematomas: a prospective study of 65 patients. J Trauma. 1998;44:868–873. doi: 10.1097/00005373-199805000-00021. [DOI] [PubMed] [Google Scholar]
- 32.Winkler JV, Rosen P, Alfry EJ. Prehospital use of the Glasgow Coma Scale in severe head injury. J Emerg Med. 1984;2:1–6. doi: 10.1016/0736-4679(84)90038-6. [DOI] [PubMed] [Google Scholar]
- 33.Winchell RJ, Hoyt DB. Endotracheal intubation in the field improves survival in patients with severe head injury. Trauma Research and Education Foundation of San Diego. Arch Surg. 1997;132:592–597. doi: 10.1001/archsurg.1997.01430300034007. [DOI] [PubMed] [Google Scholar]
- 34.Gill M, Martens K, Lynch EL, Salih A, Green SM. Interrater reliability of 3 simplified neurologic scales applied to adults presenting to the emergency department with altered levels of consciousness. Ann Emerg Med. 2007;49:403–407. 7 e1. doi: 10.1016/j.annemergmed.2006.03.031. [DOI] [PubMed] [Google Scholar]
- 35.Kerby JD, MacLennan PA, Burton JN, McGwin G, Jr, Rue LW., 3rd Agreement between prehospital and emergency department glasgow coma scores. J Trauma. 2007;63:1026–1031. doi: 10.1097/TA.0b013e318157d9e8. [DOI] [PubMed] [Google Scholar]
- 36.Ross SE, Leipold C, Terregino C, O'Malley KF. Efficacy of the motor component of the Glasgow Coma Scale in trauma triage. J Trauma. 1998;45:42–44. doi: 10.1097/00005373-199807000-00008. [DOI] [PubMed] [Google Scholar]
- 37.Gill M, Steele R, Windemuth R, Green SM. A comparison of five simplified scales to the out-of-hospital Glasgow Coma Scale for the prediction of traumatic brain injury outcomes. Acad Emerg Med. 2006;13:968–973. doi: 10.1197/j.aem.2006.05.019. [DOI] [PubMed] [Google Scholar]
- 38.Gill M, Windemuth R, Steele R, Green SM. A comparison of the Glasgow Coma Scale score to simplified alternative scores for the prediction of traumatic brain injury outcomes. Ann Emerg Med. 2005;45:37–42. doi: 10.1016/j.annemergmed.2004.07.429. [DOI] [PubMed] [Google Scholar]
- 39.Healey C, Osler TM, Rogers FB, Healey MA, Glance LG, Kilgo PD, Shackford SR, Meredith JW. Improving the Glasgow Coma Scale score: motor score alone is a better predictor. J Trauma. 2003;54:671–678. doi: 10.1097/01.TA.0000058130.30490.5D. discussion 8–80. [DOI] [PubMed] [Google Scholar]
- 40.Meredith W, Rutledge R, Hansen AR, Oller DW, Thomason M, Cunningham P, Baker CC. Field triage of trauma patients based upon the ability to follow commands: a study in 29,573 injured patients. J Trauma. 1995;38:129–135. doi: 10.1097/00005373-199501000-00030. [DOI] [PubMed] [Google Scholar]
- 41.Al-Salamah MA, McDowell I, Stiell IG, Wells GA, Perry J, Al-Sultan M, Nesbitt L. Initial emergency department trauma scores from the OPALS study: the case for the motor score in blunt trauma. Acad Emerg Med. 2004;11:834–842. doi: 10.1111/j.1553-2712.2004.tb00764.x. [DOI] [PubMed] [Google Scholar]
- 42.Caterino JM, Raubenolt A. The prehospital simplified motor score is as accurate as the prehospital Glasgow coma scale: analysis of a statewide trauma registry. Emerg Med J. 2011;29:492–496. doi: 10.1136/emj.2010.110437. [DOI] [PubMed] [Google Scholar]
- 43.Thompson DO, Hurtado TR, Liao MM, Byyny RL, Gravitz C, Haukoos JS. Validation of the Simplified Motor Score in the out-of-hospital setting for the prediction of outcomes after traumatic brain injury. Ann Emerg Med. 2011;58:417–425. doi: 10.1016/j.annemergmed.2011.05.033. [DOI] [PubMed] [Google Scholar]
- 44.Haukoos JS, Gill MR, Rabon RE, Gravitz CS, Green SM. Validation of the Simplified Motor Score for the prediction of brain injury outcomes after trauma. Ann Emerg Med. 2007;50:18–24. doi: 10.1016/j.annemergmed.2006.10.004. [DOI] [PubMed] [Google Scholar]
- 45.Nell V, Yates DW, Kruger J. An extended Glasgow Coma Scale (GCS-E) with enhanced sensitivity to mild brain injury. Arch Phys Med Rehabil. 2000;81:614–617. doi: 10.1016/s0003-9993(00)90044-3. [DOI] [PubMed] [Google Scholar]
- 46.American College of Surgeons CoT. Advanced trauma life support manual. 6th. Chicago, IL: American College of Surgeons; 1997. [Google Scholar]
- 47.McNarry AF, Goldhill DR. Simple bedside assessment of level of consciousness: comparison of two simple assessment scales with the Glasgow Coma scale. Anaesthesia. 2004;59:34–37. doi: 10.1111/j.1365-2044.2004.03526.x. [DOI] [PubMed] [Google Scholar]
- 48.Chesnut RM, Gautille T, Blunt BA, Klauber MR, Marshall LE. The localizing value of asymmetry in pupillary size in severe head injury: relation to lesion type and location. Neurosurgery. 1994;34:840–845. doi: 10.1227/00006123-199405000-00008. discussion 5–6. [DOI] [PubMed] [Google Scholar]
- 49.Jiang JY, Gao GY, Li WP, Yu MK, Zhu C. Early indicators of prognosis in 846 cases of severe traumatic brain injury. J Neurotrauma. 2002;19:869–874. doi: 10.1089/08977150260190456. [DOI] [PubMed] [Google Scholar]
- 50.Mamelak AN, Pitts LH, Damron S. Predicting survival from head trauma 24 hours after injury: a practical method with therapeutic implications. J Trauma. 1996;41:91–99. doi: 10.1097/00005373-199607000-00014. [DOI] [PubMed] [Google Scholar]
- 51.Schreiber MA, Aoki N, Scott BG, Beck JR. Determinants of mortality in patients with severe blunt head injury. Arch Surg. 2002;137:285–290. doi: 10.1001/archsurg.137.3.285. [DOI] [PubMed] [Google Scholar]
- 52.Signorini DF, Andrews PJ, Jones PA, Wardlaw JM, Miller JD. Predicting survival using simple clinical variables: a case study in traumatic brain injury. J Neurol Neurosurg Psychiatry. 1999;66:20–25. doi: 10.1136/jnnp.66.1.20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.McCrea M, Kelly JT, Randolph C Standardized Assessment of Concussion (SAC) Manual for Administration, Scoring and Interpretation. Waukesha, Wisconsin: Brain Injury Assoc.; 1997. [Google Scholar]
- 54.Department of Defense, Department of Veterans Affairs. VA/DoD Clinical Practice Guideline for Management of Concussion/Mild Traumatic Brain Injury. J Rehabil Res Dev. 2009;46:CP1–CP68. [PubMed] [Google Scholar]
- 55.McCrea M, Guskiewicz KM, Marshall SW, Barr W, Randolph C, Cantu RC, Onate JA, Yang J, Kelly JP. Acute effects and recovery time following concussion in collegiate football players: the NCAA Concussion Study. JAMA. 2003;290:2556–2563. doi: 10.1001/jama.290.19.2556. [DOI] [PubMed] [Google Scholar]
- 56.Wilde EA, McCauley SR, Kelly TM, Levin HS, Pedroza C, Clifton GL, Robertson CS, Valadka AB, Moretti P. Feasibility of the Neurological Outcome Scale for Traumatic Brain Injury (NOS-TBI) in adults. J Neurotrauma. 2010;27:975–981. doi: 10.1089/neu.2009.1193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Wilde EA, McCauley SR, Kelly TM, Weyand AM, Pedroza C, Levin HS, Clifton GL, Schnelle KP, Shah MV, Moretti P. The Neurological Outcome Scale for Traumatic Brain Injury (NOS-TBI): I. Construct validity. J Neurotrauma. 2010;27:983–989. doi: 10.1089/neu.2009.1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Jagoda AS, Bazarian JJ, Bruns JJ, Jr, Cantrill SV, Gean AD, Howard PK, Ghajar J, Riggio S, Wright DW, Wears RL, Bakshy A, Burgess P, Wald MM, Whitson RR American College of Emergency P, Centers for Disease C, Prevention. Clinical policy: neuroimaging and decisionmaking in adult mild traumatic brain injury in the acute setting. Ann Emerg Med. 2008;52:714–748. doi: 10.1016/j.annemergmed.2008.08.021. [DOI] [PubMed] [Google Scholar]
- 59.NHS, National institute for Health and Clinical Excellence. Head Injury: Triage, Assessment, Investigation and Early Management of Head Injury in Infants, Children and Adults. London: 2007. NICE clinical guideline 56. [PubMed] [Google Scholar]
- 60.Olsson T, Lind L. Comparison of the rapid emergency medicine score and APACHE II in nonsurgical emergency department patients. Acad Emerg Med. 2003;10:1040–1048. doi: 10.1111/j.1553-2712.2003.tb00572.x. [DOI] [PubMed] [Google Scholar]
- 61.Rhee KJ, Fisher CJ, Jr, Willitis NH. The Rapid Acute Physiology Score. Am J Emerg Med. 1987;5:278–282. doi: 10.1016/0735-6757(87)90350-0. [DOI] [PubMed] [Google Scholar]
- 62.Hoyt DB, Coimbra R, Potenza BM. Trauma systems, triage, and transport. In: Feliciano DV, Mattox KL, Moore EE, editors. Trauma. 6th. New York, NY: The McGraw Hill Comanies, Inc; 2008. pp. 57–82. [Google Scholar]
- 63.New Verification Site Visit Outcomes. [Accessed May 8, 2014];2012 at http://www.facs.org/trauma/verifivisitoutcomes.html. [Google Scholar]
- 64.Chambers J, Cohen SS, Hemminger L, Prall JA, Nichols JS. Mild traumatic brain injuries in low-risk trauma patients. J Trauma. 1996;41:976–980. doi: 10.1097/00005373-199612000-00006. [DOI] [PubMed] [Google Scholar]
- 65.Borg J, Holm L, Cassidy JD, Peloso P, Carroll L, Von Holst H, Ericson K. Diagnostic procedures in mild traumatic brain injury: results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. Journal of Rehabilitation Medicine. 2004;36:61–75. doi: 10.1080/16501960410023822. [DOI] [PubMed] [Google Scholar]
- 66.Stiell IG, Bennett C. Implementation of clinical decision rules in the emergency department. Academic Emergency Medicine. 2007;14:955–959. doi: 10.1197/j.aem.2007.06.039. [DOI] [PubMed] [Google Scholar]
- 67.Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, Rubin HR. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458–1465. doi: 10.1001/jama.282.15.1458. [DOI] [PubMed] [Google Scholar]

