Reliability of the visio-vestibular examination for concussion among providers in a pediatric emergency department

Abstract

BACKGROUND:

Visio-vestibular examination (VVE) deficits are common following pediatric concussion. Guidelines recommend assessing these deficits on all potentially concussed youth given their diagnostic and prognostic value, however test psychometrics of the VVE in the emergency department (ED) setting are unknown. Our objective was to determine the inter-rater (IRR) and test-retest reliability (TRR) of the VVE in a pediatric ED.

METHODS:

We enrolled 155 patients (112 IRR; 43 TRR) age 6–18 years with head injury presenting to the ED of a tertiary care children’s hospital. Exams were performed by a group of 65 attending/fellow physicians, pediatricians, and advanced practice providers. The VVE consisted of 9 maneuvers (smooth pursuits, horizontal/vertical saccades and gaze stability, binocular convergence, left/right monocular accommodation, complex tandem gait). Cohen’s kappa was calculated for IRR and TRR for each element.

RESULTS:

For IRR, 5/9 kappas (saccades, gaze stability, monocular accommodation) were in the moderate agreement range (0.40 to 0.60); remaining kappas showed fair agreement. For TRR, 6/9 maneuvers (saccades, horizontal gaze stability, monocular accommodation, tandem gait) showed substantial agreement (0.60 to 0.80). Kappas of 7/9 elements for subjects age 15–18 showed improved IRR and TRR.

CONCLUSIONS:

The individual elements of the VVE show fair to moderate agreement between providers and moderate to substantial agreement among the same provider in the ED setting. These findings suggest a role in the VVE in evaluating concussion acutely, particularly given its previously demonstrated ability to assist in risk stratification of concussed youth and the importance of early diagnosis for improved outcomes.

1. INTRODUCTION

Concussions are common childhood injuries. Nearly 2 million concussions occur per year in the pediatric and adolescent population, accounting for over 200,000 emergency department (ED) visits annually.^1,2 The diagnosis, however, remains difficult, particularly in the acute setting.^3,4 As the physiology of concussion is better understood, visio-vestibular deficits are recognized as key elements of its morbidity. These deficits are frequently present in concussed youth.^5,6 In the past 3 years, the Centers for Disease Control and Prevision, the American Academy of Pediatrics Council on Sports Medicine and Fitness, and the International Conference on Concussion in Sport have all released consensus recommendations that include performing visio-vestibular testing when diagnosing concussion.^7–9 One initial battery of visio-vestibular testing, the Vestibular/Oculomotor Screening (VOMS) examination, has been validated in the pediatric population,¹⁰ however its widespread use outside of the specialty setting has not been demonstrated. A modified version of this testing, the visio-vestibular examination (VVE), has been shown to be readily adopted by non-specialty providers in both the primary care and emergency department settings.^11,12 This testing is important for both diagnosis and risk stratification, as deficits are strongly associated with prolonged concussion symptoms.^13–15 This is especially relevant for the acute care provider, as multiple prior studies have shown early evaluation of pediatric concussion by multidisciplinary specialty clinics is associated with improved recovery times.^16,17

While some data exist showing internal consistentcy^18,19 and high test-retest reliability^20–22 of the VOMS examination, this data was collected exclusively among healthy athletes and by sports medicine providers, limiting exam generalizability. No data currently exists evaluating the test psychometrics of vestibular testing in non-athletes by providers outside of the specialty setting. In order to enable all clinicians to most accurately utilize the examination, and therefore maximize their ability to accurately diagnose and prognosticate pediatric concussion, it is imperative to understand its reliability. The primary objective of this study was to determine the inter-rater and test-retest reliability of the individual elements of the VVE in children presenting to a pediatric ED with head injury. Secondary objectives included determining the reliability of the overall examination findings, and evaluating how reliability differed by age and time between exams.

2. METHODS

2.1. Study Design and Patient Population

This was an observational, cross-sectional study of patients age 6–18 years old presenting to a pediatric ED in a tertiary care children’s hospital with a complaint of head injury over a two-year period, from 11/1/2017–12/31/2019. As we desired to establish the psychometrics of the VVE in all patients presenting with head injury (who would receive the exam in real practice), any patient with a head injury with possible concussion was included (final diagnosis of concussion was recorded). Additional inclusion criteria included Glasgow Coma Scale score of 13–15 (indicative of a mild traumatic brain injury) and presence of a parent or guardian (for patients age <18 years-old). Exclusion criteria included underlying neurologic disorders that may predispose a subject to an abnormal visio-vestibular examination (including cerebral palsy, benign paroxysmal positional vertigo, bilateral or unilateral vestibular hypofunction, strabismus, diplopia); patients who were visibly intoxicated with an illicit substance or who had positive drug toxicology testing, if performed; developmental delay precluding performance of the examination; lower extremity trauma preventing gait and balance testing; ocular trauma preventing eye tracking testing; concern for cervical spinal injury preventing vestibulo-ocular reflex and gait tasting; and non-English speaking patients or families. Subjects were screened for eligibility by trained research assistants and enrolled in a convenience sample based on the presence of study team members described in Section 2.2. Subjects were enrolled when research assistants were present in our ED, from 7 am until 12 am, 7 days per week. After written parental consent (or patient consent for those 18 years old) and verbal child assent (for subjects less than 18 years old) were obtained, a brief medical record review, as well as patient and parent interview, were completed to confirm eligibility. The study was approved by our institution’s Institutional Review Board.

2.2. Data Collection

Following screening, demographic data (including age, sex, concussion history, mechanism of injury, and any excluding chronic neurologic problems) were obtained from the patient’s electronic medical record and patient/parent interview. The visio-vestibular examination was then performed by a study investigator. There were 65 team members eligible to enroll patients, including pediatric emergency medicine attending physicians (31), fellow physicians (11), pediatricians practicing in the emergency department (5), and advanced practice providers (18). All study investigators were trained by the principal investigator, a pediatric emergency medicine physician with prior experience in sports medicine and expertise in performing the examination, during a 20-minute training session that included observation of exam performance. In addition, electronic decision support was provided in the form of an electronic template in the patient’s medical record (which lists examination elements), and a video demonstration of the VVE with a listing of abnormal findings on the institution’s acute head trauma clinical pathway.²³ After initial performance of the examination, a second examination was performed, either by a different provider for those patients in the inter-rater reliability arm, or by the same provider for those patients in the test-retest reliability arm. To prevent undue patient burden, only two total examinations were performed (the test-retest reliability arm was initiated after completion of the inter-rater reliability arm). Examinations were performed at least 15 minutes apart to allow any provoked symptoms to resolve. Study data were collected and managed using Research Electronic Data Capture (REDCap) tools hosted at our institution.²⁴

2.3. The Visio-Vestibular Examination

The visio-vestibular examination performed in this study is a modified version of the validated Vestibular/Ocular Motor Screening Assessment (VOMS),¹⁰ and consists of 9 total elements. At our institution, we have standardized its performance across specialties, including providers in primary care, emergency medicine, and sports medicine, and have demonstrated feasibility of performing the VVE in these settings.^11,12 The examination takes 3–5 minutes in total to perform. Abnormalities on each exam element are listed below, as previously described in studies evaluating the VVE in non-concussed youth.²⁵ We have previously shown that this examination can be performed developmentally in children as young as 6 years old.²⁵ Providers were instructed to perform the elements in the same order for each evaluation. Exam elements are described in full in Table 1, and are also demonstrated in video form via the following hyperlink: https://www.youtube.com/watch?v=lGUDZnZOieM. Exam elements include:

• (1)Smooth pursuits, where the patient follows the provider’s finger as it moves horizontally, progressing more rapidly, for 5 total repetitions. An abnormality on this portion of testing was defined as patient-reported symptom provocation (including headache, dizziness, eye fatigue, eye pain, or nausea), jerky or jumpy eye movements, or greater than 3 beats of nystagmus.
• (2, 3)Fast saccades, where the patient rapidly moves his or her eyes between the examiner’s fingers (held either shoulder-width apart for horizontal saccades or between the mid-forehead and sternal notch for vertical saccades). Abnormalities included patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) after 20 or fewer repetitions. The number of repetitions was chosen based on data showing that a significant proportion of non-concussed children can experience symptoms after 20 repetitions.²⁵
• (4, 5)Gaze stability testing (the vestibular-ocular reflex), where the patient shakes his or her head “yes” (for vertical gaze stability) or “no” (for horizontal gaze stability) while fixing his or her eyes on the provider’s finger. Abnormalities included patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) after 20 or fewer repetitions. The number of repetitions was chosen based on data showing that a significant proportion of non-concussed children can experience symptoms after 20 repetitions.²⁵
• (6)Near-point of convergence testing, where the patient is asked to identify the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification badge) become double. An abnormal exam was defined as doubling of letters at greater than 6 centimeters from the patient’s forehead.²⁶ Providers were asked to record distance as they would in usual practice, either via a tape measure or a general estimate.
• (7, 8)Left and right monocular accommodation, where the patient is asked to determine the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification card) become blurry. An abnormal examination was defined utilizing Hofstetter’s formula, where the cut-off for an abnormal distance increases as a child ages (>=8.7 cm for a child 6 years old, >=8.9 cm for a child 7 years old, etc. up to >=11.7 cm for a child 18 years old).²⁷ Providers were asked to record distance as they would in usual practice, either via a tape measure or a general estimate.
• (9)Complex tandem gait, assessed by having a patient walk both forwards and backwards with his or her eyes open and closed for 5 steps each. An abnormal examination was defined as errors (steps off the straight line) or sway (raising of arms for stability or any truncal movement off a vertical line extending from the crown of the subject’s head to the midline between his or her feet) during any of the four maneuvers.²⁸

Table 1.

Visio-vestibular examination elements

Exam Element	Description	Abnormalities
(1) Smooth pursuits	Patient follows the provider’s finger as it moves horizontally, progressing more rapidly, between the subject’s shoulders Examiner stops in the middle of the patient’s visual field Complete 5 total repetitions	Patient-reported symptom provocation (including headache, dizziness, eye fatigue, eye pain, or nausea) Jerky or jumpy eye movements Greater than 3 beats of nystagmus
(2,3) Fast saccades	Patient rapidly moves his or her eyes between the examiner’s fingers (held either shoulder-width apart for horizontal saccades or between the mid-forehead and sternal notch for vertical saccades) Patient is instructed to stop if significant symptom provocation occurs Complete 20 total repetitions if able	Patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) at less than or equal to 20 repetitions
(4,5) Gaze stability testing (the vestibular-ocular reflex)	Patient shakes his or her head “yes” (for vertical gaze stability) or “no” (for horizontal gaze stability) while fixing his or her eyes on the provider’s finger Patient is instructed to stop if any symptom provocation occurs Complete 20 total repetitions if able	Patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) at less than or equal to 20 repetitions
(6) Near-point of convergence testing	The patient is asked to identify the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification badge) become double. Providers record distance either via a tape measure or a general estimate	Doubling of letters at greater than 6 centimeters from the patient’s forehead.24
(7, 8) Left and right monocular accommodation	The patient is asked to determine the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification card) become blurry with one eye closed Providers record distance either via a tape measure or a general estimate	Defined utilizing Hofstetter’s formula, where the cut-off for an abnormal distance increases as a child ages (>=8.7 cm for a child 6 years old, >=8.9 cm for a child 7 years old, etc. up to >=11.7 cm for a child 18 years old)
(9) Complex tandem gait	Patient walks both forwards and backwards with his or her eyes open and closed for 5 steps each (generally in order of forward eyes open, forward eyes closed, backward eyes open, backward eyes closed)	Errors = steps off the straight line with any of the 4 maneuvers Sway = raising of arms for stability or any truncal movement off a vertical line extending from the crown of the subject’s head to the midline between his or her feet during any of the 5 maneuver

In addition to an assessment of abnormality for each individual measure, an overall abnormal determination was made for each subject utilizing the total number of abnormal elements; based on prior data from non-concussed youth, 3 or more abnormal test elements (out of the 9 total) led to a designation of an abnormal visio-vestibular test battery.²⁵

2.4. Statistical Analysis

Demographic and baseline statistics were summarized using standard descriptive statistics. Both inter-rater and test-retest reliability were assessed using Cohen’s kappa for each individual element of the examination, as well as the overall test assessment. Kappas were interpreted with the following scale: <0 = less than chance agreement; 0 = change agreement; >0–0.20 = slight agreement; >0.20–0.40 = fair agreement; >0.40–0.60 = moderate agreement; >0.60–0.80 = substantial agreement; and >0.80–0.99 = near-perfect agreement, 1.00 = perfect agreement.²⁹ A Kappa statistic was chosen for test-retest reliability as determinations were made on a categorical scale for each individual test (rather than an intraclass correlation coefficient [ICC], which would rely on continuous data). Secondary analyses of kappas for both inter-rater and test-retest reliability were performed to determine if age or time between exams affected the measurements of reliability. A univariate logistic regression was utilized to determine the odds of a final diagnosis of concussion based on the total number of abnormal examination elements, reported on a continuous scale from 0 to 9. The odds ratio determined the increased likelihood of a final concussion diagnosis based on the presence of an additional abnormal individual examination element. Additionally, unadjusted odds of a final diagnosis of concussion based on the designation of an abnormal composite examination (3 or more abnormal individual elements) were calculated. The analysis was conducted using Stata version 14.2 (StataCorp, College Station, TX).

3. RESULTS

Overall, 155 patients were recruited, 112 for the inter-rater reliability arm and 43 for the test-retest reliability arm. A flowchart of cohort derivation is shown in Figure 1. Demographics, including age, gender, concussion history, and mechanism of injury are shown in Table 2. The median time from injury to initial examination was 5.2 hours (interquartile range 3.3 hours to 20.0 hours) and the first examination was performed with 48 hours of injury in 85% of our sample. The median time between examinations was 35 minutes (interquartile range 27.0 minutes to 50.0 minutes). Overall, 63% of our sample received a diagnosis of concussion by the treating physician.

Figure 1.

Flow diagram of patients screened, approached, and enrolled into the current study.

Table 2.

Patient demographics

Characteristic	Overall, N (%)	Inter-rater reliability arm, N (%)	Test-retest reliability arm, N (%)
Patients	155	112	43
Age categories
6–8 years old	22 (14.2%)	18 (16.1%)	4 (9.3%)
9–11 years old	51 (32.9%)	36 (32.1%)	15 (34.9%)
12–14 years old	47 (30.3%)	37 (33.0%)	10 (23.3%)
15–18 years old	35 (22.6%)	21 (18.8%)	14 (32.6%)
Female gender	68 (43.9%)	55 (49.1%)	13 (30.2%)
Sports mechanism (vs. non-sports)	54 (34.8%)	41 (36.6%)	13 (30.2%)
Prior concussion history	23 (14.8%)	17 (15.2%)	6 (14.0%)
Median hours between injury and first exam (IQR)	5.2 (3.3, 20.0)	4.6 (3.0,18.0)	5.5 (4.0, 24.9)
Time between exams
Median minutes between exams (IQR)	35.0 (27.0, 50.0)	35.0 (27.0, 50.0)	35.0 (25.0, 53.0)
Exams within 30 minutes	51 (32.9%)	36 (32.1%)	15 (34.9%)
Final diagnosis of concussion	97 (62.6%)	68 (60.7%)	29 (67.4%)
Other final diagnoses
Superficial head injury	21 (13.5%)	17 (15.2%)	4 (9.3%)
Face/scalp contusion	10 (6.5%)	5 (4.5%)	5 (11.6%)
Face/scalp laceration	12 (7.7%)	9 (8.0%)	3 (7.0%)
Facial bone fracture	4 (2.6%)	3 (2.7%)	1 (2.3%)
Other	11 (7.1%)	10 (8.9%)	1 (2.3%)

IQR = interquartile range

For inter-rater reliability, overall, the kappas for 5 of the 9 exam elements fell between 0.40 and 0.60 (Table 3). Two of the elements with kappa between 0.20 and 0.40, smooth pursuits and complex tandem gait, had higher kappas when evaluating individual sub-components of the element testing (smooth pursuits kappa increased from 0.331 to 0.563 when evaluating for symptom provocation only rather than abnormal signs and symptom provocation together; complex tandem gait kappa increased from 0.386 to 0.569 when considering only errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormal elements). Several of the elements (smooth pursuits, horizontal and vertical saccades, convergence, monocular accommodation, and complex tandem gait) had kappas greater than 0.60 when considering only adolescents age 15–18 years old. In addition, several exam elements (horizontal and vertical saccades, horizontal and vertical gaze stability, convergence, and right monocular accommodation) had higher kappas when only considering those exams performed less than 30 minutes apart compared to the overall sample.

Table 3.

Inter-rater reliability for each exam element

Inter-rater reliability	% Agreement (N=113)	Overall Kappa (N=113)	Kappa Age 15–18 (n=21)	Kappa Exams Within 30 min (n=37)
Smooth Pursuits	74%	0.331	0.767	0.235
Symptom pursuits (symptoms only)*	94%	0.563	0.632	0.767
Horizontal Saccades	78%	0.512	0.798	0.643
Vertical Saccades	75%	0.472	0.806	0.634
Horizontal Gaze Stability	81%	0.535	0.483	0.679
Vertical Gaze Stability	79%	0.478	0.284	0.532
Convergence	68%	0.347	0.500	0.423
L Monocular Accommodation	85%	0.521	0.773	0.492
R Monocular Accommodation	80%	0.367	0.615	0.492
Tandem Gait (including backward eyes closed)	70%	0.386	0.529	0.235
Tandem Gait (excluding backward eyes closed)#	80%	0.569	0.586	0.729

^*Evaluates reliability for smooth pursuits when only considering symptom provocation as an abnormality

^#Evaluates reliability for tandem gait when only considering errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormalities

When considering test-retest reliability, the majority of kappas (6 of the 9 exam elements) were between 0.60 and 0.80 (Table 4). Similar to inter-rater reliability, the two lowest kappa exam elements were smooth pursuits (kappa = 0.584) and binocular convergence (kappa = 0.588). Many of the individual exam elements (including smooth pursuits, vertical saccades, horizontal and vertical gaze stability, convergence, and left and right monocular accommodation) showed higher kappas, several greater than 0.80, among adolescents age 15–18 years old. Also similar to the inter-rater reliability arm, for 5 of 9 exam elements (horizontal and vertical saccades, horizontal and vertical gaze stability, and tandem gait), those with exams performed within 30 minutes of one another showed higher kappas when compared with the overall sample.

Table 4.

Test-retest reliability for each exam element

Test-Retest Reliability	% Agreement (n=43)	Overall Kappa (n=43)	Kappa Age 15–18 (n=14)	Kappa Exams Within 30 min (n=15)
Smooth Pursuits	83%	0.584	0.689	0.588
Symptom Pursuits (symptoms only)*	87%	0.492	0.435	0.755
Horizontal Saccades	88%	0.777	0.625	0.875
Vertical Saccades	88%	0.763	0.857	0.866
Horizontal Gaze Stability	80%	0.539	0.581	0.722
Vertical Gaze Stability	83%	0.634	0.720	0.865
Convergence	79%	0.588	0.683	0.359
L Monocular Accommodation	90%	0.732	0.831	0.512
R Monocular Accommodation	92%	0.773	0.824	0.432
Tandem Gait (including backward eyes closed)	88%	0.721	0.400	0.837
Tandem Gait (excluding backward eyes closed)#	91%	0.784	0.744	1.00

^*Evaluates reliability for smooth pursuits when only considering symptom provocation as an abnormality

^#Evaluates reliability for tandem gait when only considering errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormalities

Finally, we analyzed the number of abnormal individual examination elements, stratified by final diagnosis (Table 5). When evaluating the overall examination in children with and without a final diagnosis of concussion, in a graded analysis of a unadjusted regression, a trend emerged with each additional abnormal examination element increasing the likelihood of a final diagnosis of concussion being assigned (odds ratio = 2.1 of a concussion diagnosis [95% confidence interval 1.6 to 2.7] for each additional abnormal element). We additionally found that, when classifying an abnormal overall examination as 3 or more abnormal individual elements, 61.9% of those ultimately diagnosed with a concussion had an abnormal overall examination, compared to 8.6% of those without a concussion diagnosis (OR 17.2, 95% CI 6.3 to 46.9).

Table 5.

Total number of abnormal examination elements (fsirst examination only) by final diagnosis

Number of Abnormalities	No Concussion Diagnosis (N, %)	Concussion Diagnosis (N, %)
0	34 (58.6%)	19 (19.6%)
1	12 (20.7%)	10 (10.3%)
2	7 (12.1%)	8 (8.3%)
3	5 (8.6%)	13 (13.4%)
4	0 (0 %)	11 (11.3%)
5	0 (0 %)	10 (10.3%)
6	0 (0 %)	9 (9.3%)
7	0 (0 %)	9 (9.3%)
8	0 (0 %)	5 (5.2%)
9	0 (0 %)	3 (3.1%)

4. DISCUSSION

This study, evaluating reliability of the visio-vestibular examination (VVE) for concussion in an acute care setting in children and adolescents presenting with head injury, found that the individual elements of the examination had fair to moderate agreement between two raters and moderate to substantial agreement among the same rater. A reliable bedside tool is of utmost importance in concussion management from the acute setting. Previous studies have shown this testing can improve diagnostic accuracy in both the acute and subacute timeframe.^12,30 This finding is highlighted in the current study by the increased odds of a concussion diagnosis with each abnormal individual element and significantly increased odds of a concussion diagnosis with an overall abnormal examination. As important, if not moreso, is the prognostic value of the exam; abnormalities on individual elements of the examination have been shown to correlate strongly with prolonged concussion symptoms.^5,13–15 Allowing the acute care provider to recognize those likely to have prolonged recoveries shortly after injury is particular salient; two recently published studies have shown that in those seen in multidisciplinary concussion clinics, initial evaluation within 1 week was associated with improved recovery times.^16,17

Previous studies have evaluated the test-retest reliability of the Vestibular/Oculomotor Screening (VOMS) examination, a similar test of oculomotor function, in healthy athletes in the non-acute setting. The VOMS examination relies on symptom provocation with 6 maneuvers (smooth pursuits, horizontal and vertical saccades, horizontal and vertical gaze stability, and visual motion sensitivity), as well as a measurement of near-point of convergence.¹⁰ Worst et al. evaluated 45 high school athletes (age 13–18 years old) and found levels of agreement between 49% and 89% on the individual VOMS elements compared over three time points when performed among a group of 6 testers (4 athletic trainers, a neuropsychologist, and a student).²⁰ Our overall percent agreements for the VVE are all within the upper range of these values (79% to 92% for our 9 exam elements). Yorke et al. evaluated the performance of 105 non-concussed adolescents on the VOMS exam, with 21 completing the examination twice. They found that there was perfect agreement in symptom changes for 64% of the examinations.²¹ Finally, Anderson, et al. evaluated the performance of 58 healthy athletes on the VOMS examination approximately 3 months apart, and found ICC ranging from 0.29 to 0.71.²² Our results (kappas of individual elements ranging from 0.539 to 0.773), among a much larger group of non-sports medicine providers, show similar, if not improved, test-retest reliability when compared with these studies. Of note, none of these studies evaluated inter-rater reliability of the VOMS examination.

In evaluating inter-rater reliability, we found several sub-analyses yielded improved kappa scores: evaluating only a portion of smooth pursuits and tandem gait, and considering only teenagers age 15–18 years old; these findings have implications for further exam evaluation. Our smooth pursuit testing evaluated both symptom provocation and abnormal signs, including jerky or jumpy eye movements and greater than 3 beats of nystagmus during testing. When we examined smooth pursuits for symptom provocation alone, the inter-rater reliability improved into the moderate agreement range. Similarly, our tandem gait testing showed substantial agreement on test-retest reliability, but only fair agreement on inter-rater reliability. Our complex tandem gait maneuver is unique in evaluating a patient walking forwards and backwards with eyes open and closed. The tandem gait performed in the Sport Concussion Assessment Tool, 5^th Edition (a validated sideline evaluation tool) involves a timed test of the patient walking forward with eyes open only.³¹ The latter task has been shown to have high test-retest reliability, as Howell et al. found an ICC of 0.86 in a sample of 32 healthy athletes age 9–18 years old.³² Our added components have been shown to improve sensitivity of the testing, as having the patient walk backward eyes closed has the highest sensitivity of any measure of gait or balance in evaluating concussion.²⁸ However, in the current study, we found that this element in particular reduced inter-rater reliability, as removing backward eyes closed from the battery as an abnormal finding increased inter-rater reliability from fair to moderate (kappa increased from 0.386 to 0.569). In our previous work evaluating complex tandem gait, we found a significant number of false positives (over 50% of the non-concussed sample had abnormalities) on the backward eyes closed component.²⁸ Therefore, the high prevalence of abnormalities in non-concussed patients may make identifying a true error more difficult between two examiners.

We found kappas were higher among older adolescents than younger children for the vast majority of the exam elements (7 of the 9 elements for both inter-rater and test-retest reliability). While limited previous data regarding the performance of younger children on the VOMS examination exists, Iverson et al. evaluated 387 male non-concussed athletes 8–17 years old, and found no difference in the proportion of non-concussed athletes with symptom provocation on the testing elements when stratified by age.³³ Contrary to these findings, in an evaluation of non-concussed children presenting to an emergency department setting and performing the VVE, more abnormalities were present in 9–11 and 12–14 year-old children than 15–18 year old children.²⁵ Other studies have evaluated symptom reporting in concussed children of various ages, and found internal consistency of symptom scales to be higher in teenagers; for example, Sady et al. found a higher internal consistency on the Post-Concussion Symptom Inventory among children age 13–18 when compared to children age 8–12 and children age 5–7.³³ The adolescent examination may be more reliable given the dependence on self-report of symptom provocation for most examination elements. The fact that this relationship existed for both inter-rater and test-retest reliability supports this hypothesis.

Finally, we found only fair inter-rater reliability for bincocular convergence (as well as right monocular accommodation), leading to a need for further adaption of this exam maneuver. There is a wide variation in the literature of the incidence of abnormal convergence when performing vestibular and oculomotor testing among non-concussed youth. In a sample of emergency department patients similar to the current study, 2% of non-concussed youth age 6–18 years old had abnormalities;²⁵ Vernau et al. found 11.5% of youth athletes age 6–18 years old had abnormalities;³⁴ and Iverson et al. found 22% of male athletes age 8–17 years old had an abnormal binocular convergence.³⁵ It has been hypothesized that examiner-related differences contribute to this variability.³⁵ Among these studies, there is variability in how distances are assessed; some studies utilize a single letter on an oversized tongue depressor, with distances measured using a tape measure,²¹ whereas others use an accommodative ruler with sliding card.²⁵ In this study, we encouraged providers to measure as they would in real life practice, and many reported they would estimate distances rather than measure using a ruler or tape measure. Previous literature has shown that emergency department personnel are often inaccurate when estimating physical exam findings, such as laceration length.³⁶ Taken together, these results suggest that a more standardized approach to evaluating binocular convergence as part of vestibular and oculomotor testing is needed.

There are several limitations to this study. Our study was performed in a single center, an academic tertiary care institution. As such, this may lower the generalizability of our results. In our center, all providers received electronic decision support and had access to a clinical pathway detailing the examination (in addition to specific training received for this study).²³ As providers at an academic center, they may be more likely to incorporate new examination techniques into their practice. However, we highlight the diversity of provider types and the large number of providers participating in this study. Since all our patients were enrolled from the emergency department setting, they likely represent a specific cohort of concussed children. However, this sample likely includes both athletes and non-athletes given the high proportion of non-sports injuries in our sample, increasing our overall generalizability when compared to studies focusing solely on athletes. The vast majority (85%) of our subjects were evaluated within 48 hours of their injury, therefore we cannot comment on the reliability of our examination in the subacute phase of injury. However, this is representative of the population of children who seek care in the acute setting.³⁷ Finally, we recognize that a large number of patients evaluated for head injury during enrollment period were not approached due to missed screening or lack of availability of research assistants/study team members; as the primary objective of our study was evaluating the examiners performing the test, we do not believe this significantly biased our results.

5. CONCLUSIONS

The individual elements of visio-vestibular examination (VVE) for concussion show fair to moderate agreement between providers and moderate to substantial agreement among the same provider in the pediatric emergency department setting. Elements provoking symptoms (saccades, gaze stability) have higher reliability than those evaluating physical signs (smooth pursuits) or utilizing estimated measurements (binocular convergence), and overall the examination was more reliable in older adolescents. These findings suggest a role for the VVE in evaluating concussion in the acute setting, particularly given its previously demonstrated ability to assist in risk stratification of concussed youth. Future work is needed to assess the reliability of a VVE with modifications to those elements assessing physical signs, and determine those VVE elements, along with other bedside evaluation tools, most discriminatory in diagnosing and prognosticating youth concussion

Abbreviations:

ED

emergency department

IR

inter-rater

TR

test-retest

VOMS

vestibular/oculomotor screening

VVE

visio-vestibular examination