Skip to main content
PMC home page
HHS Author Manuscripts logoLink to HHS Author Manuscripts
. Author manuscript; available in PMC: 2018 Aug 8.
Published in final edited form as: Clin Pediatr (Phila). 2017 May 19;56(9):854–865. doi: 10.1177/0009922817709555

Improving Primary Care Provider Practices in Youth Concussion Management

Kristy B Arbogast 1,2, Allison E Curry 1,2, Kristina B Metzger 1, Ronni S Kessler 1, Jeneita M Bell 3, Juliet Haarbauer-Krupa 3, Mark R Zonfrillo 4, Matthew J Breiding 3, Christina L Master 2,5
PMCID: PMC6082149  NIHMSID: NIHMS976794  PMID: 28521519

Abstract

Primary care providers are increasingly providing youth concussion care but report insufficient time and training, limiting adoption of best practices. We implemented a primary care–based intervention including an electronic health record–based clinical decision support tool (“SmartSet”) and in-person training. We evaluated consequent improvement in 2 key concussion management practices: (1) performance of a vestibular oculomotor examination and (2) discussion of return-to-learn/return-to-play (RTL/RTP) guidelines. Data were included from 7284 primary care patients aged 0 to 17 years with initial concussion visits between July 2010 and June 2014. We compared proportions of visits pre- and post-intervention in which the examination was performed or RTL/RTP guidelines provided. Examinations and RTL/RTP were documented for 1.8% and 19.0% of visits pre-intervention, respectively, compared with 71.1% and 72.9% post-intervention. A total of 95% of post-intervention examinations were documented within the SmartSet. An electronic clinical decision support tool, plus in-person training, may be key to changing primary care provider behavior around concussion care.

Keywords: primary care, electronic health record, clinical decision support, traumatic brain injury

Introduction

Estimates of the incidence of youth concussion have recently increased and health care utilization patterns indicate that families are increasingly using primary care providers (PCPs) for both initial and follow-up concussion care.1–7 The exact cause of the higher incidence is unclear, but may in part be due to heightened public awareness, state legislation requiring medical clearance before returning to play, and greater availability of timely appointments in primary care than specialty care.8,9 Regardless of the cause, clinical management of the majority of youth concussion currently rests with PCPs.6

This scenario presents a challenge to PCPs, who may have insufficient time to systematically diagnose and manage concussion patients and may lack concussion-related continuing medical education.10–13 Furthermore, PCPs have demonstrated difficulty in translating important concussion management concepts such as “cognitive rest” and “return-to-learn” into clinical practice.14 Insufficient time and training may limit adoption of best practices, including the implementation of emerging assessment techniques. This, in turn, can lead to over-referral to specialists, unnecessarily burdening the health care system.

To address challenges in PCPs’ management of concussion, we implemented a primary-care based intervention within the Children’s Hospital of Philadelphia’s (CHOP) large pediatric health care network with the primary goal of increasing CHOP PCPs’ (including nurse practitioners and physicians) adoption and systematic documentation of recommended best practice concussion management guidelines.15 The intervention included 2 key activities: (1) in-person training of PCPs that defined and recommended best practices for managing concussion and (2) development and integration of a primary care concussion-focused clinical decision support tool within CHOP’s existing electronic health record (EHR) system. Although such tools have challenges,16–18 when integrated into an EHR they can provide valuable infrastructure in a busy primary care practice to systematically convert best practice management guidelines into clinical practice, facilitate the use of structured screening and diagnostic assessments, and enable systematic documentation across a broad health care network.19–23 The primary objective of this study was to determine whether the intervention was effective in changing provider behavior surrounding 2 concussion management strategies: (1) performance of a vestibular oculomotor examination, an emerging technique in concussion assessment24,25 and (2) provision of return-to-play (RTP) and return-to-learn (RTL) guidelines to patients at the time of care. We also sought to identify relevant patient-, visit-, and provider-level predictors of these behaviors in the post-intervention period. Finally, we determined whether the introduction of an EHR-based clinical decision support tool enabled providers to systematically and consistently document these 2 practices.

Methods

The CHOP primary care health care network includes over 30 locations in southeastern Pennsylvania and southern New Jersey, serves a socioeconomically and racially diverse patient population and accepts most insurance plans, including Medicaid. CHOP PCPs function as the pediatric medical home for their patients, managing all aspects of clinical care—including all initial visits, follow-up care and subspecialty referrals—using a linked EHR system (EpicCare, Epic Systems, Inc, Madison, WI).

Intervention Development

A thorough needs assessment was performed to inform the development of the intervention, which included surveying primary care providers with regard to their current concussion knowledge, practices and comfort level in providing care. The results of this needs assessment were published previously.14 Sports medicine sub-specialty experts in concussion care were engaged with PCPs throughout the process to develop up to date and accurate clinical content for the clinical decision support tool and synthesize that content into a format that would optimize utility in the primary care setting.

Intervention Implementation

During May and June 2012, pediatric sports medicine physicians and nurse practitioner concussion specialists provided in-person training sessions to CHOP PCPs during five 2-hour sessions. Each session consisted of 60 minutes of didactic lecture, including exemplar videos of vestibular, oculomotor and balance deficits commonly seen in concussed youth.26 This was followed by 60 minutes of small group role play where providers practiced specific aspects of the concussion examination, including the vestibular oculomotor examination, which was unfamiliar to most attendees. The examination, a modified brief vestibular oculomotor examination,24 involves clinical assessment of smooth pursuits, saccades, vestibular ocular reflex function; performance on a tandem gait task with eyes open and closed, forward and backward; and a measurement of near point of convergence.27 This examination was chosen as a focus because emerging evidence suggests deficits in these systems appear to be common following concussion.22,23 Current recommendations for return-to-activity (cognitive and physical) based on Zurich guidelines were reviewed.28 One representative from each primary care practice attended at least one session. These individuals were identified as “concussion champions” and were encouraged to serve as conduits for disseminating up-to-date knowledge to other providers within their practice. A total of 90 providers attended, representing approximately half of all PCPs within the network at the time.

Concurrent with in-person trainings, we introduced a concussion-specific clinical decision support tool for PCPs—the Concussion SmartSet and referred to herein as the “SmartSet” (EpicCare, Epic Systems, Inc)—within the CHOP EHR system. Briefly, the SmartSet is Epic’s term for a clinical decision support menu of documentation and order options that helps standardize and streamline patient care and for this purpose, was designed to guide PCPs through concussion-specific patient assessments for diagnosis and management. During an office visit providers are prompted to choose the SmartSet when noting a relevant chief complaint (eg, head injury, headache, concussion) or may select the SmartSet on their own. Once chosen, the SmartSet automatically populates the EHR for that visit with a standardized template for collecting detailed history and symptoms, recording physical, vestibular oculomotor, and neurocognitive examination findings, and providing after-care patient instructions. Specific data fields exist for each subcomponent of each examination such that it is clear which aspects of the clinical evaluation were completed. The provider systematically completes the template utilizing both drop-down menus and free-text fields. Components of the template align with those elements highlighted in the trainings described above. Patient educational materials, including one that describes concussion symptoms and one that describes RTL and RTP principles, were created and reminders to provide them to the patient at the conclusion of the visit were integrated into the SmartSet. Use of the SmartSet, performance of the vestibular oculomotor examination, and provision of RTL/RTP guidelines were suggested as clinical best practice through the training but not programmed as “required to be completed” within the EHR.

PCPs were instructed to use the SmartSet at each concussion visit, with separate versions depending on whether it was an initial or follow-up visit. To ensure wide dissemination, in addition to the trainings, all providers in the network were notified about the SmartSet during regular practice meetings and via email blasts. Providers that joined the CHOP network after the training period were offered the opportunity to view a taped video of the training and were guided on the use of the SmartSet by their colleagues. The tool became available on July 1, 2012. Thus, we defined the pre-SmartSet period as the 2-year period prior to implementation (July 1, 2010 to June 30, 2012) and the post-SmartSet period as the 2-year period after implementation (July 1, 2012 to June 30, 2014).

Case Identification

We queried the EHR to identify all concussion-related office visits to CHOP’s primary care practices from July 1, 2010 through June 30, 2014 for patients, age 0 to 17 years, with an initial concussion visit during this period. Concussion visits were defined as those assigned an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis code indicative of a concussion (appendix). Initial visits were defined as the first clinical encounter for a concussion event as determined by date of the encounter within the study period; patients who were receiving ongoing concussion treatment at the onset of the study period, defined as those who had had a concussion-related CHOP visit in the 6 months before the start of the study period, were identified and excluded from the study. Once identified via their initial visit, all visits related to that concussion for that patient were included.

Outcome Measures

We chose 2 key provider practices as the main study outcomes: (1) performance of the vestibular oculomotor concussion examination and (2) discussion of RTP and RTL guidelines. These 2 components of the SmartSet were chosen as study outcomes because they represent emerging concepts (vestibular oculomotor examination) or existing guidelines (RTL and RTP) that were important to reinforce for adoption into routine clinical care for concussion. To determine if the vestibular oculomotor examination had been performed, we conducted a keyword search of the EHR provider notes for any of the following terms: saccades, smooth pursuits, gaze stability, vestibular ocular reflex (or “vor”), and convergence. For visits in the post-SmartSet period, we also determined whether the vestibular oculomotor examination was documented within the SmartSet template—indicating that the template was utilized to manage the concussion visit—versus documented only in free-text provider notes, which indicated that the template was not utilized. To determine whether guidelines on returning to activities were discussed, an automated search of the visit’s provider note was conducted for the phrases “return-to-learn,” “return-to-school,” “return-to-play,” “return-to-sports,” and “return-to-activity.” These will collectively be referred throughout this article as RTL/RTP.

Other Variables

Patient-level variables included age on the visit date (0–4, 5–11, 12–14, 15–19 years); sex; and race/ethnicity (non-Hispanic white, non-Hispanic black, Hispanic, non- Hispanic other). Visit-level variables include payor (Medicaid, private, self-pay), day of week, time of day, and concomitant injury. Concomitant injury was defined as any nonconcussion injury with an ICD-9-CM code of 800–957 excluding sprains/strains, superficial injury, and contusions. Type of provider (nurse practitioner, physician, other) was included as a provider-level variable.

Statistical Analysis

We used Pearson chi-square tests to compare the proportion of visits in the pre- and post-intervention periods in which the vestibular oculomotor examination was performed and RTL/RTP guidelines were provided, overall and by relevant characteristics. Furthermore, we determined the proportion of these visits in the post-intervention period in which the SmartSet template was utilized, overall and by month. In order to identify independent predictors of performance of the examination and provision of RTL/RTP guidelines in the post-intervention period, we estimated adjusted odds ratios (aOR) with multivariate logistic regression using generalized linear mixed models. The CHOP practice location was included as a random intercept in order to account for more similar management behaviors among providers at the same practice (using a compound symmetric correlation structure) and other covariates were included as fixed effects. Analyses were conducted in SAS Version 9.3 (SAS Institute Inc, Cary, NC). This study was approved with a waiver of consent/assent by the Children’s Hospital of Philadelphia’s Institutional Review Board.

Results

Description of Sample

From July 1, 2010 to June 30, 2014, we identified 14 527 concussion-related primary care office visits for 7284 unique patients. Table 1 describes the distribution of patient-, visit-, and provider-level characteristics, stratified by pre- and post-intervention period. Most patients were aged 5 to 19 years on the date of visit (98.5%), non-Hispanic white (76.0%), and had private insurance (86.4%); slightly more than half (52.5%) were male. In all, 1.6% of concussion visits included a concomitant injury diagnosis.

Table 1.

Visit-Level Characteristics for Concussion Primary Care Visits, Stratified by Pre- and Post-intervention Period (N = 14 527).

All Concussion Visits Pre-intervention July 1, 2010 to June 30, 2012 Post-intervention July 1, 2012 to July 1, 2014
n Proportion (%) n Proportion (%) n Proportion (%)
Overall 14 527 3744 10 783
Patient age (years) at date of visit
 0–4 225 1.5 89 2.4 136 1.3
 5–11 4031 27.7 950 25.4 3081 28.6
 12–14 5433 37.4 1362 36.4 4071 37.8
 15–19 4838 33.3 1343 35.9 3495 32.4
Patient sex
 Female 6898 47.5 1560 41.7 5338 49.5
 Male 7629 52.5 2184 58.3 5445 50.5
Patient race/ethnicity
 Hispanic 505 3.5 131 3.5 374 3.5
 Non-Hispanic white 11 041 76.0 2828 75.5 8213 76.2
 Non-Hispanic black 1659 11.4 463 12.4 1196 11.1
 Non-Hispanic other/Multiple 233 1.6 43 1.1 190 1.8
 Unknown 1089 7.5 279 7.5 810 7.5
Payor
 Medicaid 1773 12.2 398 10.6 1,375 12.8
 Private 12 558 86.4 3288 87.8 9270 86.0
 Self-pay 196 1.3 58 1.5 138 1.3
Presence of other injuries
 No 14 297 98.4 3653 97.6 10 644 98.7
 Yes 230 1.6 91 2.4 139 1.3
Visit is first concussion visit
 No 7671 52.8 1327 35.4 6344 58.8
 Yes 6856 47.2 2417 64.6 4439 41.2
Visit time of day
 07:00–10:59 4210 29.0 1096 29.3 3114 28.9
 11:00–14:59 5403 37.2 1313 35.1 4090 37.9
 15:00–18:59 4571 31.5 1236 33.0 3335 30.9
 19:00–22:59 342 2.4 99 2.6 243 2.3
Visit by weekend vs weekday
 Weekend: Sat/Sun 303 2.1 84 2.2 219 2.0
 Weekday: Mon-Fri 14 224 97.9 3660 97.8 10 564 98.0
Provider type
 Physician 12 740 87.7 3443 92.0 9297 86.2
 Nurse practitioner 1695 11.7 280 7.5 1415 13.1
 Other 92 0.6 21 0.6 71 0.7
Open in a new tab

Provider Behavior

In the pre-intervention period, performance of the vestibular oculomotor examination was documented in the EHR for only 1.8% of concussion visits. In contrast, 71.1% of visits in the post-intervention period included evidence of this examination (Table 2). The proportion of concussion visits in which the examination was performed increased over the first year after introduction of the SmartSet in July 2012, decreased slightly around July 2013, and then returned to higher levels through the end of the study period for an overall increase from July 2012 to June 2014 of 129% (Figure 1). Furthermore, the vast majority of examinations that were performed in the post-intervention period (95.3%) were documented within the SmartSet template; this was consistent across all demographic subgroups (>86% for each group), providing strong evidence that the template facilitated performance and systematic documentation of the examination components.

Table 2.

Proportion of Concussion Visits in the Post-intervention Period (N = 10 783) in Which the Vestibular Oculomotor Examination Was Performed and Return-to-Learn/Return-to-Play (RTL/RTP) Guidelines Were Provided.

Vestibular Oculomotor Examination Performed RTL or RTP Guidelines Provided
n % n %
Overall 7666 71.1 7859 72.9
Patient age (years) at date of visit
 0–4 34 25.0 37 27.2
 5–11 2217 72.0 2286 74.2
 12–14 2976 73.1 3041 74.7
 15–19 2439 69.8 2495 71.4
Patient sex
 Female 3870 72.5 3952 74.0
 Male 3796 69.7 3907 71.8
Patient race/ethnicity
 Non-Hispanic white 5910 72.0 6072 73.9
 Hispanic 231 61.8 242 64.7
 Non-Hispanic black 860 71.9 851 71.2
 Non-Hispanic Asian/American Indian/Other/Multiple race 129 67.9 139 73.2
 Unknown 536 66.2 555 68.5
Payor
 Private 6622 71.4 6813 73.5
 Medicaid 953 69.3 945 68.7
 Self-pay 91 65.9 101 73.2
Presence of other injuries
 No 7592 71.3 7786 73.1
 Yes 74 53.2 73 52.5
Visit is first concussion visit
 No 4397 69.3 4424 69.7
 Yes 3269 73.6 3435 77.4
Visit time of day
 07:00–10:59 2150 69.0 2222 71.4
 11:00–14:59 2914 71.2 2975 72.7
 15:00–18:59 2409 72.2 2465 73.9
 19:00–22:59 193 79.4 197 81.1
Visit by weekend vs weekday
 Weekend: Sat/Sun 157 71.7 163 74.4
 Weekday: Mon-Fri 7509 71.1 7696 72.9
Provider type
 Physician 6357 68.4 6543 70.4
 Nurse practitioner 1254 88.6 1263 89.3
 Other 55 77.5 53 74.6
Open in a new tab

Figure 1.

Figure 1

Open in a new tab

Monthly rate of performance of the vestibular oculomotor examination during concussion primary care visits (pre- and post-intervention) and proportion of those examinations documented within the SmartSet template.

During the pre-intervention period, 19.0% of concussion visits included EHR documentation of RTL/RTP guidelines. In contrast, in the post-intervention period these phrases were found in 72.9% of EHRs (Table 2). The temporal pattern of the proportion of visits in which RTL/RTP instructions were documented was similar to that of the vestibular oculomotor examination, with a total increase from July 2012 to June 2014 of 85% (Figure 2).

Figure 2.

Figure 2

Open in a new tab

Monthly rate of provision of return-to-learn/return-to-play (RTL/RTP) guidelines during concussion primary care visits (pre- and post-intervention) and proportion of those guidelines documented within the SmartSet template.

As shown in Table 3, the odds of performance of the oculomotor examination by providers in the post-intervention period was substantially lower with 0- to 4-year-old patients than with 15- to 19-year-old patients (aOR = 0.10; 95% CI = 0.06–0.15). It was also less commonly performed during visits: with male patients (aOR = 0.81; 95% CI = 0.73–0.90); with Hispanics compared with non-Hispanic white patients (aOR = 0.72; 95% CI = 0.56–0.93); with patients with concomitant injuries (aOR = 0.46; 95% CI = 0.31–0.68); with follow-up concussion visits (aOR = 0.67; 95% CI = 0.61–0.74); by physicians compared with nurse practitioners (aOR = 0.34; 95% CI = 0.27–0.41). Results were similar for provision of RTL/RTP guidelines, with the addition that the odds were lower during visits with patients insured by Medicaid (aOR = 0.82; 95% CI = 0.70–0.96).

Table 3.

Predictors of Performance of the Vestibular Oculomotor Examination and Documentation of Discussion of Return-to-Learn/Return-to-Play (RTL/RTP) Guidelines.a

Vestibular Oculomotor Examination Performed RTL or RTP Guidelines Provided
Adjusted Odds Ratio (95% CI)
Patient age (years) at date of visit
 0–4 0.10 (0.06, 0.15) 0.09 (0.06, 0.14)
 5–11 1.03 (0.91, 1.16) 1.00 (0.88, 1.14)
 12–14 1.18 (1.05, 1.32) 1.16 (1.04, 1.31)
 15–19 Reference Reference
Patient sex
 Female Reference Reference
 Male 0.81 (0.73, 0.90) 0.86 (0.78, 0.96)
Patient race/ethnicity
 Hispanic 0.72 (0.56, 0.93) 0.78 (0.60, 1.01)
 Non-Hispanic white Reference Reference
 Non-Hispanic black 1.19 (0.98, 1.44) 1.04 (0.86, 1.26)
 Non-Hispanic Asian/American Indian/Other/Multiple race 0.97 (0.68, 1.40) 1.15 (0.78, 1.68)
 Unknown 0.92 (0.77, 1.10) 0.95 (0.79, 1.14)
Payor
 Medicaid 0.89 (0.76, 1.04) 0.82 (0.70, 0.96)
 Private Reference Reference
 Self-pay 0.73 (0.48, 1.09) 1.01 (0.66, 1.56)
Presence of other injuries
 No Reference Reference
 Yes 0.46 (0.31, 0.68) 0.45 (0.30, 0.67)
Visit is first concussion visit
 No Reference Reference
 Yes 1.49 (1.35, 1.65) 1.73 (1.56, 1.92)
Visit time of day
 07:00–10:59 Reference Reference
 11:00–14:59 1.15 (1.02, 1.29) 1.06 (0.94, 1.19)
 15:00–18:59 1.05 (0.92, 1.18) 0.99 (0.87, 1.12)
 19:00–22:59 1.02 (0.72, 1.45) 1.04 (0.72, 1.50)
Visit by weekend vs weekday
 Weekend: Sat/Sun 0.95 (0.67, 1.34) 0.99 (0.69, 1.42)
 Weekday: Mon-Fri Reference Reference
Provider type
 Physician Reference Reference
 Nurse practitioner 2.98 (2.43, 3.66) 3.27 (2.66, 4.01)
 Other 1.13 (0.59, 2.17) 0.99 (0.53, 1.85)
Open in a new tab
a

Boldface indicates statistically significant results.

Discussion

The development of EHR-based clinical decision support tools represents a unique opportunity to provide clinical guidance to a geographically widely distributed network of providers and to change provider behavior by promoting systematic implementation and documentation of emerging recommendations and practices. This study evaluated the effectiveness of an EHR-based clinical decision support tool, coupled with in-person training, in facilitating adoption and systematic documentation of 2 youth concussion diagnosis and management strategies—the vestibular oculomotor examination and provision of RTL/RTP guidance—by a large health care network’s PCPs. Changing provider behavior around these practices is important in that it aligns with contemporary strategies of health care quality and process improvement including the use of structured screening and diagnostic assessments, the systematic and consistent documentation of care across a broad health care network, and the conversion of best practice management guidelines into clinical practice.

This effort was motivated by increased numbers of youth seeking concussion care from PCPs and the demonstrated need for additional PCP training and support.6,10,12,13 Results highlight that in the 2 years leading up to the intervention, neither management strategy was being implemented consistently or systematically. On implementation of the intervention, however, performance of the vestibular oculomotor examination and documentation of discussion of RTL/RTP guidelines increased dramatically. Furthermore, for the vast majority of visits, the SmartSet template was used to document the oculomotor examination. Since only approximately 50% of the PCPs attended the training, this suggests that PCPs found value in the SmartSet template itself and that it had utility in guiding and/or structuring clinical practice and documentation.

Systematic reviews indicate that clinical decision support tools have successfully improved health care processes, but there have been fewer assessments of their impact on clinical, economic, and efficiency outcomes.29–31 Tools integrated into EHRs have improved clinical management, including screening and diagnosis, of other pediatric conditions in the primary care setting20–23,32,33 as well as concussion in the emergency department setting.34–36 Our study is the first to demonstrate improved provider-specific behaviors when caring for pediatric concussion patients in the primary care setting. This aligns with the “5 rights” of clinical decision support, by providing: (1) the right information (evidence-based guidelines for concussion assessment and management), (2) to the right person (PCPs), (3) in the right intervention format (SmartSet coupled with training), (4) through the right channel (EHR), (5) at the right time in the workflow (when caring for concussion patients).37

Variation existed across patient-, visit-, and provider-level variables. Performance of the vestibular oculomotor examination and documentation of RTL/RTP was less common in visits with patients younger than 5 years, with male and Hispanic patients, during follow-up visits and when another injury was present. However, for all subgroups, when the examination was performed, it was done almost exclusively through the SmartSet template, suggesting that template use does encourage the provider to implement and document best practice for concussion management. The high frequency of performance of the examination and provision of guidelines on the first visit compared to follow-up visits further highlights the value of this systematic clinical decision support in helping the provider make the initial diagnosis. Administration of the vestibular oculomotor examination and provision of RTL/RTP guidelines was also 3 times more common among nurse practitioners than physicians. There is a much larger cohort of physicians in the network; it is possible that the intervention was able to more thoroughly penetrate through the nurse practitioner group versus the physician group. Collectively, these observations suggest a continued need to demonstrate value and encourage consistent use of the template for all concussion visits regardless of patient characteristics or visit type.

The intervention was less effective in changing PCP behavior when caring for children younger than 5 years. Children in this age group are not typically engaged in organized sports or formal schooling and thus there may be less of a perceived need to provide RTL/RTP guidelines. Additionally, concussion in the very young is challenging to assess as these patients cannot always describe their symptoms.38 The SmartSet template and an objective test like the vestibular oculomotor examination might be particularly advantageous in this age group given that recognition and verbalization of symptoms is challenging. In a quality improvement survey that we conducted among our network’s PCPs in May 2015, respondents revealed concerns about the reliability and applicability of the SmartSet template questions for this age group (K. Arbogast, unpublished data). Age-specific improvements to the template and strategies to increase provider usage in this age group are needed, and may include instructions related to managing a patient’s return to developmentally appropriate activities, such as preschool and playground activities.

Historically, integrating new evidence into provider behavior can take many years.39 However, in this study we demonstrated a rather robust and quick uptake of several recommended management strategies—in particular the vestibular oculomotor examination, which was a new concept to many of the network’s PCPs. We hypothesize several key characteristics that drove this success. First, this effort was driven by a clinical need. We observed that PCPs in our network did not already have an existing or systematic approach to concussion assessment and management; thus, the intervention appears to have filled a specific void. Second, our results corroborate the importance of engaging the clinical team in the design of clinical decision support tools.16 Prior to development, we broadly surveyed CHOP PCPs to understand their needs and actively engaged several PCPs in the design of the tool. Third, we believe that it was important to couple the clinical decision support tool with hands-on training, which is consistent with adult learning theory where experiential learning constitutes an essential component of acquiring new skills.40

There are several limitations of this study. First, there were other secular changes associated with concussion, including legislation and evolving societal attitudes, which occurred during the study period and may influence the findings. For example, a proportion of the observed increase in RTL/RTP documentation among Pennsylvania practices may be attributable to concussion legislation, effective July 2012, that required an injured athlete to receive medical clearance before returning to sport.9 However, a similar law that went into effect in New Jersey in December 2010 did not seem to appreciably increase documentation in the months following in those CHOP practices located in New Jersey.41 Furthermore, the observed increase in performance of the vestibular oculomotor examination would not be expected to be influenced by these laws. Although these and other societal changes may influence the number of concussion visits seen by a given provider, they are likely not the primary driver of the provider behaviors on the visit-level observed herein. The independent contribution of each factor is difficult to determine in this retrospective analysis. Second, we studied providers who are part of a single large health care network. Our network uses a single EHR system, which promotes standardized practice similar to other large health care systems. As such, the behaviors of the providers and characteristics of the patients represented in this study likely reflect those in other network that utilize a single EHR to standardize practice. Third, we chose 2 specific key indicators of best practice concussion care; these are not the only aspects of concussion management but were chosen for this analysis because they represented emerging concepts as well as existing guidelines that were important to reinforce. Finally, this analysis relied on EHR documentation of both key practices. We did not evaluate the quality of the examination or the details of the RTL/RTP guideline discussion as these data are only available via direct observation. Additionally, an examination may have been performed despite lack of documentation in the EHR. However, clear and systematic EHR documentation of all aspects of clinical management is best practice and thus represents an appropriate target to strive for in this setting. It is important to note that our analysis evaluated providers’ performance and documentation of the vestibular oculomotor examination in the context of the SmartSet template for head injuries that were ultimately diagnosed as a concussion. We did not evaluate whether initiation of the SmartSet template increased the accuracy of the concussion diagnosis, the likelihood that a diagnosis would be made, or treatment decisions and clinical outcomes. These are complex questions to answer. Although concussion severity likely influences provider behavior, it is not a concept for which there is a consensus on how to measure. Similarly, clinical outcomes are not always clearly defined; for example, assessment of time to recovery may be hampered by lingering symptoms and lack of clear recovery indicators in existing electronic health record data. These important questions should be the focus of future work.

In summary, our findings suggest that an intervention utilizing an electronic clinical decision support tool, coupled with in-person training, can effectively and quickly change provider behaviors leading to the early adoption of existing and emerging guidelines for concussion management and consistent and systematic documentation of those practices. Primary care represents an ideal target for such interventions given the increasing rates of concussion related visits to these settings. Such tools can increase PCP proficiency in concussion assessment, accelerate uptake of emerging knowledge, and promote practice consistency throughout an entire health care network. Future efforts linking these provider behaviors to improvements in clinical outcomes are necessary advancements in this line of research.

Acknowledgments

The authors would like to thank Julia Vanni of the Children’s Hospital of Philadelphia for her role in project coordination, Lara DePadilla of the CDC for her review of the manuscript and Marianne Chilutti of the Children’s Hospital of Philadelphia for her data management.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by an intergovernmental personnel act agreement between the US Department of Health and Human Services (DHHS), Centers for Disease Control and Prevention (CDC), and the Children’s Hospital of Philadelphia.

Appendix

Concussion-Related ICD-9-CM Codes.

ICD-9-CM Code Description
800.02 Fracture of vault of skull with brief (less than 1 hour) loss of consciousness
800.09 Closed fracture of vault of skull without mention of intracranial injury, with concussion, unspecified
800.52 Open fracture of vault of skull without mention of intracranial injury, with brief (less than 1 hour) loss of consciousness
800.59 Open fracture of vault of skull without mention of intracranial injury, with concussion, unspecified
801.02 Closed fracture of base of skull without mention of intracranial injury, with brief (less than 1 hour) loss of consciousness
801.09 Closed fracture of base of skull without mention of intra cranial injury, with concussion, unspecified
801.39 Closed fracture of base of skull with concussion, unspecified
801.52 Open fracture of base of skull without mention of intracranial injury with brief (less than 1 hour) loss of consciousness
801.56 Open fracture of base of skull without mention of intracranial injury with loss of consciousness of unspecified duration
801.59 Open fracture of base of skull without mention of intracranial injury, with concussion, unspecified
803.02 Other and unqualified skull fractures with brief (less than 1 hour) loss of consciousness
803.09 Other and unqualified skull fractures with concussion, unspecified
803.52 Other open skull fracture without mention of intracranial injury with brief (less than 1 hour) loss of consciousness
803.59 Other open skull fracture without mention of intracranial injury, with concussion, unspecified
804.02 Closed fractures involving skull or face with other bones, without mention of intracranial injury, with brief (less than 1 hour) loss of consciousness
804.09 Closed fractures involving skull of face with other bones, without mention of intracranial injury, with concussion, unspecified
804.52 Open fractures involving skull or face with other bones, without mention of intracranial injury, with brief (less than 1 hour) loss of consciousness
850 Concussion
850.0 Concussion with no loss of consciousness
850.1 Concussion with brief loss of consciousness
850.10 Concussion with brief loss of consciousness
850.11 Concussion, with loss of consciousness of 30 minutes or less
850.5 Concussion with loss of consciousness of unspecified duration
850.9 Concussion, unspecified
Open in a new tab

Abbreviation: ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification.

Footnotes

The findings and conclusion of this research are those of the authors and do not necessarily represent the official views of the US Department of Health and Human Services (DHHS) and the Centers for Disease Control and Prevention (CDC). The inclusion of individuals, programs, or organizations in this article does not constitute endorsement by the US federal government, DHHS, or CDC. Other than the coauthors, the sponsor had no role in the (1) study design; (2) the collection, analysis, and interpretation of data; (3) the writing of the report; and (4) the decision to submit the manuscript for publication.

Author Contributions

KBA and AEC conceptualized and designed the study, coordinated and supervised data management, drafted the initial manuscript, interpreted the data, and approved the final manuscript as submitted. CLM, MRZ, JMB, JHK, and MJB contributed to study conceptualization and design, reviewed and critically revised the manuscript, interpreted the data, and approved the final manuscript as submitted. KBM contributed towards the development of the analysis plan, managed, analyzed, and interpreted the data, drafted the statistical sections, reviewed and critically revised the manuscript, and approved the final manuscript as submitted. RSK contributed to data management, reviewed and critically revised the manuscript, and approved the final manuscript as submitted.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  • 1.Zonfrillo MR, Kim KH, Arbogast KB. Emergency department visits and head computed tomography utilization for concussion patients from 2006 to 2011. Acad Emerg Med. 2015;22:872–877. doi: 10.1111/acem.12696. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Pfister T, Pfister K, Hagel B, Ghali WA, Ronksley PE. The incidence of concussion in youth sports: a systematic review and meta-analysis. Br J Sports Med. 2016;50:292–297. doi: 10.1136/bjsports-2015-094978. [DOI] [PubMed] [Google Scholar]
  • 3.Guerriero RM, Proctor MR, Mannix R, Meehan WP. Epidemiology, trends, assessment and management of sport-related concussion in United States high schools. Curr Opin Pediatr. 2012;24:696–701. doi: 10.1097/MOP.0b013e3283595175. [DOI] [PubMed] [Google Scholar]
  • 4.Coronado VG, Haileyesus T, Cheng TA, et al. Trends in sports- and recreation-related traumatic brain injuries treated in US emergency departments: the National Electronic Injury Surveillance System–All Injury Program (NEISS-AIP) 2001–2012. J Head Trauma Rehabil. 2015;30:185–197. doi: 10.1097/HTR.0000000000000156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Kinnaman KA, Mannix RC, Comstock RD, Meehan WP. Management of pediatric patients with concussion by emergency medicine physicians. Pediatr Emerg Care. 2014;30:458–461. doi: 10.1097/PEC.0000000000000161. [DOI] [PubMed] [Google Scholar]
  • 6.Arbogast KB, Curry AE, Pfeiffer MR, et al. Point of healthcare entry for youth concussion within a large pediatric care network. JAMA Pediatr. 2016;170:e160294. doi: 10.1001/jamapediatrics.2016.0294. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Taylor AM, Nigrovic LE, Saillant ML, et al. Trends in ambulatory care for children with concussion and minor head injury from Eastern Massachusetts between 2007 and 2013. J Pediatr. 2015;167:738–744. doi: 10.1016/j.jpeds.2015.05.036. [DOI] [PubMed] [Google Scholar]
  • 8.Reisner A, Popoli DM, Burns TG, et al. The central role of community-practicing pediatricians in contemporary concussion care: a case study of children’s healthcare of Atlanta’s Concussion Program. Clin Pediatr (Phila) 2015;54:1031–1037. doi: 10.1177/0009922815573468. [DOI] [PubMed] [Google Scholar]
  • 9.National Conference of State Legislatures. [Accessed April 26, 2017];Traumatic Brain Injury Legislation. 2015 Nov 18; http://www.ncsl.org/research/health/traumatic-brain-injury-legislation.aspx.
  • 10.Zonfrillo MR, Master CL, Grady MF, Winston FK, Callahan JM, Arbogast KB. Pediatric providers’ self-reported knowledge, practices, and attitudes about concussion. Pediatrics. 2012;130:1120–1125. doi: 10.1542/peds.2012-1431. [DOI] [PubMed] [Google Scholar]
  • 11.Pleacher MD, Dexter WW. Concussion management by primary care providers. Br J Sports Med. 2006;40:e2. doi: 10.1136/bjsm.2005.019067. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Kaye AJ, Gallagher R, Callahan JM, Nance ML. Mild traumatic brain injury in the pediatric population: the role of the pediatrician in routine follow-up. J Trauma. 2010;68:1396–1400. doi: 10.1097/TA.0b013e3181cf7d1b. [DOI] [PubMed] [Google Scholar]
  • 13.Carl RL, Kinsella SB. Pediatricians’ knowledge of current sports concussion legislation and guidelines and comfort with sports concussion management: a cross-sectional study. Clin Pediatr (Phila) 2014;53:689–697. doi: 10.1177/0009922814526979. [DOI] [PubMed] [Google Scholar]
  • 14.Arbogast KB, McGinley AD, Master CL, Grady MF, Robinson RL, Zonfrillo MR. Cognitive rest and school-based recommendations following pediatric concussion: the need for primary care support tools. Clin Pediatr (Phila) 2013;52(5):397–402. doi: 10.1177/0009922813478160. [DOI] [PubMed] [Google Scholar]
  • 15.Master CL, Grady MF. Office-based management of pediatric and adolescent concussion. Pediatr Ann. 2012;41(9):1–6. doi: 10.3928/00904481-20120827-08. [DOI] [PubMed] [Google Scholar]
  • 16.Bauer NS, Carroll AE, Downs SM. Understanding the acceptability of a computer decision support system in pediatric primary care. J Am Med Inform Assoc. 2014;21:146–153. doi: 10.1136/amiajnl-2013-001851. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Krist AH. Electronic health record innovations for healthier patients and happier doctors. J Am Board Fam Med. 2015;28:299–302. doi: 10.3122/jabfm.2015.03.150097. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Zazove P, Plegue MA, Uhlmann WR, Ruffin MT. Prompting primary care providers about increased patient risk as a result of family history: does it work? J Am Board Fam Med. 2015;28:334–342. doi: 10.3122/jabfm.2015.03.140149. [DOI] [PubMed] [Google Scholar]
  • 19.Ballard DW, Rauchwerger AS, Reed ME, et al. Kaiser Permanente CREST Network. Emergency physicians’ knowledge and attitudes of clinical decision support in the electronic health record: a survey-based study. Acad Emerg Med. 2013;20:352–360. doi: 10.1111/acem.12109. [DOI] [PubMed] [Google Scholar]
  • 20.Carroll AE, Bauer NS, Dugan TM, Anand V, Saha C, Downs SM. Use of a computerized decision aid for ADHD diagnosis: a randomized controlled trial. Pediatrics. 2013;132:e623–e629. doi: 10.1542/peds.2013-0933. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Carroll AE, Biondich P, Anand V, Dugan TM, Downs SM. A randomized controlled trial of screening for maternal depression with a clinical decision support system. J Am Med Inform Assoc. 2013;20:311–316. doi: 10.1136/amiajnl-2011-000682. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Carroll AE, Anand V, Dugan TM, Sheley ME, Xu SZ, Downs SM. Increased physician diagnosis of asthma with the child health improvement through computer automation decision support system. Pediatr Allergy Immunol Pulmonol. 2012;25:168–171. doi: 10.1089/ped.2012.0143. [DOI] [Google Scholar]
  • 23.Anand V, Carroll AE, Downs SM. Automated primary care screening in pediatric waiting rooms. Pediatrics. 2012;129:e1275–e1281. doi: 10.1542/peds.2011-2875. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Mucha A, Collins MW, Elbin RJ, et al. A brief Vestibular/Ocular Motor Screening (VOMS) assessment to evaluate concussions: preliminary findings. Am J Sports Med. 2014;42:2479–2486. doi: 10.1177/0363546514543775. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Kontos AP, Sufrinko A, Elbin RJ, Puskar A, Collins MW. Reliability and associated risk factors for performance on the Vestibular/Ocular Motor Screening (VOMS) tool in healthy collegiate athletes. Am J Sports Med. 2016;44:1400–1406. doi: 10.1177/0363546516632754. [DOI] [PubMed] [Google Scholar]
  • 26.Master CL, Scheiman M, Gallaway M, et al. Vision diagnoses are common after concussion in adolescents. Clin Pediatr (Phila) 2016;55:260–267. doi: 10.1177/0009922815594367. [DOI] [PubMed] [Google Scholar]
  • 27.Corwin DJ, Wiebe DJ, Zonfrillo MR, et al. Vestibular deficits following youth concussion. J Pediatr. 2015;166:1221–1225. doi: 10.1016/j.jpeds.2015.01.039. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.McCrory P, Meeuwisse W, Johnston K, et al. Consensus statement on Concussion in Sport 3rd International Conference on Concussion in Sport held in Zurich, November 2008 [corrected] [published erratum appears in Clin J Sports Med 2010;20(4):332] Clin J Sport Med. 2009;19:185–200. doi: 10.1097/JSM.0b013e3181a501db. [DOI] [PubMed] [Google Scholar]
  • 29.Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157:29–43. doi: 10.7326/0003-4819-157-1-201207030-00450. [DOI] [PubMed] [Google Scholar]
  • 30.Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care. 2008;16:79–91. doi: 10.14236/jhi.v16i2.679. [DOI] [PubMed] [Google Scholar]
  • 31.Cresswell K, Majeed A, Bates DW, Sheikh A. Computerised decision support systems for healthcare professionals: an interpretative review. Inform Prim Care. 2012;20:115–128. doi: 10.14236/jhi.v20i2.32. [DOI] [PubMed] [Google Scholar]
  • 32.Rattay KT, Ramakrishnan M, Atkinson A, Gilson M, Drayton V. Use of an electronic medical record system to support primary care recommendations to prevent, identify, and manage childhood obesity. Pediatrics. 2009;123(suppl 2):S100–S107. doi: 10.1542/peds.2008-1755J. [DOI] [PubMed] [Google Scholar]
  • 33.Zuckerman AE. The role of health information technology in quality improvement in pediatrics. Pediatr Clin North Am. 2009;56:965–973. doi: 10.1016/j.pcl.2009.05.018. [DOI] [PubMed] [Google Scholar]
  • 34.Ip IK, Raja AS, Gupta A, Andruchow J, Sodickson A, Khorasani R. Impact of clinical decision support on head computed tomography use in patients with mild traumatic brain injury in the ED. 2015;33:320–325. doi: 10.1016/j.ajem.2014.11.005. [DOI] [PubMed] [Google Scholar]
  • 35.Melnick ER, Lopez K, Hess EP, et al. Back to the bedside: developing a bedside aid for concussion and brain injury decisions in the emergency department. EGEMS (Wash DC) 2015;3:1136. doi: 10.13063/2327-9214.1136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Schwartz L, Heard-Garris N, Grell R, Chatain V, Meyssami A, Atabaki SM. A concussion diagnostic tool and management plan implemented as clinical decision support into the electronic health record. Paper presented at: American Federation for Medical Research Eastern Regional Meeting; April 9, 2014; Washington, DC. [Google Scholar]
  • 37.Approaching Clinical Decision Support in Medication Management: Section 2—Overview of CDS Five Rights. AHRQ National Resource Center; Health Information Technology: Best Practices Transforming Quality, Safety, and Efficiency; [Accessed April 26, 2017]. https://healthit.ahrq.gov/ahrq-funded-projects/clinical-decision-support-initiative/chapter-1-approaching-clinical-decision/section-2-overview-cds-five-rights. [Google Scholar]
  • 38.Gioia GA, Collins M, Isquith PK. Improving identification and diagnosis of mild traumatic brain injury with evidence: psychometric support for the acute concussion evaluation. J Head Trauma Rehabil. 2008;23:230–242. doi: 10.1097/01.HTR.0000327255.38881.ca. [DOI] [PubMed] [Google Scholar]
  • 39.Kitson AL. The need for systems change: reflections on knowledge translation and organizational change. J Adv Nurs. 2009;65:217–228. doi: 10.1111/j.1365-2648.2008.04864.x. [DOI] [PubMed] [Google Scholar]
  • 40.Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med. 2013;88:1407–1410. doi: 10.1097/ACM.0b013e3182a368bd. [DOI] [PubMed] [Google Scholar]
  • 41.An Act Concerning the Health of Student-Athletes and Supplementing P.L. 1984 c.203 (C.45:9–37.35 et Seq.) and Chapter 40 of Title 18A of the New Jersey Statutes. [Accessed April 26, 2017];New Jersey Legislature. Chapter 94 http://www.njleg.state.nj.us/2010/Bills/PL10/94_.HTM. [Google Scholar]

RESOURCES