Concussion Management Plan Compliance: A Study of NCAA Power 5 Conference Schools

Abstract

Background:

In response to concerns over concussions and repeated head impacts that occur during sports, the National Collegiate Athletic Association (NCAA) mandated that all member institutions enact a concussion management plan (CMP). Although institutional and health care provider self-reports have been investigated, compliance with NCAA protocol recommendations has not been examined.

Purpose:

To examine the CMPs from the 65 institutions within the NCAA Power 5 conferences for compliance with the NCAA 2015 concussion guidelines.

Study Design:

Descriptive epidemiology study.

Methods:

Each institution’s publicly available CMP was obtained in 2015, reviewed, and coded for compliance with each of the required 47 components. Overall compliance rate, item-level, category-level, and institution-level compliance was assessed. Independent predictors of compliance, including each institution’s athletic training staff size, academic performance, and athletic performance, were examined with quasi-binomial regression.

Results:

CMPs varied substantially in length and level of detail. The overall compliance rate for all components across all institutions was 94.3% (2880/3055). Twelve components achieved 100% (65/65) compliance, and the lowest levels of compliance were clustered in “return to learn.” There were 22 institutions that achieved a 100% compliance rate; the lowest institutional compliance was 59.6%. There were no significant associations between the independent predictors and institutional compliance.

Conclusion:

Overall compliance with NCAA concussion management requirements was high, but there remains room for improvement. The lowest level of compliance was clustered in the return-to-learn section. There were limited details provided in the reducing head trauma component. Items with lower compliance (reducing head trauma, return to learn) tended to be outside the core competencies of the medical staff, indicating an area for improvement. Encouragingly, many institutions and specific components demonstrated full compliance.

In recent years, there has been growing concern regarding the health-related consequences of concussions and repeated head impacts sustained during sports. In response, all 50 states, the District of Columbia, and multiple sports organizations have enacted policies aimed at improving athlete safety.⁴¹ At the collegiate level, the National Collegiate Athletic Association (NCAA), created in 1910 as the governing body of intercollegiate athletics, is charged, in part, with making the game of football safer.⁴⁵ A century later, sport safety remains an ongoing challenge; however, the focus has now shifted from football fatalities to concussion and head trauma concerns across all sports. Indeed, concussions occur in all NCAA-sponsored sports, and later-life neuropathologies have been identified in former football, soccer, ice hockey, and baseball players.^29,49 The topic of sports-related concussion first appeared in the NCAA Medical Handbook in 1933, in which it was stated that the seriousness of concussions was often overlooked. The first official protocols were recommended in 1994.^8,40 Although changes occurred gradually over the next 16 years, NCAA concussion regulation has rapidly evolved since 2010.³⁴

On April 29, 2010, the NCAA enacted its Concussion Policy and Legislation, which became effective in August 2010 (NCAA Bylaw: 3.2.4.17) and stated that all member institutions must have a concussion management plan (CMP). This plan was to include the following components: (A) an annual process that ensures student-athletes are educated about concussion signs and symptoms, acknowledge receipt of information, and have a responsibility to report symptoms to a medical staff member; (B) student-athletes with suspected concussions must be removed from the current practice or game and evaluated by a medical staff member with experience in the evaluation and management of concussions; (C) student-athletes diagnosed with a concussion are precluded from same day return to play (RTP); and (D) student-athletes must be cleared by a physician or physician’s designee prior to RTP after a concussion.³⁴ Subsequent years saw minor modifications to the concussion guidelines as well as the publication of “best practices” recommendations. Bylaw 3.2.4.17 was expanded (Bylaw: 3.2.4.17.1) in January 2016 to require a member institution’s CMP to include, by August 2016, procedures for (A) compliance with 3.2.4.17; (B) preparticipation baseline testing for each student-athlete; (C) reducing exposure to head injuries; (D) education about concussion, including a policy that addresses return to learn (RTL); (E) proper and appropriate concussion management, consistent with known best practices, made available to any student-athlete who has suffered a concussion; (F) annual review of the process of identifying, removing from participation, and assessing a student-athlete for a possible concussion; and (G) certification of compliance signed by the institution’s athletics director.³³ A set of guidelines, referred to as the NCAA Concussion Safety Protocol Checklist (NCAA-CSPC), was distributed to all 65 member institutions of the autonomous 5 conferences collectively referred to as the “Power 5”: the Atlantic Coast Conference, Big Ten Conference, Big-12 Conference, Pac-12 Conference, and Southeastern Conference, to ensure their CMPs were complaint with Bylaw 3.2.4.17.1.^33,36 Each institution’s CMP was reviewed for compliance by the Concussion Safety Protocol Committee in 2015.^33,37

Since the 2010 Concussion Policy was enacted, there has been limited investigation of the evolving NCAA concussion management policies, with Baugh et al⁵ reporting high, but not total, self-reported compliance with the concussion policy among NCAA athletic department personnel. Therefore, the purpose of this study was to examine the compliance of CMPs of schools within the NCAA Power 5 conferences with the NCAA concussion guidelines as revised in 2015. Given that these CMPs had been previously reviewed for compliance by a committee within the NCAA, we hypothesized that compliance would be high. However, we expected that areas shown in previous studies to have been less well implemented would likely have lower overall compliance. Additionally, we examined the association between athletic training staff size, academic performance, athletic performance, and conference with compliance.

Methods

The Power 5 CMPs and NCAA-CSPC were obtained through the publicly available NCAA concussion safety website.³⁵ The NCAA-CSPC is divided into 8 sections with required components in each.³⁶ The administrative section, section 8, contains only submitting the plan to the NCAA which, by default, all 65 institutions performed; therefore, the administrative section was not scored in this analysis. The remaining 7 sections had a total of 47 components that were evaluated (Table 1).

TABLE 1.

Mandatory Components of the NCAA Concussion Safety Protocol Checklist^a

Component Section	No. of Components	Overall Compliance
1. Pre-Season Education	6	98.2% (383/390)
2. Pre-Participation Assessment	5	99.1% (322/325)
3. Recognition and Diagnosis of Concussion	8	97.3% (506/520)
4. Post-Concussion Management	9	92.5% (541/585)
5. Return to Play	7	98.9% (450/455)
6. Return to Learn	11	86.4% (618/715)
7. Reducing Exposure to Head Trauma	1	92.3% (60/65)
8. Administrative	2	Not analyzed

^aFor the purposes of this review, only sections 1 to 7 (N = 47) were analyzed. NCAA, National Collegiate Athletic Association.

Each component was scored as addressed (yes) or not addressed (no) in each institution’s CMP. In instances where a required component was not directly addressed (eg, “preferred to provide written care instructions to another responsible adult”), the component was scored as not addressed, since “preferred” does not meet the “required” standard. Conversely, multiple concussion management plans indicated that they were using comprehensive assessments (eg, the Sport Concussion Assessment Tool–3rd edition [SCAT-3]), and they were credited with performing each of the individual assessments (eg, balance testing) within the comprehensive assessment. Length of CMPs was determined by converting PDFs to Microsoft Word documents and taking the page and word counts. Each CMP was examined by a member of the research team to determine whether it included each of the evaluated components; a random 10% of the plans were coded by a second member of the research team and evaluated for interrater reliability, which was deemed sufficiently high (kappa: 0.976).

The athletic training staff size was obtained by reviewing the intercollegiate athletic department websites for each institution; only staff athletic trainers (ATs) who provided clinical athletic training services as of January 2016 were counted. Graduate assistants, interns, and research staff were not included in the count. The institution’s academic performance was categorized by the NCAA’s Graduation Success Rate (GSR) for the incoming 2008 recruitment class—the most recently available data.³⁸ The GSR reflects the percentage of first-time freshmen enrolled as student-athletes who graduate within 6 years regardless of transfer status.³⁸ The institution’s athletic performance was categorized by the Learfield Sports Directors’ Cup (Directors’ Cup), which awards points according to performance in NCAA championship events in 20 (10 male and 10 female) sports, with higher numbers reflecting better performance.²⁴

Data Analysis

The overall compliance rate of the 47 components across all 65 institutions was calculated based on the number of components directly addressed in the CMP divided by the total number of components. The mean institutional compliance rate was calculated as the number of components directly addressed by the institution’s CMP divided by the 47 possible components. Second, a compliance rate per NCAA-CSPC section (N = 7) and components (N = 47) was calculated by summing the number of items that the institution complied with and dividing it by the total possible items, either within the section or overall, respectively. Descriptive statistics are provided to illustrate the levels of compliance overall and within the NCAA-CSPC sections. Simple quasi-binomial regressions (log-link) were performed to investigate the institution’s overall compliance rate (0%-100%) with each of the 4 independent variables: athletic training staff size, academic performance, athletic performance, and conference. Post hoc it was decided to specifically examine compliance with the NCAA-CSPC “Return to Learn” section across these 4 independent variables, using the same methodology as described for overall compliance.

Results

The concussion management plans showed considerable diversity in length (mean ± SD, 14.2 ± 13.8 pages; range, 4-67 pages) and word count (mean ± SD, 4701.9 ± 5635.7 words; range, 1186-32,270 words). Among plans that identified their most recent update (n = 51), all but one had been updated in 2015.

Sections

Seven sections of the NCAA-CSPC were evaluated for compliance. The section with the highest compliance was “Pre-Participation Assessment” at 99.1% (322/325), while the lowest overall was “Return to Learn” at 86.4% (618/715) (Table 1).

Components

The overall component compliance was 94.3% (2880/3055). The majority of components had very high compliance (Tables 2-5).

TABLE 2.

Section 3: Recognition and Diagnosis of Concussion^a

Section: Recognition and Diagnosis of Concussion	Compliance: 506/520; 97.3%
Must be removed from practice or competition	65; 100%
Must be evaluated by ATC or team physician with concussion experience	65; 100%
Must be removed from play/practice for that calendar day if concussion is confirmed	65; 100%
Symptom assessment	65; 100%
Physical and neurological examination	63; 96.9%
Cognitive assessment	63; 96.9%
Balance assessment	65; 96.9%
Clinical assessment for cervical spine trauma, skull fracture, and intracranial bleed	55; 84.6%

^aATC, athletic trainer certified.

TABLE 3.

Section 4: Post-Concussion Management^a

Section: Post-Concussion Management	Compliance: 541/585; 92.5%
Emergency action plan including transportation for further medical care for
Glasgow coma scale <13	62; 95.3%
Prolonged loss of consciousness	61; 93.8%
Focal neurological deficit suggesting intracranial trauma	62; 95.3%
Repetitive emesis	61; 93.8%
Persistently diminished/worsening mental status or other neurological signs/symptoms	63; 96.9%
Spine injury	64; 98.4%
Mechanism for serial evaluation and monitoring following injury	64; 98.4%
Documentation of oral and/or written care to both student athlete and another responsible adult (may be roommate or parent)	50; 76.9%
Evaluation by a physician for SA with prolonged recovery in order to consider additional diagnosis and best management options	54; 83.1%

^aSA, student-athlete.

TABLE 4.

Section 5: Return to Play^a

Section: Return to Play	Compliance: 450/455; 98.9%
Final determination of RTP is from the team physician or medically qualified physician designee	65; 100%
SA has limited physical and cognitive activity until he/she has returned to baseline, then progresses with each step below without worsening or new symptoms:	65; 100%
Light aerobic exercise without resistance training	64; 98.4%
Sport specific exercise and activity without head impact	64; 98.4%
Noncontact practice with progressive resistance training	64; 98.4%
Unrestricted training	64; 98.4%
Return to competition	64; 98.4%

^aRTP, return to play; SA, student-athlete.

TABLE 5.

Section 6: Return to Learn

Section: Return to Learn	Compliance: 618/715; 86.4%
Identification of a point person within athletics who will navigate return to learn with the student-athlete	64; 98.4%
Identification of a multidisciplinary team that will navigate more complex cases of prolonged return to learn	62; 95.3%
Compliance with ADAAA	60; 92.3%
No classroom activity on same day as concussion	61; 93.8%
Remaining at home/dorm if SA cannot tolerate light cognitive activity	53; 81.5%
Gradual return to classroom/studying as tolerated	54; 83.1%
Re-evaluation by team physician if concussion symptoms worsen with academic challenges	50; 76.9%
Modification of schedule/academic accommodations for up to two weeks, as indicated, with help from the identified point person	48; 73.8%
Re-evaluation by team physician and members of the multi-disciplinary team, as appropriate, for SA with symptoms >2 weeks	53; 81.5%
Engaging campus resources for cases that cannot be managed through schedule modification/academic accommodations	56; 86.1%
Such campus resources must be consistent with ADAAA and include at least one of the following: Learning Specialists, Office of Disability Services, or ADAAA Office	57; 87.7%

^aADAAA, Americans with Disabilities Act and Amendments; SA, student-athlete.

There were 12 components that achieved 100% (65/65) compliance, primarily in the sections “Recognition and Diagnosis of Concussion” and “Return to Play.” Member institutions were frequently compliant within the “Pre-Season Education” section (98.2%; 383/390), including concussion education being provided to student-athletes (100%; 65/65), coaches (100%), team physicians (98.4%), ATs (98.5%), and the director of athletes (95.3%). Similarly, there was high compliance within the “Pre-Participation Assessment” section (99.1%; 322/325); almost all CMPs required a preparticipation assessment, including concussion history (100%; 65/65), symptom evaluation (100%), cognitive assessment (98.5%), balance assessment (100%), and the team physician determining either clearance to RTP or the need for additional testing (96.9%).

The lowest compliance items were clustered in “Return to Learn” and included “modification of schedule/academic accommodations for up to two weeks, as indicated, with help from the identified point-person” (73.8%; 48/65), “re-evaluation by team physician if concussion symptoms worsen with academic challenges” (76.9%; 50/65), “remaining at home/dorm if student-athlete cannot tolerate light cognitive activity” (81.5%, 53/65), and “re-evaluation by team physician and members of the multi-disciplinary team, as appropriate, for student-athletes with symptoms lasting longer than two weeks” (81.5%, 53/65). An additional area of low compliance was “documentation of oral and/or written care to both the student-athlete and another responsible adult” (76.9%; 50/65). All other components exceeded 83% compliance.

Institutions

The mean institutional compliance rate was 44.3 out of 47 (SD, 3.3; range, 28-47) with 22 institutions (33.8%) achieving a 100% (47/47) compliance rate. The lowest institutional compliance rate was 59.6% (28/47).

Predictors

The mean athletic training staff size was 12.1 ± 3.1 ATs (range, 7-20 ATs). The overall GSR for the 65 institutions was 85.2% ± 5.5% (range, 71%-98%). The overall Directors’ Cup score was 675.2 ± 299.6 (range, 91-1448). The overall percentage compliance by institution was not predicted by the independent variables athletic training staff size (t = 0.117, P = .907), conference (P value range, .225-.852), GSR (t = −1.41, P = .163), or Directors’ Cup score (t = −1.05, P = .300). The compliance within the “Return to Learn” category, at the institutional level, was not predicted by the independent variables athletic training staff size (t = 0.16, P = .870), conference (P value range, 0.068-0.437), GSR (t = −1.08, P = .284), or Directors’ Cup score (t = −1.696, P = .095).

Discussion

Our findings reveal high compliance with the required components of CMPs among Power 5 institutions. It is noteworthy, however, that despite the availability of a checklist and internal review of the policies, 5.7% of components were not adequately incorporated into the published CMPs. Similar to previous studies of self-reported practice patterns across a more diverse setting, these findings suggest that, while highly compliant overall, there is continued room for improvement within the Power 5 CMPs.⁵ Beyond the specific data findings, 2 themes emerged: (1) medical management of concussions demonstrated high compliance whereas administrative components had lower compliance, and (2) there were 2 general approaches presented in the CMP “Return to Play” section—procedural (eg, checklist or cutoff score driven) and clinically based (eg, clinician driven) progressions.

In the past 2 decades, there has been a substantial increase in the utilization of objective concussion assessment tools across diverse sports medicine populations. All CMPs analyzed in this study included a multifaceted preparticipation assessment with more than 98% of CMPs including concussion history, symptom evaluation, cognitive assessment, and balance testing. This represents an improvement on the 71.2% of Division I ATs who reported using multiple neurological examinations in 2011, albeit the specific examinations differed between studies.¹⁸ Across divisions in 2013, less than half of responding institutions self-reported utilizing a multifaceted battery including both balance and cognitive testing; however, Division I respondents reported using the most such tools (3.6) during concussion assessment.^6,10 In 2015, Kerr et al¹⁹ reported that only 55.2% of NCAA Division I member institutions’ preparticipation assessments were compliant with NCAA guidelines, although multiple respondents indicated they planned on incorporating preparticipation assessments in 2014; our results might reflect those adaptations. It is important to note these prior studies included many institutions outside the Power 5 conferences, which likely have fewer resources, thus limiting the direct comparability of our study to those conducted previously. After a suspected concussion, almost all CMPs in this study (≥96.9%) incorporated objective physical tests in their evaluation, as opposed to only 17% in 1999 and 79.2% in 2011.^17,18 Similarly, CMPs were highly compliant with medical clearance requirements and stepwise progressive RTP (98.9% overall). Thus, it appears that the medical management of sports-related concussion is highly compliant with NCAA best practices and consistent with consensus statement recommendations.²⁸

The sports medicine staff, typically composed of team physicians and ATs, is the cornerstone of concussion recognition, diagnosis, and management within intercollegiate athletics, and they typically possess considerable expertise and experience with concussions.^4,11,18 For ATs, the concussion-specific educational competency is related to identifying the signs, symptoms, interventions, and RTP criteria. There are, however, no educational competencies related to academic accommodations that would likely be associated with RTL protocols.³² In medical education, while the medical management of brain injury is extensively discussed, there are no medical education milestones directly or indirectly related to academic accommodations and/or RTL postconcussion.^1,2 A potential mismatch between the requirements of a CMP and current consensus might explain the lower compliance in the realm of RTL. Specifically, RTL protocols including “modification of schedule/academic accommodations” (73.8%), “gradual return to classroom/studying as tolerated” (83.1%), and “re-evaluation by team physician if concussion symptoms worsen with academic challenges” (76.9%) had the lowest compliance. Kerr et al¹⁹ reported that many (67.2%) NCAA Division I institutions had RTL programs in 2013, but only 4.8% included academic support personnel. Herein, most CMPs provided a multi-disciplinary team (95.4%), but there was limited discussion of the utilization of academic support services, neuropsychologists, or disability services for academic accommodations after a concussion. These results suggest there may be a need for increased attention to appropriate academic accommodations, within institutional guidelines, to assist with successful return to the classroom and academic environment. Indeed, Kerr et al¹⁹ have explicitly recommended that the NCAA work with member institutions to develop RTL programs.

The NCAA CMP requirements are broadly stated, thereby allowing each intercollegiate athletic department and medical staff flexibility in concussion management, given the individualistic presentation and resolution of each concussion. Two common approaches were noted within the CMPs, which we generally classified as (1) procedural or checklist driven or (2) clinically driven. Indeed, one protocol specified the exact times and tests that were to be performed after a suspected concussion as well as specific step-by-step protocols for RTP progression. In a similar manner, another institution presented sport-specific activities (eg, number of jogging laps during the initial phase of exercise) for their progressive RTP protocol. Conversely, clinician judgment–driven protocols were also presented wherein specific testing timelines were based on when the treating physician deemed they were warranted. Both approaches are consistent with the overall NCAA CMP requirements and, given the lack of evidence supporting or refuting determinants of recovery,^12,44 clinical judgment is certainly warranted so long as assessments are consistent with the test’s psychometric properties.¹⁹ Conversely, utilizing a strict objective guideline may provide sports medicine clinicians with support, offsetting pressure from coaches and student-athletes pushing for RTP.^21,31,43

Another area of high but incomplete CMP compliance was the section “Reducing Exposure to Head Trauma” (92.3% [60/65]). The CMP checklist acknowledged that reducing head trauma may be hard to quantify, and the majority of institutions stated adherence to the 2014 Inter-Association Consensus: Year Round Football Practice Contact Guidelines and Independent Medical Care Guidelines. Many CMPs made general comments about student-athlete centered or safety-first approaches; however, specific plans to reduce head trauma were not presented frequently. This is a particularly important consideration, as later life neuropathologies (eg, chronic traumatic encephalopathy) may be related to repetitive head impacts independent of concussion diagnosis; in addition, a number of studies suggest measurable changes in outcome (eg, neuroimaging or cognitive changes) after a season of sports participation regardless of whether a concussion is diagnosed.^{7,9,14,26,30,42,46,47} The quantification of head impacts could provide, at a minimum, general information about the frequency and magnitude of impacts in college football.¹⁵ Several potential strategies have been discussed that could reduce head trauma, including reduced number of contact practices, helmetless tackling,⁴⁸ and educational interventions.²⁰ Furthermore, although football is the primary focus in the head trauma reduction section, repeated head impacts frequently occur in many other NCAA-sponsored sports, including soccer, ice hockey, lacrosse, and wrestling, but few CMPs identified plans beyond football.^13,25,27 Thus, future iterations of NCAA CMPs should encourage reducing head impacts in all sports.

Although sample populations do not directly align, this study found higher rates of concussion education than that found in previous studies. For example, in 2015 Kerr et al¹⁹ indicated that 92% of their sample provided concussion education to student-athletes and 85.6% provided concussion education to coaches. In 2014, Baugh et al⁵ found that 71.5% of Division I athletes received annual concussion education. Although the vast majority of schools in this study indicated within their CMP that they provide concussion education to the relevant stakeholders, the nature of this education and the education’s effectiveness in improving concussion knowledge, attitudes, and norms is unknown.²² Previous research has indicated that even when concussion education is provided in accordance with NCAA policy, there is significant variation in both content and modality of delivery as well as questions regarding effectiveness.²² Additional research is needed to understand how the concussion education provision is operationalized at the institutional level.

Examination of variation in overall rates of compliance as well as rates of compliance within the “Return to Learning” section, in which there was the most variation, yielded no significant predictors among those evaluated. That is, the number of full-time staff ATs, the conference of competition, a measure of academic prowess, and a measure of athletic competition were not associated with the level of CMP compliance at these schools. In Division II and III institutions, a smaller staff was indicated as a limitation for incorporating multifaceted CMPs, thus it was somewhat surprising there was no relationship between staff size and CMP compliance in this study. However, this may be due to the fact that the staffs at NCAA Power 5 schools are larger than those at smaller Division II and III programs. This may additionally reinforce previous findings of heterogeneity in concussion management practices across NCAA divisions of competition.⁶ Alternatively, the small variation in compliance rates within the present sample may reduce our ability to detect statistically significant trends in CMP compliance related to staffing size.

In addition to the highly compliant medical management, there were multiple areas of improvement noted in the CMP review compared with previous self-reported responses. Specifically, herein 100% of CMPs indicated that the final determination for RTP was from the team physician or medically qualified physician designee, whereas in 2013 some respondents reported that either coaches (6.8%) or athletes (6.6%) could make the RTP decision.⁵ Current NCAA guidelines call for athletic health care providers to have “unchallengeable autonomous authority” for the medical management of student-athlete injuries, including concussions.³⁹ Furthermore, there was 100% compliance with the recommendation that student-athletes suspected of a concussion be removed from participation, evaluated by the team physician or AT, and, if a concussion diagnosis is confirmed, removed for at least the remainder of the calendar day.

It is important to note that this review focused solely on the written CMP, and the results should not be extrapolated to the actual clinical concussion management practices; the actual practice could be better or worse than is currently stated. Indeed, several lawsuits have challenged the clinical management of concussions among NCAA member institutions,^3,23 and previously high-profile apparent violations of concussion management guidelines have been reported.¹⁶ The review of CMPs can be seen as a step toward an overall enforcement by the NCAA to help protect the health and well-being of student-athletes. It is encouraging to see relatively high compliance across this subset of institutions; however, it is unclear the extent to which this compliance translates to the rest of the NCAA Division I institutions or to Division II or III, where this CMP review has not yet been implemented. To date, only this subset of the most competitive and well-resourced schools have to make their CMPs publicly available and have an NCAA-mandated review process for compliance. Moving toward a more uniform and equitable process across all levels of competition is an essential next step to ensuring compliance in concussion management for all schools and the athletes attending these schools.