Concussion management plans’ compliance with NCAA requirements: Preliminary evidence suggesting possible improvement

Precis:

This study examined the extent to which concussion management plans at National Collegiate Athletic Association (NCAA) member schools were in line with NCAA Concussion Policy and best practice recommendations in absence of any process to ensure compliance. Most schools’ concussion management plans were in compliance with 3 (60%) or 4 (25.6%) of the NCAA’s 4 required components. Annual athlete education and acknowledgement was the requirement least often included, representing an area for improvement. Further, schools tended to more often include best practices that were more medically-oriented (e.g., including baseline examination), compared to best practices that were less medical in nature (e.g., avoiding flagrant head hits).

Introduction

In recent years, concussion has been recognized as an injury with significant public health implications. In response, states and sports leagues have put in place rules and regulations to ensure that individuals who suffer a concussion receive appropriate medical attention. To date, the majority of concussion policy focuses on secondary and tertiary prevention mechanisms, with fewer attempts at primary prevention of concussion.

In 2010, the National Collegiate Athletic Association (NCAA) implemented its Concussion Policy and Legislation.¹ This was the NCAA’s first set of concussion-specific rules for its member schools, and it required each member school to have in place a concussion management plan.² Each plan is required to have, at minimum, the following four components: 1. Annual athlete education, along with athlete acknowledgement of their responsibility to report concussion symptoms; 2. Athletes who are suspected of sustaining a concussion be removed from play and evaluated by a clinician; 3. Athletes diagnosed with a concussion are not allowed to return to play (practice or game) for at least the remainder of the calendar day; and 4. Athletes diagnosed with a concussion are required to obtain clearance from a physician or physician’s designee prior to returning to play.³ The NCAA has more recently issued additional guidance on best practices for concussion management,⁴ but its 2010 policy represents the only set of rules by which all member schools are required to abide.

Initial investigations of institutional compliance with the NCAA concussion policy have found mixed results among member schools.⁵ In a 2014 survey-based study, Baugh and Kroshus et al. found that while a vast majority of schools had concussion management protocols, fewer schools implemented those protocols in full compliance with NCAA guidance.⁶ In particular, around one quarter of responding schools did not provide athletes with annual concussion education. Kelly and colleagues found that clinical practices in concussion management at NCAA Division I schools were generally in line with NCAA guidance, but could be improved in some cases through a more robust use of a multi-modal concussion examination.⁷ Buckley and colleagues had similar findings in a sample of NCAA Division II and III institutions.⁸ Additionally, previous studies have found that even when components of the concussion policy are implemented, there is significant variability in how schools chose to abide by the guidelines. For example, a study of concussion education in NCAA collegiate ice hockey players found that while all teams received education in accordance with the NCAA mandate, there was substantial variation in the modality and robustness of concussion education provided to athletes.⁹ Notably, studies to date have focused primarily on the implementation of institutional concussion management plans—that is, how these plans are put into action—rather than examining the plans themselves.

Potentially in response to mixed findings regarding concussion management in face of existing rules, the NCAA recently implemented a procedure through which a subset of concussion management plans are evaluated to ensure that they are in compliance.¹⁰ Beginning in 2015, concussion management plans from schools within the NCAA Division I “Power 5” conferences (Atlantic Coast Conference, Big Ten Conference, Big-12 Conference, Pac-12 Conference, and Southeastern Conference)—arguably the schools with the most competitive and highest profile sports programs—were evaluated for compliance. During the first year of review, all of the Power 5 concussion management plans were found by the committee to be in compliance with NCAA rules. However, prior to the enactment of this new policy, there was no mechanism to ensure that schools that had concussion management plans were abiding by NCAA rules by including the required components. Further, for the vast majority of NCAA schools, there continues to be no process for ensuring compliance of schools’ concussion management plans with NCAA policy.

This study aimed to understand whether, prior to the implementation of an evaluative mechanism, institutions’ concussion management plans included the components required by the NCAA Concussion Policy. Additionally, it examined whether compliance varied by NCAA Division of competition. Furthermore, this study evaluated whether institutional concussion management plans went “above and beyond” the minimum requirements and included best practices in concussion management as outlined in the NCAA Sports Medicine Handbook.¹¹

Methods

This study was part of a larger project assessing concussion management in NCAA-affiliated institutions. Using a distribution service provided by the NCAA Sports Science Institute, an email with a description of the research and a request to complete an online survey was sent to compliance administrators, coaches, and sports medicine clinicians (team physicians and certified athletic trainers [ATs]). In the survey, only clinicians and compliance administrators were asked to upload their school’s concussion management plan. The request was distributed to all 1066 NCAA member-institutions. The online survey was hosted on the Qualtrics survey platform, and was tailored to the individual’s position in the athletic department. The study was approved by the Institutional Review Boards at [Redacted for Review]. Participants provided informed consent before completing the survey. Surveys were collected in September and October of 2013. The survey requested the individual upload the institution’s concussion management protocol. Two follow-up emails were sent at three-week intervals after initial contact.

Measures

In addition to uploading their institution’s concussion management plan, respondents were asked to indicate the division of competition in which the majority of teams at their school compete.

Concussion Management Plan Coding

Concussion management plan content was coded using a positivist qualitative paradigm and structural coding methods.¹² A manual was developed to code a comprehensive list of content areas (i.e. concussion education, diagnosis, and management) based on a priori categories and inductive codes created based on an initial review of the concussion management plans, the NCAA Concussion Policy and best practice guidance from the NCAA Sports Medicine Handbook.¹³ Once the concussion management plans were de-identified and the coding manual was complete, two independent coders from the research team reviewed each concussion management plan on a line-by-line basis and determined whether the plans included each of the categories created. High inter-rater reliability was achieved and all discrepancies were resolved by a third member of the research team.

Statistical Analysis

Descriptive analyses are provided. Binary outcomes were compared between divisions of competition using logistic regression; Division I was used as the referent category. Scores from variables representing each of the four requirements of the NCAA Concussion Policy were summed to create a compliance sum score (range 0–4). The compliance sum score was then treated as a linear variable and compared between divisions using simple linear regression. All analyses were completed in R version 3.13 using an a priori alpha of <0.05 to indicate statistical significance.

Results

Of the 1,066 total NCAA member-institutions, 137 concussion management plans were collected from 126 responding institutions. One concussion management plan was delivered in an un-readable format, thus 125 total plans were analyzed in the present study. Respondents who provided these plans were primarily clinicians (n=117). Fifty plans were from NCAA Division I schools, 30 from Division II schools, and 39 from Division III schools. Six plans were uploaded with no identifying information, therefore division of competition was unattainable.

Plans were analyzed for their compliance with NCAA requirements (Table 1). The majority of plans (64.8%) included athlete acknowledgement of concussion education and athlete acknowledgement of their responsibility to report concussion symptoms. However, only about 30% of plans specified that this process occur annually. The requirement that athletes be removed from play was included in nearly all plans, with slightly fewer specifying that they cannot return to play until the following calendar day (at the earliest). Nearly all plans included the need for medical evaluation for athletes with a possible concussion (97.6%) and that a health care provider needed to provide medical clearance prior to resuming participation in athletics (92.8%). There were no significant differences in plan compliance with individual NCAA requirements across division of competition.

Table 1.

Compliance of School Concussion Management Plans with Requirements of NCAA Concussion Policy and Legislation (n=125)

Requirement of NCAA Policy	n	%
Concussion Education
Athlete Acknowledgement	81	64.8%
Athlete Education	81	64.8%
Annual Athlete Education	38	30.4%
Removal from Play
Removal From Play	117	93.6%
Removed for Calendar Day	111	88.8%
Medical Evaluation
Evaluation of Student Athlete	122	97.6%
Evaluation by Athletic Trainer	110	88.0%
Evaluation by Physician	98	78.4%
Remove and Evaluate	116	92.8%
Return to Play Clearance by Physician or Designee
Clearance by medical professional	116	92.8%
Clearance by Athletic Trainer	63	50.4%
Clearance by Physician	112	89.6%
Compliance Score*
Score 0	1	0.8%
Score 1	5	4.0%
Score 2	12	9.6%
Score 3	75	60.0%
Score 4	32	25.6%

^*Compliance score represents the number of required components of the NCAA policy that the concussion management plan was in compliance with. For example, a plan that included removal from play, evaluation, and medical clearance prior to return (but not annual athlete education) would receive a Compliance Score of 3.

The compliance score reflects the number of required components (0–4) from the NCAA Concussion Policy and Legislation included in the respective school’s concussion management plan (Table 1). Nearly all plans included three or four of the components (60% and 25.6%, respectively). However, a non-negligible minority included only half of the components or fewer (14.4%). There was no significant difference in compliance score across schools from different divisions of competition (p=0.60).

The extent to which schools’ concussion management plans included aspects of best practices in concussion management from the 2013–2014 NCAA Sports Medicine Handbook (the time of the plan collection) is described in Table 2. There was wide variation in the extent to which components specified in these guidelines were included. For example, only 10% of schools’ concussion management plans in this sample included language regarding the need to avoid flagrant hits to the head or neck. On the other hand, over 90% of plans included details regarding a stepwise return to activity. In general, guidelines that were broader and more medical in nature (e.g., including a pre-participation baseline examination, monitoring the athlete following a concussion, evaluation by a health care provider, and graduated return to play) were included at a higher frequency than more specific and non-medical guidelines (e.g., avoiding flagrant head and neck activity).

Table 2.

Number of schools’ concussion management plans in including aspects of 2014 NCAA Best Practice Guidelines (n=125)

Category	n	%
Avoid Flagrant Head and Neck Activity	13	10.4%
Emergency Action Plan on File	23	18.4%
Annual Review of Emergency Action Plan	28	22.4%
Both Emergency Action Plan and Annual Review	19	15.2%
Healthcare Plans on File for Each NCAA Sport	15	12.0%
Unchallengeable Authority of Health Care Provider	49	39.2%
Coach Acknowledgement of Role in Plan and Receipt of Education	28	22.4%
Pre-Participation Baseline Examination	111	88.8%
Sports Concussion Assessment Tool (SCAT I, II or III)	28	22.4%
Any Balance Examination	53	42.4%
Standard Assessment of Concussion (SAC)	28	22.4%
Symptom Checklist	28	22.4%
ImPACT	91	72.8%
Any Neurocognitive Testing	91	72.8%
Monitoring Athlete Post Concussion	115	92.0%
Serial Monitoring	84	67.2%
Written Instructions Provided to Athlete	94	75.2%
Written Instructions Provided to Roommate/Guardian	76	60.8%
Evaluation by Healthcare Provider	122	97.6%
Evaluation by Physician, specifically	98	78.4%
Stepwise Return to Play	117	93.6%
Documentation of Any Incidents	43	34.4%

Discussion

This study examined the extent to which colleges’ concussion management plans were in compliance with NCAA requirements. The concussion management plans were gathered during a time when there was no procedure through which the NCAA or any other organization was evaluating schools’ concussion management plans for their compliance. The present study found that the majority of schools’ concussion management plans included most, but not all, of the required elements. The component that the largest number of schools failed to include was annual athlete education and athlete acknowledgement of their responsibility to report symptoms; in particular, only a minority of concussion management plans specified that concussion education was to occur on an annual basis. This study also examined the extent to which schools’ concussion management plans voluntarily included elements of the NCAA’s best practices. There was a wide variation in inclusion generally trending such that more medically oriented aspects were more frequently found within schools’ concussion management plans as compared to narrower or less medically oriented guidance. This may be due to clinicians’ roles in creating, amending, and implementing the concussion management plan. Athletic trainers and sports medicine physicians may feel more confident or better able to include and implement policies that are medically-oriented (e.g., including a baseline examination) rather than policies that might rely more heavily on the involvement of other stakeholders in the athletic environment (e.g., a coach’s involvement may be required to ensure that flagrant head and neck activity is reduced or eliminated). However, this approach minimizes the inclusion of primary prevention mechanisms in college concussion management policies.

Understanding schools’ compliance with the NCAA’s concussion policy is an initial step toward understanding the policy’s possible success. A comprehensive enforcement and penalty mechanism to ensure compliance would be an important next step toward policy implementation; this has been enacted in part in recent years, as the NCAA now mandates a subgroup of schools to have their concussion management plans evaluated for compliance. However, there is much less clarity on the broader implications of the NCAA concussion policy on improving athlete health, which is presumably the ultimate policy goal. It has been previously asserted by one of this study’s authors that states and sports leagues have a moral obligation to evaluate the effects of public health laws and policies, including those relating to brain injury.¹⁴(REF) In the present case, it is imperative to evaluate whether the NCAA concussion policy is, in fact, leading to better health outcomes for collegiate athletes. Additionally, it is important to understand whether there are any unintended consequences of the policy’s implementation. For example, a rule change specific to college football that aimed to reduce concussions was found to have the unforeseen consequence of increasing lower extremity injuries.¹⁵(REF) It is possible that the broader NCAA Concussion Policy and Legislation improves, worsens, or does not affect athlete health outcomes, but empirical evaluation is required in order to understand this effect.

This is not the first study to find inconsistent compliance with NCAA concussion policy.¹⁶ The specific finding of inconsistent education or requirement of athlete acknowledgement has been observed in other studies.¹⁷ It is encouraging that a large majority of schools’ plans require athletes to be removed from play and evaluated for a suspected concussion, are not allowed to return to play for the calendar day, and are required to obtain clearance from a physician or physician’s designee before returning to play. However, only one-quarter of schools required annual athlete education and acknowledgement, representing an area for improvement. Overall compliance with the requirements of concussion management policies was not significantly different across NCAA divisions of competition.

Although many schools required that athletes receive concussion education, fewer mandated that education occur annually. It is important to point out that having some education is likely better than having no education at all; however, annual reinforcement of this information is required by the NCAA. Requiring this education annually could both directly or indirectly improve concussion health outcomes. Directly, education could reinforce information about concussion signs and symptoms, what athletes should do if they suspect they have sustained a concussion, and what they can expect from the medical personnel if they do sustain a concussion. Indirectly, repeating concussion education annually may reinforce the importance of the injury, and help indicate to the athlete that the medical staff and/or the coach are promoting a culture of safety in which concussions are seen as a medical injury warranting time and attention. Critically, evidence has also indicated that even when education is implemented, there is substantial between-institution heterogeneity in content, delivery, and efficacy.¹⁸ Simply specifying that some type of education be delivered may miss an important opportunity for risk reduction. Research is increasingly indicating that many existing concussion education programs change only knowledge but not athlete behaviors or health outcomes.¹⁹ Requiring that institutions provide more detail in their concussion management plan about the content and delivery of the concussion education they will be providing to athletes is an important step. The NCAA is encouraged to outline a list of required content areas to be covered in the required concussion education and to suggest and make accessible evidence-based education materials that meet these guidelines.

Inclusion of NCAA best practice recommendations within the concussion management plan was inconsistent across schools. Many of the recommendations were infrequently incorporated into the concussion management plans (e.g., only 12% of concussion management plans mentioned a health care plan that includes equitable athlete access to health care professionals). It is important to note that the inclusion, or lack thereof, of these best practices within the concussion management plan is not necessarily indicative of whether the best practices are used at the respective schools. It may make more sense to include some best practice information in the concussion management plan (e.g., denoting the unchallengeable authority of medical personnel to make diagnosis, management and return to play decisions) than it does for others (e.g., promotion of the use of appropriate technique and avoidance of flagrant hits to the head). Additionally, it is possible that since the best practice guidance had only been available for a short period of time, the variable inclusion was more related to a timing effect than to a purposive decision by the authors of the schools’ concussion management plans of what elements to include or exclude from their plan.

Limitations:

Although individuals at all NCAA member schools were solicited to participate in this study, only 125 concussion management plans were included, representing only a fraction of total schools. Additionally, it may be the case that the respondents that chose to upload their concussion management plans were in some way systematically different than those who chose not to upload their plans. As such this study’s primary limitation is its generalizability. Although the NCAA Concussion Policy mandates that each school have a concussion management plan including the four described components, the ultimate goal is the implementation of these aspects of concussion management into clinical practice. To this extent, it is important to note the possibility that stakeholders in the sports medicine environment may implement the tenets of the NCAA policy without it being explicitly written in their plan. For example, it is possible that schools provide athletes with concussion education without writing the requirement into their plan. Finally, the concussion management plans were collected in 2013; as such, it is possible that compliance of schools’ concussion management plans has increased or decreased in the intervening years.

Conclusion and Future Directions

This study provides preliminary evidence that most schools are in partial compliance with the NCAA’s Concussion Policy and Legislation and that there is substantial heterogeneity in schools’ concussion management plans. The extent to which this variation in policy translates into variation in practice is an important area of future research. Furthermore, understanding the extent to which the NCAA’s best practice guidelines have been incorporated into concussion management plans now that they have been publicly available for a longer period of time is an area worthy of future investigation. In particular, the present study found that plans more frequently included medically-oriented best practices that focused on concussion management and less frequently included interventions that may prevent the initial injury. Understanding the extent to which this focus on secondary and tertiary prevention persists is of public health importance. This study provides baseline information that will assist in the evaluation of whether and to what extent the NCAA’s newly implemented process for reviewing concussion management plans has, in fact, improved plans’ compliance with NCAA requirements, and whether it has translated into improved clinical practice. According to some, the NCAA has an ethical duty to ensure that its policies are resulting in their intended health outcomes.²⁰ Thus, if it is found that this review mechanism is effective in improving adherence to the concussion policy, we would hope that the NCAA consider expanding it to schools across all divisions of competition to ensure an equitable distribution of health benefits.