Validity of Research Based on Publicly Obtained Data in Sports Medicine: A Quantitative Assessment of Concussions in the National Football League

Abstract

Context:

Numerous researchers have leveraged publicly available internet sources to publish clinical research concerning incidence and recovery from injuries in National Football League (NFL) players.

Objective:

This study aims to (1) provide a comprehensive systematic review of all publicly obtained data studies (PODS) regarding concussions in NFL athletes and (2) quantify the percentage of injuries identified by these studies in comparison with published concussion data from the NFL injury database.

Study Selection:

A systematic review was conducted in accordance with PRISMA guidelines to identify all published studies utilizing publicly obtained data regarding concussions in NFL athletes.

Study Design:

Systematic review.

Level of Evidence:

Level 4.

Data Extraction:

Manuscript details, factors related to the athletes of interest (eg, study period, positions included), and results (eg, concussion rate, number of total concussions, return-to-play data) were extracted independently by 2 authors. Results were compared with incident concussions reported from 2015 to 2019 by each medical staff member to the NFL database linked to the League’s electronic health record (EHR).

Results:

A total of 20 concussion-focused manuscripts based on PODS were identified from 2014 to 2020. PODS captured between 20% and 90% of concussions (mean, 70%) reported by medical staff to the injury database. PODS reported that 55% of concussions occurred on offensive plays, 45% on defensive plays and <1% occurred during special teams plays, compared with 44%, 37%, and 18%, respectively, as indicated by published data from the NFL injury database. When analyzed by position groups, running backs and quarterbacks comprised the most over-represented positions concussed in PODS, while offensive linemen, defensive backs, and linebackers comprised the most under-represented positions.

Conclusion:

PODS captured approximately 70% of concussions reported by NFL medical staff to the NFL injury database. There is heterogeneity in the degree to which PODS were able to identify concussions, with a bias toward concussions among players at higher profile positions.

Research in the field of sports medicine is dedicated, in part, to defining the epidemiology of athletic injuries. Well-designed epidemiologic studies often rely on injury databases based on quality-reviewed reports from trained medical staff. ⁷ These studies can have broad implication, both identifying opportunities for improved player safety and monitoring, and evaluating results of injury reduction and other player health initiatives, such as rule modifications, equipment development, and playing surface modifications.^13,27 Because of the importance placed on injury data, numerous athletic organizations, such as the National Collegiate Athletic Association (NCAA), ¹³ National Football League (NFL),^7,25,33 National Basketball Association (NBA), ²⁷ Major League Baseball (MLB), ³ and Major League Soccer (MLS), ¹ maintain injury surveillance databases to provide improved care for their athletes.

It is also common for studies to review publicly available internet-based injury data (eg, Entertainment Sports Production Network [ESPN], fantasy football websites, team injury reports) in lieu of curated official league-based databases to perform sport- or league-specific sports medicine research.^14,22 Use of publicly sourced data provides convenient benefits due to their ease of access, which may entice researchers to pursue this line of inquiry; however, this research is exempt from institutional review boards (IRBs) or organizational oversight necessary to protect private health data due to the nature by which these injuries are publicized by media sources. ³⁹ Large databases from publicly obtained sources can be constructed easily to generate numerous publications from a single search.^6,35,41 Athletic performance data, as assessed by media and sports statistics after an injury or treatment, can also be obtained from these internet sources, ¹⁵ which may not be possible from some deidentified injury data sets. As a result of these potential benefits, there has been a near-exponential increase in publication rates of publicly obtained data studies (PODS). ¹⁴ The concerns regarding the utility and validity of PODS are therefore more important as these studies may possess significant methodological flaws due to inaccurate or incomplete results stemming from their dependence on nonverifiable sources and incomplete data.^14,22 As such, inaccurate or uncertain data collection potentially associated with PODS can lead to aberrant conclusions, which may have significant implications for the athletes, team or league administrators, and medical providers. ²²

The potential concerns regarding PODS are perhaps more pronounced in the setting of concussion research. Concussions and their long-term sequalae have garnered significant medical and media attention, particularly in NFL athletes.^21,28,37 Unfortunately, the totality of these concussions may be inadvertently missed by media sources or portrayed inaccurately by unaffiliated observers who have no access to league-specific data that is linked to an electronic health record (EHR). Furthermore, public disclosure of concussions may be intentionally or inadvertently vague (ie, “head injury”), leading to ambiguity in regard to their inclusion in a PODS. In addition, while the diagnosis of a concussion requires removal from participation until appropriate clinical recovery, this may not always equate to a missed competition, as game play is based on competition schedules and other factors. Being confined to this metric increases the probability that concussion-specific PODS may miss a proportion of injuries and/or only capture the subset of more severe injuries requiring an athlete to miss ≥1 competitive events. An additional concern is that PODS may also focus on higher-profile players whose relevance to the team’s success may increase the likelihood of being reported compared with lower-profile players. ¹⁴ Therefore, the purpose of this study is 2-fold: (1) to provide a comprehensive systematic review of all PODS specifically regarding concussions in NFL athletes and (2) to determine the validity of these studies through comparison with concussion data collated by the EHR-based injury database of the NFL. Our hypothesis was that NFL concussion research based solely on publicly obtained data will capture these injuries in NFL athletes at a highly variable rate.

Methods

Systematic Review

A systematic review was performed after best practices as established by the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement. ²⁹ A medical librarian at the primary institution searched all records published in standard medical literature and ‘gray’ literature (eg, reports, working papers, government document white papers, and other evaluations produced by academic centers, government agencies, private organizations, and commercial entities) for records pertaining to NFL athletes and concussion. Search strategies used a combination of keywords and controlled vocabulary in Embase.com 1947 onward, Ovid Medline 1946 onward, Scopus 1823 onward, Cochrane Central Register of Controlled Trials (CENTRAL), Cumulated Index to Nursing and Allied Health Literature (CINAHL Plus) 1937 onwards, APA PsycINFO 1800 onward, and ClinicalTrials.gov 1997 onward. All search strategies were completed on November 24, 2020, with no added limits. A total of 1300 results were found; 730 duplicate records were deleted after using the deduplication processes described in Bramer et al, ² resulting in a total of 570 unique citations included in the project library. Fully reproducible search strategies for each database can be found in the Appendix 1 (available in the online version of this article).

After completion of the structured query, 2 authors independently evaluated the manuscript title and abstract for each publication to determine eligibility for inclusion. Qualifying studies were original, English-language, full-text manuscripts regarding the epidemiology of concussions in current NFL athletes. Studies were excluded if the target population included retired NFL athletes or non-NFL athletes, data were not obtained from publicly available internet sources, or the study possessed an outcome of interest (ie, biomechanical evaluations, helmet performance, mechanism of diagnosis, etc) other than concussion incidence (overall, by position, and by game phase) and/or return to play after a concussion. After review of the abstracts, all eligible manuscripts were reviewed in full by both authors, with the reason for full-text exclusion recorded through an exclusion ladder (Appendix 2, available online). There were no cases of disagreement between the 2 authors that necessitated arbitration by a senior author. Inter-rater reliability was measured with Cohen’s kappa statistic (κ).

For each study meeting the inclusion criteria, manuscript details, including publication date, journal, authors, the journal’s impact factor (IF) if listed by the journal’s website, and whether the journal was ‘Open Access’ and available without subscription, were recorded. Study quality was determined using methodological index for nonrandomized studies (MINORS) score. ³⁸ Factors related to the athletes of interest (ie, study period, positions included, miscellaneous inclusion criteria), and results (ie, concussion rate, number of total concussions, return-to-play data) were extracted by 2 authors and verified by a third author to ensure accuracy. The 2 reviewers performing data extraction were blinded to the NFL EHR injury data during this process. No manuscript meeting the inclusion criteria provided any specific detail as to how concussions were diagnosed. Rather, each manuscript provided only a quantitative assessment based solely on the internet sources they referenced.

Comparison With the NFL Injury Surveillance System Database

Data from PODS were compared with published NFL concussion data extracted from the NFL Injury database from 2015 to 2019. This database is quality-checked and audited, with accuracy verified; all data used for comparison in this study are available within a published peer-reviewed manuscript describing the epidemiology of concussions in this timeframe. ²⁷ Concussion diagnoses were made by head team physicians based on established NFL concussion protocols. ⁸ Individual player injury information is entered into the NFL electronic medical record (EMR) system, which is a clinical EHR-based platform. The data contained in the EMR system may be used to link game injury rates to game statistics, game-day conditions, outcome information, and other performance-related data and information that is maintained outside of the EMR system. ⁷ Injury data are collected from and curated by an independent third-party data science company (IQVIA; contracted with the NFL and NFL Players Association) through a series of quality control checks and validated through audits comparing surveillance data with external reports. NFL EHR data may be stratified by either roster position or position at time of injury (ie, a safety injured during a kickoff can either be classified as a defensive back [DB] or special teams player). In cases where the outcome of interest was phase of play (ie, offense, defense, or special teams), data were stratified by position at the time of injury. Each PODS was thoroughly scrutinized for player position at the time of the concussion. Unfortunately, it became evident that no PODS offered any methodology for classifying player position and they often misclassified a significant portion of injuries during comparison by game phase. Therefore, player position was classified as roster position for all other analyses.

For each manuscript deemed eligible for inclusion in this study, the published NFL concussion data were matched to the time of season (ie, preseason, regular season, postseason) and player position(s) of interest. Given the availability of published NFL concussion data from 2015 to 2019, the average number of concussions per year was calculated as a comparison for PODS concussion data. In cases where inclusion and exclusion criteria were highly targeted,^10,15,16,34 PODS were included in qualitative analysis only.

A capture rate was generated by dividing the total number of concussions detected in a PODS by the total number of concussions reported in the NFL injury data with replicated study criteria. Studies were also stratified by game phase (ie, offense, defense, or special teams) and by position group. A relative representation ratio (R³) was calculated by dividing the relative percentage of concussions occurring at a position in a PODS by the relative percentage of concussions occurring at the same position in the dataset collated through the NFL database.

Human Subject Protection

This study did not require human subject protection through an IRB or the NFL Player Scientific and Medical Research Protocol (MRAP) as it was based on previously published data that were IRB- and MRAP-approved before publication. ²⁷

Results

Systematic Review

A total of 1300 results were identified through a structured query of the literature, with 730 duplicate records deleted through the dededuplication process; 570 unique results were ultimately reviewed (Figure 1). A total of 20 studies met criteria for inclusion after full-text review.^{4,5,9-12,15-20,30-32,34,36,39,40,42} During screening of full texts, reviewers agreed on the included/exclusion of 92.9% (κ = 0.85, almost perfect agreement) of texts.

Figure 1.

PRISMA flow diagram for the systematic review. PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses.

Study Characteristics

MINORS scores were tabulated for all included studies (Table 1). Average MINORS score was 9.8 for noncomparative studies and 15.9 for comparative studies. No studies identified for inclusion were published before 2014; 70% of studies were published after 2017 (Figure 2). Several internet-based public sources were referenced by the PODS,^{4,9-12,31,39,40} with PBS Frontline Concussion Watch being the most utilized public data source (8 of 20 studies, 40%) (Table 2). The 20 studies were published in 10 unique journals, 6 (60%) of which were considered ‘Open Access’ and 3 (30%) of which did not have an IF listed by the journal’s website (Table 3). Numerous studies overtly constructed inclusion criteria to reflect data discoverable through publicly available reports, which may be construed as allowing the research methods to drive the study question. For example, some studies excluded injuries during the final week of the regular season (Week 17),^{4,5,9,10,19,30,31} with the rationale that NFL teams are not required to publish an injury report if not eligible for the playoffs; thus, these injuries were pronounced as not discoverable.

Table 1.

MINORS scores for included studies

Author	A Clearly Stated Aim	Inclusion of Consecutive Patients	Prospective Collection of Data	Endpoint Appropriate to the Aim of the Study	Unbiased Assessment of the Study Endpoint	Follow-up Period Appropriate to the Aim of the Study	Loss of Follow-up Less than 5%	Prospective Calculation of the Study Size	An Adequate Control Group	Contemporary Group	Baseline Equivalence of Groups	Adequate Statistical Analysis	Total
Connolly 4	2	1	1	1	2	1	0	0	2	2	2	0	14
Dai 5	2	1	1	1	2	2	0	0	NA	NA	NA	NA	9
Haider 9	2	1	1	2	2	2	0	0	NA	NA	NA	NA	10
Hannah 10	2	1	1	2	2	2	0	0	NA	NA	NA	NA	10
Hanson 11	1	1	1	2	2	2	0	0	2	1	1	2	15
Heintz 12	1	1	1	2	2	1	0	0	NA	NA	NA	NA	8
Jildeh 15	2	1	1	2	2	2	0	0	2	2	2	2	18
Koschmann 16	1	1	0	2	2	1	0	0	NA	NA	NA	NA	7
Kraeutler 17	2	1	1	1	2	2	0	0	2	2	1	2	16
Kumar 18	1	2	1	1	2	1	0	0	2	2	0	2	14
Lawrence 19	2	1	2	2	2	2	0	0	NA	NA	NA	NA	11
Lawrence 20	2	1	2	2	2	2	0	0	NA	NA	NA	NA	11
Myer 30	2	1	1	2	2	2	0	2	2	2	0	1	17
Nathanson 31	2	1	1	2	2	2	0	0	NA	NA	NA	NA	10
Navarro 32	2	1	0	2	2	2	0	0	NA	NA	NA	NA	9
Reams 34	1	1	1	1	2	2	0	0	2	2	1	2	15
Sheth 36	2	2	1	2	2	2	0	0	NA	NA	NA	NA	11
Teramoto 39	2	1	1	2	2	2	0	0	NA	NA	NA	NA	10
Teramoto 40	2	1	1	2	2	2	0	0	NA	NA	NA	NA	11
Zuckerman 42	2	1	1	2	2	2	0	0	2	2	2	2	18

MINORS, Methodological index for nonrandomized studies; NA, not available.

Figure 2.

Cumulative studies utilizing publicly obtained data evaluating NFL concussions from 2012 to 2020. NFL, National Football League; POD, publicly obtained data studies.

Table 2.

Number of publications utilizing each specific internet source of concussion data

Source	No. of Publications
PBS Frontline Concussion Watch	8
NFL.com	7
NFL Injury Reports	7
Pro-football-reference.com	5
ESPN	3
wunderground.com	2
Teamranking.com	1
NFLweather.com	1
Argis.com	1
Player Profiles	1
Game Summaries	1
Kagle.com	1
NCAA.com	1
Spotra.com	1
Fantasydata.com	1
Foxsports.com	1

ESPN, Entertainment Sports Production Network.

Table 3.

Publications in which the included studies were published

Journal	Impact Factor	Open Access	No. of Publications
Orthopedic Journal of Sports Medicine	2.727	Yes	9/20 (45%)
Cureus	N/A	Yes	2/20 (10%)
The American Journal of Sports Medicine	6.202	No	2/20 (10%)
Journal of Health Economics	1.902	No	1/20 (5%)
Public Health	2.427	No	1/20 (5%)
International Journal of Sports Physiology and Performance	3.042	Yes	1/10 (5%)
Journal of Orthopaedic Sports Physical Therapy	4.751	Yes	1/20 (5%)
British Medical Journal Open Sports and Exercise Science	1.412	Yes	1/20 (5%)
Journal of Surgical Orthopaedic Advances	N/A	No	1/20 (5%)
Frontiers in Sports and Active Living	N/A	Yes	1/20 (5%)

N/A, not available.

Capture Rate

A total of 12 studies included concussion data that were amenable to comparison with the NFL EHR data.^{4,5,9,12,19,20,30,31,32,36,39,40} The PODS captured, on average, only 70.0% (±25.0%) of the concussions reported by the NFL injury data, with capture rates ranging from 20.1% to 90.2% (Table 4).

Table 4.

Capture rate of PODS when compared with the NFL database ^a

PODS	Data Period (Regular Season Unless Otherwise Stated)	PODS Concussions per Year	NFL Concussions per Year 29	Capture Rate
Connolly 4	2012-2015	142.3	166.8	85.3%
Dai 5	2012-2015	139.5	166.8	83.6%
Haider 9	2012-2015	141	166.8	84.5%
Heintz 12	2012-2015 b	160.8	253.8	63.3%
Lawrence 20	2012-2013	150.5	166.8	90.2%
Lawrence 19	2012-2013	148	166.8	88.7%
Myer 30	2012-2013	150	166.8	89.9%
Nathanson 31	2012-2013	146	166.8	87.5%
Navarro 32	2005-2016	33.5	166.8	20.1%
Sheth 36	2010-2019 b	42.5	172.8	24.6%
Teramoto 40	2012-2014 c	145.7	247.8	58.8%
Teramoto 39	2012-2015 d	161.3	253.8	63.5%
Total				70.0%

NFL EHR, National Football League electronic health record; PODS, publicly obtained data studies.

^aPosition- and phase-specific data.

^bRegular and postseason.

^cPre- and regular season.

^dAll (pre-, post- and regular season concussions).

The PODS included in this analysis recorded the players’ primary position, whereas the NFL injury data reported situational data at the time of injury, irrespective of the player’s primary position. When stratifying data by game phase, PODS indicated that 55% of concussions occurred during offensive plays, 45% occurred during defensive plays and <1% occurred during special teams play. In comparison, the NFL injury data reported that 44% of concussions occurred during offensive plays, 37% occurred during defensive plays, and 18% occurred during special teams play.

When stratified by position group (Table 5), DBs accounted for the greatest proportion of concussions in both PODS (25.5%) and the NFL injury data (28.7%), while specialists (kickers, punters, and long snappers) comprised the smallest proportion of injuries in both PODS (0%) and the NFL injury data (0.9%). Running backs (R³ = 1.46) and quarterbacks (R³ = 1.42) were identified at a relatively higher percentage by the PODS when compared with the NFL data. Alternatively, offensive linemen (R³ = 0.93), DBs (R³ = 0.88), and linebackers (R³ = 0.82) were identified at a relatively lower percentage by PODS when compared with NFL injury data (Table 6).

Table 5.

Number of injuries recorded at each position group in PODS

Position	Nathanson 31	Navarro 32	Teramoto 40	Zuckerman 42	Sheth 36	Total	Percentage
Defensive back	79	59	129	38	120	425	25%
Tight end	32	32	43	13	67	187	11%
Wide receiver	47	45	63	13	68	236	14%
Running back	29	36	43	10	43	161	10%
Quarterback	13	23	15	9	27	87	5%
Offensive line	41	42	63	21	86	253	15%
Linebacker	24	34	42	9	63	172	10%
Fullback	3	0	3	0	0	6	0%
Defensive line	24	21	34	14	36	129	8%
Specialist	0	0	0	0	0	0	0%
Other	0	15	2	0	0	17	1%
Total	292	307	437	127	510	1673	100%

PODS, publicly obtained data studies.

Table 6.

Relative proportion of injuries at each position in PODS compared with NFL EHR data

Distribution of Concussions
Position	PODS (n = 1673)	NFL EHR 27 (n = 790)	Relative Representation Ratio (R3)
Running back	9.6%	6.6%	1.46
Quarterback	5.2%	3.7%	1.42
Defensive line	7.7%	6.5%	1.19
Tight end	11.2%	10.0%	1.12
Wide receiver	14.1%	14.9%	0.94
Offensive line	15.1%	16.2%	0.93
Defensive back	25.4%	28.7%	0.88
Linebacker	10.3%	12.5%	0.82
Specialist	0.0%	0.9%	0.00

NFL EHR, National Football League electronic health record; PODS, publicly obtained data studies.

Discussion

Epidemiologic research in the field of orthopaedic sports medicine seeks to understand the injury burden and reduce the risk of injury during organized team activities or individual athletic endeavors. Elite levels of professional and collegiate athletics have curated this research by resource-intensive databases.^23,25,26 However, a novel method of sports medicine research has emerged over the past 2 decades whereby study authors utilize publicly obtained data from internet sources to generate retrospective database studies. Use of public sources for research is common in professional sports, especially those with extensive media coverage, elevated risk of significant injuries, and an engaged fan base. Recent literature, however, raises concerns that PODS may contain inherent methodological flaws that limit the utility and accuracy of the dataset and resulting conclusions of these studies.^14,22

A review of the medical literature finds that the frequency of PODS related to concussions in the NFL has increased dramatically in the last 10 years, with 90% of studies based on public data published since 2015. Moreover, these studies have appeared in a wide array of academic journals, including those devoted to sports medicine, ⁵ physical therapy, ³⁴ public health, ¹⁶ and economics. ¹¹ Of the 10 journals that published these studies, 6 were considered ‘Open Access,’ meaning their content is freely available to the general public without a subscription fee. The journals also varied widely in terms of their IF. Nevertheless, the journals in which these studies were published also varied widely in terms of relevance to their particular field of inquiry.

As demonstrated in previous literature with similar methodology, ¹⁴ the frequency with which these studies capture injury data is highly heterogenous. Though some studies demonstrated the ability to detect and report >90% of concussions in NFL athletes,^19,20,30 others failed to capture over half of all injuries.^32,36 To further demonstrate this point, 1 study spanning nearly a decade reported fewer concussions than studies spanning only 3 seasons,^4,5,9,36 which highlights the difficulty in discerning the quality and accuracy of each study. Unfortunately, such heterogeneity in capture rates prevents an understanding of the true incidence of concussions in this population as well as the change in injury rates over time.

We do note that 2 studies had a disproportionate impact on the overall capture rate, and exclusion of these studies does increase the average capture rate by approximately 10%.^32,36 Both studies span particularly long (ie, 10 years) study periods, which may preclude accurate identification of many injuries occurring in the population of interest. First, retrospectively identifying injury data produced 10 years before initiation of a research study likely presents a significant challenge. Moreover, if injury data are obtained, the resources and time required to thoroughly review the large volume of data (ie, >5000 NFL injury reports over 10 years) may be prohibitive from a practical standpoint. Finally, the long study periods may have allowed the researchers to identify a sufficient number of injuries to draw conclusions, despite suboptimal search techniques or methods of accurately identifying all eligible injuries. Therefore, shorter study duration may allow PODS to identify a greater proportion of total injuries. By comparison, PODS with higher capture rates are limited to studies that span a relatively shorter period of time and focus on injuries occurring solely during the regular season. Although this methodology allows for accurate identification of injuries from official reports generated by a league or association, such narrow inclusion may neglect injuries occurring to lesser-known players who were released from the team before the start of the regular season.

The current study demonstrates that PODS may inadvertently detect injuries in high-profile positions (ie, running backs and quarterbacks), while missing or failing to include similar injuries at other positions (ie, linebacker, offensive line, and DB). One explanation for this trend is the sources utilized for the collection of data in that researchers utilize some internet sources intimately associated with fantasy football,^{4,5,11,17,32,39,40,42} where quarterbacks and running backs are of greatest interest to a website’s audience.

Finally, PODS demonstrate significant discrepancies with respect to injuries occurring during special teams play. Though NFL injury data indicate that 18% of injuries occur during special teams play, PODS categorized injuries solely by roster positions obtained from a player’s online profile rather than the position he played when he was injured. As a result, these erroneous data have significant ramifications for player safety and meaningful research. Many NFL athletes play primarily or exclusively on special teams regardless of their primary position listed in an online player profile. Devoid of contextual data, injuries to these players overstate the risk of injury during offensive or defensive plays and understate the risk of injury during special teams plays. This is especially relevant since the kickoff is the single play in professional football responsible for the most concussions per play. ²⁷ Given the majority of player safety improvements in the last decade have occurred secondary to rule changes during special teams play, ²⁷ PODS represent a significant hindrance to improving the medical care of NFL athletes.

Thus, conclusions regarding the limitations of research based solely on publicly available data should be viewed only as a critique of study design and not study designer(s). However, the sports medicine community should be aware of the inherent flaws and pitfalls associated with this method of research and interpret future PODS from this perspective. Researchers are encouraged to utilize well-curated, prospective databases maintained by the organization of interest that is linked to an EHR rather than pursuing less consistent, though more accessible, internet sources from which inconsistent injury data can be obtained. Mechanisms exists whereby researchers may collaborate with these professional sports leagues to obtain data from their EHR databases. ²³ Such a process confers some degree of transparency through external review to protect the athletes this research is intended to assist. In this way, sport-specific clinical research of consistent quality may be produced that will benefit the sports medicine community while protecting the athletes and leagues from erroneous conclusions derived from flawed interpretation of incomplete, though readily available, injury data. Release of injury data - and publication of research based on this data - must be approved not only by league officials but also by the Players Association of each of the 5 major American professional sports leagues, which alone can introduce significant bias. All rules governing these leagues’ activities are collectively bargained. Therefore, the decision to release injury data to prospective researchers must be mutually agreed upon by both the league and Players Association irrespective of the perceived quality or relevance of the proposed research.

Limitations

As in virtually all sports medicine research, this study has major limitations. First, the study premise assumes accurate, complete reporting into the NFL EHR by NFL player medical staff who directly treat the players; this medical record system feeds the NFL injury database which represents the “gold standard” in this study. Although the NFL database is based on a clinical EHR and is curated through collaboration with an organization experienced in managing large volumes of data,^7,24 potential for error exists based on the inherent challenges of consistently diagnosing and reporting injuries across 32 separate clinical teams and data reporters caring for hundreds of NFL players each year. In addition, the idiosyncrasies of a subset of POD studies did not allow for the calculation of a mean Capture Rate. For example, 1 study included position players with 1 “touch” (handling the ball) during regular season play from 2007 to 2010. ³⁴ However, all studies in our systematic review demonstrated <100% capture rates, which is similar to capture rates reported in previous studies. ¹⁴ Finally, the concerns related to PODS likely only affect professional and elite collegiate athletic research. However, studies based on these cohorts are frequent and drive innovation across all levels of competition, thereby necessitating that such studies provide accurate, reliable data based on medical reports.

In conclusion, the frequency of clinical research studies leveraging publicly obtained injury data in NFL players has increased rapidly over the past decade. This systematic review found that the PODS captured on average 70% - and at worst, only 20% - of the concussions identified by the NFL injury database. There is significant heterogeneity in the degree to which PODS correctly identified concussions in this patient cohort with a propensity to identify those concussions occurring in players at higher profile positions. Limitations of sports medicine research that relies on publicly obtained sources must be clearly stated, and results should be interpreted with an understanding of these limitations and inherent biases.