Validity of Soccer Injury Data from the National Collegiate Athletic Association's Injury Surveillance System

Abstract

Context:

Few validation studies of sport injury-surveillance systems are available.

Objective:

To determine the validity of a Web-based system for surveillance of collegiate sport injuries, the Injury Surveillance System (ISS) of the National Collegiate Athletic Association's (NCAA).

Design:

Validation study comparing NCAA ISS data from 2 fall collegiate sports (men's and women's soccer) with other types of clinical records maintained by certified athletic trainers.

Setting:

A purposive sample of 15 NCAA colleges and universities that provided NCAA ISS data on both men's and women's soccer for at least 2 years during 2005–2007, stratified by playing division.

Patients or Other Participants:

A total of 737 men's and women's soccer athletes and 37 athletic trainers at these 15 institutions.

Main Outcome Measure(s):

The proportion of injuries captured by the NCAA ISS (capture rate) was estimated by comparing NCAA ISS data with the other clinical records on the same athletes maintained by the athletic trainers. We reviewed all athletic injury events resulting from participation in NCAA collegiate sports that resulted in 1 day or more of restricted activity in games or practices and necessitated medical care. A capture-recapture analysis estimated the proportion of injury events captured by the NCAA ISS. Agreement for key data fields was also measured.

Results:

We analyzed 664 injury events. The NCAA ISS captured 88.3% (95% confidence interval = 85.9%, 90.8%) of all time-lost medical-attention injury events. The proportion of injury events captured by the NCAA ISS was higher in Division I (93.8%) and Division II (89.6%) than in Division III (82.3%) schools. Agreement between the NCAA ISS data and the non–NCAA ISS data was good for the majority of data fields but low for date of full return and days lost from sport participation.

Conclusions:

The overall capture rate of the NCAA ISS was very good (88%) in men's and women's soccer for this period.

Key Points.

• Overall, the capture rate of the National Collegiate Athletic Association Injury Surveillance System was 88% for men's and women's soccer injuries during the study period.

• Thus, this injury-surveillance system is capable of providing reliable and valid injury statistics, at least for men's and women's soccer.

Surveillance consists of “ongoing and systematic collection, analysis and interpretation of data.”^1(p164) Accurate and timely surveillance of sports injuries is important for monitoring trends in sport injuries.¹ Surveillance data can also be used to guide and evaluate injury-prevention efforts.² Currently, several surveillance systems in the United States are used to examine sport injuries. Systems collecting information on severe sport injuries in the general population include the emergency department–based system operated by the U.S. Consumer Products Safety Commission, known as the National Electronic Injury Surveillance System–All Injury Program (NEISS-AIP),³ and the catastrophic injury registry operated by the National Center for Catastrophic Sports Injury Research.⁴ For less severe injuries, attention has typically been focused on specific settings (eg, high school or collegiate), and injury data are often collected by certified athletic trainers (ATs). Examples of surveillance systems using data from ATs include the High School Reporting Information Online (RIO; Nationwide Children's Hospital, Columbus, OH)⁵ and the Big Ten Conference Sports Injury Surveillance System database (B10-ISS).⁶

The National Collegiate Athletic Association (NCAA) Injury Surveillance System (ISS) is another setting-specific system that calls on ATs to prospectively collect data on the incidence of injury in NCAA collegiate sports.⁷ Detailed data on mechanism of injury, activity at time of injury, injury diagnosis, and the number of team exposures (games and practices) are collected. The NCAA ISS began operation in 1988 with pen-and-paper forms that were faxed or mailed to the NCAA and then entered by hand into a database.⁷ In the mid-1990s, electronic scanning of the forms was introduced. Conversion to a Web-based system was completed for all sports in 2004–2005. These data are used by NCAA committees to make decisions relevant to student–athlete welfare.⁷,⁸ The data are also used by sports medicine researchers around the world to identify and monitor important descriptive aspects of collegiate sports injuries, such as the increased incidence of anterior cruciate ligament injuries in female athletes.⁹,¹⁰ The NCAA ISS data have facilitated the implementation of measures designed to decrease the incidence of certain injuries, such as protective goggles to reduce eye injuries in women's lacrosse players.⁸,¹¹

All surveillance systems should be evaluated on a routine basis so that their performance can be assessed.²,¹²,¹³ The Centers for Disease Control and Prevention (CDC)¹² has published criteria for the evaluation of surveillance systems. An important element in evaluating a surveillance system is estimating the proportion of true cases detected by the surveillance system. The typical method for estimating this attribute is to match the cases detected by the surveillance system with another source (or sources) of injury data external to the surveillance system.¹³ If the external data source completely enumerates all true cases and contains no false-positives (ie, injuries are measured completely and without error), then it is referred to as a gold standard; however, it is very unusual to have access to a true gold standard in most evaluations of surveillance systems. More often, researchers use an external data source that is less than perfect in its detection of true cases, such as athletes' self-reports, coaches' reports, or clinical reports. In this situation, capture-recapture analysis of the data is appropriate.¹⁴

Despite the fact that all surveillance systems should be evaluated on a routine basis, few validation studies of sport injury surveillance systems have been conducted. The purpose of our study was to examine the validity of NCAA ISS data from 2 fall collegiate sports (men's and women's soccer) by comparing NCAA ISS data from a purposive sample of NCAA schools with the information recorded by other data collection systems (other software or paper records) on those same injuries.

METHODS

Capture-Recapture Analysis

We performed a validation study using record abstraction in a purposive sample of NCAA ISS colleges and universities. The validation study was limited to 15 schools reporting NCAA ISS data for men's and women's soccer. Additionally, schools were required to have contributed data to the NCAA ISS for at least 2 years to be included. A gold standard against which to validate the NCAA ISS data does not exist because both paper-based and other software-based systems probably occasionally undercount the true number of injuries. In the absence of a gold standard, capture-recapture methods were used.¹⁴ Capture-recapture analysis assumes some undercounting in both systems and estimates the true number of cases or injury events based on the completeness of each data source relative to the total number of injury events captured by both data sources combined.^15–17 Capture-recapture methods have been applied to cancer,¹⁸ infectious diseases,19 cardiovascular disease,20 and occupational injuries²¹,²²; however, they have not been not widely used in sport injury analyses. The analysis is described further in the “Statistical Analysis: Capture-Recapture Methods” section.

Study Design

In 2005–2006, 149 of an estimated 1075 NCAA colleges and universities voluntarily participated in the NCAA Web-based ISS by providing usable data for at least 1 sport. We performed a validation study using record abstraction in a purposive sample of NCAA ISS colleges and universities that provided data on both men's and women's soccer to the NCAA ISS for at least 2 years during the 2005–2006, 2006–2007, and 2007–2008 academic years. Participants included men's and women's soccer athletes and ATs at these NCAA ISS institutions.

We restricted the study to schools with at least 2 years' experience with the Web-based NCAA ISS at the time of enrollment in the validation study to ensure that schools were reasonably proficient in its use and that any discrepancies in the data due to learning effects would be minimized. In the interest of feasibility, the project was limited to 2 sports (men's soccer and women's soccer), and we included only schools that provided data to the NCAA ISS for both sports.

NCAA ISS Injury Definition

During this study, a reportable injury in the NCAA ISS was defined as one that occurred as a result of participation in an organized intercollegiate practice or contest, necessitated medical attention by ATs or physicians, and resulted in restriction of the student-athlete's participation for 1 or more days beyond the day of injury.⁷ Such injuries are hereafter referred to as time-lost medical-attention injuries.

Participants

All procedures for this study were approved by the Duke University Medical Center Institutional Review Board. We were also required to obtain approval from institutional review boards at 11 of the 15 participating schools. The 4 remaining schools did not have boards of their own; therefore, individual investigator agreements under our institution's federal-wide assurance were obtained from the school's ATs to ensure adequate oversight of human subject ethics and principles. It is important to note that this study involved abstraction from existing electronic or paper clinical records, and no athletes were directly interviewed as part of our data collection process. Nevertheless, we were required to obtain written informed consent specifically for this study from all participating athletes before any record abstraction could occur. This consent form was in addition to the NCAA form governing access by the NCAA to student–athlete injury data. All ATs completed training in human participants research ethics, and informed consent was obtained by the AT at the beginning of the fall season from current men's and women's soccer athletes. Injury data for athletes who had graduated were not abstracted because there was no ready means of obtaining consent from graduates. We also obtained consent from all the participating ATs so that they could complete a short questionnaire about their school's experience with the NCAA ISS.

Recruitment

School eligibility, recruitment, and participation are illustrated in the Figure. The NCAA provided a list of the 80 schools that entered data into the NCAA ISS for men's or women's soccer for the 2006–2007 season; of these, 45 schools entered data for both sports. In March 2007, the NCAA sent an e-mail informing the ATs who maintained NCAA ISS records for men's and women's soccer at those schools about this study. The e-mail stated that they would receive a call from a researcher in the next few weeks. Members of the research team then contacted these ATs by phone and recruited them into the study.

Figure.

National Collegiate Athletic Association (NCAA) Injury Surveillance System (ISS) validity study school eligibility, enrollment, and participation. Abbreviation: IRB, institutional review board.

All 45 schools were contacted. Of the 45 NCAA ISS schools that entered both men's and women's soccer data for 2006–2007 season, we determined that 15 (33%) were ineligible for the study because the NCAA ISS was their sole injury record-keeping system (ie, they had no other record system against which we could validate their NCAA ISS data). Reasons for ineligibility of the remaining 3 schools (7%) were not using the NCAA ISS in the coming year, not entering data for both sports, and using another data source that was insufficient for comparison. A total of 27 schools were therefore eligible for the study, and 21 were enrolled (78% school-level initial response rate). After initial recruitment, 6 schools withdrew, leaving 15 schools (56% school-level final response rate) and 37 ATs in this study. All ATs completed the AT questionnaire.

Non–NCAA ISS Data Sources

The other data sources maintained by the ATs were used to validate the NCAA ISS data. These included hard-copy AT injury-assessment forms and rehabilitation and progress notes, coaches' reports, notes from other clinicians (eg, physicians, physical therapists), and non–NCAA ISS electronic databases (eg, Sportsware Injury Tracking Software; Presagia Corporation, Montreal, QC, Canada). Seven schools used a non–NCAA ISS electronic database, and 8 schools used hard-copy records (in addition to their NCAA ISS reporting).

Injury Data Abstraction

From February 2008 to December 2008, 5 researchers (K.L.K., D.R.B., M.J.D., C.P.G., S.O.) traveled to each of the 15 study schools that agreed to participate in the study and abstracted data onsite from each school's non–NCAA ISS data source. All abstractors were ATs with prior work experience in the collegiate setting who participated in a half-day training session before data collection began.

Injury data reported to the NCAA ISS for the 2005–2006, 2006–2007, and 2007–2008 soccer seasons were compared with injury data from the ATs' paper files or some other (non–NCAA ISS) electronic injury-tracking database. Researchers reviewed injury data only for athletes who consented to participate in the study. All time-lost medical-attention sport injuries were matched by sport, athlete name or identification, and injury date. Close misspellings of the name or identification and date of injury within 1 week were considered a match. Data fields abstracted for each sport injury from the non–NCAA ISS data source included sport, date of injury, sport relatedness, time of season, event type, injury mechanism, activity before injury, whether the injury was incident or recurrent, whether the injury was chronic or acute, side of the body, whether the athlete had surgery, injured body part, type of injury, diagnosis, outcome, date of full return, and total days out due to injury. Detailed categories for all key variables are included in the Appendix. We made extensive efforts to verify that all the injury-events included in the study resulted in at least 1 day of lost time. This included verification against written (eg, AT injury-assessment forms, rehabilitation and progress notes, coach's report) or electronic sources (eg, injury-tracking software data) and personal conversations with the AT for each team. As an example, consider a scenario in which the non–NCAA ISS data source indicated a missed practice or game, but the actual date of return and the number of days lost were missing. In this case, it was clear that the athlete missed 1 day for this injury, and therefore the injury was included. If the non–NCAA ISS data source did not indicate a missed practice or game, we consulted the AT to determine whether this event was a time-lost event. The data fields for date of return and days lost were still considered missing for these injuries.

AT Questionnaire

We asked each AT to complete a short questionnaire to help us characterize the AT and the school in terms of their support for injury-surveillance activities. The background data collected included demographic data about the AT and his or her training, basic information about the institution (eg, AT staffing levels), AT knowledge and beliefs about injury surveillance, how and whether they used surveillance data, and experience using the NCAA ISS.

Statistical Analysis: Capture-Recapture Methods

Because neither data source (NCAA ISS or non–NCAA ISS data system) could serve as a gold standard that was without error, we used capture-recapture methods^15–17 to estimate the total number of injury events and the proportion of injury events captured by the NCAA ISS for men's and women's soccer participants during the study period. After matching injury events during data abstraction, we derived values for 3 cells for a 2 × 2 table comparing capture in the NCAA ISS with capture in the non–NCAA ISS data systems (Table 1). These 3 cell counts quantified injury events recorded in both systems (cell a), injury events recorded in the NCAA ISS but missing from the non–NCAA ISS source (cell b), and injury events in the non–NCAA ISS source but missing from the NCAA ISS (cell c). Based on the degree of overlap between the data systems, the hypothetical number of injury events missed or unobserved by either system was estimated (x = bc/a).¹⁵

Table 1.

Capture-Recapture Analysis Methods ^a ,¹⁵

		Captured by Non–Injury Surveillance System Data Source?
		Yes	No
Captured by Injury Surveillance System?	Yes	a	b
	No	c	x
Injury events missed by both systems (estimated) = bc/a = x
Total injury events (estimated) = a + b + c + x = N
Capture rate of injury surveillance system = (a + b)/(a + b + c + x) = (a + b)/N
Capture rate of both systems = a/(a + b + c + x) = a/N

^a Adapted from Hook EB and Regal RR, Capture-recapture methods in epidemiology: methods and limitations. Epidemiol Rev. 1995;17(2):247 (Table 1), by permission of Oxford University Press.

The overall proportion captured by the NCAA ISS was defined as the number of time-lost medical-attention injury events reported to the NCAA ISS divided by the estimated total number of such events: (a + b)/N, where N = a + b + c + x (Table 1). We also assessed whether the proportion captured by the NCAA ISS varied by 6 key variables: calendar year, sport (men's versus women's soccer), NCAA division, use of electronic database versus paper records for the non–NCAA ISS system, presence of an undergraduate athletic training program at the school, and whether data were entered into the NCAA ISS by a supervised athletic training student versus solely by the ATs. Chi-square tests were used to determine statistical differences between categories of these variables. All injuries, incident and recurrent, were included in capture-recapture analyses.

Statistical Analysis: Agreement for Injury Data Fields

In addition to the NCAA ISS capture rate, we estimated the percentage of agreement for key injury-surveillance data fields: date of injury, sport relatedness, time of season, event type, injury mechanism, activity before injury, whether the injury was incident or recurrent, whether the injury was chronic or acute, side of the body, whether the athlete had surgery, injured body part, type of injury, diagnosis, outcome, date of full return, and total days lost due to injury. Percentage of agreement between the data sources was assessed in 2 ways. First, the overall or effective percentage of agreement²³ was calculated. This was simply the proportion of values for each data field for which the NCAA ISS and non–NCAA ISS data sources were in agreement: that is, both sources recorded the same value. Second, the ϰ percentage of agreement was calculated. Kappa analysis has the advantage of accounting for agreement that occurs purely by chance²³,²⁴; however, ϰ has some limitations (see “Discussion”). Given the large number of possible category combinations for number of days lost, we created a severity variable (0, 1–7, 8–14, 15–30, or 31+ days lost) and calculated both effective agreement and ϰ. If the information for any data field was missing in either data source, we considered “missing” a valid category in both the effective agreement and ϰ analyses. This decision was relevant mainly to injury mechanism (n = 57 missing), activity before injury (n = 143 missing), outcome (n = 88 missing), date of full return (n = 254 missing), and total days lost due to injury (n = 248 missing).

Statistical Analysis: AT Questionnaire

Quantitative analyses of data from the AT questionnaire included descriptive frequencies. Narrative responses to questions were examined for patterns. The AT sport coverage and workload were quantified in 2 ways: using a sport-to-AT coverage index (# NCAA sports/# ATs, including staff and graduate assistants) and using the National Athletic Trainers' Association (NATA) unadjusted base Health Coverage Index (HCI).²⁵ The NATA's College/University Athletic Trainers' Committee developed the HCI as a means of accounting for the variability in injury rates and AT treatment among 41 collegiate sports. The HCI is composed of the estimated injury rate and treatment by sport and ranges from a minimum of 0.5 for bowling, men's golf, and rifle to a maximum of 4.0 for women's basketball and gymnastics and men's volleyball. The sum of all sport HCIs represents the overall potential AT sport workload accounting for variability by sport. A school sponsoring all 41 collegiate sports would have a maximum HCI sum of 86.2.

RESULTS

Characteristics of Schools and ATs

The final group of 15 schools included 6 Division I schools, 3 Division II schools, and 6 Division III schools. Undergraduate and graduate athletic training programs were offered by 8 and 2 schools, respectively. Schools offered a mean of 19 NCAA-sponsored sports (SD = 5.4 sports; range, 10 to 31 sports) and had a mean HCI of 42.6 (SD = 11.8; range, 25.3 to 70.2). The sport-to-AT coverage index ranged from 3 to 19 sports covered per AT, with a mean of 9 sports (SD = 4.7 sports) per AT.

Participating ATs had a mean of 3.1 (SD = 1.7; range, 0.5 to 7) years' experience using the Web-based NCAA ISS, and 49% (18/37) had participated in the paper version of the NCAA ISS. Schools entered NCAA ISS data for an average of 7 sports (SD = 5.1 sports; range, 1 to 19 sports). The top 3 reasons given for participating in the NCAA ISS were the ability to compare school incidence rates with division and national totals (24%, n = 9), providing data to the NCAA rule committees to make policy decisions (22%, n = 8), and providing data for school decisions on health and safety of the athlete (19%, n = 7). In 14 of the 15 schools, the responding AT supervised people (eg, staff, graduate assistant AT, athletic training student) who entered NCAA ISS data; in 7 of 14 schools, the person entering the data was a supervised athletic training student.

Roughly half of the 37 ATs surveyed were female (51%, n = 19). Most ATs were between 20 and 29 (51%, n = 19) or 30 and 39 (41%, n = 15) years of age, and 8% (n = 3) were 40 to 49 years of age. Participants reported a mean of 8.0 total years of experience as an AT (SD = 6.0 years; range, 0.5 to 26 years) and a mean of 4.4 years working for their current employer (SD = 3.6 years; range, 0.5 to 14 years) as a head AT (32%, n = 12), assistant or staff AT (51%, n = 19), graduate assistant AT (14%, n = 5), or program director (3%, n = 1). In addition to their athletic training certification, 76% (n = 28) had acquired a master's degree and 8% (n = 3) were also physical therapists. The majority reported attending the most recent NATA meeting (65%, n = 24).

Athletes

Of 824 men's and women's soccer athletes, 737 (89.4%) consented to provide access to their clinical records for the purposes of this study (Table 2). Athlete participation rates did not vary by either sport (χ₁ = 1.3, P = .25) or division (χ₂ = 3.5, P = .17).

Table 2.

Demographic Information for the Soccer Athletes at the 15 Colleges and universities Studied, 2005–2007

Demographic Information		Total No. of Athletes on Roster	No. of Athletes on Roster per School, Mean ± SD	Total No. of Athletes Who Consented	No. of Athletes Who Consented per School, Mean ± SD	Percentage of Athletes Who Consented
Division	I	316	26.3 ± 3.6	276	23.0 ± 4.8	87.3
	II	168	28.0 ± 2.1	149	24.8 ± 2.6	88.7
	III	340	28.3 ± 6.4	312	26.0 ± 5.8	91.8
Sex	Men	426	28.4 ± 4.0	376	25.1 ± 5.2	88.3
	Women	398	26.5 ± 5.3	361	24.1 ± 4.9	90.7
Total		824	27.5 ± 4.7	737	24.6 ± 5.0	89.4

Injury Events

We abstracted 712 injury events at 15 schools. For a variety of reasons, 48 events did not generate usable data (Table 3), leaving at total of 664 injury events for analysis.

Table 3.

Abstracted Injury Events by Inclusion Status for the 15 Men's and Women's Soccer Schools Studied, 2005–2007

Events	n, %
Included events	664 (93.3)
Excluded events	48 (6.7)
Not a time-lost event	2 (0.3)
Unable to confirm whether time was lost	2 (0.3)
Not a sport-related eventa	5 (0.7)
Only verbal information available from athletic trainer at time of abstraction	24 (3.4)
School data not included in the NCAA ISS, 2005–2006	15 (2.1)
Total abstracted	712 (100)

Abbreviation: NCAA ISS, National Collegiate Athletic Association Injury Surveillance System.

^a Includes shingles and poison ivy.

Capture-recapture analysis estimated that the NCAA ISS captured 88.3% (95% confidence interval = 85.9%, 90.8%) of an estimated total of 677 injury events (Table 4). The proportion of injury events captured (hereafter termed the capture rate) by the NCAA ISS varied by division (χ₂² = 19.2, P < .01), with greater capture in Division I and less capture in Division III. The capture rate tended to be slightly lower when the non–NCAA ISS data source included an electronic database (χ₁² = 2.6, P = .11) and was similar for both men's and women's soccer (χ₁² = 1.5, P = .22), academic year (χ₂² = 4.1, P = .13), and presence of an undergraduate program in athletic training (χ₁² = 0.09, P = .76). The NCAA ISS capture rate per school ranged from 67.9% to 100.0%.

Table 4.

Capture-Recapture Analysis for Men's and Women's Soccer Time-Lost Medical-Attention Injury Events Abstracted from the 15 Schools Studied, 2005–2007

Category	Injury Events in Both NCAA ISS and Non–NCAA ISS systems (a)	Injury Events in NCAA ISS but Not in Non–NCAA ISS system (b)	Injury Events Not in NCAA ISS but in Non–NCAA ISS system (c)	Estimated Injury Events Missed by Both NCAA ISS and Non–NCAA ISSa (x)	Estimated Total Eventsb (a + b + c + x = N)	Percentage Capture for NCAA ISS (95% Confidence Interval)	Percentage Capture for Both Systems (95% Confidence Interval)
Total	500	98	66	12.9	676.9	88.3 (85.9, 90.8)	73.9 (70.6, 77.2)
Year
2005	56	33	6	3.5	98.5	90.4 (84.5, 96.2)	56.9 (47.1, 66.6)
2006	177	36	31	6.3	250.3	85.1 (80.7, 89.5)	70.7 (65.1, 76.4)
2007	267	29	29	3.1	328.1	90.2 (87.0, 93.4)	81.4 (77.2, 85.6)
National Collegiate Athletic Association division
I	240	56	16	3.7	315.7	93.8 (91.1, 96.4)	76.0 (71.3, 80.7)
II	60	13	7	1.5	81.5	89.6 (82.9, 96.2)	73.6 (64.1, 83.2)
III	200	29	43	6.2	278.2	82.3 (77.8, 86.8)	71.9 (66.6, 77.2)
Sport
Men's soccer	264	68	30	7.7	369.7	89.8 (86.7, 92.9)	71.4 (66.8, 76.0)
Women's soccer	236	30	36	4.6	306.6	86.8 (83.0, 90.6)	77.0 (72.3, 81.7)
Non–NCAA ISS electronic database
No	253	54	27	5.8	339.8	90.3 (87.2, 93.5)	74.5 (69.8, 79.1)
Yes	247	44	39	6.9	336.9	86.4 (82.7, 90.0)	73.3 (68.6, 78.0)
Undergraduate athletic training education program?
No	271	47	37	6.4	361.4	88.0 (84.6, 91.3)	75.0 (70.5, 79.5)
Yes	229	51	29	6.5	315.5	88.7 (85.3, 92.2)	72.6 (67.7, 77.5)
Athletic trainer–supervised athletic training student entered NCAA ISS data?
No	265	65	47	11.5	388.5	84.9 (81.4, 88.5)	68.2 (63.6, 72.8)
Yes	235	33	19	2.7	289.7	92.5 (89.5, 95.5)	81.1 (76.6, 85.6)

Abbreviations: NCAA ISS, National Collegiate Athletic Association Injury Surveillance System; non–NCAA ISS, other data source used by the athletic trainer.

^a Injury events not captured in either source estimated with capture-recapture analysis.

^b Total injury events include abstracted and estimated injury events not captured in either source.

A higher capture rate was observed at schools where a supervised athletic training student entered data in the NCAA ISS compared with schools where other staff entered data (χ₁²= 9.1, P < .01). The proportion of events captured by both data sources increased over the 3-season period (χ₁² = 25.6, P < .01), possibly indicating a learning curve associated with use of the NCAA ISS, dual injury-tracking systems, or both.

Agreement for Key Data Fields

For injury events captured in both data sources (n = 500), the effective agreement between the NCAA ISS and the non–NCAA ISS data source for specific data fields ranged from 32.4% for number of days lost from sport to 99.2% for season (preseason, regular season, postseason, or other) (Table 5). Kappa percentages of agreement ranged from 35.0% for severity to 94.6% for body part and tended to be lower than the effective agreement values. For both statistics, agreement was highest for all the injury detail fields (incident or recurrent, chronic or acute, side of the body, surgery needed, body part, injury type, and diagnosis) and was substantial for injury mechanism, activity at time of injury, and outcome. Of the 124 that did not agree on the injury mechanism, 46.0% (n = 57) were unknown or missing; 36.3% (n = 45) were coded as acute noncontact; 10.5% (n = 13) were coded as contact with player, surface, or apparatus; and 7.3% (n = 9) were coded as other in the non–NCAA ISS data source. These events were coded in the NCAA ISS as acute noncontact (19.4%, n = 24); contact with player, surface, or apparatus (60.5%, n = 75); and other (20.1%, n = 25). Only 24 records did not agree for body part; of these, 29.2% (n = 7) were coded as thigh, 25.0% (n = 6) were coded as hip, 12.5% (n = 3) were coded as ankle, and 8.3% (n = 2) were coded as knee in the non–NCAA ISS data source versus 25.0% coded as thigh (n = 6) or knee (n = 6) and 20.8% (n = 5) coded as lower leg in the NCAA ISS.

Table 5.

Percentage Agreement for Injury Event, Injury Detail, and Return-to-Play Data Fields for Injury Events (n = 500) Captured by Both Data Sources for the 15 Men's and Women's Soccer Study Schools, 2005–2007

	Effective Percentage of Agreement (95% CI) Number of categories per variable is 2: agree versus noa	Kappa Percentage of Agreement (95% CI) Number of categories per variable ranged from 2 to 33a
Event details
Injury date	87.2% (84.3%, 90.1%)	NAb
Sports related	98.2% (97.0%, 99.4%)	49.4% (14.5%, 84.3%)
Season	99.2% (98.4%, 100%)	66.5% (22.8%, 100%)
Event type	89.8% (87.2%, 92.5%)	82.6% (78.2%, 87.1%)
Mechanism	75.2% (71.4%, 79.0%)	66.7% (62.0%, 71.5%)
Activity	61.6% (57.3%, 65.9%)	59.0% (54.4%, 63.6%)
Injury details
Incident or recurrent	92.6% (90.3%, 94.9%)	70.1% (61.3%, 78.9%)
Chronic	97.2% (95.8%, 98.7%)	64.7% (45.7%, 83.8%)
Side of body	93.0% (90.8%, 95.2%)	88.9% (85.3%, 92.5%)
Surgery needed	96.4% (94.8%, 98.0%)	81.2% (71.6%, 90.8%)
Body part	95.2% (93.3%, 97.1%)	94.6% (92.4%, 96.7%)
Injury type	92.4% (90.1%, 94.7%)	90.8% (88.1%, 93.6%)
Diagnosis code	90.2% (87.6%, 92.8%)	NAb
Return-to-play details
Outcome	82.2% (78.9%, 85.6%)	52.2% (43.3%, 61.2%)
Date of full return	35.2% (31.0%, 39.4%)	NAb
Number of days lost	32.4% (28.3%, 36.5%)	NAb
Severity: 0, 1–7, 8–14, 15–30, or 31+ daysc	51.8% (47.4%, 56.2%)	35.0% (29.7%, 40.3%)

Abbreviation: NA, not available.

^a See Appendix for variable categories.

^b No Κ percentage calculated for date of injury, diagnosis, date of return, or number of days out because of the large number of possible combinations.

^c Severity variable derived from number of days lost.

Irrespective of the statistic used, agreement was lowest for 2 of the return-to-play data fields: date of full return and days lost. Considering only the records with days lost not missing on both the NCAA ISS and non–NCAA ISS, 128 of 252 injury events did not agree; of these 128 events, 60.2% (n = 77) of the injury events were ± 1 to 2 days of each other, 28.1% (n = 36) were ± 3 to 7 days of each other, and 11.7% (n = 15) were ± 8 or more days of each other. The correlation between the data sources for days lost was 0.62. Effective agreement was higher for less restrictive return-to-play variables: outcome (82.2% coded as full return to play, medical disqualification, athlete no longer with team, or other) and severity (51.8% coded as 0, 1–7, 8–14, 15–30, or 31+ days lost).

Characteristics of Injuries Not Captured by the NCAA ISS

Injuries not captured in the NCAA ISS (n = 66) were predominantly sprains, contusions, and strains to the ankle, thigh, and knee (Table 6). Time-lost and event outcome information was missing for 20 events; for the remaining 46 events, all athletes returned to play with a median of 5 days lost from sport (mean = 10; SD = 16.8; range, 1 to 96 days).

Table 6.

Characteristics of Time-Lost Medical-Attention Injury Events Not Captured by the NCAA ISS for the 15 Men's and Women's Soccer Study Schools, 2005–2007

	N	%
Mechanism
Contact with player or competitor	27	40.9
Acute noncontact	13	19.7
Unknown	10	15.2
Overuse or gradual onset	10	15.2
Contact with playing apparatus	2	3.0
Contact with playing surface	2	3.0
Illness	2	3.0
Body part
Ankle	15	22.7
Thigh	11	16.7
Knee	9	13.6
Lower leg	7	10.6
Foot	5	7.6
Head or face	4	6.1
Hip	4	6.1
Chest, thoracic spine, or ribs	2	3.0
Environmental or fluids	2	3.0
Neck or cervical spine	2	3.0
Other	5	7.6
Injury type
Sprained ligament (partial or complete)	15	22.8
Contusion or hematoma	14	21.2
Strained muscle or tendon (partial or complete)	12	18.2
Other	13	19.7
Concussion	4	6.1
Compartment syndrome	2	3.0
Fracture or avulsion	2	3.0
Spasm or cramp	2	3.0
Tendinosis	2	3.0
Total	66	100.0

Abbreviation: NCAA ISS, National Collegiate Athletic Association Injury Surveillance System.

DISCUSSION

To our knowledge, this 2-sport validation study is the first to be conducted for a national Web-based sports injury-surveillance system. We found that the NCAA ISS captured 88% of time-lost medical-attention injury events in the 15 collegiate men's and women's soccer study schools during the 3-season period, a very good capture rate. The capture rate was highest in Division I and lowest in Division III, which may reflect the increased resources available to ATs in Division I. Effective agreement between the NCAA ISS and the non–NCAA ISS data sources was also high, close to or above 90% for most of the data fields examined. However, 2 interrelated data fields, date of full return and days lost, had low agreement. In view of this finding, administrators of collegiate sports injury-surveillance systems and end users of the data from these systems should carefully consider the implications of using time-loss criteria to define injury severity. Similar variability with time-loss data was noted in a previous NCAA ISS study²⁶ and is consistent with these findings. Other authors²⁷ have more globally discussed the limitations of using time-loss criteria in this manner.

The only previous validation study of sports injury did not address capture rate but rather focused on reliability of self-reported injury details.²⁸ These authors noted moderate levels of agreement for injured body part and treatment but low agreement for injury severity. Therefore, to place these findings in context, we need to go beyond the sports medicine literature to the literature on general injury surveillance. The most comparable study is a validation study of the CDC's emergency department surveillance system for all injuries (including non-sports injuries), NEISS-AIP, which had a capture rate of 83% (490/593 injury events).²⁹ This capture rate was considered very good for surveillance purposes, and NEISS-AIP is widely regarded as the most reliable system for monitoring emergency department injury visits.

Determinants of Capture Rate

We found that the capture rate was more than 10 percentage points higher in Division I schools than in Division III schools, probably reflecting the greater resources available to Division I ATs. The capture rate was 8 percentage points higher when data were entered into the NCAA ISS by a supervised athletic training student rather than entered solely by the ATs. The increase by year in the capture rates for the NCAA ISS and non–NCAA ISS systems combined may reflect a learning curve as ATs became more experienced using the new Web-based NCAA ISS and developed better methods for managing 2 data systems during the study period. In addition, increased NCAA resources and staff were directed toward training, follow-up, and monitoring of ATs and schools from 2005 to 2008 (J. Corlette, oral communication, January 15, 2010).

Data Agreement and Days-Lost Data Field

Agreement for all data fields was good except for mechanism of injury, activity before injury, date of full return, and number of days out. Date of full return and the number of days out were not available or missing more often in the non–NCAA ISS data source (n = 230 missing) than in the NCAA ISS (n = 71 missing). Maintaining accurate information on days lost was not the primary intent of the non–NCAA ISS data systems accessed in this study, and the missing information illustrates the different purposes of sports injury-surveillance systems and clinical record keeping. Also, the non–NCAA ISS data source in this study represents a variety of different methods, none of which was designed for injury surveillance. Therefore, our results concerning the low agreement for days lost and date of return should be interpreted with caution, because we acknowledge the possibility that these data fields may be more commonly and accurately recorded in the NCAA ISS than in the non–NCAA ISS data source.

Assuming it is valid, the low level of agreement for days lost suggests that using time loss as the only marker of injury severity may be unwise, at least in this population. Previous consensus statements³⁰,³¹ have made similar recommendations for specific sports and discussed more globally the distinction between incidence of injury and severity of injury.³² Although widely used as a marker of injury severity, time loss is only one method of quantifying severity.¹⁸ Other markers of severity include cost, treatment, and disability. Time loss for any particular injury is not a constant but varies among athletes, clinicians, and sports. For example, a hand injury might be severely debilitating to a tennis player, but the same injury might not result in any time loss for a runner. Additionally, advancements in prevention, treatment, and rehabilitation mean that injury severity measures based solely on days lost have an interpretation that changes over time. Observed shifts in the distribution of time loss among schools may merely reflect differences in treatment patterns rather than a true change in the underlying severity of injury. These are all important points to consider when developing and using severity measures in sports injury-surveillance systems.

Limitations

Although all researchers received training to ensure standardization and consistency in data abstraction, no formal statistical assessment of interabstractor or intra-abstractor reliability was conducted. Therefore, we do not know the degree of variability in data coding attributable to differences in abstractor coding. Most of the data coding was conducted by pairs of researchers to ensure consensus in data abstraction.

In this study, there was no gold standard. Thus, comparisons were made between the data sources with capture-recapture analysis, which accounts for assumed error in both sources. This is common practice in public health surveillance studies when neither source can be considered complete.²¹,²²,³³ However, the 88% NCAA ISS capture rate may be an underestimate if the non–NCAA ISS data source includes many injury events that were false-positives (events recorded as time loss that were, in fact, not time-loss injuries). With this possibility in mind, we went to considerable lengths to establish that all the injury events reported in the non–NCAA ISS data were true time-lost events. When we were unable to determine whether the event had resulted in time loss based on the non–NCAA ISS data source, we queried the AT to verify that the event did result in time loss. Time loss was verified with the AT regarding 12 events for which the non–NCAA ISS data source did not indicate a missed practice or game. These events were included in the study, but values for date of full return and number of days remained missing. In 2 cases in which time loss could not be confirmed in the non–NCAA ISS source or with the AT, we excluded the events from our analysis (Table 3). If the injury existed only in the NCAA ISS and we found no written record of the injury anywhere else, we sought additional information from the AT to confirm that the injury was indeed a time-lost medical-attention injury. However, verbal information from memory in the absence of a written record was not considered an acceptable comparison data source; events based only on the AT's recollection without documentation could not be qualified as time loss and were excluded from our analyses (n = 24, Table 3). As a limited sensitivity analysis, if all 26 events that we could not confirm as time loss had been included in our analysis, the estimated capture rate of the NCAA ISS would not have materially changed: (500 + 124)/(500 + 124 + 66 + 16.4) = 88.1%. The daily injury report to the coach provided a good source of information regarding lost days and date of return. However, not all schools in this study provided a formal daily report to the coach. Some relied heavily on e-mail to communicate with their coaches, and others gave verbal reports to the coach, neither of which were available or considered valid sources for comparison.

Effective agreement is a valuable tool for assessing agreement. Yet because effective agreement does not account for agreement due to chance alone, we also calculated the ϰ percentage of agreement (Table 5). Kappa has several limitations, including the fact that it is unduly conservative in some settings and tends to be low when the number of response categories is small or the prevalence of the attribute is high.²³,³⁴ The latter factor (it tends to be low when the prevalence of the attribute is high) may particularly have affected the ϰ values for sport relatedness and outcome. Missing values for data fields were considered valid entries in both agreement analyses. This was not an unreasonable assumption if the person entering the data into both sources left it missing because he or she truly did not know the value (eg, activity before the injury event was unknown). However, this assumption might not be reasonable when the value was missing in only a single data source. Restricting the analysis to nonmissing values for both data sources would have affected mainly the results for mechanism (effective agreement in nonmissing of 84.9%), activity before the injury (84.3%), outcome (98.5%), and injury severity (81.8%). Similarly, when restricted to nonmissing values, effective agreement was still low for date of full return (54.9%) and days lost (49.2%).

At the time of enrollment, schools had to have at least 2 years of experience with the NCAA ISS to be eligible for this study. Although some ATs did not have 2 years of experience with the NCAA ISS, all 15 site administrators had at least 2 years of experience (mean = 4.3; range, 2 to 7 years). The capture rate of the NCAA ISS was similar when we compared ATs with less than 2 years of experience (90.9%) with those who had 2 or more years of experience (87.8%) using the NCAA ISS.

This validation study was performed for only 2 collegiate sports: men's and women's soccer. Therefore, these results may not be generalizable to other sports or to the winter and spring seasons, in which ATs' workloads and coverage might be different. Comparing participating schools (n = 15) with nonparticipating schools (n = 30), our study group included a greater proportion of Division I (40% versus 27%) and Division II (20% versus 10%) schools and a lesser proportion of Division III schools (40% versus 63%). We also acknowledge that schools that use the NCAA ISS as the sole record-keeping system may report more accurate information; it was impossible to examine this suggestion using our study design. Finally, abstraction of practice and game exposure data was beyond the scope of this investigation. Future researchers should examine the validity and reliability of injury data in other sports and address exposure data.

CONCLUSIONS

The overall rate of capture of the NCAA ISS for the 15 schools and 2 sports in this study (88%) was very good. This finding indicates that the NCAA ISS can yield reliable and valid injury statistics. Yet the low level of agreement for days lost (32%) raises concerns about the validity of using time loss as the only marker of injury severity in surveillance systems. However, it should be noted that maintaining accurate information on days lost was not the primary intent of the non–NCAA ISS data systems accessed in this study and, therefore, our results regarding time loss should be interpreted with caution.

Appendix.

Categories for Variables Abstracted from Athlete's Injury Record and Compared with NCAA ISS Values

Sports-related

Yes—injury was sustained while participating in collegiate sport activity

No—injury was not sustained during participation in a collegiate sport activity

Season:

Traditional season (includes pre- and postseason)

Non-traditional season

Event type

Competition - Junior varsity

Competition - Varsity

Practice

Strength and conditioning

Other

Injury mechanism

Contact with player/competitor

Contact with playing surface

Contact with playing apparatus

Contact with out of bounds objects

Acute non-contact

Overuse/gradual onset

Illness

Infection

Other

Activity

Shooting

Passing

Receiving pass

Ball handling/dribbling

Defending

Blocking shot

Loose ball

Heading ball

Attempting slide tackle

Receiving slide tackle

goal tending

conditioning

general play

other

New or recurrent

New

Recurrent—this season

Recurrent—previous college season

Recurrent—previous other college

Chronic or acute

Chronic

Acute

Side of body

Right

Left

Both

NA

Surgery required

Yes

No

Body part

Abdomen (internal)

Ankle

Arm

Chest/T-spine/ribs

Ear

Elbow

Eye

Foot

Forearm

Hand

Head/face

Hip

Knee

Lower back/L-spine/pelvis

Lower leg

Mouth

Neck/cervical spine

Nose

Shoulder

Thigh

Wrist

Cardiovascular

Dermatology

Endocrinology

Environmental/fluids

GI

GU

Haematology

Infectious disease

Nervous system

Psychological

Respiratory

Rheumatology/metabolic bone

Other

Injury type

Abrasion

Blood vessel injury

Cartilage injury

Compartment syndrome

Concussion

Contusion/hematoma

Dislocation

Effusion

Epicondylitis

Stress Fracture

Fracture/avulsion

Growth plate (epiphyseal) injury

Laceration

Myositis ossificans

Nerve injury (eg, stinger, entrapment)

Organ injury

Osteochondritis

Spasm/cramp

Sprain ligament (partial/complete)

Strain muscle/tendon (partial/complete)

Subluxation

Blisters

Cysts

Disc injury

Hernia (eg, inguinal)

Impingement

Infection

Loose body

Neuroma

Overuse (eg, periostitis/shin splints)

Scar tissue

Spur

Inflammatory

Arthritis/chondromalacia

Bursitis

Capsulitis

Inflammation (general)

Necrosis (avascular)

Plantar fascitis

Synovitis

Tendinosis

Other

Event outcome

Return to play

Medical disqualification (season)

Medical disqualification (career)

Athlete chose not to continue

Athlete released from team

Permanent paralysis

Fatality

Other

Continued

Sports-related

Yes—injury was sustained while participating in collegiate sport activity

No—injury was not sustained during participation in a collegiate sport activity

Season:

Traditional season (includes pre- and postseason)

Non-traditional season

Event type

Competition - Junior varsity

Competition - Varsity

Practice

Strength and conditioning

Other

Injury mechanism

Contact with player/competitor

Contact with playing surface

Contact with playing apparatus

Contact with out of bounds objects

Acute non-contact

Overuse/gradual onset

Illness

Infection

Other

Activity

Shooting

Passing

Receiving pass

Ball handling/dribbling

Defending

Blocking shot

Loose ball

Heading ball

Attempting slide tackle

Receiving slide tackle

goal tending

conditioning

general play

other

New or recurrent

New

Recurrent—this season

Recurrent—previous college season

Recurrent—previous other college

Chronic or acute

Chronic

Acute

Side of body

Right

Left

Both

NA

Surgery required

Yes

No

Body part

Abdomen (internal)

Ankle

Arm

Chest/T-spine/ribs

Ear

Elbow

Eye

Foot

Forearm

Hand

Head/face

Hip

Knee

Lower back/L-spine/pelvis

Lower leg

Mouth

Neck/cervical spine

Nose

Shoulder

Thigh

Wrist

Cardiovascular

Dermatology

Endocrinology

Environmental/fluids

GI

GU

Haematology

Infectious disease

Nervous system

Psychological

Respiratory

Rheumatology/metabolic bone

Other

Injury type

Abrasion

Blood vessel injury

Cartilage injury

Compartment syndrome

Concussion

Contusion/hematoma

Dislocation

Effusion

Epicondylitis

Stress Fracture

Fracture/avulsion

Growth plate (epiphyseal) injury

Laceration

Myositis ossificans

Nerve injury (eg, stinger, entrapment)

Organ injury

Osteochondritis

Spasm/cramp

Sprain ligament (partial/complete)

Strain muscle/tendon (partial/complete)

Subluxation

Blisters

Cysts

Disc injury

Hernia (eg, inguinal)

Impingement

Infection

Loose body

Neuroma

Overuse (eg, periostitis/shin splints)

Scar tissue

Spur

Inflammatory

Arthritis/chondromalacia

Bursitis

Capsulitis

Inflammation (general)

Necrosis (avascular)

Plantar fascitis

Synovitis

Tendinosis

Other

Event outcome

Return to play

Medical disqualification (season)

Medical disqualification (career)

Athlete chose not to continue

Athlete released from team

Permanent paralysis

Fatality

Other

Abbreviation: NCAA ISS, National Collegiate Athletic Association Injury Surveillance System.

DISCLAIMER

Conclusions drawn from or recommendations based on the data are those of the authors and do not represent the views of the officers, staff, or membership of the NCAA or the Datalys Center for Sports Injury and Prevention.