Does the Reliability of Reporting in Injury Surveillance Studies Depend on Injury Definition?

Abstract

Background:

Choosing an appropriate definition for injury in injury surveillance studies is essential to ensure a balance among reporting reliability, providing an accurate representation of injury risk, and describing the nature of the clinical demand.

Purpose:

To provide guidance on the choice of injury definition for injury surveillance studies by comparing within- and between-team variability in injury incidence with >24-hour and >7-day time-loss injury definitions in a large multiteam injury surveillance study.

Study Design:

Cohort study (diagnosis); Level of evidence, 2.

Methods:

Injury data were reported for 2248 professional rugby union players from 15 Premiership Rugby clubs over 12 seasons. Within-team percentage coefficient of variation and mean between-team standard deviation (expressed as a percentage coefficient of variation) in injury incidence rates (injuries per 1000 player match hours) were calculated. For both variables, a comparison was made between >24-hour and >7-day injury incidence rates in terms of the magnitude of the observed effects.

Results:

The overall mean incidence across the population with a >24-hour time-loss injury definition was approximately double the reported incidence with the >7-day definition. There was a 10% higher between-team variation in match injury incidence rates with the >24-hour time-loss definition versus the >7-day definition.

Conclusion:

There was a likely higher degree of between-team variation in match injury incidence rates with a >24-hour time-loss definition than with a >7-day definition of injury. However, in professional sports settings, it is likely that the benefits of using a more inclusive definition of injury (improved understanding of clinical demand and the appropriate and accurate reporting of injury risk) outweigh the small increase in variation in reporting consistency.

Injury surveillance is considered the first step toward developing an injury prevention strategy,^6,22 and the availability of continuous high-quality longitudinal injury data⁵ is crucial for providing an assessment of the current risk of injury,²² identifying potential modifiable risk factors for injury,¹ and establishing a foundation for appropriately directed preventative interventions.⁷ For an injury surveillance system to yield meaningful outputs, the selected methodology and study definitions must be tailored to meet the requirements of the specific setting,²¹ study population,³ and research question.¹³ An important issue for epidemiologists to consider when setting up an injury surveillance study is the selection of the most suitable operational definition of injury.^2,17 Specifically, arguments have been made for both an inclusive definition (ie, a narrower definition of injury, such as >24 hours of time lost from training and/or match play) and a more exclusive definition (ie, a broader definition of injury, such as an injury that causes the player to miss at least 1 match).^9,17 The selected definition of injury defines the minimum injury severity for a case to be recorded in the study. This minimum severity may vary from “all injuries requiring medical attention” (eg, for a study comparing abrasion injuries sustained on grass and artificial playing surfaces) to “injuries leading to time loss from training or match play” (eg, an ongoing study of injuries in a professional team sport) and “injuries requiring hospital treatment and management” (eg, catastrophic spinal injuries). The first and third examples are normally easy to define and implement; however, studies related to the second example can lead to a further issue. The reliability for the reporting of mild injuries (<7 days) has been questioned, primarily because of the potential for large variability among teams in the reporting of these injuries.¹⁷ This matter is particularly important, as these injuries likely represent a large proportion of the overall injury burden sustained by a team; therefore, they have a significant impact on the clinical demand experienced by the team’s medical support team⁹ and the team’s ability to prepare for and perform in competition.

It has been suggested that injury surveillance studies adopting more inclusive definitions of injury should demonstrate the reliability of reporting between the most and least motivated teams taking part in the study,¹⁷ thereby providing an assessment of the overall reliability of the surveillance system. One way in which this can be assessed is to explore the variability and distribution of reported injury incidence rates across the various teams included in the study over time. Between-team reliability was studied in the Australian Football League¹⁸ by comparing the variation in injury reporting over 2 seasons. In 1995, the definition of injury was “any injury that resulted in missed playing time in a match or a missed training session.” In 1997, the injury definition changed to “any trauma that prevented a player from playing in a regular season match” (a match time-loss definition). This definitional change was associated with a substantial reduction in the variation of reporting among clubs, with the coefficient of variation (CV; standard deviation relative to the mean) decreasing from 101% in 1995 to 34% in 1997. This finding suggested that the “match time-loss” definition was more reliable in this setting; therefore, this definition was implemented by the Australian Football League injury surveillance system for all future studies.¹⁹

In a more recent study that assessed the methods and data quality of 15 sports injury surveillance systems, researchers concluded that there was limited information about the quality of injury surveillance data in the professional/elite sports setting.⁵ Seven of the 15 included injury surveillance systems were identified as having no publication regarding data quality. The English Professional Rugby Injury Surveillance Project (PRISP), the largest and longest-running study of injury risk in professional rugby union, was 1 of the 7 systems identified. Obtaining a truly objective assessment of data quality in this cohort is problematic owing to the lack of any independent gold standard comparison. Furthermore, the influence that different definitions of time-loss injury (eg, time loss from match and training activities vs match time loss only) may have on the variability of injury reporting within surveillance systems remains unknown. The longitudinal data set available from the PRISP study affords an opportunity to investigate reporting reliability via analysis of the between- and within-club reporting variability, as well as the effects of using different injury definitions on the reliability of reported data in a professional sport setting. This information could inform the decision-making process for the selection of the optimum operational definition of injury for future studies in similar settings among a variety of team sports. The aim of this study was therefore to compare the within- and between-team variability in the reporting injuries for >24-hour and >7-day time-loss injury definitions based on the PRISP multiteam injury surveillance study database of injuries.

Methods

Study Design and Setting

Injury data were collected over 12 seasons as part of the ongoing England PRISP study. All teams were required to submit injury and exposure data in compliance with their competition agreement to meet minimum standards with respect to player welfare. Data collected from the 12 league teams in each of the 12 seasons between 2002-2003 and 2014-2015 were included in the analysis (no data were collected during the 2004-2005 season), giving rise to a total of 15 teams taking part due to promotions and relegations during this period.

Participants

All consenting players who were members of the club’s first team squad were eligible for inclusion. The analysis included 2248 professional rugby union players from 15 professional clubs, contributing a total of 6706 player-seasons. Written informed consent was obtained from all players in the study, and all data were anonymized. The appropriate research ethics committee of the academic host institutions (University of Bath, University of Nottingham, and University of Leicester) approved the study each season, with individual player consent obtained each season regardless of whether the player had consented to the project previously.

Variables

The injury definitions and reporting procedures used in this study were constant across all seasons and were consistent with the international consensus statement for epidemiologic studies in rugby union.⁶ The primary injury definition used in this study was “any injury sustained by a player during a first-team match or training session that prevented the player from taking full part in all training activities typically planned for that day and/or match play for >24 hours from midnight at the end of the day that the injury was sustained.” Such injuries were defined as “>24-hour time-loss injuries.” Injuries that resulted in an injured player being unavailable for match selection or unable to take part in normal training sessions for >7 days were defined as “>7-day time-loss injuries.” These 2 definitions were chosen specifically, as they are the most commonly used time-loss definitions in injury surveillance systems.

The injury surveillance administrator worked with all clubs on a weekly basis to ensure the capture of accurate return-to-play dates, which allowed for the calculation of injury severity (days absence). This was then used to establish those injuries that met the time thresholds for the 2 common injury definitions analyzed in this study (>24 hours and >7 days). All injuries were recorded by qualified medical personnel at each club using a modified Orchard Sports Injury Classification System¹⁶ and standard injury report form (2002-2003 to 2012-2013) or an electronic player medical records system (Rugby Squad Medical, The Sports Office; 2013-2014 to 2014-2015). The collected data fields remained the same for the duration of the study. Individual player match exposure data and squad grouped training exposure data were reported separately from the injury data on a weekly basis by team strength and conditioning staff at each club. Injury incidence rates were then calculated and reported separately for match and training exposures as the number of injuries per 1000 player-hours of exposure.

Statistical Methods

A spreadsheet¹¹ was used to calculate the typical error (expressed as a CV percentage) in injury incidence rates reported within teams across the 12-season period (ie, what was the variability of reporting each season for the same team). Similarly, the mean between-team variation in injury incidence rates for each season (SD, expressed as a CV percentage) was calculated (ie, what was the variability of reporting among the individual teams throughout the study). For both variables, a comparison was made between >24-hour and >7-day time-loss injury incidence rates via a spreadsheet for the magnitude of effects.¹² The derived ratio of the 2 effects (>24-hour time loss / >7-day time loss) was calculated with 90% confidence limits; an effect was deemed unclear if its confidence interval overlapped thresholds for substantiveness (0.90 and 1.11) by >5%.¹⁰ Otherwise, the effect was clear and deemed to have the magnitude of the largest observed likelihood value. All values were log transformed to reduce nonuniformity of error related to the magnitude of the variable. Separate analyses were performed for match and training injuries.

Results

During the 12-season study period, the mean >24-hour time-loss injury incidence rate was 86.0 per 1000 player-hours for match injuries and 2.4 per 1000 player-hours for training injuries. The mean >7-day time-loss incidence rate was 42.8 per 1000 player-hours for matches and 1.5 per 1000 player-hours for training. Table 1 shows that a large proportion of injuries (around 50% of all match injuries and 40% of all training injuries) had a severity of <7 days. In addition, the reporting of these more minor injuries as a proportion of all reported injuries remained stable each season (with the exception of the reporting of <7-day training injuries in the 2003-2004 season). The within- and between-team variation observed for training injuries was significantly higher than that for matches (Table 2), and with the >24-hour injury definition (vs the >7-day definition of injury), there was a likely higher degree of between-team variation in match injury incidence rates (Figure 1), although this was only 10%. All other comparisons were unclear (Figure 1).

TABLE 1.

Reported <7-Day Injuries as a Proportion of Total Incidence (>24 Hours) by Season for Match and Training Injuries

	Match Incidence (per 1000 h)			Training Incidence (per 1000 h)
Season	<7 d	Total	<7 d, %	<7 d	Total	<7 d, %
2002-2003	57	100	57	1.1	3.0	38
2003-2004	45	88	51	0.2	1.6	10
2005-2006	29	75	39	1.0	2.2	47
2006-2007	47	90	52	1.0	1.9	52
2007-2008	39	83	47	1.3	2.8	45
2008-2009	48	100	48	1.0	2.4	42
2009-2010	36	80	45	1.1	2.4	45
2010-2011	44	93	47	1.2	2.8	44
2011-2012	34	82	41	0.9	2.3	38
2012-2013	26	73	36	0.9	2.6	35
2013-2014	38	91	42	0.9	2.9	32
2014-2015	33	79	42	0.9	2.3	38
Mean	40	86	47	1.0	2.4	39

TABLE 2.

Within-Team Typical Errors and Between-Team Standard Deviations as a Function of Injury Definition: >24-Hour vs >7-Day Time Loss^a

Injury: Time Loss	Within-Team Typical Error (90% CL)	Between-Team SD (90% CL)
Match
>24 h	35 (30-42)	47 (38-52)
>7 d	32 (28-38)	37 (32-44)
Training
>24 h	53 (45-66)	79 (63-119)
>7 d	52 (43-64)	76 (61-116)

^aThe results are presented for match and training injuries separately and are reported as coefficient of variation (%). CL, confidence limit.

Figure 1.

Rate ratio (using the >7-day time-loss definition as reference) for within-team and between-team reporting variability. Data labels give the percentage likelihood that the effect is lower | trivial | higher, and associated qualitative inference. Dotted lines represent thresholds for smallest worthwhile difference (0.90 and 1.11).

Discussion

The aim of this study was to investigate the within- and between-team injury reporting variability within a large multiteam surveillance study and to explore the impact that 2 divergent definitions of injury may have on reporting variability. With a >24-hour time-loss definition of injury (vs a >7-day definition), there was a likely higher degree of between-team variation in match injury incidence rates. All other comparisons were unclear and require more data to make any meaningful inference. The within- and between-team variability was higher for training injury incidence when compared with the match injury incidence for both definitions of injury.

Selecting the most appropriate injury definition is often problematic for researchers.^13,17,23 This study compared the within- and between-team variability of reporting for 2 common time-loss injury definitions. The findings highlight a likely higher variability of between-team reporting of match injuries with the more stringent >24-hour time-loss definition. It is, however, our interpretation that this 10% increase in reporting variation is not substantial when we consider the inherent natural variation attributed to the frequent changes to club and league structures, law changes, and the unpredictable nature of team sports that cannot be accounted for quantitatively. Selection of the most appropriate injury definition in any setting should take account of the balance among reporting reliability, an accurate representation of injury risk, and an understanding of clinical demand. The chosen definition of injury dictates how injury risk is presented in a particular context,² and selecting an inappropriate definition of injury may lead to an under- or overinterpretation of injury risk in that particular setting.

The impact of utilizing different injury definitions (both time loss and those that do not use time loss as an inclusion criterion) was studied in this cohort during the 2002-2004 period.² The injury definitions used and the corresponding incidence rates (displayed as injuries per 1000 hours) were as follows: player missing >1 day, 91 (95% CI, 87-96); player missing ≥1 match, 40 (95% CI, 37-43); player requiring diagnostic tests, 25 (95% CI, 22-27); and player requiring surgery, 4.5 (95% CI, 3.5-5.5). In this analysis, around 50% of all match injuries and 40% of all training injuries would have not been reported if a >7-day definition had been utilized. This highlights the importance of including those injuries that are less severe (<7 days) where possible when presenting injury incidence, as they represent a large proportion of the overall injury incidence and thus contribute significantly to the clinical workload of practitioners in assessing and treating the player.

These less severe injuries are also important from the players’ perspective because they may still affect the players’ availability for selection and could also cumulatively influence short-term tissue recovery.¹⁴ In the medium to long term, they may also be a precursor for more severe injuries. The reporting of these less severe injuries is also important for the practitioner in a more practical sense, as injuries that lead to any time lost from training and match play are likely to have an adverse effect on the performance of the player and/or team²⁴ and may have a chronic impact on the long-term well-being of an athlete.² Ultimately, in terms of the cost-benefit ratio between the slight increases in the variability of reporting associated with the more inclusive definition of injury and the considerable difference in the reported incidence of injury, this study supports the use of the more inclusive definition of injury in this professional setting.

However, it is also important for researchers to understand that one size does not fit all and that choosing the correct injury definition largely depends on (1) the context and setting of the population and (2) the parameters within which the reporting is taking place.^3,21 The importance of the latter point is clearly highlighted when this study’s findings (in support of a more inclusive definition) are compared with those from Australian rules football, which revealed that a change from an inclusive to more exclusive definition was associated with a substantial reduction in the variation of reporting among clubs.¹⁷ One important limitation to this approach is that these data relate to the absolute number of injuries reported rather than the incidence rates and therefore do not account for potential differences in exposure time and squad sizes among teams. In addition, the most appropriate choice of definition is likely to differ for the professional and amateur settings.

It has been suggested that the likelihood of not reporting less severe injuries is minimized in the professional setting owing to the consistent contact between player and clinician.^8,15 The findings from this study were largely in agreement with this statement, whereby the reported incidence of injuries with a severity of <7 days and the proportion of these injuries as a function of all injury incidence appeared to be relatively consistent among seasons for match and training injuries (Table 1). The reliability and accuracy of the reporting of less severe injuries are still questioned in professional sport, even when a significant medical resource is available and a controlled environment is commonplace.¹⁷ In the amateur game, it would be therefore reasonable to assume that the reliability and accuracy of reporting these less severe injuries are likely to be reduced in the absence of appropriate and consistent resources and medical expertise. Consequently, a more exclusive definition of injury may be better suited for use in amateur sports settings.

Overall, this study found that variation in the reporting of training injuries was greater than for match injuries. Although the reasons for this difference were beyond the scope of this study, they are likely due to the low absolute numbers of training injuries versus match injuries reported each season.²⁰ For example, if the reported incidence rate differs from 2 to 4 per 1000 hours, this represents a big relative change but a small absolute change, which is likely to have a greater impact on the reporting variability of training injuries in surveillance systems.

Limitations

Non–time loss injuries were not reported in this study; therefore, the variability of reporting of these injuries remains unknown in this setting. However, it is likely that the inclusion of these injuries would increase the variability of reporting.¹⁷ In addition, changes to medical personnel and coaches and, as a consequence, changes in playing style were not accounted for and are likely to influence the reported rate of injuries in this setting. As such, the impact of these changes on the incidence and variability of reporting remains unknown. In addition, it is plausible that the reliability of reporting certain types of injury may vary, but this analysis was beyond the scope of this study. This more detailed analysis may be warranted in the future to further understand the most appropriate definition for studies or surveillance systems that focus on particular types of injury. Furthermore, an assumption is made that training periodization is similar across teams, but recent evidence apropos to this cohort suggests that this may not be the case⁴ and should be considered when interpreting these data. In addition, injury reporting moved from a paper-based to an online-based system for the beginning of the 2013-2014 season. The impact of this change on the variability of reporting is unknown, as the sample was underpowered for this analysis, and should be investigated in the future.

Conclusion

There was a likely higher degree of between-team variation in match injury incidence rates with a >24-hour time-loss definition as compared with a >7-day definition of injury. However, we believe that in professional sports settings, the benefits of a more inclusive definition of injury (improved understanding of clinical demand and the appropriate and accurate reporting of injury risk) likely outweigh the small increase in variation in reporting consistency.