Masking in reports of “most serious” events: bias in estimators of sports injury incidence in Canadian children

Abstract

Introduction:

Surveys that collect information on injuries often focus on the single “most serious” event to help limit recall error and reduce survey length. However, this can mask less serious injuries and result in biased incidence estimates for specific injury subcategories.

Methods:

Data from the 2002 Health Behaviour in School-aged Children (HBSC) survey and from the Canadian Hospitals Injury Reporting and Prevention Program (CHIRPP) were used to compare estimates of sports injury incidence in Canadian children.

Results:

HBSC data indicate that 6.7% of children report sustaining a sports injury that required an emergency department (ED) visit. However, details were only collected on a child’s “most serious” injury, so children who had multiple injuries requiring an ED visit may have had sports injuries that went unreported. The rate of 6.7% can be seen to be an underestimate by as much as 4.3%. Corresponding CHIRPP surveillance data indicate an incidence of 9.9%. Potential masking bias is also highlighted in our analysis of injuries attended by other health care providers.

Conclusion:

The “one most serious injury” line of questioning induces potentially substantial masking bias in the estimation of sports injury incidence, which limits researchers’ ability to quantify the burden of sports injury. Longer survey recall periods naturally lead to greater masking. The design of future surveys should take these issues into account. In order to accurately inform policy decisions and the direction of future research, researchers must be aware of these limitations.

Highlights

• Surveys often collect information on only the most serious injury event within a recall period, which can reduce recall bias and shorten the survey length. However, this line of questioning can also mask less serious injuries, leading to an underestimation of the incidence of specific injury subtypes, such as those that occur during physical activity or sports.

• In this study, data from the 2002 Health Behaviour in School-aged Children (HBSC) survey and from the Canadian Hospitals Injury Reporting and Prevention Program (CHIRPP) were used to compare estimates of sports injury incidence in Canadian children, and evidence of masking bias was found.

• Without an accurate estimation of the incidence of injury subtypes, our ability to quantify injury burden and the impact of interventions is limited.

• Potential masking must be taken into consideration during the design of a survey as well as during the analysis and interpretation of survey data.

Introduction

Unintentional injuries cause acute health problems and are considered a leading cause of youth mortality worldwide.^1,2 A significant portion of pediatric injuries occur while playing or training for a sport or physical activity (“sports injuries”).^3,4,5 Unbiased information about sports injuries may help direct public health resources and evaluate the effectiveness of injury prevention efforts. Inaccurate estimates of the prevalence of specific injury subtypes make it impossible to weigh, for example, the health benefits of physical activity promotion programs against the increased risk of sports injuries.

Health measurement surveys such as the Health Behaviour in School-aged Children (HBSC) study,² the Global School-Based Student Health Survey (GSHS),⁶ the Canadian Community Health Survey (CCHS),⁵ the National Longitudinal Survey of Children and Youth (NLSCY)⁷ and the National Population Health Survey Health Promotion and Chronic Disease Prevention in Canada Vol 36, No 8, August 2016 144 Research, Policy and Practice (NPHS) have utilized a line of questioning that investigates the circumstances surrounding respondents’ “one most serious injury.”⁸ In general, injuries are defined as those that limit daily activities^5,8,9 or require treatment from a doctor or nurse.^2,6,8 However, of the surveys we considered, only the HBSC explicitly defines what is meant by the one “most serious” injury: the injury that “took the most time to get better.”² The GSHS asks respondents, “During the past 12 months, what was the most serious injury that happened to you?” (with responses such as “I had a broken bone or a dislocated joint,” etc.).⁸ The CCHS asks respondents who sustained multiple injuries to think “about the most serious injury” and answer contextual questions, including the month, year, nature of and location where the most serious injury was sustained.⁹ Similarly, the NLSCY asks, “For the most serious injury, what type of injury did he have?” (in this case, proxy reports from parents were used) prior to asking questions about the nature, context and location of the most serious injury.¹⁰

Recall bias poses a significant threat to validity in injury survey data. Recall error can result when a respondent’s response is incomplete due to memory decay (forgetting as a result of time passing). The accuracy of estimators from surveys that use longer recall periods are more affected by limitations of memory and recall.¹¹ Since memory decay decreases with increasing severity of injuries,^12-13 the “one most serious injury” line of questioning should help reduce recall bias; in addition, the respondent only has to remember and report the contextual variables to do with a single injury event, which could greatly reduce the time needed to complete the questionnaire and increase the accuracy and completeness of response.

However, gathering in-depth data on only the single most serious injury does present an important shortcoming: if someone reports a fighting-related injury as their “most serious” in the past 12 months, any “less serious” sports injuries sustained during that period would go unreported. This line of questioning results in underestimating the incidence of specific injury categories whenever a respondent sustains more than one injury during the same recall period. This issue is often overlooked, and the magnitude of this bias in sports injury incidence estimators has not been explored. Published estimates, such as the one stating that 66% of pediatric injuries are sports-related,⁵ are biased because they are based on only “most serious” injuries. Billette and Janz⁵ state in their introduction that their data do not reflect all injuries, but do not acknowledge this masking bias in their summary (“Highlights”) section.⁵ Tozija et al.¹⁴ relied on GSHS data to claim that 29.3% of injuries sustained by adolescents occurred at school, and Jildeh et al.¹⁵ used HBSC data to justify their claim that 78.5% of injuries occur at home; neither of these studies acknowledged the potential masking bias arising from collecting data on only the “most serious” injury.

Methods

In order to explore the potential extent of the masking and bias arising from a “one most serious injury” line of survey questioning, we estimated sports injury incidence rates in Canadian children using the 2002 cycle of the HBSC survey data and the 2001/2002 Canadian Hospitals Injury Reporting and Prevention Program (CHIRPP) surveillance data. We chose this period for the analysis because current HBSC and CHIRPP data are not as directly comparable. Like Pickett et al.,¹⁶ we used rates from two CHIRPP hospitals in Frontenac County, Ontario to calculate rates. These sites have been relatively good indicators of national surveillance trends in the past; together they represent the only CHIRPP sites with full community coverage.¹⁶

CHIRPP captures injury data for all people who present with an injury to an emergency department (ED) of a participating hospital; the HBSC survey captures the “most serious” injury to be treated by a doctor or nurse in a family doctor’s office, a health clinic, a hospital, a school health service, etc. and, since 2002, data on whether the reported injury was treated in an ED (see Table 1 for an overview of the demographics of those with reported sports injuries in the two data sets). This means that HBSC data can be used to estimate the incidence of all treated sports injuries in Canadian children through the percentage of HBSC respondents who reported sustaining any sports injury; the data can also be used to estimate incidence of ED-treated sports injuries. The rate of ED-treated sports injuries should be more directly comparable with the surveillance-based incidence estimate, which is calculated as the number of pediatric sports injuries presenting to the ED per 100 children in the at-risk population. In the CHIRPP data, a sports injury was defined as any injury sustained during sports (organized, practice, lessons and coaching), recreational activities (including tobogganing, trampoline, brownies, scouts, etc.), dancing in the home, playing or climbing.

Table 1. Demographic information for those with reported sports injuries in each data source .

The HBSC data include the “one most serious injury” line of questioning, which (as pointed out earlier) can lead to incomplete reporting of sports injuries, so a “naive” survey-based estimate is really a lower limit on the range of plausible estimates of sports injury incidence. By using additional HBSC data to determine whether respondents sustained multiple injuries within the recall period, we can determine the potential extent of the masking of less serious injuries including less serious sports injuries. This information can be used to identify a possible upper limit for the sports injury incidence estimate. This upper limit is calculated as the percentage of HBSC respondents who (1) reported a sports injury as their one most serious injury, or (2) reported multiple injuries and thus may have sustained a “less serious” sports injury that went unreported.

The CHIRPP data set was restricted to those aged 11 to 15 years at the time of the ED visit in order to match the target age group of the HBSC. Table 2 shows a comparison of the injury data reported in these data sources.

Table 2. Comparison of injury data as reported in the three data sources .

Results

Of the 6688 individuals who responded to the HBSC survey in 2001–2002, 1766 (26.4%) reported that their most serious injury was sustained while playing or training for sports / during a recreational activity (Table 2). However, 1994 children reported multiple injuries and only 1047 of those reported a sports injury as their most serious. Since it is possible that the other 947 children had less serious sports injuries, 14.2% of children may have had sports injuries that went unreported (i.e. they were masked) due to this “one most serious injury” line of questioning. Valid sports injury incidence rate estimates from the HBSC survey, therefore, lie between 26.4 and 40.6 injured children per 100 per year, depending on the extent of masking (Table 3).

Table 3. Sports injury incidence estimates.

If we restrict data to injuries that required ED treatment, we can estimate incidence separately using survey and surveillance data and get a better sense of the true extent of masking. In lthe 12-month recall period, 896 children (13.4% of the HBSC sample) sustained injuries that required ED treatment; 445 of these children reported a sports injury as their most serious injury (6.7% overall). Of the 451 who described their most serious injury as from another subcategory, 287 (4.3% overall) indicated they had had multiple injuries. This means that 4.3% of children could have had masked sports injury incidents (the rate of masking could theoretically be even greater if a child thought an injury that did not require an ED visit was “more serious” than one that did). Valid estimates of the incidence of sports injuries requiring ED visits, therefore, lie anywhere between 6.7 and 11.0 (or more)children per 100 per year (Table 3).

There were 884 visits to the ED as a result of pediatric sports injury from among the 8896 children living in the restricted CHIRPP catchment area—approximately 60.8% (n = 884/1453) of all injuries reported (Table 2). Therefore, our surveillance-based sports injury incidence estimate is 9.9 children per 100 per year, which is larger than the naive surveybased estimate of 6.8, but within the range produced when we recognize the potential masking arising from the “one most serious injury” line of questioning (Table 3). In fact, this larger surveillancebased estimate may still be an underestimation of the true incidence rate, since sports injuries can only be identified in those children whose guardians consent to the collection of detailed information about the injury event.

Discussion

The purpose of this paper is to highlight the existence of masking bias in surveys that gather data on only the “most serious” injury event. The potential impact of masking arising from the “one most serious injury” line of questioning is evident in our estimates of sports injury incidence. In particular, the estimate produced by CHIRPP (9.9 sports injuries per 100 children per year) falls near the top of the range produced by the HBSC ED survey data (6.7–11.0 sports injuries per 100 children per year). This suggests that masking is a potentially serious concern. Using this line of questioning in surveys may reduce recall bias, but incidence rate estimates of injury subcategories will be underestimated due to the masking of less serious injuries; regression estimators and estimators of the percent of injuries falling in a particular subcategory could be biased in either direction as a result of this masking. The extent of the masking and bias will depend on the incidence, relative severity and likelihood of the co-occurrence of specific injury subtypes, as well as on the likelihood of multiple injury occurrences within the recall period.

These factors indicate that data from the “one most serious injury” line of questioning is not appropriate when comparing incidence estimates between subgroups (for example, males versus females) that may have varying degrees of masking. Researchers should consider the potential impact of this masking bias when designing or analyzing injury surveys. Questions about whether other “less serious” injuries were experienced during the recall period could be used to produce a range of injury incidence estimates. Shorter recall periods would capture fewer individuals with multiple injuries, which would lead to fewer events being masked and to less bias. Collecting information on all injury events would eliminate masking entirely. However, this benefit needs to be weighed against the potential drawbacks of such changes to the study design including loss of power and increased recall bias.

In this article, we focus on the circumstances surrounding injury events and demonstrate a clear, but often overlooked, bias associated with estimating incidences when using data from the “one most serious injury” line of questioning. A similar bias occurs when estimating incidence of injury effects (e.g. concussion, broken bone, etc.), however, comparing survey and surveillance data would be more challenging because HBSC respondents can report up to five injury effects while CHIRPP surveillance captures at most three. Note that the direction of bias of incidence estimates is known: the masking bias results in an underestimation of the incidence of injury subgroups. However, studies of association or of relative rates could be biased in either direction if the amount of masking differs in the different exposure groups.

Limitations

The data used for our analyses are from 2001/2002, when comparable survey and surveillance data were readily available. As a result, the sports injury incidence estimates are not current.

The CHIRPP surveillance captures all people presenting to the ED with an injury; detailed information, including the context of the injury, is recorded only for children whose parents consent to be part of the study (overall consent rates at the two sites were 85% at site 1 and 60% for site 2).

The target demographic of the HBSC was students aged 11 to 15 years based on grade, but in actuality children aged 9 years and 19 years are included. To improve comparability, we limited both our CHIRPP and HBSC datasets to just 11- to 15-year-olds.

As CHIRPP surveillance only provides community coverage for Frontenac County, comparing the CHIRPP results to the HBSC national survey is problematic. The demographics captured in our CHIRPP Frontenac County and HBSC national datasets differ in some regards. Specifically, the CHIRPP data had a higher percentage of males (63.6% versus 48.8% in HBSC) and a slightly different age breakdown than the HBSC data (e.g. 17.2% of 11 year olds versus 13.4% in HBSC). Because we know that injury varies by age and sex, this may have some implications for the results. However, the differences are not so significant as to influence our overall finding that sports injury incidence estimates are likely biased and underestimate true incidence if they are underpinned by data derived from survey questions asking about the “one most serious” injury.

Conclusion

In order to design intervention programs and public policy that address key risk behaviours and outcomes in youth, researchers and policy makers must be able to rely on accurate estimates of injury incidence. As the results suggest, the difference between the HBSC study and the CHIRPP surveillance estimates of incidence rates is substantial if we do not account for potential masking in the survey data. The “one most serious injury” line of questioning may overcome some of the traditional biases associated with survey data, but estimates of sports injuries (or any specific subcategory of injuries) should only be used if the inherent masking and potential biases are recognized. Incidences of specific sports injuries will be underestimated by naive analyses of surveys that collect information on only the “one most serious event.” Such analyses should instead report ranges of plausible incidence estimates that account for the potential levels of masking. Masking from the “one most serious” line of questioning should be considered alongside issues such as recall bias when designing future surveys. We also strongly encourage collecting data on whether survey participants experienced other “less serious” events to allow for an exploration of the potential extent of masking.

Conflict of interest

None declared by any author.

Acknowledgements

Akshay Gupta was supported as a summer research intern with the CIHR Child and Youth Injury Prevention Team (http://www.childinjuryprevention.ca) and received support from the Injury Section, Health Surveillance and Epidemiology Division of the Public Health Agency of Canada (Contract 4500307663).

The authors would like to thank Wendy Thompson and Mylène Fréchette of the Public Health Agency of Canada and Dr. Robert Brison (Kingston General Hospital, Emergency Medicine) for their help getting and interpreting the necessary CHIRPP data and providing input. We would also like to thank Don McGuiness, of the South East Local Health Integration Network, for providing the catchment populations.

HBSC and CHIRPP are funded by the Public Health Agency of Canada (http://www.phac-aspc.gc.ca/index-eng .php) and Health Canada (http://www.hc -sc.gc.ca/index-eng.php). HBSC is a World Health Organization–affiliated international collaboration and, in 2002, was coordinated by Dr. Candace Currie (University of St. Andrews, Scotland). The Canadian component is coordinated by Mr. Mathew King and directed by Dr. John Freeman and Dr. William Pickett (Queen’s University, Canada).