Abstract
Objective
To study whether the methods used to select participants in research on prevalence rates of the health effects of exposure to a disaster may cause bias. This study compared background characteristics, disaster exposure, and complaints reported by (1) police officers who participated in an epidemiological study and underwent a medical examination, and (2) police officers who only participated in an epidemiological study.
Design and Setting
In 2000, an epidemiological study was started to investigate the health status among police officers who were involved in assistance work related to the air disaster in Amsterdam: the epidemiological study air disaster Amsterdam (ESADA). These police officers were personally invited to participate in the epidemiological study and were additionally offered a medical examination, providing them with the opportunity to have their health checked by a medical doctor.
Participants
Of the total group of involved police officers who were invited, 834 (70%) participated in the ESADA: 224 (26.9%) only participated in the study, but 610 (73.1%) also underwent the medical examination.
Main results
Police officers who underwent a medical examination significantly more often reported one or more musculoskeletal complaints, skin complaints, general or non‐specific complaints, had more often experienced an event with potentially traumatic impact, or had performed one or more potentially traumatic tasks.
Conclusions
The methods of selection of participants are important in research on health effects after disasters and can result in an overestimation of some of the effects, on average, by a factor of 1.5 to 2.
Keywords: selection bias, disaster research
Epidemiological studies investigating adverse health effects after exposure to a disaster are often hampered by a variety of methodological limitations.1 Some of these limitations are a logical consequence of the nature of disasters—that is, their unexpected occurrence with varying impact and exposures. In addition, the collection of epidemiological data with respect to exposures and health outcomes is often of lower priority in the immediate aftermath. This may result in lower response rates, compared with community surveys.1
As a consequence of all these factors that complicate disaster research, several epidemiological studies have focused on data that were collected during medical examinations, offered to the affected persons.2,3 However, the key question in these convenience samples is whether the participants were systematically different from the population they were intended to represent. Selection bias and reporting bias may affect estimates of the prevalence of health effects.4
Our aim is to study whether the methods used to select participants in research on the health effects of exposure to a disaster may cause bias. This study is nested within an epidemiological study that investigated the health status of disaster workers 8.5 years after they had been involved in work related to the air disaster in Amsterdam, in comparison with the health status of colleagues in reference groups. The air disaster took place on 4 October 1992, when a cargo aircraft crashed into two apartment buildings in a densely populated suburb of Amsterdam (Netherlands). In this study we focused on involved police officers only and compared the disaster exposure and background characteristics and health outcomes of involved police officers who participated in the epidemiological study, and chose to undergo a medical examination, with those of involved police officers who only participated in the epidemiological study (and decided not to undergo a medical examination). We expect that persons with adverse health behaviour (smokers), disadvantageous background characteristics (low socioeconomic status, older age, single status), a higher self reported degree of exposure to the disaster, and more complaints are more likely to attend the medical examination.5,6
Methods
Study design and population
The study design and population have been described in detail elsewhere.7 This study is nested within the ESADA, which is a historical cohort study. The main purpose of the ESADA was to investigate the health status of police officers and fire fighters 8.5 years after they had been involved in assistance work related to the air disaster in Amsterdam, and to compare their health status with that of colleagues in reference groups. These groups of disaster workers were personally invited to participate. A medical examination was also offered to all those who were involved in the disaster, providing them with the opportunity to have their health checked in one consult with a medical doctor. Participants in the ESADA were asked to fill in questionnaires concerning their physical and mental health. Both the ESADA and the medical examination took place on average 8.5 years after the disaster.7
For this study, we focused on involved police officers only. We included (1) police officers who participated in the ESADA, and underwent a medical examination, and (2) police officers who only participated in the ESADA.
Measures
The background characteristics addressed in the questionnaires were7: age, sex, ethnicity, highest level of education completed, marital status, and cigarette smoking.
The International Classification of Primary Care (ICPC)8 was used to classify the physical symptoms reported by the participants in various questionnaires, resulting in the following dichotomised symptom categories (1 = one or more complaints compared with 0 = no complaints): (1) general and non‐specific, (2) digestive system, (3) cardiovascular system, (4) musculoskeletal system, (5) nervous system, (6) respiratory tract, (7) skin, (8) appetite, and (9) urinary tract. In addition, the 20 item checklist individual strength (CIS),9,10 symptom checklist‐90 (SCL‐90),11,12 and self rating inventory for post‐traumatic stress disorder (SRIP)13,14 were used to assess fatigue symptoms, general mental health, and post‐traumatic stress symptoms.
Disaster related tasks and events were assessed by means of a questionnaire on occupational exposure to the air disaster.7 This questionnaire addressed specific disaster related tasks and events, including those with potentially traumatic impact (A1 tasks and events), based on criterion A1 of the diagnostic criteria for post‐traumatic stress disorder.15
Statistical analysis
Univariate logistic regression analyses were applied to investigate the association between background characteristics, disaster related tasks and events and health complaints, and participation in a medical examination. In the multivariate model, all measures (background characteristics, ICPC symptom categories, the CIS score, SCL‐90 subscales, the SRIP score, and disaster related tasks and events) were included and by using stepwise backward elimination (p⩾0.10), a final multivariate model was derived. Odds ratios (ORs) and the 95% confidence intervals (95% CI) were estimated. The analyses were carried out in SPSS, version 12.0, and p<0.05 (two tailed) was regarded as significant.
Results
Descriptives
Of all the police officers involved in the air disaster who were invited, 834 (70%) participated in the ESADA: 224 (26.9%) of them only participated in the ESADA, but 610 (73.1%) of them also underwent the medical examination.
Univariate and multivariate analyses
Table 1 presents the results of the univariate and multivariate logistic regression analyses. The results of the multivariate logistic regression analyses showed that police officers who had reported one or more general or non‐specific complaints, musculoskeletal complaints, skin complaints, had experienced one or more A1 event, or had performed one or more A1 task were significantly more likely to undergo the medical examination.
Table 1 Univariate and multivariate logistic regression analyses of determinants associated with the participation in both the medical examination and the epidemiological study.
| Determinant | Participants underwent a medical examination | Univariate logistic regression analysis | Multivariate logistic regression analysis | |
|---|---|---|---|---|
| Yes (n = 610) in %* | No (n = 224) in %* | OR (95% CI) | OR (95% CI) | |
| Background characteristics | ||||
| Sex (male) | 88.5 | 88.4 | 1.01 (0.63, 1.64) | – |
| Ethnicity (European) | 96.9 | 98.2 | 1.77 (0.60, 5.26) | – |
| Age (in years) | 43.9 (6.0) | 44.4 (6.7) | 0.99 (0.96, 1.01) | – |
| Smoking (yes) | 34.8 | 36.6 | 0.92 (0.67, 1.27) | – |
| Level of education (low‐moderate) | 77.0 | 80.0 | 0.84 (0.57, 1.23) | – |
| Married/cohabitation (yes) | 90.2 | 89.7 | 0.95 (0.57, 1.58) | – |
| ICPC (presence of complaints) | ||||
| General and not‐specified complaints (yes) | 46. 4 | 24.1 | 2.73 (1.93, 3.85) | 1.97 (1.33, 2.92) |
| Digestive system (yes) | 23.3 | 16.1 | 1.59 (1.06, 2.37) | – |
| Eye complaints (yes) | 16.3 | 6.3 | 2.92 (1.63, 5.22) | 1.80 (0.97, 3.36) |
| Cardiovascular (yes) | 27.9 | 19.6 | 1.58 (1.09, 2.30) | – |
| Musculoskeletal (yes) | 46.9 | 29.1 | 2.15 (1.54, 2.98) | 1.52 (1.05, 2.19) |
| Neurological (yes) | 46.7 | 45.1 | 1.39 (1.02, 1.89) | – |
| Respiratory (yes) | 33.4 | 19.6 | 2.06 (1.42, 2.98) | 1.49 (0.98, 2.25) |
| Skin (yes) | 58.0 | 35.0 | 2.57 (1.87, 3.54) | 1.98 (1.40, 2.80) |
| Appetite (yes) | 35.7 | 35.3 | 1.02 (0.74, 1.41) | 0.71 (0.50, 1.02) |
| Urinary tract (yes) | 3.6 | 1.8 | 2.06 (0.70, 6.04) | – |
| CIS‐fatigue (total score>76) | 20.0 | 7.6 | 3.04 (1.79, 5.19) | – |
| SCL‐90† | ||||
| Agoraphobia | 8.5 | 7.6 | 1.14 (0.64, 2.01) | – |
| Anxiety | 33.8 | 26.0 | 1.45 (1.03, 2.05) | – |
| Depression | 25.0 | 13.8 | 2.07 (1.36, 3.16) | – |
| Somatisation | 36. 9 | 20.1 | 2.33 (1.62, 3.36) | – |
| Thought insufficiency | 30.9 | 16.1 | 2.33 (1.57, 3.47) | – |
| Interpersonal sensitivity | 13.1 | 8.9 | 1.54 (0.92, 2.58) | – |
| Hostility | 45.9 | 33.9 | 1.65 (1.20, 2.27) | – |
| Sleeping problems | 51.1 | 40.6 | 1.53 (1.12, 2.09) | – |
| SRIP (PTSS>39) | 7.7 | 3.1 | 2.59 (1.15, 5.81) | – |
| Disaster related tasks or events | ||||
| One or more A1 event | 53.9 | 29.9 | 2.74 (1.98, 3.81) | 2.26 (1.58, 3.23) |
| One or more A1 task | 25.7 | 12.6 | 2.41 (1.56, 3.72) | 1.72 (1.06, 2.79) |
OR, odds ratio reflecting the odds of participation in both the medical examination and the epidemiological study; CI, confidence interval; – eliminated from the multivariate model. *Values are percentages unless shown as otherwise; †SCL‐90, scores above the 65th centile of the normal Dutch population were used to dichotomise subscale scores.
What is already known
Epidemiological studies investigating adverse health effects after exposure to a disaster are often hampered by a variety of methodological limitations, including the use of convenience samples. These previous studies have focused on data that were collected during medical examinations, offered to the affected persons. The key question in these convenience samples is whether the participants were systematically different from the population they were intended to represent.
What this paper adds
This study examined whether the methods used to select participants in research on the health effects of exposure to a disaster may cause bias. The results showed that an overestimation of the effects by a factor of 1.5 to 2 may result if only those participants who underwent a medical examination after a disaster are included in an epidemiological study.
Discussion
Police officers who were involved in assistance work related to the air disaster more often felt the need to undergo a medical examination in addition to participation in the ESADA, if they had reported general or non‐specific, musculoskeletal, or skin complaints, if they had performed at least one task that involved potential exposure to trauma, or had experienced at least one traumatic event.
In contrast with our expectations, participants in the epidemiological study with adverse health behaviour (smokers), disadvantageous background characteristics (low socioeconomic status, older age, single status), and psychological complaints were not more likely to undergo the medical examination. Most psychological symptoms were not associated with participation in the medical examination. However, our results suggest that police officers who were confronted with potentially traumatic tasks or events, more often underwent a medical examination.
Thus, the methods used to select participants in research on the health effects of exposure to a disaster may cause bias. Our study shows that an overestimation of some of the effects by a factor of 1.5 to 2 may result if only those participants who underwent a medical examination after a disaster are included in an epidemiological study. In contrast, including persons who participated in the epidemiological study only, might underestimate prevalence rates of health effects. One limitation of this study is the lack of information regarding police officers who did not participate in the epidemiological study, but only underwent a medical examination. We would expect that our epidemiological results would underestimate health effects among those who only underwent a medical examination. Moreover, our study was not carried out in a primary health care setting and therefore, we cannot make recommendations nor can we draw any conclusions on primary care delivery to disaster workers from the data we have.
Policy implications
To estimate the prevalence of health effects after disasters, relevant samples of the affected population should be included and not only those person who participate voluntarily in a medical examination.
Caution should be taken when interpreting the prevalence rates of health problems after disasters in studies that include data collected of participants at medical examinations only. It could be that, on average, those with more health complaints more often tend to undergo a medical examination than those with fewer health complaints.
For future studies, we recommend to include representative samples of all those who were involved in a disaster to assess prevalence rates of health consequences.
Footnotes
Funding: the study was funded by the Ministry of Health, Welfare and Sports, the City of Amsterdam, the Amsterdam‐Amstelland regional police force, and KLM Royal Dutch Airlines.
Competing Interest: none declared.
References
- 1.Logue J N, Hansen H, Struening E. Some indications of the long‐term effects of a natural disaster. Public Health Rep 19819667–79. [PMC free article] [PubMed] [Google Scholar]
- 2.Bowler R M, Mergler D, Huel G.et al Psychological, psychosocial, and psychophysiological sequelae in a community affected by a railroad chemical disaster. J Trauma Stress 19947601–624. [DOI] [PubMed] [Google Scholar]
- 3.Singh B, Raphael B. Post disaster morbidity of the bereaved. A possible role for preventative psychiatry? J Nerv Ment Dis 1981169203–212. [DOI] [PubMed] [Google Scholar]
- 4.Vlahov D, Galea S. Invited commentary: Considering bias in the assessment of respiratory symptoms among residents of Lower Manhattan following the events of September 11, 2001. Am J Epidemiol 2005162508–510. [DOI] [PubMed] [Google Scholar]
- 5.Wilson K, Elliott S, Law M.et al Linking perceptions of neighbourhood to health in Hamilton, Canada. J Epidemiol Community Health 200458192–198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Strine T W, Okoro C A, Chapman D P.et al Health‐related quality of life and health risk behaviors among smokers. Am J Prev Med 200528182–187. [DOI] [PubMed] [Google Scholar]
- 7.Slottje P, Huizink A C, Twisk J W.et al Epidemiological study air disaster in Amsterdam (ESADA): study design. BMC Public Health 20053054. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lamberts H, Wood M. eds. ICPC: International classification of primary care. Oxford: Oxford University Press, 1987
- 9.Bultmann U, de Vries M, Beurskens A J.et al Measurement of prolonged fatigue in the working population: determination of a cut‐off point for the checklist individual strength. J Occup Health Psychol 20005411–416. [DOI] [PubMed] [Google Scholar]
- 10.Vercoulen J H M, Alberts M, Bleijenberg G. De Checklist Individual Strength (CIS). Kort instrumenteel [The checklist individual strength (CIS). Gedragstherapie 19991131–136. [Google Scholar]
- 11.Derogatis L R, Lipman R S, Covi L. SCL‐90: an outpatient psychiatric rating scale–preliminary report. Psychopharmacol Bull 1973913–28. [PubMed] [Google Scholar]
- 12.Arrindell W A, Ettema J H M.Handleiding bij een multidimensionele psychopathologie indicator [Manual for a multi‐dimensional indicator of psychopathology] Lisse: Swets and Zeitlinger, 1986
- 13.Hovens J E, van der Ploeg H M, Bramsen I.et al Test‐retest reliability of the self‐rating inventory for posttraumatic stress disorder. Psychol Rep 200087735–737. [DOI] [PubMed] [Google Scholar]
- 14.Hovens J E, van der Ploeg H M, Bramsen I.et al The development of the self‐rating inventory for posttraumatic stress disorder. Acta Psychiatr Scand 199490172–183. [DOI] [PubMed] [Google Scholar]
- 15.APA Diagnostic and statistical manual of mental disorders‐IV‐TR. Washington, DC: APA, 2000