Editor—The correspondence on the issue of including observational data of harm in systematic reviews surprised us.1,2 The tone of the prose implied that this might happen in the future, and the limits of randomised controlled trials discussed told readers what anyone working in assessing adverse events knows only too well. Observational data have been included in systematic reviews of possible harms for some time, precisely for the reasons that Johnston illustrates for vaccinations.2
An increasing number of potentially damaging allegations of associations between exposure to one or more vaccines and harmful events were made recently. Evidence was scattered, seldom assessed by its methodological quality, and sometimes included in descriptive reviews. We used allegations of harmful events after immunisation with pertussis, measles, mumps, and rubella (MMR), and hepatitis B vaccines to develop methods to identify, assess, and synthesise evidence from studies of different designs, ranging from randomised controlled trials to case-only designs (www.who.int/vaccinessurveillance/ISPP/IssuesofInterest.shtml).3-5
Development of quality assessment criteria for such studies was a worthy challenge. Assessment of possible rare and unforeseen adverse events after vaccination is methodologically particularly difficult because independent controls are lacking in most cases. Most of the population is already vaccinated, and those who are not are likely to be unrepresentative of the reference population. Such difficulties can be overcome by including studies with no independent controls (before and after and case crossover designs) as no single study design is likely to answer the study question (table). All available evidence should probably be assessed and included.
Table 1.
Strengths and weaknesses of studies included in systematic reviews of harmful effects
| Method/study design | Strengths | Weaknesses |
|---|---|---|
| Case report | Early warning | Bias, differing case definitions, lack of comparators |
| Passive and active surveillance | Early warning or detection of rare events | Bias, differing case definitions, lack of comparators |
| Ecological study | Powerful, cheap | Difficulty in interpretation, confounding, bias, differing case definitions |
| Case crossover and case based studies | No need for independent controls | Lack of wide acceptance, bias, differing case definitions |
| Multiple time series | Flexible, powerful | Credibility, bias, differing case definitions |
| Case-control study | Can test hypotheses, especially rare events | Confounding, bias, differing case definitions |
| Cohort study | Powerful, cheap (if retrospective) | Confounding, bias (especially attrition), differing case definitions |
| Historical control study | Powerful, cheap | Bias, differing case definitions, difficulty in interpretation, differing case definitions |
| Randomised and clinically controlled studies | Powerful, minimisation of all biases | Short follow up, limited power, differing case definitions |
Broadening the focus of systematic reviews is not an optional feature that may come about in the future: it's here already.
Competing interests: None declared.
References
- 1.Cuervo LG, Clarke M. Balancing benefits and harms in health care. BMJ 2003;327: 65-6. (12 July.) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Electronic responses. Balancing benefits and harms in health care. bmj.com 2003. bmj.com/cgi/content/full/327/7406/65#responses (accessed 15 September 2003).
- 3.Jefferson TO, Rudin M, Di Pietrantonj C. Systematic review of the effects of DTP vaccines in children. Vaccine 2003;21: 2012-23. [DOI] [PubMed] [Google Scholar]
- 4.Demicheli V, Rivetti A, Di Pietrantonj C, Clements CJ, Jefferson T. Hepatitis B vaccination and multiple sclerosis—Evidence from a systematic review. J Viral Hepatitis 2003. (in press). [DOI] [PubMed]
- 5.Jefferson TO, Price D, Demicheli V, Bianco E. Unintended events following immunisation with MMR: a systematic review. Vaccine (in press). [DOI] [PubMed]