The scandal of poor epidemiological research

Short abstract

Reporting guidelines are needed for observational epidemiology

Something surely must be wrong with epidemiology when the new editors of a leading journal in the field entitle their inaugural offering, “Epidemiology—is it time to call it a day?”¹ Observational epidemiology has not had a good press in recent years. Conflicting results from epidemiological studies of the risks of daily life, such as coffee, hair dye, or hormones, are frequently and eagerly reported in the popular press, providing a constant source of anxiety for the public.²,³ In many cases deeply held beliefs, given credibility by numerous observational studies over long periods of time, are challenged only when contradicted by randomised trials. In the most recent example, a Cochrane review of randomised trials shows that antioxidant vitamins do not prevent gastrointestinal cancer and may even increase all cause mortality.⁴,⁵

Now Pocock et al describe the quality and the litany of problems of 73 epidemiological studies published in January 2001 in general medical and specialist journals (p 883).⁶ Perhaps the most relevant findings relate to how investigators dealt with confounding, multiple comparisons, and subgroup analyses. Confounding, the situation in which an apparent effect of an exposure on risk is explained by its association with other factors, is probably the most important cause of spurious associations in observational epidemiology. For example, a recent meta-analysis of observational studies shows how confounding could have been responsible for the belief that hormone replacement therapy provides protection against cardiovascular disease.⁷ A protective effect of hormone replacement therapy was evident in studies that did not control for socioeconomic status, but not in studies that did (figure).⁷ Higher socioeconomic position is strongly associated with both more frequent use of hormone replacement therapy and lower risk of coronary heart disease. In the large (unconfounded) Women's Health Initiative randomised trial hormone replacement therapy had no beneficial effect on cardiovascular disease.⁸

Figure 1.

Meta-analysis of cohort studies and case-control studies of hormone replacement therapy and coronary heart disease. There is little evidence for a protective effect when analyses are adjusted for, in contrast to studies not adjusted for, socioeconomic status. Adapted from Humphrey et al, reference 7

Worryingly, Pocock et al find that the rationale behind the choice of confounders is usually unclear, and that the extent of adjustment varies greatly. They also confirm that observational studies often consider several exposures, outcomes, and subgroups. This results in multiple statistical tests of hypotheses and a high probability of finding associations that are statistically significant but spurious. In many studies 20% or more of the findings may be erroneous, rather than the expected 5% false positive associations (P < 0.05).⁹

Epidemiology is not alone: and there is hope. As Altman pointed out 10 years ago, much medical research is of poor quality.¹⁰ Efforts so far to remedy what Altman described as the “scandal of poor medical research” have focused on controlled clinical trials. Empirical research has shed light on aspects of methodological quality that are crucial to prevent bias,¹¹ and Consolidated Standards for Reporting Trials (CONSORT) have been developed and endorsed widely.¹²

Clearly, such guidelines are also required for observational epidemiology. A month ago a group of epidemiologists, statisticians, and editors met in Bristol to work on a first draft of STROBE, the Standards for the Reporting of Observational Studies in Epidemiology. STROBE will provide guidance on the reporting of cohort studies, case-control studies, and cross sectional studies; a supporting document will give explanations and examples. Outcomes from the Bristol workshop will be posted on a dedicated website (www.strobe-statement.org), and you and everyone else will be invited to comment and suggest improvements before a revised version is published some time next year.

Can STROBE do for observational epidemiology what CONSORT does for randomised clinical trials? Not exactly. Both guidelines aim to promote comprehensive reporting of important methodological detail and facilitate appraisal of study quality. In the case of a large high quality randomised trial, this means that the results can be assumed to provide an unbiased estimate of the treatment effect in the population studied. This is not so in observational epidemiology. A well conducted case-control or cohort study might still produce misleading results if, for example, important confounders were not known, not measured, or imprecisely measured.

Importantly for observational studies, STROBE will also pay considerable attention to what investigators should write in the discussion section of their paper by suggesting the inclusion of statements on why methodological approaches were chosen, and why results are interpreted the way they are. The skilful interpretation of epidemiological evidence, bearing in mind theoretical considerations, resisting being seduced by possible mechanisms, and suspending beliefs to allow healthy scepticism will remain the great challenge and joy of working in epidemiology.⁸ More transparent and complete reporting of epidemiological studies, coupled with more thoughtful interpretation of their results, will help restore the reputation of a discipline that has contributed importantly to improving the health of the public, and will continue to do so in the future.