Appropriate Use and Reporting of Uncontrolled Case Series in the Medical Literature

Among epidemiological study designs, uncontrolled case series and case reports are the least methodologically robust.¹ The articles of this Series on Epidemiology spend considerable time drawing attention to the methodological problems that can bias causal inference in controlled interventional and observational clinical epidemiological studies. Uncontrolled case series, in addition to potentially suffering from these problems, have the fundamental defect of lacking a contemporaneous comparison group, leaving authors and readers to resort to historical controls or less objective considerations in order to interpret the meaning of the observations. Because of this severe limitation, uncontrolled studies typically receive little attention among epidemiologists. Nevertheless, observational case series make up a substantial proportion of publications submitted to ophthalmic journals which aspire to promulgate generalizable knowledge. Although reports of such studies frequently are rejected, when appropriately used they serve an important and legitimate purpose in furthering medical knowledge, particularly when a question of importance cannot be addressed by other methods because of ethical or logistical constraints or as a first step in clinical investigation.

Studies without “internal” controls can range in rigor from tightly formalized clinical trials (e.g., phase 1 clinical trials—discussed elsewhere²) to single case or small case series reports that are judged newsworthy for some reason. The objective of this manuscript is to discuss some of the situations wherein observational case series or case reports provide an appropriate means toward the generation of generalizable and useful clinical knowledge (see Table 1), and to provide an overview of how reports using this approach can be optimized so as to minimize (or at least identify and consider) potential biases (see Table 2).

Table 1.

Some Appropriate Settings for Use of the Case Series Study Design

Proof (or Disproof) of Concept for a New Hypothesis

Reporting of Sentinel Events Toxicities of Therapies Recognition of Epidemics Initial Identification of Previously Unrecognized Syndromes

Studying Outcomes of Rare Diseases or New Treatments (Limited Usefulness)

Table 2.

Checklist for Reporting Case Series

Explicitly State the Hypothesis/Hypotheses Under Consideration Explicitly Provide Eligibility Criteria for Subjects in the Report Precisely Describe How Treatments Were Administered or Potential Risk Factors Defined Compare Observed Results to Those in an Appropriate External Comparison Group; Discuss Potential Biases Arising from Such Comparison Perform Appropriate Statistics, Ensuring That Assumptions of the Statistical Methods are Reasonable in This Setting Discuss the Biological Plausibility of the Hypothesis in Light of the Report's Observations Explicitly Discuss the Report's Limitations, and How These Limitations Could be Overcome in Future Studies

Appropriate Uses of the Case Series Study Design

Hypothesis Generation and Proof (or Disproof) of Concept

In clinical medicine, the need to investigate an individual case is the business of the day. Thus, it is not surprising that many important hypotheses in clinical epidemiology derive from clinical observations. In this manner, single cases or a series of cases often instigate important agendas of clinical investigation leading to valuable therapeutic applications and scientific paradigms. Initial observations are particularly useful when they fit into a hypothesis with biological plausibility, in which case an important criterion of causal inference already is met. Many brilliant clinicians make major contributions by creating such hypotheses based on their clinical observations (for example, observations of Prof. J. Donald M. Gass³).

Because clinical trials, cohort studies, and even case-control studies require a considerable investment of cost and effort, characterization of a series of patients to provide proof (or refutation) of concept of the hypothesis in question is a logical first step in a research agenda, often required by funding agencies. When these early results are compelling and interesting, it is appropriate to report these results as pilot investigations, admitting the limitations of the method, and recognizing the report as an early step in a line of investigation. Such a series would carry more weight if it did not include the first observations that gave rise to the hypothesis, which would provide some degree of independent support of the initial exceptional observation(s) that provoked the research agenda and would better fit the statistical hypothesis testing paradigm (which requires that observations potentially could refute the hypothesis). In reporting such results, the critical importance of performing a definitive study thereafter must be acknowledged, as there are numerous examples of such studies refuting conclusions based on compelling initial observations (consider the case of grid macular photocoagulation for prevention of complications of age-related macular degeneration⁴). For the testing of hypotheses, case series are an important early step in the process of investigation, but are rarely definitive.

Recognition of Sentinel Events

Prospective studies, including randomized clinical trials, are limited in their ability to identify rare adverse effects of exposures (such as treatments). Adverse event reporting provides an important safety function both during such trials and after new drugs come to market, in order to identify severe adverse effects as soon as possible. Publication of such events plays a critical role in improving the safety of patients who are candidates for the new treatment. There are many examples of important reports of this nature regarding ocular toxicities of drugs.⁵ The World Health Organization has developed a system for assessing potential causality in evaluating drug-side effect associations, which should provide guidance for evaluating potential associations in reports of this nature.⁵

Likewise, observations of unexpected clusters of cases may provide clues to emerging epidemics or recognition of previously unrecognized syndromes. Armenian has provided guidance about how one would evaluate highly unusual cases, in pursuit of an explanation.⁶ A noteworthy example of a paper identifying an emerging epidemic was a small series of 5 cases of an exceptionally rare lung disease (Pneumocystis pneumonia) among a similar group of individuals (homosexual males in Southern California);⁷ this was one of the most influential papers ever written, as the first sentinel report leading to recognition of the worldwide pandemic of AIDS. An example of a report identifying a previously unrecognized syndrome was the first report of birdshot retinochoroiditis⁸—a condition that presumably was present for generations but was not recognized until 1980. In these examples, while there was no comparison group, the observations were compelling either because occurrence of such diseases in the population of interest was known to be vanishingly rare, or because of establishment of proof of concept that the syndrome existed. Prompt reporting of such observations plays an important role in management of disease outbreaks, and recognition of new clinical entities, which can be very important both for population health and clinical practice. Although reporting should await a sufficient number of observations in order to make the point, waiting for a very large amount of observation-time to accumulate in cohorts before reporting such observations would be inappropriate in these circumstances, because of health care providers need to respond promptly to the new information.

As stated previously, follow-up analytic studies should be performed in order to make sure initial conclusions in reports of this nature were correct and to expand on the observations (as was done in these instances). To understand why a compelling set of observations must be considered an exploratory observation rather than confirmation of a hypothesis, remember that if the exceptional observation(s) provoked the hypothesis, then there is no way the hypothesis could have been refuted by those observations. By definition, the observations were hypothesis-generating, rather than an activity involving generation of data to test a hypothesis. Because reports of this nature would not have been published had the results not been exceptional ( publication bias ), further studies generally should be designed to detect a smaller difference than was observed in the initial series.

Studying Outcomes of Rare Diseases or New Treatments

Perhaps the most common form of manuscript encountered by journal editors in ophthalmology is a small case series reporting the outcomes of a novel treatment, or of a rare disease. Most of these series are too small to be of much interest, because the risk of an outcome cannot be estimated precisely unless the series is large and the amount of observation long—in which case the “case series” becomes a cohort study, wherein the case definition defines entry into the cohort. To see this, consider a series of 10 cases that received a novel surgical treatment for a rare disease, four of whom had an early adverse outcome (all patients having the same amount of follow-up, so that an exact binomial confidence interval can be used). The 95% confidence interval on the best estimate of risk (40%) would be 12.1%–73.8%, leaving the reader uncertain as to whether the event is uncommon or highly frequent. Alternatively, consider an alternative scenario, in which zero complications of a new surgical procedure were observed: the 97.5% one-sided upper confidence limit would be 30.8%, supporting up to a 30% risk of complications as plausible, leaving the safety of the procedure very uncertain. Evaluation of candidate risk factors in this situation would have even less precision. On the other hand, if the outcomes of a condition were uniformly dismal, and a series of 10 cases found no instances of a bad outcome, these results would be compelling. Rather than wasting time and energy trying to estimate risk with an inadequate study design, those considering reporting a case series for this purpose should first estimate the precision that it possible based on the data they are likely to find, in order to evaluate the value of such information will provide in comparison with external controls. If an inadequate number of observations are available, collaborative study which allows reasonable sample size goals to be met (for example, a collaborative study of bevacizumab in inflammatory ocular neovascularization⁹) typically will be far more useful than a small “me-first” report. The communication facility presently available to clinicians provides relative ease in pooling rare observations over large numbers of centers to describe rare but meaningful associations that would not be established by single center series due to the limited number of observations.

Reporting of case series

If it is determined that reporting of a case series could add to generalizable knowledge, the reporting should use sound methods and disclose weaknesses, as in any clinical epidemiological report. Appropriate and inappropriate approaches are discussed more fully elsewhere.^10,11 In brief, several desirable characteristics will enhance the reporting of the study (see Table 2). When applying the case series method to a hypothesis, a clear-cut description of that hypothesis is required so that readers can interpret the observations properly. Indeed, the hypothesis also may be a primary factor in establishing why the paper is of interest and should be accepted by the journal. As with any clinical study, sharp definition of the inclusion and intervention criteria—sufficient to allow replication of the study—is necessary so that potential selection biases can be considered. Avoiding selection of patients for inclusion as much as possible by reporting of consecutive patients is a commonly used desirable design feature. Likewise, precise description of how any treatments were applied (hopefully in a uniform manner) and/or how potential risk factors assessed is critically important to make the report interpretable and generalizable. Because the study lacks internal controls, the paper should provide discussion of how the results compare to those of an appropriate external comparison group, as well as a discussion of why an external comparison group was used. Careful description of the external comparison group is needed, along with careful discussion of why such a comparison is unlikely to bias conclusions. As in any study, statistics evaluating the potential contribution of random error to the results observed are required, such as confidence intervals indicating the plausible range of values for risk estimates. Risk estimates must use the proper measure of association for the nature of the data; a common error is to use statistics which assume that the event risk is equal for all patients when in fact it is not (such as the situation where follow-up time at risk of the event differs among the patients).¹¹ As always, the assumptions of the statistical methods and hypothesis testing paradigm used should be met. The paper should clearly discuss and establish the biological plausibility of the explanation invoked to explain the findings. Limitations of the report should be clearly enunciated, and potential approaches to overcoming these limitations should be described during discussion of what confirmatory studies would be appropriate next steps along the line of clinical investigation.

In summary, case series can be highly influential when approached correctly and applied to appropriate settings. Attention to the issues discussed here can provide useful guidance about whether it is worthwhile to embark on reporting a case series, and can improve the quality of such reports.

Biography

Dr. John H. Kempen is Associate Professor of Ophthalmology and Epidemiology at the University of Pennsylvania School of Medicine, in Philadelphia, Pennsylvania, USA. He also holds an appointment as a Senior Scholar in the Center for Clinical Epidemiology and Biostatistics. He serves as Director of both the Ocular Inflammation Service and Ophthalmic Epidemiology for the University of Pennsylvania. His research centers on the effects of treatment for patients with ocular inflammatory and infectious diseases. Dr. Kempen serves as Chairman of the Systemic Immunosuppressive Therapy for Eye Diseases (SITE) Cohort Study, and as Vice-Chairman of the Multicenter Uveitis Steroid Treatment (MUST) Trial.