Impact of Small-N Studies During a Pandemic

Randomized controlled trials, the “gold standard” of evidence-based medicine, derive their strength from strict adherence to methods such as randomization and blinding, which minimize bias and external influences on study data. Even with rigorous methodology, the risk of random error caused by intrinsic data variations remains, but it can be mitigated by increasing sample size.¹ In the midst of a nascent pandemic, adequate sample size and other commonly accepted standards of medical research have been set aside for various reasons, including that of single-center attempts at studies that require more subjects than they can recruit on their own. In this article we explore the implications of clinical studies that proceed with small sample sizes during disease outbreaks.

Calculating ideal sample size for a study requires four components: alpha, power, minimal clinically relevant difference (MCRD), and estimated variability in the samples and the effect of the intervention. In a priori sample size calculations, statisticians use set values for alpha (the predetermined threshold for type 1 error and statistical significance), power (the complement of type 2 error, representing the ability of a test to correctly identify true effects), a predefined value for MCRD, and estimated variability in the data parameters to determine the ideal sample size.² Early in a disease outbreak, when there are low numbers of affected subjects, achieving adequate sample size in a single center is challenging.

With a limited sample size, one of the other generally predetermined values in the sample size calculation will necessarily vary. Because estimated sample size is inversely proportional to the square of the expected difference,² adjustment of the MCRD to a larger value will accommodate a smaller sample size but also prevent detection of smaller effects, potentially compromising the basic utility of the test. Estimated variability is usually predicted using data from a pilot study or previously performed study.² It cannot be changed arbitrarily, but it is difficult to assess early in an outbreak. Studies with small samples are inherently susceptible to effects of random error, so researchers conducting them are unlikely to allow changes to alpha, thereby reducing possibilities of reporting that any discoveries claimed were merely a result of chance. With these three components fixed, power inevitably becomes variable, and the smaller the sample size, the less powerful that study will ultimately be. Underpowered studies suffer from decreased probability of finding true effects (leading to false-negative results), low positive predictive value when an effect is claimed (false discovery rate), and inflated estimates of effect magnitude, all making results less likely to be reproducible.³

Attempts at single-center studies with limited sample size early in a pandemic result in the need for multiple clinical trials, frequently without control arms. Although forgoing the use of controls may ease the pressure of sample size constraints, it also makes studies more anecdotal and further weakens the credibility of their results. Meanwhile, as the infection spreads and mortality rates grow, the public becomes ever more desperate for some reprieve and are further inclined to acquiesce to potential safety concerns at any indication of efficacy, proven or not. During the HIV-AIDS pandemic, intense political pressure was put on researchers to permit expanded public access to experimental therapies with unproven efficacy in the face of near certain death.⁴ However, the largest controlled trial of one such therapy, zidovudine, ultimately demonstrated that the drug resulted in substantial side effects to HIV-infected patients, perhaps even death, with no beneficial effect.⁵ This example illustrates that the challenges posed by disease outbreaks may lower societal thresholds for safety and efficacy, resulting in decisions with considerable downstream effects and decreased levels of confidence. This reduction in scientific rigor, in turn, yields inopportune endorsement and use of treatments with overestimated safety and efficacy or, conversely, premature and errant abandonment of useful interventions.

As the coronavirus disease 2019 (COVID-19) pandemic burgeoned, the medical community encountered many of the challenges described. As seen in other disease outbreaks, issues related to the utility and implications of small-N studies with initially promising results emerged. The investigation into the antimalarial and immunomodulatory drug hydroxychloroquine was an example of this. Preliminary reports based on limited data were encouraging, and there was quick clinical adoption despite unproven efficacy. The decision to fast-track the use of this drug was based on early reports of its in vitro and in vivo efficacy against the virus and the disease it causes.

Several important caveats of these studies were seemingly overlooked or dismissed. The in vitro study indicated that the effective concentrations of hydroxychloroquine required for prevention of SARS-CoV-2 replication⁶ far exceed targeted human serum concentrations when used for traditional indications, suggesting the need for high-dose application of the drug. However, hydroxychloroquine has a narrow therapeutic window, and the use of high doses to approximate the concentrations found to be effective in vitro would put patients at increased risk of serious side effects. A small nonrandomized clinical trial of 36 patients to test the efficacy of hydroxychloroquine with azithromycin was conducted in March 2020. The researchers reported 100% viral clearance in patients who received both drugs, 57.1% clearance in those treated with hydroxychloroquine only, and 12.5% in the control group.⁷

The public dissemination of this small-N trial led to an abrupt surge in demand for hydroxychloroquine and its widespread use as a treatment for COVID-19 in the United States. This, in turn, led to shortages for patients chronically maintained on this medication. Subsequently, data from several larger studies have contradicted the results of the original small study, indicating that this use of hydroxychloroquine did not benefit COVID-19 patients and may have actually caused harm.^{8 , 9} As a result, the National Institutes of Health review panel, which initially indicated that there were insufficient data to either support or discourage this application of the drug, now recommends against its routine use for COVID-19 outside of clinical trials.¹⁰

Disease outbreaks disrupt the highly standardized practices of medical research that are stringently maintained under normal circumstances. One aspect of study design that becomes particularly vulnerable in such times is sample size. Ideally, research outputs during a pandemic should bring forth accurate answers to an appropriate exploratory question that can be applied to the specific population, and this cannot be achieved by small-N studies. Small sample size may, however, be the norm when researchers restrict enrollment to their own center. We suggest that outbreaks, rather than disrupting standard practices, provide enhanced opportunity for collaboration among academic centers, with the goal of providing reliable information for physicians and effective therapies for patients.