A Focus on Evaluating Major Study Limitations in Order to Apply Clinical Trials to Patient Care: Implications for the Healthcare Team

Abstract

Background: With more than a million new biomedical articles published annually, healthcare providers must stay up to date in order to provide optimal evidence-based patient care. The concise ROOTs (relevance, observe validity, obtain clinically significant results, and translate results to clinical practice) format is a valuable tool to assist with literature evaluation. Purpose: To illustrate how major study limitations found in clinical trials might inhibit the ability to adopt the findings of such studies to patient care. Methods: Examples from published clinical trials that contain major study flaws were used to illustrate, if taken at face value, would lead to erroneous assumptions, and if adopted, could potentiallly harm patients. Conclusion: When evaluating the literature, it is crucial to identify limitations in the published literature that might reduce the internal validity, affect the results, or limit the external validity of clinical trials, hence affecting the usability of literature for patient care. This article provides examples of clinical trials that contain major study limitations with potentially erroneous assumptions. These illustrations are meant to show how important it is to delve deeper into an article before conclusions are drawn.

Introduction

The amount of information published annually in the biomedical literature, especially clinical trials, continues to expand which makes it a challenge for healthcare professionals to stay informed in order to provide optimal evidence-based patient care. The ROOTs (relevance, observe validity, obtain clinically significant results, and translate results to clinical practice) format was developed to streamline the literature evaluation process of clinical trials. It provides practitioners a guide to evaluate new information and assess relevance to a clinical situation. ¹ While this tool was originally intended for journal club application, the evaluation process is equally useful for the sole practitioner or clinician reading and evaluating independently.

In the ROOTs manuscript, it mentions a few examples of study limitations that could deter extrapolation of study results to patient care, and that this piece is one of the principle shortcomings of an effective journal club. ¹ Therefore, this article will provide examples of clinical trials published in the medical literature that contain major study limitations (see Table 1) to demonstrate why the evaluation of literature is so important in assuring against erroneously adopting conclusions which are not founded on good science. The table is divided into three sections, limitations that affect internal validity, results, and external validity to assist with organizing the evaluation process. Internal validity refers to the quality of a study’s methods or study design and the subsequent confidence readers can have in the results. External validity is the degree to which the study results are meaningful to practitioners and useful for patient care. ² It should be noted that the internal validity of a study should be analyzed first prior to examining external validity because deficits in the study design might limit who the results can be applied to in the general population. The section on limitations that affect results contain errors that directly affect interpretation of the study results. Through the illustrations, clinicians should be able to accurately determine the utility of published clinical trials to improve patient care by incorporating evidence-based medicine into practice. For pharmacists working at the bedside, in clinics, in the community environment, in a managed care organization, or on a Pharmacy and Therapeutics committee, accurately evaluating study data provides support for a rational discussion to potentially negate incorrect conclusions or to provide evidence to champion why and in what circumstances a treatment makes sense. After all, it is the role of the pharmacist to promote rational and safe medication use.

Table 1.

Examples of Potential Major Clinical Trial Limitations.

Limitations that reduce internal validity
Study rationale or objective not clearly defined
Lack of randomization or randomization concealment
Unbalanced baseline characteristics
Study is not blinded, or blinding is not maintained by an adequate placebo or double-dummy
Inclusion/exclusion criteria that are too restrictive or vague
No active comparator
Treatment regimen not appropriate: comparison with suboptimally dosed control or not the standard
Treatment regimen not appropriate: intervention with suboptimally dosed treatment
No placebo control
Lack of assessment of ancillary medications that could affect outcomes
Failure to collect or report adherence
High dropout rate
Inappropriate study design (e.g., crossover, intention to treat/per protocol)
Trial stopped early
Limitations that affect results
Study duration inadequate
Small sample sizes
Wrong statistical test or tests used
Absence of inferential statistics for comparison of arms (only descriptive statistics used)
Lack of power or not reported especially when the primary outcome is not significant
Secondary outcomes show positive findings and conclusions are drawn
Secondary outcomes show no benefit when primary outcome is positive
Pharmaceutical manufacturer over-involvement*
Limitations that reduce external validity
Inclusion/exclusion criteria that are too restrictive or vague
Inappropriate study design (e.g., single-centered)
Primary outcome not appropriate, the preferred measure or accurately defined
Use of surrogate outcome measures when not proven to be tied to clinical outcomes or when clinical outcomes are the standard
Results statistically significant but the smallest effect of clinical interest not achieved (overpowered)
Number needed to treat/harm not clinically relevant
Sensitivity analyses showing minimal clinical benefit (e.g., wide confidence intervals)

^*Must be in conjunction with other major study limitations to impact results.

Limitations That Reduce Internal Validity

Treatment Regimen Not Appropriate: Comparison With Suboptimally Dosed Control or Not the Standard

One aspect of a clinical trial that should be of special importance to pharmacists is that study drugs are given at therapeutically appropriate doses. If not, it is impossible to determine if the outcome would have been different if appropriate doses were administered. Berman et al ³ conducted a study to determine if aripiprazole was effective as adjunctive therapy in patients with major depressive disorder. The rationale for the study was that 60% of patients do not respond to at least one antidepressant when given at an adequate dose. In the study, patients with an inadequate response to standard antidepressant therapy were given aripiprazole or placebo. Results showed that patients in the aripiprazole group had a significant decrease in the primary outcome compared to placebo. The methods stated that patients received one of five different antidepressants at adequate doses. However, in the results, the average doses were significantly lower than the standard dosing in all but one of the five medications. If the patients had truly received an adequate dose, the beneficial effects of adjunctive aripiprazole might not have been significantly noted.

Treatment Regimen Not Appropriate: Intervention With Suboptimally Dosed Treatment

Suboptimal dosing of the control arm will bias the results in favor of the intervention arm while suboptimal dosing of the treatment arm may result in a false negative conclusion. Kollef et al conducted a noninferiority (NI) trial to compare a 7-day doripenem infusion to an imipenem-cilastatin infusion for patients with ventilator-associated pneumonia (VAP). ⁴ NI was not established as the study was terminated early after randomizing only 274 patients due to higher rates of clinical failure and mortality in the doripenem compared to the imipenem arm. It is most likely that the failure of doripenem to show NI to imipenem was due to a lack of an advised loading dose^5,6 when administering doripenem as a 4-hour extended infusion strategy whereas imipenem was administered using the shorter 1-hour standard infusion that does not require a loading dose. Based on this study, the FDA unfortunately changed the doripenem label in January 2012 warning that doripenem “carries an increased risk of death and lower clinical cure rates compared to use of imipenem and cilastatin.” ⁷ It is now known that 7 days of therapy is sufficient for most patients with VAP as recommended by the 2016 IDSA guideline for VAP. ⁸ It is more likely that the failure of doripenem to show NI to imipenem was due to a lack of advised loading dose as previously stated.

No Placebo Control

A placebo control, when appropriate, is essential to provide an accurate estimate of the treatment effect, especially when there is no established standard of care. A placebo group is also essential for establishing the safety profile of new treatments. The ION-1 trial included four arms evaluating the efficacy and safety of sofosbuvir/ledipasvir with or without ribavirin in treatment-naïve patients with hepatitis C virus genotype 1 infection. ⁹ A similar pattern of adverse effects was observed in all groups, with the most common adverse effects of fatigue, headache, insomnia, and nausea. While these adverse effects were attributed to sofosbuvir/ledipasvir, it is important to note that none of the four arms was a placebo. In contrast, the C-EDGE trial included two arms evaluating the efficacy and safety of grazoprevir/elbasvir compared to placebo. ¹⁰ A similar pattern of adverse effects was observed including in the placebo group, with the most common adverse effects of fatigue, headache, and nausea. It is more likely that these adverse effects are attributable to hepatitis C infection rather than the drugs. Conclusions should be developed with caution when determining the adverse effect profile of a new treatment in the absence of a placebo control.

Inappropriate Study Design (e.g., Crossover, Intention to Treat/Per Protocol)

In a crossover or repeated measures study design, subjects serve as their own control. Patients receive a study treatment, experience a period with no drug treatment or placebo (i.e., washout period) to prevent a carryover effect from one arm to the next, then receive a different therapy. Chronic diseases, such as mild to moderate hypertension would be appropriate to use a crossover design if an adequate washout period (estimated to be four to five half-lives of a medication) is utilized between different drug administration. Vinson et al conducted a crossover study to determine the efficacy of a green coffee bean extract to placebo as a weight loss agent. ¹¹ Since weight should not be expected to return to baseline during a washout period, a crossover design would not be appropriate for this study and would confound the results. In addition, the authors’ blinding methods were questionable, as one of the arms received the treatment twice a day while the other two arms received their medication three times a day, and it was unclear whether the placebo was a true placebo (i.e., color, taste, appearance, smell, texture, shape, size, weight, packaging).

Trial Stopped Early

Occasionally, it is necessary to terminate a trial early if a group is unlikely to benefit from treatment or if patients are likely to be harmed with study continuation. During the course of a study, treatment effects are subject to highs and lows that confound the true treatment effect. ¹² If a large treatment effect occurs early in the trial and as a result the study is stopped, the effect size will be highly exaggerated and deviate from the true effect size. The MERINO trial was such an NI trial. It compared piperacillin/tazobactam to meropenem for definitive treatment of bacteremia caused by ceftriaxone-resistant E. coli or K. pneumoniae. ¹³ The rate of 30-day mortality was 12.3% in the piperacillin/tazobactam group and 3.7% in the meropenem group which failed to show NI. The study was terminated early after an interim review since it was determined that NI was unlikely to be demonstrated. A subsequent systematic review determined that the rate of mortality was 15.2% with definitive carbapenem treatment and 16.2% with definitive beta-lactam/beta-lactamase inhibitors. ¹⁴ It is likely that early termination of the MERINO trial resulted in unexpectedly low mortality rates in the meropenem group, and hence exaggerating the treatment effect and potentially truncating secondary and safety outcomes.

Study Limitations That Affect Results

Absence of Inferential Statistics for Comparison of Arms (Only Descriptive Statistics Used)

Inferential statistics are crucial in clinical trials to provide a quantitative method to reject or accept the null hypothesis using P-values. This provides proof (or absence) of causation rather than association of the treatment effect when guarding against false positives or negatives. In the absence of inferential statistics and a direct biological association, concluding causation is inappropriate. The ION-2 trial evaluated the efficacy of sofosbuvir/ledipasvir with or without ribavirin for either 12 or 24 weeks (four possible combinations) in patients with hepatitis C virus genotype 1 infection who had failed prior treatment. ¹⁵ While the virologic response rate in each of the four arms was shown to be superior to the adjusted historical 25% response (primary outcome), the arms were not compared to each other. For patients with cirrhosis specifically, the response rate was reported to be 86% in the 12-week arm and 100% in the 24-week arm. While there were 440 patients in the study, only 20% (about 88 patients) had cirrhosis. The difference between the two arms among cirrhotic patients was found to be statistically significant (P = .007) even though the study was not designed to compare the arms to each other. Consequently, the 2015 AASLD guidelines, based only on this study, recommended sofosbuvir/ledipasvir for 24 weeks for treatment-experienced cirrhotic patients with HCV genotype 1. ¹⁶ Further studies, designed to directly compare 12 weeks to 24 weeks of treatment, are needed to support AASLD’s recommendation.

Lack of Power or Not Reported Especially When Primary Outcome is Not Significant

When a study is underpowered and the results are not statistically significant, a follow-up study is typically needed with a larger sample size, longer duration of study, or both, to rule out a false negative. It is not uncommon to have scenarios in which it will not be feasible to have statistical power in a study when the number of events are rare or the disease is rare. A trial assigned immunocompromised patients with invasive aspergillosis (IA) to receive voriconazole with either andiulafungin (combination therapy; n = 135) or placebo (monotherapy; n = 142). ¹⁷ The study required 250 patients in the combination group to have 70% power to show an absolute reduction of 11.4% in all-cause mortality. After nearly 3 years of enrollment from 93 sites in 24 countries, 277 patients with probable or proven IA were included in the study. The rate of all-cause mortality at 6 weeks was 19.6% for combination therapy and 27.5% for monotherapy (P = .087). The power of the study was reduced due to the higher rate of mortality in the monotherapy group than the anticipated 19% rate. Therefore, the results are likely due to a false negative. While further studies would be necessary to confirm the mortality benefit with combination therapy, the rarity of this disease makes it impractical. Though the trend in reduced mortality with combination therapy is not statistically significant, the results are considered clinically relevant and were adopted into the 2016 IDSA guideline recommendations for combination therapy in certain patients (especially those with hematologic malignancy and persistent neutropenia) with documented IA. ¹⁶

Secondary Outcomes Show Positive Findings and Conclusions are Drawn

The primary outcome should be specified prior to study commencement, and any secondary outcomes can indicate potential associations that can be explored in future research or support the primary outcome. Conclusions should not be drawn from secondary outcomes as they are generally considered only hypothesis generating. In addition, study designs are not centered around nor powered to assess secondary outcomes.

The pivotal trials that provided evidence for FDA approval of fluoxetine demonstrated that a secondary outcome of weight loss was significantly higher in the fluoxetine group compared to tricyclic antidepressants (TCAs).^{18 -20} This was considered a major advantage for fluoxetine as TCAs were known to cause weight gain. Some physicians in the US started prescribing fluoxetine for weight loss in patients without depression. The design of a study to examine weight loss versus depression would look very different such as the type of patients included (depression vs obese), or excluded (depression or patients taking an antidepressant would probably be an exclusion in obesity trials), and the length of the trial in depression would be shorter, weeks compared with months to years in obesity trials. The primary outcome in a depression trial would include a rating scale such as the Hamilton Depression Rating Scale (HAMD) whereas a trial for weight loss would evaluate body mass index (BMI), weight, and/or percent body fat. When trials were designed utilizing fluoxetine for weight loss as the primary outcome, fluoxetine was found to initially cause weight loss, but caused weight gain by the end of the trials.^{21 -23} Premature adaptation of a secondary outcome applied to an unrelated condition can result in inappropriate treatments and distrust of the medical field when intended results do not come to fruition as expected. Refer to the fluoxetine and weight loss example if in the future there is a temptation to make conclusions regarding secondary outcomes. ²

Limitations That Reduce External Validity

Inclusion/Exclusion Criteria That Are Too Restrictive or Vague

Inclusion criteria are used to ensure patients in the study have the target disease being studied, and that all enrolled subjects have similar demographic and clinical characteristics. Each clinical trial should provide a complete and accurate list of entry and exclusion criteria, as this defines the study sample. These criteria help clinicians identify patients for whom the study results do not necessarily apply and for whom applying the treatment recommended could cause harm.

The SPRINT trial studied intensive (systolic blood pressure, SBP < 120 mm Hg) versus standard blood pressure control (SBP < 140 mm Hg). ²⁴ The key exclusion criteria were patients < 50 years of age with diabetes and/or history of stroke. It was estimated that due to the inclusion and extensive exclusion criteria that the results applied to approximately 20% of patients with hypertension. ²⁵ In the discussion section of the SPRINT trial, it was noted that only approximately 50% of patients with hypertension obtain the 140/90 mm Hg goal, ²⁴ but it is unclear if the 20% estimate includes this information, so extrapolation could be even more limited.

Shortly after the SPRINT trial was released, Medscape posted a Fast Five Quiz entitled: How Much Do You Know About Hypertension? ²⁶ The last question asked:

“According to the results of the Systolic Blood Pressure Intervention Trial (SPRINT) study, which of the following results in better cardiovascular outcomes?

1. BP treatment initiation and goals are 150/100 mm Hg in patients younger than 60 years with diabetes.

2. In non-black hypertensive patients aged 18 years or older, initiate treatment with a combination of ACE inhibitor and angiotensin receptor blocker.

3. Initiation of therapy in all patients to lower BP < 120/80 mm Hg.

4. In non-black hypertensive patients, thiazide-type diuretics alone are the only indicated treatment.” ²⁶

Answer 1 and 2 are not correct since patients < 50 years of age were excluded. Answer 3 is not correct since there were many exclusions so the results would not apply to all patients. Answer 4 is not correct as the study did not examine specific treatment options. The correct answer listed in Medscape was answer 3 (59% of respondents chose this answer). Medscape stated: “According to the SPRINT findings, achieving a target systolic pressure of 120 mm Hg reduced cardiovascular events (e.g., myocardial infarction, heart failure) and stroke by nearly one-third and reduced risk for death by almost one-fourth when compared with a target of 140 mm Hg.” ²⁶ It is important to utilize both the inclusion and exclusion criteria to ensure patient safety when extrapolating results to clinical practice.

Primary Outcome Not Appropriate, the Preferred Measure or Accurately Defined

The primary outcome needs to be clearly defined and appropriate for the disease state to be able to successfully answer the study objective, as well as for the reader to be able to compare the results to other studies and to extrapolate the results to clinical practice.

The 2013 REDUCE trial was a NI study that compared short-term (5 day) steroid treatment to traditional (14 day) for management of COPD in 134 patients admitted to the emergency department. The primary endpoint was time to exacerbation within 6 months. The investigators determined that NI was met for the primary outcome. ²⁷ The 2014 GOLD (Global Initiative for Chronic Obstructive Lung Disease) guidelines included only this study in their recommendation to use 5 day instead of the traditional 14 day steroid (prednisone 40 mg) treatment although it was listed as Evidence B (“Source of evidence: randomized controlled trials (RCTs). Limited body of evidence. Definition: Evidence is from endpoints of intervention studies that include only a limited number of patients, post-hoc or subgroup analysis of RCTs, or meta-analysis of RCTs. In general, Category B pertains when few randomized trials exist, they are small in size, they are undertaken in a population of the recommendation, or the results are somewhat inconsistent.”) ²⁸ Two major flaws in the study were that the primary outcome was determined using a nonpreferred measure and the study duration was inadequate. A shortened trial duration might truncate evidence for the occurrence of both the primary and secondary outcomes while the short duration might exaggerate the safety profile. The GOLD 2014 guidelines recommended that a patient’s risk of future exacerbations should be based on the rate of exacerbations as the preferred choice of the primary outcome, not the time to the next exacerbation, which was used in the Leuppi, et al study. In addition, the 2014 GOLD guidelines state that the risk of exacerbation is based on the number of exacerbations in the last 12 months and the Leuppi et al study was only 6 months in duration.^27,28 These oversights could lead to a conclusion that it was noninferior to time to exacerbation but after the first relapse could lead to even more exacerbations. It is important to verify study information in guidelines to determine the quality of information, especially when only one trial is used to change standard practice.

Use of Surrogate Outcome Measures When Not Proven to be Tied to Clinical Outcomes or Clinical Outcomes Are the Standard

A surrogate outcome is a measure that is thought to correlate with or be suggestive of a clinical outcome (e.g., elevated cholesterol levels might be a surrogate measure of atherosclerotic vascular disease); however, a correlation does not mean a true causational relationship exists between the two. Surrogate outcome measures are often used when the clinical endpoint occurs in small numbers of patients or requires longer periods of time to determine the outcome (e.g., adverse events or mortality).

Two initial studies evaluated various dosing regimens of enoxaparin among overweight patients requiring venous thromboembolism prophylaxis utilizing the surrogate primary outcome of anti-factor Xa levels rather than the standard clinical outcomes of incident of VTE, bleeding, and/or death.^29,30 In both studies, the authors concluded that 0.5 mg/kg/day was not associated with bleeding or thrombosis. However, the studies had small sample sizes, no comparator groups, and did not evaluate clinical outcomes so it is uncertain whether the surrogate marker of anti-Xa levels translates accordingly. A larger, more recent trial compared two fixed doses of enoxaparin: 40 mg daily (standard) and 60 mg daily and found consistent results with 31% versus 69% of patients achieving target anti-Xa levels (defined as 0.32 to 0.54 units/mL) for the 40 mg and 60 mg groups, respectively (P = .007). Similarly, it was concluded that the 60 mg dose was superior without more bleeding. ³¹ Not only is there variability in what target anti-Xa levels should be for enoxaparin thromboprophylaxis, there is no strong evidence to support that low levels are associated with thrombosis or high levels are associated with bleeding. Although all three of these studies recorded the incidence of bleeding events and venous thromboembolism, there were zero thrombosis events and there was only a total of one bleeding event across all three studies. Given the lack of power to evaluate these clinical outcomes, conclusions that any of these studied regimens are not associated with adverse clinical events should be made with caution. Adequately powered studies focused on the clinical outcomes of bleeding and thrombosis are needed before the decision regarding whether or not anti-Xa levels are representative of bleeding and thrombosis can be made.

Number Needed to Treat/Harm Not Clinically Relevant

Number needed to treat (NNT) or number needed to harm (NNH) are calculated from confidence intervals (CIs) to determine clinical relevance of study results. ² In most cases, NNT should be a low number whereas NNH should be a high number. A statement containing key components should be developed to obtain a complete picture of clinical relevance.

In the IMPROVE-IT study, ezetimibe 10 mg or placebo was given to patients with acute coronary syndrome receiving simvastatin 40 mg to examine the composite risk of death, major CV events and non-fatal stroke. The 7-year composite endpoint was 32.7% in the combination group versus 34.7% in the simvastatin only group, HR 0.94 (95% CI 0.89 to 0.98; P = .016). The authors concluded that ezetimibe added to simvastatin significantly lowered the risk of cardiovascular events more than simvastatin alone. ³² The authors only provided the absolute risk difference (2%) and not an NNT which was calculated to be 50. The most important component of an NNT is time. It is a very different matter for a treatment to take, for example 2 versus 7 years to reduce the number of events in 50 patients by one. The NNT statement in the IMPROVE-IT trial ³² is: 50 patients need to be treated with combination simvastatin 40 mg plus ezetimibe 10 mg for 7 years to prevent one additional composite endpoint of CV death, major coronary events or non-fatal stroke compared to treatment with simvastatin 40 mg. The results in the trial were shown to be statistically significant, but when considering the NNT, they did not show clinical relevance. When analyzing a clinical trial, a practitioner should analyze the results and, when appropriate, show why a particular recommendation supported in print should not be adopted. This conclusion of a lack of clinical relevance was validated when the FDA did not approve this new indication for ezetimibe in combination with simvastatin for a reduction in CV events based on this study. ³³

Conclusion

After examining the clinical trial excerpts in this article, it is evident how superficial scanning of studies might lead to adopting unproven therapies and claims and could potentially lead to erroneous assumptions. Possible outcomes from using unproven therapies could range from unnecessary patient and healthcare costs to crippling, avoidable morbidity, and/or mortality. It is important to realize that many studies, whether intentional or not, contain biases in the study designs that could influence the authors’ stated conclusions. It is imperative that pharmacists be able to identify major study limitations and how those limitations could affect extrapolation of results to patient care.