Secondary Analysis of a Systematic Review: Are Antifungal Noninferiority Trials at Risk of Eroding Effectiveness Because of Biocreep?

ABSTRACT

Noninferiority randomized controlled trial (RCT) effectiveness may erode when results favor the active control over time and when a decreasingly effective control arm is used in serial trials. We analyzed 32 antifungal noninferiority RCTs (NI-RCTs) for these scenarios in this secondary analysis of a systematic review. Our exploratory analysis suggests that the erosion risk in the effectiveness of antifungal noninferiority trials is uncommon. Findings are limited by small sample size and overall risk of bias.

INTRODUCTION

Since the 1990s, antimicrobial randomized controlled trials (RCTs) are typically conducted with a noninferiority design (1). This design implies that a treatment is considered at least equivalent to the comparator if its efficacy is no worse than the comparator by a predefined margin. Over time, serial noninferiority trials (NI-RCTs) may lead to loss of efficacy because of inherent weaknesses to this design in certain scenarios.

In “scenario 1” (Fig. 1a), the constancy assumption – that the treatment arm would be superior if tested in a placebo-controlled trial – is exploited. If results favor the control arm or the selected noninferiority margin (NIM) is too large, novel drugs appear falsely noninferior to older ones in serial trials.

FIG 1.

(A) In scenario 1, trials of novel antifungals violate the constancy assumption by selecting noninferiority margins that are too large or by results that favor the control arm. This leads to novel antifungal treatment effects that are consistently worse than older trials. (B) Scenario 2 illustrates the effect of biocreep in serial noninferiority trials. The antifungal used as the treatment arm in trial 1 becomes the control arm in trial 2, and so on. However, when the treatment arm in each successive trial is noninferior but the ARR < 0, the result is that over time the control effectiveness erodes to be no better than placebo. The green diamond in this panel represents the extrapolated ARR estimate for a trial comparing drug E to drug A when biocreep occurs in successive pairs of noninferiority trials. (C) Testing out scenario 1 using data from the systematic review. Only studies with a 95% CI are included in this panel. The ARR is divided by the NIM, since it allows the proportional risk reduction to be visualized. New antifungal trials (blue) represented in this panel, in order from top to bottom: Cornely 2007 (8), Reboli 2007 (27), Kohno 2010 (14), van Burik 2004 (30), Mora-Duarte 2002 (24), Rex 1994 (28), Chosidow 2003 (7), Maertens 2016 (20), Pappas 2007 (25), de la Paz Cota 2018 (9), Walsh 1999 (32), Walsh 2004 (34), Küse 2007 (18), de Wet 2005 (10), Krause 2004 (15), Vazquez 2010 (31), Kullberg 2005 (16), Walsh 2002 (33), and Kullberg 2019 (17). Older antifungal trials (red) represented in this panel, in order from top to bottom: Marks 2011 (21), Jeong 2016 (12), Molloy 2018 (23), Saliba 2015 (29), Le 2017 (19), Yim 2010 (35), Mersal 2013 (22), Kang 2020 (13), Huang 2012 (11), Yoshida 2020 (36), and Benjamin 2018 (5). Data from Buechner 2014 (6) and Queiroz-Telles 2008 (26) are not included in figure, as NIMs were published on a scale not amenable to comparison with other studies. (D) Testing out scenario 2 using data from serial noninferiority trials in the systematic review. The extrapolated point estimate is calculated by: ARR_{(B versus A) trial 1} + ARR_{(C versus B) trial 2} = ARR_{(C versus A)}. Pair 1 represented in this panel, from top to bottom, is Mora-Duarte 2002 (24) and the 100 mg micafungin arm of Pappas 2007 (25). Pair 2 represented in this panel, from top to bottom, is Mora-Duarte 2002 (24) and the 150 mg micafungin arm of Pappas 2007 (25). Pair 3 represented in this panel, from top to bottom, is Mora-Duarte 2002 (24) and Kullberg 2019 (17). ARR = absolute risk reduction; NIM = noninferiority margin. Figure adapted from Bai et al., with panels A and B reproduced with permission (2).

“Scenario 2” (Fig. 1b) requires successive NI-RCTs where the treatment in trial A becomes the control in trial B, and the treatment in trial B becomes the control in trial C, etc. If the treatment arms of each successive trial are statistically noninferior but in fact slightly inferior, the control arm's effectiveness degrades over time until it is equivalent to placebo, a phenomenon termed “biocreep.”

Our prior antibiotic NI-RCT analysis showed biocreep is rare (2). In this secondary analysis of a systematic review (40) we examined biocreep in antifungal NI-RCTs.

This study was reported per PRISMA guidelines (Text S1a, b) (3) and prospectively registered (PROSPERO CRD42020219497).

Medline, Embase, Cochrane Central, and the FDA drugs database were searched without language restrictions from inception to September 9, 2020 (Text S2). Retrieved article reference lists were screened.

We included NI-RCTs in humans comparing ≥2 antifungal regimens used for prophylaxis of an invasive fungal infection; or to treat a possible, probable, or proven fungal infection. Studies with invasive fungal infection (IFI) populations were included if the participants met the 2019 EORTC/MSG IFI consensus definition, regardless of publication year. Novel antifungal studies were defined as those conducted within 5 years of the FDA approval date for the treatment arm, whereas older antifungal studies were those occurring >5 years after – cutoffs chosen for consistency with prior literature (2). Phase I, II, and superiority design RCTs were excluded.

All assessments were performed independently in duplicate. Year of study, study centers, population, treatment and control arms, sample size calculation, NIM, and outcomes were captured. Risk of bias assessments were performed using the Cochrane RoB 2.0 tool. Disagreements were resolved by adjudication.

The coprimary outcomes were the occurrences of a consistently worse treatment effect in studies of novel antifungals compared to older ones (“scenario 1”); and of a decreasingly effective control arm in serial NI-RCTs (“scenario 2”).

We used previously described statistical analysis methods (2, 4). For scenario 1, study outcomes were converted to absolute risk reduction (ARR), where ARR <0 means the treatment arm is less effective than control. ARR was calculated as: Failure rate_treatment - failure rate_control. 95% confidence intervals were calculated using Miettinen and Nurminen's method. Fisher's exact test compared the proportion of novel antifungal studies with an ARR >0 to older antifungal studies. The ARR was divided by the NIM to allow visualization of a standardized proportional risk reduction across studies. Wilcoxon's rank-sum test compared novel and older antifungal ARRs.

For scenario 2, noninferiority trial pairs for the same infectious disease were identified where the treatment arm in an earlier trial was the subsequent control arm in a more recent trial. An extrapolated ARR estimate for the treatment arm in the later trial versus the control arm for the earlier trial was calculated: ARR_{antifungal B versus A in trial 1} + ARR_{antifungal C versus B in trial 2}.

Statistical tests were two-sided, with statistical significance defined as P < 0.05. Analysis was performed using R, version 3.6.3 (R Foundation for Statistical Computing, Vienna, Austria). Meta-analysis was not performed due to heterogeneity. Post hoc subgroup analyses stratified by systemic treatment, systemic prophylaxis, and topical treatment NI-RCTs were performed to ensure results were not influenced by including these study types.

Of 700 screened abstracts, 44 full texts were assessed, and 32 full texts were included (Fig. S1; for studies excluded at full text, Table S1) (5–36). Study characteristics and consensus extracted data set are described in Table 1 and Table S2, respectively.

TABLE 1.

Study characteristics

Characteristic	Novel antifungala studies (n = 21)	Older antifungalb studies (n = 11)
Study population exclusively adults	14	7
Multicentre	21	11
Pharmaceutical industry funding	21	6
Treatment arm median (IQR)	200.5 (131.5-183.25)	91.5 (64.75-183.25)
Infectious disease syndrome
Aspergillosis	1	0
Cutaneous mycosis	0	1
Febrile neutropenia	3	2
Invasive candidiasis (including candidemia)	11	1
Prophylaxis in nontransplant patients	1	1
Prophylaxis in transplant patients	1	4
Other	5	2
Method of diagnosis
Antigen detection	1	1
Expert opinion based on IFI consensus definition	5	6
Culture (excluding IFI)c	12	1
Microscopy	1	1
Other	2	2
Primary outcome
Clinical outcome	7	4
Microbiologic outcome	0	2
Composite clinical and microbiologic outcome	14	5
Reporting of adverse events	21	11
Conclusion by authors
Noninferiority shown	14	8
Superiority shown	4	2
Inferiority shown	3	0
Inconclusive	0	1

^aNovel antifungal studies are defined as studies undertaken within 5 years of the FDA approval date of the antifungal in question.

^bOlder antifungal studies are defined as studies undertaken ≥5 years from FDA approval date for the antifungal in the trial's treatment arm.

^cAs culture is part of, but not singularly responsible for, the EORTC/MSG diagnostic criteria for IFI, culture for other fungal infections such as candidemia or superficial   dermatophyte infection has been separated from that used as part of the diagnostic criteria for EORTC/MSG-defined IFI.

54.5% of older antifungal studies had an ARR > 0, compared with 61.9% of novel antifungal studies (Fig. 1c). The median ARR/NIM for older and novel antifungal studies was both 0.089 (IQR, older antifungal studies: −0.045–0.365; IQR, novel antifungal studies: −0.086–0.473).

Three pairs of serial NI-RCTs, all with invasive candidiasis populations, were found (Fig. 1d, Table S3). No pair had an ARR < 0 in both constituent trials; the probability of biocreep was low. Subgroup analyses stratified by systemic treatment, systemic prophylaxis, and topical prophylaxis NI-RCTs (Fig. S2) suggest results are not influenced by including these study types.

Overall risk of bias was low in 9.4%, drew some concern in 40.6%, and was high in 50% of included studies (Fig. S3). Risk of bias within individual studies is described in Table S4.

Noninferiority trials rely on assumptions of assay sensitivity – the ability of the trial to distinguish effective from ineffective treatment – and choice of an appropriate NIM and analysis set (37). They may therefore be at risk of eroding effectiveness over time. Our prior study of antibiotic RCTs (2) was congruent with others (38, 39) that biocreep is rare. Our current study suggests biocreep may be similarly uncommon with antifungal RCTs.

Our study has limitations: we examined many clinically heterogeneous syndromes, which may introduce noise. Biocreep conclusions are limited by small sample size, with three pairs of serial trials identified, all in invasive candidiasis populations. This small number of serial trials limits our conclusions to an exploratory analysis which should be interpreted cautiously. These conclusions may not be generalizable to other fungal populations. Trials were at moderate to high risk of overall bias.

Our systematic review suggests antifungal NI-RCTs may be at low risk of eroding effectiveness, supporting their continued use.