Letter to the Editor Regarding “Fluticasone Furoate/Umeclidinium/Vilanterol (FF/UMEC/VI) Triple Therapy Compared with Other Therapies for the Treatment of COPD: A Network Meta-Analysis”

Key Summary Points

The network meta-analysis (NMA) of Ismaila et al. contains limitations in design and reporting that could strongly influence the validity of the findings and conclusions.

Heterogeneity between studies was not adequately accounted for; only results from a fixed-effects model are reported, which contradicts guidance on NMA methodology for comparing studies with considerable heterogeneity and can yield misleading results.

Due to fundamental differences between the FULFIL and KRONOS studies, and a single common comparator that performed differently in the two studies, these should not have been relied upon to establish a network connection to compare fluticasone furoate/umeclidinium/vilanterol (FF/UMEC/VI) and budesonide/glycopyrronium bromide/formoterol fumarate (BUD/GLY/FOR).

Given these methodological issues and selective reporting of fixed-effects model results, we do not agree with Ismaila et al.’s assertion that FF/UMEC/VI showed statistically significant improvements in the annualized rate of combined moderate/severe exacerbations versus BUD/GLY/FOR.

The NMA findings and conclusions are not consistent with evidence from four other published NMAs of triple therapies in patients with chronic obstructive pulmonary disease (including two from independent researchers), all of which used random-effects models.

Dear Editor,

We read with interest the publication by Ismaila et al. in Advances in Therapy entitled “Fluticasone Furoate/Umeclidinium/Vilanterol (FF/UMEC/VI) Triple Therapy Compared with Other Therapies for the Treatment of COPD: A Network Meta-Analysis” [1]. Network meta-analysis (NMA) is an important statistical technique, often used to make indirect treatment comparisons when no direct head-to-head studies are available [2]. However, if not robustly conducted, NMAs can yield misleading results [3, 4].

Although clinical and methodological differences between studies are inevitable, valid NMAs rely on the assumption that the included studies are sufficiently homogenous [5, 6]. Per best practice, when NMAs are undertaken in the presence of heterogeneity, random-effects models are recommended to account for heterogeneity [3–6]. Ismaila et al. noted a large amount of between-study heterogeneity for their severe exacerbation analysis; these results were therefore not deemed robust and not reported [1]. I² is a standard statistical measure of homogeneity/heterogeneity [4, 7], and Ismaila et al. report an I² of 86.59% for the network of studies reporting combined moderate/severe exacerbation rates [1]; this is classed as considerable heterogeneity, per Cochrane (I² of 75–100%, highest category) [4], but the authors did not question the robustness of these findings [1]. Ismaila et al. subsequently analyzed a smaller group of studies with ≥ 24 weeks’ follow-up “to account for heterogeneity induced by differences in length of follow-up”, but I² increased to 94.93% [1], suggesting that heterogeneity was not substantially caused by differences in follow-up length but by other factors. However, the authors did not further explore possible reasons for this heterogeneity, nor, as detailed below, did they report the results from a random-effects model, which would account for the heterogeneity.

Ismaila et al.’s methodology states that they implemented both fixed- and random-effects models; however, they only reported results and conclusions based on the fixed-effects model [1]. The approach of reporting only the fixed-effects model findings is not clearly described nor explained. Ismaila et al. acknowledged that, where heterogeneity was present, as in the moderate/severe exacerbation analysis, the random-effects model “automatically accounted for this” [1]. However, these random-effects model results were not reported, even in cases of considerable heterogeneity, and this could alter any conclusions reached. Fixed-effects models assume similar effect sizes across studies and ignore heterogeneity; confidence intervals (CIs) from such models do not reflect the extent of heterogeneity [4].

To better understand if their results are supported by a random-effects analysis, we independently replicated the annualized moderate/severe exacerbation rate analysis with the same software and data sources, using both fixed- and random-effects models. The fixed-effects model gave similar results to Ismaila et al.’s, confirming that their reported results are from a fixed-effects model (which was not made clear). However, the random-effects model, as recommended per best practice, showed no significant treatment–effect differences, due to wide CIs driven by between-study heterogeneity. Given the methodological issues, selective reporting of fixed-effects model results, and failure to disclose the random-effects model results, which we believe to be not significant per our replicate analysis, we do not agree with Ismaila et al. that FF/UMEC/VI showed statistically significant improvements in annualized moderate/severe exacerbation rates versus budesonide/glycopyrronium bromide/formoterol fumarate (BUD/GLY/FOR) [1]. Notably, their finding contradicts four other peer-reviewed NMA publications, including two from independent researchers [8, 9], all of which used random-effects models to account for heterogeneity and reported no significant exacerbation rate differences between triple therapies in patients with chronic obstructive pulmonary disease [8–11].

Aside from the appropriateness of the model, there is the fundamental issue of whether Ismaila et al.’s network approach should be used at all when studies are discernibly heterogenous, a point highlighted by Cochrane and others [4, 12]. NMA validity can be susceptible to network structure, and when indirect comparisons between disparate parts of a network rely solely on links consisting of single studies or a single treatment informed by only two studies, close attention must be paid to these studies, because study-specific biases will be reflected in the relative effect estimates between network segments [13]. It is difficult to determine common comparators and connections in Ismaila et al.’s network diagrams. Replotting (Fig. 1) helps to identify that the connection between FF/UMEC/VI and BUD/GLY/FOR hinges solely on KRONOS and FULFIL, two lung function studies, and their single common comparator, BUD/FOR (400/12) [14, 15]. There are important differences in study design between KRONOS and FULFIL, including blinding, run-in treatments, participating countries, and exacerbation history criteria. BUD/FOR was open-label in KRONOS and blinded in FULFIL [14, 15], and whether or not a treatment is blinded could conceivably impact study results. It is also likely that between-study differences in run-in treatments impacted baseline lung function measurements and subsequent changes from baseline in lung function, evidenced by the least squares mean change from baseline in pre-dose forced expiratory volume in 1 s (FEV₁) at Week 24 being + 62 mL with BUD/FOR in KRONOS [14, 16] and − 29 mL with BUD/FOR in FULFIL [15]. Furthermore, FULFIL was conducted primarily in Eastern Europe, whereas KRONOS was conducted primarily in the US, Japan, and China [14, 15]. The countries in which studies are conducted are known to affect exacerbation results [17]. Additionally, prior exacerbations are well-recognized as a strong predictor of future exacerbation risk [18]. KRONOS did not require patients to have an exacerbation in the previous year (74% of patients did not) [14]; FULFIL required patients to have ≥ 2 moderate exacerbations or ≥ 1 severe exacerbation in the previous year if their FEV₁ was ≥ 50–< 80%, but did not require exacerbations in the previous year if their FEV₁ was < 50% (35% of patients did not) [15, 19]. Considering these studies are supposedly similar in other key aspects according to Ismaila et al., it is surprising and contrary to accepted scientific knowledge on the impact of prior exacerbations that the annual moderate/severe exacerbation rate was > 50% higher for BUD/FOR in KRONOS versus FULFIL (0.55 vs. 0.34), when 74% of KRONOS patients had no exacerbations in the previous year [14, 15]. Taken together, this highlights that there are fundamental differences between the studies, well beyond the key patient characteristics that Ismaila et al. extracted from the studies, and that these had a significant influence on the results, biasing the NMA findings.

Fig. 1.

Replotted network of evidence of Ismaila et al. informing annualized moderate and severe exacerbation analyses. Adapted from Supplementary Figure S3 of Ismaila et al.’s NMA [1], licensed under CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/). Studies/treatments that could not be linked to the network have been removed (Siler 2016 [25] [SAL/FP 50/250 and UMEC 62.5 + SAL/FP 50/250] and Sousa 2016 [26] [ICS/LABA and UMEC 62.5 + ICS/LABA]). Treatment boxes have been rearranged so that the network connections are clearer. Study labels have been adapted to focus on the connection between FULFIL and KRONOS, which hinges on a single common treatment comparator (BUD/FOR 400/12). BDP beclomethasone dipropionate, BUD budesonide, FF fluticasone furoate, FOR formoterol, FP fluticasone propionate, GLY glycopyrronium bromide, ICS inhaled corticosteroid, IND indacaterol, LABA long-acting β₂-agonist, NMA network meta-analysis, SAL salmeterol, TIO tiotropium, UMEC umeclidinium, VI vilanterol

Such differences are not limited to KRONOS and FULFIL. Ismaila et al. claim that “the similarity assumption held” for all studies in the network, including FULFIL and IMPACT, which are directly connected with the common comparator of FF/UMEC/VI [1]. One difference was that IMPACT only included patients with recent exacerbations, and the annual moderate/severe exacerbation rate with FF/UMEC/VI was 0.91 [20]. However, in a post hoc analysis of FULFIL, the annual moderate/severe exacerbation rate in patients with recent exacerbations before study entry (aligned with IMPACT) was 0.19 with FF/UMEC/VI [19], approximately 80% lower than in IMPACT. This again highlights that FULFIL is somewhat of an outlier, with fundamental differences to its connected studies; this questions how these studies could be considered similar and why heterogeneity was not further explored. FULFIL cannot be relied upon to form network connections to KRONOS or IMPACT, as these connections bias all wider relative treatment–effect estimates. Given the substantial differences in the linking studies connecting FF/UMEC/VI with BUD/GLY/FOR, Ismaila et al.’s analysis provides misleading results.

One way to ascertain whether a network of indirect relationships is reasonable is to compare direct and indirect evidence for relative treatment effects [13]. In their moderate/severe exacerbation rate comparison, there is a 25% difference (incidence rate ratio [IRR] 0.75) for FF/UMEC/VI versus UMEC/VI (per IMPACT results [20]), but a 66% difference (IRR 0.44) for FF/UMEC/VI versus GLY/FOR, where no direct comparison data exist [1]. The analysis therefore shows indirect evidence of a ~ 40% difference for UMEC/VI versus GLY/FOR. However, a 24-week head-to-head study comparing GLY/FOR versus UMEC/VI, with exacerbations as an exploratory endpoint, reported no appreciable difference in moderate/severe exacerbation rate (16.7% vs. 17.6%) or time to first moderate/severe exacerbation (hazard ratio [95% CI] 0.97 [0.73, 1.29]) [21]. The inconsistent estimates from these direct and indirect evidence sources indicate that Ismaila et al.’s network produces misleading indirect results regarding relative efficacy between FF/UMEC/VI and BUD/GLY/FOR or its components. Ismaila et al. did not report inconsistency testing of direct and indirect evidence [1], which is recommended per best practice when a network has loops [2, 22–24]. In our replicate analysis, we performed an inconsistency assessment [22] of the moderate/severe exacerbation network, which indicated statistical inconsistency between direct and indirect estimates among all network loops, further highlighting the potential for misleading results.

While other NMAs have included KRONOS and FULFIL, they employed different methods to avoid making a connection solely through KRONOS and FULFIL, thus avoiding the bias that Ismaila et al. introduce with their approach [8, 10, 11]. Heterogeneity between KRONOS and FULFIL makes them unsuitable as a critical network connection to analyze relative effects of BUD/GLY/FOR versus FF/UMEC/VI. Hence, we cast doubt on all the conclusions and statements of superior efficacy which pertain to these.