Critical Appraisal of Published Indirect Comparisons and Network Meta-Analyses of Competing Interventions for Multiple Myeloma

Abstract

Objectives:

In the field of relapsed or refractory multiple myeloma (RRMM), between-trial or indirect comparisons are required to estimate relative treatment effects between competing interventions based on the available evidence. Two approaches are frequently used in RRMM: network meta-analysis (NMA) and unanchored matching-adjusted indirect comparison (MAIC). The objective of the current study was to evaluate the relevance and credibility of published NMA and unanchored MAIC studies aiming to estimate the comparative efficacy of treatment options for RRMM.

Methods:

Twelve relevant studies were identified in the published literature (n = 7) and from health technology assessment agencies (n = 5). Data from trials were extracted to identify between-trial differences that may have biased results. Credibility of the performed analyses and relevance of the research questions were critically appraised using the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) checklist and feedback based on consultations with clinical experts.

Results:

The identified studies concerned NMAs of randomized controlled trials (RCTs; n = 7), unanchored MAICs (n = 4), or both types of analyses (n = 1). According to clinical expert consultation, the majority of the identified NMAs did not consider differences in prior therapies or treatment duration across the RCTs included in the analyses, thereby compromising the relevance.

Conclusion:

Based on the results and feedback from clinicians, the majority of NMAs did not consider prior treatment history or treatment duration, which resulted in nonrelevant comparisons. Furthermore, it may have compromised the credibility of the estimates owing to differences in effect-modifiers between the different trials. Pairwise comparisons by means of unanchored MAICs require clear justification given the reliance on non-randomized comparisons.

Introduction

Although treatments for newly diagnosed multiple myeloma (MM) patients are well established, treatment options for patients progressing beyond first-line therapy were limited before 2004, particularly for relapsed or refractory MM (RRMM), a stage of MM defined by disease that is nonresponsive while on salvage therapy or progression within 60 days of last therapy in patients who have achieved minimal response or better at some point previously.^1,2 Between 2004 and 2018, however, more than eight treatments with multiple combination options became available for use in RRMM patients. Guideline recommendations emphasize that treatment goals for patients with MM vary depending on disease stage and treatment experience, and treatment choice may depend on age, performance status, comorbidities, number of prior treatment lines, remaining treatment options, interval since last therapy, and type of relapse.^3–5 Nevertheless, recommendations regarding newer treatments are broad and do not specify which of the newer treatments is optimal for a typical second-line (2L), third-line (3L), or fourth- or subsequent-line (4L+) patient. Given the multitude of interventions for RRMM, it is important to evaluate comparative efficacy and safety to determine best practices, especially where no head-to-head clinical trials exist.

The National Institute for Health and Care Excellence (NICE) outlines two main approaches for estimating relative treatment effects between competing interventions depending on the context and evidence base.^6,7 Network meta-analyses (NMAs) are based on randomized controlled trials (RCTs) where each study directly compares a subset of, but not necessarily all, treatments of interest. Another approach is matching-adjusted indirect comparison (MAIC), where the outcomes with an index intervention of interest evaluated in one or more RCT(s) or single-arm trial(s) for which individual patient-level data (IPD) is available are compared with the outcomes for a competing intervention in another trial with study-level data. MAICs can be either anchored, where there is a shared comparator creating a connected network, or unanchored in the absence of a shared comparator (ie, disconnected network). In both cases, propensity score matching is used to estimate the relative effect between the two interventions of interest by adjusting for the differences in treatment effect modifiers (anchored and unanchored) or prognostic factors (unanchored) of the trial for which IPD is available to match the population for which study-level data is available.

There are challenges unique to RRMM that may limit the validity of treatment effect estimates based on NMAs or MAICs. For instance, some trials do not share common comparators, meaning that a standard NMA is not feasible and alternative methodologies, such as unanchored MAIC, may be required.⁸ There may also be substantial heterogeneity with regard to patient characteristics and study design. The aim of this study was to identify and critically appraise NMA and MAIC studies evaluating the comparative effectiveness of alternative interventions for the treatment of RRMM based on between-study comparisons.

Methods

A systematic literature review identified studies evaluating the comparative efficacy of pharmacologic interventions for the treatment of patients with RRMM in terms of overall survival (OS), progression-free survival (PFS), or response (see Appendix A in Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.11.003 for full details).

NMAs were evaluated using the Comparative Effectiveness Research-Collaborative Initiative Instrument created in conjunction with the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) to assess their credibility and relevance (referred to as the ISPOR checklist).⁹ Questions regarding credibility were grouped together in terms of the evidence base, network of evidence (ie, whether included studies formed a connected network), treatment-effect modifiers (ie, whether there were differences in treatment effect modifiers across treatment comparisons), analysis (ie, whether the statistical methods were used to preserve within-study randomization), and reporting (ie, whether all pair-wise contrasts were reported along with measures of uncertainty). For published MAICs, the ISPOR checklist questions were used where possible, which were complemented by recommendations from the NICE Decision Support Unit (DSU) Technical Support Document 18 to determine whether best practices were followed in terms of anchoring (appropriate methods based on whether common comparators exist), selection of covariates (ensuring all prognostic and effect modifiers were adjusted for), and reporting (covariate choice and reasoning).^7,9

The process involved consultation from both methods and clinical experts identified through a request for participation based on experience working in specialized centers or collaborating on major clinical trials in Europe and the United States. Conclusions regarding relevance and credibility were based on the consensus of two independent investigators and were then validated by clinicians. During a series of teleconferences, clinicians were presented with the NMA and MAIC research questions and asked about their relevance. Additionally, experts evaluated the credibility of each analysis based on the networks of evidence and key assumptions.

Results

Indirect Comparisons and Network Meta-Analyses

Twelve indirect comparison studies were identified comprising six publications,^8,10–14 five health technology assessment (HTA) reports,^15–19 and two conference posters^20,21 (see flow chart in Appendix A in Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.11.003), which included eight NMAs^{10–13,15,17,18,20} and five MAICs.^8,14–16,19 Details of the included studies are presented in Appendix B (in Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.11.003). Themes identified during the critical appraisal are described in terms of the evidence base, relevance, and credibility of each analysis (see details of each appraisal in Appendix C in Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.11.003).

Evidence Base

Figure 1 illustrates all 37 studies included across the different NMAs and MAICs, comprising 32 RCTs,^22–52 two single-arm studies (PANORAMA-2⁵³ and STRATUS⁵⁴), two observational studies (Gooding et al⁵⁵ and Tarant et al⁵⁶) and one non-randomized trial (GEN501⁵⁷).

Figure 1.

Clinical trials included in the studies and the reported distribution of prior treatments: two disconnected networks.

BEND indicates bendamustine; BEV, bevacizumab; BOR, bortezomib; CARF, carfilzomib; CYCLO, cyclosporine; DARA, daratumumab; DEX, dexamethasone; DOX, doxorubicin; ELO, elotuzumab; IXA, ixazomib; LEN, lenalidomide; OBLI, oblimersen; PANO, panobinostat; POM, pomalidomide; SIL, siltuximab; sub, subcutaneous; THAL, thalidomide; TX, treatment; VORI, vorinostat.

Figure 2 highlights select NMAs that used alternative methods to compare treatments across the two disconnected networks, which were centered around bortezomib in combination with dexamethasone (BOR + DEX), or lenalidomide in combination with dexamethasone (LEN + DEX).^10,12,15,18

Figure 2.

Network diagrams from selected studies. (A) Ollendorf et al¹⁸ (ICER), RRMM ≥2L: ITC using observational evidence (dotted line) and subgroup analysis for 2L (square); (B) Durand et al,¹⁵ RRMM ≥2L: ITC using observational evidence (black dotted line) and MAIC (red dotted line); (C) Botta et al,¹⁰ RRMM ≥2L: ITC with treatment classes pooled according to grey bars; (D) Maiese et al,¹² RRMM 2L: two ITCs for each separate network.

2L indicates second-line; BOR, bortezomib; CARF, carfilzomib; CYCLO, cyclosporine; DARA, daratumumab; DEX, dexamethasone; DOX, doxorubicin; ELO, elotuzumab; HDAC, histone deacetylase inhibitors; HR, hazard ratio; HTA, health technology assessment; ICER, Institute for Clinical and Economic Review; IMID, immunomodulatory agent; ITC, indirect treatment comparison; IXA, ixazomib; LEN, lenalidomide, mAB, monoclonal antibody; MAIC, matching-adjusted indirect comparison; OBS, observational; PANO, panobinostat; PFS, progression-free survival; PI, proteasome inhibitor; POM, pomalidomide; RRMM, relapsed and/or refractory multiple myeloma; sub, subcutaneous; THAL, thalidomide.

The distribution of prior treatments reported in the trials is illustrated, which was considered to be one of the most important treatment effect modifiers given the expected magnitude of the effect modification and the imbalance across the trials. Other important treatment effect modifiers and prognostic factors are summarized in Box 1. Remaining evidence networks are illustrated in Appendix D in Supplemental Materials (found at https://doi.org/10.1016/j.jval.2019.11.003), which were similar or evaluated a subset of those interventions assessed in Figure 2 ^{10,11,13,17,20}

BOX 1. Key potential treatment effect modifiers as identified by clinical experts.

Patient and outcome characteristics that may act as treatment effect modifiers and/or prognostic factors:

• Number of prior treatment lines

• Refractory to proteasome inhibitor or immunomodulatory agent

• European Cooperative Oncology Group score

• International Staging System stage

• Cytogenetic risk

• Extra-medullary disease

• Progression-free survival versus time-to-progression

Study characteristics that may act as treatment effect modifiers:

• Cross-over within randomized controlled trials (yes or no)

• Open-label versus blinded

Relevance

Experts from both Europe and the United States suggested that the comparisons between interventions in the NMAs and MAICs were often not relevant for current decision making and several were considered irrelevant even at the time of the analysis. Generally, the relevance related to the purpose of the analysis; therefore we first discuss the relevance of the HTA evaluations followed by the non-HTA publications.

The NMA by Ollendorf et al¹⁸ was performed by the Institute for Clinical and Economic Review (ICER) in the United States. This was the most recent HTA-related publication comparing carfilzomib (CARF), elotuzumab (ELO), ixazomib (IXA), and daratumumab (DARA), which evaluated subgroups based on treatment experience where possible (2L versus 3L to 4L).¹⁹ The most recent NICE submission identified (Edwards et al¹⁶) evaluated the use of DARA versus panobinostat (PANO) + BOR + DEX and pomalidomide (POM) + DEX using an MAIC among patients previously treated with a proteasome inhibitor as well as an immunomodulatory agent.¹⁴ Nevertheless, clinicians indicated that DARA monotherapy would be rare in clinical practice. Other HTA MAICs compared PANO + BOR + DEX with LEN + DEX (Durand et al,¹⁵ 2L+) and POM + DEX versus both PANO + BOR + DEX and bendamustine (Wood et al,¹⁹ 3L+ for patients with LEN and BOR experience), both of which were critiqued for not being relevant to clinical practice. Beyond the interventions compared, NICE evidence review groups often critiqued the population evaluated, which did not account for relevant treatment effect modifiers in the target population, such as age and performance status (Durand et al),¹⁵ in addition to previous treatment experience (Hoyle et al).¹⁷ Therefore none of the HTA NMAs or MAICs assessed a comparison or population that was considered to be relevant for current clinical practice.

The most comprehensive NMAs in terms of the interventions were published after 2016 (Botta et al,¹⁰ van Beurden-Tan et al,¹³ and Maiese et al¹²), which were considered to include more relevant treatments for current clinical practice. However, studies by Botta et al¹⁰ and van Beurden-Tan et al¹³ appeared to be driven by an interest in combining all available trials rather than defining a relevant question in terms of prior treatment experience. In addition, treatment classes (Botta et al¹⁰) or specific interventions (van Beurden-Tan et al¹³) were pooled, further limiting the relevance of the comparisons, which did not provide insight regarding the specific treatment comparisons of interest. Maiese et al¹² was the exception, which specified a more focused research question dependent on the backbone therapy and line of treatment; this approach precluded the need to connect the disconnected network. Clinicians agreed that treatments with a BOR backbone were more relevant to Europe, whereas treatments with a LEN+DEX backbone were more relevant for the United States. Consequently, there was no need to connect the two networks as comparisons between these regimens likely had no clinical relevance.

Credibility

Most studies identified RCTs through a systematic literature review. There is a risk that the two NMAs and two MAICs without a systematic review may have missed relevant trials, but this did not appear to be the case.

NMAs approached the two disconnected networks using alternative methods. Van Beurden-Tan et al¹³ pooled THAL with THAL + DEX, as well as BOR with BOR + DEX to connect the two networks, making the assumption that the DEX backbone in each of the treatment regimens had no effect on outcomes (see Appendix C in Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.11.003). Clinicians did not agree with this assumption; therefore there is a risk that comparisons across the disconnected network may be biased. Ollendorf et al¹⁸ included an observational study to connect the network, which compared BOR + DEX versus BOR based on a matched-comparison analysis.⁵⁸ PFS results (hazard ratio [HR] 0.56; 95% confidence interval [CI] 0.35–1.00) suggest it may not be appropriate to pool these treatments, whereas OS results were more uncertain (HR 0.96; 95% CI 0.54–1.70). Ignoring these differences may have biased results in favor of interventions compared with BOR + DEX. Clinicians preferred methods that incorporated observational evidence over pooling the treatments; however, as mentioned, comparisons across the disconnected network may not be relevant, which is why the credibility of Maiese et al¹² was not affected by these assumptions.

Ollendorf et al¹⁸ and Maiese et al¹² were the only NMAs considered relevant; they attempted to account for differences in prior treatment experience using subgroup analyses. Nevertheless, because the small number of RCTs per treatment comparison precluded a random effects models or meta-regression, the limitations to explore these differences in potential treatment effect modifiers (see Box 1) in the absence of IPD affected the credibility of the results. Also, differences in terms of backbone or active treatment were not acknowledged. For example, BOR was fixed in some trials (eight cycles in APEX⁴⁵ and CASTOR⁴³) and “treat until disease progression” in others (ENDEAVOR²⁷), which may have biased results against regimens in combination with fixed duration BOR + DEX.²³ Similarly, CARF was the only treatment that was limited to 18 cycles, whereas other treatments were administered until progression. Furthermore, OS results may have been biased due to crossover in some trials. Ollendorf et al¹⁸ predicted OS results based on the relationship between PFS and OS⁵⁹; however, a model that considered PFS and OS simultaneously or integrated external evidence may have provided a more comprehensive approach.

All MAICs were unanchored, did not provide full details regarding the methodology, and compared different treatments. Appendix E in Supplemental Materials (https://doi.org/10.1016/j.jval.2019.11.003) outlines the covariates included across the MAICs, which differed across the studies. Edwards et al¹⁶ and van Sanden et al¹⁴ had the most comprehensive models and aligned most closely with recommendations to include all treatment effect modifiers and prognostic factors.⁷ Generally, clinicians considered the covariates relevant and comprehensive; however, cytogenetic risk was an important factor not considered, likely owing to inconsistent reporting across the trials. Importantly, two of the five MAICs did not justify their choice of covariates, whereas the remaining three described a consultation with clinical experts. Generally, studies were limited by the absence of a clear process for covariate selection founded on evidence from the literature. Ultimately, although many covariates were included in the model, it was difficult to determine whether all relevant prognostic factors and treatment effect modifiers were included, highlighting the key assumption that undermines unanchored MAICs. Therefore there is a risk that MAIC estimates were still biased given that unreported or unobserved covariates may have affected the outcomes.

All NMAs and MAICs combined PFS, assuming a constant HR over time, which may not have been appropriate given that Kaplan-Meier curves of some of the included studies diverge, as illustrated in TOURMALINE-MM1⁴⁰ after 10 months. Thus combining an average effect over time may not be the appropriate way to synthesize the trial results. None of the studies other than Ollendorf et al¹⁸ evaluated the proportional hazards assumption, despite the availability of more flexible synthesis methods.^60,61

Durand et al¹⁵ was the only study that conducted both an NMA and MAIC. Nevertheless, differences between estimates could not be attributed to the different methods because the NMA included 2L to 4L patients, whereas the MAIC was restricted to 3L to 4L patients (PFS HRs for PANO + BOR + DEX comparing LEN + DEX for NMA was 1.87 versus 1.06 for the MAIC).

Discussion

Our aim was to critically appraise the relevance and credibility of published NMAs and unanchored MAICs in RRMM as identified through a systematic literature review. The critical appraisal of NMAs was based on an established ISPOR checklist. Because this checklist was not designed for MAICs, we also included questions from the NICE Decision Support Unit Technical Support Document 18.⁷ Although these provided useful tools, we felt it was important to conduct a complete feasibility assessment involving consultation with clinicians.

A majority of the NMAs and unanchored MAICs published between 2015 and 2018 were driven by the NICE HTA process in the UK and are summarized over time to illustrate the evolution of key trials and indirect comparisons (Fig. 3). Historically, the standard of care for RRMM in Europe was BOR. This was challenged in a 2008 NICE submission, based on an indirect comparison that resulted in a positive NICE recommendation of LEN + DEX for a very specific population (3L+) based on a patient access scheme.¹⁷ Between 2008 and 2015, phase III trials were published evaluating POM + DEX (vs DEX in the 3L+; MM-003³⁹) and PANO + BOR + DEX (vs BOR + DEX in 2L to 4L; PANORAMA-1⁴⁶). Nevertheless, it was not until 2015 and 2017 that each of these treatments were evaluated in separate single-technology appraisals, respectively. Between these trial publications and their respective submissions to NICE, phase III trials were published for CARF + LEN + DEX,⁴⁹ ELO + LEN + DEX,³⁷ IXA + LEN + DEX,⁴⁰ DARA + BOR + DEX,⁴³ and DARA + LEN + DEX.²⁸ Although the consultation with clinicians did not represent the UK perspective, NICE reports suggest there was a disconnect between UK HTA evaluations and clinical practice in other jurisdictions even at the time of the assessments because clinicians were interested in comparing CARF, ELO, IXA, and DARA in the 2L+ setting, which had yet to be compared in the HTA setting. The evolution of several single-technology appraisals over a period where the treatment landscape evolved rapidly was exacerbated by the fact that trials were relatively small and evaluated unique populations in terms of treatment history. RRMM highlights the need for faster comprehensive multiple technology appraisals spanning the alternative populations to better identify the place of different therapies in the treatment pathway.

Figure 3.

Timeline of key clinical trials and network meta-analyses or matching-adjusted indirect comparisons.

BOR indicates bortezomib, CARF, carfilzomib; DARA, daratumumab; DEX, dexamethasone; ELO, elotuzumab; HTA, health technology assessment; ICER, Institute for Clinical and Economic Review; ITC, indirect treatment comparison; IXA, ixazomib; LEN, lenalidomide; MAIC, matching-adjusted indirect comparison; NICE, National Institute for Health and Care Excellence; NMA, network meta-analysis; PANO, panobinostat; POM, pomalidomide.

Published NMAs were not considered to be relevant for today’s clinical decision making because they introduced strong assumptions regarding the pooling of interventions and represented a “catch-all” for the available trials rather than defining a relevant research question specific to interventions and comparators that could be used in clinical practice for a specific line of therapy. The NMA by Maiese et al¹² was considered the most relevant because it evaluated networks separately based on prior treatment experience and also targeted the research question toward populations with specific treatment experience. Dimopoulos et al⁶² published an NMA in 2018 after the systematic literature review was completed with a similar research question and approach. Although these were the most relevant studies and yielded consistent results, it is unclear whether these results can be considered credible. Given the rapid evolution of RRMM landscape, additional synthesis studies may have been performed that were not captured in our study following the systematic literature review.

Even in cases where the research question was specific and it was not necessary to connect the two disconnected networks through strong pooling assumptions or observational evidence, the credibility of the estimates resulting from these between-study comparisons was limited by the available data. Although the RCTs were generally high quality, there was a substantial risk of bias given differences in potential treatment effect modifiers, including treatment experience, crossover, duration of therapy, and cytogenetic risk. Most NMAs did not attempt to account for differences in treatment experience; where it was assessed, it was only based on subgroup data comparing previous lines of therapy (2L versus 3L+), which did not provide sufficient information regarding the type of prior therapy. None of the studies accounted for differences in duration of treatments, which ranged from fixed treatment duration (APEX,⁴⁵ CASTOR,⁴³ PANORAMA-1⁴⁶) to treatment until progression (ENDEAVOR²⁷) and may have acted as a driver of outcomes, potentially reducing BOR+DEX efficacy. Moreover, differences in cytogenetic risk were rarely reported or identified as a potential treatment effect modifier. It will be important for future clinical trials to report more clearly the biology of the disease to account for heterogeneity appropriately.

Beyond limitations due to the available study-level data, none of the studies followed a systematic or transparent feasibility assessment process for the NMA.⁶³ Although it was not feasible for synthesis studies to account for the differences given the limited number of trials overall and per treatment comparison, this was rarely acknowledged clearly in the findings or discussion.^64,65 Despite substantial between-study differences in characteristics considered to act as effect modifiers (such as treatment experience), random effects models were not feasible in most cases. Nevertheless, authors failed to highlight that NMA results should be interpreted with caution given that estimates did not account for between-study heterogeneity. Although a “best evidence” framework favors NMAs over simple naive comparisons, using this approach means it is important to clarify the underlying assumptions and limitations of the analysis.⁶³ In general, although methods for NMA models have become more standardized and it is possible to reproduce results based on study-level data, there is a clear need for more transparent assessment of effect modifiers and reporting, as per ISPOR recommendations.

Although the unanchored MAICs tended to be more targeted in terms of their research questions, the relevance to clinical practice was criticized because they made untestable assumptions. The validity of unanchored MAICs is largely dependent on how well the analysis accounts for differences in prognostic factors and treatment effect modifiers for both observed and unobserved variables between the trials indirectly compared. When conducting an NMA, randomization is preserved and therefore study-level differences in prognostic factors do not bias results; however, in an unanchored MAIC, there is uncertainty regarding the distribution of study-level differences (measured and unmeasured) that may influence the outcome of interest due to isolation of treatment arms in the analysis.⁶ In cases where the manufacturer has access to IPD from their trial(s), this additional information may allow for a more selective population matched to external studies in terms of between-study differences in patient characteristics, such as treatment experience. Nevertheless, these methods cannot be considered as strong as a standard NMA of RCTs given remaining uncertainty regarding other prognostic factors or treatment effect modifiers. Moreover, it is often the case that the external studies to which the populations are matched differ from the target populations of interest, which may limit the relevance for decision makers. Finally, the MAICs were generally not described in sufficient detail to allow for replication, which is also not feasible in the absence of IPD from the manufacturer. More detailed methods for the identification and selection of the prognostic factors and treatment effect modifiers, in addition to the model specification and selection process, are required to improve the transparency in combination with broader access to IPD.⁶⁶

An alternative approach to an unanchored MAIC would be to perform an outcomes-regression based indirect comparison or simulated treatment comparison (STC), based on a regression model for the index intervention for the outcome of interest as a function of relevant patient-related factors using the IPD of the index trial (ie, typically manufacturer trial[s]). This model could then be refit using the same IPD by centering the covariates at the mean values of the competing disconnected studies (for which only aggregate study-level data is available) to obtain the population-adjusted estimates with the index intervention for a target population as well as each of the competitor trials (ie, single-arm or disconnected RCTs). STCs may provide less biased estimates than an unanchored MAIC when the model is mis-specified (either outcome regression or propensity score model), although precision may be overestimated.⁶⁷ STCs may also be less prone to dramatic changes to Kaplan-Meier curves based on upweighting a very small number of individuals. Finally, STCs may also align more broadly with a multiple treatment framework, although this is an ongoing area of research.

Because a connected network may have been feasible by integrating evidence from an observational analysis, an anchored MAIC or a meta-regression combining study and patient-level data may provide more credible treatment estimates.^68,69 In general, access to IPD across treatment comparisons is necessary to truly evaluate the credibility of current estimates based on between-study comparisons, which would allow for a meta-regression to adjust for differences in treatment-effect modifiers and prognostic factors, as well as an assessment of the specified target population of interest in either a connected or disconnected network.^70–73

One assumption used in both the NMAs and MAICs regarding survival outcomes was that of a constant HR over time, which was not supported by the evidence informing these analyses. Time-varying HRs may have better represented the Kaplan-Meier data, and incorporation of a meta-regression accounting for previous lines of therapy may have been feasible had IPD been available.^60,61,69 When cost-effectiveness analyses integrate the synthesis estimates, outcomes are typically extrapolated over a lifetime horizon. This is particularly important in the context of RRMM, where many of the trials report immature OS. Small differences in the short term may have a meaningful impact when extrapolated over a lifetime horizon. Therefore HTA analyses informed by indirect comparisons assuming proportional hazard may be biased. Another limitation of the identified studies was that safety outcomes and observational evidence were not evaluated.

For between-trial or indirect comparisons in RRMM to yield clinically meaningful results, it is necessary to define a relevant research question with respect to all treatment effect modifiers, particularly in terms of the number of prior therapies and relevant comparators, which are likely to affect the type of prior therapy. In general, although NMAs of RCTs can provide credible comparative estimates, research questions are often defined too broadly. Although this may not be unique to the field of RRMM, the problem is accentuated in this disease area owing to the small number of trials published for each of the many treatment comparisons (given how rapidly the field has evolved). For more targeted research questions, it may be appropriate to consider anchored MAICs to achieve more balanced comparisons and to integrate single-arm studies earlier in the process. Nevertheless, this approach only makes sense if the external population to which the estimates are matched aligns with the target population of interest for decision makers.

Although much of the existing literature was driven by a UK HTA perspective based on single technology appraisals, timely multiple technology appraisals are needed for a rapidly expanding disease area. The use of cumulative or “living” NMAs (or cumulative evidence synthesis projects), whereby comparative analysis methods are agreed upon and continually updated for an entire disease area, may reduce the lag-time between new evidence being available and synthesis results.⁷⁴ This is in contrast to current standard practice whereby multiple evidence synthesis projects are ongoing simultaneously using different methods by different project teams.⁷⁵ All of these improvements will require a collaborative approach between clinicians and methodologists to ensure future evidence synthesis studies provide utility to clinicians, payers, and patients.