Perceptions of indirect treatment comparisons as an evidence base in oncology decision-making: results of an international survey of health technology assessment and payer decision-makers

Abstract

Aim:

Health technology assessment (HTA) and payer organizations are often faced with early decision-making in oncology. To design and conduct robust indirect treatment comparisons (ITCs), it is important to better understand HTA and payer decision-maker perceptions of ITCs. Here we aim to describe what individuals with HTA and payer experience see as the acceptability of ITCs for HTA and payer organization coverage and reimbursement decision-making.

Materials & methods:

This survey included 30 current and former HTA and payer decision-makers from five countries: Australia, France, Germany, the UK (n = 5 each) and the US (n = 10). Main outcomes included the ratings of acceptance of ITCs and the presence of well-defined methodological guidance for ITCs.

Results:

ITCs are generally accepted by participants in Australia and the UK but are more likely evaluated on a case-by-case basis in France, Germany and the US. Four of five participants in Germany and the UK, two of five in Australia and one of five in France reported that well-defined and prescribed criteria regarding the use of ITCs were in place.

Conclusion:

There is a need for harmonization of methods used to assess ITCs by HTA and payers, especially in the rapidly evolving treatment landscape in oncology.

Plain language summary

Survey of health technology assessment agencies and payer organizations from five countries about indirect treatment comparisons for cancer treatments

What is the article about?

People who work for health technology assessment (HTA) agencies and payer organizations often make decisions about paying for or recommending new cancer treatments. To make these decisions, they need to evaluate how the risks and benefits of new treatments compare with existing treatments. Results from head-to-head clinical studies are the preferred source of information when making these decisions. However, when head-to-head studies are limited or not available, a method called an indirect treatment comparison (ITC) can be used to compare the effectiveness and safety of different treatments. This study surveyed 30 people who have worked for HTA and payer organizations from five countries (Australia, France, Germany, the UK and the US) to find out their opinions about using ITCs to make decisions about treatment coverage and reimbursement.

What were the results of the study?

People from Australia and the UK were likely to accept results from ITCs. People from France, Germany and the US were willing to consider results from ITCs on a case-by-case basis. Four of five people in Germany, four of five people in the UK, two of five in Australia and one of five in France said that the current criteria for ITCs were well defined.

What did the study conclude?

Our study highlights the importance of harmonizing methods for ITCs, particularly for cancer treatments.

Shareable abstract

This survey of health technology assessment and payer organizations from five countries highlights the importance of harmonizing methods for indirect treatment comparisons for oncology treatments

When considering whether to cover or reimburse a new treatment, health technology assessment (HTA) bodies and payers worldwide need to evaluate how a product's risks and benefits compare with those of already available treatment options for the indication in question. Randomized controlled trials (RCTs) are considered the standard methodology used to compare healthcare interventions [1,2]. These are designed primarily to compare a treatment with a standard of care or placebo and help to inform regulatory decision-making. RCTs have the ability to reduce variation in the groups being assessed, allowing possible differences in the outcome to be ascribed to the intervention [3]. However, data from these RCTs alone are unlikely to provide enough information, given that it is usually impractical to compare the new treatment with all the available and relevant comparators across patient populations and healthcare systems. Furthermore, RCTs are not always feasible or ethical, such as in the setting of rare diseases or disease areas with high unmet need [1,4], or where there is a rapidly changing treatment landscape with multiple interventions being accepted into routine clinical practice.

Indirect treatment comparisons (ITCs) are a type of study in which a comparison of treatment effects is performed via different methods between two or among multiple interventions [5–7]. Both direct and indirect evidence can inform an ITC and data from sources other than randomized trials, such as observational studies, historical data from registries, or even more indirect evidence (such as data extrapolated from a population affected by a more common disease that shares some features with the rare disease in question), may be used as inputs for the analyses [1,8,9]. The data eventually used in the analyses also depend on the existing evidence base and guidelines [10,11], and various types of data can be used as inputs in sensitivity analyses [1,8,9]. There are also different types of ITCs, varying in methodologies. A network meta-analysis (NMA), often the most commonly used [12], compares three or more interventions and can use a combination of direct and indirect evidence [13]. A matching-adjusted indirect comparison (MAIC) involves propensity score weighting and includes both individual and aggregate patient data [14]. A simulated treatment comparison (STC) involves outcome regression and includes both individual and aggregate patient data [14].

RCTs typically require an extended period of time to complete [1]. While direct treatment comparisons should be attempted, when possible, an advantage of ITCs is that they can provide information in settings when direct evidence is unavailable. In addition, regarding coverage and reimbursement decisions, the ability of ITCs to compare treatments is an important feature [5]. Indirect treatment comparisons submitted by manufacturers are evaluated by HTA and payer organizations. However, HTA and payer organizations are often faced with early decision-making in oncology and other disease areas, frequently with immature data collected at an early stage of a clinical trial or limited follow-up period when the treatment effect may not have been fully observed. In these situations, ITCs often involve statistical methods that can integrate indirect evidence of efficacy (such as data with varying durations of follow-up) to account for the uncertainty associated with immature data and ultimately aid the estimation of the treatment effect [15].

Oncology is a unique treatment area from both a disease perspective and for HTA and payer organizations. There are many new oncology treatments, including gene therapy and precision medicine, that are being used earlier and for longer periods than more established treatments [16]. With these advances in treatment, many cancer types are moving from being a fatal disease to a chronic disease, and as such, the prevalence of cancer is increasing, along with the subsequent healthcare costs [17,18]. Oncology is often the first therapeutic area for drugs in development due to the increasing incidence, high medical need and incentives to invest in this area [19]. This evolving treatment landscape will likely require HTA and payer organizations to anticipate changes in how coverage and reimbursement decisions are assessed and made when data from RCTs are not available. This is especially true given that other study types, such as pragmatic and single-arm clinical trials, pose challenges due to potential biases and limitations [20,21].

To design and conduct the most robust ITCs, it is important to better understand HTA and payer decision-maker perceptions of ITCs. The objectives of this study were to describe what individuals from different countries with current and former HTA and payer experience currently see as the acceptability of ITCs for HTA and payer organization coverage and reimbursement decision-making. In addition, we aimed to assess the acceptability of study attributes, methods used and results of ITCs for the comparative assessment of drugs. Last, we evaluated the acceptability of ITCs for HTA and payer decision-making in the setting of three hypothetical oncology use examples.

Materials & methods

Study design

A web-based survey was conducted among 30 current and former HTA body employees, payer decision-makers, and health economists from five countries: Australia (n = 5), France (n = 5), Germany (n = 5), the UK (n = 5) and the US (n = 10). The Market Access Transformation Rapid Payer Response online portal was used to conduct the survey [22], which is a research platform with a current enrollment of approximately 2000 individuals who are or have been involved in value assessment on behalf of HTA agencies and payer organizations from around the world.

Questions were developed based on a desk review of primary research and current understanding of the payer landscape, focused on HTA-specific guidelines for ITC methods and how they can be used to inform payers' decisions. Thus, questions were asked on the following issues as perceived by HTA and payers: the acceptance of ITCs, the acceptance of different types of ITCs, data sources accepted, the preference of end points in ITCs, prioritization of criteria demonstrating ITC results of high quality, and credibility and use cases (Supplementary Table 1). For some questions relating to acceptance of ITCs, participants could select either accepted (always accepted), conditionally accepted (accepted under certain circumstances) or not accepted. Participants provided responses in multiple-choice, open-ended, Likert-type scale or ranked-choice formats to the questions (Supplementary Table 1). Participants were also asked to review three hypothetical cases in oncology and indicate whether an ITC would be accepted for HTA and payer decision-making. Case 1 was a small-molecule oral therapy studied in a single-arm trial for a small population (orphan indication) that targets a specific biomarker (i.e., first in class with a new mechanism of action) and no previous randomized clinical trials in the target indication, so an external comparator arm needs to be established. Case 2 was a biologic therapy for a cancer with high prevalence, studied with an RCT and a comparator that is no longer the standard of care and has marginal clinical benefit but signals improved efficacy in certain subgroups. Key comparators have been approved based on single-arm trials, and the manufacturer anticipates cross-trial differences between their product and comparator therapies. Case 3 was a breakthrough cell and gene therapy (high cost and potentially curative) studied in a single-arm open-label trial for a narrow patient population in the last-line setting (i.e., no treatment options, only best supportive care) and with available clinical trials of the historical standard of care in the target indication. For all questions, participants could provide open-ended responses detailing the reasoning for their choices. See Supplementary Table 2 for a copy of the survey. Participants completed the English-language, web-based survey in the period from 30 March 2023 through 10 April 2023.

Eligibility criteria

Participants for this study were selected from countries representing different HTA and payer archetypes and were recruited based on their expertise of country-specific HTA and payer organization coverage and reimbursement processes for healthcare interventions. All participants were familiar with the use of ITC methods for the assessment of healthcare interventions in their respective countries. All had at least 5 years of experience working in an HTA body or healthcare payer organization or as a health economist advising an HTA body. Each participant had at least 2 years of experience assessing healthcare interventions in their own country.

For participants from Australia, France, Germany and the UK, where any new therapy is evaluated through a public procedure by country-specific decision-making agencies, we selected participants based on their history of employment in pertinent organizations at the national and regional levels of decision-making. Participants from the US had a history of employment with private insurers, such as managed care organizations, pharmacy benefit manager services and integrated delivery networks. Participants were screened for their familiarity with current pricing and reimbursement mechanisms. In order to meet the study's eligibility criteria, participants had to rate their familiarity with current pricing and reimbursement mechanisms for healthcare interventions in their country as at least ‘somewhat familiar’ (3 out of 5). Participants were also screened for their familiarity with the use of ITCs for the assessment of healthcare interventions. In addition, participants had to rate their familiarity of current pricing and reimbursement mechanisms for healthcare interventions in their country as at least ‘somewhat familiar’ (3 out of 5).

Statistical analyses

Descriptive analyses of the participants' survey responses were performed. Categorical variables were summarized using frequencies and percentages. Continuous variables were summarized using means. Open-ended answers were reviewed and summarized. All surveys were completed and no answers were discarded from the results analysis. To ensure quality control and maximize completeness of the data, surveys were reviewed and follow-up clarifications were sent to participants if needed. Because the research was perception based, natural variations occurred across payers' interpretation of questions and their responses; therefore, the results do not reflect formal payer guidelines nor the organizations that they represent.

Results

Demographics of respondents

Table 1 shows key demographics of the respondents. Of the 30 participants, 29 had experience at the national level and 1 had experience at the regional level.

Table 1. . Participant experience by country.

Country	Participants (n)	Participant experience
Australia	5	• Former members of the Pharmaceutical Benefits Advisory Committee (n = 3) • Health economists in an advisory capacity to the Pharmaceutical Benefits Advisory Committee (n = 2)
France	5	• Former members of the Transparency Committee (Commission de la Transparence) (n = 3) • Former members of the Economic Committee for Health Products (Comité Economique des Produits de Santé) (n = 2)
Germany	5	• Former member of the Federal Joint Committee (Gemeinsamer Bundesausschuss) (n = 4) • Members of statutory health insurance organizations (e.g., association of physicians and funds) (n = 1)
UK	5	• Former committee members and advisors to the National Institute for Health and Care Excellence (n = 4) • Members and advisors of the National Health Service England (n = 1)
USA	10	• Medical and pharmacy directors from commercial plans (n = 6) • Pharmacy benefit managers who have experience in oncology and infusion or specialty pharmacy (n = 2) • Oncology pathway developers from integrated delivery networks (n = 2)

Acceptance of ITCs was most often reported in Australia (5 of 5 participants) and the UK (4 of 5 participants reported acceptance and 1 of 5 reported conditional acceptance), where respondents reported that ITCs were accepted when conducting RCTs was not feasible or the reasons for lack of direct evidence are assessed by the National Institute for Health and Care Excellence (NICE) or the Pharmaceutical Benefits Advisory Committee. In the US, 4 of 10 participants reported that ITCs were accepted to inform payer decisions, while 5 of 10 reported conditional acceptance and 1 of 10 reported not accepted. Acceptance of ITCs to inform decisions was least commonly reported in France (1 of 5 reported acceptance and 4 of 5 reported conditional acceptance) and Germany (1 of 5 reported acceptance, 3 of 5 reported conditional acceptance, and 1 of 5 reported not accepted). German respondents indicated that this lack of acceptance of ITCs was due to insufficient similarity of the compared studies or wrong comparator(s). In France, participants reported that ITCs would be accepted as a last resort or as supportive evidence and could be accepted if the infeasibility of direct comparison is justified and the methods are defined. In Australia, France, Germany, and the US, respondents indicated that ITCs were accepted when the clinical studies providing the primary data used in the ITC are of good quality and the assumptions are valid.

Four of 5 participants in Germany and the UK, 2 of 5 in Australia, and 1 of 5 in France reported that well-defined and prescribed criteria regarding the use of ITCs were in place to assess the comparative effectiveness of a new therapy. Most of the participants in Germany (4 of 5) and the UK (4 of 5) confirmed that well-defined criteria were in place. Examples of guidelines used included the Institute for Quality and Efficiency in HealthCare (IQWiG) methods guidelines, reported by participants from Germany; NICE Decision Support Unit documents, reported by participants from the UK; and Pharmaceutical Benefits Advisory Committee guidelines, reported by participants from Australia. No participants in the US reported that well-defined criteria were in place; only 3 of 10 participants reported that such criteria were present conditionally.

Participants agreed on the acceptance of ITCs when comparative clinical evidence is scarce. All participants from Australia and France, 8 of 10 in the US, 3 of 5 in the UK, and 2 of 5 in Germany reported conditional relaxation in the criteria/requirements for acceptance of ITCs in this setting. Situations for this include when an RCT is not feasible and the unmet need is important (e.g., pediatric indications or rare conditions), in ultra-orphan conditions when direct comparisons are not possible due to a small patient population, or if a new drug has a dramatic effect and there is a high unmet need. Participants from Australia reported that single-arm studies and access to patient-level data may have facilitated the relaxation of ITC criteria, such as in the case of the HTA assessment of new treatments for cystic fibrosis and chimeric antigen receptor T-cell (CAR-T) therapies. Participants from France indicated that relaxed criteria were allowed in the evaluation of CAR-T therapies and those for pediatric soft tissue sarcomas. Participants from the US indicated that relaxed criteria for ITCs have been used across therapy areas but most notably for orphan/ultra-orphan conditions when there are insufficient numbers of patients and limited therapies to conduct robust ITCs, such as in the setting of Duchenne muscular dystrophy or uveal melanoma. Participants from the UK indicated some relaxation in the application of ITC methodological guidance in the HTA assessment of CAR-T therapies and in the setting of neuroblastoma and mucopolysaccharidosis type IVA. Participants from Germany did not list any examples.

All participants in Australia, France, Germany, and the UK, and 8 of 10 in the US, reported that NMAs are accepted or conditionally accepted for HTA and payer decision-making (Figure 1A). This acceptance was reported to be driven by the transparency of the methods, adherence to guidelines and whether the results are published in peer-reviewed journals. All participants in Australia, France, the UK and the US would accept or conditionally accept results from MAICs (Figure 1A). Neither MAICs nor STCs were reported to be accepted by participants from Germany (Figure 1A).

Figure 1. . Level of acceptance of different indirect comparison methods by health technology assessment and payer decision-makers.

(A) Acceptance of NMA, MAIC and STC; and (B) unanchored indirect treatment comparison without a common comparator.

MAIC: Matching-adjusted indirect comparison; NMA: Network meta-analysis; STC: Simulated treatment comparison.

An unanchored ITC, without a common comparator, was reported as conditionally accepted by all participants in the UK, 4 of 5 in Australia and France, and 8 of 10 in the US. No participants in Germany reported conditional acceptance in this scenario (Figure 1B). This acceptance of ITCs was reported to be under specific conditions (e.g., if the burden of disease is high, the data sources are reliable and robust, if based on consensus from key opinion leaders, or if there are no available treatment alternatives currently on the market), and ITCs are considered to be a low grade of evidence and subject to significant uncertainty.

When participants were asked to rate data from different types of studies that would be accepted in different arms of ITC or relative effectiveness assessment by HTA bodies and payer organizations using a scale from 1 to 7 (1 = not accepted/low credibility; 7 = high acceptance/credibility) as a group (n = 30), they gave active-controlled RCTs the highest average rating: 6.8. The average ratings were as follows: Australia, 6.6; France, 7.0; Germany, 7.0; the UK, 6.6; and the US, 6.6 (Figure 2). Placebo-controlled RCTs ranked the second highest, with an average rating of 6.1, and pragmatic clinical, single-arm, non-randomized and observational trials were all ranked lower, with an average rating of 3.3, 3.2, 3.1 and 2.7, respectively (Figure 2). The average ratings for sources other than RCTs were lowest as reported by participants from Germany, and generally highest by participants from the US and the UK (Figure 2).

Figure 2. . Data sources accepted by health technology assessment and payer decision-makers in indirect treatment comparisons.

The diamond shows the average rating of all the countries for each study type.

RCT: Randomized controlled trial.

When participants were asked to rate on a scale from 1 to 7 (1 = not relevant; 7 = very relevant) the end points used to assess the magnitude of treatment effects with ITCs for HTA bodies and payer organizations, they rated time-to-event outcomes as being the most relevant; the average rating from all participants (n = 30) for time-to-event outcomes was 6.7 and the rating was higher than 6.0 for all countries (Australia, 6.8; France, 6.8; Germany, 7.0; the UK, 7.0; and the US, 6.2) (Figure 3). Physician-reported grade ≥3 adverse events had the second highest average rating, at 5.2, but was rated lower in the UK and the US (4.4 and 4.7, respectively) (Figure 3). Health-related quality of life (HRQoL) outcomes were rated higher in Australia, Germany and the UK (5.2, 6.2 and 6.4, respectively) and lower in France and the US (3.4 and 2.9, respectively) (Figure 3). All participants in Germany and most in the UK (4 of 5) reported that both primary and secondary end points are relevant. In both Australia and France, 3 of 5 participants reported that the relevance of primary/secondary outcomes is assessed on a case-by-case basis. In the US, 5 of 10 participants reported that both primary and secondary end points are relevant and 4 of 10 reported that only primary end points are relevant.

Figure 3. . Preferences of end points by health technology assessment and payer decision-makers in indirect treatment comparisons.

The diamond shows the average rating of all the countries for each end point.

AE: Adverse event; PRO-CTCAE: Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events.

Participants were asked to prioritize listed criteria (i.e., critical, valuable, nice-to-have, or not applicable) to demonstrate that ITC results are of high quality and credibility, are impactful, and are relevant for HTA and payer decision-making. Evidence base and methods used for ITCs were ranked as the highest priority in all countries except France (where it was the second highest after quality of included studies) (Figure 4). Quality of included studies was also commonly prioritized, whereas quality of reporting (e.g., adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses [PRISMA]-NMA [9]) and conflict of interest (e.g., sponsor of ITC) were less commonly prioritized (Figure 4).

Figure 4. . Payer prioritization of criteria demonstrating indirect treatment comparison results of high quality and credibility (overall and by country).

CI: Confidence interval; HR: Hazard ratio; ITC: Indirect treatment comparison; PRISMA-NMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Participants indicated whether an ITC would be accepted for HTA and payer decision-making for 3 hypothetical cases in oncology. Case 1 was a small-molecule oral therapy studied in a single-arm trial for a small population that targets a specific biomarker. All participants in Australia, the UK and the US, 4 of 5 in France, and 1 of 5 in Germany reported that an ITC would be accepted or conditionally accepted in this scenario (Figure 5). Case 2 was a biologic therapy for a cancer with high prevalence, studied with an RCT and a comparator that is no longer the standard of care. All participants in Australia and France, 4 of 5 in Germany and the UK, and 7 of 10 in the US reported that an ITC would be accepted or conditionally accepted in this scenario (Figure 5). Case 3 was a breakthrough cell and gene therapy studied in a single-arm open-label trial for a narrow patient population in the last-line setting. All participants from Australia, France, the UK and the US, and 3 of 5 in Germany, reported that an ITC would be accepted or conditionally accepted in this scenario (Figure 5).

Figure 5. . Acceptance of indirect treatment comparison for health technology assessment and payer decision-making in hypothetical oncology use cases.

Acceptance of ITC: results shown in n, number of responders.

BSC: Best supportive care; EHR: Electronic health record; ITC: Indirect treatment comparison; MAIC: Matching-adjusted indirect comparison; MoA: Mechanism of action; NMA: Network meta-analysis; RCT: Randomized controlled trial; RWD: Real-world data; SoC: Standard of care; STC: Simulated treatment comparison.

Discussion

These survey results provide insights into the acceptance and use of ITCs by HTA and payer decision-makers. Based on the views of participants in our study, acceptance of ITCs varies across markets, with ITCs being more accepted in the cost–effectiveness-driven markets of Australia and the UK and more commonly assessed on a case-by-case basis in France, Germany and the US. However, most participants agreed that ITCs are accepted in cases when comparative clinical evidence is scarce. The criteria for ITCs are well-defined in Germany and the UK, but most US participants noted a lack of clear methodological guidance. Participants reported that the credibility of the ITC results primarily depends on the evidence base, methods used for ITCs, and the inclusion of high-quality, relevant studies.

Participants also reported that NMAs are generally well accepted as a source of comparative effectiveness data in markets with established guidance on ITC methods (Australia, Germany, and the UK) and if they exclude sources of bias (France) or if the results are included in a peer-reviewed publication (the US). Results from MAICs and STCs were reported to be considered on a case-by-case basis in all countries except Germany. Similarly, an article assessing the use of real-world evidence in the US found that the Institute for Clinical and Economic Review also recommends NMAs for quantitative indirect comparisons in appropriate settings, such as when the patients and outcome measures are similar [23]. An evaluation of indirect comparisons in Germany, England, France, and Scotland reported that the Haute Autorité de Santé in France accepts adjusted indirect comparisons, the IQWiG in Germany recommends adjusted indirect comparisons, and NICE in England recommends using data from a mixed-treatment comparison [2].

When participants assessed data for acceptance in different arms of ITCs, active-controlled RCTs were highly valued and considered the gold-standard clinical data source. This is consistent with what has been found in other reports [23,24]. Placebo-controlled RCTs are also valued in all markets. However, there is low acceptance of study types that may introduce bias to the ITC analysis (i.e., pragmatic, single-arm, non-randomized, and observational trials), consistent with the potential biases and limitations in these types of studies [20,21]. These findings also highlight the potential need for HTA and payer organizations to evaluate how they currently, and will in the future, assess oncology treatments for coverage and reimbursement.

For participants, time-to-event outcomes (e.g., overall survival) were of the highest value when assessing the magnitude of treatment effects with ITCs. However, there were multiple end points related to oncology that were also rated relatively highly by participants, including physician-reported grade ≥3 adverse events and discontinuation rates. In contrast to a similar study surveying representatives of HTA bodies worldwide, which showed that HRQoL assessment was considered of higher importance in the US than in Europe [21], it was rated of lower importance in the US in our study. While payers indicated a willingness to accept an ITC with HRQoL outcomes, it was considered lower impact for payer decision-makers in the US and France. However, in Australia and the UK, where HRQoL is central to HTA submissions, HRQoL outcomes were rated higher.

Participants assessed the evidence base and methods, along with inclusion of high-quality studies, as the critical factors for the credibility of submitted ITC results. Another study evaluating HTA agencies' preferred methodologies for ITCs similarly found that HTA agencies often assess the quality of the ITCs [14].

When various hypothetical scenarios in oncology were assessed by participants, variation in acceptance of ITCs existed, depending on the scenario and country. In the case of a small-molecule oral therapy studied with a single-arm trial for a small population that targets a specific biomarker, inclusion of single-arm studies would generally be accepted and requirements for RCT would be relaxed for this condition. Historical real-world data will likely be required to contextualize the benefit in clinical practice; participants' preferred indirect comparison method would be an MAIC to adjust the relevant confounders. In Germany, ITCs for this case would not be accepted as no adjustment based on Bucher mechanism [5], NMA, or mixed-treatment comparison is possible. In the case of a biologic therapy for a high-prevalence cancer, studied with an RCT and a comparator that is no longer the standard of care and has marginal clinical benefit but signals improved efficacy in certain subgroups, an ITC would be conditionally accepted in most markets. This is because a considerable body of evidence from RCTs is available for high-prevalence cancer types, and thus, would supply adequate relevant data for an ITC. NMA would be the preferred method, but the manufacturer may need to justify why the standard of care is not the trial comparator. In the case of a breakthrough cell and gene therapy (high cost and potentially curative) studied in a single-arm open-label trial for a narrow patient population in the last-line setting, an ITC would be conditionally accepted in most markets, with single-arm studies and real-world data being the likely data sources due to a lack of marketed therapies in the target indication. There would likely be variability in acceptance of ITC methods, with the key criteria being the quality and relevance of the data collected. Future research could build upon the information obtained from these hypothetical scenarios and compare participant responses from our survey with a review and analysis of similar real-world cases.

Studies assessing the factors and processes for reimbursement recommendations across HTA bodies have found variability in the types of trials, end points, and conclusions drawn [25,26]. Similarly, in our study, the role of ITCs and impact on HTA and payer organization coverage and reimbursement varied by country. In Australia and the UK, where more extensive guidelines exist [14], ITCs were perceived to be more impactful. This finding is similar to that of another study, which found that ITCs were generally accepted by the Pharmaceutical Benefits Advisory Committee and NICE when head-to-head data were not available [25]. In France and Germany, ITCs were perceived to be less impactful. In Germany, head-to-head trials with direct evidence have been preferred by the IQWiG, with rigorous requirements to assess ITCs [2,14]. The results of our study are consistent, with a limited number of settings in which ITCs would be accepted. In the US, the impact of ITCs was perceived to vary on a case-by-case basis.

The responses from the participants in our study suggest that there is broad heterogeneity and lack of standardization in the methods and application of ITCs in the coverage and reimbursement decision-making processes in the US, Australia and western Europe. While there are potential limitations in using ITCs, there are also potential implications for patients if ITCs are not considered part of the coverage and reimbursement process in appropriate situations. Despite the presence of published methodological guidance on the development of ITCs for economic appraisal [27–30], there is a lack of standardization in the application of these methods by former HTA and payer decision-makers and health economists in our study. This suggests that there is a need for greater harmonization of these methods considering the evolving HTA and payer organization landscape. A recent targeted literature review conducted by Tanaka et al. [31], found similar shortcomings with inconsistencies, gaps and ambiguities in the application of current ITC guidelines [31], as reported in our cross-sectional survey. In another study, Macabeo et al. [32] analyzed HTA evaluation reports published between April 2018 and April 2021 across England, France, Germany, Italy and Spain [32]. While their methodology differed from our study, Macabeo et al. [32] also concluded that there were differences in the perception of ITC methods between the countries included in their study and that there is a need for additional clarity on ITC techniques and their assessment [32]. Macabeo et al. [6] also critically reviewed and synthesized the literature on a broad range of contemporary ITC guidelines. Some of the findings of this work mirrors those in our study, namely the importance of ITCs in the absence of head-to-head clinical trials and the need for up-to-date and standardized guidance, and highlights the role ITCs play in the comparative assessment of novel therapies across different jurisdictions [6]. While our study was limited to the opinions of the participants with expertise of country-specific HTA and payer organization coverage and reimbursement processes, future research surveying a larger sample with broader organizational archetype experience and perspective and geographical scope could add to the generalizability of our findings. In addition, comparing our findings with real-world examples of the application and influence of ITCs in payer decision-making could help to contextualize our findings within the broader HTA and payer organization landscape.

Limitations

The target markets were selected to capture a range of HTA and payer decision-makers. However, the sample size was small and there were more participants in the US than in the other individual countries surveyed. Therefore, the overall survey results are not equally weighted among the countries of interest. Due to the presence of open-ended questions, not all data could be summarized using descriptive statistics. However, this format allowed participants to provide further insight on these topics and include their own experiences. The questions could be interpreted differently by participants.

Contents related to MAICs in our survey were broadly assessed, but we did not ask participants about their perceptions of specific MAIC methodologies. Future research could examine HTA and payer decision-maker perceptions of the various types of MAICs and potential situations when they could be used.

Many aspects of our study are specific to ITCs in the oncology setting, such as the assessment of the relevance of oncology-specific end points for HTA and payer decision-makers and the hypothetical oncology cases. Additionally, many examples provided by participants were specific to oncology. However, we did include questions in the context of ITCs in general, such as those pertaining to the acceptance of ITCs, data sources accepted for ITCs, and the credibility of ITC results. As such, a benefit of our study is that there are many aspects that could be extrapolated to other disease areas.

While potential biases associated with the small sample size are recognized, these insights capture recent trends and practical applications that might not be fully reflected in formal assessments or guidelines. These complementary data can aid in understanding discrepancies, identifying gaps, and illustrating how guidelines are interpreted in practice, thereby informing future research, guideline development, and payer decisions.

Conclusion

The results of this study raise awareness surrounding the need for HTA and payer organizations to take a pragmatic approach to answer differing research questions, with greater consideration for the different and evolving methodologies employed. This is key to helping to interpret the evidence coming from indirect comparisons, in anticipation of future developments in oncological therapeutics and the evolving HTA landscape [33–35]. A full alignment between national and European Network for Health Technology Assessment guidelines is not yet a reality. The results of our study suggest that this could help standardize methodologies and best practices across countries for assessing the added value of a product and promote transparency. This can lead to increased confidence in the findings and facilitate collaboration and expertise sharing across countries.

As the use of ITCs continues to grow, future research can focus on developing and evaluating methods for conducting ITCs that are robust and transparent, investigating the impact of different methodological choices on the results of ITCs and conducting further studies to assess the impact of ITCs on oncology treatment access. This will help ensure that HTA and payer organizations use ITCs appropriately and make informed decisions about treatment access, which will ultimately help to ensure that the most appropriate treatments are available for patients. Further transparency regarding payer expectations for ITCs will support continued innovative oncology treatment development by manufacturers and ensure that the most appropriate evidence packages are developed to support treatment assessment.

Summary points

• Understanding health technology assessment (HTA) and payer decision-maker perceptions of indirect treatment comparisons (ITCs) is important to support the design and conduct of robust ITCs.

• Based on the opinions of participants in this study, acceptance of ITCs varies across markets, with ITCs being more accepted in cost–effectiveness-driven markets of Australia and the UK and more commonly assessed on a case-by-case basis in France, Germany and the US.

• Most participants agreed that ITCs are accepted in cases when comparative clinical evidence is scarce.

• The criteria for ITCs are well defined in Germany and the UK, but most participants in the US noted a lack of clear methodological guidance.

• Participants reported that the credibility of ITC results primarily depends on the evidence, the methods used for ITCs, and the inclusion of high-quality, relevant studies.

• Participants also reported that network meta-analyses are generally well accepted as a source of comparative effectiveness data in markets with established guidance on ITC methods (Australia, Germany and the UK), if they exclude sources of bias (France), or if the results are included in a peer-reviewed publication (the US).

• Results from matched-adjusted indirect comparisons and simulated treatment comparisons are considered on a case-by-case basis in most countries, except Germany.

• The results of our study suggest that there is broad heterogeneity and lack of standardization in the methods and application of ITCs in the coverage and reimbursement decision-making processes in the US, Australia, and western Europe.

• This suggests a need for greater harmonization of ITC methods, considering the evolving HTA and payer organization landscape.