Abstract
Abstract
Objectives
To investigate the association between incremental quality-adjusted life years (QALYs) predicted in economic evaluations and clinical benefits assessed by the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS), examining how accurately predicted QALYs reflect actual clinical outcomes in cancer drug reimbursement decisions.
Design
Cross-sectional observational study.
Setting
Health technology assessment (HTA) documents from Australia, Canada and England, supplemented by published economic evaluations from China. Economic evaluation data were collected from database inception to 31 December 2023.
Participants
A total of 240 economic evaluation documents were identified from Australia (n=61), Canada (n=114) and England (n=65), along with 106 published studies from China, all focused on solid tumour cancer drugs with publicly available ESMO-MCBS scores. Documents were included based on completeness and consistency of data sources; those that were incomplete or relied on external controls were excluded.
Primary and secondary outcome measures
The primary outcomes were the incremental QALYs from manufacturer submissions and HTA agency reevaluations. Secondary outcomes included associations stratified by data maturity, country, treatment setting and reimbursement recommendations.
Results
Incremental QALYs showed a moderate positive correlation with ESMO-MCBS scores (Spearman’s ρ=0.42, 95% CI: 0.31 to 0.53). All country-specific correlations were statistically significant: England (ρ=0.53), Australia (ρ=0.37), Canada (ρ=0.39) and China (ρ=0.49), all p<0.01. Stronger associations were observed in HTA agency reevaluations (adjusted OR=1.43, 95% CI: 1.15 to 1.77) compared with manufacturer submissions (OR=1.21, 95% CI: 1.09 to 1.34). Analyses limited to mature data (>70% events observed) demonstrated the strongest association (OR=1.53, 95% CI: 1.10 to 2.13). Among countries, England exhibited the highest association (OR=1.42, 95% CI: 1.15 to 1.74), followed by China (OR=1.30, 95% CI: 1.04 to 1.62), Australia (OR=1.28, 95% CI: 1.01 to 1.63), and Canada (OR=1.15, 95% CI: 1.05 to 1.26).
Conclusions
This study highlights a moderate alignment between incremental QALYs derived from economic evaluations and clinical benefit scores, emphasising the importance of rigorous reassessment, mature survival data and independent validation processes. Future research should explore strategies for enhancing data maturity and incorporating independent review mechanisms to strengthen healthcare decision-making globally.
Keywords: Health economics, Health policy, Quality in health care
STRENGTHS AND LIMITATIONS OF THIS STUDY.
The study systematically collected and analysed data from publicly available health technology assessment (HTA) reports across multiple countries.
The use of the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale, a widely validated tool, enhanced the comparability of clinical benefit assessment.
Data maturity was stratified to clearly illustrate its influence on quality-adjusted life year predictions.
The exclusion of haematological malignancies may limit the generalisability of findings.
Chinese data were obtained via a systematic literature review rather than HTA reports, which may introduce publication or selection bias.
Introduction
Economic evaluation has been widely used to inform healthcare resource allocation, price negotiation and insurance coverage decision-making for cancer drugs.1,3 In such evaluations, incremental quality-adjusted life years (QALYs) are commonly employed to measure the health benefit of drugs.4 However, previous studies have demonstrated that immature survival data and reliance on external controls can introduce significant uncertainties in the predicted incremental QALYs for cancer drugs.5 6 These uncertainties can undermine the reliability of reimbursement decisions, potentially leading to suboptimal resource allocation.5 7 8 For instance, an analysis of cancer drugs in the USA and Europe has demonstrated a lack of significant correlation between the costs of treatment for solid tumours and the clinical benefits achieved.9 These findings underscore the urgent need to evaluate whether QALYs derived from economic evaluations accurately reflect the true clinical benefits of cancer drugs.
The accuracy of predicted QALY benefits can typically be assessed by comparing extrapolated data from early clinical trials with updated results as they become available. However, the systematic implementation of this approach is often constrained by the lack of long-term follow-up data.10 A retrospective study found that 55.6% (40 out of 72) of cancer drug clinical trials did not publish updated survival data within 5 years of their initial publication.10 This limitation necessitates alternative methods for validating QALY predictions. While clinical benefit assessment tools could offer a potential solution, their retrospective nature means that they are unavailable until clinical benefit scales are established. The European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS) assesses the clinical benefit of cancer drugs based on primary and secondary endpoints in clinical trials.11 This scale is widely used in regulatory science12,14 and employs a methodology similar to that used in economic evaluation documents, where models are developed using clinical trial data to appraise the value of drugs.15 Given its robust framework and widespread use, the ESMO-MCBS offers a feasible tool to assess the validity of predicted incremental QALYs in reflecting the actual clinical benefits of cancer drugs.
This study aimed to systematically evaluate economic evaluation documents submitted for reimbursement by examining the association between ESMO-MCBS scores and incremental QALYs. By exploring this relationship, the study seeks to determine whether QALY predictions align with the clinical benefits observed, thereby contributing to the refinement of economic evaluation practices and enhancing the reliability of healthcare decision-making.
Materials and methods
Study design and data sources
This cross-sectional study selected Australia, Canada and England due to the availability of publicly accessible health technology assessment (HTA) reports. Each of these countries has established an HTA agency: the Pharmaceutical Benefits Advisory Committee (PBAC) in Australia, the Canadian Agency for Drugs and Technologies in Health (CADTH) in Canada and the National Institute for Health and Care Excellence (NICE) in England. Public documents from the official websites of PBAC16 in Australia, CADTH17 in Canada and NICE18 in England served as the primary data source for our analysis. These documents comprise clinical and economic evaluation reports submitted by pharmaceutical manufacturers, reanalysis outcomes from HTA agencies and drug reimbursement recommendations. We collected public documents from database inception until 31 December 2023 (note that PBAC started to publish economic evaluation results on 1 January 2015). To gain insights into the situation in developing countries, we selected China for a supplementary analysis. However, as China has not made the economic evaluation documents publicly available, we performed a systematic literature review (see online supplemental A for further details) of published economic evaluations conducted in China.2
We used the ESMO-MCBS to determine the clinical benefit of cancer drugs. This tool has been designed and validated to objectively assess the clinical benefit based on data from the relevant clinical trials.11 19 ESMO-MCBS scorecards are available online,20 providing ratings certified by ESMO for numerous tumour therapies.
Eligibility criteria
The following eligibility criteria were applied. First, our study was focused on economic evaluation documents related to solid tumours because the ESMO-MCBS scoring system for haematological malignancies (H Scorecard) was only introduced in 2023 and currently has limited publicly available data, whereas the version for solid tumours is extensively validated and widely adopted. Second, we chose the most recent public documents for research inclusion when multiple public documents existed for the same drug indication. Third, documents that were not complete, such as active, cancelled, pending, not filed and withdrawn, were excluded. Fourth, we included only clinical trials explicitly identified in HTA reports as pivotal for the economic evaluations and matched these with the corresponding ESMO-MCBS online scorecards. Finally, economic evaluation documents that relied on external controls were excluded to maintain consistency in data sources.
Data extraction
For studies that met the inclusion criteria, the following variables were extracted: release date, tumour type, treatment setting, clinical trial, cut-off date for analysis, number of deaths, sample size of the clinical trial, incremental QALYs submitted by pharmaceutical manufacturers, incremental QALYs reanalysed by HTA agencies (or the lower bounds of reported range if a point estimate was not provided), and final reimbursement recommendation, and the corresponding ESMO-MCBS score obtained from the online scorecard for the same clinical trial. For each eligible economic evaluation, both the manufacturer-submitted and, where available, HTA-reassessed incremental QALYs were recorded. When multiple public versions of a submission were available, the most recent version was selected. If an HTA report presented a range rather than a single value, the lower bound was taken as a conservative estimate, in accordance with the approach noted in the Results section.
Terminology and definition
Key variables that define the scope of economic evaluation documents and affect the association between incremental QALYs and ESMO-MCBS scores were selected and included in the analysis. The ESMO-MCBS assigns a score of 1 (low) to 5 (high) for drugs used in the non-curative setting and a score of C (low) to A (high) for drugs that are potentially curative. Substantial clinical benefit refers to ESMO-MCBS scores of 4–5 (non-curative setting) or A–B (curative setting), representing high levels of clinical benefit. Data maturity was defined according to a published study that examined immature survival data for anticancer drugs submitted to the NICE.6 Data maturity was defined as the proportion of observed deaths in the pivotal clinical trial. For each trial, we recorded the trial cut-off date, number of deaths and total sample size. Based on these values, data maturity was classified as follows: <20% (very immature), 20% to ≤50% (low), 50% to ≤70% (moderate) or >70% (high). The cancer site, categorised according to ICD-10 (international classification of diseases, 10th revision) codes, was digestive organs; respiratory and intrathoracic organs; melanoma and other malignant neoplasms of skin; mesothelial and soft tissue; breast; female genital organs; male genital organs; urinary tract; eye, brain and other parts of central nervous system; thyroid and other endocrine glands. Treatment setting was divided into neoadjuvant or adjuvant and advanced or metastatic. The recommendation for reimbursement was categorised as negative or positive.
Statistical analysis
We calculated Spearman’s rank correlation coefficient (ρ) between incremental QALYs and ESMO-MCBS rating for cancer drugs. Correlation coefficients (ρ) of 0.1 to <0.2, 0.2 to <0.4, 0.4 to <0.7 and ≥0.7 were interpreted as very weak, weak, moderate and strong correlations, respectively.21 22 We formally tested ρ=0 using p values computed via asymptotic t approximation. Additionally, we used multiple logistic regression to further assess the association between incremental QALYs and ESMO-MCBS scores. The dependent variable was defined as whether the cancer drugs were classified as having substantial benefit, with incremental QALYs serving as the independent variable. Covariates included cancer site and treatment setting. Considering that a one-unit increase in QALY is highly significant in economic evaluations, we scaled the QALY values in our database by a factor of 10 to reduce the sensitivity of the OR.
All documents were included in this regression analysis. The adjusted OR with 95% CIs is reported. All statistical analyses were done in R (V.4.3.1) using ggplot2 (V.3.4.4) for plots. P values below 0.05 were considered statistically significant.
Patient and public involvement
The authors declare no patient or public involvement in this study.
Results
Selection of economic evaluation documents
Our database search yielded 3469 public documents (1613 from PBAC, 1128 from CADTH and 728 from NICE). After screening the titles and removing duplicates, we found 675 potentially eligible documents. Following a review of these documents, we included 240 drug-indication pairs from the three countries in the analysis (figure 1).
Figure 1. Overview of study selection for analysis. CADTH, Canada’s Drug and Health Technology Agency; ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; NICE, National Institute for Health and Care Excellence; PBAC, Pharmaceutical Benefits Advisory Committee; QALYs, quality-adjusted life years.
Description of included documents
Our study cohort included 240 economic evaluation documents, with 61 (25.4%) from PBAC, 114 (47.5%) from CADTH and 65 (27.1%) from NICE. Among these documents, the most common cancer sites were respiratory and intrathoracic organs (n=44, 18.3%), breast (n=42, 17.5%) and digestive organs (n=37, 15.4%). Most documents (n=212, 88.3%) were for the advanced or metastatic setting. Based on data maturity, 11.7% of the documents had a maturity level below 20%, 32.9% had a maturity level between 20% and 50%, 20.4% had a maturity level between 50% and 70% and 21.3% had a maturity level above 70%. Drugs from 185 documents (77.1%) were recommended for reimbursement. The median incremental QALYs reported by manufacturers was 0.6 (IQR, 0.4–1.1), and nearly two-thirds of documents (n=156, 65.0%) had substantial clinical benefit evaluated by the ESMO-MCBS. The characteristics of the included documents are detailed in table 1 and the basic information of the included economic evaluation documents is detailed in online supplemental Table 1.
Table 1. Comparison of characteristics of cancer drug economic evaluation documents published by CADTH, NICE and PBAC.
| Characteristic | Economic evaluation documents, no. (%) | |||
|---|---|---|---|---|
| All (n=240) |
CADTH (n=114) |
NICE (n=65) |
PBAC (n=61) |
|
| ICD-10 cancer site | ||||
| C15–C26 digestive organs | 37 (15.4) | 15 (13.2) | 10 (15.4) | 12 (19.7) |
| C30–C39 respiratory and intrathoracic organs | 44 (18.3) | 21 (18.4) | 15 (23.1) | 8 (13.1) |
| C43–C44 melanoma and other malignant neoplasms of skin | 26 (10.8) | 14 (12.3) | 7 (10.8) | 5 (8.2) |
| C45–C49 mesothelial and soft tissue | 10 (4.2) | 4 (3.5) | 3 (4.6) | 3 (4.9) |
| C50–C50 breast | 42 (17.5) | 22 (19.3) | 10 (15.4) | 10 (16.4) |
| C51–C58 female genital organs | 18 (7.5) | 8 (7.0) | 3 (4.6) | 7 (11.5) |
| C60–C63 male genital organs | 20 (8.3) | 9 (7.9) | 5 (7.7) | 6 (9.8) |
| C64–C68 urinary tract | 27 (11.3) | 11 (9.7) | 9 (13.9) | 7 (11.5) |
| C69–C72 eye, brain and other parts of central nervous system | 6 (2.5) | 2 (1.8) | 2 (3.1) | 2 (3.3) |
| C73–C75 thyroid and other endocrine glands | 10 (4.2) | 8 (7.0) | 1 (1.5) | 1 (1.6) |
| Treatment setting | ||||
| Advanced/metastatic | 212 (88.3) | 98 (86.0) | 61 (93.9) | 53 (86.9) |
| Neoadjuvant/adjuvant | 28 (11.7) | 16 (14.0) | 4 (6.2) | 8 (13.1) |
| Data maturity (% death) | ||||
| <20% | 28 (11.7) | 17 (14.9) | 5 (7.7) | 6 (9.8) |
| 20%–50% | 79 (32.9) | 45 (39.5) | 17 (26.2) | 17 (27.9) |
| 50%–70% | 49 (20.4) | 23 (20.2) | 14 (21.5) | 12 (19.7) |
| >70% | 51 (21.3) | 20 (17.5) | 14 (21.5) | 17 (27.9) |
| NR | 33 (13.8) | 9 (7.9) | 15 (23.1) | 9 (14.8) |
| Reimbursement recommendation | ||||
| Negative | 55 (22.9) | 12 (10.5) | 14 (21.5) | 29 (47.5) |
| Positive | 185 (77.1) | 102 (89.5) | 51 (78.5) | 32 (52.5) |
| Substantial clinical benefit by the ESMO-MCBS | ||||
| No | 84 (35.0) | 38 (33.3) | 25 (38.5) | 21 (34.4) |
| Yes | 156 (65.0) | 76 (66.7) | 40 (61.5) | 40 (65.6) |
| Incremental QALYs, median (IQR) | 0.6 (0.4–1.1) | 0.6 (0.4–1.3) | 0.6 (0.3–1.0) | 0.5 (0.3–0.8) |
CADTH, Canada’s Drug and Health Technology Agency; ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; NICE, National Institute for Health and Care Excellence; NR indicates that the proportion of death was not reported; PBAC, Pharmaceutical Benefits Advisory Committee; QALY, Quality-adjusted life year.
Correlation analysis results
Spearman’s rank correlation analysis demonstrates a positive correlation between incremental QALYs and ESMO-MCBS scores (figure 2). The analysis included all 240 economic evaluation documents; 154 (64.2%) reported reanalysed incremental QALYs by HTA agencies. Among these, 109 documents (70.8%) were published by CADTH, and 45 (29.2%) by NICE. Incremental QALYs submitted by manufacturers exhibited a moderate positive correlation with ESMO-MCBS scores (ρ=0.42, 95% CI: 0.31 to 0.53). This correlation increases slightly when using the incremental QALYs reanalyzed by HTA agencies (ρ=0.49, 95% CI: 0.36 to 0.61).
Figure 2. Incremental QALYs stratified by clinical benefit using the ESMO-MCBS. (A) Boxplots of incremental QALYs submitted by manufacturers for cancer drugs with no substantial benefit and substantial benefit with Spearman correlation coefficients ρ and asymptotic p values. (B) Boxplots of incremental QALYs reanalysed by HTA agencies for cancer drugs with no substantial benefit and substantial benefit with Spearman correlation coefficients ρ and asymptotic p values. ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; HTA, health technology assessment; QALYs, quality-adjusted life years.
Figure 3 shows the results of the subgroups of Spearman’s rank correlation analysis between incremental QALYs and ESMO-MCBS scores. Subgroup analysis by country shows that the correlation is strongest in England (ρ=0.53, 95% CI: 0.32 to 0.69, p<0.01), followed by Canada (ρ=0.39, 95% CI: 0.22 to 0.54, p<0.01) and Australia (ρ=0.37, 95% CI: 0.12 to 0.57, p<0.01). When stratified by cancer site, the correlation is most pronounced for thyroid and other endocrine glands (ρ=0.70, 95% CI: 0.10 to 0.93, p=0.03) and eye, brain and other parts of the central nervous system (ρ=0.83, 95% CI: 0.02 to 0.98, p=0.04). In contrast, for melanoma and other malignant neoplasms of the skin, mesothelial and soft tissue, and genital organs, the correlation is not statistically significant. Regarding treatment setting, advanced/metastatic cases show a significant positive correlation (ρ=0.38, 95% CI: 0.25 to 0.49, p<0.01), as do neoadjuvant/adjuvant settings (ρ=0.39, 95% CI: 0.01 to 0.67, p=0.04). Correlation strength varies by data maturity, with the strongest correlation observed when data maturity exceeds 70% (ρ=0.54, 95% CI: 0.31 to 0.72, p<0.01). Finally, the correlation is stronger for drugs receiving negative reimbursement recommendations (ρ=0.56, 95% CI: 0.34 to 0.72, p<0.01) compared with those with positive recommendations (ρ=0.32, 95% CI: 0.18 to 0.44, p<0.01). The correlation between incremental QALYs and ESMO-MCBS scores for subgroups across countries is detailed in online supplemental figure 1–3.
Figure 3. Correlation between incremental QALYs and ESMO-MCBS scores for subgroups: digestive organs, C15–C26; respiratory and intrathoracic organs, C30–C39; melanoma and other malignant neoplasms of skin, C43–C44; mesothelial and soft tissue, C45–C49; breast, C50; female genital organs, C51–C58; male genital organs, C60–C63; urinary tract, C64–C68; eye, brain and other parts of central nervous system, C69–C72; thyroid and other endocrine glands, C73–C75. ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; QALYs, quality-adjusted life years.
Logistic regression analysis results
All 240 economic evaluation documents were included in the logistic regression. The logistic regression analysis reveals significant associations between incremental QALYs and ESMO-MCBS scores across various subgroups (figure 4). Incremental QALYs reanalysed by HTA agencies show the stronger association, with an adjusted OR of 1.43, compared with 1.21 for those submitted by manufacturers. Across different countries, England exhibits the most pronounced association (OR=1.42, 95% CI: 1.15 to 1.74, p<0.01), followed by Australia (OR=1.28, 95% CI: 1.01 to 1.63, p=0.04) and Canada (OR=1.15, 95% CI: 1.05 to 1.26, p<0.01). Data maturity also influences the strength of the association, with the highest OR observed in data sets with over 70% death (OR=1.53, 95% CI: 1.10 to 2.13, p=0.01). Additionally, the analysis reveals a strong association for drugs with negative reimbursement recommendations (OR=1.65, 95% CI: 1.15 to 2.38, p=0.01), while those with positive recommendations also show a significant, although smaller, association (OR=1.14, 95% CI: 1.05 to 1.24, p<0.01).
Figure 4. Association between incremental QALYs and ESMO-MCBS scores for subgroups. The results should be interpreted as follows: for each additional 0.1-unit increase in QALY, the likelihood of a drug being assessed as having significant clinical benefit increases by the corresponding OR. ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; HTA, health technology assessment; QALYs, quality-adjusted life years.
Supplementary analysis for China
We identified a total of 1986 published economic evaluations through our electronic database searches. After removing duplicates, we screened 1023 studies and assessed 387 full-text articles based on inclusion and exclusion criteria. Eventually, 106 economic evaluation documents involving 35 cancer drugs and 57 indications were included in the systematic review (see online supplemental Table 2 and figure 4 for further details). We found a moderate correlation between incremental QALYs and ESMO-MCBS scores for China (ρ=0.49, 95% CI: 0.25 to 0.67, p<0.01) (online supplemental figure 5). In addition, the logistic regression analysis reveals significant associations between incremental QALYs and ESMO-MCBS scores, with an adjusted OR of 1.30 (95% CI: 1.04 to 1.62, p=0.02).
Discussion
Our observational study revealed a moderate correlation between incremental QALYs and ESMO-MCBS scores, which is positively affected by both the evidence review mechanism and the data maturity of pivotal trials. Given the importance of economic evaluation evidence in drug reimbursement decisions, our findings have significant implications for evidence regulation.
Our study found a moderate correlation between incremental QALYs and ESMO-MCBS scores, indicating that while QALYs provide useful information, they contain significant uncertainty in fully capturing clinical benefit. As such, HTA agencies should interpret QALYs with caution—particularly when data maturity is low—and should more systematically integrate structured clinical-benefit measures, such as the ESMO-MCBS, into economic evaluation framework to improve decision-making reliability. Notably, 84.7% (122 out of 144) of the documents experienced a decrease in incremental QALYs following reevaluation by HTA agencies, indicating a significant sponsorship bias in economic evaluation documents. This finding aligns with prior research and highlights the critical role of re-evaluation in reducing bias and improving the reliability of economic evaluations.23 24 To address such bias, many HTA agencies convene expert review groups to appraise manufacturer-submitted models and revise results where necessary. Establishing independent, multidisciplinary committees could further enhance objectivity and methodological rigour in this process.25 26 Previous studies have found that the uncertainty in the estimates of comparative clinical efficacy was a common issue when reviewing the model by the expert group.27 28 However, our findings suggest that HTA reassessments, while valuable, achieve only modest improvements in aligning QALY estimates with clinical benefit scales (ρ=0.49 vs 0.42). This limited improvement indicates that current reassessment practices may not fully resolve the uncertainties embedded in initial evaluations. To strengthen the robustness and transparency of reimbursement decisions, greater emphasis should be placed on prioritising evaluations based on high-maturity clinical data, and on public disclosure of underlying assumptions, methodologies and data sources used in QALY estimation. Finally, cross-jurisdictional sharing of best practices may further enhance the methodological rigour and consistency of oncology HTA.
Moreover, a novel contribution of this study is the quantification of the relationship between trial data maturity and the reliability of QALY estimates. Drugs with pivotal trial data maturity exceeding 70% showed a stronger correlation between QALYs and ESMO-MCBS scores. Economic models have been widely used in economic evaluations, and model-based analyses usually require the input of survival curves observed in clinical trials to make long-term projections.29 30 Previous studies have demonstrated that the accuracy of predicted results is impacted by immature survival data when comparing model predictions with updated clinical trial results.6 10 31 Our study quantified a clear, monotonically increasing relationship between data maturity and the accuracy of incremental QALY estimates. Despite this, using immature survival data to inform reimbursement decisions is common across countries.6 32 To address this issue, governments and healthcare institutions should promote the collection of longitudinal data to improve data maturity, enabling more reliable economic evaluations and better-informed healthcare decisions. Additionally, HTA agencies should prioritise data with high maturity levels and consider establishing minimum data maturity thresholds (eg, >70% observed events) to reduce uncertainty in QALY estimates. Several sources of bias could affect our findings. First, because Spearman correlation and logistic regression assume independence and homogeneity of variance, residual overdispersion may persist and attenuate effect estimates. Second, incremental QALYs derived from immature survival data are prone to under-estimating or over-estimating long-term benefits, thereby introducing additional uncertainty. Although a sensitivity analysis excluding very immature trials could have helped mitigate this concern, we elected not to rerun the primary models for two reasons: (1) such exclusions would have substantially reduced statistical power in several subgroups, and (2) our primary objective was to describe real-world submissions, which routinely include such early-phase data with limited maturity.
In addition, we found variations in the association among different countries. These differences in reimbursement decision criteria for cancer medicines across Australia, Canada and England are likely the primary contributing factor.3 The existence of an explicit threshold may have resulted in a stronger correlation of NICE than CADTH and PBAC.33 For the manufacturers preparing submissions to CADTH or the PBAC, the cost-effectiveness threshold is uncertain, and therefore a lower ICER may be presented to increase the chances of receiving a positive decision. Unfortunately, this study could not test this hypothesis without controlling for other factors that could influence the correlation. Additionally, subgroup differences by cancer type and treatment setting may reflect underlying biological heterogeneity in prognosis and treatment response, both of which affect survival extrapolation and, consequently, the estimation of QALYs.
In developing and underdeveloped countries, the scarcity of healthcare resources amplifies the urgency of ensuring the accuracy of economic evaluation evidence in drug reimbursement decision-making. The results of our complementary analysis confirm this view. Developing countries should adopt and adapt well-established international economic evaluation frameworks.34 Implementing standardised methods and metrics will ensure consistency and scientific rigour in drug evaluations. In addition, advocating for developing countries to establish mature information disclosure mechanisms to enhance decision-making transparency is crucial for improving the quality of economic evaluation evidence and reducing bias.2
As noted in our findings, the association between incremental QALYs and ESMO-MCBS scores was stronger for cancer drugs that were not reimbursed. One possible explanation is that a higher proportion of cancer drugs with substantial clinical benefit assessed by ESMO-MCBS were recommended for reimbursement (71.9% vs 41.8%, p<0.01), and longer survival extrapolation (0.67 QALYs vs 0.40 QALYs, p<0.01) increased the uncertainty of incremental QALYs.10 Moreover, therapeutic impact is not the only factor, and decision-makers often consider other important dimensions such as cost, mechanisms for public participation and socioeconomic impact.3
Our study also has some limitations. First, while the ESMO-MCBS provides a standardised method for evaluating the clinical benefit of anticancer drugs, its assessments are not fixed and may change with the emergence of updated clinical trial results. Second, Australia, Canada and England were chosen as the primary data sources due to the availability of publicly accessible economic evaluation documents. However, it is important to note that these countries may not offer a comprehensive representation of global situations, potentially constraining the generalisability of our study findings. Third, due to the limited transparency of Chinese economic evaluation documents, we use published literature as a complementary data source. Consequently, the results of the analysis cannot be directly compared with those of the other three countries. Fourth, due to the secrecy around the drug price, some documents conceal the results, leading to a reduction in sample size. Fifth, we only included documents of solid tumours, and further evaluation of haematological malignancies is required after the publication of the ESMO-MCBS: H Scorecards. Sixth, 28.1% (32/114) of the economic evaluation documents in Canada did not directly report the results of the reanalysis of Economic Guidance Panel (EGP), which were replaced by the lower bound of the range in our analysis. The results of correlation analysis are robust.
Conclusion
Our study reinforces the importance of robust methodologies in economic evaluations and highlights actionable pathways for improving HTA processes. By integrating complementary metrics, prioritising high-maturity data and establishing independent review mechanisms, HTA agencies can ensure more reliable, equitable and transparent decision-making. These advancements have the potential to benefit healthcare systems globally, improving resource allocation and patient outcomes.
Supplementary material
Footnotes
Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2025-101678 ).
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: Not applicable.
Ethics approval: Not applicable.
Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
References
- 1.Clement FM, Harris A, Li JJ, et al. Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada. JAMA. 2009;302:1437–43. doi: 10.1001/jama.2009.1409. [DOI] [PubMed] [Google Scholar]
- 2.Chen W, Zhang L, Hu M, et al. Use of health technology assessment in drug reimbursement decisions in China. BMJ. 2023;381:e068915. doi: 10.1136/bmj-2021-068915. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Jenei K, Raymakers AJN, Bayle A, et al. Health technology assessment for cancer medicines across the G7 countries and Oceania: an international, cross-sectional study. Lancet Oncol. 2023;24:624–35. doi: 10.1016/S1470-2045(23)00175-4. [DOI] [PubMed] [Google Scholar]
- 4.Whitehead SJ, Ali S. Health outcomes in economic evaluation: the QALY and utilities. Br Med Bull. 2010;96:5–21. doi: 10.1093/bmb/ldq033. [DOI] [PubMed] [Google Scholar]
- 5.Chauca Strand G, Bonander C, Jakobsson N, et al. Assessment of the clinical and cost-effectiveness evidence in the reimbursement decisions of new cancer drugs. ESMO Open. 2022;7:100569. doi: 10.1016/j.esmoop.2022.100569. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Tai T-A, Latimer NR, Benedict Á, et al. Prevalence of Immature Survival Data for Anti-Cancer Drugs Presented to the National Institute for Health and Care Excellence and Impact on Decision Making. Value Health. 2021;24:505–12. doi: 10.1016/j.jval.2020.10.016. [DOI] [PubMed] [Google Scholar]
- 7.Yong JHE, Beca J, Hoch JS. The evaluation and use of economic evidence to inform cancer drug reimbursement decisions in Canada. Pharmacoeconomics. 2013;31:229–36. doi: 10.1007/s40273-012-0022-5. [DOI] [PubMed] [Google Scholar]
- 8.Vokinger KN, Hwang TJ, Daniore P, et al. Analysis of Launch and Postapproval Cancer Drug Pricing, Clinical Benefit, and Policy Implications in the US and Europe. JAMA Oncol. 2021;7:e212026. doi: 10.1001/jamaoncol.2021.2026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Vokinger KN, Hwang TJ, Grischott T, et al. Prices and clinical benefit of cancer drugs in the USA and Europe: a cost-benefit analysis. Lancet Oncol. 2020;21:664–70. doi: 10.1016/S1470-2045(20)30139-X. [DOI] [PubMed] [Google Scholar]
- 10.Everest L, Blommaert S, Chu RW, et al. Parametric Survival Extrapolation of Early Survival Data in Economic Analyses: A Comparison of Projected Versus Observed Updated Survival. Value Health. 2022;25:622–9. doi: 10.1016/j.jval.2021.10.004. [DOI] [PubMed] [Google Scholar]
- 11.Cherny NI, Dafni U, Bogaerts J, et al. ESMO-Magnitude of Clinical Benefit Scale version 1.1. Ann Oncol. 2017;28:2340–66. doi: 10.1093/annonc/mdx310. [DOI] [PubMed] [Google Scholar]
- 12.Nieto-Gómez P, Castaño-Amores C, Rodríguez-Delgado A, et al. Analysis of oncological drugs authorised in Spain in the last decade: association between clinical benefit and reimbursement. Eur J Health Econ. 2024;25:257–67. doi: 10.1007/s10198-023-01584-9. [DOI] [PubMed] [Google Scholar]
- 13.Tibau A, Molto C, Ocana A, et al. Magnitude of Clinical Benefit of Cancer Drugs Approved by the US Food and Drug Administration. JNCI. 2018;110:486–92. doi: 10.1093/jnci/djx232. [DOI] [PubMed] [Google Scholar]
- 14.Tibau A, Molto C, Borrell M, et al. Magnitude of Clinical Benefit of Cancer Drugs Approved by the US Food and Drug Administration Based on Single-Arm Trials. JAMA Oncol. 2018;4:1610–1. doi: 10.1001/jamaoncol.2018.4300. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Woods B, Sideris E, Palmer S, et al. Partitioned survival analysis for decision modelling in health care: a critical review. NICE DSU Technical Support Document. 2017;19:72. [Google Scholar]
- 16.PBAC Public summary documents. https://www.pbs.gov.au/pbs/industry/listing/elements/pbac-meetings/psd n.d. Available.
- 17.CADTH Reimbursement review reports. https://www.cadth.ca/reimbursement-review-reports n.d. Available.
- 18.NICE Guidance, nice advice and quality standards. https://www.nice.org.uk/guidance/published?ngt=Technology+appraisal+guidance n.d. Available.
- 19.Cherny NI, Sullivan R, Dafni U, et al. A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS) Ann Oncol. 2015;26:1547–73. doi: 10.1093/annonc/mdv249. [DOI] [PubMed] [Google Scholar]
- 20.ESMO ESMO-mcbs for solid tumours. https://www.esmo.org/guidelines/esmo-mcbs/esmo-mcbs-for-solid-tumours/esmo-mcbs-scorecards n.d. Available.
- 21.Jorda A, Bergmann F, Ristl R, et al. Association between reactogenicity and immunogenicity after BNT162b2 booster vaccination: a secondary analysis of a prospective cohort study. Clin Microbiol Infect. 2023;29:1188–95. doi: 10.1016/j.cmi.2023.05.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Overholser BR, Sowinski KM. Biostatistics primer: part 2. Nutr Clin Pract. 2008;23:76–84. doi: 10.1177/011542650802300176. [DOI] [PubMed] [Google Scholar]
- 23.Zhou T, Xie F. Sponsorship bias in oncology cost effectiveness analysis. J Clin Epidemiol. 2023;156:22–9. doi: 10.1016/j.jclinepi.2023.02.011. [DOI] [PubMed] [Google Scholar]
- 24.Xie F, Zhou T. Industry sponsorship bias in cost effectiveness analysis: registry based analysis. BMJ. 2017;377:e069573. doi: 10.1136/bmj-2021-069573. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Carroll C, Kaltenthaler E, Hill-McManus D, et al. The Type and Impact of Evidence Review Group Exploratory Analyses in the NICE Single Technology Appraisal Process. Value Health. 2017;20:785–91. doi: 10.1016/j.jval.2016.08.729. [DOI] [PubMed] [Google Scholar]
- 26.Samjoo I, Whitney S, Grima A, et al. Economic Guidance Panel Changes to Minimize Post-Progression Treatment Effects by the Pan-Canadian Oncology Drug Review in Economic Evaluations Assessing Cancer Drugs. Value Health. 2018;21:S5. doi: 10.1016/j.jval.2018.04.020. [DOI] [Google Scholar]
- 27.Saluja R, Jiao T, Koshy L, et al. Comparing Manufacturer Submitted and Pan-Canadian Oncology Drug Review Reanalysed Incremental Cost-Effectiveness Ratios for Novel Oncology Drugs. Curr Oncol . 2021;28:606–18. doi: 10.3390/curroncol28010060. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Hill SR, Mitchell AS, Henry DA. Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme. JAMA. 2000;283:2116–21. doi: 10.1001/jama.283.16.2116. [DOI] [PubMed] [Google Scholar]
- 29.Latimer N. NICE DSU technical support document 14: survival analysis for economic evaluations alongside clinical trials-extrapolation with patient-level data. Report by the Decision Support Unit. 2011 [Google Scholar]
- 30.Rutherford M, Lambert PC, Sweeting MJ, et al. NICE DSU technical support document 21: Flexible methods for survival analysis. Decision Support Unit. 2021 [Google Scholar]
- 31.Connock M, Auguste P, Capelle A, et al. Potential impact on cost-effectiveness estimates of using immature survival data: a case study based on transcatheter edge-to-edge repair (TEER) used for patients with severe mitral regurgitation at high surgical risk. BMJ Open. 2023;13:e060423. doi: 10.1136/bmjopen-2021-060423. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Meyers DE, Jenei K, Chisamore TM, et al. Evaluation of the Clinical Benefit of Cancer Drugs Submitted for Reimbursement Recommendation Decisions in Canada. JAMA Intern Med. 2021;181:499–508. doi: 10.1001/jamainternmed.2020.8588. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Wang S, Gum D, Merlin T. Comparing the ICERs in Medicine Reimbursement Submissions to NICE and PBAC-Does the Presence of an Explicit Threshold Affect the ICER Proposed? Value Health. 2018;21:938–43. doi: 10.1016/j.jval.2018.01.017. [DOI] [PubMed] [Google Scholar]
- 34.NICE Developing nice guidelines:the manual. https://www.nice.org.uk/process/pmg20/chapter/reviewing-evidence n.d. Available.