This cohort study evaluates the quality of evidence in the drug submission process for provincial reimbursement of the pan-Canadian Oncology Drug Review.
Key Points
Question
What are the differences in the clinical evidence and benefit between cancer drugs that received a positive vs those that received a negative reimbursement recommendation from the pan-Canadian Oncology Drug Review (pCODR)?
Findings
In this cohort study of 104 reimbursement recommendation decisions, 78 received a positive recommendation from the pCODR. Of these, 72 were based on phase 3 randomized clinical trials and 39 had proven overall survival benefit with median gains of 3.7 months.
Meaning
Results of this study found that only half of the pCODR-recommended cancer drugs showed improved survival, and the survival gains were usually modest; these findings suggest that, despite the pCODR framework, cancer drugs without meaningful patient benefit continue to be reimbursed in the Canadian market.
Abstract
Importance
Cancer drugs approved by the US Food and Drug Administration have come under scrutiny for marginal clinical benefits; however, the clinical benefits of cancer drugs recommended for reimbursement in Canada have not been adequately studied.
Objective
To assess the differences in the clinical evidence and benefit of cancer drugs that received a positive vs a negative recommendation for provincial reimbursement in Canada.
Design, Setting, and Participants
This cohort study obtained publicly available regulatory documents from the pan-Canadian Oncology Drug Review (pCODR) and corresponding clinical trial documentation. All cancer drugs with a solid tumor indication that were submitted from the inception of the pCODR (July 2011) to February 2020 were evaluated. To be included, submissions had to have a final reimbursement recommendation; submissions that were incomplete, were withdrawn, or had a pending decision were excluded.
Exposures
A completed reimbursement recommendation decision from the pCODR.
Main Outcomes and Measures
Final reimbursement recommendation (positive vs negative); trial characteristics; and relevant clinical outcomes (ie, overall survival [OS] and progression-free survival [PFS]), including the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS) scores available at the time of pCODR assessment.
Results
Between 2011 and 2020, the pCODR issued 104 reimbursement recommendation decisions for cancer drugs with a solid tumor indication. Among these drug submissions, 78 (75.0%) received a positive recommendation, of which 72 (92.3%) were conditional. Drugs that received a positive recommendation compared with those with a negative recommendation were more likely to have phase 3 randomized clinical trial design (92.3% [72 of 78] vs 53.8% [14 of 26]; P < .001) and have substantial benefit according to the ESMO-MCBS scores (61.5% [48 of 78] vs 19.2% [5 of 26]; P < .001). The most common primary end points associated with the successful submissions were PFS (53.9%) and OS (32.1%). Overall, 39 of 78 submissions (50.0%) that received a positive recommendation had shown OS benefit, with median (interquartile range) OS gains of 3.7 (2.7-6.5) months.
Conclusions and Relevance
This cohort study found that, although the pCODR takes into account the magnitude of clinical benefit, only half of the cancer drugs that received a positive recommendation had evidence of improved OS and the survival gains were usually modest. These results suggest that, although the pCODR helps filter out some cancer drugs with low quality of evidence and low magnitude of benefit, cancer drugs without meaningful patient benefit continue to enter the Canadian market; these findings are important for making reimbursement policy decisions globally.
Introduction
The clinical benefit and cost of cancer drugs have come under scrutiny in recent years.1,2,3,4 Although the cost of cancer drugs continues to escalate,1,2 the clinical benefits of most modern cancer drugs seem to be either modest or uncertain.3,4 Cancer drugs are prescribed to patients to improve either the duration or the quality of life, but ideally both. However, analyses of cancer drugs approved by the US Food and Drug Administration (FDA)3,5 or the European Medicines Agency6 revealed concerns that most of these drugs lacked evidence that they improved the overall survival (OS) or quality of life (QOL) of patients with cancer or that these clinically meaningful end points were not evaluated in relevant clinical trials. Instead, the use of surrogate measures, such as progression-free survival (PFS) or response rates for drug approval decisions, has increased. These surrogate measures are known to poorly correlate with improvements in OS or QOL for most tumor types.7,8,9,10
For drugs that are associated with improved OS, the gains in OS are usually minimal. An analysis of cancer drugs approved in the US between 2002 and 2014 showed that less than half of these approvals met the threshold for meaningful benefit,3 which was set by the American Society of Clinical Oncology standard.11 Notwithstanding this lack of substantial clinical benefit, new cancer drugs routinely cost upward of US $10 000 per month.1 This disconnect between benefit and cost of cancer drugs has partly been ascribed to the FDA not taking the magnitude of benefit, or the cost-effectiveness of drugs, into account in its approval process.12
In Canada, the pan-Canadian Oncology Drug Review (pCODR), a subsidiary of the Canadian Agency for Drugs and Technologies in Health, provides a reimbursement recommendation for cancer drugs that takes clinical benefit, cost-effectiveness, patient-based values, and adoption feasibility into account.13 In this cohort study, we systematically reviewed all cancer drugs submitted to the pCODR for reimbursement decision and assessed the availability and quality of clinical evidence at the time of recommendation. In addition, we assessed the differences in those metrics between drugs that received a positive and those that received a negative recommendation for provincial reimbursement.
Methods
According to the Common Rule, institutional review board approval was not required for this study because all of the data used in this study were obtained from public records and did not include individual-level, patient-identifying information. This study of published reports did not involve patient-identifying data and was not submitted for institutional review board approval. Data were collected between March 1, 2020, and May 1, 2020.
Cancer Drug Reimbursement Decision Process in Canada
For a cancer drug to be marketed in Canada, it must first be approved for sale by Health Canada, appraised for a reimbursement recommendation by the pCODR, and then funded for reimbursement at the provincial level. Although the pCODR has no mandate to assess all drugs that receive approval from Health Canada, manufacturers that wish to have their drugs in the Canadian market generally submit to the pCODR at the same time as, or after, Health Canada approval. After the pCODR assesses the submitted data for clinical and economic benefit, the pCODR Expert Review Committee provides a nonbinding recommendation (yes, conditionally yes, or no) for provincial reimbursement. The pCODR makes these reports publicly available online.14
Conditional recommendations are usually conditional on cost-effectiveness and are considered to be positive recommendations (ie, recommended for funding at the provincial level). The conditional recommendations process in Canada is not similar to the accelerated approval pathway in the US, which signifies uncertainty in the evidence and mandates a confirmatory trial for conversion to a regular approval. No postmarketing confirmatory trials are mandated for the pCODR conditional recommendations. For example, in 2019, pembrolizumab was conditionally recommended for the adjuvant treatment of advanced melanoma based on limitations in cost-effectiveness. As of September 2020, the drug has been funded in 7 provinces with various eligibility criteria, whereas 2 provinces remain in negotiations. Despite being funded, pembrolizumab remains conditionally recommended for this indication. Drugs that have received approval from Health Canada but negative recommendations from the pCODR will not be funded but can still be used in Canada through compassionate access programs, private insurance plans, or out-of-pocket costs.
Cancer Drugs and Trials Reviewed
We studied all completed submissions for cancer drugs with a solid tumor indication in adult patients from the inception of the pCODR (July 2011) to February 2020. To be included, submissions had to have received a final reimbursement recommendation; submissions that were incomplete, were withdrawn, or had a pending decision were excluded. Submissions made for biosimilar drugs, pediatric indications, or hematologic malignant neoplasms were excluded. These exclusions were made to ensure the homogeneity of the data set and to allow for assessment of clinical benefits using the European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS), a tool developed for solid tumor indications.
Data Extraction and Data Set Formulation
For drugs that met the inclusion criteria, the following variables were extracted: final reimbursement recommendation (yes, conditionally yes, or no), tumor indication, treatment setting, line of therapy, end points of the primary clinical trial informing the submission, and relevant clinical outcomes such as the medians and hazard ratios (HRs) of OS or PFS. We dichotomized reimbursement decisions as positive (including conditional) or negative. Outcome data from clinical trials must have been available at the time of the pCODR assessment. All data were extracted from the pCODR documents and cross-referenced with ClinicalTrials.gov to ensure accuracy using the trial name and national clinical trial registration number. When multiple studies were cited as contributing to the body of evidence for a particular submission, we chose the most relevant trial (on the basis of phase, recency, and trial location) for review (eTable 1 in the Supplement). Two of us (D.E.M. and K.J.) extracted the pCODR data separately, and any discrepancies were resolved by consensus with input from an oncologist (B.G.).
Calculation of Magnitude of Clinical Benefit
We designated a drug as showing improved PFS or OS if a positive difference was found between the intervention and comparator groups and if the CI of the HR did not cross 1.0. If the experimental group did not reach the median PFS and OS at the time of assessment by the pCODR, then we estimated the median PFS and OS by multiplying the median PFS and OS of the control group by the point estimate of the HR, a practice that is in keeping with the calculation of the ESMO-MCBS.15 If both the experimental and control groups had not reached the median despite statistically significant HR and CI, the drug was designated as having a clinical benefit of unknown magnitude and was excluded from the calculation of median PFS and OS gains.
We calculated the median of the median gains in PFS and OS across all the submissions and separately for submissions that received positive and negative reimbursement recommendations, only if the gains were statistically significant. For a conservative estimate, we also included the submissions for which the difference in median PFS and OS was not statistically significant by inputting a value of 0 for median PFS and OS gains for these submissions.
The clinical benefit of cancer drugs within this sample was also categorized using the ESMO-MCBS, a validated framework16 widely used in the assessment of the clinical benefit of cancer drugs. These ESMO-MCBS scores were either extracted from the ESMO website, where available, or calculated using the publicly available score cards.17 The ESMO-MCBS assigns a score of 1 (low) to 5 (high) for drugs used in the advanced or metastatic setting and a score of C (low) to A (high) for drugs that are potentially curative or used in the adjuvant setting. Scores of 4 to 5 and A to B are designated as having substantial or major clinical benefit. These scores take into account the magnitude of not only efficacy but also adverse events and quality of life.
Statistical Analysis
We compared the characteristics of the trials and the clinical benefits of submissions with positive vs negative recommendations. The differences in medians of continuous variables (ie, PFS and OS) and the dependence of categorical variables were tested using permutation testing.18 We chose this approach because we generally did not have sufficient data to rely on asymptotic testing (ie, χ2 test for categorical data). P values were not adjusted for multiple hypothesis testing, and 2-sided P < .05 was considered to be statistically significant. Analyses were performed with R, version 3.6.2 (R Foundation for Statistical Computing). Permutation testing was completed with the infer package (Creative Commons).
Results
A total of 104 cancer drug submissions met the eligibility criteria and were included in this study (Figure 1). Among these submissions, 54 (51.9%) were for new indications of drugs previously submitted to the pCODR, 41 (39.4%) were for novel drugs with no previous submissions, and 9 (8.7%) were resubmissions that either were not completed (n = 2) or received negative decisions (n = 7).
Figure 1. Identification of Completed Pan-Canadian Oncology Drug Review (pCODR) Submissions of Drugs for Solid Tumor Indications Between July 2011 and February 2020.
Of the 9 resubmissions, 7 were previously submitted and received a negative reimbursement recommendation and 2 were withdrawn from consideration before the final reimbursement recommendation was delivered.
Lung cancer was the most common primary indication (25 [24.0%]), and small molecules were the most common drug type (48 [46.2%]). Most submissions were for the advanced or metastatic setting (97 [93.3%]), with 37 (35.6%) for treatment in the first line (eTables 2 and 3 in the Supplement).
Differences in Submissions With Positive vs Negative Recommendations
Of the 104 submissions, 79 (76.0%) had OS data and 82 (78.8%) had PFS data available at the time of review. Seventy-eight submissions (75.0%) received a positive recommendation for provincial reimbursement (6 outright recommendations and 72 conditional recommendations), and 26 submissions (25.0%) received a negative recommendation. The most common reason cited for a conditional recommendation was cost-effectiveness (70 of 72 [97.2%]). During the same period, the FDA granted 163 drug approvals for adult solid tumors, of which 121 were regular (74.2%) and 42 were accelerated (25.7%) approvals. None of the 26 submissions that received a negative recommendation from the pCODR had been rejected by the FDA.
No statistically significant difference was observed in the distributions of tumor type, drug class, treatment setting, or line of therapy between submissions that received positive and negative recommendations; however, there were significant differences with regard to certain trial characteristics (Table 1).
Table 1. Comparison of Characteristics of Cancer Drug Submissions Recommended vs Not Recommended for Reimbursement by Pan-Canadian Oncology Drug Review Between 2011 and 2020.
| Characteristic | Final reimbursement recommendation, No. (%) | P value | |
|---|---|---|---|
| Positive (n = 78) | Negative (n = 26) | ||
| General submission characteristics | |||
| Time to decision, median (IQR), d | 213.0 (193.5-242.0) | 229.5 (221.0-248.8) | .06 |
| Type of submission | |||
| New drug | 30 (38.5) | 11 (42.3) | .94 |
| New indication | 41 (52.6) | 13 (50.) | |
| Resubmission | 7 (9.0) | 2 (7.7) | |
| Tumor type | |||
| Lung | 18 (23.1) | 7 (26.9) | .16 |
| Gastrointestinal | 10 (12.8) | 8 (30.8) | |
| Breast | 10 (12.8) | 4 (15.4) | |
| Genitourinary | 12 (15.4) | 2 (7.7) | |
| Melanoma | 14 (18.0) | 0 | |
| Gynecological | 5 (6.4) | 1 (3.9) | |
| Other | 9 (11.5) | 4 (15.4) | |
| Drug class | |||
| Small molecule | 33 (42.3) | 15 (57.7) | .28 |
| Immune checkpoint inhibitor | 20 (25.6) | 3 (11.5) | |
| Monoclonal antibody | 10 (12.8) | 4 (15.4) | |
| Cytotoxic | 4 (5.1) | 3 (11.5) | |
| Hormonal | 6 (7.7) | 0 | |
| Other | 5 (6.4) | 1 (3.9) | |
| Treatment setting | |||
| Neoadjuvant | 0 | 1 (3.8) | .16 |
| Adjuvant | 4 (5.1) | 2 (7.7) | |
| Advanced/metastatic | 74 (94.9) | 23 (88.5) | |
| Line of therapya | |||
| First line | 33 (44.6) | 4 (17.4) | .06 |
| Second line and beyond | 37 (50.0) | 18 (78.3) | |
| Maintenance | 4 (5.4) | 1 (4.4) | |
| Trial characteristics | |||
| Trial design | |||
| Phase | |||
| 1 | 1 (1.3) | 4 (15.4) | <.001 |
| 2 | 5 (6.4) | 8 (30.8) | |
| 3 | 72 (92.3) | 14 (53.8) | |
| Randomization | |||
| No | 6 (7.7) | 11 (42.3) | <.001 |
| Yes | 72 (92.3) | 15 (57.7) | |
| Masking | |||
| No | 35 (48.6) | 4 (26.7) | .15 |
| Yes | 37 (51.4) | 11 (73.3) | |
| Primary end point | |||
| OS | 25 (32.1) | 6 (23.1) | <.001 |
| PFS | 42 (53.9) | 6 (23.1) | |
| RR | 5 (6.4) | 10 (38.5) | |
| Other | 6 (7.7) | 4 (15.4) | |
| Clinical characteristics | |||
| PFS data available | |||
| No | 10 (12.8) | 12 (46.2) | <.001 |
| Yes | 68 (87.2) | 14 (53.9) | |
| PFS improvement | |||
| No | 11 (16.2) | 2 (14.3) | .69 |
| Yes | 57 (83.8) | 12 (85.7) | |
| PFS gain, median (IQR)b | 4.7 (2.8-8.2) | 2.9 (0.8-4.0) | .06 |
| OS data available | |||
| No | 13 (16.7) | 12 (46.2) | .01 |
| Yes | 65 (83.3) | 14 (53.9) | |
| OS improvement | |||
| No | 26 (40.0) | 6 (42.9) | .99 |
| Yes | 39 (60.0) | 8 (57.1) | |
| OS gain, median (IQR)b | 3.7 (2.7-6.5) | 1.9 (1.4-4.5) | .06 |
| Substantial clinical benefit by the ESMO-MCBS | |||
| No | 30 (38.5) | 21 (80.8) | <.001 |
| Yes | 48 (61.5) | 5 (19.2) | |
Abbreviations: ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; IQR, interquartile range; OS, overall survival; PFS, progression-free survival; RR, response rate.
If advanced or metastatic treatment setting.
Excluding values of 0 (ie, no difference between intervention and comparator) and those where neither the intervention nor comparator had reached its median value.
Of the drugs that received a positive recommendation for reimbursement, 92.3% (72 of 78) had evidence from a phase 3 randomized clinical trial (RCT) compared with 53.8% (14 of 26) of the drugs that received a negative recommendation (P < .001). The most common primary end points associated with the successful submissions were PFS (53.8%) and OS (32.1%). The most common primary end point among the submissions that received a positive recommendation was PFS (42 [53.8%]) vs response rates among those that received a negative recommendation (10 [38.5%]). In addition, drugs that received a positive recommendation were statistically significantly more likely to have data available for PFS and OS at the time of pCODR assessment (Table 1 and eTable 4 in the Supplement). No significant differences were found in the distribution of submissions with evidence of OS or PFS benefit on the basis of reimbursement recommendations.
Forty-seven submissions had documented OS improvements, of which 39 (83.0%) received a positive recommendation. Twenty-six submissions without evidence of OS benefit and 13 with no OS data available at the time of pCODR assessment also received a positive recommendation. Thus, only 50.0% (39 of 78) of cancer drugs recommended for reimbursement in Canada had shown an improvement in OS.
Magnitude of Clinical Benefit for Submissions With Positive vs Negative Recommendations
The differences between submissions that received positive and those that received negative reimbursement recommendations did not reach statistical significance for median (interquartile range) gains in PFS (4.7 [2.8-8.2] vs 2.9 [0.8-4.0]; P = .06) or in OS (3.7 [2.7-6.5] vs 1.9 [1.4-4.5]; P = .06) (Table 1, Figure 2, and Figure 3). When submissions with OS or PFS gains of 0 months were included, the median gains in PFS for submissions with positive vs negative recommendations were 3.9 vs 2.7 (P = .29) and the median gains in OS were 1.9 vs 1.4 (P = .92). These calculations included only those submissions with OS and PFS data available at the time of the recommendation. Submissions that received a positive reimbursement recommendation were more likely to have demonstrated substantial clinical benefit according to the ESMO-MCBS scores compared with those that received a negative recommendation (48 [61.5%] vs 5 [19.2%]; P < .001) (Table 1).
Figure 2. Gains in Progression-Free Survival (PFS) of 82 Pan-Canadian Oncology Drug Review (pCODR) Submissions With Data Available at the Time of Assessment.
The dashed blue line represents the overall median gain in PFS, excluding values of 0 months (ie, no statistically significant difference between the intervention and comparator). The orange vertical lines represent the first and third quartile of PFS. Median (interquartile range) PFS gains were 4.0 (2.4-7.1) months.
Figure 3. Gains in Overall Survival (OS) of 79 Pan-Canadian Oncology Drug Review (pCODR) Submissions With Data Available at the Time of Assessment.
The dashed blue line represents the overall median gain in OS, excluding values of 0 months (ie, no statistically significant difference between the intervention and comparator). The orange vertical lines represent the first and third quartile of OS. Median (interquartile range) OS gains were 3.6 (2.2-6.2) months.
Provincial Fate of Cancer Drugs Reviewed by the pCODR
Although the pCODR recommendations are nonbinding, we found 100% concordance between pCODR recommendations and provincial funding decisions for submissions that had received “recommend for funding” and “do not recommend for funding” decisions (based on available data). However, of the 72 conditional recommendations, 6 (8.3%) were generally not funded by the provinces that had completed their deliberations.
Discussion
We evaluated the characteristics of trials and the magnitude of clinical benefit of cancer drugs with a solid tumor indication submitted to the pCODR for Canadian provincial reimbursement. We found that three-quarters of all submissions received a positive recommendation, of which half demonstrated improved survival with median gains in OS of 3.7 months. Significant differences in trial characteristics were observed, such as randomization, phase, and primary end points, between submissions that received a positive recommendation and submissions that received a negative recommendation. These findings have multiple regulatory and reimbursement policy lessons globally.
These current findings must be placed into context by examining the results of similar studies that evaluated the magnitude of clinical benefit and the quality of evidence for cancer drugs approved by the FDA at different time points (Table 219,20) to facilitate a qualitative comparison. We found that 61.5% of cancer drugs (48 of 78) that received a positive recommendation from the pCODR had demonstrated substantial clinical benefit according to the ESMO-MCBS score, in contrast to the data previously reported for the FDA: 43.8% of drug approvals (46 of 105) based on RCTs21 and 33.8% of drug approvals (45 of 133) based on single-arm trials22 demonstrated substantial clinical benefit using the same scale (ESMO-MCBS). Similarly, although only 6.4% of cancer drugs (5 of 78) that received a positive recommendation by the pCODR were based on response rates, the corresponding percentage was 34.9% of drugs (22 of 63) approved by the FDA.20 For OS, 50.0% of drugs (39 of 78) recommended by the pCODR and 30.2% of drugs (19 of 63) approved by the FDA showed improved OS.20 With regard to the quality of evidence, 92.3% (72 of 78) of positive recommendations by the pCODR but only 75.8% (116 of 153) of FDA approvals were based on RCTs.
Table 2. Trial Characteristics and Clinical Benefits of Cancer Drugs Recommended for Reimbursement by Pan-Canadian Oncology Drug Review vs Cancer Drugs Approved by the US Food and Drug Administrationa.
| Characteristic | Current findings for drugs recommended by pCODR | Previous findings for drugs approved by the FDA | FDA data source |
|---|---|---|---|
| Percent of submissions that received a positive recommendation, No./total No. (%) | 78/104 (75.0) | Data unavailable | NA |
| Period of assessment | 2011-2020 | 2002-2014 | Fojo et al,3 2014 |
| 2009-2018 | Gyawali et al,19 2019 | ||
| 2009-2013 | Mailankody and Prasad,20 2015 | ||
| 2006-2016 | Tibau et al,21 2018; Tibau et al,22 2018 | ||
| Randomized trial design, No./total No. (%) | 72/78 (92.3) | 116/153 (75.8) | Gyawali et al,19 2019 |
| Improved OS | 39/78 (50.0) | 19/63 (30.2) | Mailankody and Prasad,20 2015 |
| Approved with RR | 5/78 (6.4) | 22/63 (34.9) | Mailankody and Prasad,20 2015 |
| Median gains in OS, mo | 3.7 | 2.1 | Fojo et al,3 2014 |
| 2.9 | Gyawali et al,19 2019 | ||
| Median gains in PFS, mo | 4.7 | 2.5 | Fojo et al,3 2014 |
| Substantial clinical benefit by the ESMO-MCBS, No./total No. (%) | 48/78 (61.5) | 46/105 (43.8), RCT | Tibau et al,21 2018 |
| 45/133 (33.8), single-arm trial | Tibau et al,22 2018 |
Abbreviations: ESMO-MCBS, European Society for Medical Oncology-Magnitude of Clinical Benefit Scale; FDA, US Food and Drug Administration; NA, not applicable; OS, overall survival; pCODR, pan-Canadian Oncology Drug Review; PFS, progression-free survival; RCT, randomized clinical trial; RR, response rate.
All 26 submissions that received a negative recommendation from the pCODR have been approved by the FDA, whereas no drugs that received a positive recommendation from the pCODR were not approved by the FDA. The 26 submissions that received a negative recommendation from the pCODR but were approved by the FDA included crizotinib (lung), pazopanib hydrochloride (sarcoma), regorafenib (colorectal), lapatinib ditosylate (breast), cetuximab (colorectal), aflibercept (colorectal), pertuzumab (neoadjuvant breast), sorafenib tosylate (thyroid), regorafenib (colorectal resubmission), ceritinib (lung), olaparib (ovary), trabectedin (sarcoma), alectinib hydrochloride (lung), dabrafenib mesylate plus trametinib dimethyl sulfoxide (lung), panitumumab (colorectal), TAS-102 (colorectal), pertuzumab plus trastuzumab (breast adjuvant), nivolumab (liver), lenvatinib mesylate (renal), brigatinib (lung), TAS-102 (colorectal resubmission), pembrolizumab (urothelial), atezolizumab (small cell lung), larotrectinib sulfate (histology agnostic), neratinib maleate (breast adjuvant), and lorlatinib (lung). A complete list of all drugs and indications is provided in eTable 3 in the Supplement. A submission that previously received a negative recommendation could have been resubmitted and later received a positive recommendation.
One explanation for these differences in results is that the pCODR uses a multicriteria deliberative framework that takes into account not only the presence or absence of clinical benefit but also the magnitude of benefit, quality of evidence, economic evaluation, and feedback from patient and clinician stakeholders. This approach is in contrast to that of the FDA, which considers neither the magnitude of benefit nor the cost. We found that all 26 cancer drugs that were not recommended for reimbursement by the pCODR had previously been approved by the FDA, which is similar to the findings of a previous study.23 Furthermore, although the FDA granted 163 approvals during the same period, the pCODR received only 104 submissions, of which only 78 received a positive reimbursement recommendation. These data support the notion that the pCODR framework might filter out drugs with minimal clinical utility. Integrating the ESMO-MCBS score as a preliminary tool may help the pCODR to further filter out minimally effective cancer drugs at the outset, an approach that is being used in certain countries in Europe and by the World Health Organization in formulating the essential medicine list in cancer.24
Results from the present study identified areas of improvement for the Canadian regulatory system. First, the quality of evidence for most of the submissions that received a positive recommendation was low. Less than one-third of submissions that received a positive recommendation from the pCODR had OS as the primary end point. In the US, the proportion of cancer drugs that are approved on the basis of surrogates has increased over the past decade, peaking at 93% in 2017.19 Although PFS was the most frequently used primary end point in the current study, data suggest that PFS is not a valid surrogate for clinical outcomes such as OS8,10,25 or QOL.9,26 Previous work has shown that more than half of submissions to the pCODR between 2015 and 2018 did not have data on QOL.27 None of the pCODR submissions that we evaluated had QOL as a primary outcome measure. Thus, moving forward, clinical trials should prioritize meaningful clinical end points, such as OS and QOL, to ensure that regulatory decisions are guided by high-quality evidence.
Second, cancer drugs that were recommended for reimbursement in Canadian provinces were not shown to have substantially improved clinical outcomes for patients. Across all submissions with a positive recommendation, the median OS gains were only 3.7 months and the median PFS gains were 4.7 months, which are not substantially different from the survival gains in submissions that received a negative recommendation. Furthermore, nearly 40% of cancer drugs that received a positive recommendation did not reach the threshold for substantial clinical benefit per the ESMO-MCBS. This finding demonstrates room for improvement in the Canadian system.
Previous studies have also examined the pCODR decision process from different angles and have highlighted other areas of improvement. A study by Niraula and Nugent23 found that in a cohort of 91 cancer drugs approved by the FDA, 18 (20%) received a negative recommendation by the pCODR, with higher incremental cost-effectiveness ratio being a factor associated with a negative recommendation. Other studies have found that the economic models submitted to the pCODR by drug manufacturers were often flawed as identified by the economic reviewers28 of the pCODR and cost-effectiveness appeared to be associated with a full vs conditional reimbursement decision rather than a positive vs negative decision.29 In addition, although patient values were considered in stakeholder feedback, both patients30 and clinicians31 involved in the process were shown to have relevant financial conflicts of interest.
Limitations
This study has some limitations. The study is unique in that it comprehensively summarized and quantified the clinical evidence as well as the magnitude of clinical benefit for cancer drugs that were recommended or not recommended for reimbursement in Canada. It did not, however, assess the role of cost or cost-effectiveness in this decision-making process. The pCODR bases its recommendation decision not only on quality of evidence and magnitude of benefit but also on economic evaluation, stakeholder views, and adoption feasibility; as such, this analysis may not be fully representative of all the aspects that the pCODR includes in its deliberation. We assessed only the quality of evidence on trial phase and randomization evaluated by the pCODR, but we did not assess other characteristics of the trials. A complete assessment is important given recent data that suggested that trials used to inform regulatory decisions for cancer drugs by both the European Medicines Agency32 and the FDA33 were at high risk of bias and had critical design flaws. In addition, we limited our analyses to data that were available to the pCODR at the time of deliberation and did not consider evidence that may have been subsequently generated.
Conclusions
In this cohort study, we found that 75% of submissions to the pCODR for provincial reimbursement of cancer drugs in Canada received a positive reimbursement recommendation; these submissions were mostly based on phase 3 RCTs, with half of them showing improved OS. Cancer drugs recommended for reimbursement in Canada also have issues, such as the use of surrogate end points and a modest margin of clinical benefit. Even in a regulatory system like the pCODR with a central evidence appraisal process, including assessment of evidence quality, magnitude of clinical benefit, and cost-effectiveness, cancer drugs without meaningful patient benefit are entering the market. These findings have implications for regulatory and reimbursement policy decisions in Canada and globally.
eTable 1. Examples of Trial Selection in the Present Study
eTable 2. Characteristics of the pCODR Submissions for Solid-Tumor Indications Between 2011-2020
eTable 3. List of All 104 Oncology Drugs With Final Recommendations From pCODR Between 2011-2020
eTable 4. Select Clinical Outcomes of All 104 Oncology Drugs With Final Recommendations From pCODR Between 2011-2020
References
- 1.Saluja R, Arciero VS, Cheng S, et al. Examining trends in cost and clinical benefit of novel anticancer drugs over time. J Oncol Pract. 2018;14(5):e280-e294. doi: 10.1200/JOP.17.00058 [DOI] [PubMed] [Google Scholar]
- 2.IQVIA Institute for Human Data Science . Global oncology trends 2019: therapeutics, clinical development and health system implications. May 2019. Accessed July 25, 2020. https://www.iqvia.com/insights/the-iqvia-institute/reports/global-oncology-trends-2019
- 3.Fojo T, Mailankody S, Lo A. Unintended consequences of expensive cancer therapeutics—the pursuit of marginal indications and a me-too mentality that stifles innovation and creativity: the John Conley Lecture. JAMA Otolaryngol Head Neck Surg. 2014;140(12):1225-1236. doi: 10.1001/jamaoto.2014.1570 [DOI] [PubMed] [Google Scholar]
- 4.Kumar H, Fojo T, Mailankody S. An appraisal of clinically meaningful outcomes guidelines for oncology clinical trials. JAMA Oncol. 2016;2(9):1238-1240. doi: 10.1001/jamaoncol.2016.0931 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kim C, Prasad V. Cancer drugs approved on the basis of a surrogate end point and subsequent overall survival: an analysis of 5 years of US Food and Drug Administration approvals. JAMA Intern Med. 2015;175(12):1992-1994. doi: 10.1001/jamainternmed.2015.5868 [DOI] [PubMed] [Google Scholar]
- 6.Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggarwal A. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009-13. BMJ. 2017;359:j4530. doi: 10.1136/bmj.j4530 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Chen EY, Haslam A, Prasad V. FDA acceptance of surrogate end points for cancer drug approval: 1992-2019. JAMA Intern Med. 2020;180(6):912-914. doi: 10.1001/jamainternmed.2020.1097 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Haslam A, Hey SP, Gill J, Prasad V. A systematic review of trial-level meta-analyses measuring the strength of association between surrogate end-points and overall survival in oncology. Eur J Cancer. 2019;106:196-211. doi: 10.1016/j.ejca.2018.11.012 [DOI] [PubMed] [Google Scholar]
- 9.Hwang TJ, Gyawali B. Association between progression-free survival and patients’ quality of life in cancer clinical trials. Int J Cancer. 2019;144(7):1746-1751. doi: 10.1002/ijc.31957 [DOI] [PubMed] [Google Scholar]
- 10.Booth CM, Eisenhauer EA. Progression-free survival: meaningful or simply measurable? J Clin Oncol. 2012;30(10):1030-1033. doi: 10.1200/JCO.2011.38.7571 [DOI] [PubMed] [Google Scholar]
- 11.Ellis LM, Bernstein DS, Voest EE, et al. American Society of Clinical Oncology perspective: raising the bar for clinical trials by defining clinically meaningful outcomes. J Clin Oncol. 2014;32(12):1277-1280. doi: 10.1200/JCO.2013.53.8009 [DOI] [PubMed] [Google Scholar]
- 12.Darrow JJ, Avorn J, Kesselheim AS. FDA approval and regulation of pharmaceuticals, 1983-2018. JAMA. 2020;323(2):164-176. doi: 10.1001/jama.2019.20288 [DOI] [PubMed] [Google Scholar]
- 13.Canadian Agency for Drugs and Technologies in Health . Procedures for the CADTH Pan-Canadian Oncology Drug Review. April 2020. Accessed June 10, 2020. https://www.cadth.ca/sites/default/files/pcodr/pCODR%27s%20Drug%20Review%20Process/pcodr-procedures.pdf
- 14.Canadian Agency for Drugs and Technologies in Health . Reimbursement review reports. Accessed July 25, 2020. https://www.cadth.ca/reimbursement-review-reports
- 15.European Society for Medical Oncology . ESMO-Magnitude of Clinical Benefit Scale: evaluation forms version 1.1. Accessed May 5, 2020. https://www.esmo.org/guidelines/esmo-mcbs/scale-evaluation-forms-v1.0-v1.1/scale-evaluation-forms-v1.1
- 16.Cherny NI, Dafni U, Bogaerts J, et al. ESMO-Magnitude of Clinical Benefit Scale version 1.1. Ann Oncol. 2017;28(10):2340-2366. doi: 10.1093/annonc/mdx310 [DOI] [PubMed] [Google Scholar]
- 17.European Society for Medical Oncology . ESMO-MCBS scorecards. Accessed September 23, 2020. https://www.esmo.org/guidelines/esmo-mcbs/esmo-magnitude-of-clinical-benefit-scale
- 18.Pesarin F, Salmaso L.. Permutation Tests for Complex Data: Theory, Applications and Software. John Wiley & Sons Ltd; 2010. doi: 10.1002/9780470689516 [DOI] [Google Scholar]
- 19.Gyawali B, Sharma S, Booth CM. Is the number of cancer drug approvals a surrogate for regulatory success? J Cancer Policy. 2019;22:100202. doi: 10.1016/j.jcpo.2019.100202 [DOI] [Google Scholar]
- 20.Mailankody S, Prasad V. Five years of cancer drug approvals: innovation, efficacy, and costs. JAMA Oncol. 2015;1(4):539-540. doi: 10.1001/jamaoncol.2015.0373 [DOI] [PubMed] [Google Scholar]
- 21.Tibau A, Molto C, Ocaña A, et al. Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration. J Natl Cancer Inst. 2018;110(5):486-492. doi: 10.1093/jnci/djx232 [DOI] [PubMed] [Google Scholar]
- 22.Tibau A, Molto C, Borrell M, et al. Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration based on single-arm trials. JAMA Oncol. 2018;4(11):1610-1611. doi: 10.1001/jamaoncol.2018.4300 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Niraula S, Nugent Z. New cancer drug approvals from the perspective of a universal healthcare system: analyses of the Pan-Canadian Oncology Drug Review recommendations. J Natl Compr Canc Netw. 2018;16(12):1460-1466. doi: 10.6004/jnccn.2018.7084 [DOI] [PubMed] [Google Scholar]
- 24.European Society for Medical Oncology . ESMO-MCBS and the World Health Organisation (WHO) model list of essential medicines. Accessed September 10, 2020. https://www.esmo.org/guidelines/esmo-mcbs/esmo-mcbs-and-the-who
- 25.Kemp R, Prasad V. Surrogate endpoints in oncology: when are they acceptable for regulatory and clinical decisions, and are they currently overused? BMC Med. 2017;15(1):134-137. doi: 10.1186/s12916-017-0902-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Kovic B, Jin X, Kennedy SA, et al. Evaluating progression-free survival as a surrogate outcome for health-related quality of life in oncology: a systematic review and quantitative analysis. JAMA Intern Med. 2018;178(12):1586-1596. doi: 10.1001/jamainternmed.2018.4710 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Raymakers AJN, Regier DA, Peacock SJ. Health-related quality of life in oncology drug reimbursement submissions in Canada: a review of submissions to the Pan-Canadian Oncology Drug Review. Cancer. 2020;126(1):148-155. doi: 10.1002/cncr.32455 [DOI] [PubMed] [Google Scholar]
- 28.Masucci L, Beca J, Sabharwal M, Hoch JS. Methodological issues in economic evaluations submitted to the Pan-Canadian Oncology Drug Review (pCODR). Pharmacoecon Open. 2017;1(4):255-263. doi: 10.1007/s41669-017-0018-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Skedgel C, Wranik D, Hu M. The relative importance of clinical, economic, patient values and feasibility criteria in cancer drug reimbursement in Canada: a revealed preferences analysis of recommendations of the Pan-Canadian Oncology Drug Review 2011-2017. Pharmacoeconomics. 2018;36(4):467-475. doi: 10.1007/s40273-018-0610-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Lexchin J. Association between commercial funding of Canadian patient groups and their views about funding of medicines: an observational study. PLoS ONE. 2019;14(2):e0212399. doi: 10.1371/journal.pone.0212399 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Lexchin J. Financial conflicts of interest of clinicians making submissions to the pan-Canadian Oncology Drug Review: a descriptive study. BMJ Open. 2019;9(7):e030750. doi: 10.1136/bmjopen-2019-030750 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Naci H, Davis C, Savović J, et al. Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014-16: cross sectional analysis. BMJ. 2019;366:l5221. doi: 10.1136/bmj.l5221 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Hilal T, Gonzalez-Velez M, Prasad V. Limitations in clinical trials leading to anticancer drug approvals by the US Food and Drug Administration. JAMA Intern Med. 2020;180(8):1108-1115. doi: 10.1001/jamainternmed.2020.2250 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eTable 1. Examples of Trial Selection in the Present Study
eTable 2. Characteristics of the pCODR Submissions for Solid-Tumor Indications Between 2011-2020
eTable 3. List of All 104 Oncology Drugs With Final Recommendations From pCODR Between 2011-2020
eTable 4. Select Clinical Outcomes of All 104 Oncology Drugs With Final Recommendations From pCODR Between 2011-2020