Table 3.

Refined MCDA rating tool.

Criteria		Rating Scale			Weight
		1	2	3
Group A—Criteria to Assess the Importance of the RWE Question
Criterion 1: Drug’s perceived clinical benefit (quantitative assessment) The objective of this criterion is to evaluate the perceived clinical benefit of the therapy compared to existing options. The “perceived” clinical benefit is based on the currently available objective evidence, with preference given to primary clinical trial data or indirect comparisons, a utilized in the setting of single-arm studies or to assess effectiveness in comparison to contemporary Canadian standard-of-care treatments. Assessment of the perceived clinical benefit is based on ratings derived from the European Society of Medical Oncology (ESMO) Magnitude of Clinical Benefit Scale (MCBS) version 1.1 [17]. Users are asked to provide a quantitative assessment using the ESMO-MCBS-endorsed rating. The ESMO-MCBS-endorsed rating will be provided in the corresponding vignette (if available); if not, CanREValue will provide a suggested score based on the ESMO-MCBS evaluation forms. Recognizing inter-rater variability, users are asked to double check the suggested score using the information provided in the vignette and corresponding evaluations found below. ESMO-MCBS (v1.1) scores: Curative setting: New approaches to adjuvant therapies or potential curative therapies are graded on a scale from A to C, with Grade A indicating a substantial magnitude of clinical benefit. Non-curative setting: Therapies that are not likely to be curative are graded on a scale from 1 to 5, with a Grade 4 and 5 indicating a substantial magnitude of benefit. Caveats: Please note that ESMO specifies the use of the lower limit of the 95% confidence interval of the hazard ratio (HR) (instead of the point estimate of HR) when determining the ESMO-MCBS score. Currently, ESMO-MCBS has not been validated for the evaluation of pediatric and hematological malignancies.		Rated using the European Society of Medical Oncology (ESMO) Magnitude of Clinical Benefit Scale (MCBS) v1.0. [17]			17.65
		Minimal to low clinically meaningful incremental benefit, as evidenced by ESMO-MCBS v1.1 score of: (a) Curative setting: Grade C for new approaches to adjuvant therapy or potentially new curative therapies (please refer to Evaluation Form 1 below for details on rating); or (b) Non-curative setting: Grade 1 or 2 for therapies that are not likely to be curative (please refer to relevant Evaluation Form 2a, 2b, 2/c, or 3 below, depending on primary endpoint available for details on rating).	Moderate clinically meaningful incremental benefit, as evidenced by ESMO-MCBS v1.1 score of: (a) Curative setting: Grade B for new approaches to adjuvant therapy or potentially new curative therapies (please refer to Evaluation Form 1 below for details on rating); or (b) Non-curative setting: Grade 3 for therapies that are not likely to be curative (please refer to relevant Evaluation Form 2a, 2b, 2c, or 3 below, depending on primary endpoint available for details on rating).	Substantial clinically meaningful incremental benefit, as evidenced by ESMO-MCBS v1.1 score of: (a) Curative setting: Grade A for new approaches to adjuvant therapy or potentially new curative therapies (please refer to Evaluation Form 1 below for details on rating); or (b) Non-curative setting: Grade 4 or 5 for therapies that are not likely to b e curative (please refer to relevant Evaluation Form 2a, 2b, 2c, or 3 below, depending on primary endpoint available for details on rating).
Criterion 2: Magnitude of perceived uncertainty (qualitative assessment) The objective of this criterion is to assess the degree of uncertainty regarding the endpoint in question. Users are asked to provide a qualitative assessment of whether the primary clinical trial data available is characterized by features of minimal, moderate, or substantial uncertainty. There may be features related to the design of the study that may make the results of the study or the expected clinical benefit of the drug to be more uncertain. Common attributes to consider when ascertaining uncertainty include, but are not limited to:		Minimal uncertainty	Moderate uncertainty	Substantial uncertainty	10.6
☐ Immature trial data; ☐ Single arm studies; ☐ Use of surrogate endpoints;	Phase of the trial: Consider whether the trial was early or late phase ☐ Phase I ☐ Phase II ☐ Phase III
☐ Trials lacking a relevant Canadian standard-of-care comparator; ☐ Existing controversary in the literature or clinical practice;	☐ RWE Applicability: Consider whether there are concerns with generalizability of the trial to the general unselected population in the real world ☐ Other uncertainties Consider other possible sources of uncertainty
Criterion 3: Relevance of uncertainty (qualitative assessment) The objective of this criterion is to assess the relevance of resolving the uncertainty to decision-makers. Specifically, users are asked to provide a qualitative assessment on whether the data generated from the proposed RWE study could lead to future policy change (i.e., price re-negotiation, change in funding criteria, or disinvestment), assuming that a health technology life-cycle reassessment platform existed.		Low relevance: As assessed by expert opinions, there is an expected low likelihood for the findings of the proposed RWE study to facilitate a change in policy (i.e., price re-negotiation, change in funding criteria, disinvestment).	Moderate relevance: As assessed by expert opinions, there is uncertainty in the likelihood for the findings of the proposed RWE study to facilitate a change in policy (i.e., price re-negotiation, change in funding criteria, disinvestment).	Substantial relevance: As assessed by expert opinions, there is an expected high likelihood for the findings of the proposed RWE study to facilitate a change in policy (i.e., price re-negotiation, change in funding criteria, disinvestment).	18.8
Group B—Criteria to Assess the Feasibility of the RWE Project
Criterion 4: Identification and assembly of cases and comparator control cohort (qualitative assessment) The objective of this criterion is to assess the likelihood that cases and a relevant historical Canadian comparator cohort can be identified and assembled in at least one Canadian province. A ”relevant” comparator is a group of patients that has been treated according to current Canadian standard of care regimen. To rate this criterion, users are asked to provide a qualitative assessment by assessing whether cases and the comparator cohort can be identified based on the availability and completeness of data pertaining to (a) type of cancer (i.e., primary cancer), (b) type of treatment received, and (c) clinical characteristics (i.e., biomarkers, stage, etc.). Note: Sample size of the cases and comparator control cohort should not be considered for the rating of this criterion, as it will be considered in the rating of Criterion 5 (Sample size for cases and comparator control cohort). Further, the availability of covariates and outcomes should not be considered for the rating of this criterion, as it will be considered in the rating of Criterion 6 (Availability of Data for Covariates and Outcomes).		Substantial concern	Moderate concern	Low concern	11.8
Criterion 5: Sample sizes for cases and comparator control cohort (qualitative assessment) The objective of this criterion is to assess the likelihood that there will be a sufficient sample size of patients receiving the treatment in question (cases) and that a relevant historical Canadian comparator cohort (control) can be identified within a reasonable time frame (i.e., within time to be relevant to the funding decision). Users are asked to provide a qualitative assessment by referring to the formal sample size calculation provided in the respective vignette to perform rating for this criterion.		Substantial concern: Unlikely to establish a sufficient sample size within a reasonable timeframe to effect meaningful policy change.	Moderate concern: Likely to establish a sufficient sample size (as noted in rating scale 3) but additional concern noted (e.g., concern for loss of sample size for cases due to emerging alternative novel therapies with upcoming availability through access programs or public funding).	Low concern: Very likely to establish a sufficient sample size within a reasonable timeframe to effect meaningful policy change.	14.1
Criterion 6: Availability of data for covariates and outcomes (qualitative assessment) The objective of this criterion is to assess the availability and completeness of data in at least one Canadian province for key covariates and outcome of interest. To rate this criterion, users are asked to provide a qualitative assessment by assessing whether key criteria and outcomes can be identified within currently available administrative databases pertaining to data for: (a) Covariates: Relevant patient and/or disease characteristics required to be adjusted and/or accounted for given the non-randomized nature of RWE study design (e.g., consider baseline characteristics listed in Table 1 of the pivotal clinical trial). (b) Outcome of interest Please note: If the uncertainty is related to cost, users are also to consider data for relevant costing inclusive of total healthcare costs accrued during treatment (e.g., systemic treatment, planned and unplanned healthcare resource utilization). Note: Sample size of the exposed cohort should not be considered for the rating of this criterion (as sample size is considered in the rating of Criterion 5 “Sample size for cases and comparator control cohort”).		Substantial concern	Moderate concern	Low concern	17.65
Criterion 7: Availability of Expertise and Methodology (qualitative assessment) The objective of this criterion is to evaluate the availability of required expertise (e.g., clinical experts, data analysts, and methodologists) and methodology to conduct the study. Users are asked to provide a qualitative assessment.		Substantial concern: Expected challenges to find the necessary expertise and need to develop new methods to conduct the study, with above limitations in data taken into consideration (if applicable).	Moderate concern: Expected challenges to find the necessary expertise or need to develop new methods to conduct the study, with above limitations in data taken into consideration (if applicable).	Low concern: No expected issues with the availability of the necessary expertise and no new methods required to conduct the study.	9.4

^a Please refer to pCODR clinical guidance report for data on previously conducted, relevant indirect comparisons.