Abstract
Background:
The CADTH Common Drug Review was established in 2002 to prepare national health technology assessment reports to guide listing decisions for 18 participating drug plans. The aim of this study was to compare the nonmandatory recommendations from the Common Drug Review in Canada with the listing decisions of provincial payers to determine alignment.
Methods:
We identified the recommendations issued by the Common Drug Review from Jan. 1, 2009, to Jan. 1, 2015, and compared these with the listing decisions of 3 provincial public payers (Alberta, British Columbia and Ontario) that participate in the Common Drug Review and the recommendations from Quebec.
Results:
We identified 174 medicine-indication pairs in CADTH Common Drug Review reports issued from Jan. 1, 2009, to Jan. 1, 2015; 110 of these met the inclusion criterion. Among the 110 medicine-indication pairs, listing decisions were available for 95 in Alberta, 102 in Quebec, 104 in Ontario and 106 in BC. There was moderate to substantial agreement between provincial listing decisions and Common Drug Review recommendations: 74.5% (κ = 0.47, 95% confidence interval [CI] 0.31-0.64) for Quebec, 78.8% (κ = 0.56, 95% CI 0.41-0.72) for Ontario, 78.9% (κ = 0.58, 95% CI 0.42-0.74) for Alberta and 81.1% (κ = 0.62, 95% CI 0.47-0.77) for BC.
Interpretation:
Our study showed moderate to substantial agreement between Common Drug Review recommendations and provincial listing decisions. Future studies can build on this research by evaluating the concordance between Common Drug Review recommendations and listing decisions of all participating federal, provincial and territorial drug plans.
As with payers in many other countries faced with the reality that the costs of medicines are rising, Canada's public drug plans use health technology assessment to inform reimbursement decision-making.1 Health technology assessment is a multidisciplinary field of research that generally considers the therapeutic benefits, cost-effectiveness, and social, ethical and organizational impact of a new health technology such as a pharmaceutical, medical device, or diagnostic or surgical intervention to inform health policy and reimbursement decisions. Although health technology assessment is a young field of research, regional and national bodies have been active in this area in Canada since the 1980s.2,3 Today, CADTH administers the Common Drug Review program, which conducts a centralized national health technology assessment process recognized by all public drug plans except Quebec's. Before the inception of the Common Drug Review in 2002, several provincial drug plans undertook their own health technology assessment processes to determine coverage for new drug products. CADTH established the Common Drug Review to standardize the process in Canada by which health technology assessments are conducted to review new drugs and make reimbursement recommendations, harmonize decision-making across different public drug plans, reduce duplication of work and ultimately decrease the time delay before patients can access new medicines.4
As have many other sources of health technology assessments, the Common Drug Review has been subject to criticism. In 2011, a study reported the degree of agreement between the Common Drug Review's reimbursement recommendations and the listing decisions of 3 provinces to be "no better than random chance."5 However, these findings contradict those of another study.6 In the present study, we conducted a comparison using more recent data and thus our findings are more relevant to the current environment of health technology assessment and help to determine whether the Common Drug Review is creating more standardized coverage for medicines across Canada.7 Morgan and colleagues8 argued that having multiple provincial decision-makers reduces the impact of the Common Drug Review. Similarly, Hollis and Law9 predicted that without a national Canadian formulary the Common Drug Review would only slightly improve the standardization of the coverage of medicines across provinces.
The aim of this study was to examine the impact of the Common Drug Review by comparing its nonmandatory reimbursement recommendations with the final listing decisions of provincial drug plans.
Methods
Study design
All provincial drug plans review new medicines approved by Health Canada to determine whether they will be eligible for reimbursement, and almost all public drug plans in Canada participate in the Common Drug Review. The Common Drug Review generates a clinical and economic report for each new drug and provides a reimbursement recommendation to inform decision-making at the provincial level. We chose Alberta, British Columbia and Ontario for our analysis because they are the 3 most populous provinces with public drug plans that participate in the Common Drug Review. We also included Quebec because it is the only province that does not participate in the national Common Drug Review; the Institut national d'excellence en santé et en services sociaux (INESSS) conducts health technology assessments independently. We used information from the websites of CADTH, INESSS and the governments of Alberta, BC and Ontario to identify reimbursement recommendations made through the Common Drug Review process and INESSS and to identify the listing decisions for Alberta, BC and Ontario.10-14 The study design for this research followed the STROBE Initiative's recommendations for reporting observational studies15 (available at www.cmajopen.ca/content/4/4/E674/suppl/DC1).
Data sources
CADTH was the first agency whose website we reviewed to identify the list of drug products for which a recommendation met our inclusion criteria: the recommendation had to be for an initial submission to the Common Drug Review and it had to have been issued between Jan. 1, 2009, and Jan. 1, 2015. We chose to compile data for a 6-year range to ensure we could include health technology assessment recommendations for a sizeable sample of medicines in our analysis and to capture as many of the subsequent listing decisions by the provincial drug plans as possible. However, if a resubmission was submitted up to July 2016 for any of the drug products that met the initial inclusion criteria, the latest recommendation was recorded. For each eligible medicine, the following data were recorded: generic name, proprietary drug name, indication, date of recommendation and recommendation type (positive or negative).
We next searched the websites of the governments of Alberta, BC, Ontario and Quebec to find the latest listing decision (up to Jan. 1, 2015) by the provincial drug plans for each of the medicine-indication combinations we identified in the first step. Each health technology assessment recommendation or provincial listing decision was recorded by proprietary drug name and indication and categorized as either a positive or negative recommendation/reimbursement decision. A recommendation/decision to reimburse the medicine, even if there were restrictions, was considered positive and a recommendation/decision to not reimburse was considered negative.
The health technology assessment recommendations and provincial listing decisions were collected by a single researcher (N.A.) to ensure consistency, and an independent second expert checked that all relevant data had been appropriately recorded.
Statistical analysis
We compared the Common Drug Review recommendations with provincial listing decisions to calculate how many were aligned. Because the listing decisions differed among provinces, it could be misleading to simply report the total number of decisions that were aligned with the recommendations; therefore, we calculated the percentage agreement between jurisdiction pairs (with each pair consisting of the Common Drug Review recommendations and the corresponding listing decisions of an individual province) to report the percentage of concordant recommendations. We also calculated the kappa coefficient because it determines the proportion of agreement that could be due to chance and may therefore be a more robust measure of agreement.16 For this study, we chose the Wilson score method to calculate confidence intervals because it is suitable for small n values and will not produce confidence intervals with negative values or values greater than 100%.17
Results
We identified 174 medicine-indication pairs in CADTH Common Drug Review recommendations issued from Jan. 1, 2009, to Jan. 1, 2015; 110 of these met the inclusion criterion of being an initial submission (Appendix 1, available at www.cmajopen.ca/content/4/4/E674/suppl/DC1). However, the latest reimbursement recommendation (issued by the Common Drug Review up to July 2016) was recorded for comparison for each of the medicine-indication pairs that met the inclusion criteria.
Among the 110 medicine-indication pairs, listing decisions were available for 95 in Alberta, 102 in Quebec, 104 in Ontario and 106 in BC (Figure 1). The Common Drug Review made negative recommendations for 47.3% of the pairs. Each of the 3 provinces that participate in the Common Drug Review made negative listing decisions for a smaller percentage of pairs, ranging from 31.7% for Ontario to 45.3% for BC (Table 1, Figure 1). Quebec, which does not participate in the Common Drug Review, had the lowest percentage of negative decisions (30.4%). The percentage agreement between Common Drug Review recommendations and provincial listing decisions or recommendations from Quebec was 74.5% for Quebec, 78.8% for Ontario, 78.9% for Alberta and 81.1% for BC (Table 2).
Figure 1.
Number of 110 medicine-indication pairs that received positive and negative reimbursement recommendations by the Common Drug Review issued from January 2009 to January 2015. INESSS = Institut national d'excellence en santé et en services sociaux.
Table 1: Reimbursement recommendations by the CADTH Common Drug Review and listing decisions by 4 provincial drug plans for medicine-indication pairs.
| Health technology assessment agency or provincial payer | Reimbursement recommendation or listing decision; % of medicine-indication pairs (95% CI) | |
|---|---|---|
| Positive* | Negative* | |
| CADTH (n = 110) | 52.7 (43.5-61.8) | 47.3 (38.2-56.5) |
| Alberta Health (n = 95) | 58.9 (48.9-68.3) | 41.1 (31.7-51.1) |
| British Columbia Pharmacare (n = 106) | 54.7 (45.2-63.9) | 45.3 (36.1-54.8) |
| Ontario Drug Benefit Plan (n = 104) | 68.3 (58.8-76.4) | 31.7 (23.6-41.2) |
| INESSS (Quebec) (n = 102) | 69.6 (60.1-77.7) | 30.4 (22.3-39.9) |
Note: CI = confidence interval, INESSS = Institut national d'excellence en santé et en services sociaux.
*A recommendation/decision to reimburse the medicine, even if there were restrictions, was considered positive and a recommendation/decision to not reimburse was considered negative.
Table 2: Agreement between Common Drug Review reimbursement recommendations and provincial listing decisions.
| Province | % agreement | κ value (95% CI) |
|---|---|---|
| Alberta | 78.9 | 0.58 (0.42-0.74) |
| British Columbia | 81.1 | 0.62 (0.47-0.77) |
| Ontario | 78.8 | 0.56 (0.41-0.72) |
| Quebec | 74.5 | 0.47 (0.31-0.64) |
Note: CI = confidence interval.
The kappa scores for agreement between Common Drug Review recommendations and provincial listing decisions were 0.62 (95% confidence interval [CI] 0.47-0.77) for BC, 0.58 (95% CI 0.42-0.74) for Alberta, 0.56 (95% CI 0.41-0.72) for Ontario and 0.47 (95% CI 0.31-0.64) for Quebec.
Interpretation
The listing decisions of the provincial drug plans were generally congruent with the Common Drug Review's recommendations during the study period. According to a commonly cited scale for the interpretation of kappa values,18,19 the level of agreement between Common Drug Review recommendations and provincial listing decisions can be considered substantial for BC and moderate for Alberta, Ontario and Quebec.
Price negotiations and other factors can affect the final decisions of provincial drug plans about whether or not to list a medicine. Manufacturers and provincial payers often negotiate price as a component of product listing agreements, but there is wide variation in the negotiating power of the provinces primarily because of their differing population sizes.20 Price negotiations are a key cause of the lack of congruity between Ontario's listing decisions and the recommendations of the Common Drug Review, because Ontario has the largest population in the country and thus the greatest negotiating power. Research has also shown Ontario to have the greatest proportion of medicines funded through the use of product listing agreements.21 The review process in Alberta allows manufacturers to negotiate a product listing agreement only after a formal decision on the initial price has been reached.22
The recommendations framework of the Common Drug Review has evolved over time. In November 2012, CADTH made its framework publicly available. The framework included a category of "List with criteria and/or conditions" for recommendations that may include a condition that the price be reduced to lead to a greater likelihood of a positive provincial listing decision and accommodate the price negotiations with manufacturers after the Common Drug Review.23 In addition, the framework included a category of "Do not list at submitted price" that has been used in cases in which the drug under review showed a clinical benefit comparable to that of its comparator(s) but its cost or cost-effectiveness relative to that of its comparator(s) was unacceptable. Before November 2012, "Do not list at submitted price" was a subcategory of the "Do not list" category. The Common Drug Review does not evaluate budget impact analyses and affordability, and there are no explicit willingness-to-pay thresholds.23 Before April 2015, product listing agreements could not be negotiated in Quebec before a medicine had been included in the list of medicines approved for reimbursement.24 The Quebec pricing policy also ensures that the province shall not pay more than the lowest negotiated price in Canada.25
The pan-Canadian Pharmaceutical Alliance was established in 2010 and aims to combine the purchasing power of all provinces and territories to negotiate the prices of medicines reviewed by the Common Drug Review or the pan-Canadian Oncology Drug Review.26 The work of the pan-Canadian Pharmaceutical Alliance could lead to more consistent listing decisions across Canada; however, the participating provinces and territories will still have independent budgets that will vary in size, and the prices negotiated by the pan-Canadian Pharmaceutical Alliance may still be more affordable for wealthier provinces.
The results of our study expand previous work and provide valuable insights when compared with those of previous studies with similar methodology. The degree of agreement between Common Drug Review reimbursement recommendations and provincial listing decisions was greater in our study than in a 2011 study by Gamble and colleagues.6 Gamble and colleagues6 calculated agreement between Common Drug Review recommendations and the listing decisions of 11 public drug plans for all Common Drug Review recommendations issued from the inception of the program to May 2009 using the binomial categories "listed" and "not listed." We also used a binomial approach to classification. Gamble and colleagues6 identified Ontario's public drug plan as having the lowest percentage agreement (64.2%) and kappa coefficient (κ = 0.28) with the Common Drug Review recommendations among the 11 plans they studied. We found that Ontario's more recent listing decisions are in greater agreement with Common Drug Review recommendations. The kappa coefficients calculated in our study also suggest that there is now greater alignment among the provinces in terms of their listing decisions than before the inception of the Common Drug Review.27
Other studies have evaluated agreement between provincial listing decisions, but their results are difficult to compare with ours because of differing study methodologies. Anis and colleagues27 calculated kappa coefficients for provincial listing decisions using binomial categories for the 10 provinces by directly comparing provinces because the Common Drug Review did not exist at the time of their study. They reported kappa coefficients ranging from 0.06 to 0.39 for Alberta, BC, Ontario and Quebec. The results of MacDonald and Potvin28 are also difficult to compare with ours because they used "full" and "restricted" as the 2 categories for comparison. Morgan and colleagues29 used different reimbursement categories and did not limit their comparison to new medicines issued a reimbursement recommendation by the Common Drug Review. Attaran and colleagues5 calculated percentage agreements using a multinomial classification system, a method that has been criticized because of the difficulty of accurately comparing restrictions.30
Limitations
At the time of data collection, the recommendations and listing decisions available on the websites of the Common Drug Review and the 4 provinces in the study met our inclusion criteria. However, the Common Drug Review and provincial drug plans continue to update their reports and formularies, so these results provide an insight into the health technology assessment landscape only for a defined period. Because 18 public drug plans participate in the Common Drug Review process, the fact that we reviewed data from only 3 participating provinces and Quebec is a limitation of this study. The evolution of the categories of recommendations in the Common Drug Review framework over time may pose challenges for researchers wishing to examine the agreement between recommendations and provincial listing decisions. The drug plans participating in the Common Drug Review have varying resources available to review new medicines in the context of their population; therefore, the results of our study may not be generalizable to all 18 participating drug plans. Future studies can build on this research by evaluating the concordance between Common Drug Review recommendations and listing decisions for all participating federal, provincial and territorial drug plans.
Conclusion
Our study showed moderate to substantial agreement between provincial listing decisions and Common Drug Review reimbursement recommendations. The findings suggest that the Common Drug Review provides value for participating drug plans. It could be argued that these observed scores of alignment could indicate that the provinces have become more reliant on the Common Drug Review over time and that the Common Drug Review continues to improve its ability to meet payers' needs. The fact that the provinces are able to come to different decisions on the basis of Common Drug Review recommendations illustrates the flexibility of the process. Provincial payers, which have differing budgets and patient populations, can consider their local context in their decision-making. The findings of our study could have implications for other regions with a centralized regulatory authority and a fragmented payer environment, such as Europe. Even though European countries are much more heterogeneous than Canadian provinces, the Common Drug Review provides an example of a centralized review process that generates evidence to support the common requirements of participating plans with the added advantage that each plan can incorporate context-specific evidence and budgetary considerations into its decision-making.
Supplemental information
For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/4/4/E674/suppl/DC1
Supplementary Material
Acknowledgements
The authors thank Professor Robert Peterson (Executive Director, Drug Safety and Effectiveness Network, Canadian Institutes of Health Research), CADTH, Alberta Health, British Columbia Pharmacare, the Ontario Ministry of Health and Long-Term Care and the Institut national d'excellence en santé et en services sociaux for their invaluable contributions. The authors also acknowledge the thorough and thoughtful editorial assistance of Patricia Connelly.
References
- 1.Allen N, Pichler F, Wang T, et al. Development of archetypes for non-ranking classification and comparison of European National Health Technology Assessment systems. Health Policy. 2013;113:305–12. doi: 10.1016/j.healthpol.2013.09.007. [DOI] [PubMed] [Google Scholar]
- 2.Health care technology and its assessment in eight countries. Washington (DC): Office of Technology Assessment. 1995. [Google Scholar]
- 3.Menon D, Stafinski T. Health technology assessment in Canada: 20 years strong? Value Health. 2009;12(Suppl 2):S14–9. doi: 10.1111/j.1524-4733.2009.00554.x. [DOI] [PubMed] [Google Scholar]
- 4.Spitz S. A decade of the Common Drug Review. CMAJ. 2013;185:E277–8. doi: 10.1503/cmaj.109-4428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Attaran A, Cartagena R, Taylor A. The effectiveness of the Common Drug Review in Canada's national drug strategy. Halifax: Atlantic Institute for Market Studies. 2011. [Google Scholar]
- 6.Gamble JM, Weir DL, Johnson JA, et al. Analysis of drug coverage before and after the implementation of Canada's Common Drug Review. CMAJ. 2011;183:E1259–66. doi: 10.1503/cmaj.110670. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.McMahon M, Morgan S, Mitton C. The Common Drug Review: a NICE start for Canada? Health Policy. 2006;77:339–51. doi: 10.1016/j.healthpol.2005.08.006. [DOI] [PubMed] [Google Scholar]
- 8.Morgan SG, McMahon M, Mitton C, et al. Centralized drug review processes in Australia, Canada, New Zealand, and the United Kingdom. Health Aff (Millwood) 2006;25:337–47. doi: 10.1377/hlthaff.25.2.337. [DOI] [PubMed] [Google Scholar]
- 9.Hollis A, Law S. A national formulary for Canada. Can Public Policy. 2004;30:445–52. [Google Scholar]
- 10.Common Drug Review (CDR). Ottawa: CADTH. 2016. [accessed 2016 July 31]. Available www.cadth.ca/about-cadth/what-we-do/products-services/cdr.
- 11.Alberta drug benefit list (ADBL). Edmonton: Alberta Health. 2015. [accessed 2016 July 31]. Available www.health.alberta.ca/services/drug-benefit-list.html.
- 12.PharmaCare for B.C. residents. Victoria: Government of British Columbia. 2015. [accessed 2016 July 31]. Available www2.gov.bc.ca/gov/content/health/health-drug-coverage/pharmacare-for-bc-residents.
- 13.Ontario public drug programs. Toronto: Ontario Ministry of Health and Long-Term Care. 2015. [accessed 2016 July 31]. Available www.health.gov.on.ca/en/pro/programs/drugs.
- 14.Evaluation process and criteria. Québec: Institut national d'excellence en santé et en services. 2015. [accessed 2016 July 31]. Available www.inesss.qc.ca/en/activities/drug-products/evaluation-process-and-criteria.html.
- 15.von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12:1495–9. doi: 10.1016/j.ijsu.2014.07.013. [DOI] [PubMed] [Google Scholar]
- 16.Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543–9. doi: 10.1016/0895-4356(90)90158-l. [DOI] [PubMed] [Google Scholar]
- 17.Brown LD, Cai TT, DasGupta A. Interval estimation for a binomial proportion. Stat Sci. 2001;16:101–33. [Google Scholar]
- 18.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. [PubMed] [Google Scholar]
- 19.Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37:360–3. [PubMed] [Google Scholar]
- 20.Pan Canadian drugs negotiations report. Ottawa: Health Care Innovation Working Group. 2014. [accessed 2016 Aug. 7]. Available www.pmprovincesterritoires.ca/phocadownload/pcpa/pan_canadian_drugs_negotiations_report_march22_2014.pdf.
- 21.Morgan SG, Friesen MK, Thomson PA, et al. Use of product listing agreements by Canadian provincial drug benefit plans. Healthc Policy. 2013;8:45–55. [PMC free article] [PubMed] [Google Scholar]
- 22.Alberta Health and Wellness product listing agreement policy. Edmonton: Alberta Health. 2011. [accessed 2016 Aug. 6]. Available www.ab.bluecross.ca/dbl/pdfs/PLA_policy.pdf.
- 23.Common Drug Review: recommendations options and deliberative framework. Ottawa: CADTH. 2012. [accessed 2016 Aug. 26]. Available www.cadth.ca/sites/default/files/cdr/cdr-pdf/CDEC_Deliberative_Framework_e.pdf.
- 24.Bill 28: an Act mainly to implement certain provisions of the Budget Speech of 4 June 2014 and return to a balanced budget in 2015-2016. Québec: National Assembly of Quebec. 2015. [accessed 2016 Aug. 10]. Available www2.publicationsduquebec.gouv.qc.ca/dynamicSearch/telecharge.php?type=5&file=2015C8A.PDF.
- 25.Price policy. Québec: Institut national d'excellence en santé et en services sociaux. 2016. [accessed 2016 Aug. 10]. Available www.inesss.qc.ca/en/activities/drug-products/price-policy.html.
- 26.The pan-Canadian Pharmaceutical Alliance. Ottawa: Council of the Federation Secretariat. 2015. [accessed 2016 Aug. 8]. Available www.pmprovincesterritoires.ca/en/initiatives/358-pan-canadian-pricing-alliance.
- 27.Anis AH, Guh D, Wang Xh. A dog's breakfast: prescription drug coverage varies widely across Canada. Med Care. 2001;39:315–26. doi: 10.1097/00005650-200104000-00003. [DOI] [PubMed] [Google Scholar]
- 28.MacDonald K, Potvin K. Interprovincial variation in access to publicly funded pharmaceuticals: a review based on the WHO anatomical therapeutic chemical classification system. Can Pharm J (Ott) 2004;137:29–34. [Google Scholar]
- 29.Morgan S, Gillian H, Raymond C, et al. Breadth, depth and agreement among provincial drug formularies in Canada. Healthc Policy. 2009;4:e162–84. [PMC free article] [PubMed] [Google Scholar]
- 30.Fischer KE. A systematic review of coverage decision-making on health technologies - evidence from the real world. Health Policy. 2012;107:218–30. doi: 10.1016/j.healthpol.2012.07.005. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.