Use of real-world evidence in cancer drug funding decisions in Canada: a qualitative study of stakeholders’ perspectives

Abstract

Background:

Real-world evidence (RWE) can provide postmarket data to inform whether funded cancer drugs yield expected outcomes and value for money, but it is unclear how to incorporate RWE into Canadian cancer drug funding decisions. As part of the Canadian Real-World Evidence Value for Cancer Drugs (CanREValue) Collaboration, this study aimed to explore stakeholder perspectives on the current state of RWE in Canada to inform a Canadian framework for use of RWE in cancer drug funding decisions.

Methods:

This was a qualitative descriptive study. Qualitative semistructured interviews were conducted from April to July 2018. Participants were Canadian and international stakeholders who had experience with RWE and drug funding decision-making. Thematic analysis was used to analyze data.

Results:

Thirty stakeholders participated in the study. Five themes were identified. Stakeholders indicated that RWE had value in cancer drug funding decisions. However, a cultural shift is needed to adopt RWE in decision-making. Further, the Canadian infrastructure for real-world data is currently inadequate for decision-making, and there is a need for committed investment in building capacity to collect and analyze RWE. Finally, there is a need for increased collaboration among key stakeholders.

Interpretation:

The findings of this study suggest that if RWE is to be used in drug funding decisions, there is a need for a cultural shift, improved data infrastructure, committed investment in capacity building and increased stakeholder collaboration. Together with local stakeholder engagement, application of these findings may contribute to optimizing implementation of RWE.

The costs of cancer drugs are rising rapidly.^1,2 Although some new drugs provide substantial therapeutic improvements, others confer only marginal survival benefits or improve only quality of life. Assessment of the overall costs and benefits of cancer drugs is essential for resource allocation. Drug funding decision-makers typically rely on data from clinical trials to supply clinical and economic evidence.^3,4 However, randomized controlled trials (RCTs) have highly selected populations, limiting their real-world generalizability.^5–9 Funding decisions are not revisited, and cost-effectiveness and clinical effectiveness are not re-assessed, after a drug enters the Canadian market. ¹⁰ Decision-makers have little information on whether drug investments yield expected outcomes.¹⁰ Real-world evidence (RWE) — evidence from postmarket evaluations not derived from traditional RCTs¹¹ — could fill these gaps with information on clinical effectiveness, safety, cost-effectiveness and budget impact outside of the highly controlled trial environment.^3,5,6,12

The Canadian Real-World Evidence for Value of Cancer Drugs (CanREValue) Collaboration, consisting of stakeholders in cancer control across and outside Canada (e.g., researchers, government ministries, decision-makers, payers, patients and caregivers),¹⁰ was formed to create an RWE framework to apply to cancer drug funding in Canada. The aim of this study was to inform CanREValue’s framework development through an exploration of Canadian and international stakeholders’ views and experiences with RWE, such that the framework would address end users’ needs and facilitate the uptake of RWE into cancer drug funding decision-making.

Methods

Design

Qualitative descriptive methodology (aligned with naturalistic inquiry) with thematic analysis was used to explore stakeholders’ views and experiences regarding incorporation of RWE into cancer drug funding decisions.^13–15 We chose this methodology because we wanted to understand and describe participants’ experiences and perspectives.¹⁴ Semistructured interviews were conducted between April 2018 and July 2018.

Recruitment

We aimed to recruit stakeholders of RWE, that is, people involved in using RWE in the context of drug evaluation or decision-making, or people assessing the value and implications of RWE across academia, industry, health technology assessment (HTA) and government. Our goal was to recruit a pan-Canadian sample of participants from all provinces and territories working in cancer drug evaluation, as well as international RWE experts. We sought to include people who were exposed to RWE in the context of drug evaluation (e.g., people who had used RWE in decision-making) or who worked in an academic setting assessing the value and implications of RWE.

An initial convenience sample was identified by the research team via referral, a CanREValue environmental scan on RWE and a search of websites listing memberships of HTA and advisory committees (e.g., www.ispor.org). Snowball sampling was used to recruit other stakeholders: each interview participant was asked to recommend people who had relevant experience with RWE and cancer drug evaluation. The study team then contacted these people to invite them to participate. Patient representatives with experience in cancer drug advisory groups were identified from the Cancer Care Ontario advisory board and invited to participate by the study team. International experts (people from outside of Canada with experience implementing RWE in decision-making at a national or regional level) were recruited to provide information on lessons learned from actually implementing RWE.

All people invited to participate were emailed a study invitation by K.K.W.C. or Y.B. with study information and the consent form; this was the only relationship established before study commencement. Participants were sampled until thematic saturation was reached.^14,16

Data collection

Participants took part in 1-on-1 semistructured interviews over the phone or in person (at their workplace), conducted by research coordinators (M.C., R.K.). Both interviewers had training and experience in qualitative research. Informed by a review of the background literature and feedback from the CanREValue team (including W.F.D., R.E.M., J.M.B., W.I., K.K.W.C. and Y.B.), an interview guide (English) was developed by M.C., R.K. and Y.B., which explored thoughts and experiences with RWE, perceived barriers to and facilitators of uptake, and readiness to incorporate RWE into decision-making (Appendix 1, available at www.cmajopen.ca/content/8/4/E772/suppl/DC1).

The review of the background literature was conducted using PubMed by R.K. Search terms included “real world evidence,” “real world data,” “stakeholder perspectives” and “stakeholder attitudes.” No publication date limit was imposed, and only English-language articles were reviewed. Studies that were identified were reviewed by M.C. and R.K. The aim of the literature review was to identify qualitative studies on RWE implementation in order to understand what research had been conducted on this topic and to identify areas of discussion that would merit exploration in our interviews. Interview guides were reviewed if available, and themes identified by the studies were reviewed for salient topics for discussion. Additionally, the CanREValue team had previously conducted an environmental scan, which identified topics related to RWE implementation that merited discussion.

Two of the authors (M.C., R.K.) developed interview questions to address topics related to the study aim as well as gaps identified in the literature from the review of the background literature. The interview guide was reviewed by the study team for content and flow. One of the authors (R.K.) piloted the interview guide by conducting mock interviews with members of the broader CanREValue Collaboration not directly involved in this study.

The interview guide was revised throughout data collection to capture emerging questions and domains. Interviewers took field notes after each interview. Interviews were audio recorded and transcribed verbatim. Transcripts and findings were not returned to participants for comments or correction. Repeat interviews were not conducted.

Data analysis

Data analysis was concurrent with data collection and began after the first few interviews had been conducted. The transcribed interviews were analyzed using thematic analysis. Interviews were coded by M.C., C.M. and R.K. An initial codebook was developed by M.C. and R.K. through immersion in the data and discussion at research team meetings. Initial codes were derived from topics in the interview guide and were supplemented by codes that were inductively derived from interview data. The codebook was modified as new codes and themes emerged from subsequent interviews. Codes were grouped into larger themes and patterns.

Constant comparison of the data was used to explore common and divergent themes across interviews. Barriers to and facilitators of RWE uptake described by Canadian stakeholders were compared with international experts’ experiences implementing RWE, to triangulate Canadian perspectives with experiences from other health systems. Reflexive notes were analyzed and incorporated into the study results. Research team members met periodically to review codes and discuss major themes.

Data analysis and management was conducted using HyperRESEARCH. When conducting analysis, the research team reflexively considered how their assumptions about the value of RWE for decision-making played a role in the interviews and their interpretation. It was determined that saturation was met when there was repetition in the participants’ responses and interviews no longer yielded ideas, codes or themes that had not already been identified; at this point, recruitment was stopped.

Ethics approval

Ethical approval was obtained from the Unity Health Toronto Research Ethics Board.

Results

Forty people were invited to participate. Eight did not respond to the invitation email, and 2 declined to participate. A total of 30 stakeholders (Table 1) participated in interviews (approximately 30–75 min in duration).

Table 1:

Participants’ demographic characteristics

Characteristic	No. (%) of participants n = 30
Role
Decision-maker	14 (46.7)
Academic	5 (16.7)
Industry representative	4 (13.3)
Patient advisor	4 (13.3)
International expert	3 (10.0)
Institution type
Academic	6 (20.0)
Industry	4 (13.3)
Government (e.g., ministry)	4 (13.3)
Health technology assessment or health economics not-for-profit organization	4 (13.3)
Federal or provincial health authority	6 (20.0)
Federal or provincial pharmaceutical pricing negotiation	2 (6.7)
Patient advisory board	4 (13.3)
Sex
Male	14 (46.7)
Female	16 (53.3)
Region
Central Canada	15 (50.0)
Atlantic Canada	2 (6.7)
Prairie provinces (Manitoba, Saskatchewan, Alberta)	6 (20.0)
British Columbia	4 (13.3)
Outside Canada (Scotland, England, United States)	3 (10.0)

Summary of qualitative themes

Five themes were identified related to participants’ views and experiences with RWE: RWE can be valuable in cancer drug funding decisions, there needs to be a cultural shift to adopt RWE in decision-making, the Canadian infrastructure for real-world data is currently inadequate for decision-making, there is a need for committed investment in building capacity to collect and analyze RWE, and there is a need for increased collaboration among key stakeholders. Each theme is described in detail in the following sections.

Stakeholders value RWE in cancer drug funding decisions

All stakeholders expressed enthusiasm and optimism about incorporating RWE into cancer drug funding decisions to address the limitations of RCTs (e.g., RCTs are time limited and resource intensive and have limited generalizability) and provide evidence on whether a drug provided “good value for money spent” (interviewee 5, Canadian) in the real world (Table 2; see also quote from interviewee 13, Canadian).

Table 2:

Key themes and illustrative quotes

Theme	Description	Illustrative quotes
Stakeholders value RWE in cancer drug funding decisions	Stakeholders expressed enthusiasm and optimism about the possibility of incorporating RWE into cancer drug funding decisions to address the limitations of RCTs and provide evidence on whether a drug provided “good value for money spent” (interviewee 5, Canadian) in the real world.	So, I think real-world evidence is an essential part of what we need to do in terms of bringing sort of science to real world decision-making. … We need [RWE] and I think we are in a very, we are at a point where certainly decision-makers must have that information. (Interviewee 13, Canadian)
		Rather than us accepting these [cost-effective] models that are fanciful, in which there’s tremendous uncertainty in them, you’d be better to make an additional recommendation to fund and collect data prospectively in the real world to see, What is the duration of survival? What is the duration of treatment? What are the long-term toxicities? Things that you don’t pick up in a clinical trial. … There has to be much more certainty of their benefits and with certainty on the benefits, then we could … if we’re going to require drugs to be cost-effective, we’re in a better position to negotiate what those prices might be and get it into what we consider a cost effective range. (Interviewee 2, Canadian)
A cultural shift is required to adopt RWE in decision-making	Although participants were enthusiastic about RWE’s potential for greater external validity than RCTs, they recognized that a cultural shift is required for decision-makers to move beyond “gold standard” (interviewee 11, Canadian) evidence from RCTs.	I guess it’s just easier with randomized controlled trials, because I think the approach and the accepted analytic methods are much better known. And there’s a lot more debate, and I guess uncertainty about what the best methods would be in real-world evidence because there’s so many variables. It creates a situation where it’s easy to criticize any analysis that’s done. (Interviewee 14, Canadian)
Canadian RWE data infrastructure is currently inadequate for decision-making	Participants saw challenges with data quality and access as the biggest barriers to using RWE at present. As such, participants were uncertain of how our current data infrastructure could be transformed so that it can be used to inform quality decisions.	I also think that there’s still a scarcity of data, that we don’t have data for everything yet. We have a lot of data, but it seems to be unorganized and lack of consistency of how people are gathering data. So, until we really can get our data together, and that it’s shared, it’s consistent, it’s gathered in the same way, and it’s pool-able, until that is done I think it can be challenging to really use the data. (Interviewee 10, Canadian)
		Linking the different data sets and different electronical health records together is still a huge challenge especially here in the US where you have so many different payers, so many different systems. (Interviewee 19, international)
Committed investment in building capacity is required	Stakeholders perceived the Canadian drug funding decision-making system as stretched beyond capacity in terms of finances, expertise and leadership and saw these factors as a barrier to the adoption of an RWE framework.	So, it’s a kind of problematic issue right now to be pursuing real-world evidence-based agreements for too many products because they’re very complex and it takes a long time and it takes a lot of resources because capacity is very stretched. (Interviewee 7, Canadian)
There is a need for increased collaboration among key stakeholders	Participants noted that systems are currently operating in silos and emphasized the need to increase engagement among stakeholders. There was a diversity of opinion on whether and how to engage industry.	If you’re going to do these studies, there has to be a change in the attitudes between the players. Like, the payers, government and cancer agencies, typically don’t have a really good relationship with industry … only if you have that kind of collaborative environment, would you actually be able to undertake some of these studies efficiently and effectively.” (Interviewee 2, Canadian)

Note: RCT = randomized controlled trial, RWE = real-world evidence, US = United States.

RWE was described as a valuable source of supplemental data that can inform postmarket decisions about continued funding, price renegotiations or delisting of drugs currently on the formulary. Participants described how RWE could provide postmarket data to reduce uncertainty about a drug’s long-term performance and assist the payer in price negotiations (Table 2, quote from interviewee 2, Canadian).

However, some participants expressed concern that if drugs were delisted on the basis of RWE, treatment options would be taken away, and patients and the public would “fight tooth and nail” to maintain access to currently listed medications (interviewee 12, Canadian). Others noted that RWE might have limited utility in provinces that fund fewer drugs and are seeking to expand the number of available therapies. Participants expressed a need for clarity about the intended outcomes of incorporating RWE into decision-making.

A cultural shift is required to adopt RWE in decision-making

Although participants were enthusiastic about the greater external validity of RWE compared that of RCT evidence, they recognized that a cultural shift was required for decision-makers to move beyond the traditional, “gold standard” (interviewee 11, Canadian) evidence provided by RCTs. In contrast, RWE was perceived as being susceptible to bias and confounding, with inconsistent data collection, analytic methods and conclusions. To adopt RWE in decision-making, decision-makers would need to trust RWE and accept RWE’s uncertainty (Table 2, quote from interviewee 14, Canadian). Overall, participants suggested that a culture shift away from sole reliance on RCT data was needed.

Some participants perceived that the incorporation of RWE into decision-making might serve as a catalyst for transforming the collection and use of health care data in Canada. These participants suggested developing mechanisms to manage uncertainties through conditional approval, whereby results can continue to be captured until the data have matured to inform the final decision about public funding for a drug.

Canadian RWE data infrastructure is currently inadequate for decision-making

Participants saw data infrastructure and access as the biggest barrier to using RWE at present. In Canada, real-world data are collected by multiple organizations operating in different provinces. Participants described how current data collection procedures were not built for evaluation: real-world data are not standardized, are not embedded into clinical workflows and are not consistently collected. Participants noted that key measures (e.g., patient-reported outcomes) are often missing. Many participants described the current Canadian data infrastructure as patchy and unreliable, limiting its utility (Table 2, quote from interviewee 10, Canadian).

Participants feared that effective drugs would be delisted or ineffective drugs would continue to be funded if inconclusive or incorrect data were used in decision-making. These participants preferred to wait for consistent collection of all necessary outcomes for a decisive RWE evaluation. Participants projected that it would take 3–7 years to generate suitable RWE for decision-making, although some acknowledged that certain Canadian jurisdictions currently collect RWD of sufficient quality. Some participants expressed comfort in using international data to fill Canadian data gaps, while others noted that RWE is extremely contextual.

Although recognizing the need for data protection, many stakeholders expressed frustration with the time and financial resources wasted because of difficult and inefficient procedures for data access. Academic and industry representatives described the importance of timely access, as RWE cannot inform decisions throughout a product’s life cycle if it takes 2–3 years to access it. With respect to data protection, patient representatives stated that patients would be willing and eager to share personal data for research, provided it was anonymized, used in aggregate and protected from insurers and employers. However, some patient representatives feared that the sharing of personal data with industry could result in its misuse. Participants suggested that patient groups be involved in revising data access procedures.

International experts corroborated the concerns about data infrastructure raised by the Canadian stakeholders, reporting that fragmented data sets were a major barrier to using RWE (Table 2, quote from interviewee 19, international). Canadian and international stakeholders suggested key improvements necessary for uptake of RWE: guidance on the collection of thorough and relevant data, a unified pan-Canadian data collection infrastructure and a learning health system approach where RWE could be used to make funding decisions.

Committed investment in building capacity is required

Participants described a need for investments in system-wide capacity building to support RWE; current system readiness was seen as “very poor” (interviewee 17, Canadian). Stakeholders perceived the Canadian decision-making system for drug funding as stretched beyond capacity in terms of finances, expertise and leadership, inhibiting the adoption of an RWE framework (Table 2, quote from interviewee 7, Canadian).

All participants discussed a lack of capacity to cover the costs needed to generate and use RWE, and a lack of clarity regarding how to divide costs and roles between public and private sectors. Some participants stated that industry should be responsible for RWE-associated costs. Others supported public funding for RWE, to reduce perceptions of bias associated with industry-generated evidence.

Participants noted that few people have the expertise required to generate and analyze RWE appropriately. They highlighted a need to invest in training programs to build capacity for RWE analyses across Canada.

Finally, participants discussed the need for strong leadership and clear roles and responsibilities. Otherwise, participants stated, different groups would use RWE in varying ways, undermining the benefits of a unified approach to its uptake in decision-making. Participants suggested that RWE could first be used at a provincial level to work out issues on a smaller scale before pan-Canadian adoption.

There is a need for increased collaboration among key stakeholders

All participants expressed a desire for collaboration across organizations. The current siloed state of RWE was identified as a substantial barrier to its adoption (Table 2, quote from interviewee 2, Canadian). All participants acknowledged that the relationship between industry and the public sector is strained, but they recognized that RWE use in Canada would not be feasible without industry participation. Some participants expressed concern about loss of public control of data if industry acted as gatekeepers of RWE and that industry would withhold RWE if it worked against their financial interest. Others saw RWE as an opportunity to improve relations with the private sector. Such a partnership could give the public sector access to data and technical expertise currently limited to the private sector. International experts also recognized the need to involve industry in generating and collecting RWE, but they raised questions about data ownership, data governance and which sector would pay for data collection.

Industry participants were eager to partner with academic and government organizations. Industry participants stated that their teams had the resources and experience needed to work with RWE and that they had much to offer to the development and implementation of an RWE framework. To industry participants, partnership could improve the currently fragmented state of RWE, benefiting both sectors.

Participants across the public and private sectors suggested that stakeholders determine early on what role industry will play in the development of an RWE framework (Box 1). Belgium was raised as a model of true partnership where both sectors benefit from each other by sharing data and costs.

Box 1: Overall suggestions to incorporate real-world evidence into cancer drug funding decisions.

• Clarify the intended outcome of using real-world evidence

• Improve data collection mechanisms

• Determine when and how real-world evidence would be used

• Balance the need for real-world evidence with needs of privacy

• Involve patient groups and industry

• Apply real-world evidence at a provincial level first

• Build infrastructure, capacity and expertise in real-world evidence

• Use “conditional reimbursement”

Interpretation

Stakeholders in this study were interested in using RWE to fill gaps in the processes currently used to make cancer drug funding decisions to achieve better patient and economic outcomes. However, barriers (e.g., data quality, stakeholder collaboration) must first be addressed in any framework that aims to guide effectively RWE use in decision-making by Canadian stakeholders.

Stakeholders identified barriers to use of RWE in decision-making, including lack of expertise in RWE methodology, lack of universally accepted methodologic standards, challenges in accessing data and issues of bias and confounding, consistent with the literature.^17,18 Another concern raised by participants was the use of RWE to replace RCT data to demonstrate efficacy, given the risk of bias and uncertainty. Participants suggested using conditional reimbursement to manage uncertainty associated with RWE. This is consistent with how some European HTA agencies have used RWE; they have not commonly used it to understand treatment effects or in initial reimbursement decisions.¹⁹

Other solutions proposed by our participants included better governance of data access and building capacity for RWE analysis, as described elsewhere.¹⁸ A contribution that our study makes to the literature is the identification by stakeholders of the need for a cultural shift away from sole reliance on RCT data in decision-making. Another contribution is the finding that our participants were unaware of ongoing efforts to address the quality of real-world data in Canada. Efforts to improve data quality must be made more transparent to assuage decision-makers’ concerns about the readiness of RWE for use in decision-making.^20,21

Limitations

Study limitations include lack of representation from the health care systems of Quebec and the territories. We reached thematic saturation within our sample, but there was variation within different groups of respondents (e.g., government, industry). Future research may be needed to uncover new perspectives. As with all qualitative research, the study’s findings are not generalizable, but they are valuable for understanding the challenges and potential solutions to RWE adoption in drug funding.

Conclusion

Incorporating RWE into a health care system’s decision-making process is complex. In this study, stakeholders showed that a cultural shift would be needed to include evidence beyond RCTs in drug-funding decisions. In addition, stakeholders require improved data infrastructure, a committed investment to building the necessary financial, leadership and expert capacity to implement RWE, and increased stakeholder collaboration (particularly between the private and public sectors). The findings of this study suggest that if RWE is to be used in drug funding decisions, there is a need for a cultural shift, improved data infrastructure, committed investment in capacity building and increased stakeholder collaboration. Together with local stakeholder engagement, application of these findings may contribute to optimizing implementation of RWE.