The prioritization preferences of pan-Canadian Oncology Drug Review members and the Canadian public: a stated-preferences comparison

Abstract

The pan-Canadian Oncology Drug Review (pcodr) is responsible for making coverage recommendations to provincial and territorial drug plans about cancer drugs. Within the pcodr process, small groups of experts (including public representatives) consider the characteristics of each drug and make a funding recommendation. It is important to understand how the values and preferences of those decision-makers compare with the values and preferences of the citizens on whose behalf they are acting.

In the present study, stated preference methods were used to elicit prioritization preferences from a representative sample of the Canadian public and a small convenience sample of pcodr committee members. The results suggested that neither group sought strictly to maximize quality-adjusted life year (qaly) gains and that they were willing to sacrifice some efficiency to prioritize particular patient characteristics. Both groups had a significant aversion to prioritizing older patients, patients in good pre-treatment health, and patients in poor post-treatment health. Those results are reassuring, in that they suggest that pcodr decision-maker preferences are consistent with those of the Canadian public, but they also imply that, like the larger public, decision-makers might value health gains to some patients more or less highly than the same gains to others. The implicit nature of pcodr decision criteria means that the acceptability or limits of such differential valuations are unclear. Likewise, there is no guidance as to which potential equity factors—for example, age, initial severity, and so on—are legitimate and which are not. More explicit guidance could improve the consistency and transparency of pcodr recommendations.

BACKGROUND

The pan-Canadian Oncology Drug Review (pcodr), part of the Canadian Agency for Drugs and Technologies in Health, is responsible for making coverage recommendations about cancer drugs to provincial and territorial drug plans. The pcodr review process is designed “to bring consistency and clarity to the assessment of cancer drugs” and emphasizes four dimensions of value in their decision criteria: clinical benefit, economic evaluations, patient-based values, and adoption feasibility¹. The pcodr guidelines state that there is no weighting scheme for the criteria and no threshold that must be met for any single element of the review. Rather, decisions should be made on the basis of the individual drug, disease, and context¹. In that regard, the pcodr could be described as taking an implicit approach to decision-making. Proponents of implicit approaches to decision-making argue that some ambiguity is necessary to address the inherent complexity of priority-setting, allowing for individual decision-makers to exercise appropriate contextual judgment^2–4. However, advocates of more explicit processes argue that being vague or implicit can lead to inconsistency in decision-making and has the potential to create an invisible class of “others,” who might be victims of injustice or discrimination without knowing it^5,6.

Just a handful of jurisdictions—including England, Ireland, and Thailand^7,8—explicitly define acceptable thresholds, usually with respect to efficiency in terms of cost per quality-adjusted life-year (qaly) gained. For example, the National Institute for Health and Care Excellence in England has set an acceptable cost-effectiveness “range” of £20,000 to £30,000 per qaly gained⁹. Submissions below £20,000 are most often accepted; submissions greater than £20,000 per qaly must demonstrate other aspects of value. Submissions greater than £30,000 per qaly are most often rejected. The Institute also organizes “citizen’s juries” to identify broad societal values and to understand acceptable trade-offs between efficiency and various aspects of equity¹⁰, although criticisms of how well small groups of citizens can represent the preferences of society as a whole have been expressed¹¹. On the basis of feedback from one such jury, the National Institute for Health and Care Excellence makes an explicit allowance for a higher acceptable threshold for life-extending treatments at the end of life¹². Again, questions about the economic and ethical justifications for that policy have been raised^13,14, but the presence of an explicit policy has the advantage of being transparent and stimulating debate.

In making its recommendations, pcodr, like many other decision-making bodies, relies on a process of “procedural objectivity,” whereby societal decision-making is delegated to small groups of experts (often including members of the lay public) appointed on the basis of their knowledge, expertise, and professionalism^11,12. In that context, pcodr decision-makers can be seen to be acting as “agents” on behalf of society, ensuring that limited societal resources—in this case, funding for cancer drugs—are put to their most valuable uses¹³. That approach, though, concentrates decision-making authority in the hands of a relatively small number of experts. Arguably, the responsibility of societal decision-makers is not always to reflect what citizens would do, but rather what they ought to do¹⁴. That is, the knowledge and expertise of the decision-makers can in some cases lead to decisions at odds with public opinion. However, the allocation of societal resources is, even within a putatively objective process, an unavoidably subjective issue, and therefore the preferences of decision-makers should be broadly consistent with the values and preferences of society¹⁵.

In light of those issues, understanding how the preferences of pcodr decision-makers compare with those of the Canadian public with respect to the allocation of scarce health care resources is of interest. Within that overall objective, the first aim of the present study was to understand the degree to which the allocation choices of respondents might be driven by the principles of qaly maximization. The qaly maximization framework requires that resources be allocated in the way that produces the greatest aggregate qaly gain, which combines years of life, quality of life, and the total number of beneficiaries into a single summary measure of health gains. Critically, the qaly maximization framework presumes “distributive neutrality,” or a societal indifference to how health gains are distributed between various individuals or groups^16,17. The qaly maximization paradigm is dominant in health care priority-setting and has been shown to play an important role in drug review processes in the United Kingdom and Australia^18,19. The qaly maximization framework is also reflected in pcodr’s consideration of relative cost-effectiveness, although it is important to note that pcodr does not define an acceptable cost-effectiveness threshold.

However, recent research has suggested that the qaly maximization framework might not be consistent public preferences and that the public is not indifferent to how health gains are distributed. Rather, there is evidence of a willingness to forego some qaly gains so as to distribute health gains more fairly or equitably^16,20–22. If most respondents to that elicitation did not appear to make allocation choices on the basis of strict qaly maximization, the second aim of the present work was to compare the degree to which pcodr decision-makers and the Canadian public are willing to forego efficiency for greater equity. That sacrifice is known as an “equity–efficiency trade-off,” and it is based on the idea that equity and efficiency are commensurate concepts and that more of one can compensate for less of the other²³. That is, society might be willing to prioritize a relatively inefficient program if it were to be associated with a fairer or more equitable outcome. The trade-off has limits, though, and at some point, a gain in equity is too minor to justify the sacrifice in efficiency, or alternatively, a gain in efficiency is so great that it justifies some inequity. How might the limits of that equity–efficiency trade-off then differ between the public and their agents?

METHODS

The preferences of an age- and sex-representative sample of the Canadian public and of a convenience sample of pcodr committee members were elicited using stated preference methods. Those methods ask respondents to make choices between hypothetical alternatives, each described in terms of sets of attributes and levels. By systematically varying the levels of the attributes in a series of tasks, the weight that respondents give to various attributes and levels in their choices can be inferred²⁴.

This research was approved by the Capital Health Research Ethics Board, Halifax, Nova Scotia, and by the University of Sheffield Research Ethics Committee, Sheffield, United Kingdom. It was funded in part by the Canadian Centre for Applied Research in Cancer Control and the Capital Health Research Fund, Halifax, Nova Scotia.

Experimental Design

The process of identifying the patient and program attributes used to describe each patient group has been detailed elsewhere²². Briefly, the attributes were identified through a process of empirical ethics, whereby each attribute had to have empirical evidence of public support and had to be consistent with some coherent and defensible theory of justice²⁵. An empirical filter ensures that the attributes are relevant to society; an ethical filter contributes to the legitimacy of the attributes and avoids the “moral relativism” that might result from strictly empirical methods, whereby what is morally right or wrong is reduced to social consensus or a simple majority²⁶.

A “pearl growing” search strategy was used to identify patient and program characteristics with empirical evidence of societal relevance, building on the references and forward citations of a handful of key articles. The ethical justification for those characteristics was then subjectively assessed on the basis of prominent theories of distributive justice, including need, maximization, and egalitarian principles. During the process, patient age, severity before or without treatment, final health state with or after treatment, and the distribution of health gains were judged to have evidence of public support and a defensible ethical justification. To distinguish between “severity” as proximity to death and as a poor health state, that factor was deconstructed into life expectancy without treatment and initial health state. Distribution of benefit was considered in terms of the number of patients who could be treated. Finally, duration of benefit was included to facilitate the calculation of qaly gains—despite some ambiguity concerning its interpretation in that context.

It is worth noting that a number of attributes included in similar elicitations were excluded from the process used here. Most notably, prioritization on the basis of a patient’s lifestyle or culpability for their illness had empirical support, but was judged to have little ethical justification, and priority on the basis of social role or productivity had a utilitarian ethical justification, but little empirical support.

To balance information about the shape of the utility function with statistical efficiency, each attribute was assigned 3 levels, evenly spaced across plausible ranges, to allow for the identification of nonlinear preferences in a minimal number of scenarios. The aggregate qaly gain was calculated as a function of the other attribute levels and included as an additional attribute in each alternative, but was excluded from the statistical models to avoid multi-collinearity with the component attributes.

Given 6 attributes of 3 levels each, 729 (3⁶) scenarios were possible. Given that a survey with that many possible combinations would be beyond the capacity of any respondent to complete, a fractional factorial experimental design was developed in SAS²⁷. The final design consisted of 22 paired-choice tasks, which were divided into 2 blocks of 11 tasks each. Each block included a test of dominance or nonsatiation, in which one alternative was unambiguously better than the others in terms of health gain, and a test of consistency, in which respondents were presented with the same alternatives seen in an earlier task. Those tests were included for descriptive purposes, but were not used to exclude “irrational” respondents²⁸.

Data Collection

Two groups were recruited to participate in the elicitation. First, individuals associated with the pcodr review process, including members of the expert panel and the clinical and economic review committees, were invited in targeted e-mail messages to participate. Second, an age- and sex-representative sample of the Canadian general population was drawn from an online survey panel maintained by Research Now, a market research firm.

The elicitations were administered over the Internet. Respondents were asked to imagine themselves as a decision-maker responsible for allocating a fixed budget between two competing health care programs. They were told that both programs had the same cost and that the budget was not large enough to fund both of them. To provide a uniform context, respondents were told that the groups each had some form of cancer; however, specific diagnoses were not mentioned, and the alternatives were presented simply as program A and program B. Although labelled alternatives have the advantage of making hypothetical choice tasks more realistic and concrete, respondents can also use such labels to infer information that was not presented or intended as part of the task. At the extreme, respondents might ignore trade-offs between attributes and make their choices based on their perceptions of the labels alone²⁹.

Participants were randomized to either a discrete-choice experiment (dce) or a constant-sum paired comparison (cspc) questionnaire. The dce tasks asked respondents to identify the patient group that they would prefer to prioritize in each of a series of paired alternatives. In the cspc tasks, the alternatives were presented in the same manner as they were in the dce, but respondents were asked to by move a slider that allocated budget percentages between the two programs. Respondents could allocate 100% of the budget to program A or program B, or to some combination of the two, including a 50%–50% allocation. Tables i and ii show examples of the two tasks.

TABLE I.

Discrete choice experiment

Variable	Program A	Program B
Average age of the patients	70 years	40 years
Quality-of-life without/before treatment	9 out of 10	5 out of 10
Quality-of-life with/after treatment	9 out of 10 [no change]	9 out of 10 [4 levels higher]
Life expectancy without/before treatment	10 years	5 years
Gain in life expectancy with treatment	10 additional years	1 additional year
Number of patients that could be treated	5,000	2,500
Total quality-adjusted life years gained	45,000	7,250
□I would prefer to fund Program A      □I would prefer to fund Program B

TABLE II.

Constant-sum paired comparison experiment

Variable	Program A	Program B
Average age of the patients	70 years	40 years
Quality-of-life without/before treatment	9 out of 10	5 out of 10
Quality-of-life with/after treatment	9 out of 10 [no change]	9 out of 10 [4 levels higher]
Life expectancy without/before treatment	10 years	5 years
Gain in life expectancy with treatment	10 additional years	1 additional year
Number of patients that could be treated	3,600	700
Total quality-adjusted life years gained	32,400	2,030
Share of budget to Program A	72% |------------------------ < >----------------------------------------------------| 28%	Share of budget to Program B

Analysis

The two groups were compared in terms of the number of choices from among the 11 tasks seen by each respondent that prioritized the qaly-maximizing alternative. In each choice task, one alternative was always associated with a greater potential aggregate qaly gain. It was anticipated that pcodr decision-makers would be more familiar with the principles of qaly maximization and would therefore be more likely to make choices on that basis. The number of qaly-maximizing choices made by respondents in the two groups was compared by permutation t-test, and the proportion of respondents in each group who prioritized the qaly-maximizing alternative in most of their choices (≥6 of 11) was compared by 2-sample z-test of proportions.

To maximize the statistical power of the analysis, given the small number of pcodr members who could potentially participate, all responses to the cspc questionnaire were transformed to discrete choices on the basis of the alternative that was allocated the greater proportion of the budget. Those transformed responses were combined with the dce responses into a single dataset. Equal 50%–50% responses in the cspc questionnaire were excluded from the analysis, because no preference was expressed for either alternative.

Choice responses were modelled using a pooled multinomial logit, which presumes that the likelihood of an alternative being chosen is proportional to its relative attractiveness³⁰. Simply, the more attractive a given alternative is relative to the another alternative or alternatives in the choice task (that is, the greater its relative utility), the more likely it is that a respondent will choose it. From this choice model, the strength of the preferences for various attribute levels was calculated in terms of compensating variation (cv)³¹,

$${\text{CV}}_{x:\text{LYg}}=1/{\beta }_{\text{LYg}}\left[{\nu }^0-{\nu }^1\right],$$

where β_LYg is the coefficient of the life-years gained attribute (the constant marginal utility of an additional life-year gained), and v⁰ and v¹ are the aggregate utility of the scenario before and after a change in the level of a specific attribute from x⁰ to x¹, holding all other attributes constant at the baseline (middle) level. The cv represents the amount of some valued good (known as the “numeraire”—in this case, life-years gained) that a respondent would be willing to forego to prioritize a particular level of attribute x. A negative cv indicates that respondents would be willing to give up some potential life-year gains to prioritize the new level (x¹) over the baseline level (x⁰). Conversely, a positive cv indicates that compensation would be needed to accept greater priority for x¹ over x⁰—or equivalently, that respondents would be willing to forego some life-year gains to maintain priority for the baseline level of the attribute. The further from zero the cv for a particular level is found, the stronger the preference (cv < 0) or the aversion (cv > 0) for that level. Note that these life-years would accrue to the hypothetical patients in the choice tasks, not the respondents. Confidence intervals were calculated using the delta method³². Preferences for a particular attribute level were judged to be significantly different between the groups if the confidence intervals did not overlap. All statistical analyses were performed using the R software application (version 2.13.3: The R Foundation, Vienna, Austria).

RESULTS

In total, 21 pcodr members responded to the survey; 11 completed a dce questionnaire, and 10 completed a cspc. In the representative public sample, 656 respondents completed a dce questionnaire, and 662 completed a cspc. Table iii presents a demographic and socioeconomic comparison of the public sample and the overall Canadian population.

TABLE III.

Demographic and socioeconomic comparisons, Canadian population³³ and public sample surveyed

Age group	2012 Canadian population (%)		Survey sample [n (%)]
	Men	Women	Men	Women	No answer
18–24 Years	6	6	62 (5)	79 (6)	0 (0)
25–34 Years	9	9	109 (8)	125 (9)	0 (0)
35–44 Years	9	8	97 (7)	213 (16)	5 (0)
45–54 Years	10	10	113 (9)	117 (9)	3 (0)
55–64 Years	8	8	89 (7)	100 (8)	2 (0)
65–74 Years	5	5	68 (5)	31 (2)	0 (0)
≥75 Years	3	5	32 (2)	71 (5)	0 (0)
No answer	—	—	0 (0)	0 (0)	2 (0)
Subtotal	49	51	570 (43)	736 (56)	12 (1)
TOTAL				1318

Note: The proportion of the public respondents that had graduated college or university was 35%, identical to the 2006 Canadian census population (p=0.98)³⁴; however, the median family income category for the sample ($60,000–$64,999) was lower than the 2010 Canadian median family income ($76,950)³⁵. Income and education were not collected from pCODR respondents, but compared with the same attributes for the general population, both were likely to be somewhat higher.

Of 231 cspc allocations by the pcodr sample, 13 (5.6%) were excluded because of an equal 50%–50% allocation; of 14,267 allocations by the public sample, 823 (5.8%) were excluded. No pcodr respondents and only 8 public respondents (1.3%) equalized cspc allocations in every task, suggesting that most respondents did not hold strict egalitarian preferences.

Consistency with QALY Maximization

Figure 1 shows the distribution of total qaly maximizing choices for each respondent, by group.

FIGURE 1.

Distribution of quality-adjusted life-year (QALY)–maximizing choices per respondent, by group. The proportion of respondents in each group by the number of QALY-maximizing choices they made over the 11 tasks in each questionnaire.

On average, pcodr respondents were more likely than the public to choose the qaly-maximizing alternative (6.7 vs. 5.9 of 11 tasks), but the difference did not achieve statistical significance (p = 0.08). Most respondents in both groups chose the qaly-maximizing alternative in most of their choices: 62% of pcodr respondents and 58% of the public respondents chose the qaly-maximizing alternative in 6 or more of their 11 choices (p = 0.83). A notably larger proportion of pcodr respondents chose the qaly-maximizing alternative in all or almost all of their choices: 48% of pcodr respondents and 23% of the public respondents made 8 or more qaly-maximizing choices (p = 0.02).

Respondent Preferences by Attribute

Figure 2 illustrates compensating variations by respondent groups, attribute, and level. In the figure, preferences for the high and low levels of each attribute relative to the middle (baseline) level are shown. Points in the top half of each figure indicate more-preferred levels, and points in the lower half indicate less-preferred levels, relative to the middle level.

FIGURE 2.

Preferences of the public and of pan-Canadian Oncology Drug Review members, by attribute level. The graphs show compensating variations (CVs) associated with an upward and downward change in the level of each attribute, relative to a baseline state with all attributes at their middle level, by respondent group. The y axis is reversed to show more-preferred differences above zero and less-preferred differences below zero.

The cvs in Figure 2 show that public respondents were, for example, willing to forego a potential gain of 1.17 life-years to 40-year-olds so as to prioritize 10-year-olds. Conversely, they would require 70-year-olds to gain an additional 5.29 life-years to justify priority over 40-year-olds (alternatively stated, they would be willing to forego a gain of 5.29 life-years to a 70-year-old to prioritize 40-year-old individuals). The preference for 10-year-olds over 40-year-olds was not statistically significant, given that the 95% confidence interval includes 1; however, the aversion to 70-year-olds was significant. The pcodr respondents had a similarly significant aversion to elderly beneficiaries and were willing to forego a gain of 5.27 life-years to prioritize 40-year-olds over 70-year-olds.

Overall, the preferences of the pcodr and public respondents were observed to closely correspond, with no statistically significant differences for any attribute levels. Both groups had a significant aversion to prioritizing older patients, patients in a good initial health state or those who would finish treatment in a poor health state, and smaller patient groups. Conversely, a significant preference for greater priority was evident for patients in a more severe initial health state and for larger patient groups. No significant preferences for untreated life expectancy in either group was observed.

DISCUSSION AND CONCLUSIONS

The present work addresses the issue of whether the Canadian public and pcodr committee members acting on behalf of the public are strict health (qaly) maximizers or whether they appear willing to forego some degree of efficiency so as to prioritize specific patient characteristics. That question is important in light of pcodr’s commitment to an implicit decision-making framework, which gives individual committee members considerable latitude in assessing the value of cancer drugs in various patient populations. Although that latitude allows decision-makers to exercise context-specific judgments, it could also lead to recommendations at odds with the values and preferences of the broader society they represent.

The question of which decision criteria have the greatest impact in predicting pcodr decisions is not addressed here. Such studies have previously been conducted in the United Kingdom and Australia^18,19, and a Canadian study using publicly-available pcodr data is currently ongoing. But whereas studies of the impact of various decision criteria focus on the characteristics of the drugs, including clinical benefit, economic factors, patient preferences, and adoption feasibility¹, the present study explores differences in the perceived value of health gains to various patients. In this sense, drugs with the same clinical and economic characteristics might be valued differently because of differences in the relative value of health gains to the patients they treat. For example, consideration of a patient’s culpability for their illness was specifically excluded from the present analysis, but it is conceivable that society or pcodr decision-makers (or both) might view the value of health gains differently depending on whether they accrue to a heavy smoker with lung cancer or to a child with leukemia.

It is important to acknowledge that the statistical power of this analysis is limited by the very small sample of decision-makers. That limitation was unavoidable, because the centralization of provincial oncology drug review processes within pcodr means that the potential pool of “decision-makers” in Canada is relatively small. The number of responses was further reduced by the exclusion of 50%–50% allocations in converting cspc responses to discrete choices. That approach would have had the effect of excluding more moderate preferences and emphasizing the extremes. However, the exclusions did not affect the count of qaly-maximizing choices, and the relatively small and similar proportions of exclusions among the pcodr and public respondents suggests that they should not have had a substantive or differential impact on the observed preferences of the two groups.

The proportion of all pcodr members represented by the respondents to the study’s elicitation is also not clear. The pcodr Expert Review Committee currently has 16 members³⁶, and the clinical and economic review panels take on additional members on a per-submission basis. Given the relatively small number of pcodr members at any one time, it is most likely that the study’s 21 respondents include some mix of current and former members. That group is arguably a meaningful proportion of pcodr members, although it is not possible to claim that they are necessarily a representative sample.

Within the foregoing limitations, the results suggest that the preferences of the sampled pcodr respondents are not substantively different from those of a representative sample of the Canadian general public. The pcodr respondents appeared, on average, only slightly more likely than the general public to make their choices on the basis of qaly maximization, although a significantly greater proportion of the pcodr respondents appeared to be relatively strict qaly-maximizers. Both groups were willing to forego some potential life year-gains to give greater priority to particular patient groups, suggesting a willingness to sacrifice some degree of efficiency for greater fairness and equity. That finding is reassuring, because it suggests that the preferences of pcodr decision-makers are consistent with those of the larger Canadian public they represent.

Critically, however, the study results also imply that health gains to some patients might be valued more or less highly than the same gains to other patients. The pcodr decision framework does not define the potential equity and fairness considerations that are legitimate in funding recommendations or how much weight should be given to such factors. For example, the results implied that both groups would be willing to pay more for health gains accruing to younger patients than for the same gains accruing to patients 70 years of age. The acceptability and limits of such differential valuations are not addressed in the pcodr guidelines. More explicit guidance could improve the consistency and transparency of pcodr recommendations, and in turn, public trust in the pcodr decision-making process^6,37. Such transparency could also stimulate constructive debate about societal values pertaining to the allocation of public health care resources.

CONFLICT OF INTEREST DISCLOSURES

I have read and understood Current Oncology’s policy on disclosing conflicts of interest, and I declare that I have none.