Engaging the public in priority‐setting for health technology assessment: findings from a citizens’ jury

Abstract

Objectives  To assess the feasibility of using a citizens’ jury to elicit public values on health technologies and to develop criteria for setting priorities for health technology assessment (HTA).

Methods  Sixteen individuals were selected from 1600 randomly sampled residents of the Capital Health Region in Alberta, Canada. They participated in a 2½ day jury which comprised presentations by ‘expert witnesses’, who represented innovators, patients, health‐care policy‐makers and clinicians, as well as a series of small and large group priority‐setting exercises based on actual examples of technologies that had recently been considered for assessment by local and national HTA bodies. The session was audio‐taped, and transcripts were independently reviewed by two researchers using content analytical techniques in order to ensure that no important concepts expressed by individual jurors were missed during group development of the final list of priority‐setting criteria. Jurors evaluated the process by completing self‐administered, semi‐structured questionnaires at the end of the session. Responses were analysed using qualitative methods.

Results  The jury identified 13 criteria, which they subsequently ranked in order of importance. The top two criteria included ‘potential to benefit a number of people’ and ‘extends life with quality’. Based on feedback from questionnaires, jurors valued the opportunity to become engaged in such a process, and expressed interest in participating in future juries.

Conclusions  Citizens’ juries offer a feasible approach to involving the public in priority‐setting for HTA. Furthermore, technologies that may benefit a number of people and improve quality of life appear to be of greatest importance to the public.

Introduction

In Canada, the call for greater public involvement and accountability in health‐care decision making has never been clearer. Over the past 7 years, all five commissions appointed by Canada’s federal and provincial governments to examine the state of public health care and offer recommendations for strengthening it have reached similar conclusions. ¹ , ² , ³ , ⁴ , ⁵ Public confidence in the Canadian health‐care system is eroding, and citizens are increasingly questioning decisions that involve funding some services but not others. Furthermore, there is heightened interest in and scrutiny over how and by whom such decisions are made. ⁶ , ⁷ , ⁸ Consequently, federal and provincial governments face pressures not only to demonstrate transparency and legitimacy in decision making, but also to formalize a role for the public in the process. ⁹

Now more than ever, managing the rapidly growing availability of and demand for new, often high‐cost health technologies has become a key priority for policy‐makers across Canada and abroad. ¹⁰ Such technologies continue to represent a major source of health spending, and their rate of development far exceeds that at which they can be introduced into the public health‐care system. ¹¹ , ¹² Therefore, as with all health services, decisions around which technologies to include in the publicly funded basket must be made. ¹³ Health technology assessment (HTA) is a tool to aid such decisions‐to help policy‐makers determine whether or not a technology offers ‘good value for money’. ¹⁴ , ¹⁵ Since its inception over 30 years ago, HTA has employed explicit analytical frameworks, adapted from evidence‐based medicine and health economics, to provide information on the costs, effectiveness and broader impact of health technologies. However, the field is beginning to evolve with a changing health technology environment. ¹⁶ It has become well recognized that choices between competing technologies, all of which may offer some health benefit, require subjective judgments that are often value‐laden (e.g. How important is the clinical benefit to patients and society as a whole? Does the technology do the ‘right’ job?, Are its costs fair?). ¹⁷ , ¹⁸ , ¹⁹ Therefore, to ensure that HTA remains an effective tool for informing health technology policy, the field is dedicating significant efforts to finding ways of incorporating the values of citizens into its existing processes. ²⁰

Briefly, the HTA process comprises three main phases: (1) Selection of technologies to be assessed, (2) Performance of the assessment and (3) Communication and implementation of the findings. ²¹ The first phase involves setting priorities for HTA. Canada’s HTA‐producing organizations, including the Canadian Agency for Drugs and Technologies in Health, and its counterparts around the world, receive more requests than they can complete with the resources available to them. Thus, these organizations, together with decision‐makers, must determine which technologies should receive priority. Once a technology is selected, its ‘value’, from a health outcomes and economic perspective, can then be assessed (i.e. Phase 2). A role for the public in one or both of these first two phases has been proposed. Therefore, those involved in HTA are asking the question, ‘At what level is the public willing to take part in developing health technology policy?’ ²²

In Canada and abroad, there is a growing body of literature suggesting that, while the public is prepared to be engaged in health‐care policy development, and more specifically priority‐setting, it would prefer not to be involved in making explicit decisions about which technologies to provide (e.g. choosing between a new cancer treatment and home haemodialysis). ⁶ , ⁷ , ⁸ , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ Rather, it would like to participate in formulating criteria to guide funding decisions. ³¹ , ³² , ³³ , ³⁴ Putting these findings into the HTA context, efforts to determine a role for the public might most appropriately begin with the development of ways to engage citizens in the establishment of criteria for setting HTA priorities.

This paper reports on the first attempts to accomplish this in Canada. We describe our use of a citizens’ jury to formulate HTA priority‐setting criteria for Canada’s HTA‐producing organizations. Citizens’ juries, as with legal juries, are based on the idea that ‘once a small sample of the population has heard the evidence, its subsequent deliberations can fairly represent the conscience and intelligence of the general public’. ³⁵ , ³⁶ , ³⁷ They consist of 12–16 individuals recruited to be broadly representative of their community. Typically charged with addressing complex questions, jurors meet over a 2‐ to 4‐day period during which they hear from a variety of expert ‘witnesses’, who present a range of perspectives on a particular issue, engage in deliberations among themselves, and, ultimately, come up with a ‘common ground’ set of findings. Therefore, unlike opinion polls, surveys and focus groups, citizens juries offer a way of seeking informed public views using a democratic, deliberative process. The approach has been rigorously evaluated for fairness and competence through external reviews of deliberations from previously conducted juries. ³⁵ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ The results demonstrated that jurors had equal opportunity to participate in the process and express their views. They became actively engaged in debates, were able to recall small details about information presented to them over the jury’s time period and developed a sense of community, shifting their views from more self‐interested ones to socialistic ones.

Objectives

The purpose of this project was to pilot the citizens’ jury as an approach to engaging the public in priority‐setting for HTA. Specifically, it aimed at:

• 1assessing the willingness of citizens to participate in setting priorities for HTA;
• 2determining the feasibility of conducting a citizens’ jury to elicit the views of the public on priorities for HTA; and
• 3developing a set of criteria to guide priority‐setting for HTA.

Methods

Assembly of jury

Sixteen residents of the Capital Health Region (which services a population of approximately 1.6 million and includes both urban and outlying rural communities) were recruited to comprise a demographically representative jury through the following process.

Assembly of jury pool

Letters of invitation to participate in a telephone screening survey were mailed to 1600 randomly selected residents of the Capital Health Region. The sample size was calculated based upon response rates reported in previously published citizens’ juries, which ranged from 2% to 25%. ³⁹ Names and mailing addresses were extracted from a commercial database of registered telephone numbers using simple random sampling techniques (random numbers table). The letter of invitation included a description of the screening survey, as well as the main project (i.e. the jury session). To minimize volunteer bias, it also stated that individuals selected to participate in the jury would each receive a stipend of $375 Cdn. Consent forms, pertaining to both the screening survey and jury session, accompanied each letter. Respondents were asked to complete and return these forms within 3 weeks of the postage date.

Selection of jurors

Fifteen‐minute telephone screening surveys were conducted with all consenting respondents who had confirmed their willingness and availability to participate in the jury session during the days specified in the invitation letter. Each survey was administered by one of two experienced researchers using a pre‐tested standard interview script. As the aim of the jury was to elicit the views of ‘ordinary citizens’ (i.e. individuals with no particular axe to grind and whose voices might not otherwise be heard), interview questions were designed to gather information on not only characteristics such as age, gender, ethnicity, education, income, employment status and family structure (including number and age of dependents), but also potential affiliations with special interest/patient advocacy groups and employment in a health‐care delivery organization or government as a health‐care professional (exclusion/ineligibility criteria). To select 16 jurors with a collective demographic and socioeconomic profile comparable to that of the Capital Health Region, a combination of purposive and random sampling techniques was employed. Eligible respondents (i.e. non‐health‐care professionals and individuals not involved in any special interest/advocacy movements) were first categorized into groups based upon trends in the above characteristics across the Capital Health Region according to census data reported by Statistics Canada. From within each group, potential jurors were then purposefully selected to roughly match the distribution of the data. Random sampling (random numbers table) was used in cases where two or more respondents exhibited the same characteristics.

Conduct of the jury session

Held in Edmonton, Alberta, over 2½ days, the jury session included presentations from expert witnesses, who described how decisions for new health technologies were made at the provincial and regional health authority levels, scenario‐based priority‐setting exercises and opportunities for deliberations among jurors, both in small and large groups. Assistance facilitating the session was provided by two moderators who had been involved in running the Citizens Council for the National Institute for Health and Clinical Excellence (NICE) in the United Kingdom. The Citizens Council comprises 15 men and 15 women ‘from all walks of life’ who meet twice a year to express their views on issues related to the development of NICE guidance on care that citizens covered by the National Health Service can expect to receive. ³⁷ Council members are appointed for 1–3 years.

The length of the jury session and the programme outlined below were based upon a review of published literature, which reported on lessons learned from previous citizens’ juries and personal communication with authors.

Day 1(half day)

The first day commenced with welcomes and introductions by the investigative team, followed by a series of ‘ice‐breaker’ exercises intended to give jurors an opportunity to develop a comfort level working together. It ended with presentations that: (1) discussed the need to make tough but fair decisions regarding which health technologies to fund publicly, (2) defined HTA and its role in informing such decisions (i.e. what HTA is and why it is done), (3) introduced the main HTA‐producing organizations in Canada and (4) outlined the steps involved in a citizens jury (i.e. what jurors could expect over the next 2 days).

Day 2 (full day)

The second day began with presentations from expert witnesses. Senior administrators and policy‐makers from the Capital Health Authority and the Alberta Ministry of Health and Wellness described how priority‐setting decisions for new technologies are made at the regional and provincial levels in Alberta. At the end of each presentation, jurors had an opportunity to ‘interrogate’ witnesses during a question and answer period. They then engaged in their first scenario‐based priority‐setting exercise. The purpose of this exercise was to identify criteria, in no particular order, that might be used to guide priority‐setting for HTA. To accomplish this, jurors were presented with 13 mini technology scenarios (Table 1), taken from actual HTA requests submitted by regional and provincial policy‐makers within the past year. Each scenario comprised one paragraph describing the technology and indications for its use, the number of individuals anticipated to benefit, and, where possible, its estimated unit or per case cost. The level of information provided reflected that typically received by those involved in setting HTA priorities for the province. ⁴⁶ After independently rating the importance of each technology on a scale of 1–5, jurors met in small groups to share and explain their choices and compile a list of criteria based on their rationales or reasons. They then reconvened to deliberate over and agree upon which criteria to include in an initial draft set.

Table 1.

 Summary of technologies used for the mini technology scenarios

Technology	Indication
Laparoscopic adjustable gastric banding	Morbid obesity
Birmingham hip replacement	Arthritis of the hip
Fetal fibronectin testing	Test for preterm delivery in patients with symptoms of preterm labour
Uracyst® (Stellar Pharmaceuticals, London, ON, Canada)	Interstitial cystitis (painful bladder syndrome)
Newborn hearing screening	Detection of hearing problems in babies
Accomplia® (Sanofi‐Aventis, Paris, France) (active ingredient: Rimonabant)	Smoking cessation and weight loss
Combretastatin [vascular targeting agent (attacks blood vessels in tumours)]	Colon cancer and thyroid cancer Possibly other solid tumour cancers
MammoSite® (Proxima Therapeutics, Alpharetta, GA, USA) targeted radiation therapy	Internal radiation of cavity Remaining breast tissue following a lumpectomy
Vagus nerve stimulation	Severe depression
Implantable cardioverter defibrillator	Prevention of sudden death from cardiac arrest due to ventricular fibrillation
Positron emission tomography	Detection of changes in the cellular function of tissues in the body
Ceredase® (Genzyme, Cambridge, MA, USA)	Gaucher’s disease (a genetic condition that causes fatty deposits to accumulate in different parts of the body, including the liver, spleen and bone marrow)
Titanium rib expandable prosthesis	Expansion of the rib cage and straightening of the spine in growing children with severe chest wall, rib or spine abnormalities

Day 3 (full day)

The third day was dedicated to ‘testing’ out and subsequently refining the draft set of criteria to create a ranked list. To accomplish this, jurors engaged in a second scenario‐based exercise comprising two in‐depth case studies derived from local technology issues. They included: (1) the use of drug‐eluting stents over bare metal stents for the treatment of coronary artery disease, a common condition characterized by the hardening and narrowing of arteries that supply blood to the heart muscle, and (2) the use of sildenafil (Viagra®; Pfizer Inc., New York, NY, USA) for the treatment of primary pulmonary hypertension, a rare condition that results in the progressive narrowing of the blood vessels of the lungs, causing high blood pressure in such vessels, which eventually leads to heart failure and death. These two technologies were selected because they facilitated ‘trade‐off’ discussions around the importance of different aspects of a technology and the condition it treats, including disease prevalence and seriousness, availability of existing treatments, and potential clinical and economic benefit of the technology (e.g. improves quality of life and/or survival, offers cost savings). For each technology, jurors heard from and asked questions to a patient with the condition, a health‐care provider who treats the condition, a policy‐maker involved in determining the reimbursement status of new technologies and the manufacturer of the technology. To familiarize expert witnesses with the jury process and ensure presentations captured a broad range of perspectives, mock sessions, overseen by the two moderators from NICE, were held prior to the actual session. Following presentations from expert witnesses, each juror was asked to decide which of the two technologies should receive higher priority for assessment and explain his/her decision. Jurors then split into four small groups to: (1) discuss how they had applied the initial draft set of criteria to their own decision‐making process, (2) deliberate over any necessary modifications to the criteria and (3) rate each revised criterion as ‘extremely important’, ‘quite important’, ‘important’ or ‘not important’. The jury then reconvened to review the findings from each group and establish, through consensus, a final ranked set of criteria. This was achieved by resolving discrepancies in terminology across groups and weighting each criterion, multiplying the frequency with which it appeared on the groups’ lists by the magnitude of the importance ‘score’ it received [3 for ‘extremely important’, 2 for ‘quite important’, 1 for ‘important’, and −2 for ‘not important’ (suggesting that it should not be used)]. Criteria were then ordered according to the sum of their weighted scores [highest sum (most important) to lowest sum (least important)] to generate a ranked list, which was subsequently finalized by the jury (Table 2).

Table 2.

 Number of times each criterion appeared in criteria lists compiled by jury ‘break out’ groups and the relative importance it received

Criterion	Importance (weighting)								Sum
	Extremely important (3)		Quite important (2)		Important (1)		Not important (−2)
	n*	Weighted score	n*	Weighted score	n*	Weighted score	n*	Weighted score
Lack of an alternative	1	3	2	4	0	0	0	0	7
Completeness of data on adverse events	0	0	1	2		0	0	0	2
Cost	0	0	0	0	0	0	1	−2	−2
Potential to extend life with quality	2	6	2	4	0	0	0	0	10
Potential to extend life	1	3	0	0	0	0	0	0	3
Potential to detect a condition which, if treated early, averts future costs	1	3	1	2	1	1	0	0	6
Potential clinical benefit over existing treatments	1	3	2	4	1	1	0	0	8
Potential to improve quality of life	2	6	1	2	1	1	0	0	9
Potential for additional applications	1	3	1	2	1	1	1	−2	4
Potential to benefit to a number of people	3	9	1	2	0	0	0	0	11

*Number of jury ‘break‐out’ groups (four jurors per group).

Analysis of jury findings

Audiotapes of the entire jury session were transcribed and analysed by two independent reviewers using content analytical techniques, which involved manually identifying key chunks of information and categorizing them into emerging themes and sub‐themes, and constant comparison techniques to ensure that all the jurors’ views had been appropriately captured in the final set of criteria.

Evaluation of the jury session

At the end of the last day, each juror completed a self‐administered, semi‐structured, feedback questionnaire designed to assess the ‘trustworthiness’ of the jury session results. To ensure that jurors had sufficient time to respond, they were given the option of taking the questionnaires home and returning them using the provided self‐addressed, postage‐paid envelopes. No juror identification information was collected on questionnaires to increase the likelihood of obtaining open and honest responses. Survey questions included: ‘How helpful did you find the presentations, full jury deliberations, and small group discussions in clarifying your views?’ (very helpful, helpful, neutral, unhelpful, very unhelpful); ‘What did you think about the amount of time allowed for the jury session?’ (far too much, a bit too much, adequate, too little, far too little); ‘What did you think about the amount of time allowed for jury deliberation?’ (far too much, a bit too much, adequate, too little, far too little); ‘What did you think about the main question for the jury session?’ (far too broad, a bit too broad, about right, too narrow, far too narrow); ‘What did you think about the number of witnesses?’ (too many, about right, too few); ‘How did you feel about the balance of information presented?’ (a good balance, a poor balance, vital perspective missing, overlap); ‘How much clearer do you feel about the issues discussed during the jury session than you did before?’ (much clearer, clearer, somewhat clearer, a bit clearer, no clearer); ‘Overall, to what extent did you feel: (a) welcomed, (b) informed, (c) able to contribute and (d) challenged?’ (a lot, enough, not enough, not at all); ‘In general, how could the jury session be improved?’; and ‘What made you accept the offer to take part in the jury?” Responses to Likert‐type questions were analysed quantitatively while those to open‐ended questions were analysed qualitatively using content analytical techniques.

Results

Of the 1600 individuals to whom letters of invitation were sent, 476 replied within 2 weeks of the initial mail‐out. Twelve additional responses were received after the deadline but prior to completion of the screening survey. Among respondents, 420 consented to the screening survey and indicated that they would be available to participate during the days scheduled for the jury, while 68 consented to the survey but stated that they would not be able to participate unless the dates for the jury were changed. Of the 1112 non‐respondents, 982 were unreachable, as letters returned by the post office were marked with ‘no known address’ or ‘change of address with no forwarding address’ stamps. This may, in part, be explained by the fact that names and addresses were obtained from a commercial database, which relied upon billing information corresponding to registered telephone numbers. While it was not possible to verify the accuracy of this information prior to the study, analyses comparing the geographical distribution of respondents with that of non‐respondents suggested that the two groups were similar. All 420 consenting and available respondents completed the telephone screening survey.

Profile of jury

Sociodemographic characteristics of the 16 citizens selected to participate in the jury are presented in Table 3. As in the Capital Health Region, half were men and half were women, ranging in age from 18 to 75 years. They included two stay‐at‐home moms, a university student, a retired farmer, a retired teacher, a realtor, a social worker, an architect, a computer programmer, a legal assistant, an electrician, a construction worker, a plumber, a security guard, a long haul truck driver and an oil field driller. They represented various education and household income levels, also distributed similar to that of the Capital Health Region, and a mix of ethnic backgrounds and family structures (e.g. single, married with and without children, divorced with and without children, widowed, etc.).

Table 3.

 Sociodemographic profile of the jury

Characteristic	Number of jurors
Gender
Male	8
Female	8
Age
18–24	2
25–34	3
45–54	4
55–64	4
65–74	2
>74	1
Ethnicity
African	1
Asian	1
Caucasian	12
First Nations (Aboriginal)	1
Spanish	1
Employment status
Employed	12
Unemployed	4
Education (highest level)
<High school	2
High school	4
Post‐secondary training	8
Postgraduate training	2
Annual income ($ Cdn)
<20 000	2
20 000–40 000	3
40 000–60 000	5
60 000–80 000	4
>80 000	2
Dependents
No	4
Yes	12
Total	16

Final set of criteria for setting HTA priorities

At the end of the 2½ days, the jury presented a unanimously agreed‐upon final set of criteria for setting HTA priorities, which is outlined in Table 4. Of greatest importance was the ‘potential to benefit a number of people’, reflecting the jury’s view that technologies for highly prevalent conditions within a population should be assessed over those for less prevalent ones. Both the second and third highest ranked criteria, ‘potential to extend life with quality’ and ‘potential to improve quality of life’, respectively, demonstrated the importance of ‘quality of life’ to jurors. Notably, the ‘potential to extend life’, alone, appeared near the bottom of the list (eighth). Furthermore, jurors felt it necessary to distinguish improvement in quality of life from other outcomes, creating a separate, fourth‐ranked criterion, ‘potential clinical benefit over existing treatments’, for the latter. This criterion was followed by ‘lack of an alternative’, suggesting that the availability of existing treatments should be taken into account. The sixth and seventh criteria, ‘potential to detect a condition which, if treated early, averts future costs’ and ‘potential for additional applications’, highlighted the concept of investment in health. The jury valued technologies it considered to be good long‐term investments, such as those for screening and prevention or treating conditions other than the ones for which they were initially indicated (such as some chemotherapy agents). With respect to the ninth and final criterion, ‘completeness of data on adverse events’, the jury felt that technologies with more established safety profiles should receive priority over those with less established ones, as the latter would be too difficult to assess (i.e. the information needed to properly evaluate the technology may not yet be available). Lastly, the jury identified one criterion that, in its view, should not be considered during priority‐setting for HTA, ‘cost’. It indicated that the per‐patient cost of a technology, alone, provides little insight into what the economic impact of its introduction might be, or whether it might offer value for money.

Table 4.

 Final set of criteria for setting priorities for HTA in ranked order (highest to lowest)

Criteria to be used
1. Potential to benefit a number of people
2. Potential to extend life with quality
3. Potential to improve quality of life
4. Potential clinical benefit over existing treatment(s)
5. Lack of an alternative
6. Potential to detect a condition which, if treated early, averts costs in the future
7. Potential for additional applications
8. Potential to extend life
9. Completeness of data on adverse events
Criteria not to be used
1. Cost

HTA, health technology assessment.

Feedback from jurors on the jury session

Sixteen completed questionnaires were returned, the main findings of which are presented in Table 5. All 16 jurors indicated that the scope of the question they had been asked to address (i.e. what criteria should be used for setting priorities for HTA?), the time allowed for deliberations and the full jury session, and the number of witnesses from whom they heard were ‘adequate’ or ‘about right’. Furthermore, all of them found the presentations, jury deliberations and small group discussions helpful or very helpful. Fifteen of the jurors stated that the information presented was balanced, while one thought there was overlap, explaining that arguments offered by the witness who represented the decision‐makers’ perspective were similar for both case studies. Nevertheless, all 16 felt welcomed, informed, able to contribute and challenged, and walked away from the experience feeling clearer or much clearer about the issues addressed than they had prior to the session. Although jurors were asked to offer ways in which the jury session could be improved, no suggestions were received. Based on responses collected to the question ‘What made you accept the invitation to take part in the jury?’ jurors participated out of interest in the topic (14) or for the money (2). Lastly, jurors had an opportunity to provide any additional feedback. The following comments were made:

Table 5.

 Summary of jurors’ responses to feedback questionnaire

Questions	Responses (total n = 16)
How helpful did you find the presentations, full jury deliberations, and small group discussions in clarifying your views?
	Very helpful	Helpful	Neutral	Unhelpful	Very unhelpful
Presentations	16
Full jury deliberations	15	1
Small group discussions	16
What did you think about the amount of time allowed for the jury session and jury deliberations?
	Far too much	A bit too much	Adequate	Too little	Far too little
Jury session			16
Jury deliberations			16
What did you think about the main question for the jury?
	Far too broad	A bit too broad	About right	Too narrow	Far too narrow
Jury question			16
How much clearer do you feel about the issues discussed during the jury session than you did before?
	Much clearer	Clearer	Somewhat clearer	A bit clearer	No clearer
Clarity	15	1
How did you feel about the balance of information presented?
	A good balance	A poor balance	Vital perspective missing	Overlap
Balance	15			1
What did you think about the number of witnesses?
	Too many	About right	Too few
Witnesses		16
Overall, to what extent did you feel welcomed, informed, able to contribute, and challenged?
	A lot	Enough	Not enough	Not at all
Welcomed	16
Informed	15	1
Able to contribute	15	1
Challenged	16

This was an extraordinary opportunity to participate in an important approach to allow the general public to become better informed or and to be actively involved in decision‐making processes.

A great way to get involved and have some input as a member of the general public.

I could hardly sleep from the information I got and continually think about. Thanks for the opportunity.

The exercises were excellent because they were presented in a real way.

Discussion and conclusions

To our knowledge, this project represents the first attempt to involve the public in setting priorities for HTA in Canada and to apply the citizens jury technique to the Canadian health technology policy context. Therefore, we compared the jury’s findings with existing international criteria established by health‐care decision‐makers and HTA producers in different jurisdictions around the world. ⁴⁷ , ⁴⁸ Similar to the jury’s findings, such criteria included elements related to the number of people expected to benefit, the anticipated effectiveness of the technology over existing treatments and its potential diffusion (including its application to other conditions). However, in contrast to the jury’s findings, none distinguished between types of health benefit, such as increased length of life or improved quality of life. Moreover, they all contained at least one criterion related to the cost of the technology at the patient level, population level or both, suggesting significant differences either between the values of the public and those of decision‐makers and HTA producers or between cultures and health‐care systems. None of the existing criteria had originated from Canada, and according to recently published work in this area, variations in such criteria can be attributed to health‐care system and cultural differences across countries. ⁴⁹ While every effort was made to assemble a jury whose views represented those of the broader public, it was not possible to confirm the reliability or generalizability of the jury’s findings within the project’s parameters. This would have required conducting more than one jury with the same jurors and conducting several juries with different jurors respectively.

As demonstrated in the results of the feedback questionnaire, jurors viewed their experience as a positive one. Expert witnesses, who were informally surveyed after the jury, also responded favourably, expressing enthusiasm and a willingness to participate in future juries. At the provincial policy‐making level, plans to fund citizens juries on other health technology‐related issues are now underway, with proposed resource requirements at levels similar to that employed in the present study.

Based on the findings from this study, it seems reasonable to conclude that citizens juries offer a feasible approach to involving the public in setting local HTA priorities.