A descriptive review on methods to prioritize outcomes in a health care context

Abstract

Background

Evidence synthesis has seen major methodological advances in reducing uncertainty and estimating the sizes of the effects. Much less is known about how to assess the relative value of different outcomes.

Objective

To identify studies that assessed preferences for outcomes in health conditions.

Methods

Search strategy: we searched MEDLINE, EMBASE, PsycINFO and the Cochrane Library in February 2014. Inclusion criteria: eligible studies investigated preferences of patients, family members, the general population or healthcare professionals for health outcomes. The intention of this review was to include studies which focus on theoretical alternatives; studies which assessed preferences for distinct treatments were excluded. Data extraction: study characteristics as study objective, health condition, participants, elicitation method, and outcomes assessed in the study were extracted.

Main results

One hundred and twenty‐four studies were identified and categorized into four groups: (1) multi criteria decision analysis (MCDA) (n = 71), (2) rating or ranking (n = 25), (3) utility eliciting (n = 5) and (4) studies comparing different methods (n = 23). The number of outcomes assessed by method group varied. The comparison of different methods or subgroups within one study often resulted in different hierarchies of outcomes.

Conclusions

A dominant method most suitable for application in evidence syntheses was not identified. As preferences of patients differ from those of other stakeholders (especially medical professionals), the choice of the group to be questioned is consequential. Further research needs to focus on validity and applicability of the identified methods.

Background

Whenever we want to know whether a health intervention, that is a pharmaceutical, a medical product or a procedure, is valuable or not, we have to assess it against specific outcomes. A drug can effectively lower blood pressure but it can fail to prevent strokes. Another drug might prevent strokes but can cause severe fatigue as an adverse effect. Depending on the choice of these outcomes, we will categorize the health intervention as valuable or not. Most often, we are not dealing with just a single outcome but with several outcomes at once. This leads to the question about the relative importance of these outcomes.

In clinical practice, primary research or evidence synthesis, doctors and researchers mostly adhere to an intuitive/implicit approach to choose and balance outcome parameters. Given the relevance of such a choice this is rather remarkable, especially as the concordance between patients and providers is not always high.1 In trials, the outcome parameter is traditionally decided on by sponsors, manufacturers, providers or researchers, whereas only rarely were the patients involved.2 This fact is doubly surprising: The patients are the ones who are affected, and, at the same time, they are valuable experts in assessing the importance of the effects and adverse effects of the health interventions they are treated with.3, 4 The importance of patient involvement in research is widely accepted now, but not always implemented yet.5, 6, 7

As the results of evidence synthesis are often used for reimbursement decisions, there is a rising debate among academics and HTA‐agencies how to better incorporate patients' perspectives.8 So far, involving patients into the HTA process, if reported at all, has been limited predominantly to involving organized groups or lay representatives. The involvement consists frequently of consultation or communication, only seldom on participatory approaches.9 A systematic and empirical method to elicit preferences was rarely applied.10, 11

A systematic review conducted by Ryan et al.12 in 2001 aimed to identify techniques to assess public views on the provision of healthcare. As the focus of the review by Ryan et al. was on the provision of healthcare rather than on the prioritisation of specific outcomes in one specific health condition, different techniques might be applicable in the context of evidence synthesis.

With this descriptive review, we aimed to identify studies which assessed preferences of consumers of health care (patients, family members, caregivers or the general population) and those delivering it for specific outcomes in health conditions. We were interested in which methods were used and how often and in which health conditions, how many outcomes were included in the preference elicitation tasks and what strengths and weaknesses the methods have. If different methods were investigated in the same study, we were interested in how far the results were similar or if they diverged.

Methods

Eligibility criteria

We included studies evaluating preferences for outcomes in specific health conditions without restriction on methods used. Outcomes were defined as health condition related aspects, ranging from clinical outcomes such as mortality, morbidity, quality of life, to various other aspects regarding the health condition and its treatment as, for example, driving time to treatment, dosage flexibility, mode of administration or impact on everyday life. The preferences of patients, their families, caregivers or the general population were considered; whether this was the appropriate participant group for the health condition studied (e.g. the general population is appropriate for prevention questions). Preferences of healthcare professionals were also considered, but only as additional information. We restricted the publication date to year 2000 onwards, to assess which methods were currently used. The methods used in studies with a publication date before 2000 might have been revised or might not be applicable anymore.

We excluded studies, which only described the methodological aspects of preference elicitation, but did not perform the task or did not report in sufficient detail on it. Furthermore, editorials, letters without empirical findings or conference abstracts were excluded. Studies which elicited preferences with different objectives (e.g. prioritization of research) or studies which conducted a preference elicitation but did not report the results for the separate outcomes but only an overall assessment (e.g. utility score) were excluded. Studies which elicited preferences for health states rather than outcomes were also excluded. The intention of this review was to include studies which focus on theoretical alternatives; studies which assessed preferences for distinct treatments were excluded.

If more than one publication on the same study was identified, only the publication providing the most complete data was included.

Search strategy and study selection

We searched MEDLINE, EMBASE, PsycINFO and the Cochrane Library databases until February 2014 for the relevant literature. The full MEDLINE search strategy is presented in Table 1 and was adapted for the other databases.

Table 1.

Search strategy MEDLINE (Ovid 1950 to February 2014)

No.	Searches	Results
1	exp Patient Satisfaction/	59 022
2	exp Consumer Participation/	30 960
3	exp Decision‐Making/	116 830
4	(patient* preference* or patient* rating* or patient* weighting or patient* participation or personal preference*).kw,ti,ab.	8948
5	((patient* preference* or patient* rating* or preference*) adj3 (measuring or incorporating or involving or integrating or assessment or assessing)).ti,ab.	876
6	(patient* priorit* or patient* perspective*).ti,ab.	4159
7	1 or 2 or 3 or 4 or 5 or 6	205 435
8	(preference* elicit* method* or preference* assess* method* or (design adj3 measuring adj3 preference*) or Preference‐based approach* or preference‐based method* or patient* weighting of importance or (patient rating adj5 preference*)).ti,ab.	58
9	((patient* participation adj5 decision‐making) or (elicit* adj5 preference*) or (participatory adj5 decision‐making)).ti,ab.	1035
10	((preference adj5 (evaluat* or determin* or assess*)) or ((patient* adj3 preference assessment) or patient* preference* survey).ti,ab.	3302
11	((patient* preference* or patient* rating*) adj5 (method* or measuring or incorporating or involving or integrating or assess*)).ti,ab.	600
12	(ranking* or rating* or visual analogue scale* or likert scale*).ti,ab.	116 080
13	(choice base* technique* or choice base* method* or conjoint analys* or analytic hierarchy proces* or standard gamble or time trade‐off or willingness‐to‐pay or allocation point*).ti,ab.	3907
14	8 or 9 or 10 or 11 or 12 or 13	123 488
15	exp Treatment Outcome/or (treatment* outcome* or outcome*).ti,ab.	1 226 421
16	((test* or treatment* or therapy* or medication*) adj5 attribute*).ti,ab.	3497
17	(attribute* profile* or attribute* level*).ti,ab.	82
18	medication* characteristic*.ti,ab.	40
19	(symptom* or consequence*).ti,ab.	915 092
20	important factor*.ti,ab.	42 556
21	(importance adj3 aspects adj3 therapy).ti,ab.	3
22	15 or 16 or 17 or 18 or 19 or 20 or 21	2 019 483
23	7 and 14 and 22	5164
24	limit 23 to humans	5112
25	limit 24 to yr = “2000 ‐Current”	4271

To identify further studies, reference lists of the relevant studies were scrutinized.

Two reviewers (IJ, MSG) independently screened titles and abstracts of the retrieved citations to identify potentially eligible publications. The full texts of these articles were obtained and independently evaluated by two reviewers (IJ, MSG, FS). Disagreements were resolved by consensus.

Data extraction and presentation

We extracted the following study characteristics from all included studies: objective, health condition, participants (e.g. patients, general population), number of participants performing the preference task and number of participants who have successfully completed the preference tasks and were included in the analysis, elicitation method used, as well as the outcomes which were assessed in the study. A standardized form was used for data extraction. After inclusion of the studies, they were categorized according to the methods used to elicit preferences. Study characteristics between different elicitation methods were compared, specifically regarding the number of outcomes assessed as well as the number of participants successfully completing the task (Table 2). An impression of the comparisons between different participant groups or different methods in individual studies as described by the authors is given in separate sections of the results. Details of other study characteristics, such as objective, sort of outcomes assessed and health condition are presented in detail in supplementary material (Tables S1–S8).

Table 2.

Included studies, information on method, participants and number of outcomes

Study	Method	n completed task (n included)a		No of outcomes
		Patients	Others
Multiple criteria decision analysis methods
Aristides (2004)90	DC	235 (290)		5
Augustovski (2013)91	DC	240		7
Bederman (2010)22	CA	129 (164)	333 (HP)	6
Beusterien (2007)72	ACA & CAR	288 (323)		13
Beusterien (2012)92	ACA	108 (121)		12
Bhargava (2006)93	CARK	82		5
Bishop (2004)18	DC	253 (291)b	94 (98) (HP)	4
Bogelund (2011)94	DC	270 (325)		8
Bridges (2012)95	CA	89 (100)		8
Brown (2011)30	DC	20	33 (CG)	12
Burnett (2012)14	DC		170 parents	6
Carroll (2013)31	DC	65	40 partner	4
Chancellor (2012)23	DC	242 (306)	270 (303) (HP)	5
Deal (2013)96	DC	361 (366)		6
De Bekker‐Grob (2008)97	DC	117 (120)c		5
De Bekker‐Grob (2009)24	DC	117 (120)	39 (40) (HP)	5
Eberth (2009)98	DC	100		7
Essers (2010)99	DC	615		5
Faggioli (2011)25	DC	160	30 (HP); 102 family	6
Fraenkel (2001)73	ACA	65 (103)d		9
Guo (2011)100	DC	194 (204)		6
Hauber (2011)101	DC	143 (150)		7
Hauber (2013)102	DC	289 (294)		7
Haughney (2005)103	DC	122 (125)		9
Hawley (2008)104	CAR	205 (212)		5
Herbild (2008)75	DC	65		4
Hill (2012)26	DC	335 (350)	181 (193) (HP)	4
Hodgkins (2012)105	DC	400		6
Hummel (2012)27	AHP	11 (12)	5 (7)	11
Jendle (2010)106	DC	461 (537)		11
Johnson (2007)74	DC	580 (614)		7
Johnson (2007)107	DC	523		7
Johnson (2010)108	DC	576 (589)		8
Kauf (2012)109	DC	276 (284)		5
King (2012)110	DC	422		7
Kleinman (2002)111	DC	204 (205)		4
Laba (2013)112	DC	188		7
Lancsar (2007)113	DC	57		10
Lichtenstein (2010)114	DC	252		9
Lloyd (2005)115	DC	148		7
Lloyd (2007)79	DC	479		6
Lloyd (2011)116	DC	248 (252)		7
Manjunath (2012)117	DC	193 (263)		7
Marshall (2007)15	DC		547 (GPop); 200 (HP)	6
Marshall (2009)21	DC		1588 (Gpop)	9
McKenzie (2001)118	DC	162 (174)		5
McTaggart‐Cowan (2008)119	DC	157		6
Meister (2002)19	CARK	175	25 (HP)	6
Mohamed (2013)120	DC	383 (400)		8
Nayaradou (2010)17	DC		656 (Gpop)	7
Ossa (2007)84	DC	110		6
Petrou (2009)76	DC	630 (648)		7
Phillips (2002a)51	DC	339 (365)		6
Pieterse (2007)20	ACA	66 (70)	61 (HP)	4
Pieterse (2010)121	ACA	98 (114)		4
Radcliff (2012)122	SMART	36		5
Ratcliff (2004)123	DC	412		5
Schaarschmidt (2011)124	DC	163 (163)		11
Sculpher (2004)125	DC	129		8
Seston (2007)77	DC	126		6
Snoek (2008)78	DC	49 (53)		7
Stanek (2000)29	CAR	50 (51)	47e	4
Sweeting (2011)126	DC	58		5
Taylor (2013)127	CARK	21 (21)		5
Thrumurthy (2011)28	DC	81 (82)	90 (108) (HP)	6
Walzer (2007)13	DC	42 parents		4
Watson (2004)16	DC		211 (GPop)	7
Wirostko (2011)128	ACA	161 (171)		11
Wittink (2010)129	DC	86		5
Wouters (2013)130	ACA	241		8
Zimmermann (2013)131	DC	227 (255)		8
Rating or ranking methods
Bergmark (2002)132	R	256		22
Carvalho (2005)32	RK	82 (93)b		10
Cooper (2000)133	R	76		126f
Davis‐Michaud (2004)134	RK	29		18
Fischer (2011)34	RK	182	35 (HP)	9
Houwert (2010)35	R	40	20 (HP)	17
Howell (2012)135	RK	57 (57)		47g
Jenkins (2001)136	RK	355		10
Kinter (2009)81	RK	25		13
List (2000)83	RK	131 (137)		12
Mahadev (2011)137	R	69		16
Martin (2009)138	R	150	9 (HP)	146h
Opmeer (2007)139	R	29 (30)		50i
Pyne (2008)140	RK	38		12
Rashiq (2003)37	RK	109 (131)	30 (HP)	9
Rodrigue (2011)141	R	104		4
Rosenheck (2005)142	RK	1281		10
Rozen (2006)143	R	150		10
Sanderson (2011)144	RK	26		5
Sherer (2005)145	R	387		9
Shumway (2003)39	R	53j	100 (PM)	6
Singer (2000)146	R	724		8
Turk (2008)82	R	959 (1148)		19
Wagner (2007)33	RK	142 parents		5
Watt (2007)38	R	80	15 (HP)	138k
Utility assessment methods
Cykert (2000)44	SG	64		16
Gu (2009)43	TTO	50	16 (HP)	10
Holt (2010)41	WTT	258 (262)		13
Lenert (2003)40	WTP	257 (395)		6
Volk (2004)42	TTO	168 (men)	168 (wives)	8
Comparison of various methods
Beusterien (2005)49	R vs. CA	35 (42)		12
Bijlenga (2011)46	DC vs. VAS vs. TTO	24	30 (HP); 27 (GPop)	5
Duarte (2007)64	RK vs. R	3000		7
Fegert (2011)53	R vs. DC	121 parents		23 R; 6 DC
Ijzerman (2012)67	AHP vs. CA	86 (89)		6 AHP; 8 CA
Jampel (2003)61	RK vs. WTP	230		7
Kuppermann (2000)66	TTO vs. WTP	206 parents		19
Lewis (2006)50	RK vs. CA	220 (305)b	175 (195) (HP)	3
Merlino (2001)60	R vs. TTO/STO	107	12 (HP)	14
Mühlbacher (2008)47	R vs. CA	282		16 R; 8 DC
Mühlbacher (2009)56	R vs. DC	219 family		23 R; 6 DC
Mühlbacher (2010)54	R vs. DC	329		23 R; 6 DC
Mühlbacher (2011)55 , l	R vs. DC		243 (HP)	16 R; 8 DC
Mühlbacher (2013)57	R vs. DC	214 (218)		26 R; 6 DC
Nguyen (2010)58	R vs. SG		24 (GPop)	7
Palumbo (2011)62	CA vs. WTP	160		5
Pham (2003)59	R vs. SG	90 (91)	59 (60) MW; 31 HP	8
Phillips (2002b)147	RK vs. CA	365		6
Pignone (2011)48	R/RK vs. DC	104		6
Pignone (2013)45	R/RK vs. DC vs. BS		911 (1036) (GPop)	4
Silverman (2012)52	RK vs. CA	367		4
Vermeulen (2007)63	CA vs. WTP	74	200 (HP)m	6
Weiss (2006)65	RK vs. R	999		8

ACA, adaptive conjoint analysis; BS, balance sheet; CAR, conjoint analysis rating; CARK, conjoint analysis ranking; CG, caregivers; FA, family; GPop, general population; HP, healthcare professionals; MW, midwives; PM, policy makers; R, rating; RK, ranking; SMART, Single attribute rating technique; WTT, willingness‐to‐travel.

^aOnly when more patients were included in the study, this is noted here; otherwise, either all patients completed successfully or study did not give any information about incomplete results.

^bPregnant women.

^cElderly women.

^dThe missing 38 patients were used for validation; they are not counted as incomplete results.

^eHealthy control patients.

^f126 outcomes from 7 domains.

^g47 outcomes from 5 domains.

^h146 outcomes from 5 domains.

ⁱ50 outcomes from 3 domains.

^jPrimary stakeholders: patients, family members, mental healthcare providers.

^k138 outcomes from 15 domains.

^lStudy succeeding Mühlbacher 2008.

^m50 clinicians and 150 nurses.

Results

Search results

The systematic literature search retrieved 9740 publications; after exclusion of duplicates and screening for relevance in title and abstract, 219 publications were further appraised in full text. From these, 124 publications were included (see Fig. 1).

Figure 1.

Flowchart of included and excluded publications.

Description of included studies

The included studies investigated the preferences of patients, healthcare professionals, family members or healthy persons concerning specific outcomes. The studies were categorized into four methodological groups (see Fig. 1): the first group used multi criteria decision analysis (MCDA) methods [including discrete choice (DC) methods, conjoint analysis (CA), analytical hierarchy process (AHP)] to elicit preferences, the second group used rating or ranking methods and the third group used utility eliciting methods, namely time trade‐off (TTO), standard gamble (SG) or willingness‐to‐pay (WTP). The fourth group was comprised of studies in which a comparison of different methods was made. About 57% (n = 71) of identified studies used a MCDA method, 20% (n = 25) used a rating or ranking technique and only 4% (n = 5) a utility assessment method. The remaining 19% (n = 23) of studies compared at least two different methods (see Table 2). The median number of outcomes was largest with rating and ranking methods, where a median of 12 outcomes was assessed. Studies using MCDA methods or comparing various methods used a median of 6 outcomes in their elicitation task. The number of outcomes of each study is shown in Table 2, a comparison between the four methodological groups is illustrated in Fig. 2. More details on the studies included in this review are described in the following sections.

Figure 2.

Number of outcomes assessed by the studies, categorized into 4 groups. Boxplots for the methodological groups, line in the box represents the median value; the box marks the 1st and 3rd quartile. Outliers and extreme values not depicted.

Studies using MCDA methods

MCDA methods elicit preferences for various attributes from individuals. The judgements are obtained using rating, ranking or choice exercises. Choice‐based questions have widely been used within the CA framework in healthcare. The techniques used are either DC analysis or adaptive CA.

Of the 71 included studies, 64 used choice‐based methods, mainly DC or adaptive CA (one of these studies also used CA rating), two used CA rating methods and three CA ranking methods. One study used AHP and another study used a single attribute rating technique. Asthma, diabetes or colorectal cancer screening are health conditions or situations which were often chosen to assess preferences. Further health conditions were mainly chronic conditions (depression, hepatitis C, HIV/AIDS, osteoarthritis), but also screening or management settings (osteoporosis prevention, Down's syndrome screening or miscarriage management), see Table S1. Fifty‐four studies elicited preferences of patients only, including two studies which used parents as proxies of children with asthma or juvenile idiopathic arthritis.13, 14 Three studies used preferences of the general population only.15, 16, 17 Fourteen studies compared preferences of different groups to each other: in 11 studies preferences of patients, pregnant women or the general population were compared with preferences of clinicians,18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28 one study compared preferences of patients with congestive heart failure with that of patients with other conditions.29 One study compared preferences of patients with haemophilia with that of their caregivers and another study assessed preferences for Down's syndrome screening in pregnant women and compared it with preferences of their partners.30, 31 Results of the group comparisons are described later in this section. In median, 194 participants were included in the studies (range between 16 and 1788). Thirty‐seven of the 71 studies reported that participants had problems completing the preference task as indicated by discontinued tasks or by inconsistent results which could not be analysed. The remaining studies did not report these numbers or all of the participants successfully completed the tasks. However, in those studies reporting that some participants had difficulties with the task and were not analysed, the number of participants successfully finishing the task was in median 94% (range 63–99.5). With the MCDA methods only a limited number of outcomes were assessed, in median 6 outcomes (range 4–13) (Table 2); for details, see Table S2.

Studies using rating or ranking methods

Simple ranking exercises use an ordinal ranking, where the highest rankings are viewed as the most important. When using rating scales, participants are presented criteria, scenarios or statements and are asked to respond according to their opinions or attitudes on either a numerical or semantic scale.

Of the 25 identified studies, 13 studies used rating and 12 studies used ranking methods to elicit preferences for outcomes. Most studies (n = 19) included patients only, including preferences of pregnant women32 and parents of children (as proxies).33 Five studies compared preferences of patients with those of healthcare professionals.34, 35, 36, 37, 38 One study compared preferences of primary stakeholders as patients, family members or mental healthcare providers with preferences of policymakers in schizophrenia.39 The methods were assessed in a variety of health conditions, ranging from chronic conditions such as cancer, schizophrenia, HIV/AIDS or chronic pain to more acute conditions as caesarean delivery, postoperative recovery or anaesthesia (see Table S3). With a median number of 131 participants (range 25–1281), the rate of successfully completed tasks was high and failures pertained to only a few participants. Only five of 25 studies reported data on patients who had to be excluded from the analysis. In those studies reporting data on failures, in median 88% (range 84–97) of patients were able to complete the rating or ranking task successfully. A large set of outcomes, in median 12 (range 4–146) was assessed (see Table 2 and Table S4).

Studies using utility assessment methods

Five studies were identified in this group that comprises SG, TTO, WTP and Willingness‐to‐travel (WTT) methods. With SG methods, participants are asked to choose between remaining in a state of ill health for a certain period of time or a medical intervention which has a certain chance of either restoring them to perfect health or causing their death. The TTO method requires participants to choose between remaining in a state of ill health for a certain period of time or being restored to perfect health and having a shorter life expectancy. With WTP or WTT methods, participants are presented with a choice of the intervention not being performed or having the intervention applied but giving up a certain amount of money or having a longer travel time. The money they are willing to forfeit or the time they are willing to travel to have the intervention is their willingness to pay (travel) for that intervention.

In all studies, non‐chronic, more acute health conditions or situations were assessed, such as prostate cancer screening, abdominal aortic aneurysm screening, hip arthroplasty, migraine headache or postoperative outcomes after lung resection (see Table S5). The number of outcomes assessed in the individual studies was in median 10 (range from 6 to 16) (see Table S6). The studies were rather large, 257 participants in median (range 64–395), incomplete tasks were reported in two studies.40, 41 However, in one study, 35% were not able to complete the task successfully,40 whereas in the second study 99% of participants succesfully completed the eliciation task.41 The two TTO studies42, 43 compared preferences between groups. The other three studies used WTP,40 WTT41 and SG methods44 to assess preferences of patients only.

Studies using and comparing various method groups or specific methods

Twenty‐two studies compared two methods in the same population; two studies45, 46 compared three methods. The method group most analysed was ranking or rating (n = 18), followed by MCDA (n = 16) and utility assessment (n = 8). In median, 6 or 8 outcomes were assessed, depending on the number of outcomes taken into account. Some studies used different outcomes for the compared methods (e.g. Mühlbacher et al.47 used 16 outcomes for the rating exercise, but only 8 for the DC task). Considering the minimum number of outcomes per study results in a median number of outcomes of 6 (range 3–19) for the whole group; accordingly, taking the maximum number into consideration increases the median value to 8 outcomes (range 3–26) (Table 2 and Table S7). In the following sections, the different method group comparisons are presented; for further information on the included studies, see Tables S7 and S8.

Ranking or rating vs. MCDA methods

In this set of twelve studies, eight studies compared rating methods with CA or DC methods, two studies compared a ranking method with CA and two studies compared a combined rating and ranking exercise with a DC task.45, 48 Additionally, one of the latter studies45 also compared the two mentioned methods with a third method, a balance sheet exercise. Five studies45, 49, 50, 51, 52 reported differences in the elicited preferences between the studied methods which would lead to different conclusions. Two studies found largely consistent preferences assessed with the compared methods, which would lead to the same conclusions, even if the hierarchy of outcomes differed to a small extent.48, 53 The five studies conducted by Mühlbacher et al. did not compare the same number of outcomes with both methods (rating vs. DC). Therefore, it is not possible to analyse the agreement of those methods for every outcome.47, 54, 55, 56, 57 The authors concluded that both methods complement each other.

Ranking or rating vs. utility assessment

Four studies compared a utility assessment with a rating or ranking method.

Nguyen 2010 compared a rating method with standard gamble in non‐seminomatous germ cell testicular cancer. The study found considerable variation in the response for post‐treatment health states.58

Pham 2003 compared a rating method with SG in the context of birth outcomes in three participant groups: postnatal women, midwives and medical staff. According to the authors, the comparison of eight birth outcomes showed highly correlated results for both methods in the same participant groups.59

The study by Merlino 2001 compared rating to TTO (or sleep trade‐off, STO) to assess preferences of rheumatoid arthritis patients for 14 different outcomes. With the STO, the participants were asked to trade the time which was lost due to a non‐refreshing sleep pattern. TTO was used for chronic health states, as, for example, diabetes or septic arthritis; the STO method was used for temporary health states such as hip or rib fractures or urinary tract infections. Hierarchies obtained with the two methods differed to a great extent; only one outcome was found in the top three lists of important outcomes with both of these methods.60

Jampel 2003 was the only study comparing ranking to the utility assessment method, in this case, to WTP. Ranking was used to validate the results obtained by the WTP method. Using the WTP method, patients with glaucoma were asked to state their preferences for specific aspects of a therapy using eye drops. At the end of the questionnaire, patients were also asked to rank the different aspects. In the case of an obvious disparity, the interviewer asked the patient to reconsider his or her estimation of the ranking. However, only four of the 230 subjects reordered their ranking.61

MCDA vs. utility assessment methods

Of the three studies in this set, two compared a CA method to a WTP method.

The third study46 compared a DC method to TTO and additionally to a visual analogue scale. With the data presented in the studies, a direct comparison of preferences assigned to the outcomes by the studied methods was not possible for the study Palumbo et al.62 and Vermeulen et al.63 The study Bijlenga et al.46 found different preference patterns with the three methods analysed.

Ranking vs. rating methods

Two studies (Duarte et al.64 and Weiss et al.65) published the data of one large study, performed in the US (Weiss 2006) and in France, Germany, Mexico, Spain and the United Kingdom (Duarte 2007). Both publications evaluated patients' preferences for osteoporosis medications among postmenopausal women.

In both publications, the two methods showed generally similar results regarding the highest rated outcomes. Discrepancies were rather small and were observable only as small differences in the rank order.64, 65

TTO vs. WTP methods

In the study Kuppermann 2000, the preference of parents for outcomes associated with childhood vaccinations were assessed by two different utility assessment methods. In this study, the correlation between results obtained by those two methods was highly significant.66

AHP vs. CA methods

In the study Ijzerman 2012, the preference of patients in stroke rehabilitation was assessed by two different MCDA methods, AHP and CA. In this study, the correlation of preferences obtained with these methods was high, as stated by the authors.67

Comparison of different groups

Twenty‐eight studies compared preferences of different participant groups, mostly patients vs. health care professionals. Among the 22 studies evaluating preferences of patients and health care providers, only one stated that the preferences assessed were identical,37 and four found at least some agreement concerning the most valued outcomes.25, 27, 34, 35 The remaining 17 studies showed differences between the groups studied, either in differing hierarchies or in strength of preferences. Furthermore, most studies described major differences in preferences for specific outcomes such as safety or timing of a test18, 50 or severe health states.43 When patient preferences were compared with preferences of family members or healthy control patients, a difference was again observable.29, 42 The study Shumway 2003 merged ratings of patients, family members and health care providers into one group (primary stakeholders) and compared them to a group of policy makers. In this study, very similar preferences among the primary stakeholders were found. Comparing primary stakeholders and policy makers, however, showed significant differences.39

Discussion

Summary of findings and summary analysis

Of the 124 included studies, 71 used a MCDA method to elicit preferences. In 25 studies, a rating or ranking technique was used, in 5 studies a utility assessment method. The search strategy identified many more studies using utility assessment methods to elicit preferences. As they did not present results on specific outcomes but a general utility score instead, these studies were not included in this review. With this review, we aimed to identify methods, which enable a prioritisation of preferences for specific outcomes in a health care context. If a drug is superior in one specific outcome but does show no effects or even inferior effects in others, it is important to know how patients or health care providers weight these outcomes. Only when this consideration is made, the net value of a health intervention can be assessed. The incorporation of preferences can have an influence on the development of new treatments, the regulation and reimbursement of treatments, and eventually on the treatment choice for individual patients.

Only few reviews were identified which studied methods to elicit preferences in a medical context. The review by Ryan et al. 2001 investigated which methods could be used to elicit public views on the provision of health care. The authors concluded that there was no method that was generally superior to the others. They recommended that the methods should be further evaluated regarding their applicability and validity.12 A recent update focused on methods used by economists to value patient experiences of health care processes. Although the authors again highlight the importance of including patient preferences, a number of methodological issues needing further attention were identified.68 The review by Opmeer et al. 2010 presents an overview of clinical studies that assessed preferences for non‐health outcomes. They concluded that there is a diversification in methods and application fields which reflects a lack of standardization. They propose that methodological standards and criteria to evaluate the methodological quality and clinical validity of studies are needed.69 This need is also emphasized by Hauber et al.70, who gives an overview of preference elicitation methods used to quantify benefit‐risk preferences for medical interventions. Mühlbacher et al.,71 who assessed the difference in preferences of patients and physicians, stated that the methodology to compare preferences is diverse.

Strengths and limitations of included studies

The reason for using MCDA methods is often described as the possibility to assess the relative values of preferences for specific attributes, similar to trade‐offs made by consumers in the real world.15, 18, 72, 73 Furthermore, the reconstruction of complex decisions in MCDA methods is considered as strength 72, 73 because it enables the participants to rate the relative importance of each attribute. However, these methods are complex and making trade‐offs between a number of attributes and levels is experienced as challenging by participants.74 The preference elicitation tasks were often not completed or simplifying rules were used, for example, a dominant attribute was chosen with the risk of introducing bias.49, 51, 74, 75, 76, 77, 78 To reduce the complexity of the method, the number of attributes and their levels had to be carefully chosen. As the selection of outcomes is often done by the investigators, relevant outcomes might therefore not be considered in the preference task, or relevant levels of these outcomes as they were perceived by patients, might not be considered.19, 73, 79 The results of the elicitation task might therefore provide answers to the wrong questions. Using focus groups with patients to identify relevant outcomes in a first step might be useful to prevent this problem.80

Rating or ranking methods can include a large number of outcomes; therefore, the risk of selecting too few and therewith not all important outcomes is low. Additionally, the simplicity of this method compared with more complex methods as, for example, CA is discussed in some studies.39, 81 The main problems regarding this elicitation method are the variability of prioritization between individuals and a ceiling effect.65, 82, 83 Without a forced ranking component in the elicitation task (rating and ranking), the creation of a hierarchy of outcomes might be difficult.

Utility assessment methods are also feasible to assess preferences for specific attributes. The hypothetical character of the utility assessment methods was described as the major problem by the identified studies.40, 42, 61, 84

The comparison of methods showed an ambiguous picture. Most studies found greatly differing results with the compared methods; however, some studies showed very similar hierarchies. Framing effects, the choice of outcomes (e.g. which attributes and levels used) or the type of survey (e.g. online, paper‐and‐pencil, interview) might have a large influence on differing results.

Strengths and weaknesses of this review

This review gives a comprehensive overview of methods used to prioritize outcomes in a health care context. It informs the reader not only on possible methods, but also holds information in which health conditions these methods have been applied, how many outcomes have been assessed and if there were clues for problems with the feasibility. However, this review has several potential limitations. First, although the identification of a diversity of methods was our aim, the diversity limited the interpretability of results. Apart from different participants included, different outcomes assessed and different health conditions studied, also the different scales and measures in the same methodological groups made a universal statement hardly possible. As quality assessment tools to evaluate the correct handling of preference elicitation methods do not exist, a quality assessment of included studies was not performed. A discussion of the theoretical foundation of the included studies was also not intended. The aim of our descriptive review was confined to the description of the research field. To achieve this aim, the search and the screening process have been systematic; however, the data presentation and interpretation were based on an iterative and more qualitative process, as they evolved throughout the preparation of the review. Our qualitative approach allowed us to incorporate growing insights, but may have introduced some inconsistencies in the description of study aspects or interpretation of differences between methods and participant groups. As we only reported the conclusions of the study authors, we cannot evaluate whether the results were statistically rigorous, the quality of the studies sufficient and the application of the methods appropriate. The advantages and disadvantages of methods as described in the discussion part were only based on examples of included studies, not on a rigorous extraction of additional aspects of studies.

To assess the feasibility of methods, we had a priori planned to extract the number of patients who successfully completed the preference task and to compare this to the number of included patients. We assumed that this could give a first impression of the feasibility, however, this number might have been biased by different participant groups included in the studies (some might have been cognitively more capable to perform more difficult tasks than others) and the different quality of reporting of this information. In some studies, the participants were asked to state their ease to perform the preference task; this estimation could also give an impression of the feasibility of the method. In further reviews, this might be a better criterion for feasibility.

We tried to perform a comprehensive literature search, but we only included medical databases and limited the publication dates to the years 2000 onwards. Our intention was to assess which methods are currently used to elicit preferences for outcomes of specific health conditions in the research field. Publications before 2000 or those not indexed in medical databases were considered less important. Although we might have missed some studies, the objective of this review was to give an overview of possible methods used. Our literature search was thorough, but it was not intended to be exhaustive.

Unanswered questions

A major problem identified while conducting this review concerns the validity of the results obtained by the applied methods. Commonly used criteria are plausibility, internal validity, convergence validity, predictive validity, model fit and dominance tests. Although some study authors concluded that the results had a good validity, no gold standard for validity assessment was applied. Validity checks were performed either by comparing the obtained results with those found by other methods or previous results of other studies or by verifying the obtained results with hypotheses that the authors themselves found reasonable. Some studies assessed the test–retest correlation or construct validity, but a rigorous validation of the instruments used was only performed by very few studies; the interpretation of obtained results is therefore problematic. In particular, the interpretation of different results obtained by two or more methods in the same population is challenging. An ongoing debate exists in the field of MCDA methods about the validity and applicability of the methods and many studies highlight the need for further research.85, 86, 87, 88 A first step is the publication of a guidance for the standardization of experimental designs of DC experiments.89

Apart from the methodological problem to test the validity of preferences, the studies showed interesting results about the variability of preference prioritization by different subgroups. A differing ranking of outcomes was especially observable between patients and health care providers. The prioritization of outcomes might determine whether the technology is rated valuable or not. As most studies showed that there are differences in preferences for the specific outcomes depending on the subgroup asked it is important to make transparent whose preferences are used for the prioritisation of outcomes. This conclusion was also made by Mühlbacher et al. 2013, who evaluated whether patient preferences differed from physicians' judgements by identifying studies that used different preference elicitation methods and highlighting differences. They found that most studies revealed a divergence between preferences of patients and those of physicians.71 To ensure patient‐centred health decisions or evaluations of health technologies, preferences of patients have to be elicited as they differ from those of other stakeholders.

Conclusions

Different methods to elicit the prioritization of outcomes by patients or other stakeholders have been identified. Multi criteria decision analysis methods are the most widespread methods used in the included studies; however, other methods as ranking, rating or utility assessment were also applied. A dominant method most suitable for application in evidence synthesis was not identified. The application of different methods often leads to different hierarchies of outcomes in the same population. The question, how to compare varying hierarchies obtained by different methods is still unanswered. As important differences were found between preferences of patients and other stakeholders, especially health care professionals, the choice of the group to be involved remains consequential. Further research regarding the validity and applicability of the identified methods is needed before they can be implemented in evidence synthesis.

Conflicts of interests

The authors declare that they have no conflict of interests.

Source of funding

No external funding received.

Supporting information

Table S1. Details on the objectives, health conditions and participants of included studies using conjoint analysis.

Table S2. Details on assessed outcomes of included studies using conjoint analysis.

Table S3. Details on the objectives, health conditions and participants of included studies using rating or ranking methods.

Table S4. Details on assessed outcomes of included studies using rating or ranking methods.

Table S5. Details on the objectives, health conditions and participants of included studies using utility assessment methods.

Table S6. Details on assessed outcomes of included studies using utility assessment methods.

Table S7. Details on the objectives, health conditions and participants of included studies using various methods.

Table S8. Details on assessed outcomes of included studies using various methods.