On what basis should the effectiveness of decision aids be judged?

Abstract

Objective Attempts to synthesize the evidence on the effects of decision aids have been hampered by the lack of consensus regarding how such effectiveness should be measured. This paper seeks to describe and critically assess the range of measures of effectiveness used in randomized controlled trials of decision aids.

Search strategy The published systematic reviews of the field were used to identify primary studies evaluating the effects of decision aids.

Inclusion criteria Non‐randomized trials were excluded from this review. As were abstracts and theses of subsequently published studies, methodological papers and reports of subgroups of a study's main publication.

Main results A wide range of measures were used to evaluate the effectiveness of decision aids. The most commonly used measures sought to assess treatment decisions, patient's knowledge and the decision‐making process. This pattern was repeated when primary measures of effectiveness were examined. No study attempted to measure the extent to which decisions made were consistent with patient's values.

Conclusions Within the current literature there is little consensus on what the aims of decision aids should be. If we can agree that the aim of a decision aid is to help patients make specific personal treatment choices, then evaluations of decision aids should measure the primary effectiveness of their interventions in terms of the extent to which they enable patient's to undergo treatments that agree with their values.

Introduction

The first randomized controlled trial (RCT) of a decision aid was conducted in 1983. ¹ Since then more than 30 subsequent studies investigating the effectiveness of such interventions have been published. These trials have employed a wide range of measures under which the effectiveness of decision aids has been judged, reflecting the breadth and complexity of evaluating interventions aimed at the doctor–patient decision‐making process. However, the lack of an accepted primary measure of effectiveness against which the effects of decision aids can be compared across these studies has hampered research in the field. ²

In many ways this is not surprising as decision aids have been promoted as tools for increasing the involvement of patients in treatment decision‐making, and this call for greater patient participation has come from a range of perspectives, including patient consumerism, ³ medical ethics, ⁴ evidence‐based medicine ⁵ and decision theory, both normative ⁶ and descriptive. ⁷ Some have put the case for specific models of decision‐making, ⁸ , ⁹ whilst others have argued from more pragmatic standpoints, seeing increased patient involvement as a vehicle through which research findings could be implemented, ¹⁰ specific health‐care decisions promoted ¹ , ¹¹ , ¹² or health status improved. ¹³

Due to these wide ranging perspectives, there has been little consensus on what the aims of decision aids are, and hence under what criteria their effectiveness should be judged. The aims of this paper are to provide: a summary of the measures of effectiveness that have been used by previous evaluations of decision aids and the particular measures chosen as the primary measure of effectiveness; a critical discussion on the strengths and weaknesses of these measures; and an assessment of whether recommendations can be made on the choice of primary measure of effectiveness for future evaluations.

Methods

There have been four systematic reviews of decision aid RCTs, all published in 2000/1. Although the Cochrane review was first published in 1999, ² the 2001 ¹⁴ update has been used in this study. Two of the reviews (Molenaar et al. ¹⁵ and Estabrooks et al. ¹⁶ ) included reports from non‐randomized evaluations of decision aids. Only the randomized studies from these reviews have been included in this paper as many of the studies were either pilot evaluations carried out by teams who subsequently published randomized trials of the same decision aids, or were designed to assess the reliability, validity and acceptability of the aids rather than to measure effectiveness. The other review concentrated only on decision aids for patients with cancer. ¹⁷ The Cochrane review's list of RCTs of decision aids awaiting publication ¹⁴ was also used to identify studies published in 2001 and after.

To assess the effectiveness measures used within the published randomized trials of decision aids; data was abstracted on those measures for which results were presented. The effectiveness measures were classified as follows: knowledge; decision process; decision; health status; and other, the collection of economic data was also noted (see Box 1). Further data on the format of the measure, its source and the characteristics of the clinical situation faced by study subjects were also abstracted. Where the primary measure was not explicitly identified in the trial report, the order and prominence of the measures described in the introduction and methods sections were used in identification.

Table Box 1  .

Classification of effectiveness measures

Knowledge	Treatment or condition‐related knowledge
	Expectation of likely outcomes
	Perception of the risks and benefits of the treatment options
Decision process	Perceptions of the decision‐making process
	Satisfaction with the decision‐making process
	The level of control subjects had over the decision
	The level of control subjects had over the decision‐making process
	Subject's preferences for participation in the decision‐making process
Decisions	Treatment intention
	Treatment undergone
	Whether a treatment decision had been made
	Adherence to the decision made
Health status	Condition‐specific health status
	General health status
Economic variables	Intervention costs
	Resource use

Included studies

The Cochrane review included 27 articles published between 1983 and 2001, the Estabrooks review identified nine RCTs reported between 1986 and 1999, and the Molenaar review covered 14 trial reports published between 1983 and 1998. The Effective Health Care Bulletin identified five RCTs of decision aids relating to cancer, published between 1995 and 2001, and two further articles published in 2001 were identified from the Cochrane review's list of RCTs of decision aids awaiting publication. Each of these five sources contributed at least one report not included in any other.

In total, 38 articles were identified and 33 were included in this review of effectiveness measurement. ¹¹ , ¹² , ¹³ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ One study included in the Cochrane review was excluded as it was a report of the results of a subgroup of the main trial report, ⁴⁷ another because it was a methodological paper rather than a trial report ⁴⁸ and one as it was a thesis of a subsequently published paper. ⁴⁹ One study was excluded from each of the Molenaar and Effective Health Care reviews as they were abstracts of studies, the full reports of which were identified from the Cochrane review. ⁵⁰ , ⁵¹

Study characteristics

The majority of the trials examined the effects of decision aids on either patients with a diagnosed condition facing a choice between alternative treatment options (12/33), ¹³ , ²¹ , ²³ , ²⁷ , ²⁸ , ³² , ³⁴ , ³⁷ , ³⁹ , ⁴⁰ , ⁴⁴ , ⁴⁶ or potentially eligible population samples interested in screening or some other preventative health‐care intervention (13/33). ¹¹ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁹ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁸ , ⁴¹ , ⁴² , ⁴⁵ The other studies looked at parents or pregnant women making health‐care decisions for their children or unborn child (5/33) ¹ , ¹² , ²⁴ , ³⁰ , ³³ and volunteers set a hypothetical decision related to treatment or entry to a clinical trial (3/33). ¹⁹ , ²⁰ , ²²

Cancer was the subject of approaching half of the aids (14/33), ¹⁹ , ²⁰ , ²² , ²³ , ²⁶ , ²⁸ , ²⁹ , ³⁴ , ³⁶ , ³⁸ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ communicable diseases (4), ¹¹ , ¹⁸ , ²⁵ , ³³ cardiovascular disease (3) ³² , ³⁷ , ⁴⁰ and the menopause (3) ³¹ , ³⁵ , ⁴⁵ were the subject of a number of studies each. The other studies focused on potential birth defects (Down's syndrome and cystic fibrosis, 2), ²⁴ , ³⁰ benign prostatic hyperplasia (2), ²⁷ , ⁴⁶ back pain, ³⁹ facial deformity, ²¹ ulcer disease ¹³ and circumcision (2). ¹ , ¹²

Effectiveness measures

The most frequently used effectiveness measures related to the decisions faced by subjects. All but four studies collected data on decisions, ¹ , ¹¹ , ¹² , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁹ , ³⁰ , ³¹ , ³² , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ as treatment intentions (14), treatments undergone (17), whether a treatment decision had been made (1), or as adherence to the decision made (2) (Table 1). Data were mostly collected in terms of the treatment options subject's intended to undergo or had undergone. However, three of the studies that assessed intentions, used likert type scales to assess strength of intention, rather than providing discrete choice options. One study ²⁹ used provision of a blood sample to be used for future genetic testing, as a proxy measure for intention, in addition to a likert scale measure.

Table 1.

Primary and secondary measures of effectiveness

Article	Primary measure	Format	Secondary measures	Format
Herrera et al. 1	Treatment undergone	Options
Maisels et al. 12	Treatment undergone	Options	When decision made	Options (pre/postnatal)
			Perceptions of treatment risks/benefits	Scale developed for study
			Perceptions of treatment risks/benefits	Scale developed for study
Greenfield et al. 13	Decisional control	Coded from recordings	Decisional control	Coded from recordings
			Decisional control	Coded from recordings
			General health status	Published scale (Functional status index 52 )
			Condition‐specific health status	Scale developed for study
			Decisional control	Scale developed for study
			Knowledge	Scale developed for study
			Satisfaction with care	Scale developed for study
Carter et al. 11	Treatment undergone	Options
Clancy et al. 18	Treatment undergone	Options
Fetting et al. 19	Intention	Options	Perceived likelihood of outcomes	VAS
			Perceived likelihood of outcomes	VAS
Llewellyn‐Thomas et al. 20	Knowledge	Scale developed for study	Intention	Options
			Perceptions of treatment risks/benefits	Scale developed for study
			Perceptions of treatment risks/benefits	Scale developed for study
Phillips et al. 21	Perceived likelihood of outcomes	Published scale (long‐term expectations scale 53 )	Intention	Options
Sebban et al. 22	Intention	Likert scale	Satisfaction with decision	Likert scale
			Reliability	Options
Street et al. 23	Knowledge	Scale developed for study	Optimism	Published scale (Optimism disposition scale 54 )
			Decisional control	Coded from recordings
			Decisional control	Published scale (Perceived Involvement in Care scale 55 )
			Decisional control	Published scale (Perceived decision control instrument 56 )
			Intention	Options
Thornton et al. 24	Treatment undergone	Options	Treatment undergone	Options
			Treatment undergone	Options
			Treatment undergone	Options
			Anxiety	Published scale (Hospital Anxiety and Depression Scale 57 )
			Anxiety	Published scale (State Trait Anxiety Inventory 58 )
O'Connor et al. 25	Treatment undergone	Options	Perceived likelihood of outcomes	Probability scale
			Decisional conflict	Published scale (Decisional Conflict Scale 59 )
			Condition‐specific health status	Published scale (Influenza vaccine symptom scale 60 )
			General health status	Absenteeism
Wolf et al. 26	Intention	Likert scale
Barry et al. 27	Intention	Options	Treatment undergone	Options
			Knowledge	Scale developed for study
			Satisfaction with the decision‐making process	Scale developed for study
			Condition‐specific health status (1)	Published scale (AUA symptom index 61 )
			Condition‐specific health status (2)	Published scale (Benign Prostatic Hyperplasia Impact index 62 )
			General health status	Published scale (SF‐36 63 )
			Decisional control	Published scale (Autonomy Preference Index 64 )
			Satisfaction with decision	Scale developed for study
Davison and Degner 28	Decisional control	Published scale (Control preferences scale 65 )	Anxiety	Published scale (State Trait Anxiety Inventory 58 )
			Depression	Published scale (Centre for epidemiologic studies depression scale 66 )
Lerman et al. 29	Knowledge	Published scale (Inherited breast cancer and BRCA1 testing knowledge scale 67 )	Perceived likelihood of outcomes	Likert scale
			Perceptions of treatment risks/benefits	Published scale (BRCA1 testing attitudes scale 67 )
			Intention	Likert scale
			Intention	Provision of blood sample
Michie et al. 30	Knowledge	Scale developed for study	Anxiety	Published scale (State Trait Anxiety Inventory 68 )
			Systematic decision	Scale developed for study
			Satisfaction with the decision‐ making process	Scale developed for study
			Treatment undergone	Options
Rothert et al. 31	Knowledge	Scale developed for study	Decisional conflict	Subscale of published scale (Decisional Conflict Scale 58 )
			Satisfaction with decision	Published scale (Satisfaction With Decision scale 69 )
			Satisfaction with most recent encounter with health practitioner	Published scale (Satisfaction Scale 70 )
			Decisional control	Published scale (Self efficacy Scale 71 )
			Adherence	Options
Bernstein et al. 32	Satisfaction with the decision‐ making process	Published scale (Decision‐making process questionnaire 27 )	Satisfaction with decision	Published scale (Satisfaction With Decision Scale 27 )
			Knowledge	Published scale (Knowledge questionnaire 49 )
			Condition‐specific health status	Published scale (Seattle Angina Questionnaire 72 )
			General health status	Published scale (SF‐12 73 )
			Treatment undergone	Options
Dunn et al. 33	Knowledge	Scale developed for study
Maslin et al. 34	Anxiety and depression	Published scale (Hospital Anxiety and Depression Scale 59 )	General health status	Published scale (SF‐36 74 )
			Treatment undergone	Options
			Decisional control	Scale developed for study
			Satisfaction with decision	Not described
O'Connor et al. 35	Decisional conflict	Published scale (Decisional conflict scale 58 )	Perceived likelihood of outcomes	Probability estimate
			Knowledge	Scale developed for study
			Intention	Options
			Perceptions of treatment risks/ benefits	VAS
Davison et al. 36	Decisional control	Published scale (Control preferences scale 65 )	Anxiety	Published scale (State Trait Anxiety Inventory 75 )
			Decisional conflict	Published scale (Decisional Conflict Scale 59 )
			Treatment undergone	Options
Man‐Son‐Hing et al. 37	Whether decision made	Yes/no	Treatment undergone	Options
			Knowledge	Scale developed for study
			Perceived likelihood of outcomes	Probability estimate
			Decisional conflict	Published scale (Decisional Conflict Scale 59 )
			Satisfaction with the decision‐ making process	Scale developed for study
			Adherence	Options
Volk et al. 38	Knowledge	Scale developed for study	Intention	Options
			Intention	Options
Deyo et al. 39	Condition‐specific health status	Published scale (Roland disability scale 76 )	Treatment undergone	Options
			General health status	Absenteeism
			Condition‐specific health status	Published scale (Back pain symptom satisfaction 77 )
			Satisfaction with the decision‐ making process	Scale developed for study
			Satisfaction with care	Scale developed for study
			Health‐care utilization	Resource use
Morgan et al. 40	Satisfaction with the decision‐making process	Published scale (Decision‐making process questionnaire 27 )	Knowledge	Scale developed for study
			Intention	Options
			Treatment undergone	Options
			General health status	Published scale (SF‐36 63 )
			Condition‐specific health status	Published scale (Canadian Cardiovascular Angina scale 78 )
			Agreement of doctor recommendation and intention	Options
Pignone et al. 41	Treatment undergone	Options	Intention	Options
			Intention	Likert scale
			Whether screening discussed	Options
			Agreement of intention and treatment undergone	Likert scale
Schapira and Vanruiswyk 42	Treatment undergone	Options	Knowledge	Scale developed for study
			Perceptions of treatment risks/benefits	Scale developed for study
Wolf and Schorling 43	Intention	Likert scale	Intention	Options
			Knowledge	Scale developed for study
			Perceptions of treatment risks/benefits	Likert scale
Goel et al. 44	Decisional conflict	Published scale (Decisional conflict scale 59 )	Knowledge	Published scale (Breast Cancer Information Test – Revised 79 )
			Satisfaction with decision	Scale developed for study
			Anxiety	Published scale (State Trait Anxiety Inventory 58 )
			Treatment undergone	Options
Murray et al. 45	Treatment undergone	Options	Decisional conflict	Published scale (Decisional Conflict Scale 59 )
			Decisional control	Scale developed for study
			Total cost	Cost
			Anxiety	Published scale (State Trait Anxiety Inventory 68 )
			General health status (1)	Published scale (SF 36 74 )
			General health status (2)	Published scale (EQ‐5D 80 )
			Condition‐specific health status	Published scale (Menqol 81 )
Murray et al. 46	Condition‐specific health status	Published scale (AUA symptom index 61 )	Anxiety	Published scale (State Trait Anxiety Inventory 68 )
			Decisional conflict	Published scale (Decisional Conflict Scale 59 )
			Decisional control	Scale developed for study
			Total cost	Cost
			General health status (1)	Published scale (SF 36 74 )
			General health status (2)	Published scale (EQ‐5D 80 )

Measures assessing the decision‐making process were used in more than half the studies (18/33). ¹³ , ²² , ²³ , ²⁵ , ²⁷ , ²⁸ , ³⁰ , ³¹ , ³² , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁹ , ⁴⁰ , ⁴⁴ , ⁴⁵ , ⁴⁶ The most commonly used measure was the Decisional Conflict Scale used in nine studies (Table 1). Other outcome measures focused on the level of control subjects had over the decision or decision‐making process, or their desire to participate in decision‐making (9). Measures included coding schemes applied to tape recordings of the consultation, item response scales specially developed for the studies and a range of published scales (Table 1). Ten studies assessed the subjects’ satisfaction with the decision or the decision‐making process using likert scales, study specific item response scales or published scales. One trial developed and used a measure of the extent to which the decision was made in a systematic manner.

Subjects’ knowledge and beliefs were also used as effectiveness measures in more than half the studies (20/33). ¹² , ¹³ , ¹⁹ , ²⁰ , ²¹ , ²³ , ²⁵ , ²⁷ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁵ , ³⁷ , ³⁸ , ⁴⁰ , ⁴² , ⁴³ , ⁴⁴ These were generally specially developed item response scales. Other measurement methods included visual analogue scales, probability scales, likert scales and four studies used previously published scales (Table 1).

Health status was measured in just under half the studies (14/33). ¹³ , ²⁴ , ²⁵ , ²⁷ , ²⁸ , ³⁰ , ³² , ³⁴ , ³⁶ , ³⁹ , ⁴⁰ , ⁴⁴ , ⁴⁵ , ⁴⁶ Seven trials used published condition‐specific measures relevant to the conditions being studied (one study developed a condition‐specific scale for their project) and seven trials assessed the effects of the interventions on anxiety or depression (Table 1). Seven trials measured general health status using standard published scales and two studies used absence from work as a proxy measure for health status.

Other measures of effectiveness were: satisfaction with care (2); ¹³ , ³⁹ optimism (1); ²³ subjects’ satisfaction with their most recent encounter with their health practitioner (1); ³¹ whether the condition under study was raised as a subject during the consultation (1); ⁴¹ when the treatment decision was made (1); ¹² agreement between the clinician's recommendation and the subject's treatment intention (1); ⁴⁰ the agreement between treatment intentions and treatment undergone (1); ⁴¹ and one study assessed intention after a second application of the intervention to assess test–retest reliability. ²² Three studies collected data on economic variables. ³⁹ , ⁴⁵ , ⁴⁶ Two calculated total costs, intervention costs and subsequent resource use, and one collected data on resource use only (Table 1).

Primary measure of effectiveness

Primary measures covered the range of main categories described above (Table 1). The most frequently used primary measures related to the decision (15/33). Five studies assessed differences in treatment intentions, nine in treatments undergone and one trial used the extent to which a decision had been made as the primary measure. The effectiveness of the decision aids to improve subjects’ knowledge was also used for a number of studies (8/33). The majority used general measures of treatment‐related knowledge (7/8) and one, the subjects’ perceptions of the likelihood of the potential outcomes.

Seven studies assessed the effect of their interventions on the decision‐making process. Three used the decisional conflict scale, two the level of decisional control held by the subject and two more the subjects’ satisfaction with the decision‐making process. Two studies used condition‐specific health status measures, the Roland disability scale in a study of treatment for back pain and the American Urological Association symptom index in a study of treatment for BPH; another assessed anxiety and depression (HAD scale) in a study of treatment for breast cancer.

Discussion

The wide range of measures of effectiveness used in these RCTs reflects the varying aims of the decision aids and the different effects potentially attributable to them. Whilst this may be useful for secondary measures of effectiveness, this lack of consensus was also evident in the selection of the primary measure of effectiveness which limits the generalizability of the findings of these evaluations and the scope for using meta‐analysis. This is likely to continue until decision aid researchers resolve the debate surrounding what decision aids are supposed to do.

A previous critique of effectiveness measures for evaluations of decision aids, by Entwistle et al. ⁸² argued that in a health‐care system with a primary aim to improve health and well‐being, decision aids should be considered in the context of whether they can improve health status. In addition, the use of generic health status measures to assess effectiveness permit the benefits of these interventions to be compared with other potential uses of health‐care resources. Few studies used health status as their primary effectiveness measure, perhaps because any improvement in health status associated with a decision aid is likely to be small and may not accrue for many years after the decision under evaluation has been made. Studies would therefore require large numbers of subjects and involve follow‐up for many years.

Decision aids are often applied to conditions for which there is no clinically best treatment, and therefore the patient's values will often be the determining factor in the choice between treatment options. In this situation observing a difference in health status as a result of receiving a decision aid is highly unlikely. In addition, in situations where there may be differences in the clinical effectiveness of available treatment options, certain characteristics of the options may make the choice between them less straightforward. For example, should the most effective treatment option involve a long recuperation period, patients may choose to undergo a less effective option that would interfere less with their life or lifestyle, or should the most effective treatment involve changes in diet and behaviour that the subjects value highly, they may choose a less effective treatment option without these limitations. The problem with using health status as the primary measure of effectiveness is that patients might quite rationally choose a treatment option that does not maximize their future health status.

A commonly employed definition of what constitutes a good decision, proposed by O'Connor et al. ⁸³ is that a decision should be informed, should agree with subject's values and should be implemented. Two aspects of this definition map onto the effectiveness measures employed in the evaluations described above. A measure that tries to incorporate all three of these features has undergone initial validation, but has yet to be used in RCT. ⁸⁴

Knowledge‐related outcomes attempt to assess the extent to which decisions made have been informed. However, improving knowledge should not be the primary aim of a decision aid. Knowledge may help patients to form treatment preferences but it does not ensure that patients are able to play a significant role in the clinical encounter or ensure that they receive the treatment that they want. ⁸⁵ The use of knowledge measures also assumes that patients are able to use the information to come to a decision, whereas in practice subjects may need additional help to combine this evidence with their values to choose between the options. ⁸⁶ The use of knowledge as a measure should be confined to the intervention development process, and the ability of a decision aid to increase knowledge should be established long before it undergoes evaluation in RCT.

Decision‐related measures, whether as treatment intention or treatment undergone, have been used to assess the extent to which the decision made is implemented. The use of intention rather than treatment undergone tends to overestimate the size of this effect. Whilst treatment intentions are highly correlated with treatments undergone, a significant proportion of subjects expressing an intention towards a treatment option, do not follow through their intention. However, treatments undergone cannot be used as the basis on which a decision aid should be evaluated. The purpose of a decision aid is not to promote one treatment option over another. A decision aid should provide evidence on the risks and benefits of all the options, the right choice for any individual patient will depend on how he or she values the risks and benefits of these options.

The one aspect of the O'Connor definition that has not been adopted as a primary outcome measure, is the extent to which decisions made agree with subject's values. Of the 33 studies included in the review none attempted to measure this outcome directly, however, the use of some decision process measures, may be interpreted as attempts to measure this outcome indirectly. The measures assessing decisional control and satisfaction with the decision‐making process may be tapping into a slightly different characteristic of the decision process, one of autonomy and participation in the consultation. Such measures are inappropriate as primary measures of effectiveness as again the aim of a decision aid is not to promote one model of decision‐making over any other. Individual patients will have preferences over the role they wish to play in the decision‐making process, and this role may even change during the course of this process, increasing as they become more informed or decreasing as they become more anxious. ⁸

Measures assessing satisfaction with the decision and the effective decision‐making subscale of the Decisional Conflict Scale could be interpreted as attempts to assess the underlying construct of the extent to which subject's values were incorporated into the decision. Neither of these measures quite achieves this.

Satisfaction with the decision does not assess the extent to which the decision corresponded with the patient's values. As with all assessments of satisfaction a whole range of other factors may influence the perception and it is subject to a range of potential biases. ⁸⁷ The Decisional Conflict Scale measures subject's perceptions of the extent to which they are uncertain about which option to choose, the factors contributing to this uncertainty and the effectiveness of their decision. Of the 16 items that make up the scale just one, on the decision effectiveness subscale, addresses whether the choice reflected the patient's values, and this particular subscale exhibits low discriminant abilities. ⁵⁹ Therefore, the most important contribution of this scale may be in quantifying uncertainty and factors contributing to uncertainty, rather than measuring the extent to which patient's values were incorporated into the decision.

If we adopt the O'Connor et al. argument that the purpose of a decision aid is to help patients to make ‘a specific, personal choice between options’, then the effectiveness of decision aids should be judged by the extent to which patients undergo treatments that are consistent with their values for the potential outcomes of the available options.

At its most basic a measure to assess agreement could take the form of a ranking of the most important characteristics influencing the decision, and effectiveness judged in terms of the extent to which the treatment undergone satisfied these characteristics. Such an approach was used by two of the observational studies included in the Molenaar and Estabrooks reviews. ⁶² , ⁸³ At its most complex utility measurement would be the measure of value and decision aids evaluated in terms of maximizing expected utility. A methodological paper conducted as part of the Rothert study, ⁴⁸ made the case for measuring agreement between values and the treatments undergone in terms of the correlation between subjective expected utility for potential treatment outcomes and the likelihood of undergoing therapy. However, no formal validations of such measures have so far been undertaken. The methodological challenge will be to ensure that the effectiveness measure used, is valid, in that it includes all the risks and benefits of importance to the patient, not just those that are health related; is unbiased, for example not influenced by framing effects or attitudes to risk; is sufficiently sensitive; and is practical for use in large RCTs.

In conclusion, if decision aid researchers can agree that the primary purpose of a decision aid is to help patients make specific personal choices between different treatment options, then decision aids should be evaluated in terms of the extent to which patients undergo treatments that agree with their values.