A systematic review of information in decision aids

Abstract

Objective  We completed a systematic review of information reported as included in decision aids (DAs) for adult patients, to determine if it is complete, balanced and accurate.

Search strategy  DAs were identified using the Cochrane Database of DAs and searches of four electronic databases using the terms: ‘decision aid’; shared decision making’ and ‘patients’; ‘multimedia or leaflets or pamphlets or videos and patients and decision making’. Additionally, publications reporting DA development and actual DAs that were reported as publicly available on the Internet were consulted. Publications were included up to May 2006.

Data extraction  Data were extracted on the following variables: external groups consulted in development of the DA, type of study used, categories of information, inclusion of probabilities, use of citation lists and inclusion of patient experiences.

Main results  68 treatment DAs and 30 screening DAs were identified. 17% of treatment DAs and 47% of screening DAs did not report any external consultation and, of those that did, DA producers tended to rely more heavily on medical experts than on patients’ guidance. Content evaluations showed that (i) treatment DAs frequently omit describing the procedure(s) involved in treatment options and (ii) screening DAs frequently focus on false positives but not false negatives. About 1/2 treatment DAs reported probabilities with a greater emphasis on potential benefits than harms. Similarly, screening DAs were more likely to provide false‐positive than false‐negative rates.

Conclusions  The review led us to be concerned about completeness, balance and accuracy of information included in DAs.

Introduction

Patient decision aids (DAs) aim to facilitate patient involvement in decisions about their health care, with the goal that each person's decision be informed and consistent with his/her values. ¹ , ² , ³ The information provided in the DA, therefore, must be relevant, accurate and complete. With the increasing rate of DA production in recent years, ⁴ we were interested to determine what guidance is available in the literature for new developers in how to select and present information in their DAs.

For treatment decisions, determining relevant information is guided, to some extent, by the legal and ethical obligations of informed consent. ⁵ , ⁶ Legal requirements typically specify that the content covered includes a description of the medical condition, all treatment options including their procedures, potential benefits and harms or side effects. The ethical considerations imply that the information should be complete, balanced and as accurate as possible. While there is no legal doctrine related to screening decisions, they are subject to the same ethical principles, and one would expect parallel types of information to be included.

Although legal and ethical obligations broadly define what information should be provided to patients, it is the patients themselves and the health professionals involved in treating them who can best determine which details should be covered. To provide best information to patients, input from all relevant groups is preferable as empirical studies suggest that the different perspectives yield different concerns such as is seen, for example, in the setting of early stage prostate cancer. ⁷ , ⁸ , ⁹ In addition, because information has to be relevant for patients, consultations with them will determine if there are categories of information that should be addressed beyond those defined by informed consent considerations. ¹⁰

Concerns over the accuracy of the information include the notion that the patient should understand how accurate it is. And the quality of the evidence depends, in part, on the method used to generate it. Hierarchies of evidence ¹¹ have been designed to help people understand which studies are most likely to provide the most valid and reliable data. The methods range from the most accurate large randomized trials and meta‐analyses to the least accurate case studies and anecdotes. Although there are concerns about accuracy of anecdotal evidence, the use of patient narratives has been advocated. ¹² The recommendation follows from the fact that it is easier for people to understand the concrete events of one person's experience than the more accurate but abstract presentation of a group's experience. Patient narratives of specific health states are particularly useful to help others understand a state they have never experienced before. ¹³ Thus, there is tension about how best to convey the possible consequences of particular choices in DAs.

Accuracy of information is also affected by how recently it was generated. This is especially important for medical DAs where continual research makes information around screening and treatments quickly out‐of‐date. One strategy used by DA producers to make clear when their information was produced is to state it and/or provide citations, either within the patient DA or in a separate resource.

To gain perspective on the readily available guidance to new DA developers, this study reviews what is reported in the literature about information included in DAs, both treatment and screening. In that vein, the study has three goals. First, to determine the extent to which DA producers have sought guidance about what information to include in their documents and what methods they used to obtain the guidance. Secondly, to determine how widely the categories of information demanded by informed consent legislation representing a minimum standard, are included. And thirdly, to determine how DAs have described the accuracy of the information presented. To reach these goals, we reviewed the information included both in treatment and in screening DAs as they have been reported in the published literature and, for those DAs easily available to the public on the Internet, we viewed them directly. We recognize that reports of what people have included in their DAs are not necessarily the most accurate description of what they have done, and that viewing a DA does not provide insight into the methods used to develop it. Thus, the most comprehensive and accurate descriptions were derived using both types of data sources. However, our intention was to review what is readily available because it is the most likely guide to new developers.

Methods

Decision aid identification

We began with the DAs identified by the Ottawa Health Research Institute for the 2003 Cochrane Review with an update for 2004. Two independent reviewers (SB and KM) then systematically searched the literature for DAs not included in the review or published after its latest update. Four databases were searched including CINAHL, psycINFO, Ovid Medline and Embase using the following key words: ‘decision aid’, ‘shared decision making and patients’, and ‘multimedia or leaflets or pamphlets and patients and decision making’. The search was limited to English articles only. There was no time restriction.

A priori inclusion criteria were the following: any primary research specifically describing the development or implementation of a DA, DA designed for a competent adult making a medical screening or treatment decision for themselves. Eighty DAs identified from the Cochrane Review met our inclusion criteria. About 129 unique citations from the literature search met our inclusion criteria and these papers represented another 18 DAs not already accounted for. Thus, in total 98 DAs were identified for review: 68 treatment‐focussed and 30 screening‐focussed DAs.

We also sought to obtain all publications reporting the development of a DA referred to in the DA primary report; 48 DAs had papers reporting explicitly on their development. We also viewed actual DAs (n = 33) if they were reported as publicly available on the Internet.

Evaluations

The resulting papers and DAs were reviewed independently by two reviewers (SB and KM) who then met to resolve discrepancies. They searched for evidence relating to our three major concerns.

Guidance to information selection

We were interested in identifying the type of guidance used in the development phase of the DA and we defined the development phase to include guidance sought before pilot or pre‐testing of the DA. In particular, we identified whether guidance was sought from professionals and/or patients on the issues that the DA should address, and whether guidance was sought through a direct process, either qualitatively through interviews or focus groups, quantitatively through surveys, or both. In some instances developers described basing their development on the literature, and when we could determine the types of studies used, we reclassified them as above. When we could not determine the types of studies the literature referenced or when the literature use was related to the actual outcomes rather than the selection of information, we simply left the classification as ‘literature’. Other developers identified theory as guiding their selection of information.

Content

Because a minimum standard for content would reflect informed consent requirements, we investigated the categories typically demanded by informed consent legislation. Thus, we searched for a description of the medical condition, identification of various treatment options, including procedures, benefits and harms. We note that we could not judge the comprehensiveness of the treatment options offered in the DA including whether or not a ‘no treatment option’ was explicitly addressed. Although informed consent legislation focuses on treatment decisions, we used that as our guide and drew parallels for the screening DAs. Thus, we searched for description of the condition including the probability of its occurrence. Regarding test details, we searched for test options being presented including a no‐test option, a description of test procedures, potential benefits and harms, what the test was measuring and its potential inaccuracies, including a description of false positives and false negatives along with their probabilities. Finally, we also looked for interpretation of test results including potential strategies for addressing the result.

Information accuracy

We searched for indicators that conveyed accuracy of information, such as confidence intervals around quantities, type of study that produced the data, or a rating scheme that described how accurate the information is. We also searched for indirect quality indicators such as provision of a list of sources, either within or additional to the DA, from which the data were gathered, and we determined when patient narratives were included.

Results

Table S1 lists the treatment‐related DAs and Table 1 the screening‐related DAs that we found, including the papers reporting their development and/or their content. As Tables S1 and 1 show, the most comprehensive descriptions we could find, including both a direct view of the DA and reports of its development and implementation, were available for 16 of 68 (24%) treatment DAs, and for only two of 30 (7%) screening DAs.

Table 1.

 Screening Decision Aids

Decision aid		Source		Guide		Condition		Test details													Evidence
Condition	Producer	DA	Rep	Pt	Exp	Desc	Prob	Opt	No test	Proc	Ben	Harm	Meas	Inacc	False+	False−	Prob+	Prob−	+Res	−Res	Cite	Pt Exp
BRCA 1/2 genetic testing	Green 145 , 146 , 147		D, I		Q	•	•	•	•	•	•	•							•
	Lerman et al. 148		I				V	•			•	•		•					•	•
	Schwartz et al. 149		I			•		•			•	•		•					•	•
Breast cancer: mammography	Lawrence et al. 150		D	Q	Q	•	•	•	•	•	•	•		•			•	•	•
	Lewis/Pignone 151	•	I		L			•		•	•	•		•	•		•		•
	Rimer et al. 152 , 153		I		T		•				•	•		•	•	•			•			•
Cervical cancer	Adab et al. 154		I				•			•	•	•		•			•	•	•
	Holloway et al. 155		I				•
Colon cancer	Dolan 156 , 157		I		Q, L	•		•	•	•	•	•		•	•		•		•
	Kim et al. 158		D	Q	L	•	•	•		•	•	•	•						•			•
	Pignone et al. 159	•	I	Q	T	•	•	•		•	•	•	•						•	•		•
	Wolf and Schorling 160		I		Q		•	•	•	•	•	•	•	•	•	•	•	•	•
Prenatal testing	Bekker et al. 161		I				•	•	•			•							•
	Drake 162 , 163		D, I		Q, L	•	•	•	•		•	•							•		•	•
	Hewison et al. 164		I		Q, L	•	•					•	•	•	•	•			•	•
	Kuppermann et al. 165		I			•	•					•
	Leung et al. 166		I			•		•	•										•
	Michie et al. 167		I			•					•	•							•
	MIDIRS/NHS 71 , 72	•	D, I	Q	Q, L	•		•	•	•	•	•	•	•	•	•		•	•		•
	Thornton et al. 168		I				•	•	•		•	•		•	•	•
Prostate cancer	Davison et al. 169		I					•			•	•
	Gattellari and Ward 170	•	I		L	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•
	Hamm 171	•	D	Q		•	•	•	•	•	•	•	•	•	•	•	•	•	•
	Schapira and Van Ruiswyk 172		I	Q	T	•	•	•			•	•		•			•	•	•
	FIDM (videodisc) 173 , 174	•	I	Q	Q, L		•				•	•		•					•			•
	FIDM (Internet) 175		I	Q	Q, L		•				•	•		•					•
	Partin et al. 176	•	I	S, Q	Q	•	•	•	•		•	•	•	•	•	•			•
	Sheridan et al. 177	•	I	Q	L	•	•	•	•	•	•	•	•	•					•			•
	Wolf et al. 178		I		Q, L	•	•	•	•		•	•		•	•		•		•
Ultrasound	MIDIRS/NHS 71 , 72	•	D, I	Q	Q, L	•		•	•	•	•	•	•	•	•	•	•	•			•
Total (N)	30	9	7/27	11	19	19	22	22	15	13	25	28	10	20	12	9	10	8	25	5	4	6
Percentage		30	23/90	37	63	63	73	73	50	43	83	93	33	67	49	30	33	27	83	17	13	20

Source – DA, actual decision aid was viewed; Rep, type of report: D, development; I, implementation; Guide – Pt, patient sources: S, survey (quantitative method); Q, qualitative method; A, ad hoc; Exp, expert sources: S, survey (quantitative method); Q, qualitative method; L, literature; T, theory; O, other; Condition – Desc, description of the condition; Prob, probability of its occurrence; V, verbal descriptor (note: numeric report assumed if probability mentioned); Test details – Opt, optional tests presented; No Test, no test was explicitly provided as an option; Proc, procedure described; Ben, potential benefits of test; Harm, potential harms of test; Meas, what is being measured; Inacc, inaccuracy of measurement is mentioned; False+, false positives explained; False−, false negatives explained; Prob+, probability of false positives (note: numeric probability assumed if probability mentioned); Prob−, probability of false negatives (note: numeric probability assumed if probability mentioned); +Res, interpretation of a positive test result; −Res, interpretation of a negative test result; Evidence – Cite, citation list; Pt Exp, patient experience reported in a narrative.

Guidance to information selection

Both tables identify whether patient and/or medical expert guidance was sought during the development phase of the DA and the method by which it was obtained. As Table S1 shows, information included in treatment DAs has, for the most part, been guided by consultations with external groups, although nine of 68 (13%) of the treatment DAs did not report any such external consultations and an additional five of 68 (7%) reported being guided only by the literature and/or theory. Table 1 shows that external consultations were reported less frequently in the development of screening DAs, with 10 of 30 (33%) not reporting any external consultation and an additional three of 30 (10%) being guided by only the literature and/or theory. When an external group was consulted, DA producers reported relying more heavily on experts’ than on patients’ guidance: of the treatment DAs, 45 of 68 (66%) involved consultation with patients while 57 of 68 (84%) involved consultations with experts; and, of the screening DAs, 11 of 30 (37%) reported consulting patients while 19 of 30 (63%) reported consulting experts. All but two treatment DAs and one screening DA that reported consulting patients also reported consulting experts.

Both tables show the types of studies used in the consultations identified above: only five of 68 (7%) treatment DAs and one of 30 (3%) screening DAs reported using both qualitative and quantitative studies of patients. An additional four of 68 (6%) treatment DAs and no screening DA reported surveying patients without mentioning a qualitative study as underlying construction of the survey. An additional 35 of 68 (52%) treatment DAs and 10 of 30 (33%) screening DAs reported using only qualitative studies to obtain patient input, most commonly focus groups. In obtaining the views of experts about what information to include, only two of 68 (3%) treatment DAs and no screening DA reported using surveys built either on a qualitative study or on the literature. The most common method for obtaining external guidance from experts was from a small group (like a panel) which was used by 48 of 68 (71%) treatment DAs and by 12 of 30 (40%) screening DAs; most qualitative studies of experts was accompanied by guidance from the literature and a few were also guided by theory.

Content

While one might expect that issues related to obtaining external guidance would be very similar for treatment and screening DAs, issues related to actual content would differ in important ways. Thus, we review their content separately.

Treatment decision aids

As can be seen in Table S1, while 48 of 68 (71%) treatment DAs reported describing the condition, 39 (57%) reported including a description of how the condition would likely unfold over time. Similarly, although 59 (87%) reported identifying different treatment options, 37 (54%) reported describing the procedures involved with any of the options. Finally, almost all treatment DAs reported some description of benefits and harms. However, a detailed analysis showed that only 42 (62%) reported a description of the benefits and only 44 (65%) their likelihood(s), using either verbal or numeric labels. Reporting of harms was similar, with 63 (93%) mentioning something about harms but 44 (65%) providing a description of each potential harm/side effect and 45 (66%) providing a likelihood description.

Screening decision aids

As can be seen in Table 1, within screening DAs, 19 of 30 (63%) reported describing the condition being tested and 22 (73%) included the likelihood of the condition. Twenty‐two (73%) reported presenting options, and 15 (50%) mentioning the possibility of no test although only in some cases presented as an explicit option. Interestingly, 10 (33%) reported identifying what the test measured, although 20 (67%) reported acknowledging inaccuracies in the measurement. Providing details about measurement inaccuracies was reported less frequently: 12 (49%) mentioned defining false positives and, of those, 10 provided the rate; similarly nine (30%) reported defining false negatives with eight providing the rate. Interpretation of a positive test result, with or without describing possible treatment options, was reported in 25 (83%) of the DAs but only five (17%) reported interpreting a negative result possibly with a follow‐up strategy.

Information accuracy

Accuracy issues, similar in treatment and screening DAs, are reviewed together. In attempts to convey the quality of the study design that produced the data (and, hence, have an impact on how accurate the data are likely to be), a few DA producers developed their own simple rating scheme, such as medals (gold–bronze) to convey the different levels of quality ¹⁴ , ¹⁵ or explicitly marking the evidence that was consensus‐based ¹⁶ (i.e. not necessarily very accurate but the best estimate available). Others brought users’ attention to accuracy concerns by identifying particular information as being based on unclear evidence (i.e. Tamoxifin and BRCA 1/2 carriers, ¹⁷ herbal remedies and Hormone Replacement Therapy (HRT) and Alzheimer's ¹⁶ ). Some developers explicitly reported using verbal labels to convey outcome probabilities as a strategy to convey uncertainty around the estimate ¹⁸ while others reported providing numeric intervals around the estimates but then deleted them because patients did not consider them helpful. ¹⁶

Both tables further show which DAs reported including patient narratives/experiences, examples, or quotes. Table S1 shows that 29 of 68 (43%) treatment DAs and six (20%) screening DAs included some aspect of individual patient experience.

Finally, both tables indicate which DAs include a citation list, either as part of the DA or in an accompanying document. Table S1 shows that only 14 of 68 (21%) treatment DAs and four of 30 (13%) screening DAs included a list of citations. We did not find any other method used by DA producers to convey to their users how recently the data in the DA were generated.

Discussion

Guidance to information selection

Although most developers reported some type of external consultation in deciding what information to include in the DA, an appreciable portion did not include consultations with the patients they intended to assist. This is of concern for several reasons. First, patient concerns can be quite different from what a clinical model might predict (e.g. decision analytic models of treatment for early stage prostate cancer ⁹ ). Secondly, comparison of patient priorities to those of health‐care professionals suggests that they may differ significantly (e.g. patient ¹⁹ , ²⁰ vs. professional ⁸ , ²¹ ). Because experts and patients have unique but complementary experiences with the situation, patients facing a new health‐care situation would be best informed if they could learn from both. Finally, studies of information priorities within professional and patient groups suggest wide variability. The wide variability means that the priorities of a few individuals do not necessarily reflect the opinion of a larger group, thus, consulting only a few may be inadequate.

Concerns about the strength of evidence that pre‐occupy the ‘evidence‐based’ approach to treatments are also important when deciding what information to include in DAs. Qualitative studies are best to generate ideas about which issues should be addressed and to ensure depth of understanding about the issues. Quantitative studies to determine the prevalence of issues identified in the qualitative studies then ensure that the DA will address the most frequently appearing concerns. Although most rigorous, a qualitative–quantitative approach is very time‐consuming and only addresses part of the development process for DAs. A reasonable alternative is to build on existing studies of information priorities of the various relevant groups, if they exist, recognizing that cultural differences ²² and geographic variations in practice ²³ may limit the potential applicability of the published results to other groups. Quantitative studies are also valuable in determining if there is wide variability of concerns such that found both amongst professionals treating ⁸ and amongst patients with early stage prostate cancer. ⁹ , ²⁰ In such cases, the DA needs to be flexible in order to also address the less common but important concerns of individuals.

Decision aid content

This review suggests that the content within the DAs frequently does not reflect balance and may not be including some information that would be critical to informed decision‐making. We recognize that the review only captures what is reported about what has been included in the DAs; however, the imbalance in reporting raises the possibility of a basic orientation in the DA producers that allows such imbalances. For example, in the treatment DAs, only about 1/2 of those that identify treatment options actually report describing treatment procedures. Similarly, among the screening DAs, there appears to be more emphasis on false positives than on false negatives when describing test characteristics. It is interesting to note that although there are many more DAs now than when O'Connor et al. reviewed those used in randomized trials in 1999, ²⁴ their observation that slightly over half of the DAs provided outcome probabilities has not changed.

We reviewed whether or not screening DAs include test error as part of their content rather than as part of the accuracy of the information that the DAs provide. We did so because the notion that a test result may not be correct may be unexpected, thus, it is an additional idea that should be introduced to patients. Interestingly, in a study of information to be included in the informed consent process before Prostate‐Specific Antigen (PSA) screening, the experts thought information about false negatives should be provided while the patients thought information about false positives should be included. ²⁵

Information accuracy

Concerns about the accuracy of the information provided and how that accuracy is conveyed need to be weighed against the likelihood of overloading the patient with information that may be secondary to the task at hand. Some DA producers have attempted to reduce the cognitive burden of processing probabilities by using verbal labels. However, it is clear that verbal labels are understood differently by different people, ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ thus, this strategy of reducing the cognitive burden in fact reduces the accuracy of what is understood.

There is little empirical data to clarify the implication of presenting uncertainty around numeric estimates but studies of presenting quality of life information to patients suggesting that patients (i) understand the information without the uncertainty more accurately than when the uncertainty is included and (ii) prefer not to have the uncertainty included. ³⁹ , ⁴⁰ The attempt to convey uncertainty with the quantity estimates is similar to attempts to convey randomness with the quantity through, for example, the use of random faces. ⁴¹ In this situation, empirical study showed that trying to convey both messages at once compromises patients’ ability to understand the size of the quantity, ⁴² thus, it is not recommended. One potential solution is to label the numeric value as an ‘estimate’ but only present the uncertainty estimates to patients who request the additional information.

Providing a citation list is a strategy used by DA producers to convey how recently the data were generated. In addition, the list provides a means for those interested to search further. Attempts to limit the cognitive burden created by the information, which is secondary to the actual decision, have resulted in the list being provided as an adjunct to the DA.

Concern about how accurately patients understand their situations suggest caution about including patient narratives. Narratives have been promoted as a means to help people better understand health states they have not yet experienced, ⁴³ and evidence suggests patients like them in some contexts. ¹⁰ However, the narratives can have substantial impact but we do not understand why at this point. ⁴⁴ , ⁴⁵ , ⁴⁶ The ¹⁴ biasing effect has been demonstrated in systematic studies, ⁴⁷ , ⁴⁸ and in non‐health care contexts where merchandise sales are increased by including testimonials in infomercials. ⁴⁹ Additional methodological issues surrounding the use of testimonials include the use of outcomes collected retrospectively, which often biases interpretation, and the use of outcomes that are entangled in the larger context of the experience of the whole health‐care system. ⁵⁰ In spite of attempts to collect patient testimonials, ⁵¹ and even to create a database of them, ¹² patients themselves at times question their credibility: only 13% of health consumers sampled in the UK Judge Project for developing health website quality guidelines rated information written from personal viewpoints as being ‘very important’ to them. ⁵² The resulting guidelines developed from the project warn health consumers about interpreting personal experiences (http://www.judgehealth.org.uk/how_judge_personal.htm). The potentially biasing impact of narratives has led other guidelines and quality criteria to specify that their inclusion is undesirable (e.g. complimentary and alternative medicine, ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ diets, ⁵⁷ , ⁵⁸ hearing aid sales guidelines ⁵⁹ ). Thus, it seems desirable for DAs to avoid using patient narratives until their impact is better understood in order to ensure that their benefits can be achieved without distorting the patient's understanding of the situation.

Conclusions

The results suggest that DA developers frequently do consult both patients and experts but as a group they tend to rely more on the medical experts’ guidance for selection of information to present. Content evaluation suggests that DAs frequently omit describing the procedure(s) involved. While benefits and risks are often described, the likelihood of these events are often not identified. Sources of information, in terms of quality evidence citations, are rarely provided. While we recognize that the development and use of DAs is at an early stage of health adoption, attention to content completeness and accuracy of information while avoiding (even inadvertent) bias is likely to improve patients’ understanding of available health‐care options and the quality of their decisions.

Supporting information

Table S1. Treatment decision aids (word document).

Supporting info item