Evaluation of the quality of patient information to support informed shared decision‐making

Abstract

Objectives (a) To find out how much patient information material on display in family physicians’ offices refers to management choices, and hence may be useful to support informed and shared decision‐making (ISDM) by patients and (b) to evaluate the quality of print information materials exchanged during the consultation, i.e. brought in by patients or given out by family physicians.

Design All print information available for patients and exchanged between physicians and patients was collected in a single complete day of the office practices of 21 family physicians. A published and validated instrument (DISCERN) was used to assess quality.

Setting and participants Community office practices in the greater Vancouver area, British Columbia, Canada. The physicians were purposefully recruited by their association with the medical school Department of Family Practice, their interest in providing patients with print information and their representation of a range of practice types and location.

Main variables studied The source of the pamphlets and these categories: available in the physicians’ offices; exchanged between physician and patient; and produced with the explicit or apparent intent to support evidence‐based patient choice.

Main outcome measures The quality of the print information to support ISDM, as measured by DISCERN and the ease of use and reliability of the DISCERN tool.

Results and conclusions Fewer than 50% of pamphlets available in these offices fulfilled our minimum criteria for ISDM (mentioned more than one management option). Offices varied widely in the proportion of pamphlets on display that supported ISDM and how particular the physician was in selecting materials. The DISCERN tool is quick, valid and reliable for the evaluation of patient information. The quality of patient information materials used in the consultation and available in these offices was below midpoint on the DISCERN score. Major deficiencies were with respect to the mention of choices, risks, effect of no treatment or uncertainty and reliability (source, evidence‐base). Good quality information can be produced; some is available locally.

Introduction

Legal and ethical trends related to informed consent and autonomy require that patients be fully involved in making decisions about their health‐care management. Good decision‐making that takes into account medical best‐evidence and individual patient factors requires good quality information that includes detailed explanations about the patient’s condition, and risks and benefits of different treatment options. Patients generally seek and respect their physicians’ advice and opinions about health information more than any other single source, ¹ and the doctor–patient encounter is the ‘teachable moment’.

In a direct observational study of primary care Braddock et al. found that 48.6% of decisions were ‘basic’, e.g. about follow‐up appointments, routine laboratory tests or activities of daily living counselling. ² The others were classified as intermediate or complex and deemed to require some discussion of alternatives and the ‘pros and cons’ of these. Such discussion of alternatives, risks, benefits and uncertainties and exploration of patient preferences are essential components of informed shared decision‐making (ISDM). ³ However, there are barriers to the physician providing patients with the information they need if they are to make informed choices. In a typical primary care consultation, time for informing the patient is limited. Even family practitioners skilled in communication and up‐to‐date with the best evidence would be hard pressed to convey orally all the information which is required to enable a patient to share in an informed decision. Good quality written information that outlines management options could help overcome the problem.

However, the literature suggests that written information to support ISDM is poor in quality and quantity. In the United Kingdom (UK) a major King’s Fund study evaluated 54 patient information materials for 10 common conditions or treatments. The methodology included focus groups for patients with experience of the conditions, review using a structured checklist by clinical or academic specialists who were familiar with the research evidence, and a questionnaire to developers of the materials. ⁴ The materials evaluated in this study omitted relevant data, failed to give a balanced view of the effectiveness of different treatments and ignored uncertainties; few actively promoted a participative approach to decision‐making by referring to treatment choice. ⁵

Our incidental experience in British Columbia supported this finding. However, we had no evidence or systematic information that described the situation in our setting, i.e. family physicians in the greater Vancouver area, or that specifically examined the quality of print information used during the consultation.

The purpose of this study was (a) to find out how much patient information material available in family physicians’ offices refers to management choice and (b) to evaluate the quality, using a validated instrument, of print information materials exchanged during the consultation (i.e. brought in by patients or given out by family physicians). We purposely studied offices that were likely to represent good practice. Our hypothesis was that print information available and used in ‘typical good practice’ falls short of what patients and physicians need to support informed shared decision‐making.

Methods

Collection of patient information materials

Patient information materials were collected during the course of a 1‐day office study of each of 21 family physicians in the greater Vancouver area of British Columbia during summer 1999. The physicians in the study were associated with and identified through the University of British Columbia Department of Family Practice. They were told that the study was about information sharing and recruited because they said they provide their patients with written information. They were also selected for range of location (inner city to suburban) and practice type (clinic, group and solo). In each of the 19 different offices studied (three physicians shared a clinic), researchers recorded what patient information materials were on display or otherwise available and collected examples that mentioned more than one management option (of relevance to decision‐making). They also noted the arrangements made within the practice to provide patient information materials.

In order to study in more detail the information exchanged during the consultation, researchers administered questionnaires to patients and physicians in the same offices. One office was used as a pilot site to refine the methodology and the results are not included. Consenting patients – 287 of 372 who were approached (77%) – were asked to identify what print information they brought into the consultation and copies were obtained for evaluation. The physicians were asked to record what print information they gave to each patient, and what they wanted but did not have. The protocol was approved by The University of British Columbia Behavioural Research Ethics Board.

Selection of the evaluation instrument

Various instruments are available to evaluate the quality of consumer health information, but most work has focused on criteria such as readability or presentation. If information is to be used to support informed decision‐making it not only needs to be well presented, but also comprehensive and scientifically accurate. We identified two instruments that had been specifically designed to evaluate the quality of patient information materials to promote patient participation in treatment decision‐making. Both use scaled responses to items regarding the quality of information. The DISCERN instrument, developed by researchers at Oxford University, Oxford, England, is specifically designed to judge the quality of written consumer health information on treatment choices. ⁶ It is a 16‐item questionnaire, all graded on a five‐point scale, and the final question is an ‘intuitive summary’ of previous responses. Its use does not depend on specialist knowledge about the conditions or treatment options described. ⁷ The second instrument, the structured checklist for clinical‐academic specialists, developed by the King’s Fund, London, as part of the Materials for Informed Choice – Evaluation (MICE) study is intended to judge the scientific reliability of written information for patients. ⁴ It is composed of 18 items, including 14 scaled items, three qualitative items and one item with four sub‐items; there is no overall score. The instruments rate similar aspects of patient information. Both have been validated and published. Both were developed in the UK but do not contain any items that are likely to be specific to the UK health‐care system or sociocultural context.

Four patient information pamphlets were independently evaluated by each of three raters (investigators of this study) using the DISCERN and MICE tools. In addition, raters recorded the time taken to evaluate the information, their rationale for scoring and any items in the tools that were ambiguous. The results of the evaluations were compiled into tables. Items where there was no agreement between raters or where there was one outlier rating were discussed and guidelines developed for agreement.

Guidelines for DISCERN are that the overall rating of the publication as a source of information about treatment choices should be an ‘intuitive summary’. ⁶ However, the guidelines state, for example, that a High (5) should be given if ‘the publication rated high (4 or above) on the majority of questions’ and a Moderate (3) if ‘the publication rated high and low on a similar number of questions, or the majority of questions rated in the mid‐ranges’. The authors report that an expert panel achieved ‘acceptable reliability’ (Cohen’s kappa=0.53) with this intuitive summary. ⁷ Since these guidelines appeared to us to direct one to an approximation of the overall score we used an actual score from the sum of ratings for all questions.

On average DISCERN was quicker to use (average 10 min per pamphlet) than MICE (average 19 min). DISCERN appeared more reliable than MICE for each of the pamphlets evaluated: there were more items on which raters agreed and fewer on which they disagreed. The MICE checklist did not allow for a cumulative score and several of the items appeared to be intercorrelated; each of the DISCERN items appeared to be distinct. As the MICE checklist was specifically designed to be used by specialists (content experts), at times it was difficult for the non‐specialist to evaluate the quality of the evidence presented.

We concluded that DISCERN is the more reliable, valid and easy to use instrument for the evaluation of patient information designed for evidence‐based patient choice. Disagreements between raters on individual items could be resolved by reference to the accompanying handbook for the DISCERN instrument which gives detailed information about the meaning of each item and provides examples of materials which were rated high to low. ⁶

Training and calibration of raters to use DISCERN

Two students (in their final year of Bachelor’s programmes in science and pathology) were trained to use DISCERN. Following an orientation to its purpose and use, each student was given six pamphlets to evaluate. Pamphlets were selected to represent a range of quality and included examples which had been highly rated in the King’s Fund study. Each student rated the pamphlets independently and their scores on each item were compared. Differences of one point were discussed during initial calibration. However, they often could not be clearly assigned a single rank by reference to the guidelines and we averaged them to generate the final assigned score. Differences >1 (on a five‐point scale) were counted as a ‘disagreement’ and signified a problem with the raters’ interpretation of the guidelines. These were resolved by discussion to identify the reason for the disagreement, to clarify and, if necessary, re‐interpret the guidelines for scoring of the item and to agree a final score. The number of initial disagreements per pamphlet ranged from two to eight; overall there were 32 items in the six pamphlets that differed by >1. The same exercise was repeated with a further five pamphlets. This time the number of disagreements per pamphlet ranged from none to three and there were nine items (in the five pamphlets) that differed by >1.

Evaluation of information with DISCERN criteria

Three categories of materials were obtained during the office study and evaluated using DISCERN. Exchanged: information that was exchanged between patients and doctors in the office study (i.e. information brought in by patients or given to patients by physicians) (53 pamphlets). Wanted: print information that was wanted (but not available) by patients and doctors, as revealed by the office study questionnaires, that was available in other offices (14 pamphlets). Offers choices: a sample of the information collected from displays in the physicians’ offices that stated an explicit intent to assist decision‐making or that had an inferred intent though presentation (e.g. tabulated comparison of risks and benefits) (twelve pamphlets).

In addition, a fourth category of materials was selected and evaluated in order to provide a benchmark by which to judge the quality of the materials. This category was named EB Patient Choice. It comprised a sample of materials from the UK that was specifically designed to help evidence‐based patient decision‐making. These were pamphlets identified as superior examples in the King’s Fund study or produced by organizations with explicit intent for informed patient choice. Of the eight pamphlets in this category, six were evaluated as part of the training and calibration process but the scores reported in the results are the final scores following discussion and agreement with reference to the guidelines.

Results

Quality of pamphlets on display in offices

A total of 663 pamphlets were recorded from the 19 offices (range 10–117 per office; 15 offices had <40; two had >100). Of these, 298 (45%) fulfilled our minimum criteria for supporting ISDM, i.e. mentioned more than one management option; 365 (55%) did not (Table 1). The proportion of pamphlets in each office which at least provided this minimal support for ISDM ranged from 14% to 79%. In five offices more than 50% of the pamphlets supported ISDM and two offices had more than 75%.

Table 1.

 The source, number and relevance to informed shared decision‐making (ISDM) of all patient information materials collected

About 80% of the information on display was supplied by commercial companies (usually pharmaceutical), government and professional organizations (Table 1). The most frequently available pamphlets that supported ISDM were supplied by commercial companies, although the information was frequently noted to be biased, and professional organizations. The most frequently available pamphlets that were not relevant to ISDM (e.g. were designed to describe a particular service or product) were supplied by government. In the professional organizations category, two suppliers (British Columbia Medical Association and College of Family Physicians of Canada) produced the majority of leaflets available.

Nine of the physicians said they were not particular about what information was available for patient pick up: the materials were acquired by unsolicited mailing from the suppliers or by being left by representatives of suppliers and were usually put on display by clerical staff in the office. Nine of the physicians said they were very particular about what was available and were involved in the selection of materials on display.

Reproducibility of the DISCERN instrument

The 53 pieces of information that were exchanged between doctor and patient were scored by both raters following training and calibration. Forty of 53 pamphlets (75%) had ≤2 disagreements (difference in score of >1) on the 16 items. Five pamphlets had four or more disagreements.

The overall interrater reliability (before discussion of disagreements and averaging of items differing by 1 point) for the exchanged pamphlets (n=53) after calibration was 0.74 (Spearman’s rho). The final score assigned each pamphlet was refined by discussion of items and reference to the explanatory guidelines when the raters differed by >1 point.

Problems with DISCERN

Of the 53 exchanged pamphlets evaluated with DISCERN, 15 pamphlets yielded a disagreement (initial difference between raters of >1) on Question 14: ‘Is it clear that there may be more than one possible treatment choice?’. Ten pamphlets yielded disagreements on Question 9: ‘Does it describe how each treatment works?’. On all the other questions there were ≤8 pamphlets for which the raters initially disagreed.

Quality of information used in the consultation

The possible total scores on DISCERN are from 16 to 80; the midpoint is 48. For the Exchanged material 87% scored midpoint or below (Fig. 1). The highest score was 59 for The Asthma Handbook (BC Lung Association). The lowest was 24 for Visit 1: Instructions for the Patient (Parke Davis & Pfizer) (preparation for blood cholesterol measurement).

Figure 1.

 The quality of printed consumer health information as measured by the DISCERN total score of 16 items each with a range of 1–5. The box plot shows outliers and percentiles: 90, 75, median, 25 and 10; dotted line = mean. Exchanged (n=53) was information exchanged between physician and patient during the encounter. Wanted (n=14) was information physicians or patients wanted, did not have, but was available in another office. Of fers choices (n=12) was information, culled from the whole collection of 663 pamphlets, which apparently was intended to provide treatment alternatives and choice. EB Patient Choice (n=8) was information published in the UK that explicitly intended to assist evidence‐based patient choice.

Material that physicians and patients expressed a need for that was available from other offices (Wanted) was of a similar quality. By comparison, the material selected that was apparently intended to support choice (Offers choices) scored significantly higher (Fig. 1).

Upon item analysis of all 92 publications evaluated (including calibrators) only four of 16 items had mean scores ≥ 3.0 (midpoint or above). These items related to the following questions:

• Aims clear?

• Achieves its aims?

• Relevant to consumers?

• Describes the benefits of treatment?

The following questions were the lowest scoring:

• Date of information explicit? (mean score 1.9)

• Describes what would happen without treatment? (mean score 2.0)

• Sources of information explicit? (mean score 2.1)

• Refers to areas of uncertainty? (mean score 2.2)

• Describes the risks of treatment? (mean score 2.2)

No publication scored higher than mid‐point (3) on the ‘no treatment’ options question. For all other items the highest score (5) was achieved by at least one publication.

Discussion and conclusions

Use of DISCERN

DISCERN has good interrater reliability. It is easy to train raters, fairly quick to use and is appropriate for the evaluation of consumer health information produced outside the UK. There are a few items that consistently caused problems and required more guidance than was available in the handbook. DISCERN was specifically designed to address treatment choices. Though the DISCERN handbook says that ‘Publications describing one particular treatment choice can be acceptable’ it also notes this is true only ‘if the author has acknowledged that other treatment choices may be available’. Our experience indicates that the particular question most related to this intent (Question 14: Is it clear that there may be more than one possible treatment choice?) caused the greatest difficulty in rating when applied to pamphlets not specifically aimed at treatment choice. These pamphlets sometimes included investigation options (and we extended the guidelines to include these as valid ‘choices’). Some pamphlets were apparently designed to provide patients with information in the form of lists, e.g. various treatments – diet, exercise, medication — would be named but no indication given to weight that information. It is not clear to what extent such mere ‘mentions’ constitute choices and whether they ought to be considered as offering a choice between alternatives or whether patients are expected to follow all the treatments. Thus the finally agreed score for Question 14 on such pamphlets was usually only arrived at after some debate. Nevertheless, it should be noted that a piece of patient print information could have a fairly high overall score by careful attention to the DISCERN criteria that are not choice‐specific.

Questions that tended to score high related to aims and benefits. Questions that tended to score low related to information about risks, uncertainty, no‐treatment options and the evidence‐base for this information. These are areas where the materials we looked at tended to fail and they are important deficiencies also identified by patients. ⁵

Quality of information

The broad criteria used in the DISCERN tool are: ‘Is the publication reliable (up to date, evidence‐based)?’ and ‘How good is the quality of information on treatment choices (includes risks and benefits, consequence of no treatment, support for shared decision‐making)?’. In general the written information for patients that was exchanged and available in this study rated fair to poor with respect to these criteria. Items most deficient are about decision‐making. Even our substantially less rigorous criterion of ‘mentions more than one management option’, shows that the majority of information available in physicians’ offices does not obviously intend to support informed and shared decision‐making.

This was a snapshot study of physicians who said they used printed patient information. The quality and amount of information they had cannot provide an estimated average that is generalizable to all physician practices, though the range in amount available is similar to that seen in the study of family practices by McVea et al. ⁸ We expect that we have seen a better quality of patient information in this study than one would find in the general population. The physicians were associated with an academic Department of Family Practice, were informed about the nature of the study and were recruited on the basis of their expressed interest in information sharing with patients. They also represented a range of practice types and geographic locations.

Good quality pamphlets can be produced. Some examples from the UK which we used as benchmarks are Understanding Coronary Heart Disease (British Medical Association, London, England) and The Total Hip Replacement Operation, Questions and Answers (Royal College of Surgeons, London, England); both scored > 65, i.e. in the top quartile. Some reasonably good quality information produced in Canada was available in the offices we studied although these only occurred in single offices: Enjoy Life Through the Menopause (Menopause Clinics of Saint‐Luc and Ottawa General Hospitals, Ottawa) scored 69, and Facts on Erectile Dysfunction (Janssen‐Ortho Inc., Toronto) scored 62.

Recommendations

Organizations and agencies that currently produce or fund the production of pamphlets should institute guidelines (for example based on the criteria used in the DISCERN tool) to improve the usefulness of their products. Attention to a few fundamentals, especially indicators of reliability (e.g. dates and sources of information) and discussion of uncertainty and risks could make a big difference.

Acknowledgements

This work was supported by grants from the British Columbia Medical Services Foundation and The Max Bell Foundation. Medical students, Tanya Orton and Niriksha Chand, played a major role in recruiting physicians and diligently collected the pamphlets and data in their offices. Greg Doho and Sukhwinder Singh Sangha read and evaluated the many pamphlets; Catherine Elliott organized and analysed their data. Dr Duncan Etches was especially helpful by providing his counsel and his practice for the testing of procedures and questionnaires.