Assessing decision quality in patient-centred care requires a preference-sensitive measure

Abstract

A theory-based instrument for measuring the quality of decisions made using any form of decision technology, including both decision-aided and unaided clinical consultations is required to enable person- and patient-centred care and to respond positively to individual heterogeneity in the value aspects of decision making. Current instruments using the term ‘decision quality’ have adopted a decision- and thus condition-specific approach. We argue that patient-centred care requires decision quality to be regarded as both preference-sensitive across multiple relevant criteria and generic across all conditions and decisions. MyDecisionQuality is grounded in prescriptive multi criteria decision analysis and employs a simple expected value algorithm to calculate a score for the quality of a decision that combines, in the clinical case, the patient’s individual preferences for eight quality criteria (expressed as importance weights) and their ratings of the decision just taken on each of these criteria (expressed as performance rates). It thus provides an index of decision quality that encompasses both these aspects. It also provides patients with help in prioritizing quality criteria for future decision making by calculating, for each criterion, the Incremental Value of Perfect Rating, that is, the increase in their decision quality score that would result if their performance rating on the criterion had been 100%, weightings unchanged. MyDecisionQuality, which is a web-based generic and preference-sensitive instrument, can constitute a key patient-reported measure of the quality of the decision-making process. It can provide the basis for future decision improvement, especially when the clinician (or other stakeholders) completes the equivalent instrument and the extent and nature of concordance and discordance can be established. Apart from its role in decision preparation and evaluation, it can also provide real time and relevant documentation for the patient’s record.

Background

The increase in the range of options available for health and disease management, coupled with the shift towards greater patient involvement in recent years, has led to a profusion of decision aids and related support systems aimed at patient, clinician and medical team.^1–3 This has been followed by the development of instruments and checklists to assess the quality of such aids. Most have focused on the internal quality of the particular decision aid as appraised by a set of normative criteria, with the International Patient Decision Aid Standards instrument (IPDASi)⁴ emerging as the most prominent of such checklists. Where empirical evaluation has been undertaken or is proposed, most attention has been on particular process and outcome aspects (e.g. acceptability, involvement, conflict, knowledge), rather than on the comparative performance of aided and unaided decision making in relation to overall decision quality. (A list of evaluation measures is available on the Ottawa Patient Decision Aids website, http://decisionaid.ohri.ca/eval.html).

None of the clinical trials included in the Cochrane systematic review of decision aids has evaluated the quality of the decisions derived from different decision technologies (including decision aids) using a single index measure of overall decision quality. There is therefore a need for evaluation measures that address the overall quality of decisions, as distinct from measures that address particular aspects of decision making.

Adopting the Berwick⁵ philosophy of patient-centred care implies that all decisions should be regarded as preference-sensitive, with the relative importance a patient attaches to various outcomes and processes having a large influence on, if not determining, what is decided. This philosophy also necessitates that decision quality be regarded as preference-sensitive and that the relevant preferences are those of the patient facing the decision, as opposed to the average preferences of a group of patients with the same condition or those of the health professional(s) involved in the decision.

In the case of drugs or medical devices the main purpose of the intervention is to achieve a health benefit. It follows that the primary outcome of intervention research is to determine its effectiveness in accomplishing that task. We suggest that, in the case of decision aids, the main purpose should be to enhance the overall quality (‘goodness’) of the aided decision relative to that which can be achieved without it – by some other ‘decision technology’ – and that this should be assessed at the time of decision making and not in terms of any subsequent change or outcome.⁶ Hence, decision quality should become the primary outcome of studies comparing decision-aided and unaided practices. This essay therefore argues for the use of decision quality as a directly measurable patient-reported measure for all health conditions. It could be regarded as a Patient-reported outcome measure if the decision is regarded as an outcome of the decision-making process⁷ or, alternatively and more conventionally, as a patient-reported experience measure.⁸ While policy makers and researchers may benefit from a generic measure, the fundamental reason for it is so that the individual is able to assess the quality of the various health-care decisions they face in, and through, their life, using the same instrument, whatever the context, condition, timing in life or role occupied. We describe and illustrate the application of the MyDecisionQuality (MDQ) instrument to this task.

The development of MDQ followed an assessment of the available instruments for evaluating decision aids – mainly on the Ottawa website – which was designed to establish whether any of these instruments generated a generic and preference-based index of overall decision quality; as opposed to ones that were: (i) condition-, setting- or decision-specific; or (ii) measured one or more possible aspects of decision making such as preferred involvement in decision,⁹ satisfaction with the decision¹⁰ or decision conflict experienced,¹¹ rather than overall decision quality or (iii) did not weight their components to produce an index measure (i.e. were profile instruments) or, if they did enable weighting, did not elicit weights from the specific patient on the specific decision occasion.

None of the instruments identified in the search constituted such a personalized preference-based measure of decision quality. The only instruments uncovered that used the label ‘decision quality’ per se were those developed by Sepucha and colleagues.¹² Their condition- and decision-specific decision quality instruments (DQIs) include items that assess: (i) knowledge – the extent to which the patient was ‘well informed’, (ii) concordance – the level of agreement between the patient’s goals and concerns and their treatment and (iii) involvement – the extent to which the patient was involved in decisions about their care. In addition to the fact that they are not generic, these DQIs are not preference-based. The scores that are produced relate to particular segments of the DQI and are not aggregated, by weighting, into a single overall index measure of decision quality for the individual patient. Moreover, the score for the concordance component is calculated only for patient populations, not for any specific patient.

We do not regard the Decisional Conflict Scale¹¹ and its 4-item form SURE¹³ as measures possessing construct validity in relation to decision quality because they penalize decision processes and support systems that leave the decision maker in a state of warranted equipoise.

Decision satisfaction is not an appropriate measure for assessing the quality of a decision aiding or making process, and will often reward suppression of uncertainties rather than their expression.¹⁴ For example, item two of the widely used Satisfaction with Decision instrument (‘The decision I made was the best possible choice for me’) makes it unsuitable in the evaluation of decision quality because it denies the possibility that there may be two or more best possible choices.¹⁰ We regard patient empowerment as incompatible with the notion that the role of the physician is to provide a confident and single recommendation.

Defining decision quality

We take the view that both the definition and measurement of decision quality should be treated as preference-sensitive. Accordingly, in principle the measurement of decision quality will require the decision owner to choose which criteria to include in the instrument. A DQI based on a set of consensus-led items and weights (including the equal weights implied in most checklists) seems incompatible with truly personalized patient-centred care. While Edwards and Elwyn¹⁵ raise a number of legitimate issues concerning the operationalization of decision quality, many of these stem from their view that a ‘good decision’ exists, but is yet to be defined. In contrast, we take the view that decision quality – defined tautologically as the goodness of a decision – does not exist and should not be defined in a positivistic way. ‘Decision quality’ is a multicriterial construct and all we can do, given the necessity to assess it, is to propose a set of items that appeal to our – and others’ – value judgements as to what should be included. (In this respect it parallels constructs like ‘health-related quality of life’ that instruments such as EQ-5D simultaneously define and measure.) Beyond this immediate challenge, the next task that any one constructing a DQI faces is operationalizing the measurement and synthesizing its components. We interpret Edwards and Elwyn as agreeing with us that abdicating from this task because of the substantial operational challenges it poses is not an option, and we regard ‘subjective’ numerical calculation as a vital complement to their ‘subjective’ verbal deliberation approach in responding to these challenges.

Developing a personalized and generic decision quality measure

MDQ is a web-based generic instrument for the individual-specific measurement of decision quality based on Multi-Criteria Decision Analysis (MCDA).

Dolan recently explored the potential of MCDA as the basis for decision support systems in health care, including Shared Decision Making (SDM).^16,17 He outlined the current portfolio of multicriteria methods and commented on their respective merits and problems. His assessment led him to favour the Analytic Hierarchy Process (AHP) because it offered greater flexibility, ease of use and strength of measurement compared with the other methods. However, he acknowledged that one of the main problems with using the AHP in clinical decision support is that its pairwise comparison elicitation processes, the main source of its strength in measurement, is both time consuming and cognitively demanding.

In conjunction with the online survey program called Elicia© in which it is embedded, Annalisa© forms an interactive, online decision support template that was explicitly designed to make MCDA-based decision support less temporally and cognitively demanding.¹⁸ It was not developed solely with applications in the health-care setting in mind, but it was designed to be the basis for tools which were practical in both time and resource-pressured situations – as well as relaxed ones. Annalisa meets one of Bates et al.’s¹⁹ commandments for the satisfactory delivery of a decision aid in stressing visualization and presenting all aspects of the decision (preferences, evidence and options) on a single screen. Annalisa-based decision aids provide a structured analytical framework for decision deliberation and hence for the balancing of the two main contributors to decision making – analytical modelling and intuitive judgement.²⁰ Above all, Annalisa tools seek the requisite balancing of normative rigour and operational practicality.^21,22 A study involving an Annalisa tool for prostate cancer screening has confirmed its ease of use.²³

MDQ is a dually personalized DQI based on MCDA and currently implemented in Annalisa©, though in principle it could be implemented in any form of online spreadsheet. (By saying MDQ is based on MCDA we mean that there is always an implicit alternative decision process (option) with which the MDQ result for the currently implemented option should be compared.) The assessor (e.g. patient) is responsible not only for (i) weighting the criteria of decision quality in terms of their relative importance but also (ii) rating the quality of a decision just made on the criteria. MDQ is generic in the sense that the criteria are phrased without reference to any particular decision or context. Information relating to the specific decision (such as one in a particular health-care setting and population) is to be provided outside the MDQ instrument, but in the larger decision support system in which MDQ will often be situated.¹⁸

As with all implementations of the simple ‘weighted-sum’ version of MCDA, MDQ combines a set of importance weights for multiple criteria with performance ratings for each option on these criteria and calculates the overall score as the expected value of these components. The patient’s weightings for the eight criteria of decision quality are elicited as early as possible in the decision-making process and their ratings on how well the decision made performed on these criteria as soon as possible after it was made. The MDQ Score, unique to the patient and to the particular occasion, is automatically calculated as the summed multiplication of criterion weightings and ratings. A worked example is provided in Figure 1.

Figure 1.

Screen capture of Annalisa presentation of MDQ for patient PCS2880.

Both MDQ and the decision aids developed in Annalisa are accessible via the internet from any operating system, browser and on mobile tablets (iPad, Android). The resulting summary picture of the decision quality assessment (showing Weightings, Ratings and MDQ Score) can be printed and/or downloaded as an image for later use, including sharing and formal clinical documentation.

The desire to make MDQ practical in pressured situations such as a health-care clinic determined the number of criteria included. The number that an individual could realistically be asked to weight and rate at the time of decision making was initially set at 10. The review of the most commonly used instruments in relation to patient involvement and participation in health decision making helped us to generate a list of candidate criteria. This list was reduced to 10 on the basis of either conceptual redundancy or inappropriateness for inclusion in a universal (i.e. not specifically health) decision quality measure. These 10 included six items which remained when it was later decided that eight was the maximum practical number for a user to weight and rate at the time of decision making and hence the maximum number of items to include in a decision quality measure. This number is within Miller’s magical number seven plus or minus two and is endorsed in the Cochrane Handbook.²⁴ The shorthand labels for these six criteria are: ‘Options’, ‘Effects’, ‘Importance’, ‘Trust’, ‘Control’ and ‘Commitment’. Of the remaining four items in the original 10, an Uncertainty criterion was subsumed in a ‘Chances’ criterion and an Emotional Support item in a general ‘Support’ criterion.

Of the eight criteria in the current version of MDQ (Figure 2), the first four match the structural requirements for any MCDA implementation in any context (Options, Criteria, Weightings and Ratings). These criteria also appear, in one form or another, in all checklists for developing decision aids for health decisions, including IPDASi. The last four criteria relate to other aspects of the decision process and are also explicitly or implicitly included in most checklists for decision aids. The ‘Commitment’ criterion creates for investigating concordance at the point of decision and correspondence with future actions and outcomes.²⁵

Figure 2.

Screen capture of MyDecisionQuality weightings items (Ratings items are rephrased in past tense). Scale: 0 = Not at all important, 5 = moderately important and 10 = extremely important.

As with all the existing instruments referred to earlier there is no intention in MDQ to capture or assess the subjective experience of the patient (fear, anxiety, etc.). The patient expresses their views as to the support they received in relation to their feelings and emotions – and all other aspects of the decision experience – by their weighting and rating on the Support criterion.

After discussions of the provisional eight items with immediate colleagues, we uploaded an online survey incorporating the initial MDQ on the Facebook page of the SDM Group, and emailed an invitation to comment to those on the lists of the Society for Medical Decision Making (SMDM) and Society of Judgment and Decision Making in mid-December 2011. Allowing for crossover, we estimated that this provided us with a few hundred potential respondents. Twenty individuals completed the questionnaire (latest in mid-January 2011) and nine also provided comments on the MDQ screen. Their feedback was incorporated in the re-development of the MDQ, when it was compatible with the underlying framework and construct.

Figure 1 shows the Weightings component of MDQ as now adopted. The equivalent Ratings component of MDQ is not presented here, but it is identical to the Weightings except that the descriptions are phrased in the past tense (e.g. ‘I was clear about the Options available to me and the processes they involve’).

While it is possible to have Users enter their weightings and ratings directly into Annalisa, early testing of MDQ using a convenience sample of academics showed that it was easier for their data to be elicited on a 0–10 scale in the survey program (Elicia) within which the Annalisa for MDQ is embedded. The responses were then mapped directly on to a 0–1 scale and ported into Annalisa using the software bridge between the two programs.

There are always tensions between what can be expected from development of a practice-relevant tool and one that is also used for research. Since the first (Weightings) part of MDQ is designed to be administered as early as possible in the decision-making process it constitutes an intervention in itself, whether or not any other intervention (e.g. decision aid) is involved before the Rating part is administered. It is not at all clear what should be the primary outcome, in terms of decision quality, of a trial of MDQ-supplemented decision making and standard practice. The challenge of validating a patient-specific, preference-based instrument such as MDQ does not appear to have been addressed in the literature thus far and we continue to seek assistance in this respect. Given the personalized character of MDQ, we are particularly interested in exploring the use of N-of-1 study designs.²⁶

Illustration of the application of MDQ

Figure 1 presents the MDQ screen of a patient from a randomized controlled trial (RCT) using two Annalisa-based decision aids for prostate cancer screening in Australia.²³ The Weightings shown are as they were entered in Elicia in non-normalized format, that is, they do not add to one. They are normalized in Annalisa to add to one as the Score calculation is always using normalized weights.

After seeing the MDQ Score, the patient has the option of viewing its breakdown into the contributions made by each criterion, that is, their ‘part-worths’, to establish whether, and if so how, they might be able to improve their MDQ Score in the future, provided the opportunity exists. In Figure 3, the MDQ Score for the patient is partitioned into eight segments. If a segment is relatively large then the criterion is making a larger contribution to the individual’s MDQ Score. On the other hand, if it is relatively small then it is making a smaller contribution to their MDQ Score. By placing the cursor over any segment of the MDQ score bar, the relevant criterion label and the weightings and ratings for it are highlighted (column 1 in Figure 3; video demonstration at http://bit.ly/17yKWNm). If the patient has assigned the criterion a small weighting but also a low rating there is no need for concern. However, if they have assigned the criterion a small rating but a moderate or large weighting, they may want to think about how they might change things for the better, for example, by prioritizing the seeking of more information about the effects, more value clarification about criterion importance, moving towards their preferred level of control or whatever the criterion relates to.

Figure 3.

Screen capture of Annalisa presentation of MDQ for patient PCS2880 showing MDQ Score breakdown and highlighting of part-worth, weighting and rating for Options.

This feedback can be formalized in the form of the Incremental (Expected) Value of Perfect Rating (IVPR) (see Figure 4). The IVPR for a criterion indicates the amount by which the overall MDQ Score could be increased if, given the respondent’s weighting of a particular criterion, they achieved a (perfect) rating of one. In the case of this patient, this would suggest priority being given to improving the Rating for Options, followed by Effects and Support.

Figure 4.

Incremental (Expected) Value of Perfect Ratings for each criterion for respondent PCS2880, shown as priority distribution. The longest bar indicates biggest gain from perfect rating.

Discussion

As we move towards patient-centred care it is important that we respond positively and wholly to patient heterogeneity in the value aspects of decision making.²⁷ Developing a portfolio of instruments to evaluate the overall quality of decisions in a transparent and preference-sensitive manner is a growing area of research. MDQ is offered as one that may be able to contribute to this process.

Given its preference-sensitive nature how might MDQ be used to advance patient-centred research? Patients are heterogeneous in both their biophysical profile and their preferences. Because of the preference-sensitive nature of the instrument, the usefulness of average results (Weightings, Ratings and MDQ Scores) from MDQ depends on the purpose of using it in the first place. If a patient uses MDQ to assess the quality of a decision made after using an online decision aid and prints out the MDQ screen, this may be used during a clinical consultation follow-up to form the basis of a tailored conversation about particular aspects of the decision. In this case, it is only the individual patient’s Weightings, Ratings and Score that matter. On the other hand, for an RCT involving simple and complex versions of a decision aid, average MDQ scores may be used to compare decision quality across the two arms of the trial. However, these results still need to be carefully interpreted in light of the personalized criterion weightings they embody. In principle, the average results matter most where health-care resources are being consumed or a relevant group- or population-level policy decision is to be taken.²² In this respect, cluster and latent class analysis²⁸ may be used to establish meaningful patterns of preferences in the community,²⁹ leading to an ‘epidemiology of preferences’.

In relation to the Annalisa-based decision aids we are developing, we envisage an alternative route where the individual selects the criteria most important to them from a longer list rather than having a set of criteria provided by researchers. We refer to this as a ‘Pick Your Own’ version of a decision aid. This version formed one of the arms of the clinical trial involving two decision aids for prostate cancer screening.³⁰ A ‘Pick Your Own’ version of (MDQ) captures the essence of person-centred care but it is essential that any such alternative patient-reported MDQ measure retains its theoretical and prescriptive basis in MCDA.

Declaration of Conflicting Interests

Annalisa and Elicia are both © Maldaba Ltd (www.elicia.org.uk). JD has a financial interest in the Annalisa software but did not benefit from its use in the study from which the illustrative data are drawn.

Funding

MMK's PhD study is funded by the Region of Southern Denmark, the University of Southern Denmark and The Health Foundation (Helsefonden). The contribution of GS and MC was supported by the Screening and diagnostic Test Evaluation Program (STEP) funded by the National Health and Medical Research Council of Australia under program grant number 633003.