Impact of Timing on Measurement of Decision Quality and Shared Decision Making: Longitudinal Cohort Study of Breast Cancer Patients

Abstract

Purpose.

The objective of this study was to examine whether scores of shared decision making measures differ when collected shortly after (one month) or long after (one year) breast cancer surgical treatment decisions.

Methods.

Longitudinal, multi-site survey of breast cancer (BC) patients, with measurements at 1 month and 1 year after surgery at four cancer centers. Patients completed the BC Surgery Decision Quality Instrument (used to generate a knowledge score, ratings of goals, and concordance with treatment preferences) and Shared Decision Making (SDM) Process survey at both time points. We tested several hypotheses related to the scores over time, including whether the scores discriminated between sites that did and did not offer formal decision support services. Exploratory analyses examined factors associated with large increases and decreases in scores over time.

Result.

Across the four sites, 229 patients completed both assessments. The mean total knowledge scores (69.2% (SD16.6%) at 1-month and 69.4% (SD17.7%) at 1-year, p=0.86), SDM Process scores (2.7 (SD1.1) 1-month versus 2.7 (SD1.2) 1-year, p=0.68), and the percentage of patients receiving their preferred treatment (92% at 1-month and 92% at 1-year, p=1.0) were not significantly different over time. The site using formal decision support had significantly higher knowledge and SDM Process scores at 1-month, and only the SDM Process scores remained significantly higher at 1-year. A significant percentage of patients had large changes in their individual knowledge and SDM Process scores, with increases balancing out decreases.

Conclusion.

For population-level assessments, it is reasonable to survey breast cancer patients up to a year after the decision, greatly increasing feasibility of measurement. For those evaluating decision support interventions, shorter follow-up is more likely to detect an impact on knowledge scores.

Over the last decade, there has been growing interest in and endorsement of shared decision making (SDM) as a way to enhance patient-centered care and improve healthcare quality.^1-3 The National Quality Forum in the United States has identified SDM as a key measurement gap and in March 2018, released a playbook entitled, “Shared Decision Making in Healthcare” complete with recommendations, case studies, and a dedicated section on tracking, monitoring, and reporting.⁴ Specific recommendations regarding SDM measurement include tracking the distribution and use of decision aids; documentation of SDM-related patient and clinical outcomes; and measuring patient decision quality (e.g., knowledge, involvement, and preferences). Additionally, systematic SDM data collection and reporting is mentioned as a performance indicator for key service lines.

Current tools for measuring SDM include, but are not limited to, observation and coding of clinical encounters by independent raters, dyadic approaches that query both the physician’s and patient’s perspective, and patient-focused approaches that assess the patient’s experience via surveys.^5-7 Observational approaches and the physician measures are expensive to collect and analyze and are often only feasible as part of funded research studies. Patient-reported measures may be more easily integrated for use as a SDM performance measure or as part of quality improvement projects.

Although there are several patient-reported measures of SDM, there is gap in evidence regarding the validity of measures. A key aspect of validity for SDM measures is the ability to discriminate among sites or clinicians who do and do not provide decision support.^7-10 Another critical challenge in measurement of SDM is that for many decisions, reliable identification of patients is only feasible after treatment has occurred.¹¹ However, surveying patients after they have received treatment may result in biases for both knowledge and preference assessments. For example, studies suggest that patients do not retain information, and may only retain the gist without the details, implying assessment of knowledge may also depend on timing.^12,13 More research is needed to determine how health-related knowledge “sticks” and in particular, which aspects of knowledge are maintained over time versus being transient.

Breast cancer is a common disease that requires several major decisions. For the majority of women with invasive breast cancer, mastectomy (with or without radiation) and lumpectomy with radiation are reasonable options for local treatment.¹⁴ The decision about breast cancer surgery is considered preference sensitive because each option has differing risks, benefits, tradeoffs, and recovery considerations; yet, the survival rates are the same.^15,16 Accordingly, women should be informed of the options and the pros and cons of each, ideally with a shared decision making process. ^17,18

The goal of shared decision making is to ensure a high quality decision, or one that is informed, and reflects the patients’ values and preferences.¹⁹ The Breast Cancer Surgery Decision Quality Instrument (BCS-DQI) is a patient reported survey developed to assess the degree to which breast cancer patients are informed about their surgical options and received treatment that matches their goals. ^20-22 The Shared Decision Making (SDM) Process survey is a short, patient reported survey that measures the involvement of patients in the decision making process. The objective of this study was to examine the discriminant validity of the BCS DQI and the SDM Process scores and to examine how the scores are impacted if women with breast cancer were surveyed shortly after (one month) or long after (one year) surgical treatment decisions. This research is important because it will inform the validity and stability of patient-reported decision quality measures over time, which may in turn have implications for the feasibility of measurement efforts.

METHODS

The study follows STROBE guidelines for observational studies.²³ The purpose of this study was to gather evidence of the psychometric properties of the BCS DQI. A longitudinal survey of breast cancer (BC) patients was conducted at four cancer centers in the United States from February 2010 through February 2011. Each site is a National Cancer Institute-designated comprehensive cancer center, two are located in the Northeast, one in the Southeast and one on the West coast. Staff screened surgical schedules to identify potentially eligible patients and obtained permission to contact patients from the treating surgeon. Eligible subjects received an initial packet by mail within one month after their surgery. The packet included a consent form, a survey and a small incentive (booklet of stamps). Subjects were able to opt-out if they were not interested in participating. All subjects who did not opt out received a reminder call two weeks after the mailing and a reminder packet that included another copy of the survey at four weeks. Respondents were surveyed again about 1 year later following a similar protocol.

One site (site 4) had a formal decision support program in place that provided patient decision aids to new patients in advance of the initial surgery visit and offered decision coaching.²⁴ The coaches (trained student interns) identified patients from clinic schedules, mailed a copy of the decision aid, called patients to remind them to review the decision aid, and called them back to type up their questions and concerns in advance of the visit. A coach then accompanied the patient to their appointment, typed up notes and made an audiotape of the visit. Each patient received a copy of their question list, the coach’s visit notes, and the audio-recording. No other site used formal patient decision aids or coaching.

Patients were eligible for inclusion if they were adult females with a diagnosis of Stage I-III breast cancer within four weeks of their definitive surgery. At one site (Site 4), patients also needed to have received either a patient decision aid or decision coaching as part of their care to be considered eligible. Women with DCIS only, metastatic (Stage IV) disease, those unable to read or speak English or those with confirmed bilateral breast cancer, prior radiation or neoadjuvant therapy were excluded.

Each site sent copies of de-identified surveys to the [site name redacted] for centralized data entry. Each site obtained human subjects approval from its ethics committee. Analyses were conducted by a statistician who was not involved in the data collection using SAS version 9.4 (SAS Institute, Cary, NC) and a two-sided p value of 0.05 or less was considered as statistically significant.

Limited data were collected on non-responders including age, stage at diagnosis and definitive surgical treatment.

Measures

Demographics, cancer stage and treatment history were collected by patient report and confirmed via medical chart review. The following surveys were administered at 1 month and 1 year.

BC Surgery Decision Quality Instrument (research version): includes 10 decision-specific, multiple-choice knowledge items and 2 open-ended knowledge items, 5 decision-specific goals and concerns rated on an importance scale from 0 (not at all important) to 10 (extremely important) and 1 treatment preference item. The DQI was developed with considerable input from patients and providers^25,26 based on the conceptual framework of shared decision making.^27,28 Both the full, research version and a brief 6-item version (with 5 knowledge items and 1 treatment preference item) have demonstrated strong psychometric properties (e.g., retest reliability, validity, sensitivity to change) and clinical sensibility (e.g., acceptability and feasibility).^22,29 The BCS DQI is included in the appendix and is available at www.mghdecisionsciences.org. For these analyses a total knowledge score (0–100%) was calculated for patients who answered at least half of the items. Three knowledge items assess understanding of quantitative risk (including complications of radiation, local recurrence after mastectomy and local recurrence after lumpectomy and radiation). A concordance score (% of patients who received preferred treatment) was calculated. For concordance, patients who responded that they preferred mastectomy and received mastectomy were considered ‘concordant’, similarly for those who preferred and received lumpectomy. Patients who responded that they were “not sure” for their preferred treatment were not considered to be concordant.

Shared Decision Making (SDM) Process survey: 4 items assess discussion of treatment alternative, pros and cons of the treatment received, and patients’ treatment preferences. The items were developed in conjunction with the DQIs, using similar process, and have evidence of reliability and validity across a range of clinical situations.^25,30-33 Total score ranges from 0–4 with higher scores indicating more SDM. Any survey with 1 or more missing items did not receive a total score.

Analysis

Sample size: With a minimum of 100 responses at both time points, the study had 85% power to detect a decrease in knowledge or SDM Process of 6% (an effect size of about 0.3) with 0.05 level of significance assuming a standard deviation of 0.2 for the scores and assuming interpatient correlation of 0.5. We based the effect size on evidence from 2014 Cochrane systematic review of studies that compared patient decision aids to simpler interventions. These studies found 5.5% difference between complex and simple decision aids which we used to estimate a minimally important decline in knowledge.³⁴

Sample evaluation: First, the sample characteristics were compiled and two-sample t-tests and chi-square tests were used to compare characteristics of responders and non-responders. Chi-square tests were used to compare the response rates and patient characteristics among sites. We also examined the interaction between patient characteristics and response rate by site.

We tested the following hypotheses regarding the scores:

1. Knowledge scores would decline from baseline to one year.

2. The decline in knowledge scores over time would be greater for quantitative items (i.e. the 3 knowledge items that assess understanding of numerical risk) than qualitative (or gist) items (i.e. 7 other knowledge items).

3. The SDM process scores would decline from baseline to 1 year.

4. The ratings of the individual goals and the reported preferred treatment would become more aligned with actual treatment received over time.

5. Site 4, with formal decision support services, would have higher knowledge, higher concordance (percentage of patients receiving their preferred treatment), and SDM Process scores at 1 month and 1 year compared to the other sites.

Analyses examined the hypotheses for the full BCS-DQI, that is often used in research studies, and the brief version, that is often used in quality improvement or clinical practice. Exploratory analyses examined whether any demographic or clinical factors (such as treatment, site, age, education) were associated with those individuals who had changes in knowledge and SDM Process scores, (≥10% for knowledge and ≥1 point for SDM Process).

Results

Staff screened 882 patients and sent a packet to 451 who met the study criteria. We had 274/451 (61%) responses to the 1-month assessment, of which 7 were deemed ineligible after reviewing the completed survey for a final baseline sample of 267. We had 229/267 (86%) responses to the 1-year assessment. These analyses are restricted to respondents at both time points.

Response rates differed across sites, from 55%−75%, Chi Square p=0.008. Non responders did not differ significantly from responders by age, surgery type, nor tumor stage (p>0.2 for all analyses). Non responders to the 1-year survey also did not differ significantly from responders by clinical or demographic factors (including surgery type, tumor stage, race, education, or age).

Knowledge scores were generally stable over time

The mean total knowledge scores were similar at each time point (see Table 2). Knowledge scores for the quantitative items decreased slightly over time (4.1 points, p=0.09) and scores for the qualitative items increased over time (1.9 points, p=0.08) as predicted by our hypotheses; however, the differences were small and did not reach statistical significance. The brief (5-item) version also had stable scores (68.2% (SD 21.0) at 1 month vs. 69.1% (SD 20.3) at 1 year, p=0.53).

Table 2.

Knowledge, SDM Process, and Concordance Scores by Site at 1 Month and 1 Year^*

	Total	Site 1	Site 2	Site 3	Site 4
Knowledge score (0-100%) (n=218), mean (SD)
1 month	69.0 (16.9)	67.2 (15.1)	69.9 (15.4)	66.7 (18.5)	75.4 (18.4)
1 year	69.9 (16.9)	67.7 (16.0)	72.1 (13.9)	67.8 (20.6)	73.7 (15.2)
SDM process score (0-4) (n=211), mean (SD)
1 month	2.7 (1.1)	2.4 (1.1)	2.8 (1.1)	2.8 (1.1)	3.2 (0.8)
1 year	2.7 (1.2)	2.5 (1.2)	2.8 (1.1)	2.6 (1.3)	3.2 (0.8)
Concordance, % (n=219)
1 month	91.8% (201/219)	93.5% (58/62)	91.5% (54/59)	94.0% (63/67)	83.9% (26/31)
1 year	92.2% (202/219)	95.2% (59/62)	93.2% (55/59)	92.5% (62/67)	83.9% (26/31)

^*Patients with scores available at both time points, 1 month and 1 year.

Site 4, which had a formal SDM program, had significantly higher knowledge at 1 month compared to the other sites (mean difference=7.5 points, p=0.03). At 1-year, the difference was no longer significant (mean difference=4.5 points, p=0.17). The results were similar for the brief knowledge score that includes 5 items (mean difference 1 month=8.7 points, p=0.02 and at 1 year=−0.4 points, p=0.82).

About half (53.2%) of respondents had a large change in knowledge scores, 53/218 (24.3%) decreased 10% points or more, and 63/218 (28.9%) increased 10% points or more. Exploratory analyses did not find an association between any of the demographic variables (age, education, race) nor clinical variables (site, stage of disease, surgical treatment, radiation treatment) and respondents who had a large change in scores.

SDM process scores significantly higher at site with formal decision support services

The mean SDM Process scores were similar at 1 month and 1 year (2.7 (SD 1.1) vs. 2.7 (SD 1.2), p=0.74). Table 2 shows SDM Process scores by site. Site 4 had significantly higher SDM Process scores both at 1-month (mean difference = 0.58 (SE=0.18), p=0.002) and 1-year (mean difference = 0.61 (SE=0.18), p=0.002). About half of the respondents had SDM Process scores that changed by ≥1 point (54/216 (25.6%) had scores decrease by ≥1 point and 56/216 (26.5%) had scores increase by ≥1 point). Exploratory analyses did not find any significant associations between the demographic or clinical variables for respondents who had large changes in scores.

Patient’s goals and treatment preferences

Three goals discriminated significantly between patients who had a mastectomy and those who had lumpectomy (see Figure), p<0.001 for all three comparisons at 1-month and 1-year. One of the goals, avoid radiation, was significantly less important at 1 year for all patients (for lumpectomy patients, mean difference=−0.43, p=0.03 and for mastectomy patients, mean difference=−1.5, p=0.01). It was also less important both for patients who had radiation (mean difference = −0.5, p=0.01) and those who did not have radiation (mean difference = −0.8, p=0.1). Mastectomy patients also rated importance of removing breast for peace of mind significantly lower at 1 year (mean difference = −1.00, p=0.03).

Figure.

Ratings of goals and concerns for lumpectomy and mastectomy patients at 1 month and 1 year. Scores range from 0 to 10, with higher scores indicating more importance. *P < 0.05 for 1-month v. 1-year ratings for both lumpectomy and mastectomy. **P = 0.03 for 1-month v. 1-year rating for mastectomy patients only.

Overall, the majority of patients preferred lumpectomy at 1 month (72%) and 1 year (70%). Concordance, or the percentage of patients who received their preferred treatment, was similar over time (92% at 1 month vs. 92% at 1 year, McNemar p=1.0). The site with formal decision support services had a lower percentage of concordance compared to the other sites (see Table 2), but the differences were not statistically significant at either time point (p=0.09 at 1-month and p=0.24 at 1-year).

Discussion

In this study, we examined the measurement of decision quality and SDM in sample of women with breast cancer. On average, patients at these cancer centers were well informed and the majority received their preferred treatment. The site that had a formal SDM program had significantly higher knowledge scores and higher SDM Process scores compared to the other sites at 1 month, providing evidence of discriminant validity for the scores. Contrary to our hypotheses, the mean scores for the sample were stable over time.

Patients had high mean knowledge scores at 1-month and 1-year. The knowledge items included in the BCS-DQI test for both gist (i.e. high-level summaries of key ideas) and verbatim knowledge (e.g. detailed knowledge of the numerical chance of having a recurrence or complication). Fuzzy trace theory suggests that verbatim knowledge would not be kept over the long term, rather only the ‘gist’ would remembered.^12,13 However, in this sample, patients actually had similar quantitative (or verbatim) knowledge scores at 1 month and 1 year. These results provide evidence that patients are able to maintain key aspects of knowledge over time, an important component of quality decision making.

Another key aspect of shared decision making is eliciting patients’ goals and preferences. Psychological theories such as cognitive dissonance and bolstering, predict that importance ratings of goals and concerns would become more aligned with actual treatment choices over time. However, this study found relatively modest changes in ratings. Further, for one goal, ‘avoiding radiation,’ both lumpectomy and mastectomy patients’ importance ratings decreased over time. This finding held for those who did and did not actually receive radiation therapy as part of their treatment. It suggests that patients’ concerns about radiation are subject to a consistent error in affective forecasting, such that the actual concern is lower than predicted. A key implication of this result is that decision support efforts may need to help newly diagnosed patients better understand the lived experiences of patients who undergo radiation therapy. In doing so, it may improve patients’ ability to forecast how they will feel during and after radiation.^35-37

The majority of patients at these sites preferred lumpectomy and received their preferred treatment. Across sites, about 5–15% of these women, who were eligible for either approach, did not receive their preferred treatment. The lack of increase in the rate of concordance over time is potentially due to the limited room for improvement given the high baseline rates across sites. For surgical decisions, particularly ‘preference-sensitive’ and elective procedures, it is important that all patients are well informed and have a clear preference for the procedure. Ideally, 100% of patients would be concordant. The findings here suggest a failure of informed consent and that 1 in 10 patients may have a ‘preference misdiagnosis’.³⁹ A preference misdiagnosis occurs when the surgeon or physician recommends a treatment that does not meet or align with the patients’ preferences. In the case of surgery, a treatment that is not reversible, this may lead to performing the wrong operation, and is a serious safety issue. More work is needed to understand the extent of preference misdiagnosis and whether routine measurement of decision quality with feedback to physicians would help improve performance in this regard.

The timing of the assessment did not have a meaningful impact on scores, and this finding has considerable implications for the feasibility of decision-making surveys. From a decision theoretic perspective, surveying patients after a decision has been made but before they have received the treatment is ideal. However, in practice, identifying and surveying cancer patients at that time is extremely difficult, because of the compressed time window, as well as heightened patient anxiety. The empirical data from this study provide support for surveying breast cancer patients about their decisions up to one year after surgery. This expanded time window may allow centers to incorporate SDM items in existing patient experience surveys or patient registries greatly increasing feasibility.

Although the mean knowledge scores were similar over time, the ability of the scores to discriminate among sites with and without decision support was higher at 1 month. A key implication is that using the decision-making measures to evaluate an intervention would benefit from administration close in proximity to the delivery of the intervention.

Despite growing interest from groups including UK’s National Health Service,⁴⁴ National Quality Forum⁴ and the Centers for Medicare & Medicaid Services,⁴⁵ measurement of SDM in current clinical practice has been limited. This may be due in part to the paucity of data on how to measure SDM in a way that is feasible and does not unduly burden patients or clinicians.⁴⁶ As patient reported outcome measures become more common for evaluating cancer patients’ treatment experiences,^47,48 our findings may inform best practices around when to survey patients. For example, while it may be desirable to survey breast cancer patients shortly after treatment for certain outcomes of interest, the findings from this study suggest that recall is not compromised when women are surveyed up to 1 year later. Health systems may have some flexibility for deciding when to follow up with patients. From a physical and emotional recovery standpoint, patients may benefit from a longer time horizon for follow up.

The current literature on SDM measurement is small, but emerging. Our research raises new questions for how SDM process measures can be implemented and assessed outside of a research context. For example, health systems are increasingly exploring novel uses of the electronic medical record to assess various aspects of the patient experience including health-related quality of life.^49,50 Additionally, some health systems are using patient portals to “prescribe” decision aids and other types of educational materials patients. In the future, electronic medical record platforms may be leveraged to assess decision making for breast cancer surgery patients within 1 year of treatment and this may be connected to health system’s patient experience and patient reported outcomes assessments.⁵¹ A more systematic approach to collecting SDM process measures across sites may allow for better regional and national assessment of decision quality, which may have implications for improving quality and safety, and thereby reducing adverse events (e.g., physical or psychological) for cancer patients. Such measures could potentially be integrated into annual surveys of patient experience. For sites already collecting patient-reported outcomes measures (PROMS), it may be beneficial to separately assess certain measures by critical windows of time (e.g. specific knowledge scores within 1–3 months, and general SDM process scores within 6–12 months).

Strengths, Limitations, and Future Directions

Strengths of this study include the fact that women were recruited from geographically diverse NCI-Designated Cancer Centers in the United States and followed longitudinally. Additionally, the instruments used to assess decision quality and the SDM have demonstrated strong psychometric properties and were developed with input from both patients and clinicians. Despite the strengths of this study, some limitations should be noted. Patients were not randomly assigned to receive decision support across these sites, rather this was a natural experiment comparing sites that have different baseline practices. Further, all of the assessments were post surgery and it is not clear whether the findings are reflective of the knowledge and preferences of patients prior to surgery. The site with decision support did not have an explicit values clarification exercise, and that may have limited the potential impact on preference-related outcomes. Moreover, these findings may not fully generalize to women seen in non-NCI designated cancer centers, nor to other conditions.

There are many opportunities to expand upon this research. First, future work should examine SDM measurement timing among a broader sociodemographic population of women with breast cancer and examine decisions about other treatments. Second, additional experimental or observational studies that include patients who used decision support with formal preference elicitation techniques (and with different techniques) and that survey patients before the treatment is received would provide important evidence of the validity of the DQI scores. Finally, evaluating measurement timing for other common decisions both cancer and non-cancer related would be important to establish the generalizability of the findings across clinical conditions, gender, and other factors.

CONCLUSION

Several groups are calling for better measurement of SDM performance in clinical practice and this study provides important data on the validity and timing of measurement in a sample of breast cancer patients. For population-level assessments of breast cancer surgical decisions, the results suggest it is reasonable to survey patients up to a 1 year after the decision, greatly increasing feasibility of measurement. However, when evaluating decision support interventions for breast cancer, administering the surveys closer to the intervention will be most effective. Continued research and practical strategies are needed to better support decision making for women with breast cancer.

Table 1.

Sample Characteristics by Site

Characteristic	Total N=229	Site 1 n=65	Site 2 n=63	Site 3 n=67	Site 4 n=34
Mean Age, y (SD)	56.9 (12.3)	57.9 (13.4)	54.1 (12.1)	59.6 (11.9)	54.8 (10.4)
White, non-Hispanic, No. (%)	205 (89.5)	62 (95.4)	61 (96.8)	54 (80.6)	28 (82.4)
Education, No. %
College graduate or more	137 (59.8)	38 (58.5)	38 (60.3)	40 (59.7)	21 (61.8)
Some college	65 (28.4)	22 (33.8)	16 (25.4)	18 (26.9)	9 (26.5)
High school or less	26 (11.4)	4 (6.2)	9 (14.3)	9 (13.4)	4 (11.8)
Stage of disease, No. (%)
Stage I	136 (59.4)	41 (63.1)	38 (60.3)	34 (50.7)	23 (67.6)
Stage IIa	67 (29.3)	19 (29.2)	16 (25.4)	22 (32.8)	10 (29.4)
Stage IIb	19 (8.3)	2 (3.1)	5 (7.9)	11 (16.4)	1 (2.9)
Stage III	7 (3.1)	3 (4.6)	4 (6.3)	0	0
Final surgical treatment, No. (%)
Lumpectomy	160 (69.9)	46 (70.8)	38 (60.3)	52 (77.6)	24 (70.6)
Mastectomy	69 (30.1)	19 (29.2)	25 (39.7)	15 (22.4)	10 (29.4)
Had Reconstruction	48 (21.0)	13 (20.0)	20 (31.7)	7 (10.4)	8 (23.5)
Systemic therapy, No. (%)
Chemotherapy	109 (47.6)	28 (43.1)	33 (52.4)	31 (46.3)	17 (50.0)
Hormone therapy	182 (79.5)	59 (90.8)	52 (82.5)	45 (67.2)	28 (76.5)