Development of instruments to measure the quality of breast cancer treatment decisions

Abstract

Background

Women with early stage breast cancer face a multitude of decisions. The quality of a decision can be measured by the extent to which the treatment reflects what’s most important to an informed patient. Reliable and valid measures of patients’ knowledge and their goals and concerns related to breast cancer treatments are needed to assess decision quality.

Objective

To identify a set of key facts and goals relevant to each of three breast cancer treatment decisions (surgery, reconstruction, and adjuvant chemotherapy and hormone therapy) and to evaluate the validity of the methods used to identify them.

Methods

Candidate facts and goals were chosen based on evidence review and qualitative studies with breast cancer patients and providers. Cross-sectional surveys of patients and providers were conducted for each decision. The accuracy, importance, and completeness of the items were examined.

Results

38 facts (11–14 per decision) and 27 goals (8–10 per decision) were identified. An average of 17 patients and 21 providers responded to each survey. The sets of facts were accurate and complete for all three decisions. The sets of goals and concerns were important for surgery and reconstruction, but not chemotherapy/hormone therapy. Patients and providers disagreed about the relative importance of several key facts and goals.

Conclusion

Overall, breast cancer patients and providers found the sets of facts and goals accurate, important, and complete for three treatment decisions. Because patients’ and providers’ perspectives are different, it is vital that instrument development should include items reflecting both views.

Introduction

A diagnosis of early stage breast cancer sets off a series of preference-sensitive treatment decisions, including: (1) initial surgical choice between mastectomy and breast-conserving surgery; (2) whether to have breast reconstruction if mastectomy is chosen; and (3) whether or not to have adjuvant chemotherapy, hormone therapy (CHT), or both. Clinical guidelines and quality measures recognize that there is not one “right” answer for any of these decisions.(1–3) Rather, consensus is growing that a high-quality decision for such preference-sensitive conditions is one that reflects the considered preferences of well-informed patients.(4, 5) In order to evaluate the quality of preference-sensitive decisions, instruments are needed that assess whether the patient is informed of the choices, understands the likelihood of positive and negative outcomes of each decision, and whether the treatment she gets reflects what is most important to her.

Although a number of studies have examined the degree to which breast cancer patients are informed about their treatment options, few have used measures of knowledge with proven reliability and validity. Four studies have reported knowledge for the surgical decision, (6–10) one has assessed knowledge of reconstruction,(10) two have assessed knowledge of chemotherapy,(11, 12) and none has assessed knowledge of hormone therapies. On close inspection of these studies’ measures, only one was a previously-validated knowledge instrument. It was validated in 1990 and is outdated for our purposes.(6) The other instruments were created specifically for their particular studies. Two studies described how the items were generated, and both were developed using medical expert input without patient input.(6–9) None of the studies reported reliability or validity of the knowledge instruments for the study sample.

Studies of patients’ preferences about breast cancer treatments have also been limited by the quality of measures. Several cross-sectional studies have identified patient concerns associated with choice of surgical treatment.(13–17) Two studies have examined patients’ preferences and their association with decisions about reconstruction.(18, 19) Only one of these studies used previously validated instruments and reported information on the reliability or validity of the data.(18) The others developed the items specifically for the study and did not report on reliability or validity. Although the issues explored by these studies had significant conceptual overlap, the wording and scaling tasks differed.

Studies of decision making about adjuvant chemotherapy and hormonal therapy (CHT) have taken the approach of measuring how much benefit a patient would need to gain in order to make the side effects of treatment worthwhile.(20–24) Most studies have used structured interviews and time-tradeoff exercises to estimate the required level of benefits. Although the importance of weighing the benefits and harms seems obvious, the benefits and harms used in the exercises varied. Furthermore, all of the patients in these studies had taken the treatment (chemotherapy and/or hormonal therapy). How well these approaches would work in patients actually facing CHT decisions is not known.

A more transparent and rigorous process is needed to identify the key pieces of information that patients should understand and the salient issues that may determine their preference for one treatment over another. The authors and their colleagues have designed such a process for the development of decision quality instruments (DQIs) specific to common clinical decisions, including breast cancer treatment decisions.(25) The development process for the instruments is summarized in the methods and Table 1. The first step in developing the instruments is to generate a core set of facts that is essential for patients to understand before making a decision, as well as a core set of salient goals and concerns that lead patients to prefer one treatment over another.

Table 1.

Development process for Decision Quality Instruments

Phase of instrument development	Development steps
1. Identification of key facts and goals	Review clinical evidence and decision making literature. Review literature on patient perspective, and conduct focus groups and interviews with patients and providers. Draft candidate facts and goals across key domains and review with medical experts and decision-making experts. Conduct cross-sectional surveys of patients and providers to assess accuracy, importance, and completeness of candidate facts and goals.
2. Drafting of questions	Draft survey items for key facts and goals. Conduct cognitive testing of survey items with patients. Revise items based on literacy review by expert. Revise items based on medical review by providers. Field test with patients to assess reliability and validity. Refine instruments based on field test results.
3. Validation of instrument	Evaluate performance in diverse samples of patients. Conduct broad-scale review by providers. Review items for accuracy and completeness on annual basis, including patient and provider input as needed.

Here, we report the results of cross-sectional surveys of breast cancer survivors and providers for three decisions: surgery for early stage breast cancer; breast reconstruction, and adjuvant chemotherapy and hormone therapy for early stage breast cancer. The purpose of the surveys was to assess the accuracy, importance, and completeness of the set of candidate facts and goals. A secondary aim was to determine whether or not a small, core set of information and goals, which providers and patients would both agree is essential, exists.

Methods

Purpose and item development

The rationale for the DQIs,(5) conceptual underpinnings of the approach,(26) and pilot results(25, 27) have been described. The underlying conceptual framework builds on the systems approach to decision making described by Mulley in 1989.(28, 29) This framework attempts to bridge the tension between normative (or rational theories of decision making, which promote fully informed choices that maximize expected utility) and behavioral decision theories (which describe the heuristics, biases and traps that people often use when actually making decisions).(30) For example, it recognizes that people have limited cognitive capacity and limited resources and will never reach the normative ideal. In addition, it recognizes that factors other than utilities for health states may appropriately influence choices. We refer to these factors more generally as ‘‘goals and concerns” in order to distinguish them from the formal utilities of decision analysis. We specifically do not use “preferences” to describe these items, as the term has many different meanings, including preferred treatment, preferred role in decisions, time and risk preferences, and preferences for specific outcomes and attributes.

The DQIs are designed to measure the extent to which treatments reflect informed patients’ goals and concerns. The instruments are composed of a set of decision-specific knowledge questions and a set of subjective assessments of patients’ goals and concerns. The facts cover the following five domains: Disease (such as the prevalence, lethality, and natural history), Choices (such as a description of the treatment options and what’s involved with each), Benefits of the choices (such as survival, symptom relief, likelihood of outcomes), Harms of the choices (such as serious and permanent problems, temporary and common problems, the likelihood of problems), and, the Decision situation (such as the urgency to treat, patient’s role in the decision, recognition of decision).

The goals and concerns include good and bad health outcomes and other factors that patients reported were critical to their decisions, phrased in their language. The goals cover the following categories: Benefits (such as symptom relief due to a treatment), Harms (such as long-term, adverse health states possible after a treatment, or temporary adverse health state during or shortly after a treatment), Other attitudes about non-health states (e.g., time spent getting treatment or in recovery, costs), Holistic attitudes toward treatment or approach to treatment (e.g., always want to do as much as possible, avoiding medication), and Influence of others (e.g., following doctor’s or spouse’s recommendation).

The candidate facts and goals were generated by patients, medical experts and decision making experts, and were based on reviews of the clinical evidence and qualitative studies of breast cancer decision making and experiences of breast cancer care.(25)

Sample and Design

Providers

For each decision, a convenience sample of breast cancer providers, including breast cancer nurses, medical oncologists, surgical oncologists, general surgeons, and plastic surgeons, was identified. Providers were identified through colleagues and through academic and community websites. Eligible providers were mailed the survey and a $10 incentive. Non-responders were sent a reminder and then another copy of the survey after four weeks. Responders received an additional $40 for completing the survey.

Patients

A convenience sample of patients was recruited through a combination of newspaper advertisements, flyers, registries, providers, and patient support groups in the areas around Dartmouth Hitchcock Medical Center and the University of Massachusetts Boston. Respondents were screened by phone. Those who were older than 21, with a history of early stage breast cancer diagnosed within five years prior to contact, and who could speak and read English were eligible. We limited the time from diagnosis to five years so that participants would have faced the same treatment options available to current patients. Because we were developing instruments, we felt that enrolling newly-diagnosed patients would pose unjustified burden at a time of great stress. In addition, because patients who are still making decisions would not have enough perspective to reflect on the information they needed, we sought some hindsight to guide the rating of facts and goals. For the reconstruction survey, participants must have had a mastectomy. Although we were aware that knowledge about breast reconstruction could affect decisions between mastectomy and breast conservation, we chose to exclude women who had breast conservation from the reconstruction survey, to maximize the relevance of responses in our small sample. Each eligible patient was mailed a survey for one decision. Non-responders were sent a reminder and then a second copy of the survey again after four weeks. Responders were given $10 to $25 for completing the survey.

Data collection

Participants rated the importance of each fact and goal on a four-point scale (1 = Not Important, 2 = Somewhat Important, 3 = Very Important, 4 = Extremely Important), selected their top three most important facts and goals, and added any additional facts and goals and concerns that they thought were salient to the decision.

Analysis

The primary objective of analysis was to assess the degree to which the set of candidate facts and goals was accurate, important, and complete. Secondary analyses examined the amount of agreement between patients and providers regarding the importance of each candidate fact and goal.

Accuracy

Participating providers were asked to identify any inaccuracies in the candidate facts. Although we did not specifically ask patients to identify inaccuracies, we considered any inaccuracies that they noted. If anyone (regardless of specialty) raised a concern about the accuracy of a fact, we sought evidence to clarify the concern. Here, published peer reviewed studies trumped personal experience – so for example, if a surgeon said their complication rate was lower but did not have published evidence of their rates to support the claim, we did not consider the fact inaccurate. Adding a fact to the list required more than one person to mention it, thus requiring it to be not a unique concern, but one that is shared or more widely recognized. An expert in breast cancer evidence and clinical care also evaluated any suggested inaccuracies to determine whether or not the clinical evidence supported the suggested change. Corresponding changes were then made to the next version of the DQI, to be used in subsequent testing. If one or fewer changes were made to items, the set of facts was considered accurate. The set of candidate goals was not assessed for accuracy because they are subjective.

Importance

A weighted median was calculated for patients’ and providers’ ratings of importance for each candidate fact and goal, for each condition. The weighted median gave equal weight to the patient sample and the provider sample. All items that had a weighted median greater than 2 were considered important. If an item had a weighted median of 2 or less, and fewer than 10% of patients and fewer than 10% of providers placed the item in their top three, then the item was considered a candidate for deletion. A set of facts and goals for a given condition was considered “important” if two or fewer facts and two or fewer goals were candidates for deletion.

Completeness

Additional facts and goals provided by patient or provider participants in response to the open ended questions were evaluated to determine if they were accurate and relevant, and if they added new content to the existing set of items. Accurate, relevant, and additional suggestions were added to the candidate list only if two or more respondents (patients or providers or both) offered the same or similar suggestions, indicating wider importance. A set of facts and goals for a given condition was considered “complete” if two or fewer facts and two or fewer goals were added to the candidate list.

Agreement between patients and providers: First, for each candidate fact and goal, we determined the percentage of times the fact or goal was reported, by patients and providers, respectively, to be among the three most important facts or goals. Based on these results, we identified the three most commonly selected facts and goals among patients and among providers to see how much overlap there was in the top three ranked items. Then, for each item, the percentage of patients who placed the item in their top three was compared to the percentage of providers who placed the item in their top three. The asymptotic 95% confidence interval (CI) around the difference between the patient and provider percentages was calculated for each item. Confidence intervals excluding 0 were considered evidence of statistically significant differences in importance ratings between patients and providers.

Results

Response rates and sample

Across the three decisions, the patients’ response rate was 79% and the providers’ response rate was 77%. Tables 2a and 2b describe the demographics of the patient and provider samples. Fifteen providers, including 6 surgeons, completed the surgery survey. Twenty providers, including 11 plastic surgeons, completed the reconstruction survey. Twenty-seven providers, including 11 medical oncologists, completed the CHT survey. Four providers completed two surveys (surgery and reconstruction). Each patient completed only one survey.

Table 2a.

Characteristics of patients

Topic	Patients
	N	Mean age (range)	Race (% white)	Treatment	Education (% HS or less)	Mean years since dx
Surgery	14	64 (40–83)	100	43% mast	36	1.7
Reconstruction	21	50 (35–69)	92	85% (11/13) recon**	15	2.7
Chemotherapy/hormone therapy*	17	53 (40–63)	92	62% CH 77% HT 59% both 15% none	0	2.5

dx=diagnosis; HS = high school; mast = mastectomy; recon = surgical breast reconstruction; CH = chemotherapy; HT = hormone therapy; none = no systemic therapy

^*For CHT, demographic data were available for 13 of the 17 patients.

^**For reconstruction, treatment data were available for 13 of the 21 patients.

Table 2b.

Characteristics of providers

Topic	Providers
	N	Mean age (range)	Gender (% male)	Mean years in practice (range)	Median yearly volume	Specialty
Surgery	15	49 (30–59)	27	24 (5–30)	30	55% physician 45% surgery*
Reconstruction	20	43 (32–58)		15 (5–38)	42	50% physician 50% plastic surgery*
Chemotherapy/hormone therapy*	27	47 (30–65)	27	16 (2–30)	200	60% physician 67% medical oncology*

^*percentage of the physicians

Performance of the facts

Accuracy

Providers had very few comments on the items (2 for surgery, 3 for CHT and 4 for reconstruction) and did not identify significant inaccuracies. For the surgery item “Radiation involves treatments five days a week for 5 to 6 weeks”, one provider noted that partial breast irradiation techniques do not take several weeks. As a result, we added a qualifier “traditional” to the description of radiation. For the reconstruction item, “Prosthesis can provide a ‘natural look’ in clothes”, one provider did not agree with this statement and another remarked that it depended on the clothes. As a result, we added a qualifier “most” to “clothes”. None of the other comments resulted in changes to the items. Patients did not note any inaccuracies. Overall, the set of facts for each of the three decisions was considered accurate.

Importance

The overall median importance rating of the facts was 3 (range 3–4). Since none of the items had a median importance score of 2 or less, no facts were candidates for deletion for any decisions. Overall, both patients and providers considered each set of facts important.

Completeness

Respondents wrote in 21 comments on the surveys (4 for surgery, 9 for CHT, and 8 for reconstruction). Only one resulted in an additional fact - that reconstruction does not interfere with cancer surveillance or impact the likelihood of recurrence. Some suggestions were already covered by the existing items. For example, two patients suggested that the trauma and after-effects of flaps needed more emphasis. We added additional details to the existing item “Implants require less extensive surgery than flaps.” Other suggestions were not relevant to the primary purpose of the instruments. For example, both patients and providers wrote in that they wanted more information on aromatase inhibitors in the CHT facts. The DQI is intended for use at the level of selecting types of medications (i.e., chemotherapy, hormone therapy, or no medication), rather than specific agents, so we did not add specific items on aromatase inhibitors. None of the other suggestions was mentioned by more than one respondent. Overall, the set of facts for each decision was considered complete.

Performance of the goals

Importance

For surgery, the weighted median of the importance ratings ranged from 2 to 4, and no items were candidates for deletion. For reconstruction, the weighted median of the importance ratings ranged from 2 to 3, and no items were candidates for deletion. For CHT, the weighted median of the importance ratings ranged from 2 to 4. Three items, “avoid short-term side effects of chemotherapy’”, “maintain fertility”, and “do what your doctor thinks is best” were candidates for deletion. The item on the short-term side effects of chemotherapy was deleted, but the other two items were kept. Although fertility is only relevant for younger women (a small minority of breast cancer patients), we decided that the issue was significant enough for those women to warrant keeping it. The item about the doctor’s recommendation is a theme across many different conditions and was also kept, but as a preference about participation in decisions, and not as a goal. The sets of goals for surgery and for reconstruction were considered important.

Completeness

For surgery, providers suggested one additional goal, and patients suggested four. The items were fairly general (e.g., the experience of the doctor and the hospital’s MRSA infection rate) and would not necessarily lead a woman to choose one surgery over another. None was mentioned by more than one respondent. For reconstruction, providers wrote in four goals and patients wrote in two. The one item that was mentioned by two respondents, “wanting to avoid complications of surgery”, was added to the set of goals.

For CHT, providers added one goal, and patients added six. Three patients added the goal of balancing benefits and side effects of adjuvant therapy. Two providers wrote in consideration of the absolute risks for an individual patient. Because seven of the ten goals already covered the benefits or side effects of adjuvant therapy, including asking patients about knowledge of the absolute benefit of therapy, we considered these additions to be redundant. No additional goals were added. Overall, the sets of goals and concerns for each decision were considered complete.

Agreement among and between providers and patients

Surgery facts

Patients and providers ranked several different facts as being among the three most important, with some overlap among the facts most often selected by both types of respondents (see table 3a). Significantly more providers than patients (53% versus 14%, 95% CI of the difference −70, −8) felt that the fact “for most women with early stage breast cancer, waiting 4 weeks to make a treatment decision would not affect their chances of survival” was important.

Table 3a.

Surgery: percentage of patients and providers who ranked each fact and goal in the top 3.

Surgery Fact*	Patient (n=14) %	Provider (n=15) %	95% CI of the difference
Patients who have lumpectomy live as long as those who have mastectomy.	43	73	−65	4
Waiting 4 weeks to make a treatment decision does not affect survival.	14	53	−70	−8
The chance of cancer coming back in the treated breast is slightly higher after lumpectomy and radiation.	36	27	−25	43
If 100 women have lumpectomy and radiation, 5 to 15 will have cancer come back in 10 years.	29	27	−31	35
Most women are candidates for both types of treatment and have a choice.	36	27	−25	43
Some who have lumpectomy will need more than one operation for margins.	7	27	−46	7
If cancer comes back in the breast after lumpectomy, it is usually treated with mastectomy.	29	27	−31	34
If 100 women have mastectomy, 2 to 10 will have cancer come back in 10 years.	21	13	−19	36
Radiation involves treatment five days a week for 5–6 weeks.	21	13	−36	19
Side effects of radiation include fatigue and cosmetic changes to the breast.	7	13	−16	28
Breast reconstruction is an option that can be done at the time of the mastectomy or later.	29	7	−5	49
Most women are very satisfied with the way their breast looks after lumpectomy.	14	0	−33	4
Serious problems caused by radiation are rare.	7	0	−6	21
Surgery Goal*	Patient (n=14) %	Provider (n=15) %	95% CI of the difference
Minimize the chance of cancer coming back in the treated breast.	100	93	−21	6
Be able to say you did everything possible.	64	79	−19	47
Keep the breast.	7	71	−92	−37
Avoid side effects and complications of radiation therapy.	0	21	−43	0
Do what your doctors think is best.	86	14	−97	−46
Remove the entire breast to gain peace of mind.	14	14	−26	26
Avoid the hassle of radiation therapy.	0	7	−21	6
Avoid breast reconstruction.	14	0	−4	33

CI = confidence interval

^*shortened from the original wording

Bold print denotes statistically significant difference, p<0.05.

Surgery goals

Patients and providers almost unanimously felt that minimizing the chance of recurrence was the key goal for this decision. This item was the only one for all surveys that reached greater than 90% for both groups. Patients were significantly less likely than providers to consider “keep your breast” as a top goal when choosing surgery (7% vs. 71%, 95% CI of the difference −92, −37). Patients were significantly more likely than providers to select “do what your doctor thinks is best” in their top 3 goals (86% versus 14%, 95% CI of the difference −97, −46).

Reconstruction facts

Only one reconstruction fact, “about 1/3 of patients who have reconstruction will have a major complication,” was most frequently ranked in the top 3 by both providers and patients (Table 3b). Otherwise, patients and providers appeared to focus on different aspects of the decision. Providers were more concerned about the impact of radiation on the success of the reconstruction than patients (60% vs. 24%, 95% CI of the difference −64, −8). There was a trend toward greater patient interest in the fact that patient satisfaction is the same whether reconstruction is done immediately or delayed (24% vs. 5%, 95% CI of the difference −2, 39). Patients also tended to place more importance than providers did on understanding that the data on complications is limited (24% vs. 5%, 95% CI of the difference −2, 39).

Table 3b.

Reconstruction: percentage of patients and providers who ranked each fact and in the top 3.

Reconstruction Fact*	Patient (n=21) %	Provider (n=20) %	95% CI of the difference
Radiation can increase complications and affect cosmetic result of reconstruction.	24	60	−64	−8
About 1/3rd will have a major complication in the two years after reconstruction.	67	40	−56	3
Reconstruction often requires multiple procedures over multiple visits to complete.	33	35	−27	31
Reconstruction can be at the time of mastectomy or delayed for months or years.	43	35	−22	38
Women who do not have reconstruction generally as satisfied as women who do.	5	30	−47	−3
Women who have flap are more satisfied with the look and feel than women who have implant.	29	30	−26	29
Immediate reconstruction offers more natural look and feel than delayed.	38	30	−21	37
Implants require less extensive surgery than flaps.	5	10	−21	11
Women who delay reconstruction are as satisfied as women who have immediate.	24	5	−2	39
Prosthesis can provide a “natural look” in clothes.	10	5	−11	20
The data available to provide estimates of complications for reconstruction is limited.	23	5	−2	39
Reconstruction Goal*	Patient (n=21) %	Provider (n=20) %	95% CI of the difference
Look natural in clothes.	43	60	−13	47
Minimize the number of surgeries.	71	60	−40	18
Minimize recovery time.	19	45	−54	2
Look natural without clothes.	23	40	−12	44
Avoid a lengthy process.	38	30	−21	37
Use your own tissue to create a breast.	43	30	−16	42
Do what your doctor(s) think is best.	24	15	−15	33
Do what your spouse thinks is best.	5	10	−21	11
Avoid using a prosthesis.	33	0	13	54

CI = confidence interval

^*shortened from the original wording

Bold print denotes statistically significant difference, p<0.05.

Reconstruction goals

The two most frequently selected goals were the same for patients and providers: “minimize the number of operations” and “look natural in clothes”. Patients placed greater importance on avoiding a prosthesis (33% vs. 0%, 95% CI of the difference 13, 54). There was a trend toward less patient concern about “looking natural without clothes” compared to providers (24% vs. 40%, 95% CI of the difference −12, 44).

CHT facts

For both patients and providers, “the best choice depends on medical factors and patient’s feelings about benefits and side effects” and “realistic estimate of recurrence with chemotherapy or hormone therapy” were two of the three facts most frequently selected (Table 3c). While 24% of patients felt that “chemotherapy can cause rare but serious side effects” was critical, none of the providers selected that in their top three (95% CI of the difference 3, 44). None of the providers selected any of the four items that covered common and serious side effects of chemotherapy and hormone therapy in their top three, whereas 6–24% of patients put them in their top three. Providers were more likely to put benefits in their top three compared to patients (for benefits of chemotherapy: 38% versus 12%, 95% CI of the difference −52, 1; for benefits of hormone therapy: 33% versus 12%, 95% CI of the difference −47, 4).

Table 3c.

Chemotherapy and hormone therapy: percentage of patients and providers who ranked each fact and goal in their top 3.

Chemotherapy/Hormone Therapy Fact*	Patient (n=17) %	Provider (n=27) %	95% CI of the difference
Best choice depends on medical factors and patient’s feelings about benefits and side effects.	41	52	−20	43
Realistic estimate of the risk of recurrence with CHT.	41	48	−38	25
Chemotherapy can reduce recurrence and improve survival.	12	38	−52	0
HT can reduce recurrence, improve survival in women with hormone receptor positive tumors.	12	33	−47	4
Size of the benefit from CHT depends on size of the risk of recurrence.	29	33	−26	34
Realistic estimate of the prognosis, recurrence, or death without CHT.	18	33	−43	11
60–75% will be cancer free 10 years after surgery without CHT.	29	29	−28	30
CHT reduces recurrence more than it increases survival.	6	14	−27	10
Realistic estimate of how much CHT would help lengthen life.	29	14	−42	11
Women with serious health problems may gain less benefit from CHT.	35	5	6	55
CH can cause short term, temporary side effects (nausea, vomiting, hair loss, fatigue).	12	0	−4	27
CH can cause rare, serious side effects (heart problems, cancers, infection, clots).	24	0	3	44
HT can cause temporary side effects (hot flashes, vaginal discharge, sexual problems).	6	0	−5	17
HT can cause rare, serious side effects (blood clots, endometrial cancer, bone loss).	6	0	−5	17
Chemotherapy/Hormone Therapy Goal*	Patient (n=20) %	Provider (n=27) %	95% CI of the difference
Live as long as possible.	59	96	12	62
Reduce the chance that the cancer will come back.	82	91	−13	30
Avoid serious side effects of chemotherapy.	29	41	−41	18
Be able to say that you did everything possible.	35	41	−25	36
Avoid serious side effects of hormone therapy.	25	9	−4	45
Do what your doctor(s) think is best.	6	9	−20	13
Avoid short-term side effects of chemotherapy.	0	5	−13	4
Avoid lengthy treatment.	24	5	−3	41
Maintain fertility.	6	5	−13	16
Avoid short-term side effects of hormone therapy.	12	0	−4	27

CI = confidence interval; CHT = chemotherapy and/or hormone therapy; CH = chemotherapy; HT = hormone therapy

^*shortened from the original wording

Bold print denotes statistically significant difference, p<0.05.

CHT goals

Patients and providers ranked the goals similarly, although the emphasis was slightly different. Significantly more providers felt that “live as long as possible” was the main goal for CHT decisions (95% vs. 59%, 95% CI of the difference 12, 62). There was a trend toward patients being more concerned than providers about short term side effects (12% vs. 0%, 95% CI of the difference −4, 27) and serious side effects (25% vs. 9%, 95% CI of the difference −41,18) of hormone therapy and about the length of treatment (24% vs. 5%, 95% CI of the difference −3,41).

Discussion

Our approach to identifying the key facts and goals for three breast cancer treatment decisions is a significant step forward in the measurement of the quality of these decisions and of preference-sensitive decisions in general. The systematic process we used to identify the salient facts and goals resulted in three sets of items that were accurate, important, and complete. In addition, we observed several interesting differences in the perspectives of patients and providers regarding which facts and goals were most important.

Item development

Previous measures of decision making about breast cancer treatments have been limited by lack of validation (7–12) or involvement of providers only in their development(6, 9). The DQIs follow a consensus-based framework that engages patients and providers in generating items,(25) is based on principles of survey development,(31) and is guided by decision theory (25, 32)

In this first phase of instrument development, we identified three sets of facts and goals. The facts for all three decisions were accurate, with a few clarifying comments added. The importance of the facts and goals was also high, with only CHT having items to delete. Although provider and patient participants wrote in several suggested additions, most were already covered by the existing items or were not relevant to selecting among treatments. These robust findings confirm the validity of the item generation process for breast cancer decisions, a process that has also been validated for symptom-driven conditions. (32)

Significant variability existed within the patient and provider groups, as well as between the groups in prioritizing the facts and goals. In fact, every fact and goal was selected in the top three by at least one patient or provider, yet no item was placed in the top three by 100% of patients and providers. The closest was the surgery goal, “minimize the chance of having cancer come back in the breast,” which was placed in the top 3 by 100% of patients and 92% of providers. Despite this variability, we did identify a core set of facts and goals which met criteria for importance, as rated by patients and providers.

How to handle goals considered highly important by only a few participants is a challenge in the development of decision quality instruments. We were able to identify a set of goals that most patients felt were important, but some goals were ranked in the top three by only one or two patients. Decision quality instruments should be broad enough to accommodate variation in a population, change over time, and differences among populations, but at the same time, they cannot measure every possible goal that patients may have. At this stage of development, we erred on the side of inclusion and deleted an item only if its weighted median importance rating was less than 2 (“somewhat important”), and if fewer than 10% of patients and 10% of providers ranked it in their top 3. We also asked patients to add any items they felt were missing, which resulted in no additions.

A strength of our approach to item development has been the inclusion of patients’ perspectives at each step. Our finding of differences between patients’ and providers’ rankings of the facts and goals affirms the importance of including both perspectives in item generation and item reduction. The differences in rankings may have implications for development of decision aids as well. Many decision aids have been developed by first consulting with provider experts and then obtaining feedback from patients later.(33) Our findings suggest that earlier input from patients in order to frame the issues may be beneficial, as they bring a unique perspective to the process.

Significant limitations to our approach include limited precision and the potential for bias due to small, nonrandom samples. We used convenience samples drawn from and around academic institutions. Patients tended to be white and well-educated, most patients in the reconstruction sample had undergone reconstruction, and all of the patients in the CHT sample were younger than 65 with some college education. Less than half of the providers in the surgery sample were surgeons. Whether or not the issues and emphasis would be the same with a more representative sample of patients is unclear. However, a more diverse sample would probably not yield increased consensus, so we believe that the main insights regarding item development would likely hold. The sample sizes were large enough to examine the content validity of the items, but not for the purpose of drawing conclusions about patient or provider opinion more generally. The samples were also too small to test for associations between a patient’s ranking of the goals and her treatment choice. In the next phase of instrument development, we will test instruments in larger, more diverse populations.

Patient and provider perspectives on facts and goals

Providers tended to be more uniform in their opinions about the relative importance of the facts and goals, while patients’ ratings showed more diversity across items. For example, for the CHT goals, 95% of providers placed “live as long as possible” in the top three, 91% of providers placed “minimize recurrence” in the top three, and almost no providers placed “avoid lengthy treatment”, “maintain fertility”, or “avoid short-term side effects of chemotherapy” in the top three. Providers’ homogeneity, in terms of medical education and training, may account for some of this uniformity, although other factors not accounted for here may have also contributed. The finding that the patient sample had more variability in their opinions affirms the value of including patients in the processes of item generation and reduction.

We were surprised by the relative lack of importance patients assigned to the goal of keeping one’s breast. Seven percent of patients ranked this item as a top three goal, and women who had breast conservation were no more likely to do so. In a recent study that used an early version of the surgery DQI, women who felt strongly about keeping their breast were more than five times as likely to choose breast conservation.(27) Similarly, cross-sectional studies of breast cancer patients have found that fear of cancer, fear of recurrence, concern about losing a breast, cosmetic result, and body image were associated with choice of surgery.(13, 14, 16, 17, 34) In the next phase of investigation, when we validate the DQIs in large samples, we will have a better understanding of the importance of this goal.

Another somewhat surprising finding was the difference in importance patients placed on “doing what the doctors think is best” about surgery, compared to providers. We have found this discrepancy in other conditions, with patients placing more emphasis on the provider than the providers did.(26) In contrast, patients considered this less important for the other decisions, with 24% placing it in the top three for reconstruction and 6% placing it in the top three for CHT. In a recent population-based survey, many survivors reported playing a small role in the decision about surgery, and women who played a greater role were more likely to have had a mastectomy.(9, 35) Since the publication of guidelines favoring breast conservation,(36) surgeons may be more likely to recommend breast conservation without eliciting patients’ preferences.(37)]

Given the technical nature of surgery, one might expect patients to defer decisions about surgery to the surgeon, but we did not see the same result for reconstruction. This discrepancy may reflect a general absence of discussions between patients and providers about reconstruction.(38, 39) Alternatively, patients may see the reconstruction decision as more personal than the surgery decision. Ultimately, we believe that “doing what doctors think is best” is a preference about participation in decisions, and not a fundamental goal, so we have moved this item to a section on participation in decisions, in subsequent versions of the DQIs.

With regard to reconstruction, patients appeared to place more importance on practical matters than providers did. A third of patients placed “avoid using a prosthesis” in the top three concerns, whereas not a single provider did. More providers (40%) than patients (24%) ranked the goal of looking natural without clothing in the top three, although this difference was not statistically significant. More providers (60%) than patients (25%) ranked “radiation can affect the outcome of reconstruction” in the top three. Patients may be less concerned about the effects of radiation on appearance of the reconstructed breast. Alternatively, patients may have ranked this fact highly only if they had had radiation, whereas providers would have been more aware of growing indications for post-mastectomy radiation.(40) How a patient feels about body image is associated with breast reconstruction,(8) but we believe that patients may value reconstruction as a means of returning to their normal lives without having to worry about the clothes they wear or whether they look unbalanced or asymmetric. In general, patients may have less concern than providers think they do about how the reconstructed breast looks without clothing.

Conclusion

This part of the survey development process is replicable and generated accurate, important, and complete sets of facts and goals for each of three breast cancer treatment decisions. Because patients and providers differ in what they see as most important to these decisions, including patients in the process of identifying the key facts and goals is critical to the instrument development process.

Survey instruments that can reliably examine knowledge and patients’ preferences for different attributes of treatment are important. As guidelines and performance metrics increasingly emphasize shared decision making and the importance of informed patients, having adequate measures to determine the extent to which we can achieve those goals will be equally important.