Breast cancer specialists' views on and use of risk prediction models in clinical practice: A mixed methods approach

Abstract

Purpose

Risk prediction models (RPM) in breast cancer quantify survival benefit from adjuvant systemic treatment. These models [e.g. Adjuvant! Online (AO)] are increasingly used during consultations, despite their not being designed for such use. As still little is known about oncologists' views on and use of RPM to communicate prognosis to patients, we investigated if, why, and how they use RPM.

Methods

We disseminated an online questionnaire that was based on the literature and individual and group interviews with oncologists.

Results

Fifty-one oncologists (partially) completed the questionnaire. AO is the best known (95%) and most frequently used RPM (96%). It is used to help oncologists decide whether or not to recommend chemotherapy (> 85%), to inform (86%) and help patients decide about treatment (> 80%), or to persuade them to follow the proposed course of treatment (74%). Most oncologists (74%) believe that using AO helps patients understand their prognosis.

Conclusion

RPM have found a place in daily practice, especially AO. Oncologists think that using AO helps patients understand their prognosis, yet studies suggest that this is not always the case. Our findings highlight the importance of exploring whether patients understand the information that RPM provide.

Deciding about adjuvant systemic therapy for breast cancer can be a difficult balancing act between potential survival gains and side effects. Many risk prediction models (RPM) have been developed to primarily aid oncologists' decision-making about adjuvant systemic treatment [1]. RPM seem to meet a need and appear to have been widely adopted in clinical practice. For example, the Dutch breast cancer adjuvant systemic treatment guidelines are largely based on Adjuvant! Online's (AO) survival and treatment benefit estimates [2]. The American National Comprehensive Cancer Network (NCCN) guidelines have incorporated Oncotype Dx in their adjuvant systemic treatment decision-making algorithm [2,3]. The British National Institute for Health and Clinical Excellence (NICE) has incorporated the Nottingham Prognostic Index in their decision algorithm and both NICE and NCCN endorse the use of AO to support estimations of individual prognosis and absolute benefit of adjuvant treatment [3,4].

A 2005 questionnaire amongst American medical oncologists found that 80% had ever used Oncotype Dx, and that 78% used AO [5]. A small questionnaire study amongst 25 British medical oncologists from 13 oncology centers found that 96% of the participants used AO to calculate mortality estimates and 36% also used it to calculate relapse probabilities. Most participants (≥ 84%) were confident that AO estimates are accurate [6].

Most RPM offer graphical representations of prognostic information, and this increases their appeal for use in the consultation to convey prognostic information to patients. The UK-based questionnaire found that 92% of participants regularly discussed the survival probabilities and treatment benefit estimates from AO with their patients, and a quarter also said they provided patients with the printout from AO [6]. Not much is known about such use of RPM during the consultation (i.e. frequency and reason for use) and similarly, little is known about how well patients understand prognostic information from RPM. The information these models provide is complex and could cause confusion if risk communication is not done properly, and increase patients' anxiety. Patients tend to have problems understanding probabilities, in part due to limited understanding of health statistics [7,8]. Two small studies (< 30 patients) assessing patients' understanding of prognostic information before and after receiving results from AO reported that 43–65% were not able to accurately recall recurrence-free (RFS) and/or overall survival (OS) immediately after the consultation with their medical oncologist [9,10]. In a few patients the use of AO printouts led to heightened confusion and decreased comprehension [10]. Simplifying AO's printout resulted in significantly more accurate recall [11], although at the cost of information loss.

A drawback of RPM is that the point estimates they provide reflect average outcome probabilities derived from groups of similar patients [7]. AO provides survival estimates as point estimates without the confidence interval surrounding the estimates. Knowing the width of the confidence interval could help oncologists gauge how robust AO's survival estimates are. Yet, it is unknown if oncologists are interested in this type of information and if and how they would disclose the associated uncertainty to their patients. Many patients have difficulties understanding uncertainty [7]; and the effect of and how best to share uncertainty with patients is unknown [12,13].

Given the lack of information on the use of RPM to communicate prognosis to patients, and the pitfalls if not done appropriately, we assessed oncologists': a) familiarity with and use; b) reasons for use, for themselves and with patients; c) views on the (dis)advantages of RPM; and d) wish for uncertainty estimates and views about communicating these to patients.

Methods

Questionnaire development

Given the limited literature on this subject, we first conducted semi-structured interviews (N = 10) with surgical and medical oncologists. We aimed to conduct a minimum of 10 interviews, and during the analysis process we also observed that after 10 interviews new categories, themes or explanations stopped emerging (data saturation). Subsequently, we held two online focus groups with a new group of surgical and medical oncologists (8 active participants of 20 who agreed to participate). Oncologists attending the 2011 Dutch Medical Oncology congress and members of the Comprehensive Cancer Centre The Netherlands (IKNL) medical oncology and breast cancer working parties were invited to participate via e-mail, if they wanted to participate they indicated their preference for either an interview or focus group. IKNL has a nationwide coverage, facilitating the recruitment of our target population throughout The Netherlands.

The themes explored in the interviews were oncologists': a) familiarity with and use; b) reasons for use, both for themselves and with patients; c) views on the (dis)advantages of RPM; and d) wish for uncertainty estimates and views about communicating these to patients. We used the information obtained in the interviews to formulate statements, which we posted on a website especially created for these online focus groups. The online focus group participants were asked to post their views about the statements during a four-week period. They were also able to respond to other participants' posts. Participants were not aware of each other's identity. The data from the interviews and online focus groups were independently coded by two researchers using NVivo 9 software, and an open coding system. Discrepancies in coding were resolved by consensus.

Next, we used the data from the interviews and online focus groups to develop an online questionnaire. With the online questionnaire we explored all the themes (a–d) described above (Supplementary Appendix 1, available online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2014.964810). We also assessed participants': a) characteristics; and b) general reluctance to disclose uncertainty [14]. To limit participants' time investment, most questions were multiple choice; answering categories were based on the findings of our qualitative analyses. Participants were also offered the option of providing open answers.

Recruitment of participants online questionnaire

The Comprehensive Cancer Centre The Netherlands sent out an invitation on our behalf to the members of all regional medical oncology and breast cancer working parties. Medical and surgical oncologists were eligible to participate in the current study. Participants could anonymously complete the questionnaire online or on paper. Four weeks after sending the initial invitation, a reminder was sent to the working parties.

Data analysis

For privacy reasons we could not access data on the size and composition of the working parties; and are unable to estimate the response rate. The proportion of surgical and medical oncologists in our sample was similar to the distribution of the specialties in a reference sample of IKNL-working parties across The Netherlands. Participants who only partially completed the online questionnaire were included in the analyses if they had answered at least the questions on the (dis)advantages of RPM in general. Descriptive analyses were performed, as well as comparisons between groups, using χ² or Fisher's Exact Tests for categorical variables and Student's t-test for continuous variables, all using SPSS 20.

In the results we will focus on the RPM that the majority of oncologists use most frequently illustrate oncologists' views on and how they use RPM in general. Further, we will present quotes from the interviews and online focus groups to illustrate the quantitative findings.

Results

Fifty-one participants were included (Supplementary Appendix 2, available online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2014.964810) and 77% of them completed all questions. There were no significant differences between the participants who had fully or partly completed the questionnaire (Supplementary Appendix 3, available online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2014.964810). On average the participants were 49 years old, 44% were female, and 82% worked in teaching hospitals (general or university) (Table I). We found no significant difference in socio-demographic and work-related characteristics between surgeons and medical oncologists.

Table I.

Participants' characteristics (N = 51)^∗.

	Surgeons N (%)	Medical oncologists N (%)
Average age in years (range)	50 (37–64)	48 (31–62)
Age unknown	8 (32)	5 (19)
Gender
Male	12 (71)	10 (48)
Experience with breast cancer care in years
< 5	5 (20)	10 (39)
6–10	9 (36)	9 (35)
> 10	11 (44)	7 (27)
Number of consultations with early-stage breast cancer patients per month
1–5	1 (4)	3 (12)
6–10	7 (7)	12 (46)
> 10	17 (68)	11 (42)
Type of hospital
General teaching hospital	10 (59)	12 (55)
University medical center	4 (24)	6 (27)
General non-teaching hospital	3 (18)	4 (18)
Total	25 (49)	26 (51)

^∗Participants do not add up to 51 due to missing data.

No significant differences between surgical and medical oncologists, hence p-values not reported.

Familiarity with and use of RPM in clinical practice

The best-known RPM amongst oncologists were AO (95%) and MammaPrint (88%). About one third were familiar with Oncotype Dx and 19% with the Nottingham Prognostic Index. Overall, 71% of surgical oncologists reported to sometimes or regularly use RPM, compared to 100% of medical oncologists (p = 0.007; Fisher's exact test) (Table II). Of those who use RPM, medical (100%) and surgical (95%) oncologists indicated that they most frequently use AO. If MammaPrint was used, in most cases it was to supplement AO. For example, if the patient and/or the oncologist were leaning towards foregoing chemotherapy, the MammaPrint results were decisive in determining the probability that forgoing chemotherapy would negatively affect RFS.

Table II.

Frequency of RPM use (in N (%)).

	Surgeons N = 24∗	Medical oncologists N = 25∗	P#
Never	4 (17)	0	0.007
Ever	3 (13)	0
Sometimes	9 (38)	7 (28)
Regularly	8 (33)	18 (72)

^∗Participants do not add up to 51 due to missing data.

^#Comparison made using Fisher's exact test.

We asked participants which estimates, 10-year OS or RFS, they most frequently consulted a) before and b) during consultations with patients. Both surgical (63%) and medical (71%) oncologists reported that they usually consulted both estimates before the consultation. If only one was consulted, it most frequently was OS (21%). The majority indicated that they preferred OS because the main aim of adjuvant systemic treatment is to improve OS. There were also some concerns about the robustness of the relapse estimates, as in AO no distinction is made between loco-regional and distant recurrences. One in three oncologists indicated that they habitually showed patients only the OS estimates and about half reported to show patients both the OS and RFS estimates. Oncologists indicated that AO estimates are not too difficult to show to patients (Table III). Some think that estimates from AO should always be disclosed to patients, except if the patient strongly objects to hearing this information. Most medical (63%) and surgical (74%) oncologists indicated that one should ask patients if they want to hear AO estimates, and if so, provide them with the estimates.

Table III.

Oncologists' views on using Adjuvant! Online (AO) during the consultation (in %).

Oncologists should:		Surgical oncologists (N = 19$)			Medical oncologists (N = 24$)
		Disagree	Neutral	Agree	Disagree	Neutral	Agree
A	not show AO estimates to patients as it is too difficult for them	84	16	0	83	17	0
B	not show AO estimates to patients as people cling too much to the estimates	53	47	0	75	21	4
C	never show AO estimates to patients, it is best to use verbal labels# instead	42	42	16	71	21	8
D	not show AO estimates to patients if these estimates are too hard to face	47	32	21	63	12	25
E	only show AO estimates to highly educated patients as they are best capable of understanding this information	63	21	16	83	12	4
F	offer to show AO estimates to patients and show the estimates if the patient wants to see it	16	10	74	17	20	63
G	always show AO estimates, unless the patient absolutely does not want to hear this	53	26	21	79	8	13
H	always show AO estimates to breast cancer patients ≤ 40 years, as this information is most relevant for these patients	53	36	11	71	16	13
I	always show AO estimates if the patient asks for information on prognosis	0	32	68	17	8	75

^$Participants do not add up to 51 due to missing data.

^#Verbal labels are terms used to denote likelihoods, e.g. “small chance that x will happen” or “it is likely that x will happen”.

The category “disagree” comprises of those that selected either “totally disagree” or “disagree”. And the category “agree” comprises of those that selected either “agree” or “totally agree”.

No significant differences between surgical and medical oncologists were found, hence p-values Fisher's exact test are not reported.

Of medical oncologists, 42% indicated that they ask patients if they want a printout to take home, compared to 11% of surgical oncologists [(p = 0.04); Fisher's exact test]. Most surgical oncologists (61%) indicated that they do not actively offer a printout, but provide it if asked. Moreover, many participants (63% of medical and 47% of surgical oncologists) feel that oncologists should disclose AO estimates to patients even if they forecast a bleak outlook. As an oncologist said: “Before I disclose AO's estimates I tell patients that the estimates could be quite hard to stomach and check whether they still want to hear it…. if they still do, I discuss them”.

Reasons for using RPM for themselves or with patients

More than 90% of oncologists sometimes use AO to prepare the consultation; one in four medical oncologists always use AO to prepare the consultation. Oncologists predominantly consult AO before the consultation, to decide whether or not to recommend chemotherapy alone (87%) or in combination with endocrine therapy (91%). AO is also consulted to decide about endocrine monotherapy (60%). Up to one in four oncologists (surgical more often than medical oncologists) also use AO to decide about neo-adjuvant systemic therapy. Overall, 85% of surgical and 76% of medical oncologists indicated that their treatment preference sometimes changed after consulting a RPM. If there was a shift in medical oncologists' treatment preference, it was caused by either viewing the results of AO alone (42%) or in combination with MammaPrint (58%).

Surgical oncologists indicated to regularly use AO to help patients decide whether or not undergoing chemotherapy is worthwhile (73%) (Table IV). Medical oncologists stated to use AO to provide patients with prognostic information (100%) and/or to help patients decide whether or not to undergo chemotherapy (96%). Additionally, 75% of medical oncologists indicated that they sometimes/regularly use AO to convince patients that undergoing chemotherapy is not necessary and 83% also use it occasionally to convince patients of the benefit of their proposed treatment plan.

Table IV.

Oncologists' reasons for using Adjuvant! Online (AO) (in %).

		Surgical oncologists (N = 19$)					Medical oncologists (N = 24$)
		Never	Rarely	Sometimes	Often	Always	Never	Rarely	Sometimes	Often	Always	P#
Oncologists use AO before the consultation to:
A	prepare for the consultation	11	21	37	26	5	8	8	25	33	25	0.371
Oncologists use AO during the consultation to:
B	inform patients	16	16	53	16	0	0	0	38	50	13	0.003
C	inform patients who ask about prognosis	16	16	26	37	5	0	4	21	54	21	0.099
D	present the survival probabilities graphically	16	16	47	11	11	4	30	22	39	4	0.070
E	convince patients that undergoing chemotherapy is not necessary	42	11	26	21	0	4	21	63	8	4	0.006
F	help patients decide whether or not to undergo chemotherapy	21	5	47	26	0	4	4	25	63	4	0.050
G	convince patients of the benefits of my treatment plan	32	5	42	21	0	0	17	58	21	4	0.024

^$Participants do not add up to 51 due to missing data.

^#Comparison made using Fisher's exact test.

Not significant.

Medical (96%) and surgical (75%) oncologists reported that the output of RPM not only influenced their own decisions, but also those of their patients. In all, 56% of surgical and 70% of medical oncologists indicated that they frequently observe hesitation with regard to chemotherapy, yet after seeing AO's prognostic estimates patients change their minds.

Over 70% of oncologists think that AO helps patients to understand their prognosis better. Conversely, about 14% think that AO does not make it easier for patients to understand their prognosis, but makes it easier for them to discuss prognosis with patients.

Views on the (dis)advantages of RPM

The two most frequently cited concerns about RPM were: 1) estimates only provide insights at a group level (34%); and 2) those based on genetic profiles, e.g. MammaPrint or Oncotype Dx, are not yet sufficiently validated for use in clinical practice (36%). Twelve percent of medical oncologists indicated that another important drawback of RPM is that they give patients a false sense of security: “As you can imagine, when people who feel the need to keep a tight grip on their illness or their life find themselves in a situation in which all certainties have been taken away, that they desperately look for something to cling to… it's very hard to get them to put these estimates in perspective”.

We asked oncologists to indicate their main concerns with regard to AO specifically. They consistently indicated that AO is one of the best RPM currently available, but far from perfect. The accuracy of AO's estimates in some patient populations, e.g. in the elderly (> 65 years), is possibly suboptimal. Some felt that it would be informative, especially for younger patients and those with hormone receptor positive disease, if AO were to provide prognostic estimates up to 20-years follow-up, instead of only 10-year estimates. The majority (85%) indicated that AO is currently missing important prognostic factors, particularly her2neu receptor status. Also, preferably AO should take the effect of trastuzumab into account. More than three quarters indicated that the way prognostic factors are categorized in AO is not ideal, or that it is unclear how the categories should be interpreted. Many felt the categorization of nodal status too crude (i.e. 0 positive; 1–3 positive; 4–9 positive and > 9 positive nodes). “A patient with one positive node would reasonably be expected to have a better prognosis than a patient with three positive nodes.” It is currently unclear how to classify patients with micro-metastases; classifying them as node negative might yield prognostic estimates that are too optimistic, but classifying them as having 1–3 positive nodes seems to be a gross exaggeration.

It was often mentioned as an asset that AO takes comorbid conditions into account, but most participants do not know how to interpret the categories AO uses (i.e. perfect health; minor problems; average for age; major problem + 10; major problem + 20 and major problem + 30). “If a patient has well- managed diabetes, is that a minor problem or is it a major problem?” Over 80% of oncologists indicated that they tend to use the default setting, namely “minor problems”. However, if a patient has significant comorbidities, choosing a comorbidity category is often a bit of guesswork; oncologists try out multiple categories to see what happens with the estimates, and stick with the one they think yields the most realistic survival estimates.

Views on communicating uncertainty around the estimates from RPM

One in three (37%) thought that a confidence interval would be of no added value to them, with most indicating that they assume that AO's estimates are sufficiently accurate because the Dutch breast cancer guidelines are partly based on AO. Half (49%) would want to know the width of the confidence intervals to determine for themselves how much credence they should give the estimates.

One in five oncologists are highly reluctant to disclose uncertainty to patients; yet, 95% of surgical and 100% of medical oncologists indicated that they currently discuss the uncertainty associated with AO's estimates with their patients in general terms. One oncologist said: “Uncertainties are a part of consultations with patients. We should not shy away from communicating them.” Using an open-ended question, we asked oncologists to briefly describe how they communicate uncertainty around AO's estimates to patients. The two most frequently reported methods were: 1) telling patients that the estimates do not say anything about an individual, they are true at a group level (46%); and 2) telling patients the estimates are based on statistics (14%). An oncologist provided the following illustration of their current explanation of uncertainty associated with AO's estimates: “these probabilities apply to groups and not to a specific individual. Survival or recurrence is for an individual 0% or 100% – it either happens or it does not. I tell them that AO is a tool that facilitates discussion about adjuvant systemic treatment, it does not predict your fortune”. Other frequently mentioned phrases used to explain the uncertainty associated with AO's estimates to patients are: “each case is unique, you never know how a specific patient will react to treatment”, “AO's estimates are only a general guideline” and “I cannot predict in which category a patient will end up… either cured without treatment, those that are cured because of treatment or those that will relapse or die in spite of treatment”.

If estimates of the confidence interval were available, over 75% of oncologists say that they would disclose the confidence interval surrounding AO's estimates to patients, whom they think are capable of understanding this. It is worth noting that this high willingness to communicate the actual confidence interval to patients would not necessarily translate to a high rate of uncertainty communication during consultations with patients as oncologists are only willing to share this information if they feel that a patient is able to understand it. During the interviews almost all oncologists expressed the view that many patients are not able to grasp the concept of uncertainty, e.g.: “I think that for patients, for 99% of patients, the information on confidence intervals would be completely lost on them”. A medical oncologist poignantly remarked: “Sometimes I think patients can't handle uncertainty, but doctors probably struggle with it even more…”.

Discussion

We assessed oncologists' views on RPM and their use of these tools. AO is the most frequently used RPM, with many oncologists using it to prepare their consultation and use AO in the encounter to inform and/or help patients decide about treatment. About half sometimes use AO to convince patients of the merits of the proposed treatment plan. Surgical and medical oncologists' role in decision-making about adjuvant systemic treatment differs, hence we found some differences in frequency and motivation for using RPM. Unexpectedly, we found that up to a quarter of oncologists also used AO to decide about neo-adjuvant systemic therapy. AO has not been validated for this purpose, and it is not known whether the estimates hold in the neo-adjuvant setting.

MammaPrint was the best-known RPM based on a gene profile, but was rarely used. Most oncologists indicated that such RPM do not yet have sufficient scientific underpinning to guide treatment decision-making. Many indicated that they are awaiting the results of the Mindact trial¹ and TAILORx trial², to know whether high risk patients according to AO but low risk according to MammaPrint or Oncotype Dx, respectively, can be spared chemotherapy without negatively affecting RFS.

Oncologists expressed concern about the validity of AO's estimates in specific subgroups and felt some key prognosticators were missing, inappropriately categorized or it is difficult to categorize patients into. These views are in agreement with the results of our recent systematic review [1]. In spite of these limitations, most felt that AO is a helpful tool and that no matter how complete the RPM, it will always be impossible to provide patients with a 100% certainty about disease outcome or treatment effect.

Most felt that using AO during consultations helped patients understand their prognosis better. Moreover, in general oncologists did not think that the complex nature of AO's estimates and the fact that these estimates could be hard to hear for patients, are reasons not to use AO during consultations. Oncologists even reported high willingness to communicate about the uncertainty surrounding the estimates of RPM to patients.

There are not many studies we can compare our findings to. A study that assessed the communication of uncertainty about risks and benefits of various treatments in outpatient clinics found that uncertainty was discussed in about 1% up to 16% of consultations depending on the difficulty of the decision at hand [15]. It would be interesting to get insights in how and how often oncologists actually discuss uncertainty in daily practice, since there are no guidelines available on how uncertainty should best be communicated [12]. Moreover, it is unclear to what extent patients understand the uncertainty around RPM estimates and how information on uncertainty affects them personally as well as their final treatment decision.

Unfortunately, we were unable to determine our response rate. Also, the number of participants was relatively small. This is partly explained by the fact that we recruited participants via the IKNL-working parties which consist of a highly motivated, yet relatively small subgroup of experienced oncologists.

In conclusion, RPM have found their way into the consultation. It is encouraging that oncologists are driven to obtain the best possible prognostic estimates to guide their own decision-making and to communicate this information to patients, which in turn may facilitate patient participation in decision-making. However, clinicians assume that using RPM during consultations helps patients understand their prognosis better. Studies on patient understanding of prognosis [9,10] suggest that using AO does not necessarily facilitate or improve patient understanding. Large observational studies of the communication process between oncologists and patients involving RPM are urgently needed to get insight into whether patients indeed understand the risks communicated during the consultation, and whether this enhances their participation. Additionally, studies assessing patients' understanding and acceptance of communication about uncertainties are needed to guide practice on communicating uncertainties.

Supplementary material available online

Supplementary Appendices 1–3 available online at http://informahealthcare.com/doi/abs/10.3109/0284186X.2014.964810.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

E.G. Engelhardt is funded by a grant from the Dutch Cancer Society (grant UL2010-4805).