The Predictive Accuracy of PREDICT: A Personalized Decision-Making Tool for Southeast Asian Women With Breast Cancer

Abstract

Web-based prognostication tools may provide a simple and economically feasible option to aid prognostication and selection of chemotherapy in early breast cancers. We validated PREDICT, a free online breast cancer prognostication and treatment benefit tool, in a resource-limited setting.

All 1480 patients who underwent complete surgical treatment for stages I to III breast cancer from 1998 to 2006 were identified from the prospective breast cancer registry of University Malaya Medical Centre, Kuala Lumpur, Malaysia. Calibration was evaluated by comparing the model-predicted overall survival (OS) with patients’ actual OS. Model discrimination was tested using receiver-operating characteristic (ROC) analysis.

Median age at diagnosis was 50 years. The median tumor size at presentation was 3 cm and 54% of patients had lymph node-negative disease. About 55% of women had estrogen receptor-positive breast cancer. Overall, the model-predicted 5 and 10-year OS was 86.3% and 77.5%, respectively, whereas the observed 5 and 10-year OS was 87.6% (difference: −1.3%) and 74.2% (difference: 3.3%), respectively; P values for goodness-of-fit test were 0.18 and 0.12, respectively. The program was accurate in most subgroups of patients, but significantly overestimated survival in patients aged <40 years, and in those receiving neoadjuvant chemotherapy. PREDICT performed well in terms of discrimination; areas under ROC curve were 0.78 (95% confidence interval [CI]: 0.74–0.81) and 0.73 (95% CI: 0.68–0.78) for 5 and 10-year OS, respectively.

Based on its accurate performance in this study, PREDICT may be clinically useful in prognosticating women with breast cancer and personalizing breast cancer treatment in resource-limited settings.

INTRODUCTION

In early breast cancers, accurate assessment of individual patient prognosis is key to good clinical decision making. At present, a number of prognostic prediction tools are available either commercially or as free online programs to aid adjuvant chemotherapy-related decision making.¹ Oncotype-Dx² and Mammaprint,³ the 2 widely used commercial microarray gene expression-based prognostic tests, are steadily gaining popularity in Western oncology practices as powerful tools in risk stratification, and selection and optimization of drug regimens for women with early breast cancer.⁴ Nevertheless, these assays are inaccessible for a vast majority of the global oncology community, as they are expensive.⁵ A study in France had recently demonstrated that Adjuvant! Online, a free, web-based prognostic calculator, is far more cost effective than Mammaprint.⁶ Furthermore, the clinical utility of multigene assays remains unclear in Asian settings. In Asia, a substantial proportion of women are diagnosed with estrogen receptor (ER)-negative tumors, in part owing to the younger age at diagnosis of breast cancer in the population.^7–9 It has been previously shown that almost half of the breast cancers in several Asian settings including Hong Kong, Singapore, and India occurred in premenopausal women^7–9 and approximately 45% of patients had ER-negative breast cancers.^7,8

Online prognostication tools may therefore be valuable in clinical practice and have been proposed to guide adjuvant chemotherapy decision making, at institutional and even at national levels in affluent countries (http://www.predict.nhs.uk/technical.html).¹⁰ Previous studies in the United States and the Netherlands have shown that the use of prognostic prediction tools were common among the oncologists.^11,12 At present, there are several free web-based prognostication programs to predict the survival of early breast cancer such as Adjuvant! Online,¹³ PREDICT,¹⁴ and CancerMath.net.¹⁵ These prognostic models have been validated in Western settings and generally seem to possess good calibration and discriminatory accuracy,^16–21 with very few exceptions.^12,21,22 However, a series of validation studies conducted within multiethnic Asian patients with breast cancer had consistently shown that Adjuvant! Online was overoptimistic in predicting survival.^23–25 The utility of PREDICT albeit remains unclear given that it has not been validated in Asian settings.

PREDICT is a web-based prognostication tool, which estimates the probability of survival for individual patients with breast cancer and the impact of systemic treatment choices on their survival probability (http://www.predict.nhs.uk/). Compared with Adjuvant! Online, PREDICT uses an algorithm that, additionally, includes mode of detection, human epidermal growth factor receptor (HER2) status, as well as Ki67 status, but does not include patient's comorbidity. An advantage of PREDICT is that it was developed for 2 end users, the clinicians and the patients with breast cancer, involved in chemotherapy decision making. The model was developed using a population-based cancer registry in the United Kingdom, and externally validated in a second United Kingdom-based registry and also a large cohort of Canadian patients with breast cancer.^18,19

Given that PREDICT is free, user friendly, and easily accessible, it may provide an economically feasible option to guide adjuvant chemotherapy decision making in resource-limited settings. Malaysia is a middle-income country in Southeast Asia that comprises 3 major ethnic groups, that is, Malays, Chinese, and Indians.²⁶ The prognostic prediction performance of PREDICT was assessed using a large hospital-based prospective cohort of multiethnic patients with breast cancer in Malaysia.

METHODS

This study obtained ethics approval from the Ethics Committee of the University Malaya Medical Centre (UMMC), Kuala Lumpur, Malaysia. As the study relies on nonidentifiable registry-based data, the need to obtain informed consent was waived.

Study Participants

Data from the UMMC Breast Cancer Registry were used. The UMMC is an academic tertiary hospital situated in the relatively affluent part of Kuala Lumpur, Malaysia, and caters to a predominantly middle-class urban population. The UMMC Breast Cancer Registry is a prospective hospital-based registry of consecutive women who were newly diagnosed with breast cancer since 1993. Details of this registry have been described elsewhere.⁷ The registry has been approved by the ethical review committee of the institution and encompasses data on patient's demography (including age and ethnicity), tumor characteristics (including pathological data on tumor size, number of involved lymph nodes, tumor grade based on Bloom–Scarff–Richardson classification, ER status, and HER2 status based on immunohistochemistry testing), as well as treatment. Treatment data include type of treatment (surgery, chemotherapy, radiotherapy, and endocrine therapy), type of surgery (mastectomy, breast-conserving surgery), chemotherapy regimen, and type of endocrine therapy. Data on Ki67 status were not available as it was not tested in a vast majority of patients with breast cancer. Although data on use of trastuzumab were also not available in this registry, its use in the hospital was extremely low during the study period.

For this study, consecutive patients who were newly diagnosed with invasive nonmetastatic breast cancer from 1998, when second-generation (anthracycline-based) chemotherapy was introduced in the hospital, until 2006, allowing at least 5 years of follow-up (or until 2002, allowing 10 years of follow-up) were identified. Patients were included if they had undergone complete surgical treatment (ie, mastectomy or breast-conserving surgery followed by radiotherapy). We excluded women with ER-positive tumors who did not receive hormone therapy, and those with missing information on chemotherapy regimen. Patients receiving first-generation chemotherapy were also excluded given that first-generation regimes were not included in PREDICT's algorithm. The final validation dataset for 5-year survival validation comprise 1480 women. For validation of 10-year survival, only 472 women who were diagnosed until 2002 were included.

Follow-Up

In this study, the vital status of patients were obtained by direct linkage with the National Registration Department in Malaysia that has the mortality records of all Malaysians, using patients’ unique identity card number. As passive follow-up was done, we were able to follow-up all patients. Follow-up time was calculated, starting at date of diagnosis with breast cancer until death (all cause), or censored at the end of follow-up (March 2013, date of linkage with national mortality registry).

Predicted 5 and 10-Year OS

For each patient, data on age (continuous), mode of detection (screen detected, symptomatic, unknown), tumor size (continuous), number of involved lymph nodes (continuous), ER status (positive, negative, undefined), tumor grade (grade 1, grade 2, grade 3, undefined), HER2 status (positive, negative, undefined), Ki67 status (entered as undefined for all patients), and adjuvant chemotherapy regimen (no chemotherapy, second-generation chemotherapy, third-generation chemotherapy) were manually entered. For every entry, the program predicted 5 and 10-year overall survival (OS) for 4 different scenarios, that is, survival with no adjuvant treatment, benefit of adjuvant hormone therapy, additional benefit of adding adjuvant chemotherapy to adjuvant hormone therapy, and additional benefit of adding trastuzumab to adjuvant chemotherapy and hormone therapy. The survival probability corresponding to the actual treatment received by the individual patient was recorded. For instance, in the event a patient did not receive adjuvant chemotherapy, the predicted OS for no adjuvant treatment will be extracted from the output charts generated by PREDICT tool. In order to ensure accuracy, all the PREDICT scores were calculated by 2 research personnel, and further audited by a third person in a random sample of 10% of patients.

Statistical Analysis

Kaplan–Meier analysis was used to estimate the observed 5 and 10-year OS in the entire study population and within subgroups. The predicted 5 and 10-year OS were derived from the median of individual predicted survival probabilities from PREDICT. To assess the calibration of the PREDICT model, the observed and predicted 5 and 10-year OS rates were compared.^16,23 A χ² goodness-of-fit test was also performed to assess the fit of PREDICT model.¹⁹ A P value of <0.05 for this test was considered to represent a significant difference between the observed and predicted mortality. Model calibration was further assessed within strata of other prognostic subgroups. The observed 5 and 10-year mortality estimates were plotted against quintiles of predicted mortality.¹⁹

Receiver-operating characteristic (ROC) analysis was performed to assess the discriminatory performance of PREDICT (ie, its ability to discern patients having good prognosis—alive after 5 or 10 years—from those having poor prognosis—death within 5 or 10 years).^27,28 The area under the ROC curve (AUC) gives an indication of the discriminatory performance of the model, whereby it can be interpreted as the proportion of patients who are correctly predicted to be alive or dead at 5 and 10 years. An AUC of 0.5 indicates no discriminative performance whereas an AUC of 1.0 indicates perfect discrimination.

The National Institutes of Health Consensus Development Conference recommends administration of adjuvant chemotherapy for node-negative breast tumors when tumor size is >1 cm, irrespective of hormone receptor status.²⁹ Within this subgroup of women, the accuracy of PREDICT was assessed. Subsequently, the program was used to predict the OS benefit conferred by chemotherapy in women in the above subgroup.

The utility of PREDICT is less clear in patients who have received neoadjuvant chemotherapy, as the program per se does not specify whether the model is suitable for use in this subset of patients. Therefore, the performance of PREDICT within women receiving neoadjuvant chemotherapy was also determined. The tumor size and lymph node status entered in this analysis comprised values following administration of neodjuvant chemotherapy.

P values of <0.05 were considered statistically significant. Analyses were performed using SPSS for Window version 20.0 (SPSS Inc., Chicago, IL).

RESULTS

In this cohort of Asian women with early breast cancer, the median age at diagnosis was 50 years (quartile 1: 44 years, quartile 3: 58 years). A large proportion was Chinese (n = 994, 67%), followed by Malays (n = 281, 19%), Indians (n = 192, 13%), and other races (n = 13, 1%). A vast majority of patients were symptomatic at presentation (n = 1421, 96%), whereas only 4% of women in this study had mammographic screening-detected breast cancer.

The median tumor size at presentation was 3 cm (quartile 1: 2 cm, quartile 3: 4 cm), and 663 patients had lymph node involvement (46%). Data on ER and HER2 status were not available in 47 and 179 patients, respectively, whereas tumor grade was unknown in 259 women. Within patients with available information, ER and HER2 were expressed in approximately 771 (54%) and 496 (38%) patients, respectively, whereas 484 (40%) patients had high-grade tumors. Among 1054 patients who received adjuvant chemotherapy, 17 patients (1.6%) did not complete their chemotherapy cycles. Only 38 women (4%) had received taxane-based (third-generation) regimes, whereas 1016 women received anthracycline-based (second-generation) regimes. A majority of patients who received endocrine therapy were given tamoxifen, whereas 25 received aromatase inhibitors and 20 received tamoxifen followed by aromatase inhibitors.

Figure 1 shows the distribution of the predicted 5 and 10-year OS as estimated by PREDICT in our patients. Although most patients had high predicted 5 and 10-year OS, a smaller number of women were predicted to have low OS. As the predicted scores were negatively skewed, we reported the median score.

FIGURE 1.

Distribution of predicted (A) 5 and (B) 10-year overall survival in Asian women with early breast cancer by PREDICT tool.

In patients with ER-positive breast cancers, endocrine therapy was estimated to confer an additional survival benefit of 2.6% and 5.3% in 5 and 10 years, respectively. Administration of adjuvant chemotherapy in this group of women was predicted to provide an additional OS benefit of 1.7% and 3.7% at 5 and 10 years, respectively. Among patients with ER-negative breast cancers, the program predicted that adjuvant chemotherapy would confer an additional OS benefit of 9.5% and 11.3% at 5 and 10 years, respectively.

Although PREDICT tool had predicted that of 1480 patients with complete follow-up of 5 years, 203 would have died, we had observed 184 deaths. Among 472 patients with complete follow-up of 10 years, the predicted number of deaths was 106, whereas in reality, a total of 122 deaths were observed.

Overall, PREDICT was accurate in predicting short-term survival; the predicted 5-year OS was 86.3% versus the actual observed 5-year OS that was 87.6% (difference: −1.3%). The P value for goodness-of-fit test was 0.18. However, the tool slightly overestimated long-term survival; the predicted 10-year OS was 77.5% whereas the observed 10-year OS was 74.2% (difference: 3.3%). The corresponding P value for goodness-of-fit test for 10-year OS was 0.12. The program seems to have performed fairly well in patients with good prognosis and only displayed overoptimism in patients with the poorest prognosis (lowest quintile) (Figure 2).

FIGURE 2.

Calibration plot of observed mortality with 95% confidence interval against predicted mortality by quintiles of the predicted value, at (A) 5 and (B) 10 years after diagnosis.

PREDICT was also accurate in most subgroups of patients, except in several subgroups wherein a tendency of the program to be overoptimistic was noted (Tables 1 and 2). For instance, in women aged <40 years, 5-year OS was overestimated by 6.8% whereas 10-year OS was overestimated by 17.2%. The program also seems to be overoptimistic in predicting 10-year OS in Malay patients, with a difference of about 10% (P = 0.06). Although the model seems to have underestimated short-term survival in patients with ER-negative tumors, the underestimation was not seen in prediction of long-term survival. Analysis based on HER2 status, however, yielded conflicting results (Tables 1 and 2).

TABLE 1.

Calibration of PREDICT in Asian Women With Early-Stage Breast Cancer: 5-Year Survival

TABLE 2.

Calibration of PREDICT in Asian Women With Early-Stage Breast Cancer: 10-Year Survival

Within patients with node-negative tumors measuring >1 cm, PREDICT was accurate whereby the predicted 5 and 10-year OSs were 90.3% and 82.0%, respectively, whereas the observed 5 and 10-year OSs were 93.3% (95% confidence interval [CI]: 92.5–96.1) and 82.2% (95% CI: 76.5–87.9), respectively. Within this subset of patients, the program had predicted that adjuvant chemotherapy would only confer an additional survival benefit of 1.2% at 5 years, and 3.0% at 10 years irrespective of hormonal receptor status.

It was found that PREDICT was persistently overoptimistic in predicting both short-term and long-term OS among patients who had received neoadjuvant chemotherapy, by approximately 6% (Tables 1 and 2).

The ROC analysis of PREDICT showed that the model discriminated good and poor survivors fairly well, with an AUC of 0.78 (95% CI: 0.74–0.81) and 0.73 (95% CI: 0.68–0.78) for 5 and 10-year survival, respectively.

DISCUSSION

Within this large cohort of women with early-stage breast cancer from a middle-income setting, PREDICT performed well in terms of calibration and discrimination. However, PREDICT substantially overestimated survival in very young patients with breast cancer, as well as in patients who have received neoadjuvant chemotherapy.

The current finding that PREDICT is accurate in predicting OS in early breast cancers and performs robustly across a majority of patient subgroups suggest that the program may be clinically useful in similar Asian settings. It is felt that the program can safely be used to refrain chemotherapy administration in patients demonstrating low survival benefit from adjuvant chemotherapy. For instance, the National Institutes of Health Consensus Development Conference had recommended administration of adjuvant chemotherapy for node-negative breast tumors when tumor size is >1 cm, irrespective of hormone receptor status.²⁹ However, in this subset of patients within the current study, PREDICT had predicted that the additional OS benefit conferred by adjuvant chemotherapy at 10 years was only 3%, underscoring the importance of tailoring treatment approaches based on individual patient prognoses. Nevertheless, the margin of benefit that is deemed appropriate for decision-making is arbitrary and depends on individual physicians and patients. The Cambridge Breast Unit (United Kingdom), for instance, uses the absolute 10-year survival benefit from chemotherapy to guide decision making for adjuvant chemotherapy; no chemotherapy if <3%; chemotherapy discussed as a possible option if 3% to 5%; and chemotherapy recommended if >5%.¹⁸

Based on the current findings, PREDICT is not recommended for use in very young Asian women with breast cancer. A common weakness shared between PREDICT and Adjuvant! Online is that both models poorly performed in very young patients, not only in Asians but also in whites.^16–18,23 Patients diagnosed with breast cancer at a very young age have poor survival,^30–32 and the current variables included in the PREDICT algorithm seem unable to adequately prognosticate this subgroup of patients.

In this study, PREDICT marginally overestimated 10-year OS by about 3%. A previous validation study of Adjuvant! Online within our setting had shown that the model was generally overoptimistic across most subgroups of patients, whereby the 10-year OS was overestimated by approximately 7% (95% CI: 3.0–10.4).²² These results, taken together with the accurate performance of PREDICT in predicting 5-year OS, seem to suggest that systematic differences in background mortality (including comorbidities), between our study population and populations where the above models were developed (ie, United Kingdom and United States), are more likely to play a central role in explaining the overoptimism pertaining to long-term survival predictions. For instance, life expectancy in 2010 for a female at 50 years of age in Malaysia was 78.4 years, whereas in the United Kingdom and United States, it was 83.9 and 83.1 years, respectively (http://www.worldlifeexpectancy.com/world-healthrankings).

Nevertheless, differences in lifestyle after breast cancer between the Western and Asian populations may also explain disparities in long-term survival.³³ This is in view that studies from other high-income Asian settings such as Korea and Taiwan have also shown that Western-derived long-term survival prediction models may still be overoptimistic in Asian populations.^24,25 Differences in life expectancy and lifestyle after diagnosis of breast cancer may also explain the overoptimism of PREDICT in predicting long-term survival among patients of Malay ethnicity.^23,34

Although information on HER2 status was available, the testing was not routine prior to 2005 in UMMC and was highly selective, whereby patients with the highest and lowest baseline prognosis were not tested.³⁵ This may explain the conflicting results based on HER2 status in the current study. We are uncertain of the influence of lack of data on Ki67 status on our results, given that the model was still largely able to accurately prognosticate our patients with breast cancer. Although PREDICT was recently adapted to include Ki67 status in its algorithm, it is important to note that the added prognostic value of Ki67 is yet to be externally validated due to lack of appropriate populations (http://www.predict.nhs.uk/technical.shtml). Furthermore, there is a general lack of clarity in standardizing assessment of these factors for routine clinical practice, particularly in determining optimal cutoff values to establish Ki67 status.³⁶ It is also acknowledged that the number of patients in some subgroup analyses were relatively small, which may have resulted in unstable estimates.

Furthermore, an inherent issue with PREDICT is that the model does not include patient's comorbidity. This maybe problematic in multiethnic Asian women given that Asian populations have been found to have higher risk of cardiovascular diseases compared with their white counterparts.³⁷ Considering that breast cancer therapy, particularly radiotherapy, chemotherapy, and HER2-targeting therapy, may be associated with cardiac toxicity,³⁸ it may be necessary that PREDICT is optimized to take into account the baseline cardiovascular risk profiles of Asian patients. This may be done using newer sophisticated techniques such as artificial neural networks and machine learning.^39–41

Online breast cancer prognostic prediction tools may play a valuable role in clinical practice to aid chemotherapy-related decision making as they are more cost effective than molecular profiling.⁶ Although previous studies, which validated Adjuvant! Online, consistently showed that the program is overoptimistic in middle and high-income Asian settings, the current validation exercise of PREDICT reaffirms that Western-derived prognostic scoring systems may still be relevant and applicable in Asian patients. Nevertheless, it is important to note that validation of such tools are extremely crucial and a prerequisite for adoption into clinical practices in other settings.

In conclusion, PREDICT possesses appreciable clinical utility in resource-limited settings. The program may potentially serve as a clinical decision aid for both patients with breast cancer and physicians in therapeutic decision making. However, PREDICT may not be suitable for use in very young patients (ie, <40 years at diagnosis), and patients who have received neoadjuvant chemotherapy. Future studies are required to optimize the model in Asian settings by inclusion of data on comorbidities and assess the impact of using PREDICT on multidisciplinary tumor board decisions. It is also pertinent to determine the usefulness of the program to patients, in understanding their prognoses and helping them make adjuvant treatment-related decisions.¹