External validation of the first prognostic nomogram for older adults with cancer

Abstract

Background:

The geriatric oncology population tends to be complex because of multimorbidity, functional and cognitive decline, malnutrition and social frailty. Prognostic indices for predicting survival of elderly cancer patients to guide treatment remain scarce. A nomogram based on all domains of the geriatric assessment was previously developed at the National Cancer Centre Singapore (NCCS) to predict overall survival (OS) in elderly cancer patients. This nomogram comprised of six variables (age, eastern cooperative oncology group performance status, disease stage, geriatric depression scale (GDS), DETERMINE nutritional index and serum albumin).

Objectives:

To externally validate the NCCS prognostic nomogram.

Design:

This is a prospective cohort study.

Methods:

The nomogram was developed based on a training cohort of 249 patients aged ⩾70 years who attended the NCCS outpatient geriatric oncology clinic between May 2007 and November 2010. External validation of the nomogram using the Royston and Altman approach was carried out on an independent testing cohort of 252 patients from the same clinic between July 2015 and June 2017. Model misspecification, discrimination and calibration were assessed.

Results:

Median OS of the testing cohort was 3.1 years, which was significantly higher than the corresponding 1.0 year for the training cohort (log-rank p < 0.001). The nomogram achieved a high level of discrimination in the testing cohort (0.7112), comparable to the training cohort (0.7108). Predicted death probabilities were generally well calibrated with the observed death probabilities, as the joint test of calibration-in-the-large estimates at year 1, 2 and 3 from zeros and calibration slope from one was insignificant with p = 0.432. There were model misspecifications in GDS and serum albumin.

Conclusion:

This study externally validated the prognostic nomogram in an independent cohort of geriatric oncology patients. This supports the use of this nomogram in clinical practice.

Introduction

The population of older adults is growing rapidly worldwide, with the World Health Organization estimating that one in six people will be aged 60 years or over by 2030. ¹ While this trend had initially started in high-income countries, low- and middle-income countries are now experiencing this transition. Singapore is no exception, where the proportion of citizens aged 65 and above had increased to 18.4% in 2022 and is estimated to further increase to 23.8% by 2030. ² As cancer risk increases with age, the incidence of cancer has naturally increased with the growing elderly population. ³

The geriatric oncology population presents unique challenges to the management of their cancers due to complexities associated with multimorbidity, functional and cognitive decline, socioeconomic frailty, malnutrition risk and polypharmacy. ⁴ Their under-representation in clinical trials also result in a lack of evidence to support the use of certain treatments.^5,6 Functional status assessment using the eastern cooperative oncology group (ECOG) or Karnofsky Performance Status scores have been shown to poorly correlate with comorbidity and predict functional impairment in the elderly. ⁷ The international society of geriatric oncology and other clinical practice guidelines now recommends a comprehensive geriatric assessment (GA) to guide oncologists’ evaluation and management of elderly cancer patients.^8,9

Prognostic indices for prediction of survival in the geriatric oncology population remain scarce. Brunello et al. ¹⁰ developed the oncological multidimensional prognostic index (Onco-MPI) score that predicts 1-year mortality of geriatric oncology patients in an Italian population. In the Asian population, the first nomogram to predict overall survival (OS) of elderly patients with cancer was developed based on a retrospective analysis of comprehensive GA data collected from 249 consecutive new patients who attended the outpatient geriatric oncology clinic at the National Cancer Centre Singapore (NCCS) between May 2007 and November 2010. ¹¹ Six independent factors were included in the nomogram: age, serum albumin levels, ECOG performance status, disease stage, malnutrition risk as assessed using the DETERMINE nutritional index, and the geriatric depression scale (GDS). The nomogram was successfully validated internally based on simulated data via bootstrapping. This present study was carried out to externally validate this prognostic nomogram in order to boost its clinical validity and application.

Methods

Patient cohort

This study examined two patient cohorts viz. a training cohort and a testing cohort. The training cohort comprised of the 249 patients involved in the development of the nomogram. The testing cohort for validating the nomogram comprised of 252 new patients who attended the same outpatient geriatric oncology clinic at the NCCS between July 2015 and June 2017.

Patients for the testing cohort were recruited based on the same criteria used for the training cohort which had been described previously. ¹¹ Briefly, patients aged 70 years or older with a diagnosis of cancer at any stage were included in the study. Patients who met the inclusion criteria were invited to participate in the study by a trained research coordinator. The GA questionnaire was administered to patients who agreed to participate in the study in English or Mandarin by the research coordinator. GA data were prospectively collected during patients’ first visit to the clinic.

GA questionnaire

The questionnaire used for the testing cohort was broadly similar to the one used for the training cohort. It consisted of seven distinct domains, viz. functional status, comorbidity, cognitive status, affective status, polypharmacy, nutritional status and geriatric syndromes (dementia, delirium, depression, failure to thrive, neglect or abuse, osteoporosis, falls and incontinence). Functional status was evaluated using ECOG performance status (score range 0 to 4; higher score indicating worse functional status), the Barthel index of activities of daily living (ADL) (score range 0 to 100; 0 indicating total dependence, 100 indicating full independence) and Lawton and Brody instrumental ADL (score range 0 to 8; lower score indicates a higher degree of dependence). Physical performance tests such as the Get Up and Go test, Balance test and the dominant hand grip strength test were also administered. Comorbidity was scored according to the Charlson comorbidity index, and patients were classified into comorbidity classes ranging from low (0 points), medium (1–2 points), high (3–4 points) and very high (⩾5 points). Cognitive status was evaluated using Folstein’s mini mental state examination (MMSE) (a score of less than 24 out of 30 points was used for the diagnosis of cognitive impairment), clock drawing test, ¹² and abbreviated mental test. The patient’s mood was screened using the GDS short form 15 (with a score of more than five suggesting depression). Polypharmacy was detailed in terms of number of medications, its appropriateness, and its interactions. Nutritional status was assessed using the body mass index and the DETERMINE nutritional index. Patients were classified into good (0–2 points), moderate (3–5 points) and high (⩾6 points) nutritional risk groups. Geriatric syndromes assessed were those as described by Balducci and Yates. ¹³ Demographic and clinical characteristics including selected blood tests were also collected in the questionnaire. All-cause mortality data for the testing cohort were obtained from the Singapore Registry of Births and Deaths (SBD).

Statistical analysis

Continuous and categorical characteristics between the two patient cohorts were compared using the Mann–Whitney U test and Fisher’s exact test, respectively. Mortality data from SBD were cut-off as at 12 April 2019 for analysis. OS was defined from date of diagnosis to date of death, and alive patients were censored at 12 April 2019. OS distribution was estimated using the Kaplan-Meier method, and compared using the log-rank test. Cox proportional hazards (PHs) regression was performed to estimate the hazard ratio to assess the association of various factors with OS. PH assumption was verified based on Schoenfeld residuals.

The prognostic model underlying the nomogram, which was developed based on multivariable Cox PH regression, was externally validated according to the approach by Royston and Altman. ¹⁴ The prognostic index (PI), that is, the total nomogram score, was first calculated for each patient in the testing cohort. Model misspecification was assessed by fitting a Cox model on the testing cohort with the six prognostic factors in the nomogram, after offsetting PI, included as covariates. The nomogram was not misspecified if the coefficient estimate of all the predictors in the fitted model were all zeros. Discrimination ability of the nomogram was evaluated based on the Harrell’s concordance index for censored data (c-index). Accuracy of the predictions from the nomogram was evaluated based on calibration-in-the-large and calibration slope. A generalised linear regression model with a complementary log-log link function was fitted on observed death probabilities with linear predictor ${\gamma }_0{\gamma }_1\ln \widehat{H}\left(t\right),$ where ${\gamma }_0$ was the calibration-in-the-large, ${\gamma }_1$ was the calibration slope, $\ln \widehat{F}\left(t\right)=\ln \left[-\ln \left\{1-\widehat{F}\left(t\right)\right\}\right]$ was the estimated log cumulative hazard function, $\widehat{F}\left(t\right)=1-{\widehat{S}}_0{\left(t\right)}^{\exp \left(PI\right)}$ was the predicted death probabilities, ¹⁵ and ${\widehat{S}}_0\left(t\right)$ was the baseline survival function modelled in the training cohort using a second-degree fractional polynomial. The observed death probabilities were estimated using the method of pseudo-observations by Anderson and Perme. ¹⁶ The predicted death probabilities were well calibrated to the observed death probabilities if ${\gamma }_0=0$ and ${\gamma }_1=1.$

Missing data

Only patients in the testing cohort with complete data for all the prognostic factors in the nomogram were included in the validation analyses. Sensitivity analyses were performed to examine the impact of excluding patients with missing data from the validation analyses. Two additional sets of PI were derived for each patient in which we assumed (a) the lowest score and (b) the highest score for each missing variable based on the nomogram.

All p-values were two sided and a p-value < 0.05 was considered statistically significant. All analyses were performed using Stata version 14.2 (StataCorp, College Station, TX, USA).

Results

Comparison of testing and training cohort

Compared with the training cohort, patients in the testing cohort were younger (median: 76 versus 77 years; p = 0.048). More patients in the testing cohort were diagnosed with stage 1–2 disease (28% versus 15%; p < 0.001), had better ECOG performance status 0–1 (73% versus 33%; p < 0.001), better functional status (instrumental ADL score ⩾ 7: 54% versus 12%; p < 0.001), and normal serum albumin levels (72% versus 21%); p < 0.001). Fewer patients in the testing cohort were GDS depressed (14% versus 28%; p < 0.001), had cognitive impairments based on MMSE (16% versus 31%; p < 0.001) and had more than four prescribed drugs (47% versus 60%; p = 0.002). The two cohorts were comparable in terms of DETERMINE nutritional risk groups. Details of other clinical characteristics of the two cohorts are listed in Table 1.

Table 1.

Patient characteristics by patient cohort.

Patient characteristics	Training (N = 249)		Testing (N = 252)		p Value
	No.	%	No.	%
Age at GA, years
Median (range)	77 (70–94)		76 (70–94)		0.048
Gender
Male	153	61.4	144	57.1	0.363
Female	96	38.6	108	42.9
Primary tumour site
Head and neck	6	2.4	7	2.8	<0.001
Gastrointestinal tract	167	67.1	97	38.5
Breast	5	2.0	31	12.3
Lung	29	11.6	49	19.4
Genitourinary	12	4.8	36	14.3
Others	30	12.0	31	12.3
Missing	0	–	1	0.4
Stage of disease
Early (I–II)	38	15.3	70	27.8	<0.001
Late (III–IV)	210	84.3	177	70.2
Missing	1	0.4	5	2.0
ECOG performance status
0–1	83	33.3	184	73.0	<0.001
2–4	166	66.7	68	27.0
Instrumental activities of daily living
⩾7	30	12.0	142	56.3	<0.001
<7	219	88.0	110	43.7
Clock-drawing test
Normal (⩽2)	96	38.6	150	59.5	<0.001
Abnormal (>2)	134	53.8	68	27.0
Missing	19	7.6	34	13.5
Mini mental state examination
Normal (⩾24)	163	65.5	181	71.8	<0.001
Abnormal (<24)	78	31.3	40	15.9
Missing	8	3.2	31	12.3
Geriatric depression scale
Normal (⩽5)	177	71.1	186	73.8	<0.001
Depressed (>5)	70	28.1	35	13.9
Missing	2	0.8	31	12.3
Body mass index, kg/m2
<27.5	231	92.8	222	88.1	0.222
⩾27.5	17	6.8	27	10.7
Missing	1	0.4	3	1.2
DETERMINE nutritional index
Good	67	26.9	56	22.2	0.065
Moderate risk	110	44.2	106	42.1
High risk	72	28.9	85	33.7
Missing	0	–	5	2.0
Charlson comorbidity index
Low	88	35.3	75	29.8	0.061
Medium	111	44.6	110	43.7
High	34	13.7	56	22.2
Very high	16	6.4	11	4.4
Polypharmacy (>4 prescribed drugs)
No	98	39.4	134	53.2	0.002
Yes	150	60.2	118	46.8
Missing	1	0.4	0	–
Geriatric syndromes
Absence	98	39.4	150	59.5	<0.001
Presence	151	60.6	101	40.1
Missing	0	–	1	0.4
Caregiver burden
Little or no burden	188	75.5	63	25.0	<0.001
Mild to moderate burden	55	22.1	29	11.5
Moderate to severe burden	1	0.4	5	2.0
Missing	5	2.0	155	61.5
Serum hemoglobin, g/dL
Normal (⩾12)	106	42.6	132	52.4	0.004
Abnormal (<12)	139	55.8	108	42.9
Missing	4	1.6	12	4.8
Serum albumin, g/L
Normal (>35)	53	21.3	182	72.2	<0.001
Abnormal (⩽35)	178	71.5	62	24.6
Missing	18	7.2	8	3.2
Creatinine clearance test, mL/min
Normal (⩾60)	156	62.7	101	40.1	<0.001
Abnormal (<60)	73	29.3	121	48.0
Missing	20	8.0	30	11.9

ECOG, eastern cooperative oncology group; DETERMINE, disease, eating poorly, tooth loss/mouth pain, economic hardship, reduced social contact, multiple medicines, involuntary weight loss/gain, needs assistance in self-care, elder years > 80; GA, geriatric assessment.

At the time of analysis, median follow-up of the training and testing cohort was 2.7 years [interquartile range (IQR): 2.0–3.3 years] and 3.1 years IQR: 2.7–3.6 years), respectively. Median OS of the testing cohort was 3.1 years, which was significantly higher than the corresponding 1.0 year for the training cohort (p < 0.001) (Figure 1). The 1-year, 2-year and 3-year OS estimates of the testing cohort were consistently higher than those of the training cohort.

Figure 1.

Overall survival by patient cohort.

External validation of the nomogram

Of the 252 patients, 207 had complete data for all the prognostic factors in the nomogram and were included in the validation analyses. There was evidence of model misspecification as some of the coefficient of the prognostic factors in the nomogram estimated based on the testing cohort after offsetting PI were significantly different from zeros (joint test’s p = 0.003) (Table 2). The misspecified factors were albumin (Wald test’s p = 0.019) and GDS (Wald test’s p < 0.001). The nomogram achieved a high level of discrimination in the testing cohort (0.7112), which was similar to the level obtained for the training cohort (0.7108). Predicted death probabilities were well calibrated with the observed death probabilities, especially at year 2 and 3 since diagnosis (Figure 2). Overprediction of death probabilities was evident at year 1. Calibration-in-the-large were estimated as −0.29 (95% CI: −0.65 to 0.07) at year 1, −0.13 (−0.38 to 0.12) at year 2, and −0.09 (−0.32 to 0.14) at year 3 since diagnosis, and the calibration slope was estimated as 0.98 (0.70–1.26). A joint test showed that the calibration-in-the-large estimates were not significantly different from zeros, and the calibration slope was not significantly different from 1 (p = 0.432) (Table 3).

Table 2.

Multivariable Cox regression of overall survival on risk factors in the nomogram with offsetting of prognostic index in the testing cohort.

Risk factors	Hazard ratio (95% CI)	p Value
Age at GA (per year increase)	0.98 (0.94–1.02)	0.262
Albumin (abnormal versus normal)	1.73 (1.09–2.74)*	0.019
ECOG performance status (2–4 versus 0–1)	1.08 (0.66–1.76)	0.758
Geriatric depression scale (depressed versus normal)	0.35 (0.20–0.61)	<0.001
Stage at diagnosis (III–IV versus I–II)	1.68 (0.94–2.99)	0.080
DETERMINE nutritional index (moderate risk versus good)	0.88 (0.49–1.59)	0.668
DETERMINE nutritional index (high risk versus good)	0.73 (0.40–1.35)	0.320
Prognostic index	1	–
Overall model $	–	0.003

^*Violated proportional hazards assumption.

^$Joint test whether the coefficient estimates of all the risk factors after offsetting prognostic index were all zeros.

CI, confidence interval; ECOG, eastern cooperative oncology group; DETERMINE, disease, eating poorly, tooth loss/mouth pain, economic hardship, reduced social contact, multiple medicines, involuntary weight loss/gain, needs assistance in self-care, elder years > 80; GA, geriatric assessment.

Figure 2.

External calibration of death probability at 1, 2 and 3 year post diagnosis. Smooth estimates of observed death probability (dash line) with 95% pointwise confidence interval were plotted against predicted death probability. Solid line was the line of identity denoting perfect calibration.

Table 3.

Calibration tests in the testing cohort at year 1, 2 and 3 since diagnosis.

Calibration tests	Estimate (95% CI)
Calibration-in-the-large (γ0)
At year = 1 (γ01)	−0.29 (−0.65 to 0.07)
At year = 2 (γ02)	−0.13 (−0.38 to 0.12)
At year = 3 (γ03)	−0.09 (−0.32 to 0.14)
Calibration slope (γ1)	0.98 (0.70–1.26)
Joint test: γ01 = 0, γ01 = 0, γ01 = 0, γ1 = 1	p value = 0.432

CI, confidence interval.

Sensitivity analysis of patients with missing data for prognostic factors in the nomogram

The 45 patients with missing data who were excluded from the nomogram validation tended to be older, had ECOG 2–4 and had lower albumin values as compared with the 207 analysed patients. The majority of the 45 patients had missing data for GDS (n = 31). OS of the excluded patients were not significantly different from the patients included in the validation analyses (p = 0.277). Outcomes of the sensitivity analyses suggested that the exclusion of patients with missing data from the validation analyses would not make a large difference to the validation results (Supplemental Table 1).

Potential new prognostic factors

Several new variables such as smoking history, serum bilirubin and dominant handgrip strength test that were not previously examined in the training cohort, were found to be significantly associated with OS in the testing cohort after adjustment for PI, based on multivariable Cox regression analysis (Supplemental Table 2).

Discussion

We had previously developed this prognostic nomogram in 2011 for elderly patients with cancer. This was based on a comprehensive evaluation of all domains of the GA and was the first clinical scoring method for prognostication in this population. This study externally validated the nomogram in an independent cohort of geriatric oncology patients in Singapore.

The nomogram achieved a high level of discrimination in the testing cohort similar to the training cohort. Predicted death probabilities were well calibrated with the observed death probabilities in the testing cohort, despite the fact that the testing cohort differed significantly in terms of OS, type and stage of disease, cognitive and functional status, and biochemical (serum albumin) status from the training cohort. The nomogram overestimated the death rates at 1 year, and this may have been due to the testing cohort having more favorable characteristics than the training cohort in terms of age, functional impairment, cognitive impairment, disease staging and polypharmacy.

Of the six variables in the nomogram, GDS and albumin were misspecified. An abnormal GDS score was identified as an independent adverse prognostic factor in the training cohort. Risk of death among depressed patients was 1.81 times (95% CI: 1.29–2.56) that of patients with normal GDS score. In the testing cohort, however, an abnormal GDS score was associated with lower death risk compared to a normal score. It is possible that compared to the training cohort from 8 years prior, this testing cohort received better access to diagnosis, treatment and psychosocial support for depression which reversed the poor outcomes associated with low mood. Similarly, better nutritional interventions could have mitigated the poor outcomes associated with a low serum albumin level in this cohort. In addition, the diagnosis of depression in oncological patients is challenging due to confounders from cancer symptoms.^17,18 The optimal cut-off scores for the 15-item GDS has been shown to vary slightly in different populations, depending on the prevalence and severity of depression, as well as cross-cultural differences. ¹⁹ Hence, the true specificity of an abnormal GDS for the diagnosis of clinical depression in the local geriatric oncology population might differ from that of the general population.

Some potential new prognostic factors were identified on Cox regression of OS based on the testing cohort. Current smoking status at diagnosis has been shown to increase the risk of both overall mortality and disease-specific mortality. ²⁰ Serum bilirubin is a marker of liver function and is used in various prognostic scoring for hepatocellular cancers (HCCs) and liver disease, ²¹ but its use in prognosticating non-HCC cancers is unclear. Grip strength is one of the physical phenotypes of frailty but may not have additional value in predicting survival when used with an already extensive GA. ²² Overall, the incremental value of these new variables to the survival prediction of older cancer patients remains to be validated.

Survival prediction models for geriatric oncology patients remain scarce in the current literature. Besides the prognostic nomogram, the Onco-MPI is the only other such model developed in 2015 by Brunello et al. ¹⁰ to predict 1-year mortality using a 12-variable score after studying more than 600 Italian cancer patients with a GA assessment. Onco-MPI scores had recently been found to be associated with chemotherapy administration in a cohort of older patients with metastatic colorectal cancer, where high-risk patients were more likely to receive less chemotherapy. ²³ The Onco-MPI is currently undergoing external validation, and evaluation of its prognostic ability beyond 1-year mortality. We are not able to validate the performance of Onco-MPI on our patients, as some of the variables included in the Onco-MPI score such as the cumulative illness rating scale is not routinely assessed in our centre.

A key limitation of our study was the substantial number of patients who were excluded from the analysis due to missing data in their GDS, although this was unlikely to have a huge impact on the validation outcome as shown in the sensitivity analysis. Another limitation was the testing cohort was recruited from the same centre and country as the training cohort, albeit over a different time period. This could limit the applicability in populations with different ethnicities than those found in Singapore. More collaborative work would have to be performed to validate the nomogram in a different country and setting, so as to affirm its universal applicability.

To our knowledge, the prognostic nomogram remains the only model with good prediction of survival over 3 years that was developed from analysis of data gathered from comprehensive GA of geriatric oncology patients. This information could help guide decisions on treatment recommendations. It is possible to digitalize the nomogram into a mobile app calculator that is convenient and easy to use. The six variables identified in the nomogram may help clinicians cue in to the factors that most affect survival. As survival is only one of many considerations in clinical decision-making, further studies looking into outcomes such as patient quality of life and functional independence in geriatric oncology patients could further refine shared decision-making between clinician and patient. Lastly, as most prognostic scales are typically scored at the initial clinic visit and does not take into account changes further down the line, further research into serial scoring of the nomogram looking at the evolution of the prognostic parameters across a patient’s journey and its impact on outcomes can be explored in future studies.

Conclusion

This study externally validated the prognostic nomogram in an independent cohort of geriatric oncology patients. With this validation, the nomogram is ready for use in clinical decision-making.