A Two-Step Frailty Assessment Strategy in Older Patients With Solid Tumors: A Decision Curve Analysis

PURPOSE

The intended clinical value of frailty screening is to identify unfit patients needing geriatric assessment (GA) and to prevent unnecessary GA in fit patients. These hypotheses rely on the sensitivity and specificity of screening tests, but they have not been verified.

METHODS

We performed a cross-sectional analysis of outpatients age ≥ 70 years with prostate, breast, colorectal, or lung cancer included in the ELCAPA cohort study (ClinicalTrials.gov identifier: NCT02884375) between February 2007 and December 2019. The diagnostic accuracy of the G8 Geriatric Screening Tool (G8) and modified G8 scores for identifying unfit patients was determined on the basis of GA results. We used decision curve analysis to calculate the benefit of frailty screening for detecting unfit patients and avoiding unnecessary GA in fit patients across different threshold probabilities.

RESULTS

We included 1,648 patients (median age, 81 years), and 1,428 (87%) were unfit. The sensitivity and specificity were, respectively, 85% (95% CI, 84 to 87) and 59% (95% CI, 57 to 61) for G8, and 86% (95% CI, 84 to 87) and 60% (95% CI, 58 to 63) for the modified G8 score. For decision curve analysis, the net benefit (NB) for identifying unfit patients were 0.72 for G8, 0.72 for the modified G8, and 0.82 for GA at a threshold probability of 0.25. At a threshold probability of 0.33, the NBs were 0.71, 0.72, and 0.80, respectively. At a threshold probability of 0.5, the NBs were 0.68, 0.69, and 0.73, respectively. No screening tool reduced unnecessary GA in fit patients at predefined threshold probabilities.

CONCLUSION

Although frailty screening tests showed good diagnostic accuracy, screening showed no clinical benefits over the GA-for-all strategy. NB approaches, in addition to diagnostic accuracy, are necessary to assess the clinical value of tests.

INTRODUCTION

Geriatric assessment (GA; a comprehensive, multidimensional evaluation of a patient's health status) can identify undetected age-related factors such as frailty.¹ GA in older patients with cancer influences treatment selection, guides geriatric interventions,² and predicts mortality³ and chemotherapy toxicity.^4-6 Hence, American Society of Clinical Oncology,⁷ National Comprehensive Cancer Network,⁸ and International Society of Geriatric Oncology (SIOG)¹ have recommended GA to optimize decision making in older patients with cancer. However, GA is resource- and time-consuming. Hence, a two-step strategy using frailty screening has been recommended to distinguish between unfit patients (who should undergo GA) and fit patients (who should avoid GA).⁹ Although many frailty screening tools are available, in two systematic reviews, the G8 Geriatric Screening Tool (G8) score was one of the most robust because of its high sensitivity and acceptable specificity.^9,10 The G8 score is recommended by the SIOG,^11,12 the European Association of Urology,¹³ the European Society of Surgical Oncology, and the European Organisation for Research and Treatment of Cancer.^14,15

CONTEXT

• Key Objective

• Typically, the clinical value of screening tools relies on diagnostic accuracy measures such as sensitivity and specificity. However, it is unclear which accuracy level justifies tests' implementation or leads to decisions that would do more harm than good. In older patients with cancer, the recommendation of using frailty screening tools to identify patients who should undergo geriatric assessment (GA) relies on the diagnostic accuracy of screening tools. However, their clinical utility has not been verified.

• Knowledge Generated

• We used decision curve analysis to evaluate the utility of frailty screening tools for detecting unfit patients or avoiding unnecessary GAs in fit patients. Despite the good diagnostic accuracy of screening tools, we did not observe any benefits of frailty screening over a geriatric-assessment-for-all approach, regardless of the tumor type, clinical stage, or age group.

• Relevance (S.B. Wheeler)

• This analysis examining the utility of frailty screening before GA failed to demonstrate clinical value of prescreening for frailty across multiple tumor types, disease stages, and age groups.*

  *Relevance section written by JCO Associate Editor Stephanie B. Wheeler, PhD, MPH.

Typically, the evaluation and recommendation to use one screening test over another relies on sensitivity and specificity. However, it is unclear which level of sensitivity and specificity justifies its clinical implementation.¹⁶ In addition, these statistics are of limited clinical value in decision making as they lack any diagnostic meaning because they are conditioned by the future (the confirmed diagnosis) when predicting the past (test result).¹⁷ A physician's decisions should rely on post-test probabilities, that is, the probability of having a disease, given the test results, rather than the contrary.¹⁷ Moreover, sensitivity, specificity, and area under the curve do not provide information about the clinical utility of tests and do not account for the risks or benefits of decisions on the basis of test results.¹⁸

In decision curve analysis (DCA), an intervention's expected risks and benefits are weighted across a range of physician or patient preferences.¹⁸ Moreover, DCA enables simultaneous evaluation of several strategies so that physicians and patients can select the best one.¹⁹ Finally, DCA provides insight into whether making decisions on the basis of test results would do more harm than good.¹⁸

Therefore, the objective of this study was to evaluate the value (as assessed in a DCA) of frailty screening on the basis of the G8 score or the modified 6-item G8 score (hereafter referred to as the modified G8 score) to identify unfit individuals needing GA and to avoid GA in fit individuals.

METHODS

Design, Setting, and Participants

In this cross-sectional study, we analyzed outpatients with cancer of the multicenter, prospective ELCAPA cohort between February 2007 and December 2019. The ELCAPA study was conducted at 19 geriatric oncology clinics in the Greater Paris region of France. All patients were age ≥ 70 years, had a confirmed cancer diagnosis, and were referred to the geriatric oncology unit for GA before treatment selection. We only included patients for whom complete GA, G8, and modified G8 data sets were available. All patients provided informed consent before inclusion in the study. The study protocol was approved by an independent ethics committee (CPP Ile-de-France I, Paris, France; reference: 2019 mai-MS121). The ELCAPA study is registered at ClinicalTrials.gov (identifier: NCT02884375).

Outcome Ascertainment

GA (the reference test).

The GA in the ELCAPA study included seven validated scoring tools covering important health domains for older patients with cancer, assessed by an experienced geriatrician using a standardized case report form. The instruments, thresholds, and definition of unfit patients were consistent with development studies for the G8²⁰ and modified G8 scores.²¹ As in the study by Bellera et al,²⁰ we defined unfit patients as those with at least one abnormal score in the GA domains: activities of daily living (ADL; dependency on at least one item); instrumental activities of daily living (IADL; dependency on at least one item); the Mini-Mental State Examination (≤ 23 out of 30 points); the Mini Geriatric Depression Scale (Mini GDS, ≥ one out of four points); the Mini Nutritional Assessment (MNA, ≤ 23.5 out of 30 points); the Cumulative Illness Rating Scale for Geriatrics (CIRS-G, at least one grade 3 or 4 comorbidity); and the Timed Up and Go test (more than 20 seconds). In patients with prostate cancer (and as recommended by the SIOG), we used only bathing, dressing, toileting, transferring, and feeding items to evaluate ADL. Management of money, medications, transportation, and telephone use were used to evaluate IADL.²²

Screening tools.

The G8 score included the following eight items: food intake and weight loss over the last three months, reduced mobility, neuropsychologic problems, body mass index, administration of more than three daily drugs, self-rated health status, and the patient's age.²⁰

The modified G8 score included the following six items: weight loss over the past three months, neuropsychologic problems, administration of more than six daily drugs, self-rated health status, Eastern Cooperative Oncology Group performance status, and the presence of heart failure or coronary disease.²¹

Abnormal G8 and modified G8 scores were defined using the recommended cutoffs (≤ 14 of 17 points and ≥ 6 of 35 points, respectively).^20,21

Covariates.

We included the following baseline covariates to describe the study population and perform sensitivity analysis: tumor type, clinical stage (metastatic disease or nonmetastatic/unknown), node involvement (positive or negative/unknown), age, and sex.

Statistical Analysis

Summary statistics were used to describe participants' demographic and clinical characteristics. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated (95% CIs) for each screening tool.

DCA

We used the DCA technique described by Vickers et al.¹⁸ We assumed that reasonable threshold probabilities were 0.25 and 0.33, indicating that missing an unfit patient was three and two times worse than exposing a fit patient to an unnecessary GA (odds of 1:3 and 1:2, respectively). We also assessed a threshold probability of 0.5 (odds 1:1), assuming that missing an unfit patient was the same as exposing a fit patient to an unnecessary GA (this probability corresponds to the underlying hypothesis of evaluating the tests' usefulness with sensitivity and specificity).

We compared the net benefit (NB) of frailty screening with GA-for-all and GA-for-none approaches. The NB in terms of true positives (TP; identified unfit patients) represented the percentage of unfit individuals identified by the test if the test's false-positive rate was zero. NB in terms of true negatives (avoided GAs in fit patients) represented the percentage of fit individuals identified by the test if the test's false-negative (FN) rate was zero. Finally, the tradeoff corresponded to the number needed to test to make the GA worthwhile, given its additional cost.²³

Sensitivity Analyses

We performed analyses of clinical stage, tumor type, and age group. We performed analyses in which the criterion for abnormal GA was two impaired domains rather than one. We also used an alternative definition of abnormal GA, as described by Martinez-Tapia et al.²⁴ This definition included recommendations issued by physicians in four clinically relevant domains: nutritional, home, neuropsychologic, and social support. Prescription of recommendations in at least one of the four domains defined an abnormal GA. We compared the geriatric and demographic characteristics of FN and TP patients of the G8 and the modified G8 scores. Finally, we performed pooled analyses of published data for the Vulnerable Elders Survey-13 (VES-13),^25-27 Flemish version of the Triage Risk Screening Tool (fTRST),²⁸ Cardiovascular Health Study (CHS) instrument,²⁷ and Senior Adult Oncology Program 2 (SAOP2).²⁹ See the Data Supplement for a description of these tools (online only). All statistical analyses were performed using the Stata software (version 14.2, Stata Corp, College Station, TX).

RESULTS

Participants

Between February 2007, and December 2019, 2,168 outpatients with breast cancer (n = 1,059 [49%]), colorectal cancer (n = 507 [23%]), prostate cancer (n = 348 [16%]), and lung cancer (n = 254 [12%]) were included in the ELCAPA study. Complete data sets were available for 1,648 patients (76%; Fig 1). The median (interquartile range [IQR]) age was 81 (77-85) years, 1,067 patients (65%) were females, 559 (34%) had metastatic disease, and 1,428 (87%) were unfit. CIRS-G grade 3 or grade 4 comorbidities were the most common impaired domains (58%). For G8, the median (IQR) score was 12 points (10-14), and 1,308 patients (79%) were considered unfit. For the modified G8, the median (IQR) score was 14 points (6-22), and 1,309 patients (79%) were considered unfit. The other characteristics of the study population are summarized in Appendix Table A1 (online only).

FIG 1.

Flow diagram of participants and available data.

Excluded patients (n = 520) were older (median age: 83 years, v 81 years for included patients) and were less likely to have CIRS-G grade 3 or grade 4 comorbidities (45% v 58%), an impaired Mini GDS (22% v 32%), an impaired MNA (27% v 47%), an impaired G8 score (67% v 79%), and an impaired modified G8 score (63% v 79%).

Diagnostic Accuracy of Screening Tools for Identifying Unfit Patients

The G8 score had a sensitivity of 85% (95% CI, 84 to 87), a specificity of 59% (95% CI, 57 to 61), a PPV of 93% (95% CI, 92 to 94), and an NPV of 38% (95% CI, 36 to 41). The modified G8 score had a sensitivity of 86% (95% CI, 84 to 87), a specificity of 60% (95% CI, 58 to 63), a PPV of 93% (92 to 95), and an NPV of 39% (95% CI, 37 to 42).

DCA

For the G8 and modified G8 scores, NB at a threshold probability of 0.25 (odds of 1:3) was 0.72. This result equates to screening 72 patients per 100 with the G8 score, all of whom were found to be unfit. The GA-for-all strategy yielded an NB of 0.82. At a threshold probability of 0.33, the NB for the G8, modified G8 score, and GA-for-all approaches were 0.71, 0.72, and 0.80, respectively. With a threshold probability of 0.50, the NB were 0.68 for G8, 0.69 for modified G8, and 0.73 for the GA-for-all strategy, respectively. The NB data are shown in Figure 2.

FIG 2.

Decision curve for the NB of three different strategies (relative to not doing GA for any patient), with different threshold probabilities. The odds of 1:3 and 1:2 mean that missing an unfit patient is, respectively, three and two times as bad as exposing a fit patient to an unnecessary GA. The odds of 1:1 mean missing an unfit patient is the same as exposing a fit patient to an unnecessary GA. The NB represents the number of patients (per 100) who will be found to be unfit using a given frailty screening tool. G8, G8 Geriatric Screening Tool; GA, geriatric assessment; NB, net benefit.

We observed negative values when calculating the ability of the screening tools to avoid GAs in fit patients. These negative values represent the number of unfit patients lost if screening was used to decide who should undergo GA. For the G8 score, the numbers of lost patients at probabilities of 0.25, 0.33, and 0.50 were 27, 18, and 5, respectively; for the modified G8 score, they were 26, 17, and 4, respectively (Table 1).

TABLE 1.

Diagnostic Accuracy and Net Benefit for Detecting Unfit Patients and Avoiding Unnecessary GA in Fit Patients for the G8 and the Modified G8 Scores, Using One- and Two-Abnormal-Domain Cutoffs and an Alternative Definition for an Abnormal GA

Sensitivity Analyses

At probabilities under 0.5, neither of the scores showed NBs over GA, regardless of the tumor type, clinical stage, or age group (Appendix Tables A2 and A3, online only). When using the alternative definition of abnormal GA (according to Martinez Tapia et al²⁴), we did not observe NBs for screening over GA. When applying a GA cutoff of two impaired domains, both scores showed favorable NBs at threshold probabilities under 0.5 (Table 1).

Compared with FNs, TPs for G8 were more likely to have impaired results in all GA domains, except for CIRS-G grade 3 or grade 4 comorbidities. For the modified G8, TPs were more likely to have impaired results in all GA domains, except for Mini-Mental State Examination and MiniGDS. See Appendix Table A4 (online only).

At threshold probabilities under 0.5, only SAOP2 (cutoff of ≥ 3 impaired domains in GA) showed a benefit for identifying unfit patients and reducing unnecessary GAs in fit patients (Table 2).

TABLE 2.

Diagnostic Accuracy and NB for Detecting Unfit Patients and Avoiding Unnecessary GA in Fit Patients, Using the Various Frailty Screening Tools

DISCUSSION

In the DCA of an outpatient population of older patients with prostate, breast, colorectal, or lung cancer, despite the good diagnostic accuracy of screening tools, we did not observe any NBs of two-step frailty screening over a one-step, GA-for-all strategy.

In our study, the sensitivity and specificity of G8 and modified G8 (85% and 59%; and 86% and 60%, respectively) coincided with those reported in development studies (85% and 57%²⁰; and 89% and 79%,²¹ respectively). Only the specificity of the modified G8 was lower in our study, but in contrast to the study by Martinez-Tapia et al,²¹ we included only outpatients. Similar to other studies, we observed variations in these measures across tumor types, clinical stages, and age groups.^25,30,31

High sensitivity is considered the most relevant parameter for identifying unfit patients at risk for adverse outcomes. High specificity is required to limit the number of fit patients who undergo unnecessary GA.⁹ However, we showed that these parameters do not necessarily translate into better decisions, thus reinforcing the argument that accuracy measures play a limited role in test implementation in clinical practice.

In our study, the prevalence of unfit patients with at least one impaired GA domain was 87%. Other studies have reported similar results, with a median unfit patient prevalence (IQR) of 88% (87-94).^{20,21,25,32,33} When we applied an alternative definition (on the basis of the prescription of geriatric interventions), the prevalence was 75%. We did not observe an NB for screening in these settings relative to the GA. When we applied a cutoff of two impaired GA domains, the prevalence of unfit patients was lower (65%). Several studies have explored this cutoff for defining unfit patients.^25,26,28 Russo et al²⁹ used a cutoff of ≥ 3 impaired domains in GA to define an unfit patient when evaluating the SOAP2. In settings using higher cutoffs, we observed NB of screening over GA. However, these results can be explained by the decrease in prevalence and increased NPV produced by increasing the number of impaired domains defining an unfit patient. Although this approach would allow for the identification of the frailest patients, this strategy would rule out GA in many unfit patients who might benefit from targeted interventions.

This study has some limitations. First, we only analyzed patients with complete data sets. However, our study's unfit patient prevalence and diagnostic accuracy results are consistent with those reported in the literature. Second, our population was relatively old (median age: 81 years), so collecting information on younger patients might have been more valuable. However, our sensitivity analyses produced similar results for the various age groups. Third, our data came solely from centers with expertise in geriatric oncology. It would have been interesting to analyze data from primary or secondary care centers, where the prevalence of unfit patients would be lower. However, studies with a lower prevalence of unfit patients have reported similar results.^25,27,28 In addition, the prevalence of unfit adults age ≥ 70 years in the general population is estimated to be 71%.³⁴ Fourth, the choice of evaluation tools, the number of GA domains to be assessed, and the definition of an unfit patient are still subject to debate. Thus, our results apply to screening tools with the same cutoffs and domains as those studied here. Finally, there is no consensus regarding the threshold probabilities to use. However, we used reasonable threshold probabilities (under 0.5) because those above 0.5 mean that exposing a fit patient to an unnecessary GA is worse than missing an unfit patient.

Our study had some strengths. In particular, we used DCA to directly assess the clinical value of frailty screening using robust, externally validated tools and four of the most frequent types of cancer in older patients.

Our findings suggest that performing GA in all patients (irrespective of frailty screening results) is the most reasonable approach in populations with a high prevalence of unfit individuals. Performing GA on all patients would lead to better decisions and provide the highest benefits if there was a high degree of concern about unfit patients. In addition, performing GA in at least 10 patients would make it worthwhile compared with screening, given its additional costs. However, this tradeoff could be considered high, especially in centers that cannot perform GA. Hence, a time- and resource-saving strategy such as screening is desirable. Nevertheless, direct diagnostic procedures are recommended in high-prevalence settings, as appears to be the case for unfit older patients with cancer.³⁵

Although the NBs of GA are higher than those of screening, unfit patients identified by screening would also benefit from GA, which is better than usual care according to two recent randomized controlled trials.^5,6 Additionally, screening can (1) create a framework for the evaluation of older patients with cancer in a busy practice, (2) raise awareness of the importance of GA among physicians, and (3) stimulate discussions about the choice of cancer treatments.³⁶

Although accuracy measures do not provide insight into implementing tests in clinical practice or making better decisions, most research papers regarding diagnostic tests or predictive models published in the Journal of Clinical Oncology, and other similar journals, report only these performance measures without evaluating clinical utility. In geriatric oncology, for instance, several frailty classifications³⁷ and predictive tools for chemotherapy toxicity exist^38,39; some are recommended in practice guidelines,^7,13 whereas others are used in practice. In addition to calibration and discrimination, no study evaluating these tools assessed their clinical usefulness. Moreover, none of the 17 studies included in the most recent systematic review of frailty screening tools¹⁰ evaluated the usefulness of these tools. Therefore, it is hard to know whether using those tests or models is of clinical value or justifies its implementation in clinical practice. As suggested by previous editorials in Journal of Clinical Oncology,^40,41 researchers should evaluate the clinical value (eg, using NB approaches) when developing instruments for decision making.

Furthermore, the number, domains, and instruments assessed in a GA differ in their importance for cancer outcomes (eg, toxicity, quality of life, or survival).⁴² Therefore, a more comprehensive definition of an abnormal GA is required—perhaps by assigning different weights to various impaired dimensions. In addition, screening tools do not capture all GA domains. Hence, improvements in the identification of impairments in missed domains are desirable. However, when developing decision-making tools, researchers must consider the trade between gaining information and making better decisions, as a good decision tool leads to the best decisions with as little information as possible.

In conclusion, despite the good diagnostic accuracy of screening tools, frailty screening did not provide a benefit over a GA-for-all approach for detecting unfit patients or avoiding GA in fit patients. Optimizing current frailty assessment strategies and evaluating their usefulness with NB approaches is necessary to improve decision making for older patients with cancer in busy practices with limited human resources.

APPENDIX

TABLE A1.

Demographic and Clinical Characteristics of the Study Participants

TABLE A2.

Diagnostic Accuracy and NB for Detecting Unfit Patients and Avoiding Unnecessary GA in Fit Patients for the G8 and the Modified G8 Scores

TABLE A3.

Diagnostic Accuracy and NB for Detecting Unfit Patients and Avoiding Unnecessary GA in Fit Patients for the G8 and the Modified G8 Scores

TABLE A4.

Demographic and Clinical Characteristics of TP and False-Negative Patients for the G8 and the Modified G8 Scores

DATA SHARING STATEMENT

Restrictions apply to the availability of these data. Data were obtained from the ELCAPA Study Group and are available from the corresponding author with the permission of the ELCAPA Study Group investigators.

AUTHOR CONTRIBUTIONS

Conception and design: Adolfo González Serrano, Thomas Barnay, Florence Canouï-Poitrine

Provision of study materials or patients: Marie Laurent, Pascaline Boudou-Rouquette, Thomas Aparicio, Philippe Caillet, Elena Paillaud, Florence Canouï-Poitrine

Collection and assembly of data: Marie Laurent, Thomas Aparicio, Philippe Caillet, Elena Paillaud, Florence Canouï-Poitrine

Data analysis and interpretation: Adolfo González Serrano, Marie Laurent, Thomas Barnay, Florence Rollot-Trad, Claudia Martínez-Tapia, Etienne Audureau, Boudou-Rouquette, Pierre Soubeyran, Carine Bellera, Philippe Caillet, Elena Paillaud, Florence Canouï-Poitrine

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

A Two-Step Frailty Assessment Strategy in Older Patients With Solid Tumors: A Decision Curve Analysis

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/jco/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).