Optimizing the G8 Screening Tool for Older Patients With Cancer: Diagnostic Performance and Validation of a Six-Item Version

In an effort to develop an optimized G8 tool combining a systematic statistical approach with expert judgment to ensure optimal discriminative power and clinical relevance, a large prospective cohort study was conducted. The improved screening tool achieves high sensitivity, high specificity, better homogeneity across cancer types, and greater parsimony with only six items needed, facilitating selection for a full geriatric assessment.

Abstract

Background.

A multidimensional geriatric assessment (GA) is recommended in older cancer patients to inventory health problems and tailor treatment decisions accordingly but requires considerable time and human resources. The G8 is among the most sensitive screening tools for selecting patients warranting a full GA but has limited specificity. We sought to develop and validate an optimized version of the G8.

Patients and Methods.

We used a prospective cohort of cancer patients aged ≥70 years referred to geriatricians for GA (2007–2012: n = 729 [training set]; 2012–2014: n = 414 [validation set]). Abnormal GA was defined as at least one impaired domain across seven validated tests. Multiple correspondence analysis, multivariate logistic regression, and bootstrapped internal validation were performed sequentially.

Results.

The final model included six independent predictors for abnormal GA: weight loss, cognition/mood, performance status, self-rated health status, polypharmacy (≥6 medications per day), and history of heart failure/coronary heart disease. For the original G8, sensitivity was 87.2% (95% confidence interval, 84.3–89.7), specificity 57.7% (47.3–67.7), and area under the receiver-operating characteristic curve (AUROC) 86.5% (83.5–89.6). The modified G8 had corresponding values of 89.2% (86.5–91.5), 79.0% (69.4–86.6), and 91.6% (89.3; 93.9), with higher AUROC values for all tumor sites and stable properties on the validation set.

Conclusion.

A modified G8 screening tool exhibited better diagnostic performance with greater uniformity across cancer sites and required only six items. If these features are confirmed in other settings, the modified tool may facilitate selection for a full GA in older patients with cancer.

Implications for Practice:

Several screening tools have been developed to identify older patients with cancer likely to benefit from a complete geriatric assessment, but none combines appropriate sensitivity and specificity. Based on a large prospective cohort study, an optimized G8 tool was developed, combining a systematic statistical approach with expert judgment to ensure optimal discriminative power and clinical relevance. The improved screening tool achieves high sensitivity, high specificity, better homogeneity across cancer types, and greater parsimony with only six items needed, facilitating selection for a full geriatric assessment.

Introduction

A majority and growing number of cancers are diagnosed in patients aged 65 years or more [1]. Population aging in industrialized countries and the increase in cancer incidence with advancing age make the management of older patients with cancer a major public health challenge. Older patients vary widely regarding multiple aspects of the aging process and health status. Individual treatment tailoring is therefore particularly important to ensure optimal efficacy and to minimize toxicity in this age group.

To assess the profile of each individual patient, a multidimensional geriatric assessment (GA) has been recommended since 2005 by the International Society of Geriatric Oncology [2, 3]. The GA produces an inventory of health problems and an assessment of physical, psychosocial, and functional capabilities [4]. The GA has proved useful for characterizing health and functional impairments potentially associated with oncological outcomes [2, 5–7]. Nevertheless, it requires considerable time and human resources and is actually not required in all patients [6].

Screening tools have therefore been developed to discriminate between fit older patients who are likely to tolerate standard therapy and vulnerable patients who would benefit from specific treatment tailoring based on findings from a complete GA [8]. The ideal screening tool is easy to perform, requires little time, covers all the domains routinely assessed by geriatricians, and effectively separates fit from vulnerable patients. Although many tools have been described, none combines both appropriate sensitivity and specificity for predicting an abnormal GA [8, 9]. The G8, a screening tool specifically developed for older patients with cancer [10, 11], is among the most sensitive but lacks specificity [8, 9, 12]. Importantly, most screening instruments, including the G8, use items selected based on expert opinion and/or existing assessment tools (e.g., mini-nutritional assessment-short form [MNA-SF] in the G8) that were originally validated to detect outcomes other than GA impairment.

Here, our objectives were to evaluate the performance of the G8 in identifying older cancer patients likely to have abnormal GAs and to determine whether modifications to the G8 might improve this performance. We used a systematic statistical approach to simultaneously test the original G8 items and a set of additional, potentially relevant items in a large cohort of older patients with cancer.

Materials and Methods

Patients

We analyzed data from the ELCAPA (ELderly CAncer PAtients) study, a prospective open-cohort survey of consecutive patients aged 70 years or older with solid cancers or hematologic malignancies who were referred by an oncologist, surgeon, radiotherapist, or other specialist to one of two geriatric oncology clinics in teaching hospitals in the Paris urban area. A multidimensional GA was performed by geriatricians specialized in oncology. Then a multidisciplinary meeting was held to determine the best treatment strategy. For the present analysis (ELCAPA-07), we included patients for whom complete G8 and GA information were available in the electronic database. Patients enrolled from January 2007 to October 2012 were considered as the training set, whereas patients subsequently recruited until July 2014 were considered as the validation set. Informed consent was obtained from all study patients prior to inclusion. The study was approved by the appropriate ethics committee (Comité de Protection des Personnes [CPP] Ile-de-France I, France).

GA Reference Procedure

The GA consisted of a set of seven validated tests covering a variety of important health domains in older cancer patients [13] and consistent with the questionnaires and thresholds used in the primary G8 validation study [11]. Abnormal GA was thus defined as an impaired score on at least one of the following tests: Activities of Daily Living (ADL ≤5/6), Instrumental Activities of Daily Living (IADL ≤7/8), Mini Mental State Examination (MMSE ≤23/30), Mini Geriatric Depression Scale (Mini GDS ≥1), Mini Nutritional Assessment (MNA ≤23.5/30), Cumulative Illness Rating Scale for Geriatrics (CIRS-G; at least one comorbidity grade 3 or 4), and Timed Up-and-Go test (>20 seconds). Sensitivity analyses were performed to test the robustness of the results, by using an alternative cutoff of ≥2 impaired tests to define abnormal GA or by omitting the CIRS-G or ADL/IADL from the reference GA.

Candidate Items

We identified 22 candidate items for a modified G8, based on both the literature and clinical expertise, to avoid the overfitting to the training data set seen when item selection relies solely on statistical significance [14]. In addition to the G8 items (depression/dementia, body mass index [BMI], anorexia, weight loss, age, medications [>3 per day], mobility, and self-rated health status), we selected 14 items routinely collected during geriatric evaluations and known to be clinically relevant for assessing older cancer patients: asthenia, incontinence, fall risk (single-leg stance time < 5 seconds), history of fall(s) in the past 6 months, Eastern Cooperative Oncology Group Performance Status (ECOG-PS), gender, living alone at home, metastatic status, and a selection of six comorbidities (diabetes, hypertension, heart failure and/or coronary heart disease [CHD], complete arrhythmia with atrial fibrillation [CAAF], chronic renal failure, and chronic respiratory failure).

Statistical Analysis

Training Set Analysis

First, we performed a descriptive univariate analysis to assess associations between candidate items and the reference GA, using χ² or Fisher’s exact tests as appropriate and estimating the crude odds ratios (ORs) with their 95% confidence intervals. Receiver-operating curve (ROC) analysis was performed for the number of medications to assess alternative cutoffs to the one used in the original G8. Second, we used multiple correspondence analysis (MCA) to investigate correlations between candidate items and identify redundancies across conceptually close qualitative variables, thus helping to decide which variables should be combined, dichotomized, or omitted. Additional details are shown in supplemental online Figure 1.

Remaining candidate items were then entered into a multivariate logistic regression model, using a stepwise backward procedure to sequentially remove items based on a p < .05 level until the final model was obtained. Model discrimination was assessed by the area under the ROC curve (AUROC) and calibration by the Hosmer-Lemeshow χ² test. Regression coefficients were considered for use as weights to compute the final score. We rescaled (multiplied) and rounded them to the closest integer, using the algorithm described by Cole to find the optimal solution that both improved simplicity of use in the clinical setting and preserved initial model accuracy [15].

We internally validated our model using bootstrapping procedures with 300 replications to estimate the amount of optimism in our measurement of model discrimination and to compute the bias-corrected AUROC accordingly [16]. The modified G8 was finally applied to the validation set population in which AUROC, sensitivity, specificity, positive predictive values (PPV), and negative predictive value (NPV) were calculated.

There were no missing data for G8 items or GA findings. Few data were missing for the 14 additional items: their proportion ranged from 0% to 4.9% (chronic renal failure) in the training set and 4.1% (health perception status) in the validation set. We imputed missing values using 10-fold multiple imputation by chained equations and combining the estimates using Rubin’s rules [17]. Data were assumed to be missing at random, conditional on other predictors and on the outcome. All analyses were performed using Stata v12.1 (StataCorp, College Station, TX, http://www.stata.com) at the two-tailed p < .05 level. This observational study is reported according to the STARD checklist for diagnostic accuracy studies.

Results

Patient Characteristics

Between January 2007 and October 2012, 1,056 patients were included into the ELCAPA cohort (training set), of whom 729 had complete G8 data available at the time of our analysis. Between November 2012 and July 2014, 442 patients were included (validation set), of whom 414 had complete G8 data (supplemental online Fig. 2). Table 1 reports the general characteristics of the study populations. Overall, 632 patients (86.7%) had at least one impaired GA test in the training set (14.8% had 1 impaired test, 30.1% had 2 or 3 impaired tests, and 41.8% had 4 or more impaired tests) and 390 (94.2%) had at least one impaired GA test in the validation set (14.5% had 1 impaired test, 35.3% had 2 or 3 impaired tests, and 44.4% had 4 or more impaired tests).

Table 1.

Patient characteristics in the ELCAPA-07 cohort study

Univariate Analysis

Table 2 reports the main results for the original G8 items and additional candidate items. Of the G8 items, 7 were significantly associated with an abnormal full GA; the remaining item was BMI (p = .06). Of the 14 additional items, 12 were significantly associated with an abnormal full GA: asthenia, fall risk, ECOG-PS, incontinence, heart failure/CHD, CAAF, hypertension, diabetes, chronic renal failure (Table 2), fall(s) in the 6 past months (p < .001), metastasis (p = .02), and chronic respiratory failure (p = .04). No significant association was found for gender (p > .10), and living alone was significant only in patients with urinary tract cancers (p = .001; p > .2 for all other cancer sites). ROC analysis of the number of medications per day showed an optimal cutoff of ≥6, as shown by the maximized Youden’s index (33.0 [≥6] vs. 9.9 [>3]) and minimal distance between the ROC curve and upper left corner.

Table 2.

Univariate analysis of the ability of candidate items to predict impairment of the geriatric assessment (training set; N = 729)

Multiple Correspondence Analysis

MCA was conducted to assess the correlations between items (supplemental online Fig. 1). Because several variables were located in close proximity, only those items exhibiting the highest discriminative power and showing a graded distribution along the first MCA axis were kept for the multivariate analysis. These items were the ECOG-PS, weight loss, and fall risk. Thus, we omitted the G8 BMI, anorexia, mobility, and history of fall in the past 6 months.

Multivariate Model

Based on the previous steps, we selected 16 items for the multivariate analysis. A backward stepwise approach showed that six items were independently associated with an abnormal GA. We merged several categories when similar OR values were found across adjacent modalities or for certain modalities that almost perfectly predicted an abnormal GA (ECOG-PS, weight loss, dementia/depression, and self-rated health status), yielding the final model shown in Table 3. Model calibration was excellent (χ² = 69.8; p = .97). The regression coefficients were then rescaled and rounded to integers to provide weights suitable for use in clinical practice. Among tested multiplication coefficients, 3.5 proved optimal. The final six-item questionnaire is shown in supplemental online Table 1.

Table 3.

Final multivariate logistic model for predicting impairment of the geriatric assessment (training set; N = 729)

Diagnostic Performance of the Original versus Modified G8

Using the recommended score cutoff of ≤14 [11], the original G8 demonstrated the following indices: sensitivity, 87.2% (84.3–89.7); specificity, 57.7% (47.3–67.7); PPV, 93.1% (90.7-95.0); and NPV, 40.9% (32.6–49.6). For the modified G8, the cutoff of ≥6 of 35 points maximized sensitivity and yielded the following characteristics: sensitivity, 89.2% (86.5–91.5); specificity, 79.0% (69.4–86.6); PPV, 96.5% (94.7–97.9); and NPV, 52.8% (44.3–61.2). Using the higher cutoff of ≥7 of 35 produced the following values: sensitivity, 85.8% (82.8–88.5); specificity, 88.4% (80.2–94.1); NPV, 48.8% (41.2–56.6); and PPV, 98.0% (96.4–99.0). The AUROC was 86.5% (83.5–89.6) for the original G8 and 91.6% (89.3; 93.9) for the modified G8 (p = .0002) (Fig. 1). When we analyzed each cancer site, we found that the modified G8 yielded consistently higher AUROC values with greater uniformity (from 87.1% in colorectal cancers to 96.2% in urinary tract cancers) compared with the original G8 (from 78.4% in colorectal cancer to 93.6% in liver and upper gastrointestinal tract cancer) (Fig. 2).

Figure 1.

ROC curves for predicting impairment of the geriatric assessment, used as the reference test: original versus modified G8 questionnaire (training set; n = 729).

Abbreviations: AUROC, area under the ROC curve; ROC, receiver-operating characteristic.

Figure 2.

Area under the ROC curve by cancer site: original versus modified G8. *, p < .05.

Abbreviation: ROC, receiver-operating characteristic.

Internal Validation and Sensitivity Analyses

The internal validation procedure showed no evidence of overoptimism (optimism = 0.89% ± 0.13%). The bias-corrected AUROC was 90.7% for the final prediction model. Sensitivity analyses found an AUROC of 91.4% (89.2–93.6) when the CIRS-G was removed from the GA, 90.4% (88.0–92.8) when the ADL/IADL was removed from the GA, and 90.3% (88.0–92.6) when an abnormal GA was defined as impairment of ≥2 tests. Applying the modified G8 to the validation set produced the following values: AUROC, 92.8% (88.4–97.2); at the ≥6 of 35 cutoff: sensitivity, 91.4% (88.0–94.1); specificity, 75.0% (53.3–90.2); PPV, 98.2% (96.0–99.3); NPV, 37.5% (24.0–52.6); at the ≥7 of 35 cutoff: sensitivity, 88.0% (84.1–91.2); specificity, 87.5% (67.6–97.3); PPV, 99.0% (97.2–99.8); and NPV, 33.3% (22.0–46.3). Model calibration was excellent (χ² = 3.12; p = .93).

Discussion

We developed a modified version of the G8 based on six simple items that are routinely collected by geriatricians. The modified G8 demonstrated a sensitivity of 89.2% and a specificity of 79.0% at the optimized cutoff of ≥6 of 35 points, with evidence of homogenous performance across tumor sites.

A wide variety of screening tools have been evaluated to identify patients likely to benefit from a complete GA. The G8, the Vulnerable Elders Survey-13 [18], and the Groningen Frailty Indicator [19] are among the most extensively assessed. Although none of the available tools is markedly better than the others [8], the G8 has the theoretical advantage of having been specifically developed for older patients with cancer, with a selection of items covering important domains in this population [11]. A recent review identified eight studies evaluating the ability of the G8 to predict an abnormal GA [8]. Sensitivity was usually high, with a range of 65%–92% (median, 85.5%), but specificity was lower, ranging from 3% to 75% (median, 59.5%). Similarly, in our study, the original G8 was 87.2% sensitive but only 57.7% specific. It is noticeable that the G8 was derived from the MNA-SF questionnaire, because of its known high prognostic value in older patients [11]. The fact that the MNA-SF was not designed to specifically detect an abnormal GA probably explains the lack of specificity of the G8 as a screening instrument.

Our objective was to develop a variant of the G8 that would improve the identification of patients requiring a full GA. We used a systematic approach, as typically applied for developing clinical rules or prediction models. This approach consisted of the initial selection of candidate items based on clinical reasoning, multivariate analyses to identify items conveying independent information, internal validation based on bootstrapping techniques to prevent overfitting [14], and reassessment of the model performance on a validation set population. Special attention was given to weights computation to obtain an easy-to-use tool while limiting the loss of information inevitably associated with rounding [15]. Finally, we used multiple imputation at each step of model development to maintain an effective sample size and to control their potential influence on the final model [14].

The modified G8 has only six items yet covers multiple domains included in the GA, namely, nutritional status, mood or cognition, comorbidities, and polypharmacy, in addition to self-rated health status and a simplified version of the ECOG-PS. Interestingly, alterations in these items have been shown to predict adverse outcomes [12, 20–22]. For the number of medications per day, instead of the >3 cutoff used in the G8, we found that the ≥6 cutoff improved discrimination, in keeping with conclusions from a recent expert consensus conference [23]. Similarly to the original G8, most items in the modified G8 are subjective and therefore not well suited as criteria for individual diagnosis [11]. The assessment of a past history of heart failure/CHD should not be viewed as an abbreviated version of comorbidity assessment tools such as the CIRS-G but rather as a marker predicting an impaired GA when used in combination with the other items. Despite being associated with an abnormal GA in univariate analysis, several items (age, BMI, anorexia, and mobility) were omitted from the modified G8 because they had minimal independent discriminative power, given their close correlations with other variables, as visualized by MCA.

The modified G8 was robust to sensitivity analyses involving changes in the definition of GA/abnormal GA. This point is of particular interest, because various definitions have been used in previous studies assessing the performance of the G8 in the absence of a clearly defined reference GA [8, 9]. Moreover, the modified G8 showed homogeneity across tumor sites, including various solid tumors and hematological malignancies, whereas evidence for heterogeneity was previously reported for the original G8 [12, 24, 25]. However, despite those apparent benefits, these findings should still be interpreted cautiously in the absence of external validation studies, which are crucial to verify the stability of the model in other populations. Should the improved screening performance of the modified G8 be confirmed, this new tool may encourage the actual use of a two-step approach, in which the results of screening determine whether a full GA is performed [8]. High discriminative power is essential to avoid performing time-consuming unnecessary GAs (false positives) and to ensure that no patients requiring a GA are missed (false negatives). This last point is of major importance, given the consistently high prevalence of abnormal GA findings in several studies conducted in various settings (>80% [11, 12, 24]; 86.7% in our study).

Our study has several limitations. First, in keeping with our study objective, we confined our sample to patients for whom the GA and G8 items were available, which resulted in 327 patients being excluded from the original sample of 1,056 patients. However, we found no statistically significant differences between included and excluded patients regarding the main demographic and clinical features (age, gender, cancer type, and cancer spread; data not shown), suggesting minimal selection bias. Second, several potentially relevant variables were not entered in our database at the time of the analysis, including specific items from validated scales or details on the social environment. These variables deserve investigation in future studies.

Conclusion

We developed a modified G8 screening tool that exhibited better diagnostic performance across a variety of tumor sites and greater parsimony, with only six items instead of eight. Our work illustrates the usefulness of combining in-depth statistical analyses with expert judgment to ensure both optimal discriminative power and clinical relevance. Further research is needed to confirm the features of the modified G8 in other populations and to measure its prognostic value and its impact on treatment decisions and health outcomes.

See http://www.TheOncologist.com for supplemental material available online.

This article is available for continuing medical education credit at CME.TheOncologist.com.

Author Contributions

Conception/Design: Claudia Martinez-Tapia, Florence Canoui-Poitrine, Sylvie Bastuji-Garin, Pierre Soubeyran, Simone Mathoulin-Pelissier, Elena Paillaud, Etienne Audureau

Provision of study material or patients: Christophe Tournigand, Elena Paillaud, Marie Laurent

Collection and/or assembly of data: Christophe Tournigand, Elena Paillaud, Marie Laurent

Data analysis and interpretation: Claudia Martinez-Tapia, Florence Canoui-Poitrine, Sylvie Bastuji-Garin, Etienne Audureau

Manuscript writing: Claudia Martinez-Tapia, Florence Canoui-Poitrine, Sylvie Bastuji-Garin, Pierre Soubeyran, Simone Mathoulin-Pelissier, Christophe Tournigand, Elena Paillaud, Marie Laurent, Etienne Audureau

Final approval of manuscript: Claudia Martinez-Tapia, Florence Canoui-Poitrine, Sylvie Bastuji-Garin, Pierre Soubeyran, Simone Mathoulin-Pelissier, Christophe Tournigand, Elena Paillaud, Marie Laurent, Etienne Audureau

Disclosures

Pierre Soubeyran: Teva, Celgene (C/A), Pierre Fabre Oncology (other). The other authors indicated no financial relationships.

(C/A) Consulting/advisory relationship; (RF) Research funding; (E) Employment; (ET) Expert testimony; (H) Honoraria received; (OI) Ownership interests; (IP) Intellectual property rights/inventor/patent holder; (SAB) Scientific advisory board