A systematic review of predictive accuracy via c-statistic of preoperative frailty tests for extended length of stay, post-operative complications, and mortality

ABSTRACT

Frailty, as an age-related syndrome of reduced physiological reserve, contributes significantly to post-operative outcomes. With the aging population, frailty poses a significant threat to patients and health systems. Since 2012, preoperative frailty assessment has been recommended, yet its implementation has been inhibited by the vast number of frailty tests and lack of consensus. Since the anesthesiologist is the best placed for perioperative care, an anesthesia-tailored preoperative frailty test must be simple, quick, universally applicable to all surgeries, accurate, and ideally available in an app or online form. This systematic review attempted to rank frailty tests by predictive accuracy using the c-statistic in the outcomes of extended length of stay, 3-month post-operative complications, and 3-month mortality, as well as feasibility outcomes including time to completion, equipment and training requirements, cost, and database compatibility. Presenting findings of all frailty tests as a future reference for anesthesiologists, Clinical Frailty Scale was found to have the best combination of accuracy and feasibility for mortality with speed of completion and phone app availability; Edmonton Frailty Scale had the best accuracy for post-operative complications with opportunity for self-reporting. Finally, extended length of stay had too little data for recommendation of a frailty test. This review also demonstrated the need for changing research emphasis from odds ratios to metrics that measure the accuracy of a test itself, such as the c-statistic.

Introduction

Frailty is a syndrome of reduced physiological reserve that is present in 20% of patients undergoing emergency laparotomies aged 65 and above.[1,2] A recent systemic review has identified frailty as the strongest risk factor for developing post-operative morbidity in older patients.[3] Major stresses, such as surgery, temporarily decrease physiological reserves, meaning that the combination of frailty and surgery can result in significant mortality and morbidity. A diagnosis of frailty can increase 90-day post-operative mortality by a factor of 3.18.[2] Additionally, it is well-known that age is directly associated with the severity of frailty. Indeed, geriatric people may make up almost a quarter of the population by 2060 in the United States with more than 50% of this population requiring at least one surgery in their lives.[4,5] As such, frailty poses a significant threat to patients and the health systems of nations.

To combat this, surgical and anesthetic international societies have recommended preoperative frailty assessment since 2012.[6,7] However, their use in practice has been hampered by the sheer number of frailty tests available (a previous study found 35 alone) and the lack of census for which of these to use.[8,9] Currently, there exist three predominant models of frailty [Table 1]. There are two modalities of frailty assessment: clinical, where the assessor examines the patient in-person, and administrative, where hospital database information can be used to calculate a score. The closest to a gold standard for frailty assessment is the comprehensive geriatric assessment (CGA), a multidisciplinary process assessing the domains of multimorbidity, polypharmacy, nutrition, mobility, physiologic function/reserves, neurocognition, and psychological health to identify and manage these risk factors.[10] The CGA has already been employed in perioperative settings with great success, decreasing morbidity and mortality.[11,12] However, in the perioperative setting, the CGA can be an unwieldy program that is time-consuming and requires geriatrician expertise not commonly available in surgical teams.[13] As such, a frailty test needs to be tailored for the perioperative environment; tailored for the anesthesiologist who is best placed to accompany the patient throughout the entire perioperative journey. The ideal frailty test for the anesthesiologist needs to be feasible (able to be completed quickly with little extra training or equipment); universal (able to be applied to any surgical population); and accurate (able to correctly classify frail patients and predict post-operative outcomes). Finally, with the dawn of digital medicine, another desirable trait is digital interface of frailty tests, such as completion via an app on the phone, as well as easy online accessibility for physicians.

Table 1.

The three most popular models of frailty according to the literature

Model	Definition	Archetypal Test
Phenotype of Frailty[14]	A disease-like syndrome consisting of energy depletion and inflammation, which exhibits itself as “weakness, decreased endurance, and slow performance.”	Fried’s Phenotype of Frailty
Accumulation of Deficits[15]	The accumulation of disabilities and conditions with emphasis on the number rather than the nature of the deficits.	Frailty Index
Multidimensional[16]	A dynamic state of loss affecting 1 or more areas of functioning such as the cognitive, physical, and social domains.	Comprehensive Geriatric Assessment

A current survey of the literature demonstrates an emphasis on feasibility because the use of odds ratios makes differentiation of predictive accuracy difficult. McIsaac et al.[3] commented that, despite only moderate agreement between frailty tests (Cohen’s kappa = 0.1–0.8), many studies had found no difference in effect sizes for length of stay, post-operative complications and mortality. Indeed, odds ratios assess prevalence of an event in a population rather than the predictive accuracy of a test itself, the most commonly reported of such a metric being the c-statistic.[17] By appealing to feasibility, most reviews and guidelines have recommended the Clinical Frailty Scale for preoperative assessment.[18,19] In contrast, it is the aim of this systematic review to rank preoperative frailty tests according to their predictive accuracy, in the form of the c-statistic, as well as feasibility.

Materials and Methods

Search Strategy

Search terms were derived from initial scoping of previous systematic reviews covering preoperative frailty tests.[18,20,21,22] The search method was applied to Medline and EMBASE databases from inception to March 10, 2023. A summary of the search strategy has been included [Supplementary Table 1]. Reference lists of related systematic reviews and primary articles discovered in systematic search were also inquired for other studies not covered by the search method. No language restrictions were applied.

Supplementary Table 1.1.

Search terms used for Ovid Medline (R). Search period from 1946 to March 10, 2023

Search Step	Search Terms	Total Studies, n
1	(Preoperative Care or preoperative period).sh.	74390
2	(preoperat* or pre-operat*).tw.	385505
3	1 or 2	418662
4	(frailty or frail elderly).sh.	19638
5	frail*.tw.	32568
6	4 or 5	37347
7	geriatric assessment.sh.	32057
8	(test* or screen* or assess* or index* or indicator* or rule* or measur* or tool* or instrument* or scale* or score* or metri* or rating or resignation or phenotype).tw.	11619843
9	7 or 8	11627309
10	(mortality or death or morbidity or complication* or adverse event* or length of stay).tw.	2842226
11	3 and 6 and 9 and 10	1080

Study selection

Eligible studies were included if they: (1) studied a surgical population with a mean or median age greater than 60 years; (2) included a frailty instrument explicitly described or used according to its original publication and its result recorded before the surgery; (3) reported a predictive accuracy outcome in the form of the c-statistic for length of stay, 3-month or less post-operative complication or mortality.

Predictive accuracy of 3-month or less post-operative mortality was the primary outcome and 3-month or less post-operative complications, as defined by greater than or equal to grade 2 on the Clavien-Dindo classification model, and extended length of stay, as defined by a greater than 75^th percentile length of stay, were secondaries.[23] Other secondary outcomes included feasibility parameters: completion time, equipment, training, database compatibility, and cost for frailty tests, which were recorded from original publications of frailty tests.

Studies were excluded if they: (1) included mixed populations with less than 50% of patients undergoing surgery; (2) included samples with greater than 50% of patients undergoing cardiac or major thoracic and abdominal vascular surgery (since frailty has a larger influence on post-operative outcome in these surgeries); (3) included samples with greater than 50% of patients with a cancer diagnosis or undergoing surgery specifically for cancer resection; (4) determined frailty by the CGA (since this is inappropriate for the perioperative environment); (5) determined frailty by a single laboratory or imaging technique (e.g., ultrasound scan for sarcopenia); (6) determined frailty using a score specific to a surgical subpopulation (e.g., Nottingham Hip Fracture Score). Conference abstracts or other grey literature were not included due to incomplete descriptions of methodology.

Data extraction and quality assessment

Screening of papers was conducted first by title and abstract and then by full text using Covidence. Removal of duplicate articles was done automatically by Covidence as well as manually by screeners. Data extracted included basic study and study population parameters and primary and secondary outcomes as above. Updated versions of frailty tests, such as modified frailty index 11-item and 5-item, were combined into one frailty test for analysis. Risk of bias was analyzed using the Quality in Prognosis Studies tool.[24]

Data analysis

The c-statistic is a measure of the discriminatory power of a predictive model calculated from the area under the receiver operating characteristic curve, which can be summarized as: “the proportion of all pairs of patients where one patient experienced the event of interest and the other patient did not experience the event, and the patient with the lower risk score was the one who did not experience the event.”[25] For use in comparing predictive accuracy of frailty tests in this review, a c-statistic of 0.80 and above was defined as excellent predictive accuracy; 0.70 and above as good predictive accuracy; 0.60 and above as fair accuracy; 0.50 and above as poor accuracy.

The finding of the best frailty tests involved rounds of elimination based on desirable properties: (1) frailty tests must have data from at least three studies for predicting a single outcome (length of stay, post-operative complication or mortality); (2) total number of studies for predicting an outcome by a frailty test must not be composed by more than 50% of the same surgery type; (3) the frailty test must not take more than 5 minutes to complete. The remaining frailty tests were then ordered by their mean c-statistic in each outcome, and the 5 best were chosen for comparison.

Results

A total of 1772 records were screened after 590 duplicates were removed [Figure 1]. 304 full-text articles were assessed, and 35 studies were included from the systematic search. A further 17 studies were included after analyzing citations of reviews and primary articles. Thus, 52 studies in total have been included. Overall, included studies consist of 2,168,912 participants and were published between 2008 and 2023 with 2022 being the most common year of publication. A full summary of studies has been included [Table 2].

Figure 1.

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram for inclusion and exclusion of papers

Table 2.

Summary of 52 included studies for assessment of predictive accuracy of frailty tests. Where there were multiple frailty measurements, the lowest prevalence was used

Study	Study design	Study Size, n	Mean Age, Year	Sex, % Female	Frail patient, %	Surgical procedure	Procedure urgency	Frailty Measure
2008 Burgos[26]	P	232	85	85	-	Hip	EM	Barthel, CCI
2008 Dasgupta[27]	P	125	77.4	58	12.8	Non-cardiac	EL	EFS
2013 Robinson 1[28]	P	201	74	2	33.3	Colorectal	EL	Robinson
2013 Robinson 2[29]	P	98	74	4	26.5	Colorectal	EL	TUGT
2015 Kenig[30]	P	184	76.9	53.2	50	Abdominal	EM	BFI, G8, GFI, RFT, VES
2015 Revenig[31]	P	351	63	39	27.3	Abdominal	M	FP
2017 Hall[32]	P	1021	60.2	4.2	31	Mixed	EL	RAI-A, RAI-C, mFI-11
2017 Kapoor[33]	P	403	72	48.9	18	Mixed	EL	FP, LLFDI-F
2018 Gilbert[34]	R	1013590	84.1	57.4	57.6	Mixed	EM	HFRS
2018 Han[35]	P	176	69.5	53.4	23.7	Abdominal	EL	FP
2018 Kenig[36]	P	315	77	52.4	60.3	Abdominal	EM	G8
2018 Ondeck 1[37]	R	68580	65.1	55.8	3.9	THA	EL	CCI, Elixhauser, mFI-11
2018 Ondeck 2[38]	R	49738	82	72.3	5	Hip fracture	EM	CCI, Elixhauser, mFI-11
2018 Zattoni[39]	P	556	81	57.3	29	Abdominal	EM	CCI, TRST
2019 Al-Hamis[40]	R	295490	61	52	18	Colorectal	EL	mFI-5
2019 Amin[41]	R	158855	64.6	22.9	16.9	Urological	M	mFI-5, mFI-11, RAI-A
2019 Fu[42]	R	10527	69.2	56.4	2.5	Shoulder	EL	CCI, mFI-11
2019 Katlic[43]	P	513	80.5	63.7	47.4	Mixed	EL	CCI, FP
2019 Lima[44]	P	229	69	55	-	Mixed	EL	CFS
2020 Arya[45]	R	6856	60.7	3.6	19.9	Non-cardiac	EL	RAI-C, RAI-C Rev
2020 Barazanchi[46]	R	758	62	50.1	-	Laparotomy	EM	mFI-11
2020 Choi[47]	R	648	76.6	52.8	-	Mixed	EL	6-min walk, MFS
2020 He[48]	P	134	76.9	50	29.1	Abdominal	M	EFS
2020 Lu[49]	P	136	77.5	67	36.8	Hip fracture	EM	FI
2020 McIsaac[50]	P	645	74	49.8	36.6	Non-cardiac	EL	CFS, FI
2020 Rogozinski[51]	R	451	65.1	7.1	-	THA, TKA	EL	CCI, mFI-11
2020 Roopsawang[52]	P	200	72	78	43	Orthopedic	EL	Self-reported EFS
2021 Aguilar-Frasco[53]	P	140	72.7	47.1	35	Abdominal	EL	RFI
2021 Arteaga[54]	P	92	78.7	53.3	14.1	Abdominal	EM	FRAIL, FI, TRST, CFS
2021 Costa[55]	P	240	77.6	47.9	-	Abdominal	EM	EmSFI
2021 Lee[56]	R	4664	80	40.5	47.6	Mixed	EM	OFRS
2021 Pandit[57]	R	8681	76	32	24.5	LEA	EL	mFI-5, mFI-11
2021 Tse[58]	R	47197	66	1.1	-	LEA	M	RAI-A Rev
2021 Wu[59]	R	397	83.5	63	90	Hip fracture	EM	CCI, CFS, KPS
2021 Yi[60]	R	3893	68	77.6	24.8	Shoulder	EM	CCI, mFI-5
2021 Yin[61]	P	194	79	53.6	32.5	Abdominal	EL	CFS, FI, FRAIL
2022 Conlon[62]	R	6571	64	42.7	5.4	Spine	EM	mFI-5, RAI-A, RAI-A Rev
2022 Cotton[63]	R	298	67	0	65.8	LEA	EL	mFI-11, RAI-C
2022 Forssten[64]	R	2365	84	67.7	47	Hip	EM	CCI, mFI-5
2022 Ikram[65]	P	1577	83.6	56.4	44.3	Hip	EM	CFS
2022 Iwasaki[66]	R	476	92.4	64.3	32.5	Non-cardiac	M	ECOG-PS, mFI-5
2022 Kweh[67]	R	272	73.5	45.6	20.6	Spinal	M	mFI-5, mFI-11
2022 Le[68]	R	37186	67.9	51.4	20.2	Abdominal	M	FI, HFRS, mFI-5, RAI-A
2022 Lee[69]	R	1557	80.4	60.2	14.9	Mixed	EM	CCI, HFRS, OFRS
2022 Li[70]	R	923	73.5	37.6	24.4	GI	EL	CRI, mFI-11
2022 Palaniappan[71]	R	1434	65	51	10.6	Abdominal	EM	CFS
2022 Ruiz[72]	P	100	61.3	51	-	Abdominal	EM	UEF
2022 Wei[73]	R	4195	73.9	38.6	-	Abdominal	M	RAI-A Rev
2022 Yin[74]	P	194	77	53.6	37.6	Abdominal	EL	CFS, FI, FRAIL
2023 Darbyshire[75]	R	1508	66	54.1	-	Bowel	EM	HFRS
2023 McConaghy[76]	R	433311	>60	55.5	-	THA, TKA	EL	CCI, Elixhauser, mFI-5
2023 Sirisegaram[77]	R	535	72	40.2	21.1	Mixed	EL	EFS

P, prospective study; R, retrospective study; –, information not available; EL, Elective; EM, Emergency; M, Mixed; BFI, Balducci Frailty Index; CCI, Charlson Comorbidity Index; CFS, Clinical Frailty Scale; CRI, Composite Risk Index; ECOG-PS, Eastern Cooperative Oncology Group performance status; EFS, Edmonton Frailty Scale; EmSFI, Emergency Surgery Frailty Index; FI, Frailty Index; FP, Fried’s Phenotype; G8, Geriatric 8; GFI, Groningen Frailty Indicator; HFRS, Hospital Frailty Risk Score; KPS, Karnofsky Performance Status; LEA, Lower Extremity Amputation; LLFDI, Late-Life Function and Disability Instrument; mFI-5, 5-Item Modified Frailty Index; mFI-11, 11-Item Modified Frailty Index; MFS, Multidimensional Frailty Score; OFRS, Operation Frailty Risk Score; RAI-A, Risk Analysis Index – Administrative; RAI-C, Risk Analysis Index – Clinical; RFT, Rockwood Frailty Test; THA, Total Hip Arthroplasty; TKA, Total Knee Arthroplasty; TRST, Triage Risk Screening Tool; TUGT, Timed-up-and-go test; UEF, Upper Extremity Function; VES, 13-Item Vulnerable Elders Survey; Rev, Revised

Surgical and patient populations

Both abdominal and orthopedic surgical patients were the most studied populations (14 studies each [27%]), followed by mixed surgical patients (13 studies [25%]). Elective was the most studied surgical urgency population (24 studies [46%]), followed by emergency (20 studies [38%]) and mixed (8 studies [15%]). The average study population age ranged from 60.2 to 92.4 years, and the proportion of female patients ranged from 0% to 98% with the averages being 73.2 years and 48.7%, respectively. Frailty prevalence ranged from 2.5% to 90% with the average being 30.5%.

Frailty tests

Our review revealed twenty-nine unique frailty tests [Table 3]. The most prevalent was the Modified Frailty Index (30 data from 18 studies), followed by the Charlson Comorbidity Index (23 data from 12 studies), Frailty Index and Clinical Frailty Scale (9 data from 5 and 7 studies, respectively). The most common model of frailty used was the accumulation of deficits (13 frailty tests) followed by multidimensional (11 frailty tests) and phenotype of frailty (5 frailty tests). Seven frailty tests were database compatible, all of which used the accumulation of deficits model. Three frailty tests (Clinical Frailty Scale, Phenotype of Frailty and Upper Extremity Function) required training, two of which were due to the use of specialist equipment. Three multidimensional frailty tests (Composite Risk Index, Robinson Frailty Test and Multidimensional Frailty Score) required the use of imaging or blood tests. No tests used proprietary content.

Table 3.

A summary of 29 frailty tests and their usefulness in preoperative screening. Accuracy of predicting post-operative length of stay, complication, and mortality is represented by the mean c-statistic and its range. Mortality and complication outcomes are measured within 3 months or less of surgery. Electronic database compatibility is defined as being based on International Classification of Diseases diagnosis codes or National Surgical Quality Improvement Program variables. Electronic database scores were not recorded as having any time since they are not used manually. Cost is defined as using any proprietary tests or the use of blood tests or imaging

Test Number	Test Name	Duration (min)	Items, n	Extended Length of Stay			Increased Post-Operative Complications			Increased Mortality			Database use	Training	Equipment	Cost
				Mean	Range	n	Mean	Range	n	Mean	Range	n
Phenotype of Frailty
1	FRAIL[78]	<5	5	0.81			0.68			0.80	(0.72-0.87)	2
2	TUGT[79]	5	1	-			0.78			-
3	UEF[80]	5	6	-			-			0.74				X	X
4	6MWT[81]	10	1	-			0.67			-
5	PF[14]	15	5	0.74			0.67	(0.60-0.76)	4	0.75	(0.68-0.81)	2		X	X
n = 5 tests
Accumulation of deficits
6	Bal[82]	<5	4	-			0.55			0.56
7	TRSI[83]	<5	5	-			0.61	(0.52-0.69)	2	0.65	(0.52-0.72)	3
8	RAI-C[32]	<5	14	-			0.64			0.74	(0.70-0.82)	4
9	ESFI[55]	5	9	-			0.54			0.76
10	Bar[84]	5	10	-			0.67			0.69
11	VES[85]	5	13	-			0.68			0.64
12	MFI[86]	-	5	0.61	(0.53-0.76)	8	0.65	(0.52-0.88)	10	0.74	(0.59-0.96)	12	X
13	RAI-A[32]	-	14	0.67			0.61	(0.57-0.66)	6	0.76	(0.60-0.98)	6	X
14	CCI[87]	-	17	0.61	(0.50-0.72)	7	0.60	(0.52-0.71)	6	0.70	(0.59-0.82)	7	X
15	ECI[88]	-	31	0.66	(0.56-0.74)	3	0.57			0.62	(0.62-0.62)	2	X
16	FI[89]	-	40	0.85			0.71	(0.64-0.75)	3	0.80	(0.73-0.85)	5	X
17	HFRS[34]	-	109	0.68			0.71			0.67	(0.55-0.82)	4	X
18	OFRS[56]	-	111	-			-			0.72	(0.62-0.81)	2	X
n = 13 tests
Multidimensional
19	CFS[90]	<5	1	0.88			0.75			0.77	(0.63-0.91)	7		X
20	ECOG[91]	<5	1	-			-			0.98
21	KIPS[92]	<5	1	-			-			0.82
22	RFI[93]	<5	1	-			0.52			0.57
23	G8[94]	<5	8	-			0.70	(0.56-0.83)	2	0.75	(0.57-0.92)	2
24	EFS[95]	<5	11	-			0.73	(0.69-0.81)	3	0.75
25	LLFDI[96]	<5	40	-			0.67			-
26	GFI[97]	5	15	-			0.60			0.58
27	RFS[28]	15	7	-			0.70			-				X	X	X
28	CRI[70]	20	8	-			0.65			-				X	X	X
29	MFS[98]	20	9	-			0.75			-				X	X	X
n = 11 tests

–, information not available; 6MWT, 6-minute walk test; Bal, Balducci frailty test; Bar, Barthel frailty test; CCI, Charlson Comorbidity Index; CFS, Clinical Frailty Scale; CRI, Composite Risk Index; ECOG-PS, Eastern Cooperative Oncology Group performance status; EFS, Edmonton Frailty Scale; EmSFI, Emergency Surgery Frailty Index; FI, Frailty Index; FP, Fried’s Phenotype; G8, Geriatric 8; GFI, Groningen Frailty Indicator; HFRS, Hospital Frailty Risk Score; KPS, Karnofsky Performance Status; LEA, Lower Extremity Amputation; LLFDI, Late-Life Function and Disability Instrument; mFI-5, 5-Item Modified Frailty Index; mFI-11, 11-Item Modified Frailty Index; MFS, Multidimensional Frailty Score; OFRS, Operation Frailty Risk Score; RAI-A, Risk Analysis Index – Administrative; RAI-C, Risk Analysis Index – Clinical; RFT, Rockwood Frailty Test; TRST, Triage Risk Screening Tool; TUGT, Timed-up-and-go test; UEF, Upper Extremity Function; VES, Vulnerable Elders Survey

Supplementary Table 1.2.

Search terms used for Embase. Search period from 1947 to March 10, 2023

Search Step	Search Terms	Total Studies, n
1	(preoperat* OR “pre-operat*”)	547321
2	frail*	51617
3	(old OR elderly OR geriatric OR aged)	5217295
4	(test* OR screen* OR assess* OR index* OR indicator* OR rule* OR measur* OR tool* OR instrument* OR scale* OR score* OR metri* OR rating OR resignation OR phenotype)	14682871
5	(complication* OR “adverse event”) AND (“post-operative”)	65120
6	(mortality OR “length of stay”)	1725801
7	5 OR 6	1773169
8	1 AND 2 AND 3 AND 4 AND 7	1282

Length of stay

A total of 24 data from ten frailty tests were found for predicting length of stay. The Modified Frailty Index was the frailty test with most data (8 values) followed by Charlson Comorbidity Index (7 values) and Elixhauser Comorbidity Index (3 values). Overall, predictive ability ranged from 0.50 to 0.88 with underreported frailty tests like Clinical Frailty Scale and FRAIL scale reporting excellent discrimination [Table 3.19: 0.88; Table 3.1: 0.81]. The three frailty tests with the most data all had fair discrimination [Table 3.12: 0.61; Table 3.14: 0.61; Table 3.15: 0.66]. Notably, these three tests are all of the accumulation of deficits frailty model and are database compatible. Although these three tests are automatically calculated and do not take time, Elixhauser Comorbidity Index required the most data followed by Charlson Comorbidity Index and Modified Frailty Index [Table 3.15: 31; Table 3.14: 17; Table 3.12: 5]. None of these tests have extra costs. The Modified Frailty Index has the best ratio of data required to accuracy for extended length of stay.

Post-operative complications

A total of 53 data from 25 frailty tests were found for predicting post-operative complications. The Modified Frailty Index was the frailty test with the most data (10 values) followed by the administrative Risk Analysis Index and Charlson Comorbidity Index (both 6 values), and the Phenotype of Frailty (4 values). Overall, predictive ability ranged from 0.52 to 0.88 with most underreported tests exhibiting low to fair predictive accuracy. The most reported tests had fair predictive abilities [Table 3.12: 0.65; Table 3.13: 0.61; Table 3.14: 0.60]. All frailty tests with ≥3 data were either automatically calculated via database with under 20 data items required or took less than 5 minutes to complete. Edmonton Frailty Scale was included among these with good discrimination [Table 3.24: 0.73]. The exceptions were: Frailty Index has an extensive item requirement of 40 but with good discrimination; Phenotype of Frailty has a time requirement of 15 minutes but with only fair discrimination [Table 3.17: 0.71; Table 3.5: 0.67]. Phenotype of Frailty also requires training. None of these tests have extra costs. Edmonton Frailty Scale has the best ratio of time required to accuracy for post-operative complications.

Post-operative Mortality

A total of 71 data over 23 frailty tests were found for predicting post-operative mortality. The Modified Frailty Index was the frailty test with the most data (12 values) followed by the Charlson Comorbidity Index (10 values), and the administrative Risk Analysis Index and Clinical Frailty Scale (both 7 values). Overall, predictive ability ranged from 0.52 to 0.98 with most underreported tests exhibiting poor to good predictive accuracy; some exceptions such as Eastern Cooperative Oncology Group Performance score and Karnofsky’s Index of Performance Status had excellent discrimination [Table 3.20: 0.98; Table 3.21: 0.82]. The top three most accurate frailty tests with ≥3 data were Frailty Index, Clinical Frailty Scale, and administrative Risk Analysis Index [Table 3.17: 0.80; Table 3.19: 0.77; Table 3.13: 0.76]. Again, Frailty Index suffers from requiring 40 items compared to the others which take less than 5 mins or have fewer than 20 items. However, Clinical Frailty Scale also requires training. None of these tests have extra costs. Clinical Frailty Scale proved to have the best ratios of time to accuracy for mortality.

Assessment of best frailty tests

Of a total of 29 frailty tests, only six were included in the final assessment after excluding undesirable findings and properties [Figure 2]. 18 of the tests were excluded due to insufficient data for predicting either length of stay, post-operative complication or mortality. Four frailty tests were also rejected due to overrepresentation of specific surgical populations. Notably, Charlson and Elixhauser Comorbidity indices overrepresented orthopedics. Finally, the Phenotype of Frailty was excluded due to it taking ≥5 minutes to complete.

Figure 2.

Flowchart of process for selection of top five frailty tests. Process has been outlined in methods. LoS, length of stay; POC, post-operative complications

Of the remaining six tests, five were included in the final ranking [Table 4]. Forms used to complete these frailty tests have also been included [Supplementary Table 2.1-5] The excluded frailty test, Hospital Frailty Risk Scale, had a predictive ability for mortality far below the remaining 5 (0.67) and not enough data to predict length of stay or post-operative complication. The top five include three clinical frailty tests (Clinical Frailty Scale, clinical Risk Assessment Index, and Edmonton Frailty Scale) and two administrative frailty tests (Modified Frailty Index and administrative Risk Assessment Index). Overall, the best frailty test for predicting increased mortality and increased post-operative complications was the Clinical Frailty Scale and Edmonton Frailty Scale, respectively. Both had good predictive ability (0.77 and 0.73). Only the Modified Frailty Scale had enough data for predicting extended length of stay, the ability of which was found to be poor (0.61).

Table 4.

Summary of top five frailty tests with average c-statistic. Average c-statistics with ranges are derived from Table 3. All tests take no longer than 5 minutes to complete or are derived automatically from administrative data. All tests have been assessed in diverse surgical populations. Assessor accessibility is a measure of the ability for populations to be able to complete the test

Ranking	Test Name	Extended Length of Stay	Increased Post-operative Complications	Increased Mortality	Assessor Accessibility			Advantages	Disadvantages
					Doctor	Nurse	Patient
Clinical Frailty Tests
1	CFS	-	-	0.77 (0.63-0.91)	X	X		Very quick to complete (44 s on average).[50] Visual chart available for reference. Online training module available that takes <30 minutes.[99] App available for phone or tablet.[100]	Concerns over inter-rater reliability (0.78-0.79).[101,102] Training recommended. Lacking self-reporting data. Likely difficult due to training recommended.
2	RAI-C	-	-	0.74 (0.70-0.82)	X	X		Form available for completion. Recent revision completed.[45]	Lacking self-reporting data.
3	EFS	-	0.73 (0.69-0.81)	-	X	X	X	Form available for completion. Validated for self-reporting.[77] Includes nutrition and cognitive components.
Administrative Frailty Tests
1	MFI	0.61 (0.53-0.76)	0.65 (0.52-0.88)	0.74 (0.59-0.96)	X	X		Automatically generated from hospital database using NSQIP codes.	Requires hospital database.
2	RAI-A	-	0.61 (0.57-0.66)	0.76 (0.60-0.98)	X	X		Automatically generated from hospital database using NSQIP codes. Recent revision completed.[45]	Requires hospital database. Requires more items for calculation. More likely to have missing data.

CFS, Clinical Frailty Scale; EFS, Edmonton Frailty Scale; MFS, Modified Frailty Index; RAI-A, Risk Assessment Index – Administrative; RAI-C, Risk Assessment Index – Clinical

Supplementary Table 2.1.

Clinical Frailty Scale

Supplementary Table 2.5.

Administrative risk analysis index

RAI Variable	MDS Variable	VASQIP Variable	RAI-A Scoring System*
1. Sex	Sex	SEX	+5 if "male"
2. Age	Age	AGE	Continuous (Scored as interaction with Cancer Diagnoses as per table in eFigure 1)
3. Cancer (excluding skin cancer, except for melanoma)	Cancer diagnosis with or without metastasis	DISCANCER or RADIO or CHEMO	1= any of the 3 variables "yes" 0 = all of the 3 variables "no"
4. Weight Loss ("Have you had unintentional weight loss in the past 3 months >10 pounds?"	Weight loss	WTLOSS	+5 if "yes"
5. Renal Failure	Renal failure	RENALFAIL or DIALYSIS	+6 if either variable "yes"
6. Chronic/Congestive Heart Failure	Chronic heart failure	HXCHF	+4 if "yes"
7. Poor Appetite	Poor appetite	WTLOSS	+4 if "yes"
8. Shortness of Breath at Rest	Shortness of breath	DYSPNEA	+8 if "yes"
9. Residence other than Independent Living	Recent admission to nursing home	TRANST	+8 if transferred to the hospital for the index operation from a nursing home, chronic care facility, spinal cord injury unit or intermediate care unit
10. Cognitive Deterioration ("Have your cognitive skills or status deteriorated over the last 3 months?")	Cognitive Deterioration	IMPSENS or COMA or CVANEURO	"yes" if any of 3 variables "yes" "no" if all of 3 variables "no" (Scored as interaction with Activities of Daily Living as per table below)
11. Activities of Daily Living (Mobility, Eating, Toileting, Personal Hygiene)	Short-Form ADL Scale in 4 dimensions	FNSTATUS	Without Cognitive Decline +16 = totally dependent +8 = partially dependent +0 = independent With Cognitive Decline +21 = totally dependent +10 = partially dependent -2 = independent

The RAI-A score is calculated the same way as the RAI-C, and both scores range between 0 and 81

Supplementary Table 2.2.

Clinical risk analysis index

Scoring Instructions: To calculate the RAI-C score, first look at the age/cancer table to determine the single value between 2 and 20 that corresponds to the patient's age and cancer status. Record this single value in the appropriate line for item 3. Next look at the ADL table and sum the scores (0–4) for the four ADLs queried in items 10–13. This sum is the ADL Score and should range between 0 and 16. Next look at the ADL/Cognitive-Decline table to determine the single value between - - 2 and 21 that corresponds to the patient's ADL Score and cognitive decline. Record the value in the appropriate line for item 14. Finally, sum the values for items 1,3-9, and 14 to yield a final RAI-C score between 0 and 81

Supplementary Table 2.3.

Edmonton frail scale

Supplementary Table 2.4.

Modified Frailty Index

Modified Frailty Index
Item	Score
Functionally dependent	1
History of diabetes	1
Chronic obstructive pulmonary disease	1
Congestive heart failure	1
Hypertension	1
Total:

A score of >2 designates a frail person.

All of the top five frailty tests are very quick to perform either being automatically calculated from a database using software or taking ≤5 minutes in preoperative clinic. However, database tests with numbers of items greater than ten, such as the administrative Risk Analysis Index, can struggle with missing data. Only the Edmonton Frailty Scale has been validated for self-reporting and only the Clinical Frailty Scale has an app available for tablet or phone. Although the frailty test recommends training, which is freely available and short, the Clinical Frailty Scale combines the best predictive ability with the best feasibility of fastest time to complete and phone app availability.

Risk of bias analysis

Risk of bias results according to the Quality in Prognosis Studies tool has been reported [Supplementary Table 3]. A major contributor to high risk of bias was not reporting confounding factors like duration and stress of surgery and method of anesthesia and not accounting for these confounding factors in analysis. Poor reporting of missing preoperative frailty test data also commonly contributed to high risk of bias. The removal of high risk of bias studies from the ranking analysis [see Supplementary Table 3 in red] did not change the ranking of preoperative frailty tests.

Supplementary Table 3.

Risk of bias assessment of 30 studies involved in top 5 ranking analysis according to Quality in Prognosis Studies tool. Green, yellow, red, and white represent low, moderate, and high and unclear risk of bias, respectively

Study	Study Participation	Study Attrition	Prognostic Factor Measurement	Outcome Measurement	Study Confounding	Statistical Analysis and Reporting	Overall Risk of Bias
Al-Hamis 2019
Amin 2019
Arteaga 2021
Arya 2020
Barazanchi 2020
Conlon 2022
Cotton 2022
Dasgupta 2008
Forssten 2022
Fu 2019
Hall 2017
He 2020
Ikram 2022
Iwasaki 2022
Kweh 2022
Le 2022
Li 2022
Lima 2019
McConaghy 2023
McIsaac 2020
Ondeck 2018
Ondeck 2018 2
Palaniappan 2022
Pandit 2021
Rogozinski 2020
Roopsawang 2020
Tse 2021
Wu 2021
Yi 2021
Yin 2021

Discussion

In this systematic review that compared predictive accuracy of preoperative frailty tests using c-statistic values from 52 studies, we found that: (a) Clinical Frailty Scale combined the best predictive accuracy for mortality with the best feasibility thanks to speed of completion and phone app availability; (b) Edmonton Frailty Scale was the best predictor for post-operative complications and potentially has similarly excellent feasibility due to its validated ability for self-reporting; (c) at this time, we would not recommend any frailty test for predicting extended length of stay due to poor accuracy of tests and lacking data; (d) Modified Frailty Index was found to be the best administrative frailty test overall but administrative Risk Analysis Index outperforms it in mortality prediction though it requires far more data items, which predisposes it to missing data.

Previously, the choice of frailty test had been obscured using odds ratios, from which no statistical difference could be detected between frailty tests despite poor-to-modest agreement between them.[3] In our analysis of predictive accuracy, we were able to rank tests by their average c-statistic, but not able to calculate statistical differences due to the small number of studies. Moreover, the small differences in average c-statistic are unlikely to translate into any observable difference in predictive ability in practice. As such, despite the use of the c-statistic, we have found similar results to other reviews such as Aucoin et al.,[18] who found that, whilst the Clinical Frailty Scale has the highest odds ratio of 4.89 for mortality, it was not statistically different from other frailty tests. In head-to-head cohort studies comparing predictive ability of different frailty tests, the Clinical Frailty Scale is also consistently better than other frailty tests, but only by differences in c-statistic of 0.01–0.02, which have little observable clinical effect.[103] Indeed, despite the use of the c-statistic, our ranking of frailty tests is still based on feasibility, for which the Clinical Frailty Scale excels at. However, if more studies were available presenting c-statistics and their confidence intervals, statistical differences may be able to be calculated. Especially now that the effects of frailty on post-operative outcomes are well established via odds ratios, the emphasis of research needs to be on predictive accuracy of frailty tests so that such statistical tests can be done in future reviews.

Another large contributing factor to difficulty choosing frailty tests for preoperative screening is the explosion of frailty tests designed for different surgical populations. Tests such as the Nottingham Hip Fracture Score and Addenbrookes Vascular Frailty Score, which include significant frailty components, have been designed for predicting post-operative complications in orthopedic and vascular populations.[104,105] Despite the increased accuracy that these may provide, we believe they are not feasible for the anesthesiologist working mixed caseloads. The widespread use of the American Society of Anesthesiologists physical status score can be partly attributed to its use in all surgery types.[106] Indeed, rather than having very many frailty tests for each surgical population, it is more prudent to have a single frailty test, which has a component that quantifies the risks of different surgeries. We recommend surgery-specific frailty tests be reserved for surgeons.

If the Clinical Frailty Scale is found to be the most accurate frailty test, the issue of bias due to its judgment component needs to be addressed. Although the scale’s training program emphasizes that physicians look out for leniency and central tendency bias effects, two separate studies have reported good inter-rater reliability for the Clinical Frailty Scale, finding kappa values between 0.74 and 0.85 after standardized training.[101,102] In context, the inter-rater reliability of the American Society of Anesthesiologists physical status score has been found to be 0.40, 0.61, and 0.21–0.4 in different studies, suggesting that the Clinical Frailty Scale may be more reliable than one of the most widely used preoperative risk scores.[107,108,109]

The future of frailty tests should take advantage of the multidimensional nature of frailty, as exemplified by the many categories of the CGA, so that the accuracy of the Clinical Frailty Scale is further improved. Some tools found in this systematic review have already tried to combine multiple techniques to increase risk prediction but lacked sufficient data to be properly analyzed [Table 2.27-29]. While we need to balance feasibility, it is possible and desirable for a frailty test to include other dimensions like nutrition and cognition, both of which in poor condition can increase post-operative mortality by 3.86 in hip fracture surgery and 1.6 in any elective surgery, respectively.[110,111] Edmonton Frailty Scale already includes the clock drawing test for cognition and a basic screening question for nutrition, but such tests need to be further refined, especially for difficult to detect conditions like mild cognitive impairment, which contributes significantly to post-operative complications like post-operative delirium.[112] The ideal preoperative assessment would be a “mini-CGA,” which is able to combine all dimensions into a single score that can predict surgical risk.

Other possible additions to this mini-CGA could be emerging imaging and blood biomarkers for frailty. Sarcopenia, the loss of muscle mass related to age, which commonly coincides with frailty, can be measured with computed tomography or ultrasound scans.[113] The measurement of quadriceps depth with ultrasound, which suffers from requiring a trained sonographer, can accurately predict post-operative delirium with a c-statistic of 0.89.[114] Turning to blood biomarkers, serum albumin, which is very commonly measured, is a composite of nutrition and liver and kidney condition and its preoperative to post-operative change has been found to predict post-operative complication.[115] Newer frailty biomarkers, like interleukin-6 and alpha-1-acid glycoprotein, may have better accuracy, a recent trial finding a 0.781 c-statistic for both predicting morbidity, but may not be available in most pathology labs.[116] Other experimental neurological biomarkers such as neurofilament light chain and glial fibrillary acidic protein may be more useful for anticipating post-operative delirium and cognitive dysfunction.[117,118] All such additions would significantly improve post-operative risk prediction but may have feasibility issues.

The problems of feasibility associated with further additions to different dimensions to the mini-CGA could be attenuated by allowing patients to rate their own frailty, which could be accommodated using digital apps. The Edmonton Frailty Scale was recently validated as a self-reporting tool if the timed-up-and-go and clock drawing tests are removed.[77] This missing data could be collected by phone apps: walking pace, which has strong agreement with the timed-up-and-go, can be tracked via accelerometer; and many apps exist to test cognition such as the Cogstate Brief Battery, which has better sensitivity and specificity for detecting mild cognitive impairment than the clock drawing test.[119,120,121] Instruction to use these apps could be provided before preoperative consultation so that the data may be used for frailty assessment. Current barriers include the proprietary nature of cognitive testing phone apps and technological illiteracy in elderly populations.[122] Nevertheless, such additions could completely revolutionize feasibility and accuracy.

Finally, an area of preoperative assessment that needs significant accuracy improvement is prediction of extended length of stay. In this review, extended length of stay had less than half the data than post-operative complications and mortality and had poor accuracy in frailty tests with enough data for assessment. A common cause of extended length of stay is low severity post-operative complications.[123] According to Mah et al.,[124] the Modified Frailty Index may be very good at predicting Clavien-Dindo grade 3–5 complications (0.92) but inclusion of grade 2 reduces accuracy significantly (0.74). A possible explanation for this is that frailty may have poor association with minor post-operative complications. However, since some of the best frailty tests, like Clinical Frailty Scale, have little to no extended length of stay data, this cannot be completely verified. As such, more research should be focused on assessing preoperative frailty screening tools, especially clinical frailty tests, for predicting extended length of stay.

Strengths and Limitations

This systematic review is the first of its kind to assess predictive accuracy of post-operative outcomes via the c-statistic as its primary outcome. This allows a quantification of accuracy of specific tests without the interference of outcome prevalence in a sample, which is the problem with odds ratios.[17] To avoid interference via confounding factors, this review had well-defined inclusion criteria, which excluded surgeries where frailty is more likely to influence post-operative outcomes such as cardiac and major thoracic and abdominal vascular surgery. We also assessed feasibility and displayed important test properties for the reference of anesthesiologists. Where possible, we followed the systematic review best practice by including risk of bias assessment, PRISMA and displaying of our complete search methodology.

While we did attempt to measure predictive accuracy, other insightful statistical metrics exist such as predictive values, likelihood ratios and calibration.[125] Indeed, similar c-statistics may not necessarily have comparable positive predictive values. Lack of available data made use of these metrics unfeasible for review at this time. While we can rank accuracy based on the average c-statistic, many tests differed by small values, which may have little effect in practice. Finally, since the average was used, there was no analysis of heterogeneity, and all studies were weighted the same. It can be predicted that, due to the wide variety of surgical types and different cutoffs for frailty tests, heterogeneity may be high.

Additionally, feasibility of frailty tests in the emergency surgery setting was not explored, which poses unique challenges with timing and communication with patients of reduced consciousness. A systematic review of feasibility in acute trauma suggests that the Clinical Frailty and FRAIL scales can be completed in between 71% and 100% and 62% and 100% of cases, respectively.[126] Problems may arise in longer frailty tests such as the Frailty Index, which was found to have a completion rate of only 31.9%.[127] Additionally, institutionalization, a metric of patient post-operative quality of life, was not assessed. Despite a finding of 22% of elderly patients being institutionalized after abdominal surgery, this metric had poor data availability during initial scoping research and requires further exploration.[128]

Conclusions

This is the first systematic review to rank preoperative frailty tests according to a metric of predictive accuracy in addition to their feasibility. Clinical Frailty Scale was found to be the best for predicting mortality; this, alongside its standout time efficiency and phone app availability, made it the definitive preoperative frailty test. Another notable test was Edmonton Frailty Scale, which had the best predictive ability for post-operative complications and represents future opportunities for feasibility via self-reporting. Research emphasis must continue to move away from odds ratios to predictive accuracy metrics like the c-statistic, especially for extended length of stay.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.