Frailty Screening Using the Risk Analysis Index: A User Guide

Abstract

The Risk Analysis Index (RAI) has emerged as the most thoroughly validated and flexible assessment of surgical frailty, proven feasible for at-scale bedside screening and available in a suite of tools, that effectively risk stratifies patients across a wide variety of clinical contexts and data sources. This user guide provides a definitive summary of the RAI’s theoretical model, historical development, validation, statistical performance, and clinical interpretation, placing the RAI in context with other frailty assessments and emphasizing some of its advantages. Detailed instructions are provided for each RAI variant, along with a systematic review of existing RAI–related literature.

BACKGROUND

Frailty is a clinical syndrome of decreased physiologic reserve whereby small deficits accumulate in multiple adaptive systems, any one of which might be clinically insignificant, but that together produce significant vulnerability to catastrophic decompensation when a stressor occurs, such as surgery.¹ Frailty has multiple causes and contributors, characterized by diminished strength, endurance, nutrition, and cognitive status, with an emphasis on functional performance. Although frailty increases with age, it is more than just age and the sum of comorbidities, and it can explain why older robust patients recover from the same surgery that overwhelms the physiologic reserve of younger, frail patients (Figure 1).

Figure 1:

As people age (x-axis), they expend physiologic reserve (y-axis) until it is exhausted at the time of death. Although the average person expends their reserve at a set rate (black line), the slope of this line can vary between individuals, with lower slopes indicating more robust patients (green) and higher slopes indicating more frail patients (yellow). This can explain why the stress of a similar surgical procedure (red lines) may overwhelm the reserves of a younger frail patient even as an older robust patient is able to recover.

Originally conceptualized and measured by geriatricians, frailty is increasingly recognized as one of the strongest predictors of postoperative outcomes, including complications,² length of stay,³ readmission,⁴ loss of independence,⁵ failure to rescue,⁶ mortality,³ and cost.⁷ As such, there is a critical need for a rapid, point-of-care, frailty assessment to screen predominantly robust populations of patients considering elective surgery. Whereas surgeons have always been attentive to physiologically stressful, high-risk surgeries such as pancreatectomy, esophagectomy, or pneumonectomy, frailty screening permits clinicians to recognize patients at high risk even after moderately stressful procedures. As such, frailty screening becomes the critical first step in setting realistic expectations and mitigating adverse outcomes.

The Risk Analysis Index (RAI) was developed in 2012 to facilitate rapid preoperative frailty screening as part of a quality improvement initiative at the US Department of Veterans Affairs (VA) Omaha VA Medical Center (VAMC).⁸ It has subsequently emerged as one of the most thoroughly validated measures of surgical frailty^9–14 with proven ability to screen large volumes of clinic patients in entire health care systems without disrupting clinic flow.^{9, 15} The key to widespread implementation has been a bedside survey instrument administered by a nurse or physician,¹⁶ now validated as a patient-facing tool.^{15, 16} In addition, alternate versions of the RAI can be applied to retrospective datasets, such as the Surgical Quality Improvement Programs^17–19 and administrative billing data using the 10th edition of International Classification of Disease (ICD) codes.²⁰ This suite of RAI tools permits clinicians and investigators to ask and answer frailty-related questions in readily available data to identify quality improvement opportunities and inform development of targeted interventions.

TOOL DEVELOPMENT

At the time of RAI development, tools existed for quantifying frailty in retrospective datasets,²¹ but the primary prospective measure of frailty²² was not clinically feasible because of the time and specialized equipment required (for example, dedicated research personnel, grip dynamometers). Literature review identified the Minimum Data Set Mortality Risk Index (MMRI) as a prospective survey validated to predict 180-day mortality at the time of admission to a nursing home.²³

Although not originally validated in surgical populations, the MMRI was modified by leadership at the Omaha VAMC, to eliminate a question about dehydration considered not applicable to the elective surgical setting, and leadership made the score mandatory for scheduling elective surgery. This modified MMRI was renamed the RAI. At the same time, each item on the RAI survey was mapped to variables contained within the VA Surgical Quality Improvement Program (VASQIP), permitting the retrospective calculation of RAI scores for VASQIP cases (Table 1) and setting a frailty threshold value corresponding to the highest risk decile.^24–28

Table 1. Components, Definitions, and Operationalizations Across the Four Risk Analysis Index (RAI) Variants.

	Definition	RAI-C	RAI-A	RAI-VQI	RAI-ICD
Age	Chronological age in years.	Patient’s stated age or age recorded in electronic record; clinicians should choose the most accurate integer value.	AGE	AGE	—
Sex	Biological sex as a binary construct, not to be confused or conflated with gender identity or legal sex; clinicians are encouraged to use clinical judgment in ascertaining this variable respectfully in cases in which gender identity or legal sex are different than biological sex.	Patient’s reported sex or sex recorded in the electronic record; clinicians should choose the most accurate. (Response values: “male” or “female.”)	SEX	GENDER	—
Cancer	Intended to indicate any active cancer not definitively in remission. Originally operationalized in VASQIP as any neaodjuvent chemo- or radiotherapy or disseminated disease, subsequent work aimed at maximizing sensitivity focused on any cancer not in remission. (See Estock et al.13 for exploration of how best to interpret the cancer variable.)	In the past 5 years, have you been diagnosed with or treated for cancer?	DISCANCER or RADIO or CHEMO	—	C77–C79
Unintentional Weight Loss	Operationalized as more than 10 pounds in past 3 months.	In the past 3 months, have you lost 10 pounds or more without trying? (Yes/No)	WTLOSS	WTKG and HTCM and HTIN	R63.4, R63.6, R64, R62.7
Poor Appetite	Any patient- or caregiver-reported decline in appetite.	Is your appetite currently poor? (Yes/No)	WTLOSS	—	R63.0, R63.39, R63.8
Renal Failure	Operationalized to include not just renal failure and dialysis but also renal insufficiency, although no specific threshold of glomerular filtration rate or creatinine has been established. Patient facing survey operationalized as any visit to a “kidney doctor” for a condition other than renal calculi.	Have you ever seen a nephrologist (kidney doctor) or have a history of kidney problems? (Yes/No) If yes, was this for kidney stones or another problem? (Response values: “Kidney Stones,” “Other,” “Both kidney stones and another problem.”)	RENALFAIL or DIALYSIS	PREOP_CREAT or PREOP_DIALYSIS or DIALYSIS	I12.0, I13.11, I13.2, N18.4 - N18.6, Z49, Z99.2
Congestive Heart Failure	Any reported history of heart failure regardless of AHA classification, especially if there is associated deficit in functional performance.	Do you have chronic (long-term) congestive heart failure (CHF)? (Yes/No)	HXCHF	PRIOR_CHF	I50
Dyspnea	Any reported shortness of breath with minimal exertion. May also include any use of supplemental oxygen.	Do you have shortness of breath while resting or with minimal activity? (Yes/No) Prompt: “Do you have trouble catching your breath when you are resting or doing minimal activities (for example, walking to the bathroom or mailbox”)?	DYSPNEA	COPD	J96.1, R06.0, Z99.81
Other Living Setting	Any residence other than independence. Duration and location are not required for scoring but typically ascertained to inform the clinician.	Do you live in a nursing home, skilled nursing facility, or another assisted living environment? (Yes/No) If yes, where do you live? (Response values: “nursing home,” “skilled nursing facility,” “assisted living.”) If yes, “When did you begin living in there?” (Response values: “Less than 3 months ago,” “3 months to 1 year ago,” “Greater than 1 year ago.”)	TRANST	LIVING STATUS or TRANSFER or PREOP_AMBUL	—
Cognitive Decline	Any observable and chronic cognitive decline or dementia. Clinician or caregiver opinion is often preferred, given the challenges of self-report.	During the last 3 months, has it become more difficult for you to remember things or organize your thoughts?	IMPSENS or COMA or CVANEURO	—	F01.50, F01.51, F02.80, F02.81, F03.9, F03.91, G30, G31, R41.81
Functional Status and Activities of Daily Living *	Functional Status: Totally Dependent	—	FNSTATUS = 3, totally dependent	FUNCSTATUS	G12, G80.0– G80.2, G81, G82.21, G82.50– G82.54, R53.2, Z74.01, Z74.09, Z74.3, Z99.3, Z99.89
	Functional Status: Partially Dependent	—	FNSTATUS = 2, partially dependent	FUNCSTATUS	G82.22, G83.1, G83.2, R26.2, R26.8, R26.9, R53.1, R54, Z73.6, Z74.1, Z74.2, Z91.81
	Functional Status: Independent	—	FNSTATUS = 0, independent	FUNCSTATUS	—
	Mobility	Can get around without any help. (Score: 0) Needs help from a cane, walker, or scooter. (Score: 1) Needs help from others to get around the house or neighborhood. (Score: 2) Needs help getting in or out of a chair. (Score: 3) Totally dependent on others to get around. (Score: 4)	—	—	—
	Eating	Can plan and prepare own meals. Needs help planning meals. Needs help preparing meals. Needs help eating meals. Totally dependent on others to eat meals.	FNSTATUS	—	—
	Toileting	Can use the toilet without help. Needs help getting to or from the toilet. Needs help to use toilet paper. Cannot use a standard toilet, but with help can use a bedpan/urinal. Totally dependent on others to manage toileting.	FNSTATUS	—	—
	Personal Hygiene	Can shower or bathe without prompting or help. Can shower or bathe without help when prompted. Needs help preparing the tub or shower. Needs some help with some elements of washing. Totally dependent on others to shower or bathe.	FNSTATUS	—	—

C, clinical; A, administrative; VQI, Vascular Quality Initiative; ICD, International Classification of Diseases; VASQIP, US Department of Veterans Affairs Surgical Quality Improvement Program; AHA, American Heart Association; ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program.

^*Ideally conceptualized as activities of daily living, but approximated in VASQIP, ACS NSQIP, VQI, and ICD-10 with variables that indicate independent, partially dependent, or totally dependent functional status.

As described elsewhere,^{8, 29, 30} patients with RAI scores exceeding the frailty threshold were reviewed by the chief of surgery or a designee to identify opportunities for care plan optimization, substantially improving postoperative survival.^{8, 29} Based on this initial success, prospective frailty screening using the RAI survey was implemented across five hospitals at the University of Pittsburgh Medical Centers (UPMC) as well as the VA Pittsburgh Healthcare System. During implementation, cognitive interviewing techniques were used to edit the RAI for patient self-report, with the items simplified to the sixth-grade reading level.^{15, 16} At the same time, the RAI was recalibrated in a national sample of surgical patients to generate a scoring system based on a surgical sample, rather than the sample of nursing home admissions on which the MMRI was developed.⁹ The revised scoring system not only improved model discrimination and calibration, it also increased the mean score substantially, requiring a new and higher threshold to identify the highest risk decile.

TOOL DESCRIPTION

RAI Components, Performance, and Interpretation

From its inception, the RAI has included 11 parameters: age, sex, living location, appetite, weight loss, cognitive decline, dyspnea, congestive heart failure, renal insufficiency, cancer history, and activities of daily living (ADL) (Figure 2). As such, it is consistent with the deficit accumulation model of frailty, but unlike the frailty indices developed by Rockwood³¹ and others³² that give equal weight to each variable, the RAI is scored on a weighted scale with two statistical interactions that generate a score ranging from 0 to 81 (Table 2) calibrated to predict mortality. Although quasi-continuous RAI scores retain the maximum statistical power to predict and adjust for frailty-associated risk, clinically relevant categories for counseling can be identified by stratifying scores into four categories: robust, normal, frail, and very frail. Thresholds for each stratum are chosen such that the frail and very frail categories correspond respectively to mortality rates two and four times the overall mean mortality rate (usually, the mortality rate for the normal group). Robust thresholds are set to approximately half the overall mean mortality rate.

Figure 2:

Panels A to C represent three validated frailty assessments: physical frailty, cumulative deficit frailty, and the Risk Analysis Index (RAI). Each assessment includes five domains, which are color coded to aid comparison across tools. Physical frailty includes functional performance measures (green), weight loss and nutrition (orange), and physical activity (blue). Cumulative deficit frailty adds cognitive decline (purple), and disability (red). The RAI adds demographics and living location (pink). CHF, congestive heart failure; CKD, chronic kidney disease.

Table 2. Scoring Paradigm for the Risk Analysis Index (RAI) in Each Variant.

	RAI-C		RAI-A		RAI-VQI	RAI-ICD
	Without Cancer	With Cancer	Without Cancer	With Cancer		Without Cancer	With Cancer
Age in Years
≤ 19	0	28	0	28	0	1	39
20–24	1	29	1	29	1	3	39
25–29	4	29	4	29	4	5	39
30–34	6	30	6	30	6	6	39
35–39	8	30	8	30	8	9	40
40–44	10	31	10	31	10	10	40
45–49	12	31	12	31	12	12	40
50–54	14	32	14	32	14	14	40
55–59	16	32	16	32	16	15	40
60–64	18	33	18	33	18	17	40
65–69	20	34	20	34	20	19	40
70–74	22	34	22	34	22	21	40
75–79	24	35	24	35	24	23	40
80–84	26	35	26	35	26	25	40
85–89	28	36	28	36	28	27	40
90–94	30	36	30	36	30	28	41
95–99	32	37	32	37	30	30	41
≥ 100	34	37	34	37	30	41	41
Biological Sex (Male)	3		3		3	1
Unintentional Weight Loss	4		4		—	10
Poor Appetite	4		4		—	9
Nutrition	—		—		—	—
BMI < 20	—		—		8	—
BMI 20–34.9	—		—		0	—
BMI ≥ 35	—		—		8	—
Renal Failure	8		8		8	5
Congestive Heart Failure	5		5		5	5
Dyspnea	3		3		3	3
Other Living Setting	1		1		1	—
Functional Status	—		Without Cognitive Decline	With Cognitive Decline	—	Without Cognitive Decline	With Cognitive Decline
Independent	—		0	5	—	0	6
Partially Dependent	—		7	11	—	4	8
Totally Dependent	—		14	16	—	8	11
ADL Score	Without Cognitive Decline	With Cognitive Decline	—		—	—
0	0	5	—		—	—
1	1	6	—		—	—
2	2	6	—		—	—
3	3	7	—		—	—
4	4	8	—		—	—
5	4	8	—		—	—
6	5	9	—		—	—
7	6	10	—		—	—
8	7	11	—		—	—
9	8	11	—		—	—
10	9	12	—		—	—
11	10	13	—		—	—
12	11	13	—		—	—
13	11	14	—		—	—
14	12	15	—		—	—
15	13	15	—		—	—
16	14	16	—		—	—
Cognition (Preoperative Ranking)	—		—			—
0–1	—		—		0	—
2	—		—		2	—
3	—		—		5	—
4	—		—		10	—
5	—		—		16	—

C, clinical; A, administrative; VQI, Vascular Quality Initiative; ICD, International Classification of Diseases; BMI, body mass index; ADL, activities of daily living.

There are currently four RAI versions that have the same parameters but are applicable to varying data. The prospective RAI survey is called the RAI-C (for clinical). The RAI-A (for administrative) can be applied to variables available within the noncardiac VASQIP and American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) datasets that have similar data definitions of RAI variables. Subsequently, the RAI has been used in the Vascular Quality Initiative (RAI-VQI) and administrative billing data (RAI-ICD) using ICD codes. Although the qualifying descriptors (-C, -A, -VQI, -ICD) delineate the specific methods for RAI calculation, each scoring system represents a universal, five-domain conceptual framework (social, nutritional, physical, functional, and cognitive deficits) with associated weighted scoring. Therefore, the qualifying descriptors are not required if the RAI methodology used is clear and appropriately cited.^*

Across all RAI versions, the model is well-calibrated to predict mortality, with predicted and observed mortality within close tolerances. The RAI-A and RAI-C are calibrated to predict 180-day mortality, but the RAI-ICD is calibrated to a composite of 30-day and in-hospital mortality because administrative data is limited in its time horizon. VQI outcomes are assessed at one year, and the calibration of the RAI-VQI is currently in progress. In previously published calibration curves,⁹ the RAI model tends to overpredict mortality in the upper range of scores. However, consistent with useful screening instruments, predicted and observed mortality have a monotonic relationship, so clinicians can be confident that high RAI scores are consistent with a high mortality rate.

As shown in Table 3, overall model discrimination (c-statistic) for RAI-C, RAI-A, RAI-VQI, and RAI-ICD is 0.804,⁹ 0.842,⁹ 0.733,³³ and 0.784,²⁰ respectively. Each variant has its own specific thresholds, each selected to make assessments comparable across versions and datasets with corresponding sensitivity, specificity, positive and negative predictive values, F1, and Matthews correlation coefficient (MCC) (Table 3). Although performance may change when applied to different datasets, the RAI is, overall, highly specific and modestly sensitive, meaning that patients categorized as normal or robust are likely to experience low rates of postoperative complications. Those identified as frail deserve closer scrutiny, but our clinical recommendation emphasizes that the limited sensitivity of the RAI precludes its use as a singular determinant of surgical candidacy. Surgeons can and do operate on even the frailest patients, with 70% of very frail patients surviving two years after surgery.^{16, 28, 34}

Table 3. Risk Analysisi Index (RAI) Model Performance^*.

		Robust	Normal	Frail	Very Frail
RAI-A	2010–2014 VASQIP sample, N = 480,731
	Score Range	≤ 20	21–29	30–39	≥ 40
	Predicted 30-day Mortality, %	0.1	0.8	3.8	69.9
	Predicted 180-day Mortality, %	0.3	3.1	13.9	91.5
	Sensitivity, %	99.7	76.9	43.1	0.6
	Specificity, %	7.6	76.5	95.9	100.0
	PPV, %	3.9	10.8	27.9	79.0
	NPV, %	99.8	98.9	97.8	96.4
	F1	0.6826	0.7672	0.5850	0.0113
	MCC	0.0366	0.2667	0.1946	0.0028
RAI-C	July-December 2016 UPMC Sample, N = 8,172
	Score Range	≤ 29	30–36	37–44	≥ 45
	Predicted 180-day Mortality, %	0.6	3.6	7.7	43.8
	Sensitivity, %	98.6	68.1	32.9	0.0
	Specificity, %	22.9	77.0	94.0	99.9
	PPV, %	3.3	7.3	12.8	0.0
	NPV, %	99.8	98.9	98.1	97.4
	F1	0.7153	0.7127	0.4737	0
	MCC	0.1075	0.2255	0.1345	-0.0005
RAI-ICD	2020 National Inpatient Sample, N = 1,771,081
	Score Range	< 27	27–35	36–45	> 45
	Predicted Mortality, %	0.9	3	7.2	30.2
	Sensitivity, %	95.3	59	32.2	0.8
	Specificity, %	35.5	82	92.7	99.9
	PPV, %	2.3	14	29.7	82.4
	NPV, %	99.8	98	93.5	73.2
	F1	0.7336	0.663	0.4617	0.0155
	MCC	0.1539	0.2025	0.1247	0.0036

^*Threshold values for calculating dichotomous statistics set at the middle of the range of RAI score (for example, robust, normal, frail, and very frail) correspond to threshold values of RAI-A = 11, 25, 35, 61; RAI-C = 15, 33, 41, 63; and RAI-ICD = 14, 31, 41, 63, respectively). VASQIP, US Department of Veterans Affairs Surgical Quality Improvement Program; PPV, positive predictive value; NPV, negative predictive value; MCC, Matthews correlation coefficient; A, administrative; C, clinical; ICD, International Classification of Diseases.

Although biomarkers are sometimes important for risk assessment, investigation supplementing the RAI with serum albumin, hematocrit, and creatinine showed no clinically meaningful improvement in model performance, arguing against adding financial and time costs associated with measuring biomarkers and emphasizing the strength of frailty screening using the RAI parameters.¹⁴

Interpretation of the RAI can be challenging in the setting of a cancer diagnosis because the age^*cancer interaction can account for 37 of the possible 81 points. For this reason, and particularly when the identified cancer might be effectively treated by the surgery under consideration, it can be helpful to report a separate RAI_{(without cancer)} score that omits the cancer-related points.¹³ By considering both scores simultaneously, the clinician can immediately ascertain how the cancer diagnosis affects the RAI score and then apply their best clinical judgment with regard to the treatment choices at hand.

RAI in Context with Other Frailty Measures

Consensus regarding the definition of frailty remains elusive,^{35, 36} and apart from the 60- to 90-minute comprehensive geriatric assessment (administered by a trained geriatrician)³⁷, there is no gold standard measure of frailty. As such, available tools for measuring frailty have proliferated from various theoretical frameworks.^{38, 39} For example, some tools include (or exclude) assessment of age, functional performance (for example, gait speed or grip strength), or health care utilization (for example, prior hospitalization or nursing home use). However, recent evidence suggests that when existing frailty indices are supplemented with age and utilization to make fair comparisons, they demonstrate essentially equivalent model performance, even with only partial intersection of the sets of patients identified as frail.^{32, 40} Consequently, a recent National Institutes of Health consensus statement argued against the creation of any new frailty tools without a clearly defined need that could not be met by an existing tool.³⁵

Although a variety of frailty tools have been applied to surgical contexts, the RAI has several advantages. First, the RAI is granular across a range of 81 points, permitting calibration of cutoffs depending on population and context. For example, the prevalence of frailty among vascular surgery cohorts is substantially higher than in orthopedic cohorts.¹¹ The choice of RAI threshold can be easily shifted up or down to meet particular clinical or research needs, whereas a tool like the Johns Hopkins assessment of physical frailty²² lacks such flexibility because it has only five points in three categories (normal, pre-frail and frail).

Second, the prospective application of the RAI survey is rapid and proven feasible at scale across entire health care systems, taking as little as 30 seconds to quantify in the context of typical clinic workflows.¹⁶ The Johns Hopkins assessment of physical frailty and the Edmonton Frail Scale⁴¹ require substantially more time and dedicated equipment to measure grip strength and gait speed. The single-item Clinical Frailty Scale³¹ is likely feasible at scale, having been applied in clinical practice,⁴² but it is fundamentally a single-item gestalt assessment of frailty by a provider, which, like other single-item assessments, lacks the rigor and precision afforded by multifactor, patient-centered assessments.^43–45

Third, across 11 parameters, the RAI measures five domains of frailty (Figure 2), and the prospective survey version measures ADLs in four dimensions (Tables 1 and 2). As such, it more robustly captures the syndrome of frailty than the modified frailty index (mFI) initially developed by Velanovich and colleagues.^{46, 47} The mFI was among the first applications of frailty to surgical populations, adapted from ACS NSQIP, and was constructed based on Rock-wood’s model of deficit accumulation.³¹ The available variables were limited to those extant in ACS NSQIP, of which only 11 could be mapped to one of the 70 frailty-associated domains originally identified by Rockwood, and these were primarily measures of comorbidity rather than functional performance, leading some to criticize the mFI as being more appropriately conceptualized as a comorbidity index, like Charlson or Elixhauser, and not truly a measure of frailty.⁴⁸ The limitations of the mFI were exacerbated in 2012 when 6 of the original 11 ACS NSQIP variables were abandoned. Although attempts were made to rescue the mFI using only the remaining 5 variables of comorbidity,⁴⁹ we suggest that the mFI-5 is obsolete due to its limited theoretical framework and comparatively poor model performance^50–55, especially compared to the RAI. Callahan and colleagues criticize the RAI and mFI as not including “a sufficiently broad set of age-related deficits,”^{56(p. 1361)} recommending at least 30 items. This certainly applies to the mFI, but with ADLs assessed in four dimensions the RAI contains 14 distinct variables—which is at the upper range of the 10 to 15 items Rockwood demonstrated to be sufficient for frailty indices (with additional items providing little incremental improvement in model discrimination or calibration).⁵⁷ In addition, increasing variable capture could significantly affect workflow and data entry in a clinic setting, potentially discouraging frailty screening of large populations.

Fourth, each RAI variable maps to potential consult and intervention, solidifying the face validity of the RAI in ways that facilitate clinical adoption—for example, stimulating appetite, reversing weight loss, improving dyspnea, or supporting ADLs. Other frailty indices, such as the Clinical Frailty Scale and Hospital Frailty Risk Score (HFRS), are difficult to interpret because the deficits may not be specified³¹ or because machine learning algorithms are using nonlinear relationships and nonclinical variables.⁵⁸ Regard-less of the underlying statistical model, when tools include thousands of predictors,^{32, 58} it is challenging to intervene on individual patient risks.

Fifth, the RAI is validated in a variety of forms, applying a consistent conceptual framework across a variety of contexts, including surgical quality registries, administrative billing data, and prospective clinical surveys. This permits investigators and clinicians to ask frailty-related questions with immediate quality improvement opportunities using their own real-world data.

Finally, the RAI has proven to be an effective way to adjust multivariable models for patient-level risk, without the problems that typically limit model fit and convergence when large numbers of individual risk variables are added to the models.^{11, 59–61} In multiple cohorts and approaches, including least subset regression and a variety of machine learning techniques, the RAI is consistently identified as the single most important predictor of a variety of postoperative outcomes. These include mortality, readmission, complications, and Desirability of Outcome Ranking (DOOR) score (Figure 3).^{62, 63} This gives solid support from a clinical and statistical perspective that frailty can be considered the single biggest predictor of surgical outcomes.

Figure 3:

These graphs show the relative importance of Risk Analysis Index (RAI) compared to other predictors of post-operative outcomes according to multiple machine learning techniques for predicting (A) postoperative mortality and (B) Desirability of Outcome Ranking (DOOR) score. “Case Status” reflects urgent, emergent, or elective surgery; “Gagne Score” is a measure of comorbidity; “Specialty” denotes the surgical specialty of the operating physician. PASC, preoperative acute serious condition; DNR, do not resuscitate order; BMI, body mass index; OSS, Operative Stress Score; ADI, Area Deprivation Index.

Although not intuitively obvious, there is increasing concern that clinical algorithms may contain unrecognized bias in their performance within and between demographic groups.⁶⁴ We recently examined differences in RAI performance when stratified by Black and White racial categories, finding better accuracy among Black patients.⁶⁵ We also found that at extreme RAI values, the direction of the bias reversed. This suggests that assessment of racial bias should be attentive to the thresholds under consideration. To our knowledge, the RAI is the only frailty tool that has explored the possibility of algorithmic racial biases, and our findings suggest that the RAI is unlikely to exacerbate racial disparities.

HOW TO

Prospective Survey (RAI-C)

The RAI-C is a simple survey that can be completed either by clinicians based on elicited or documented history or by patients with or without the assistance of their caregivers or companions. It can be formatted onto a single page and has been translated into Spanish, Portuguese, and Chinese.^† Although it can be completed in real time in as little as 2 minutes, we recommend distributing the survey to patients so they can complete it as they wait to see their clinician. The patient-reported responses can then be entered into one of several available computerized tools (see below) to rapidly compute and record the score. This approach takes a median 30 seconds to complete and does not disrupt clinic work flow.¹⁶ Most importantly, a risk score is immediately available for the clinician to guide counseling and decision-making.

Four of the RAI-C survey items deserve discussion. First, biological sex is recorded and should not be confused or conflated with gender identity.⁶⁶ We leave it to the clinicians involved to ascertain this variable in a way that respects each patient, regardless of their gender identity. Second, the cancer item is intended to maximize sensitivity by asking for any cancer diagnosis not in remission; this approach will alert clinicians to any relevant diagnosis, allowing them to exercise clinical judgment as to the pertinence of the cancer diagnosis, as described above and elsewhere.¹³ Third, the item pertaining to renal failure is intended to capture any kind of renal insufficiency. Cognitive interviewing revealed that many patients were unfamiliar with their diagnosis but could identify contact with a “kidney doctor,” which is counted as evidence of renal insufficiency unless the indication was for isolated renal calculi. Finally, the item for living location asks for details about the kind and duration of nonindependent living locations, but any nonindependent living location is sufficient to accrue the points associated with this item. The additional detail can be omitted, but we recommend retention to better inform the clinician of the nature of the living location.

Administration and calculation of the RAI-C can be facilitated in a variety of ways. Paper-and-pencil application is subject to operator error in calculating the score, but we recommend this as a first approach, as it is immediately implementable, and errors can be minimized using scoring tables printed on the back of the survey. After data are obtained, hand calculations can be facilitated by the free online calculator.⁶⁷ The RAI is also available in electronic health records. As part of the Epic Clinical Program, the module can be installed without charge by contacting Epic Technical Support assigned to your institution.⁶⁸ For those working in Cerner environments, API interfaces can be developed between Cerner and a freestanding Web-based tool as implemented by the University of New Mexico, as explained in the RAI User Guide Repository.⁶⁹ The Repository also contains details sufficient to program the RAI-C in either REDCap, a free shareware supported by Vanderbilt university,^{70, 71} or Microsoft’s PowerApp. Finally, for those working in the Veterans Health Administration, the RAI is available in every instance of the Computerized Patient Record System and searchable under the template “RAI Frailty Tool.”

Addition details about RAI-C implementation are found in Appendix 1 (available in the online article), including details about test-retest performance and the reasons to avoid attempts to (1) triage the RAI-C to a more narrowly tailored population or (2) enforce unnecessary precision in response to survey questions.

Surgical Quality Registries (RAI-A and RAI-VQI)

The RAI-A and RAI-VQI are calculated based on variables present in the VASQIP, ACS NSQIP, or VQI quality registries, according to the definitions in Table 1. Computation of the RAI-A and RAI-VQI can be facilitated using code in STATA, SAS, and R. (See the RAI User Guide Repository.⁶⁹) Care must be taken to account for slight differences in variable names and categorizations between VASQIP and ACS NSQIP, particularly as it pertains to some of the variables that were abandoned by ACS NSQIP after 2012. These include variables for neoadjuvant chemo- or radiotherapy as well as variables for decreased levels of consciousness and cerebral vascular accidents with neuro-logical deficits. Extensive sensitivity analyses demonstrate that overall model performance increases when these variables are included, but their elimination does not degrade model discrimination significantly. Therefore, we recommend that these variables be included when present, but that missing values should not preclude the calculation of a valid score. (See the RAI User Guide Repository.⁶⁹ )

Investigators should recognize that VASQIP, ACS NSQIP, and VQI data are recorded at the level of the surgical case, and therefore a single unique patient may have more than one recorded case in any given sample of data. It is impossible to adjust for this duplication in the ACS NSQIP Participant User File (PUF) because all data are de-identified. However, cases are identifiable in both VASQIP and the site-specific data supplied by each participating hospital in ACS NSQIP, with rates of duplication representing approximately 18.4% in VASQIP and 8.2% across three participating ACS NSQIP sites. In the case of duplication, the presence of a duplicate RAI value constitutes a bias in favor of survival. Depending on the analytic strategy, this bias can be managed by taking the first or last RAI value, but we typically take one value at random.

The VQI includes 14 datasets capturing cases typically performed by those treating a variety of vascular pathologies, ranging from dialysis access to aortic aneurysmal repair. Each dataset includes its own disease- and procedure-specific variables, which can be mapped to the 11 parameters included in the RAI. The RAI-VQI has been validated in the carotid endarterectomy and aneurysm datasets^72–74 and extrapolated to the peripheral artery disease datasets.^{72, 75} A broader validation of the RAI in the VQI is actively in process, but it is important to note that none of the VQI datasets include domains for cognition or cancer, and these shortcomings need to be addressed in future research.

Administrative Billing Data (RAI-ICD)

The RAI-ICD is calculated based on a limited selection of 380 diagnosis codes available in the National Inpatient Sample (NIS) as defined by the ICD-10-CM (Tables 1 and 2).²⁰ Codes were selected to map directly to each RAI variable through a process of consensus methodology, with an emphasis on parsimonious selection of codes that reliably indicate the presence of each variable and only that variable. Other administrative frailty indices include as many as 25,000 codes,^{76, 77} making interpretation of resulting scores challenging. Because no ICD-10 code reliably indicates a living location other than home, this parameter was omitted from the RAI-ICD model (Table 1). In addition, three different versions of the cancer variable were constructed according to decreasing rates of five-year survival to correspond to mild, moderate, and severe cancer biology.²⁰ Model discrimination was maximized using the severe cancer variable, which is analogous to the RAI-A variable corresponding primarily to disseminated cancer. However, the moderate or mild cancer variables may better approximate responses to the RAI-C, which defines the presence of cancer more permissively. Documentation of all three cancer variables, as well as Stata code for computing each of the three RAI-ICD variants, are available in the RAI User Guide Repository.⁶⁹

Although calibrated and validated in the NIS with strong discrimination and good calibration, the RAI-ICD model performance was even better in a sample of data from UPMC. These differences are likely multifactorial. First, although they improved with ICD-10-CM over ICD-9-CM, substantial regional and institutional variations exist in commonly applied ICD codes.^78–81 Second, an unlimited number of potential ICD-CM-10 codes can be applied to each hospital admission at UPMC when compared to the maximum of 20 codes applied to the NIS. Third, the UPMC data were not limited to hospitalizations with surgical procedures, demonstrating not only validity but improved performance in a broader patient population. The RAI-ICD extends the RAI to any conceivable clinical condition, surgical or otherwise. In fact, preliminary work has explored frailty in samples of patients after coronary artery bypass grafts, a procedure not included in the VASQIP dataset used to calibrate the RAI-A. Additional work is required to explore how the RAI-ICD performs in other ICD datasets, but the possibilities are extensive.

RESULTS AND LESSONS

Supplemental Figure 1 summarizes a systematic review of peer-reviewed publications pertaining to the RAI.²⁵ In accordance with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines,⁸² one investigator [C.A.J.] systematically searched the PubMed electronic database between May 6 and May 15, 2024. Searches were conducted to find articles at the intersection of the RAI (search terms including Risk Analysis Index, Risk Assessment Index, and RAI) and frailty (search terms including frail, frailty, frailness). We also performed forward citation searches beginning with the three seminal manuscripts validating and recalibrating the RAI.^{9, 10, 28} Candidate publications were reviewed to include only those that (1) developed or validated the RAI, (2) risk adjusted for frailty with the RAI in patients either undergoing surgery or attending surgical consults, (3) compared the RAI against different frailty indices in surgical patients, or (4) investigated RAI implementation for surgical quality improvement. Articles were excluded if they presented data unrelated to the RAI or were systematic reviews of previously published data.

The final search strategy detailed in Supplemental Table 1 yielded a total of 201 unique articles, of which 121 met inclusion criteria (see Supplemental Figure 1 for details about exclusion). Each included publication was carefully reviewed to abstract the relevant RAI variant, cohort/population, outcomes, and primary findings and whether an author was part of the team that originally developed the RAI. Citations were then organized by thematic focus and sorted by publication date (Supplemental Table 2 and RAI User Guide Repository⁶⁹ ).

Out of 121 included articles, the RAI-A (n = 79, 65.3%) and RAI-C (n = 31, 25.6%) were the most common versions under study. Many studies involved multiple surgical specialties (n = 37, 30.6%); studies limited to a single surgical specialty were most commonly neurosurgery (n = 34, 28.1%) and vascular (n = 24, 19.8%). Mortality was the most common outcome (n = 74, 61.2%), followed by complications (n = 43, 35.5%), discharge destination (n = 25, 20.7%), and length of stay (n = 25, 20.7%). RAI studies have been steadily increasing (13 in 2020, 17 in 2021, 18 in 2022, 38 in 2023), with 22 articles in the first 5 months of 2024, and most written by authors unaffiliated with the RAI creation and validation group (n = 68, 56.2%). The vast majority were published in the United States, but various articles from non-U.S. countries suggest the potential for international implementation. Readers are encouraged to use the summary in Supplemental Table 2 to guide deeper reading according to interest and need.

SUMMARY AND NEXT STEPS

In summary, the RAI in its several variants constitutes a suite of tools that effectively risk stratify patients in a wide variety of clinical contexts and across different data sources. It is the most thoroughly validated and flexible assessment of surgical frailty available to date, and emerging data support its application outside surgery. The RAI is proven feasible for bedside screening at scale without disrupting clinic workflow, and the RAI-ICD holds promise for those health care systems willing to invest in automating electronic health records. The RAI is intended as a rapid test to identify patients at risk for poor surgical outcomes, to inform shared decisions between surgeons and patients. However, the RAI should never be used as a single-item metric to establish surgical candidacy because surgeons can and do operate safely on patients with extremely high RAI scores.

Future work should include deeper comparisons between the RAI variants themselves, as well as to other frailty assessments and frailty indices. Prior work has shown that although a variety of frailty indices identify the highest risk decile of patients with similar discrimination and calibration, the set of patients classified into this highest decile intersect only partially.^{28, 32} Additional work could focus on ensuring that the calibration of each variant is tuned to populations of interest to ensure that application in new populations is valid and to guard against concept and data drift where validation has already occurred. Finally, it might be possible to use the RAI components to develop separate models that predict a variety of relevant performance metrics, such as readmission, discharge to nursing homes, postoperative complications, and long-term loss of independence. Accurate estimates of these events could further inform perioperative shared decision-making between surgeon and patient. Data sufficient for these analyses are rapidly accruing as the RAI–based Surgical Pause³⁰ is diffusing rapidly across the Veterans Health Administration and a growing number of private sector hospitals, with The Joint Commission and National Quality Forum recognizing the Surgical Pause as the 2023 John M. Eisenberg Award for Innovation in Patient Safety and Quality at a National Level.³⁰

Supplementary Material

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jcjq.2024.12.005.