Validation of the Risk Analysis Index for Evaluating Frailty in Ambulatory Patients

STRUCTURED ABSTRACT

Background

Frailty is a marker of dependency, disability, hospitalization and mortality in community dwelling older adults. However, existing tools for measuring frailty are too cumbersome for rapid point-of-care assessment. The Risk Analysis Index (RAI) of frailty is validated in surgical populations, but its performance outside surgical populations is unknown.

Objective

Validate the RAI in ambulatory patients.

Design, Setting, and Participants

Observational cohort study of outpatient surgical clinics within the University of Pittsburgh Medical Center Healthcare System between July 1 and December 31, 2016. Frailty was assessed using the RAI. Current Procedural Terminology codes following RAI assessment identified patients with and without minor office-based procedures (e.g., joint injection, laryngoscopy).

Main Outcomes and Measures

All cause 1-year mortality assessed by stratified Cox proportional hazard models.

Results

Of 28,059 patients, 13,861 were matched to a minor, office-based procedure and 14,198 did not undergo any procedure. The mean (SD) age was 56.7 (17.2) years; women constituted 15,797 (56.3%) of the cohort. Median time (IQR 25th-75th percentile) to measure RAI was of 30 (22–47) seconds. Mortality among the frail was 2 to 5 times that of patients with normal RAI scores. For example, the hazard ration for frail ambulatory patients without a minor procedure was 3.69 (95% CI 2.51–5.41), corresponding to 30-, 180- and 365-day mortality rates of 2.9%, 11.2% and 17.4%, respectively compared to 0.3%, 2.3% and 4.0% among patients with normal RAI scores. Discrimination of mortality (overall, and censored at 30-, 180- and 365 days) was excellent, ranging from c=0.838 (0.773–0.902) for 30-day mortality after minor procedurs to c=0.909 (0.855–0.964) without a procedure.

Conclusion

RAI is a valid, easily administered tool for point-of-care frailty assessment in ambulatory populations that may help clinicians and patients make better informed decisions about care choices—especially among patients considered high risk with a potentially limited life span.

Introduction

Frailty is a syndrome of decreased physiologic reserve making patients vulnerable to adverse health outcomes such as disability, dependency, falls, the need for long-term care, and mortality.^1–6 Although a precise definition eludes consensus, an international expert panel convened in 2011 agreed frailty is a multidimensional construct of 6 domains (physical performance, gait speed, mobility, nutritional status, mental health, and cognition) that together indicate increased risk of death, disability, and institutionalization⁷. In addition, studies evaluating trajectories of disability in the last year of life found the most common condition leading to death is frailty^8,9. It is therefore critical to equip clinicians with frailty assessment tools that can inform real-time, patient-centered decision making.

As recently reviewed by Walston et al,¹⁰ the variety of tools for measuring frailty fall primarily into either the deficit accumulation model^11–13 or the physical frailty model^14–16. However, few of the existing tools have proved feasible for rapid, real-time, point-of-care frailty screening in busy clinical practices comprised of predominantly robust (i.e.: non-frail) patients. Our previous work among patients scheduled for major surgery demonstrates that the Risk Analysis Index (RAI) is an effective and efficient risk stratification tool that takes only 30 seconds to measure and can inform clinical decisions.¹⁷ We have also shown that these frail patients suffer astonishingly high morbidity and mortality after even low stress surgical procedures,¹⁸ suggesting that routine frailty screening is imperative not only for patients considering major surgery, but also for those considering lower stress interventions typically performed by non-surgeons. Furthermore, given the increasing demands on primary care physicians, the benefits of earlier and more consistent palliative care in the last months of life^19–21 suggest that these resources might be more efficiently deployed if clinicians could identify the small portion of their practice at greatest risk for medium-term mortality who might benefit most from efforts to ensure that care in the last months to year of life is consistent with patient values.^22–25

The RAI might meet this need for a rapid, point-of-care frailty assessment in primary care and medical sub-specialty settings, but its performance outside surgical populations is unknown. The RAI can be calculated from either a brief patient-facing survey instrument or from variables commonly collected in large clinical registries^26–28. While the RAI was initially developed using patients from a single Veterans Affairs (VA) hospital, it has been subsequently recalibrated and externally validated using nationally representative patient populations within the VA and the private sector with excellent performance characteristics²⁸. Importantly, since prospective frailty assessment using the RAI takes under 1 minute to administer,^17,27 it is an appealing option for implementation as a system-wide frailty screening tool in a wider variety of clinical settings. Thus, the purpose of this investigation is to evaluate the RAI’s ability to predict near- and medium-term mortality in ambulatory patients with or without minor office-based procedures. Our hypothesis was that the RAI would effectively risk stratify these patients with similar discrimination to that observed in patients obtaining major surgery.

Methods

As described elsewhere,¹⁷ and as part of a system-wide quality improvement (QI) project, surgeons from the University of Pittsburgh Medical Center (UPMC) recently started measuring the RAI on all new patients presenting to outpatient surgical clinics, regardless of whether or not they were eventually scheduled for surgery. Clinic personnel used patient responses to a paper version of the RAI questionnaire to complete an online tool programmed in REDCap²⁹ that computed the RAI score, clarifying responses as needed by direct patient interview. RAI scores were computed according to the revised scoring system published in our previous work²⁸ and is derived using the following variables: age, sex, loss of appetite, unintentional weight loss, diagnosis of cancer (a composite of disseminated cancer, preoperative chemotherapy, and preoperative radiation therapy), renal failure, congestive heart failure, shortness of breath, chronic care facility status, presence of cognitive deterioration, and functional status. Most RAI scores were computed by medical assistants, but depending on clinic flow, registered nurses, advanced practice providers or physicians computed RAI scores for some patients. Some patients visited multiple outpatient clinics, yielding multiple RAI values while other clinics calculated the RAI on multiple visits based on revised responses to survey items. Thus, when any individual patient had multiple RAI values, we retained one assessment selected at random. After securing approval from the UPMC Quality Improvement Review Committee (QRC#986), RAI values were downloaded and linked to other parts of the electronic medical record (EMR).

We queried the EMR for Common Procedural Terminology (CPT) codes performed within 90 days of each RAI assessment. CPT codes representing a major surgical procedure requiring general anesthesia, conscious sedation, or neuraxial blockade were not included in this analysis, but because only a minority of patients seen in surgical clinics need or go on to have major surgery, our sample included a large number of ambulatory patients who either had a minor office-based procedure (Table 1) or were only seen in consultation. Our analysis of major surgical procedures is reported elsewhere,¹⁷ allowing us in this study to focus on those ambulatory patients who either had a minor, office-based procedure or no procedure at all. Our intent was not to compare RAI within these two cohorts (i.e.: those with and without office-based procedurse), but rather to evaluate whether the RAI could provide useful information in the ambulatory setting.

Table 1.

Top Ten Most Common Procedures among the Minor Procedure cohort

CPT Rank	CPT Code(s)	CPT Description	N
1	20610, 20600, 20605	Arthrocentesis, aspiration and/or injection of small, intermediate or major joints	1,916
2	31575, 31579	Laryngoscopy +/− stroboscopy	1,344
3	69210	Removal impacted cerumen requiring instrumentation	1,106
4	51798	Measurement of post-voiding residual urine by ultrasound	1,060
5	31231	Diagnostic nasal endoscopy	915
6	36415	Venipuncture	519
7	52000	Cystourethroscopy	334
8	51701	Insertion of non-indwelling bladder catheter	192
9	43239	Esophagogastroduodenoscopy	150
10	55250	Vasectomy	135

In the event that more than one CPT code was noted within 90 days of RAI assessment, we selected the procedure dated closest to the RAI assessment. Among this sample, we linked each record to the patient’s vital status and date of death, if deceased, using a proprietary file maintained by UPMC that combines the social security death index with other sources to render the best available record of vital status. We used these death dates to calculate survival length from the date of procedure (for those matched to a CPT code) or date of RAI assessment (for those with no identified CPT code). Linked data also included the department of the surgeon in whose clinic the patient’s RAI was assessed. After inspecting the data for out of range and missing values, we generated summary statistics for key demographic data along with the individual response items used to calculate the RAI scores. Time stamps within the online tool used to calculate the RAI permitted comparison of the time required to compute the RAI. The Kaplan-Meier method and log-rank test were used to compare survival distributions. Finally, we built stratified Cox proportional hazard models to evaluate the association between RAI scores and risk of death. Model discrimination was assessed with Harrell’s C statistic, a nonparametric technique for estimating C in censored survival models as a weighted area under an “incident/dynamic” Receiver Operating Characteristic Curve with weights depending on the study-specific censoring distribution. Robust standard errors were used to account for clustering of patients within surgical departments. Because the RAI is a composite variable that incorporates data from many of the variables that would typically be included as model covariates (e.g., age, sex, and some variables indicating comorbidity), we chose, a priori, to report unadjusted models to avoid issues related to collinearity. However, we did perform sensitivity analyses in which we included age and sex as model covariates. As described in our previous work, RAI scores were stratified into five categories: Robust (RAI< 29), Normal (RAI 30–36), Pre-Frail (RAI 37–44), Frail (45–52) and Very Frail (RAI ≥ 53)^17,28. All analyses were conducted using Stata 15.1 (StataCorp, LLC), all p-values were 2-sided, and significance was defined as p<0.05.

Results

From July 1 to December 31, 2016, the RAI was assessed in 28,059 ambulatory patients who did not have major surgery after presenting to surgical clinics across five hospitals within the UPMC Healthcare System. Of these patients, 13,861 were matched to a minor, office-based procedure and 14,198 did not undergo any procedure. The ten most common minor procedures are listed in Table 1 according to decreasing prevalence and include arthrocentesis, cerumen disimpaction and various endoscopic procedures. Demographic characteristics of the cohort are detailed in the Supplementary Table S1. The mean (S.D.) age of the overall cohort was 56.7 (17.2) years, there were 15,797 women constituting 56.3% of the total cohort. 23,330 (83.1%) patients were white with black patients comprising 10.6% of the cohort. A larger proportion of patients in our surgical cohort (12.1%) had RAI scores >36 compared to patients in the non-surgical cohort (9%). The mean (S.D.) RAI in the non-surgical and minor procedure cohort were 20.4 (10.7) and 22.6 (11.0) respectively. In the surgical cohort 0.4% and 2.1% of the patients died within 30-days and 180-days from the procedure compared to 0.2% and 1.0% in the non-surgical cohort. Overall, the mean (S.D.) time required to collect and compute the RAI score was 43.7 (49.9) seconds with a median (IQR 25th-75th percentile) of 30 (22–47) seconds.

The RAI-associated survival-risk is represented for each of the cohorts in the survival curves plotted in Figure 1 (A & B). In each of the cohorts, the survival curves demonstrated worse survival with rising RAI scores (all p<0.001). Hazard ratios and mortality rates for each strata of RAI score are reported in Table 2. The models were stratified by 18 hospital-specific departments (Table 3) and demonstrate a consistent rise in mortality with each strata of RAI score such that the hazard ratios for pre-frail, frail and very frail patients were 2 to 5 times greater than those with normal RAI scores. For example, the hazard ratio for frail ambulatory patients without a minor procedure was 3.69 (95% CI 2.51–5.41), corresponding to 30-, 180- and 365-day mortality rates of 2.9%, 11.2% and 17.4%, respectively, compared to 0.3%, 2.3% and 4.0% among patients with normal RAI scores, respectively.

Figure 1.

Mortality stratified by Risk Analysis Index (RAI) score. A, Ambulatory patients without minor procedure. B, Ambulatory patients with minor procedure.

Table 2:

Mortality Hazards and Rates Stratified by RAI Score

		Ambulatory Patients without Minor Procedure				Ambulatory Patients with Minor Procedure
		Hazard Ratio	30-Day Mortality	180-Day Mortality	365-Day Mortality	Hazard Ratio	30-Day Mortality	180-Day Mortality	365-Day Mortality
		(95%CI)	N (%)	N (%)	N (%)	(95%CI)	N (%)	N (%)	N (%)
RAI	≤ 29 Robust	0.16 (0.11, 0.23)	4 (< 0.1%)	25 (0.2%)	62 (0.5%)	0.19 (0.14, 0.25)	16 (0.2%)	59 (0.6%)	107 (1.0%)
	30–36 Normal	ref	4 (0.3%)	32 (2.3%)	56 (4.0%)	ref	7 (0.4%)	58 (3.3%)	108 (6.2%)
	37–44 Pre-Frail	1.87 (1.29, 2.69)	5 (0.5%)	43 (4.6%)	84 (8.9%)	1.87 (1.48, 2.34)	20 (1.7%)	88 (7.6%)	146 (12.7%)
	45–52 Frail	3.69 (2.51, 5.41)	7 (2.9%)	27 (11.2%)	42 (17.4%)	2.76 (2.31, 3.30)	11 (2.8%)	53 (13.3%)	82 (20.6%)
	≥ 53 Very Frail	5.89 (3.96, 8.78)	9 (9.3%)	20 (20.6%)	27 (27.8%)	4.83 (3.44, 6.79)	8 (6.5%)	31 (25.0%)	38 (30.6%)
Harrell’s C		0.859 (0.830, 0.888)	0.909 (0.855, 0.964)	0.883 (0.852, 0.914)	0.872 (0.842, 0.903)	0.841 (0.817, 0.866)	0.838 (0.773, 0.902)	0.858 (0.826, 0.891)	0.848 (0.818, 0.879)

Note: Values reported from stratified Cox proportional hazards models with robust standard errors clustered on hospital-specific surgical departments (e.g., general, cardiac, thoracic, orthopedic, vascular, etc). Mortality rates reported on available data after censoring for sufficient follow up. C statistics reported for models using RAI as a continuous variable. Hazard ratios reported from models using binned RAI values.

Table 3:

Hospital-Specific Department Groups Use to Stratify Hazard Models

Hospital-Specific Department Group	Ambulatory Patients without Minor Procedure	Ambulatory Patients with Minor Procedure	Overall
Orthopaedic Surgery - Not Spine	3,489	3,167	6,656
Otolaryngology	2,044	3,966	6,010
Urological Surgery	2,073	2,612	4,685
Neurological Surgery	1,465	641	2,106
Surgical Oncology (Solid Tumor)	930	924	1,854
Vascular Surgery	1,118	395	1,513
Gyencological Surgery	497	643	1,140
Orthopaedic Surgery - Spine Surgery	618	251	869
Plastic Surgery	453	233	686
General Surgery- St. Margaret Hospital	178	188	366
Non-Cardiac Thoracic Surgery	161	201	362
Bariatric Surgery	249	89	338
General Surgery/Trauma-Mercy Hospital	216	121	337
Hepatobiliary Surgery	163	165	328
Cardiac Surgery	225	70	295
General Surgery - McKeesport Hospital	150	105	255
General Surgery/Trauma-Presbyterian Hospital	103	47	150
Transplant Surgery	66	43	109

Discrimination of mortality (overall, and censored at 30-, 180- and 365 days) in each cohort was excellent ranging from c=0.838 to c=0.909, depending on the cohort and the outcome (Table 2). For example, for the ambulatory patients without a minor procedure, Harrell’s c-statistics (95% C.I.) for 30- and 180- and 365-day day mortality were 0.909 (0.855–0.964), 0.883 (0.852–0.914) and 0.872 (0.842–0.903), respectively. Sensitivity analyses that included age and sex in the model along with RAI rendered essentially identical c-statistics, differing no more than 0.003. However, after isolating the effect of age (HR 1.02 [1.01–1.02] and 1.02 [1.01–1.03] with and without minor procedures, respectively), the hazard ratios associated with RAI were slightly lower. For example, among the cohort without a minor procedure, the hazards associated with pre-frail, frail and very frail patients were 1.64 (1.11–2.42), 2.98 (2.04–4.36), and 4.50 (2.88–7.05), respectively.

Discussion

This investigation was designed to determine the RAI’s ability to predict near- and medium-term mortality in ambulatory patients with and without minor office-based procedures. We found that the RAI accurately discriminated between frail and non-frail patients, suggesting it can be used to effectively to risk-stratify patients in both cohorts. Our purpose in this study was not to compare patients who did or did not undergo an office-based procedure, but rather to show that the RAI is valid in every kind of setting. These data validate the RAI as an effective frailty screening tool not only for patients undergoing major surgery, but importantly also in patients undergoing minor office-based procedure or seen for face-to-face consultation in ambulatory settings.

The RAI’s predictive power is similar to other published frailty indices such as the Edmonton Frailty Scale (EFS)³⁰, the Fried Frailty Phenotype (FFP)¹⁴ or the Canadian Study of Health and Aging (CSHA) frailty index¹². However, these indices have been used primarily in either retrospective studies (e.g., CSHA) or research settings that afford the luxury of time and resources to capture the physical performance metrics on which they are based (e.g., the EFS and FFP require grip dynamometers, walking tracks and stop watches to measure grip strength and gait speed). As such, none of the available tools have proven feasible for universal point-of-care testing of predominantly robust populations to inform real-time clinical decision making. They are also validated primarily in hospitalized populations.^30–32

By contrast, the RAI was developed to minimize the burden of administration: It requires no special equipment and can be calculated in less than 60 seconds without disrupting the workflow of busy outpatient clinics. As such the RAI is intended to provide point-of-care risk stratification to aid clinical decisions in real time, and it is the only tool proven feasible for sytem-wide implementation across entire health care systems.^17,26,27 Our previous work demonstrated its feasibility and validity in predicting outcomes after surgery,^18,26,28 but because there is also a need for risk stratification in ambulatory patients, we designed this study to show that the RAI is effective in risk stratifying patients typical of ambulatory outpatient clinics, contexts and settings in which major surgical procedures are not performed. Additional validation among primary care or specifically geriatric populations will be useful to further demonstrate the utility of the RAI for frailty screening, but given the consistency of the findings from our prior work with those from this study, we find little reason to suspect that generalizability to these populations should be a concern.

Our study presents data across the entire spectrum of age for three reasons. First, one critical advantage of frailty screening is its focus on global function (e.g., dyspnea rather than emphysema) which acknowledges that older individuals can be robust and younger patients can be rendered frail by the burden of their disease, comorbidity or lifestyle. For example, in our prior work, we have found that one in five frail patients undergoing surgery is under the age of 55 years.³³ Although comparatively rare, our approach has identified younger patients at increased risk by virtue of their frailty, and among all ages, the RAI effectively identifies substantial frailty-associated risk at “pre-clinical” levels that are not immediately apparent to either the patient or clinician. Second, to the degree that quality improvement lives or dies by the feasibility and simplicity of the proposed intervention, we thought it would be easier to screen every patient rather than adding the cognitive load of stratifying the implementation procedures according to age. Given that it took only 30 seconds per patient, we believe our intuition was proved accurate, and in fact, as other healthcare systems have consulted with us in implementing RAI screening, we continue to recommend that they screen everyone regardless of age. Finally, our previous work recalibrating the RAI²⁸ demonstrates that model performance improves substantially when age is included as a continuous variable as opposed to categorizing ages above or below 65 as in the tool from which the RAI was adapted.³⁴

We elected not to control for age or sex in our Cox models to avoid problems with collinearity because these variables are included as part of the composite measure that is the RAI. We recognize that this approach is different from some models of frailty that attempt to isolate the syndrome from these demographic variables.^14,30 However, with the RAI, we are not attempting to define a gold standard for frailty, but in line with the recent recommendation to move beyond the enduring debate about frailty definitions,¹⁰ our purpose was to design a parsimonious tool to risk stratify patients and inform clinical decisions in real time at the point of care. The inclusion of age and sex adds significant predictive power and may explain why the c-statistics reported here are marginally higher than those reported for tools like the FFP or EFS.

The RAI was implemented at our institution as part of a quality improvement project. However, the data reported here are derived from the first 6 months of our frailty screening program where our only goal was to socialize the RAI and reach a benchmark of RAI assessment in at least 80% of all patients newly presenting to outpatient surgical clinics. Details of this implementation are described elsewhere,¹⁷ and although surgeons were clearly aware of the RAI score during this phase of the project, it is unlikely that such knowledge changed the care of the patients reported here for several reasons: First, academic detailing on the RAI and an explicit request to modify the care of frail patients did not occur until 6 months after this cohort was closed. Second, when we did ask surgeons to change treatment, the focus was exclusively on patients scheduled for major surgery, and thus the patients reported here were deliberately excluded from our efforts to incentivize improved quality.

When designing our frailty screening initiative, we considered building a tool that would automatically calculate frailty based on data in the electronic record. Such an automated tool would be welcome and observational studies suggest this might be possible,^35,36 but to our knowledge, no tool has yet been effectively deployed to provide real time, automated assessment. This capacity will almost certainly become available in the future, but any such tool would depend on data already in the electronic record with missing data points in at least 30% of cases.³⁵ Thus we chose feasibility over automaticity and mandated the use of this simple survey instrument. Concerns about employee burden were quickly assuaged given that it took a median 30 seconds to complete, and with more than 350,000 assessments to date, the RAI is poised to inform care plans at our center and is being implemented at a variety of hospital systems across the country. Until automated frailty assessments are widely available and reliably implemented, we commend the RAI as an immediately implementable tool for system-wide frailty screening.

Frailty assessment is important because frailty is associated with poor outcomes, including falls, hospitalization, disability, placement in a nursing home, as well as mortality in community dwelling adults, critically ill patients in intensive care units and patients undergoing surgical procedures^{2–6,13,33,37–41}. As such, it is vitally important for clinicians to ascertain patient frailty in order to inform shared decisions in the course of medical care. This has been well demonstrated in patients with advanced cancer where geriatric assessment of function, comorbidity, falls, depression, cognition and nutrition has been shown to improve outcomes and inform decisions for chemotherapy.⁴² In some cases, it may be possible to reverse certain aspects of frailty through interventions aimed at increasing strength, balance and overall physiologic reserve. For example, the Ambulatory Geriatric Assessment- Frailty Intervention Trial (Age-FIT) used the Comprehensive Geriatric Assessment (CGA) to screen for frailty and deploy cost-neutral interventions that resulted in longer survival and fewer days in the hospital^43–45. In other cases, and based on data that demonstrate patients’ priorities shift in the setting of life-limiting illness^46–49, a diagnosis of frailty can trigger earlier implementation of palliative care measures that result in significant improvements in quality of life, decreased healthcare expenditures, and increased satisfaction with care^19,25,50. However, the critical first step in any of these interventions is frailty assessment, a task that has heretofore proven difficult to incorporate into the routine workflow of entire healthcare systems.

Frailty assessment is also relevant from the perspective of health economics because approximately 13% of the $1.6 trillion spent on personal health care costs in the United States is devoted to the care of individuals in their last year of life^22,23. Patterns of functional decline prior to death are thought to follow a limited range of trajectories in the final year of life with frail individuals having a steady decline in function over time^24,51. Further evaluation of the decline in physical function of these groups before death found that frail decedents are relatively more disabled in the final year and especially dependent during the last month⁵². Hence, if a screening tool like the RAI were widely implemented to assess frailty across a health system in all ambulatory clinics, there may be opportunity to allocate resources based on frailty in order to improve outcomes and decrease health care costs. For example, one recent study demonstrated that a higher RAI score was associated with increased cost of care and net hospital income in frail patients undergoing elective surgical procedures, leading to frailty-sensitive care pathways aimed at mitigating some of these costs.⁵³

Our study has notable limitations. First, while it demonstrates the feasibility of frailty screening across a large health system, it does not provide any information about the degree to which frailty may be modifiable; future work will explore the value of system-wide implementation of interventions to address frailty. Second, our work appears to support the use of the RAI in ambulatory outpatient settings, but our data were derived from patients seen in surgical clinics and thus generalizability to primary care or geriatric settings merits consideration. Third, the RAI relies predominantly on self report without objective biomarkers; as such the RAI signal likely includes significant noise. However, this limitation would only decrease model discrimination, and despite the noise, the RAI demonstrates excellent discrimination. Finally, the calculation and presentation of the RAI score may have changed the care rendered to frail patients, thus biasing the findings presented here. However, as described above, we believe this to be a rare event.

In summary, the RAI is a simple and easily administered frailty assessment tool that can be used in ambulatory patient populations to deliver reliable, real-time, point-of-care risk stratification. By risk stratifying patients, the RAI can help identify those who might benefit from interventions aimed at improving outcomes by mitigating or reversing some of the physiological changes associated with frailty. Additionally, the RAI can help clinicians, patients and their caregivers locate patients on the overall trajectory of life, thus making possible more informed decisions about interventions and care choices among high risk patients with potentially limited life spans.

Impact Statement.

We certify that this work is a novel clinical investigation. It is unique in the eminent ease of implementing the RAI—a tool that we show takes just 30 seconds to provide the same predictive power as more intensive frailty assessments that rely on functional performance measures.