Utility of Functional Status for Classifying Community Versus Institutional Discharges After Inpatient Rehabilitation for Stroke

Abstract

Objective

To evaluate the ability of patient functional status to differentiate between community and institutional discharges after rehabilitation for stroke.

Design

Retrospective cross-sectional design.

Setting

Inpatient rehabilitation facilities contributing to the Uniform Data System for Medical Rehabilitation.

Participants

Patients (N = 157,066) receiving inpatient rehabilitation for stroke from 2006 and 2007.

Interventions

Not applicable.

Main Outcome Measure

Discharge FIM rating and discharge setting (community vs institutional).

Results

Approximately 71% of the sample was discharged to the community. Receiver operating characteristic curve analyses revealed that FIM total performed as well as or better than FIM motor and FIM cognition subscales in differentiating discharge settings. Area under the curve for FIM total was .85, indicating very good ability to identify persons discharged to the community. A FIM total rating of 78 was identified as the optimal cut point for distinguishing between positive (community) and negative (institution) tests. This cut point yielded balanced sensitivity and specificity (both=.77).

Conclusions

Discharge planning is complex, involving many factors. Identifying a functional threshold for classifying discharge settings can provide important information to assist in this process. Additional research is needed to determine if the risks and benefits of classification errors justify shifting the cut point to weight either sensitivity or specificity of FIM ratings.

Functional status at admission is instrumental in predicting resource needs and determining reimbursement under the prospective payment system for inpatient rehabilitation. Patients are assigned to a case-mix group based on their primary impairment category and admission motor functional rating. Age and admission cognitive functional rating are additional criteria for select case-mix groups within certain impairment groups.¹ Intuitively, functional status should be similarly important to the timing and setting of discharge from inpatient rehabilitation. However, there is little published information describing the influence of discharge functional status on these decisions. This lack of research may be in part because discharge disposition is a complex decision involving many factors that are difficult to quantify and are beyond the scope of rehabilitative care; for example, socioeconomic status, family and social supports, prior living situation, and home physical environment.^2-5

ROC curves are commonly used in biomedical research to assess the discriminative abilities of potential diagnostic measures. Examples range from determining the effectiveness of the tuberculin skin test to differentiate between tuberculosis positive and negative cases⁶ to evaluating the utility of a clinical prediction rule to diagnose venous thromboembolism.⁷ The ROC curve plots the sensitivity (true positive rate) as a function of 1 − specificity (false-positive rate) at each level of the diagnostic variable.⁸ Essentially, the ROC curve displays how well diagnostic variable scores can distinguish between persons in dichotomous groups; for example disease versus no disease and community discharge versus institutionalization. The AUC, which summarizes the overall discriminative ability of the measure, is the most familiar ROC curve statistic.⁹ From a clinical application perspective, however, it is also important to identify the specific cut point within the distribution of diagnostic variable scores that differentiates a “positive” test from a “negative” test. The utility of the cut point is assessed in terms of the sensitivity and specificity of this threshold value for correctly classifying positive and negative cases, respectively.⁸

There are 2 well-known methods for identifying the “optimal” cutoff score. One method finds the point on the ROC curve closest to perfect classification: (0,1) on the x,y axes. The other method finds the point on the ROC curve farthest from chance classification: Youden Index.^8,10 The term optimal, therefore, refers to the cut point that maximizes correct classifications and/or minimizes incorrect classifications. Without definitive knowledge on the costs of misclassifying cases (either false positives or false negatives), the classification of positive cases and negative cases is considered equally important. Thus, the optimal cut point maximizes the overall rate of correct classifications (sensitivity+specificity) and minimizes the overall rate of misclassifications ([1 − sensitivity] + [1 − specificity]).¹⁰

Patients with stroke are an ideal population to investigate the effectiveness of functional status in discriminating between persons discharged to the community versus to an institutional setting. Stroke is a leading cause of disability among adults¹¹ and the most frequent impairment category in inpatient rehabilitation.¹² In addition, patients with stroke often experience long-term functional deficits in both motor and cognitive domains.

The purpose of this study was to evaluate the ability of functional status to differentiate between patients discharged to the community versus those discharged to an institutional setting after inpatient rehabilitation for stroke. Objectives included (1) comparing the discriminative abilities of the motor subscale, cognitive subscale, and total functional scale scores and (2) identifying the cut point in functional scores that optimizes success in terms of total correct and incorrect classification rates.

Methods

Data Source

Data were obtained from the UDSMR. The UDSMR is the largest nongovernmental national registry for inpatient medical rehabilitation information in the United States.¹³ The UDSMR includes information for more than 13 million patient records from 1987 to 2009 for approximately 1400 rehabilitation hospitals or facilities.¹³ The CMS has published several reports examining the representativeness and validity of the UDSMR database.^14,15 The University's Institutional Review Board on Human Protection and Research Ethics approved the study.

Study Sample

The study sample included adults (aged ≥18y) with stroke (International Classification of Disease, version 9 codes 430.0–434.9, 436.0–438.9) who were living in the community prior to stroke onset and were discharged from inpatient rehabilitation in 2006 and 2007. There were 172,840 patients in the database who met these criteria. Patients were excluded if this was not their initial rehabilitation admission (n=9923) or if they experienced atypical lengths of stay, including less than 3 days or outside of 3 SDs of logarithm length of stay (n=5851). The final sample included 157,066 patients, which represents approximately 91% of the original sample.

Functional Assessment

Patient functional status was assessed with the FIM instrument, the standardized inpatient rehabilitation measure for disability and burden of care.^14,15 Over the past 25 years the FIM instrument has been used to document functional status at admission and discharge from inpatient medical rehabilitation.^16-18 In 2002, FIM instrument items were incorporated into the inpatient rehabilitation facility–patient assessment instrument.¹ Admission FIM ratings are used to formulate Medicare reimbursements under the prospective payment system.

The FIM instrument is administered within 3 days of admission and 3 days of discharge. It contains 18 items covering 6 functional domains: self-care, sphincter control, transfers, locomotion, communication, and social cognition. The first 4 domains constitute the motor subscale and the latter 2 the cognitive subscale. All items on the FIM instrument are measured on a 7-point scale ranging from 1 (total assistance) to 7 (complete independence): range, 18 to 126. The reliability and validity of the FIM instrument have been studied extensively in patients with stroke,¹⁹ as well as with other common inpatient rehabilitation impairments.²⁰

Outcome

Discharge setting was used as the dependent variable. The inpatient rehabilitation facility–patient assessment instrument includes 14 options for discharge setting. For this study, discharge setting was dichotomized into 2 groups: those who were discharged to the community and those discharged to an institutional setting. Community is defined as home, board and care, and transitional or assisted living. Institution includes intermediate care, skilled nursing facility, acute unit own facility, acute unit another facility, chronic hospital, rehabilitation facility, alternate level of care, subacute setting, and other.

Data Analysis

Patient characteristics were stratified by discharge setting and examined with descriptive statistics: independent t tests for numerical variables and chi-square tests for categorical variables.

Three ROC curves were developed to display the potentials for discharge FIM motor, FIM cognitive, and FIM total scores to differentiate between patients discharged to the community versus those discharged to institutional care. The ROC curves show the relationships between true- and false-positive rates for classifying patients into the 2 discharge groups at all levels of the functional scales; in other words, they plot sensitivity along the y axis and 1 − specificity along the x axis across the entire range of functional scores. Both axes range from 0 to 1. The diagonal line (from 0,0 to 1,1) represents the “chance” or no discrimination line. AUCs were computed to assess the relative effectiveness of the 3 functional scales in discriminating between discharge groups. AUC can range from 0.5 (no discrimination) to 1.0 (perfect discrimination). As a general rule, an AUC greater than 0.9 is high, 0.7 to 0.9 is moderate, 0.5 to 0.7 is low, and 0.5 is none.⁸

Next, we used 2 different methods to identify the cut point within the distribution of FIM total ratings that best discriminates between patients discharged to the community versus to an institutional setting. This cut point can be considered the threshold value that differentiates a positive test (community=1) from a negative test (institution=0). The first method, the closest-to-(0,1) criterion, identifies the point on the ROC curve that is closest to the top left corner (0,1) of the figure. The ROC curve for a point that perfectly classifies both positive and negative cases would be a vertical line from (0,0) to (0,1) joined by a horizontal line from (0,1) to (1,1) with an AUC of 1. The closest-to-(0,1) criterion is based on the logic that the point on a less-than-perfect ROC curve closest to the perfect standard will provide the best differentiation of cases. The second method, the Youden Index, identifies the point on the ROC curve that is the greatest vertical distance from the chance line. The ROC curve for a cut point that provides no discrimination ability would fall on the diagonal from (0,0) to (1,1) resulting in a 50-50 chance of correctly classifying positive or negative cases. The Youden Index is based on the logic that the point farthest from chance will provide the most discriminative ability. Detailed descriptions of these 2 methods are available in recent articles by Akobeng⁸ and by Perkins and Schisterman.¹⁰ The formulas for identifying the respective cut points are as follows:

$$\text{closest}-\text{to}-(0,1)=\text{minimum}[{(1-\text{sensitivity})}^2+{(1-\text{specificity})}^2]$$

$$\text{Youden Index}=\text{maximum}(\text{sensitivity}+\text{specificity}-1)$$

Lastly, we plotted the FIM total probability density functions for both community and institutional discharge groups based on a method described by Schisterman et al.²¹ This process allowed us to observe the distributions of group FIM ratings relative to each other and relative to the newly identified cut point. In addition, it provided a model to visualize the tradeoff between sensitivity and specificity that would occur with adjustments to the cut point.

Results

Approximately 71% of the sample was discharged to the community after inpatient rehabilitation for stroke. Table 1 shows patient characteristics stratified by discharge setting. Compared with patients discharged to an institutional setting, those returning to the community were younger, less likely to be women, less likely to be white, and more likely to be married. In addition, community-discharged patients experienced shorter lengths of stay and demonstrated higher functioning at discharge than patients discharged to an institutional setting.

Table 1. Patient Characteristics by Discharge Setting.

		Discharge Setting
Characteristics	Total	Institution	Community
n	157,066	45,626	111,440
Age, y	70.5±13.8	73.9±12.6	69.1±14.0
Women	51.8	53.4	51.2
Race/ethnicity
White	71.6	75.4	70.0
Black	15.2	13.4	15.9
Hispanic	5.6	4.6	6.0
Asian	3.9	3.1	4.2
Other/unspecified	3.7	3.5	3.8
Married	49.6	42.3	52.5
Length of stay, d	17.1±9.5	18.8±10.3	16.5±9.1
Discharge FIM motor	56.3±18.6	39.6±16.7	63.1±14.5
Discharge FIM cognition	24.2±7.3	20.0±7.7	26.0±6.5
Discharge FIM total	80.5±23.6	59.7±21.5	89.1±18.5
Discharge FIM total≥78	61.4	22.9	77.1

NOTE. Values are mean ± SD or percentages. P<.001 for all univariate comparisons.

ROC curves and corresponding AUCs for discharge FIM total, FIM motor, and FIM cognition are shown in figure 1. FIM total and FIM motor demonstrated moderately high discriminative abilities. FIM cognition was less effective in differentiating between community and institutional discharges.

Fig 1.

ROC curves comparing the ability of discharge FIM total, FIM motor, and FIM cognition to classify discharge setting after inpatient rehabilitation for stroke. AUC quantifies the relative strength of the test to distinguish between patients discharged to the community versus an institution.

The cut point discerning positive tests from negative tests was identified using 2 different methods and with equal weight given to errors of sensitivity and specificity. Figure 2 shows that the closest-to-(0,1) and Youden Index approaches identified the same point on the ROC curve for discharge FIM total. This point (FIM=78) yielded identical sensitivity and specificity (.77) values for classifying community and institutional discharges, respectively.

Fig 2.

ROC curve for discharge FIM total scores to differentiate between community and institutional discharges. The same cut point (FIM=78) was identified by the closest-to-(0,1) and Youden Index approaches, which locate the point on the ROC curve closest to perfect discrimination and furthest from chance discrimination, respectively.

Figure 3 illustrates the relationship between the identified cut point (FIM=78) and the FIM score probability density functions for the 2 discharge groups. It is evident that when errors in sensitivity and specificity are considered equally important, the optimal cut point is positioned at the intersection of the probability density functions for the 2 groups. Accordingly, this graphic also displays the tradeoffs in acceptable error rates with adjustments to the cut-point value. A higher cut point would decrease the false-positive rate, but it would also result in a relatively larger increase in the false-negative rate. Conversely, a lower cut point would decrease the false-negative rate at the expense of a relatively larger increase in the false-positive rate.

Fig 3.

Probability density functions for FIM total scores from the 2 discharge groups. The cut point shown represents the optimal FIM score for differentiating positive and negative tests when errors in sensitivity and specificity are equally weighted. The graph also shows the tradeoff between these errors (false-negative and false-positive rates) when moving the cut point from its current value.

Discussion

An important goal of rehabilitation is to maximize functioning to enable independence. There is little information, however, in the rehabilitation literature demonstrating a functional status threshold that embodies community discharge after inpatient rehabilitation. In this article we present methodology that is routinely used in biomedical research to judge the ability of biomarkers to diagnose disease, and then we apply that methodology to rehabilitation-relevant measures and outcomes. Our purpose was to evaluate the effectiveness of functional status ratings associated with community versus institutional discharges after inpatient rehabilitation for stroke. We compared the discriminative abilities of motor, cognitive, and total FIM ratings. We then used 2 different approaches to identify the cut point in FIM total scores that optimized the overall success rate of correct and incorrect classifications.

Our findings suggest that FIM total rating performs as well as or better than the motor and cognitive subscales in distinguishing between community and institutional discharges (see fig 1). Our findings further suggest that a FIM total rating of 78 is the ideal threshold to separate a positive test from a negative test. We realize that the FIM total rating does not reflect a single construct, and that FIM instrument ratings are more appropriately interpreted as motor (13 items) and cognitive (5 items) subscales. Figure 1 includes the AUC for the total FIM along with the motor FIM and cognitive FIM subscales. The AUC for the total FIM and motor FIM is the same. We elected to use the total FIM rating in the cut-point analysis for 2 reasons. First is ease of interpretation for clinicians and consumers. While we understand that functional independence, as assessed by the FIM, is not a unidimensional construct empirically or statistically, we believe that the general concept of functional independence has value. Functional independence is obviously multifactorial and includes more components than are contained in the FIM instrument. The FIM instrument represents one way to operationalize this construct. For the purpose of differentiating the probability of discharge to community versus noncommunity (institutional) settings, we felt it was appropriate to consider both motor and cognitive function in aggregate. Developing separate cut points for motor versus cognitive subscales could be clinically confusing. Second, previous research identifying “burden of care,” or the amount of assistance (time) required from another person to complete various tasks associated with the FIM has been based on total FIM ratings.¹⁸

Approximately 70% of patients in our study returned to the community. This is comparable to prior studies examining discharge after stroke rehabilitation, with reported percentages ranging from 65% to 85%.^22-24 Other investigators have also looked at the association between discharge disposition and various FIM metrics.^23,25-29 However, none of these prior investigations used empirical methods to evaluate the utility of the FIM as a diagnostic tool for classifying discharge setting.

Traditionally (anecdotally), a FIM discharge rating of 80 has generally been considered the target value associated with community discharge after inpatient rehabilitation for stroke. But we are not aware of any empirical evidence to justify this or any other functional threshold. Previous research involving persons with stroke has suggested 80 as the target FIM total rating associated with returning home.^23,29,30 This value was based on clinical experience and/or descriptive statistics for patients who returned home. Black et al,³⁰ for example, reported the sensitivity and specificity of discharge FIM ratings using cutoff values of 75, 80, and 85 from a sample of 234 stroke patients within a single rehabilitation hospital. The cut point of 80 produced a sensitivity of .94 and a specificity of .65. Bottemiller et al²³ reported that 81% of patients with a FIM rating at or above 80 were discharged home over a 5-year period. In that study, the cut point of 80 yielded a sensitivity of .88 and a specificity of .62.²³

Obviously, better functioning (higher FIM rating) is a universal goal for every patient. Thus, all else being equal, a discharge FIM rating of 80 or 85 is a better outcome than a FIM rating of 78. However, the goal of our study was to evaluate the utility (diagnostic accuracy) of the standardized inpatient rehabilitation functional measure for differentiating the probability of community versus institutional discharges. In other words, a FIM of 78 was identified as the optimal cut point for classifying both positive and negative cases, not as the optimal outcome for patients. The cut point simply represents the value that maximizes information from the FIM instrument for characterizing the 2 discharge groups. Altering the cut point has both statistical and clinical implications.

From a statistical perspective, sensitivity and specificity are inversely related, such that changing the cut point to increase one will decrease the other⁸—at least within the range of diagnostic variable scores that yield sensitivity and specificity values less than 1. Figure 3 provides a visual of this dynamic relationship. Shifting the cut point to the right, for example, will increase the proportion of patients discharged to an institutional setting with negative tests, which increases specificity and decreases the false-positive rate. Conversely, this shift will also decrease the proportion of patients discharged to the community with positive tests, which decreases sensitivity and increases the false-negative rate. It is clear from the figure that for any potential cut point to the right of optimal (FIM=78), the reduction in sensitivity is greater than the gain in specificity. For values to the left of optimal the reverse is true; the reduction in specificity is greater than the gain in sensitivity.²¹ Thus, without definitive reasons to place a greater value (weight) on either sensitivity or specificity, the cut point that maximizes the overall rate of correct classifications and minimizes the overall rate of misclassifications provides the most useful information.¹⁰

From a clinical perspective, the FIM cut point we identified (FIM=78) is 2 points below the informal standard and previously examined threshold (FIM=80) associated with community versus institutional discharges.^23,30 Two points on the total 108-point range for the FIM may seem inconsequential until one considers it in the context of the requirements for rehabilitation-based FIM gains. A recent national benchmark report for stroke rehabilitation shows that patients, on average, gain slightly less than 2 FIM points per day when computed over the entire length of their stays.³¹ Thus, it is reasonable to assume that it generally requires at least 1 additional day in rehabilitation to progress from a FIM of 78 to a FIM of 80.

The decision to adjust the cut point to favor either sensitivity or specificity requires one to quantify (weight) the risks associated with false positives (premature discharge) and false negatives (unnecessary care). Both of the methods we used to identify the optimal cut point can accommodate differential weighting of sensitivity and specificity.¹⁰ Currently, however, comprehensive risk-benefit analyses quantifying the (dis)advantages of additional care (higher functioning) versus earlier discharge to the community (lower functioning) are not available in the rehabilitation literature. This is an area in need of additional research. Information on the comparative risks associated with extended facility-based care (eg, infections, falls, deconditioning) and early community discharge (eg, caregiver burden, falls, rehospitalization) would improve the diagnostic ability of functional status to differentiate between discharge settings and corresponding levels of independence. For example, a cut point that yields a lower false-negative rate (higher sensitivity) would be more beneficial if it is determined that the risks of additional rehabilitative care exceed those of earlier community discharge. On the other hand, a cut point that yields a lower false-positive rate (higher specificity) would be more beneficial if it is determined that the risks of earlier community discharge exceed those of additional rehabilitative care. It is also important to note that all of this needs to be considered within the cost structure of prospective payment.

Study Limitations

A limitation, or caution, in interpreting the current findings involves the fact that sensitivity and specificity provide only population-level information about the diagnostic ability of a measurement instrument. In other words, they do not provide an estimate of the probability that an individual patient will experience a particular outcome. Once sensitivity and specificity are established for an instrument, however, they can be used (via likelihood ratio) along with outcome prevalence information to determine patient-level probability. Another limitation relates to reducing discharge setting to 2 groups. Our study evaluated community discharge against a variety of other noncommunity settings. We did not perform subanalyses examining differences in alternate discharge settings such as skilled nursing facilities. We selected community versus institutional settings for this initial investigation because discharge to the community is a global goal for all patients and an important quality indicator for program evaluation and accreditation reviews. Last, there are several nonrehabilitation-related factors influencing discharge decisions that were not included in the current study. Comprehensive assessments of discharge disposition require sophisticated analyses with complex interpretation and limited clinical application. Our approach mirrored that of the biomedical field, wherein the diagnostic ability of a particular biomarker for disease is established and the information from that test is combined (integrated) with other factors such as medical history and several clinical signs and symptoms to ultimately determine patient status. In our case, we simply applied this methodology to relevant rehabilitation indicator (functional status) and outcome (community discharge) variables.

Conclusions

Discharge planning is a complex process involving many factors. We view this study as the first step in examining the utility of the FIM instrument in differentiating community and institutional discharges after inpatient rehabilitation for stroke. Overall, the FIM performed well in classifying patients by discharge setting. With an equal emphasis on sensitivity and specificity, a FIM total rating of 78 was identified as the ideal cut point to distinguish positive from negative tests. Both positive (FIM≥78) and negative (FIM<78) tests correctly classified more than three quarters of patients discharged to the community and to institutional settings, respectively. Additional research is necessary to determine if the risk-benefits ratio of false positives and/or false negatives justify shifting the cut point to account for the consequences of early discharge or extended care.

List of Abbreviations

AUC

area under the curve

CMS

Centers for Medicare and Medicaid Services

ROC

receiver operating characteristic

UDSMR

Uniform Data System for Medical Rehabilitation