Abstract
Objective:
The Improving Medicare Post-Acute Care Transformation Act of 2014 (IMPACT Act) mandates using standardized patient functional data across post-acute settings. This study characterized similarities and differences in clinician-observed scores of self-care and transfer items for the standardized Section GG functional domain and the Functional Independent Measure® (FIM) at inpatient rehabilitation facilities (IRFs).
Design:
We conducted secondary analyses of 2017 Uniform Data System for Medical Rehabilitation (UDSMR) national data. Patients were assessed by clinicians on both Section GG and FIM at admission and discharge. We identified seven self-care items and six transfer items in Section GG conceptually equivalent with FIM. Clinician-assessed scores for each pair of items were examined using score distributions, Bland-Altman plot, correlation (Pearson coefficients), and agreement (kappa and weighted kappa) analyses.
Setting and Participants:
In all, 408,491 patients were admitted to UDSMR-affiliated IRFs with one of the following impairments: stroke, brain dysfunction, neurologic condition, orthopedic disorders, and debility.
Measures:
Section GG and FIM.
Results:
Patients were scored as more functionally independent in Section GG compared to FIM, but change score distributions and score orders within impairment groups were similar. Total scores in Section GG had strong positive correlations (self-care: r=0.87 and 0.95; transfer: r=0.82 and 0.90 at admission and discharge, respectively) with total FIM scores. Weak to moderate ranking agreements with total FIM scores were observed (self-care: kappa= 0.49 and 0.60; transfers: kappa= 0.43 and 0.52 at admission and discharge, respectively). Lower agreements were observed for less able patients at admission and for higher ability patients of their change scores.
Conclusions and Implications:
Overall, response patterns were similar in Section GG and FIM across impairments. However, variations exist in score distributions and ranking agreement. Future research should examine the use of GG codes to maintain effective care, outcomes, and unbiased reimbursement across post-acute settings.
Keywords: Subacute Care, Medicare Payment Advisory Commission, Health Services Administration, Outcome and Process Assessment, Health Care, Mobility, Self-Care, Critical Care Outcomes
Brief Summary:
Evaluating response patterns of the IMPACT Act mandatory standardized functional data sets up baseline evidence to further determine quality reporting and maintain effective care, outcomes, and unbiased reimbursement across post-acute settings.
Introduction
The Improving Medicare Post-Acute Care Transformation Act of 2014 (IMPACT Act) is changing how functional status, care and services will be evaluated and reimbursed across post-acute settings.1 To resolve the concerns of using different functional assessments across post-acute settings, the IMPACT Act mandates the use of standardized patient assessment data elements (SPADEs) across the continuum of post-acute care.1-4 Section GG was developed as part of the SPADEs to measure functional abilities and goals across inpatient rehabilitation facilities (IRF), skilled nursing facilities, home health agencies, and long-term care hospitals.2,3
Since the IRF prospective payment system (IRF-PPS) was implemented in 2002, the Centers for Medicare and Medicaid Services (CMS) incorporated 11 items from the Functional Independence Measure® (FIM) in the Inpatient Rehabilitation Facility-Patient Assessment Instrument (IRF-PAI).5 These FIM items measure patients’ performances of fundamental daily activities and the amount of needed assistance (burden of care).6 Although the entire IRF-PAI includes more than FIM items, for convenience, this manuscript refers to the prior site-specific IRF-PAI functional items as FIM and the new standardized functional items in the IRF-PAI as Section GG. CMS mandated the use of Section GG beginning October 2016 and officially replaced the FIM functional items in October 2019.6 There is limited information of direct comparison between Section GG and FIM. A 2019 report of FY 2017 Medicare data, found a small difference between the models predicting costs of care using the CMGs based on the original IRF-PPS (FIM items) versus the model using the CMGs based on SPADEs (GG items)7.
In June 2019, the Medicare Payment Advisory Commission (MedPAC) indicated a concern with the fidelity of reported functional data across post-acute providers.8 IRFs began co-collecting Section GG and FIM functional data in 2016 as part of the IRF Quality Reporting Program.9 This simultaneous data collection for three years at IRFs permits unique examination of the impact of this transition on how clinicians and facilities evaluate and report functional outcomes. We used this simultaneous collection of both Section GG and FIM data at IRFs in 2017 to examine response patterns at nearly 900 U.S. IRFs.
This study aims to: 1) develop an algorithm to identify scoring discrepancies between Section GG and FIM; 2) characterize the distribution of standardized functional outcomes at admission, discharge, and functional change across impairment conditions; and 3) examine similarities and differences between clinician-observed scores in Section GG and FIM for the same IRF patient.
Methods
Data Source
We analyzed data retrieved from the 2017 Uniform Data System for Medical Rehabilitation (UDSMR®). UDSMR keeps the largest non-governmental database covering approximately 70% of U.S. IRFs for inpatient medical rehabilitation outcomes, including free-standing inpatient rehabilitation hospitals and acute hospital-based rehabilitation units. The study was approved by the university Institutional Review Board. A Data Use Agreement with the UDSMR was also obtained.
Study Population
The patients’ primary admission diagnosis was one of the following: stroke, brain dysfunction, neurologic condition, orthopedic disorders, and debility; accounting for >90% of our IRF cohort. Each patient was evaluated on items from both Section GG and FIM at admission and/or discharge (Suppl. Table 1).
Developing an Algorithm to Identify Scoring Discrepancy
To resolve MedPAC data fidelity concerns, we developed an algorithm to determine inconsistent and clinically implausible ratings between Section GG and FIM (Table 1). This algorithm was based on the scoring criteria from the IRF-PAI manuals11 and the clinical judgement from our research team. We determined, for instance, clinically implausible differences in the ratings if a patient scored a 7 (complete independence) on FIM but a 1 (complete dependence) in Section GG for equivalent items. The first author and the second author applied clinical judgements to come to a consensus on which scoring combinations were clearly implausible (Table 1). Suppl. Table 2 provides detailed rating scale structures for Section GG and FIM Items.
Table 1.
Algorithm to Determine Consistent and Implausible Ratings between Section GG and FIM.
| Functional Independence Measure Rating Scale |
Section GG Rating Scale | O= Consistent X= Implausible |
|---|---|---|
| 7 Complete Independence | 6 Independent | O |
| 5 Set up clean up | O | |
| 4 Supervision or touching | X | |
| 3 Partial/Moderate | X | |
| 2 Substantial/maximal | X | |
| 1 Dependent | X | |
| 6 Modified Independence | 6 Independent | O |
| 5 Set up clean up | O | |
| 4 Supervision or touching | X | |
| 3 Partial/Moderate | X | |
| 2 Substantial/maximal | X | |
| 1 Dependent | X | |
| 5 Supervision | 6 Independent | X |
| 5 Set up clean up | O | |
| 4 Supervision or touching | O | |
| 3 Partial/Moderate | X | |
| 2 Substantial/maximal | X | |
| 1 Dependent | X | |
| 4 Minimal Assistance (>75%) | 6 Independent | X |
| 5 Set up clean up | X | |
| 4 Supervision or touching | O | |
| 3 Partial/Moderate | O | |
| 2 Substantial/maximal | X | |
| 1 Dependent | X | |
| 3 Moderate Assistance (51-75%) | 6 Independent | X |
| 5 Set up clean up | X | |
| 4 Supervision or touching | O | |
| 3 Partial/Moderate | O | |
| 2 Substantial/maximal | X | |
| 1 Dependent | X | |
| 2 Maximal Assistance (26-50%) | 6 Independent | X |
| 5 Set up clean up | X | |
| 4 Supervision or touching | X | |
| 3 Partial/Moderate | O | |
| 2 Substantial/maximal | O | |
| 1 Dependent | O | |
| 1 Total Assistance (0-25%) | 6 Independent | X |
| 5 Set up clean up | X | |
| 4 Supervision or touching | X | |
| 3 Partial/Moderate | X | |
| 2 Substantial/maximal | O | |
| 1 Dependent | O |
Abbreviations: FIM=Functional Independence Measure® scores; Section GG: functional ability scores in the standardized patient assessment data elements (SPADEs) assessment instrument. Please refer to Methods for details regarding how this algorithm was developed.
Standardized Functional Data: Section GG
Section GG has two domains of self-care and mobility, with seven self-care and 17 mobility items (Suppl. Table 3). To calculate comparable scores between Section GG and FIM, we extracted six transfer items from Section GG (roll left and right, sit to lying, lying to sit on side of bed, sit to stand, chair/bed-to-chair transfer, and toilet transfer) to compare with three transfer items (bed/chair/wheelchair transfer, toilet transfer and tub/shower transfer) in FIM. If no exact corresponding item was found, the most relevant items were chosen (Suppl. Table 3).
Total scores of self-care and transfer for both Section GG and FIM at admission and at discharge were calculated. Change scores were calculated by subtracting the admission score from the discharge score. Total GG raw scores range 7-42 in self-care and 6-36 in transfer. Total FIM scores range 6-42 in self-care and 3-21 in transfer. To adjust the differences in rating scales (i.e., Section GG: 1-6; FIM: 1-7) (Suppl. Table 2), we recalibrated scores as a 0-100% ratio for fair comparison. Each total score was subtracted from the mean domain score, then divided by the domain score range. We divided the recalibrated total scores into quartiles based on the distribution, to compare ranking agreement.
Statistical Analysis
Demographics and clinical characteristics were stratified by impairments for descriptive analyses. We replaced missing and the score of ‘0’ (activity does not occur) of FIM as a score of 1 (the lowest function) following the CMS approach.12 To improve fidelity of the analyzed data, we excluded patients with >3 items with implausible ratings within a single domain based on the developed algorithm (Table 1). Because the goal was to compare responses across all items in a domain, we excluded patients with any missing (Step 2, Suppl. Table 1). We generated separate subsamples by domain (self-care and transfer) and by evaluation time (admission and discharge) for correlation and agreement analyses. We used Pearson coefficients to examine the associations of recalibrated total scores between Section GG and FIM with Mukaka13 criteria (0.3-0.5 as low, 0.5-0.7 as moderate, and 0.7-0.9 as high). Since correlations only examine the relationship between two variables but not the differences, we used Cohen’s kappa/weighted kappa criteria14 (0.4-0.59 as weak, 0.6-0.79 as moderate, and 0.8-0.9 as strong) to indicate the levels of ranking agreements. We used Bland-Altman plots to further examine whether low agreement occurred at the lower-end, middle-, or higher- end of ability levels.15
Results
Demographics
Table 2 presents demographics of 408,491 patients receiving inpatient rehabilitation services from January 2017 through December 2017 in the UDSMR. The mean age was 70.0 (SD 14.4) years, the majority were non-Hispanic white (77.1%), women (53.2%), married (47.8%), and living with family/relatives prior to acute care hospitalization (67.8%). The mean Elixhauser comorbidity index was 3.8 (SD 2.0). Stroke was the most prevalent rehabilitation impairment, representing 29.8% of the cohort, followed by orthopedic disorders (27.3%), neurologic conditions (17.4%), brain dysfunction (14.5%), and debility (11.1%) (Table 2).
Table 2.
Demographic Characteristics by Impairment Groups (N=408,491).
| All (n=408,491) |
Stroke (n=121,640) |
Brain Dysfunction (n=59,184) |
Neurologic Condition (n=70,886) |
Orthopedic Disorders (n=111,581) |
Debility (n=45,200) |
|
|---|---|---|---|---|---|---|
| Age at admission – CMS Calendar (years) | ||||||
| Mean (SD) | 70.0 (14.4) | 69.1 (13.3) | 66.7 (16.9) | 69.5 (14.3) | 71.9 (14.1) | 72.6 (13.3) |
| Median (Q1, Q3) | 72 (62, 81) | 70 (60, 79) | 70 (58, 79) | 72 (62, 80) | 74 (64, 83) | 75 (65, 83) |
| Sex, n (%) | ||||||
| Male | 191358 (46.9) | 63206 (52.0) | 32415 (54.8) | 34447 (48.6) | 39598 (35.5) | 21692 (48) |
| Female | 217062 (53.2) | 58404 (48.0) | 26757 (45.2) | 36434 (51.4) | 71967 (64.5) | 23500 (52) |
| Race, n (%) | ||||||
| Non-Hispanic white | 315002 (77.1) | 83985 (69.0) | 45439 (76.8) | 56715 (80.0) | 92505 (82.9) | 36358 (80.4) |
| Non-Hispanic Black | 47880 (11.7) | 20451 (16.8) | 6260 (10.6) | 8042 (11.3) | 8116 (7.3) | 5011 (11.1) |
| Hispanic | 21103 (5.2) | 7298 (6) | 3326 (5.6) | 3538 (5.0) | 5177 (4.6) | 1764 (4.0) |
| Others | 24506 (6) | 9906 (8.1) | 4159 (7.0) | 2591 (3.7) | 5783 (5.2) | 2067 (4.6) |
| BMI, n (%) | ||||||
| Underweight | 19971 (5.0) | 5055 (4.2) | 3461 (5.9) | 3571 (5.1) | 5457 (5.0) | 2427 (5.4) |
| Normal weight | 120403 (29.9) | 35101 (29.3) | 20545 (35.2) | 19612 (28.0) | 32240 (29.3) | 12905 (28.9) |
| Overweight | 118677 (29.5) | 39002 (32.5) | 17549 (30.1) | 19274 (27.5) | 30699 (27.9) | 12153 (27.2) |
| Obesity | 143976 (35.7) | 40700 (44.0) | 16789 (28.8) | 27524 (39.3) | 41788 (37.9) | 17175 (38.5) |
| Marital Status, n (%) | ||||||
| Never Married | 70747 (17.8) | 21837 (18.5) | 12155 (21.1) | 12260 (17.7) | 17208 (15.9) | 7287 (16.6) |
| Married | 189662 (47.8) | 58953 (50.0) | 28971 (50.4) | 33580 (48.6) | 48677 (44.9) | 19481 (44.3) |
| Widowed | 91701 (23.1) | 23164 (19.6) | 10351 (18.0) | 15320 (22.2) | 30667 (28.3) | 12199 (27.7) |
| Separated | 4161 (1.1) | 1522 (1.3) | 547 (1.0) | 699 (1.0) | 971 (0.9) | 422 (1.0) |
| Divorced | 40839 (10.3) | 12490 (10.6) | 5461 (9.5) | 7271 (10.5) | 11019 (10.2) | 4598 (10.5) |
| Comorbidity Tier, n (%) | ||||||
| No cost | 195044 (47.8) | 66933 (55.0) | 21813 (36.9) | 25274 (35.7) | 66339 (59.5) | 14685 (32.5) |
| Lowest cost | 20278 (5.0) | 49784 (40.9) | 3598 (6.1) | 6060 (8.6) | 2785 (2.5) | 3966 (8.8) |
| Medium cost | 33368 (8.2) | 1054 (0.9) | 12796 (21.6) | 9616 (13.6) | 5124 (4.6) | 4778 (10.6) |
| Highest cost | 159801 (39.1) | 3869 (3.2) | 20977 (35.4) | 29936 (42.2) | 37333 (33.5) | 21771 (48.2) |
| Region, n (%) | ||||||
| CT, RI, MA, ME, NH, VT | 20747 (5.1) | 6627 (5.5) | 2761 (4.7) | 2611 (3.7) | 6188 (5.6) | 2560 (5.7) |
| NY, NJ | 25543 (6.3) | 9064 (7.5) | 3698 (6.3) | 2969 (4.2) | 7443 (6.7) | 2369 (5.2) |
| PA, WV, VA, DC, MD, DE | 52267 (12.8) | 14560 (12.0) | 7022 (11.9) | 10566 (14.9) | 14837 (13.3) | 5282 (11.7) |
| KY, TN, MS, AL, GA, SC, | ||||||
| NC, FL | 87458 (21.4) | 24263 (20.0) | 11655 (19.7) | 15730 (22.2) | 25124 (22.5) | 10686 (23.6) |
| MN, WI, MI, IL, IN, OH | 55224 (13.5) | 18586 (15.3) | 9712 (16.4) | 9467 (13.4) | 11028 (9.9) | 6431 (14.2) |
| NM, OK, AR, TX, LA | 83305 (20.4) | 19722 (16.2) | 10113 (17.1) | 17479 (24.7) | 25966 (23.3) | 10025 (22.2) |
| NE, IA, KS, MO | 20837 (5.1) | 6818 (5.6) | 3319 (5.6) | 3050 (4.3) | 5063 (4.5) | 2587 (5.7) |
| MT, ND, WY, SD, UT, CO | 12770 (3.1) | 3975 (3.3) | 2316 (3.9) | 1589 (2.2) | 3681 (3.3) | 1209 (2.7) |
| NV, CA, AZ | 43770 (10.7) | 14375 (11.8) | 7367 (12.5) | 6985 (9.9) | 11410 (10.2) | 3633 (8.0) |
| WA, OR, ID | 6570 (1.6) | 3650 (3) | 1221 (2.1) | 440 (0.6) | 841 (0.8) | 418 (0.9) |
| Pre-hospital Living With, n (%) | ||||||
| Alone | 110955 (27.5) | 30093 (25.0) | 13426 (23.0) | 18449 (26.5) | 35401 (32.0) | 13586 (30.6) |
| Family/Relative | 273379 (67.8) | 84697 (70.5) | 41875 (71.7) | 47874 (68.8) | 70228 (63.6) | 28705 (64.7) |
| Friends | 7902 (2.0) | 2779 (2.3) | 1427 (2.4) | 1200 (1.7) | 1800 (1.6) | 696 (1.6) |
| Attendant | 5004 (1.2) | 1023 (0.9) | 747 (1.3) | 1028 (1.5) | 1460 (1.3) | 746 (1.7) |
| Other | 5858 (1.5) | 1604 (1.3) | 943 (1.6) | 1079 (1.6) | 1585 (1.4) | 647 (1.5) |
| Discharge Living With, n (%) | ||||||
| Alone | 11135 (9.6) | 2877 (7.1) | 1311 (6.9) | 1886 (11.2) | 3633 (12.1) | 1428 (14.0) |
| Family/Relatives | 90655 (77.7) | 33237 (82.1) | 15537 (81.2) | 12145 (72.2) | 22345 (74.5) | 7391 (72.3) |
| Friends | 2649 (2.7) | 978 (2.4) | 483 (2.5) | 319 (1.9) | 679 (2.3) | 190 (1.9) |
| Attendant | 1842 (1.6) | 589 (1.5) | 344 (1.8) | 251 (1.5) | 438 (1.5) | 220 (2.2) |
| Othersa | 10374 (8.9) | 2795 (6.9) | 1466 (7.7) | 2214 (13.2) | 2911 (9.7) | 988 (9.7) |
| Pre-hospital Living Setting, n (%) | ||||||
| Homeb | 403098 (98.7) | 120196 (98.8) | 58418 (98.7) | 69630 (98.2) | 110474 (99.0) | 44380 (98.2) |
| Skilled Nursing Facility (SNF) | 1265 (0.3) | 372 (0.3) | 203 (0.3) | 267 (0.4) | 233 (0.2) | 190 (0.4) |
| Home under care of organized home health | ||||||
| service organization | 266 (0.1) | 116 (0.1) | 25 (0.0) | 71 (0.1) | 35 (0.03) | 19 (0.0) |
| Othersa | 3862 (1.0) | 956 (0.8) | 538 (0.9) | 918 (1.3) | 839 (0.8) | 611 (1.4) |
| Admitted to Rehab From, n (%) | ||||||
| Homeb | 13768 (3.4) | 2888 (2.4) | 1156 (2.0) | 4881 (6.9) | 2843 (2.6) | 2000 (4.4) |
| Short-term General Hospital | 381965 (93.5) | 115467 (94.9) | 56233 (95.0) | 62044 (87.5) | 106631 (95.6) | 41590 (92.0) |
| Skilled Nursing Facility (SNF) | 3980 (1.0) | 1473 (1.2) | 483 (0.8) | 789 (1.1) | 909 (0.8) | 326 (0.7) |
| Othersa | 8778 (2.2) | 1812 (1.5) | 1312 (2.2) | 3172 (4.5) | 1198 (1.2) | 1284 (2.8) |
| Discharge Setting, n (%) | ||||||
| Homeb | 116655 (28.6) | 40476 (33.3) | 19141 (32.3) | 16815 (23.7) | 30006 (26.9) | 10217 (22.6) |
| Skilled Nursing Facility (SNF) | 51579 (12.6) | 20619 (17.0) | 6572 (11.1) | 6753 (9.5) | 13645 (12.2) | 3990 (8.8) |
| Home under care of organized home health service organization | 386 (0.1) | 168 (0.1) | 68 (0.1) | 42 (0.1) | 72 (0.1) | 36 (0.1) |
| Othersa | 239871 (58.7) | 60377 (49.6) | 33403 (56.4) | 47276 (66.7) | 67858 (60.8) | 30957 (68.5) |
| Total Length of Stay, R&D Calc. | ||||||
| Mean (SD) | 13.0 (7.3) | 15.2 (8.7) | 13.1 (8.8) | 12.5 (6.6) | 11.7 (4.9) | 11.3 (5.4) |
| Median (Q1, Q3) | 12 (9, 16) | 14 (9, 20) | 12 (8, 16) | 12 (9, 15) | 12 (8, 15) | 11 (8, 14) |
| Actual Net Length of Stay | ||||||
| Mean (SD) | 13 (7.3) | 15.2 (8.7) | 13.1 (8.8) | 12.5 (6.6) | 11.7 (4.9) | 11.3 (5.4) |
| Median (Q1, Q3) | 12 (9, 16) | 14 (9, 20) | 12 (8, 16) | 12 (9, 15) | 12 (8, 15) | 11 (8, 14) |
| Expected Net Length of Stay | ||||||
| Mean (SD) | 15.3 (4.6) | 17.9 (5.7) | 15.3 (4.4) | 14.7 (3.7) | 13.7 (3.1) | 13.5 (3.0) |
| Median (Q1, Q3) | 15 (12, 17) | 17 (14, 24) | 14 (12, 18) | 15 (13, 17) | 14 (11, 16) | 13 (12, 16) |
| Number of Elixhauser comorbidity | ||||||
| Mean (SD) | 3.8 (2.0) | 4.2 (1.9) | 3.8 (2.0) | 4.1 (2.1) | 3.2 (1.9) | 4.2 (2.0) |
| Median (Q1, Q3) | 4 (2, 5) | 4 (3, 5) | 4 (2, 5) | 4 (3, 5) | 3 (2, 4) | 4 (3, 6) |
All demographic characteristics were significantly different by impairment groups (p< 0.05).
‘Others’ included short-term general hospital, hospice (home and institutional facility), another inpatient rehabilitation facility, long-term care hospital (LTCH).
‘Home’ included private home/apartment, board/care, assisted living, group home, transitional living.
Abbreviations: SC: Self-Care; Trans: Transfer.
Using Algorithm to identify Scoring Discrepancy between Assessments
Suppl. Table 4 shows the percent of implausible scores between Section GG and FIM. Overall, clinicians scored self-care items more consistently than transfer items. In respective self-care and transfer domains, Section GG lower-body dressing (89.1%) and toilet transfer (88.7%) were the most consistently scored items, while oral hygiene (72.7%) and roll left and right (58.3%) were the least consistently scored. After applying the developed algorithm, transfer change scores had the largest improvement of correlations and agreements, especially in orthopedic disorders (Table 3, Suppl. Table 6).
Table 3.
Correlation and Ranking Agreement between Section GG and FIM by Impairment (after Removing Discordant Scores).
| All | Stroke | Brain Dysfunction | Neurologic condition | Orthopedic Disorders | Debility | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Correlation | n | Corr., (95 CI) | n | Corr., (95 CI) | n | Corr., (95 CI) | n | Corr., (95 CI) | n | Corr., (95 CI) | n | Corr., (95 CI) |
| Admission Score | ||||||||||||
| Self-Care | 291,836 | 0.87 (0.87, 0.87) | 87,392 | 0.90 (0.90, 0.90) | 41,055 | 0.89 (0.89, 0.89) | 49,704 | 0.85 (0.85, 0.85) | 82,071 | 0.81 (0.81, 0.82) | 31,614 | 0.84 (0.84, 0.85) |
| Transfer | 258,234 | 0.82 (0.81, 0.82) | 79,284 | 0.85 (0.85, 0.85) | 37,876 | 0.82 (0.82, 0.82) | 45,753 | 0.79 (0.79, 0.79) | 66,642 | 0.78 (0.78, 0.79) | 28,679 | 0.76 (0.76, 0.77) |
| Discharge Score | ||||||||||||
| Self-Care | 348,357 | 0.95 (0.95, 0.95) | 103,322 | 0.95 (0.95, 0.96) | 49,200 | 0.95 (0.95, 0.95) | 60,011 | 0.94 (0.94, 0.94) | 98,022 | 0.93 (0.93, 0.93) | 37,802 | 0.93 (0.93, 0.93) |
| Transfer | 291,424 | 0.90 (0.90, 0.90) | 88,736 | 0.92 (0.92, 0.92) | 41,718 | 0.90 (0.90, 0.90) | 50,399 | 0.89 (0.89, 0.90) | 78,905 | 0.88 (0.88, 0.88) | 31,666 | 0.87 (0.87, 0.87) |
| Change Score | ||||||||||||
| Self-Care | 265,572 | 0.80 (0.79, 0.80) | 79,873 | 0.80 (0.80, 0.81) | 36,994 | 0.82 (0.82, 0.82) | 44,523 | 0.80 (0.80, 0.80) | 75,749 | 0.76 (0.75, 0.76) | 28,433 | 0.78 (0.78, 0.78) |
| Transfer | 200,369 | 0.73 (0.73, 0.74) | 62,570 | 0.73 (0.72, 0.73) | 29,142 | 0.73 (0.72, 0.73) | 34,828 | 0.73 (0.72, 0.73) | 51,974 | 0.72 (0.72, 0.72) | 21,855 | 0.70 (0.69, 0.71) |
| Agreement | n | Kappa (95 CI) Weighted Kappa (95 CI) |
n | Kappa (95 CI) Weighted Kappa (95 CI) |
n | Kappa (95 CI) Weighted Kappa (95 CI) |
n | Kappa (95 CI) Weighted Kappa (95 CI) |
n | Kappa (95 CI) Weighted Kappa (95 CI) |
n | Kappa (95 CI) Weighted Kappa (95 CI) |
| Admission Score | ||||||||||||
| Self-Care | 291,836 | 0.49 (0.49, 0.50) | 87,392 | 0.56 (0.55, 0.56) | 41,055 | 0.54 (0.53, 0.54) | 49,704 | 0.45 (0.45, 0.46) | 82,071 | 0.41 (0.40, 0.41) | 31,614 | 0.45 (0.45, 0.46) |
| 0.68 (0.68, 0.68) | 0.73 (0.72, 0.73) | 0.71 (0.70, 0.71) | 0.64 (0.64, 0.65) | 0.60 (0.60, 0.60) | 0.64 (0.64, 0.65) | |||||||
| Transfer | 258,234 | 0.43 (0.43, 0.43) | 79,284 | 0.48 (0.48, 0.49) | 37,876 | 0.43 (0.42, 0.43) | 45,753 | 0.41 (0.40, 0.42) | 66,642 | 0.42 (0.42, 0.43) | 28,679 | 0.36 (0.35, 0.37) |
| 0.62 (0.62, 0.62) | 0.67 (0.67, 0.68) | 0.61 (0.61, 0.62) | 0.59 (0.59, 0.60) | 0.62 (0.62, 0.63) | 0.54 (0.53, 0.55) | |||||||
| Discharge Score | ||||||||||||
| Self-Care | 348,357 | 0.60 (0.60, 0.60) | 103,322 | 0.68 (0.68, 0.69) | 49,200 | 0.61 (0.60, 0.61) | 60,011 | 0.61 (0.60, 0.61) | 98,022 | 0.56 (0.55, 0.56) | 37,802 | 0.57 (0.57, 0.58) |
| 0.76 (0.76, 0.76) | 0.81 (0.81, 0.82) | 0.76 (0.76, 0.77) | 0.76 (0.76, 0.77) | 0.73 (0.73, 0.73) | 0.74 (0.74, 0.74) | |||||||
| Transfer | 291,424 | 0.52 (0.51, 0.52) | 88,736 | 0.58 (0.57, 0.58) | 41,718 | 0.55 (0.55, 0.56) | 50,399 | 0.59 (0.58, 0.59) | 78,905 | 0.53 (0.53, 0.54)a | 31,666 | 0.51 (0.50, 0.52) |
| 0.70 (0.70, 0.70) | 0.74 (0.74, 0.74) | 0.72 (0.72, 0.72) | 0.75 (0.74, 0.75) | 0.64 (0.63, 0.64)a | 0.69 (0.69, 0.70) | |||||||
| Change Score | ||||||||||||
| Self-Care | 265,572 | 0.40 (0.40, 0.40) | 79,873 | 0.41 (0.40, 0.41) | 36,994 | 0.42 (0.41, 0.43) | 44,523 | 0.40 (0.39, 0.40) | 75,749 | 0.36 (0.35, 0.36) | 28,433 | 0.37 (0.37, 0.38) |
| 0.58 (0.58, 0.58) | 0.58 (0.58, 0.59) | 0.59 (0.59, 0.60) | 0.58 (0.57, 0.58) | 0.54 (0.53, 0.54) | 0.55 (0.55, 0.56) | |||||||
| Transfer | 200,369 | 0.32 (0.31, 0.32) | 62,570 | 0.32 (0.32, 0.33) | 29,142 | 0.31 (0.30, 0.32) | 34,828 | 0.31 (0.30, 0.31) | 51,974 | 0.31 (0.30, 0.31) | 21,855 | 0.29 (0.28, 0.30) |
| 0.49 (0.48, 0.49) | 0.50 (0.49, 0.50) | 0.49 (0.48, 0.49) | 0.48 (0.47, 0.48) | 0.47 (0.46, 0.47) | 0.45 (0.45, 0.46) | |||||||
Abbreviations: Corr.= Correlation.
Created by using tertile.
Recalibrated Total Score Distributions
Section GG and FIM self-care and transfer total scores were recalibrated (0-100%) in distributions at admission, discharge, and change (Figure 1). Suppl. Table 5 provides detailed score distributions of Section GG and FIM, including medians, means, SD, skewness, and kurtosis. Both self-care and transfer scores at admission were relatively normally distributed, but FIM scores had a fatter tail at the lower-functioning end (left side), suggesting less discrimination of lower-functioning patients. For self-care at discharge, both assessments produced non-normal distributions, with Section GG having a pronounced ceiling effect. For transfers at discharge, both assessments produced non-normal score distributions, but both had pronounced “ceilings,” albeit the FIM score ceiling was at about 80. Self-care change score distributions for both assessments were almost identical, although Section GG captured more patients who declined in self-care performance. Transfer change score distributions were relatively normal and similar, although FIM change scores were more peaked; both assessments detected some degrees of decline in transfer performance (Figure 1).
Figure 1. Section GG and FIM Recalibrated Score Distributions (0-100%) (Top: Admission, Middle: Discharge, Bottom: Change; Red: Section GG; Blue: FIM).
Figure 1 demonstrates Section GG and FIM recalibrated self-care and mobility score distributions at admission, discharge, and the change between admission and discharge (discharge – admission). Red color represents Section GG and blue color represents FIM. Abbreviations: FIM: Functional Independence Measure.
Recalibrated Scores by Impairment
Across impairments, admission scores differed more between Section GG and FIM than the discharge and change scores (Suppl. Figure 1). Overall, both assessment scores across impairment groups were as expected (e.g., orthopedic patients were most functional in self-care but the least functional in transfer at admission compared to other impairments). Similar patterns were observed across impairments for both Section GG and FIM (e.g., patients with debility scored the highest at admission in self-care, following by orthopedic disorders, brain dysfunction, neurological condition and stroke for both assessments) (Suppl. Figure 1, self-care panel). Patients scored more independent at discharge compared to admission, regardless of assessment used or impairment. Section GG showed a wider score range compared to FIM across impairments (e.g., admission self-care in stroke: ranged 31-42 in Section GG vs. 31-34 in FIM), but the ranges for change scores were similar (Suppl. Figure 1).
Correlation & Ranking Agreement
Using recalibrated total scores, Section GG had moderate-to-strong correlations with FIM at admission and discharge (>0.8) and for change scores (range 0.70-0.82). Discharge scores had stronger correlations than admission, regardless of functional domain or impairment (Table 3).
Section GG had weak-to-moderate agreements with FIM; the lowest agreement was in transfer change scores (weighted kappa range 0.45-59). Patients with stroke had the strongest correlations and agreements, followed by brain dysfunction and neurological conditions (Table 3).
Bland-Altman Plots of Recalibrated Total Scores
Bland-Altman plots evaluate a bias between the mean differences from two assessments and estimate 95% agreement intervals of two assessments (± 2 SD).15,16 We found the means of B-A plots did not equal zero, indicating that patients tended to be scored higher (more functionally independent) in Section GG than in FIM. Patients with lower average ability levels had lower agreement between two assessments at admission (Figure 2, admission plots: lower left below the lower 95% CI), especially if patients scored higher in FIM. Similarly, patients with higher average ability levels had lower agreement for change score (Figure 2, change score plots: lower right below the lower 95% CI), especially if patients with greater change scores and higher FIM scores.
Figure 2. Bland-Altman Plot between Section GG and FIM Recalibrated Scores (0-100%) (Top: Admission, Middle: Discharge, Bottom: Change).
Figure 2 demonstrates Bland-Altman plots between Section GG and FIM recalibrated self-care and mobility scores at admission, discharge, and the change between admission and discharge (discharge – admission). The x-axis of the Bland-Altman plot is the average of two measures ([Section GG+ FIM]/2) and the y-axis is the difference between the two measures (Section GG – FIM). Red line represents mean difference between Section GG and FIM and blue line represents the upper and the lower bounds within ± 2s of the mean difference. Abbreviations: FIM: Functional Independence Measure.
Discussion
The alignment of clinician-observed scores in self-care and transfer between standardized Section GG and site-specific FIM functional data were as expected, implying most inpatient rehabilitation providers accurately report patients’ function based on the given instructions of each assessment. Section GG and FIM scores mostly aligned regardless of evaluation timing or impairment. We recommended future studies accounting for variations in functional score distributions when analyzing Section GG data across impairments. Patients were scored more frequently as independent using Section GG relative to FIM, because patients are considered independent in Section GG even if using assistive devices. The Section GG rating also distinguishes patients who are dependent or need 2-person assistance from those who can contribute some physical effort to the activity. The latter will tend to receive a score of 2 (substantial assistance) on Section GG and a score of 1 (total assistance) on FIM. Thus, at both the upper and lower ends of the rating scale, patients receive higher scores on Section GG than on FIM. The relatively higher scores do not imply quantitatively “better” patient function but reflect the difference in rating scale step definitions.
Further, Section GG scores reflect usual performance while FIM scores reflect the most dependent performance; thus, Section GG scores will tend toward being higher. Additionally, there may be incentive for facilities to report most dependent performances (such as during an evening shift), since lower FIM scores result in patient assignment in higher-paying CMS categories.
We also found that Section GG scores tended to be more discriminating for patients with less self-care and transfer ability, whereas FIM scores tended to be more discriminating for patients with greater self-care and transfer ability. This difference likely reflects the relative challenge of items within each assessment (e.g., FIM transfer items include more challenging tub/shower transfer compared Section GG, in which the most challenging item is toilet transfer).17,18 Section GG self-care has more items and rating scale categories to discriminate less able patients while FIM transfer has more items and rating scale categories to discriminate more able patients. Each of these factors — rating scale categories and definitions, challenge levels of the items, and usual vs. most dependent performance — likely contributes to our findings of difference between two assessments.
We also found stronger correlations between Section GG and FIM in self-care than in transfer. This may reflect the greater alignment in the content of self-care items than in transfer items. Furthermore, GG mobility items were designed to capture patients’ ability across the full post-acute continuum, from long term care, skilled nursing, and hospitals to home care. Thus, GG items tend to reflect a wider range of mobility performance than FIM items, which were originally designed primarily for inpatient rehabilitation. This does not imply that one assessment is more valid or accurate than the other, but rather reflects the different purposes for which the assessments were originally built.
The stronger correlation in discharge scores than in admission scores may in part be due to the non-normal distributions observed in both assessments at discharge. Scores at discharge tended to be compressed into a smaller region of the 0-100 range, further magnifying differences in item challenge and the impact of category differences between the Section GG rating scales of 6 and the FIM rating scales of 7. Both Section GG and FIM self-care scores are skewed negatively, suggesting that many patients achieved close to total independence in self-care prior to discharge. The clear ceiling effect observed for Section GG but not for FIM self-care items at discharge is likely a result of rating scale category differences.
These data implied that many patients were discharged using an assistive device to independently complete self-care activities. For transfers at discharge, both Section GG and FIM were negatively skewed and “ceiling” effects are apparent; on Section GG, the “ceiling” is at 100 and for FIM, the “ceiling” is at about 80 (recalibrated scores). Practically, this can be interpreted as, for Section GG, patients are mostly independent in toilet transfers (with or without an assistive device) at discharge, but on FIM, most patients need assistance with the more challenging tub-shower transfer item.
Overall, the Bland-Altman plots were consistent with other findings. At admission and discharge for self-care and transfers, Section GG scores were higher than FIM. Neither assessment showed any apparent systematic bias. Scores that exceed the limits of agreement reflect the issues with rating scale categories and definitions, item challenge, and usual vs. most dependent performance, as discussed.
Finally, by providing an effective algorithm for identifying implausible scoring, our study makes a significant contribution to examining similar functional data at inpatient rehabilitation settings. Our study found 11-27% discordance in self-care and 11-42% in transfer of records/scores between the two assessments. During 2017 (the period our data were collected), IRFs were paid for reporting Section GG scores only, not for patient performance on quality measures. As a result, incentive to ensure the accuracy of the Section GG scores may have been limited. In contrast, during the same period, facilities were paid in large part based on the case-mixed grouping to which a patient is assigned, mainly determined by the FIM scores at admission. This likely created a stronger incentive to ensure the accuracy of FIM scores and to report the most dependent performances at admission. Our algorithm provides an effective approach for identifying inconsistent functional data reporting for future studies that examine Section GG and FIM.
The transition to using Section GG and its effect on the IRF prospective payment system and on how patients are evaluated, treated, and reimbursed remains understudied. Evaluating response patterns between Section GG and FIM is necessary to set up baseline evidence to further determine the impact of transitioning into using the standardized functional data for quality reporting and payment across post-acute settings.
Study Limitations
CMS is constantly updating items in the standardized functional assessments across post-acute settings. Our data reflected the Section GG items in the IRF-PAI version 1.4 mandated in October 2016. While the most recent version of IRF-PAI is version 4.0, effective October 2020, we found no significant differences for the items analyzed in this study. Thus, our findings remain relevant in examining standardized functional outcomes. Second, we used only six of 17 Section GG mobility items to compose the transfer scores. Lacking information on the other mobility items may limit our understanding of the spectrum of mobility performance. Future study should conceptually and methodologically control for missing data in Section GG mobility items. Additionally, the UDSMR data were collected for billing purpose; we are unable to control for potential confounders (e.g., rater bias). This study also has intrinsic limitations, as do other studies analyzing retrospective data.19-21 For instance, we can only analyze variables provided by the UDSMR system. Therefore, case-mixed information and facility data may be limited. However, the UDSMR system captures a majority of IRF data across the nation.10
Conclusions and Implications
This study represents an early attempt to examine the transition to, and the use of, standardized functional data on quality reporting in IRFs. We found clinician-observed scores in Section GG and FIM mostly aligned regardless of evaluation timing or impairment, implying that most IRF providers are accurately reporting patients’ function relative to the given instructions of each assessment. Patients were scored more functionally independent using Section GG relative to FIM but the relatively higher scores simply reflected the difference in rating scales rather than “better” function. We suggest repeating this study for patients receiving services in other post-acute settings such as skilled nursing or home health, to determine whether Section GG can maintain the quality and outcomes of post-acute services and assist treatment planning across multiple settings.
Supplementary Material
Acknowledgments:
The authors would like to acknowledge Sarah Toombs Smith, PhD, a board-certified Editor in the Life Sciences (bels.org), at the Sealy Center on Aging, University of Texas Medical Branch, for her assistance in reviewing and editing the manuscript prior to our submission without salary compensation.
In memory of Carl V. Granger, MD, founder of the Uniform Data System for Medical Rehabilitation.
Funding Disclosure:
This study was funded, in part, by the National Institutes of Health (P2CHD065702, K01HD101589).
Abbreviations
- CMS
Centers for Medicare and Medicaid Services
- FIM®
Functional Independence Measure
- IMPACT Act
Improving Medicare Post-Acute Care Transformation Act of 2014
- IRF
Inpatient rehabilitation facilities
- IRF-PAI
Inpatient Rehabilitation Facility-Patient Assessment Instrument
- IRF-PPS
Inpatient Rehabilitation Facility Prospective Payment System
- MedPAC
Medicare Payment Advisory Commission
- SPADEs
Standardized patient assessment data elements
- UDSMR®
Uniform Data System for Medical Rehabilitation
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of Interest Disclaimer:
All authors declare that they follow the International Committee of Medical Journal Editors (ICMJE) authorship guidelines and have no potential conflicts of interest with respect to publish this paper.
References
- 1.U.S. Government. Public Law 113-185. Improving Medicare Post-Acute Care Transformation Act of 2014. Issued October 6, 2014. Retrieved October 27 2020 from https://www.govinfo.gov/content/pkg/PLAW-113publ185/pdf/PLAW-113publ185.pdf [Google Scholar]
- 2.IMPACT Act Standardized Patient Assessment Data Elements. Centers for Medicare and Medicaid Services; website. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Post-Acute-Care-Quality-Initiatives/IMPACT-Act-of-2014/-IMPACT-Act-Standardized-Patient-Assessment-Data-Elements.html. Updated January 11, 2019. Accessed June 4, 2019. [Google Scholar]
- 3.IMPACT Act of 2014 data standardization & cross setting measures. Centers for Medicare and Medicaid Services; website. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Post-Acute-Care-Quality-Initiatives/IMPACT-Act-of-2014/IMPACT-Act-of-2014-Data-Standardization-and-Cross-Setting-Measures.html. Updated December 11, 2018. Accessed June 4, 2019. [Google Scholar]
- 4.DeJong G Coming to terms with the IMPACT Act of 2014. Am J Occup Ther. 2016; 70(3): 7003090010p1–6. DOI: 10.5014/ajot.2016.703003. [DOI] [PubMed] [Google Scholar]
- 5.Granger CV, Hamilton BB, Linacre JM, Heinemann AW, Wright BD. Performance profiles of the functional independence measure. Am J Phys Med Rehabil. 1993. Apr;72(2):84–9. doi: 10.1097/00002060-199304000-00005. [DOI] [PubMed] [Google Scholar]
- 6.RTI International. Analyses to Inform the Use of Standardized Patient Assessment Data Elements in the Inpatient Rehabilitation Facility Prospective Payment System. Report Prepared for Center for Medicare and Medicaid Services (CMS), Assistant Secretary for Planning and Evaluation (ASPE). Issued March 1 2019. Retrieved April 10 2021 from https://www.hhs.gov/guidance/sites/default/files/hhs-guidance-documents/IRFPPSAnalysis2019RTI032219.pdf. [Google Scholar]
- 7.RTI International. Analysis of Crosscutting Medicare Functional Status Quality Metrics Using the Continuity and Assessment Record and Evaluation (CARE) Item Set. Prepared for Centers for Medicare and Medicaid Services, Division of Chronic & Post-Acute Care. November 2012. Report. RTI Project Number 0212050.022.000.001. Accessed on 1/10/2022: https://aspe.hhs.gov/reports/analysis-crosscutting-medicare-functional-status-quality-metrics-using-continuity-assessment-record-0 [Google Scholar]
- 8.Medicare Payment Advisory Commission (MedPAC). Report to the Congress: Medicare Payment Policy. Chapter 9: Payment issues in post-acute care. https://www.medpac.gov/wp-content/uploads/import_data/scrape_files/docs/default-source/reports/jun19_ch9_medpac_reporttocongress_sec.pdf Published June 2019. Accessed June 10, 2021. [Google Scholar]
- 9.Centers for Medicare & Medicaid Services. Medicare Program; Inpatient Rehabilitation Facility (IRF) Prospective Payment System for Federal Fiscal Year 2020 and Updates to the IRF Quality Reporting Program. Federal Register 42 CFR Part 412. Final Rules and Regulations Issued August 8 2019. Retrieved April 10 2021 from https://www.govinfo.gov/content/pkg/FR-2019-08-08/pdf/2019-16603.pdf. [Google Scholar]
- 10.Uniform Data System for Medical Rehabilitation (UDSMR). Accessed April 24, 2020. 1999-2021. UDSMR, a division of UB Foundation Activities, Inc.. http://udsmr.org. [Google Scholar]
- 11.Centers for Medicare and Medicaid Services. Inpatient Rehabilitation Facility Patient Assessment Instrument (IRF-PAI) and IRF-PAI Manual. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/IRF-Quality-Reporting/IRF-PAI-and-IRF-PAI-Manual Update July 20, 2021. Accessed August 2021. [Google Scholar]
- 12.Centers for Medicare & Medicaid Services. IRF-PAI. 2015 IRF-PAI Information. Updated Sections of the IRF-PAI Training Manual - Effective Oct 1, 2015. Section 3: Finalized the FIM Instrument. Accessed April 24, 2020. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/InpatientRehabFacPPS/IRFPAI [Google Scholar]
- 13.Mukaka MM. Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi Med J. 2012;24(3):69–71. [PMC free article] [PubMed] [Google Scholar]
- 14.Cohen J A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46. [Google Scholar]
- 15.Giavarina D Understanding Bland Altman analysis. Biochem Med (Zagreb). 2015. Jun 5;25(2):141–51. doi: 10.11613/BM.2015.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Gerke O Reporting Standards for a Bland-Altman Agreement Analysis: A Review of Methodological Reviews. Diagnostics (Basel). 2020. May 22;10(5):334. doi: 10.3390/diagnostics10050334. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Centers for Medicare & Medicaid Services. Inpatient Rehabilitation Facility PPS. IRF-PAI. 2017 & 2018 IRF-PAI Information (ZIP) [Zip File]. Accessed April 24, 2021. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/InpatientRehabFacPPS/IRFPAI [Google Scholar]
- 18.Linacre JM, Heinemann AW, Wright BD, Granger CV, Hamilton BB. The structure and stability of the Functional Independence Measure. Arch Phys Med Rehabil. 1994. Feb;75(2):127–32. [PubMed] [Google Scholar]
- 19.Abu-Arafeh A, Jordan H, Drummond G. Reporting of method comparison studies: a review of advice, an assessment of current practice, and specific suggestions for future reports. Br J Anaesth. 2016. Nov;117(5):569–575. doi: 10.1093/bja/aew320. [DOI] [PubMed] [Google Scholar]
- 20.Lieberman D Pitfalls of Using Administrative Data for Research. Dig Dis Sci. 2010;55(6):1506–1508. doi: 10.1007/s10620-010-1246-x [DOI] [PubMed] [Google Scholar]
- 21.Cheng HG, Phillips MR. Secondary analysis of existing data: opportunities and implementation. Shanghai Arch psychiatry. 2014;26(6):371–375. doi: 10.11919/j.issn.1002-0829.214171 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.