Development and Validation of the Nursing Home Minimum Data Set 3.0 Mortality Risk Score (MRS3)

Abstract

Background

To develop a score to predict mortality using the Minimum Data Set 3.0 (MDS 3.0) that can be readily calculated from items collected during nursing home (NH) residents’ admission assessments.

Participants

We developed a training cohort of Medicare beneficiaries newly admitted to United States NHs during 2012 (N = 1,426,815) and a testing cohort from 2013 (N = 1,160,964).

Methods

Data came from the MDS 3.0 assessments linked to the Medicare Beneficiary Summary File. Using the training dataset, we developed a composite MDS 3.0 Mortality Risk Score (MRS3) consisting of 17 clinical items and patients’ age groups based on their relation to 30-day mortality. We assessed the calibration and discrimination of the MRS3 in predicting 30- and 60-day mortality and compared its performance to the Charlson Comorbidity Index and the clinician’s assessment of 6-month prognosis measured at admission.

Results

The 30- and 60-day mortality rates for the testing population were 2.8% and 5.6%, respectively. Results from logistic regression models suggest that the MRS3 performed well in predicting death within 30 and 60 days (C-Statistics of 0.744 [95% confidence limit (CL) = 0.741, 0.747] and 0.709 [95% CL = 0.706, 0.711], respectively). The MRS3 was a superior predictor of mortality compared to the Charlson Comorbidity Index (C-statistics of 0.611 [95% CL = 0.607, 0.615] and 0.608 [95% CL = 0.605, 0.610]) and the clinicians’ assessments of patients’ 6-month prognoses (C-statistics of 0.543 [95% CL = 0.542, 0.545] and 0.528 [95% CL = 0.527, 0.529]).

Conclusions

The MRS3 is a good predictor of mortality and can be useful in guiding decision-making, informing plans of care, and adjusting for patients’ risk of mortality.

Nursing homes (NHs) provide care to patients with a wide spectrum of clinical care needs, many of whom are in their last year of life (1,2). Understanding the clinical profile and risk of mortality upon NH admission can stimulate discussions designed to establish patients’ goals of care, as well as guide end-of-life decision-making and advance care planning with patients and family members (3,4). However, predicting a patient’s survival is fraught with difficulty and notoriously imprecise. Recognizing this difficulty, many mortality risk scores and models have been developed to assist physicians outside of the NH setting (eg, GRACE (5), TIMI (6), Ranson’s criteria for pancreatic mortality (7), Surgical Apgar Scale (8–10), Veterans Aging Cohort Study (VACS) index (11,12), and POMPE-C Tool) (13,14). In addition to their utility for clinical decision-making, mortality risk scores and prediction models are also used to adjust for a patient’s risk of adverse outcomes and as inclusion criteria in randomized control trials (15–17). Furthermore, mortality risk scores can be useful in calibrating facility case mix, thereby allowing for more valid comparisons of outcomes across providers (18). As such, they are often used as adjusters in outcomes research and publicly reported quality measures (19).

While it is important to have valid scoring systems that predict mortality and signal a referral for advanced care planning or hospice, it is critically important that such systems are constructed from readily available data routinely obtained in clinical practice. The mandated NH resident assessment process, utilizing the Minimum Data Set version 3.0 (MDS 3.0) instrument (20), captures rich clinical and functional information about patients to inform clinical decision-making and care planning. Previous efforts to develop mortality risk scores for the NH setting have relied on sub-samples of patients (eg, those with advanced dementia) (21,22) and items derived from a previous version of the MDS resident assessment data (MDS 2.0) (23–30). However, many of the items in the MDS 2.0 used in previous mortality risk scores were eliminated in the new version of the MDS implemented in October 2010. As such, we have not had a mortality risk score for the NH population since the introduction of the MDS 3.0. The purpose of this study was to develop and evaluate a mortality risk score based upon a national population of newly admitted NH patients, which can be readily calculated from items collected during the admission assessment process using the MDS 3.0 in order to guide decision-making, better inform plans of care, and adjust for facility case-mix severity.

Method

Data

Data for this study come from the MDS 3.0 and the Medicare Master Beneficiary Summary File. The MDS instrument is a standardized, comprehensive assessment that must be completed for all persons who receive care in a Medicare and/or Medicaid-certified nursing facility. The MDS captures information about residents’ physical, cognitive, psychological, and psychosocial functioning, in addition to medical diagnoses, clinical symptoms, and any treatments or therapies prescribed and received. Assessments are conducted on all residents at admission and discharge, in addition to other time intervals (eg, quarterly and annually). While initially designed for care planning, the MDS is also used for payment, quality monitoring, and research. The MDS 3.0, the current version of the assessment instrument, has been extensively validated (31–38).

From the MDS 3.0, we obtained the first full assessment after its introduction (October 1, 2010) for each individual. We linked individual’s first MDS assessment from 2012–2013 to the 2012–2014 Medicare Master Summary Beneficiary File to identify NH residents who were Medicare beneficiaries and to determine beneficiaries’ dates of death. Importantly, the Medicare Master Beneficiary Summary File allows us to identify NH residents who die, regardless of where or when.

Study Samples

With the MDS 3.0, we identified a cohort of Medicare beneficiaries who were newly admitted to the NH in 2012 and 2013 with no NH stay since the introduction of the MDS 3.0 in October 2010. We split the sample to derive “training” and “testing” populations. The “training” population used to develop the prediction model consisted of all Medicare beneficiaries newly admitted to NHs during 2012 (N = 1,426,815). The population used to “test” the sensitivity and specificity of the prediction model included all Medicare beneficiaries newly admitted to NHs during 2013 (N = 1,160,964).

Study Variables

Measures of mortality were created to examine whether the new resident admitted to the NH died within 30 days and within 60 days of the assessment. Thirty-day mortality was used to develop the score and we tested the performance of the score using the 30- and 60-day measures of mortality.

To develop the score, the research team, including a practicing geriatrician and palliative care physician, identified clinical indicators that were reported across assessments in the MDS 3.0 believed to be related to the likelihood of death. A total of 46 candidate measures was selected and grouped into three categories: 1) Cognitive and Physical Function, 2) Diagnoses, and 3) Health Conditions (see Supplementary Table 1, for a complete list of measures).

For ease of calculation, we dichotomized ordinal variables measuring cognitive and physical function. The cognitive and physical function variables were dichotomized into categories representing severe functional limitation (a 28-point Activities of Daily Living [ADL] (39) score greater than or equal to 23) and severe cognitive impairment (a Cognitive Function Scale score of 4) (37). We collapsed variables that measured a similar construct. For example, three shortness of breath measures (lying flat, sitting, and with exertion) were collapsed into one measure to capture whether or not patients experienced any of these symptoms. We also created a composite measure to indicate if a patient experienced any swallowing disorder, and a composite measure to indicate if a patient had at least one stage 3, stage 4, or unstageable pressure ulcer.

Analyses

Approach to develop a mortality risk score

With the training population, we conducted bivariate analyses, with facility fixed effects to remove any influence of the facility on mortality, predicting death at 30 days for each of the measures that were candidates for inclusion in the score. Only statistically significant variables (p < .0001) with odds ratios greater than 1.2 were retained for the multivariate models. A total of 21 clinical indicators were entered into a backwards, stepwise regression model predicting death within 30 days. Variables with a p-value greater than .2 were dropped from the model. The remaining 17 measures were tested in a logistic regression model with facility fixed effects predicting death within 30 days. We calculated the adjusted odds ratio for each variable to estimate the risk of death for a given variable controlling for coexistent cognitive and physical function, diagnoses, and health conditions.

We created a weighted score, the MDS 3.0 Mortality Risk Score (MRS3), to represent the number and seriousness of a patient’s clinical indicators in addition to a patient’s age. Specifically, the adjusted odds ratios were employed as weights for the different clinical indicators. Indicators with an odds ratio of ≥1.2 and <1.5 were assigned a weight of 1 (n = 5); indicators with an odds ratio of ≥1.5 and <2.5 a weight of 2 (n = 7); indicators with an odds ratio of ≥2.5 a weight of 3 (n = 5). Age groups were assigned the following weighted values: Ages <65 = 1, 65–74 = 2, 75–84 = 3, 85–94 = 4, and 95+ = 5. (See Table 1, for a list of indicators and weighted values). Weighted values were summed for each individual to create a score that could range from 1 to 39 with 1 being least and 39 being the highest risk of mortality. However, given the very small number of individuals with a score greater than 20 (less than 0.001%), we censored the higher values of the score so that any above 20 were set to 20.

Table 1.

Scoring the MRS3

Component	MRS3 Value
Age <65 years	1
Age 65–74 years	2
Age 75–84 years	3
Age 85–94 years	4
Age 95+ years	5
Severe physical impairmenta	3
Acute change in mental status	3
Shortness of breath, any	3
Dehydrated	3
Severe cognitive impairmentb	3
Internal bleeding	2
Swallowing disorder, any	2
Vomiting	2
Fever	2
Unplanned weight loss	2
Heart Failure	2
Worst pressure ulcer, slough, or necrotic	2
Pneumonia	1
Any falls since admission or prior assessment	1
Viral hepatitis	1
Hyponatremia	1
At least one stage 3, 4, or unstageable pressure ulcer	1

Notes: Weighted values are summed to create the MDS 3.0 Mortality Risk Score (MRS3) that could range from 1 to 39 with 1 being least and 39 being the highest risk of mortality. A MRS3 score of 8 or higher is associated with a 5.4 times increase in the odds of 30-day mortality compared to a score of 7 or lower (OR = 5.408, CI = 5.726, 5.543) and a 4.2 times increase in the odds of 60-day mortality (OR = 4.215, CI = 4.140, 4.291).

^aActivities of Daily Living Score greater than or equal to 23 (range 0–28, 0 = total independence, 28 = total dependence).

^bCognitive Function Scale score of 4 (range 1–4; 1 = intact cognition, 4 = severe cognitive impairment).

Approach to External Validation

We assessed the accuracy of the score with measures of discrimination and calibration. To do this, we conducted logistic regression analyses with the MRS3 to predict the odds of death at 30 and 60 days. We tested the discrimination of the score using the area under the receiver operating characteristic (ROC) curve (or concordance statistic, C). An area under the ROC curve of 0.5 indicates no discrimination, whereas a C-Statistic of 1 represents perfect discrimination. For comparison of the score’s discrimination to other established measures, we conducted separate logistic regression analyses using the Charlson Comorbidity Index and the 6-month prognosis indicator in the MDS assessment to predict death within 30 and 60 days. The Charlson Comorbidity Index is derived using the list of active diagnoses that are reported in the MDS 3.0 assessment. The 6-month prognosis variable is an indicator that is completed by a clinician and reported in the MDS 3.0 assessment. This variable is used to identify whether or not the patient has a life limiting illness or condition and is not expected to survive longer than 6 months. We also tested performance of the MRS3 in terms of calibration. Calibration refers to how closely the predicted mortality risk agrees with observed mortality. We assessed calibration of the MRS3 by plotting the observed proportions versus predicted probabilities of death at 30 and 60 days and by tabulating the 30-day mortality ratio (predicted to actual) for each value of the MRS3 score. We used the Youden Index (J) (40) to identify a cut point that discriminates between low versus high risk of mortality and examined its performance in logistic regression models predicting 30- and 60-day mortality.

Sensitivity Analyses

In sensitivity analyses, we tested a logistic regression model predicting mortality using the detailed coefficients for each indicator.

Results

Table 2 describes the training sample of newly admitted NH patients in terms of their demographic, clinical, and functional characteristics as well as the observed 30-, 60-day, and 1-year mortality rates. The mortality rate for the testing population was 2.9% within 30 days and 5.7% within 60 days. The mean MRS3 score was 4.74 (SD = 2.49) with a median of 4. Over one-third of the testing population was 85 years or older and the most frequently reported indicator was shortness of breath (18.34%) followed by heart failure (18.14%). The bivariate relationship with mortality for each variable included in the MRS3 can be found in the Supplementary Table 1.

Table 2.

Descriptive Characteristics of Sample

	Training Population: New Admissions, 2012	Testing Population: New Admissions, 2013
Number of observations (N)	1,426,815	1,160,964
30-day mortality (%)	5.72	2.88
60-day mortality (%)	9.76	5.69
365-day mortality (%)	25.55	21.94
Age groups
Age: <65 (%)	9.08	9.17
Age: 65–74 (%)	21.84	22.3
Age: 75–84 (%)	35.1	34.78
Age: 85–94 (%)	30.46	30.26
Age: 95+ (%)	3.53	3.49
Demographics
Male	36.88	31.10
American Indian/Alaska  Native	0.34	0.30
Asian	1.37	1.54
Black	9.04	9.62
Hispanic or Latino	3.76	4.09
Native Hawaiian or Pacific  Islander	0.25	0.28
White	82.81	80.88
Cognitive and physical function
Severe cognitive impairmenta (%)	3.62	3.01
Acute change in mental status (%)	2.14	1.59
Severe Physical Impairmentb (%)	7.95	6.47
Any falls since admission or  prior assessment (%)	6.99	7.2
Diagnoses
Heart failure (%)	18.20	18.14
Pneumonia (%)	9.1	8.74
Hyponatremia (%)	2.24	2.3
Health conditions
Dehydrated (%)	0.42	0.33
Fever (%)	2.79	2.34
Internal bleeding (%)	0.93	0.82
Vomiting (%)	2.59	2.29
Shortness of breath, any (%)	19.38	18.34
Shortness of breath when lying  flat (%)	9.04	8.53
Shortness of breath when  sitting (%)	5.64	5.01
Shortness of breath with  exertion (%)	16.08	15.23
Swallowing disorder, any (%)	5.46	4.76
Choking (%)	2.35	2.01
Painful swallowing (%)	2.76	2.48
Holding food (%)	1.19	1.01
Lost liquid (%)	0.53	0.43
Unplanned weight loss (%)	4.47	4.05
At least one stage 3, 4, or  unstageable pressure ulcer (%)	3.34	3.44
Worst pressure ulcer, Slough or  Necrotic (%)	3.15	2.99
Worst pressure ulcer:   Slough (3) (%)	1.51	1.5
Worst pressure ulcer:   Necrotic (4) (%)	1.63	1.48

Notes: ^aCognitive Function Scale score of 4 (Range 1–4; 1 = intact cognition, 4 = severe cognitive impairment).

^bActivities of Daily Living Score greater than or equal to 23 (Range 0–28, 0 = total independence, 28 = total dependence).

Results from our logistic regression models suggest that the MRS3 performed well in predicting death within 30 and 60 days (with C-statistics of 0.744 [95% CL = 0.741, 0.747] and 0.709 [95% CL = 0.706, 0.711]), respectively (see Table 3). When comparing the performance of the MRS3 models in predicting mortality to those using either the Charlson Comorbidity Index or the clinicians’ assessment of 6-month prognosis, we found that the MRS3 was superior (see Table 3). Specifically, the C-statistic for the model using the Charlson Comorbidity Index to predict 30-day mortality was 0.611 (95% CL = 0.607, 0.615), and it was even lower for the model using a clinician’s assessment of 6-month prognosis (C = 0.543, 95% CL = 0.542, 0.545).

Table 3.

Performance of MRS3 in Predicting Death among the Testing Population Compared to the Charlson Comorbidity Index and the 6-Month Prognosis Indicator

	30-Day Mortality	60-Day Mortality
MRS3	0.744 (0.741, 0.747)	0.709 (0.706, 0.711)
Charlson Comorbidity Index	0.611 (0.607, 0.615)	0.608 (0.605, 0.610)
6-Month Prognosis	0.543 (0.542, 0.545)	0.528 (0.527, 0.529)

Notes: Values represent the C-statistics and 95% Confidence Intervals.

Figure 1 plots the observed versus predicted probability of death for each value of the MRS3 score in the testing sample. The smooth, 45-degree line represents a perfect prediction model where the observed value and predicted value are the same, while the shapes represent the actual probability of death at each time period at each value of the score. Because the models calibrate to the data well, the shapes and the line lie close to each other. As can be seen in Figure 1, the MRS3 performs well in predicting mortality with the greatest deviations seen among the highest values, which have fewer observations.

Figure 1.

Calibration curve for observed versus predicted probability of death at 30 and 60 days. Notes: Values plotted are the average predicted and average observed probability for each level of the MRS3 (1–26).

We present the observed and predicted 30-day mortality rates, by MRS3 value, in Table 4. Results suggest that individuals with a low MRS3 have a very low observed rate of death in the short term. For example, of the 16% of individuals with an MRS3 score of 1 or 2, less than 1% had died within 30 days. In contrast, 34% of individuals with an MRS3 score of 15 or higher had died within 30 days.

Table 4.

Observed and Predicted 30-Day Mortality, by MRS3 Score

MRS3 Score	N	Predicted Risk of 30-Day Mortality	Observed 30-Day Mortality Rate	Predicted 30-Day Mortality	Observed 30-Day Mortality	Ratio of Observed to Expected Mortality
1	46,255	0.59	0.58	271	267	1.01
2	124,129	0.79	0.79	986	976	1.01
3	199,554	1.08	0.93	2,150	1,864	1.15
4	207,875	1.46	1.28	3,035	2,658	1.14
5	124,279	1.98	2.12	2,456	2,628	0.93
6	113,835	2.67	2.89	3,039	3,287	0.92
7	80,065	3.60	3.90	2,880	3,125	0.92
8	51,509	4.83	5.05	2,488	2,600	0.96
9	38,639	6.46	6.72	2,496	2,598	0.96
10	21,656	8.59	8.83	1,860	1,912	0.97
11	11,687	11.33	11.25	1,325	1,315	1.01
12	7,157	14.81	15.15	1,061	1,084	0.98
13	4,072	19.13	17.68	780	720	1.08
14	2,134	24.35	21.46	520	458	1.13
15	1,361	30.46	27.55	415	375	1.11
16	785	37.34	34.65	294	272	1.08
17	439	44.77	35.54	197	156	1.26
18	209	52.44	43.54	110	91	1.20
19	136	60.01	43.38	82	59	1.38
20	137	67.12	59.85	92	82	1.12

We identified a cut point of 7 to indicate low versus high risk of mortality. A MRS3 score of 8 or higher was associated with a 5.4 times increase in the odds of 30-day mortality compared to a score of 7 or lower (OR = 5.408, CI = 5.726, 5.543) and a 4.2 times increase in the odds of 60-day mortality (OR = 4.215, CI = 4.140, 4.291).

In sensitivity analyses, we found a relatively small loss of precision by working with the MRS3 (a simpler model and scoring system) that we developed as opposed to the logistic regression model predicting mortality using the detailed coefficients for each indicator (results from the fully weighted model can be found in the Supplementary Table 3).

Discussion

As NHs increasingly care for more clinically complex and frail patients, a tool to enable clinicians to identify patients who may be close to the end of life could potentially serve as a trigger for advance care planning. Furthermore, with the variation in the types of patients admitted to NHs, a score, or set of items, that strongly predict mortality is needed to adequately adjust for patients’ risk and facility case mix. Results from our development and validation study suggest that the MRS3 is a useful new tool to predict 30-day mortality and adjust for patient risk using the MDS 3.0.

This mortality risk score, developed for patients newly admitted to the NH, is of benefit to clinicians and health services researchers for multiple reasons. First, the MRS3 can be a helpful tool not only to provide prognosis information that can aide in guiding care, but it can also assist with planning for care. For example, a patient’s MRS3 score can serve as a signal for end of life and advanced care planning discussions with the patient and their family. Second, the MRS3 can inform goals of care, particularly for patients with high MRS3 scores and predicted to be near the end of life. Where to establish the cut point to identify patients who should be approached about advance care planning or for whom a hospice discussion might be necessary is an issue that will require further investigation. However, our results suggest that a score of 8 and above is associated with over five times the increased odds of mortality and might serve as this cut point. While this cut point maximizes the predictive value, the decision of the cut point must take into account the goals. For example, health providers may want to focus on sensitivity rather than specificity to ensure all that would benefit from advance care planning are offered the chance to state their wishes prior to a period of possible future mental incapacity. While the way the score was developed maximizes its clinical applicability, it is also useful to researchers who will benefit from a score that can be readily calculated and used with the MDS 3.0. This score has the potential to be used as adjustment for mortality in examining the impact of a new treatment or comparing quality across facilities.

As with the development of any score, there are some limitations to note. First and foremost, the MRS3 was designed as a clinical aid and clinicians may have additional knowledge that better informs prognostication (eg, knowing that a patient is stopping dialysis). This score was not designed to take the place of a clinical opinion and training for appropriate prognostication and therefore should be approached and interpreted as such. Secondly, this score is not based solely on biological-based markers or objective physiological measures; the MRS3 relies on subjective clinical appraisals for at least some of the indicators. However, MDS coordinators and those who complete the assessments have been trained to obtain this information for care planning, payment, and public reporting; therefore, it can be argued that there is some standardization and objectivity in collecting this information and the MDS has been validated for its accuracy in previous studies (31–34,36,37). In addition, there are several items that comprise the MRS3 that are potentially resolvable or treatable. Therefore, future research should examine the use of the MRS3 as a time-dependent covariate to determine the impact of changes in the MRS3 score over time on improving the prediction of mortality.

In addition, there were limitations of our approach to validation of the MRS3. First, the score was developed and tested on an admission cohort of NH patients. Therefore, the score may not be appropriate or as accurate in predicting mortality for long-stay NH residents as previous work suggests that long-stay residents tended to be older, male, white, unmarried, low-income subsidy recipients, have multiple comorbidities, and have higher rates of mortality than residents with shorter stays (41). Future work is needed to develop a separate mortality risk score for other cohorts of NH residents such as long-stay residents and those readmitted to the facility. In addition, the score relied only on items present on every assessment. There are a number of items present on the Federal OBRA-required full admission assessment that are not included on the 5-day prospective payment assessment that may have enhanced the performance of this score (ie, Cancer, Cirrhosis, and Renal Insufficiency, Renal Failure, or End-Stage Renal Disease).

Despite these limitations, there are many strengths to the development and testing of this score that are worthy of mention. First, this score was developed to predict mortality with population-based administrative information for the national population of new admissions to NHs. Second, it was validated with an external, national population of new admissions to NHs in the following year. Third, this score is comprised of real-time data on health conditions and functioning in the MDS 3.0 that are routinely collected for every patient in every facility by trained staff, unlike other mortality scores or frailty indices requiring information from various data sources including surveys and lab results (42–44). Fourth, we were able to model a patient’s risk of death, regardless of location at time of death, for all Medicare beneficiaries admitted to NHs. Fifth, the transformation of a logistic regression model to an index provides greater clinical utility for decision-making and care planning. Finally, because the score is not a disease-specific measure, unlike many mortality risk scores with good performance characteristics, it allows for prognostication and risk adjustment across different diseases and risk profiles.

In summary, the MRS3 is a potential useful tool for identifying mortality risk among patients admitted to NHs. The MRS3 provides valuable information that clinicians, service providers, researchers, and policy makers can use to inform decisions related to the care of patients admitted to NHs without imposing additional workload or assessment burden.

Funding

This work was supported by the National Institute of Aging at the National Institutes of Health (P01AG027296); and the U.S. Department of Veterans Affairs Health Services Research and Development Service (CDA14-422 to K.S.T.). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the National Institutes of Health, or the U.S. government.

Conflict of Interest

Vince Mor is the Chair of the Independent Quality Committee at HCR ManorCare and am compensated for this service. HCR is a provider of short-term, post-hospital services, and long-term care. The company has a network of more than 500 nursing and rehabilitation centers, assisted living facilities, outpatient rehabilitation clinics, and hospice and home health care agencies.

Vince Mor is a paid consultant to NaviHealth, Inc. and chair their Scientific Advisory Board. NaviHealth is wholly owned by Cardinal Health. The company offers post-acute care (PAC) management and services to more than 1.5 million beneficiaries in all regions of the country through its partnerships with health plans and health systems.

Vince Mor is a former Director at PointRight, Inc. While he no longer provides any services or holds any positions at PointRight, he still has a small amount of equity in the company. PointRight is a private company based in Cambridge, MA. It provides predictive analytics solutions to thousands of post-acute providers, long-term care providers, hospitals, payers and insurance organizations.