Agreement of Minimum Data Set 3.0 Depression and Behavioral Symptoms with Clinical Diagnosis in a Nursing Home

Abstract

Objectives:

While Minimum Data Set (MDS) 3.0 has adopted Patient Health Questionnaire (PHQ)-9 to screen for depression and rephrased language for behavioral symptoms among nursing home residents, it remains unclear how well the assessment data agree with external sources. This study aimed at assessing the agreement of these two mental health conditions between MDS 3.0 and medical records.

Design:

Using a retrospective review of MDS 3.0 linked to medical diagnostic records between October 2010 and November 2017, we included residents with at least one quarterly or short-term (day 30 or day 60) MDS-3.0 assessment of depression PHQ-9 (n=446) or behavioral symptoms (including psychosis, physical, verbal or other aggressive behaviors) (n=460). For each resident of each cohort, we randomly selected an eligible MDS-3.0 depression and behavioral symptom assessment and compared against the respective medical diagnoses recorded within 30 days before the MDS-3.0 assessment.

Results:

Percent agreement was high for depression (90.1%) and behavioral symptoms (89.3%). The negative agreement was also high for depression (94.8%) and behavioral symptoms (94.3%), while the positive agreement was low for both conditions (4.3% and 10.9%).

Conclusion:

MDS-3.0 assessed depression and behavioral symptoms had a high overall and negative agreement, but a low positive agreement with clinician diagnoses. MDS 3.0 data may be useful in ruling out depression and behavioral symptoms. Confirmation of the findings in a representative sample of nursing homes is warranted.

Introduction

The minimum data set (MDS), a clinical instrument mandated in every Medicare- and Medicaid-certified nursing home, accounts for almost all (95%) of nursing home facilities in the United States (US).(Rahman and Applebaum, 2009) Under Omnibus Budget Reconciliation Act (OBRA) and Medicare’s nursing home prospective payment system (PPS), all participating facilities are required to assess their residents’ health conditions at admission, regular intervals during Medicare-covered short-term stays, quarterly thereafter, and upon discharge or a significant change in health status for reimbursement and care planning purpose.(The Centers for Medicare and Medicaid Services) In addition, MDS has become the primary source to gauge the quality of nursing home care.(Rahman and Applebaum, 2009). MDS assessments have been used by the Center for Medicare and Medicaid Services (CMS) to construct Nursing Home Quality Measures to generate a star rating system to help consumers select and compare nursing home facilities.(The Centers for Medicare and Medicaid Services)

Central to concerns about nursing home quality is the management of residents with depression and behavioral symptoms. Depression is reported in up to 50% of nursing home residents,(Levin et al., 2007) and behavioral symptoms are commonly seen (Lyketsos et al., 2002) and represent one of the major reasons for patients with Alzheimer’s disease and related dementia (ADRD) to be admitted to nursing homes.(Gaugler, Yu, Krichbaum, & Wyman, 2009) Improving upon MDS 2.0 version, MDS 3.0 implemented in October 2010 was designed for obtaining resident self-reports through an interview with a validated mental health tool—Patient Health Questionnaire (PHQ)-9 to screen for depression, supplemented with nursing staff observation for patients with difficulty in verbal communication.(Saliba, DiFilippo, et al., 2012) MDS 3.0 behavioral symptoms, solely relying on nursing staff assessment, were improved through rephrasing in non-stigmatizing language, adding operational definitions of psychosis for clarity, and regrouping symptoms to make them consistent with recognized factors.(Saliba and Buchanan, 2012) Both MDS-3.0 depression and behavioral items have shown excellent nurse-to-nurse interrater reliability (kappa >0.90).(Saliba and Buchanan, 2008) MDS-3.0 PHQ-9 measure showed a higher validity than MDS- 2.0 depression measure when both were compared to standard geriatric depression assessments.(Saliba, DiFilippo, et al., 2012) The MDS 3.0 PHQ-9 was also validated in nursing home settings, showing a high specificity (90% or higher) although poor sensitivity (11% or lower) compared with the presence of depression diagnosis in Medicare claims.(Belanger, Thomas, Jones, Epstein-Lubow, & Mor, 2019)

Despite the potential improvements on MDS 3.0 depression and behavioral symptoms, it remains unclear how well the assessment data agree with external sources. This study aims to examine the agreement of these two conditions assessed in MDS 3.0 by residents or registered nurse staff against clinical diagnoses assessed by a geriatric team and recorded in medical charts. The geriatric team is dependent from the nursing home facility in assessing residents’ depression and behavioral symptoms.

Methods

A retrospective review of electronic medical records (EHRs) was conducted in a regional, federally-certified nursing home. The EHR contains both resident-level MDS 3.0 assessments and medical diagnostic records. Individual de-identified data were extracted by research team members. The study was approved by the institutional review board.

Study sample

Two separate condition cohorts were created based on having at least one quarterly or short-term (day 30 or day 60) MDS-3.0 assessment for 1) depression and 2) behavioral symptoms screening between October 1, 2010, and November 30, 2017. In each cohort, we reviewed the presence or absence of the respective clinical condition indicated by its ICD-9 or ICD-10 codes within 30 days before the date of a randomly selected eligible MDS assessment for each resident. The choice of days before an MDS 3.0 assessment is because MDS 3.0 assessed depression and behavioral symptoms in a recall period (14- and 7-day, respectively) prior to each assessment. We used the prior 30 days, rather than the recall periods, to provide a better capture of clinical diagnoses that were made during regular monthly clinic visits by the geriatric team.

We excluded MDS-3.0 admission, discharge, and any assessments (5-day and 14-day) administrated within the first month of post-acute nursing home care following hospitalizations, during which the studied clinical conditions might not be fully detected and thus diagnosed by the nursing home’s geriatrician team. Residents with incomplete MDS information on the studied conditions (n=17 for depression cohort; n=3 for behavioral cohort) were also excluded in each cohort (Figure 1).

Figure 1.

Flow chart of sample selection

MDS-3.0 depression and behavioral symptoms assessment

MDS-3.0 depression was measured by resident self-assessment of mood status in the past 14 days using PHQ-9, a validated tool to detect major depressive disorder with high sensitivity and specificity (both >85%).(Kroenke, Spitzer, & Williams, 2001) Depression status of non-verbal residents was measured by staff-assessed PHQ-9. Each of the PHQ-9 items scored from 0 (not at all) to 3 (nearly every day), with a total score ranging from 0 to 27. Patients whose PHQ-9 score was ≥10 were classified as having moderate-to-severe depression; otherwise, they had no or mild depression.(Kroenke, et al., 2001)

In MDS 3.0, nursing staff assessed the presence of four common behavioral symptoms during the past 7 days, including (1) psychosis; (2) physical behavioral symptoms directed toward others; (3) verbal behavioral symptoms directed toward others; and (4) other behavioral symptoms not directed toward others.(Gruber-Baldini, Boustani, Sloane, & Zimmerman, 2004) Residents were considered having a behavioral symptom if they exhibited any of the four behaviors; otherwise, no behavioral symptom.

Clinical diagnosis

For all eligible residents, research team members reviewed the presence or absence of clinical diagnosis documented in the nursing home’s EHR for major depressive disorder (ICD-9 codes: 296.2x, 296.3x, 300.4, 309.1 and 311.xx)(Crystal, Sambamoorthi, Walkup, & Akincigil, 2003), which demonstrated a high sensitivity compared to a PHQ-9 score of ≥10.(Kroenke, et al., 2001; Phelan et al., 2010) Data were also extracted on behavioral symptoms due to dementia (ICD-9 codes: 290.x, excluding 290.0, 290.10, and 290.40), transient mental disorders (i.e., delirium) (293.x), persistent mental disorders (294.x, excluding 294.0 and 294.10), delusional disorders (297.x), other nonorganic psychoses (298.x), unspecified disturbance of conduct (312.9), and altered mental status (780.97). The ICD-9 codes listed for both conditions were converted to corresponding ICD-10 diagnoses.

Resident characteristics

Characteristics included demographics, pain severity, and cognitive, physical and mental health status, assessed based on the randomly selected MDS 3.0 assessment during the study period. Demographics included age, sex, and race. MDS 3.0-assessed pain was classified as no-to-mild and moderate-to-severe pain.(Edelen and Saliba, 2010) Cognitive function measured by Brief Interview for Mental Status was categorized into intact/borderline (13–15), moderate impairment (8–12), and severe impairment (0–7) groups.(Saliba, Buchanan, et al., 2012) Physical dependence measured by the 7-item activities of daily living (ADL) was categorized into no (0–7), mild (8–14), moderate (15–21), and severe (≥21) dependence.(Carpenter, Hastie, Morris, Fries, & Ankri, 2006) We also described two factors associated with depression and behavioral symptoms: 1) receiving a diagnosis of ADRD based on its active diagnosis flag on MDS 3.0 Section I; and 2) having self-reported or staff-assessed PHQ-9 (for depression cohort only). Finally, in each disease cohort, we provided the following variables measured during the study period, including 1) average nursing home stay per resident, 2) average number of MDS-3.0 assessment per resident, 3) proportions of residents with any positive MDS-3.0 depression screen(s) and behavioral screen(s), and 4) proportions of residents with any clinical diagnosis for depression and behavioral symptoms.

Statistical analysis

At the resident level, we examined the agreements by comparing the presence or absence of major depressive disorder and behavioral symptoms assessed in the randomly selected MDS 3.0 assessment versus medical diagnoses that occurred within 30 days prior to the assessment. Because no single statistic adequately captures agreement,(Chen, Faris, Hemmelgarn, Walker, & Quan, 2009) we reported three agreement indices with corresponding 95% confidence intervals —1) percent agreement; 2) positive agreement (PA), defined as the average proportion of both MDS 3.0 and medical records in identifying positive cases of a disease condition; and 3) negative agreement (NA), defined as the average proportion of both MDS 3.0 and medical records in identifying non-cases of a disease condition. The calculation of both PA and NA was corrected for chance as detailed by Cicchetti and Feinstein.(Cicchetti and Feinstein, 1990) Each agreement statistic ranged from 0 to 100%, with a higher percentage indicating a higher degree of agreement.

We used PA and NA, rather than Cohen’s Kappa, because a kappa value would be artificially low despite a high observed agreement when there is an imbalance in marginal distributions.(Feinstein and Cicchetti, 1990). In our nursing home sample, the ratios of total positive versus negative depression cases confirmed by MDS 3.0 (23/423) and by medical diagnosis (23/423) are far away from 1, an indication of imbalance (Table 2). The imbalance was also observed for behavioral symptoms. To overcome the Kappa’s paradox, Cicchetti and Feinstein suggested reporting PA and NA indexes, which are analogous to sensitivity and specificity concepts in assessing diagnostic accuracy against a gold standard. (Cicchetti and Feinstein, 1990) Finally, we performed subgroup analyses to determine differences in the percent agreement between the MDS 3.0 and medical diagnostic codes using the chi-square test or Fisher’s exact test if the number of samples was small. P< .05 were considered statistically significant, and all tests were 2-sided. All analyses were performed using SAS 9.4 (SAS Institute, Cary, NC).

Table 2.

Agreement of Major Depressive Disorder and Behavioral Symptoms Assessed in Minimum Data Set (MDS) 3.0 versus Medical Diagnostic Records at the Resident level

Medical diagnoses	MDS-3.0* Assessed Major Depressive Disorder (MDD)
	% (No. of Residents with MDS-3.0 Assessment / Total No. of Residents)		Percentage (%) Agreement
	Yes (PHQ-9 ≥10)	No (PHQ-9 <10)	General (95% CI)	Positive (95% CI)	Negative (95% CI)
Prior 30 days
Yes	0.2 (1/446)	4.9 (22/446)	90.1 (87.4,92.9)	4.3 (−1.5,10.2)	94.8 (93.3,96.3)
No	4.9 (22/446)	89.9 (401/446)
Medical diagnoses	MDS-3.0* Assessed Behavioral Symptoms*
	% (No. of Residents with MDS-3.0 Assessment / Total No. of Residents)		Percentage (%) Agreement
	Presence of any behaviors	Absence of any behaviors	General (95% CI)	Positive (95% CI)	Negative (95% CI)
Prior 30 days
Yes	0.7 (3/460)	2.4 (11/460)	89.3 (86.5, 92.2)	10.9 (2.7, 19.1)	94.3 (92.8, 95.9)
No	8.3 (38/460)	88.7 (408/460)

Abbreviations: MDS, Minimum Data Set; PHQ-9, Patient health questionnaire

^*A randomly selected MDS assessment per resident was used.

^†including the presence of psychosis (hallucinations or delusions), physical and verbal behavior directed toward others, and other behavioral symptoms not directed toward others

Results

We identified 446 eligible nursing home residents with at least one valid MDS-3.0 assessment of PHQ-9 for depression screening and 460 residents for behavioral symptoms. Of residents with PHQ-9 depression or behaviors assessment, most were female (68%) and whites (95%), with a mean age of 85 ±7.8 years old (Table 1). Based on the randomly selected MDS 3.0 assessment in both disease cohorts, one-third (33%) had moderately to severely impaired cognitive function, one in ten (13–16%) had moderate-to-severe physical dependence, and one-third (33–34%) had moderate-to-severe pain. About a quarter (24%) of the eligible nursing home residents in both disease cohorts had an active diagnosis of ADRD. The majority (94.2%) of residents had self-reported MDS-3.0 depression status. During the study period, residents in both disease cohorts had an average nursing home stay of 9 months (272–277 days). Around one in five had ≥1 positive MDS-3.0 depression (21–22%) or behavioral symptoms (19%), which were higher than the prevalence (14%) of the diseases diagnosed at any time during the nursing home stay.

Table 1.

Characteristics of eligible nursing home residents from October 1, 2010 to November 30, 2017

Characteristics*	Sample for MDS 3.0 PHQ-9 Depression Assessment	Sample for MDS 3.0 Behavioral Symptom Assessment
	N=446 (100%)	N=460 (100%)
Age, y
Mean (SD)	85.3 (7.8)	85.5 (7.8)
≤ 74	46 (10.3)	47 (10.2)
75–84	133 (29.8)	137 (29.8)
≥85	267 (59.9)	276 (60.0)
Female sex	305 (68.4)	312 (67.8)
Race
Whites	426 (95.5)	440 (95.7)
Non-whites	20 (4.5)	20 (4.3)
BIMS cognitive score
Intact/borderline (13–15)	297 (66.6)	305 (66.3)
Moderate impairment (8–12)	81 (18.2)	89 (19.3)
Severe impairment (0–7)	68 (15.2)	66 (14.3)
ADL dependence
No (0–7)	192 (43.0)	185 (40.2)
Mild (8–14)	195 (43.7)	201 (43.7)
Moderate to severe (15–28)	59 (13.2)	74 (16.1)
Pain
No	104 (23.3)	112 (24.3)
Mild	191 (42.8)	184 (40.0)
Moderate to severe	146 (32.7)	157 (34.1)
ADRD (Yes)	105 (23.5)	112 (24.3)
Source of PHQ-9 assessment
Having a self-reported assessment	420 (94.2)	443 (96.3)
Having a staff-assessed	26 (5.8)	17 (3.7)
Length of NH stay in days§ (Mean, Median)	277, 60	272,60
Number of MDS assessments§ (Mean, Median)	3, 1	3,1
Having ≥ 1 positive MDS-3.0 depression §	94 (21.1)	102 (22.2)
Having ≥ 1 positive MDS-3.0 behavioral symptoms §	83 (18.6)	88 (19.1)
Having ≥ 1 diagnosis for depression §	63 (14.1)	66 (14.3)
Having ≥ 1 diagnosis for behavioral symptoms §	60 (13.5)	67 (14.6)

Abbreviations: MDS, Minimum Data Set; PHQ-9, Patient health questionnaire; BIMS, brief interview for mental status; ADL, activities of daily living; ADRD, Alzheimer’s disease and related dementia

^*All characteristics (except for those with superscript ^§) were measured based on the randomly-selected observed MDS 3.0 assessment used for agreement analysis.

^†including the presence of psychosis (hallucinations or delusions), physical and verbal behavior directed toward others, and other behavioral symptoms not directed toward others

^§Variables were measured during the study period.

We reviewed medical records within 30 days before a randomly selected MDS assessment for each resident in the cohort for depression (n=446) and behavioral symptoms (n=460) (Table 2). The proportion of residents with disease was similar between MDS 3.0 and diagnostic data for depression (5.2% [23/446] vs 5.2% [23/446]) but was higher in MDS 3.0 than in diagnostic data for behavioral symptoms (9.1% [42/460] vs. 3.0% [14/460]). MDS 3.0 showed a good percent agreement (90.1%) and negative agreement (NA) (94.8%), but a low positive agreement (PA) (4.3%) for major depressive disorder, compared with diagnostic records of the prior 30 days. Likewise, we observed a good percent agreement (89.3%) and NA (94.3%), but a low PA (10.9%) on behavioral symptoms between MDS-3.0 and medical diagnostic records.

Table 3 shows a percent agreement for depression and behavioral symptoms between MDS 3.0 and prior 30-day medical diagnosis measure based on characteristics at the resident level. A significantly lower percent agreement of major depressive disorder was observed among residents who had moderate-to-severe pain versus no-to-mild pain (83.6 % vs 93.2%, P-value=0.001). When comparing the percent agreement of behavioral symptoms by resident characteristics, the agreement was significantly lower among residents who had moderate-to-severe (versus no-to-mild) cognitive impairment (81.7% vs. 95.9%, P-value<.001) and diagnosis of ADRD (73.2% vs 94.5%, P-value < .001).

Table 3.

Characteristic Differences in Percent Agreement for Major Depressive Disorder and Behavioral Symptoms Between MDS 3.0^* and Prior 30-day Medical Diagnostic Records at the Resident level

Characteristics	Major Depressive Disorder (Total No. of Residents =446)			Behavioral Symptoms† (Total No. of Residents=460)
	No. of Residents	Percent agreement, %	P-value§	No. of Residents	Percent agreement, %	P-value§
Age, y
<85	179	89.9	0.91	184	91.3	0.27
≥85	267	90.3		276	88.0
Sex
Females	305	89.5	0.43	312	91.0	0.09
Males	141	91.5		148	85.8
Race
Whites	426	89.9	0.71	440	89.1	0.71
Non-Whites	20	95.0		20	95.0
BIMS cognitive score
Intact/borderline	378	89.7	0.45	394	93.4	0.00
Moderate-to-severe impairment	68	92.6		66	65.2
ADL dependence
No to mild	387	91.2	0.05	386	90.9	0.01
Moderate to severe	59	83.1		74	81.1
Pain
No to mild	295	93.2	0.00	296	88.5	0.29
Moderate to severe	146	83.6		157	91.7
Active diagnosis of ADRD
Yes	105	91.4	0.61	112	73.2	0.00
No	341	89.7		348	94.5
Source of PHQ-9 assessment
Self-reported	420	90.5	0.33	-	-	-
Staff-assessed	26	84.6		-	-

Abbreviations: MDS, Minimum Data Set; PHQ-9, Patient health questionnaire; ADL, activities of daily living

^*A randomly selected MDS assessment per resident was used.

^†including presence of psychosis (hallucinations or delusions), physical and verbal behavior directed toward others, and other behavioral symptoms not directed toward others

^§P-value was determined by Chi-square test or Fisher’s exact test as appropriate.

Discussions

This is among the first studies to investigate how depression and behavioral symptoms assessment in MDS 3.0 agree with geriatrician clinical diagnosis among nursing home residents. The study showed MDS 3.0 had a high agreement with clinician diagnosis documented in the prior 30-days, with about 90% agreement for major depressive disorder and behavioral symptoms. We also observed a high negative agreement for both depression and behavioral symptoms, indicating that MDS 3.0 and medical records provided a similar result in noting no evidence of both conditions. The result of high negative agreement suggests that using both data sources may be informative to rule out depression or behavioral symptoms.

We observed, however, a low positive agreement between MDS 3.0 and medical records, with the former being likely to note depression and behavioral symptoms as present. One possible explanation of this discrepancy is that clinicians may reserve a definitive diagnosis for patients showing clear clinical signs or symptoms of diseases, while residents and nurse staff tend to describe any symptom, some of which may not need clinical attention, during MDS assessments. Our result echoes the result of a recent validation study indicating a poor sensitivity of MDS-3.0 PHQ-9 in identifying depression disorders compared with the presence of depression diagnosis in Medicare claims.(Belanger, et al., 2019) Our results of low positive agreements for depression and behavioral symptoms suggest that using either MDS 3.0 or medical diagnoses to confirm a positive result in nursing home residents is non-informative.

Our subgroup analyses identified several potential characteristics that might influence agreement between MDS 3.0 and clinical diagnoses of the studied conditions. Residents who had moderate-to-severe pain (vs. no/mild) showed a significantly lower agreement of depression. We also observed a lower agreement of behavioral symptoms among residents who had moderate-to-severe (versus no-to-mild) cognitive function, and among those with ADRD (versus without). These identified characteristics suggest a potential opportunity for improvement in MDS-3.0 assessment for depression and behavioral symptoms. Future investigations are needed to test the independent effect of these characteristics.

The study is limited in that the findings were generated from local nursing home residents of predominantly White race. The prevalence of depression and behavioral symptoms based on clinical diagnosis was low (<5%), compared to the prevalence estimates (12.4% and 34.0%, respectively) of other nursing home sample composed of individuals of various socio-economic status.(Gruber-Baldini, et al., 2004; Thakur and Blazer, 2008). Nevertheless, this study demonstrated the feasibility of comparing MDS 3.0against medical records. Future studies that replicate our study findings in a representative sample of nursing home residents are warranted.

Conclusions

This study showed that MDS-3.0 depression and behavioral symptoms had a high overall and negative agreement, but a low positive agreement with clinician diagnoses. The MDS 3.0 data may be useful in ruling out depression and behavioral symptoms. Confirmation of the findings in a population-based nursing home sample is warranted.

Data availability statement:

The data that support the findings of this study are available from a regional nursing home. Restrictions apply to the availability of these data, which were used under license for this study. Data are available from the corresponding author with the permission of the contracted regional nursing home.