Identifying dementia severity among people living with dementia using administrative claims data

Abstract

Objectives:

There is currently no reliable tool for classifying dementia severity level based on administrative claims data. We aimed to develop a claims-based model to identify patients with severe dementia among a cohort of patients with dementia.

Design:

Retrospective cohort study.

Setting and participants:

We identified people living with dementia (PLWD) in US Medicare claims data linked with Minimum Data Set (MDS) and Outcome and Assessment Information Set (OASIS).

Methods:

Severe dementia was defined based on cognitive and functional status data available in MDS and OASIS. The dataset was randomly divided into training (70%) and validation (30%) sets, and a logistic regression model was developed to predict severe dementia using baseline (assessed in the prior year) features selected by generalized linear mixed models (GLMMs) with least absolute shrinkage and selection operator (LASSO) regression. We assessed model performance by area under the receiver operating characteristic (AUROC), area under precision-recall curve (AUPRC), and precision and recall at various cut-off points, including Youden Index. We compared the model performance with and without using Synthetic Minority Oversampling Technique (SMOTE) to reduce the imbalance of the dataset.

Results:

Our study cohort included 254,410 PLWD with 17,907 (7.0%) classified as having severe dementia. The AUROC of our primary model, without SMOTE, was 0.81 in the training and 0.80 in the validation set. In the validation set at the optimized Youden Index, the model had a sensitivity of 0.77 and specificity of 0.70. Using a SMOTE-balanced validation set, the model had an AUROC of 0.83, AUPRC of 0.80, sensitivity of 0.79, specificity of 0.74, PPV of 0.75, and NPV of 0.78 when at the optimized Youden Index.

Conclusions and implications:

Our claims-based algorithm to identify patients living with severe dementia can be useful for claims-based pharmacoepidemiologic and health services research.

Brief summary:

We developed a claims-based dementia severity algorithm to identify patients with severe dementia so that dementia severity can be considered in drug effectiveness and safety studies using claims data.

Introduction

Dementia is characterized by significant cognitive decline that is not due to delirium or other mental disorders and interferes with independence in everyday activities.^1,2 It is estimated that there are >6 million people living with dementia (PLWD) in the United States.² The Functional Assessment Staging Test (FAST), one of the tools to determine dementia severity, categorizes dementia into mild, moderate, and severe stages.³ Mild dementia is characterized by decreased job function, organizational capacity, and ability to perform complex tasks such as cooking, cleaning, and traveling. Moderate dementia is characterized by difficulty with bathing, dressing, and toileting, and it may be associated with urinary or fecal incontinence. Severe dementia is characterized by limited speech and loss of ambulatory ability. Given the distinct symptoms and health status at different stages, dementia management as well as the management of chronic co-existing conditions should be matched to patient needs at each stage.

There are several dementia severity rating scales such as the FAST, the Clinical Dementia Rating (CDR) Scale, and the Dementia Severity Rating Scale that are used in clinical practice. However, the use of these tools may be influenced by other pressing considerations during the clinic visit, reimbursement incentives, as well as clinician specialty and training.⁴ Moreover, their application in large population studies is limited.^5,6,7 As newer dementia treatments are being introduced, and because PLWD are severely underrepresented in clinical trials,^8,9,10 it is important to generate real-world evidence regarding effectiveness, safety, and treatment heterogeneity with varying dementia severity using routinely collected data. Insurance claims data have been traditionally used to generate such evidence using diagnostic codes for dementia as disease indicators. However, these codes may falsely capture reversible or non-cognitive neuropsychiatric symptoms, rather than true neurodegenerative disease.⁴ Further, it is not possible to ascertain dementia severity using diagnostic codes alone as there are no distinct International Classification of Diseases (ICD) 10 codes that reflect dementia severity. Thus, there is a critical need for a scalable dementia severity algorithm, using information available in claims data, to identify those with severe dementia, in order to study the effectiveness and safety of pharmacological and non-pharmacotherapy to manage comorbidities and manifestations of dementia. In this study, we aimed to develop a model that can be applied to routinely collected claims data to identify patients with severe dementia.

Methods

Data source and study population:

We used Medicare claims data linked with Minimum Data Set (MDS) and Outcome and Assessment Information Set (OASIS). MDS is part of the federally mandated process for clinical assessment of functional capabilities of all residents in Medicare or Medicaid certified nursing homes. OASIS is a group of data elements that contain core items of comprehensive assessment results for patients receiving adult home care, and includes sociodemographic, environmental, support system, health status, and functional status attributes, with selected attributes of health service utilization.¹¹ The Medicare claims data contain information on demographics, enrollment start and end dates, dispensed medications, performed procedures and medical diagnoses.

For this study, we identified patients aged ≥ 65 years who either had an MDS admission, quarterly or annual assessment or an OASIS start, resumption of care, or discharge assessment that did not occur within 30 days of a hospitalization from 2014 to 2018. The cohort entry (index) date was the date of the first MDS or OASIS assessment. To be included in the study, we required that patients have at least one diagnosis of dementia in Medicare claims data in 2 or more distinct settings (inpatient, outpatient, nursing home, emergency department) at any time prior to (including) the index date (AUROC [95% CI]: 0.81 [0.78–0.84], positive predictive value: 77.5%).¹² Dementia was identified using the Chronic Conditions Warehouse (CCW) definition.¹³ We required that patients have at least 365 days of continuous enrollment in Medicare Part A (inpatient), Part B (outpatient), and part D (prescription) plans before the index date to allow for covariate assessment as well as 90 days after the index date to allow for dementia severity assessment. Further, we excluded patients with Parkinson’s disease, post stroke paresis, and hip fracture in the 365 days prior to (including) the index date, as functional deficits in these patients cannot be completely attributed to dementia.

Outcome variable:

The primary outcome of interest was severe dementia which was operationalized as having a Cognitive Function Scale (CFS) of 4 in the MDS dataset or moderate to severe cognitive impairment in the OASIS dataset, and dependency in three early activities of daily living (ADL) (dressing, bathing, toileting)¹⁴ plus dependency in at least one advanced activity of daily living (incontinence, speech, feeding, transferring and ambulation) in both datasets (Table S1). The components of the outcome were measured between (and including) the index date and 90 days after index date (Figure S1). If a patient had more than one MDS or OASIS assessment in the outcome measurement window, the assessment closest to the index date was chosen.

Candidate predictors:

There were 278 candidate predictors in our analyses, including age, sex, race, US region of residence, the Kim claims-based frailty index and its 93 components,¹⁵ medication use, and healthcare utilization. Medication use was defined based on the Anatomical Therapeutic Chemical (ATC) classification system.¹⁶ All the candidate predictors were assessed from the Medicare claims component in the 365 days prior to (and including) the index date (Figure S1). While there were changes to Medicare payment incentives for documentation of dementia during the study period, the analysis was adjusted for calendar year of cohort entry, thus accounting for the effect of policy change in a specific year.

Missing data:

Missing outcome values were imputed by multiple imputation¹⁷ using frailty score components, under the missing at random assumption. Model development and validation were performed in the imputed dataset.

Model development:

The study cohort was randomly split into training (70%) and validation (30%) sets. The balance of patient characteristics between development and validation datasets was quantified in terms of absolute standardized difference (ASD). We used generalized linear mixed models (GLMMs) with least absolute shrinkage and selection operator (LASSO) logistic regression with leave-cluster-out cross-validation (by US geographical regions) for feature selection in the training set to account for nonindependence of observations with geographical regions and non-linearity relationships. The LASSO-selected features were then included in a logistic regression model to predict the probability of severe dementia.

Evaluation of model performance:

The tuning parameter (penalty term) of the LASSO model was optimized using cross-validated area under the receiver operating curve (AUROC). Model discrimination was assessed by AUROC and area under the precision-recall curve (AUPRC) in the training and validation sets. Model calibration was assessed by Hosmer-Lemeshow test. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated at the optimal (maximum) Younden Index¹⁸ as well as at varying cut-off points in the validation set. Data cleaning and processing were done using SAS version 9.4. All other statistical analyses were carried out using GLMMLasso package in R version 4.3.2 (2023–10-31) and Python version 3.11.5 and.

Sensitivity analyses:

To address the low number of outcome events in our study cohort, we repeated our analyses using Synthetic Minority Oversampling Technique (SMOTE) that oversamples the dataset to boost outcome prevalence.¹⁹ We also compared the model performance with other commonly used machine-learning methods, including random forest²⁰ and gradient boosting.²¹

Results

Characteristics of the study population:

Our study cohort included 254,410 patients, with 178,087 patients in the training and 76,323 patients in the validation set. Of these, 17,907 (7.0%) patients were classified as having severe dementia (Table 1). Patient characteristics between the training and validation datasets were comparable (ASD ≤0.01 for all measured variables). The mean age was 83 years in the training and validation sets. Majority of patients in both training and validation sets were female (62.7% in the training and 62.4% in the validation set) and White (86.0% in both training and validation sets). Hypertensive diseases and arthropathies were the most common pre-existing comorbidities in both development and validation sets. The mean number of impaired ADLs at baseline was 4.0 in both training and validation sets. The most frequently occurring ADL impairment in both training and validation sets was requiring assistance for dressing (71%) and the least frequent was requiring assistance with feeding (17%). 16% of patients had dependencies in all three early ADLs (dressing, bathing, and toileting, without dependencies in any advanced ADLs) and 60% had a dependency in at least one advanced ADL (speech, feeding, ambulation, sitting, incontinence). 10% of patients in both training and validation sets had severe cognitive impairment at baseline. 2,578 patients had missingness for severe dementia, which was imputed.

Table 1:

Patient charateristics

Characteristics	Training dataset N = 178087	Validation dataset N = 76323	ASD
Severe dementia, n (%)	12593 (7.1)	5314 (7)	0.006
Severely impaired cognitive function, n (%)	17980 (10.1)	7567 (9.9)	0.009
Demographics
Age in years, mean (SD)	82.5 (7.5)	82.6 (7.5)	0.006
Female, n (%)	111647 (62.7)	47656 (62.4)	0.007
Race, n (%)
White	153197 (86)	65533 (85.9)	0.007
Black	15616 (8.8)	6726 (8.8)	0.002
Asian	3231 (1.8)	1394 (1.8)	0.001
Unknown/others	1952 (1.1)	857 (1.1)	0.004
Pre-existing conditions
Frailty score, mean (SD)	0.3 (0.1)	0.3 (0.1)	0.005
Disorders of thyroid gland, n (%)	64158 (36)	27275 (35.7)	0.009
Purpura and other hemorrhagic conditions, n (%)	14140 (7.9)	6182 (8.1)	0.008
Organic psychotic conditions, n (%)	123549 (69.4)	52842 (69.2)	0.004
Hypertensive disease, n (%)	159774 (89.7)	68503 (89.8)	0.002
Ischemic heart disease, n (%)	76072 (42.7)	32599 (42.7)	<0.001
Cerebrovascular disease, n (%)	73790 (41.4)	31684 (41.5)	0.002
Pneumonia and influenza, n (%)	40753 (22.9)	17274 (22.6)	0.008
Chronic obstructive pulmonary disease and allied conditions, n (%)	54781 (30.8)	23286 (30.5)	0.008
Nephritis, nephrotic syndrome, and nephrosis, n (%)	70050 (39.3)	29827 (39.1)	0.007
Infections of skin and subcutaneous tissue, n (%)	29131 (16.4)	12422 (16.3)	0.003
Arthropathies and related disorders, n (%)	121437 (68.2)	52024 (68.2)	0.001
Dorsopathies, n (%)	83503 (46.9)	35981 (47.1)	0.007
Healthcare utilization
Number of hospitalizations, mean (SD)	1.2 (1.1)	1.1 (1.1)	0.083
Number of ER visits, mean (SD)	2.3 (2.1)	2.3 (2.1)	0.01
Number of office visits, mean (SD)	9.2 (7.8)	9.2 (7.8)	0.007
Number of unique drugs dispensed, mean (SD)	11.3 (6.3)	11.3 (6.3)	0.005
Number of days spent at home in the past 365 days, mean (SD)	332.1 (37.4)	332.2 (37.1)	0.002
Nursing home stay for more 100 days, n (%)	3449 (1.9)	1468 (1.9)	0.001
ADL dependence
Number of impaired ADLs, mean (SD)	4 (2.4)	4 (2.4)	0.003
Grooming, n (%)	103899 (58.3)	44320 (58.1)	0.008
Dressing, n (%)	125983 (70.7)	53902 (70.6)	0.004
Bathing, n (%)	155489 (87.3)	66569 (87.2)	0.004
Toileting, n (%)	109169 (61.3)	46688 (61.2)	0.004
Transferring, n (%)	95367 (53.6)	40989 (53.7)	0.004
Feeding, n (%)	30481 (17.1)	12784 (16.7)	0.014
Ambulation, n (%)	98348 (55.2)	42253 (55.4)	0.004
Early ADL, but no advanced, n (%)	28074 (15.8)	11950 (15.7)	0.004
Any advanced ADL, n (%)	107209 (60.2)	45975 (60.2)	0.001
Early plus any advanced ADL, n (%)	59850 (33.6)	25434 (33.3)	0.008

Early ADL dependence includes dependence in dressing, bathing, and toileting. Advanced ADL dependence includes dependence in feeding, speech, ambulation, incontinence and transferring. Dementia severity, cognitive function and ADL dependence were assessed in Minimum Data Set (MDS) or Outcome and Assessment Information Set (OASIS). See Table S1 for specific definitions.

All other variables were assessed in Medicare claims.

ASD = absolute standardized difference; SD = standard deviation; ADL = activities of daily living.

Development of prediction model:

Of the 266 candidate predictors, 191 were selected by GLMMs LASSO feature selection. Dementia patients with nursing home stays longer than 100 days had 1.75 times the odds of having severe dementia compared to those with nursing home stays less than 100 days (Tables S2, S3). Other positive predictors of severe dementia include a history of organic psychotic conditions (OR: 1.45), antipsychotic medication use (OR: 1.60), number of unique medication use (OR: 1.43). In contrast, non-organic psychoses (not organic psychotic disorders; OR: 0.78), receiving inpatient care (OR: 0.62), use of systemic antihistamine (OR: 0.72), disorders of the peripheral nervous system (OR: 0.80), and receiving care from an ophthalmologist (OR: 0.75) were among the negative predictors.

Model performance:

The AUROC of the model was 0.81 and 0.80 in the training and validation sets, respectively (Figure 1). The corresponding AUPRC was 0.24 in both the training and validation sets (Table 2). The highest Younden Index was obtained at a predicted risk ⩾ 0.07, with a sensitivity of 0.77 and specificity of 0.70. The chi-squared statistic (χ²) of Hosmer-Lemeshow goodness of fit test in the validation set was 48.26 (p-value <0.001) (Figure 2).

Figure 1 -.

Model performance (AUROC) in the training and validation datasets

Description: The AUROC of the model was 0.81 and 0.80 in the training and validation sets, respectively.

Table 2.

Model performance

Model/ Approach	AUROC in Training	AUROC in Validation*	AUPRC in Training	AUPRC in Validation*
Primary analysis
LASSO w/o SMOTE + Logistic w/o SMOTE	0.806	0.802	0.239	0.244
Sensitivity analyses
LASSO w/ SMOTE + Logistic w/ SMOTE	0.778	0.777	0.219	0.223
LASSO w/ SMOTE + Logistic w/o SMOTE	0.806	0.802	0.239	0.243
LASSO w/o SMOTE + Logistic w/ SMOTE	0.778	0.775	0.219	0.222
Random forest w/ SMOTE	0.966	0.800	0.680	0.224
Random forest w/o SMOTE	0.972	0.800	0.754	0.236
Gradient boosting w/ SMOTE	0.82	0.805	0.26	0.233
Gradient boosting w/o SMOTE	0.842	0.812	0.245	0.309

LASSO, least absolute shrinkage and selection operator; SMOTE, synthetic minority oversampling technique; AUROC, area under the receiver operating characteristic; AUPRC, area under precision-recall curve.

^*Validation dataset is not SMOTE balanced.

Figure 2 -.

Observed vs. predicted risk of severe dementia by deciles of predicted probability

Description: The chi-squared statistic (χ²) of Hosmer-Lemeshow goodness of fit test in the validation set was 48.26 (p-value <0.001).

Sensitivity analyses:

Using a SMOTE-balanced training set did not improve model performance in the validation set (Table 2). In the validation dataset, the AUROC of the random forest model was 0.80 and that of the gradient boosting model was 0.81, with and without SMOTE, which was comparable to the performance of LASSO-logistic regression model without SMOTE (AUROC=0.80). When applied to a SMOTE-balanced validation set, our primary model had an AUROC of 0.83, AUPRC of 0.80, overall accuracy of 0.76, sensitivity of 0.79, specificity of 0.74, PPV of 0.75, and NPV of 0.78 at the optimized cut point determined by Youden Index.

Discussion

We have developed a claims-based dementia severity algorithm to identify patients with severe dementia using Medicare claims data linked with MDS or OASIS. The algorithm demonstrated a high calibration performance with an AUROC of 0.82 in the independent validation set. Strong positive predictors of severe dementia include nursing home stay for more than 100 days, diagnosis codes of organic psychotic conditions, and use of antipsychotic medications. Strong negative predictors of severe dementia include other psychoses and receiving inpatient care. Repeated analyses using SMOTE yielded models with comparable performance. Since various machine-learning models yielded comparable performance, The LASSO-logistic regression model was selected for its clear interpretability and ease of applicability.

Dementia exists in a continuum of severity, and the management for dementia of different severity levels are substantially different. Routinely used medications for dementia are designed to prevent progression from early to moderate dementia but their role in the management of severe dementia is somewhat unclear. Some randomized control studies showed clinically meaningful benefit of memantine for patients with moderate to severe dementia.²² However, there have also been reports of worsening hallucinations and delusions with memantine.^23,24 Since 2021, two drugs that slow dementia progression have been approved for use in the US by the Federal Drug Administration (FDA),^25,26 with other drugs in the pipeline.²⁷ While these drugs performed well in clinical trials, those with severe dementia were not included in these studies.^27,28,29 The efficacy and safety of these conventional and newer drugs needs to be evaluated in the real-world setting via large-scale studies using routinely collected data, such as administrative claims data or EHR, which would be representative of patients with moderate to severe dementia. The treatment effect heterogeneity in such a study can only be evaluated with a reliable and scalable algorithm to identify patients with severe dementia based on routinely collected data. The administration of currently available tools, like the CDR, to determine dementia severity, is resource-intensive and requires a patient encounter, which limits their scalability in real-world research settings. While several machine learning models have been developed to determine dementia severity,^7,30 these models utilized features that may not be available in routine administrative data (e.g., clinical cognitive assessment results) and their generalizability may be limited. Our model used features readily available in administrative claims data, increasing its applicability to other routinely collected healthcare data. Researchers can use our algorithm to identify patients with severe dementia in claims data and assess subgroup effects in this vulnerable population.

By generating claims-based real-world evidence on treatment effect heterogeneity by dementia severity, our algorithm may benefit a wide range of comparative safety and effectiveness research questions that are clinically relevant among PLWD. For instance, the considerations of prescribing pharmacotherapy for primary prevention vary based on dementia severity. Also, the benefits of drugs used in primary prevention, like aspirin, statins, bisphosphonates, or oral anticoagulants (OACs) in those with concomitant atrial fibrillation must be weighed differentially against the risks at different dementia severities. Poor medication adherence, limited insight into health conditions, and behavior disturbance associated with advanced dementia may present additional challenges to healthcare providers and caregivers. ^31,32,33,34 For example, dementia and frailty are among the leading reasons for withholding OAC when anticoagulation is indicated. ^{35,36,37,38,39} This is because PLWD are at higher risk of falls, traumatic intracranial bleeding, medication errors, and poor adherence.^40,41 Therefore, it is important to stratify the analyses by dementia severity when studying the safety and effectiveness of OACs. Another example is the investigation of drug safety for delirium or behavior disturbance in PLWD. Dementia has been identified as a barrier to discontinuing antipsychotics used to treat post-discharge delirium.⁴² As the risk of delirium and severity of behavior increases with dementia severity, monitoring the safety and effectiveness of psychotropics needs to be stratified by dementia severity. Our dementia severity classification algorithm can be useful to researchers studying drug use, health service utilization, effectiveness, and safety of medical and surgical treatments in patients with severe dementia. These studies would ultimately inform safe prescribing and deprescribing, and optimal management and interventions in patients with severe dementia who are especially vulnerable to adverse events from these treatments.

Our study has several limitations. Our study is based on administrative claims data and many factors that can be predictive of dementia severity may not be recorded in claims data⁴ and hence not included in our algorithm. These factors include patient factors such as awareness of decline, access to care and familial support, which affect how symptoms are recognized and reported to providers. Further, there is potential for misclassification of dementia when functional deficits due to other causes (like post stroke paresis, Parkinson disease or hip fracture) are wrongly attributed to dementia. To overcome these limitations, we required a diagnosis of dementia in 2 distinct settings and excluded those with recorded post stroke paresis, Parkinson’s disease, or hip fracture in the 365 days before cohort entry. Additionally, our outcome was based on clinical assessments recorded in MDS and OASIS. While these data sources have relevant information regarding the severity of cognitive impairment and degree of functional status impairment, they were not originally designed to determine dementia severity. Many PLWD may often transition from one care setting to another. For example, a community-dwelling person with dementia may be hospitalized and discharged to a short-term skilled nursing facility (where the MDS assessment was done) and then discharged home, receiving home health care (where the OASIS assessment was done). In order to develop a model that can be applicable to a wide spectrum of PLWD, not just those institutionalized or those who are very healthy and rarely hospitalized, we elected to develop the model in the MDS-OASIS combined dataset. It is possible that a model developed in a specific population (e.g., institutionalized people living with dementia) may have a better performance in that population, but it is likely less generalizable to other related population (e.g., community-dwelling people living with dementia). Lastly, our algorithm was developed in a cohort of US Medicare beneficiaries, that was predominantly White, and its validity among patients of other races, those outside of the US and those with other insurances should be established in future studies.

Conclusions and implications

Based on a national cohort of US Medicare beneficiaries, aged 65 years or older, we developed a claims-based algorithm to identify patients with severe dementia with good discrimination and calibration. Our model can be useful for identifying vulnerable study populations with severe dementia for claims-based health services research and comparative effectiveness and safety studies to optimize prescribing and deprescribing decisions.