Abstract
Introduction
This study aimed to develop and validate a 3‐year dementia risk score in individuals with mild cognitive impairment (MCI) based on variables collected in routine clinical care.
Methods
The prediction score was trained and developed using data from the National Alzheimer's Coordinating Center (NACC). Selection criteria included aged 55 years and older with MCI. Cox models were validated externally using two independent cohorts from the Prospective Registry of Persons with Memory Symptoms (PROMPT) registry and the Alzheimer's Disease Neuroimaging Initiative (ADNI) database.
Results
Our Mild Cognitive Impairment to Dementia Risk (CIDER) score predicted dementia risk with c‐indices of 0.69 (95% confidence interval [CI] 0.66–0.72), 0.61 (95% CI 0.59–0.63), and 0.72 (95% CI 0.69–0.75), for the internally validated and the external validation PROMPT, and ADNI cohorts, respectively.
Discussion
The CIDER score could be used to inform clinicians and patients about the relative probabilities of developing dementia in patients with MCI.
Keywords: dementia, mild cognitive impairment, risk prediction, routine care, validation
1. BACKGROUND
Dementia is typically preceded by mild cognitive impairment (MCI). On average, the annual risk of progression in specialist clinic populations is 9.6% and and in community‐based studies is 4.9%, 1 with progression usually occurring within 2 to 3 years. 2 Persons with MCI often request information about their risk of developing dementia. 3 Accurate knowledge of being at a higher personal risk might serve as motivation to adopt beneficial preventive lifestyle modifications as well as planning for the future by completing advance care directives. 4
There is currently no dementia risk score for individuals with MCI that is suitable for application in routine clinical practice. 5 Systematic reviews 5 , 6 identified two dementia risk scores for individuals with amnestic MCI (excluding studies without any validation process or in which the number of events is smaller than 100), but neither was externally validated, and both used information unlikely to be available in a routine clinical practice.
The objective of this study was to develop and validate a pragmatic and clinically useful score, Mild Cognitive Impairment to Dementia Risk (CIDER), to predict dementia risk in persons with MCI, suitable for application in a care setting without access to advanced imaging, cerebrospinal fluid analysis, or neuropsychological testing. Additionally, we sought to determine whether two commonly used global cognitive screening tools, the Folstein Mini‐Mental State Examination (MMSE) 7 and the Montreal Cognitive Assessment (MoCA) 8 , provide predictive information on the prognosis of MCI.
2. METHODS
2.1. Data sources
We used data from three sources: the National Alzheimer's Coordinating Center's Uniform Data Set (NACC‐UDS; naccdata.org), the Prospective Registry of Persons with Memory Symptoms (PROMPT) of the Cognitive Neurosciences Clinic at the University of Calgary; and the Alzheimer's Disease Neuroimaging Initiative (ADNI). NACC was established by National Institute on Aging (NIA)‐funded Alzheimer's Disease Research Centers (ADRCs) that recruit and collect data on subjects with cognitive function ranging from normal to dementia. The NACC‐UDS is a longitudinal dataset that includes demographic and standardized clinical data collected approximately annually. All test centers administered standardized forms and informed consent was collected from all subjects and their informants. Detailed information on the cohort and neuropsychological battery of tests included in the UDS is described elsewhere. 9 , 10 , 11 The PROMPT registry was established in July 2010 and enrolls patients referred to the Cognitive Neurosciences Clinic, operating in two tertiary care centers in Calgary, for assessment of cognitive or behavioral dysfunction. 12 Consecutive patients are approached for consent, and all patients attending the clinic are eligible to participate. The ADNI was used as another independent external validation dataset. 13 The primary goal of ADNI has been to test whether biomarkers can be combined to measure the progression of MCI and early Alzheimer's disease (AD).
2.2. Ethical considerations
The NACC database itself is exempt from institutional review board review. The PROMPT has been approved by the University of Calgary Conjoint Health Research Ethics Board. All ADNI subjects or their proxies provided written informed consent.
2.3. Derivation cohort
As the NACC is one of the largest and most comprehensive longitudinal datasets for dementia research, it was used to develop our models. There were 6222 patients with MCI, aged 55 years and older, who had at least one follow‐up after the baseline visit within 3 years, and had either MoCA or MMSE scores. MCI was defined based on criteria from the NIA‐Alzheimer's Association (NIA‐AA). 14 There are three versions of NACC‐UDS (version 1: 2005–2008; version 2: 2008–2015; version 3: 2015‐present). The MoCA was administered from 2015 on, while the MMSE was collected in the previous versions. Test administration in Spanish was allowed. The “NACC‐MoCA” cohort (N = 1233) refers to those enrolled from 2015 onward, and the “NACC‐MMSE” cohort (N = 4989) refers to those enrolled earlier. Figure S1 in supporting information shows the identification of the study cohort for model development.
2.4. Validation cohorts
In PROMPT, there were 452 individuals aged 55+ years with MCI based on NIA‐AA criteria. 14 MCI participants were not required to have AD biomarkers or a working diagnosis of AD. Figure S2 in supporting information shows the identification of this validation cohort for model testing. Figure S3 in supporting information shows the identification of the ADNI validation cohort. Participants included in ADNI were between 55 and 90 years of age, English or Spanish speakers, and were accompanied by study partners. We identified a sample of 598 individuals with MCI that was defined as having a MMSE score between 24 and 30, reported subjective complaints, objective memory deficits defined as Wechsler Memory Scale Logical Memory II scores below education‐adjusted thresholds, and a Clinical Dementia Rating score of 0.5.
2.5. Outcome measures
For the NACC cohort, the outcome was time to dementia of any cause within 3 years of an MCI diagnosis made at their first (baseline) NACC visit. In NACC, trained clinicians make diagnoses in accordance with published research diagnostic criteria, and most ADRCs at this time used Diagnostic and Statistical Manual 4th edition (DSM‐IV) criteria for dementia. In addition, AD was determined using criteria from either the National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer's Disease and Related Disorders Association (NINCDS‐ADRDA) 15 or NIA‐AA. 16 Patients were followed approximately annually.
In PROMPT, a dementia diagnosis based on DSM‐IV criteria was made by experienced clinicians. Patients were followed about every 6 to 12 months for the occurrence of dementia. To ensure complete follow‐up, we linked PROMPT clinic participants to Alberta health administrative data for surveillance of new dementia diagnoses. We excluded any dementia diagnoses within 30 days of the PROMPT first visit at which MCI was diagnosed, reasoning that they probably reflected miscoded MCI. The administrative dementia case definition was based on that used by the Public Health Agency of Canada (PHAC). 17 Specifically, dementia was defined as one or more hospitalizations with any dementia‐related International Classification of Diseases codes, three or more physician claims for dementia at least 30 days apart within a 2‐year time period, and/or prescription of a cholinesterase inhibitor. 17 The date of dementia was defined as the date of first diagnosis of dementia from either PROMPT or the administrative data.
In the ADNI validation cohort, dementia was defined as patients who had an MMSE score between 20 and 26, Clinical Dementia Rating score of 0.5 or 1, and met DSM‐IV criteria. Dementia status was ascertained at a prospective research visit conducted at 6, 12, 18, 24, and 36 months. For those without dementia, follow‐up time was censored at their last follow‐up date (if less than 3 years), at 3‐year event‐free follow‐up (right censoring), or at time of death.
RESEARCH IN CONTEXT
Systematic Review: Based on recent systematic reviews on dementia risk prediction models, there are currently no pragmatic, externally validated dementia risk scores to use in routine care for individuals with mild cognitive impairment (MCI).
Interpretations: The authors developed and externally validated the Mild Cognitive Impairment to Dementia Risk (CIDER) score to predict dementia within 3 years in patients with MCI. The CIDER score was designed to use variables collected in most clinical care, and it was based on data from three independent MCI cohorts. The CIDER score could be used to inform clinicians and patients about the relative probabilities of developing dementia in patients with MCI.
Future Directions: Further research needs to further validate the CIDER score in general clinical practice and to determine the added value of additional testing and more advanced markers such as neuropsychological testing, neuroimaging, and biomarkers of Alzheimer's disease.
HIGHLIGHTS
The Mild Cognitive Impairment to Dementia Risk (CIDER) score was derived and validated to predict risk for dementia in 3 years in persons diagnosed clinically with mild cognitive impairment (MCI).
The input variables use simple clinical information and cognitive test scores often used in routine clinical practice, making the score pragmatic for use in a general neurology or geriatric setting.
Using the CIDER score, clinicians can communicate an objectively derived assessment of risk to their patients, facilitating discussion of prognosis, risk factor modification, and advance care planning.
2.6. Predictors
To create the CIDER score, we restricted candidate predictors to variables that are obtainable during the first office visit of a patient with suspected MCI: age; sex; education (in years); marital status at the time of assessment (married/common‐law vs. never married, separated divorced, or widowed); smoking (former, current, and never); MoCA or MMSE score; and presence or absence (yes/no) of diabetes mellitus, hypertension, traumatic brain injury (TBI), cardiovascular diseases, mood disorder, and history of alcohol abuse. Full operational definitions for the predictor measures used in the three cohorts are provided in Table S1 in supporting information. Hypertension, diabetes mellitus, and TBI were defined based on self‐report, documentation in the medical records, or use of a medication for that condition. Cardiovascular disease was defined if at least one of the following was reported: congestive heart failure, atrial fibrillation, angioplasty/endarterectomy/stent, cardiac bypass procedure, pacemaker, or heart valve replacement or repair. A mood disorder was defined if at least one of the following was reported in the subject health history: depression, anxiety disorder, or the use of an anti‐depressant medication. Smoking history was categorized as current (if individuals reported they smoked cigarettes in the last 30 days), former, or never. Alcohol abuse was defined as alcohol use during the last 12‐month period resulting in a failure to fulfill major role obligations at work, school, or home. Every predictor was measured at baseline (first) visit across the three cohorts.
2.7. Statistical analysis
Cox regression models were used to estimate the risk of dementia within 3 years. The linear relationship between continuous candidate predictors and outcome were assessed using the restricted cubic splines. The assumption of proportional hazards was checked based on Schoenfeld residuals. 18 Variable selection was based on a combination of clinical knowledge, Akaike information criterion (AIC), and the availability of the predictors in the three cohorts to obtain the most parsimonious models, while retaining age and sex. All variables were included in the model prior to selection and any possible two‐way interactions were tested and retained if statistically significant (at α = 0.05) and deemed clinically relevant and supported by previous literature. 19 , 20 , 21 Hazard ratios with 95% confidence intervals (CIs) were reported. We evaluated predictive accuracy based on measures of discrimination (the ability to distinguish dementia high‐risk individuals from the low‐risk individuals), and calibration (the agreement between predicted and observed dementia [survival] probability). The discriminatory performance was evaluated using Harrell's C‐index and calibration was assessed by graphically comparing predicted and observed values based on a flexible hazard regression approach. 22 , 23 Bootstrapping (B = 500) was used to obtain overfitting‐corrected estimates of discrimination index and calibration. Independent model development and validation processes were carried out for including either the MoCA or the MMSE, which are referred as the “CIDER‐MoCA” and the “CIDER‐MMSE” models, respectively. The PROMPT and ADNI database were used to externally validate the developed models in terms of discrimination and calibration. Monograms were reported and for each predictor in the model, the range of the scoring were restricted to be 0 to 100. Analyses were performed using the R studio “rms” package. 21
3. RESULTS
Table 1 shows that for the NACC‐MoCA cohort (N = 1233), 25% developed dementia over 3 years. The mean age at baseline was 73 years (standard deviation [SD] = 7.5), about half were females, and the median number of years of education was 16 years. For the NACC‐MMSE cohort (N = 4989), MCI conversion rate was higher (31%) but demographic characteristics were similar to the NACC‐MoCA cohort. In the PROMPT registry (N = 452), 42% were diagnosed with dementia over 3 years, mean age at baseline was 67 (SD = 8), approximately 43% were females, and median number of years of education was 13 years. In ADNI (N = 598), 37% were diagnosed with dementia over 3 years, mean age at baseline was 74 years (SD = 8.1), about 40% were females, and median years of education was 16. The median follow‐up time was 17.4 (interquartile range [IQR] = 14.3), 29.6 (IQR = 41.3), 17.3 (IQR = 37.5), and 24.0 (IQR = 35.5) months, in the NACC‐MoCA, NACC‐MMSE, PROMPT, and ADNI cohorts, respectively.
TABLE 1.
Descriptive characteristics for individuals with MCI
| Derivation cohorts | External Validation cohort | |||
|---|---|---|---|---|
| NACC MoCA N = 1233 | NACC MMSE N = 4989 | PROMPT N = 452 | ADNI N = 598 | |
| Dementia diagnosis in 3 years, n (%) | 303 (24.6) | 1528 (30.6) | 189 (41.8) | 222 (37.1) |
| Follow‐up in months, median (Q1–Q3) | 17.3 (12.8–27.1) | 29.6 (15.6–57.1) | 17.3 (6.3–43.8) | 24.0 (12.2–47.7) |
| Died in 3 years from initial visit, n (%) | 28 (2.3) | 227 (4.6) | 29 (6.4) | 14 (2.3) |
| Age, mean (SD) | 72.6 (7.5) | 74.0 (8.3) | 67.2 (8.1) | 73.9 (7.5) |
| Female sex, n (%) | 619 (50.2) | 2463 (49.4) | 194 (42.9) | 234 (39.1) |
| Education in years, median (Q1–Q3) | 16 (14–18) | 16 (12–18) | 13 (12–16) | 16 (14–18) |
| Marital status, n (%) | ||||
| Married/common‐law | 860 (69.8) | 3341 (67.0) | 329 (72.8) | 467 (78.1) |
| Others a | 366 (29.7) | 1605 (32.2) | 116 (25.7) | 129 (21.6) |
| Missing | 7 (0.6) | 43 (0.9) | 7 (1.6) | 2 (0.3) |
| MoCA score, median (Q1–Q3) | 23 (20–25) | 22 (19–24) | 22 (20–25) | |
| MMSE score, median (Q1–Q3) | 28 (26–29) | 28 (25–29) | 27 (26–29) | |
| Vascular diseases, n (%) | ||||
| Yes | 270 (21.9) | 1387 (27.8) | 122 (27.0) | 415 (69.4) |
| No | 957 (77.6) | 3599 (72.1) | 330 (73.0) | 183 (30.6) |
| Missing | 6 (0.5) | 3 (0.1) | 0 | 0 |
| Hypertension, n (%) | ||||
| Yes | 774 (62.8) | 3268 (65.5) | 232 (51.3) | 288 (48.2) |
| No | 458 (37.2) | 1719 (34.5) | 220 (48.7) | 310 (51.8) |
| Missing | 1 (0.1) | 2 (<0.1) | 0 | 0 |
| Diabetes, n (%) | 200 (16.2) | 773 (15.5) | 73 (16.2) | |
| Mood disorder, n (%) | ||||
| Yes | 543 (44.0) | 1983 (39.8) | 195 (43.1) | 152 (25.4) |
| No | 688 (55.8) | 3006 (60.3) | 257 (56.9) | 446 (74.6) |
| Missing | 2 (0.2) | 0 | 0 | 0 |
| Smoking status, n (%) | ||||
| Never | 738 (59.9) | 2624 (52.6) | 209 (46.2) | 332 (55.2) |
| Current | 40 (3.2) | 159 (3.2) | 50 (11.1) | 164 (27.4) |
| Former | 448 (36.3) | 2185 (43.8) | 134 (29.7) | 51 (8.5) |
| Missing | 7 (0.6) | 21 (0.4) | 59 (13.1) | 51 (8.5) |
| History of alcohol abuse | ||||
| Yes | 45 (3.7) | 284 (5.7) | 66 (14.6) | 19 (3.2) |
| No | 1183 (96.0) | 4694 (94.1) | 386 (85.4) | 579 (96.8) |
| Missing | 5 (0.4) | 11 (0.2) | 0 | 0 |
| Hispanic/Latino ethnicity | 89 (7.2) | 395 (7.9) | 3 (0.7) | 14 (2.3) |
Abbreviations: Q1, the first quartile; Q3, the third quartile; ADNI, the Alzheimer's Disease Neuroimaging Initiative; MCI, mild cognitive impairment; MMSE, Mini‐Mental State Examination; MoCA, Montreal Cognitive Assessment; NACC, National Alzheimer's Coordinating Center; PROMPT, Prospective Registry of Persons with Memory Symptoms; SD, standard deviation.
Notes: There were three patients missing education, 38 patients missing MoCA info in the NACC MoCA cohort. There were 15 patients missing education, 204 patients missing MMSE info in the NACC MMSE cohort.
Others included widowed, separated, divorced, and never married.
3.1. Missing data
For the development of the CIDER‐MoCA model, we retained 96% (1185/1233) of eligible individuals who had complete data. In the creation of the CIDER‐MMSE model 95% (4737/4989) were retained. Comparisons were made between individuals included and excluded because of missing data and are shown in Tables S4 and S5 in supporting information for NACC‐MoCA and NACC‐MMSE cohort, respectively.
3.2. Derivation in NACC
The CIDER‐MoCA model included age, sex, education, marital status, hypertension, and mood disorder in addition to the MoCA score. The C‐indices were 0.69 (95% CI 0.66–0.73) and 0.69 (95% CI 0.66–0.72) for the derivation cohort and bootstrapped validated models, respectively (Table 2). The CIDER‐MMSE model (where the MMSE score was used rather than MoCA) had the same set of predictors in addition to the cognitive test result. The C‐index was 0.67 (95% CI 0.65–0.69) for the derivation cohort and bootstrapped validated models. Figure 1 depicts calibration curves, which showed that all models had excellent calibration after bootstrapping validation. The median predicted 3‐year dementia risk of CIDER‐MoCA was 38%. The average risk within the lower quartile was 21% and within the upper quartile was 68%, respectively. The median predicted 3‐year dementia risk of CIDER‐MMSE was 32% with average risks within lower and upper quartiles of 16% and 61%, respectively.
TABLE 2.
CIDER dementia risk models—developed in NACC and internally validated based on bootstrapping, externally validated in PROMPT and ADNI
| CIDER MoCA | CIDER MMSE | |
|---|---|---|
| C‐index 95% CI | ||
| Derivation cohort | 0.69 [0.66, 0.73] | 0.67 [0.65, 0.69] |
| Index corrected | 0.69 [0.66, 0.72] | 0.67 [0.65, 0.69] |
| PROMPT | 0.61 [0.59, 0.63] | 0.62 [0.60, 0.64] |
| ADNI | 0.72 [0.69, 0.75] | 0.61 [0.59, 0.63] |
| HR (95% CI) | HR (95% CI) | |
|---|---|---|
| MoCA a | 0.44 [0.37, 0.51] | |
| MMSE b | 0.35 [0.31, 0.40] | |
| Age | 1.23 [1.03, 1.47] | 1.26 [1.17, 1.36] |
| Education | 1.40 [1.14, 1.72] | 1.34 [1.22, 1.47] |
| Sex | ||
| Female vs. male | 0.93 [0.72, 1.20] | 1.11 [1.00, 1.25] |
| Marital status | ||
| Married c vs. others | 1.78 [1.30, 2.43] | 1.46 [1.28, 1.65] |
| Hypertension | ||
| Yes vs. no | 0.78 [0.61, 1.00] | 0.90 [0.81, 1.00] |
| Mood disorder | ||
| Yes vs. no | 1.24 [0.98, 1.56] | 1.43 [1.29, 1.59] |
Abbreviations: ADNI, Alzheimer's Disease Neuroimaging Initiative; CI, confidence interval; CIDER, Mild Cognitive Impairment to Dementia Risk; HR, hazard ratio; MMSE, Mini‐Mental State Examination; MoCA, Montreal Cognitive Assessment; NACC, National Alzheim'r's Coordinating Center; PROMPT, Prospective Registry of Persons with Memory Symptoms.
For MoCA, age, and education years, HRs corresponded to each variable's interquartile range (IQR). For MoCA, IQR = 5; for age, IQR = 11 years; for education, IQR = 5 years; for MMSE, IQR = 3.
MMSE score was modeled using restricted cubic splines (with 3 degrees of freedom).
Married included “married” and “common‐law partnership”.
FIGURE 1.
Calibration plots from internal validation (bootstrapping corrected overfitting)—model developed based on NACC. Black: Observed line shows the relationship between the (not overfitting corrected) predicted 3‐year dementia risk and the actual dementia risk. Gray: Ideal line for predicted versus observed risks. Blue: Relationship between the overfitting corrected predicted 3‐year dementia risk and the actual dementia risk, based on bootstrapping (B = 500) and subsetting predictions into intervals by interpolation (containing 50 individuals on average). The blue dotted line above and below the blue line shows the 95% confidence interval based on the mean prediction error. CIDER, Mild Cognitive Impairment to Dementia Risk; MMSE, Mini‐Mental State Examination; MoCA, Montreal Cognitive Assessment; NACC, National Alzheimer's Coordinating Center
3.3. Validation in PROMPT and ADNI
External validation of the CIDER‐MoCA model resulted in C‐indices of 0.61 (95% CI 0.59–0.63) and 0.72 (95% CI 0.69–0.75) for the PROMPT and ADNI cohorts, respectively. External validation of the CIDER‐MMSE model resulted in C‐indices of 0.62 (95% CI 0.60–0.64) and 0.61 (95% CI 0.59–0.63) for PROMPT and ADNI, respectively. Figure 2 shows calibration curves resulting from the direct application of our models to the PROMPT and ADNI cohorts. CIDER‐MoCA showed overly extreme estimates when applied to the PROMPT cohort where it had a tendency of overestimating dementia risk for patients with a higher risk, and underestimating dementia risk for patients with a lower risk. On the other hand, the estimated risk from CIDER‐MoCA for ADNI was not sufficiently extreme. Moreover, while CIDER‐MMSE showed perfect calibration in internal validation, when externally validated, it consistently underestimated dementia risk in the two external cohorts. For CIDER‐MoCA (CIDER‐MMSE), the median predicted 3‐year dementia risk and the average risks within lower and upper quartiles were 37% (28%), 19% (14%), and 71% (60%) for PROMPT; and 43% (37%), 21% (17%), and 67% (60%) for ADNI, respectively.
FIGURE 2.
Calibration plots external validations (PROMPT + ADNI)—model developed based on NACC. Solid blue line shows the trend curve between the observed and predicted 3‐year risk of dementia; the dotted blue line above and below the solid blue line shows the 95% confidence interval based on the mean prediction error; dashed line shows the ideal line between observed and predicted risks. Risks was calculated as one minus the survival probabilities. ADNI, Alzheimer's Disease Neuroimaging Initiative; CIDER, Mild Cognitive Impairment to Dementia Risk; MMSE, Mini‐Mental State Examination; MoCA, Montreal Cognitive Assessment; NACC, National Alzheimer's Coordinating Center; PROMPT, Prospective Registry of Persons with Memory Symptoms
Figures 3 and 4 show the nomogram for CIDER based on using either the MoCA or MMSE, with higher scores suggesting higher dementia risk. The total points and corresponding 3‐year dementia risk are shown. For example, 70‐year‐old married males with a bachelor's degree (no hypertension or mood disorder) who had MCI with a MoCA score of 20 at baseline had about 60% risk (95% CI 0.51–0.69) of converting to dementia in the following 3 years.
FIGURE 3.
Point system from nomogram for CIDER risk score using MoCA. CIDER, Mild Cognitive Impairment to Dementia Risk; MoCA, Montreal Cognitive Assessment
FIGURE 4.
Point system from nomogram for CIDER risk score using MMSE. CIDER, Mild Cognitive Impairment to Dementia Risk; MMSE, Mini‐Mental State Examination
4. DISCUSSION
In this study we developed and validated a pragmatic risk score, CIDER, to predict dementia within 3 years in patients with MCI using data from three volunteer cohorts similar in characteristics to those seen in a memory clinic. CIDER was designed to be operationalized in a care setting using routinely collected data, without requiring specialized information such as the results of comprehensive neuropsychological testing, positron emission tomography, or cerebrospinal fluid testing. The CIDER score with MoCA has better accuracy, but in settings in which only MMSE is available, the CIDER score with MMSE can be used with only minor losses in accuracy. CIDER could potentially help inform clinicians and patients with MCI about their dementia risk in the 3 years after their baseline visit. An online calculator can be accessed at cider‐dementia.org.
We found that the MoCA or MMSE score is of paramount importance in discriminating MCI converters from non‐converters. Some additional variables also significantly predicted risk for conversion to dementia. Mood disorder is found to be associated with higher risk for dementia in those with MCI, which is consistent with findings from previous reviews. 3 , 24 Hypertension predicted a lower risk of progressing from MCI to dementia, which is also consistent with prior literature. 25 This may be because the presence of hypertension indicates a preventable vascular contribution is present, or that a neurodegenerative disease process reducing blood pressure is absent. This hypertension effect is the reverse of that seen in the general population in which hypertension increases the risk of dementia. 25 In this study, more education was associated with higher risk, even though education has been consistently associated with lower risk in the general population. This may be because highly educated persons with MCI have exhausted their cognitive reserve and have a high amount of brain pathology present leading to a more rapid rate of subsequent decline. 26 Some variables (e.g., presence of vascular disease) associated with higher dementia risk in the general population were not predictive in our models. Potential explanations include limited specific discriminative ability in predicting the occurrence of dementia or the possibility that vascular disease might indicate that the patient suffers from vascular MCI rather than AD‐MCI, with vascular MCI having a better prognosis.
The ability of CIDER to discriminate risk of dementia was moderate, and score results should not be overinterpreted especially in a group with an overall high risk for developing dementia. Given the inherent variability due to the complex interplay of age, genetics, and environment including biological and social factors, it is not possible to predict precisely and with high certainty which MCI patients will develop dementia, particularly when limiting predictors to those that can be collected at a routine office visit. Nonetheless, CIDER with MoCA scores can discriminate average risks for those with MCI from ≈21% to 68%, which we suggest could be clinically and personally (to the patient) relevant. In other areas of medicine, risk scores with similar discriminated are recommended as useful for clinical care. For example, the CHA2DS2‐VASC score for atrial fibrillation is recommended as the basis for anticoagulation decisions 27 but has an area under the curve of about 0.64, 28 which is lower than CIDER.
Strengths of this study include the internal and external validation of our developed models, thereby avoiding methodological shortcomings of prior models as documented in multiple systematic reviews. 5 , 6 , 29 A recent review shows that among 138 dementia risk prediction models, a third of the studies did not perform internal or external validation (46/138), and >10% of studies used a sample <100 individuals. 5 Moreover, 44% (60/138) of previous models were derived using information from subsamples drawn from the ADNI data. This highlights another strength of the current study. We based our models on the NACC database, which is one of the largest and most comprehensive longitudinal datasets for dementia research. Finally, the CIDER score will provide a good basis to investigate incremental prognostic value of advanced biomarkers. 19 , 30
One limitation of the study is that NACC may be not representative for the general population. There are few large, well‐characterized cohorts of MCI derived from the general population. This study was unable to consider culture‐specific variation in cognitive test performance or variation due to language of test administration; however, controlling for Hispanic ethnicity made no difference in model performance. There are some inherent differences between PROMPT (or ADNI) and NACC, which may explain why CIDER‐MoCA and CIDER‐MMSE gave imperfectly calibrated risk prediction in the two cohorts. It is possible that differences in MCI and dementia assessment methods could underlie some of the variance in model performance across the cohorts. In addition, ADNI is a volunteer‐based research cohort focused on AD that excluded patients with clinically diagnosed vascular cognitive impairment. PROMPT is a memory clinic registry enrolling all patients with MCI, making it more heterogeneous and representative of other memory clinics. The discriminative ability of CIDER is lower than previously published models. We think this is because we intentionally restricted candidate variables to those that can be captured using routinely collected data. In future work, we will determine the added value of additional testing and more advanced markers such as neuropsychological testing, amyloid, and tau, as well as explore the potential for machine learning. Prior work in ADNI suggests that C statistics of 0.85 or higher may be achievable when adding the results of biomarkers and neuropsychological testing. 31
5. CONCLUSIONS
CIDER is a simple, feasible risk score that we have validated for the purpose of predicting a gradient of 3‐year risk for progressing to dementia in persons with MCI seen in memory clinics. It may help clinicians and patients in this setting to convey and understand individualized prognosis. This might help motivate patients at higher risk to take actions that might reduce their likelihood of progression (such as adopting healthier lifestyles) and do advance care planning. How best to present this data and support patients will require additional study.
CONFLICTS OF INTEREST
The authors have no conflicts of interest to report.
Supporting information
Supporting Information
ACKNOWLEDGMENTS
The NACC database is funded by NIA/NIH Grant U01 AG016976. NACC data are contributed by the NIA‐funded ADCs (see Supplemental Materials for complete list). Data collection and sharing for this project was funded by the ADNI (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH‐12‐2‐0012). ADNI is funded by the NIA, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie; Alzheimer's Association; Alzheimer's Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol‐Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann‐La Roche Ltd. and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC; Johnson & Johnson Pharmaceutical Research & Development LLC; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California. The PROMPT registry is funded by the Kathy Taylor Chair in Vascular Dementia the University of Calgary with additional support from the Brain and Mental Health Research Clinics initiative of the Hotchkiss Brain Institute, University of Calgary.
Wang M, Sajobi TT, Ismail Z, et al. A pragmatic dementia risk score for patients with mild cognitive impairment in a memory clinic population: Development and validation of a dementia risk score using routinely collected data. Alzheimer's Dement. 2022;8:e12301. 10.1002/trc2.12301
Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp‐content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf
REFERENCES
- 1. Mitchell AJ, Shiri‐Feshki M. Rate of progression of mild cognitive impairment to dementia – meta‐analysis of 41 robust inception cohort studies. Acta Psychiatr Scand. 2009;119(4):252‐265. [DOI] [PubMed] [Google Scholar]
- 2. Lopez OL, Becker JT, Chang YF, et al. Incidence of mild cognitive impairment in the Pittsburgh Cardiovascular Health Study – Cognition Study. Neurology. 2012;79(15):1599‐1606. 10.1212/WNL.0b013e31826e25f0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Cooper C, Sommerlad A, Lyketsos CG, Livingston G. Modifiable predictors of dementia in mild cognitive impairment: a systematic review and meta‐analysis. Am J Psychiatry. 2015;172(4):323‐334. [DOI] [PubMed] [Google Scholar]
- 4. Arvanitakis Z, Shah RC, Bennett DA. Diagnosis and management of dementia. JAMA. 2019;322(16):1589‐1599. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Goerdten J, Čukić I, Danso SO, Carrière I, Muniz‐Terrera G. Statistical methods for dementia risk prediction and recommendations for future work: a systematic review. Alzheimers Dement Transl Res Clin Interv. 2019;5(1):563‐569. 10.1016/j.trci.2019.08.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Hou XH, Feng L, Zhang C, Cao XP, Tan L, Yu JT. Models for predicting risk of dementia: a systematic review. J Neurol Neurosurg Psychiatry. 2019;90(4):373‐379. 10.1136/jnnp-2018-318212 [DOI] [PubMed] [Google Scholar]
- 7. Folstein MF, Folstein SE, McHugh PR. “Mini‐mental state”: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12(3):189‐198. 10.1016/0022-3956(75)90026-6 [DOI] [PubMed] [Google Scholar]
- 8. Nasreddine ZS, Phillips NA, Bédirian V, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53(4):695‐699. 10.1111/j.1532-5415.2005.53221.x [DOI] [PubMed] [Google Scholar]
- 9. Weintraub S, Salmon D, Mercaldo N, et al. The Alzheimer's Disease Centers’ Uniform Data Set (UDS): the neuropsychological test battery. Alzheimer Dis Assoc Disord. 2009;23(2):91‐101. 10.1097/WAD.0b013e318191c7dd [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Beekly DL, Ramos EM, van Belle G, et al. The National Alzheimer's Coordinating Center (NACC) Database: an Alzheimer disease database. Alzheimer Dis Assoc Disord. 2004;18(4):270‐277. [PubMed] [Google Scholar]
- 11. Morris JC, Weintraub S, Chui HC, et al. The Uniform Data Set (UDS): clinical and cognitive variables and descriptive data from Alzheimer Disease Centers. Alzheimer Dis Assoc Disord. 2006;20(4):210‐216. 10.1097/01.wad.0000213865.09806.92 [DOI] [PubMed] [Google Scholar]
- 12. Sheikh F, Ismail Z, Mortby ME, et al. Prevalence of mild behavioral impairment in mild cognitive impairment and subjective cognitive decline, and its association with caregiver burden. Int Psychogeriatr. 2018;30(2):233‐244. 10.1017/S104161021700151X [DOI] [PubMed] [Google Scholar]
- 13. Petersen RC, Aisen PS, Beckett LA, et al. Alzheimer's Disease Neuroimaging Initiative (ADNI). Neurology. 2010;74(3):201‐209. 10.1212/WNL.0b013e3181cb3e25 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Albert MS, DeKosky ST, Dickson D, et al. The diagnosis of mild cognitive impairment due to Alzheimer's disease: recommendations from the National Institute on Aging‐Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement. 2011;7(3):270‐279. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. McKhann G, Drachman D, Folstein M, Katzman R, Price D, Stadlan EM. Clinical diagnosis of Alzheimer's disease: report of the NINCDS‐ADRDA Work Group under the auspices of Department of Health and Human Services Task Force on Alzheimer's Disease. Neurology. 1984;34(7):939‐944. 10.1212/wnl.34.7.939 [DOI] [PubMed] [Google Scholar]
- 16. McKhann GM, Knopman DS, Chertkow H, et al. The diagnosis of dementia due to Alzheimer's disease: recommendations from the National Institute on Aging‐Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement J Alzheimers Assoc. 2011;7(3):263‐269. 10.1016/j.jalz.2011.03.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Public Health Agency of Canada. Canadian Chronic Disease Surveillance System (CCDSS): Case Definitions. Published online 2017. https://health‐infobase.canada.ca/ccdss/data‐tool/
- 18. Grambsch PM, Therneau TM. Proportional hazards tests and diagnostics based on weighted residuals. Biometrika. 1994;81(3):515‐526. 10.1093/biomet/81.3.515 [DOI] [Google Scholar]
- 19. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer International Publishing; 2019. 10.1007/978-3-030-16399-0 [DOI] [Google Scholar]
- 20. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170(1):W1‐W33. 10.7326/M18-1377 [DOI] [PubMed] [Google Scholar]
- 21. Harrell FE, Jr . Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Springer; 2015. [Google Scholar]
- 22. Stone CJ, Hansen MH, Kooperberg C, Truong YK. Polynomial splines and their tensor products in extended linear modeling. Ann Stat. 1997;25(4):1371‐1425. [Google Scholar]
- 23. Kooperberg C, Stone CJ, Truong YK. Hazard regression. J Am Stat Assoc. 1995;90(429):78‐94. [Google Scholar]
- 24. Li JQ, Tan L, Wang HF, et al. Risk factors for predicting progression from mild cognitive impairment to Alzheimer's disease: a systematic review and meta‐analysis of cohort studies. J Neurol Neurosurg Psychiatry. 2016;87(5):476‐484. 10.1136/jnnp-2014-310095 [DOI] [PubMed] [Google Scholar]
- 25. Kennelly SP, Lawlor BA, Kenny RA. Blood pressure and the risk for dementia: a double edged sword. Ageing Res Rev. 2009;8(2):61‐70. 10.1016/j.arr.2008.11.001 [DOI] [PubMed] [Google Scholar]
- 26. Mungas D, Gavett B, Fletcher E, Farias ST, DeCarli C, Reed B. Education amplifies brain atrophy effect on cognitive decline: Implications for cognitive reserve. Neurobiol Aging. 2018;68:142‐150. 10.1016/j.neurobiolaging.2018.04.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. January CT, Wann LS, Calkins H, et al. 2019 AHA/ACC/HRS Focused Update of the 2014 AHA/ACC/HRS guideline for the management of patients with atrial fibrillation: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines and the Heart Rhythm Society. J Am Coll Cardiol. 2019;74(1):104‐132. 10.1016/j.jacc.2019.01.011 [DOI] [PubMed] [Google Scholar]
- 28. van Doorn S, Debray TPA, Kaasenbrood F, et al. Predictive performance of the CHA2DS2‐VASc rule in atrial fibrillation: a systematic review and meta‐analysis. J Thromb Haemost JTH. 2017;15(6):1065‐1077. 10.1111/jth.13690 [DOI] [PubMed] [Google Scholar]
- 29. Tang EYH, Harrison SL, Errington L, et al. Current developments in dementia risk prediction modelling: an updated systematic review. PLoS One. 2015;10(9):e0136181. 10.1371/journal.pone.0136181 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Knopman DS, Haeberlein SB, Carrillo MC, et al. The National Institute on Aging and the Alzheimer's Association research framework for Alzheimer's disease: perspectives from the research roundtable. Alzheimers Dement. 2018;14(4):563‐575. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Zhao A, Li Y, Yan Y, et al. Increased prediction value of biomarker combinations for the conversion of mild cognitive impairment to Alzheimer's dementia. Transl Neurodegener. 2020;9(1):1‐10. 10.1186/s40035-020-00210-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supporting Information