How good are medical and death records for identifying dementia?

Abstract

INTRODUCTION:

Retrospective studies using administrative data may be an efficient way to assess risk factors for dementia if diagnostic accuracy is known.

METHODS:

Within-individual clinical diagnoses of Alzheimer’s disease (AD) and all-cause dementia in ambulatory (outpatient) surgery, inpatient, Medicare administrative records and death certificates were compared with research diagnoses among participants of Cache County Study (1995–2008, N=5092).

RESULTS:

Combining all sources of clinical health data increased sensitivity to identify all-cause dementia (71%) and AD (48%), while maintaining relatively high specificity (81% and 93%, respectively). Medicare claims had the highest sensitivity for case identification (57% and 40%, respectively).

DISCUSSION:

Administrative health data may provide a less accurate method of identifying individuals with dementing disease than a research evaluation, but accuracy is improved by combining health data sources. Assessing all-cause dementia versus a specific cause of dementia such as AD will result in increased sensitivity, but at a cost to specificity.

1. Introduction

The prevalence of dementia, including Alzheimer’s disease (AD) and related dementias (RD) is increasing worldwide, with a trebling of the expected number of people living with dementia over the next 35 years [1]. Consequently, aging researchers are advocating for more methodologically sound studies assessing risk factors for dementia in hopes of reducing dementia related morbidity and mortality [2].

While increasing age and family history are significant predictors of dementia [3], socio-demographic and behavioral factors also play a role [4,5]. Currently, it is estimated that at least 40% of dementia risk can be attributed to potentially modifiable factors, many of which begin at early and mid-life [2,6]. The time lag between these early and mid-life exposures and the onset of dementia, and the limited size of epidemiological studies, make routinely collected health data available in much larger cohorts a promising vehicle for carrying out risk factor research [7]. However, before such research can be undertaken, an understanding of how well administrative health data accurately capture dementia within population based cohorts is needed [7,8].

We are aware of nearly 40 studies assessing accuracy for dementia case identification within administrative databases [8–15]. The majority of these studies focus on all-cause dementia [7], and draw from a single administrative dataset [7,8]. Furthermore, administrative databases in different countries vary in their accuracy of dementia diagnoses due to different clinical practices and variation in EHR recording. Therefore, we set out to compare clinical diagnosis (test method) of AD, RD, and all-cause dementia found in outpatient, inpatient, Medicare records, and death certificates with research diagnoses (reference method) in the same individuals who took part in the US-based Cache County Study on Memory, Health, and Aging (CCSMHA). Our primary objective was to assess the accuracy of AD, RD, and all-cause dementia captured via each source as well as all sources combined, with the assumption that information from different data sources could be complimentary and give a more accurate picture.

2. Methods

2.1. Study Population

Data for this study were obtained from the CCSMHA [3,4], a prospective epidemiological study of dementia that is remarkable for its comprehensive study of a complete population, with 90% (N=5,092) of the Cache County, Utah population aged ≥ 65 as of January 1, 1995 participating. Participants were followed over four triennial waves (12 years) for prospective ascertainment of cognitive impairment [4]. As reported previously, CCSMHA was 99% white, 58% female, and ranged in age from 65–105 years at baseline, with a mean (SD) of 75.9 (7.3) years at enrolment [4]. Range of formal education was 0–20 years, with a mean (SD) of 13.1 (2.9) years [4], and 65% were married, 30% widowed, 3% divorced or separated, and 2% had never married [16].

CCSMHA participants’ definitive dementia diagnoses, age of enrollment, and gender were linked with the Utah Population Database (UPDB) inpatient hospital claims, ambulatory surgery records, death certificates and Medicare claims (1995–2008). The University of Utah Institutional Review Board and Resource for Genetic and Epidemiologic Research in addition to the Utah State University Institutional Review Board approved this study.

2.2. CCSMHA Dementia Diagnoses (Reference Method, “Gold Standard”)

CCSMHA investigators assessed prevalent and incident dementia using gold-standard criteria [3,4]. The initial interviews occurred in 1995 (prevalence) and three follow-up waves 3, 7, and 10-years later (incidence); the last year of the assessments was 2008. For each wave, a multi-stage dementia ascertainment protocol was employed (Supplemental Table 1) [3,4,17–21].

Over the course of the four triennial waves, 942 (18.5%) persons were identified with dementia (335 prevalent cases; 607 incident cases). Of these, 58% had possible/probable AD, 11% had AD comorbid with RD, and 31% had RD [3, 4]. The remaining 4,150 participants (81.5%) were deemed cognitively normal or cognitively impaired no dementia (CIND) by end of follow up. Prior work has shown high sensitivity and specificity for identifying dementia over the first two waves of ascertainment [22, 23].

2.3. Death Certificate Dementia Diagnoses (Test Method 1)

Death certificates from 1996 to present included multiple causes of death (underlying, associated, and contributing causes of death) [24] Deaths occurring in 1995 among CCSMA participants listed only the underlying cause of death. International Classification of Diseases (ICD) 9 was used to categorize causes of death from 1995–1998 while ICD-10 was used to categorize causes of death from 1999–2008 (Table 1). Participants still alive by 2008 (16%) were excluded from the death certificate validation assessment.

Table 1:

ICD-9 and ICD-10 codes used for identifying Alzheimer’s disease and related dementia in administrative databases

ICD-9	Definition	ICD-10	Definition
331.0	Alzheimer’s disease	G30.0	Alzheimer’s disease with early onset
		G30.1	Alzheimer’s disease with late onset
		G30.9	Alzheimer’s disease, unspecified
ICD-9	Definition	ICD-10	Definition
290	Senile dementia, uncomplicated	G30.8	Other Alzheimer’s disease
290.0	Senile dementia, uncomplicated	F01	Vascular dementia
290.1	Presenile dementia (brain syndrome w/presenile dementia)	F01.5	Vascular dementia
290.10	Presenile dementia, uncomplicated	F01.50	Vascular dementia without behavioral disturbance
290.11	Presenile dementia with delirium	F01.51	Vascular dementia with behavioral disturbance
290.12	Presenile dementia with delusional features	F02	Dementia in other diseases classified elsewhere
290.13	Presenile dementia with depressive features	F02.8	Dementia in other diseases classified elsewhere
290.2	Senile dementia with delusional features	F02.80	Dementia in other diseases classified elsewhere w/out behavioral disturbance
290.20	Senile dementia with delusional features	F02.81	Dementia in other diseases classified elsewhere with behavioral disturbance
290.21	Senile dementia with depressive features	F03	Unspecified dementia
290.3	Senile dementia with delirium	F03.9	Unspecified dementia
290.4	Vascular dementia, uncomplicated	F03.90	Unspecified dementia without behavioral disturbance
290.40	Vascular dementia, uncomplicated	F03.91	Unspecified dementia with behavioral disturbance
290.41	Vascular dementia, with delirium	F05	Delirium due to known psychological condition
290.42	Vascular dementia, with delusions	F10.27	Alcohol dependence with alcohol-induced persisting dementia
290.43	Vascular dementia, with depressed mood	F19.97	Other psychoactive substance use, unspecified with persisting dementia
290.8	Unspecified dementia, without behavioral disturbance	G10	Huntington’s dementia
290.9	Unspecified dementia, without behavioral disturbance	G12.21	Amyotrphic lateral sclerosis (ALS) with dementia
291.2	Alcohol-induced persisting dementia	G31.0	Frontotemporal dementia
292.82	Drug-induced persisting dementia	G31.01	Pick’s disease
294.0	Amnestic syndrome	G31.09	Other frontotemporal dementia
294.1	Dementia in conditions classified elsewhere w/out behavioral disturbance	G31.1	Senile degeneration of brain, not elsewhere classified
294.10	Dementia in conditions classified elsewhere	G31.83	Dementia with Lewy bodies
294.11	Dementia in conditions classified elsewhere	G91.2	Normal pressure hydrocephalus (NPH)
294.2	Dementia, unspecified, without behavioral disturbance
294.21	Dementia, unspecified, with behavioral disturbance
294.8	Other specified organ brain syndrome (chronic)
331.1	Frontotemporal dementia
331.11	Pick’s disease
331.19	Other frontotemporal dementia
331.2	Senile degeneration of the brain
331.82	Dementia with lewy bodies
331.5	Normal pressure hydrocephalus (NPH)
333.4	Huntington’s dementia
335.20	Amyotrphic lateral sclerosis (ALS) with dementia
797	Senility without mention of psychosis

2.4. Ambulatory Surgery Records, Inpatient Hospital Claims, and Center for Medicare & Medicaid Services Claims (Test Methods 2–3)

Inpatient hospital claims and ambulatory (outpatient) surgery records [25] along with Medicare claims [26] did not transition to using ICD-10 until October 1, 2015. Thus, ICD-9 codes were used for dementia identification in these three sources throughout the study period (Table 1). We used the CCSMHA list of AD and RD dementia diagnoses to inform the AD and RD dementia diagnoses we identified in our administrative databases. Details on database management for administrative health records can be found in Supplemental Table 1.

2.6. Definitions of AD, AD Mixed, RD, and All-Cause Dementia within Administrative Databases

As per Table 1, AD included Alzheimer’s Disease while RD included vascular dementia, frontotemporal dementia/Pick’s, dementia with Lewy bodies, and other RDs. AD/AD mixed was AD only or AD comorbid with RD; RD was RD only with no AD; and all-cause dementia was any dementia (AD and/or RD).

2.7. Statistical Analysis

We calculated the prevalence (n [%]) of AD/AD Mixed, RD, and all-cause dementia for each of our administrative databases as well as the administrative databases combined. All dementia-related records were used for the classification of dementia subtypes. We calculated encounters per person by individual data sources and combined (median, IQR, and range) as well as whether there was a difference in number of encounters by all-cause dementia diagnosis (t-test). To visualize agreement of dementia diagnoses between the various databases, we created proportional Venn diagrams using eulerAPE [27].

We used several measures to assess the accuracy of the administrative databases (individually and overall) in correctly identifying dementia diagnoses (AD/AD Mixed, RD, and all-cause dementia) compared to our CCSMHA gold standard assessment including sensitivity, specificity, negative/positive predictive values (NPV/PPV), Area Under the Curve (AUC), and Cohen’s κ statistic. We used Landis and Koch’s [28] guidelines for interpreting κ statistics.

In addition to the analyses for the full sample, we performed stratified analyses by sex (male and female), median age of enrollment (≥77 vs <77 years), and 1) restricted to only administrative records that occurred prior to CCSMHA dementia diagnosis, or 2) incident cases only. We also assessed accuracy of administrative databases to identify key RD dementia subtypes (vascular dementia, frontotemporal dementia, and dementia with Lewy bodies) as compared to CCSMHA diagnoses. All analyses were conducted using SAS 9.4.

3. Results

3.1. Population Characteristics and ADRD Prevalence and Agreement

Our study population included 5,011 (98%) CCSMHA participants who could be linked to at least one of the administrative data sources (Table 2). Across the study period (1995–2008), all 5011 participants had at least one ambulatory surgery, inpatient hospital, or Medicare claim encounter (median=49; IQR=22,100; range=1–626). 3199 (64%) of the participants had ≥1 ambulatory surgery encounter (median=8; IQR=4,15; range=1–113), 3851 (77%) had ≥1 inpatient hospital encounter (median=18; IQR=9,30; range=1–190), and 3825 (76%) had ≥1 Medicare encounter (median=43; IQR=20,83; range=1–429). When considering all databases combined, participants with a CCSMHA all-cause dementia diagnosis had slightly fewer encounters compared to those without dementia (68.6 versus 73.5, P=0.051).

Table 2.

Baseline characteristics among CCSMHA participants linking to UPDB administrative databases, n=5011 (1995–2008)

Characteristics	Total (N=5,011)
Age at enrollment (CCSMHA)	5,011	77.2 ± 7.3
Gender
Male	2,135	42.6
Female	2,876	57.4
Deceased (as of 2008)
Yes	4,192	83.7
No	819	16.3
Age at the dementia diagnosis (CCSMHA)	928	86.0 ± 6.5
Cache County Study of Memory, Health and Aging
Alzheimer’s Disease/Alzheimer’s Disease Mixed*	628	12.5
Related Dementias	300	6.0
UPDB Administrative Databases
Alzheimer’s Disease/Alzheimer’s Disease Mixed†
Overall (Ambulatory or Inpatient or Death Records or Medicare)	609	12.2
Ambulatory Surgery (ICD9)	19	0.4
Inpatient (ICD9)	208	4.2
Death Records (ICD9-ICD10) ‡	193	4.6
Primary Causes of Death (among Death Records) ‡	62	1.5
Medicare (ICD9)	512	10.2
Related Dementias
Overall (Ambulatory or Inpatient or Death Records or Medicare)	832	16.6
Ambulatory Surgery (ICD9)	17	0.3
Inpatient (ICD9)	317	6.3
Death Records (ICD9-ICD10) ‡	236	5.6
Primary Causes of Death (among Death Records) ‡	45	1.1
Medicare (ICD9)	595	11.9

CCSMHA: Cache County Study of Memory, Health, and Aging; UPDB: Utah Population Database

^*n=529 AD and n=99 AD Mixed;

^†n=56 AD and n=553 AD Mixed;

^‡Out of those who are deceased as of 2008 (n=4192)

The prevalence of AD/AD Mixed and RD in the linked CCSMHA was 12.5% and 6.0%, respectively, while overall for the administrative databases it was 12.2% and 16.6%, respectively (Table 2). Of the 628 AD/AD Mixed in the CCSMHA, 529 (84%) were diagnosed with AD alone and 99 (16%) were diagnosed with AD Mixed whereas in the administrative databases, 56 (9%) were diagnosed with AD alone and 553 (91%) were diagnosed with AD Mixed.

Overlap between inpatient hospital claims, Medicare claims, and CCSMHA diagnoses was much higher compared to ambulatory surgery, death certificates, and CCSMHA diagnoses, with the former having a 16% overlap of all three sources (among the 1534 all-cause dementia cases) while the latter having only a 1% overlap of all three sources (among the 1078 all-cause dementia cases) (Figures 1 and 2). When just considering the administrative health databases, Medicare and inpatient hospital claims had the greatest overlap, followed by inpatient hospital and death certificates. Ambulatory surgery records had the lowest overlap with any of the other records (Supplemental Figures 1 and 2).

Figure 1:

Venn diagram representing overlap of All-cause Dementia diagnoses in Inpatient, Medicare, and Cache County Study on Memory, Health, and Aging (N=5011).

Figure 2:

Venn diagram representing overlap of All-cause Dementia diagnoses in Outpatient, Death Certificates, and Cache County Study on Memory, Health, and Aging among deceased participants (N=4192).

The sensitivity of administrative databases overall relative to the CCSMHA for AD/AD Mixed, RD, and all-cause dementia was 48%, 36%, and 71%, respectively; while the specificity was 93%, 85%, and 81% respectively (Table 3). Medicare claims had the highest sensitivity, followed by death certificates, and inpatient claims with ambulatory surgery records providing the lowest sensitivity. Specificity across record types was more consistent (range 93% to 100% for AD/AD Mixed, 85% to 100% for RD, and 81% to 100% for all-cause dementia).

Table 3.

Validity of dementia diagnoses in the UPDB administrative databases using CCSMHA diagnoses as the gold standard

	Alzheimer’s Disease/Alzheimer’s Disease Mixed
Statistics	Overall UPDB vs CCSMHA	AS vs CCSMHA	IC vs CCSMHA	DC vs CCSMHA*	Medicare vs CCSMHA
n (%)	609 (12%) vs 628 (13%)	19 (0.3%) vs 628 (13%)	208 (4%) vs 628 (13%)	193 (5%) vs 615 (15%)	512 (10%) vs 628 (13%)
Sensitivity	47.9	1.9	18.0	19.2	39.5
Specificity	93.0	99.8	97.8	97.9	94.0
PPV	49.4	63.2	54.3	61.1	48.4
NPV	92.6	87.7	89.3	87.6	91.6
AUC	0.70	0.51	0.58	0.59	0.67
Cohen’s k	0.41	0.03	0.22	0.24	0.36
	Related Dementias
Statistics	Overall UPDB vs CCSMHA	AS vs CCSMHA	IC vs CCSMHA	DC vs CCSMHA*	Medicare vs CCSMHA
n (%)	832 (17%) vs 300 (6%)	17 (<0.1%) vs 300 (6%)	317 (6%) vs 300 (6%)	236 (6%) vs 298 (7%)	595 (12%) vs 300 (6%)
Sensitivity	35.7	0.003	14.7	19.1	25.7
Specificity	84.6	99.7	94.2	95.4	89.0
PPV	12.9	5.8	13.9	24.2	12.9
NPV	95.4	94.0	94.6	93.9	95.0
AUC	0.60	0.50	0.54	0.57	0.57
Cohen’s k	0.11	0.001	0.09	0.16	0.10
	All-Cause Dementia
Statistics	Overall UPDB vs CCSMHA	AS vs CCSMHA	IC vs CCSMHA	DC vs CCSMHA*	Medicare vs CCSMHA
N (%)	1441 (29%) vs 928 (19%)	36 (1%) vs 928 (19%)	525 (11%) vs 928 (19%)	429 (10%) vs 913 (22%)	1107 (22%) vs 928 (19%)
Sensitivity	71.0	2.4	30.2	30.2	57.1
Specificity	80.9	99.7	94.0	95.3	85.9
PPV	45.7	61.1	53.3	64.1	47.9
NPV	92.5	81.8	85.6	83.0	89.8
AUC	0.76	0.51	0.62	0.63	0.71
Cohen’s k	0.43	0.03	0.29	0.31	0.40

AS: Ambulatory (Outpatient) Surgery; AUC: Area Under the Curve; CCSMHA: Cache County Study of Memory, Health, and Aging; Cohen’s k=Cohen’s Kappa Coefficient; DC: Death Certificates; IC: Inpatient Hospital Claims; NPV: Negative Predictive Value; PPV: Positive Predictive Value; UPDB: Utah Population Database

^*Among only those who are deceased. κ between 0.00 and 0.20 indicates slight agreement; κ between 0.21 and 0.40 denotes fair agreement; κ between 0.41 and 0.60 characterizes moderate agreement; κ between 0.61 and 0.80 defines substantial agreement and a value of κ greater than 0.80 equates to almost perfect agreement.

The PPV of administrative databases overall relative to the CCSMHA for AD/AD Mixed, RD, and all-cause dementia was 49%, 13%, and 46%, respectively; while the NPV was 93%, 95%, and 93% respectively (Table 3). Ambulatory surgery records and death certificates tended to have the highest PPVs followed by in-patient records and Medicare claims. PPVs for RD were low across all sources.

Slight to moderate agreement was found between administrative databases combined and CCSMHA gold standard assessment for AD/AD Mixed (κ=0.41), RD (κ=0.11), or all-cause dementia (κ=0.43).

3.3. Sensitivity analyses; stratified analyses based on gender and age

Our stratified analyses indicated that women had slightly better agreement than men in regards to AD/AD Mixed but similar estimates for RD or all-cause dementia (Table 4). Similarly, younger participants (<77 years) at the age of enrollment had better agreement than older participants for all-cause dementia but similar estimates for RD.

Table 4.

Validity of dementia diagnoses in the UPDB administrative databases* compared to CCSMHA diagnoses as the gold standard

	Alzheimer’s Disease/Alzheimer’s Disease Mixed
Statistics	Women UPDB vs CCSMHA	Men UPDB vs CCSMHA	Age ≥77 years UPDB vs CCSMHA	Age <77 years UPDB vs CCSMHA	Restricted to prior diagnoses UPDB vs CCSMHA
n (%)	400 (14%) vs 414 (14%)	209 (10%) vs 214 (10%)	409 (17%) vs 474 (19%)	200 (8%) vs 154 (6%)	529 (11%) vs 628 (13%)
Sensitivity	50.5	43.0	45.2	56.5	38.7
Specificity	92.2	93.9	90.2	95.3	93.5
PPV	52.3	44.0	52.3	43.5	45.9
NPV	91.7	93.7	87.4	97.1	91.4
AUC	0.71	0.68	0.68	0.76	0.66
Cohen’s k	0.43	0.37	0.37	0.45	0.35
	Related Dementias
Statistics	Women UPDB vs CCSMHA	Men UPDB vs CCSMHA	Age ≥77 years UPDB vs CCSMHA	Age <77 years UPDB vs CCSMHA	Restricted to prior diagnoses UPDB vs CCSMHA
n (%)	522 (18%) vs 156 (5%)	310 (15%) vs 144 (7%)	579 (24%) vs 205 (8%)	253 (10%) vs 95 (4%)	814 (16%) vs 300 (6%)
Sensitivity	39.1	31.9	39.5	27.4	30.3
Specificity	83.0	86.7	78.0	90.7	84.7
PPV	11.7	14.8	14.0	10.3	11.2
NPV	96.0	94.6	93.4	97.0	95.0
AUC	0.61	0.59	0.59	0.59	0.57
Cohen’s k	0.11	0.12	0.10	0.10	0.08
	All-Cause Dementia
Statistics	Women UPDB vs CCSMHA	Men UPDB vs CCSMHA	Age ≥77 years UPDB vs CCSMHA	Age <77 years UPDB vs CCSMHA	Restricted to prior diagnoses UPDB vs CCSMHA
N (%)	922 (32%) vs 570 (20%)	519 (24%) vs 358 (17%)	988 (40%) vs 679 (28%)	453 (18%) vs 249 (10%)	1343 (26%) vs 928 (19%)
Sensitivity	74.0	66.2	71.3	71.3	60.5
Specificity	78.3	84.1	71.8	71.8	80.9
PPV	45.8	45.7	49.0	49.0	41.8
NPV	92.4	92.5	86.8	86.8	90.0
AUC	0.76	0.75	0.72	0.79	0.71
Cohen’s k	0.42	0.43	0.38	0.43	0.35

AUC: Area Under the Curve; CCSMHA: Cache County Study of Memory, Health, and Aging; Cohen’s k=Cohen’s Kappa Coefficient; NPV: Negative Predictive Value; PPV: Positive Predictive Value; UPDB: Utah Population Database. κ between 0.00 and 0.20 indicates slight agreement; κ between 0.21 and 0.40 denotes fair agreement; κ between 0.41 and 0.60 characterizes moderate agreement; κ between 0.61 and 0.80 defines substantial agreement and a value of κ greater than 0.80 equates to almost perfect agreement.

3.4. Sensitivity analyses; restricting to administrative health data prior to CCSMHA ADRD diagnoses or incident cases only.

Restricting analyses to include only administrative records recorded prior to CCSMHA AD and/or RD diagnoses resulted in decreased sensitivity for AD/AD Mixed (39%), RD (30%) and all-cause dementia (61%) compared to including all records regardless of whether diagnosis came before or after CCSMHA dementia diagnosis but with little change in specificity (Table 4). Restricting the sample to only incident dementia cases resulted in a slight decrease in sensitivity for AD/AD Mixed (42%) and all-cause dementia (69%), but no change in specificity.

3.5. Sensitivity analyses; RD subtypes

We found little agreement when looking at specific related dementia subtypes. While vascular dementia, frontotemporal dementia, and dementia with Lewy bodies prevalence were similar within the UPDB versus CCSMHA (3.6% versus 3.9%, 0.1% versus 0.2%, and 0.5% versus 0.2%, respectively), sensitivity was low (14%, 0%, and 8%, respectively) while specificity was high (97%, 100%, 100%), respectively.

4. Discussion

4.1. Summary of findings

In this population-based study assessing the accuracy of dementia diagnoses within four different health/vital record data sources, we found relatively high sensitivity (71%) but low PPV (46%) for all-cause dementia compared to prior studies [7,8]. Medicare records contributed the greatest to sensitivity (57%) followed by hospital, death certificates (both 30%), and outpatient (2%) data. Similar to prior studies, we found higher sensitivity and PPV for AD/AD mixed compared to RD as well as a higher PPV among older versus younger patients [7]. Restricting our administrative databases to include only information collected prior to our gold standard dementia diagnosis resulted in only a slight reduction in our all-cause dementia sensitivity (71% reduced to 61%) and PPV (46% to 42%).

4.2. Prevalence

Prevalence of AD/AD mixed and RD within CCSMHA (12.5% and 6.0%) is comparative to other population-based studies with detailed clinical evaluations [29, 30]. Our administrative databases reported near equal prevalence of AD/AD Mixed (12.2%) as compared to the CCSMHA (12.5%). The higher prevalence for RD (as well as AD Mixed compared to AD alone) found in our administrative databases (16.6%) as compared to the CCSMHA (6.0%) may have been due to our reliance on final diagnoses for the CCSMHA as opposed to any diagnosis for the administrative database. Additionally, research studies with rigorous clinical evaluation for dementia are better able to phenotype dementia as compared to administrative databases. Future work within our administrative databases requiring minimal number of times (e.g., ≥ 2 ) a diagnosis code must occur to be counted as an AD or RD case [26], or using only the last diagnosis to count as an AD or RD case may be warranted. However, given the high AD or RD specificity others and we found [8], this may increase the false negative rate and lead to greater misclassification bias.

4.3. Sensitivity

Of the prior fourteen studies assessing sensitivity of dementia diagnoses in administrative health databases, only three scored higher sensitivities than our study [8,32,33]. While sensitivity does not depend on prevalence of disease within a population, it is influenced by characteristics of the population. Given that prior studies have found that dementia is more likely to be recorded correctly in routinely collected health records among white, married, and female populations [8], the make-up of our population sharing these characteristics may have contributed to our higher sensitivity.

Additionally, prior US research has shown that studies using Medicare data [14, 32, 33] report higher sensitivities compared to hospital admission or death certificate data [7]. Specifically, Taylor et al found 86% sensitivity when comparing AD/AD mixed diagnoses from the Aging Demographics and Memory Study (n=758) with Medicare claims (2001–2003) [33], Lee et al found 85% sensitivity comparing clinical diagnosis of dementia versus Medicare claims (2007–2012) among a sample of 147 University of Southern California Alzheimer Disease Research Study patients [32]; while Zhu et al found sensitivity of Medicare claims-identified dementia was 51% as compared to clinical assessment among the Washington Heights-Inwood Columbia Aging Project cohort (n=2196, 1999–2010) [14].

While Medicare claims consistently show the strongest ability to detect dementia in the US [7], hospital administrative databases may be adequate in countries with universal health coverage. For example, our sensitivity for hospital admission data was less than half that found in a recent UK study of 21,387 people from a large mental health care database linked to 2008–2016 hospital data [8]. It is intuitive that a clinic-based sample of patients being referred for psychiatric care will result in a higher sensitivity compared to a population cohort that includes those with and without mental health disorders as in the CCSMHA, providing a potential more “real world” sensitivity. Other reasons for the discrepancy between our and the UK study may be due to the differences in study time period, given the increasing rates of dementia diagnoses in UK hospitals [8]; the more organized system of memory assessment clinics in the UK [34], or simply due to better capture of EHR diagnoses within the UK versus US healthcare system.

4.4. Specificity

Specificity was relatively high as has been found in prior studies [33]. Despite high specificity, 13% (n=629) of participants who were considered as non-dementia patients by CCSMHA assessment had a dementia diagnosis within the administrative health records.

Prior research supports that the majority of individuals with ADRD will not seek a medical diagnosis until at least 2 years after symptom onset [36, 37], with median survival time from diagnosis to death being 5 years, ranging from 2.7 to 9.4 years depending on age and gender [38]. Consequently, the CCSMHA triennial waves were an appropriate timeframe for dementia assessment. However, there are still individuals within a triennial timeframe who will receive a dementia diagnosis and die among a cohort aged 65 years or older. Given prior research and our research indicating high specificity for ADRD, we assume that the vast majority of those diagnosed with dementia in the administrative data are true positives. Prior healthy community-based samples with relatively long life expectancies, such as the CCSMHA have found that 55% (95% CI, 49–60) of individuals aged 60 years go on to develop some type of dementia during their lifetime [39]. With 84% of our population deceased by 2008 and a combined overall prevalence of 41% for all-cause dementia if all the administrative health records and CCSMHA data sources are used, we are in line with other population-based studies for lifetime dementia risk [39, 40]. Consequently, while further research is needed to corroborate our findings, we believe that future ADRD risk factor research consider combining administrative healthcare records with ADRD research cohorts, when available, for casting the widest net possible in accurately detecting dementia cases.

4.5. Positive Predictive Value

Our PPV of 49% for AD/AD mixed, 13% for RD, and 46% for all-cause dementia is relatively low compared to prior studies assessing PPV [7,11,41], which has averaged internationally at around 77% (range: 35% to 100%). Even a direct comparison of our study with other US studies that occurred during the same time frame [7,33,41–43] averaging 78% (range: 69% to 93%), reveals our low PPV. Given that PPV increases with prevalence of disease, studies using clinical populations [41] or population-based studies with high dementia prevalence [33] are not comparable.

4.6. Strengths and Limitations

Our study had many strengths. First, our population was community based, enrolling 90% of total residents in a US county. Given that the diagnosis and treatment of dementia is predominately community based, we may have been able to identify cases that would not appear in a solely clinically-based cohort [7]. However, our use of a true community based population also may explain our relatively lower PPVs compared to prior US studies. Of note, if the goal of a high PPV is to reduce the number of false positives (non-dementia individuals misidentified as dementia cases) [7], our high specificity (93% for AD/AD Mixed and 71% for all-cause dementia) demonstrates our administrative databases’ overall ability to correctly identify non-dementia among true non-dementia cases. Additional strengths of our study, compared to prior research, include our ability to assess agreement using a combination of data sets as well as for dementia subtypes including AD and vascular dementia [7]. This speaks to the value of having population registries comprising linked records from a variety of sources. Linked records bring insight on the full spectrum of patient journey coming from various stakeholders including payer, healthcare providers, and governmental organization.

Finally, our study was able to investigate identification of both prevalent and incident dementia diagnoses. The resulting decrease in sensitivity we found when removing the prevalent cases is intuitive, as prevalent cases would have a longer time to show up in the administrative health databases prior to enrollment compared to incident cases. However, our study design in which we restricted UPDB records to the same time frame by year as the CCSMHA study (1995–2008) resulted in very little change in sensitivity compared to if we had pulled all prior records available to us (e.g., Medicare data going back to 1992). Our sensitivity analysis restricting to only incident cases gave us reassurance that our results were not overly biased by combining the prevalent and incident cases.

Our study does have several limitations. First, we were not able to include medication prescription data or unstructured data such as full-text medical records inclusive of clinical notes. Prior research has found that use of medication data to identify dementia cases can lead to a PPV of 97% [44], while use of natural language processing of clinical notes with coded data can produce a PPV of >92% [45]. Secondly, our study may lack generalizability in regards to time and location. Specifically, our study assessed validity of administrative databases from 1995–2008. The utility of medical records data is likely to improve further and become even more valuable. Given prior research indicating an improvement in sensitivity of dementia captured from mortality data from 2006 to 2013 [8], healthcare professionals may be increasing their ability to both adequately diagnose and record diagnoses related to cognitive impairment and dementia in healthcare records. Advances in diagnostic technology and better and more specific drug treatments also will enhance the value of health records. Additionally, while our population is representative of an entire county of Utah having enrolled 90% of the county residents over age 65 years, it has limited racial/ethnic diversity and thus caution is warranted in extrapolating our findings to non-white and/or non-Hispanic populations.

Finally, while our assignment of non-dementia including both individuals who were cognitively normal and those who had CIND follows prior research [7], a better understanding for the impact of including individuals with CIND as dementia cases is needed given close proximity of codes between dementia subtypes and CIND (e.g., ICD-9 code for mild cognitive impairment is 331.83 while code for dementia with Lewy bodies is 331.82). With only 35 (0.7%) total cases of mild cognitive impairment within our administrative databases, we were limited in our ability to assess how including versus excluding CIND in our comparison (non-dementia) population affects accuracy.

4.7. Conclusion

In this true population cohort, with an all-cause dementia prevalence of 19% over 12-years of follow up, we found moderate agreement and a relatively high sensitivity and specificity for detecting all-cause dementia diagnoses, notably when using a combination of linked databases including inpatient, outpatient, death, and insurance claims data. While using a combination of linked databases can increase accuracy in identifying individuals with dementing disease, further work to minimize false positives (in order to reduce distortion of risk estimates) while at the same time reducing false negatives (in order to ensure dementia cases are representative and to maximize statistical power) is warranted, potentially via incorporating multiple EHR components in addition to ICD codes not considered in this study, such as clinical notes through natural processing and specific medications identified from pharmacy records [7,43].

HIGHLIGHTS:

• Electronic health records (EHR) are increasingly being used in dementia research

• Trade-off between sensitivity (“rule in”) and specificity (“rule out”)

• High specificity needed for etiological studies; high sensitivity for detection

• 48% sensitivity, 93% specificity for Alzheimer’s disease/Alzheimer’s disease comorbid with a related dementia; and 71% sensitivity, 81% specificity for all-cause dementia comparing clinical (EHR) versus research assessed diagnoses.

RESEARCH IN CONTEXT:

1. Systematic Review: A recent systematic review identified 27 studies assessing all-cause dementia diagnostic accuracy within administrative health databases with sensitivities ranging 21–86% and PPVs ranging 33–100%. Our population-based study was novel in assessing accuracy of clinically diagnosed dementia in multiple sources—hospital admissions, outpatient, death, and insurance (Medicare) records— as compared to gold-standard research diagnoses.

2. Interpretation: We found an overall sensitivity of 71% for all-cause dementia and a PPV of 46%. Given that PPV is dependent on disease prevalence, our relatively low dementia prevalence (19%) may partly explain relatively poor PPV. Medicare claims had the highest sensitivity (57%) followed by hospital admission and death certificates (30%), and outpatient (2%) data.

3. Future Directions: Incorporating multiple health record components in addition to ICD codes not considered in this study, such as clinical notes through natural language processing and specific medications identified from pharmacy records, is warranted.

ABBREVIATIONS:

AS

Ambulatory (Outpatient) Surgery

AD

Alzheimer’s Disease

AUC

Area Under the Curve

CCSMHA

Cache County Study on Memory, Health, and Aging

CIND

Cognitive Impairment Not Demented

DC

Death Certificates

EHR

Electronic Health Records

IC

Inpatient Hospital Claims

NPV

Negative predictive value

PPV

Positive predictive value

RD

Related Dementias

UPDB

Utah Population Database