The diagnostic accuracy of the Ascertain Dementia 8 questionnaire for detecting cognitive impairment in primary care in the community, clinics and hospitals: a systematic review and meta-analysis

Abstract

Background

The prevalence of cognitive impairment is increasing due to the aging population, and early detection is essential clinically. The Ascertain Dementia 8 (AD8) questionnaire is a brief informant-based measure recently developed to assess early cognitive impairment, however, its overall diagnostic performance is controversial. The objective of this meta-analysis was to assess the diagnostic accuracy of the AD8 for cognitive impairment.

Methods

All relevant studies were collected from databases including MEDLINE, EMBASE and the Cochrane Library up to April 2017. We used QUADAS-2 to assess the methodological quality after the systematic search. The accuracy data and potential confounding variables were extracted from the eligible studies which included those in English and non-English. All analyses were performed using the Midas module in Stata 14.0 and Meta-DiSc 1.4 software.

Results

Seven relevant studies including 3728 subjects were collected, and classified into two subgroups according to the severity of cognitive impairment. The overall sensitivity (0.72, 0.91) was superior to specificity (0.67, 0.78). The pooled negative likelihood ratio (0.17, 0.13) was better than the positive likelihood ratio (2.52, 3.94). The areas under the summary receiver operating characteristic curve were 0.83 and 0.92, respectively. Meta-regression analysis showed that location (community versus non-community) may be the source of heterogeneity. The average administration time was less than 3 minutes.

Conclusion

Our findings suggest that the AD8 is a competitive tool for clinically screening cognitive impairment and has an optimal administration time in the busy primary care setting. Subjects with an AD8 score ≧2 should be highly suspected to have cognitive impairment and a further definite diagnosis is needed.

Introduction

Dementia is a public health issue worldwide. The prevalence continues to increase due to the aging population, resulting in higher rates of disability and dependency among the elderly. According to a report by the WHO in April 2016, 47.5 million people have dementia with 7.7 million new cases being diagnosed every year (1). However, a major challenge of slowing progression of the devastating effects of dementia is how to best detect its earliest stage by primary care providers. And advanced stages of the disease are related to greater social and economic impacts and poorer response to current treatment (1,2). According to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-V) in 2013 updated by the American Psychiatric Association’s (APA), the term ‘dementia’ has been replaced by ‘has been replaced by ssociatio and the less severe type ‘nd the less severe type ciationl of Mental Disordersf the devahe preclinical stage or pre-prodromal phase of dementia (3,4). The change and evolution of the diagnostic criteria emphasizes the importance of earlier prevention, detection and treatment of cognitive impairment in primary care (2).

Currently, many cognitive screening tools are used to identify people at risk of cognitive impairment in primary care in the community and clinics, however, each has its limitations. For example the Mini-Mental State Examination (MMSE) is a commonly used performance-based cognitive screening measure, however, it has a ceiling effect which makes it difficult to detect the earliest signs of cognitive impairment, especially in the highly educated elderly (5). The Montreal Cognitive Assessment (MoCA) requires trained staff and takes about 10–15 minutes to administer, which is inappropriate for use in a busy primary care setting (6,7). The capacity of the Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) to detect very mild dementia has not been validated, and its cut-off points remain unclear, and the sensitivities of the Clock Drawing Test and Mini-Cog are too low to allow their use as appropriate screening tools for cognitive impairment (6).

The Ascertain Dementia 8 (AD8) questionnaire was developed by Washington University in St Louis in 2005. It is a brief informant-based measure and is considered to be better than performance-based tools (8,9). With only eight questions including domains of judgement, hobby/activity level, repetitive conversations, learning ability, memory in relation to date/appointments, finances and daily thought processes, it takes less than 3 minutes to complete (9). The English version of the AD8 has been translated into Taiwanese, Korean, Portuguese and Japanese, and it has been validated in the community, primary health care centres and hospitals (5,10,11). The apparent benefits of the AD8 are the simple scoring system, minimal training required and being less prone to bias from different cultures and education level (7,12,13). However, the test performance of the AD8 has been shown to differ considerably due to differences in the clinical setting, severity of disease, reference standards and even cut-off points (14–16). To the best of our best knowledge, no synthesis of the diagnostic accuracy of the AD8 has been performed. Therefore, the objective of this systematic review was to determine the diagnostic accuracy of the AD8 in all-cause cognitive impairment in a primary care setting in the community, clinics and hospitals. We further investigated the potential heterogeneity and analyzed subgroups of different stages of cognitive impairment and different clinical settings to clarify the accuracy of the AD8.

Methods

Data source and search strategy

We used the search term ‘AD8’ without restrictions of language to conduct a comprehensive search of UpToDate, Cochrane Library, PubMed/Medline, Embase, PsycINFO, PerioPath Index to Taiwan Periodical Literature, Airiti Library and Google Scholar. We also checked the reference lists of all relevant studies to identify further articles for possible inclusion in this review. With regards to secondary evidence (filtered resources), we only found one section about the validity and reliability of the questionnaire in an article in UpToDate by the developer of the AD8 (17). We searched 20 Cochrane Reviews, however only one matched the topic, and this was not included as it was only a protocol (13). Regarding the systematic review of PubMed/Medline, one synthesis by the US Preventive Services Task Force (USPSTF) in 2013 related to screening cognitive impairment in older adults was identified (6). However, the authors could not comment on the consistency of the findings of the AD8 because it was only evaluated in one study which was conducted in a geriatric emergency department in 2011 (18).

Therefore, we searched the primary evidence (unfiltered resources) of the aforementioned databases from August 2005 when the AD8 was first published (9) to April 2017 for all studies related to the AD8, including those in English, traditional Chinese, simplified Chinese, Japanese and Spanish. Two authors independently searched the databases twice and resolved conflicts after consensus with a third author.

Criteria for study selection and data extraction

We excluded repetitive and unrelated articles first, and the same two authors independently used the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool to assess the methodological quality of each included study (19). The QUADAS-2 incorporates four domains including participant selection, index test, reference standard and ‘flow and timing’. We followed the guidelines of the protocol published in the Cochrane Database of Systematic Reviews and assessed the risk of bias of each domain and the applicability of the first three domains (13). We then excluded non-consecutive or case–control studies and studies without consistently applied reference standards, because the levels of evidence in such studies are lower (Levels 3 and 4) according to the Centre for Evidence-Based Medicine (CEBM) published by the University of Oxford in 2011 (20). Studies conducted in a non-community setting, clinics or hospitals which were not relevant to our topic were also excluded. If two studies were from the same target population, only one study was included. The study flow diagram is shown in Figure 1.

Figure 1.

Study flow diagram of the meta-analysis of the diagnostic accuracy of Ascertain Dementia 8.

We then extracted clinical setting in which the study was conducted, age and size of the target population, the prevalence of cognitive impairment and the reference diagnosis. The diagnosis of mild cognitive impairment (MCI) was based on the published criteria (21,22) and the diagnosis of dementia was defined using DSM-IV (23) and National Institute on Aging-Alzheimer’s Association (NIA-AA) criteria (24). The diagnostic accuracy was also extracted including true positives, false positives, false negatives and true negatives. We calculated these as needed according to the sensitivity, specificity and number of subjects in each included study. The time it took to administer the AD8 and comparisons with other screening tools in the same study were also recorded if available.

Statistical analysis

Data were analyzed using the Midas module and binreg command in Stata 14.0 (Stata Corporation, College station, TX) and Meta-DiSc 1.4 software (25). The true positives, false positives, false negatives and true negatives in each study were pooled to obtain sensitivity, specificity, positive likelihood ratio (PLR) and negative likelihood ratio (NLR) to estimate the usefulness of the AD8 in detecting cognitive impairment. The likelihood ratios indicated how much an index test would increase or decrease the probability of having disease (PLR > 10 and NLR < 0.1, 5 < PLR < 10 and 0.1 < NLR < 0.2, 2 < PLR < 5 and 0.2 < NLR < 0.5, indicated high, moderate and low diagnostic informativeness, respectively) (26). The diagnostic odds ratio (DOR) was calculated as the ratio of the odds of having the disease relative to that of not having the disease. The DOR ranged from 0 to infinity, with a higher value indicating better test performance (27).

Summary receiver operating characteristic (SROC) curves were used to summarize the diagnostic accuracy of the results, and the area under the curve (AUC) was estimated to evaluate the diagnostic performance (28). AUC values of ≧0.97, 0.93–0.96, and 0.75–0.92 were considered to be excellent, very good and good diagnostic accuracy, respectively (29). Heterogeneity among studies was assessed using I², and statistically significant heterogeneity was considered if I² > 50% (30). Potential sources of heterogeneity were investigated by meta-regression and sub-group analyses. Study-specific covariates such as where the study was conducted and the prevalence of disease were used to explore the meta-regression. We used Deek’s funnel plots to investigate publication bias, and a P value of <0.10 for the slope coefficient was taken to indicate significant asymmetry (31).

Results

Study selection and quality assessment

A total of 485 published studies associated with the AD8 questionnaire were retrieved from the nine electronic databases (Fig. 1). After excluding irrelevant and duplicate studies, 29 studies were selected for careful full text review. We then excluded a further 22 studies considered to lack suitability or as consisting of lower levels of evidence according to the CEBM (20). Finally, a total of seven studies published between August 2005 and April 2017 met the inclusion criteria (10–12,32–35).

With regards to the assessment of the methodological quality of the seven included studies (Supplement 1), the cut-off point of the AD8 questionnaire was not pre-specified in the study by Chan et al. (12), which conveyed an unclear risk of bias in the ‘index test’ domain. In the study by Meguro et al. (11), the staff who collected the AD8 questionnaire had few possibilities to assess the degree of cognitive impairment, resulting in a little risk of bias in the ‘reference standard’ domain. However, the other domains of these two studies and all domains of the other five studies were considered to have a low risk of bias and low risk of applicability, which indicated the high quality of all seven included studies.

Study characteristics

The demographics and diagnostic data of the included studies are shown in Table 1. The seven studies were conducted in Asia, Europe, North and South America, and the target populations were recruited from the community, clinics and hospitals. All of the participants were older than 60 years of age, except for those from the study conducted by Galvin et al. (32) and Larner (33) which also recruited young and middle-aged participants. The prevalence of cognitive impairment (including MCI and dementia) ranged from 20% to 30% in the studies conducted in Singapore, China and Brazil (10,12,35), consistent with that reported in Taiwan (5). The prevalence rates in the other four studies were all more than 50%. Of these, the study conducted in Japan recruited participants older than 75 years of age (11), and the studies conducted by Razavi et al. (34) and Larner (33) were performed in neurology departments, which may have been the reason for the high prevalence. With regards to the cut-off point of the AD8 questionnaire, two studies (10,12) differed from the original research by the AD8 developer, who set the cutoff point at 2 (9). Of these, the most optimal cut-off point was determined by ROC curve in the Singapore study (12). The authors of the Brazilian study considered that the median education and economic level of the population were lower than that in the USA, so they set the AD8-Brazil cutoff point at 3 in advance (10). Those who completed the AD8 in China were participants themselves (35), but the others in all other six studies were informants, that is caregivers or families rather than the participants themselves (10–12,32–34).

Table 1.

Basic information of the eligible seven studies of the diagnostic accuracy of Ascertain Dementia 8 for cognitive impairment

Study	Country	Location	Age	Size	Prevalence	Diagnosis	True positive	False positive	False negative	True negative
Galvin et al. (32)	USA	Community	76.8 ± 8.9	325	0.54	Dementia	148	26	28	123
Correia et al. (10)	Brazil	Community	Above 65	109	0.29	MCI + dementia	25	21	7	56
					0.14	Dementia	15	31	0	63
Razavi et al. (34)	USA	Community-based neurology/memory clinic	77.8 ± 8.2	186	0.76	MCI + dementia	141	10	1	34
					0.69	Dementia	128	10	1	34
Meguro et al. (11)	Japan	Community	above 75	572	0.63	MCI + dementia	182	38	176	176
					0.12	Dementia	61	38	8	176
Larner (33)	UK	Regional neuroscience centre	64.5 (16–92)	212	0.62	MCI + dementia	127	67	4	14
					0.33	Dementia	67	67	2	14
Chan et al. (12)	Singapore	Primary health care centres	71.7 ± 8.2	309	0.14	Dementia	40	23	4	242
Yang et al. (35)	China	Community	79.5 ± 7.6	2015	0.22	Dementia	398	339	46	1232

Diagnostic accuracy of the AD8 in differentiating cognitive impairment

Seven studies including 3728 participants were included in the pooled analysis. A random effect model was used to investigate the overall diagnostic performance of the AD8 questionnaire for cognitive impairment. We divided the subjects into two subgroups according to the severity of cognitive impairment according to reference diagnosis. The first subgroup was to differentiate normal cognition from MCI and dementia, and the other subgroup was to differentiate non-dementia from dementia.

Four studies were included in the first subgroup (10,11,33,34), and the pooled sensitivity, specificity and the DOR were 0.72 (95% CI: 0.68–0.75), 0.67 (95% CI: 0.63–0.72) and 13.7 (95% CI: 3.88–48.40), respectively (Fig. 2A, B and E). The AUC was 0.83, indicating that the AD8 had good diagnostic accuracy for MCI and dementia (Fig. 2F). Pooled PLR and NLR values were 2.52 (95% CI: 0.93–6.82) and 0.17 (95% CI: 0.05–0.64), respectively, indicating that the AD8 had small informational value in confirming MCI and dementia but moderate informational value in excluding it (Fig. 2C and D).

Figure 2.

Diagnostic performance of the Ascertain Dementia 8 for differentiating normal cognition from MCI and dementia. Forest plots of overall sensitivity (A), specificity (B), PLR (C), NLR (D), DOR) (E) and SROC curve (F).

Seven studies were included in the second subgroup (10–12,32–35), and the pooled sensitivity, specificity and DOR were 0.91 (95% CI: 0.89–0.92), 0.78 (95% CI: 0.76–0.80) and 37.23 (95% CI: 21.34–64.94), respectively (Fig. 3A, B and E). The AUC was 0.92, indicating that the AD8 had very good diagnostic accuracy in differentiating dementia from non-dementia (Fig. 3F). Pooled PLR and NLR vales were 3.94 (95% CI: 1.97–7.87) and 0.13 (95% CI: 0.09–0.19), respectively, indicating that the AD8 had small informational value in confirming dementia but moderate informational value in excluding it (Fig. 3C and D).

Figure 3.

Diagnostic performance of the Ascertain Dementia 8 for differentiating non-dementia from dementia. Forest plots of overall sensitivity (A), specificity (B), PLR (C), NLR (D), DOR (E) and SROC curve (F).

Publication bias

Deek’s funnel plots were created for these two AD8 subgroups, and no asymmetric distribution was apparent (Supplement 2). The P values for the slope coefficients in the two subgroups were 0.41 and 0.50, respectively, indicating no evident publication bias.

Heterogeneity analysis and subgroup analysis

In this meta-analysis, all of the I² values in the overall pooled sensitivity (98.6%, 82.4%), specificity (97.4%, 96.7%), PLR (97.8%, 98.8%), NLR (92.5%, 51.4%) and DOR (84.7%, 60.5%) exceeded 50%, indicating substantial inter-study heterogeneity. The Spearman correlation coefficients in the two subgroups were 0.400 (P = 0.600) and 0.679 (P = 0.094), respectively, suggesting that no threshold effect contributed to the heterogeneity.

We then conducted meta-regression analysis of the two subgroups to identify the sources of the heterogeneity. The location of where the study was conducted (community versus non-community setting) and the prevalence of cognitive impairment (>0.5 versus <0.5) were reasonable factors for heterogeneity. Based on these results (Table 2), the sensitivity of the AD8 when conducted in the community was significantly less than when it was conducted in a non-community setting in the two subgroups (cognition normal versus MCI and dementia, non-dementia versus dementia) with P values of <0.001. The specificity of the AD8 when conducted in the community was significantly greater than that when conducted in a non-community setting in the subgroup of differentiating cognition normal from MCI and dementia (P < 0.001). The trend of the specificity was similar in the subgroup of differentiating non-dementia from dementia but non-statistically significant (P = 0.065). In other aspects, the sensitivity and specificity of the AD8 when conducted in an area of higher prevalence (>0.5) was not significantly different from that conducted in an area of lower prevalence in both of the subgroup.

Table 2.

Subgroup analysis of the Ascertain dementia 8 in detecting cognitive impairment of different severity

	Sensitivity (95% confidence interval)	P	Specificity (95% confidence interval)	P
MCI + dementia
Location
Community	0.53 (0.32–0.73)	<0.001	0.80 (0.72–0.86)	<0.001
Non-community	0.98 (0.96–0.99)		0.38 (0.30–0.47)
Prevalencea
High	0.71 (0.52–0.85)	0.407	0.66 (0.54–0.77)	0.263
Low	0.78 (0.61–0.89)		0.73 (0.62–0.81)
Dementia
Location
Community	88 (0.78–0.94)	<0.001	0.79 (0.74–0.82)	0.065
Non-community	0.97 (0.94–0.99)		0.74 (0.70–0.78)
Prevalencea
High	0.90 (0.87–0.93)	0.942	0.81 (0.77–0.85)	0.232
Low	0.91 (0.88–0.93)		0.78 (0.76–0.79)

^aPrevalence: high: > 0.5; low: < 0.5.

The administration time of the AD8 questionnaire was only reported in the three studies conducted by Galvin et al. (32), Correia et al. (10), and Chan et al. (12), and the average time was less than 3 minutes.

Discussion

In the current meta-analysis, the data were pooled from 3728 participants in seven studies, and then further classified into two subgroups according to the severity of cognitive impairment based on the currently used diagnostic criteria (21–24). Our goal was to demonstrate whether the AD8 is a useful questionnaire for primary care providers to detect all-cause cognitive impairment in the community, clinics and hospitals. We also wanted to investigate the diagnostic accuracy of the AD8 in differentiating different stages of cognitive impairment and in different clinical setting. Based on the QUADAS-2 tool, all of the seven included studies had good quality methodologically. In the overall analysis, the sensitivity (0.72, 0.91) was greater than the specificity (0.67, 0.78) in individual subgroups, which indicated that false positive results were likely. The NLR (0.17, 0.13) was better than the PLR (2.52, 3.94), indicating that the probability of false negative results was relatively low. In other words, the AD8 had moderate informational value to exclude cognitive impairment or dementia if the AD8 result was negative. The AUC of the subgroup (non-dementia versus dementia) was 0.923, suggesting that the AD8 had good to very good diagnostic performance in discriminating dementia and non-dementia. The AUC of the subgroup (cognition normal versus MCI and dementia) was 0.829, suggesting that the AD8 had good diagnostic accuracy in discriminating cognitive impairment from normal cognition.

A test of heterogeneity demonstrated that inter-study heterogeneity existed in this meta-analysis, without a threshold effect. We therefore performed meta-regression analysis to explore the source of heterogeneity, and the location of where the study was conducted may have played a role. The AD8 had greater sensitivity in differentiating normal cognition from MCI or dementia when used in clinics or hospitals than when it was used in the community. The greater sensitivity of the AD8 in non-community was probably due to the severity difference of cognitive impairment. The degree of cognitive impairment must be severe enough to be detected by the informant, so the elderly were then taken to the clinics or hospitals. Like the study by Razavi et al. (34), at neurology/memory clinic (the prevalence of MCI and dementia: 0.76 versus dementia: 0.69), most of the participants belonged to the moderate to severe form of cognitive impairment, so their AD8 scores were easily positive with scores far more than two. Another study by Meguro et al. (11), in the community (the prevalence of MCI and dementia: 0.63 versus dementia: 0.12), most of the participants belonged to the very mild to mild form of cognitive impairment. If the informants are not paying close attention to the elderly, it is very likely that they will be unaware of any mild cognitive changes. Therefore, the AD8 scores will very likely be underestimated. Thus, there should be caution in screening for MCI in the community because false negative results are likely, especially if the elderly is not being fully understood by the informant. In the other subgroup of dementia versus non-dementia, sensitivity was still significantly greater in non-community, but the difference of sensitivity between community and non-community became much smaller as compared to the subgroup of MCI and dementia versus normal cognition. We conclude that the cognition difference between normal and dementia is greater than those between normal and MCI, so the informant could easily detect the cognitive change in more severe patients. Therefore, these severe patients had higher rate of positive AD8 scores.

Given the predicted increase in the prevalence of dementia worldwide, the need for a definite diagnosis at the individual patient level may conflict with the need for easy access to a diagnosis at the population level. In current clinical practice, a two-stage process is often used with initial triage assessment that is suitable for primary care providers. In the second stage, those who require further evaluation are assessed by a multidisciplinary team and undergo tests including neuroimaging and biomarker assessment which can only be performed in a specialist memory service (36).

Various tools are currently used for initial cognitive screening, however, they often take the form of brief and direct cognition testing such as the MMSE and MoCA which only provide a ‘snapshot’ of cognitive function (6,13). However, a definite evaluation of cognitive impairment requires assessments of changes in cognition and neuropsychology over time. Therefore, an appropriate approach is to question collateral sources with sufficient knowledge of the person (13). Informant-based questionnaires such as the AD8 or IQCODE aim to assess changes in function retrospectively.

The studies included in this meta-analysis demonstrated that the AD8 was more sensitive than the MMSE (0.97: 0.53) but that it had poorer specificity (0.15: 0.75) in differentiating MCI and dementia from normal cognition (33). In detecting dementia, the sensitivity (1.00: 0.93, 0.90: 0.93) and specificity (0.67: 0.52, 0.78: 0.81) were comparable to the MMSE (10,35). The AUC of AD8 was superior (0.97: 0.92, P = 0.047) or comparable (0.89: 0.89, 0.84: 0.87) to that of the MMSE in differentiating dementia from non-dementia in three of the studies (11,12,35). One study showed that the AUC of the AD8 was comparable to that of the IQCODE (0.95: 0.93) in detecting dementia from normal cognition, however the sensitivity (0.99: 0.79) and negative prediction value (0.97: 0.59) of the AD8 were superior to that of the IQCODE (34). With regards to detecting MCI, the sensitivity of the AD8 was also greater than that of the IQCODE (1.00: 0.46) in the same study (34). With regards to the administration time for one person, the AD8 takes less than 3 minutes compared to 7–10 minutes for the MMSE, 10 minutes for the MoCA, and 20 minutes for the IQCODE (6), which illustrates the benefit of using the AD8 in a busy clinical setting.

To the best of our knowledge, this is the first meta-analysis to summarize the diagnostic accuracy of the AD8 for cognitive impairment in the community, clinics and hospitals. However, there are some limitations to this study. First, only seven high-quality studies were included even though we searched for studies in Chinese, Japanese and Spanish in addition to English. Further analysis of studies conducted in different countries and in other languages is needed to validate our results. Second, not every included study defined MCI in their target population (12,32,35), which may have weakened the accuracy of the AD8 to detect MCI and also deviated from the original intention of the developer who constructed the AD8. Third, ‘reference diagnosis’ would affect the accuracy of the AD8 and be a potential source of heterogeneity. Most of the included studies used DSM-IV as reference diagnosis and the other two studies used NIA-AA criteria (34,35). Therefore, further studies are needed to clarify the effect of reference standard.

The Canadian Task Force On Preventive Health Care recommends not screening asymptomatic adults 65 years of age or older for cognitive impairment (strong recommendation, low quality evidence) (37). However, this recommendation does not apply to people with symptoms suggestive of cognitive impairment (e.g. loss of memory, language, attention, executive or visuospatial function, or behavioural or psychological symptoms) or those suspected of having cognitive impairment by clinicians, family or friends. Therefore, clinicians should still pay more attention to the elderly who may have cognition impairment in primary care. The AD8 is one of the competitive screening instruments we can choose if subjects are suspected to have cognitive impairment.

In conclusion, this meta-analysis indicates that the sensitivity of the AD8 is superior to specificity in screening for cognitive impairment. But poorer specificity and PLR may result in false positive results. Therefore, similar to all other screening tools, if the result is positive, a further diagnosis is necessary. A better negative predictive value and NLR of the AD8 suggests that false negative results are less likely, a main characteristic that every screening tool should possess. In addition, the shorter administration time (<3 minutes) make the AD8 more competitive in a busy clinical setting. Taken together, the AD8 appears to be a competitive tool that can be used in primary care to screen cognitive impairment in the community, clinics and hospitals. Participants with an AD8 score ≧2 should be highly suspected to have cognitive impairment and a further definite diagnosis is needed.

Supplementary Material

Supplementary data are available at Family Practice online.

Declaration

Funding: none

Ethical approval: none.

Conflict of interest: none.