Time that tells: critical clock-drawing errors for dementia screening

Abstract

Background

Clock-drawing tests are popular components of dementia screens but no single scoring system has been universally accepted. We sought to identify an optimal subset of clock errors for dementia screening and compare them with three other systems representative of the existing wide variations in approach (Shulman, Mendez, Wolf-Klein), as well as with the CDT system used in the Mini-Cog, which combines clock drawing with delayed recall.

Methods

The clock drawings of an ethnolinguistically and educationally diverse sample (N = 536) were analyzed for the association of 24 different errors with the presence and severity of dementia defined by independent research criteria. The final sample included 364 subjects with ≥5 years of education, as preliminary examination suggested different error patterns in subjects with 0–4 years of education and inadequate numbers of normal controls for reliable analysis.

Results

Eleven of 24 errors were significantly associated with dementia in subjects with ≥5 years of education, and six were combined to identify dementia with 88% specificity and 71% sensitivity: inaccurate time setting, no hands, missing numbers, number substitutions or repetitions, or refusal to attempt clock drawing. Time setting was the most prevalent error at all dementia stages, refusal occurred only in moderate and severe dementia; and ethnicity and language of administration had no effect. All critical errors increased in frequency with dementia stage. This simplified scoring system had much better specificity than two other systems (88% vs 39% for Mendez’s system –63% for Shulman’s) and much better sensitivity than Wolf-Klein’s (71% vs 51%). Stepwise logistic regression found the simplified system to be more strongly predictive of dementia than the three other CDT systems of dementia. Substituting the new CDT algorithm for that used in the original CDT Mini-Cog improved the Mini-Cog’s specificity from 89 to 93% with minimal change in sensitivity.

Conclusions

Only six errors need be assessed to capture most of the power of clock drawing to discriminate between people with dementia and normal subjects, and improves specificity over older systems in subjects with ≥5 years of education. These errors require minimal conceptual classification and are easily detected and scored by non-specialists.

Introduction

Clock-drawing tasks (CDTs) are widely used in screening for cognitive impairment and dementia (Shulman et al., 1993; Ferrucci et al., 1996; Shulman, 2000), and ranked first among 20 instruments in ease and speed, and second only to the Mini-mental State Examination (MMSE) in frequency of use reported by geriatric experts (Shulman et al., 2006). However, among many proposed CDT systems (e.g. Sunderland et al., 1989; Mendez et al., 1992; Tuokko et al., 1992; Borson et al., 2000; Scanlan et al., 2002; see Shulman et al., 2006), there is no consensus as to which is the most useful for dementia screening. An ideal system would be rapidly administered in diverse settings, minimize false positives, optimize dementia detection, and reduce or eliminate the influence of education, language or ethnicity (Ainslie and Murden, 1993; Borson et al., 1999; Shulman, 2000; Paganini-Hill et al., 2001).

In addition to their use as a stand-alone screen for dementia, CDTs are frequently incorporated into longer test batteries or included in composite cognitive screens such as the Mini-Cog, which performs better than the CDT alone (Borson et al., 2000). Improvements in the effectiveness of the CDT could improve the performance of any screening approach that includes it as a component.

The goal of the present study was to improve the utility, efficiency and scoring consistency of the CDT as a screen for dementia in the elderly and as a component of the Mini-Cog. We sought to determine which errors provide maximum discrimination of cognitive status, which can be disregarded because of confounding by education or other demographic factors such as ethnicity, and how severity of cognitive impairment is reflected in error patterns.

Methods

Participants

The clock drawings of 536 older adults enrolled in the University of Washington Memory Disorders Clinic and the Alzheimer’s Disease Research Center (ADRC) Satellite were examined. Initially, subjects were classified into four groups by years of education: 0–4, 5–8, 9–12, and ≥12. Those in the lowest education stratum (n = 91) were all non-English speaking late-life immigrants (Chinese, Korean or Hispanic). The remainder (n = 445) represented white Hispanic and non-Hispanic, African-American, and several Asian ethnic subgroups (see Table 1).

Table 1.

Algorithm sample descriptors, means (SD)

	NO DEMENTIA	DEMENTIA
N	138	226
Age	72.5 (8.0)	77.9 (8.6)
Education (years)	12.75 (3.6)	12.2 (3.8)
Gender (% female)	66%	65%
MMSE	27.4 (2.5)	18.2 (11.0)
Mini-Cog	3.7 (1.3)	1.2 (1.4)
Ethnicity	% OF TOTAL SAMPLE
Chinese	32
White Non-Hispanic	22
African-American	23
Non-Chinese Asian	9
Hispanic	9
Other	6
Primary language spoken
English	43%
Chinese	36%
Other Asian	10%
Spanish	11%

Procedures

All participants were asked to “draw a clock: draw a circle, fill in the numbers, set the hands to show 8:20 (or 11:10).” All subjects were evaluated using established research protocols (ADRC sample; for details see Borson, 1999; 2000) or structured multidimensional clinical diagnostic methods for the dementia clinic sample. This process yielded three diagnostic groups: no cognitive impairment (n = 154), subsyndromal cognitive impairment (n = 101), or dementia syndrome (n = 281). The Clinical Dementia Rating Scale (CDR; Hughes et al., 1982) was used to classify stage of illness as not impaired (0), subsyndromal impairment (0.5), mild dementia (1), moderate dementia (2), and severe and very severe dementia (3+). Cases of dementia were further classified by etiology using the DSM-IV (American Psychiatric Association, 1994) and NINCDS-ADRDA (McKhann et al., 1984) for Alzheimer’s disease, research criteria for vascular dementia (Román et al., 1993); and published criteria for mixed (Zekry et al., 2002), Lewy body (Campbell et al., 2001; Del Ser et al., 2000) and frontotemporal (Lund and Manchester Group, 1994) dementias. Unclassifiable cases were grouped as dementias of unknown etiology.

Clock errors were drawn from three systems (Mendez et al., 1992; Tuokko et al., 1992; Shulman et al., 1993) selected for superior performance in comparative studies (Royall et al., 1998; Storey et al., 2001; Scanlan et al., 2002) or for comprehensive examination of a large number of possible errors (Tuokko et al., 1992). Twenty-three unique error types were recognized; number spacing was further divided into any and major errors (see Appendix for a complete description). This final set of 24 separate errors was then applied to all 536 de-identified clock drawings by a trained rater who was unaware of the subjects’ cognitive status.

Data analysis

EXCLUDING VERY LOW EDUCATION SUBJECTS (≤4 YEARS) AND SUBJECTS WITH SUB SYNDROMAL COGNITIVE IMPAIRMENT (COGNITIVE IMPAIRMENT/NODEMENTIA)

Initial analyses showed a relationship between years of education and many error types. One-way ANOVAs were used to test for error frequencies as a function of education group (0–4, 5–8, 9–12, and ≥12 years) and dementia status. Analysis of variance was also used to examine possible non-linear or threshold effects (i.e. patterns that differed or reversed in low versus high education groups, or errors that functioned differently in one group as compared with all others). ANOVAs showed that education effects were explained by subjects with <5 as opposed to ≥5 years of education. Relative to more educated subjects, many useful errors showed unacceptably high false positive classification rates among participants with no dementia with low education (e.g. time, no hands, and missing numbers which had false positive rates of 77%, 46% and 23%, respectively). The <5 years education group also had few non-impaired participants (16/91), reducing statistical power and overall confidence in the stability of the results. We therefore excluded these subjects from any further analysis. In addition, cognitively impaired subjects without dementia (subsyndromal, n = 81), irrespective of educational group, were excluded from analyses designed to create the optimal algorithm for dementia detection. These exclusions yielded a final sample of 138 normal and 226 demented subjects (total n = 364) for algorithm development.

IDENTIFYING SIGNIFICANT ERRORS

After excluding very low education and non-demented/impaired subjects, CDT errors were examined in stepwise logistic regressions for prediction of dementia, and for possible confounding influences of language (English, Chinese and Spanish), ethnicity and education. In these analyses the individual error was treated as the dependent variable, and dementia status, education, language and ethnicity were used as predictors. Significance was assigned at p ≤ 0.01 for all analyses.

CREATING THE ALGORITHM

After isolating CDT errors that strongly discriminated dementia with minimal confounding by education, language and ethnicity, our goal was to combine them into an algorithm that optimized specificity and sensitivity. Stepwise logistic regression was again used, with dementia status (normal = 0, dementia = 1) as the dependent variable and all significant CDT errors as predictors. Then, the algorithm was tested for specificity and sensitivity for discriminating normal subjects from those with dementia, and sensitivity to subsyndromal cognitive impairment and to dementias of varying etiologies.

COMPARING THE ALGORITHM WITH OTHER CDT SYSTEMS

Once the new CDT algorithm was created, we compared its relative specificity and sensitivity with three previously studied systems (Mendez, Shulman, and Wolf-Klein) and with the CDT used in the original Mini-Cog (Borson et al., 2000), using their published scoring rules. All CDT systems chosen for comparison have long histories of use, and include one previously shown to be sensitive but less specific (Mendez et al., 1992), one with balanced sensitivity and specificity (Shulman, 2000), and one with high specificity but low sensitivity (Wolf-Klein et al., 1989; data from Scanlan et al., 2002). We then examined the specificity and sensitivity of a revised Mini-Cog (three-item delayed recall + the new CDT algorithm) with the original version. All analyses were performed with SPSS Version 13.

Results

Significant and non-significant errors: bivariate analyses

For participants with ≥5 years of education, error types fell into two main groups with respect to their association with dementia and confounders (Table 2; all errors shown in rank order of χ² association with dementia). Significant errors included 11 of the original 24 errors. Of these top 11 errors, all occurred in ≤10% of non-demented subjects, and none showed significant confounding by language, ethnicity, or education. Refusal to draw a clock was relatively rare in this sample, but eliminated the possibility of other errors when it occurred and therefore was retained as a significant predictor. Non-significant errors included 13 that did not significantly distinguish between normal subjects and those with dementia, or occurred more often in normal subjects than in those with dementia (e.g. hand length, face geometry, distance between circumference and numbers, number rotation, centering, second tries, and aggregated number spacing errors of all types).

Table 2.

Frequency of clock error types in subjects with ≥5 years of education (n = 364)

	NO DEMENTIA* (N = 138)	WITH DEMENTIA (N = 226)	RATIO (WD/ND)	X2	P VALUE
Discriminating errors
Time incorrect	9	55	6.11	75.4	0.001
No hands	2	30	15	42.5	0.001
Missing numbers	4	30	7.5	34.2	0.001
Repeated numbers	1	16	16	22.4	0.001
Substitution	2	18	9	19.8	0.001
Number orientation	1	13	13	16.6	0.001
Number order	0	9	NA	12.9	0.001
Numbers outside circle	3	13	4.33	10.9	0.001
Clock-like figure	0	7	NA	10.2	0.001
Number spacing (major)	6	17	2.83	9.4	0.003
Refusal to do	0	5	NA	7.6	0.01
Non-discriminating errors
Shape incorrect	1	7	7	5.98	NS
Extra marks	11	21	1.91	5.6	NS
Number perseveration	1	6	6	5	NS
Hands in center	38	29	0.76	2.8	NS
Number of hands	7	11	1.57	1.74	NS
Hand length	44	36	0.82	1.4	NS
Number rotation	17	13	0.76	1.03	NS
Face geometry	53	57	1.08	0.28	NS
Number distance	51	50	0.98	0.12	NS
Number spacing (all)	44	45	1.02	0.10	NS
Second try	21	22	1.09	0.06	NS
Military time	0	0	N/A	N/A	NS
Anchoring	0	1	1	N/A	NS

^*Subjects with subsyndromal cognitive impairment were excluded.

Classification algorithm

Stepwise logistic regression identified four errors highly predictive of dementia: wrong time setting, number substitution, number repetition, and no hands. Since not all significant errors appeared in the regression equation, we considered additional variables that might add information and improve discrimination if included in the final algorithm. Refusal, an error that occurred only in participants with dementia, could not be included in the regression because if no drawing was made no other error could occur. Refusal was therefore added to the algorithm post hoc. In addition, the “missing numbers” error did not appear in the regression equation, but was the third most potent bivariate predictor. When this error was added to the regression-derived algorithm enhanced by “refusal,” sensitivity was improved (2–3%) with negligible loss of specificity. Comparisons of this algorithm with a simple combination of all CDT errors significantly associated with dementia in bivariate analyses demonstrated the superior performance of the algorithm, as the sum of all significant errors had 10% lower specificity with only 6% higher sensitivity. The final algorithm, therefore, included six of the original 24 errors: the four identified by regression and two others that were strong in bivariate analyses.

APPLYING THE ALGORITHM

As a test of the utility of this final algorithm, a composite binary score was constructed: if any included error was present, the subject was classified as having dementia; if none of the six errors was present, the subject was classified as not having dementia. Using this approach, the algorithm’s specificity for dementia (defined by research criteria) was 88% and sensitivity was 71%. When applied to subjects with varying etiologies of cognitive impairment (Table 3), the algorithm was most sensitive to probable AD (82%) and least sensitive to subsyndromal cognitive impairment (35%).

Table 3.

Sensitivity of the algorithm to cognitive disorders (n = 307)

	TOTAL CDT TESTS	ALGORITHM DETECTS (FREQUENCY)	ALGORITHM DETECTS (PERCENTAGE)
Probable AD	132	108	82%
Mixed dementia	27	22	81%
Vascular dementia	19	13	68%
Other dementias	48	22	46%
SCI	81	28	35%

AD: Alzheimer’s disease; SCI: subsyndromal cognitive impairment.

Other dementias: Lewy body, frontotemporal dementias of any type, and dementia of unknown type.

Comparing the algorithm with other CDT systems

Table 4 compares the performance of the new algorithm with three other CDT systems and the Mini-Cog in our sample. It shows the performance of the original Mini-Cog (Borson et al., 2000) and a revised Mini-Cog, substituting the new CDT algorithm. The new algorithm had better specificity than Mendez and Shulman and better sensitivity than the Wolf-Klein system, and improved the performance of the Mini-Cog as a dementia screen.

Table 4.

Comparison of new algorithm with existing CDT systems and Mini-Cog

	SPECIFICITY	SENSITIVITY
Stand-alone CDTs (N = 364)
New algorithm	88	71
Mendez	39	87
Shulman	63	80
Wolf-Klein	85	51
Mini-Cog* (N = 220)
With new algorithm	93	93
With original CDT	89	94

^*Three-word delayed recall + CDT.

When the four CDT systems were allowed stepwise entry in a logistic regression predicting dementia, the new system was the strongest predictor (χ² = 130, df = 1, p < 0.001). Once the new CDT system entered, none of the other systems significantly increased the variance accounted for.

As an additional test of the new CDT system we compared the relative performance of our older Mini-Cog test with one which incorporated this new algorithm. In stepwise logistic regression predicting dementia, the “new” Mini-Cog entered the equation first (χ² = 188, df = 1, p < 0.001), suggesting that this modification may be an improvement over the original Mini-Cog, although the differences are relatively small (see Table 4).

ERRORS AND DEMENTIA SEVERITY

Figure 1 depicts the prevalence of specific predictive errors and of a positive score using the composite screening algorithm as a function of CDR stage (0–3+). Time setting was the most frequent error in all stages, and the only critical error to separate participants with subsyndromal cognitive impairment from normal participants (McNemar test, p ≤ .001). Refusal to draw did not appear until CDR 2 (moderate dementia). All other errors except repetition increased in frequency with dementia severity.

Figure 1.

Percentage of subjects exhibiting errors.+, Whole clock scoring algorithm; ◆, Incorrect time setting; □, Missing numbers; ▲, Substitution; ×, Repetition; ¥, No hands; ○, Refusal.

Errors were ranked according to frequency and plotted in terms of the percentage of individuals at each CRD stage identified by the error made. The error algorithm was then plotted against the composite errors individually.

Discussion

To our knowledge, this study is the largest thus far to examine the CDT as a dementia screen, and the only one to isolate specific errors that best discriminate elderly subjects with dementia from those without. It considers all major error types in analyses with sufficient statistical power to suggest generalizability. The data yield a new CDT scoring algorithm limited to six errors, all of which met stringent criteria for dementia discrimination and were free of linguistic, ethnic and educational biases in individuals with ≥5 years of education. The top six errors (wrong time, no hands, missing numbers, number substitutions, repetition and refusal) are all easily observed by untrained individuals and require little judgment or subjective interpretation. An algorithm using these errors, plus refusal, had good specificity (88%) and sensitivity (71%) for dementia in this sample. These findings also suggest that many non-discriminating errors may be safely ignored when using the CDT to screen for dementia.

The new algorithm, derived from a highly heterogeneous geriatric sample, performed better than three previously tested stand-alone CDT systems as a dementia screen. These older systems might have performed less well in the present sample than they did in the original descriptive studies because they were developed on smaller, more homogeneous, and less ethnolinguistically and medically complex samples; additionally, reports on two systems (Shulman, Wolf-Klein) did not mention education, and the third (Mendez) excluded subjects with <8 years. Furthermore, the two systems (Mendez and Shulman) with lower specificity than the new algorithm appear to have compromised their specificity by including errors that were not highly informative with respect to dementia. About a third of the errors scored in those two systems were found, in our sample, to have no discriminating power. Conversely, the system with the lowest sensitivity, the Wolf-Klein, has omitted the most important CDT error, correct time, from its algorithm.

We do not advocate the use of any CDT as a stand-alone dementia screen. This study was designed to simplify and make transparent the optimal rules for scoring clocks in screening applications. Here, as in the original Mini-Cog paper (Borson et al., 2000) and a subsequent comparison of eight CDT systems (Scanlan et al., 2002), all systems thus far examined were less effective than the Mini-Cog, which combines the CDT with three-item delayed recall. Using the new algorithm in place of the old Mini-Cog CDT, we did observe some increase in specificity (from 89% to 93%) and the “new” system was chosen over the old in stepwise logistic regressions.

We observed that CDT errors were related to dementia severity, consistent with the findings of Rouleau et al. (1996) and Yamamoto et al. (2004), whose sample sizes were smaller and error analyses less transparent than in the present study. We and others (Seigerschmidt et al., 2002; Powlishta et al., 2002) find that subsyndromal cognitive impairment cannot be effectively distinguished by the CDT alone, and requires different and more complex screening tests (De Jager et al., 2003). The CDT scoring system developed here was most successful in identifying patients with Alzheimer type dementia (AD + mixed dementia), and less so for other types (pure vascular, other) in this sample.

The limitations of this study are those generally associated with the use of non-random samples, e.g. over-representation of individuals with dementia, and the small number of non-impaired but poorly educated individuals. Our data do not resolve questions about optimizing the CDT for screening individuals with subsyndromal cognitive impairment, or those with <5 years of schooling (developmental studies suggest that clock-drawing skills usually reach normal “adult” levels around grade 5; Cohen et al., 2000).

Conclusions

Results of this study indicate that, for individuals with at least five years of formal education, a simple CDT scoring system need not sacrifice crucial information derived from much more complex systems and can be applied to persons of widely varying ethnicity and language without loss of performance. The algorithm developed here performs better as a dementia screen than three other established systems. By illuminating the clock errors which most accurately discriminate dementia from those which are non-discriminating and can be safely ignored, the CDT can be kept simple, easily learned and used, and comparable across varying populations.

Appendix. Clock-drawing errors evaluated for inclusion in the final algorithm

Twenty-four categorical errors were identified from three popular clock scoring methods (Mendez et al., 1992; Tuokko et al., 1992; Shulman et al., 1993) and incorporated into a single list. All 536 clock drawings were systematically inspected by a trained rater for the presence or absence of each error type, scored independent of any other error scored on the same clock. Error descriptions are given below, rank ordered, for clarity, by χ² association with cognitive classification taken from Table 2; italicized errors did not significantly discriminate dementia.

ERROR	DESCRIPTION
Time incorrect	Hands set incorrectly (or no attempt to depict).
No hands	No hands drawn.
Missing numbers	One or more numbers missing; includes tick marks in place of numbers.
Repeated numbers	Same number appears more than once.
Substitution	Symbols or marks used in place of numerals, or time written out rather than shown by hands.
Number orientation	Numbers counterclockwise.
Number order	Number sequence incorrect.
Numbers outside circle	Numbers placed outside exterior boundary of circle.
Clock-like figure	Image, figure, symbols, or characters drawn do not resemble a clock or features expected on an analog clock face.
Number spacing (major)	Gross error of spacing including a completely empty quadrant.
Refusal	Test form blank, or tester recorded refusal to start or finish a partial attempt.
Shape incorrect	Clock face not round or not a closed figure.
Extra marks	Uninterruptible extraneous marks, symbols, or figures anywhere on clock face.
Number perseveration	Any number past 12 (13, 14 …) unless military time used throughout (also an error).
Hands in center	Hands not connected, do not radiate from center, or originate somewhere other than center of clock face.
Number of hands	Only one or more than two hands drawn.
Hand length	Both hands same length, hour hand not clearly shorter than minute, or hands extended past boundary of circle.
Number rotation	Numbers backwards or upside down, or orientation rotated as if paper turned while subject writing.
Face geometry	Clock gestalt present, but internal geometry skewed (e.g., 12 and 6 not aligned at north and south poles).
Number distance	Numbers not equidistant from center and edge of clock face.
Number spacing (all)	Any uneven spacing or gapping.
Second try	Subject started over (2 starts seen or tester recorded).
Military time	Numbers indicating an entire 24-hour period on analog clock (1,2 … 23, 24)
Anchoring	Only anchoring numbers present (12, 3, 6, 9); intermediate numbers omitted.