Does MRI Increase the Diagnostic Confidence of Physicians in an Outpatient Memory Clinic

Abstract

Background and Aim

Data showing the usefulness of MRI to improve the accuracy of the diagnostic process in cognitive disorders were derived from studies in tertiary referral centers. MRI is widely used as a diagnostic tool in everyday practice, but it is unknown what the actual added value of MRI is. We studied the usefulness of MRI in the diagnostic process by measuring the change of confidence of the physician.

Methods

Physicians indicated confidence in their diagnosis before and after presentation of MR images using a visual analogue scale from 0-100%.

Results

Use of MRI increased the level of confidence by 3% in experienced clinicians and by 9% in inexperienced physicians. In 2/125 cases, MRI showed an unexpected finding.

Conclusion

MRI is a useful diagnostic tool in everyday practice of diagnosing cognitive disorders.

Introduction

In the international guidelines on the recommended diagnostic approach of patients presenting with cognitive disorders, neuroimaging is mentioned as a tool to exclude treatable causes of a dementia syndrome [1, 2, 3, 4, 5]. Neuroimaging is also a diagnostic method to increase the accuracy of diagnostic classification [6, 7]. The data to support this are mainly derived from studies in tertiary referral centers [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21].

The availability of MRI and CT outside tertiary referral centers has led to the routine use of these tools in the diagnostic process of cognitive dysfunction. The most recent version of the Dutch outpatient memory clinics survey reported that more than 80% of patients undergo neuroimaging [22]. However, to our knowledge there are no data available on how physicians outside tertiary referral centers use MRI in the diagnostic process of cognitive dysfunction.

We studied the usefulness of MRI in a memory clinic diagnostic program in a community-based hospital by asking physicians to what extent neuroimaging supports their diagnostic confidence. We used a probabilistic method to mimic the clinical weighing of MR results in a diagnostic process.

Subjects and Methods

Patients

We included all consecutive patients seen between January 2012 and April 2013 at the Medical Center Alkmaar memory clinic after they gave written informed consent Exclusion reasons were refusal or inability to undergo MRI.

The diagnostic program comprised of taking a medical history, physical and neurological examination, a neuropsychological evaluation, clinical laboratory tests, ECG and MRI. The local ethics committee approved the study.

Neuropsychological Test Battery

The neuropsychological test battery included the Visual Association Test and the direct Word List Learning, delayed verbal recall and recognition, Trailmaking A and B, Stroop, Rey Complex Figure, CAMCOG gnosis, Clock Drawing Test, the Verhage Education Level, parts of the Groningen Intelligence Test to estimate premorbid IQ, parts of the Amsterdam Dementia Screenings Test being orientation, meander, fluency and copy drawing. The Cognitive Screenings Test was used to extrapolate the MMSE score. The Clinical Dementia Rating Scale, Geriatric Depression Scale, Neuropsychiatric Inventory-Questionnaire, Informant Questionnaire of Cognitive Decline (IQ-CODE) and the Lawton IADL and Katz ADL were also used in this test battery.

MRI

MR images were acquired on a 1.5-Tesla machine. The scanning protocol included (1) a 3D T1-weighted sequence: coronal orientation, whole brain coverage with 1.5 mm slice thickness, (2) a FLAIR sequence with a 5 mm slice thickness and slice gap of 0.5 mm, and (3) a transverse T2-weighted turbo spin echo sequence with 5 mm slice thickness and slice gap of 0.5 mm. All images were acquired with an in-plane resolution of 1 × 1 mm.

Images were scored using the assessment of the Global Cortical Atrophy (GCA) four-point rating scale developed by Pasquier et al. [23], assessment of the Medial Temporal Atrophy (MTA) five-point visual rating scale described by Scheltens et al. [18], estimation of white matter hyperintensities (WMH) by using the Fazekas score [12] and number and localization of lacunar infarcts. GCA and WMH were assessed on a FLAIR image. The MTA score was assessed in the coronal reconstructions of the T1-weighted MPRAGE perpendicular to the hippocampus axis. Lacunas were defined as hypointense lesions on FLAIR and hyperintense lesions on T2 images with a minimum diameter of 3 mm and were distinguished from Virchow-Robin spaces based on their location and the presence of a hyperintensive rim on the axial 2D FLAIR image.

Diagnostic Classification

Patients were diagnosed with probable or possible Alzheimer's disease (AD) if they fulfilled NINCDS-ADRDA criteria [4], frontotemporal lobe dementia using the criteria of Neary et al. [24], vascular dementia (VaD) by the NINCDS-AIREN criteria [25], dementia with Lewy bodies using the criteria of McKeith et al. [26], Parkinson's disease dementia complex, and dementia without a distinctive etiology. Non-demented patients either were categorized as healthy or having subjective cognitive complaints, symptoms of a cognitive disorder probably caused by a developmental psychological disorder, a psychiatric or neurological disorder or patients were diagnosed with mild cognitive impairment (MCI) according to the Petersen and the revised Petersen criteria [27, 28].

Measuring Confidence

All available diagnostic information was discussed in a standard order by a multidisciplinary team of geriatricians, geriatric medicine trainees, a neurologist, an old age psychiatrist, a neuropsychologist and a specialized nurse. Before this meeting, the physician was unaware of the results of MRI and the test battery. After presentation of the basic clinical information, the physician indicated on a visual analogue scale (VAS) a level of confidence about the clinical diagnosis, ranging from 0–100%.

The VAS is a frequently used method to indicate pain [29, 30, 31, 32, 33], experienced physical health [34] or indicate mood [35]. It has also been used to measure the confidence of clinicians in the diagnostic process of ankle [36] or carotid pathology [37].

After indication of a first measure of confidence, MR images were always presented by the same physician (first author, L.B.). All available MRI sequences were presented in the same way using the various described scales and in the same order. The physicians then gave a second estimation of the extent of their confidence. Afterwards, the neuropsychologist presented the results of the test battery and following this, when available, additional biomarker results were discussed. Finally, the physicians were asked to indicate which of the MRI, the test battery or additional biomarker results predominantly influenced their confidence in the diagnosis.

The left end of a 10-cm-long VAS signifies no confidence at all in the clinical diagnosis, the right side represents complete confidence in the diagnosis. The VAS scores were used as percentage scores of diagnostic certainty.

The level of certainty may increase, decrease or remain unchanged after the assessment of MR images. If the number of increased and decreased changes and their extent would be balanced, this could result in a mean change of 0%, thereby obscuring possible clinically relevant effects of MRI on level of confidence. We therefore also computed absolute differences where increased and decreased levels of confidence were added up and negative results were multiplied by −1, giving a measure of change of confidence.

We also studied the usefulness of MRI by counting the number of diagnoses that changed due to assessment of MRI. The measure of change of confidence might be small butin this case it would nonetheless be clinically relevant when a number of diagnoses were changed after assessment of MRI.

In the analysis of the measures of confidence, we distinguished a group of experienced and a group of inexperienced physicians, as we expected that experience would influence the measure and the change of diagnostic certainty by using MRI. Physicians were considered experienced when they had been a consultant for more than 5 years. We checked for an effect of experience in a post hoc analysis.

Prior to the start of the study, the experienced raters were asked to give a measure of change in their certainty using MR results that would be clinically relevant to them. They all indicated independently that 20% was a clinically relevant change in certainty. This percentage, an alpha of 0.05 and a power of 0.80 were used to make a power calculation. The number needed to analyze was 50. To compare change of certainty in the two groups of physicians, we aimed at an inclusion of 100 patients.

A possible problem of a VAS is that the position of the first indication of the confidence level on the VAS might influence the positioning of the second indication, resulting in a smaller change in confidence level. To control for this possibility, we performed a vignette study where we presented four experienced geriatricians 50 vignette patients unknown to them. They were asked to write down a diagnosis after reading the vignettes and indicate a level of confidence. This procedure was repeated after 12 months to make sure they had no recollection of their former assessment. In the second round, they were given the same vignettes and indicated a level of confidence after they read the vignettes and were shown the MR images. They were also asked to give an indication of confidence level after providing them with the test results so to be able to quantify the clinical importance of these results in comparison to the results of MRI.

Statistical Analysis

We used the SPSS version 20 (for Windows) for statistical analysis. Results are expressed as n (%) or means in SD unless indicated otherwise. Frequency distributions for categorical variables were compared with the Fisher exact test To compare means, either the dependent or independent t test statistic was used and when a small number of cases was studied, the non-parametric Kruskal-Wallis statistic was applied. ANOVA was used to compare age and MMSE between groups.

The main study outcome was the proportion of change in the level of confidence in the clinical diagnosis before and after assessment of the MR images.

Results

MRI data were available for 125 consecutively referred patients. Reasons for exclusion were a pacemaker (n = 5) and refusal (n = 4). Another 6 had recently undergone CT imaging. In these 6 patients, CT imaging was performed to check the location of a deep brain stimulus device (n = 1), to rule out a cerebrovascular event (n = 1), to check for cerebral metastases in a patient suffering of lung cancer (n = 1), and to minimize scan time because of frailty due to progressive chronic obstructive pulmonary disease, panic disorder and behavioral aggressive disorder, respectively. Demographic characteristics of the sample are presented in table 1.

Table 1.

Demographics, MMSE and diagnosis before and after discussion of MR results in the two physician groups

	Experienced (n = 3)	Inexperienced (n = 3)
Patients, n	66	59
Mean age ± SD, years	73±8.1	72±8,7
Female sex	32 (48)	22 (37)
Mean MMSE ± SD	25±4.1	25±4.5
Pre-MRI diagnosis
Subjective symptoms	6 (9)	8 (14)
MCI	38 (58)a	22 (37)
AD	10 (15)b	10 (17)
VaD	0	6 (10)
Other dementias	5 (8)c	5 (8)
Other neurology	1 (2)	3 (5)
Psychiatry	5 (8)	2 (3)
Other diagnosis	1 (2)	3 (5)
Post-MRI diagnosis
Subjective symptoms	9 (14)	7 (12)
MCI	38 (58)	23 (39)
AD	10 (15)	10 (17)
VaD	0	2 (3)
Other dementias	5 (8)	5 (8)
Other neurology	0	5 (8)
Psychiatry	3 (5)	3 (5)
Other diagnosis	1 (2)	4 (7)

Values are shown as n (%), unless otherwise indicated.

^aMCI = MCI-a + MCI-md + vascular cognitive impairment.

^bAD = AD + mixed AD.

^cOther dementias = dementia with Lewy bodies + frontotemporal lobe dementia + Parkinson's disease dementia complex.

We compared diagnoses across the groups of experienced and inexperienced clinicians. As the number of patients in some diagnostic groups was low, comparisons were made for four groups: subjective symptoms, MCI, AD and other dementias. Diagnoses made by the two groups of physicians did not differ significantly for the pre-MRI diagnosis groups (ϧ² = 2.71, p > 0.05) or for the post-MRI diagnoses (χ² = 1.27, p > 0.05). Results of the indications of diagnostic certainty before and after presentation of MR images are presented in table 2. In 2 patients, no post-MRI indications of diagnostic certainty were available which allowed us to study the results of 123 patients.

Table 2.

Physician confidence in pre- and post-MRI diagnosis in mean percentages and SD, t test statistic and significance level

	n	Mean % (SD)	t test	p
All physicians
Pre-MRI	123	–	–4.789	<0.001
Post-MRI	123	75 (14.7)
Experienced physicians
Pre-MRI	66	69 (13.9)	–1.827	0.072
Post-MRI	66	72 (15)
Non-experienced physicians
Pre-MRI	57	70 (12.1)	–5.038	<0.001
Post-MRI	57	79 (13.4)
Absolute differences pre- and post-MRI
All physicians	123	11 (9.6)
Experienced physicians	66	9 (8.1)	1.767	0.08
Non-experienced physicians	57	12 (10.9)

The physicians indicated a mean 5% increase in their diagnostic certainty (independent-samples t test, t = −4.789, p < 0.001). There was an effect of level of experience. The mean increase of confidence in the experienced group was 3% (dependent-samples t test, t = −1.827, p = 0.072) and in the inexperienced group it was 9% (dependent-samples t test, t = −5.038, p < 0.001). In absolute differences, the change of diagnostic certainty was 9% in the experienced group and 12% in the inexperienced group (independent-samples t test, t = 1.767, p = 0.08). Combined for both groups, this difference was 11%.

In 18% of cases, the diagnosis was changed due to assessment of MR images. For the diagnosis groups mentioned in table 1, the number of cases with a new diagnosis after MRI out of the total number of cases in this diagnosis group before MRI were, respectively, subjective symptoms 2/14, MCI 9/60, AD 4/17, VaD 3/5, other dementias 0/10, other neurology 1/3, psychiatry 2/7.

Pre- and post-MRI certainty did not differ for different age groups. All mean absolute differences varied around 11%. On visual inspection of the number of increased, decreased and unchanged levels of confidence and the corresponding boxplot, there seemed to be an effect of age in the number of assessments with an increased confidence level. This was highest in the youngest age group (65%), a figure that varied around 50 % in the older age groups. The number of decreased levels in the youngest age group [11%) doubled in the oldest age group (table 3). This trend is illustrated in the boxplot where the number of absolute differences above the median of 10% in the youngest age group is greater than in the oldest age group (fig. 1). This effect, however, was not statistically significant.

Table 3.

Mean absolute differences in confidence levels before and after MRI, and number of assessments with an increased, decreased or equal confidence level before and after MRI in different age groups

	Mean absolute difference in CL, % (SD)	IL, n	DL, n	NL, n
All patients	11 (9.6)	55	16	29
Age group
<70 years (n = 43)	12 (8.6)	65	11	24
70–79 years (n = 45)	11 (10.6)	49	16	35
≥80 years (n = 35)	10 (9.4)	51	23	26

CL = Confidence level; IL = increased confidence level; DL = decreased confidence level; NL = equal confidence level.

Fig. 1.

Boxplot of mean absolute differences in confidence levels before and after MRI for three age groups.

Experienced physicians indicated in 51 (80%) cases the test battery as most helpful in increasing their confidence level. Inexperienced physicians indicated in 38 (66%) cases the test battery as most helpful (χ² = 3.096, p = 0.078).

Results of the Vignette Study

Table 4 shows the results of the case vignette study. The case vignette study showed a mean increase in confidence before and after MRI of 7% (paired-samples t test, t = −5.046, p = 0.000). In an absolute figure, this amounted to 11%. The level of confidence increased from 69% after MRI to 82% after additional presentation of test results, which represented a significant difference in a paired-samples t test (t = −5.998, p = 0.000). In an absolute figure, this difference was 16%.

Table 4.

Vignette substudy, levels of confidence in percentages and SD for mean increase and mean absolute change, t statistic and level of significance

	n	Mean (SD)	t test	p
Pre-MRI	50	62 (9.9)	–5.046	0.000
Post-MRI	50	69 (11.2)	–5.998	0.000
Post-MRI test results	50	82 (11.7)
Absolute change pre- and post MRI	50	11 (7.1)
Absolute change pre- and post-MRI test results	50	16 (11.4)

In 2/125 cases, MRI showed an unexpected finding. In 1 patient, a glioblastoma multiforme was diagnosed on MRI as a cause of a rapidly progressive cognitive disorder. Another MRI suggested cerebral toxoplasmosis in a patient that received a kidney transplantation in 2003 and since then used medication that impaired the immune system. Four patients were diagnosed with VaD. In 6 patients who underwent CT imaging, no treatable causes of cognitive symptoms were found.

Discussion

In this study, we evaluated how MRI findings influence the clinical diagnostic certainty. Physicians indicated a statistically significant mean increase of 5% in their level of confidence, this was an 11% absolute difference.

The effect of using MRI was probably larger due to two factors. In the main study, the pre- and post-MRI indications of confidence were given on the same form. The position of the first indication could have influenced the positioning of the second indication resulting in a smaller change of confidence. The vignette study was performed to check for this psychological mechanism. The results of the vignette study suggest that the position of the first indication indeed influenced the positioning of the second indication. The percentages’ mean increase and absolute difference were larger in the vignette study in comparison to findings in the experienced group in the main study. Also, given our finding that inexperienced physicians indicated even higher percentages, the mean increase of 7% and the 11% absolute difference of the vignette study are therefore probably reliable indications of the effect of using MRI assessment on the experienced level of confidence. The second factor is the skewedness of the distributions of confidence. This compresses the distances between the indicated confidence levels. When these differences would be projected on a line with normal distribution, the differences pre- and post-MRI would probably be greater.

In the literature, we found one report in which the minimum clinically significant difference in VAS pain scores for acute pain were studied [32]. These authors suggest that 9 mm, in our study 9%, is the minimum change to be regarded as being clinically relevant. A VAS has not been used before in the way we did and we therefore cannot make any comparisons. We consequently feel, given that the percentages we found are in the range of 9%, that our results are clinically relevant and indicate that MRI is a useful diagnostic tool in the setting of our practice.

As far as we know, this is the first study that tried to quantify the subjectively experienced clinical usefulness of MRI in everyday practice of diagnosing cognitive syndromes in an outpatient memory clinic outside a tertiary referral center. We hypothesized that the level of diagnostic confidence would differ between experienced and inexperienced physicians. This proved to be the case, although in absolute figures the difference was not significant. We furthermore studied the reliability of our method by performing a vignette study. Another strength of this study was the inclusion of all consecutively referred patients to the memory clinic. No patients refused to sign an informed consent form and all but 6 patients underwent MRI. We therefore do not expect an important selection bias in our results.

We did not study the effect of patients’ age on the change of the feeling of confidence in the diagnosis by using MRI. In retrospect, we feel this is a weakness of the study as it is well known that with increasing age there is an increasing overlap between anatomical changes due to age and due to the presence of a neurodegenerative disease. Age could therefore influence the way physicians use MRI in diagnosing disease. It would be valuable to know the effect of age on the way MRI is used and consequently have an indication how to use MRI in younger and older patients.

Although on inspection of table 3 and figure 1 there seemed to be an age effect on change of the confidence level, this was not a statistically significant effect. This might be due to the small number of cases per age group available in this analysis.

As a side result, we confirmed earlier studies that reported on the number of treatable causes of a cognitive syndrome diagnosed by imaging the brain [38, 39, 40]. In our study, we found unexpected findings in only 2 of 125 MRI cases. The glioblastoma was deemed untreatable at the moment of discovery in the 86-year-old patient The immune-compromised patient thought to have a cerebral toxoplasmosis was eventually diagnosed suffering from a cerebral B-cell lymphoma and died shortly afterwards. In the 6 patients that underwent a CT of the brain previously to this study, no treatable causes of dementia were found. Also, in patients showing cerebral lacunar infarcts secondary preventive treatment for cerebrovascular disease was started in an earlier phase. No large cerebral infarcts were found.

We conclude that in a community-based memory clinic, MRI increases the level of certainty of physicians about their diagnosis. The results suggest that inexperienced clinicians gain more confidence than experienced colleagues using this diagnostic tool. It would be worthwhile to study the effect of age of the patients on the change of confidence using MRI. In addition, we confirmed earlier studies reporting the low yield of finding treatable causes using neuroimaging.

Disclosure Statement

The authors declare no conflicts of interest for this article.