The diagnostic accuracy of CTseg segmentation software for dementia in a New Zealand memory service

Abstract

This study examines the accuracy of CTseg segmentation software to diagnose Alzheimer's disease dementia and other dementias using routine CT scans from a New Zealand memory service. Analyzing 168 scans (89 with dementia and 79 without dementia) the software segmented the brain to produce total brain volume and hippocampal volume. CTseg-derived total brain volume (sensitivity 72%, specificity 58%) and hippocampal volume (sensitivity 71%, specificity 62%) were reasonably effective at differentiating dementia from non-dementia at time of diagnosis. Our findings suggest that CTseg automated volumetric analysis has some potential to aid dementia diagnosis in real-world clinical settings.

Introduction

Visual assessment of computer tomography (CT) and magnetic resonance imaging (MRI) has long been employed to improve the diagnostic accuracy of dementia by identifying patterns of atrophy and excluding other brain pathologies. More recently, there has been interest in using automated brain volumetric analysis to assist diagnosis. ¹ The capability of automated methods to provide reliable results without the need for highly specialized clinical expertise also suggests potential as a biomarker for dementia.^2,3 Until now much of the research in artificial intelligence (AI)-assisted diagnosis of dementia has been conducted using MRI but most patients in real-world clinical settings only have access to lower cost CT scans. Recent research suggests that CT may provide comparable results with MRI when used with automated neuroimaging software. ⁴ The CTseg software (https://github.com/WCHN/CTseg), openly available as an extension on the widely used Statistical Parametric Mapping (SPM) 12 neuroimaging software, now offers a means of automatically segmenting CT scans.^5–7 There is evidence of good correlation between CTseg-derived and manually segmented brain volumes, ⁸ and the utility of the software has also been successfully tested in a mortality prediction model. ⁹

Due to the rapidly changing technologies, there has been relatively little research evaluating the diagnostic accuracy of CT segmentation software in dementia. A few recent studies have demonstrated good accuracy in assessment of cortical atrophy and for identification of Alzheimer's disease^4,8,10,11; however, only one of these ⁸ used open-source software. Most of these studies examined scans from people with Alzheimer's disease dementia or normal controls diagnosed in a research center setting, whereas many people seen in clinical practice will have different subtypes of dementia and often mixed dementia if aged over 80. ¹² For a diagnostic test to be useful, it must be applicable in a real-world clinical setting (for example, in an AI-assisted diagnostic system) and be able to distinguish patients with true dementia from those with without dementia within a full spectrum of brain pathology, while also minimizing false positives and false negatives. If shown to be diagnostically accurate in most common subtypes of dementia, CT segmentation software could aid diagnosis in real-world settings that may not have the same resources as a research clinic.

The aim of this study is to assess the diagnostic accuracy of CTseg automated segmentation in distinguishing all-cause dementia from non-dementia in a real-world sample of patients with a mixture of common subtypes of dementia attending a New Zealand (NZ) memory service.

Methods

Setting

De-identified health data from were extracted from patients assessed at a memory service based at a general hospital in New Zealand. Patients undergo a comprehensive diagnostic assessment that includes a clinical interview and examination by a memory team clinician (including collateral history), tests for cognitive and functional impairment, review of comorbidities including substance misuse and prescribed medications, blood tests, and a CT head scan. The diagnosis is made by consensus at the weekly multidisciplinary team meeting.

Sample

We selected de-identified CT scans and health data from a consecutive sample of recently assessed patients attending the memory service. Only patients of European background were included in this preliminary study to reduce the as yet unknown impact of ethnicity, which may be a confounding factor. The sample included scans from people with dementia comprising Alzheimer's disease dementia, vascular dementia, mixed Alzheimer's disease and vascular dementia, and from people without dementia (including patients diagnosed with mild cognitive impairment who were included in the group with no dementia). Patients who had not had a CT head scan within three months of their diagnosis were excluded.

Scans

CT head scans were performed using Siemens or Philips CT Scanners. The imaging was performed with standard CT head scan contrast negative protocol: patient position headfirst supine, tube voltage 120 kv, standard dose modulation, scan extant C2 to vertex, scan direction caudocranial, scan geometry 1 mm thickness and 0.5 increments. The acquired data was reformatted into 1 mm or 3 mm axial, sagittal, and coronal reformats and stored in the PACS – Agfa Impax system. ¹³

CT scans within the Agfa Impax system were transferred to the Syngo.Via application ¹⁴ which was used to export de-identified 1 mm or 3 mm Axial CT scans in a DICOM format. The de-identified images were compressed into .rar format using the WinRAR application ¹⁵ and securely transferred to the University of Auckland (UoA) environment for image processing using ShareFile app ¹⁶ hosted by HealthAlliance New Zealand.

Image processing

The DICOM CT images were Nifti reformatted using the SPM_dicom_convert tool in SPM12 ⁷ and then segmented using the CTseg pipeline. CTseg ⁷ is an extension of the unified segmentation algorithm for SPM12 ¹⁷ and includes several enhancements such as improved registration, priors on Gaussian mixture model parameters, and an atlas that has been trained on from both MRI and CT scans. ¹⁸ The CTseg pipeline produces several outputs, including the total intracranial volume (TIV), total brain volume (TBV) and various tissue maps: grey matter, white matter, cerebrospinal fluid (CSF), bone, soft tissue, and background. ⁸ To address any mask misfitting, images were co-registered and resliced to the Automated Anatomical Labelling (AAL) atlas ¹⁹ using SPM12, and then the hippocampus mask from the AAL atlas applied to calculate the hippocampal volume. The image processing pipeline is summarized in Figure 1.

Figure 1.

Image processing pipeline. This figure outlines the process for extracting brain volumes from CT scans. The steps include: (1) loading the images into CTseg software, (2) Extraction of total brain volume (TBV) and total intracranial volume (TIV), grey matter volume (GM), and white matter volume (WM) from CTseg (3) normalization of grey matter (GM) and white matter (WM) outputs from CTseg to Montreal Neurological Institute (MNI) space, (4) application of the hippocampal mask from the Automated Anatomical Labeling (AAL) atlas, and (5) extraction of hippocampal volumes.

Statistical analysis

Age, sex, dementia diagnosis, subtype and severity were extracted from routine health data. All data were de-identified prior to analyses. T-tests and Pearson chi-squared tests were applied to investigate baseline differences between groups with dementia and without dementia. Differences in TBV, hippocampal volume (HV), grey matter (GM), and white matter (WM) volumes were assessed by dementia status and severity, as well as by age and sex.

We used One-way Analysis of Covariance (ANCOVA) to compare TBV, HV, GM, and WM volumes by dementia status, adjusted for age and TIV to account for head size variations. ²⁰ Levene's test for equality of variances and tests for normality were performed to ensure the assumptions were met. We estimated the association of dementia status with brain volumes using stepwise logistic regression, adjusted for age, and TIV. The regression coefficients were then used to assess diagnostic accuracy of various combinations of brain volumes for dementia vs non-dementia estimating specificity, sensitivity, and the area under the receiver operating characteristic curve (AUC). Statistical analysis was performed on SPSS Version 29.0.1.0 (171) ²¹ and R statistical package version 4.3.2. ²²

Ethics

Ethical approval for this study was given by the New Zealand Health and Disability Ethics Committee (HDEC: 17/NTB/191) on 18 October 2017. Individual consent in this study was waivered by the Ethics Committee as it involved analysis of retrospective routinely collected data.

Results

Sociodemographic and clinical characteristics

The dementia group (n = 89) included Alzheimer's disease (n = 32), vascular dementia (n = 33), and mixed dementia (n = 24). The non-dementia group (n = 79) consisted of 40 people diagnosed with mild cognitive impairment and 39 with no neurocognitive disorder. There was a difference in the mean ages between the non-dementia group (78.4 years, sd = 6.65) and the dementia group (81.7 years, sd = 5.79), p < 0.001) but no difference in sex between dementia (55% male) and non-dementia groups (47% male, p = 0.29).

Brain volumes

Table 1 presents the mean CTseg brain volumes (TBV, HV, GM, and WM) by sociodemographic and clinical characteristics. The table shows that, compared to the non-dementia group, the volumes for HV and WM were significantly smaller in the dementia group, but there was no significant difference for TBV and GM. Subgroup analysis revealed differences in brain volumes by sex, severity and dementia subtype (Supplemental Figure 1). After adjustment for age and TIV, there were significant differences between dementia and non-dementia groups for all brain volumes. After adjustment for age and TIV, the association with dementia severity was also significant for TBV (p = 0.011), HV (p < 0.001), GM (p = 0.007), and WM (p = 0.002).

Table 1.

Mean CTseg brain volumes by sociodemographic and clinical characteristics.

Variable	Dementia (n = 89) mean (95%CI)	Non-dementia (n = 79) mean (95%CI)	p for differences between dementia and non-dementia (unadjusted except *)
Total brain volume (cm3)
All (unadjusted mean)	993 (970–1015)	1018 (994–1042)	0.127
Adjusted mean	998 (991–1006)	1012 (1004–1020)	0.013*
Male (n = 86)	1055 (1029–1080)	1090 (1060–1119)	0.037
Female (n = 82)	917 (894–941)	955 (933–978)	0.023
65–80 years (n = 87)	1018 (980–1056)	1030 (996–1064)	0.636
81 + years (n = 81)	973 (948–998)	999 (967–1032)	0.204
Mild severity (n = 70)	999 (975–1024)
Moderate severity (n = 19)	969 (923–1016)
Alzheimer's disease (n = 32)	1000 (963–1038)
Vascular dementia (n = 33)	992 (955–1029)
Mixed dementia (n = 24)	984 (940–1027)
Mild cognitive impairment (n = 40)		1019 (986–1053)
No neurocognitive disorder (n = 39)		1017 (983–1051)
Hippocampal volume (cm3)
All (unadjusted mean)	12.4 (11.9–12.8)	13.5 (13.0–13.9)	<0.001
Adjusted mean	12.5 (12.1–12.7)	13.3 (13.0–13.7)	<0.001*
Male (n = 86)	13.2 (12.7–13.8)	14.4 (13.8–15.0)	0.005
Female (n = 82)	11.3 (10.8–11.8)	12.8 (12.2–13.2)	<0.001
65–80 years (n = 87)	12.5 (11.9–13.2)	13.9 (13.3–14.5)	0.003
81 + years (n = 81)	12.2 (11.7–12.7)	12.8 (12.1–13.4)	0.175
Mild severity (n = 70)	12.6 (12.1–13.0)
Moderate severity (n = 19)	11.6 (10.8–12.5)
Alzheimer's disease (n = 32)	12.6 (11.9–13.3)
Vascular dementia (n = 33)	12.7 (12.0–13.4)
Mixed dementia (n = 24)	11.5 (10.8–12.3)
Mild cognitive impairment (n = 40)		13.5 (12.7–14.3)
No neurocognitive disorder (n = 39)		13.5 (12.9–14.1)
Grey Matter volume (cm3)
All (unadjusted mean)	798 (780–815)	815 (797–835)	0.166
Adjusted mean	801 (793–809)	813 (804–821)	<0.001*
Male (n = 86)	847 (827–867)	872 (849–895)	0.097
Female (n = 82)	738 (719–757)	766 (748–784)	0.034
65–80 years (n = 87)	817 (787–846)	818 (792–845)	0.943
81 + years (n = 81)	782 (762–804)	812 (786–839)	0.085
Mild severity (n = 70)	805 (786–824)
Moderate severity (n = 19)	770 (730–812)
Alzheimer's disease (n = 32)	809 (778–841)
Vascular dementia (n = 33)	801 (772–830)
Mixed dementia (n = 24)	778 (746–810)
Mild cognitive impairment (n = 40)		819 (797–842)
No neurocognitive disorder (n = 39)		812 (779–846)
White Matter volume (cm3)
All (unadjusted mean)	550 (535–564)	573 (558–588)	0.027
Adjusted mean	550 (548–560)	569 (562–575)	0.001*
Male (n = 86)	586 (568–635)	614 (594–635)	0.038
Female (n = 82)	505 (490–520)	537 (522–552)	0.004
65–80 years (n = 87)	566 (542–591)	587 (565–609)	0.210
81 + years (n = 81)	537 (522–552)	551 (533–571)	0.227
Mild severity (n = 70)	554 (538–570)
Moderate severity (n = 19)	533 (503–564)
Alzheimer's disease (n = 32)	555 (529–582)
Vascular dementia (n = 33)	552 (531–574)
Mixed dementia (n = 24)	538 (514–561)
Mild cognitive impairment (n = 40)		571 (552–590)
No neurocognitive disorder (n = 39)		575 (548–603)

Diagnostic accuracy of CTseg

Figure 2 presents the receiver operating characteristic (ROC) curve for diagnosis of dementia versus no dementia: TBV had 72% sensitivity (95%CI:61–81), 58% specificity (95%CI:47–69) and an AUC of 0.68 (95%CI:0.60–0.76), whereas HV had 71% sensitivity (95%CI:60–80), 62% specificity (95%CI:50–73) specificity, and AUC of 0.72 (95%CI:0.64–0.80). Stepwise logistic regression sequentially adding all brain volumes (TBV, HV, GM, and WM, adjusted for age and TIV) into the model did not improve the overall accuracy. For the model that included all variables (TBV, HV, GM, and WM, plus age and TIV), the AUC was 0.74 (95%CI:0.66–0.81), sensitivity was 69% (95% CI:59–78) and specificity was 60% (95% CI:50–71).

Figure 2.

ROC curves for total brain volume and hippocampal volume in the diagnosis of all cause dementia versus no dementia. (A) ROC curve using total brain volume adjusted for age, AUC of 0.68 (95%CI: 0.60–0.76). (B) ROC curve using hippocampal volume adjusted for age, AUC of 0.72 (95%CI: 0.64–0.80).

Discussion

The results of our study suggest that the CTseg software produces feasible estimates for brain volumes in our sample of patients with mixed subtypes of common dementias. Mean brain volumes were lower in the dementia subgroup compared to the non-dementia subgroup, and volumes in moderate dementia severity were lower than for mild dementia severity. The mixed dementia subgroup had lower mean brain volumes than the vascular dementia subgroup which were lower than the Alzheimer's disease subgroup, this presumably reflects the extent of structural changes in the brain. The mean age of our dementia group (81.7) was significantly older than the non-dementia group (78.4) but the difference was adjusted for in the analysis.

The diagnostic accuracy of the model that included all CT-seg brain volumes was reasonable with low to moderate sensitivity (69%, 95% CI:59–78) and low specificity (60%, 95% CI:50–71). In a sample of 1000 untested patients (of whom we expect 10% to have dementia) this translates to 429 screen positives, of whom only 69 would have true dementia but 360/429 will not have dementia (false positives). False positives, particularly in the diagnosis of an irreversible untreatable disease such as dementia, may cause psychological harm and should be avoided.^23,24 Therefore this model could not be used alone. However similar models that also include sociodemographic and health data (for example comorbidities, health utilization data, lab tests and prescribed medications) have been shown to have high specificity of 85% which would greatly reduce the number of false positives, improving acceptability and cost-effectiveness. ²⁵ The addition of amyloid, tau and other biomarkers for dementia would likely further improve the accuracy, however these are not available in many public sector services including our own.

Due to CT segmentation software being a relatively new technology in dementia, there are very few studies to compare our findings. Srikrishna et al. (2024) ¹⁰ was the only study we found that reported the diagnostic test accuracy of CT segmentation software to differentiate dementia (of various diagnostic subtypes) from non-dementia. Using their own in-house software and a combination of TBV, WM, and GM (adjusted for CSF and/or ventricle size), they reported an AUC of 0.95, with approximately 90% sensitivity and 90% specificity for the composite score (estimated from published ROC curve). Our findings (sensitivity 60%, specificity 69%) were lower than these, but Srikrishna et al. (2024) ¹⁰ excluded mild cognitive impairment from the non-dementia group, which can increase the apparent diagnostic accuracy of the tool due to spectrum bias (a lack of people in the grey area between disease and non-disease). It was also unclear whether they only examined people with Alzheimer's disease dementia. We included patients with Alzheimer's disease, vascular and mixed dementias in our dementia group (and mild cognitive impairment in our non-dementia group) to enhance the generalizability of our findings ²⁶ but at the potential expense of apparent diagnostic accuracy.

While the strength of our study is the use of deidentified routine health data and minimization of exclusions, the main weakness of our study is the possibility of incorporation bias. The CT brain scans used in the study would have been viewed to help make the dementia diagnosis in the memory service and therefore the CT-seg derived volumes were not completely independent of the gold standard. We have therefore planned future research to conduct external validation in an independent sample where the CT scans were not used as part of the diagnostic work-up, and to potentially improve the accuracy of the model by addition of other routinely collected health data (for example, comorbidities, health utilization data, lab tests and prescribed medications). In addition, as our exploratory study included only NZ European subjects to minimize potential confounding by ethnicity, we intend to apply the CTseg software to other ethnic groups living in NZ and elsewhere to ensure generalizability to the whole population. Further research, especially multicenter studies conducted in a variety of clinical settings, are urgently required to confirm that these predictive models are accurate in dementia. If the AI models are found to be diagnostically accurate, their deployment in the clinical workflow needs to be considered, involving compliance with legal regulations, establishing user trust, and ensuring data security. ²⁷

Conclusion

Our study suggests that CTseg automated volumetric analysis has potential to aid the diagnosis of dementia in real-world clinical settings, but the low specificity limits its current applicability due to the potentially high number of false positives. However, the technique has great potential to enhance AI-assisted dementia diagnosis, especially if enhanced with routinely collected health data. These approaches are likely to become part of mainstream practice within a few years. Further research is urgently required to confirm that these predictive models are accurate.