We have studied the manuscript of Nicholas et al. [1] very attentively; here are our comments:
The authors have used a different dataset (ADNI-3, rather than ADNI-2 used in [2]). The protocols of ADNI-2 and ADNI-3 datasets are not fully consistent [3]. The ADNI MR data set includes a wide range of scanner platforms; however, there has been a broad gap between older MRI systems and the state-of-the-art systems within each vendor’s product line. In ADNI-3, the “ADNI 3 Basic” and “ADNI 3 Advanced” protocols were used. The authors failed to mention if the images they used were made using a protocol compatible with ADNI-2. The dMRI spatial resolution was improved between ADNI-2 and ADNI-3 by reducing the voxel size from 2.7 × 2.7 × 2.7 mm to 2.0 × 2.0 × 2.0 mm [4]. This may have influenced the results. Moreover, the classification results among these studies are not directly comparable, because they differ in terms of the sets of participants.
We fully agree that the replication of important findings by multiple independent investigators is fundamental to the accumulation of scientific evidence [5]. Deep learning network models are notoriously known for being difficult to replicate, even if the same sets of parameters are used. The training of neural network models is not deterministic, so the models are likely to produce differing results [6]. The strive of the authors to precisely replicate the results may not be achievable.
Considering the description of the training process described in their manuscript, we have tried our model on the ADNI-3 dataset using both cross-validation procedures used by To et al., However, we failed to replicate their results (see the result Table 1 and Figure 1).
Our result is not exceptional. In fact, it is in line with the state-of-the-art studies, which achieved a similar high performance in the ADNI dataset by using 2D CNN, ResNet-18 [7] and custom CNN [8], as well as in other datasets such as OASIS [9,10]. We are somewhat puzzled as to why the performance reported by To et al. on the ADNI dataset is so low.
Table 1.
Replicated results.
| Binary Classes | Accuracy (%) | Sensitivity (%) | Specificity (%) |
|---|---|---|---|
| EMCI vs. LMCI | 70.62 | 68.98 | 95.23 |
| CN vs. EMCI | 77.30 | 73.50 | 92.03 |
Figure 1.
Confusion matrices of replicated result.
Funding
This research received no external funding.
Conflicts of Interest
The authors declare no conflict of interest.
Footnotes
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Nicholas P.J., To A., Tanglay O., Young I.M., Sughrue M.E., Doyen S. Using a ResNet-18 Network to Detect Features of Alzheimer’s Disease on Functional Magnetic Resonance Imaging: A Failed Replication. Comment on Odusami et al. Analysis of Features of Alzheimer’s Disease: Detection of Early Stage from Functional Brain Changes in Magnetic Resonance Images Using a Finetuned ResNet18 Network. Diagnostics 2021, 11, 1071. Diagnostics. 2022;12:1094. doi: 10.3390/diagnostics12051094. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Odusami M., Maskeliūnas R., Damaševičius R., Krilavičius T. Analysis of Features of Alzheimer’s Disease: Detection of Early Stage from Functional Brain Changes in Magnetic Resonance Images Using a Finetuned ResNet18 Network. Diagnostics. 2021;11:1071. doi: 10.3390/diagnostics11061071. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.ADNI|Alzheimer’s Disease Neuroimaging Initiative. [(accessed on 23 March 2021)]. Available online: http://adni.loni.usc.edu/
- 4.Zavaliangos-Petropulu A., Nir T.M., Thomopoulos S.I., Reid R.I., Bernstein M.A., Borowski B., Jack C.R., Jr., Weiner M.W., Jahanshad N., Thompson P.M., et al. Diffusion MRI Indices and Their Relation to Cognitive Impairment in Brain Aging: The Updated Multi-protocol Approach in ADNI3. Front. Neuroinform. 2019;13:2. doi: 10.3389/fninf.2019.00002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.U.S. National Academies of Sciences . In: Engineering, and Medicine Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Schwalbe M., editor. The National Academies Press; Washington, DC, USA: 2016. [PubMed] [Google Scholar]
- 6.Wen J., Thibeau-Sutre E., Diaz-Melo M., Samper-González J., Routier A., Bottani S., Dormont D., Durrleman S., Burgos N., Colliot O., et al. Convolutional neural networks for classification of Alzheimer’s disease: Overview and reprod ucible evaluation. Med. Imag. Anal. 2020;63:101694. doi: 10.1016/j.media.2020.101694. [DOI] [PubMed] [Google Scholar]
- 7.Ramzan F., Khan M.U.G., Rehmat A., Iqbal S., Saba T., Rehman A., Mehmood Z. A Deep Learning Approach for Automated Diagnosis and Multi-Class Classification of Alzheimer’s Disease Stages Using Resting-State fMRI and Residual Neural Networks. J. Med. Syst. 2019;44:37. doi: 10.1007/s10916-019-1475-2. [DOI] [PubMed] [Google Scholar]
- 8.Basaia S., Agosta F., Wagner L., Canu E., Magnani G., Santangelo R., Filippi M., Alzheimer’s Disease Neuroimaging Initiative Automated classification of Alzheimer’s disease and mild cognitive impairment using a single MRI and deep neural networks. NeuroImag. Clin. 2019;21:101645. doi: 10.1016/j.nicl.2018.101645. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fulton L.V., Dolezel D., Harrop J., Yan Y., Fulton C.P. Classification of Alzheimer’s Disease with and without Imagery Using Gradient Boosted Machines and ResNet-50. Brain Sci. 2019;9:212. doi: 10.3390/brainsci9090212. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Zhang Y., Teng Q., Qing L., Liu Y., He X. Lightweight deep residual network for alzheimer’s disease classification using sMRI slices. J. Intell. Fuzzy Syst. 2022;42:1885–1893. doi: 10.3233/JIFS-211247. [DOI] [Google Scholar]