Diagnosing Multiple Sclerosis from Magnetic Resonance Imaging Images: Highlights from the Second Isfahan Artificial Intelligence Event 2024

Abstract

Background:

Multiple sclerosis (MS) is an autoimmune disease of the central nervous system which is the main reason of disabilities of young adults. MS occurs when the immune system attacks the central nervous system and destroys the myelin sheaths of neurons. Loss of myelin sheaths results in appearing several lesions in different parts of the brain. The place and amount of lesions are important criteria for determining the level and progression of the disease. These parameters are usually determined manually by an expert which can be time-consuming and inaccurate.

Methods:

Considering the effectiveness of artificial intelligence (AI)-based methods in diagnosing and predicting different diseases, and the increasing need for driving new and effective diagnostic methods, this challenge, entitled “Diagnosing MS from magnetic resonance imaging (MRI) Images,” has been organized by Isfahan Province Elites Foundation in collaboration with Medical Image and Signal Processing Research Center of Isfahan University of Medical Sciences, as a part of Isfahan AI 2024 event, held in October 2024 in Isfahan, Iran. The challenge has been dedicated to find new AI-based methods for the segmentation and localization of lesions in MRI images of patients with MS. The challenge had three steps, where in the first and second steps, the teams received the train and test datasets, respectively. Finally, the selected teams were invited to the last round of the competition, held in person, and received the last test dataset.

Results:

Based on the received results, the best achieved dice score was 0.33, best sensitivity was 0.349, best precision was 0.3, and the lowest centroid distance was 53.025. In addition, the best accuracy for lesion detection in periventricular, deep white matter, juxtacortical, and infratentorial parts of the brain was 80.282%, 74%, 63.492%, and 62.5%, respectively.

Conclusion:

Several methods, mostly based on deep learning, have been submitted. The results show that AI has the ability for the segmentation and localization of lesions. However, the received results are still far from the desired accuracy, which shows a need for further improvement and studies in this field.

Introduction

Multiple sclerosis (MS) is an autoimmune disorder which affects the central nervous system.[1,2,3,4,5,6,7] The disease is the main responsible of most of disabilities in young adults and occurs when the immune system incorrectly attacks the central nervous system. As a result of this attack, the myelin sheaths of neurons are destroyed which results in scares or lesions in different parts of the brain.[1,2,3,4,5,6,7]

The exact reason of MS is uncovered yet. There are some possibilities of the disease occurs due to genetic records, environmental situation, vitamin D shortage, smoking, obesity, or infectious agents.[1,8,9] The symptoms of the disease vary for different people; however, the common symptoms include blurred vision, muscle weakness, fatigue, inability to maintain balance, dizziness, and numbness. The disease in its higher levels can highly destroy the normal life of a patient and impose high emotional and financial expenses to the families and society. Therefore, diagnosing MS in its early stages and preventing its harmful effects are of high importance.

Based on the McDonald criterion, the place and amount of lesions are important factors for MS diagnosis.[10,11,12,13] By injecting Gadolinium, the lesions can be viewed as bright stains in magnetic resonance imaging (MRI) images of patients.[14] Detecting and localizing the lesions are usually done manually by a physician, which is a time-consuming procedure and can be accompanied by error.

Recently, artificial intelligence (AI)-based methods have received high consideration in different biomedical areas. The methods are designed for detecting and localizing anomalies, disease classification and diagnosis, and even the prediction of the occurrence or reoccurrence of different cancers. Studies show the success of AI-based tools in all of the mentioned areas in addition to their lower cost and processing time comparing to the conventional methods.[15,16,17]

The AI-based methods have also been exploited for MS lesion segmentation and localization. However, the existing methods are not still as effective as expected. This shows a need for newer and more applicable methods in this field.[18,19,20,21,22,23,24]

Considering this issue, the challenge “Diagnosing MS from MRI Images” has been organized by Isfahan Province Elites Foundation in collaboration with Medical Image and Signal Processing Research Center of Isfahan University of Medical Sciences, as a part of Isfahan AI (IAI 2024) event, held in October 2024 in Isfahan, Iran. The challenge was dedicated to the segmentation and localization of lesions in periventricular, infratentorial, deep white matter, and juxtacortical regions of the brain using MRI images. This paper is devoted to the description of the challenge, the dataset, evaluation metrics, and the achievements of the participating teams.

The paper has been organized as: In Section II, a description of the challenge and the dataset has been presented. In Section III, details of the winner methods have been presented. Evaluation metrics and the results are presented in Sections IV and V, respectively. Finally, Sections VI and VII are devoted to the discussion and conclusion, respectively.

Dataset and Challenge Description

Dataset description

The dataset exploited for this challenge consisted of MRI images of 90 patients with neurologist-confirmed MS, meeting the 2017 McDonald criteria. Imaging was conducted using a 1.5 Tesla Siemens Avanto scanner (Siemens Healthcare, Erlangen, Germany) at Kashani Hospital, Isfahan, Iran, using a 12-channel coil.

Three-dimensional (3D) FLAIR sequences were acquired for each patient using the following parameters: repetition time = 5000 ms, echo time = 331 ms, inversion time = 1800 ms, and a 256 mm × 256 mm field of view with a 1-mm slice thickness. The images were transformed to the Montreal Neurological Institute (MNI) space (matrix size 181 × 217 × 181 and were stored in the NIFTI format.

The initial lesion segmentation was performed using Volbrain. Subsequent manual correction of segmented lesions was conducted by two neurologists specializing in MS, utilizing ITK-SNAP software (version 3.8.0; http://www.itksnap.org). Preprocessing steps comprised denoising (using a spatial adaptive nonlocal means filter), MNI space registration (using Advanced Normalization Tools), inhomogeneity correction (using N4 bias field correction), and intensity normalization.

Each 3D image has a mask image with the same size, which shows the place of lesions in the brain. Sample of an MRI slice and its corresponding mask have been shown in Figure 1. Lesions in different parts of the brain have been shown with different intensities, where intensities 1–4 correspond to lesions in periventricular, deep white matter, juxtacortical, and infratentorial regions, respectively.

Figure 1.

Lesion mask with four different intensities

Challenge description

The challenge had three steps. In the first step, the teams received the training dataset, contained 70 MRI images with their corresponding masks. The teams had to prepare and train their networks for the detection and localization of lesions. Then, in the second step, the teams received the test dataset, which contained MRI images of 10 patients without their respective masks.

All participants were required to submit lesion masks containing bounding boxes. A bounding box for each lesion is defined as a cubic region which completely filled its respective lesion. Sample for a lesion and its corresponding bonding box have been illustrated in Figure 2. The intensities of the masks varied from 1 to 4, showed the place of the lesion in different parts of the brain. The lesion masks were required to have the same dimensions (matrix size) as the patient’s FLAIR image.

Figure 2.

A sample for a lesion (colored in red) and its corresponding bonding box (colored in green)

The teams also had to send a comprehensive report of their approaches including the details of their methods implementation and parameter selection, in addition to a 5-min video explaining their methods.

Twenty-four teams submitted their results to the organizers. Among them, 10 teams were removed due to several technical issues. After a comprehensive evaluation using different metrics (see the next section), the following five teams were selected as finalists and invited to the last round of the competition, held at the Isfahan Science and Technology Town:

• BME-UT

• CBRC2

• TeaRelaxation

• Physic pezeshki

• Aimedic.

In the last round of the competition, the finalists received a new test dataset consisted of 7 MRIs (without their respective masks) to test their methods and drive the outputs in 1-h time in the presence of the judges. Based on the final evaluations, the finalists were ranked as:

• 1-BME-UT

• 2-CBRC2

• 3-TeaRelaxation.

Technical Highlights of Winners

The methods exploited by the winners have been reviewed in the following three subsections.

CBRC2

The method used by “CBRC2” consisted of 4 steps: 1 – Preprocessing, 2 – Pretraining, 3 – Fine-tuning, and 4 – Postprocessing.

In the preprocessing step, the data were normalized into the range (0–1), and the size of the data was reduced in a way that the margins with no information were removed. Furthermore, the data were augmented by the constructive learning method.

A vision transformer was then used in the pretraining step. Using the resulting weights, a UNETR architecture[25] was fine-tuned and exploited for lesion segmentation.

Finally, in the postprocessing step, the bounding boxes corresponding to segmented lesions were extracted. The method exploited by “CBRC2” is summarized in Figure 3.

Figure 3.

The method exploited by “CBRC2”

TeaRelaxation

The “Tearelaxation” team tested the lesion segmentation by several two-dimensional (2D) and 3D models. Among the 2D models, UNET++,[26] which is an improved version of the UNET model, showed considerably better segmentation performance comparing to the other 2D methods. However, since 2D structures were unable in 3D data processing, their performances were not as high as expected.

The team also exploited 3D structures for lesion segmentation. Among the tested methods, SWIN-UNETR[27] had the best performance. The SWIN-UNETR architecture is a combination of the SWIN transformer and UNET network, where the SWIN transformer is a Self-Attention mechanism for driving complicated and non-linear relations among different parts of an image. The overall approach is presented in Figure 4.

Figure 4.

The overall approach of the TeaRelaxation team

BME-UT

The method of “BME-UT” included 1 – Preprocessing of data, 2 – Data augmentation, and 3 – Lesion segmentation.

In the preprocessing step, the skull was removed from images using the method of.[28] After normalization of the images by z-normalization, the background regions of the images (black parts) were removed.

Data augmentation was done by adding Gaussian noise, Gamma transform (for changing the intensities of the image), and applying different transforms such as rotation.

Finally, the UNET structure was exploited for lesion segmentation. The used UNET consisted of five blocks in the encoder, where the first block contained 32 filters, where the number of filters was doubled in each block. The block diagram of the “BME-UT” method has been illustrated in Figure 5.

Figure 5.

The block diagram of the method exploited by the team “BME-UT”

Evaluation Metrics

Lesion masks submitted by the participants were evaluated using a reference mask derived from clinically confirmed data. Some participant masks failed to completely cover the lesions or detected additional, nonexistent lesions. Therefore, their bounding boxes were occasionally too large, encompassing multiple lesions, or too small, partially covering the actual lesion. To enhance the accuracy of comparisons, the dice coefficient method, defined as follows, was innovatively applied. The dice score was calculated for each lesion to quantify the degree of overlap between the reference and detected lesion masks.

In addition to dice, sensitivity, and precision, defined as follows, were identified as appropriate metrics for evaluating the accuracies of selected bounding boxes:

where the true positive percentage is defined as the common volume between the ground-truth mask and the resulting bounding box. The true negative (TN) is defined as the background area outside the lesion. False negative is defined as parts of lesions which are outside of the bounding box, and false positive is the parts of the bounding box which do not related to the ground-truth lesion. For a better understanding, please see Figure 6. Due to the typically small size of lesions, the TN value often constituted a large proportion of the overall volume. As a result, parameters such as specificity and accuracy were deemed less informative and did not considered here.

Figure 6.

Illustrations of true positive, false positive, false negative, and true negative. The green box is the estimated bounding box, and the brown lesion is the ground-truth mask for the lesion

The centroid distance was also employed to evaluate localization accuracy. This metric is defined as the distance between the centroids (geometric centers) of the ground-truth lesion and the corresponding bounding box, providing a quantitative assessment of spatial alignment in medical imaging.

To ensure that all groups automatically generated lesion bounding box masks, a final test was conducted under direct supervision. One hour after participants submitted their results, an additional FLAIR image was released. This image was one of the seven images of the final test datasets, whose name was altered. This was done for the assessment of the results honesty and reproducibility. Participants were required to create a corresponding lesion mask in the presence of a supervisor. Groups that manually corrected their results were identified and disqualified from the competition.

Participating Results

In this section, the results of the three best groups for the last round of the competition have been presented. Dice scores, sensitivity, and precision for the results of the finalists in four different parts of the brain have been reported in Tables 1-3. As the results show, the team “CBRC2” had the best averaged evaluation parameters.

Table 1.

Dice scores of the results of the three finalists for lesion segmentation in different parts of the brain

Team name	Brain part
	Periventricular	Deep white matter	Juxtacortical	Infratentorial	Average
CBRC2	0.349	0.411	0.3	0.26	0.33
TeaRelaxation	0.311	0.360	0.34	0.185	0.3
BME-UT	0.315	0.355	0.273	0.242	0.3

Table 3.

Precision percentages of the results of the three finalists for lesion segmentation in different parts of the brain

Team name	Brain part
	Periventricular	Deep white matter	JuxtaCortical	Infratentorial	Average
CBRC2	0.289	0.41	0.26	0.241	0.3
TeaRelaxation	0.237	0.329	0.329	0.053	0.239
BME-UT	0.252	0.282	0.239	0.055	0.207

Table 2.

Sensitivity percentages of the results of the three finalists for lesion segmentation in different parts of the brain

Team name	Brain part
	Periventricular	Deep white matter	JuxtaCortical	Infratentorial	Average
CBRC2	0.519	0.422	0.378	0.081	0.349
TeaRelaxation	0.515	0.334	0.288	0.073	0.303
BME-UT	0.515	0.391	0.381	0.104	0.347

The centroid distances of the results of the finalists for localizing the lesions in different parts of the brain have also been reported in Table 4. Lower centroid distance shows a higher accuracy in lesion localization. As the results show, the team “CBRC2” had the lowest centroid distance comparing to the other two finalists.

Table 4.

Centroid distances of the results of the three finalists for lesion segmentation in different parts of the brain

Team name	Brain part
	Periventricular	Deep white matter	JuxtaCortical	Infratentorial	Average
CBRC2	38.908	42.594	57.44	73.16	53.025
TeaRelaxation	39.429	49.314	53.636	91.723	58.525
BME-UT	37.811	51.497	54.359	88.477	58.036

Finally, the ability of each method for the detection and segmentation of the lesions in different parts of the brain has been evaluated. For this aim, the number of actual lesions in each part of the brain has been compared with the number of segmented and detected lesions by each team. Note that segmented lesions are the lesions which have been segmented by the teams (correctly or incorrectly), but the detected lesions are the lesions which have been segmented correctly. The results have been reported separately for each part of the brain and presented in Tables 5-8.

Table 5.

Comparing the number of segmented and detected lesions with the actual number of lesions in the preventricular

Team name	Periventricular
	Number of ground-truth lesions	Number of segmented lesions	Number of detected lesions	Percentage of detected lesions
CBRC2	71	52	49	69.014
TeaRelaxation	71	68	55	77.465
BME-UT	71	63	57	80.282

Table 8.

Comparing the number of segmented and detected lesions with the actual number of lesions in the infratentorial

Team name	Infratentorial
	Number of ground-truth lesions	Number of segmented lesions	Number of detected lesions	Percentage of detected lesions
CBRC2	24	26	15	62.5
TeaRelaxation	24	20	10	41.667
BME-UT	24	21	14	58.333

Table 6.

Comparing the number of segmented and detected lesions with the actual number of lesions in the deep white matter

Team name	Deep white matter
	Number of ground-truth lesions	Number of segmented lesions	Number of detected lesions	Percentage of detected lesions
CBRC2	50	77	35	70
TeaRelaxation	50	78	37	74
BME-UT	50	79	37	74

Table 7.

Comparing the number of segmented and detected lesions with the actual number of lesions in the juxtacortical

Team name	Juxtacortical
	Number of ground-truth lesions	Number of segmented lesions	Number of detected lesions	Percentage of detected lesions
CBRC2	63	42	25	39.682
TeaRelaxation	63	84	40	63.492

Discussion

The results of the three finalists have been extensively evaluated and compared with each other. As the results show, the best dice (0.33), sensitivity (0.349), precision (0.3), and centroid distance (53.025) have been achieved by “CBRC2.” In addition, “CBRC2” had the best performance in lesion detection in infratentorial (62.5%) and the lowest percentage of detected lesions in the preventricular, deep white matter, and juxtacortical.

The team “BME-UT” achieved the best performance in lesion detection in preventricular (80.282%) and deep white matter along with the “TeaRelaxation” team (74%), and the second best place for lesion detection in juxtacortical and infratentorial. Considering dice and sensitivity, the team BME-UT had the second-best place with slightly lower indices in comparison to CBRC2.

The “TeaRelaxation” had the best lesion detection performance in deep white matter (74%) and juxtacortical (63.492%), and the second best place for lesion detection in preventricular. The team achieved the second place in terms of Dice and Precision, and the lowest performance in terms of Sensitivity and centroid distance.

Considering the achieved results, it seems that still there is a need for more improved and accurate methods for lesion localization and segmentation. The accuracies of the proposed methods and their performances are not sufficient for a reliable diagnosis, and this shows a gap between the existing needs and available methods.

Conclusion

The details and descriptions of the challenge “Diagnosing MS from MRI Images” have been presented. This challenge has been organized by the Isfahan Province Elite Foundation in collaboration with the Medical Image and Signal Processing Research Center, as a part of IAI 2024. The challenge has been organized to familiarize young researchers with the importance of lesion segmentation and localization in MRI images of patients with MS. Several AI-based methods, mostly based on deep learning, have been developed by participants which show improvements in the segmentation and localization accuracies. However, considering the accuracy needed for an effective MS diagnosis, the proposed methods are still far from a desired applicable method. This gap can be filled in future studies by exploiting more complicated and well-designed methods.

Conflicts of interest

The authors declare the following potential conflicts of interest:

• MT was the organizer of the IAI 2024 competitions on behalf of the Isfahan Elite Foundation

• FD, IA, FS, and HR served as scientific committee members for the Challenge “Diagnosing MS from MRI Images.” They were responsible for evaluating the methodologies and results of all participant teams

• ZGH, HA, MPH, MGH, KMD, MR, MHR, MHA, AS, ER, HS, MB, AB, and SPZ are members of the winning teams in this challenge. None of the organizers and scientific committee members (FD, IA, MT, FS, and HR) contributed to the development of the methods used by the participating teams.

The final decision regarding the winners was made by the policy council members based on the following criteria:

• Technical contribution in developed models by teams

• The results on initial and final test data of each team

• The submitted reports and teams’ presentations.

The authors have disclosed these relationships to ensure transparency and maintain the integrity of the research.

Funding Statement

Nil.