High‐Field‐Blinded Assessment of Portable Ultra‐Low‐Field Brain MRI for Multiple Sclerosis

ABSTRACT

Background and Purpose

MRI is crucial for multiple sclerosis (MS), but the relative value of portable ultra‐low field MRI (pULF‐MRI), a technology that holds promise for extending access to MRI, is unknown. We assessed white matter lesion (WML) detection on pULF‐MRI compared to high‐field MRI (HF‐MRI), focusing on blinded assessments, assessor self‐training, and multiplanar acquisitions.

Methods

Fifty‐five adults with MS underwent pULF‐MRI following their HF‐MRI. Two neuroradiologists independently assessed pULF‐MRI images in an evaluation process, including initial assessment blinded to HF‐MRI, self‐training with reference to HF‐MRI and evaluation of 20 cases with additional T2‐fluid‐attenuated inversion recovery in an additional plane. A third rater conducted cross‐referenced analysis with HF‐MRI data to determine true‐positive lesions, false‐positive areas, and case‐level sensitivity and positive predictive value.

Results

The mean age of participants was 50 years (standard deviation: 11; 74% women). Initially, Rater 2 marked more false‐positive areas than Rater 1 (p = 0.003). After self‐training, both raters embraced a conservative approach, with Rater 2 marking fewer false‐positive areas (p = 0.01). Both raters maintained 100% case‐level sensitivity and positive predictive value for detecting at least one WML, particularly in periventricular areas. Multiplanar acquisitions reduced both false‐positive areas and true‐positive lesions. True‐positive lesions and false‐positive areas had similar contrast‐to‐noise ratios in the juxtacortical region (p = 0.73) but not in periventricular, deep parenchymal regions (p = 0.004, p = 0.01).

Conclusion

With adequate training, radiological interpretation of pULF‐MRI has high sensitivity and positive predictive value for MS lesions but should be approached conservatively. These results suggest utility for patient triage, potentially reducing diagnostic delay, and screening high‐risk individuals.

1. Introduction

Multiple sclerosis (MS) is an immune‐mediated disorder characterized by demyelination within the central nervous system. It is the most prevalent progressive neurological disorder in young adults worldwide [1]. MRI at high magnetic field strength (≥1.5 tesla [T]) is a crucial diagnostic tool employed in the assessment and monitoring of MS, allowing clear visualization of imaging characteristics of MS and enabling the evaluation of disease activity and progression [2]. The radiological diagnosis of MS relies on the spatial distribution of white matter lesions (WMLs), characterized as “dissemination in space” (DIS), and the temporal features of WML, including active or inactive lesions and the emergence of new lesions, described as “dissemination in time.” [3]

Recent advancements in compact, cost‐effective [4], and portable ultra‐low field (pULF) (64 millitesla [mT]) MRI scanners have opened promising possibilities for utilizing point‐of‐care MRI in the diagnosis and monitoring of several neurological conditions. Reduced infrastructure requirements, such as radiofrequency‐shielded rooms and coil‐cooling systems, along with the affordability and portability of pULF‐MRI scanners [5], render them a viable option for the implementation of point‐of‐care MRI in the management of neurological diseases. The cost‐effectiveness of the scanner comes from its lower price; based on 2023 prices, it theoretically costs one‐sixth and one‐twelfth of the price of 1.5 T and 3 T scanners, respectively—as well as from reduced maintenance, the absence of specialized long‐duration training to operate the scanner, and lower logistical costs for patients [6]. For MS, especially on a global scale, increased access to MRI may facilitate early diagnosis, leading to the early initiation of disease‐modifying treatments. Consequently, this could lower the global economic burden of the disease by improving prognosis with early detection of disease [7, 8].

Current research supports the feasibility and diagnostic capability of pULF‐MRI in various clinical settings, including intensive care units [9, 10] and the assessment of cerebrovascular disease [11, 12]. Notably, when directly compared to high‐field (HF) MRI, pULF‐MRI has shown significant agreement in WML burden estimation via automated segmentation methods and high sensitivity in detecting WML larger than 4 mm [13]. While these results underscore the remarkable potential of pULF‐MRI in the realm of MS, it remains uncertain how effectively pULF‐MRI can detect WML in a real‐world HF‐MRI blinded scenario.

In this study, we evaluated the WML detection and diagnostic capabilities of pULF‐MRI for MS in a clinical imaging environment where HF‐MRI data were concealed, such that assessments were exclusively reliant on pULF‐MRI. Subsequently, we examined the impact of assessor self‐training and the influence of utilizing additional acquisition planes on the accuracy of WML detection.

2. Methods

2.1. Participants

Imaging, demographic, and clinical data were collected with Institutional Review Board approval and after written, informed consent, as part of the National Institute of Neurological Disorders and Stroke's “Evaluation of Progression in Multiple Sclerosis by Magnetic Resonance Imaging” protocol (NCT00001248). In this protocol, participants with MS [3] undergo annual HF‐MRI of the brain with a variety of advanced research pulse sequences and may receive additional assessments for comparison with the HF‐MRI data, including pULF‐MRI. The imaging data of 15 participants in this cohort has been previously reported [13], with a focus on assessing the accuracy of WML segmentation and the sensitivity of pULF‐MRI to WML when evaluations were done in tandem and unblinded to HF‐MRI. Here, we report the sensitivity of pULF‐MRI to WMLs under blinded conditions, where raters were unaware of the HF‐MRI information.

2.2. MRI Acquisition

HF‐MRI was performed on a 3 T scanner (Skyra; Siemens, Erlangen, Germany). The whole‐brain imaging protocol included 3‐dimensional (3D) T1‐weighted (T1w) fast‐spin‐echo (FSE), T1w magnetization prepared 2 rapid gradient echo (MP2RAGE), and T2w fluid‐attenuated inversion recovery (T2‐FLAIR) sequences, as well as T2*w echo planar images (EPI) and multislice dual‐echo FSE scan for proton density‐ and T2‐weighted (PD‐T2w) images. Gadolinium‐based contrast agent (gadobutrol, 0.1 mmol/kg) was administered intravenously during the HF‐MRI, followed by 3D T1w‐FSE and T2‐FLAIR scans. Subsequently, pULF‐MRI scans were obtained on a 64 mT scanner (Swoop 1.6; Hyperfine, Inc., Guilford, CT) with axial T1w FSE and T2‐FLAIR images. An additional plane of T2‐FLAIR was acquired in a subset of participants when time allowed. A summary of sequence parameters and acquisition times for both pULF and HF‐MRI is provided in Table 1.

TABLE 1.

Acquisition parameters of pulse sequences at high‐field MRI (3 T) and portable ultra‐low‐field MRI (64 millitesla).

Magnetic field strength	Sequence name	Acquisition plane	Repetition time (ms)	Echo time (ms)	Inversion time (ms)	Flip angle (degree)	Slice thickness (mm)	Pixel size (mm)	Scan time (min:s)
3T	T1w‐MP2RAGE ( * )	Axial (3D)	5000	2.9	700/250	0	1.0	1.0 × 1.0	5:38
3T	PD‐T2w ( * )	Axial (2D)	5000	82	N/A	150	3.0	0.34 × 0.34	4:29
3T	T2*‐EPI ( * )	Sagittal (3D)	64	35	N/A	10	0.65	0.65 × 0.65	5:46
3T	T2‐FLAIR ( ** )	Sagittal (3D)	4800	352	1800	120	1.0	1.0 × 1.0	7:12
3T	T1w‐FSE ( ** )	Sagittal (3D)	700	26	N/A	120	0.67	0.67 × 0.67	4:59
64 mT	T2‐FLAIR ( *** )	Axial (3D)	4000	183	1400	90	5.0	1.5 × 1.5	11:19
64 mT	T1WI ( *** )	Axial (3D)	1500	4	300	90	5.0	1.5 × 1.5	6:24
64 mT	T2‐FLAIR ( *** )	Sagittal (3D)	4000	229	1400	90	5.0	1.5 × 1.5	10:13
64 mT	T2‐FLAIR ( *** )	Coronal (3D)	4000	214	1400	90	5.0	1.5 × 1.5	11:10

Abbreviations: 3D, 3‐dimensional; EPI, echo planar imaging; FSE, fast‐spin echo; MP2RAGE, magnetization‐prepared 2 rapid gradient echo; mT, millitesla; N/A, not applicable; PD, proton density; T, Tesla; T1WI, T1‐weighted images; T2‐FLAIR, T2w, T2‐weighted; T2‐weighted fluid attenuated inversion recovery.

Acquisition ⁽*⁾ only before, ⁽**⁾ before and after, ⁽***⁾ only after gadobutrol administration.

2.3. MRI Evaluations

Two neuroradiologists, “Rater 1” (R1) and “Rater 2” (R2), with a minimum of 8 years of experience each, were blinded to clinical/demographic information and HF‐MRI data. The assessments followed a three‐step process (Steps 1, 2, and 3), with a minimum 4‐week wash‐out period between each step. They performed evaluations of pULF‐MRI using the Osirix software (Version: 13, Bernex, Switzerland, https://www.osirix‐viewer.com/) [14]. In each step, they annotated the WML areas they identified, marking and documenting putative WML in specific brain regions, including periventricular, juxtacortical (“touching” or involving the cortex), infratentorial, and deep‐parenchymal (deep gray and deep white matter) areas. Additionally, they noted their assessment of DIS based on the 2017 revised McDonald Criteria for MS [3].

• In Step 1, the blinded raters assessed all pULF‐MRI images using axially reformatted T1w FSE and T2‐FLAIR. They could also incorporate additional multiplanar reformatted images into their assessments.

• In Step 2, first, the raters reassessed 10 selected cases and then were unblinded to 10 different cases for self‐training. Subsequently, they performed a blinded assessment of the 10 cases they had evaluated before the self‐training.

• In Step 3, the raters assessed a subset of 20 scans from the Step 1 cohort, selected because they had an additional T2‐FLAIR acquired in a second plane (sagittal or coronal), which had not been available in Step 1. There were no overlapping cases between Steps 2 and 3.

• After Step 1, a neurologist specializing in MS neuroimaging, with 5 years’ experience (SVO), selected cases for Steps 2 and 3. The neurologist then examined the markings generated by R1 and R2 in all steps, recording the counts of both regional and total true‐positive lesions as well as false‐positive areas. This evaluation was conducted after coregistering HF‐MRI and pULF‐MRI T2‐FLAIR images and involved assessing corresponding images from both scanners side‐by‐side, using ITK‐SNAP (Version: 3.8.0, Philadelphia, PA, http://www.itksnap.org/pmwiki/pmwiki.php) [15]. A graphical representation of the stepwise evaluation process is provided in Figure 1.

FIGURE 1.

(A) Flow chart of study participants. (B) Stepwise evaluation process. See text for details. n, number; SD, standard deviation; F, female; M, male (Created in BioRender.com).

2.4. Qualitative and Quantitative HF‐MRI White Matter Lesion Burden Estimation

We estimated the total WML volume with Classification using DErivatives‐based Features (C‐DEF), a locally developed machine‐learning algorithm [16] that employs T1w‐MP2RAGE, T2‐FLAIR, PD‐T2, and T2*‐EPI to segment WML. C‐DEF has demonstrated robust segmentation performance across diverse imaging datasets obtained from different scanners, a finding that has been further corroborated by visual assessments [16, 17, 18]. Furthermore, the qualitative assessment of WML burden at 3T (conducted by SVO) involved categorizing lesions into three groups: mild, moderate, and severe. The classification was based on counting the number of WML and visually evaluating the burden of confluent lesions when present. Definitions for WML burden categories were as follows: mild (fewer than 10 WML), moderate (10–20 WML and/or less than 25% confluent WML), and severe (more than 20 WML and/or more than 25% confluent WML). Validation was performed by comparing WML volumes among the three defined categories.

2.5. Contrast‐to‐Noise Ratio (CNR) and Raters’ Lesion Size Analyses

Using the raters' annotations as a guide in 12 selected scans with higher rates of true‐positive and false‐positive markings, regions of interest (ROIs) were manually delineated on the markings and adjacent parenchymal tissue on T2‐FLAIR images in ITK‐SNAP (by AAT and SVO). Spherical ROIs with an 8 mm radius (per each ROI) were drawn in four quadrants outside the tissue and within the coil coverage area to encompass “noise” areas at the lesion level. Values for true‐positive lesions and false‐positive areas in the periventricular (n = 22), juxtacortical (n = 19), and deep‐parenchymal (n = 23) regions were calculated using the CNR formula.

$$\begin{array}{ccc}& & \mathrm{CNR}=\left(\mathrm{signal}\mathrm{intensity}{\left[\mathrm{SI}\right]}_{\mathrm{ROI}}-\mathrm{S}{\mathrm{I}}_{\mathrm{adjacent}\mathrm{parenchyma}}\right)\hfill \\ & & /\mathrm{standard}\mathrm{deviation}{\left[\mathrm{SD}\right]}_{\mathrm{noise}}\hfill \end{array}$$

Pre‐training and post‐training markings from both raters, performed on 10 scans used in Step 2 evaluations, were manually segmented (by SVO) in the native pULF‐MRI space using the voxel‐intensity thresholding feature of 3D Slicer (https://www.slicer.org/). Confluent lesions were excluded. The volumes of true‐positive lesions and false‐positive markings were then compared.

2.6. Statistical Analyses

Distribution analyses were done using the Shapiro–Wilk and Kolmogorov–Smirnov tests. The Wilcoxon signed‐rank test was used to compare the counts of true‐positive lesions and false‐positive areas between raters, as well as to assess intrarater changes in Steps 2 and 3. Correlation analyses were performed using the Spearman ρ. Wilcoxon signed‐rank (paired), and Mann–Whitney U (unpaired) comparison tests were performed for categorical group analyses when two categories were compared. Standard p values from correlation and bicategorical comparison tests are reported. Tukey's test, applied subsequent to one‐way analysis of variance, was employed for pairwise comparisons across multiple groups, when applicable. Adjusted p values are reported. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value were calculated for each region, referencing HF‐MRI as the “gold standard.” Logistic regression models were employed to analyze true‐positive lesion volumes and false‐positive area volumes, generating receiver operating characteristic (ROC) curves and corresponding area under curve (AUC) values, from which optimal thresholds were extracted using the Youden J‐index.

3. Results

3.1. Demographic, Clinical, and HF‐MRI Characteristics

Fifty‐five participants (74% women), with a mean age of 50 (SD: 11), comprising various MS phenotypes, underwent pULF‐MRI scans immediately after clinical HF‐MRI, with a mean post‐gadolinium delay of 58 (SD: 21) min. Of the 55 cases, solely based on 3T brain scans, 52 (94%) met the 2017 DIS criteria, indicating the presence of at least one periventricular along with one juxtacortical and/or infratentorial lesion. Only three cases (6%) required a spinal cord lesion to meet DIS. In the qualitative assessment, 23 cases had mild (41%), 22 moderate (40%), and 10 (19%) severe WML burden. Median WML volume was 1.8 mL with interquartile range (IQR) 0.8–3.5 mL. No cases had gadolinium‐enhancing WML in their HF‐MRI. Detailed characteristics of the study cohort are provided in Table 2.

TABLE 2.

Study cohort characteristics.

Age, mean ± SD (years)	50 ± 11
Sex (F/M) (n, %)	41/14, 74%/26%
Clinical phenotype
Clinically isolated syndrome (n, %)	2, 3%
Relapsing‐remitting MS (n, %)	43, 78%
Secondary progressive MS (n, %)	7, 12%
Primary progressive MS (n, %)	3, 5%
EDSS (median, range)	1.5, 0–6
WML burden
Mild (n, %)	23, 41%
Moderate (n, %)	22, 40%
Severe (n, %)	10, 19%
WML volume, median, IQR (mL)	1.8, 0.8–3.5

Note: Demographic, clinical, and high‐field MRI characteristics of the study cohort.

Abbreviations: EDSS, expanded disability status score; F, female; IQR, interquartile range; M, male; MS, multiple sclerosis; n, number; SD, standard deviation; WML, white matter lesion.

3.2. Distinct Rater Approaches and High Periventricular Lesion Sensitivity in Step 1

R1 marked 454 foci and R2 638 foci, with median counts of 7 (IQR: 3–23) for R1 and 9 (IQR: 4–16) for R2 (p = 0.003). The two raters identified a similar number of true‐positive lesions in total (n/median/IQR: R1, 419/7/2–12; R2, 509/7/2–14) (p = 0.12). R2 exhibited a higher median number of false‐positive areas (n = 129) compared to R1 (n = 35) (p < 0.001). Both raters had good case‐level sensitivity for detecting at least one WML (90% for both), especially in the periventricular areas (75% and 95% for R1 and R2, respectively). R1 detected DIS within the brain field‐of‐view in 49% and R2 in 51% sensitivity based solely on true‐positive lesions. The topographical distribution of true‐positive lesions and false‐positive areas marked by the raters and their comparisons is summarized in Table 3. Figure 2 displays illustrative cases featuring examples of true‐positive lesions and false‐positive areas across varying lesion burdens.

TABLE 3.

Regional variation in the marking of true‐positive lesions and false‐positive areas.

Region	Rater 1 (n, median, IQR)	Rater 2 (n, median, IQR)
Total
True‐positive lesions	419, 7, 2–12	509, 7, 2–14
False‐positive areas	35, 7, 3–13**	129, 9, 4–16**
Sensitivity	90%	89%
Specificity	N/A	N/A
PPV	93%	83%
NPV	NA	NA
Periventricular
True‐positive lesions	261, 4, 0–9	329, 5, 2–9
False‐positive areas	8, 0, 0–0	8, 0, 0–0
Sensitivity	75%	95%
Specificity	N/A	N/A
PPV	100%	90%
NPV	N/A	N/A
Juxtacortical
True‐positive lesions	55, 0, 0–1*	71, 0, 0–2*
False‐positive areas	10, 0, 0–0**	69, 1, 0–2**
Sensitivity	46%	76%
Specificity	50%	16%
PPV	80%	52%
NPV	17%	36%
Infratentorial
True‐positive lesions	3, 0, 0–0	4, 0, 0–0
False‐positive areas	4, 0, 0–0	12, 0, 0–0
Deep parenchymal
True‐positive lesions	100, 1, 0–3	105, 1, 0–3
False‐positive areas	13, 0, 0–0**	40, 0, 0–1**

Note: Rater 2 had more true‐positive lesion detections, albeit at the expense of more false‐positive area markings, particularly notable in the juxtacortical and deep‐parenchymal (subcortical white matter and deep gray matter) regions.

Abbreviations: IQR, interquartile range; n, [total] number; N/A, not applicable; NPV, negative predictive value; PPV, positive predictive value

Wilcoxon signed rank test, *: p ≤ 0.05, **: p ≤ 0.001.

FIGURE 2.

Representative cases comparing portable ultra‐low‐field MRI (at 64 millitesla) and high‐field MRI (at 3 tesla), illustrating true‐positive lesions and false‐positive areas. (A) A 47‐year‐old man with relapsing‐remitting MS and mild lesion burden on MRI. Both raters identified periventricular and juxtacortical true positive lesions (yellow arrows), accompanied by a false‐positive area in a juxtacortical region (red arrowhead). (B) A 67‐year‐old woman with relapsing‐remitting MS and moderate lesion burden. Both raters marked 2 periventricular true‐positive lesions (yellow arrows) with additional deep parenchymal lesions (not shown). (C) A 49‐year‐old woman with secondary progressive MS and severe lesion burden. Both raters detected true‐positive lesions in periventricular, juxtacortical, and deep parenchymal areas (yellow arrows), with a false‐positive area in the juxtacortical region marked by both raters (red arrowhead).

WML volume assessed at 3T exhibited correlations with number of total, periventricular, juxtacortical, and deep‐parenchymal true‐positive lesions as identified by both R1 (total: ρ = 0.70, p < 0.0001, periventricular: ρ = 0.63, p < 0.001, juxtacortical: ρ = 0.60, p < 0.0001, deep‐parenchymal: ρ = 0.64, p < 0.0001, infratentorial: ρ = 0.32, p = 0.01) and R2 (total: ρ = 0.78, p < 0.0001, periventricular: ρ = 0.71, p < 0.0001, juxtacortical: ρ = 0.68, p <0.0001, deep‐parenchymal: ρ = 0.65, p < 0.0001, infratentorial: ρ = 0.34, p = 0.01). Note that no correlations were observed for false‐positive areas marked by either rater.

Median (IQR) WML volumes were 0.7 (0.4–1.8) mL for mild, 2.2 (1.4–3.5) mL for moderate, and 6.0 (4.9–8.8) mL for severe WML groups (mild vs. moderate, p = 0.02, moderate vs. severe, p < 0.0001, mild vs. severe, p < 0.0001). Not surprisingly, for both raters, the total number of true‐positive lesions was higher in moderate and severe WML burden cases than those with mild WML burden (R1, mild vs. moderate: p = 0.002, moderate vs. severe: p < 0.0001, mild vs. severe: p<0.0001; R2, mild vs. moderate: p = 0.0008, moderate vs. severe: p < 0.0001, mild vs. severe: p < 0.0001). The number of false‐positive areas did not show any difference in different lesion burden groups for either rater. Correlations between WML volume and total true‐positive/false‐positive markings, grouped comparisons per WML burden for each rater, as well as region‐specific correlations and comparisons of WML volume across different WML burden categories, are shown in Figure 3.

FIGURE 3.

Relationship between white matter lesion (WML) burden and true‐positive lesions and false‐positive areas marked by each rater. (A) True‐positive lesion detection exhibited positive correlations in total and supratentorial topographical distributions with WML volume for both raters. These correlations were absent for false‐positive areas marked by either rater. (B) Grouped by WML burden based on high‐field qualitative judgments, there were a higher number of true‐positive lesion detections, especially in moderate WML burden compared to mild WML burden. (C) Positive correlations are observed among true positive lesions identified by both raters in periventricular, juxtacortical, deep‐parenchymal, and infratentorial regions. Notably, such correlations are absent for false‐positive areas marked by either rater. (D) Grouped differences in white matter lesion volume measurements across mild, moderate, and severe white matter lesion burden categories. One‐way analysis of variance, adjusted p values with Tukey's multiple comparisons test; ns: not significant, *: p ≤ 0.05,** p ≤ 0.01, ****p ≤ 0.001, ****: p ≤ 0.0001.

3.3. Self‐Training Harmonized Approaches Among Raters in Step 2

In a sample of 10 cases, both raters identified lesions in various topographical areas, including periventricular, juxtacortical, deep‐parenchymal, and infratentorial. After self‐training, neither R1 nor R2 exhibited a significant change in the overall number of true‐positive lesions (R1: p = 0.37; R2: 0.10), although the number of true‐positive periventricular and juxtacortical lesions decreased for R2 post‐training (p = 0.005 and p = 0.007, respectively). While R1's false‐positive areas showed no significant change (p = 0.09), R2's false‐positive areas decreased from 37 to 5 after self‐training (p = 0.01). Both raters demonstrated 100% case‐level sensitivity and positive PPV for WML detection both before and after training, with variations observed in sensitivity and PPV for periventricular and juxtacortical areas. For DIS, R1 showed a pre‐training sensitivity of 60% and a PPV of 100%, compared to a 50% sensitivity and 100% PPV post‐training. R2 exhibited a pre‐training sensitivity and PPV of 80%, compared to 50% sensitivity and 100% PPV post‐training. Table 4 displays comprehensive results and detailed findings.

TABLE 4.

Comparison of lesion markings before and after self‐training.

	Rater 1			Rater 2			Rater 1 vs. Rater 2
Total markings (n, median, IQR)	Pre‐training	Post‐training	p value	Pre‐training	Post‐training	p value	Pre‐training	Post‐training
	140 (13, 6–8)	125 (9, 6–19)	0.10	158 (15, 11–19)	102 (8, 6–15)	0.007	0.20	0.34
	True‐positive lesions
Total (n, median, IQR)	125 (14, 4–18)	120 (9, 5–19)	0.37	121 (13, 8–14)	97 (8, 6–15)	0.10	0.67	0.37
Periventricular (n, median, IQR)	74 (7, 4–11)	71 (6, 4–11)	0.65	97 (10, 7–12)	69 (7, 4–10)	0.005	0.003**	0.95
Juxtacortical (n, median, IQR)	20 (1, 0–3)	11 (0, 0–1)	0.21	23 (1, 0–3)	4 (0, 0–1)	0.007	0.37	0.25
Deep parenchymal (n, median, IQR)	31 (3, 0–4)	38 (3, 1–6)	0.24	32 (3, 1–4)	24 (1, 0–3)	0.17	>0.99	0.10
	False‐positive Areas
Total (n, median, IQR)	15 (1, 0–3)	5 (0, 0–1)	0.09	37 (4, 1–6)	5 (0, 0–1)	0.01	0.04*	>0.99
Periventricular (n, median, IQR)	3 (0, 0–0)	1 (0, 0–0)	0.75	5 (0, 0–1)	2 (0, 0–0)	0.53	0.75	>0.99
Juxtacortical (n, median, IQR)	4 (0, 0–1)	0 (0, 0–0)	0.12	16 (1, 0–3)	1 (0,0–0)	0.01	0.03*	>0.99
Deep parenchymal (n, median, IQR)	7 (0, 0–1)	3 (0, 0–0)	0.50	15 (1, 0–2)	2 (0, 0–0)	0.09	0.37	>0.99
	Sensitivity
Whole brain	100%	100%	–	100%	100%	–
Periventricular	90%	90%	–	100%	90%	–
Juxtacortical	77%	44%	–	100%	30%	–
Dissemination in space	60%	50%	–	80%	50%	–
	Positive predictive value
Whole brain	100%	100%	–	100%	100%	–
Periventricular	100%	100%	–	100%	100%	–
Juxtacortical	33%	100%	–	80%	100%	–
Dissemination in space	100%	100%	–	100%	100%	–

Note: Significant disparities between Rater 1 and Rater 2 were observed in total false‐positive areas, periventricular true‐positive lesions, and juxtacortical false‐positive areas before self‐training in Step 1. Notably, after self‐training, no differences were observed in the true‐positive lesion and false‐positive area markings, either overall or across topographical distributions, between the two raters.

Abbreviations: IQR, interquartile range; n, [total] number.

*: p ≤ 0.05, **: p ≤ 0.01, Wilcoxon signed‐rank paired comparison tests.

3.4. Both Raters Marked Fewer False‐Positive Areas and True‐Positive Lesions With Multiplanar Evaluations in Step 3

In 20 cases with an additional T2‐FLAIR plane, R1 had a decrease in true‐positive lesions (177 to 159, p = 0.07) between Steps 1 and 3, while total false‐positive areas showed no significant difference (13 to 5, p = 0.17). R1's true‐positive and false‐positive markings in periventricular, juxtacortical, and deep‐parenchymal areas varied across steps without consistent significant differences in periventricular and juxtacortical regions. Conversely, R2 exhibited a substantial decrease in true‐positive lesions (214 to 117, p <0.0001) between Steps 1 and 3, with an important reduction in total false‐positive areas (46 to 4, p = 0.002). R2 displayed reductions in true‐positive and false‐positive markings across different areas with reductions except in periventricular false‐positive areas. Figure 4 presents examples of multiplanar image acquisitions and illustrates their influence on lesion ratings. Table 5 details the number of true‐positive lesions and false‐positive areas marked by the raters across the steps and topographical differences.

FIGURE 4.

Impact of additional plane T2‐FLAIR acquisitions. (A) Representative images from a 49‐year‐old woman with relapsing‐remitting MS highlighting white matter lesions marked by both raters in axial and sagittal slices (yellow arrows). (B) A 45‐year‐old woman with relapsing‐remitting MS, emphasizing a juxtacortical lesion magnified on portable ultra‐low‐field MRI (at 64 millitesla) slices, particularly visible on coronal T2‐FLAIR acquisition (indicated by yellow arrows) compared to coronal reformat of the axial T2‐FLAIR acquisition. (C) Paired analyses from 20 scans show a decrease in both raters' total markings and true‐positive lesion numbers, with only Rater 2 showing a reduction in false‐positive area numbers in Step 3 compared to Step 1 ratings. T2‐FLAIR, T2‐weighted fluid‐attenuated inversion recovery. Wilcoxon signed‐rank paired comparison tests; ns: not significant, *: p ≤ 0.05, **: p ≤ 0.001, ***: p ≤ 0.0001.

TABLE 5.

Comparison of Rater 1 and Rater 2 lesion markings in Step 1 and Step 3.

	Rater 1			Rater 2
Total markings (n, median, IQR)	One plane	Two planes	p value	One plane	Two planes	p value
	190 (7, 2–14)	164 (5.5, 1–8)	0.007**	260 (11, 3–17)	121 (5, 0–9)	<0.0001***
True‐positive lesions
Total (n, median, IQR)	177 (6, 1–13)	159 (5.5, 1–8)	0.07	214 (8, 2–14)	117 (4, 0–9)	<0.0001***
Periventricular (n, median, IQR)	103 (3.5,0–8)	100 (3, 1–6)	0.75	129 (5, 1–9)	94 (3.5,0–7)	0.003**
Juxtacortical (n, median, IQR)	22 (0.5, 0–1)	20 (0, 0–1)	0.80	33 (0.5, 0–2)	14 (0, 0–0)	0.003**
Deep parenchymal (n, median, IQR)	50 (1.5, 0–3)	38 (0.5, 0–2)	0.04*	52 (2, 0–3)	9 (0, 0–0)	0.002**
False‐positive areas
Total (n, median, IQR)	13 (0, 0–1)	5 (0, 0–0.7)	0.17	46 (1.5, 0–4)	4 (0, 0–0)	0.002**
Periventricular (n, median, IQR)	3 (0, 0–0)	0 (0, 0–0)	0.75	7 (0, 0–0)	4 (0, 0–0)	0.61
Juxtacortical (n, median, IQR)	4 (0, 0–0)	2 (0, 0–0)	0.75	25 (0.5, 0–2)	0 (0, 0–0)	0.002**
Deep parenchymal (n, median, IQR)	5 (0, 0–0)	3 (0, 0–0)	0.75	14 (0, 0–1)	0 (0, 0–0)	0.007**

Abbreviations: IQR: interquartile range, n: [total] number.

*: p ≤ 0.05, **: p ≤ 0.01, ***: p ≤ 0.001 Wilcoxon signed‐rank paired comparison tests.

3.5. CNR Is Similar for True‐Positive Lesions and False‐Positive Areas in Juxtacortical Regions: False‐Positive Areas Are Smaller Than True‐Positive Lesions Across All Regions

Across representative juxtacortical, deep‐parenchymal, and periventricular regions, the CNR of 9 true‐positive lesions and 10 false‐positive areas, 10 true‐positive lesions and 13 false‐positive areas, and 10 true‐positive lesions and 12 false‐positive areas were calculated, respectively. Mean CNR (SD) of true‐positive lesions in juxtacortical, deep‐parenchymal, and periventricular regions were 30 (8), 33 (20), and 47 (26), respectively. CNR was not different in true‐positive lesions across different regions (p = 0.18). On the contrary, false‐positive areas in juxtacortical regions had higher CNR than false‐positive areas in deep‐parenchymal and periventricular regions (p = 0.02, p = 0.01, respectively). CNR of true‐positive lesions and false‐positive areas in the juxtacortical region were not different (p = 0.71). In deep‐parenchymal and periventricular regions, however, true‐positive lesions had higher CNR than the false‐positive areas (p = 0.01, p = 0.004, respectively).

A total of 87 true‐positive lesions (10 juxtacortical, 31 deep‐parenchymal, and 46 periventricular) and 37 false‐positive areas (17 juxtacortical, 16 deep‐parenchymal, and 4 periventricular) were segmented on the native‐space pULF‐MRI T2‐FLAIR. True‐positive lesions were larger than false‐positive areas in juxtacortical regions (mean volume ± SD: 0.19 ± 0.12 mL and 0.09 ± 0.03 mL, respectively, p = 0.02). True‐positive lesions were also larger than false‐positive areas in deep‐parenchymal regions (mean volume ± SD: 0.22 ± 0.18 mL and 0.09 ± 0.05 mL, respectively, p = 0.0009) and periventricular regions (mean volume ± SD: 0.41 ± 0.31 mL and 0.12 ± 0.08 mL, respectively, p = 0.03). With a similar approach, a total of 72 true‐positive lesions and 8 false‐positive areas marked in Step 2 after self‐training were segmented. True‐positive lesions were larger than false‐positive areas in general (mean volume ± SD: 0.39 ± 0.35 mL vs. 0.09 ± 0.06 mL, respectively, p = 0.0003). There were no differences in the sizes of true‐positive lesions (p = 0.19) and false‐positive areas (p = 0.47) before and after self‐training. Analyses of ROC curves revealed that in pre‐training settings, areas exceeding 0.80 mL achieved optimal accuracy, with a lesion‐based sensitivity of 56% and specificity of 97%. In contrast, post‐training evaluations indicated that areas larger than 0.87 mL demonstrated improved accuracy, with a sensitivity of 75% and specificity of 87%. Figure 5 presents CNR comparison analyses and a representative image of the juxtacortical true‐positive lesion and false‐positive area in the same case with CNR comparison. The figure also illustrates differences in size between true‐positive lesions and false‐positive areas marked during the pre‐training phase of the Step 2 cohort and ROC curves in post‐training markings.

FIGURE 5.

Comparison of contrast‐to‐noise ratio (CNR) between true‐positive lesions and false‐positive areas. Panel A shows that CNR is not significantly different between true‐positive lesions and false‐positive areas in juxtacortical regions. However, in deep‐parenchymal and periventricular regions, CNR is higher in true‐positive lesions, suggesting that true‐positive lesions in juxtacortical regions may be less distinguishable to raters than deep‐parenchymal and periventricular regions during visual assessment. Panel B illustrates a representative case of a 35‐year‐old man with MS, showing a juxtacortical true‐positive lesion (top row) and a false‐positive area (bottom row) on 64 millitesla T2‐FLAIR images, confirmed by coupled 3 tesla T2‐FLAIR images. Signal intensities of marked hyperintense areas (green and red circles) and their surrounding tissue (yellow circles) were used for CNR calculation. Both areas have similar CNR values and appearances visually. Panel C shows size differences between true‐positive lesions and false‐positive areas in various regions. Panel D illustrates the ROC curve analysis of the volume measurements obtained from the post‐self‐training markings in Step 2. AUC: area under curve, FLAIR: fluid‐attenuated inversion recovery; ROC: receiver operating characteristic, SE: standard error. Mann Whitney U unpaired comparison tests; ns: not significant, *: p ≤ 0.05, **: p ≤ 0.01, ***: p ≤ 0.001, ****: p ≤ 0.0001.

4. Discussion

In this study, we assessed the performance of pULF‐MRI in an HF‐MRI‐blinded setting in established MS, mimicking to some extent a clinical image evaluation scenario. We then investigated the impact of self‐training on rater performance in identifying true‐positive lesions and false‐positive areas. Additionally, we explored the potential additional benefit of an extra T2‐FLAIR plane for accurate WML identification.

Our findings yield several key insights. Initially, even without self‐training, pULF‐MRI consistently exhibited high sensitivity and PPV in detecting at least one WML, particularly in the periventricular region, despite the raters using different evaluation approaches. Specifically, one rater (R1) adopted a more conservative strategy with fewer true‐positive lesions, whereas the other (R2) used a less conservative approach, detecting more true‐positive lesions at the cost of more false‐positive areas, particularly in the juxtacortical regions. The count of true‐positive lesions from both raters was correlated with the WML burden assessed both qualitatively and quantitatively. In contrast, false‐positive areas did not exhibit such a correlation. In addition, false‐positive areas were smaller across all regions, and optimal accuracy may be reached by restricting calls to candidate lesions larger than ∼0.8 mL (approximating a sphere of diameter ∼10 mm). While our previous study demonstrated that a 4‐mm diameter cut‐off achieved 100% case‐level sensitivity [13], its reliance on coupled HF‐MRI does not clarify how many false‐positive areas might also exceed 4 mm in size. These findings suggest that false‐positive areas are not influenced by WML burden but are instead associated with the current implementation of pULF‐MRI technology. In particular, the similarity of CNR in true‐positive lesions and false‐positive areas in the juxtacortical region, which has a high false positivity rate, may be associated with technical issues such as noise, distortion, and artifacts like cerebrospinal fluid pulsation, which may mimic WML appearances. Our findings that juxtacortical areas have the highest rate of false‐positive areas support the previous conclusion, as pULF‐MRI images have a poorer delineation of cerebrospinal fluid and brain parenchyma due to reduced signal‐to‐noise ratio [19]. Due to our clinical care workflow, we were able to perform research pULF‐MRI after clinical HF‐MRI for those with gadolinium injection, allowing us to leverage same‐day acquisition and ensuring stable findings for comparison. As a trade‐off, post‐gadolinium pULF‐MRI scans were conducted with an approximately post‐gadolinium 1‐hour delay. Since this study focuses on hyperintense WML on T2‐FLAIR images, we believe the potential effect of gadolinium on our findings is negligible.

In the second step, when raters compared their own markings with HF‐MRI data, a harmonized and more conservative approach emerged, particularly for the originally less conservative rater (R2), resulting in fewer false‐positive areas at the expense of a decrement in true‐positive areas. Despite this shift, in the Step 2 cohort, raters maintained high sensitivity and PPV for periventricular lesions. Notably, both raters achieved an overall 50% success rate in identifying DIS based solely on true‐positive lesions in both Steps 1 and 2. Finally, in the third step, in which an additional T2‐FLAIR plane was introduced, raters marked fewer true‐positive lesions and false‐positive areas. This reduction may also be attributed to the self‐training effect, which synchronized the conservative approach among the raters.

The high sensitivity of pULF‐MRI to periventricular lesions presents several potential advantages. It can serve as an effective screening tool, particularly given that the periventricular area is the most common location among the regions where the presence of WML is required for DIS [20]. Importantly, frequent delays in the initial diagnosis, which are linked to a worse prognosis [21], are often associated with limited accessibility to HF‐MRI [22], which has a high cost, infrastructure requirements, and unequal distribution worldwide [23]. The utilization of pULF‐MRI offers a cost‐effective option for screening and triaging patients, potentially expediting access to HF‐MRI and contributing to shorter diagnosis times for individuals with MS [24]. Early detection of MS can also facilitate the prompt initiation of disease‐modifying therapies, known to be effective in delaying progression [25]. On a wider scale, early diagnosis may contribute to lowering the economic burden associated with disease severity [26].

Challenges in distinguishing true‐positive lesions and false‐positive areas may arise from reduced image quality, characterized by lower signal and contrast‐to‐noise ratios due to the lower magnetic field as well as the absence of physical radiofrequency shielding [27]. A recent Fourier transformation algorithm applied to multiple orthogonal images to create a super‐resolution image in pULF‐MRI has shown promising results in enhancing WML and surrounding tissue contrast and sharpness, both qualitatively and quantitatively [28]. Further exploration is needed to determine whether this approach or other computational methods aiming to improve image quality [19, 29] can increase the detection of true‐positive lesions while reducing false‐positive areas.

Our cohort lacks a control group, and as a result, the specificity of pULF for MS in the diagnostic setting— not the aim of the current study, which was designed to compare scans in established MS—remains unknown. The false‐negative rates are also undetermined because the lesion markings were manually performed, and the comparison between pULF‐ and HF‐MRI involved manual classification of these markings as true‐positive or false‐positive.

In conclusion, we find that pULF‐MRI is sensitive to periventricular WML in MS and that self‐training leads to the adoption of a more conservative approach in the evaluation of pULF‐MRI as applied to MS, enhancing vigilance for potential false‐positive areas without compromising case‐level sensitivity for WML, especially in periventricular areas. This is particularly significant as MRI accessibility contributes to delayed MS diagnoses, and such accessibility varies across different regions globally.

Disclosure

J.M.S. has received support from sponsored research agreements with Hyperfine, Inc. D.S.R. has received research funding from Abata and Sanofi, unrelated to the current study. S.V.O., G.N., K.D.K., A.A.T., C.A.D., and M.I.G. have nothing to disclose. This study was performed under a Research Collaboration Agreement with Hyperfine, Inc., which did not involve the transfer of funds. Statistics were performed by S.V.O. and supervised by D.S.R., who is an adjunct professor of biostatistics at Johns Hopkins University. Images from 15 of 55 participants in this cohort were used in Arnold et al. [13]. The research questions, methodology, and analyzed imaging are different in the current paper, and 40 additional participants have been added.

Conflicts of Interest

The authors declare no conflicts of interest.