Automated MRI Classification in Progressive Supranuclear Palsy: A Large International Cohort Study

Abstract

Background:

The Magnetic Resonance Parkinsonism Index is listed as one of the most reliable imaging morphometric markers for diagnosis of progressive supranuclear palsy (PSP). However, the use of this index in diagnostic workup has been limited until now by the low generalizability of published results because of small monocentric patient cohorts, the lack of data validation in independent patient series, and manual measurements used for index calculation. The objectives of this study were to investigate the generalizability of Magnetic Resonance Parkinsonism Index performance validating previously established cutoff values in a large international cohort of PSP patients subclassified into PSP–Richardson’s syndrome and PSP-parkinsonism and to standardize the use of the automated Magnetic Resonance Parkinsonism Index by providing a web-based platform to obtain homogenous measures around the world.

Methods:

In a retrospective international multicenter study, a total of 173 PSP patients and 483 non-PSP participants were enrolled. A web-based platform (https://mrpi.unicz.it) was used to calculate automated Magnetic Resonance Parkinsonism Index values.

Results:

Magnetic Resonance Parkinsonism Index values showed optimal performance in differentiating PSP–Richardson’s syndrome and PSP-parkinsonism patients from non-PSP participants (93.6% and 86.5% of accuracy, respectively). The Magnetic Resonance Parkinsonism Index was also able to differentiate PSP–Richardson’s syndrome and PSP-parkinsonism patients in an early stage of the disease from non-PSP participants (90.1% and 85.9%, respectively). The web-based platform provided the automated Magnetic Resonance Parkinsonism Index calculation in 94% of cases.

Conclusions:

Our study provides the first evidence on the generalizability of automated Magnetic Resonance Parkinsonism Index measures in a large international cohort of PSP–Richardson’s syndrome and PSP-parkinsonism patients. The web-based platform enables widespread applicability of the automated Magnetic Resonance Parkinsonism Index to different clinical and research settings.

There is an urgent need to have reliable diagnostic neuroimaging biomarkers for early identification of atypical parkinsonisms.¹ In the last few years, several studies have provided important evidence for the usefulness of the Magnetic Resonance Parkinsonism Index (MRPI) in supporting the clinical diagnosis of patients with progressive supranuclear palsy (PSP).^2–11 Since its first description in 2008 as a highly accurate measure in distinguishing patients with PSP from those with Parkinson’s disease (PD) or multiple system atrophy (MSA),² the MRPI is now recognized as one of the most reliable morphometric biomarkers for identifying features of PSP–Richardson’s syndrome (PSP-RS), supporting clinical diagnoses in both the early and the late stages of the disease.¹² Indeed, in the past years several studies have found high performance of the MRPI in discriminating PSP from probable and possible PD and from the Guadeloupean variant of parkinsonism.^3,13 Moreover, MRPI showed high accuracy in predicting clinical evolution of unclassifiable parkinsonisms in PSP, even in the stages of the disease when clinical features such as isolated postural instability with falls or isolated slowing of vertical saccades did not allow making a diagnosis of PSP using established consensus criteria.¹⁴ The MRPI was more powerful than clinical features in predicting the appearance of vertical supranuclear gaze palsy in PSP with predominant parkinsonism (PSP-P) on an individual basis.¹⁵

Despite good performance overall, most studies investigating the usefulness of the MRPI in supporting the clinical diagnosis of PSP have been conducted on small patient cohorts, and MRPI values were obtained using a manual measurement process.^2–5,8 As such, it is hard to compare MRPI values across sites and among different groups of subjects and then to generalize the results obtained. Recently, we proposed a fully automated method for the segmentation and measurement of brain regions involved in the calculation of MRPI.^6,7 This approach allows more widespread use of the MRPI in clinical practice, overcoming both the time-consuming aspects as well as the operator dependence associated with manual measuring. The automated procedure was evaluated on a large Italian cohort of PSP, PD, and controls showing the accuracy of automated MRPI values above 90% in differentiating PSP from PD.⁶ However, in this multicenter study PSP patients were not classified using the recent criteria for PSP,¹⁶ and thus the performance of the MRPI in distinguishing different PSP phenotypes from PD was not considered. Moreover, recruitment was limited to Italian movement disorders centers, reducing the diversity of a sample that would exist in measurements from different centers across Europe, the United States, and Canada.

The present study was conducted to explore for the first time the performance of the automated MRPI in an international cohort of PSP and non-PSP participants (173 and 483 participants, respectively), one of the largest cohorts to investigate the performance of an automated MR morphometric marker in an atypical parkinsonian disorder. Patients with PSP were also classified as PSP-RS and PSP-P patients according to newly published MDS-PSP criteria.¹⁶ Moreover, we developed a web-based platform (https://mrpi.unicz.it) to calculate MRPI values using a web browser for data upload and results download. By using the web-based application, authorized users could easily upload their medical images anytime and anywhere, and automated MRPI calculation was performed via the internet on a remote server.

Methods

Study Design and Participants

This study involved 283 patients affected by idiopathic PD, 173 patients affected by PSP, 52 patients with MSA, and 148 healthy controls. All participants were recruited between 2010 and 2017. Clinical diagnoses for all patients were performed by movement disorder specialists using international diagnostic criteria.^16–18 Patients enrolled before 2017 were reclassified according to the recent diagnostic criteria for PSP.¹⁶ In particular, patients with PSP-RS were characterized by falls within the first 3 years of the disease and vertical ocular dysfunction (vertical supranuclear gaze palsy or slowness of vertical saccades), whereas patients with PSP-P were characterized by similar ocular abnormalities as in PSP-RS, and a levodopa-responsive or levodopa-resistant parkinsonism, in the absence of falls within the first 3 years.

MR images were uploaded on the web-based platform by 6 international research centers (Table S1). In each center PSP patients were classified as PSP-RS and PSP-P using new consensus criteria for the clinical diagnosis of PSP.¹⁶ Other PSP variants were not included given their relative rarity and overlap with other disorders.

For each participant, a complete medical history was also available. Neurological examination and clinical assessment using the Movement Disorder Society–sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) and Hoehn and Yahr (HY) rating scale were performed in all patients.^19,20

All participants gave written informed consent according to the Declaration of Helsinki for the use of their medical records for research purposes. The study was approved by the local institutional review board and ethics committee.

MRI Imaging Protocol

The MRI protocol included a 3-dimensional T1-weighted MR sequence used for morphological calculations. MRI acquisitions were performed using 1.5T or 3T scanners from different manufactures (Philips, General Electrics and Siemens). See Supplementary Material for further details.

Web-Based Framework for the Automated MRPI Calculation

MRPI calculation service was provided through a web platform composed of a website on the front end and an ensemble/set/cluster of high-performance servers running the MRPI algorithm core on the back end.⁶ Before the automated MRPI calculation, MR images were rigidly registered (using a 6-parameter affine registration) to standard space (MNI) based on the mutual information metric using FSL software (FMRIB Software Library). Then the volume of images was corrected for fluctuations in intensity using the FreeSurfer software package.²¹

The service is available worldwide on request of registration. Once approved and logged in on the MRPI website, each user has a dedicated page to run the MRPI calculation. MRI data must be provided already anonymized in a zip folder containing all DICOM files related to a single-subject T1-weighted acquisition. For each sequence, a new MRPI task is generated and scheduled for execution. To maintain scalability and serve multiple requests concurrently, the platform automatically selects a server based on the current load information. Users can monitor their tasks, access all results, and download a summary report per task on their personal site area.

Statistical Analysis

The difference in sex distribution between non-PSP participants (controls and PD and MSA patients) and PSP patients was evaluated with the chi-square test. The differences in age at examination, age at onset, disease duration, H-Y score, and MDS-UPDRS score between non-PSP participants (PD and MSA patients) and PSP patients were assessed by pairwise Wilcoxon rank sum tests. Assessment for differences in midbrain area, pons area, middle cerebellar peduncle (MCP) width, superior cerebellar peduncle (SCP) width, and MRPI between non-PSP subjects (controls amd PD and MSA patients) and PSP patients was conducted using a similar approach. Resulting P values were corrected according to the Bonferroni method.

Sensitivity, specificity, accuracy, and positive and negative predictive values (PPV and NPV, respectively) of automated MRPI values for differentiating PSP from non-PSP were determined using the optimal cutoff values reported in previous studies.^6,9 In this way, we could validate the generalizability of previous suggested MRPI cutoff values in a larger independent cohort of PSP and non-PSP participants. In particular, a cutoff >13.42 was used to distinguishing all PSP patients from non-PSP patients.⁶ An MRPI ≥13.88 and ≥ 12.38 was used to differentiate non-PSP participant from PSP-RS and PSP-P participants, respectively.⁹ To further stress the usefulness of the MRPI in the classification of PSP in an early stage of disease, classification analyses were also performed considering only patients with a disease duration fewer than 4 years.

Of note, MRPI performance in distinguishing PSP patients from non-PSP patients was first estimated in cases in which the automated method successfully completed the segmentation of brain structures involved in the MRPI calculation. Next,we explored the accuracy of the automated MRPI calculation approach considering algorithm failures in the calculation of MRPI performance.

All analyses were performed considering all MRI data as a single sample without considering the site as a factor in our analyses. Indeed, in a previous study no differences were found in automated MRPI values evaluated on MR images acquired with different scanner manufacturers (GE, Philips, Siemens) or magnetic fields (1.5T and 3T).⁶

Results

Demographic and clinical data of patients and controls uploaded and then measured using the web-based platform are shown in Table 1. The online platform provided the automated MRI measures in 94% of cases (616 of 656 total uploaded MRI DICOM volumes). The algorithm failed in only 40 of the cases (see Table S1). Most of these failures were principally because of individual variation in anatomical features or peculiarities of the MRI acquisition process such as motion artefacts that did not allow correctly identifying the anatomical landmarks used for segmenting the pons and midbrain areas.

TABLE 1.

Demographic and clinical data of non-PSP participants and PSP patients

	non-PSP			PSP
	HC	PD	MSA	All PSP	PSP-RS	PSP-P
Participants	139	270	43	164	108	56
Sex (men/women)	65/74	168/102	24/19	96/68	55/53	41/15
Age at examination (years)a	65.1 ± 10.7	65.2 ± 8.9	65.2 ± 8.7	70.8 ± 6.1	70.2 ± 5.9	71.8 ± 6.3
Disease duration (years)a	NA	6.4 ± 4.3	3.5 ± 2.9	4.5 ± 2.9	3.7 ± 2.1	6.1 ± 3.5
Age at onset (year)a	NA	58.8 ± 9.1	61.7 ± 8.3	66.2 ± 6.0	66.5 ± 5.9	65.7 ± 6.2
H-Y scorea	NA	2.3 ± 0.7	3.3 ± 0.7	3.4 ± 0.7	3.4 ± 0.6	3.3 ± 0.8
MDS-UPDRS scorea	NA	29.4 ± 12.2	56.1 ± 15.9	43.5 ± 12.6	43.1 ± 12.7	44.2 ± 12.5

PSP-P, progressive supranuclear palsy–parkinsonism; PSP-RS, progressive supranuclear palsy–Richardson’s syndrome; PD, Parkinson’s disease; MSA, multiple system atrophy; HC, healthy controls; MDS-UPDRS, Movement Disorder Society–sponsored revision of the Unified Parkinson’s Disease Rating Scale; H-Y, Hoehn-Yahr; NA, not applicable.

^aP < 0.001 (non-PSP vs PSP) Mann-Whitney test.

Hence, 616 participants (270 PD patients, 164 PSP patients, 43 MSA patients, and 139 healthy controls) were included in the following analyses. Automated MRI measures for patients and controls are presented in Table 2. Automated measurements for the midbrain area and SCP width were smaller in patients with PSP than in non-PSP participants. In the PSP group, smaller values for the midbrain area and SCP width were observed in patients with PSP-RS compared with those with PSP-P. In the non-PSP group, MSA patients showed a decrease in the pons area and MCP width compared with PD patients and controls. By contrast, MSA group showed values of the MCP width similar to those observed in PSP patients. In each group of non-PSP participants, MRPI values were lower than those observed in PSP-P and PSP-RS patients.

TABLE 2.

Automated measurements of brain structures and MRPI in non-PSP participants and PSP patients

	non-PSP			PSP
Brain structures	HC	PD	MSA	All PSP	PSP-RS	PSP-P
Midbrain area (mm2)a	128 ± 20	127 ± 22	113 ± 23	75 ± 20	70 ± 16	85 ± 22
Pons area (mm2)a	497 ± 49	514 ± 58	419 ± 112	472 ± 59	465 ± 54	486 ± 65
MCP width (mm)a	8.97 ± 0.77	9.12 ± 0.84	8.25 ± 1.10	8.52 ± 0.84	8.48 ± 0.83	8.61 ± 0.86
SCP width (mm)a	3.75 ± 0.41	3.86 ± 0.41	3.64 ± 0.45	3.15 ± 0.58	3.00 ± 0.56	3.44 ± 0.51
MRPIa	9.55 ± 2.01	9.87 ± 2.12	8.75 ± 3.02	18.55 ± 5.96	20.10 ± 5.64	15.55 ± 5.43

PSP-P, progressive supranuclear palsy–parkinsonism; PSP-RS, progressive supranuclear palsy–Richardson’s syndrome; PD, Parkinson’s disease; MSA, multiple system atrophy; HC, healthy controls; MCP, middle cerebellar peduncle; SCP, superior cerebellar peduncle; MRPI, Magnetic Resonance Parkinsonism Index.

All data are expressed as mean ± standard deviation.

^aP < 0.001 (non-PSP vs PSP) Mann-Whitney test.

Sensitivity, specificity, accuracy, PPV, NPV, and AUC for automated MRPI values in distinguishing PSP from non-PSP participants are shown in Figure 1 and Table 3. Automated MRPI had 82.3% sensitivity and 93.8% specificity in discriminating PSP from non-PSP patients. Sensitivity and specificity of 88.9% and 94.7%, respectively, were observed when only PSP-RS participants were compared with non-PSP participants. MRPI values showed lower sensitivity and specificity in discriminating PSP-P patients from non-PSP patients (76.8% sensitivity and 87.6% specificity). Focusing on patients at an early stage of disease (disease duration of less than 4 years), we found that MRPI values had 86.9% sensitivity and 89.4% specificity in discriminating patients with PSP from non-PSP participants. Sensitivity and specificity of 89.8% and 90.4%, respectively, were observed when PSP-RS patients were compared with non-PSP participants. MRPI values showed 82.3% sensitivity and 86.5% specificity in discriminating PSP-P patients from non-PSP patients (Table 4). Slightly lower MRPI performance was observed when algorithm failures were considered as misclassification cases (see Table S3 and Table S4).

FIG. 1.

Receiver operating characteristic curves of MRPI values. (A) Receiver operating characteristic curves considering all PSP and non-PSP participants; (B) Receiver operating characteristic curves considering patients with duration of disease fewer than four year.

TABLE 3.

Cutoffs and diagnostic properties of MRPI for the differentiation of patients with PSP from non-PSP participants

Cutoff and statistical values	Automated MRPI
	164 PSP vs 452 non-PSP
Cutoff value	> 13.42
Sensitivity	82.3%
Specificity	93.8%
Accuracy	90.7%
PPV	82.8%
NPV	93.6%
	108 PSP-RS vs 452 non-PSP
Cutoff value	≥ 13.88
Sensitivity	88.9%
Specificity	94.7%
Accuracy	93.6%
PPV	80.0%
NPV	97.3%
	56 PSP-P vs 452 non-PSP
Cutoff value	≥ 12.38
Sensitivity	76.8%
Specificity	87.6%
Accuracy	86.5%
PPV	43.4%
NPV	96.8%

PSP, progressive supranuclear palsy; PSP-P, progressive supranuclear palsy–parkinsonism; PSP-RS, progressive supranuclear palsy–Richardson’s syndrome; non-PSP, Parkinson’s disease patients + multiple system atrophy patients + healthy controls; MRPI, Magnetic Resonance Parkinsonism Index; PPV, positive predictive value; NPV, negative predictive value.

TABLE 4.

Cutoffs and diagnostic properties of the MRPI for the differentiation of patients with PSP from non-PSP patients in early stage of disease (disease duration less than 4 years)

Cutoff and statistical values	Automated MRPI
	76 PSP vs 104 non-PSP
Cutoff value	> 13.42
Sensitivity	86.9%
Specificity	89.4%
Accuracy	88.3%
PPV	85.7%
NPV	90.4%
	59 PSP-RS vs 104 non-PSP
Cutoff value	≥ 13.88
Sensitivity	89.8%
Specificity	90.4%
Accuracy	90.1%
PPV	84.1%
NPV	94.0%
	17 PSP-P vs 104 non-PSP
Cutoff value	≥ 12.38
Sensitivity	82.3%
Specificity	86.5%
Accuracy	85.9%
PPV	50.0%
NPV	96.7%

PSP, progressive supranuclear palsy; PSP-P, progressive supranuclear palsy–parkinsonism; PSP-RS, progressive supranuclear palsy–Richardson’s syndrome; non-PSP, Parkinson’s disease patients + multiple system atrophy patients; MRPI, Magnetic Resonance Parkinsonism Index; PPV, positive predictive value; NPV, negative predictive value.

Discussion

In the current study, we investigated for the first time the generalizability of MRPI values in a large international cohort of PSP patients and non-PSP participants.

Our findings showed that PSP patients were characterized by midbrain and SCP atrophy compared with healthy controls and patients with PD. Of note, the values of the midbrain area and the SCP width in PSP-P patients were intermediate between those of PSP-RS and non-PSP participants. These findings are in agreement with postmortem studies in PSP patients, confirming that the PSP-P phenotype was associated with a less severe pattern of atrophy compared with PSP-RS.^22,23 Moreover, our results are in line with previous MRI studies demonstrating morphometric and diffusivity alterations of SCP in PSP patients compared with PD patients and controls.^2,24–26 The more severe atrophy in PSP-RS patients compared with those with PSP-P may explain the clinical manifestations that distinguish the 2 phenotypes of the disease. More specifically, the decrease in SCP width found in PSP-RS compared with PSP-P could be related to the balance and posture deficits that characterize PSP-RS already in the early stages of disease. This finding is in line with our recent study showing the key role of SCP in developing postural instability in PSP.²⁴ Moreover, the more severe reduction in midbrain area in PSP-RS may be related to vertical gaze abnormalities possibly occurring later in the course of PSP-P.^23,24,27

Concerning the MRPI, we observed that its values were significantly higher in PSP patients than in non-PSP participants, with a low overlap between PSP and non-PSP subjects. The MRPI showed optimal performance in differentiating PSP-RS patients from non-PSP subjects, whereas it had lower performance in distinguishing between PSP-P patients and non-PSP participants, confirming the results already described in previous studies.^4,9,15 Indeed, some authors in a retrospective study on a small sample of PSP-P patients found that the MRPI differentiated these patients from those with PD with low diagnostic sensitivity (70%) and specificity (68%),⁴ and others reported 74% accuracy using the volume and fractional anisotropy of the superior cerebellar peduncles.²⁸ In our previous report, we demonstrated that about 50% of patients with a clinical diagnosis of PSP-P had MRPI values indistinguishable from those in patients with PD.¹⁵ Recently, we also found that the MRPI was much less accurate in differentiating PSP-P patients with ocular slowness from PD patients compared with PSP-RS patients.⁹ To overcome the low accuracy of the MRPI in differentiating PSP-P from PD, we recently introduced a new MRI biomarker called MRPI 2.0, which more accurately differentiates PSP-P patients with ocular slowness from patients with PD,⁹ thus helping clinicians consolidate a clinical diagnosis of PSP-P in the early stage of the disease. MRPI 2.0 calculation was performed multiplying the MRPI value by the third ventricle width/frontal horns width ratio. However, to date ventricular values used in the MRPI 2.0 calculation are evaluated using a manual procedure. Then, future studies using an automated approach for the calculation of this new index on a large data set of PSP-P patients should provide the strongest evidence about its usefulness in clinical practice. Moreover, despite the high diagnostic accuracy of the MRPI and MRPI 2.0, there is still room for improvement. Previous diffusion-imaging studies (reflecting microstructural damage) as well as iron-sensitive MRI approaches were shown to be helpful in discriminating patients with atypical parkinsonian disorders from PD patients.²⁹ Because different MR sequences provide unique kinds of information on tissue changes, approaches using the MRPI or MRPI 2.0 together with multimodal MR imaging to assess complementary tissue characteristics may be a promising approach to improve diagnostic accuracy in the differential diagnosis of neurodegenerative parkinsonian disorders, particularly early in the disease course.

Our study shows several strengths. First, our participants were collected from several international research groups. This allowed obtaining the largest cohort of participants ever used to automatically quantify atrophy of the midbrain, pons, MCP, and SCP in PSP patients and then to explore MRPI performance in distinguishing these patients from non-PSP participants. Second, the use of an automated method for the MRPI calculation also allowed resolving conflicting results reported in previous studies, mainly because of either the different expertise of raters manually measuring the brain-stem structures involved in MRPI calculation, or small samples of participants used in classification analysis. Moreover, the automated measurement of the MRPI may overcome the operator dependence associated with manual measuring, avoiding the use of different raters blinded to clinical diagnosis and thus allowing the attainement of more reliable results at a lower cost. Third, MRPI classification analyses were based on cutoff values determined by other studies to investigate the robustness and generalizability of previous MRPI cutoff values in a larger independent cohort of PSP and non-PSP participants. Fourth, we explored for the first time the performance of these MRPI cutoff values in a subcohort of patients within 4 years from disease onset. Our results demonstrated the usefulness of the MRPI for also diagnosing PSP-RS in the early stages of the diseases, enabling PSP-RS patients to be included in clinical trials for promising disease-modifying therapies. Finally, the use of a web-based platform to collect PSP and non-PSP participants allowed us to obtain automated MRPI values in a set of MRI data updated by several research centers around the world in a very simple way representing an excellent solution for incorporating MRPI values in diagnostic decision-making and clinical trial designs.

Despite these strengths, our study also has some limitations. Our cases were not pathologically confirmed, even if the clinical diagnosis was based on international established consensus criteria.^16,17 There was also a significant difference in age at examination between PSP patients and participants from other groups. However, in a previous study we found that MRPI performance in distinguishing PSP patients from non-PSP values were not influenced by age.³⁰ Moreover, the automated approach showed failure in 40 subjects (6% of the sample), in whom it failed without measuring the brain structures. The sensitivity and specificity of the MRPI in differentiating PSP patients from non-PSP participants evaluated including algorithm failures showed slightly lower values than those obtained using only cases in which the automated method successfully completed the segmentation of brain structures. However, algorithm failures in the automated MRPI calculation can be resolved using a manual approach performed by an expert neuroradiologist at server side. Finally, in recent years, several structural and functional imaging studies have shown the important role of supratentorial brain regions in differentiating neurodegenerative parkinsonian disorders.^12,29 Then, new approaches combining MRPI values with advanced imaging solutions assessing supratentorial characteristics may represent a useful tool to improve accuracy in the differentiation between PSP and PD patients.

In conclusion, we provide strong evidence of the generalizability of automated MRPI measures in a large international cohort of PSP-RS and PSP-P patients, demonstrating that this MR morphometric marker is able to accurately distinguish between PSP and non-PSP patients already in the early stage of disease. Moreover, the use of a web-based platform to perform the automated MRPI measurement represents an interestingly solution for incorporating MRPI values in diagnostic decision-making and in patients selection for clinical trial.