External validation of the PLASMIC score: A clinical prediction tool for thrombotic thrombocytopenic purpura diagnosis and treatment

Summary

Background

The PLASMIC score was recently published to distinguish patients with severe ADAMTS-13 deficiency from those without for early identification of thrombotic thrombocytopenia purpura (TTP).

Objective

We performed an independent external validation of the PLASMIC score for clinical prediction of severe ADAMTS-13 deficiency.

Patients/Methods

We studied an independent cohort of 112 consecutive hospitalized patients with suspected thrombotic microangiopathy and appropriate ADAMTS-13 testing (including 21 patients with TTP diagnosis).

Results

The PLASMIC score model predicted severe ADAMTS-13 deficiency with a c statistic of 0.94 (0.88–0.98). When dichotomized at high (score 6–7) versus low-intermediate risk (score 0–5), the model predicted severe ADAMTS-13 deficiency with positive predictive value of 72%, negative predictive value of 98%, sensitivity of 90%, and specificity of 92%. In the low-intermediate risk group (score 0–5), there was no significant improvement in overall survival associated with plasma exchange.

Conclusions

The PLASMIC score model had excellent applicability, discrimination, and calibration for predicting severe ADAMTS-13 deficiency. The clinical algorithm allowed identification of a subgroup of patients who lacked significant response to empiric treatment.

Introduction

Acquired thrombotic thrombocytopenic purpura (TTP) is a rare hematologic disorder whose hallmark is severe ADAMTS-13 protease deficiency that leads to accumulation of ultra-large von Willebrand Factor, microvascular thrombosis, and hemolytic anemia [1,2]. Bendapudi et al. recently published a validated clinical algorithm known as the PLASMIC score which was shown to be an excellent diagnostic model to distinguish patients with severe ADAMTS-13 deficiency from those without [3]. However, it remains unclear if it can also serve as a prognostic marker for predicting response to plasma exchange (PEX) such that it is safe to withhold treatment in the lower-risk group. In this independent external validation of the PLASMIC score, we aim to reproduce its diagnostic value for severe ADAMTS-13 deficiency in suspected TTP patients and examine for the first-time its predictive value for response to PEX.

Patients and Methods

This study was approved by the University of Washington (UW) institutional review board. We identified 239 consecutive adult patients with at least one ADAMTS-13 testing from 2007 to 2016 in our laboratory registry. We then excluded patients who did not meet laboratory criteria of microangiopathic hemolytic anemia (MAHA as defined by thrombocytopenia with platelet count <150 × 10⁹ per L and concurrent schistocytes on blood smear) or had outpatient assay testing (Fig 1). We allowed for inclusion of patients who received prior fresh frozen plasma (FFP), provided they did not receive PEX before testing. Patients were selected from UW Medical Center (academic tertiary care center) and Harborview Medical Center (major county hospital and regional trauma center). ADAMTS-13 assays were performed at the BloodWorks Northwest (BWNW) using the Flourescence Resonance Transfer Assay (FRET). Severe deficiency was defined as an ADAMTS-13 activity level <15% in this study before data collection given possible interference from FFP as one prior study showed that 2–3 units of FFP could increase ADAMTS-13 activity by an average of 8% [4].

Figure 1.

Patient selection criteria for PLASMIC score validation

For clinical chart review to identify the 7 components of the PLASMIC score, we had a single reviewer (PRK) blinded to the ADAMTS-13 results who collected the earliest available laboratory values for up to 3 days if the test was sent within the first 3 days of hospitalization or the closest laboratory values within 5 days if the test was sent after 3 days. Two reviewers (AL, PRK) also individually reviewed clinical charts to determine the clinical diagnoses and discrepancies were adjudicated by a third reviewer (DAG). We used an algorithm to assign definitions similar to the one published by Bendapudi et al. whereas clinical TTP diagnosis was made independently from ADAMTS-13 activity based on clinical chart review of admission and discharge summaries [5]. For patients receiving PEX, the procedures were performed by trained apheresis physicians from BWNW. Typically, patient received 1 daily exchange with 1.3 plasma volume. Per BWNW protocol, human serum albumin (HSA) was used as the first half of the replacement fluid followed by FFP for the remainder to achieve a final 50:50 HSA:FFP ratio [6]. The decision to taper was made by the apheresis service in consultation with the clinical hematologist. Overall survival was assessed as a surrogate for the response to PEX. Patients were followed until death or were censored at the time of last available inpatient or outpatient clinical visit.

To determine the performance of the PLASMIC score in this external dataset, we tested the original model for discrimination and moderate calibration. Since there were <5% missing data, we used complete-case analysis. We first created a logistic regression to test the association between severe ADAMTS-13 deficiency and the nominal 7-point PLASMIC score. C statistic was calculated from area under the receiver operator curve (AUC of ROC). Bias-corrected 95% confidence interval was estimated using 1000 bootstrap resamples. We then assigned patients into the 3-risk categories to test the goodness-of-fit with the calibration curve and Hosmer-Lemeshow (HL) test. Kaplan Meier estimator was used for survival analysis and the difference was compared with the log-rank test.

Results and Discussion

We identified 112 patients who met the appropriate MAHA criteria out of 239 consecutive patients (Fig 1). Among them, 108 (96%) had complete data for all 7 components of the PLASMIC score assessment. In contrast to the derivation cohort of 214 patients, our validation cohort was drawn from a different geographic location and a different reference laboratory. Other notable differences in the validation cohort included allowance for prior FFP infusion, higher median ADAMTS-13 activity, lower proportion of severe deficiency, lower median lactate dehydrogenase, and higher median international normalized ratio (Table 1). Twenty-seven patients received FFP prior to ADAMTS-13 testing. Twenty patients had severe ADAMTS-13 deficiency (including 2 patients with activities at 11% and 12%; both received FFP prior to testing). Twenty-one patients had a clinical diagnosis of TTP (all had severe ADAMTS-13 deficiency except 1 patient who had ADAMTS-13 activity of 21%, but met our clinical definition of TTP and responded to PEX). The percentage of patients receiving PEX treatment and the distribution of clinical diagnoses in each group were summarized in Table 2.

Table 1.

Comparison of the original derivation cohort (Bendapudi et al) and current external validation cohort (Li et al)

	Derivation Cohort (n=214)	Validation Cohort (n=112)	P value
Demographic Features
Geographic Location	New England	Pacific Northwest
Age (years)	51 (38–63)	48 (34–58)	0.050
Female Sex	129/214 (60%)	71/112 (63%)	0.633
White Ethnicity	157/204 (77%)	72/106 (68%)	0.102
Clinical Data
Received FFP Prior to Testing	0/214 (0%)	27/112 (24%)	<0.001
Received Plasma Exchange	124/214 (58%)	58/112 (52%)	0.291
Recent Cancer Treatment	61/214 (29%)	28/112 (25%)	0.516
Previous Transplant	37/214 (17%)	22/112 (20%)	0.650
Laboratory Data
Reference Laboratory	BloodCenter of WI	BloodWorks NW
ADAMTS-13 Activity (%)	44 (0–63)	59 (33–82)	<0.001
Severe Deficiency †	62/214 (29%)	20/112 (18%)	0.032
Inhibitor Present ‡	52/214 (24%)	18/112 (16%)	0.090
Platelet Count (×10^9/L)	34 (18–59)	42 (19–71)	0.383
Creatinine (mg/dL)	1.7 (1.1–3.4)	2.0 (1.2–3.6)	0.312
Lactate Dehydrogenase (U/L)	963 (637–1594)	787 (450–1537)	0.032
International Normalized Ratio	1.1 (1.1–1.3)	1.3 (1.1–1.5)	<0.001
Bilirubin (mg/dL)	1.8 (1.0–3.1)	1.8 (1.0–2.9)	0.967
Completeness of Data
All Components of Score (%)	200/214 (93%)	108/112 (96%)	0.317

Continuous variables are expressed as median (IQR) and categorical variables are expressed as number/total (%). Continuous variables are compared using the rank sum test. Categorical variables are compared using the Fisher’s exact test.

^†Severe deficiency is defined as ADAMTS-13 ≤10% in the derivation study and ≤15% in the external validation study.

^‡A positive inhibitor is defined as a titer >0.4 Bethesda units.

Table 2.

External validation of the PLASMIC score using 108 patients with TMA

	PLASMIC Score Risk Prediction
	Low Risk Score 0–4 (n=49)	Intermediate Score 5 (n=34)	High Risk Score 6–7 (n=25)
ADAMTS-13 Testing
Severe ADAMTS-13 Deficiency †	0 (0%)	2 (6%)	18 (72%)
ADAMTS-13 <10%	0 (0%)	2 (6%)	16 (64%)
Positive ADAMTS-13 Inhibitor ‡	2 (4%)	2 (6%)	14 (56%)
Treatment Received
Plasma Exchange (PEX)	15 (31%)	19 (56%)	22 (88%)
FFP Prior to Testing	10 (20%)	13 (38%)	4 (16%)
Clinical Diagnosis *
Autoimmune TTP	0	2	19
HUS or aHUS	3	2	2
Rheumatologic	1	1	2
Malignant Hypertension	3	5	0
Drug Associated	1	1	0
Transplant Associated	17	2	0
Sepsis +/− DIC	11	10	1
Cancer +/− DIC	3	3	0
Obstetric +/− DIC	2	3	0
Other DIC	3	2	1
TMA-Mimic	5	3	0

The PLASMIC score separated patients into 3 different cohorts with distinctively different ADAMTS-13 results, plasma exchange utilization, and associated clinical diagnoses. The high-risk for TTP (score 6–7) group predicted 72% of patients with TTP whereas the low-risk for TTP (score 0–4) group predicted 100% of patients without TTP.

^†Severe deficiency is defined as ADAMTS-13 <15% in this external validation study.

^‡A positive inhibitor is defined as a titer >0.4 Bethesda units.

^*TTP: thrombotic thrombocytopenic purpura; HUS/aHUS: hemolytic uremic syndrome, atypical hemolytic uremic syndrome; DIC: disseminated intravascular coagulation; TMA-mimic: clinical conditions that mimicked the diagnosis of thrombotic microangiopathy (e.g. severe B12 deficiency).

The PLASMIC score stratified 108 TMA patients with diverse clinical diagnoses into 3 risk categories: of the 25 patients with high-risk (score 6–7), 72% had severe ADAMTS-13 deficiency; of 49 patients classified as low-risk (score 0–4), none had severe ADAMTS-13 deficiency (Table 2). There were only 2 cases of severe ADAMTS-13 deficiency in the intermediate-risk group (score 5). One patient in this group had a platelet count of 33,000/μL on transfer from outside hospital (dropped to below 30,000/μL the next day); the other patient had unclear interventions from two outside hospitals in the week prior to transfer. When dichotomized at high (score 6–7) versus low-intermediate risk (score 0–5), the model predicted severe ADAMTS-13 deficiency with positive predictive value of 72%, negative predictive value of 98%, sensitivity of 90%, and specificity of 92%.

The 7-point PLASMIC score model had excellent discrimination with a c statistic of 0.94 (0.88–0.98). The 3-category risk stratification had a near-perfect moderate calibration curve (HL test P of 0.79). For applicability, we compared the ease of the PLASMIC score calculation (available in 108 patients or 96% cohort) with two other published risk prediction tools for TTP. The Coppo score could only be assessed in 53 patients (47% of the cohort) [7] and the Bentley score could only be calculated in 37 patients (33% of the cohort) [8]. Both alternative scores misclassified 1 severe ADAMTS-13 deficiency as low-risk but no formal comparisons could be made due to missing values. As a sensitivity analysis, we used clinical TTP diagnosis (instead of severe ADAMTS-13 deficiency) as our prediction endpoint and re-examined the PLASMIC score after including the 4 patients who were originally excluded for ADAMTS-13 testing after PEX. The PLASMIC score (n=112) retained a c statistic of 0.95 (0.90–0.98). Among the 4 patients with invalid ADAMTS-13 testing, the PLASMIC score identified 3 as high risk and 2 of them had clinical TTP (1 had atypical hemolytic uremic syndrome) and 1 as intermediate risk who did not have TTP (rheumatologic TMA).

Finally, we examined the treatment effect of PEX on survival probability separately in the high-risk (6–7) and low-intermediate risk (0–5) groups (Fig 2). In the high-risk group, despite the limited numbers, treatment with PEX versus not led to significantly improved survival (log-rank P of <0.01). In contrast, in the low-intermediate risk group where patients were predicted to not have severe ADAMTS-13 deficiency, treatment with PEX versus not did not have a significant impact on the survival (log-rank P of 0.50).

Figure 2.

Overall survival by PLASMIC score risk and plasma exchange (PEX)

In the high-risk for TTP group (PLASMIC score 6–7), treatment with PEX led to significantly longer survival (log rank P-value <0.01). In the low-intermediate risk for TTP group (PLASMIC score 0–5), treatment with PEX did not lead to a difference in survival (log rank P-value 0.50) and both treated and untreated groups had worse prognosis.

Our study had several limitations. First, we had small sample size with limited events due to the inherent nature of rare disease. Using the same algorithm as the original PLASMIC score derivation, we screened all patients who had at least 1 ADAMTS-13 testing who also had signs of TMA. While we could have missed patients with TMA without ADAMTS-13 testing, that subgroup of patients was very unlikely to have TTP. As the key referral hospital for the pacific northwest region, we captured a diverse patient population from a different geographic region to extend the generalizability of the original prediction model. Second, due to the inherent differences in disease prevalence between our validation cohort and the original validation cohort (18% vs. 47%), the current validation study had slightly lower PPV (72% vs. 82%) and slightly higher NPV (98% vs. 89%) when we dichotomized patients by the high-risk category [3]. However, we confirmed that the prediction score retained its original high sensitivity to “rule out” severe ADAMTS-13 deficiency (90% vs. 88% at the high-risk cut-off and 100% vs. 97% at the low-risk cut-off). Third, we modified the inclusion criteria a priori to allow prior FFP administration during patient transfer from community hospitals; consequently, we defined severe ADAMTS-13 deficiency as <15% instead of <10%. Fortunately, these changes allowed additional inclusion of 27 patients and only affected 2 outcome classifications. When the study was reanalyzed using the more stringent <10% cut-off, the discriminative value was not affected. Despite broadening the inclusion criteria, the PLASMIC score model retained its discrimination for predicting severe ADAMTS-13 deficiency with c statistic 0.94 (0.88–0.98) when compared to the original derivation c statistic 0.96 (0.92–0.98). Similarly, when we used clinical TTP diagnosis as the outcome in our additional sensitivity analysis, the discriminative value remained unchanged. Finally, our analysis to predict the effect of plasma exchange on patient survival by PLASMIC model remained exploratory. Given the small sample size in the low-intermediate risk group (n=83 with 37 deaths), we could only detect a statistically large effect size on survival with a hazard ratio of 0.37 at 80% power and 5% alpha error. Nonetheless, the lack of apparent clinical benefit for PEX in the low-intermediate PLASMIC score group, which corresponded to patients without severe ADAMTS-13 deficiency, was consistent with results from a previously study in a similar patient population [9]. We acknowledge that the post-PEX survival rate of patients with a high PLASMIC score in our study was somewhat higher than might be expected based on previous reports of outcomes in patients who received PEX for TTP [10]. This discrepancy may be explained in part by the fact that none of the previous trials stratified patients by PLASMIC score and most of them allowed inclusion of patients without severe ADAMTS-13 deficiency.

In summary, in our independent external validation study of the PLASMIC diagnostic model, a high score (6–7) correlated well with severe ADAMTS-13 deficiency (and clinical TTP diagnosis in the sensitivity analysis) and predicted a high rate of response to PEX. In contrast, patients with a low or intermediate PLASMIC score (0–5) were very unlikely to have severe ADAMTS-13 deficiency and did not appear to respond to PEX. The ease of calculating the PLASMIC score makes it a valuable clinical tool when ADAMTS-13 testing is not available and urgent therapy is needed. The results of our analysis support the recommendations from the original PLASMIC score authors: to forgo ADAMTS-13 activity testing in low-risk patients; to measure ADAMTS-13 activity while trending daily labs with best clinical judgment in the intermediate-risk patients; and to treat empirically with PEX in the high-risk patients unless pre-treatment ADAMTS-13 activity turns out to be normal [11]. Future prospective trials should consider incorporating the PLASMIC score as a risk-stratification to determine its true predictive potential.

Essentials.

• Severe ADAMTS-13 deficiency is key to thrombotic thrombocytopenic purpura (TTP) diagnosis.

• PLASMIC score predicts ADAMTS-13 deficiency in suspected TTP with high discrimination.

• PLASMIC score is more generalizable with fewer missing data than alternative clinical scores.

• PLASMIC score identifies a subgroup of patients lacking significant response to plasma exchange.