Second international collaborative study evaluating performance characteristics of methods measuring the von Willebrand factor cleaving protease (ADAMTS-13)

Summary.

Background:

Over the last 4 years ADAMTS-13 measurement underwent dramatic progress with newer and simpler methods.

Aims:

Blind evaluation of newer methods for their performance characteristics.

Design:

The literature was searched for new methods and the authors invited to join the evaluation. Participants were provided with a set of 60 coded frozen plasmas that were prepared centrally by dilutions of one ADAMTS-13-deficient plasma (arbitrarily set at 0%) into one normal-pooled plasma (set at 100%). There were six different test plasmas ranging from 100% to 0%. Each plasma was tested ‘blind’ 10 times by each method and results expressed as percentage vs. the local and the common standard provided by the organizer.

Results:

There were eight functional and three antigen assays. Linearity of observed-vs.-expected ADAMTS-13 levels assessed as r² ranged from 0.931 to 0.998. Between-run reproducibility expressed as the (mean) CV for repeated measurements was below 10% for three methods, 10–15% for five methods and up to 20% for the remaining three. F-values (analysis of variance) calculated to assess the capacity to distinguish between ADAMTS-13 levels (the higher the F-value, the better the capacity) ranged from 3965 to 137. Between-method variability (CV) amounted to 24.8% when calculated vs. the local and to 20.5% when calculated vs. the common standard. Comparative analysis showed that functional assays employing modified von Willebrand factor peptides as substrate for ADAMTS-13 offer the best performance characteristics.

Conclusions:

New assays for ADAMTS-13 have the potential to make the investigation/management of patients with thrombotic microangiopathies much easier than in the past.

Introduction

ADAMTS-13 (A Disintegrin And Metalloprotease with ThromboSpondin type 1 motifs 13) is a plasma protease [1,2] that functions as an in vivo regulator of the platelet-adhesive protein von Willebrand factor (VWF) under conditions of high shear stress [3,4]. As a result of severe congenital or acquired ADAMTS-13 deficiency VWF is not properly cleaved and circulates as ultralarge multimers that promote disseminated platelet aggregation and microvascular thrombosis, leading to thrombotic thrombocytopenic purpura (TTP) [5,6]. Upon the identification and characterization of ADAMTS-13 several methods have been designed for its measurement; some of them measure directly or indirectly the VWF cleaving properties and others measure the antigenic concentration. In a previous international collaborative study we had investigated the performance characteristics of the methods described until the year 2003 [7]. Since then several novel methods have been described and this prompted us to organize and carry out a second international collaborative study that includes some of the previous methods [8,9]and others described from 2004 until now. Altogether we report on the performance characteristics of 11 methods run in eight laboratories.

Materials and methods

Participants

Newly published methods were identified by searching the international literature and the corresponding authors were invited to participate. Eight laboratories agreed to join this project; there are eight new methods and three (methods 5, 6 and 11) that had already been investigated in the previous international collaborative study (Table 1). There are eight functional (methods 1–7 and 11) and three antigen assays (methods 8–10) (Table 1). Some of the participating laboratories performed more than one method and some used assays based upon the same principle (i.e. FRET, collagen binding, or antigen assays). However, in this study they were classified as different methods because they have been set up in different laboratories and with different reagents.

Table 1.

Main characteristics of the investigated methods

Method	Substrate, monoclonal or polyclonal antibodies	Type of methods	Measurement of cleaved VWF	Ref.
1	Fluorogenic VWF73 peptide (residues 1596–1668 of VWF)	Functional	Fluorescence resonance energy transfer (FRET) assay	[10]
2	VWF78 peptide conjugated with HRP (N-terminus) and labelled with biotin (C-terminus) (HRP-VWF78-biotin)	Functional	Streptavidin-agarose absorption and measurement of HRP activity in solution	[16]
3	Recombinant VWF73 (residues 1596–1668 of VWF) tagged with both GST and His (GST-VWF73-His)	Functional	Anti-N10 MoAb and measurement of HRP activity	[15]
4	Fluorogenic VWF73 peptide (residues 1596–1668 of VWF)	Functional	Modified fluorescence resonance energy transfer (FRET) assay	[14]
5	Purified (plasma-derived) VWF	Functional	SDS agarose electrophoresis/immunoblotting	[9]
6	Recombinant VWF	Functional	Collagen binding	[8]
7	Fluorogenic VWF73 peptide (residues 1596–1668 of VWF)	Functional	Fluorescence resonance energy transfer (FRET) assay	[10]
8	Polyclonal rabbit antihuman ADAMTS-13	Antigen		[17]
9	MoAb antihuman ADAMTS-13	Antigen		[11]
10	MoAb antihuman ADAMTS-13	Antigen		[11]
11	Plasma-derived VWF	Functional	Collagen binding	[8]

VWF, von Willebrand factor; HRP, horseradish peroxydase; GST, glutathione-S-transferase; His, histidine.

Test plasmas

Blood was obtained from two Iranian brothers born of a consanguineous marriage. The first was a 26-year-old male, who developed his first TTP episode at the age of 23 years and had six subsequent recurrent episodes without precipitating events. The second was a 31-year-old male who developed TTP at the age of 29 years in association with an episode of pulmonary infection. Both patients were in remission at the time of blood donation and had not received fresh frozen plasma (within the last 4 weeks). ADAMTS-13 activity and antigen were undetectable by collagen binding [8], FRET [10] and ELISA [11] assays as measured on different occasions at the coordinating laboratory. No ADAMTS-13 antibodies were detected by Western blotting and the Bethesda method [8]. Genotype analysis performed by PCR followed by sequence analysis showed that they are homozygotes for a deletion mutation of six nucleotides GTGCCC at position 2930–2935 (c.2930_2935del GTGCCC) in exon 23, leading to the replacement of Cys977 residue by a Trp and the deletion of two amino acids Ala978 and Arg979 (p.C977W+p.A978_R979del) in the TSP1–6 repeat domain of the ADAMTS-13 protein [12], causing a severe ADAMTS-13 deficiency. The study was carried out with the patients’ consent and after approval by the Institutional Review Board. Blood was collected into plastic tubes and anticoagulated with trisodium citrate (109 mM) at a proportion of 1:9 (anticoagulant:blood) and centrifuged (controlled room temperature) for 20 min at 2000 × g. Supernatant plasmas from the two patients were pooled in plastic containers. Concurrently, aliquots of citrated plasmas from four healthy volunteers were pooled to prepare a normal plasma. For the purpose of this study the deficient and normal plasma were arbitrarily assigned values of 0% and 100% ADAMTS-13 activity (or antigen), respectively. Normal and deficient plasmas were stored frozen at −70 °C until the preparation of the test plasmas that was carried out at the coordinating laboratory (Milano) 3 months after blood collection. Normal and deficient plasmas were thawed and mixed appropriately to prepare six test plasmas (provisionally coded from A to F) with ADAMTS-13 levels ranging from A = 100% (normal plasma) to F = 0% (deficient plasma only). The intermediate ADAMTS-13 levels in the remaining plasmas were arranged to correspond to B = 40%, C = 20%, D = 10%, and E = 5%. Each of the six plasmas was then aliquoted in plastic tubes and assembled into sets of 60. Each set included 10 copies of the six original plasmas. Plasmas in each set were eventually coded from 1 to 60 and stored frozen at )-70 °C until they were sent in dry ice to the participants, who were unaware of the procedure adopted to prepare and assemble the sets of test plasmas.

Common reference standard

A suitable volume of the pooled normal plasma was aliquoted in plastic tubes, snap frozen and stored as above. The ADAMTS-13 activity (or antigen) for this plasma was arbitrarily set at 100%.

Design of the study

Sets of coded test plasmas were sent to the participants together with suitable amounts of the common standard and a detailed testing protocol with the following instructions. (i) ADAMTS-13 levels had to be measured using the local method run according to the local procedure (number of readings/dilutions for each sample). (ii) The method had to be calibrated against the local and the common standard. (iii) The whole set of measurements had to be divided into 10 runs. (iv) On each run the method had to be calibrated with a new set of local and common standards. (v) Coded plasmas had to be tested in ascending consecutive order (first run, 1–6; second run, 7–12; through to the last run, 55–60), so that each group of six included one set of the original test plasmas (A–F).

Because of this design, at the end of the 10 runs the participants obtained (in blind) 10 replicate determinations for each test plasma. Results were recorded in a data collection form as ADAMTS-13 percentage activity (or antigen) relative to the local and the common standard. One laboratory expressed antigen results when measured against the local standard as ng mL⁻¹ and one laboratory did not provide results against the local standard.

Data analyses

Results were centralized and decoded at the coordinating center. Statistical analyses were carried out as described in the previous international collaborative study [7]. Briefly, ADAMTS-13 activity or antigen values were used to evaluate the following characteristics. (i) The linearity of expected vs. observed values, evaluated by calculating for each method the correlation coefficient expressed as r². (ii) The between-run reproducibility, evaluated by calculating the mean and standard deviation for the 10 repeated measurements obtained for each test plasma with each method; reproducibility was expressed as the coefficient of variation (CV) calculated as the standard deviation divided by the mean ADAMTS-13 value and multiplied by 100. (iii) The ability of each method to distinguish between different ADAMTS-13 levels, evaluated by assessing for statistical significance the difference between mean values obtained for each test plasma (one-way analysis of variance, or t-test); the F-(or the t-) value generated for each method was taken as an index of the ability of each method to distinguish different ADAMTS-13 levels (the greater the F-(or the t)-value the better the discrimination). (iv) The between-method comparability of results, evaluated as the CV value calculated for each test plasma. (v) The influence of local or common standards on the between-method comparability of results, evaluated by comparing the between-method CV values calculated on results obtained by using the local or the common standard. ADAMTS-13 levels that were below the detection limits for each method were not included in the above analyses.

Results

Linearity of expected vs. observed values

Plots of expected vs. observed ADAMTS-13 values and squared correlation coefficients for each method are shown in Fig. 1. r² values ranged from 0.931 (method 9) to 0.998 (method 2). Table 2 shows results for each method obtained on the plasma with an expected ADAMTS-13 level set arbitrarily at 0% (i.e. the deficient plasma pooled from the patients with congenital recurrent TTP) and on the plasmas with expected levels of 5% and 10%. Methods 3, 2 and 1 detected the lowest measurable mean values for the deficient plasma (3.0%, 3.2% and 5.8%, respectively). The mean values obtained by the above methods for the 5% plasma were 6.3%, 7.6% and 9.9%, respectively. All the other methods obtained ADAMTS-13 mean levels for the deficient plasma that were below their detection limits. Methods 4, 5, 6 and 11 measured ADAMTS-13 mean levels of 5.9%, 7.4%, 10.4% and 9.4%, respectively for the 5% plasma. Methods 9 and 10 consistently measured ADAMTS-13 levels of less than 1% for the 10%, 5% and 0% plasmas (Table 2).

Fig. 1.

Expected vs. observed ADAMTS-13 levels recorded with 11 methods for test plasmas. Panel A, functional assays; panel B, antigen assays. Data points for each test plasma represent 10 repeated determinations taken on different occasions. Horizontal bars represent mean values. Plasmas with observed levels below detection limits were omitted from calculation. Observed ADAMTS-13 levels were expressed as % of the local reference standard, except for methods 5 and 8, which were expressed as % of the common standard.

Table 2.

ADAMTS-13 levels (% of the local standard) by different methods for test plasmas with expected 0%, 5% and 10%

		Method
Replicate	Sample	1	2	3	4	5*	6	7	8*	9	10	11
1	0%	2.4	3.5	3.0	<5	<5	<5	<5	<5	<1	<1	<6
2		13.4	2.9	3.2	<5	<5	<5	<5	<5	<1	<1	<6
3		2.6	3.6	2.9	<5	<5	<5	<5	<5	<1	<1	7
4		5.5	3.5	3.3	<5	<5	<5	<5	<5	<1	<1	<6
5		3.0	2.2	3.0	<5	<5	<5	<5	<5	<1	<1	6
6		3.2	3.0	3.1	<5	<5	<5	<5	<5	<1	<1	<6
7		5.4	3.7	3.0	<5	<5	<5	<5	<5	<1	<1	<6
8		8.2	3.1	2.4	<5	<5	<5	<5	<5	<1	<1	<6
9		9.3	3.8	2.0	<5	<5	<5	<5	<5	<1	<1	<6
10		5.4	3.2	3.6	<5	<5	<5	<5	<5	<1	<1	<6
Mean		5.8	3.2	3.0	–	–	–	–	–	–	–	–
1	5%	9.1	8.6	6.4	6	6	10	<5	6	<1	<1	14
2		12.2	7.4	6.2	7	6	10	<5	<5	<1	<1	8
3		8.4	7.2	6.5	6	6	7	<5	<5	<1	<1	<6
4		9.3	7.8	6.8	5	9	12	<5	<5	<1	<1	7
5		10.8	6.8	6.8	5	7	13	<5	6	<1	<1	<6
6		15.1	7.2	6.0	6	8	10	<5	<5	<1	<1	6
7		6.0	8.3	6.3	5	10	9	<5	<5	<1	<1	9
8		7.7	8.4	7.1	6	9	12	<5	6	<1	<1	12
9		11.8	6.7	4.2	6	7	10	<5	6	<1	<1	9
10		8.7	7.6	7.1	7	6	11	<5	<5	<1	<1	10
Mean		9.9	7.6	6.3	5.9	7.4	10.4	–	–	–	–	9.4
t value†		3	17	11	–	–	–	–	–	–	–	–
Mean	10‡	13.6	12.4	10.3	10.6	11.1	15.9	6.9	9.7	<1	<1	11.3
SD		2.8	1.2	1.0	0.5	2.2	2.2	2.7	1.1	–	–	2.5

^*Results expressed as % of the common standard.

^†Statistical analysis to test for differences between mean values of plasmas with an expected ADAMTS-13 level of 0% and 5%. The greater the t value, the better the discrimination.

^‡For the 10% plasma only mean and SD of the 10 replicate measurements are provided.

Reproducibility

CV values obtained for the repeated measurements of the six test plasmas are shown in Fig. 2. In general, the reproducibility for all methods improved (lower CV value) with increasing ADAMTS-13 levels. Mean CV values below 10% were obtained by methods 4, 2 and 8; methods 6, 3, 9, 5 and 10 obtained CV values from 10% to 15% and methods 7, 11 and 1 obtained mean CV values from 17% to 22%.

Fig. 2.

Between-run reproducibility for the 11 methods expressed as coefficient of variation (CV) calculated on repeated determinations for each test plasma. Panel A, functional assays; panel B, antigen assays. CV values for test plasmas with an observed value below the detection limits were not calculated.

Discrimination

The ability of each method to distinguish between ADAMTS-13 levels over the range of measurements from 5% to 100% is summarized in Fig. 3. F-values ranged from 965 (method 2) to 137 (method 9). High F-levels (above 1000), denoting excellent discrimination, were recorded for methods 2, 5, 8 and 4. In particular, the ability of each method to distinguish between the two plasmas with expected 0% and 5% ADAMTS-13 levels could be assessed by statistical analysis only for methods 1–3, as most of the others recorded values for the former plasma that were below the detection limits (Table 2). Overall, the ability to distinguish 5% from 0% is statistically significant (t values 17, 11 and 3) for methods 2, 3 and 1, respectively, and may be subjectively rated as good for methods 4–6 and 11; methods 7–10 were unable to distinguish the 5% and 0% plasmas; methods 9 and 10 were unable to distinguish 10%, 5% and 0% plasmas (Table 2).

Fig. 3.

Discrimination between ADAMTS-13 levels recorded for 11 methods. Columns represent F-values (numbers represent actual values) as determined by one-way analysis of variance for 10 repeated determinations on test plasmas from 5% to 100%. The greater the F-value, the better the discrimination. Statistical analysis for methods 7 and 9–10 was performed for the range of values 10–100% and 20–100%, respectively.

Between-method comparability of results and influence of standards

The between-method variability, expressed as the CV value, amounted to 24.8% (range 15.1–38.8%) when calculated according to the local standard and to 20.5% (range 6.8– 32.8%) when calculated according to the common standard (Table 3). Overall, the variability recorded with the common standard decreased (lower CV values) more for the plasmas with high ADAMTS-13 levels than for the plasmas with low ADAMTS-13 levels (Table 3).

Table 3.

ADAMTS-13 levels (%) obtained by participants who completed the whole set of measurements for test plasmas using the local and common standard

	Local standard				Common standard
Test plasmas (expected value)	n	Mean (%)	SD	CV (%)	n	Mean (%)	SD	CV (%)
A (100%)	9	92	13.9	15.1	9	100	6.8	6.8
B (40%)	9	36	6.6	18.3	9	40	6.0	15
C (20%)	9	18	5.3	29.0	9	20	5.7	28.2
D (10%)	7	12	2.8	24.4	7	13	2.5	19.2
E (5%)	6	8	1.9	23.1	6	9	1.9	21.2
F (0%)	4	5	1.8	38.8	4	5	1.8	32.8
Overall mean CV				24.8				20.5

Discussion

More than 4 years have elapsed since the first international collaborative study organized and carried out to investigate the performance characteristics of ADAMTS-13 methods [7]. At that time all methods were functional assays using purified (plasma-derived, or recombinant) VWF as substrate for plasma ADAMTS-13 and the detection of the cleaved protein was mainly based on the measurement of the residual collagen binding [8] or ristocetin cofactor activity [13] as compared with the uncleaved protein, or the quantitative immunoblotting of the degraded VWF multimers [9]. These methods are, however, complex, and require skill and relatively long incubation times with denaturing agents to demonstrate ADAMTS-13 activity. More recently other functional assays have been developed that employ small synthetic substrates (VWF73 or VWF78) mimicking the portion of VWF containing the physiologic ADAMTS-13 cleavage site (see Table 1). In one of these methods, called fluorescence resonance energy transfer (FRET), the synthetic peptide (VWF73) is chemically modified to reveal (upon cleavage by ADAMTS-13) the fluorescence, which is quenched in the intact peptide [10,14]. This approach makes the direct measurement of the cleaved peptide produced by ADAMTS-13 easier, more sensitive and presumably more reproducible. In another method, the synthetic substrate is tagged with glutathione S-transferase (GST) and histidine (His), making the GST-VWF73-His modified peptide a suitable substrate that upon cleavage by plasmatic ADAMTS-13 may be recognized by a specific horseradish peroxidase (HRP)-labelled monoclonal antibody [15]. Finally, a third method employs a newly developed recombinant peptide (VWF78)-derivative, conjugated with HRP at the N-terminus and labelled with biotin at the C-terminus (here called HRP-VWF78-biotin) [16]. Upon cleavage by ADAMTS-13 and removal of the uncleaved substrate by adsorption with streptavidin-agarose, the cleaved substrate may be quantitated by measuring the unabsorbed HRP activity remaining in solution [16].

Besides the above innovative methods, the production and characterization of polyclonal [17] or monoclonal [11] antibodies to ADAMTS-13 have made possible the antigenic measurement of the metalloprotease by ELISA [11,17] (see Table 1).

The above innovations prompted us to organize and carry out the second international collaborative study evaluating the performance characteristics of the newly developed methods. When we started organizing this study, other innovative methods were available [18,19], but their authors did not join this international collaborative study.

To compare the merits of the new methods with those of the previous ones we elected to apply to the present study the same protocol that was successfully used for the first study [7]. The protocol has some distinct features. First, all methods were performed in those laboratories who developed and described the new methods or in those laboratories that participated in the previous international collaborative study [7], thus ensuring the necessary expertise to make comparison at the highest level. Second, all the methods tested the same set of well-characterized plasmas. Third, all plasmas were coded by the coordinating laboratory and the operators at each participating laboratory were neither cognizant of the ADAMTS-13 levels they were about to measure, nor of the numbers of replicate measurements performed for each test plasma, thus ensuring unbiased estimates of all performance characteristics. Fourth, the measurements had to be taken with methods calibrated vs. the local or the common standard supplied to the participants by the coordinating laboratory, thus giving the opportunity to assess the value of establishing an international reference standard for ADAMTS-13 antigen and activity.

However, there are limits of the present protocol that should be recognized. First, test plasmas were dilutions of one ADAMTS-13 deficient plasma (with no autoantibodies to ADAMTS-13) made into one pooled normal plasma, so that the estimate of some performance characteristics (see below) may be biased by the absence of variable ADAMTS-13 deficiencies and/or the presence/absence of autoantibodies to ADAMTS-13 that would have occurred had test plasmas been obtained from different patients with TTP. On the other hand, the collection of test plasmas from different ADAMTS-13 deficient patients in such large amounts as needed for the study involving many participants and methods would have been difficult and far reaching. Second, ADAMTS-13 levels in test plasmas (expected levels) were arranged on the assumption that the deficient plasma the pooled normal plasma had 0% and 100% ADAMTS-13 levels, respectively. Although it is reasonable to assume that plasma pooled from the two patients used in the study had a severe deficiency of ADAMTS-13 activity and antigen because both patients had recurrent TTP and were both carriers of two homozygous deletion mutations [12],we cannot exclude that residual ADAMTS-13 activity is still present in the plasma. Indeed, mean levels ranging from 3% to 5.8% have been measured by three participating laboratories (see Table 2) These limitations notwithstanding, we believe that the protocol maintains its validity because any effect brought about by the above issues would presumably affect all the investigated methods to the same extent.

Overall, this international collaborative study shows that the performance of the last generation of methods measuring ADAMTS-13 activity is better than that observed in the first study evaluating previous methods [7].First, the analysis of the linearity of expected vs. observed ADAMTS-13 levels, expressed as r², which ranged from 0.390 to 0.970 in the previous study [7], is now much better, ranging from 0.931 to 0.998. Second, the reproducibility of current methods has improved compared with that recorded in the previous study; five of 10 methods had CV value below 15% [7] as opposed to eight of 11 in the present study. Third, the ability of current methods to discriminate between ADAMTS-13 levels is much better than that of the previous ones; the F-value that may be taken as an index of the discriminatory capacity (the higher the better) ranged from 599 to 8 in the previous [7] and from 3965 to 137 in this study. Fourth, the between-method comparability of results expressed as (the overall) CV value that formerly was 40% [7] is now reduced to 25%. The beneficial effect of the common (over the local) reference standard in harmonizing results across methods as shown in this study is rather poor (CV = 25% with the localstandardvs.20% with the common standard) as it was in the previous study (CV = 40% vs. 33%) [7], suggesting that the influence of different standards is not the major determinant of the between-method variability. This consideration is further corroborated by the observation that between-method variability was relatively high even for those laboratories that used the same methodological principle (i.e. the FRET assay that was used by three participants) and supports the view that the variability is due to the way individual laboratories set up the same assay rather than due to between-assay variability. Perhaps, between-laboratory variability could be minimized by the availability of commercial kits assembling all the components needed for the FRET or other assay methods. However, to draw firm conclusions the same kit should be used in different laboratories to test the same set of well-characterized plasmas.

Looking at the performance characteristics of the individual methods involved in this study it appears that all of them were able to detect the most severe deficiency of ADAM-TS-13, as shown by the very low levels consistently measured on the deficient plasma. In particular, methods 1, 2 and 3 recorded low (yet measurable) mean levels of ADAM-TS-13 (i.e. 5.8% 3.2% and3%,respectively)(Table 2).The collagen binding[8] (methods 6 and 11) and the SDS-agaros immunoblotting [9] (method 5) confirmed the good performance characteristics shown in the previous study [7]. The new methods based on (modified) VWF peptides (i.e. the HRP-VWF78-biotin [16] (method 2), the FRET [10,14] (methods 4, 7 and 1), or the GST-VWF73-His [15] (method 3)) displayed excellent performance characteristics. There is little to choose between them, although the HRP-VWF78-biotin (method 2) [16] and the modified-FRET (method 4) [14] displayed the overall best performances (see Fig. 1 and Table 2). The two antigen methods (methods 9 and 10) that used the same monoclonal antibody to capture ADAMTS-13 [11] displayed acceptable reproducibility and linearity of expected vs. observed ADAMTS-13 levels (see Fig. 1), but were unable to distinguish plasma levels below 10% (see Table 2). The other ELISA (method 8) employing a polyclonal antibody [17] displayed excellent reproducibility and linearity of expected vs. observed levels (see Fig. 1), but was unable to distinguish between 5% and 0% ADAM-TS-13 antigen (see Table 2).

Conclusions

The results of this study involving eight new methods for ADAMTS-13 activity or antigen measurement, run in expert laboratories and testing the same set of coded and well-characterized plasmas, demonstrate that the methods performed quite well. Small, but measurable differences in the performance characteristics of the different methods make those based on modified VWF peptides the most suitable to measure ADAMTS-13. Compared with 4 years ago it can be concluded that the quality of methods for the measurement of the VWF-cleaving protease has considerably improved thanks to the development of specific VWF peptides and polyclonal/monoclonal antibodies. These new assays have the potential to make the investigation and management of patients with TTP much easier than in the past.