A Multi-Center Ring Trial of Allergen Analysis using Fluorescent Multiplex Array Technology

Abstract

Background

Consistent performance of allergen assays is essential to ensure reproducibility of exposure assessments for investigations of asthma and occupational allergic disease. This study evaluated intra- and inter-laboratory reproducibility of a fluorescent multiplex array, which simultaneously measures eight indoor allergens in a single reaction well.

Methods

A multi-center study was performed in nine laboratories in the US and Europe to determine the inter-laboratory variability of an 8-plex array for dust mite, cat, dog, rat, mouse and cockroach allergens. Aliquots of 151 dust extract samples were sent to participating centers and analyzed by each laboratory on three separate occasions. Agreement within and between laboratories was calculated by the concordance correlation coefficient (CCC).

Results

Results were obtained for over 32,000 individual allergen measurements. Levels covered a wide range for all allergens from below the lower limit of detection (LLOD=0.1 - 9.8ng/ml) to higher than 6800ng/ml for all allergens except Mus m 1, which was up to 1700ng/ml. Results were reproducible within as well as between laboratories. Within laboratories, 94% of CCC were ≥0.90, and 80% of intra-laboratory results fell within a 10% coefficient of variance (CV%). Results between laboratories also showed highly significant positive correlations for all allergens (∼0.95, p<0.001). Overall means of results were comparable, and inter-laboratory CV% for all allergens except Rat n 1 ranged between 17.6% and 26.6%.

Conclusion

The data indicate that performance criteria for fluorescent multiplex array technology are reproducible within and between laboratories. Multiplex technology provides standardized and consistent allergen measurements that will streamline environmental exposure assessments in allergic disease.

1 Introduction

Exposure to dust mite, pet, rodent and cockroach allergens has been identified as an important risk factor for allergic sensitization and exacerbation of asthma (Platts-Mills et al., 1997). Allergen exposure assessments have played an essential role in multiple epidemiologic studies of asthma in the US, Europe and New Zealand (Eggleston et al., 1998; Phipatanakul et al., 2000; Arbes, Jr. et al., 2003; Arbes, Jr. et al., 2004; Zock et al., 2006; Sears et al., 2003; Almqvist et al., 2003; Woodcock et al., 2004; Illi et al., 2006; Celedon et al., 2002).

Allergen measurements are routinely performed as part of indoor air quality investigations and occupational health monitoring (Curtin-Brosnan et al., 2010; Olmedo et al., 2011), and for standardization of allergenic products. Until recently, these measurements were made using enzyme-linked immunosorbent assay (ELISA). While ELISA has been used successfully for many years, separate tests are required for each allergen, and the process is time-consuming. Monitoring the performance of allergen assays is essential to ensure reproducibility of allergen measurements. Few prior data on the intra- and inter-laboratory variability of ELISA are available (Codina and Lockey, 2007; Pate et al., 2005). A proficiency testing study compared ELISA results for six indoor allergens between eight US laboratories and found significant differences between study sites, with CVs ranging between 61% and 93% (Pate et al., 2005). The study also included the dust handling and extraction process, and use of separate calibrators, which may have contributed to the high levels of variability observed.

Recently, fluorescent multiplex array technology has been developed that allows the simultaneous detection of multiple allergens in a single reaction well, with significantly increased sensitivity (Earle et al., 2007), which is increasingly being used for allergen detection both in homes, schools and occupational health settings (Permaul et al., 2012, Samadi et al., 2010; Wright et al., 2009). Fluorescent multiplex array technology is being extensively used in allergy and immunology research to measure cytokines, growth factors or respiratory viruses (Lalvani et al., 2008). Commercial kits are available for measurement of up to 50 cytokines and growth factors. While several studies investigate intra-laboratory performance of multiplex assays, or compare commercial multiplex kits between manufacturers or with other detection methods (Wong et al., 2008; Djoba Siawaya et al., 2008; Lewczuk et al., 2008; Johnson et al., 2007), few systematic studies of intra- and inter-laboratory performance of this fluorescent bead-based multiplex technology have been published (Fichorova et al., 2008). Systematic studies however are essential for the development of reliable methods and the direct comparison of results from different studies.

Here, we evaluate the precision and reproducibility involving intra- and inter-laboratory variance of a Multiplex ARray for Indoor Allergens (MARIA) which simultaneously measures allergens of dust mites (Der p 1, Der f 1 and Mite Group 2), cat (Fel d 1), dog (Can f 1), rat (Rat n 1), mouse (Mus m 1) and German cockroach (Bla g 2). The objectives of this study were to conduct an international multi-center ring trial to assess the performance of MARIA technology, and document the intra- and inter-laboratory variability of allergen measurements.

2 Methods

2.1 Allergen measurements using fluorescent multiplex array (MARIA)

The MARIA is based on xMAP® technology (Luminex Corp. Austin TX) which uses polystyrene microspheres that are internally labeled to create distinct sets of microspheres. Separate bead sets are covalently coupled with allergen-specific monoclonal antibodies, enabling the simultaneous capture and detection of multiple allergens in a single sample (Earle et al., 2007). The MARIA 8-plex used here allowed the simultaneous detection of allergens of dust mite (Der p 1, Der f 1, Mite Group 2), cat (Fel d 1), dog (Can f 1), mouse (Mus m 1), rat (Rat n 1) and German cockroach (Bla g 2). While results obtained by MARIA are comparable to ELISA within the dynamic range of the ELISA standard curves (Earle et al., 2007), MARIA is significantly more sensitive (Table A.1). The array uses a Universal Allergen Standard to quantify allergens (Filep et al., 2012; Chapman et al., 2008; van Ree et al., 2008; Earle et al., 2007).

2.2 Sample set

Since the intent of this study was to examine MARIA 8-plex assay performance alone, variability associated with sample processing (collection, sieving, extraction) were eliminated by providing pre-processed dust extracts to all study sites. Dust extracts from a bank of reservoir dust samples collected in households primarily in central Virginia were prepared at the coordinating center using established procedures (Vojta et al., 2002). In brief, 100mg of fine dust were extracted in 2ml of PBS-0.05% Tween 20, centrifuged and the resulting supernatant used in this study. 151 samples were selected to create a set of specimens that covered a range of allergen concentrations including undetectable to very high for all eight tested analytes. As none of the available house dust samples contained detectable rat allergen, a number of samples were spiked using animal room bulk dust provided by Dr. Anne Renström (Karolinska Institute, Stockholm, Sweden). All samples were aliquoted into batches of 200µl following thorough mixing, to create identical sets of 151 extract specimens for each of the participating laboratories. Samples were stored frozen at -20°C until their use in the study.

2.3 Study design

Ten US and European laboratories with access to xMAP® instruments were recruited to participate in the study. Nine of the ten facilities completed the study and provided data (Table A.2). The tenth laboratory did not submit data within 18 months of sample receipt and was excluded from the study to avoid further delays. All personnel involved in the analyses were trained to perform MARIA 8-plex analyses by the study coordinator (Dr. Eva King, Indoor Biotechnologies) in Charlottesville, VA. Following training, each participating laboratory received an identical package of materials: a set of 151 dust extracts, all reagents required for the study (pre-mixed 8-plex MARIA beads, Universal Allergen Standard, pre-mixed 8-plex detection antibodies, streptavidin-phycoerythrin and filter plates), instructions for sample and reagent storage, MARIA protocol, data analysis instructions and an Excel template for compiling analysis results. Each laboratory was asked to measure all 151 samples using MARIA 8-plex (at three dilutions: 1:10, 1:100 and 1:10,000) on three separate occasions, in order to provide information both about reproducibility within each study site as well as between laboratories. The timing between sample receipt and reporting results varied between study sites and ranged between two and twelve months. This approach generated over 3,600 individual allergen measurements for each of the participating study sites and more than 32,000 for the entire study.

2.4 Data handling and statistical analysis

Each study site compiled their analysis results in the Excel template provided. Data from all sites were collected at the coordinating center (Indoor Biotechnologies) and forwarded to an independent statistical center for analysis (Rho Inc., NC). The statisticians assigned a blinded code (A-I) to each laboratory's data set. The goal of the study was to assess the agreement within laboratory (intra-laboratory) and between laboratories (inter-laboratory) to evaluate the reproducibility of the MARIA 8-plex. Agreement on a continuous measure was estimated using the concordance correlation coefficient (CCC) (Lin, 1989) which measures how close each pair falls along a 45-degree line from the origin (or a slope of exactly 1). The CCC scales can vary from −1 to +1, as a Pearson correlation coefficient r does, but the CCC has no ability to surpass r in absolute value. For statistical purposes, non-detectable values were treated as the lower limit of detection divided by two (LLOD/2). Allergen concentrations showed a right-skewed distribution therefore we performed a log 10 transformation of those measurements to obtain symmetrical, approximately Gaussian distributions. Analyses were performed with R Version 2.13.1 (R Development Core Team, 2011) and figures were constructed using the R package lattice (Sarkar, 2008).

3 Results

3.1 Distribution and range of allergen levels in the sample set

Initial determination of the concentration range for all eight allergens in the sample set was performed at the coordinating center. Concentrations for all eight allergens tested ranged from below the lower limit of detection (LLOD) of the assays to very high. Sample numbers above detection limit (n) as well as range of allergen concentrations for the individual allergens were: Der p 1: n=105, <0.6-45,385 ng/ml ; Der f 1: n=107, <0.6-6,850 ng/ml ; Mite Group 2: n=127, <0.2-21,255 ng/ml ; Fel d 1: n=128, <0.2-20,037 ng/ml ; Can f 1: n=105, <0.6-25,138 ng/ml ; Mus m 1: n=85, <0.1-1,744 ng/ml, Rat n 1: n=14, <0.2-8,873 ng/ml ; Bla g 2: n=23, <4.9-14,808 ng/ml. The small number of samples with detectable Rat n 1 (n=14) had an effect on the statistical validity of data for this analyte.

3.2 Reproducibility within study sites

Intra-laboratory reproducibility was evaluated using concordance correlation coefficients (CCC) and coefficients of variation (CV%) between triplicate analyses of the 151 samples performed by each laboratory. Figure 1 shows the entire data set of the study: Each of the 72 cubes represents triplicate results of one laboratory for a single allergen, while each axis of a cube represents one of three separate sample measurements. The resulting points should ideally be located along the diagonal through the three-dimensional matrix. Mean CCCs for each triplicate data set are displayed with each cube. Table 1 demonstrates descriptive statistics for the three separate measurements of each allergen by each laboratory. These correlations, as well as mean CV%s demonstrated that the overall reproducibility within each laboratory was good. Over 22% (16/72) of mean CVs fell within 5%, and 80% (57/72) of data was within the 10% CV margin, indicating a high level of intra-laboratory precision (Table 1, Table A.2). The actual level of reproducibility varied between laboratories: Mean CCCs between triplicate data ranged from 0.98 for sites A and C, 0.97 for D and E, 0.96 for B, F, 0.95 for I and 0.94 for G and H. These site-specific differences were also reflected in the CV% results (Table 1, Table A.3).

Figure 1.

Intra-laboratory comparisons between triplicate measurements (run 1, 2 and 3) for all 151 samples. Solid is an identity line indicating perfect concordance between each triplicate. Cells with mean concordance correlation coefficient (CCC) below 0.90 are highlighted.

Table 1. Descriptive Statistics for the 151 samples run on 3 separate occasions by each laboratory.

Allergen	Statistic	Run	Lab A	Lab B	Lab C	Lab D	Lab E	Lab F	Lab G	Lab H	Lab I
Bla g 2 (ng/ml)	Mean*	1	7.3	11.5	10.1	15.4	12.4	10.0	21.8	10.9	9.9
		2	7.2	13.8	11.1	14.0	13.4	11.5	17.2	11.4	9.2
		3	7.6	13.3	11.1	13.4	13.2	10.9	15.8	11.4	10.3
	CV%†		6.5%	7.8%	7.8%	5.1%	4.3%	4.7%	9.1%	4.1%	7.8%
Can f 1 (ng/ml)	Mean	1	11.3	12.6	11.7	13.1	11.5	9.5	17.7	9.0	14.9
		2	12.2	13.4	13.0	13.0	13.7	11.1	18.1	11.1	15.8
		3	10.3	11.0	13.2	13.8	12.4	11.0	14.4	11.1	16.7
	CV%		4.9%	3.8%	4.4%	6.6%	6.8%	5.5%	14.4%	7.1%	7.1%
Der f 1 (ng/ml)	Mean	1	4.7	5.4	4.9	5.3	4.4	4.1	7.2	5.0	4.9
		2	4.8	6.1	5.2	5.5	4.6	4.8	6.6	5.3	4.8
		3	4.4	5.9	5.3	5.6	4.6	4.7	6.1	5.2	5.5
	CV%		3.9%	5.3%	5.5%	8.8%	7.2%	6.2%	12.5%	11.6%	9.3%
Der p 1 (ng/ml)	Mean	1	5.4	6.1	5.8	6.6	5.3	4.6	8.8	5.6	5.9
		2	5.4	6.4	6.2	7.2	5.9	5.4	7.8	5.0	6.5
		3	5.1	6.3	6.5	7.2	5.8	3.5	7.0	5.4	6.5
	CV%		4.6%	5.4%	5.7%	9.4%	5.5%	12.1%	11.1%	11.6%	7.3%
Fel d 1 (ng/ml)	Mean	1	11.4	12.5	11.2	13.6	12.5	8.2	14.1	9.3	12.1
		2	11.2	14.6	12.4	13.8	14.4	10.0	14.3	10.4	17.9
		3	9.9	11.2	12.5	13.8	13.6	10.0	11.7	11.3	21.8
	CV%		5.8%	9.4%	6.7%	12.3%	8.5%	6.0%	13.7%	9.9%	17.3%
Mite Gr 2 (ng/ml)	Mean	1	7.2	8.6	8.7	6.8	7.6	5.0	11.9	8.6	9.4
		2	7.2	9.1	9.6	7.2	8.6	5.8	10.9	7.2	10.6
		3	6.8	9.5	10.2	7.0	8.7	5.5	9.1	6.6	10.6
	CV%		4.6%	10.4%	6.3%	8.6%	7.8%	7.8%	11.3%	14.0%	9.2%
Mus m 1 (ng/ml)	Mean	1	0.9	1.0	1.0	1.0	1.0	0.8	1.4	0.8	1.0
		2	1.0	1.1	1.1	1.1	1.1	0.9	1.3	0.9	1.1
		3	1.0	1.0	1.1	1.1	1.0	0.8	1.1	1.0	1.1
	CV%		3.5%	4.1%	6.0%	6.4%	5.4%	4.1%	11.2%	7.2%	5.8%
Rat n 1 (ng/ml)	Mean	1	0.9	1.7	1.2	1.6	1.0	1.0	1.6	1.0	1.2
		2	0.9	1.5	1.3	1.5	1.2	0.9	1.5	0.9	1.1
		3	0.9	1.6	1.4	1.5	1.1	0.9	1.1	1.0	1.1
	CV%‡		0.3%	7.1%	7.9%	3.2%	5.0%	3.2%	21.6%	4.4%	10.1%

^*Geometric Mean

^†The coefficient of variation was calculated as the mean of the standard deviation to the mean of the 151 samples.

^‡The coefficient of variation for Rat n 1 should be interpreted with caution given the low number of specimens above the level of detection.

3.3 Reproducibility between study sites

Inter-laboratory reproducibility was evaluated using concordance correlations. Median CCC between laboratories tended to be in the order of 0.95 and displayed narrow ranges around the median (Figure 2, Figure A.1). A wider than usual range of correlations was observed for Bla g 2. The effect was apparent between study sites with long versus short or no transit times during shipping and lead to discrepant positive vs. negative classification in Bla g 2 levels close to the lower limit of detection. The coordinating center recreated this effect experimentally by exposing Bla g 2 reagents to temperatures outside the recommended storage conditions overnight, which demonstrated that both accidental freezing (−20°C) and room temperature storage may cause the observed low level discrepancies. Following these findings, shipping procedures were modified to stabilize internal package temperatures during transit. Overall means of results were comparable between study sites (Table 1), but revealed a slight systematic difference of results for Lab G compared to all other sites. Analysis of median CVs between study sites by allergen showed that results between laboratories were in good agreement, with mean CVs between 17.6% and 26.6% (Figure 3). A higher CV of 36.3% was observed for Rat n 1, but was not statistically significant due to small number of samples above LLOD.

Figure 2.

Mean inter-laboratory agreement (CCC) between every pairwise laboratory.

Figure 3. Inter-Laboratory precision: Median CVs between study sites by allergen. Note: Rat n 1= n.s., due to small total specimen number (n=14).

4 Discussion

The aim of our study was to evaluate intra- and inter-laboratory reproducibility of MARIA using an international multi-center ring trial. Our results showed high levels of reproducibility both within and between laboratories for all allergens. While levels of reproducibility were high for all study sites (80% of intra-laboratory results fell within a 10% CV margin), the assay precision was dependent on the operator. Potential contributors to variability were identified during the study, principally: 1) Inter-laboratory variations in results for low Bla g 2 levels led to the identification of a storage/shipping temperature sensitivity issue of the Bla g 2 reagents. The issue has since been addressed by modifying packaging materials and procedures during shipping. 2) Systematic differences in results were observed for Lab G, which may partly be attributable to this study site having mistakenly stored their dust extracts under refrigeration but not frozen for three months prior to analysis. 3) The minor differences observed in reproducibility within laboratories point towards operator-dependent variability in precision that might be expected. Not all laboratories had access to the same data handling software on their multiplex instruments. However, curve fitting parameters were identical for all study sites and no systematic effect of the data handling software on the comparability of data between laboratories was identified. Overall, this study showed that results produced by MARIA were reproducible both within and between laboratories with inter-laboratory CVs below 30%.

Consistent procedures for handling and interpretation of results are important to ensure reproducibility of results. This was particularly the case for Der f 1 results, where a small percentage of samples (<5%) appeared to demonstrate non-parallelism between results of different sample dilutions. This effect is due to a low level background signal inherent to the matrix of certain samples, which is mathematically amplified by the data handling software during calculation of increasing sample dilutions. The matrix effect is addressed by examining the appropriate decrease of fluorescent signal intensity in relation to increasing sample dilutions in the raw data output (Table A.4).

The present study design intentionally excluded variability introduced by dust sample handling and extraction, which may be considered a limitation. The focus on reproducibility of the immunoassay itself, however, provides an opportunity to more clearly identify potential contributors to assay variability. Another potential limitation lies in the small number of samples with detectable levels of rat allergen, which weakened the statistical validity of results for this analyte.

Few peer-reviewed data are available on the intra- and inter-laboratory variability of allergen analysis, and currently no specific laboratory accreditation or proficiency testing program exists. The necessity for more standardized laboratory procedures, larger studies of reproducibility and proficiency testing programs has been previously pointed out by other authors (Codina and Lockey, 2007; Pate et al., 2005). MARIA reproducibility has been shown to be significantly enhanced over previously reported studies involving the ELISA.

Considering the current absence of formal accreditation programs, MARIA performance has been closely monitored at the coordinating center over the past three years. The ISO 17025-compliant quality control program involves routine sample duplicate and quality control sample analyses using Westgard rules. This internal quality control program has since confirmed the reproducibility reported in this study (King et al., 2010).

While fluorescent multiplex array technology has been used for numerous biomarkers for more than ten years (Kellar and Lannone, 2002), few systematic, peer-reviewed trials of inter-laboratory reproducibility have been published for any commercial multiplex kit. Several studies investigated intra-laboratory performance of multiplex assays and compared commercial multiplex kits between manufacturers or with other detection methods (Wong et al., 2008; Djoba Siawaya et al., 2008; Lewczuk et al., 2008; Johnson et al., 2007). However, few studies of intra-and inter-laboratory performance of multiplex technology have been published. One study (Fichorova et al., 2008) investigated biological and technical variables affecting immunoassay recovery of IL-1ß and IL-6 in spiked samples using various immunoassays, including a commercial multiplex cytokine kit. The study included 12 laboratories and concluded that a 6- fold concentration difference was required to detect differences between laboratories. Our study has shown significantly higher levels of reproducibility of the allergen detection technology, and to our knowledge, represents the most comprehensive multi-center study of fluorescent multiplex array technology performed to date.

The U.S. National Heart Lung and Blood Institute's Expert Panel Report on Asthma Education and Prevention (2007) significantly strengthened guidelines recommending allergen avoidance as an important goal of asthma management. Educating homeowners and patients about allergens present in their environment is an important step towards more effective management of allergic disease. Multiplex testing of the most relevant household allergens in a single assay analysis is a cost-effective approach towards achieving this goal. It provides a productive starting point for successful avoidance and intervention. Validated procedures have been developed that can reduce exposure to allergens, and reduce symptom scores and medication use (Krieger et al., 2010; Tovey and Marks, 2011).

The results indicate that MARIA has the potential to significantly improve the reproducibility of environmental allergen detection to the benefit of allergic patients.

Abbreviations

MARIA

Multiplex array for indoor allergens

ELISA

Enzyme-linked immunosorbent assay

CV

Coefficient of variation

CCC

Concordance Correlation Coefficient

LLOD

Lower limit of detection

PBS

Phosphate buffered saline

NAEPP

National Asthma Education and Prevention Program