Impact of Analytical Variability on Clinical Interpretation of Multiplex Pneumococcal Serology Assays

Abstract

The response to pneumococcal vaccination can be used to assess a patient's humoral immune response to polysaccharide antigens. Multiplex assays measuring serotype-specific levels of pneumococcal antibodies are often used for this purpose, and clinical algorithms have been published to assist in the definition of an adequate immune response. We evaluated whether interlaboratory variability in multiplex pneumococcal serology assays would affect the clinical classification of the immune response. Specimens from 57 patients were analyzed at three reference laboratories with different multiplex assays to measure pneumococcal serology. Analytical correlation and clinical agreement in the classification of a patient's vaccination status by the three methods were compared. Although substantial variation in the quantitative antibody levels measured by different laboratories was seen, the qualitative classification of individual serologic results showed a high degree of agreement between labs and the ultimate classification of a patient as “protected” or “nonprotected” was the same for most patients. The majority of discordant classifications were driven by a systematic bias in results from one of the assays rather than by random error. These data suggest that the use of integrated assessments based on multiple serotypes can compensate for much of the analytical variability seen between laboratories. Knowledge of the analytical performance characteristics of a particular assay is most important when evaluating patients with results near clinical cut points.

INTRODUCTION

Pneumococcal serology can be used to measure the vaccination response to polysaccharide antigens in patients who are being evaluated for defects in humoral immunity. Pneumococcal vaccines such as Pneumovax 23 and Prevnar 13 contain a mixture of antigens from multiple pneumococcal serotypes to provide coverage against a broad range of commonly encountered strains (1). The earliest measurements of serotype-specific antibodies were performed with individual enzyme-linked immunosorbent assays (ELISAs) for different serotypes (2). However, the increasing availability of multiplex immunoassay techniques has led to the development of assays that can provide quantitative, serotype-specific measurements of multiple antipneumococcal antibodies in a single reaction (3, 4). Although this level of detail was initially used primarily for research into the nature of the immune response to pneumococcal vaccines (5, 6), multiplex pneumococcal serology assays are increasingly being measured in clinical practice to assess a patient's response to pneumococcal vaccination.

Clinical interpretation of pneumococcal serology assay results can be complex, and several guidelines have been proposed to define an adequate vaccination response when using serotype-specific measurements (7–10). Although the details of these proposals vary, most approaches incorporate two primary factors, the level of the antibody response achieved against a given pneumococcal serotype and the number of serotypes for which this level is reached. A combination of these factors is then used to determine if the patient has achieved an adequate response to vaccination (10). A key parameter in such approaches is the definition of the clinically relevant cutoff that indicates an adequate antibody response. One strategy is to define a “protective” level of antibody that has been associated with reduced rates of a given clinical outcome on the basis of epidemiologic data. In adult patients, a level of 1.3 μg/ml has been proposed as a relevant cutoff on the basis of correlation with a reduced likelihood of primary infection and pneumococcal colonization in early studies (10, 11). A lower cut point of 0.35 μg/ml is often cited for pediatric patients on the basis of studies correlating this level with the prevention of invasive pneumococcal disease in infants (12, 13).

The earliest multiplex assay for measuring pneumococcal serology was developed on a bead-based flow cytometric assay platform (14), and most currently available clinical assays are based on this approach. At present, there is only one FDA-cleared in vitro diagnostic (IVD) kit for multiplex pneumococcal serology assay available in the United States. This assay measures antibodies against 14 pneumococcal serotypes (xMAP Pneumo14; Luminex Corporation, Austin, TX). In addition to the IVD assay, several commercial reference laboratories have developed laboratory-developed tests (LDTs) that are also based on the Luminex platform and use reagents and assay conditions developed and validated by the individual laboratories under Clinical Laboratory Improvement Amendments regulations. A recent interlaboratory comparison evaluated the performance of three multiplex pneumococcal assays with a small series of reference standards provided by the WHO (15). The three assays showed 42 to 55% agreement with the WHO-assigned values, with various levels of correlation between serotypes. However, in practical terms, it was not clear whether the level of variation seen was large enough to have an impact on the clinical use of these results when evaluating vaccine responses. To address this issue, we performed multiplex pneumococcal serology testing of a large series of clinical samples to determine whether interlaboratory variability could impact the clinical classification of patients when using published clinical algorithms.

MATERIALS AND METHODS

Sample acquisition.

Specimens were obtained from residual portions of clinical test material sent to the laboratory for multiplex pneumococcal serology assay. Initial sample selection was performed without knowledge of patient clinical history or vaccination status. Samples were acquired from a total of 57 patients (generating 741 individual serology assay results per laboratory) for use in the final validation set. The majority of these specimens were single, unpaired specimens from adult patients, reflecting the population that our laboratory serves. All samples were collected according to local Institutional Review Board policies governing the use of residual material for assay validation.

Pneumococcal serology testing.

Samples were tested with three different multiplex assays for the measurement of serotype-specific pneumococcal IgG antibodies. Samples were tested on site at the Cleveland Clinic with the xMAP Pneumo14 pneumococcal immunity panel (Luminex Corporation, Austin, TX). This IVD assay detects antibodies against 14 pneumococcal serotypes, i.e., 1, 3, 4, 6B, 7F, 8, 9N, 9V, 12F, 14, 18C, 19A, 19F, and 23F. Standard values for the assay are calibrated to FDA-89 reference serum, and assay buffers incorporate cell wall polysaccharide and polysaccharide type 22 as blocking agents. The performance characteristics of this assay have been reported previously (14, 16, 17). In addition, aliquots of each sample were sent in a blinded fashion to two commercial reference laboratories that perform multiplex pneumococcal serologic testing. Both reference laboratories use LDT assays based on the Luminex platform, but the specific analytical details of these assays have not been published. Both LDT assays A and B measure antibodies against the following 14 serotypes, 1, 3, 4, 5, 6B, 7F, 8, 9N, 9V, 12F, 14, 18C, 19F, and 23F, which differ from the IVD assay by one serotype, substituting serotype 19A for serotype 5.

Statistical analysis.

Method comparisons and correlation statistics for quantitative results were calculated with EP Evaluator software (release 9). Comparison of population medians between assays was performed by one-way analysis of variance (ANOVA) on ranks with SigmaPlot 11.2 software. For serotypes where significant differences were identified by ANOVA, pairwise multiple-comparison testing (Tukey's test) was performed to identify the group that differed from the others.

RESULTS

Interlaboratory variation in quantitative results.

We initially compared the absolute levels of pneumococcal antibodies measured for each serotype in order to determine agreement in quantitative results between laboratories. A total of 2,223 serotype results were generated from the 57 samples in the test set. The 13 serotypes that were common to all of the assays were evaluated independently, and correlation statistics were calculated for all of the possible pairwise combinations of the assays. Substantial variation between laboratories was seen, with only 61% of the comparisons achieving r values of >0.8 (Table 1). However, the degree of variation differed widely between serotypes. While some serotypes showed good correlation across all of the laboratories, others showed very little agreement between assays (Fig. 1). In addition to this sporadic serotype-specific variability, a systematic negative bias was seen for LDT B relative to the other two assays across multiple serotypes (Table 1). This bias was most evident when comparing the median values for each serotype in the sample set across the three assays, where the values generated by LDT B were lower for most of the serotypes (Fig. 2).

Table 1.

Interassay correlation statistics^a

Serotype	IVD vs LDT A				IVD vs LDT B				LDT A vs LDT B
	r	Slope	Intercept	Bias	r	Slope	Intercept	Bias	r	Slope	Intercept	Bias
1	0.73	2.12	−1.72	1.55	0.57	0.49	1.36	−0.12	0.46	0.19	1.95	−1.67
3	0.71	0.41	0.78	−1.43	0.56	0.14	0.55	−2.49	0.70	0.36	0.23	−1.21
4	0.93	1.31	−0.01	0.35	0.98	0.73	−0.06	−0.38	0.94	0.55	−0.05	−0.72
6B	0.89	0.62	0.65	−1.20	0.90	0.44	0.74	−2.44	0.85	0.95	−0.24	−0.42
7F	0.83	2.19	−2.29	1.99	0.85	0.64	−0.60	−1.90	0.79	0.36	0.17	−3.99
8	0.88	1.12	−0.39	−0.01	0.65	0.46	0.37	−1.80	0.76	0.75	−0.52	−1.42
9N	0.89	1.26	−0.74	0.12	0.93	0.74	−0.54	−1.38	0.91	0.59	−0.11	−1.49
9V	0.71	1.52	−0.88	0.84	0.93	0.63	−0.23	−1.45	0.95	0.37	0.12	−2.44
12F	0.92	0.29	0.88	−1.58	0.98	0.25	0.22	−2.52	0.73	0.51	−0.12	−1.35
14	0.74	0.95	−0.71	−1.04	0.84	0.53	−0.26	−3.43	0.80	0.54	0.26	−2.39
18C	0.79	3.30	−3.25	6.62	0.90	0.90	−1.10	−1.55	0.60	0.21	0.43	−8.17
19F	0.39	4.97	−16.98	3.44	0.53	0.77	−0.82	−2.02	0.91	0.35	0.10	−5.46
23	0.90	1.12	0.03	0.34	0.85	0.68	0.15	−0.67	0.92	0.63	0.08	−1.01

^aAll possible pairwise comparisons of the three assays are shown for each serotype. Regression statistics were calculated by Deming analysis, while bias is presented as the average absolute bias for each comparison.

Fig 1.

Quantitative correlation of representative serotypes. Examples of serotypes showing good (serotype 4, r > 0.9) and poor (serotype 19F, r < 0.6) correlations between assays are shown.

Fig 2.

Comparison of pneumococcal antibody levels. For each assay, the median value of the 57 samples tested is shown for each serotype. The results for serotypes marked with an asterisk were statistically significantly different (P < 0.05) from the median results of the other assays.

Interlaboratory variation in qualitative serotype classification.

In order to determine whether the variations in pneumococcal antibody measurements would impact clinical decision making, we examined how each assay classified serotypes as “protective” or “nonprotective” versus a clinically defined cut point. Because our sample set was composed primarily of single time point values from adult patients, we chose to use the classification scheme suggested by Paris et al., which defines an adequate response as an antibody level greater than 1.3 μg/ml (10, 18). This type of qualitative classification improved the overall agreement between laboratories, with rates of concordance near 80% for all pairwise comparisons between laboratories (Table 2). When comparing IVD and LDT A, discrepancies between the two assays were fairly evenly distributed in terms of which assay fell above the “protective” cutoff. In contrast, the results of comparisons involving LDT B were more skewed, with 93% of the discrepancies involving samples where LDT B gave a result below the cutoff. This is consistent with the systematic negative bias of the LDT B results seen in the quantitative comparison.

Table 2.

Interassay agreement in qualitatitive classification of serotypes^a

Serotype	IVD vs LDT Ab				IVD vs LDT Bb				LDT A vs LDT Bb
	Agree	IVD+	LDT A+	% Agreement	Agree	IVD+	LDT B+	% Agreement	Agree	LDT A+	LDT B+	% Agreement
1	49	3	5	86	45	4	8	79	51	2	4	89
3	48	6	3	84	42	15	0	74	45	11	1	79
4	45	3	9	79	51	6	0	89	47	10	0	82
6B	49	6	2	86	50	7	0	88	51	4	2	89
7F	48	7	2	84	35	22	0	61	41	16	0	72
8	49	6	2	86	46	11	0	81	48	9	0	84
9N	55	1	1	96	53	4	0	93	53	4	0	93
9V	49	2	6	86	49	8	0	86	41	15	1	72
12F	36	11	10	63	42	15	0	74	42	13	2	74
14	50	6	1	88	42	15	0	74	49	8	0	86
18C	49	1	7	86	48	9	0	84	41	16	0	72
19F	44	7	6	77	34	22	1	60	35	22	0	61
23	54	2	1	95	50	6	1	88	50	6	1	88
Total	625	61	55	84	587	144	10	79	594	136	11	80

^aSerotype results were classified as protective or nonprotective for each sample by using a cut point of 1.3 μg/ml. Results that were classified identically by both assays are listed under “Agree,” while samples classified as protective by one assay and nonprotective by the other are listed under the assay giving the protective result (IVD⁺, LDT A⁺, or LDT B⁺).

^bEach value is the number of samples unless otherwise specified.

Interlaboratory variation in the clinical classification of patients as “protected” versus “nonprotected.”

Because the interpretation of a patient's vaccination status incorporates information from multiple serotypes, we evaluated whether variations in serotype classifications between assays would impact the final categorization of a patient's immune response. On the basis of the previously referenced criteria, we classified a patient as “protected” if 70% of the serotypes (9/13) achieved a level of >1.3 μg/ml. By these criteria, 82% of the patients (47/57) were identically classified by all three assays, with 96% agreement (55/57) between IVD and LDT A. The majority of the discordant results were cases where patients were classified as nonprotected by LDT B and protected by the other two assays (Fig. 3).

Fig 3.

Classification of vaccination statuses of individual patients. Serotypes with protective results are green, while nonprotective ones are red. Each row represents results from an individual patient. Patients below the horizontal bar have nine or more positive serotypes and were classified as protected by the consensus of the three assays. The 10 patients with discrepant classifications are outlined in blue.

DISCUSSION

Multiplex pneumococcal serology assays are among the most analytically complex assays used in clinical practice, and the use of differing methods introduces the potential for analytical variability between laboratories. In this study, we evaluated whether this variability is significant enough to impact the final clinical interpretation when assessing vaccination responses. Substantial variation was seen in the absolute values of pneumococcal antibodies reported by the three different assays. Both serotype- and assay-specific variations were observed. The underlying causes of the differences are not clear but may relate to differences in the specific reagent formulations and calibration materials used by the different laboratories to validate the assays. A more consistent approach to the standardization of pneumococcal antibody assays could potentially help to reduce these differences. Detailed recommendations in this regard have previously been published by the WHO for the standardization of ELISAs (www.vaccine.uab.edu), and reference standards are available for this purpose.

Despite the variability seen in the reported quantitative results of the various assays, the overall classifications of patients' pneumococcal immune status were remarkably similar between assays. This appears to be due primarily to two factors. First, although pneumococcal serology assays provide quantitative results, many clinical algorithms use these results in a qualitative fashion by converting results into “protective” or “nonprotective” on the basis of a clinically defined cut point. The data shown here suggest that interpreting results in this manner helps to mitigate much of the analytical variability seen between assays. In addition, the evaluation of multiple serotypes limits the impact that any one serotype has on the overall classification of the patient status. Because multiple serotypes are considered in the final classification, isolated differences in individual serotypes are usually offset by the larger agreement across the remaining serotypes, leading to an overall concordance in the classification.

LDT B provided an interesting illustration of these benefits. Despite the systematic negative bias seen in this assay, results from LDT B still agreed with the consensus classification for >80% of the patient samples. In addition, most of the discordant samples were from patients given an overall classification of “nonprotective” by LDT B but achieving protective levels for eight serotypes (narrowly missing the nine required for 70%). Clinicians should be aware of the specific characteristics of the assay they are using, as an awareness that an assay tends to give lower results might lead them to give further consideration to patients with “borderline” results.

A potential limitation of this study is that we did not examine the impact of analytical variability on algorithms that use the difference between pre- and postvaccination levels to define a successful immune response (19), which many authors have suggested is the most sensitive way to evaluate an immune response to pneumococcal variation (9). Unfortunately, such samples are uncommon in the patient population we serve, as less than 5% of the samples our laboratory receives are paired specimens (data not shown). One would hypothesize that the use of paired samples might further improve the concordance between labs, since comparison of pre- and postvaccination results would reduce the significance of the actual quantitative values. However, this needs to be examined further for individual assays.

The goal of any diagnostic test is to maximize analytical accuracy and precision to levels sufficient for clinical use, and multiplex pneumococcal serology assays have a high level of inherent complexity, which could potentially affect the clinical interpretation of results. This study suggests that although individual and systematic bias between assays does exist, this variability has a relatively minor impact on the final classification of patients as “protected” or “nonprotected” when using diagnostic algorithms based on the qualitative comparison of serotypes against a predefined threshold. Clinicians should be most aware of the potential for discordant results for patients with values near the clinical cut point or patients with an intermediate number of protective serotypes. Ultimately, improved assay standardization traceable to qualified reference materials could help minimize this issue, particularly for laboratories using LDTs.