Evaluation of Commercial Assays for Single-point Diagnosis of Pertussis in the US

Abstract

Background:

Pertussis serodiagnosis is increasingly being used in the US despite the lack of an FDA approved, commercially-available assay. To better understand the utility of these assays in diagnosing pertussis, serology assays were evaluated for analytical parameters and clinical accuracy.

Methods:

Forty-three antigen-antibody combinations were evaluated for single-point diagnosis of pertussis. Serum panels included sera from laboratory-confirmed cases, an international reference standard, and healthy donors. Phase I panel (n=20) of sera was used to assess precision, linearity, and accuracy; phase II panel (n=226) followed with positive (PPA) and negative percent agreement (NPA) estimates. Analytical analyses included coefficients of variation (CV) and concordance correlation coefficients (r_c).

Results:

Intra-analyst variability was found to be relatively low among samples per assay, with only 6% (78/1240) having CV > 20%, primarily with the highly concentrated IgG anti-pertussis toxin (PT) specimens and IgM assays. r_c measurements to assess linearity ranged between 0.282–0.994, 0.332–0.999, and −0.056–0.482 for IgA, IgG, and IgM, respectively. Analytical accuracy for calibrated IgG anti-PT assays was 86–115%. PPA and NPA varied greatly for all assays; PPA/NPA ranges for IgA, IgG, and IgM assays, with culture and/or PCR-positivity as control, were 29–90/13–100, 26–96/27–100, and 0–73/42–100, respectively. In IgG assays, mixing filamentous hemagglutinin (FHA) antigen with PT increased PPA, but decreased NPA.

Conclusions:

Seroassays varied substantially under both analytical and clinical parameters; however, those that were calibrated to a reference standard were highly accurate. Our findings support incorporation of calibrated pertussis seroassays to the pertussis case definition for improved diagnosis and surveillance.

INTRODUCTION

Pertussis, caused by the bacterium Bordetella pertussis, has been on the rise in the US since the early 2000s, with cases now reaching record-breaking numbers last observed in the 1950s [1]. Outbreaks have occurred in multiple states throughout the country and both California and Washington experienced state-wide epidemics [2, 3]. Reporting of confirmed and suspect cases is based on clinical data and laboratory confirmation. Currently, the Council of State and Territorial Epidemiologists (CSTE) case definition for a pertussis laboratory-confirmed case only includes positive results from bacterial isolation and polymerase chain reaction (PCR)-based assays [4]. Only the state of Massachusetts uses their in-house serology assay as part of their definition for reporting pertussis cases for adolescents and adults with over two weeks of cough [5].

Ongoing surveillance indicates that, regardless of exclusion from the CSTE case definition [4], serology is used in every state for pertussis diagnosis [6]. However, there are currently no serological assays FDA approved for pertussis diagnosis. Much is unknown about the serological assays being used for diagnostic purposes. The assays are either laboratory developed tests (LDT) not available for distribution or are sold commercially for research purposes only with little information on their analytical and clinical performance. These assays vary between targets, antibody response, assay conditions, and whether results are qualitative or quantitative.

Serology has been shown to be beneficial for pertussis case confirmation, and has been supported in many public health recommendations world-wide [7–9]. However, obtaining acute and convalescent specimens can be logistically difficult and cost prohibitive, leaving many cases unconfirmed. Furthermore, patients often present too late to obtain an acute serum specimen. Therefore, many countries have moved toward single-point diagnostic assays that required one serum sample obtained from a patient during a specified window of time following illness onset [8–13].

A serology LDT developed and analytically validated in a collaboration between FDA and CDC is currently being used by the CDC’s Pertussis and Diphtheria Lab (PDL) as a critical component of their multi-test algorithm, which also includes culture and real-time PCR, for the diagnosis of pertussis. It is an IgG anti-pertussis toxin (PT) enzyme-linked immunosorbent assay (ELISA) [14] with diagnostic cut-offs proposed at 49 and 94 International Units (IU)/mL [15] for indeterminate and positive interpretations, respectively. The assay is ideal for retrospective diagnosis for adolescents and adults, as a single-point assay, when the timing, recommended as two through eight weeks following cough onset, is too late for consistently accurate diagnosis by culture or PCR [10, 16]. Public Health Laboratories are hesitant to routinely adopt similar serology assays for pertussis diagnosis because of potential burden related to case investigations and treatment and, most importantly, the absence of serology in the current CSTE definition.

Our objective was to assess the analytic and clinical accuracy and validity of pertussis serodiagnostic assays in the US with an overall goal of harmonizing diagnostic methods in the US and, ultimately, world-wide [13]. A two-phase approach was followed for this validation study. Initially, a pilot test was run with a small panel of serum specimens collected from de-identified culture-positive or negative individuals for analysis of analytical parameters: precision, linearity, and accuracy. In the second phase, a larger panel was tested to calculate positive (PPA) and negative percent agreement (NPA).

MATERIALS & METHODS

Study Protocol

Phase I assessed analytical parameters, including linearity, precision, and percent accuracy. Four plates were run per commercial assay, with two plates run per analyst (total= 2 analysts). The four assays were run on the same day. To assess intra-plate variability, two dilutions per sample were run on each plate. The microsphere-based assay was performed by one analyst with one assay per day and two sample dilutions per assay over four days.

Phase II assessed clinical accuracy; positive and negative percent agreement were measured. In this phase, the serum panel was tested once per assay.

Assay Testing

Thirty-five total assays, including the CDC LDT [14], were assessed representing 43 different antigen-antibody combinations, including IgG, IgA, or IgM antibodies against filamentous hemagglutinin (FHA) only, PT only, both PT and FHA, or whole cell lysates (Table 1). All but one was performed at CDC; this remaining assay was a microsphere-based, multi-plex assay that was performed at Focus Diagnostics (Cypress, CA) which offers four separate measurements of IgG or IgA antibodies against PT or FHA [17].

Table 1.

Characteristics of 35 analyzed serological assays stratified by antigen/antibody combinations (n=43).

Antigen*	Immunoglobulin type (Ig)	Commercial assays**	FOCUS assay†	CDC assay	Antigen/antibody combinations
FHA only	A	1	1	.	2
	G	1	1	.	2
PT only	A	6	1	.	7
	G	9	1	1	11
	M	1	.	.	1
PT and FHA	A	4	.	.	4
	G	4	.	.	4
	M	1	.	.	1
WC	A	3	.	.	3
	G	3	.	.	3
	M	5	.	.	5
Total		38	4	1	43

^*Antigens: PT=pertussis toxin; FHA=filamentous hemagglutinin; WC=whole cell lysate

^**11 total commercial companies provided 33 ELISAs. All commercial assays were tested at CDC.

^†The Focus Diagnostics serological assay is a microsphere-based, multi-analyte immune detection system that individually measures IgG or IgA antibodies against PT or FHA. Testing was completed at Focus Diagnostics.

Phase I included 25 ELISAs, along with the Focus and CDC assays. Manufacturers included Ammunolab/MP Biomedicals (Santa Ana, CA), Euro Diagnostica (Malmö, Sweden), Euroimmun US (Morris Plains, NJ), IBL-America (Minneapolis, MN), Novum DRG Branch Labs (Springfield, NJ), R-Biopharm (Darmstadt, Germany), Savyon Diagnostics (Ashdod, Israel), Genzyme/Sekisui Virotech (Rüsselsheim, Germany), and Virion\Serion (Würzburg, Germany). During Phase II, testing included 33 assays alongside the Focus and CDC assays. Manufacturers included those in Phase I plus IBL International (Toronto, Canada) and Statens Serum Institute (Copenhagen, Denmark).

For both phases, assays were performed according to the manufacturer’s instructions, following quality control guidelines provided by the manuals. Interpretations of the results were based on the information provided by the enclosed manual for each kit. Assay cut-offs for negative, intermediate, and positive results were typically given and varied greatly between assays. If additional cut-offs were also provided, such as those that recommended 40 IU/mL for intermediate and 100 IU/mL for positive [18], then the sample interpretations were also analyzed against these cut-offs. Focus Diagnostics followed their clinical cut-offs of PT IgG <45 IU/mL, PT IgA <10 IU/mL, FHA IgG <90 IU/mL, and FHA IgA <50 IU/mL [17].

Serum Panels

Human sera and plasma were collected previously as either part of public health response or from studies to develop and evaluate pertussis diagnostics. These studies were approved by the necessary institutional review board (IRB) committees. The Phase I panel consisted of 20 serum specimens: five from a two-fold dilution series of a culture-positive case, 10 from culture-positive cases, four from healthy donors, and the WHO International Reference Standard (WHO IS) 06/140 (NIBSC, UK) [19]. The sera of culture-positive cases were collected at 4–6 weeks after cough onset. The Phase II panel consisted of 226 blood specimens: 70 from culture and/or PCR-positive symptomatic patients with appropriately-timed nasopharyngeal (NP) specimens (i.e. NP specimens that were collected at the optimal time for capturing culture and/or PCR positivity, 0–4 weeks after cough onset); 130 from culture and/or PCR-negative symptomatic patients with appropriately-timed NP specimens; and 26 from healthy donors. Four specimens were plasma; the remaining specimens were serum. Timing of case patient serum specimens included acute and/or convalescent specimens, with a range of 4–55 days of cough (mean and median=27 days) for 68/70 culture and/or PCR-positive symptomatic patients that had cough onset dates available, and a range of 5–41 days of cough (mean and median=17 days) for all 130 culture and/or PCR-negative symptomatic patients. Both healthy donors and patients were not previously vaccinated in the last year before blood collection.

Statistical Analyses and Study Exclusions

In Phase I, precision for each assay was calculated for within and between-analyst variability, stratified by analyst, immunoglobulin type, and antibody concentration (specifically, IgG anti-PT). For linearity, a two-fold dilution series of positive sera was made and concordance correlation coefficients (r_c) were measured to determine the level of agreement between the measured and expected concentrations for each of the dilutions. Expected values for each of the diluted samples in the dilution series originated from the most concentrated sample that could give a result by each assay. Once determined, each following dilution was then calculated accordingly for each given assay. An average value was calculated from the 16 replicates for each dilution. The available IgG anti-PT assays that were calibrated to either CBER lot3 or WHO IS 06/140 were assessed for analytical accuracy by measuring the percent difference between the observed and expected values of the blinded undiluted specimen of WHO IS 06/140 in the Phase I panel (335 IU/mL). Some assays were not assessed as they were not available at the time of Phase I testing or the upper limit of quantification did not reach 335 IU/mL, the IgG anti-PT value of the WHO IS 06/140.

In Phase II, diagnostic interpretations were considered based on the assay’s provided cut-offs and/or any additional clinical cut-offs described in the manual, when applicable. The 26 healthy controls were analyzed as either part of the negative clinical sera or were excluded from analysis. Due to the lack of a true reference standard for diagnosing pertussis, either appropriately-timed culture and/or PCR positivity or the CDC ELISA results were considered the non-reference standard, and the measurements of agreement between culture and/or PCR or CDC ELISA results with each test assay were henceforth considered the positive (PPA) and negative percent agreement (NPA) estimates for each test assay [20]. In the absence of a gold reference standard, these measurements assess the level of agreement between the comparison test and the non-reference standard and are informative in a manner similar to measuring percent sensitivity and specificity. Intermediate interpretations were analyzed in three ways: excluded from the analysis (data not shown), considered positive (Intermediate_positive), or considered negative (Intermediate_negative) [20, 21].

RESULTS

Overall, 35 assays were assessed representing 43 different antigen-antibody combinations, consisting of ELISAs and a multi-analyte, micro-sphere based assay (Table 1). In Phase I, analysis of precision revealed that precision, while predominantly low throughout, was highly variable (Figure 1). Between-analyst and between-plate %CV remained less than 20% for all assays; however, within-analyst %CV varied drastically between assays, suggesting that the type of assay and the analyst can greatly affect precision. When the data from Figure 1 was broken down and stratified by specimen, analyst, and assay type, to analyze precision, a few patterns emerged (Table 2). Seventy-eight out of 1240 (6.3%) potential %CV values, from 17 different assays, had variability >20% for the two analysts. While the total number of values was low for both analysts, Analyst 1 had a higher number of %CV values than Analyst 2 (49 vs. 29), indicating that Analyst 1 did not perform as well as Analyst 2 during this testing (Table 2). This finding suggests that variability can be user-dependent. Additionally, assays that measured IgM had a much higher percentage of variability than those that measured either IgG or IgA. Finally, samples that contained a higher concentration of antibodies appeared to also give higher variability.

Figure 1.

Phase I precision measurements between-analyst, between-plate, and within-analysts. Precision was capped at 40% to allow for better differentiation of between-plate and between-analyst measurements at the lower percentages. Assays included: Ammunolab/MP Biomedicals (AL), Eurodiagnostica (TRO), Euroimmun (EI), Genzyme/Sekisui Virotech (GEN), IBL America (IBL), Novum DRG Branch Labs (NOV), R-Biopharm (BIO), Savyon Diagnostics (SAV), Virion\Serion (VIR), and CDC. P=Pertussis toxin (PT) antigen only; F=Filamentous hemagglutinin (FHA) antigen only; PF=PT and FHA as mixed antigens; WC=whole cell

Table 2.

Precision measurements for intra-analyst variability. Number (%) of values with precision >20% for the 26 assays tested at CDC during Phase I testing, stratified by analyst, immunoglobulin type, and antibody concentration (n=1240 values). Highest variability was observed by analyst 1, with IgM assays, and with higher IgG anti-pertussis toxin (PT) concentrations.

Immunoglobulin type	IgG anti-PT concentration (IU/mL)	# (%) of values with precision >20%
		Analyst 1	Analyst 2	Both Analysts
IgA		9	2	11 (0.9)
IgG		10	12	22 (1.8)
	0–42				18 (1.5%)
	57–214				19 (1.5%)
	228–910				41 (3.3%)
IgM		30	15	45 (3.6)
Total		49 (4)	29 (2.3)	78 (6.3)*

^*For the two analysts, a total of 78 values from 17 different assays had intra-analyst variability >20%. The 1240 total values came from 31 different antigen-antibody combinations, testing 20 samples, and run by two analysts.

Concordance correlation coefficients (r_c) were measured for each assay and the ranges of r_c, stratified by immunoglobulin type, are shown in Figure 2. Those assays with r_c values closer to 1 were considered closer to perfect agreement. For both IgA and IgG, inclusion of PT, either alone or with FHA, offered the highest r_c values.

Figure 2.

Concordance correlation coefficients (r_c)* ranges, stratified by immunoglobulin and antigen type, to assess linearity for the 25 assays with quantifiable results. A 2-fold dilution series of positive sera was included as part of the Phase I panel. Expected values for each of the diluted samples in the dilution series originated from the most concentrated sample that could give a result by each assay. Once determined, each following dilution was then calculated accordingly for each given assay. An average value for each dilution was calculated from 16 replicates. Concordance correlation coefficients were measured to determine the level of agreement between the measured and expected concentrations for dilutions 1:4–1:32 (“4 Points”) and 1:8–1:32 (“3 Points”). Perfect agreement would yield an r_c of 1.

Analytical accuracy was measured to assess how well an assay could accurately measure the concentration of a specimen with known concentration. Four assays, at the time of testing, fit the criteria for inclusion in this assessment: Focus, CDC, Genzyme/Sekisui Virotech (GEN PT), and Virion\Serion (VIR PT) (Table 3). The observed results for all four assays were ≤15% different from expected values.

Table 3.

Accuracy measurements for calibrated IgG anti-pertussis toxin (PT) assays.

Assay*	Observed Concentration	Expected Concentration	% Difference**
CDC	355	335	106
FOCUS PT	384	335	115
GEN PT	30	27.5	109
VIR PT	287	335	86

^*Assessed only for the IgG anti-PT assays that are calibrated to either CBER lot3 or WHO IS 06/140. Note: some assays were not assessed as they were not available at the time of Phase I testing or the upper limit of quantification did not reach 335 IU/mL. GEN=Genzyme/Sekisui Virotech; VIR=Virion\Serion

^**Undiluted WHO IS 06/140 was included in the Phase I sera panel. Observed concentration was the average of 16 values taken from four tests run by two analysts, two dilutions per plate, and each dilution run in duplicate wells. Percent difference was calculated as observed/expected x 100 to determine analytical accuracy.

For Phase II, PPA and NPA measurements were calculated to assess agreement similar to measuring percent sensitivity and specificity, respectively. No major differences in PPA and NPA were found between the inclusion and the exclusion of the 26 healthy donors as negative sera (results for exclusion of healthy donor sera not shown). With culture and/or PCR results as the standard, both PPA and NPA varied greatly, depending on the assay (Figure 3). Overall, IgG anti-PT assays were both high in PPA and NPA. IgA assays overall had lower PPA and NPA, while IgM assays had high NPA, but very poor PPA. Results using CDC ELISA results as the standard proved to be highly similar (data not shown). Addition of FHA with PT in IgG assays appeared to increase PPA, but decreased NPA, as observed with the R-Biopharm (BIO), Genzyme/Sekisui Virotech (GEN), Savyon Diagnostics (SAV), and Virion\Serion (VIR) IgG assays.

Figure 3.

Phase II positive percent agreement (A) or negative percent agreement (B) with culture and/or PCR as the non-reference standard. Diagnostic interpretations were based on the provided assay cut-offs. Error bars represent 95% CI bounds. Assays are stratified by immunoglobulin and antigen type. Healthy donor specimens were included in the analysis as negative clinical sera. “Intermediate_positive” analysis (gray bars) considers the intermediate results as positive, while “Intermediate_negative” analysis (black bars) assigns them as negative interpretations. Assays included: Ammunolab/MP Biomedicals (AL), Eurodiagnostica (ED & TRO), Euroimmun (EI), Genzyme/Sekisui Virotech (GEN), IBL America (IBL), IBL International (IBL INT), Novum DRG Branch Labs (NOV), R-Biopharm (BIO), Savyon Diagnostics (SAV), Statens Serum Institute (SSI), Virion\Serion (VIR), CDC, and Focus Diagnostics (FOCUS). P=Pertussis toxin (PT) antigen only; F=Filamentous hemagglutinin (FHA) antigen only; PF=PT and FHA as mixed antigens; WC=whole cell

Interestingly, one particular assay, Savyon Diagnostics IgG anti-PT & FHA, provided a “positive” and “high positive” interpretation versus inclusion of an “indeterminate” or “intermediate” result. Therefore, that particular assay was analyzed in two ways, either keeping the lower positive cut-off of 10 BU as a positive interpretation or changing it to an intermediate cut-off and making the high positive cut-off of 50 BU the new positive cut-off. Analysis showed that while PPA was unaffected with the change in interpretation, NPA increased dramatically (data not shown). Evaluating the data in this way suggests that a higher cut-off for an assay that has FHA and PT mixed in the same well, where PPA is already high, may also benefit the NPA estimate.

DISCUSSION

A total of 43 antigen-antibody combinations, in ELISA or microsphere-based assays, were analyzed for analytical and clinical accuracy and utility as single-point diagnostics. Assays were highly variable, depending on precision, linearity, accuracy, and positive/negative percent agreement. Precision measurements indicated that variability of assays is affected by individual analysts, type of immunoglobulin analyzed (IgM assays more variable), or antibody concentration (highly-concentrated samples produce more variability). The high intra-analyst variability observed is not unexpected, considering the analysts had no previous experience running these specific assays. It should also be noted that the precision cut-off we applied of 20% was rather stringent. Nevertheless, these results suggests that the expertise of the analyst with serology and the specific experience of a particular assay could greatly affect results. Additionally, previous comparisons between different assays suggested that assay characteristics and sample antibody concentrations needed to be considered when assessing assay precision [22], and our findings to date continue to support this conclusion. Interestingly, no assay protocol provided information on the quality of the antigens or the sources from where they came, making it difficult to ascertain if differences observed here may also be due to antigen purity. Regardless of the assay implemented, analysts should follow proper procedures for training, competency, and proficiency and be well-versed in the methodology.

When compared to both non-reference standards, IgG anti-PT ELISAs appeared to be high in both PPA and NPA compared to IgA or IgM assays. These findings strongly suggest that IgG anti-PT ELISAs will provide acceptable clinical sensitivity and specificity as a stand-alone, single-point assay. On the contrary, the PPA and NPA values were highly variable with IgA assays, never reaching the levels of IgG assays and the PPA of IgM assays proved to be exceptionally poor, suggesting that IgM assays would not be useful in diagnosing pertussis (Figure 3). IgA assays tested previously showed similar results, further suggesting the use of IgG assays as the optimal serodiagnostic tool [12, 23]. Our results also showed that the inclusion of additional antigens in IgG assays, such as FHA with PT, increased PPA but drastically decreased NPA (Figure 3). In other findings, FHA was not found to increase positivity compared to PT alone; in fact, positivity was observed in high concentrations in the control group, suggesting non-specific cross-reactivity [24]. For serology, a tool that is ideal for retrospective diagnosis, high specificity is critical to avoid falsely-attributing unknown outbreaks and cases to pertussis, as well as unnecessary follow-up of case contacts and prophylaxis measures.

Furthermore, quantitative results for IgG anti-PT ELISAs that were calibrated to an international reference standard proved to be highly accurate, suggesting that assay results may be comparable between different assays. Previous publications have also suggested that different assays can be harmonized if assays are rigorously validated, calibrated to a reference standard, and use highly purified antigens [25, 26]. Indeed, production of these assays does appear to be increasing, as between the time of testing of Phase I and Phase II panels, the number of commercially-available, calibrated, IgG anti-PT only kits doubled, likely due to the high demand for them [8, 13, 23, 27, 28].

A valid concern does arise with the use of anti-PT ELISAs and the recommended one-time Tdap booster for adolescents and adults. Should serodiagnosis be used for a clinically-suspect adolescent or adult that was previously vaccinated with Tdap, it may be difficult to discern if the observed titers are due to infection, vaccination, or a combination of both. A previous study following healthy adult subjects who were vaccinated with Tdap for up to two years post-vaccination determined that titers did not reach the high diagnostic cut-offs for acute infection, suggesting that vaccination should not confound diagnostic results [29]. Nonetheless, waiting periods from six months to three years post-vaccination have been recommended to ensure the likelihood of decreased antibody titers [8, 9, 29].

In conclusion, the qualities of the kits that are deemed most promising for diagnosing pertussis are that they measure IgG antibodies against PT only and that they are calibrated to a reference, because these assays show 1) high PPA compared with IgA and IgM assays, 2) higher NPA compared to ELISAs with multiple coating antigens, and 3) the potential to compare results with other calibrated assays. Awareness among clinicians and public health specialists about the advantages and limitations of pertussis serodiagnostics is increasing. One can speculate that in the US, a similar harmonization of serodiagnostics will occur as it did with PCR diagnostics in the last decade [30, 31]. When PCR and bacterial isolation during the later phases of disease are severely limited, serology can be highly effective when clinically validated assays are used [9]. Our findings provide insight to manufacturers about the ideal assay characteristics for pertussis serodiagnosis, hopefully leading the way to finding clinically validated FDA approved assays on the US market in the near future. Serology, combined with sound guidance and informed decisions, may offer the complementary diagnostic tool for older, vaccinated populations, allowing public health officials to more accurately assess the burden of pertussis across the age spectrum.