Discordant performance of mpox serological assays

Abstract

Background:

Since July 23, 2022, global mpox cases reached 92,546, with over 31,000 in the United States. Asymptomatic carriage is a critical mechanism influencing the global dissemination of mpox. Seroprevalence studies are crucial for determining the epidemic’s true burden, but uncertainties persist in serologic assay performance and how smallpox vaccination may influence assay interpretation.

Objectives:

Our study aimed to assess the performance of several diagnostic assays among mpox-positive, vaccinated, and pre-outbreak negative control samples. This investigation sought to enhance our understanding and management of future mpox outbreaks.

Study Design:

Serum samples from 10 mpox-positive, five vaccinated uninfected, and 137 pre-outbreak controls were obtained for serological testing. The mpox-positive samples were obtained around 100 days post symptom onset, and vaccinated patients were sampled approximately 90 days post-vaccination. Multiple diagnostic assays were employed, including four commercial ELISAs (Abbexa, RayBioTech, FineTest, ProteoGenix) and a multiplex assay (MesoScale Diagnostics (MSD)) measuring five mpox and five smallpox antigens.

Results:

Three commercial ELISA kits had low specificity (<50%). The Proteogenix ELISA targeting the E8L antigen had a 94% sensitivity and 87% specificity. The E8L antigen on the MSD assay exhibited the greatest distinction between exposure groups, with 98% sensitivity and 93% specificity.

Conclusions:

None of the assays could distinguish between mpox-positive and vaccinated samples. The MSD assay targeting the MPXV E8L antigen demonstrated the greatest differentiation between mpox-positive and pre-outbreak negative samples. Our findings underscore the imperative to identify sensitive and specific assays to monitor population-level mpox exposure and infection.

Background

Mpox is a zoonotic virus within the Orthopoxvirus (OPXV) genus and has primarily been identified in endemic regions of Central and West Africa (1, 2). The 2022 outbreak was declared a Public Health Emergency of International Concern, marking the first multi-country virus spread, impacting over 104 countries (3). Over 92,546 cases have been reported globally, with over 31,000 in the United States (US) (4). Serologic prevalence is one way to understand the proportion of people who may have been exposed, even in the absence of clear clinical manifestations (5, 6). Serological tools capable of discriminating between OPXV species are further complicated, however, by the vaccination efforts for both smallpox and mpox (7). Given the size and geographic distribution of the 2022 outbreak, developing serologic assays specific to monkeypox virus (MPXV) is essential for diagnosing, treating, monitoring immunity to and evaluating community transmission of mpox infections, especially among vaccinated populations (6–11).

The OPXV genus has been extensively researched, particularly in the context of smallpox eradication efforts in 1980 (12). Research considering overlapping genetic and antigenic characteristics within the OPXV genus has revealed the cross-protective immunity conferred by prior OPXV infection or vaccination (12). With the cessation of smallpox vaccination in 1972, the number of people with cross-protective vaccinia-induced immunity has decreased, especially in younger generations without exposure to vaccinia virus (VACV) (13). In addition to vaccine concerns, there has been an increase in incidence of OPXV infections among different species, most notably mpox cases in humans (14).

Given the evidence of cross-protective immunity from smallpox vaccination to other OPXV, a targeted immunization program for people at the highest risk for mpox was initiated following the identification of mpox cases in non-endemic countries (13). A crucial objective in measuring population-level immunity and exposure to mpox, particularly among high-risk populations, is to identify the presence of MPXV and VACV-specific antibodies in mpox-positive or vaccinated patients. Moreover, developing and validating sensitive and specific diagnostic tools for MPXV is essential for evaluating the population-level exposure, asymptomatic carriage, and transmission of mpox (10, 11).

(10)(11, 15)(10)

Materials and Methods

Study Samples

Serum samples from symptomatic patients who were PCR-positive for mpox from the Johns Hopkins Hospital (JHH) in December 2021 in Baltimore, Maryland, were obtained as positive controls (mpox-positive). Serum samples from mpox-positive patients were deidentified and unlinked for research use.

Serum samples from healthy donors (HDs) at the JHH who received the JYNNEOS vaccine were deidentified, unlinked and included as additional positive controls (VACV-positive). Three HDs received the complete regimen of JYNNEOS, including two subcutaneous doses administered 28 days apart. Two patients received only one dose due to an adverse reaction and prior smallpox vaccination history. Given the small sample size (n=5), VACV-positives were analyzed together regardless of the number of doses received. VACV-positives had no prior exposure to mpox. The multiplex assay evaluated VACV-positives solely to assess any differentiation between VACV and MPXV antibody concentrations.

Pre-outbreak negative samples (NC) were deidentified, unlinked remnant serum samples from adult patients (>18 years) attending the JHH Emergency Department in Baltimore, Maryland in December 2021 and had blood drawn for clinical purposes (16). Demographic data was abstracted from the administrative database before sample deidentification, and the unlinked samples were stored at −70°C for later testing. NC samples were later stratified into two age groups to account for potential smallpox vaccination ending in 1972 (NC≥50, NC<50). All specimens were processed, stored, and serologically tested in a biosafety level two laboratory.

Ethics

The Johns Hopkins University School of Medicine IRB approved this study. The mpox-positive and vaccinated serum samples were obtained under the Johns Hopkins protocol IRB00245545. All individuals gave written informed consent.

Collection of the pre-outbreak negative control samples was permitted via a consent waiver. We obtained remnant clinical blood specimens and paired de-identified demographic and administrative data with their research identifier to characterize the local prevalence of disease (16–18).

Serological Methodology

Four commercial ELISAs (Abbexa A29L MPXV IgG [Cambridge UK], RayBioTech MPXV A29L IgG [Peachtree GA], FineTest non-specific IgG [Wuhan, China], ProteoGenix Recombinant Monkeypox virus [Miami FL]) were performed per manufacturer’s protocol. None of the ELISA kits included certified external quality controls. A manufacturer-provided positivity cutoff of 1.88 ng/mL was used for the Abbexa and FineTest assays. The ProteoGenix IgG MPXV ELISA included five antigen-specific plates (A29L, A35R, E8L, M1R, H3L). The signal produced in this ELISA is inversely proportional to the optical density generated in the well, as the subject’s antibodies are competing with the biotinylated antibodies specific to the target antigen. The subject’s antibody concentration was calculated per standard curve extrapolation. The manufacturer provided no positivity cutoff.

The study also evaluated a multi-array electro-chemiluminescent technology (MesoScale Diagnostics (MSD), Gaithersburg, MD) to assess IgG antibodies to ten viral antigens (five MPXV [A29L, A35R, M1R, E8L, B6R] and five VACV [A27L, A33R, L1R, D8L, B5R]). The multiplex assay can quantify IgG to MPXV and VACV antigens simultaneously, allowing us to evaluate the potential distinction between mpox-positive and vaccinated samples. The assay was run per manufacturer protocol, and calculated concentrations are presented in arbitrary binding units per milliliter (AU/mL). The manufacturer provided no positivity cutoffs. Manufacturer provided certified external quality controls were run per protocol (two positives, 1 negative) and an evaluation of diagnostic performance is provided in the supplement.

Statistical Analysis

Concentrations were calculated per manufacturer protocol using the provided standards. Manufacturer provided positivity cutoffs were used when available (Abbexa, FineTest). Sensitivity, specificity, positive and negative likelihood ratios (PLR, NLR) were calculated for each antigen tested (19). Optimal positivity cutoffs for the assays were selected by calculating the maximum sum of sensitivity and specificity for each cut point. To identify the MSD assay optimal cutoffs, the mpox-positive and VACV-positive samples were combined as true positives. Receiver operating characteristic (ROC) curve analysis was employed and area under the curve (AUC) were calculated to assess the assays’ ability to distinguish positive from negative samples (20). Kruskal-Wallis and Mann-Whitney non-parametric comparison tests were performed on the highest performing antigens from the MSD assay to evaluate the distinction of mpox-positive from vaccinated samples (21). A two-sided P-value of <0.05 was considered statistically significant in all analyses. All analyses and figures were performed and created in RStudio 4.3.0 (Rstudio, Boston, MA) and Prism 10.2.2 (GraphPad Software, Boston, MA).

Results

Of the 137 negative pre-outbreak controls, 44% (n=60/137) were under 50 years of age (NC<50) (Supplemental Table 1). Samples from mpox-positive patients were collected a mean of 110 days (range 55–160) post symptom onset (Supplemental Table 1).A majority (n=7/10) of mpox-positive patients were also living with HIV. Persons living with HIV had viral loads obtained within 90 days of mpox-positive PCR-test and clinical blood draw. Persons living with HIV had a mean CD4 count of 554 (range 360–1025), with 71% (n=5/7) having undetectable HIV viral loads (HIV-1 RNA <20 copies/mL). Nine of the patients identified as men who have sex with men, and one identified as a transgender female.

Vaccinated patients obtained one or two of the JYNNEOS vaccines, and blood samples were collected a mean of 90 days (range 67–131) post final vaccine dose (Supplemental Table 2). There was no statistically significant difference in the antibody concentrations between the VACV-positive samples who received one or two doses. All VACV-positive samples were obtained from HDs between ages 21 and 60.

Standard ELISA outcomes ([Abbexa-A29L, FineTest-IgG, RayBioTech-A35R)

The average calculated antibody concentration for mpox-positive samples was 5.0 [A29L; 95%CI: 3.7, 6.2], 3.0 [IgG MPXV non-specific; 95%CI: 1.9, 4.2], 38.2 [A35R; 95%CI: 28.6, 47.8], for Abbexa, FineTest and RayBioTech, respectively (Table 1). These values were comparable to NC≥50: 11.9 [A29L; 95%CI: 10.1, 13.8], 3.5 [IgG MPXV non-specific; 95%CI: 2.8, 4.3], 43.1 [A35R; 95%CI: 27.1, 59.1], for Abbexa, FineTest and RayBioTech, respectively. Concentrations for NC<50 samples were similar for each ELISA. The ELISAs could not distinguish between mpox-positive and pre-outbreak negative samples (Table 1, Figure 1). The PLR for Abbexa was 0.88, indicating a positive A29L antibody concentration (>1.88 ng/mL) was not associated with having an mpox infection. The PLR for FineTest was 1.01, and 1.73 for RayBioTech. While these results are above 1, neither demonstrated strong evidence for detecting an mpox infection.

Table 1.

Positivity cutoffs and diagnostic evaluation metrics for each diagnostic kit used.

Standard ELISA	Positivity Cutoff	Sensitivity (%)	Specificity (%)	PLR	NLR	AUC
Abbexa (A29L)a	1.9	88	0	0.9	-	0.85
FineTest (IgG)a	1.9	78	31	1.0	0.96	0.58
RayBioTech (A35R)	23.3	100	42	1.7	0	0.61
Competition ELISA
Proteogenix A29L	828.6	100	0	-	1	0.66
Proteogenix A35R	1,385.8	100	0	1	-	0.59
Proteogenix E8L	179.3	95	86	6.7	0.1	0.96
Proteogenix M1R	137.0	70	71	0.98	1.1	0.74
Proteogenix H3L	65,985.6	10	98	4.2	0.9	0.78
MSD Assayb
MPXV A29L	211.1	76	64	2.1	0.4	0.68
MPXV A35R	843.7	84	76	3.5	0.2	0.80
MPXV E8L	1,312.0	98	93	13.7	0.0	0.97
MPXV M1R	70.3	84	53	1.8	0.3	0.70
MPXV B6R	372.0	91	61	2.3	0.2	0.79
VACV A27L	247.4	71	64	1.9	0.4	0.67
VACV A33R	314.1	87	58	2.1	0.2	0.73
VACV D8L	719.1	93	85	6.1	0.1	0.71
VACV L1R	176.7	64	70	2.2	0.5	0.78
VACV B5R	251.1	98	58	2.3	0.0	0.94

Abbreviations: ELISA, enzyme-linked immunosorbent assay; MSD, Meso-Scale Diagnostics; VACV, vaccinia virus; MPXV, monkeypox virus; PLR, positive likelihood ratio; NLR, negative likelihood ratio; AU, arbitrary MSD binding unit of measurement.

^aCutoff reported by manufacturer

^bCutoff listed by AU/mL, determined by ROC analysis

Figure 1.

Log concentration of antibody in ng/mL for commercial ELISA’s (Figure a. Abbexa, FineTest, RayBioTech; Figure b. Proteogenix) for mpox PCR positive patients and Johns Hopkins Emergency Department pre-pandemic negative controls, stratified by age. Abbreviations: NC, negative pre-outbreak control (stratified by age in years to account for potential smallpox vaccination); ng/mL, nanogram per milliliter.

Outcomes for the Proteogenix ELISA [A29L, A35R, E8L, M1R, and H3L antigens)

Of the antigen-specific ProteoGenix ELISA’s, the E8L antigen had the greatest ability to discriminate between mpox-positive and pre-outbreak negative samples (Table 1, Figure 1b). The E8L antigen had 95% sensitivity and 86% specificity. The average concentration of mpox-positive samples was 684.2 [95%CI: 433.4, 935.0], and the average for NC≥50 was 173.5 [95%CI: 128.5, 218.5]. The average concentration for NC<50 was 145.8 [95%CI: 137.8, 153.7]. The ProteoGenix E8L ELISA demonstrated greater evidence for identifying an mpox infection (PLR=6.65, NLR=0.06).

MSD Multiplex Assay outcomes (MPXV [A29L, A35R, E8L, M1R, B6R]; VACV [A27L, A33R, L1R, D8L, B5R])

To identify optimal positivity cutoffs, mpox-positive and VACV-positive samples were defined as true positives. To compare the differential antigen concentrations for each assay, the results from each exposure group are reported separately (Table 1, Figure 2). Additional information on the calculated concentrations is presented in the supplement.

Figure 2.

Log concentration of antibodies in AU/mL for ten viral antigens on MSD Multiplex Assay (Figure a. MPXV; Figure b. VACV) for mpox PCR positive patients, vaccinia virus vaccinated patients, and Johns Hopkins Emergency Department pre-outbreak negative controls, stratified by age. Abbreviations: VACV, vaccinia virus; MPXV, monkeypox virus; NC, negative pre-outbreak control (stratified by age in years to account for potential smallpox vaccination); MSD, Meso-Scale Diagnostics; AU, arbitrary MSD binding unit of measurement.

The E8L antigen exhibited the greatest differentiation of antibody concentrations among exposure groups in the MSD assay (Table 1, Figure 2a). The average concentration reported in AU/mL, for mpox-positive samples was 21,876 [95%CI: 9974, 33779], and 15,070 [95%CI: 6524, 23616] for VACV-positives. The average concentration for NC≥50 samples was 779 [95%CI: 374, 1184], and the average for NC<50 was 88 [95%CI: 52, 123]. Additional information on the performance of the MSD E8L assay is presented in Table 1.

D8L performed similarly to its homologous antigen pair E8L (sensitivity=93%, specificity=85%) (Table 1, Figure 2b). Average concentration for mpox-positive samples was 3,001 [95%CI: 2427, 3575], and 53,124 [95%CI: 14193, 92054] for VACV-positives. The concentration for NC≥50 samples was 909 [95%CI: 454, 1363], and for NC<50 was 74 [95%CI: 40, 108]. The PLR was 6.1 and the NLR was 0.1, demonstrating strong predictability of mpox status given an antibody concentration above the optimal positivity cutoff. The AUC demonstrated moderate predictability (0.71).

The B6R MPXV antigen demonstrated distinction between exposure groups, and mpox-positive samples had the highest average concentration (5,791 [95%CI: 3798, 7784]). VACV-positive samples had an average of 3,934 [95%CI: 2003, 5866]. Average concentrations for NC≥50 samples were 3,957 [95%CI: 1184, 6731], and 76 [95%CI: 39, 113] for NC<50. Compared to E8L, the B6R antigen was less specific for distinguishing positive from negative samples (sensitivity=91%, specificity=61%). The PLR for B6R was 2.3, and a NLR of 0.2. The AUC was also lower (0.79).

The B5R VACV antigen performed similarly to its B6R homologous pair (Table 1, Figure 2). The average concentration among mpox-positive samples was 3,103 [95%CI: 2302, 3904], and the VACV-positive samples averaged 4,540 [95%CI: 2113, 6967]. The average concentration for NC≥50 samples was 3,896 [95%CI: 859, 6933], and 71 [95%CI: 33, 108] for NC<50. The PLR was 2.3.

Distinction between VACV and Mpox in MSD Assay

Given the increased sensitivity and specificity to detect both MPXV and VACV antigens in the MSD assay, we performed additional analyses on the two antigens demonstrating the greatest distinction between mpox-positive, VACV-positive, and NC<50 in the preliminary analyses. We found a significant difference in the concentrations to E8L and D8L by exposure group. Per the Kruskal-Wallis non-parametric test of comparison with two degrees of freedom, the test statistic was 35.55 for E8L and 35.13 for D8L (p<0.0001). Using the Mann-Whitney non-parametric test of comparison, we found no significant difference between E8L concentrations among mpox-positives and VACV-positives (U=18.5, p=0.47). Similarly, there was no significant difference in D8L concentrations between mpox-positives and VACV-positives (U=15, p=0.25).

Per the Kruskal-Wallis non-parametric test of comparison with two degrees of freedom, the test statistic was 33.84 for B6R and 33.51 for B5R (p<0.0001). Using the Mann-Whitney non-parametric test of comparison, we found no difference between B6R concentrations among mpox-positives and VACV-positives (U=22.5, p=0.81). There was no significant difference in B5R concentrations between the mpox-positive and VACV-positives (U=23, p=0.86). These data illustrate the ability for the MSD assay to distinguish between mpox-positive and pre-outbreak negatives but are unable to differentiate infected and vaccinated samples.

Discussion

The 2022 mpox outbreak precipitated a need for diagnostic tools to monitor the population-level spread of mpox beyond its endemic regions into the global sphere. Historically, mpox diagnosis has been largely limited to case-based surveillance employing PCR testing for symptomatic patients (6, 10). However, using serological methods may help ascertain the disease landscape that enabled the 2022 global mpox outbreak. Serology can uncover historical exposure to MPXV and provide insight into the number of sub-clinical and asymptomatic cases circulating in the community. (11, 22) Further, the immediate vaccination recommendation for high-risk groups, combined with the limited availability of serological tools capable of distinguishing a mpox antibody response from that induced by smallpox vaccination, has complicated our understanding of population-level immunity (9, 23). In contrast to to the existing literature focuses on evaluating in-house assays (24–28), our study provides a comparative analysis of commercially available serodiagnostic tools. Identifying sensitive and specific serodiagnostic tools to mpox infection is imperative to accurately measure the population exposure to disease (10), as well as the number of symptomatic and asymptomatic infections in a largely vaccinated population (6).

Our findings support prior literature elucidating the varied performance of diagnostic tools to identify mpox infections (6, 7, 11). A recent review summarized available diagnostic tools, emphasizing the scalability of serology-based methods to evaluate the scope of future outbreaks, while also noting the challenges of serological development given the limited availability of inactivated MPXV particles and the cross-reactivity of the OPXV genus (11). In our assessment, none of the standard ELISAs proved suitable for identifying Clade IIb positive samples, but further evaluation of Clade I and Clade IIa are needed. The E8L antigen demonstrated the best performance, though a specificity of 86% (ProteoGenix) would limit its utility. The MSD E8L, D8L, B6R and B5R assays exhibited distinction between mpox-positive and negatives (Table 1), yet were unable to discriminate vaccinated and positive samples. These results were similar to the results from a previous study which used an in house assay with multiple different antigens, which showed that the E8 antigen had the greatest discrimination between negative controls and samples from vaccinated or mpox infected individuals. (29) Additionally, no single antigen could distinguish samples from vaccinated from infected individuals.

Our data suggests the potential utility of combining multiple antigens in the MSD assay to distinguish exposure groups. However, the limited sample size prevented a conclusive determination of the efficacy of this method, and additional research using a greater number of mpox-positive and VACV-positive samples is needed. Serological tools capable of differentiating mpox-positive, vaccinated, and unexposed samples could inform future outbreak strategies by evaluating vaccination interventions, estimate the population at risk, and inform public health resource allocation (15). These tools are crucial for identifying people who may require additional vaccination, determining duration since infection, assessing antibody levels associated with protection, understanding the durability of the antibody response, and identifying breakthrough infections (15). It is also important to consider if certain clinical characteristics or medications could interfere with a patient’s antibody response to mpox infection. The small cohort of mpox-positives evaluated here, including only one patient using immunosuppressive drugs, did not elucidate any major patterns on antibody concentrations. Determining the window of infection or vaccination to immunity is necessary to understand the pathogenesis and transmission dynamics of mpox (9). The gap in serological data and subsequent misinterpretation of non-specific diagnostics can lead to an underestimation of mpox incidence among vaccinated people (9, 15, 23). Understanding the role of neutralizing antibodies in the antibody response to mpox infection will be critical for future case management and surveillance.

Our findings have some limitations. First, our number of samples from mpox-positive and vaccinated patients were limited due to a lack of available clinical and commercial samples. Second, our analyses assumed the NC≥50 patients previously received the smallpox vaccine, so having samples from individuals known to have been vaccinated to smallpox prior to 1974 would aid in the precision of the study’s findings. Additional studies involving larger cohorts of known infected and vaccinated participants are essential to ascertain whether these diagnostic tools can differentiate exposure groups.

Our study indicates a discordant performance of serodiagnostic tools to detect antibodies resulting from mpox infection. Our preliminary evaluation of the available serodiagnostic tools revealed E8L as the highest performing antigen to distinguish mpox or VACV-positive from unexposed individuals. The MSD assay demonstrated a promising opportunity for population-level surveillance of mpox infections, but future research including a more robust set of mpox-positive and VACV-positive samples are essential to determine if those with mpox infection can be serologically differentiated from recently vaccinated individuals.

Highlights.

• Evaluated assays had varied performance for the detection of antibodies to mpox

• Assays with the E8L antigen had the best performance

• No assay was able to distinguish mpox-positive from recently vaccinated patients