Prospective Evaluation of Molecular Assays for Diagnosis of Vaginitis

Molecular tests to diagnose conditions involving the disruption of normal microbiota are difficult to optimize. Using Nugent-scored Gram stain (NS) as the reference standard, we evaluated the performance of 3 molecular assays for the diagnosis of bacterial vaginosis (BV) and examined the impact of an incremental increase in bacterial targets.

ABSTRACT

Molecular tests to diagnose conditions involving the disruption of normal microbiota are difficult to optimize. Using Nugent-scored Gram stain (NS) as the reference standard, we evaluated the performance of 3 molecular assays for the diagnosis of bacterial vaginosis (BV) and examined the impact of an incremental increase in bacterial targets. The BD Affirm assay includes a DNA probe for Gardnerella vaginalis, the Hologic transcription-mediated amplification (TMA) analyte-specific reagent (ASR) assay adds a second Lactobacillus sp. target, and the recently cleared in vitro diagnostic use (IVD) Aptima BV assay includes a third target (Atopobium vaginae). The diagnosis of vulvovaginal candidiasis (VVC) by the Affirm and Candida vaginitis Hologic TMA ASR assays was assessed using microscopy for yeast as the reference standard. From May to December 2018, 111 women with vaginitis symptoms prompting the clinician to order an Affirm test were enrolled with informed consent for the collection of additional specimens. Clinicians accurately predicted BV as the most likely diagnosis for 71% of the 45 patients with BV. Coinfection occurred in 13.5% of patients. For BV, the specificity of the Aptima IVD assay (86.3%) was higher than the Affirm assay (60.6%, P = 0.0002), but sensitivities were not significantly different. For VVC, the sensitivity of the ASR assay (100%) was higher than Affirm (75.9%; P = 0.023) and the specificity of the Affirm assay (98.8%) was higher than the ASR assay (86.6%; P = 0.004).

INTRODUCTION

Molecular tests to diagnose conditions that represent a disruption of normal microbiota are difficult to optimize. Bacterial vaginosis (BV) is a common cause of vaginitis associated with alteration of the normal microbiome, namely, a decrease in Lactobacillus spp. accompanied by an increase in Gardnerella vaginalis and other bacteria (1). The reference standard for diagnosing BV is Nugent scoring of a Gram stain prepared with vaginal secretions (2). Research using broad-range molecular techniques has uncovered more bacteria associated with BV (Atopobium vaginae, Megasphaera spp., Leptotrichia amnionii, Sneathia sanguinegens, and Porphyromonas asaccharolytica), including 3 new species designated “bacterial vaginosis-associated bacteria” (BVAB1, BVAB2, and BVAB3) (3). An evaluation of a quantitative PCR targeting 7 bacteria associated with BV concluded G. vaginalis (≥10⁹ copies/ml) and A. vaginae (≥10⁸ copies/ml) had the best predictive value for diagnosing BV with Nugent score as the reference method (4).

Current options to diagnose BV in our institution include Gram stain with Nugent score or DNA probe (BD Affirm vaginal pathogens III) targeting G. vaginalis. The Affirm assay targets 2 additional infectious etiologies of vaginitis, namely, Candida spp. and Trichomonas vaginalis. The Affirm assay is ergonomically difficult for technologists to perform, and recent microbiome studies suggest that using only a single target to diagnose BV is probably suboptimal.

Using the Nugent score as the gold standard, we evaluated the performance of 3 molecular assays for the diagnosis of BV and examined the impact of an incremental increase in bacterial targets; the Affirm assay targets G. vaginalis, the Hologic transcription-mediated amplification (TMA) analyte-specific reagent (ASR) assay adds a second Lactobacillus sp. target, and the recently cleared in vitro diagnostic use (IVD) Aptima BV assay contains a third target (A. vaginae). Additionally, a TMA ASR assay targeting Candida spp. was compared to the Affirm assay for the detection of vulvovaginal candidiasis (VVC) using microscopy for yeast as the reference standard. The prevalence of coinfections and the ability of clinicians to predict the etiology of vaginitis were also examined.

(The study was presented, in part, as a poster presentation at the ASM Microbe on June 23, 2019 in San Francisco, CA.)

MATERIALS AND METHODS

Clinical specimens.

From May to December 2018, Cleveland Clinic outpatients with vaginitis symptoms that would normally prompt their provider to order the BD Affirm assay were invited to participate in this institutional review board (IRB)-approved study. After informed consent was obtained, 2 vaginal swabs (Copan swab in Amies transport for Gram stain and Aptima vaginal swab for BV and VVC testing) were collected in addition to a swab for the Affirm assay. On a questionnaire submitted with patient samples to the main campus microbiology laboratory, providers were asked to record a description of vaginal discharge, whiff test results, and vaginal pH; and use clinical judgment to indicate the most likely syndrome (rank order BV, VVC, and trichomoniasis).

BD Affirm.

The Affirm VPIII microbial identification test was performed according to the manufacturer’s instructions and reported in the patient record according to laboratory protocol. The Affirm assay incorporates single-stranded DNA probes immobilized on a bead embedded in a probe analysis card, which contains a separate bead for each target organism and positive and negative controls. After hybridization, an enzyme conjugate-binding step occurs to enable detection. Observing visible blue color on the target organism bead is a positive result. Negative results indicate <10,000 CFUs of Candida cells, <200,000 CFUs of G. vaginalis, and <5,000 trichomonads.

Microscopy.

Every Gram stain was read by 2 technologists who were blind to all of the molecular test results. Bacteria and yeast observed were quantified based on the number present per oil immersion field (i.e., rare, few, moderate, and many) (5). Any quantity of yeast was considered a positive result. The Nugent score system guidelines were applied to interpret the significance of bacteria observed and a third technologist (blind to the initial 2 Nugent score results) reviewed the slide if the initial 2 readings were not concordant (positive, negative, and intermediate). The presence of clue cells was noted but not factored into the assigned score. A predominance of medium-to-large Gram-positive bacilli suggestive of lactobacilli (score, 0 to 3) is consistent with normal vaginal flora. A predominance of Gram-negative or Gram-variable bacilli suggestive of G. vaginalis, Mobiluncus species, and Bacteroides species (score, 7 to 10) is consistent with bacterial vaginosis. An intermediate Nugent score (4–6) suggests a transition from normal vaginal flora and was considered positive for BV if ≥2 Amsel’s criteria (positive whiff test, clue cells on Gram stain, or vaginal pH of >4.5) were present (6).

Hologic ASR BV assay.

The ASR BV assay for the Panther instrument (Hologic) utilizes transcription-mediated amplification (TMA) technology. After release of RNA in specimens, hybridization occurs to target organisms. After target amplification, results are reported as T-times, utilizing a scale of 0 to 100 (lower values indicate less time to detection; T-time = 100 if no target detected). The ASR BV algorithm uses the Lactobacillus results only if a medium organism burden of G. vaginalis is detected. There were 2 scenarios when the assay was considered positive for BV, namely, (i) if a high burden of G. vaginalis was detected (T-time sample < T-time high control) or (ii) if a medium amount of G. vaginalis (T-sample < T-time low control) was detected along with no Lactobacillus sp.

Aptima IVD BV assay.

In May 2019, the remnant specimens (stored at −70°C) were tested using a new investigational use only (subsequently cleared as IVD) Aptima BV assay that contains A. vaginae along with the G. vaginalis and Lactobacillus sp. targets in the ASR assay initially tested. Software was installed on the Panther instrument that provided a revised algorithm for interpreting results as negative or positive for BV and allowed an additional (fourth) test to be run on a specimen.

Aptima ASR Candida assay.

The VVC assay utilizes TMA targeting 5 Candida species (Candida albicans, Candida parapsilosis, Candida tropicalis, Candida dubliniensis, and Candida glabrata). Probes provide detection of C. glabrata or other Candida spp. The Candida vaginitis (CV) assay was an ASR when utilized in our study and received FDA approval as a combination test with T. vaginalis on May 29, 2019.

Data analysis.

Using microscopy as the reference standard, we determined the sensitivity, specificity, positive predictive value, and negative predictive value for all assays. Confidence intervals were calculated using the modified Wald method; McNemar’s test was applied to analyze differences between test performance (sensitivity and specificity), with P values of ≤0.05 considered statistically significant (https://www.graphpad.com). The rate of coinfection was determined using the T. vaginalis result from the Affirm assay and microscopy for BV and VVC.

RESULTS

A total of 111 women were enrolled in the study. To achieve concordance for Nugent score interpretation (positive, negative, or intermediate), a third reader was required for 24% of specimens. There were 40 patients with positive Nugent scores (7–10) consistent with BV and 52 patients with low scores (0 to 3) interpreted as normal vaginal microbiota. Only 5 of the 19 specimens with intermediate Nugent scores were positive by Amsel’s criteria and classified as positive for BV; the other 14 specimens were considered negative.

For BV, the sensitivities and specificities were 75.6% (34/45) and 81.8% (54/66) for the Hologic ASR assay, 86.7% (39/45) and 60.6% (40/66) for the Affirm assay, and 84.4% (38/45) and 86.3% (57/66) for the Aptima BV IVD assay, respectively (Table 1). Although the differences in sensitivity did not reach statistical significance, the specificity of the Aptima BV IVD assay was higher than the BD Affirm assay (P = 0.0002). Excluding intermediate Nugent score, the sensitivities and specificities for BV were 82.5% (33/40) and 84.6% (44/52) for the Hologic ASR assay, 87.5% (35/40) and 63.4% (33/52) for the Affirm assay, 90% (36/40) and 92.3% (48/52) for the Aptima BV IVD assay, respectively (Table 2). This subset analysis also demonstrated higher specificity of the Aptima BV IVD assay than the BD Affirm assay (P = 0.0003).

TABLE 1.

Comparison of BV molecular assays to NS Gram stain for 111 specimens^{a ,b}

Assay	TP	FP	TN	FN	Sensitivity (%)	Specificityc (%)	PPV (%)	NPV (%)
Aptima IVD	38	9	57	7	84.4 (70.9–92.6)	86.3 (75.9–92.9)	80.9 (67.2–89.8)	89.1 (78.8–94.9)
BD Affirm	39	26	40	6	86.7 (73.5–94.1)	60.6 (48.5–71.5)	60.0 (47.8–71.0)	87.0 (74.0–94.3)
Hologic ASR	34	12	54	11	75.6 (63.6–87.6)	81.8 (70.7–89.4)	73.9 (59.6–84.5)	83.1 (72.0–90.5)

^aBV, bacterial vaginosis; NS, Nugent score; TP, true positive; FP, false positive; TN, true negative; FN, false negative; PPV, positive predictive value; NPV, negative predictive value. Values in parentheses are 95% confidence intervals.

^bIntermediate NSs (4–6) were considered positive for BV if ≥2 of Amsel’s criteria (positive whiff test, clue cells on Gram stain, and vaginal pH of >4.5) were present.

^cThe specificity of Aptima IVD was higher than BD Affirm (P = 0.0002).

TABLE 2.

Comparison of BV molecular assays to positive and negative NS Gram stain for 92 specimens^a

Assay	TP	FP	TN	FN	Sensitivity (%)	Specificityb (%)	PPV (%)	NPV (%)
Aptima IVD	36	4	48	4	90.0 (76.4–96.6)	92.3 (81.3–97.5)	90.0 (76.4–96.6)	92.3 (81.2–87.5)
BD Affirm	35	19	33	5	87.5 (73.4–95.0)	63.4 (49.8–75.2)	64.8 (51.5–76.2)	86.8 (72.2–94.7)
Hologic ASR	33	8	44	7	82.5 (67.7–91.6)	84.6 (72.2–92.3)	80.5 (65.7–90.0)	86.3 (74.0–93.5)

^aThe 19 specimens with intermediate NSs (4–6) were excluded. BV, bacterial vaginosis; NS, Nugent score; TP, true positive; FP, false positive; TN, true negative; FN, false negative; PPV, positive predictive value; NPV, negative predictive value. Values in parentheses are 95% confidence intervals.

^bThe specificity of Aptima IVD was higher than BD Affirm (P = 0.0003).

The Aptima ASR assay detected Candida spp. in all 29 specimens with yeast observed by microscopy, while only 22 (75.9%; P = 0.023) were detected by the Affirm DNA probe (Table 3). There were more false positives (n = 11) and lower specificity with the Aptima ASR assay than Affirm assay (86.6% versus 98.8%; P = 0.004) for VVC.

TABLE 3.

Comparison of molecular assays to microscopy for detection of vulvovaginal candidiasis^a

Assay	TP	FP	TN	FN	Sensitivityb (%)	Specificityc (%)	PPV (%)	NPV (%)
Aptima ASR	29	11	71	0	100 (86.1 to 100)	86.6 (77.4 to 92.5)	72.5 (57.0 to 84.0)	100 (93.0 to 100)
BD Affirm	22	1	81	7d	75.9 (57.6 to 88.1)	98.8 (92.8 to >99.9)	95.7 (77.3 to >99.9)	92.1 (84.2 to 96.3)

^aBV, bacterial vaginosis; NS, Nugent score; TP, true positive; FP, false positive; TN, true negative; FN, false negative; PPV, positive predictive value; NPV, negative predictive value. Values in parentheses are 95% confidence intervals.

^bThe sensitivity of Aptima ASR was higher than BD Affirm (P = 0.023).

^cThe specificity of BD Affirm was higher than Aptima ASR (P = 0.004).

^dQuantities of yeast observed by microscopy but not detected by Affirm were rare (n= 4), few (n = 1), moderate (n = 1), and many (n = 1).

Coinfection occurred in 15 (13.5%) patients (12 BV and VVC, 2 BV and trichomoniasis, and 1 trichomoniasis and VVC). Clinicians accurately chose BV as the most likely diagnosis for 71% of the 45 patients with BV.

DISCUSSION

Despite the higher cost, there are a number of reasons why molecular assays to diagnose vaginitis have been embraced by clinicians and laboratorians. The high percentage of women that may have BV by Nugent criteria and, yet, are without symptoms (up to 75%) explains why clinicians appreciate having the option of ordering syndromic testing covering multiple etiologies (7–9). The importance of diagnosing and treating BV is underscored by the association with preterm delivery and acquisition of HIV and other infections (10–12).

The gold standard for BV is labor-intensive, and many laboratories have difficulty finding technologists competent and confident in reading Gram stains. The subjective nature of Nugent scoring is reflected by 24% of specimens in the current study needing a third reader to achieve a concordant category result. It can be difficult to get agreement for specimens with Nugent scores adjacent to a categorical cutoff value (6 and 7 or 3 and 4). The quality of research studies without multiple readers for each Gram stain should be questioned.

There was little change in Affirm assay performance with the elimination of intermediate Nugent specimens (from 86.7% to 87.5% sensitivity and 60.6% to 63.4% specificity). Although the Affirm assay is FDA approved and performed by many laboratories, there is limited published data on the accuracy of this method to compare with our results. Sensitivity and specificity of the Affirm assay compared with the Nugent criteria determined in a study of 195 Korean women was 75.0% and 88.9% for G. vaginalis (13). The Affirm package insert reports 83.8% sensitivity and 100% specificity for G. vaginalis compared with Gram stain and 89.0% sensitivity and 99.1% specificity compared with culture of G. vaginalis. Culture is not recommended or performed routinely as a test for diagnosing BV since G. vaginalis can be recovered from cultures collected from healthy women.

Of the 3 molecular assays we evaluated for BV, Aptima BV IVD demonstrated the highest specificity, which may reflect value for the A. vaginae target unique to that assay. It is difficult to predict whether the difference observed in the Aptima IVD and Affirm assay positive predictive values would be seen in other settings (85% of our patients were enrolled from general gynecology clinics and 15% from a vulvovaginitis specialty clinic). The sensitivity and specificity of the Aptima IVD reached 90% only with elimination of intermediate Nugent-scored specimens (Table 2). This approach (eliminating intermediate Nugent-scored specimens) was used for an evaluation of the BD Max assay containing 2 additional BV targets (BVAB2 and Megasphaera-1) and reported 92.7% sensitivity and 91.5% specificity compared with Nugent score (14). Although our study had fewer subjects, the Aptima IVD results (90.0% sensitivity, 92.3% specificity) showed similar performance to that reported for BD Max. Although assays that incorporate more bacterial targets are attractive since they reflect the bacterial diversity that has been reported in BV, it is uncertain whether they will provide better diagnostic accuracy to offset the higher cost usually charged for additional targets. Third party reimbursement of multitarget nucleic acid amplification tests (NAAT) for BV may be denied by insurers who want to see evidence of better patient management or improved clinical outcomes to support NAAT as an alternative to traditional approaches.

For VVC, our Affirm assay sensitivity compared with microscopy was 75.9%; the sensitivity stated in the Affirm package insert is 80.6% compared with culture. A comparison of Affirm assay to KOH microscopy reported similar performance, with a sensitivity of 82% and specificity of 100% (15). Our Affirm assay specificity compared with microscopy (98.8%) was similar to the Affirm package insert comparison to culture (98.2%) and higher than the TMA assay (86.6%); false positives are not surprising when NAAT is compared with microscopy as the reference standard.

A major limitation of our study is the uncertainty regarding whether microscopy is the optimal gold standard for BV and VVC. Hopefully future studies will be performed that are designed to determine whether clinical outcomes are improved in populations where a highly sensitive NAAT method rather than microscopy is used to diagnose vaginitis. More subjects and clinical outcome data would enrich the study but were prevented by limited clinician time to enroll patients and the amount of funding.

Strengths of this evaluation include a relevant, prospective study design with a comparison of a new commercially available molecular assay (Aptima IVD) to an established, commonly ordered assay (Affirm) with little published performance data. The reference standard of microscopy was performed in a rigorous manner (minimum of 2 technologists blind to the molecular results). Comparisons of the performance of the Affirm assay with one target (G. vaginalis), Hologic BV ASR with 2 targets (Lactobacillus sp. and G. vaginalis), and Aptima BV IVD with a third target (A. vaginae) to a reference standard of microscopy using current clinical samples are a novel demonstration of the utility that may be associated with individual targets for diagnosing BV. Many laboratories use the Affirm assay but few have the resources needed to conduct a prospective study requiring informed consent to collect additional specimens. The ergonomic complaints reported by technologists performing the Affirm assay illustrates the relevance of comparing those results to more automated NAAT solutions.

Although coinfections in our study population were uncommon, ordering patterns at our institution for Affirm demonstrate clinicians’ preference for syndromic tests that cover multiple etiologies. Although workflow and some performance parameters for the IVD TMA assays were better than the DNA probe, studies demonstrating improved clinical outcomes with NAAT testing for vaginitis may be needed to ensure reimbursement.