Performance of Three Nucleic Acid Amplification Tests for Detection of Chlamydia trachomatis and Neisseria gonorrhoeae by Use of Self-Collected Vaginal Swabs Obtained via an Internet-Based Screening Program

Abstract

Use of self-obtained vaginal specimens processed by nucleic acid amplification tests (NAATs) has significantly increased the utilization of nontraditional locations for Chlamydia trachomatis and Neisseria gonorrhoeae screening programs. One important emerging source of such venues includes home-based self-sampling kits available via the Internet. The objective of our study was to evaluate the performance of three commercially available NAATs (Becton-Dickinson ProbeTec SDA, Gen-Probe Aptima Combo2 TMA, and Roche Amplicor PCR) for detection of C. trachomatis and N. gonorrhoeae in vaginal samples obtained via an Internet-based screening program. From July 2004 to August 2005, 500 self-collected vaginal swabs were tested for C. trachomatis and N. gonorrhoeae by using all three NAATs. Another 500 samples were collected between August 2005 and November 2007 and tested by ProbeTec and Combo2; PCR testing was discontinued due to low specificity for N. gonorrhoeae. All tests were conducted according to the manufacturers' procedures; the “gold standard” for an infected C. trachomatis or N. gonorrhoeae patient was defined as ≥2 positive NAAT results. Of the first 500 swabs submitted, 46 were C. trachomatis infected (9.2%) and 5 were N. gonorrhoeae infected (1.0%), and 3 of these were coinfected (0.6%). All C. trachomatis and N. gonorrhoeae Combo2-positive/ProbeTec-negative samples were confirmed as true positives by an alternative NAAT. For C. trachomatis, ProbeTec, Combo2, and PCR had sensitivities of 82.6%, 100%, and 100%, with specificities of 100%, 100%, and 99.3%, respectively. For N. gonorrhoeae, ProbeTec, Combo2, and PCR had sensitivities of 80%, 100%, and 100%, with specificities of 100%, 100%, and 98.8%, respectively. Of the total 1,000 swabs submitted, 92 were C. trachomatis infected (9.2%) and 15 were N. gonorrhoeae infected (1.5%), and 7 of these were coinfected (0.7%). There were no ProbeTec-positive/Combo2-negative samples. For C. trachomatis, ProbeTec and Combo2 had sensitivities of 81.5% and 100%, with specificities of 100% and 100%, respectively. For N. gonorrhoeae, ProbeTec and Combo2 had sensitivities of 80% and 100%, with specificities of 100% and 100%, respectively. Overall, ProbeTec had 17 C. trachomatis false-negative results (1.7%) and 3 N. gonorrhoeae false-negative results (0.3%), while Combo2 had none. Our results were consistent with the sensitivities and specificities stated by the manufacturers. NAATs perform well for detection of chlamydia and gonorrhea with self-obtained vaginal swabs shipped in a dry state to a laboratory. For 1,000 self-collected vaginal swabs tested by NAATs, the sensitivities for C. trachomatis and N. gonorrhoeae for Combo2 were 100% and 100%, while they were 81.5% and 80%, respectively, for ProbeTec. For 500 PCR samples, the C. trachomatis sensitivity was 100% and the N. gonorrhoeae sensitivity was 100%, with specificities of 99.3% and 98.8%, respectively.

Each year, an estimated 3 million Chlamydia trachomatis infections occur in the United States, with over 1 million reported in 2006 surveillance reports (2). In these same reports, 358,366 cases of Neisseria gonorrhoeae infection were reported to occur in the United States (2). Since many sexually transmitted infections (STIs) are asymptomatic, especially chlamydial infections, traditional approaches to clinic-based diagnosis of STIs omit a whole segment of the population that would not ordinarily get tested. Prior to the emergence of nucleic acid amplification tests (NAATs), symptomatic persons or contacts of infected patients would be the only persons seeking care for STIs, usually at a sexually transmitted disease clinic or family planning clinic.

Through the use of noninvasive urogenital samples, it is now possible to use alternative venues for screening programs outside the clinic where a clinician is not required to obtain the specimens. Utilization of self-obtained specimens for C. trachomatis and N. gonorrhoeae testing, which employ the highly sensitive and specific NAATs, has enhanced the use of noninvasive urogenital samples (9, 17). Studies have reported that self-collected vaginal specimens are acceptable and can be successfully used to diagnose STIs, especially chlamydia and gonorrhea, when they are used with NAATs (3, 14, 17, 18, 26). This can eliminate the necessity for a clinician-performed pelvic examination for women for sample collection and may represent cost savings when a woman does not require a Pap test or have symptoms (1).

Expansion of alternative venues in which chlamydia testing can be performed may even be taken to Internet recruitment for home collection of vaginal swabs for mailing to a laboratory for testing (11, 13). This alternative venue may be popular among adolescents who search the Internet for health care information and information regarding STIs (20). The Internet is also used by people seeking sex partners, and these users appear to be at greater risk for STIs (22). Internet use statistics are staggering; the Pew American Life Project reports that 168 million people are on the Internet per day (http://www.pewinternet.org/reports.asp). Internet use is estimated at 83% of those aged 18 to 29 years, the prime age range in which STIs are most prevalent (http://www.pewinternet.org/reports.asp).

We have successfully used an Internet-based recruitment method for home collection of self-obtained vaginal swabs for chlamydia testing (11). While such samples collected at home are not yet FDA cleared, self-obtained vaginal swabs collected in a clinic are FDA cleared for one type of NAAT (26). We therefore decided to compare the performances of the three commercially available NAATs for C. trachomatis and N. gonorrhoeae testing with vaginal samples obtained at home through an Internet screening program and mailed in a dry state (10, 17, 29) to the laboratory for testing.

MATERIALS AND METHODS

Between July 2004 and November 2007, 1,000 self-collected vaginal swabs were mailed to the laboratory from individuals who had accessed the Internet-based self-screening website www.iwantthekit.org (11). The prenumbered kit consisted of a contact form for return of results, a consent form, collection instructions, a swab, a questionnaire, and a postage-paid, preaddressed return mailer (11). The data are represented in two stages. Stage 1 consisted of the first 500 samples that were received from July 2004 to August 2005; samples were tested by three NAATs. Stage 2 represented the second 500 samples, which were received from August 2005 to November 2007; samples were tested by only two NAATs. Each swab was expressed in 800 μl Tris-EDTA buffer; 200 μl was placed into SDA lysis diluent, 200 μl was placed into TMA specimen transport medium, and 200 μl was DNA extracted using MagNA PureLC (Roche Molecular Corporation, Indianapolis, IN) (6). All of the samples were tested for C. trachomatis and N. gonorrhoeae by using three commercially available NAATs: Becton Dickinson ProbeTec SDA (Becton Dickinson, Sparks, MD), Gen-Probe Aptima Combo2 TMA (Gen-Probe Inc., San Diego, CA), and Roche Amplicor PCR (Roche Molecular Diagnostics, Indianapolis, IN). All tests were performed according to manufacturers' procedures. Testing by the Amplicor PCR was discontinued after the first 500 samples. For this study, we defined the “gold standard” for a C. trachomatis or N. gonorrhoeae “infected patient status” as two or more positive NAAT results. When only two NAATs were used (with the second 500 samples), the discordant specimens were tested by another stand-alone Aptima NAAT, either ACT or AGC (Gen-Probe Inc.), which targets alternative gene sequences (26). Sensitivity, specificity, positive predictive values (PPVs), and negative predictive values (NPVs) were calculated in comparison to the “gold standard.” Data analyses were performed by Stata version 10.0 (Stata Corp., College Station, TX). Exact binomial confidence intervals were also calculated using Stata. One-sided, 97.5% confidence intervals were calculated when the sensitivity, specificity, and PPV or NPV values were 100%.

RESULTS

Of the first 500 swabs submitted, 46 were C. trachomatis infected (9.2%) and 5 were N. gonorrhoeae infected (1.0%), and 3 of these were coinfected (0.6%). Of the total 1,000 swabs submitted, 92 were C. trachomatis infected (9.2%) and 15 were N. gonorrhoeae infected (1.5%), and 7 of these were coinfected (0.7%) (Table 1).

TABLE 1.

Detection of true-positive Chlamydia trachomatis and Neisseria gonorrhoeae test results

Test	No. of samples testing positivea
	C. trachomatis		N. gonorrhoeae
	First 500	Total 1,000	First 500	Total 1,000
ProbeTec SDA	38	75	4	12
Aptima Combo2	46	92	5	15
Amplicor PCR	46	NAb	11	NA

^aThe numbers of true positives were as follows: for the first 500 samples for C. trachomatis, 46; for all 1,000 samples for C. trachomatis, 92; for the first 500 samples for N. gonorrhoeae, 5; and for all 1,000 samples for N. gonorrhoeae, 15.

^bNA, not applicable.

For the first 500 samples, of the 46 true positives for Chlamydia trachomatis, ProbeTec detected 38, while Combo2 and PCR each detected 46. The sensitivities of ProbeTec, Combo2, and PCR for C. trachomatis were 82.6%, 100%, and 100%, and the specificities were 100%, 100%, and 99.3%, respectively (Table 2).

TABLE 2.

NAAT performance statistics for Chlamydia trachomatis

Test	% (no. of positive samples confirmed/total no. of positive samples) (CIa)
	Sensitivity		Specificity		PPV		NPV
	First 500	Total 1,000	First 500	Total 1,000	First 500	Total 1,000	First 500	Total 1,000
ProbeTec SDA	82.6 (38/46) (68.6-92.2*)	81.5 (75/92) (72.1-88.9*)	100 (454/454) (99.2-100**)	100 (908/908) (99.6-100**)	100 (38/38) (90.7-100**)	100 (75/75) (95.2-100**)	98.3 (454/462) (96.6-99.2*)	98.2 (908/925) (97.1-98.9*)
Aptima Combo2	100 (46/46) (92.3-100**)	100 (92/92) (96.1-100**)	100 (454/454) (99.2-100**)	100 (908/908) (99.6-100**)	100 (46/46) (92.3-100**)	100 (92/92) (96.1-100**)	100 (46/46) (92.2-100**)	100 (908/908) (99.6-100**)
Roche Amplicor PCR	100 (46/46) (92.3-100**)	NAb	99.3 (451/454) (98.1-99.9*)	NA	93.9 (46/49) (83.1-98.7*)	NA	100 (451/451) (99.2-100**)	NA

^a*, 95% confidence interval (CI); **, 97.5% CI.

^bNA, not applicable.

For the total 1,000 samples collected for C. trachomatis, ProbeTec had a sensitivity and specificity of 81.5% (75/92) and 100% (908/908), respectively. Combo2 had a sensitivity and specificity of 100% (92/92) and 100% (908/908), respectively (Table 2). All C. trachomatis Combo2-positive/ProbeTec-negative samples for the last 500 samples were confirmed as true positives by the alternative stand-alone NAAT, ACT.

For the first 500 samples, of the five true positives for Neisseria gonorrhoeae, ProbeTec detected four, while Combo2 and PCR each detected five. The sensitivities of ProbeTec, Combo2, and PCR for N. gonorrhoeae were 80%, 100%, and 100%, and the specificities were 100%, 100%, and 98.8%, respectively (Table 3).

TABLE 3.

NAAT performance statistics for Neisseria gonorrhoeae

Test	% (no. of positive samples confirmed/total no. of positive samples) (CIa)
	Sensitivity		Specificity		PPV		NPV
	First 500	Total 1,000	First 500	Total 1,000	First 500	Total 1,000	First 500	Total 1,000
ProbeTec SDA	80 (4/5) (28.4-99.5*)	80 (12/15) (51.9-95.7*)	100 (495/495) (99.3-100**)	100 (985/985) (99.6-100*)	100 (4/4) (39.8-100**)	100 (12/12) (73.5-100**)	99.8 (495/496) (98.9-99.9*)	99.7 (985/988) (99.1-99.9*)
Aptima Combo2	100 (5/5) (47.8-100**)	100 (15/15) (78.2-100**)	100 (495/495) (99.3-100**)	100 (985/985) (99.6-100**)	100 (5/5) (47.8-100**)	100 (15/15) (78.2-100**)	100 (495/495) (99.3-100**)	100 (985/985) (99.6-100**)
Roche Amplicor PCR	100 (5/5) (47.8-100**)	NAb	98.8 (489/495) (97.4-99.6*)	NA	45.5 (5/11) (16.7-76.6*)	NA	100 (489/489) (99.2-100*)	NA

^a*, 95% confidence interval (CI); **, 97.5% CI.

^bNA, not applicable.

For the total 1,000 samples for N. gonorrhoeae, ProbeTec had a sensitivity and specificity of 80% (12/15) and 100% (985/985), respectively. Combo2 had a sensitivity and specificity of 100% (15/15) and 100% (985/985), respectively (Table 3). The three N. gonorrhoeae Combo2-positive/ProbeTec-negative samples for the last 500 samples were confirmed as true positives by the alternative stand-alone NAAT, AGC.

Overall, for the total 1,000 samples collected for C. trachomatis and N. gonorrhoeae testing, ProbeTec had 17 C. trachomatis false-negative results (18.5%) and 3 N. gonorrhoeae false-negative results (20.0%), while Combo2 had none (Table 4). For the first 500 samples, ProbeTec produced eight false negatives for C. trachomatis (17.4%) and one false negative for N. gonorrhoeae (20.0%) (Table 4). PCR yielded three C. trachomatis false-positive test results and six N. gonorrhoeae false positives. Combo2, however, had no false test results.

TABLE 4.

False-negative test results for Chlamydia trachomatis and Neisseria gonorrhoeae

Test	No. (%) of false-negative samplesa
	C. trachomatis		N. gonorrhoeae
	First 500	Total 1,000	First 500	Total 1,000
ProbeTec SDA	8 (17.4)	17 (18.5)	1 (20.0)	3 (20.0)
Roche Amplicor PCR	0	NAb	0	NA
Aptima Combo2 TMA	0	0	0	0

^aThe numbers of true positives were as follows: for the first 500 samples for C. trachomatis, 46; for all 1,000 samples for C. trachomatis, 92; for the first 500 samples for N. gonorrhoeae, 5; and for all 1,000 samples for N. gonorrhoeae, 15.

^bNA, not applicable.

DISCUSSION

Ultimately, the results obtained were fairly consistent with the sensitivities and specificities stated by the manufacturers. When considering the sensitivities and specificities for the three NAATs (ProbeTec SDA, Aptima Combo2, and Amplicor PCR), one assay suffered from reduced sensitivity for chlamydia and another demonstrated poor specificity for gonorrhea. The PCR test is well documented to have decreased specificity for gonorrhea, as was demonstrated in our study (8, 24, 30). Thus, for this assay, we concluded that routine confirmation is recommended (7). On the other hand, as in our study, for the Aptima Combo2, excellent specificity for gonorrhea was shown by Golden et al., who demonstrated that the assay had a PPV of 97.4% for gonorrhea in 59,664 specimens by confirming 258/265 positive Combo2 gonorrhea results via AGC in a low-prevalence population (0.47%), indicating that confirmation is not required even in populations with very low prevalences (15). Additionally, Moncada et al. showed that the use of an alternate gene target for confirmation was either comparable to or slightly less sensitive than a repeat test and that the Aptima Combo2 and AGC assays were better at confirmation than SDA and PCR (23). However, only 89.5% of PCR-positive gonorrhea samples were confirmed. Alternatively, for Aptima Combo2-positive samples, the AGC test confirmed 95.7% of their positive samples (23). Our population was a screening population, and the PPV of NAATs in low-prevalence settings has been controversial, especially for gonorrhea. However, we believe that positive gonorrhea results for Aptima Combo2 do not need confirmation by another NAAT. This is supported by the conclusion by others mentioned above that the Aptima Combo2 for N. gonorrhoeae performs well in a low-prevalence (screening) population and does not need further confirmation (15, 23).

ProbeTec appeared to be less sensitive for detection of C. trachomatis than our defined “gold standard” (P < 0.05), and the results were lower than expected (82.6% for 500 samples and 81.5% for 1,000 samples). For gonorrhea, the ProbeTec sensitivity was also lower than the “gold standard” (80%) as well as lower than the sensitivity demonstrated by the Aptima Combo2 assay. We cannot explain this lower sensitivity for ProbeTec in this study. However, our findings are supportive of the report that all NAATs do not necessarily demonstrate the same sensitivity (27). The Amplicor PCR assay was very sensitive for the first 500 samples for chlamydia and gonorrhea. The performance of the Aptima Combo2 assay was excellent compared to the “gold standard” of having positive results for two assays or a confirmed positive result by the ACT assay (100% for both sensitivity and specificity) for all 1,000 samples.

The CDC has previously recommended confirming NAAT-positive samples in populations of low prevalence through supplemental testing, which can increase PPV and decrease the possibility of false positives (19). We were able to use the confirmatory nature of a second positive test result to substantiate the PPV of the NAATs in this population. It appeared that the Aptima Combo2 TMA was the superior NAAT in this study, since it demonstrated no false-positive or false-negative test results, compared to our definition of a “gold standard.” We realize there is no perfect test and that no test has 100% sensitivity and specificity. Since the absence of a perfect “gold standard” requires that one choose a definition for a “gold standard,” we chose “two positive tests” as the standard. These were from two different platforms for the first 500 samples (three tests performed on all samples). For the second 500 samples (two tests performed on samples), we chose either (i) positives on two different platforms or (ii) one positive test platform, confirmed as positive by a second-target test for the Aptima platform (with discordant or discrepant analysis, which is not favored by some statisticians) (16). The choices of a “gold standard” and the number of comparators required for determining a “true positive” have always been problematic in the evaluation of diagnostic tests and will probably continue to be so (21).

The high sensitivity of NAATs has made it possible to use noninvasive specimens, such as urine or vaginal swabs, for diagnosis and screening for major bacterial STIs. This increase in detection of positive infections from screening assays must not be overlooked in assessing the utility of outreach screening programs. When NAATs were first introduced, it was shown that they could be used to test first-catch urine (FCU) specimens from men and that the performance profiles were very similar to those seen with urethral swabs (5), and now, FCU is considered the specimen of choice for testing men for chlamydia (12). Shortly after, FCU specimens from females, when tested by NAATs, were shown to have performance profiles similar to those reported for endocervical swab specimens (4). The ability to use FCU specimens from men and women meant that these specimens could be collected in places other than traditional clinical settings. Noninvasive collection of specimens has made it far easier to screen asymptomatic individuals and has also made it possible to perform studies aimed at determining the population-based prevalence and even the incidence of these infections (20a). After FCU specimens were found to be useful, a number of researchers evaluated vaginal swab specimens, and these too were found to be highly effective for detecting both chlamydial and gonococcal infections via NAATs (26).

It is important to note that there has been a proliferation of Internet-based STI testing services that are advertised as private, safe, and confidential. These sites exist because noninvasive specimens can be self-obtained and can be stored and transported at ambient temperatures (25). Outreach screening may be one tool for approaching the chlamydia epidemic. The WHO estimates that 92 million new cases of curable chlamydia and 62 million cases of gonorrhea occur every year, >90% of which occur in settings with no or limited access to STI laboratory services (25) (http://www.who.int/vaccine_research/diseases/soa_std/en/index1.html), thus making Internet-based STI testing services an attractive alternative to clinic visits for some persons. For women, vaginal swabs may be preferred over FCU specimens for this type of screening (26). This and many other studies have demonstrated that self-collected vaginal swabs are suitable specimens for chlamydia and gonorrhea testing by NAATs, and their use has been recommended by the NIH and the CDC (17).

In summary, this paper reports a comparison of three commercially available NAATs for use with self-obtained vaginal swabs mailed to a testing site. We reported that all three NAATs can be used with good sensitivity and specificity to test self-collected vaginal specimens collected at home and mailed to a laboratory site for analysis. Our comparison demonstrated that the most superior assay platform was the Aptima Combo2. Although the ProbeTec assay performed with less sensitivity than the Aptima Combo2 assay, both assays demonstrated high specificity. The Amplicor PCR assay performed well for detection of chlamydia but cannot be recommended for detection of gonorrhea, because of its lower specificity, unless positives are confirmed. The approach of expanding STI testing venues to be home based via the Internet or other outreach programs could promote further inclusion of persons at risk. Outreach screening requires increased public health support for more research to address the increasing rates of STIs.