Using a rapid communication approach to improve a POC Chlamydia test

Abstract

Chlamydia trachomatis is a common sexually transmitted infection (STI) in young women. Available point-of-care (POC) diagnostic tests perform poorly, but development of new devices can be costly and time consuming. We explored the feasibility (user friendliness) and test characteristics (sensitivity, specificity) of a new prototype device to detect Chlamydia in adolescent women by using small numbers of subjects and rapid communication with the manufacturer. We compared cervical POC test results to the gold standard (cervical nucleic acid amplification testing). We also assessed the accuracy of the POC test on self-collected vaginal swabs by comparing results to cervical NAAT and to the cervical POC test. We frequently reviewed user experience and test results with the manufacturer.

RESULTS

Feasibility

Initial device malfunctions were identified and corrected. The device would be easy to use in a non-clinical setting, as it is self-contained and the color change for some specimens was dramatic and immediate.

Accuracy

Initial prototypes demonstrated low sensitivities (38%) for vaginal and cervical swabs. After feedback, the company developed new prototypes with improved sensitivity (80%). However, the increased sensitivity was accompanied by a high percentage of indeterminate results and false positives that lowered specificity.

I. Introduction

Chlamydia trachomatis (CT) is the most frequently reported bacterial sexual transmitted infection (STI) in the U.S.A. [1]. Young women are disproportionally affected, with recent estimates that up to 4.7 % of women age 18-26 have asymptomatic Chlamydia infections [2]. According to the Centers for Disease Control and Prevention (CDC) in 2008, Chlamydia was reported most often in women 15 to 19 years old (3,276 cases per 100,000 women) and followed by women 20 to 24 years of age (3,180 cases per 100,000 women) [1]. Despite its high prevalence, especially in younger women, many cases go untreated due to its mild and often asymptomatic presentation. The long term effects of an infection can be devastating for women and include ectopic pregnancy, pelvic inflammatory disease, and infertility. Current recommended testing for Chlamydia is nucleic acid amplification test (NAAT), which is highly sensitive and specific when performed on endocervical, urine, or vaginal specimens [3]. However, the disadvantages to NAAT, in addition to being expensive, are that it takes days to obtain results. Women tested with NAAT face an average of 3-14 days between testing and treatment [4], [5]. This delay between visit and treatment is especially problematic for adolescents, as up to 20% of the adolescent women are lost to follow up after initial testing.

In order to increase diagnosis and reduce time for treatment, point-of-care (POC) tests which can be performed and read on the same day are now available for detecting Chlamydia trachomatis thus providing immediate results and consultation with a clinician. Currently, three POC Chlamydia tests have been cleared by the United States Food and Drug Administration (FDA). Compared to NAAT, these tests have low sensitivities (25-65%), but good specificities (97-100%) when testing was done with endocervical swabs [6]—[8]. However, these sensitivities are too low to be useful in clinical settings where NAAT is available.

POC tests would be even more useful if they performed well on self-collected vaginal samples, as this would expand the settings where such tests could be used. One study of an available POC test found a similarly disappointing sensitivity (33%) and high specificity (100%) when using vaginal swabs [8]. A POC test available in Europe has shown more promising results, with sensitivity of 83.5% and specificity of 99% using vaginal swabs [9]. In the same study, self-collected vaginal swabs were as sensitive as clinician-collected vaginal swabs (81% vs. 78%) compared to urine NAAT [9].

Because of the limitations of currently available POC tests, research and development of new devices is urgently needed, but development of large preclinical trails can be costly and time intensive. In this study, we aimed to explore the feasibility (user friendliness) and test characteristics (sensitivity, specificity) of a new prototype device to detect Chlamydia in adolescent women by using small numbers and rapid communication with the device manufacturer (BioVei, Inc, Oakley, CA,). The device is a self-contained, two-step enzyme-based detection system that the company has developed for detection of other pathogens. The Chlamydia prototype contains a chromogenic substrate for the specific enzyme coupled to a fluorescent tag in an aqueous solution. When exposed to Chlamydia, the substrate undergoes an enzymatic reaction, which, when coupled with the developer, produces a blue or blue/mottled color change.

Prior to study start-up, the manufacturer had already evaluated the device with stored endocervical samples. They report the device to have an accuracy of 90% or greater when compared to NAAT/PCR (http://www.biovei.com/faq.htm ). Also prior to study start-up, representatives from the company visited study staff at Cincinnati Children’s and performed serial dilutions of isolated, highly concentrated Chlamydia (provided by the Gaydos’ laboratory). There was no problem detecting a positive result, as dilutions of 10⁶, 10⁵, 10⁴, 10³, 10², 10, 1, and 10⁻¹ all produced an obvious blue color change. Dilutions performed at 10⁻² and 10⁻³ initially yielded a negative result, but when the test was performed again with these dilutions, the results demonstrated the positive blue color change.

The aims of this study were to assess the feasibility and test performance of the prototype devices with clinical samples in real time. First, we compared cervical POC device test results to those of cervical NAAT as the gold standard. Secondly, because self-collection of vaginal swabs is a newly recommended strategy for STI testing, we assessed the accuracy of the prototype POC test on self-collected vaginal swabs by comparing results to the gold standard cervical NAAT and to the cervical POC test collected by a clinician. At frequent intervals, we reviewed user experience and test results with the manufacturer.

II. Methods

Women were recruited from the City of Cincinnati’s Health Department STI Clinic, and Cincinnati Children’s Hospital Medical Center’s Teen Health Center and Emergency Department. Written informed consent was obtained with a wavier of parental permission for participants under 18 years of age. Women were eligible for participation if they were between the ages of 14-30, reported a history of vaginal sex within the past three months, were getting a pelvic exam with Chlamydia testing on the day of their visit, and were willing to self-collect vaginal swabs. Potential participants were ineligible if they had used oral antibiotics or topical vaginal medications within two weeks prior to recruitment.

Each participant contributed a brief survey including demographics and clinical history items, a self-collected vaginal swab (collected either before or after the pelvic exam depending on clinical flow) and a clinician-obtained cervical swab (obtained during the pelvic exam). The swabs were tested with prototype devices supplied by the manufacturer, following their established protocol. In brief, after the swabs were placed in the self-contained device (Figure 1), study staff broke the ampoule with the self-contained reagent A. The device was then kept in an upright position for 10 minutes as the reagent ran down the swab tip. After 10 minutes, the device was turned upside down and study staff broke the ampoule containing reagent B, and kept the device upside down (with B at the top) for one to two minutes before comparing any color on the swab tip to the color chart. A positive result was blue and no color change (or faint yellow or pink) was negative. The reference Chlamydia NAAT test (Strand Displacement Assay (SDA), BD Probetek ET, Becton Dickinson, Sparks, MD) was sent to the clinical lab, and read per usual procedures. The lab and clinicians were blinded to the POC test results. Chlamydia infection was defined as a positive cervical NAAT. Sensitivity, specificity, 95% confidence intervals (CI), and percent agreement between vaginal and cervical swabs were calculated for the device. In addition, we recorded subject and staff comments about device performance.

Fig. 1.

Prototype Self-contained collection and testing kit

III. RESULTS

We recruited 84 women; one was withdrawn from the study because she was on her menses, which, per device instructions, may interfere with the reading of test results. The mean age of these 83 participants was 18.3 years and 87% were black. Thirteen (15.7%) were Chlamydia NAAT positive.

A. Feasibility

We used the first prototype device (A) with the first seven participants. However, the research staff reported that the color change was not distinguishable (i.e. couldn’t tell a positive result from a negative result) so the company provided a second version (B). We recruited 27 women using device B. The readability problem was greatly improved; however, we continued to experience problems with the device. When research staff crushed the ampoule A that contained the substrate, glass shards would puncture the ampoule and, on two occasions, cut the fingers of the staff members, and reagents leaked out of the device. We halted study recruitment and communicated our results to the company, which promptly sent two more prototypes: C was accompanied by specialized crushers used to break the ampoule, and D contained a smaller pludget so there would be more room in the device for the substrate fluid to easily run down to the tip of the swab. These changes successfully solved the problems of staff injury and reagent leakage. However, readability again became a problem with 5 (10%) of 50 samples declared indeterminate by study staff.

B. Test performance

Because of difficulties with device performance described above, we analyzed results stratified by type of prototype device. Table I contains the raw data for POC and NAAT test results, stratified by prototype and source (cervical or vaginal). In order to assess test performance, because of small sample size and the similarities between devices A and B, and between C and D, we combined these devices (Table II). Of the 33 participants who used either device A or B, the device was 38% sensitive (CI: 8-76%) compared to NAAT for both cervical and vaginal samples, while specificity was 77% (CI: 54-92%) for cervical and 64% (CI: 40-83%) for vaginal swabs (Table II). Agreement between vaginal and cervical swab results was moderate (68% agreement, kappa 0.62).

TABLE I.

test results by device type

	N	POCT Vaginal +	POCT Cervical +	NAAT +	NAAT −
Device A	7	7	6*	2	5
Device B	26	6	2^	6	20
Device C	15	10	8#	0	15
Device D	35	27*	22	5	30

^*one result was missing

^{^}two results were indeterminate

^#four results were indeterminate

TABLE II.

Test performance for prototype devices compared to NAAT results: TP=true positive, TN= true negative, FN=false negative, FP=false positive

Device	Specimen type	POC result compared to NAAT Positive (n)	Sensitivity %	95% CI	POC result compared to NAAT negative (n)	Specificity %	95% CI
Device A and B (N=33)	Cervical*	TP (3)	38%	8-76%	TN (17)	77%	54-92%
		FN (5)			FP (5)
	Vaginal	TP (3)	38%	8-76%	TN (15)	60%	40-83%
		FN (5)			FP (10)
Device C and D (N=50)	Cervical**	TP (4)	80%	28-99%	TN (15)	37%	22-53%
		FN (1)			FP (26)
	Vaginal***	TP (4)	80%	28-99%	TN (11)	25%	13-40%
		FN (1)			FP (33)

^*two results were indeterminate, one result was missing

^**four results were indeterminate

^***one result was indeterminate

Another 50 participant samples were tested with prototypes C and D (those with new crushers and smaller pludgets). Four cervical swab POC results and one vaginal swab result were indeterminate. The new devices were 80% sensitive (CI: 28-99%) compared to NAAT for both cervical and vaginal samples. Specificity was 37% for cervical (CI: 22-53%) and 25% (CI: 13-40%) for vaginal swabs (Table II). Percent agreement between self-collected (vaginal) and clinician-collected (cervical) was 80% (kappa 0.72). Because the two groups exhibited markedly different sensitivity and specificity, we did not analyze all samples as a single group.

IV. DISCUSSION

Based on our results, the device is not yet ready for clinical use.. The rapid communication strategy was efficient in that after feedback from our research staff regarding device performance, the company did develop new prototypes that showed a much increased sensitivity (80% with both vaginal and cervical swabs), and prevented staff injury by using a “crusher” device to break the ampoules. Although device sensitivity appeared to improve using this tactic (from 38% to 80%), due to small sample sizes this increase was not statistically significant. In addition, the increased sensitivity was accompanied by a high percentage of indeterminate results and false positives that lowered specificity from 60-77% to 25-37%, which was statistically significant at p=.05. Therefore, while safety issues improved, overall device performance is not ready for larger clinical trials.

If the problems with accuracy are resolved, this device has good potential as a point-of-care test in the future. The device would be fairly easy to use in a home or non-clinical office setting, as it is self-contained and the color change for some specimens was dramatic and immediate (within minutes). This would be a clear, visual way to determine if Chlamydia infection is present at the point-of-care, a strategy that would greatly improve STI care. The goal then for the company would be to try to find a way to generate higher sensitivity without sacrificing specificity. In fact, a highly specific test with moderate sensitivity (75-80%) would be good enough to dramatically reduce the possibility that a patient would be lost to follow-up after testing.

The device test performance is not much different than what is reported for other POC tests for Chlamydia. Currently, three tests have been cleared by the United States FDA: the optical immunoassay (OIA) (Inverness [formerly Biostar], Princeton, NJ), Clearview Chlamydia (Inverness, Princeton, NJ), and QuickVue (Quidel, San Diego, CA). These three POC tests for Chlamydia have reported low sensitivities (25-65%) compared to NAAT, but good specificities (97-100%) and most testing was done using endocervical swabs [6]—[8]. The only study that tested vaginal swabs with Clearview® found a similarly disappointing sensitivity (33%) and high specificity (100%) [8]. The POC test available in Europe (Chlamydia Rapid Test, Diagnostics for the Real World, LTD, Cambridge, UK ) is much more accurate (sensitivity 83.5 % and specificity of 99% using vaginal swabs, compared to NAAT). However, there is no indication that the company intends to seek FDA clearance.

One limitation to our study was our small sample size. Part of the reason for this was that when we encountered problems with a prototype, we stopped recruitment until we communicated with the manufacturer and they sent us a new prototype which resolved the problem. In the case of devices A and B, this was mainly technical issues (could not determine color change, ampoules breaking, etc.). Although these were fixed with the next two prototypes, C and D, the tests were continuing to generate poor specificities, with a high number of false positive results. As these small samples demonstrated rough estimates of test performance, we believed that it was a waste of time and resources to study a larger number of women with poorly performing prototypes. However, our small sample size (n=83) results in very imprecise estimates of sensitivity and specificity as the small sample size leads to an overlapping of confidence intervals. Thus, further testing on a larger sample of adolescents is needed to determine more reliable estimates of the properties of these tests.

Another limitation is that we did not use all four prototypes on the same patient sample, which could have affected device results. For example, it is possible that certain prototypes were able to detect positive infections only if the patient had a high bacterial load present. Nor did we collect data on co-infections or symptoms which could have affected results. Future research on this POC test should examine the affect of such compounding factors on device performance.

POC testing for STIs, especially for an infection as highly prevalent as Chlamydia, enables patients to receive immediate results, care, counseling, and can help prevent reinfection. This method of testing is particularly beneficial to adolescent women, who often are disproportionally affected, tend not to come in for follow-up results, and may often even initially resist clinical testing with a pelvic examination. There is an urgent need for continued development of new devices while refining the feasibility and accuracy of existing devices. Ultimately, we wish to create easy to use, sensitive POC tests that can provide both quick results in a clinical setting and eventually allow individuals to test themselves in non-clinical settings as well.

We showed that an academic-industry collaboration with frequent communication is an important step between bench science and large clinical trials. Although preliminary results from our industry partner looked quite promising, by using the device in a clinical setting in real time, we uncovered three problems that must be resolved before large scale manufacture and trials are performed. In order to move the POC STI test device field forward, other researchers and companies may want to consider this strategy to improve the chances of developing an accurate and user-friendly device.