Abstract
Objective
To describe a Clostridium difficile infection (CDI) pseudo-outbreak caused by a faulty toxin assay lot, and to determine the effect of sensitivity, specificity, and repeat testing for C. difficile on CDI burden, positive predictive value (PPV), and false positive results.
Design
Outbreak investigation and criterion standard
Patients
Patients hospitalized at a tertiary-care hospital who had at least one toxin assay test for C. difficile from July 1, 2004 through June 30, 2006.
Methods
Run-control chart methodology and chi-square tests were used to compare CDI rates and proportion of positive tests before, during, and after the pseudo-outbreak. The impact of repeat testing was evaluated using three hypothetical models with a sample of 10,000 patients and various assay sensitivity and specificity estimates.
Results
In November of 2005, the hospital CDI rate increased from 1.5/1000 patient-days to 2.6/1000 patient-days (p<0.01) and the proportion of positive tests increased from 13.6% to 22.1% (p<0.01). Investigation revealed a pseudo-outbreak caused by a faulty toxin assay lot. A decrease of only 1.2% in the specificity of the toxin assay would result in a 32% increased in perceived CDI incidence at this institution. Using the manufacturer's specificity and sensitivity and this institution's testing practices, the PPV of the test decreased from 80.6% in a first test to 4.1% in patients who received three tests.
Conclusion
Specificity is as important as sensitivity when testing for CDI. False positive CDI cases can drain hospital resources and adversely affect patients. Repeat testing for C. difficile should be performed with caution.
Introduction
The incidence of Clostridium difficile infection (CDI) has been increasing (1–3). The diagnosis is typically established by testing for C. difficile and/or toxins A and/or B in a patient with diarrhea. In the United States, the diagnosis of CDI is most commonly established by enzyme-linked immunosorbent assays (ELISA) that detect toxins A and/or B (4). Due to concerns about poor sensitivity of toxin ELISAs, which can range from 50–95% (5), physicians commonly repeat testing in patients with persistent diarrhea if prior stool specimens tested negative. However, toxin ELISAs are not 100% specific and false positive results can occur (5). The false positive rate can increase, with a resulting drop in the positive predictive value, when testing is conducted in populations with a low prevalence of CDI or if the specificity of the assay decreases.
A low positive predictive value may adversely impact both patient care and CDI surveillance. Patients with a false positive test may be started on treatment for CDI and placed in contact precautions. Unnecessary treatment for CDI may result in adverse drug events, increase the risk of acquiring vancomycin-resistant enterococci, and may actually increase the risk of CDI after treatment is stopped (6;7). Contact precautions are associated with decreased patient satisfaction, decreased contact with healthcare workers, and increased adverse events (8–10). False positive tests increase the observed incidence of CDI for a healthcare facility as well, diverting resources away from other infection prevention and control problems. An increase in CDI rates due to false positive tests can also adversely impact the validity of public reporting of CDI (11;12).
The purpose of this study is to describe a five-week increase in CDI incidence corresponding to a faulty toxin assay lot, and the steps taken to identify the increase as a pseudo-outbreak. Because the false positive tests significantly increased the CDI rate and because it is common at our institution for repeat toxin assays to be ordered without a critical re-evaluation of the patient, we also determined the effect repeat testing for C. difficile had on positive predictive value, perceived CDI burden, and false positive results based on toxin testing ordering patterns at our institution.
Methods
Data collection
Barnes-Jewish Hospital (BJH) is a 1250-bed tertiary care, adult hospital that has used the Remel ProSpecT C. diff Toxin A/B Microplate Assay (Remel Inc., Lenexa, Kansas) to detect C. difficile toxin since July 1, 2004. The microbiology laboratory only tests diarrheal stools for the presence of C. difficile toxin. According to the manufacturer's package insert, this assay has a sensitivity of 90.3%, and specificity of 96.2%. An increase in healthcare onset CDI (defined as CDI onset >48 hours from admission (13)) was noted in November 2005 through routine Infection Prevention and Control surveillance. The pseudo-outbreak and investigation are described.
We collected all C. difficile toxin assay results from a retrospective cohort of patients hospitalized at BJH between July 1, 2004 and June 30, 2006. A CDI testing episode was defined as an episode of testing in which another assay was performed within seven days of a previous negative assay, up to a total of three assays. A patient could have more than one testing episode during a single admission if there were greater than seven days between toxin assays if all prior toxin assays were negative. Results of toxin assays after the first positive toxin assay during a single admission were excluded. These data were used to determine the proportion of patients at BJH with a negative assay that had a second toxin assay performed, and the proportion of patients with a second negative assay that had a third assay performed. The CDI prevalence over the study period was calculated from the results of the first toxin assay performed per CDI testing episode as follows:
d = [x{(1–0.903)/0.903}-y]/[{(1–0.962)/0.962}{(1–0.903)/0.903}−1]
f = d/0.962
e = z – f
CDI prevalence = e/z
Where x = the number of positive assays, y = the number of negative assays, d = the number of true negative assays, f = the number of episodes without disease, e = the number of episodes with CDI, z = the total number episodes.
Statistical analysis
Run-control chart methodology and chi-square tests were performed to determine if the increase in CDI rate and proportion of positive toxin assays associated with the suspect toxin assay lot were significantly different from before and after the specific lot was used. Any points greater than three standard deviations from the mean were considered abnormal by the run-control chart methodology.
To assess the impact of repeat testing on positive predictive value, perceived CDI burden, and false positive tests, three different scenarios were modeled. All models included a population of 10,000 patients with diarrhea in a healthcare facility tested for CDI that had identical baseline CDI prevalence and the same proportion of patients with repeat testing. The models differed by the performance characteristics of the C. difficile toxin assay. The first model used the test performance characteristics reported in the package insert for the assay used at our facility (sensitivity = 90.3%, specificity = 96.2%). In the second model the sensitivity was decreased to 75% while the specificity remained at 96.2%. In the third model the specificity was decreased to 95% and the sensitivity remained at 90.3%. Data was analyzed using SPSS version 15.0 (SPSS Inc, Chicago, IL) and EpiInfo version 3.4.1(July 3, 2007, EpiInfo™, CDC, Atlanta, GA).
Results
CDI Pseudo-outbreak
Between July 2004 and October 2005, the average monthly CDI incidence rate was 1.5/1000 patient days (pt-days). In November 2005, the CDI rate increased to 2.6/1000 pt-days (p<0.01) (Figure 1). This increase was noted by the BJH Infection Prevention and Control Department and was hospital wide. The Infection Prevention and Control Department contacted the BJH microbiology laboratory to determine if there were any changes in C. difficile testing. The increase in the number of positive toxin assays correlated with a change in the toxin assay lot number, which was used from November 4, through December 11, 2005. The proportion of positive assays between July 2004 and October 2005 was 13.6% (1490/10942). In November of 2005, the proportion of positive cases increased to 22.1% (148/671) (p<0.01). Over the next seven months the CDI rate and proportion of positive toxin assays decreased to an average of 1.3/1000 pt-days (p<0.01) and 11.8% (624/5284) (p<0.01), respectively, after the lot in question was no longer used. The increase in CDI rate and proportion of positive assays in November of 2005 was greater than three standard deviations above the mean (Figure 1, p<0.01). The manufacturer of the toxin ELISA was contacted and verified that the lot in question was prone to degradation with storage, causing negative specimens to appear weakly positive (telephone communication, January 19, 2006). Our laboratory confirmed a high number of weakly positive assays associated with the lot in question.
Impact of Sensitivity, Specificity, and Repeat Testing on Toxin Assay Results
There were 12,898 CDI testing episodes identified during the period from July 1st, 2004 to June 30th, 2006 (excluding the pseudo-outbreak period). There were 8,529 episodes in which one assay was performed, 2,910 episodes in which two assays were performed, and 1,459 episodes in which three assays were performed (Table 1). A second assay was performed on 41% of the initial 7,103 negative assays. A third assay was performed on 53.2% of the 2,744 second negative assays. Based on the sensitivity (90.3%) and specificity (96.2%) of the ELISA assay reported in the package insert, the calculated prevalence of CDI based on results of the first run assays was 14.9%.
Table 1.
Test number | Positive | Negative |
---|---|---|
First | 1,426 | 7,103 |
Second | 166 | 2,744 |
Third | 59 | 1,400 |
If 10,000 patients were tested for CDI using an assay with a sensitivity of 90.3%, specificity of 96.2%, and CDI prevalence of 14.9%, there would be 1,345 true positives, 323 false positives, and 145 false negatives for a positive predictive value of 80.6% and negative predictive value of 98.3% (Table 2). The resulting prevalence of CDI in patients with a negative assay would be 1.7%. If a second stool sample was tested on 41% of the patients with a first negative assay, there would be 53 true positives, 127 false positives, and 6 false negatives for a positive predictive value of 29.6% and negative predictive value of 99.8%. The resulting prevalence of CDI in patients with a negative assay would be 0.2%. If a third stool sample was tested on 53.2% of the patients with a second negative assay, there would be 3 true positives, 65 false positives, and no false negatives for a positive predictive value of 4.1% and negative predictive value of 100%. Overall, of the 1,918 positive assays, a total of 515 (26.9%) would be false positives.
Table 2.
Test Number | CDI Prevalence | Positive n | Negative n | False Positive n (%) | PPV | NPV |
---|---|---|---|---|---|---|
Sensitivity = 90.3%, Specificity = 96.2% | ||||||
First | 14.9% | 1,669 | 8,331 | 323 (19.4) | 80.6% | 98.3% |
Second | 1.7% | 181 | 3,232 | 127 (70.2) | 29.6% | 99.8% |
Third | 0.2% | 68 | 1,719 | 65 (95.6) | 4.1% | 100.0% |
Sensitivity = 75%, Specificity = 96.2% | ||||||
First | 14.9% | 1,441 | 8559 | 323 (22.4) | 77.6% | 95.6% |
Second | 4.4% | 242 | 3265 | 127 (52.5) | 47.3% | 98.8% |
Third | 1.2% | 80 | 1655 | 65 (81.3) | 18.9% | 99.7% |
Sensitivity = 90.3%, Specificity = 95.0% | ||||||
First | 14.9% | 1,771 | 8,229 | 426 (24.1) | 76.0% | 98.2% |
Second | 1.8% | 219 | 3,371 | 166 (75.8) | 24.4% | 99.8% |
Third | 0.2% | 87 | 1,676 | 84 (97.7) | 3.2% | 100.0% |
|
CDI = C. difficile infection, PPV = positive predictive value, NPV = negative predictive value
In the model where the sensitivity of the assay was reduced to 75% (Table 2), the CDI prevalence for patients with one prior negative assay was 4.4% and 1.2% for patients with two prior negative assays. As a result, the negative predictive value for the first, second, and third test were 95.6%, 98.8%, and 99.7%, respectively, and the positive predictive values were 77.6%, 47.3%, and 18.9%, respectively. The total number of false positive assays (515) was unchanged. In the model were the specificity of the assay was reduced to 95% (Table 2), the CDI prevalence for patients with one prior negative assay was 1.8% and 0.2% for patients with two prior negative assays. The negative predictive values in this model were 98.2%, 99.8%, and 100% and the positive predictive values were 76.0%, 24.4%, and 3.2% for the first, second, and third assays, respectively. The total number of false positive assays increased to 676 (32.6% of all positive assays).
Discussion
Many clinicians are concerned about the relatively low sensitivity of C. difficile toxin assays and the risk of false negative test results. However specificity is also an important characteristic, but few data exist on how specificity affects the number of false positive results, the positive predictive value of the test, and perceived CDI burden when testing for C. difficile (5;14–17). In addition, there are few data that attempt to assess how the prevalence of CDI impacts the interpretation of C. difficile toxin assays (5;14–17). After a pseudo-outbreak of CDI due to a decrease in the specificity of a toxin ELISA, we quantified the impact declines in sensitivity, specificity, and CDI prevalence have on performance of C. difficile toxin testing based on toxin assay ordering patterns at our healthcare facility.
This study demonstrates the importance of specificity when testing for C. difficile. An absolute decrease in specificity of only 1.2% resulted in an additional 103 false positives tests during just the first round of testing (Table 2). Overall when accounting for repeat testing this decrease in specificity resulted in an additional 161 false positive tests, a 32% increase in the perceived CDI incidence. Based on the increase in the proportion of positive tests during our pseudo-outbreak, we estimate the specificity of the assay decreased from 96.2% to between 84% and 87%. There are several factors that can lead to a decrease in specificity of toxin ELISAs, including assay malfunction, improper handling of stool specimens (18), and insufficient cleaning of the plate washer (email communication, Roberta Carey, PhD, Centers for Disease Control and Prevention, January 12, 2006). The former will cause an increase in perceived CDI incidence in all facilities that use that particular assay. The latter two can cause increases that may be operator dependent, and therefore more difficult to identify.
Repeat testing also may significantly increase the number of false positive test results, with the number of false positives far outnumbering the number of true positives on repeat testing. This occurs because the prevalence of CDI decreases each time patients with a positive test are removed from the population being tested. As the prevalence decreases, the positive predictive value of the test also decreases. When calculations were performed using the reported sensitivity and specificity of the assay used at our facility during the study period, the positive predictive value of the second and third tests were 30% and 4%, respectively. The argument used to support repeat testing is true cases are missed because of the low sensitivity of toxin ELISAs. However, when the sensitivity for the calculations was reduced to 75%, the negative predictive value for the first test was still 96%. Conversely, the positive predictive value of the second test fell to 47% and to 19% for the third test. The risk of missing true cases despite an excellent negative predictive value needs to be balanced by the significant decrease in positive predictive value.
False positive tests are not without risk. It is recommended all patients with CDI be placed in contact precautions. Several studies have identified patients in contact precautions may be at increased risk for adverse outcomes compared to patients not in contact precautions. Kirkland and Weinstein found healthcare workers were significantly less likely to enter the room and have direct contact with patients in contact precautions (8). Saint et al found attending physicians at a teaching hospital were significantly less likely to examine patients in contact precautions (9). Stelfox et al found patients in contact precautions were more likely to have vital signs incompletely recorded or not recorded at all compared to patients not in contact precautions (10). Patients in contact precautions were also less satisfied with their care than patients not in contact precautions and were more likely to experience an adverse event (10). A false positive test for CDI may also result in unnecessary CDI treatment. In addition to placing a patient at risk for an adverse drug reaction, the patient may be at higher risk for CDI when the treatment is stopped (7). False positive test results may also negatively impact a healthcare facility. They result in enhancement of CDI prevention measures that are not necessary (19). False positive testes may also impact the validity of CDI surveillance and reporting (11;12).
There are several potential limitations to this study. Toxin assay test performance estimates were calculated based on retrospectively obtained data. The CDI prevalence was calculated based on the reported sensitivity and specificity of the toxin ELISA without confirmation by stool culture for C. difficile. However the calculated prevalence of CDI (15%) is consistent with previous literature (6). It is possible physicians are more likely to repeat testing if there is a high index of suspicion for CDI, and therefore the positive predictive value of repeat testing would be higher than calculated for this study. In one study physicians were more likely to initiate empiric therapy for CDI while tests results were pending if the patient had fever, an abnormal abdominal examination, or an abnormal leukocyte count (14). All of these signs were significantly associated with having a positive test for CDI. This indicates physicians are able to identify patients whose diarrhea may be more likely due to CDI than other causes. The proportion of tests positive for CDI with repeat testing would be higher than otherwise expected if clinical index of suspicion is taken into account when repeat testing is ordered. However the actual proportion of positive tests on repeat testing during the study period (5.7% for the second test and 4% for the third test) was almost identical to the calculations for the first model (5.3% and 4%, respectively), indicating clinical index of suspicion is not taken into account when repeating testing for CDI occurs at our institution. This is consistent with observations of the authors (VJF, DKW, ERD). This also suggests the reported sensitivity and specificity of the toxin ELISA are close to the true sensitivity and specificity of the ELISA when performed by our microbiology laboratory.
Several recent studies have reported that repeat testing is low yield and discourage the practice (14–16;20). These studies did not attempt to assess whether positive results on repeat testing represent true positives or false positives. Our study suggests most positive tests on repeat testing are false positives. This provides further evidence that repeat testing for C. difficile should not be ordered routinely since false positive tests may result in patient harm or negatively impact a healthcare facility. In addition, some studies have found results of repeat tests often do not alter clinical management of patients or patient outcomes (14).
This study was prompted by a pseudo-outbreak of CDI due to a faulty lot of toxin ELISA assays. Its findings have changed the approach to CDI testing at BJH, but it also has potential implications for CDI surveillance efforts in the US. As a result of this pseudo-outbreak, our microbiology laboratory now monitors the proportion of positive tests as a quality control measure. The findings of this study have also changed our institutional recommendations for C. difficile testing. Although not officially recommended, the practice of writing an order for ”C. diff × 3” to ensure up to three stool specimens were sent to the microbiology laboratory for testing if the prior test came back negative was tolerated. We now actively discourage the practice of repeat testing unless there is a high index of suspicion for CDI and the results of repeat testing will alter patient management. This study also has implications for CDI surveillance in the US because toxin ELISA assays are the most commonly used method to diagnose CDI in the US (4). Declines in specificity and false positive assays due to repeat testing may both falsely elevate CDI prevalence. Currently it is unknown how frequently the specificity of toxin ELISA assays may vary, therefore it is unclear how declines in specificity may have impacted current CDI prevalence estimates (3). However it is very possible repeat testing has had a significant impact since repeat testing appears to be a common occurrence based on the experience of the authors of this study.
Although sensitivity is an important test characteristic, specificity and prevalence of disease are important as well. Even with a sensitivity of 75%, the negative predictive value of toxin ELISAs remains high enough to be clinically useful. Repeat testing should not be routinely performed because the decrease in positive predictive value that occurs with repeat testing decreases the usefulness of a positive test result, and may result in harm to the patient.
Acknowledgments
Financial support: This work was supported by grants from the Centers for Disease Control and Prevention (UR8/CCU715087-06/1, 1U01C1000333-01) and the National Institutes of Health (KL2RR024994-01, K24AI067794-01)
Footnotes
Potential conflicts of interest: E.R.D.: consultant – Merck, Becton-Dickenson, Salix; research – Viropharma
V.J.F.: speakers' bureaus – Verimetrix, Steris
D.K.W.: consultant – 3M Healthcare, Chitopure Inc, Cardinal Health; research – 3M Healthcare, Sage Products
Preliminary data was presented in part at the 18th Annual Meeting of the Society for Healthcare Epidemiology of America, Orlando, FL (April 5-8, 2008).
Reference List
- (1).Goorhuis A, Van der KT, Vaessen N, Dekker FW, Van den BR, Harmanus C, et al. Spread and epidemiology of Clostridium difficile polymerase chain reaction ribotype 027/toxinotype III in The Netherlands. Clin Infect Dis. 2007;45(6):695–703. doi: 10.1086/520984. [DOI] [PubMed] [Google Scholar]
- (2).Loo VG, Poirier L, Miller MA, Oughton M, Libman MD, Michaud S, et al. A predominantly clonal multi-institutional outbreak of Clostridium difficile-associated diarrhea with high morbidity and mortality. N Engl J Med. 2005;353(23):2442–2449. doi: 10.1056/NEJMoa051639. [DOI] [PubMed] [Google Scholar]
- (3).McDonald LC, Owings M, Jernigan DB. Clostridium difficile infection in patients discharged from US short-stay hospitals, 1996–2003. Emerg Infect Dis. 2006;12(3):409–415. doi: 10.3201/eid1203.051064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (4).Gerding DN. New definitions will help, but cultures are critical for resolving unanswered questions about Clostridium difficile. Infect Control Hosp Epidemiol. 2007;28(2):113–115. doi: 10.1086/512550. [DOI] [PubMed] [Google Scholar]
- (5).Peterson LR, Manson RU, Paule SM, Hacek DM, Robicsek A, Thomson RB, Jr., et al. Detection of toxigenic Clostridium difficile in stool samples by real-time polymerase chain reaction for the diagnosis of C. difficile-associated diarrhea. Clin Infect Dis. 2007;45(9):1152–1160. doi: 10.1086/522185. [DOI] [PubMed] [Google Scholar]
- (6).Gerding DN, Johnson S, Peterson LR, Mulligan ME, Silva J., Jr. Clostridium difficile-associated diarrhea and colitis. Infect Control Hosp Epidemiol. 1995;16(8):459–477. doi: 10.1086/648363. [DOI] [PubMed] [Google Scholar]
- (7).Johnson S, Homann SR, Bettin KM, Quick JN, Clabots CR, Peterson LR, et al. Treatment of asymptomatic Clostridium difficile carriers (fecal excretors) with vancomycin or metronidazole. A randomized, placebo-controlled trial. Ann Intern Med. 1992;117(4):297–302. doi: 10.7326/0003-4819-117-4-297. [DOI] [PubMed] [Google Scholar]
- (8).Kirkland KB, Weinstein JM. Adverse effects of contact isolation. Lancet. 1999;354(9185):1177–1178. doi: 10.1016/S0140-6736(99)04196-3. [DOI] [PubMed] [Google Scholar]
- (9).Saint S, Higgins LA, Nallamothu BK, Chenoweth C. Do physicians examine patients in contact isolation less frequently? A brief report. Am J Infect Control. 2003;31(6):354–356. doi: 10.1016/s0196-6553(02)48250-8. [DOI] [PubMed] [Google Scholar]
- (10).Stelfox HT, Bates DW, Redelmeier DA. Safety of patients isolated for infection control. JAMA. 2003;290(14):1899–1905. doi: 10.1001/jama.290.14.1899. [DOI] [PubMed] [Google Scholar]
- (11).Final Report of Rates of Clostridium difficile for Ohio Hospitals and Nursing Homes, January 1 -December 31, 2006. 2007. [Google Scholar]
- (12).Eggertson L. Hospitals to report C. difficile and MRSA. CMAJ. 2007;176(10):1402–1403. doi: 10.1503/cmaj.070507. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (13).McDonald LC, Coignard B, Dubberke E, Song X, Horan T, Kutty PK. Recommendations for surveillance of Clostridium difficile-associated disease. Infect Control Hosp Epidemiol. 2007;28(2):140–145. doi: 10.1086/511798. [DOI] [PubMed] [Google Scholar]
- (14).El Gammal A, Scotto V, Malik S, Casey KC, Cody R, Alcid DV, et al. Evaluation of the clinical usefulness of C. difficile toxin testing in hospitalized patients with diarrhea. Diagn Microbiol Infect Dis. 2000;36(3):169–173. doi: 10.1016/s0732-8893(99)00129-7. [DOI] [PubMed] [Google Scholar]
- (15).Cardona DM, Rand KH. Evaluation of repeat Clostridium difficile enzyme immunoassay testing. J Clin Microbiol. 2008;46(11):3686–3689. doi: 10.1128/JCM.00931-08. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (16).Aichinger E, Schleck CD, Harmsen WS, Nyre LM, Patel R. Nonutility of repeat laboratory testing for detection of Clostridium difficile by use of PCR or enzyme immunoassay. J Clin Microbiol. 2008;46(11):3795–3797. doi: 10.1128/JCM.00684-08. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (17).Bennett RG, Laughon BE, Mundy LM, Bobo LD, Gaydos CA, Greenough WB, III, et al. Evaluation of a latex agglutination test for Clostridium difficile in two nursing home outbreaks. J Clin Microbiol. 1989;27(5):889–893. doi: 10.1128/jcm.27.5.889-893.1989. [DOI] [PMC free article] [PubMed] [Google Scholar]
- (18).Mayer J, South B, Mooney B, Deryke C, Alexander D, Rubin M, et al. Surveillance of Clostridium difficile-Associated Disease Based ib Toxin Enzyme Immunoassay Results: Did a Problem with Testing Lead to a Pseudo-Epidemic?. Society for Healthcare Epidemiology of America Annual Meeting.2008. [Google Scholar]
- (19).Dubberke ER, Gerding DN, Classen D, Arias KM, Podgorny K, Anderson DJ, et al. Strategies to prevent clostridium difficile infections in acute care hospitals. Infect Control Hosp Epidemiol. 2008;29(Suppl 1):S81–S92. doi: 10.1086/591065. [DOI] [PubMed] [Google Scholar]
- (20).Mohan SS, McDermott BP, Parchuri S, Cunha BA. Lack of value of repeat stool testing for Clostridium difficile toxin. Am J Med. 2006;119(4):356–358. doi: 10.1016/j.amjmed.2005.08.026. [DOI] [PubMed] [Google Scholar]