Evaluation of four algorithms to identify incident syphilis among HIV-positive men who have sex with men engaged in primary care

Abstract

We developed four algorithms for syphilis among HIV-positive MSM engaged in primary care in 2016–2017. Clinician-based diagnosis from chart reviews was the gold standard. Sensitivities ranged from 74.2% to 93.9%. Specificities were >99% with positive and negative predictive values >95%. Algorithms that incorporated treatment data performed best.

Short Summary

A study of four algorithms for syphilis showed that algorithms incorporating testing and treatment data perform best.

Introduction

Since the early 2000s, the incidence of syphilis in the United States has reached levels not seen since the early 1990s.^1,2 In 2016, men who have sex with men (MSM) comprised 58% of primary and secondary syphilis diagnoses and 47% of MSM with primary and secondary syphilis were HIV-positive. Multi-site clinical cohorts and registries based on electronic medical record systems represent an opportunity to evaluate the secular trends in syphilis diagnoses among key populations; however, there are no consistent diagnostic criteria for incident syphilis that can be readily applied to these data without laborious individual chart review. For example, diagnosis codes reflecting syphilis may not always be entered into the medical record; clinical interpretation of non-treponemal test (i.e., rapid plasma reagin [RPR] and venereal disease research laboratory [VDRL]) titers may differ by provider and site; use of traditional and reverse testing algorithms may not be consistent across laboratories; and, treatment may be presumptive without clinical or laboratory confirmation. Therefore, we sought to develop standardized algorithms for the identification of incident syphilis for use with clinical cohort and registry data.

Methods

Our study population included HIV-positive MSM enrolled in the Center for AIDS Research Network of Integrated Clinical Systems (CNICS) cohort at Fenway Health (FH) in Boston, MA.³ We conducted a retrospective analysis of syphilis diagnoses among participants who had at least one clinical visit in 2016 or 2017 with screening for syphilis.

FH uses the traditional syphilis screening algorithm with an initial non-treponemal test (RPR) followed by a treponemal test (fluorescent treponemal antibody absorption [FTA-ABS]).⁴ We developed algorithms for the diagnosis of syphilis based on RPR results. We felt this approach most appropriate due to the high risk of incident and recurrent syphilis among HIV-positive MSM.^5,6 We applied all four algorithms to the RPR and medication data extracted from the electronic medical record system (Centricity, General Electric, Boston, MA). We then performed a chart review to assess clinician-based diagnosis of syphilis as the gold standard to which we compared our algorithms. We reviewed the charts of all patients, including those who did not meet criteria for incident syphilis in order to fully evaluate the performance of the algorithms. Based on chart review, we were also able to detect syphilis cases diagnosed and/or treated at a location other than FH.

Algorithm A had three criteria: (1) an RPR at entry into the cohort of greater than 1:8; (2) an RPR of 1:4 or greater after a prior non-reactive RPR regardless of whether the patient had ever had a reactive RPR in the past; and, (3) an increase in RPR titer of four-fold or greater from one titer to the next. Algorithm B had four criteria: (1) an RPR at entry into the cohort of greater than 1:8; (2) an RPR of 1:4 or greater after a prior non-reactive RPR among patients with a prior reactive RPR; (3) any positive RPR after a prior non-reactive RPR among patients who have never had a prior reactive RPR; and, (4) an increase in RPR titer of four-fold or greater from one titer to the next. Algorithm C captured treatment with benzathine penicillin 2.4 million units administered intramuscularly for one to three doses or doxycycline 100 mg by mouth twice per day for 14–28 days subsequent to (1) any positive RPR and (2) an increase in RPR titer of four-fold or greater from one titer to the next.⁴ Finally, algorithm D was the same as algorithm C but allowed for the four-fold increase in RPR titer to evolve over repeated titers rather than from one RPR to the next.

We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), of each algorithm with and without outside cases. Because ignoring outside cases might inflate the performance of the algorithms, the inclusion of outside cases estimates their performance without this expected bias in settings were HIV and STI care are not always co-located.

We compared sensitivities of the algorithms using a two-sample test of proportions with a P<0.05 level of statistical significance. We used STATA 14.2 (StataCorp, College Station, TX).

Results

During the study period, 1719 men were screened for syphilis; there were 202 (11.8%) cases of syphilis that were true positives based on chart review. Twenty-two (10.9%) of the true positives were diagnosed and/or treated for syphilis at an outside location.

Tables 1 and 2 present the 2×2 contingency tables, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of each algorithm. Specificities, PPVs, and NPVs were similarly high across algorithms. The sensitivities were 74.2%, 77.8%, 82.2%, and 83.7% for algorithms A, B, C, and D, respectively, when including outside cases (Figure 1). Excluding the outside cases resulted in sensitivities of 75.6%, 79.4%, 92.2%, 93.9% for algorithms A, B, C, and D, respectively. There were statistically significant differences between algorithms A and D when including (P=0.025) and excluding outside cases (P<0.001) and between algorithms A and C when excluding outside cases (P<0.001). There were no other statistically significant differences between algorithms.

Table 1.

2 × 2 contingency tables and performance characteristics of four algorithms (including outside cases) for the identification of incident syphilis among HIV-positive men who have sex with men, Boston, MA, 2016–2017.

Algorithm A				Algorithm B
	Chart review				Chart review
Algorithm	Positive	Negative	Total	Algorithm	Positive	Negative	Total
Positive	150	5	155	Positive	157	6	163
Negative	52	1518	1570	Negative	45	1517	1562
Total	202	1523	1725	Total	202	1523	1725
Sens, %	Spec, %	PPV, %	NPV, %	Sens, %	Spec, %	PPV, %	NPV, %
74.2	99.7	96.8	96.7	77.7	99.6	96.3	97.1
Algorithm C				Algorithm D
	Chart review				Chart review
Algorithm	Positive	Negative	Total	Algorithm	Positive	Negative	Total
Positive	166	4	170	Positive	169	5	174
Negative	36	1520	1556	Negative	33	1518	1551
Total	202	1524	1726	Total	202	1523	1725
Sens, %	Spec, %	PPV, %	NPV, %	Sens, %	Spec, %	PPV, %	NPV, %
82.2	99.7	97.6	97.7	83.7	99.7	97.1	97.9

NPV, negative predictive value; PPV, positive predictive value; RPR, rapid plama reagin; sens, sensitivity; spec, specificity

Table 2.

2 × 2 contingency tables and performance characteristics of four algorithms (excluding outside cases) for the identification of incident syphilis among HIV-positive men who have sex with men, Boston, MA, 2016–2017.

Algorithm A				Algorithm B
	Chart review				Chart review
Algorithm	Positive	Negative	Total	Algorithm	Positive	Negative	Total
Positive	136	5	141	Positive	143	6	149
Negative	44	1526	1570	Negative	37	1525	1562
Total	180	1531	1711	Total	180	1531	1711
Sens, %	Spec, %	PPV, %	NPV, %	Sens, %	Spec, %	PPV, %	NPV, %
75.6	99.7	96.4	97.2	79.4	99.6	96.0	97.6
Algorithm C				Algorithm D
	Chart review				Chart review
Algorithm	Positive	Negative	Total	Algorithm	Positive	Negative	Total
Positive	166	4	170	Positive	169	5	174
Negative	14	1520	1534	Negative	11	1518	1529
Total	180	1524	1704	Total	180	1523	1703
Sens, %	Spec, %	PPV, %	NPV, %	Sens, %	Spec, %	PPV, %	NPV, %
92.2	99.7	97.6	99.1	93.9	99.7	97.1	99.2

NPV, negative predictive value; PPV, positive predictive value; RPR, rapid plama reagin; sens, sensitivity; spec, specificity

Figure 1.

Sensitivities of algorithms to identify incident syphilis among HIV-positive men who have sex with men, Boston, MA, 2016–2017.

Discussion

We developed four algorithms for the identification of incident syphilis based on RPR results and treatment data extracted from electronic medical records of HIV-positive MSM accessing primary care at the largest HIV clinic in New England. When compared to clinician-based diagnosis as the gold standard, sensitivities ranged from 74.2% to 83.7% when we included patients with syphilis that were diagnosed and/or treated outside our clinical site. When we excluded such patients, sensitivities ranged from 75.6% to 93.9%. For all of the algorithms, specificities were greater than 99% with PPV greater than 95% and NPV greater than 96%.

Algorithms that ignore outside cases result in a lower proportion of false negatives and a greater proportion of true positives, resulting in increased sensitivity compared to those that include outside cases. The difference in sensitivity estimates between algorithms that include and exclude outside cases is the bias expected from assuming that all syphilis diagnosis and/or treatment is co-located with HIV care. Thus, performance of the algorithms excluding outside cases represent a best-case scenario in situations where no patients are diagnosed and/or treated outside the clinical site of interest.

Algorithms C and D, the two algorithms that incorporated treatment data, had greater sensitivity compared to the two algorithms that used only RPR testing (algorithms A and B). The sensitivities of algorithms C and D were similar; thus, capturing a four-fold increase over several titers may not provide a significant performance improvement over a four-fold increase between two consecutive titers. Thus, algorithm C seems to represent the best candidate algorithm for use in future studies.

The differences in sensitivities of the algorithms that include and exclude outside cases were smaller for algorithms A and B than for algorithms C and D. Unlike algorithms C and D, algorithms A and B captured some of the outside cases. For example, a participant may have had an RPR of 1:32 with treatment at an outside location. At a later visit to FH, the RPR was 1:8 after a non-reactive RPR, but they were not treated. Algorithms A and B would capture this participant as a case, but algorithms C and D would not since treatment was not administered.

The evaluation of our proposed algorithms is limited in its use of data from a single site where the majority of HIV-positive patients receive their HIV and STI care in the same place. As noted above, the algorithms may perform less well when applied to clinical sites where more patients are tested and treated for syphilis at outside locations. HIV-positive MSM are at higher risk for incident and recurrent syphilis than HIV-positive heterosexual people and HIV-negative people.^5,6 Therefore, the performance of these algorithms is likely not generalizable to populations at lower risk for syphilis. HIV-positive people may also be more likely to have false negative and false positive RPR results impacting the performance of RPR-based algorithms.^7–13 Additionally, the algorithms may perform differently in cohorts with less complete laboratory and antibiotic treatment data, or with the use of the reverse screening algorithm.

The effect of expanding the algorithms to capture treponemal testing in settings using the reverse algorithm would likely be complex. Initial screening with a treponemal test identifies more patients with reactive results compared to the traditional algorithm; however, a significant proportion of patients who have reactive treponemal screening have a non-reactive RPR.^9–11,13 Such discordant results could mean very early syphilis, late latent syphilis, previously treated syphilis, or no syphilis. Not only are HIV-positive patients more likely to have this pattern of discordant testing,¹³ but also may be more likely to have non-reactive treponemal testing in the setting of reactive non-treponemal testing.¹² Thus, the incorporation of treponemal testing to algorithms in settings using the reverse algorithm warrants further exploration.

We developed four algorithms for the identification of incident syphilis among HIV-positive patients enrolled in an open, longitudinal clinical cohort. Algorithms that incorporate treatment data perform better than those that used testing data only. While these algorithms require further evaluation in other settings, they provide a method to evaluate trends and predictors of syphilis using laboratory and treatment data captured through electronic medical record systems as part of a clinical cohort or registry.