The Development of an Online Risk Calculator for the Prediction of Future Syphilis among a High-Risk Cohort of Men Who Have Sex with Men and Transgender Women in Lima, Peru

Abstract

Background

Syphilis incidence worldwide has rebounded since 2000, particularly among men who have sex with men (MSM). A predictive model for syphilis infection may inform prevention counseling and use of chemoprophylaxis.

Methods

We assessed data from a longitudinal cohort study of MSM and transgender women meeting high-risk criteria for syphilis followed quarterly for two years. Incidence was defined as a four-fold increase in rapid plasma reagin (RPR) titers, or new RPR reactivity if two prior titers were non-reactive. We used generalized estimating equations to calculate rate ratios (RR) and develop a predictive model in 70% of the dataset, which was validated in the remaining 30%. Furthermore, we developed an online risk calculator for the prediction of future syphilis.

Results

Among 361 participants, 22.0% were transgender women and 34.6% were HIV-infected at baseline. Syphilis incidence was 19.9 cases per 100-person years (95% CI 16.3–24.3). HIV infection (RR 2.22; 95% CI 1.54 – 3.21) and history of syphilis infection (RR 2.23; 95% 1.62 – 3.64) were significantly associated with incident infection. Our final predictive model for syphilis incidence in the next three months included HIV infection, history of syphilis, number of male sex partners and sex role for anal sex in the prior 3 months, and had an area under the curve of 69%. The online syphilis risk calculator based on those results is available at: www.syphrisk.net.

Conclusions

Our predictive model for future syphilis has a moderate predictive accuracy, and may serve as the foundation for future studies.

Summary:

Using data from a longitudinal cohort study among a population at high-risk for syphilis infection in Peru, we developed a predictive model and online risk calculator for future syphilis infection.

Introduction

Syphilis incidence worldwide has rebounded since 2000 (1). Untreated syphilis has significant long-term consequences, including neurosyphilis, cardiovascular syphilis, and gummatous syphilis (2). Additionally, syphilis has been shown to increase the transmission and acquisition of HIV infection (3, 4). Men who have sex with men (MSM) and transgender women are particularly affected by syphilis (5).

The incidence rate of syphilis among MSM and transgender women in Peru has been estimated to be as high as 8.4 per 100 person-years (5), while prior reports have estimated the prevalence of early syphilis infection to be between 7.4–10.5% (6, 7), compared to a 0.5% prevalence of any syphilis among men in the general population (8, 9). Syphilis reinfection among MSM and transgender women in Peru is also common, with an estimated annual incidence rate of 35.3 per 100 person-years (10). It has been estimated that 60.9% of syphilis cases in one year in Peru occur among the 13.3% of men that have sex with men or condomless sex with female sex workers (9). Given the high prevalence and incidence among MSM and transgender women, it has been postulated that those populations constitute a core group of high-risk individuals that may sustain disease transmission in the general population (11).

Factors associated with syphilis among MSM and transgender women include HIV infection, a high numbers of sex partners, condomless receptive anal intercourse, and a history of syphilis infection (3, 10–12). Further synthesis of the current epidemiologic understanding of syphilis into a predictive model may be clinically useful. A prior predictive model for syphilis among women reported good predictive accuracy (13). Developing a predictive model for syphilis among core populations may be particularly useful as predictive modeling approaches for sexually transmitted infections (STIs) can inform risk prevention counseling (14), and identify specific subsets of the population to target with screening interventions, which is of particular importance as increasing the frequency of screening among high-risk individuals may be effective in reducing the incidence of syphilis (15). A predictive model for syphilis may also direct possible chemoprophylaxis strategies, such as the potential use of doxycycline chemoprophylaxis among high-risk individuals (16). We therefore aimed to develop a predictive model of future syphilis among high-risk MSM and transgender women.

Methods

Data

The Picasso Study (17) enrolled 404 MSM and transgender women meeting high-risk criteria for syphilis defined as the presence of at least three of the following, reported at the time of screening: 1) HIV-infection, 2) a history of syphilis in the prior two years, 3) any current ulcerative STI, 4) any STI diagnosed in the prior six months, 5) more than five years of sexual activity, and, during the preceding 3 months, 6) more than five sex partners, or 7) more than five occurrences of condomless anal sex. Participants were recruited from one of two clinic sites in Metropolitan Lima: one public STI clinic for high-risk populations (Barton Health Center in the district of Callao in the far west of Metropolitan Lima); the other a Gay Men’s community center (Epicentro located in the district of Barranco in the south of Lima). Enrollment and follow-up took place between June 2013 and July 2016 (17). Baseline characteristics of our population have been published previously (18). To augment retention, regular follow-up reminders were provided via cell-phone text messages, phone calls, e-mail and Internet chats, as supported by previous studies (19).

Participants answered a self-administered, standardized, computer-based survey at baseline and during follow-up visits at three-month intervals for 24 months. The survey collected data on socio-demographics, healthcare seeking behavior, history of current/past syphilis and HIV infection, sexual behavior, alcohol and substance use. High-risk alcohol use was defined as a score of greater than or equal to 8 on the World Health Organization (WHO) AUDIT questionnaire (20).

Each study visit also included laboratory testing for HIV (among those with no prior HIV diagnosis) and syphilis serology regardless of reported symptoms. Pre-test counseling along with rapid HIV testing were provided to participants, and those subsequently diagnosed with HIV infection were referred to the Peruvian National HIV Treatment Program sites. Syphilis testing was performed on whole blood samples using rapid plasma reagin (RPR) and confirmatory T. pallidum-Particle Agglutination (TPPA) tests. For participants diagnosed with syphilis, on-site treatment was offered with Penicillin G Benzocaine 2.4 million units intramuscularly once in accordance with Peruvian National STI Treatment Guidelines (21).

The study was approved by the institutional review board of the Universidad Peruana Cayetano Heredia. For the present analysis, the University of California, Los Angeles institutional review board exempted this secondary data analysis from review as it entailed the analyses of de-identified data only.

Statistical Analysis

Measures used in the statistical analysis were based on past reports of factors associated with syphilis (3, 7, 10, 12, 22) and included younger age, condomless receptive anal sex, recent sexually transmitted infections, HIV infection, a history of syphilis, and a high number of male sex partners. Incident syphilis was defined as at least a four-fold increase in RPR titers, or any new RPR reactivity among participants with at least two consecutive prior non-reactive RPR titers. Furthermore, individuals had to be TPPA positive. For the analysis, individuals with repeat infection were included when we had documented treatment and response to treatment of the prior infection, defined as a four-fold decrease in RPR titers. Since participants were assessed for incident syphilis at 3-month intervals, the outcome of interest was incident syphilis in 3 months. To assess that outcome, participants were included in the analysis only if they had one or more follow-up visits after baseline enrollment. Descriptive statistics are reported for the variables of interest. We calculated incidence and incidence rate ratios (RRs) in the full cohort dataset using generalized estimating equations (GEE) with a Poisson regression and an exchangeable correlation structure to account for within participant correlation of multiple follow-up visits.

The modeling strategy employed followed a predictive modeling approach (23), striving to yield the most parsimonious model with the best predictive accuracy as well as balancing predictive ability with ease of obtaining the included variables, as opposed to modeling for confounding control. We also emphasized variables that would be readily available to clinicians or individuals to assure model utility. We split the dataset into a training dataset (randomly selected 70% of participants) and validation dataset (the remaining 30% of participants), and used the same GEE with a Poisson regression to develop a multivariable predictive model of syphilis. We subsequently validated the selected model in the validation dataset by calculating the predictive probability per patient-visit and comparing the areas under the curve calculated for each model to determine the best combination of predictors of syphilis in the following 3 months. Importantly, because syphilis is an event that can repeat, the reported sensitivities and specificities were calculated based on person-visit, not the individual.

We developed several candidate predictive models and selected the model with the best area under the curve. We then applied the best model to MSM and transgender women separately to evaluate difference in predictive accuracy between the two groups. Once the final model was internally validated, we used the beta coefficients from that model to develop an online risk calculator for estimating probability of the outcome (Appendix).

To assess model transferability, we applied the model to two other datasets. The first was the complete cohort baseline dataset, which provides additional information on the reproducibility of the model for cross-sectional data. In that dataset, the outcome used was recent syphilis defined as an RPR titer ≥ 1:8. The second dataset was composed of a lower-risk cohort of men who were 18 years of age or older, reported having sex with a man in the past year, identified as either homosexual or transgender, and were enrolled from community settings instead of STI clinics. Applying the model to that dataset provides information on the historic transportability as well as spectrum transportability of our model (23).

In addition, to assess prediction in a different framework we performed classification and regression tree analysis to identify important predictors for syphilis. A binary tree was constructed through recursively partitioning the covariates into disjoint regions and correspondingly splitting the data into nodes (groups).

Finally, we performed a retention analysis comparing risk factors reported at the baseline visit among participants who never returned after enrollment to risk factors reported by participants who missed one or zero follow-up visits. Proportions were compared using chi-square test. Fisher’s exact test was used instead of chi-square test when the expected values in cells were less than five. Wilcoxon’s Rank Sum Test was used for non-parametric data. The classification and regression tree analysis was conducted using Package “rpart” in the R statistical software environment version 3.3.1. All other analyses were performed using STATA software version 14.2 (StataCorp®, College Station, TX, USA).

Results

Among 404 participants, 89 (22.0%) were transgender women. We had follow-up data for 361 (89.4%). At baseline, 125 (34.6%) had HIV infection. The median number of follow-up visits per participant was 6 (interquartile range: 3 – 8 follow-up visits) where the maximum number of study visits per participant was 8. The mean age of the study population was 29 years. There were a total of 111 new cases of syphilis during the study period, resulting in an incidence of 19.9 cases per 100-person years (95% CI 16.3–24.3). Of the 111 new cases, 18 were cases of re-infection. The results of our bivariate analysis of incident syphilis within the complete dataset can be found in Table 1.

Table 1:

Bivariate model for risk factors associated with incident syphilis among a high-risk cohort of MSM and transgender women in Lima, Peru, using the entire dataset.

Risk Factor	No. Cases	Incident Rate (95% CI)	RR Crude	95% CI
Age (years)
< 30	47	23.21 (17.61 – 30.61)	ref	-
30–39	20	15.81 (10.15 – 24.61)	0.68	0.40 – 1.15
40–49	8	10.91 (5.43 – 21.90)	0.47	0.22 – 1.00
≥ 50	18	27.59 (17.64 – 43.18)	1.19	0.70 – 2.01
HIV Infection*
No	44	14.31 (10.67 – 19.21)	ref	-
Yes	49	30.53 (23.30 – 40.00)	2.22	1.54 – 3.21
Taking Anti-Retroviral Medication*
No	52	16.48 (12.56 – 21.61)	ref	-
Yes	36	29.98 (21.95 – 40.95)	1.82	1.20 – 2.75
History of Syphilis Infection*
No	40	13.28 (9.76 – 18.07)	ref	-
Yes	53	32.23 (24.82 – 41.87)	2.23	1.62 – 3.64
Receptive Anal Sex with a Man, past 3 months*
Had No Sex	28	14.09 (9.76 – 20.35)	ref	-
Only Had Protected Sex	53	24.75 (19.08 – 32.10)	1.34	0.75 – 2.36
Had Unprotected Sex	12	22.18 (12.96 – 37.97)	1.54	0.70 – 3.41
Insertive Anal Sex with a Man, past 3 months*
Had No Sex	59	22.19 (17.29 – 28.48)	ref	-
Only Had Protected Sex	29	17.53 (12.22 – 25.15)	0.79	0.51 – 1.23
Had Unprotected Sex	5	13.47 (5.87 – 30.90)	0.61	0.26 – 1.44
Transgender Identity
No	73	19.93 (15.90 – 24.97)	ref	-
Yes	20	19.91 (12.85 – 30.85)	1.00	0.61 – 1.64
Anal Sex Role*
Insertive	9	9.98 (5.26 – 18.93)	ref	-
Receptive	39	23.34 (17.21 – 31.65)	2.34	1.15 – 4.76
Versitile	43	20.77 (15.47 – 27.89)	2.08	1.03 – 4.20
Number of Male Sex Partners, past 3 months*
0	3	9.97 (3.30 – 30.15)	0.49	0.15 – 1.59
1	23	20.22 (13.54 – 30.19)	ref	-
2 – 3	19	16.32 (10.52 – 25.33)	0.81	0.45 – 1.46
4 – 9	20	20.82 (13.70 – 31.64)	1.03	0.58 – 1.84
>10	26	24.40 (16.76 – 35.51)	1.21	0.70 – 2.08
High-Risk Alcohol Use*†
No	57	18.54 (14.38 – 23.91)	ref	-
Yes	36	22.41 (16.18 – 31.03)	1.21	0.80 – 1.83
Drug Use Prior to Sex*
No	90	19.77 (16.13 – 24.23)	ref	-
Yes	1	10.20 (1.47 – 70.96)	0.52	0.73 – 3.63
Earns less than Peruvian minimum monthly wage*
>250 USD	83	20.51 (16.59 – 25.36)	ref	-
≤ 250 USD	5	15.67 (6.63 – 37.04)	0.76	0.32 – 1.85

Legend for Table 1:

^*Indicates variables with missing data

^†High risk alcohol behavior was defined as a score of ≥ 8 on an alcohol AUDIT

Predictive Modeling

Table 2 shows the variables considered for the predictive model, which were based on our bivariate analysis as well as on past reports of factors associated with incident syphilis. The area under the curve and 95% confidence interval for several predictive models (1 to 6) of incident syphilis are also shown in Table 2. The total number of participants included in all of the models was 246 (70% of the analytic sample). Model 6 was selected as the best model (Figure 1), and included the following variables: HIV infection, a history of syphilis, number of male sex partners, and sex role for anal sex. That model was selected as it had the highest area under the curve (AUC=69%; 95% CI 61% – 78%) and was more parsimonious than several of the other models.

Table 2:

Variables considered, resulting area under the curve, and 95% confidence intervals for six models tested for the prediction of incident syphilis among a high-risk cohort of MSM and transgender women in Lima, Peru.

Variables Included	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6 (best model)
HIV Infection	X	X	X	X	X	X
History of Syphilis Infectiona	X	X	X	X	X	X
Use of Antiretroviral Therapy	X
Number of Male Sex Partners in Prior 3 Months	X	X	X*	X*	X	X
Condom Use for Receptive Anal Intercourse in Prior 3 Months		X		X	X
Sex Role for Anal Sex in Prior 3 Months	X	X		X		X
Age (years)a				X
Transgender Identitya					X
AUC (95% CI)	68% (59% – 76%)	68% (59% – 76%)	59% (49% – 69%)	59% (49% – 69%)	69% 60% – 77%)	69% (61% – 78%)

Legend for Table 2: The total number of participants included in each model was 246 (70% of the analytic sample).

^*Number of male sex partners was categorized as: 0, 1, 2–3, 4–9, >10

^aData for this variable was collected at baseline.

Figure 1:

The receiver-operating curve for our final predictive model (model 6) for predicting incident syphilis in the next 3 months among a high-risk cohort of MSM and transgender women in Lima Peru, enrolled between 2013 and 2016.

The optimal sensitivity and specificity of our model is for a predicted probability of syphilis in the next 3 months of 4.1%, which resulted in 71% sensitivity and 63% specificity. At that cutoff, 64% of the members of the cohort were correctly classified as either had or did not have incident syphilis in the following 3 months. There were 10 cases of syphilis with risk scores lower than 4.1%.

When applied to only transgender women, our final model had an area under the curve of 68% (95% CI 51% – 86%). A risk of 4.3% among only transgender women resulted in a 100% sensitivity and a specificity of 44.8% for predicting syphilis in the following 3 months. Additionally, when applied to only MSM, our final model had an area under the curve of 72% (95% CI 63% – 81%). A risk of 4.1% among only MSM resulted in a sensitivity of 70.4% and a specificity of 65.7% for predicting syphilis in the following 3 months.

The resulting tree of our classification and regression tree analysis contained six predictors with an area under the curve of 65% (95% CI 51% – 79%). However, as the classification and regression tree approach did not account for the correlated structure of the data and the tree model was less parsimonious than the Poisson regression model, we did not choose it as our final predictive model.

Construction of the Risk Calculator

Estimated rate ratios were used to calculate the estimated probabilities of syphilis, which we then used to create an online risk calculator. Those at highest risk had a 1-in-10 probability of syphilis in the next three months. The online syphilis risk calculator based on the results of our model is available at: www.syphrisk.net (in English) and www.sifriesgo.net (in Spanish).

Further Validation

We then applied model 6 to the complete baseline dataset for further model validation. The prevalence of recent syphilis at baseline was 20% (95% CI 17% – 25%). The area under the curve for predicting recent syphilis in the complete baseline dataset was 69% (95% CI 63% – 75%). Model 6 was also applied to the lower-risk cohort dataset with a population of 569 men and transgender women reporting sex with a man in the prior year. The prevalence of syphilis in that dataset was 11.0 per 100 person years (95% CI 8.4 – 14.5). When model 6 was applied to that dataset, the resulting area under the curve of 63% (95% CI 52%−74%) for predicting syphilis in the next three months.

Retention Analysis

Only 43 (10.6%) of the 404 participants did not have a follow-up visit after baseline enrollment. We found that 31% of participants who reported exclusively insertive anal intercourse were lost to follow-up compared to 12% who reported exclusively receptive anal sex and 14% who reported sex role versatility (p-value=0.008). We also found that 39% of participants who were recently diagnosed with HIV at baseline were lost to follow-up compared to 15% participants without a recent diagnosis of HIV infection at baseline (p-value=0.02). Furthermore, 23% of participants reporting high-risk alcohol use were lost to follow-up compared to 11% not report high-risk alcohol use (p-value=0.01).

Discussion

We report the development of a predictive model and risk calculator for future syphilis among a high-risk population of MSM and transgender women. Our final model included HIV infection, a history of syphilis, number of male sex partners in the prior 3 months, and sex role for anal sex in the prior 3 months, and had an area under the curve of 69% (95% CI 61% – 78%). The area under the curve is a representation of how well our model distinguishes between those that did not become infected and those that did (24). An area under the curve of 69% represents a 69% probability that our model correctly predicts future syphilis in a randomly chose individual that indeed became infected (25).

Our final model may not have sufficient discriminatory power to be clinically useful at this point. However, our model does provide the foundation for predicting syphilis among populations with a high burden of disease. There have been other predictive modeling studies for syphilis. One study concluded that only HIV infection is a predictor of syphilis reinfection among MSM (11); however that study did not comment on predictive accuracy. Another study evaluated the utility of a pre-determined set of risk factors in identifying maternal syphilis and reported an area under the cure of 78% (13); that study included similar risk factors used in our model (prior syphilis, number of sex partners, and HIV infection), however it also included several factors that were not applicable to our population (history of abortion, history of stillbirth, history of neonatal death). Furthermore, that study did not report any further validation of their model outside of their initial dataset. Other studies have focused on predicting neurosyphilis with varying success (26–28).

The reproducibility of our results in the complete cohort baseline dataset, as well as the historic and spectrum transferability of our model into a dataset with a lower incidence of syphilis is encouraging. Further research to refine and improve the model is warranted, and may increase the clinical utility of the risk calculator. Specifically, we were unable to include clinical symptoms of syphilis at the time of presentation or a recent STI in a sex partner, which may be worth evaluating in future studies. A recent STI in a sex partner has been shown to be a predictor of maternal syphilis (13). Similarly, recent CD4 counts and viral loads may be useful predictors among individuals living with HIV; however, as those values may not be accessible to all individuals, including those variables in a predictive model may limit the model utility in non-clinical settings.

The use of antiretroviral therapy may warrant further investigation as a predictor for syphilis. A recent modeling study hypothesized that the spike in syphilis incidence among individuals living with HIV may be in part due to antiretroviral therapy suppression of the innate immune system and macrophage opsonization, thus predisposing to syphilis (29). That hypothesis is substantiated by previous work demonstrating a reduction in both macrophage phagocytosis and intracellular killing as a result of nucleotide reverse transcriptase inhibitors (30–32). The results of our bivariate analysis showing a higher incidence rate of syphilis among patients taking antiretroviral therapy further support that hypothesis. Furthermore, our model that included the use of antiretroviral therapy (model 1) had an area under the curve that was similar to our final model.

We also evaluated the predictive accuracy of our model among MSM and transgender women separately. In both populations, our model demonstrated similar predictive accuracy and discriminatory capability. The high sensitivity for predicting future syphilis among transgender women may indicate a higher degree of risk factor homogeneity among transgender women than among cisgender MSM. However, future studies may be able to add additional predictive factors to the model among transgender women in order to improve the specificity.

Current recommendations from the Centers for Disease Control and Prevention (CDC) include periodic screening for syphilis every 3–6 months for MSM at high-risk of infection, defined as those with HIV infection or who report that they or their partners have multiple sex partners (33). Our findings support those recommendations, and provide data for possibly further defining populations requiring increased screening. Two recent modeling studies demonstrated that increased frequency of syphilis screening among core populations may reduce syphilis incidence more than by expanding syphilis screening interventions to populations who are under-screened (15, 34). Therefore, a predictive model may be useful for further defining core populations and thereby guiding screening recommendations. Refining screening recommendations may also improve the cost effectiveness of screening. Similarly, targeted screening and a predictive model for syphilis might also inform HIV prevention efforts, given syphilis is a risk factor for incident HIV infection (35, 36).

An additional potential application of a predicative model for syphilis infection involves chemoprophylaxis. The utility of doxycycline chemoprophylaxis to prevent syphilis among individuals at high risk has been suggested (16). Risk calculation may inform future efforts to delineate the populations most likely to benefit from chemoprophylaxis, similar to what has been accomplished with the pre-exposure prophylaxis calculator (www.IsPrEPforMe.org) for HIV infection (37).

The development of an online risk calculator also has several potential benefits including promoting targeted risk behavior counseling by enabling a discussion of patient-specific risk factors with counselors. Furthermore, an online risk calculator for self-assessment of risk may promote healthcare seeking, including STI/HIV screening. Other online calculators for predicting risk of Chlamydia trachomatis infection or STI risk in general exist (38, 39) and have similar potential benefits, but there is not yet a widely accepted calculator for syphilis risk. A recent systematic review found that web-based educational and behavioral interventions have a greater impact on risk behavior reduction than non-web-based interventions (40), possibly due to individual use in non-clinical settings. That finding potentially supports the utilization of our online calculator for education and behavior modification interventions.

We also report a higher incidence of syphilis infection among MSM and transgender women in Lima, Peru compared to previous reports (3, 5). That high incidence is likely due to the high-risk inclusion criteria of the present study. Perhaps the clinical utility of our risk calculator may be improved by pre-screening questions to identify similarly high-risk populations.

Limitations

There were several limitations to our study. Primarily this study was done exclusively among MSM and transgender women, and therefore our findings may not be generalizable to other populations, particularly those at lower risk for syphilis. Additionally, the area under the curve of our final model may not represent sufficient predictive accuracy for clinical implementation; therefore, further work is necessary to refine that model. Furthermore, though we validated the results of our study in one external dataset, future studies are necessary to further assess the external validity and reproducibility of our results. We were also unable to include all variables of interests, including clinical symptoms, a recent STI in a sex partner, use of antiretroviral therapy, and last CD4 count and viral load. Future studies should attempt to evaluate the utility of those predictors. Our sample size was also not large enough to run separate models for non-transgender MSM and transgender women, which constitute populations at different risk for infection (41). Thus, a predictive model focusing on either population may further refine screening recommendations.

Finally, our retention analysis demonstrated significant differences in those lost to follow-up. Specifically, a higher proportion of participants reporting exclusively insertive anal sex, high-risk alcohol use, or recent HIV infection diagnosis at baseline did not return for a follow-up visit. The participants reporting exclusively insertive anal sex were likely in a lower risk category for syphilis; therefore, our results may have slightly over-estimated the incidence of syphilis. On the other hand, participants reporting high-risk alcohol use as well as those with a recent diagnosis of HIV infection were likely in a higher risk group for syphilis, thus potentially underestimating the incidence of syphilis. However, since the number of participants who did not return for follow-up was a small proportion of the overall sample, those differences likely did not significantly impact our results.

Conclusion

We developed a predictive model for future syphilis with moderate predictive accuracy as well as a web-based tool for model implementation. Although our model may not be clinically useful in its current form, it may provide the groundwork for future studies, which may be used to inform future screening and prevention strategies.