Validation of the San Diego Early Test Score for Early Human Immunodeficiency Virus Infection Among Amsterdam Men Who Have Sex With Men

Abstract

The San Diego Early Test score is a simple risk-assessment tool for acute, and early human immunodeficiency virus (HIV) infection. Validation in a prospective cohort of Amsterdam men who have sex with men showed fair prediction of HIV seroconversion (AUC, 0.701). This score can help prioritize and target HIV-prevention strategies.

If one wants to target human immunodeficiency virus (HIV)–prevention strategies, knowing the risk factors for HIV acquisition is a roadmap for making these decisions. Specifically, scores predicting HIV acquisition may help target costly interventions. Several HIV-acquisition risk scores for men who have sex with men (MSM) have been developed [1–6]. However, some include symptoms of acute and early HIV infection (AEHI) [3–5, 7], are rather complex [6], or generalizability may be uncertain due to lack of validation [2, 3]. The San Diego Early Test (SDET) score is a simple, 4-item risk-assessment tool to predict AEHI among MSM and includes only variables concerning risk behavior: condomless receptive anal intercourse (CRAI) with an MSM positive for HIV (3 points), the combination of CRAI plus 5 or more male partners (3 points), 10 or more male partners (2 points), and diagnosis of bacterial sexually transmitted infection (STI) (2 points), all self-reported for the previous 12 months [1]. Validation showed fair discriminative ability in San Diego [1], while the discriminative ability of this and other validated scores was poor in Atlanta [8]. The SDET score has, to date, not been validated outside the United States. In this study, we validated the SDET score in a prospective cohort study for MSM who are negative for HIV in Amsterdam, the Netherlands.

METHODS

Validation Cohort

We validated the SDET score by using data from the Amsterdam Cohort Studies (ACS). This is an open, ongoing prospective cohort study for MSM negative for HIV [9]. Biannual visits included HIV antigen (since January 2001) and antibody testing, and a standardized questionnaire on risk behavior and STI in the previous 6 months. STI testing at study and interim visits has been included since October 2008. Written informed consent is obtained from every participant at enrollment.

Selection of Visits

For this analysis, we included participants who were HIV negative at study entry and had at least 1 follow-up visit. Variables included in the SDET score were consistently included in the questionnaire since January 2003; thus, visits between January 2003 and September 2018 were included. Any visit with an interval of more than 12 months following the preceding visit was excluded.

Statistical Analysis

A seroconversion visit was defined as a visit with a first positive HIV test result after a previous visit with an HIV-negative result. We included STI self-report, as was done in San Diego. Reporting of risk factors was compared between seronegative and seroconversion visits and between the San Diego cohort [10] and ACS using χ ² statistics. The SDET score was calculated for each ACS visit, using the SDET score variables point values [1]. Performance of the SDET score was assessed by areas under the receiver operating characteristic curves (AUCs), sensitivity, and specificity. The optimal cutoff was determined by the Youden index [11]. We split the dataset in 2 quantiles, based on age, and assessed the performance of the SDET score in both datasets. If a value was missing, we assumed this risk factor was not present.

Sensitivity Analyses

We conducted 4 sensitivity analyses: (1) imputation of missing SDET score values using chained equations (we included age, calendar year, number of partners, and seroconversion in the imputation model, assuming a Poisson-distributed SDET score, which resulted in 20 imputed datasets); (2) inclusion of bacterial STI test results at a study visit (ie, biannual scheduled research visits) or at interim visits (ie, unscheduled STI clinic visits) in the previous 6 months (2008–2018), instead of self-reported bacterial STI; (3) adjustment of 2 variables (“the combination of CRAI plus 5 or more male partners” becoming “the combination of CRAI plus 3 or more male partners” and “10 or more male partners” becoming “5 or more male partners”) to take into account the shorter risk reporting period in ACS (6 months) compared with the San Diego cohort (12 months); and (4) exclusion of visits with report of pre-exposure prophylaxis (PrEP) use (n = 278). AUCs were compared using the DeLong method [12] or Hanley MacNeil [13], where appropriate. Statistical analysis was performed using Stata version 15.1 (StataCorp); 2-tailed significance level was P < .05.

RESULTS

Demographic Characteristics

A total of 1071 participants contributed to 14 619 seronegative and 76 seroconversion visits. The median number of visits per participant was 9 (interquartile range [IQR], 4–17). The median age was 37 (IQR, 32–44) years at seronegative visits and 33 (IQR, 29–39) years at seroconversion visits. During their first visit, the majority of participants reported to have completed a college or university degree (74.3%) and to have been born in the Netherlands (83.9%).

San Diego Early Test Score Risk Factors

Compared with seronegative visits, at seroconversion visits men more often reported CRAI with an MSM who was HIV positive (13.2% vs 3.0%, P < .001), the combination of CRAI plus 5 or more male partners (50.0% vs 20.1%, P < .001), 10 or more male partners (51.3% vs 9.7%, P < .001), and diagnosis of bacterial STI (14.5% vs 4.1%, P < .001). The proportion of missing values for each SDET score variable ranged from 1.8% to 4.0%; the SDET score values were missing in 8.4% of visits. Compared with the San Diego cohort, ACS participants were less likely to report CRAI with an MSM who was HIV positive (3.0% vs 5.5%, P < .001), the combination of CRAI plus 5 or more male partners (20.2% vs 31.5%, P < .001), 10 or more male partners (29.8% vs 33.4%, P < .001), and diagnosis of bacterial STI (4.2% vs 12.2%, P < .001).

Performance San Diego Early Test Score

When applying the SDET score to ACS, the overall AUC was 0.701 (95% confidence interval [CI], .639–.762) (Table 1). The optimal cutoff was 3 or greater. At this cutoff, sensitivity was 54.0%, specificity was 77.9%, positive likelihood ratio was 2.4, and negative likelihood ratio was 0.6 for seroconversion. The AUC was not significantly different between visits in the lower age quantile (AUC, 0.717; 95% CI, .644–.790) than in the higher age quantile (AUC, 0.693; 95% CI, .579–.806; P = .723). The optimal cutoff was 2 or greater for the lower age quantile and 3 or greater for the higher age quantile. Compared with the main analysis, sensitivity analyses did not show substantial differences in AUCs: (1) AUC of 0.719 (95% CI, .658–.779); (2) AUC of 0.721 (95% CI, .624–.818; P = .273 compared with the main analysis); (3) AUC of 0.706 (95% CI, .646–.766; P = .731); and (4) AUC of 0.708 (95% CI, .646–.769).

Table 1.

Performance of San Diego Early Test Score Among 1071 Men Who Have Sex With Men With 14 695 Visits in Amsterdam Cohort Studies, 2003–2018

Cutoff	Sensitivity, %	Specificity, %	Positive Likelihood Ratio	Negative Likelihood Ratio	Visits With a Score of at Least the Cutoff, n (%)
≥2	71.1	60.2	1.8	0.5	5873 (40.0)
≥3	54.0	77.9	2.4	0.6	3267 (22.2)
≥4	44.7	85.7	3.1	0.6	2129 (14.5)
≥5	43.4	86.7	3.3	0.7	1977 (13.5)
≥6	15.8	97.0	5.3	0.9	445 (3.0)
≥7	10.5	97.6	4.3	0.9	364 (2.5)
≥8	5.3	98.5	3.6	1.0	217 (1.5)
≥10	2.6	99.7	9.4	1.0	43 (0.3)

DISCUSSION

The SDET score predicted recent HIV acquisition fairly in a cohort of Amsterdam MSM. The SDET score showed similar discriminative ability for recent HIV acquisition in ACS (AUC, 0.701; 95% CI, .639–.762) as for AEHI in a San Diego–based validation cohort (AUC, 0.703; 95% CI, .625–.781) [1], and better than in an Atlanta-based validation cohort (AUC, 0.55; 95% CI, .44–.66) [8]. The latter low performance was partly explained by a high proportion of seroconversions among black MSM in Atlanta, who reported low-risk behavior. The optimal cutoff was a score of 3 or greater, which is lower than the recommended cutoff (≥5) by Hoenigl et al [1]. This could be explained by the significantly lower reporting of SDET risk factors in ACS than in San Diego. MSM were eligible for participation in ACS if they reported at least 1 male partner in the 6 months prior to enrolment; thus, the ACS did not specifically recruit high-risk MSM. MSM seeking HIV testing services may report higher-risk behavior (whether in San Diego or Amsterdam), as risk behavior can be a trigger to seek testing. The optimal cutoff was even lower (≥2) when we applied the SDET score to younger MSM in ACS. Although these younger MSM reported lower-risk behavior than older MSM in ACS, they accounted for a larger number of seroconversions, and the SDET score predicted seroconversion fairly in both younger and older MSM in ACS.

Limitations include a large number of missing values, which we addressed by assuming a risk factor was not present if a value was missing. This conservative approach may have led to an underestimation of performance of the SDET score. Another limitation is the difference in risk-reporting periods between ACS and San Diego. We addressed this by adjusting the number of partners in the SDET score; this did not affect the performance of the SDET score. Last, ethnicity was not included in the ACS questionnaire, but country of birth suggests a predominantly white study population. A strength of this study is the inclusion of STI test results at and between visits, which may be more reliable than self-reported diagnosis of STI. This did not alter the performance of the SDET score, suggesting that self-reported STI is a suitable alternative for STI test results.

Pre-exposure prophylaxis has not been available in the Netherlands outside of study settings, and low PrEP uptake in our study sample did not alter the performance of the SDET score. Other scores have been developed and validated to specifically predict AEHI [3, 4, 7]. These scores include symptoms of AEHI and showed generally better performance (AUCs ranging from 0.78 to 0.88). While these symptom-based scores may be used to allocate resources for AEHI testing (eg, targeted AEHI-awareness campaigns or targeted HIV-RNA testing), risk-based scores can help identify MSM in need of or qualifying for PrEP, or promote uptake of regular HIV testing. The SDET score, which is publicly available online (http://sdet.ucsd.edu), is an easy-to-use risk-assessment tool that can help prioritize and target HIV-prevention strategies. Further validation in other settings, including in black MSM, nonurban settings, and in settings with higher PrEP uptake, is needed to determine the further generalizability of this score.

Notes

Author contributions. M. H. and G. J. d. B. designed the study. M. D. conducted the analysis and drafted the manuscript. M. F. S. v. d. L. and M. H. supervised the analysis. All authors critically read and approved the manuscript.

Acknowledgments. The authors acknowledge the acute coronary syndrome study participants and study nurses who collected the data (currently Samantha de Graaf and Leeann Storey). The authors also thank Dominique Loomans, Ertan Ersan, Maartje Basten, Liza Coyer, and Ward van Bilsen for their extensive help with data management, and Anders Boyd for his assistance in data analysis.

Financial support. The ACS is financially supported by the Centre for Infectious Disease Control of the Netherlands, National Institute for Public Health and the Environment, Bilthoven, the Netherlands. M. H. reports additional support from the National Institute of Health (National Institute of Mental Health grant MH113477 and National Institute of Allergy and Infectious Diseases grants AI106039 and AI036214).

Potential conflicts of interest. G. J. d. B. reports grants from Bristol-Myer Squibbs and Mac Aids fund. M. H. reports grants from Gilead, outside the submitted work. All other authors report no potential conflicts. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.