THE AMSTERDAM SYMPTOM AND RISK-BASED SCORE PREDICTS FOR ACUTE HIV INFECTION IN MEN WHO HAVE SEX WITH MEN IN SAN DIEGO

Abstract

Objective:

Dijkstra and colleagues recently described a risk- and symptom-based score moderately predictive for HIV seroconversion in the preceding 6 to 12 months in men who have sex with men (MSM) in Amsterdam. Our objective was to determine whether this “Amsterdam Score” could also predict for acute HIV infection (AHI) in MSM.

Design and Setting:

This study is a case-control analysis of a prospectively enrolled cohort of MSM who voluntarily presented for HIV testing in San Diego. The study sample was composed of MSM who screened HIV antibody-negative and then either tested positive with AHI (HIV nucleic acid test [NAT]-positive), or tested HIV NAT-negative.

Methods:

The Amsterdam Score was calculated for each participant in the study sample. Score performance was assessed using receiver operating characteristic (ROC) curves and their area-under-the-curve (AUC). An optimal cut-off was determined using Youden’s index.

Results:

757 MSM (110 AHI, 647 HIV NAT-negative) were included in the analysis. AHI and HIV-negative cases were similar in age (median 32 years [interquartile range 26–42] vs 33 [27–45], respectively, P=.082). The Amsterdam Score yielded a ROC curve with an AUC of 0.88 (95%CI 0.84 to 0.91). An optimal cut off of ≥1.6 was 78.2% sensitive and 81.0% specific.

Conclusions:

The risk- and symptom-based Amsterdam Score was highly predictive (AUC of 0.88) of AHI in MSM in San Diego. The Amsterdam Score could be used to target NAT utilization in resource-poor settings among MSM who test HIV antibody-negative, though the potential cost savings must be balanced with the risk of missing AHI diagnoses.

Introduction

Diagnosis and treatment of acute HIV infection (AHI) has important implications for patient care and public health. HIV nucleic acid testing (NAT) is the most reliable method of detecting AHI ¹, although its routine use is costly. There are identifiable risk factors associated with AHI ², and risk-behavior based scores such as the San Diego Early Test (SDET) score ³ can be used to target resources among men who have sex with men (MSM). Risk-behavior based scores, however, may not generalize well to populations that have different demographics than the derivation population⁴. The addition of symptoms to these scores could enhance their discriminative ability and/or generalizability, as symptoms reflect the acute retroviral syndrome (ARS).

Dijkstra and colleagues recently described a risk- and symptom-based score (“Amsterdam Score”) that was moderately predictive for seroconversion in the 6 to 12-month period preceding follow-up in men who have sex with men (MSM) ⁵. Their validation cohort was comprised of MSM in select U.S. cities (not including San Diego). It was proposed that the Amsterdam Score may be cost saving by reducing HIV NAT utilization while increasing diagnostic yield. In the present study, we validated the Amsterdam Score in a cohort composed of MSM presenting for voluntary community-based HIV testing in San Diego who were either HIV-uninfected or diagnosed with AHI.

Methods

Study Population / Data Collection

This was a case-control analysis of a cohort study and comprised 1) individuals who tested positive for AHI (defined as HIV antibody [Ab]-negative, HIV NAT-positive) from 2007 to 2017 through the Early Test program and 2) testing encounters that resulted in a negative HIV NAT from January to July 2017, also through the Early Test program. The Early Test is a free and voluntary community-based screening program in San Diego, California in which participants are prospectively enrolled to receive universal point-of-care rapid HIV antibody (Ab) testing, followed by reflex HIV NAT in those who test Ab-negative; blood samples for NAT are obtained at the time of Ab testing. Participants were interviewed regarding risk behaviors for the preceding 3-month period at the time of testing, and symptoms were assessed via questionnaire for the 14 days prior to testing as described in previous works ^3,6–9. HIV-negative participants and AHI participants were included from differing time periods as symptoms were not systematically and universally assessed before 2017.

Amsterdam Score and Adjustments

The Amsterdam Score was calculated for each participant using point values described in Dijkstra et al ⁵ (Table 1). Symptoms and risk behaviors were both assessed for the 6 months prior to testing in the derivation and validation cohorts used in the original work. To take into account the 3 month risk reporting period in the Early Test, we created an “adjusted Amsterdam Score” by adjusting one original variable (“>5 sexual partners in the previous 6 months”) to “>3 sexual partners in the previous 3 months”, retaining the same point value ⁵. Participant-reported oral thrush was not assessed in the San Diego cohort.

Table 1: Amsterdam score variables and their prevalence in the San Diego cohort.

CRAI, condomless receptive anal intercourse; HIV, human immunodeficiency virus; LAD, lymphadenopathy.

		HIV-negative (n=647a)		Acute HIV infection (n=110)
Variable	Score valueb	Number reporting Y (%)	Number missing (%)	Number reporting Y (%)	Number missing (%)
Symptoms (Y/N)c
Fever	1.6	28 (4.3)	3 (0.5)	66 (60.0)	1 (0.9)
LAD	1.5	29 (4.5)	4 (0.6)	31 (28.2)	1 (0.9)
Weight lossd	0.9	10 (1.5)	3 (0.5)	25 (22.7)	2 (1.8)
Oral thrushe	1.7	-	-	-	-
Risk factors in preceding period (Y/N)f
Gonorrhea	1.6	19 (2.9)	4 (0.6)	5 (4.5)	21 (19.1)
>3 sex partnersg	0.9	299 (46.2)	0 (0)	61 (55.0)	1 (0.9)
>5 sex partners	0.9	141 (21.8)	0 (0)	43 (38.7)	1 (0.9)
CRAI	1.1	300 (46.4)	4 (0.6)	92 (83.6)	11 (10.0)

^aThese 647 testing encounters represented 591 individuals.

^bPer Dijkstra et al. BMC Infectious Diseases (2017) 17:425 DOI 10.1186/s12879–017-2508–4.

^cSymptoms were assessed for the 14-days prior to testing in the San Diego cohort, compared to the previous 6 months in the original work by Dijkstra et al.

^dWeight loss was defined as >=2.5kg in the past 14 days

^eOral thrush was not assessed in the San Diego cohort.

^fRisk behaviors were assessed for the previous 3 months in the San Diego cohort, compared to the previous 6 months in the original work by Dijkstra et al.

^gThis variable was not included in the original work.

Statistical Analysis

Statistical analysis was performed using R version 3.1.1 and package “pROC” ^10,11. The two-tailed significance level was P<0.05.

Values that were “missing” (collected in the San Diego cohort but were not reported by certain participants) were handled using three methods: 1) retention of unreported variables with their values coded as “0” (meaning “negative” or “absent”) in order to produce a more conservative analysis, 2) listwise deletion of cases with any missing variables, and 3) multiple imputation with chained equations using five iterations and five imputations.

Risk behavior prevalence was compared between the original validation cohort (the Multicenter Aids Cohort Study [MACS]⁵) and the San Diego cohort using Pearson Chi-square statistics. MACS data were supplied by Dijkstra and colleagues.

Performance of the Amsterdam Score was assessed using receiver operating characteristic (ROC) curves and areas-under-the-curve (AUC) with 95% confidence intervals (95%CI). Four ROC curves were generated, between the Amsterdam Score, the adjusted Amsterdam Score, and the two strategies for non-reported variables. Optimal cut off scores were determined using the Youden index. As a sensitivity analysis, AUCs were compared using the DeLong method¹². The hypothesis of this study was that the Amsterdam Score would be at least moderately predictive for AHI in the San Diego cohort (receiver operating characteristic [ROC] area under the curve [AUC] >=0.70). We found that ROC AUCs did not differ significantly by adjusted/unadjusted Amsterdam Scores, nor by missing data methods. Therefore, for clarity, we only discuss in detail results using the unadjusted Amsterdam Score and the sample with retained missing variables. Results of sensitivity analyses are shown in Supplemental Figure 1 and Supplemental Table 1. University of California, San Diego Human Research Protections Program approved the study protocol, consent, and all study-related procedures.

Results

Demographics

757 MSM cases (110 AHI, 647 HIV NAT-negative) were included in the analysis. The 647 HIV-negative cases represented 591 unique individuals. AHI and HIV-negative cases did not differ significantly in age (median 32 years [interquartile range {IQR} 26–42] vs 33 [IQR 27–45], respectively, P=.082), Hispanic ethnicity (33.6% vs 31.8%, P=.71) and White race (69.7% vs 63.7%, P=.22).

Symptom / Risk Variables

Proportions of participants among AHI and HIV-negative cases who met the definition of each Amsterdam Score variable, as well as the proportion of missing variables, are shown in Table 1. Compared to the MACS cohort, the San Diego cohort had significantly greater prevalence of gonorrhea (3.2% vs 0.4%, P<.001) and condomless receptive anal intercourse (51.8% vs 23.9%, P<.001), while the proportion who reported having >5 sexual partners was similar (24.3% vs 26.6%, P=.17).

Amsterdam Score Performance

The unadjusted Amsterdam Score yielded a ROC AUC of 0.878 (95% confidence interval [CI] 0.844 to 0.913) when applied to the sample with missing variables retained. Sensitivity analyses using the adjusted/unadjusted Amsterdam Scores and two missing data handling methods produced ROC curves with a range of AUCs from 0.877 to 0.896; these ROC curves are shown in Supplemental Figure 1.

Table 2 shows the sensitivity, specificity, positive likelihood ratio and negative likelihood ratio at each unadjusted Amsterdam Score cut-off when applied to the sample in which missing variables were retained. The optimal cut off was ≥1.6, yielding a sensitivity of 78.2%, specificity of 81.0%, positive likelihood ratio (+LR) of 4.11, and negative likelihood ratio (-LR) of 0.24 for AHI. At a cut off of ≥1.6 (e.g. fever or gonorrhea or any combination of two variables), 23/110 AHI cases would have been missed, while 123/647 participants without HIV infection would have received NAT. At a cut off of ≥0.9 (e.g. any one variable), 108/110 AHI cases would have been submitted for NAT testing, while 392/647 participants without HIV would have received NAT. Sensitivity analyses of optimal cut offs and diagnostic parameters at these cut offs for are shown in Supplemental Table 1.

Table 2: Performance of the unadjusted Amsterdam Score.

Parameters shown are for the Amsterdam Score applied to the entire sample (N=757) with missing variables retained. The optimal cut-off using Youden’s index is underlined. +LR, positive likelihood ratio; -LR, negative likelihood ratio.

Cut off	Specificity(%)	Sensitivity(%)	+LR	-LR
≥0	0	100	1	N/A
≥0.9	39.4	98.2	1.62	0.05
≥1.1	48.7	95.5	1.86	0.09
≥1.5	79.9	79.1	3.94	0.26
≥1.6	81.0	78.2	4.11	0.27
≥2	82.8	74.5	4.35	0.31
≥2.4	92.6	65.5	8.82	0.37
≥2.5	93.2	64.5	9.49	0.38
≥2.6	93.8	61.8	10.00	0.41
≥2.7	94.7	60	11.42	0.42
≥2.9	96.6	51.8	15.24	0.50
≥3.1	96.9	51.8	16.76	0.50
≥3.5	97.4	49.1	18.68	0.52
≥3.6	97.7	46.4	20.00	0.55
≥4	98.8	21.8	17.65	0.79
≥4.1	98.8	20.9	16.91	0.80
≥4.2	98.9	20.9	19.33	0.80
≥4.4	99.2	13.6	17.65	0.87
≥4.5	99.4	13.6	22.06	0.87
≥5.1	99.4	10.9	17.65	0.90
≥5.6	99.8	3.6	23.53	0.97
≥5.8	100	3.6	N/A	0.97
≥6	100	2.7	N/A	0.97

Discussion

The risk- and symptom-based Amsterdam Score was predictive for AHI when applied to a San Diego-based cohort of MSM undergoing voluntary, community-based HIV screening who tested Ab-negative. Even when utilizing a shorter risk-behavior reporting period in this cohort as well as a conservative approach to missing data, the AUC of 0.878 found in this study exceeded the AUC described by Dijkstra et al (0.78, 95%CI 0.74 to 0.82) in their U.S. based validation cohort (the Multicenter AIDS Cohort Study [MACS]) ⁵. The better performance in the San Diego MSM cohort is likely explained by our “AHI-positive” case and “HIV-negative” control definitions based on routine HIV-NAT testing, compared to definitions used in the derivation and validation cohorts in Dijkstra et al. In that work, positives were defined by a new positive HIV-1 antibody test followed by confirmatory Western blot, while HIV-negative cases were defined by a negative HIV-1 antibody test ⁵. The Amsterdam Score was also expected to perform better in the San Diego cohort as symptoms were assessed for the previous 14 days (vs 6 months in MACS).

The optimal cut-off for the Amsterdam Score in our cohort was only slightly higher than that in the original validation cohort (≥1.6 points) ⁵, despite our omission of the “oral thrush” variable. This may reflect either the greater prevalence of risk behaviors in the San Diego cohort, or that oral thrush was reported only in 2.3% of the original Amsterdam derivation sample. At our respective optimal cut-offs, we found a much higher sensitivity (78.2% vs 56.2%, P<.001) and relatively small decrease in specificity (81.0% vs 88.8%, P<.001) compared to when the Amsterdam Score was applied to the MACS cohort. If the purpose of the Amsterdam Score is to reduce NAT utilization among those who test Ab-negative, the trade-off of potentially missing AHI diagnoses must be balanced with the cost-savings of avoiding unnecessary NAT tests in those who are ultimately HIV negative. In our sample, a more clinically appropriate cut off may be ≥0.9, which had over 98% sensitivity but would still help to spare over 35% of NAT tests among those who first test Ab-negative. The risk of a false positive HIV NAT after a negative HIV-Ab test is minimal, and would be of even lesser consequence when followed by a confirmatory western blot. Similarly, the risk of a false negative HIV NAT is small to minimal¹³.

We showed that the performance of the Amsterdam Score was maintained and stable even when using different reporting periods for risk-behaviors (3 months in this cohort vs 6 months in Dijkstra et al⁵) and for symptoms (14 days in this cohort vs 6 months in Dijkstra et al). Similar AUCs in adjusted and non-adjusted analyses may also indicate that the original, unadjusted Amsterdam Score could be used in different settings that assess risk behaviors over 3 months, without significant loss of predictive power. Using a 3 month risk assessment period may be advantageous as i) participant recollection may be more accurate, and ii) it can exclude those who engaged in risk behaviors in the previous 4 to 6 months but not in the 3 months prior to testing. Similarly, a symptoms reporting period of 14 days compared to 6 months like improved specificity for AHI and reduced the noise that would be associated with other common causes of fever, such as upper respiratory infection. All in all, we would recommend using the adjusted Amsterdam Score for its shorter reporting periods, and because it is simpler with the exclusion of oral thrush. Validation of the adjusted Amsterdam Score in other cohorts and other periods of assessment would lend further support to its use in different settings. While scores that incorporate symptoms may be more generalizable in theory, symptoms still have the potential to under-predict for AHI, especially in regions with different HIV-1 subtype epidemics¹⁴.

There are important limitations to note, including missing data and differing assessment periods between cohorts, which we aimed to address with sensitivity analyses. An additional limitation of this analysis was that the HIV-negative cases and AHI cases were recruited across different time periods. This was because symptoms among HIV-negative cases were only collected starting in 2017. This has the potential to bias our results if patterns in the reporting of behaviors or symptoms differed across time. A subanalysis of MSM recruited during the same period was not possible because only two participants were diagnosed with AHI in 2017. Finally, our validation sample did not include participants who tested Ab-positive. Therefore, our findings were limited only to validating the use of the Amsterdam Score in MSM who initially test Ab-negative.

Conclusions

The Amsterdam Score was predictive of AHI in MSM in San Diego despite a shorter risk-behavior assessment period and a conservative analysis of missing variables. The improved performance is likely attributable to our definition of positive cases as Ab-negative, NAT-positive AHI, compared to seroconversion in the preceding 6–12 month period in the original work. Combined risk- and symptom-based scores may demonstrate improved generalizability across different populations compared to existing risk-based scores³, and when applied to MSM who test Ab-negative, may increase the yield of AHI detection while reducing testing costs in settings that do not routinely test for AHI.