External validation of a prediction tool to estimate the risk of human immunodeficiency virus infection amongst men who have sex with men

Supplemental Digital Content is available in the text

Abstract

A human immunodeficiency virus (HIV) risk assessment tool was previously developed for predicting HIV infection among men who have sex with men (MSM), but was not externally validated. We evaluated the tool's validity for predicting HIV infection in an independent cohort.

The tool was assessed using data from a retrospective cohort study of HIV-negative adult MSM who were recruited in Beijing, China between January 2009 and December 2016.

High-risk behaviors occurring within 6 months before the survey were evaluated. Area under curve (AUC) of the receiver operating character curve (ROC) was used to quantify discrimination performance; calibration curve and Hosmer–Lemeshow statistic were used for calibration performance valuation; and decision curve analysis (DCA) was used to evaluate clinical usage.

One thousand four hundred forty two participants from the cohort were included in the analysis; 246 (17.1%) sero-converted during follow-up. External validation of the tool showed good calibration, the Hosmer–Lemeshow test showed no statistical difference between observed probability and tool-based predictive probability of HIV infection (X² = 4.55, P = .80). The tool had modest discrimination ability (AUC = 0.63, 95% confidence interval [CI]: 0.61–0.66). The decision curve analysis indicated that implementing treatment measures based on the tool's predicative risk thresholds ranging from 10% to 30% might increase the net benefit of treatment when compared with treating all or no MSM.

The HIV risk assessment tool can predict the actual risk of HIV infection well amongst MSM in China, but it has a moderate ability to discriminate those at high risk of HIV infection.

1. Introduction

Men who have sex with men (MSM) are disproportionately affected by human immunodeficiency virus (HIV). MSM accounted for almost 30%,^[1] of newly diagnosed, HIV positive adults in 2015 in China, or an estimated 32,600 cases.^[2] This group is considered a key population, as they have the highest risk of contracting and transmitting HIV and yet, also tend to lack access to preventative services. Therefore, efforts addressing the specific needs of MSM can help reduce the spread of HIV and related morbidity and mortality. In particular, HIV infection can better be prevented and managed when MSM at high-risk are identified as early as possible.^[3,4]

HIV testing is the first step for identifying new cases of HIV and improving access to prevention and care.^[5,6] Increasing the number of HIV positive individuals who are aware of their status may help reduce risky sexual behaviors.^[7] Additionally, improving diagnosis and linkage to care can further reduce the pool of viral infection, thus decreasing transmission and disease burden.^[7] The Chinese Center for Disease Control and Prevention (China CDC) recommends sexually active MSM be tested at least once every 3 to 6 months.^[8] Despite this guideline, the frequency of HIV testing amongst MSM in China remains low,^[9] leaving many men who have sex with men undiagnosed. A nationwide study carried out in mainland China in 2018 amongst 1100 MSM determined three main barriers to HIV testing existed: MSM did not believe themselves to be at risk for HIV, they had a fear of being diagnosed of HIV positive, and feared their privacy would be violated during testing.^[10] Additionally, other studies in the region have found that the low rate of HIV testing amongst MSM is associated with a lack of awareness of HIV infection.^[11] Due to these barriers, other sources aimed at identifying high-risk individuals for HIV infection and linking them to care are needed.

Because of these reasons, HIV infection risk assessment tools have been developed to identify MSM at different levels of risk for acquiring HIV. Risk assessment tools can be used by health staff to communicate with MSM about their risks for contracting HIV, subsequently improving their awareness of HIV infection risk. Risk assessment tools can also be used to inform HIV interventions by stratifying MSM based on their individual risk. However, there are some limitations of previously published tools, for example, many prior HIV risk assessment tools were developed by a single database,^[12–15] limiting their generalizability to other subgroups of MSM. Additionally, these tools included risk behavior variables, which occurred on average, 1.6 years prior to HIV infection.^[12] This may not explain HIV sero-conversion amongst participants because of the time lag between when the risky behaviors occurred and actual infection. Previously, Li et al,^[16] developed an HIV infection risk assessment tool for MSM in China using a combination of systematic reviews and Delphi expert consultations. However, their risk tool was not further externally validated for its prediction ability. External validation is essential for further clinical utility.^[17] Therefore, we performed external validation of the published HIV risk assessment tool created by Li et al^[16] using an independent, retrospective cohort study.

2. Methods

This study used retrospective data from a community-based cohort study to externally validate an HIV risk assessment tool.

2.1. Original model

The HIV infection risk assessment tool (Supplement Table 1) was developed by Li et al,^[16] by systematic reviews and 2 rounds of Delphi expert consultations with the purpose of identifying risk behaviors that can help predict the likelihood of HIV infection. Variables included in the HIV risk assessment tool were: number of homosexual partners (anal or oral intercourse), HIV-positive homosexual partners, unprotected anal intercourse (UAI) with a man, commercial male sex behaviors, diagnosis of sexually transmitted diseases, sex role during anal sex with a man, recreational drug usage, and group sex with men. All variables were reported for the prior 6 months. In the original risk assessment tool, risk scores for each item were reported as average odds ratios, which were extracted from studies included in systematic reviews during the tool development process, with the total risk score for an individual being the summed scored of each individual score.

2.2. Construction of the validation cohort

In order to externally validate the HIV infection risk assessment tool, an external validation cohort was created from a previous study, which has been reported elsewhere.^[18] Briefly, HIV negative MSM were recruited and attended follow-up visits every 3 months for HIV testing and answered a questionnaire. Data on demographic information, sexual behavior, injection drug use, and sexually transmitted infections were collected and HIV testing was performed during each follow-up visit. Participants in the cohort were at least 18 years of age or older, man at birth, tested negative for HIV at the start of entering the cohort, and finished at least 2 successive follow-ups. Exclusion criteria included those who did not finish the survey, ended follow-up within 6 months after entering the cohort, or completed only one follow-up session. As this was a secondary analysis of an existing cohort, consent was not needed.

Survey answers available from the external validation cohort were matched with variables in the HIV risk assessment tool. Li et al HIV risk assessment tool evaluated risk behaviors 6 months before the survey, while this external validation cohort collected data every 3 months. As such, the survey data from the last 2 follow-up visits of the external validation cohort were extracted. Table 1 provides a description of variables used in Li et al tool and the redefinition used in the external validation cohort study. Additionally, in this cohort study, no participants reported recreational drug use or group sex, two variables included in Li HIV risk assessment tool. Subsequently, we recoded the data such that all participants reported engaging in recreational drug use and none were exposed to group sex. This decision was based on previous findings in China that recreational drug use amongst MSM remains high, while participation in group-sex remains low.^[19,20] Though this imputation may lead to bias, this bias is assumed to be minor.^[21]

Table 1.

Variables in Li^∗ HIV risk assessment tool compared with variables in the external validation cohort.

2.3. Outcome of interest

HIV sero-conversion was defined as a participant testing HIV positive between 3 months after entering the cohort and 7 years of follow-up in the cohort. Standard HIV type 1/2 ELISA was used to screen HIV cases (Abbott recombinant HIV-1/2 third generation, Vironostka HIV Uni-Form II plus O), and western blot kits were used for HIV anti-body confirmation (Gene labs, HIV Blot 2.2, AE2029). HIV infection was considered as a binary outcome (yes or no).

2.4. Statistical analysis

Frequencies with percentages were reported for categorical variables, and the differences between HIV-positive and HIV-negative MSM at the end of follow-up in the cohort were assessed using Pearson Chi-square test. In order to evaluate the performance of the HIV infection risk assessment tool in the cohort; discrimination, calibration, and decision curve analysis (DCA) were performed. Area under the receiver characteristic (ROC) curve (AUC) was used to evaluate the tool's discrimination performance, which measures the ability of a risk assessment tool to correctly distinguish between HIV-positive and HIV-negative participants. A tool has better discrimination if the value of AUC is close to one. Calibration measures the agreement between observed probabilities and predictive probabilities, which is often assessed graphically with a calibration plot. Observed probability represents the proportion of participants in the external validation cohort who sero-converted during follow-up, while the predictive probability means the likelihood of a person being infected with HIV based on the HIV infection risk assessment tool. The calibration plot has mean prediction probabilities on the x-axis and mean observed probabilities on the y-axis. A 45° line indicates perfect calibration, any deviation above or below the 45° line indicates under or overestimation. Hosmer–Lemeshow statistic was also used to test the calibration performance, which tested the fitting precision between the observed and predicted outcome. DCA assesses the net benefit of a tool on a range of threshold risks. The net benefit is defined as follows: 

where n is the sample size of the study and the weight factor is calculated as: threshold risk/(1 – threshold risk). The weight factor indicates how the relative harms of false-positives (classifying a person as eligible for future preventions who does not develop HIV infection) and false negatives (classifying a person as non-eligible for preventions who develops HIV infection) are valued at a given threshold. The threshold risk is used to classify a participant into positive or negative outcomes based on the risk assessment tool. In clinical practice, DCA is used to decide at which range of threshold risks using the prediction tool provides a better net benefit of prevention and treatment measures over not using the tool for selecting individuals for interventions and treatments.^[22]

During the HIV risk assessment tool development by Li et al,^[16] authors did not report the regression intercept, thus we calculated the intercept by refitting the logistic regression model to the validation cohort. The calculated intercept, along with the predetermined log-odds in Li et al tool, were used to calculate the absolute risk of HIV infection for each participant. Finally, a sensitivity analysis was conducted, in which we excluded the 2 missing variables (recreational drug usage and group sex) from the tool, and evaluated the updated tool's prediction performance. A two-sided P value ≤.05 was regarded as statistically significant. Analyses were performed using the statistical package for R (version 3.5.1, Institute for Statistics and Mathematics, Vienna, Austria).

3. Results

3.1. Baseline characteristics between HIV-positive and HIV-negative MSM

As shown in Fig. 1, of the 2966 subjects who were enrolled in the validation cohort between 2009 and 2016, 791 were excluded because they did not complete the baseline survey, 613 were excluded because they were followed-up for <6 months, and 120 were excluded for only finishing one follow-up visit. The remaining 1442 participants were included in the final analysis. Overall, 246 of 1442 (17.1%) participants sero-converted during follow-up visits.

Figure 1.

Study enrollment and follow up.

Descriptive baseline characteristics of the external validation cohort are shown in Table 2. Mean age was 29.6 years old (standard deviation, SD: 8.06). Fewer HIV positive men were born in Beijing (11.38% vs 16.72%; P = .04). There were no statistical differences of age, ethnicity, educational level, occupation, and marital status between HIV-positive and HIV-negative MSM.

Table 2.

Baseline demographic characteristics between HIV-positive and HIV-negative men who have sex with men in the external validation cohort (N = 1442).

3.2. External validation of the HIV infection risk model

A sensitivity analysis was conducted to assess the impact of the 2 missing variables (recreational drug use and group sex) on the tool's prediction performance. Because the result was similar with that in the previous analysis (without the 2 variables, results not shown), we only report results from the analysis in which the tool included the 2 imputed variables.

When the HIV risk assessment tool was applied to the external validation cohort, the tool had an AUC of 0.63 (95% CI: 0.60–0.67), on a scale of 0 to 1, indicating modest discrimination ability to accurately distinguish between those whom are HIV positive or negative. Additionally, the tool calibrated well, particularly for those with low to moderate predicted risk of HIV infection (Fig. 2). As seen in Fig. 2, the tool-based nonparametric line almost overlapped with the ideal line, and the goodness-of-fit Hosmer–Lemeshow statistic showed there was no statistical significance between the observed and expected probabilities of HIV sero-conversion (X² = 4.55, P = .80), further supporting the tool's good calibration.

Figure 2.

The calibration of the HIV risk assessment tool, when applied to the external validation cohort. The plot illustrates the agreement between predicted and observed probability of HIV infection. The ideal line is a 45° line, which shows perfect concordance between the observed and predicted probabilities. The nonparametric line and the ideal line overlap in low predicted probabilities for HIV infection, but the disagreement appears when the predicted probability grows higher, indicating the tool-based (nonparametric) predicted probability overestimates an individual's actual probability of HIV infection. HIV = human immunodeficiency virus.

3.3. Decision curve

The decision curve in Fig. 3 presents the net benefit of implementing prevention or treatment interventions (e.g., HIV testing) on different levels of risk thresholds. The HIV risk tool had a higher net benefit across a range of risk threshold probabilities, ranging from approximately 10% to 30%.This indicates that the risk assessment tool is valuable for implementing prevention or treatment interventions if the benefit of detecting one HIV positive case is 7 to 9 times better than unnecessarily treating on person, calculated as threshold risk (e.g., 10%–30%)/1 – threshold risk (e.g., 10%–30%).

Figure 3.

The decision curve of the predicted probability in the validation cohort. The “All” line assumes all participants are at high risk of HIV infection and therefore prevent or treat everyone; the “None” line assumes all participants are at low risk of HIV infection and therefore treat no one; the “validation data” line means different net benefits of interventions or treatment based on different risk thresholds. HIV = human immunodeficiency virus.

4. Discussion

We evaluated the prediction performance of an HIV infection risk assessment tool for HIV acquisition amongst MSM in China. The HIV risk assessment tool was previously developed by methods of systematic reviews and Delphi-expert consultations for identifying high risk behaviors associated with HIV infection, without a formal external validation.^[16] Thus, we used data from an independent cohort study to assess the diagnostic performance of the tool amongst MSM in Beijing, China. Our results showed that the diagnostic accuracy of the tool was 63% and had good calibration. Additionally, if the risk threshold probability ranges between 10% and 30% (e.g., if an individual who has a predicted risk of HIV infection of >15% will receive further prevention or other treatment measures, and those whose risk is <15% will not receive prevention or treatment), the net benefit of prevention and treatment implementation based on this risk threshold will be higher than providing interventions to all or no MSM.

The diagnostic accuracy of the tool was 63%, showing moderate ability to distinguish between HIV positive and negative MSM. Predictors in Li et al HIV infection risk assessment tool were mainly extracted from 2 systematic review studies, recruiting different subgroups of MSM in different parts of China by a variety of recruitment methods (online recruitment, voluntary counseling, and testing clinics, etc.).^[23,24] Population heterogeneity and geographic differences are important characteristics of Li et al tool. However, in our study, we only included MSM recruited from 1 cohort study in Beijing, China. This external validation cohort is more homogenous in terms of population and geographic characteristics, which may result in similar risk levels of HIV infection. Consequently, the tool cannot discriminate these risks well. Studies have reported that frequent use of geo-social networking applications are important predictors for HIV infection risk.^[25] Incorporating social media usage in the HIV risk assessment tool may improve the tool's discrimination ability. Additionally, this external cohort validation tool received a similar or higher AUC score when compared with previous studies,^[12,15,26] thus indicating that our tool is still acceptable in clinical application.

Calibration of risk assessment tools tests the association between predicted and observed probabilities and represents another important variable in model evaluation. Overall, the HIV risk assessment tool evaluated in this study showed good calibration, especially among individuals with low to moderate risk of HIV infection. It should be noted however, that the Calibration Plot in Fig. 2 revealed an overestimation of predictive HIV infection probability among observed high-risk MSM. This means that if an individual has a high risk of HIV infection, then the tool is likely to predict a higher risk of HIV infection. This may be beneficial for HIV prevention amongst MSM, as it could impact and improve their HIV awareness and knowledge. However, this may increase those individuals’ psychological burden because of the exaggerated risk of infection. Thus, the tool requires careful consideration in clinical-decision making.

Discrimination and calibration performance focus on the predictive accuracy and reliability of a prediction tool from the perspective of statistical measures, but do not indicate whether a model is worth using in clinical practice.^[27] As such, we conducted DCA, a method to evaluate the net benefit of a prediction model across a range of threshold risks to facilitate clinical decision. In our study, if researchers used the risk threshold of 10% or lower to divide participants into high and low risks of HIV infection, the HIV risk assessment tool is of no value on net benefits of HIV prevention and treatment when compared with treatment for all, as the lower the risk threshold, the more false-positive cases the prediction model generates. Additionally, if the cutoff of risk division for further intervention and treatment is 30% or higher, the net benefit may be equal to that of treating no MSM, as the higher the threshold risk, the more false negative MSM who would miss the chance of receiving further interventions. In this study, it is more important to find the true HIV-positive cases than it is to avoid unnecessary interventions for HIV-negative individuals. Thus, the lower threshold risk for higher net benefit is acceptable. DCA analysis has been widely reported in leading medical journals^[27–29] and is also recommended by the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guideline, a set of recommendations for reporting studies which develop, validate, or update prediction models for diagnostic or prognostic reasons.^[30] Though DCA has been widely used in model predictions for different disease areas, such as urology and nephrology,^[31] oncology,^[32] and internal medicine,^[33] its application to prediction models of HIV sero-conversion among MSM is limited.^{[12,14,15,34,35]} Further studies using DCA to evaluate HIV-risk prediction models are warranted.

The HIV infection risk assessment tool has many implications in clinical usage. First, the tool can be used by health staff to educate MSM about their individual risk of HIV infection and related risk factors, thus improving MSM's understanding of risk for HIV infection and the need for prevention in future self-management of HIV. Second, the tool can efficiently allocate public health resources to MSM at different levels of risk of HIV infection. Moreover, the HIV risk assessment tool validated in this article was developed by systematic reviews and expert consultation, which may overcome geographical restriction of study populations, and are more broadly applicable to different MSM populations.

Some limitations to this study exist. First, only MSM from Beijing, China were recruited for external validation, restricting the generalization of these findings to other MSM throughout China. Validation of this tool using other MSM populations is warranted. Second, 2 variables (recreational drug usage and group sex) in the tool were not available in the external validation cohort, which may lead to bias. In our study, we imputed the missing variables using the prevalence of recreational drug use and group sex amongst MSM reported in previous studies in the same area. Moreover, previous studies in China have reported significant associations between group sex, recreational drug usage, and UAI.^[36] In our study, we collected information on UAI, which may partly explain the relationship between the 2 missing variables and HIV infection.

In summary, we reported the external validation of an already developed HIV risk assessment tool using an independent, retrospective cohort study. The tool showed good calibration and moderate discrimination ability for HIV infection, and is still acceptable for further clinical usage.

Acknowledgments

The authors thank staff at the You’an Hospital, Beijing, China for their assistance in data collection, and all the participants who contributed to the cohort study.

Author contributions

Conceptualization: Hao Wu, Zunyou Wu.

Data curation: Xiaojie Huang, Hao Wu.

Formal analysis: Qianqian Luo, Guodong Mi, Zunyou Wu.

Funding acquisition: Zunyou Wu.

Investigation: Qianqian Luo, Xiaojie Huang, Lingling Li.

Methodology: Qianqian Luo, Yingying Ding, Guodong Mi, Yan Zhao, Keming Rou, Na He, Zunyou Wu.

Project administration: Qianqian Luo, Xiaojie Huang, Lingling Li.

Writing – original draft: Qianqian Luo, Sarah Robbins Scott.

Writing – review & editing: Qianqian Luo, Xiaojie Huang, Lingling Li, Yingying Ding, Guodong Mi, Sarah Robbins Scott, Yan Zhao, Keming Rou, Na He, Hao Wu, Zunyou Wu.

Zunyou Wu orcid: 0000-0002-0839-4548.

Qianqian Luo orcid: 0000-0003-1424-4797.

Sarah Robbins Scott orcid: 0000-0002-0455-644x.

Xiaojie Huang orcid: 0000-0002-5045-9209.

Hao Wu orcid: 0000-0001-7006-2027.