Ending the HIV Epidemic: Development and Evaluation of CyberGIS-HIV, a Web-Based Prediction Application

Abstract

Officials from US state and county health departments perceive the need to adopt new methods for making epidemiological decisions about HIV. Hence, a web-based prediction modeling application, CyberGIS-HIV, was developed and systematically compared with currently used approaches (e.g., community consultation) in a sample of 42 state and federal public health officials as well as senior and junior modelers. Overall, CyberGIS-HIV had more favorable public health and modeling ratings than other approaches currently used by the participants.

We developed a web-based Geographic Information System (GIS) application—CyberGIS-HIV—based on Sass and colleagues’¹ two-part model using nine data sources,² including the National Center for HIV, Viral Hepatitis, STD, and Tuberculosis Prevention of the Centers for Disease Control and Prevention (CDC). The CyberGIS-HIV application, backed by the CyberGISX platform,^3,4 adopts an open science framework that allows users to download source codes from GitHub.

INTERVENTION AND IMPLEMENTATION

The main features of CyberGIS-HIV include analyzing and visualizing spatiotemporal HIV rates and displaying results based on user-defined scenarios. Users can visually compare the spatial differences by displaying two maps of HIV rates from different years (e.g., 2020 and 2026) for a particular county. For example, two maps can be displayed side by side: the 2020 HIV rates on the left and the 2026 HIV rates on the right (Figure 1). The application also automatically shows line graphs to illustrate temporal changes in HIV rates and each key predictor from 2020 to 2026. In addition, CyberGIS-HIV offers lasso or zoom features that enable users to select one or more counties to visualize spatiotemporal and temporal patterns of HIV rates in those circumscribed areas. Four training videos were created²: logging in to CyberGISX, running CyberGIS-HIV, plotting spatial patterns, and exploring temporal patterns.

FIGURE 1— HIV Rates Across US Counties: (a) 2020 Actual Data and (b) 2026 Predicted Data.

Source. CyberGIS-HIV Web User Interface.

Note. Other plots are provided in Figure B, available as a supplement to the online version of this article at http://www.ajph.org.

An HIV prediction application should meet several goals. We defined our goals according to the guidelines of the World Health Organization,⁵ national health agencies,⁶ and the Johns Hopkins Bloomberg School of Public Health.⁷ From a public health perspective, a new HIV prevention method should offer population benefit by providing information that improves prevention planning. It should also support the identification of vulnerable regions and be useful at a large scale (e.g., by all relevant health department personnel). Moreover, models should offer automation and efficiency and be statistically sound, with precise statistical estimates. Thus, we conducted a study in which users evaluated CyberGIS-HIV versus the method they were currently using to forecast HIV patterns and make service provision decisions.

PLACE, TIME, AND PERSONS

The evaluation leveraged a within-participants design to obtain 360-degree feedback on CyberGIS-HIV.^8,9 We preregistered the study at https://osf.io/nr4hw. Participants used their system-generated credentials to log into CyberGIS-HIV and self-paced through the training. Next, they completed 12 evaluation items (six for their current approach and six for CyberGIS-HIV) using a 10-point scale and provided open-ended comments about the new method. Each participant received $200 for taking part in the study.

Public health personnel, senior modelers, and junior modelers (i.e., junior researchers with modeling expertise) were recruited at a ratio of 5:1:4. We used a snowball sampling method to recruit health departments and other agencies. We contacted researchers through their online profiles and via coordinators of public health programs (details are provided in the appendix, available as a supplement to the online version of this article at http://www.ajph.org). Forty-two participants completed the study (22 public health personnel, five senior modelers, and 16 junior modelers), constituting an approximately 70% completion rate and a 38% response rate.

We used a weighted-decision matrix with six items rated on a scale from 1 (does not describe the approach) to 10 (truly describes the approach). Three items measured public health goals: (1) population benefit, (2) constituency perspectives, and (3) administrative feasibility. Another three measured modeling goals: (1) automation and efficiency, (2) statistical soundness, and (3) satisfactory level of uncertainty (see the appendix). For each item, respondents were asked about the approach they were currently using and the CyberGIS-HIV application. The raw scores for each rating were summed after multiplying each item by its weight (public health at 0.3, 0.5, and 0.2 and modeling at 0.33, 0.33, and 0.33; see the appendix). These indexes were then compared across the CyberGIS-HIV application and the approach respondents were using at the time.

PURPOSE

The prevalence and incidence of HIV infections in the United States remain troubling and inequitable despite the availability of successful biomedical interventions.¹⁰ Difficulties achieving critical health milestones (e.g., the National HIV/AIDS Strategy¹¹) can be attributed in part to delayed surveillance data reporting within regions, lags in preexposure prophylaxis use or HIV status awareness, and limited access to Federally Qualified Health Centers. Some of these problems might have been avoided if public health officials had been able to forecast them. Integrating a predictive model of HIV epidemiology into a cyber-based GIS framework^12,13 enables public health officials and researchers to make more accurate predictions about future HIV trends. It also allows them to assess the impact of changes in intervention strategies. Thus, the public health system needs methods to make service decisions by predicting HIV epidemiology within and across areas,^14,15 considering social determinants of health, and contemplating different prediction scenarios.

To date, however, existing methods involve spreadsheets and, occasionally, heat maps of regional epidemiology in addition to informal knowledge of the area in question and insights from community partners. As noted, we collected field data on the perceived need for statistical prediction tools, developed a web-based GIS application, and evaluated the application among public health personnel and modelers.

EVALUATION AND ADVERSE EFFECTS

Table 1 reports descriptive statistics by evaluation item and approach. Examples of current approaches included relying on data from America’s HIV Epidemic Analysis Dashboard, AIDSVu, the Enhanced HIV/AIDS Reporting System, and CDC’s HIV reports. Thirty-seven percent of the approaches currently used by respondents involved displaying surveillance data without any predictive modeling. These applications provided data mainly at the state level and included no predictive results.

TABLE 1—

Evaluation Item Responses From 42 State and Federal Public Health Study Respondents: CyberGIS-HIV Web User Interface and Other Approaches, 2020

	CyberGIS-HIV Item Score, Mean (SD)	Current Approach Item Score, Mean (SD)	Difference Between Means (95% CI)
Public health items
Public health personnel	7.91 (1.63)	4.03 (2.98)	3.88 (2.62, 5.15)
Senior modelers	8.23 (0.96)	5.05 (3.11)	3.18 (−0.62, 6.97)
Junior modelers	8.53 (0.75)	5.92 (2.93)	2.61 (0.89, 4.34)
Overall	8.18 (1.31)	4.85 (3.03)	3.33 (2.41, 4.25)
Modeling items
Public health personnel	7.92 (1.68)	3.06 (2.66)	4.86 (3.58, 6.15)
Senior modelers	8.00 (0.47)	4.08 (2.54)	3.92 (−0.29, 8.12)
Junior modelers	8.54 (0.89)	5.44 (2.91)	3.10 (1.65, 4.55)
Overall	8.17 (1.36)	4.06 (2.91)	4.11 (3.21, 5.00)

Note. CI = confidence interval. Public health items measured population benefit, constituency perspectives, and administrative feasibility, whereas modeling items measured automation and efficiency, statistical soundness, and satisfactory level of uncertainty.

On our 1 to 10 weighted scale, CyberGIS-HIV mean scores ranged from 7.91 to 8.54, whereas scores on currently used approaches ranged from 3.06 to 5.92. Thus, across dimensions, CyberGIS-HIV was rated more favorably than the approach respondents were using at the time, and the findings were similar for different user populations (Table 1). In addition, considering each rating individually, CyberGIS-HIV received more positive ratings than the approach currently used for all six evaluation items (Figure A, available as a supplement to the online version of this article at http://www.ajph.org).

Many respondents also provided comments about the application. These comments were classified as pertaining to functionality, visualization, or statistics. Table A (available as a supplement to the online version of this article at http://www.ajph.org) indicates whether the comments had been addressed in a new system. Most comments suggested making minor layout changes and adding details to CyberGIS-HIV, and these suggestions have been implemented. Moreover, CyberGIS-HIV was developed on the Python Jupyter Notebook user interface. The interface requires users to enter codes or commands and obtain results. A few comments requested a more straight-forward user interface without any coding and with side-by-side scenario comparisons, which we are developing for CyberGIS-HIV 2.0. There were two comments about user-friendliness, which can be fully addressed only with a major update of the CyberGISX system in the future.

This study has several limitations. First, the cross-sectional nature of the evaluation, even with a within-participants design, may have influenced the findings. In addition, the visualization of national county maps does not consider population sizes or densities, which could affect data interpretation if users do not consider the other provided charts and figures. Furthermore, the proposed tool cannot provide predictions over shorter time periods (e.g., monthly or quarterly intervals) because of limitations in the input data. The use of the Python Jupyter Notebook interface, rather than a “what you see is what you get” interface, might also affect usability. Finally, participants’ ratings were subjective and might not correspond to real-world service design and implementation. The relationship between the evaluation ratings and real-world implementation has not yet been explored and is an empirical question for future research.

SUSTAINABILITY

CyberGIS-HIV allows for the analysis and visualization of spatiotemporal HIV rates based on user-defined scenarios in real time. The administrative feasibility item raw scores were higher for CyberGIS-HIV (mean = 7.94, SD = 1.77) than for the current approach respondents were using (mean = 4.64, SD = 3.25), t(41) = 6.10, 95% confidence interval [CI] = 2.21, 4.39, suggesting that CyberGIS-HIV is easier to use and scale up than most approaches. However, 30% of our participants did not complete the study and stopped at the video tutorial stage. Future studies can investigate these difficulties and ways to overcome them.

PUBLIC HEALTH SIGNIFICANCE

Here we have reported fieldwork about public health prediction tools, specifically comparing CyberGIS-HIV with the approach our study respondents were currently using. CyberGIS-HIV outperformed currently used approaches on public health and modeling criteria and among public health personnel, senior modelers, and junior modelers. These findings underscore the potential for CyberGIS-HIV to significantly improve HIV epidemiology decision-making after refinement of the system’s interface.

To maximize its impact, CyberGIS-HIV should be disseminated at public health conferences, journals, and Centers for AIDS Research to ensure that it remains accessible to researchers and public health officials at no cost. It is crucial to regularly update the associated data and codes to maintain their relevance and accuracy. Public health agencies and nongovernmental organizations should take advantage of these data in their decision-making processes and publish their results. Finally, modeling researchers are encouraged to enhance prediction algorithms and integrate them with other implementation data to improve forecasting capabilities within a dynamic social context.