Application of Machine Learning Techniques in Classification of HIV medical care status for People Living with HIV (PLWH) in South Carolina.

Abstract

Objectives

Ending the HIV Epidemic requires innovative use of data for intelligent decision-making from surveillance through treatment. This study sought to (1) examine the usefulness of using linked integrated PLWH health data to predict PLWH’s future HIV care status and (2) model and compare the performance of machine learning methods for predicting future HIV care status for SC PLWH.

Design

We employed supervised machine learning for its ability to predict PLWH’s future care status by synthesizing and learning from PLWH’s existing health data. This method is appropriate for the nature of integrated PLWH data due to its high volume and dimensionality.

Methods

A data set of 8,888 distinct PLWH’ health records were retrieved from an integrated PLWH data repository. We experimented and scored seven representative machine learning models including Bayesian Network, Automated Neural Network, Support Vector Machine, Logistic Regression, LASSO, Decision Trees and Random Forest to best predict PLWH’s care status. We further identified principal factors that can predict the retention-in-care based on the champion model.

Results

Bayesian Network (F=0.87, AUC=0.94, precision=0.87, recall=0.86) was the best predictive model, followed by Random Forest (F=0.78, AUC=0.81, precision=0.72, recall=0.85), Decision Tree (F=0.76, AUC=0.75, precision=0.70, recall=0.82) and Neural Network (cluster) (F=0.75, AUC=0.71, precision=0.69, recall=0.81).

Conclusions:

These algorithmic applications of Bayesian Networks and other machine learning algorithms hold promise for predicting future PLWH HIV care status at the individual level. Prediction of future care patterns for SC PLWH can help optimize health service resources for effective interventions. Predictions can also help improve retention across the HIV continuum.

INTRODUCTION

Efforts towards ending the Human Immunodeficiency Virus (HIV) Epidemic can benefit from the innovative use of data for intelligent decision-making from surveillance through treatment. The goal of HIV treatment for people living with HIV (PLWH) is to achieve viral suppression. PLWH who achieve and maintain viral suppression have effectively little to no risk of transmitting HIV to others and live an improved quality of life. [1–2]To achieve this goal, the current approach used for HIV prevention in the United States has adjusted to engaging PLWH across the HIV care continuum. [3] The HIV care continuum depicts discrete steps from initial HIV diagnosis, linkage to care via a HIV healthcare provider within one month after diagnosis, receipt of HIV medical care (antiretroviral therapy), engagement in care (active PLWH participation), retention in care, and viral suppression.[4]

However, the HIV care continuum in the United States continues to face challenges. The Centers for Disease Control and Prevention (CDC) estimated that about 64% of all PLWH in the United States received HIV medical care, but only 49% were retained in care, of whom the viral suppression rate was 62%. [3–4] Upwards of 50% PLWH who are diagnosed with HIV infection did not remain in HIV care across the care continuum, including those who failed in linkage to care, those who received care but subsequently fell out of care, and those who reengaged in care or transferred from other settings. [4] An ongoing iEngage trial showed that among those who linked to HIV care, 86% of individuals maintained viral suppression after 48 weeks of diagnosis. [5] Disparities in linkage to and retention in care persist with studies indicating that a significant proportion of African Americans were not in care, or for those in care, not achieved viral suppression. [3,6–7] These findings collectively suggest that high volumes of drop-offs during the HIV care drive the low overall rates of viral suppression. Statistics from South Carolina support this suggestion. Cumulatively, only 66% of SC PLWH received any care across the treatment continuum; of these people, 54% received continuous HIV care; and 57% were virally suppressed. [6–8] Accordingly, in the 2020 update of the National HIV/AIDS Strategy for the United States, “prevention through linkage and retention” was included as an imperative objective. [9] To improve the HIV care continuum, research has centered on the factors that influence retention in care as well as effective clinical, behavioral, and structural interventions to PLWH adhere to care. Studies suggest that factors that influence retention in care can be summarized into an adapted socioecological framework, which consists of individual level, relationships, community-level, health care system, and healthcare policy. [4] This framework outlines numerous findings of poor engagement across the care continuum. For example, poor linkage and retention have been discovered in key populations, including youth and adolescents [10], heterosexual men [11,12], transgender women [13], African Americans [11–14], and Hispanics/Latinos [11,12,15] in the United States. A number of factors are associated with poor linkage and retention, including mental illness [16–19], stigma and fear [20], place of residence [21–23], substance use [15,18,24], and access to health insurance [25]. These findings have informed interventions for improving retention in care with various foci, including linkage care management, medical case management, intensive outreach, peer and paraprofessional patient navigation, and clinic-wide messaging and culture. [4] However, many of these studies were focused on a few contributing factors and/or population samples without considering integrated evidence from clinical, administrative, and community-based datasets. This limitation has led to several missed opportunities for identifying those at risk for not engaging in care, and those at risk for disengaging in care. The reasons for these missed opportunities may be many. First, surveillance data across the HIV care continuum often fails to capture a complete portrait of PLWH’s health status because of the lack of interoperability among datasets provided by different health service agencies. Second, it is often difficult to systematically identify missed opportunities for HIV testing and treatments as well as vulnerabilities of HIV care services across the care continuum. Third, it is difficult to proactively intervene PLWH who are at high risk of dropping out of HIV care using limited data and predictive models with only linear combinations. This is important with studies showing reasonable costs and cost savings associated with retention in care for PLWH. [2, 26] Recent literature suggests that the plan for ending the HIV epidemic in the United States could improve when Big Data science is used to advance data integration and treatment, address gaps in transmission risks/behavior and improve HIV testing. [27]

To fill these knowledge gaps, our effort has led to an integrated data repository that accumulates individual level data of all types (i.e., clinical, administrative, claims and community-based health services data) for PLWH in South Carolina as reported elsewhere. [28] A unique advantage of this data repository is that it enables the prediction of future care status for PLWH through the rich, integrated, and high volume of individual level data. Despite sparse applications of predictive modeling in epidemiology and infectious diseases [29–30], no existing research has reported the development of computational methods used for predicting retention-in-care using a comprehensive data repository.

In the present study, we sought to develop and identify the best machine-learning based predictive model that can predict PLWH’s HIV medical care status by learning from historical data from the established data repository. The out-performed model was selected by benchmark comparison among seven supervised machine learning algorithms for the ability of accurately predicting PLWH’s retention-in-care. The out-performed model was further analyzed to identify variables that are most effective in prediction and clinically meaningful to HIV medical care. The expected outcomes of this study hold potential to establish the benchmark evidence for predictive analysis of PLWH’s care continuum, which further extends to evidence-based interventions and HIV health services underpinned by integrated clinical, administrative, claims, and community-based health data.

METHODS

Data repository

The complete architecture of the South Carolina PLWH Big Data repository is described elsewhere. [28] Below we highlight information important to the present study. The data repository received data from (1) South Carolina HIV/AIDS electronic reporting system (e-HARS), which is a laboratory-based reporting system that collects statewide CD4 and viral load (VL) tests since January 2004; (2) Ryan White HIV/AIDS Program Service Reports (RSR), which collect clinical data provided by Ryan White-funded entities; and (3) Health Sciences South Carolina (HSSC) clinical data warehouse, which integrated clinical records from six of the state’s largest health systems (AnMed Health, McLeod Health, Medical University of South Carolina Hospital Authority, Palmetto Health, Self Regional Healthcare, and Spartanburg Regional Healthcare System). The South Carolina office of Revenue and Fiscal Affairs (RFA) integrated PLWH data from these sources with an all payer healthcare inpatient database, Medicaid claims data (including demographic files, visits files, and pharmacy files), state employee health services plan, Department of Corrections data (crime rates, prison history, etc.), Department of Mental Health, etc.

Study population

The data repository includes PLWH whose residence at diagnosis was in South Carolina and whose age was 13 or older since 2005. We selected 2005 because it was the first year South Carolina state law mandated the reporting of all CD4 and VL tests to e-HARS. We identified data qualified for machine learning experiments based on several criteria. First, we defined care status for this experiment using CD4 or VL test as a proxy measure. Second, we used linkage to care as an inclusion criterion for all PLWH in the database. As a result, a total of 8,888 distinct PLWH records with 3,670,845 observations were identified. To prepare for modeling, we cleaned the data, addressed missing values issues, and generated a total of 3,640,102 observations in the final study sample.

Machine learning experiments

There were 40 input variables and 1 output (target) variable included in the machine learning experiments. Table 1 shows a summary of selected variables. The target variable is PLWH care status. We used the definition of HIV medical care as recommended by the CDC: “documentation of at least two CD4 cell counts or viral load tests performed at least three months apart during the year of evaluation” to define the care status. [23] All data records were annotated for care status (in care vs not-in-care) per this definition.

Table 1:

Selected Machine Learning Variable Attributes for adult PLWHs in South Carolina

Variable Name	Role	Level	Level (N)
Sociodemographics
Age	INPUT	INTERVAL
Gender	INPUT	NOMINAL	2
Race and ethnicity (White, African Americans, Other, Hispanic)	INPUT	NOMINAL	5
Education	INPUT	NOMINAL	7
Marital Status	INPUT	NOMINAL	6
County of Residence at HIV Diagnosis	REJECTED	UNARY	0
Type of Primary Caregiver	INPUT	NOMINAL	8
South Carolina Resident (Yes/No)	INPUT	BINARY	2
Clinical Factors
Retention in Care Status (In care vs. Not in Care)	TARGET	BINARY	2
Age at HIV Diagnosis	INPUT	INTERVAL
Age at AIDS Diagnosis	REJECTED	INTERVAL
AIDS Category (Yes/No)	INPUT	NOMINAL	3
HIV Transmission Risk Categories	INPUT	NOMINAL	6
Group (Care Groups)	INPUT	NOMINAL	4
HIV Viral Load	INPUT	INTERVAL
CD4 count	INPUT	INTERVAL
Year (Calendar year of HIV Diagnosis)	INPUT	NOMINAL
Supply (Days of drug supply)	REJECTED	INTERVAL
Years of retention in care (count)	INPUT	BINARY	2
Retention in Care Sequence for follow-up year	INPUT	INTERVAL
Health System Utilization
Zero Income Eligibility (Yes/No)	REJECTED	BINARY	2
Poverty (Percent)	INPUT	INTERVAL
Medicare Dual Eligibility	REJECTED	BINARY	2
Ever on AIDS Drug Assistance Program	INPUT	BINARY	2
# of Prescription Drug Refills used	REJECTED	INTERVAL
Emergency Room Flag (Yes/No)	INPUT	BINARY	2
DMH Payor Event 1	INPUT	NOMINAL	6
DMH Payor Event 2	INPUT	NOMINAL	5
DMH Payor Event 3	INPUT	BINARY	2
DMH Payor Event 4	INPUT	NOMINAL	4
DMH Payor Event 5	INPUT	BINARY	2
Time (Time from diagnosis date to linkage)	INPUT	INTERVAL
Hospital Length of Stay (Days)	INPUT	INTERVAL
Hospital Payor	INPUT	NOMINAL	6
# of comorbidities	INPUT	NOMINAL	7
# of diagnoses	INPUT	INTERVAL
Therapeutic Drug Class	REJECTED	INTERVAL
# of Prescription Drugs	INPUT	INTERVAL
Corrections Information
Jail (Yes/No)	INPUT	BINARY	2
Jail (Length of Stay in Days)	INPUT	INTERVAL

All data cleaning/management and machine learning experiments were performed using SAS 9.4 and SAS Viya 3.4. Since missing values can create bias during predictive modeling, we imputed values for variables with 10% or less of missing data using the tree surrogate method [31] for categorical variables and median for interval variables. We employed seven supervised machine learning algorithms that represent a variety of approaches including Bayesian Network, Neural Network, Support Vector Machine, Logistic Regression, least absolute shrinkage and selection operator (LASSO), Decision Trees and Random Forest.

Data Partitioning

The 10-fold cross validation approach was employed as the framework for machine learning experiments to measure unbiased accuracy. Figure 1 illustrates the diagram of machine learning experiments used for this study. Specifically, we randomly generated ten equal splits of the data. We chose to partition our data into three groups namely training, validation and testing to avoid overfitting that could occur using only training and validation data (Figure 1). For each algorithm, six splits (N=2,184,061, [60%]) were used for training, three splits (N=1,092,031 [30%]) were used for validation, and one split (N=364,010 [10%]) was randomly selected for testing. We used F-1 measure, precision, recall, and the area under the receiver operating characteristic (AUC) to assess the model performance. These measures were generated based on a contingency table that specifies whether or not the model prediction is correct based on the true values in the dataset (see Figure 2). The AUC shows how true positive rates and false positive rates interact with each other.

Figure 1.

Diagram of machine learning experiments.

Figure 2.

Measures used for algorithm performance evaluation.

Algorithms

Bayesian Network is a supervised learning approach that fits a Bayesian Network model of inputs to our nominal target (care status). Bayesian Network is a directed, acyclic graphical model where the structure is based on conditional dependency between two random variables. Details about its recent utility and promise in personalized medicine for healthcare is documented by several studies [37–40]. Neural Network is a supervised learning method we used to assess the connections between input variables, hidden layers, and an output layer (care status), and the connections between each of these similar to the biologic structure of a human brain. Its utility is also well documented in areas like cancer, diabetes and electronic health records based studies. [41–45] For Neural Network, prior variable selection is preferred, so we used the fast-supervised selection method [46–47] that identified input variables (cluster) which jointly explained the largest variance for care status and compared it to the Neural Network with no selection group. We used decision trees (classification and regression trees [CART]) as a non-parametric supervised learning method to create a tree model for relevant input variables associated with our outcome (care status) using a series of rules. Each rule assigns an observation to a segment, based on the value of one input. [48–49] Random Forest is an extension of decision trees and consists of multiple decision trees using different samples with its use also reported in healthcare. [50] Support vector machine is a well-known optimization-based supervised learning method for fitting binary classification problems. [51–54] It does this by identifying a decision boundary with the maximum possible margin between the data points and has also been applied to healthcare. Logistic regression is a member of discriminative models in machine learning [33] and was used here to predict care status as a function of input variables.

Feature analysis

To determine the relative importance of variables in predicting care status, we employed different measures for each algorithm. Consistent with our data driven approach, we ran machine learning algorithms against the variable selection options (fast-selection) and compared it to machine learning algorithms for all the variables without prior selection. We ranked the importance of input variables by the performance decrease when taking each input variable out of the model. Top-ranked input variables were candidate principal factors for predicting PLWH’s care status.

RESULTS

We present the selected results of these algorithms and compared model performance in predicting care status for PLWH using the selected input variables. Results presented differ based on types of algorithms. Figure 3A and 3B show decision tree models for the data driven input variable selection (cluster) and for all input variables. Notable tree splits differed in the highlighted branches based on the misclassification rate by variables. Both trees had varying inputs of relative importance. Both had years of retention in care but differed at the main split between hospital payor and age at HIV diagnosis. Other split differences are annotated in the figures below. For ease of understanding, we display a forward feed single layer neural network showing weights associated by input variable level to our target (care status) see Figure 4A. Input variables of importance included years of retention in care, age at HIV diagnosis, hospital payor, type of transmission risk, marital status and education. Similarly, figure 4B displays the Bayesian Network diagram for classifying/specifying the conditional relationships/dependencies between the input and target variables. Dependencies show intricate but important relationships between input variables and target (care status) highlighted in yellow. This result indicates that predictive relationship is often codependent on other relevant input variables. The variable selection by relative importance also differed for other algorithms.

Figure 3.

Visualization of trained Decision Trees.

Figure 4.

Visualization of trained Neural Network.

Machine learning performance

Consistent with the primary aims of this study, we compared and scored each algorithm for performance. Bayesian Network (F=0.87, AUC=0.94, precision=0.87, recall=0.86) has outperformed other algorithms, followed by Random Forest (F=0.78, AUC=0.81, precision=0.72, recall=0.85), Decision Tree (F=0.76, AUC=0.75, Precision=0.70, Recall=0,82) and Neural Network (cluster) (F=0.75, AUC=0.71, precision=0.69, recall=0.81). Table 2 shows the results for each algorithm. The AUC across the experimented algorithms are consistent with the F scores. Figure 5 illustrates a diagrammatic comparison of AUC among algorithms for training, validation and test data.

Table 2.

Prediction performance for all the algorithms

Algorithms	Precision	Recall	F score	AUC
Bayesian Network	0.87	0.86	0.87	0.94
Random Forest	0.72	0.85	0.78	0.81
Decision Tree	0.70	0.82	0.76	0.75
Neural Network (cluster) 1	0.69	0.81	0.75	0.71
Logistic Regression	0.69	0.82	0.75	0.71
Support Vector Machine	0.69	0.80	0.74	0.67
LASSO	0.56	1.00	0.72	0.50
Neural Network2	0.56	1.00	0.72	0.50

¹Neural Network with fast variable selection;

¹Neural Network without fast selection

Figure 5.

Areas under Receiver Operating Characteristic (AUC) curves.

Recall scores are consistently higher than precision across all algorithms except for Bayesian Network, suggesting that these algorithms generally performed well in predicting as many PLWH who would remain in care as they will actually do so yet performed less well in assuring that all machine-identified retention-in-care individuals who actually will remain in care. Notably, for LASSO and Neural Network, the recall scores have achieved 1.00 yet the precision scores are low, which significantly impeded the balanced performance as indicated by the F scores and AUC. This result suggests that a considerably large portion of in-care PLWH that is identified by LASSO and Neural Network do not actually remain in care. Figure 5 shows AUC curves for all the algorithms and data splits (training, validation and testing). The AUC curve is a plot of sensitivity (the true positive rate) against 1-specificity (the false positive rate), which both are measures of classification based on the confusion matrix. These measures are calculated at various cutoff values. An AUC curve that rapidly approaches the upper-left corner of the graph, where the difference between sensitivity and 1-specificity is the greatest, indicates a more accurate model. Consistently, the Bayesian Network outperformed other models. Figure 6 illustrates the input variables by relative importance for the champion model – Bayesian Network. This plot shows the 23 most important input variables, as determined by the relative importance. The relative importance is calculated using a one-level decision tree for each input to predict the predicted value as a global surrogate model. The most important input variables for this model is years of retention-in-care with a relative importance score of 1, followed by a number of important variables with a relative importance scores ranging from 0.1 to 0.3, including CD4 count, Calendar year of HIV diagnosis, number of diagnoses, marital status, gender, number of prescription drugs, zero income eligibility (yes/no).

Figure 6:

Relative Importance Champion Model - Bayesian Network

DISCUSSION

To help end the HIV epidemic, predictive healthcare decisions and interventions for PLWH must be driven by evidence and data. Studies suggest decision-making in healthcare can benefit from predictive models and artificial intelligence. [47, 55–56] In this study, we compared different machine learning algorithms, and demonstrated important differences in model performance when predicting PLWH care status. Important predictive factors for care status are typically not linearly associated with care status suggesting more complex relationships (dependencies) may exist. Our study assessed both traditional and novel statistical methods of prediction that hold implications for the field particularly with the Bayesian Network performing best for prediction. The slight differences between the ranked variables of importance hold implications for designing future interventions.

Principal findings

The present study is among the first to leverage machine learning and Big Data analytics to predict HIV medical care status of PLWH using their previous health records consisting of clinical, health administrative, and community-based health data. The study demonstrated that PLWH’s future state of HIV care utilization is predictable by using supervised machine learning and integrated health data. Additionally, we identified factors, including last year of retention in care, CD4 count, calendar year of HIV diagnosis, number of diagnoses, marital status, gender, number of prescription drugs, zero income eligibility (yes/no) as important inputs that play a key role in indicating whether or not a PLWH is likely to retain in care. These identified key predictive factors demonstrate that nuanced information from PLWH’s clinical care, social determinants of health, and activities in social care jointly contributes to a precise predictive capability of PLWH’s future HIV medical care status.

Supervised machine learning algorithms are feasible for predicting retention-in-care. Algorithms such as Bayesian Network, Random Forest, Decision Trees, and Neural Network (cluster) showed superior predictive performance. However, algorithms such as LASSO and Neural Network did not show effectiveness in the prediction task. One possible reason for the substandard performance for the Neural Network could be that we only tested a basic three-layer model with default parameters. We believe there may be deeper temporal relationships that need detection to further improve our predictive models. [57] To provide a systematic investigation on algorithms that heavily relies on initial parameters and architecture (i.e., Neural Network), follow-up studies should be directed to develop tailored Neural Network models. Overall, the comparatively lower scores in precision over recall are a major source of error that affected the balanced performance. Considering the fact that we only selected a limited subset of input variables available from the integrated PLWH data repository, the problem of low precision could be improved if follow-up studies would include input variables that would contribute to the accurate identification of negative output variable (i.e., not in care).

Clinical and policy implications

Several of these predictive factors have been identified in literature as important for retention in care. [58]However, the challenge for ending the HIV epidemic goes beyond linking PLWH into care, to retaining them in HIV medical care. Identifying and predicting PLWH at high-risk for being not-in-care after linkage allows for reallocation of resources to keep such high-risk population in HIV medical care. This leads to cost savings from reduced transmissibility, improved service for PLWH, and cost savings from preventing individuals dropping out of care. From evidence-driven healthcare and precision-medicine perspectives, our study shows that the application of powerful machine learning techniques (in this case, Bayesian Network and Random Forests) makes enough difference to explore and justify their potentials for predicting future HIV care status for any PLWH population using integrated data sources. It also holds promise for predicting at the individual level, which could aid preventing dropouts in HIV medical care. To decipher the complex probability relationships observed from the decision trees, Neural Network and Bayesian Network for improving the prediction of future care status, Deeper Learning is needed. This work provides evidence for the increased use of machine learning techniques when large-scale integrated datasets exist. Observed performance differences also justify the need to apply machine learning methods on large-scale integrated data from multiple sources.

Limitations and future direction

This study presents a few limitations. First, as a pilot study, we only selected a limited number of input variables for machine learning experiments guided by domain experts. There may be important variables that can be used to improve prediction but were not included in machine learning. Second, other states in HIV care continuum such as linkage-to-care and recipient-of-care are equivalently important for HIV care adherence but were not tested in the machine learning experiments. Third, there might be intricate relations (e.g., temporal relations) among input variables across the years in care. The present study did not aim to investigate these relations. Fourth, identified predictive factors need to be further evaluated to confirm their clinical value. To address these limitations, future studies are directed to (1) expand the input variables and other states in HIV care continuum as the PLWH data repository continues to include and link data, (2) develop advanced predictive models (e.g., Deep Learning) to solve the computational complexity raised by the expanded dataset, and (3) conduct a comprehensive evaluation on the identified predictive factors.