Machine learning algorithms using routinely collected data do not adequately predict viremia to inform targeted services in postpartum women living with HIV

Abstract

Background

Adherence to antiretroviral treatment (ART) among postpartum women with HIV is essential for optimal health and prevention of perinatal transmission. However suboptimal adherence with subsequent viremia is common and adherence challenges are often under-reported. We aimed to predict viremia to facilitate targeted adherence support in sub-Saharan Africa during this critical period.

Methods

Data are from PROMISE 1077BF/FF which enrolled perinatal women between 2011–2014. This analysis includes postpartum women receiving ART per study randomization or country-specific criteria to continue from pregnancy. We aimed to predict viremia (single and confirmed events) after 3 months on ART at >50, >400 and >1000 copies/mL within 6-month intervals through 24 months. We built models with routine clinical and demographic data using the least absolute shrinkage and selection operator (LASSO) and SuperLearner (which incorporates multiple algorithms).

Results

Among 1321 women included, the median age was 26 years and 96% were WHO Stage 1. Between 0–24 months postpartum, 42%, 31% and 28% experienced viremia >50, >400, and >1000 copies/mL at least once, respectively. Across models, the cross-validated area under the receiver operating curve (AUC) ranged from 0.74 (95% confidence interval [CI]: 0.72–0.76) to 0.78 (95% CI: 0.76–0.80). To achieve 90% sensitivity predicting confirmed viremia >50 copies/mL, 64% would be classified as high risk.

Conclusions

Using routinely collected data to predict viremia in >1300 postpartum women with HIV, we achieved moderate model discrimination, but insufficient to inform targeted adherence support. Psychosocial characteristics or objective adherence metrics may be required for improved prediction of viremia in this population.

INTRODUCTION

Roughly 1.2 million women living with HIV in sub-Saharan Africa gave birth in 2020, and nearly 90% had access to antiretroviral treatment (ART).¹ With adequate adherence, ART during pregnancy and postpartum optimizes maternal health² and virtually eliminates perinatal and sexual HIV transmission.^3–9 However, adherence to daily medication can be particularly challenging during the postpartum period, given the disruption of routine with a newborn. Indeed, suboptimal adherence and subsequent viremia are frequently observed in postpartum women.^2,10–15 Evidence-based approaches to improve ART adherence, such as peer support, pill boxes, and enhanced counselling,^16–18 can be resource-intensive, and are not always appropriately targeted to those in need of support, in part because inadequate adherence is not readily identified. Both recall bias and social desirability bias can lead to under-reporting of “unhealthy” behaviors, and, as such, self-reported adherence can be unreliable.^19,20 Scalable strategies to identify postpartum women in need of adherence support, before virologic failure, antiretroviral resistance,²¹ or HIV transmission during breastfeeding occurs, are urgently needed.

A risk score using readily collected information to predict which women are most likely to experience viremia could facilitate timely and targeted enhanced adherence interventions to those most in need. In addition to reaching the correct individuals to prevent viremia, appropriately targeting services could improve the quality of delivery of resource-intensive interventions which may be challenging to deliver well to a wider population. Indeed, differentiated service delivery according to patient need is increasingly important as universal ART places high demands on constrained health facilities in resource-limited settings such as sub-Saharan Africa.^22–24 Risk scores predicting virologic failure in non-pregnant or postpartum populations have been developed to inform targeted virologic monitoring with moderate prediction performance,^25–30 although their implementation has not been reported. The postpartum period presents unique challenges to adherence with new responsibilities and disruptions in routines, often including travel to stay with relatives post-delivery who may be unaware of the mother’s HIV status. Simultaneously, adherence is especially critical during this time for prevention of HIV transmission to infants while breastfeeding. Thus, factors linked to the postpartum experience, such as disclosure of HIV status to partner or relatives, breastfeeding status, or pregnancy outcomes, may help predict viremia during this unique period.

We therefore aimed to develop a risk score to predict viremia using routinely collected clinical and demographic data, including pregnancy-related predictors, to support the identification of women who might benefit most from enhanced adherence interventions. This analysis leverages data from a large cohort of postpartum women on ART across sub-Saharan Africa enrolled in the Promoting Maternal-Infant Survival Everywhere (PROMISE) Study,^3,5,31 and uses machine learning methods to formulate a score to predict viremic events within 6 months, applicable from delivery through 2 years postpartum.

METHODS

Study setting, population, and procedures.

Data are from the International Maternal Pediatric Adolescent AIDS Clinical Trials Network (IMPAACT) PROMISE 1077BF/1077FF Study (NCT01061151), which was conducted between 2011–2015 at 14 sites where breastfeeding was standard (in Zimbabwe, Malawi, South Africa, Uganda, Zambia, Tanzania, and India) to examine optimal strategies for prevention of perinatal transmission of HIV and improving maternal health, prior to universal treatment guidelines.³² The study enrolled pregnant and postpartum women living with HIV who were not yet eligible for ART at the time. Thus, women with CD4 counts <350 cells/mm (or a higher threshold, according to individual country guidelines) and women with any clinical signs or symptoms that designated WHO clinical Class IV status³³ were excluded. Study procedures and primary results have been reported.^3,5,31 Briefly, PROMISE 1077BF included three components with three separate randomizations: during pregnancy (antepartum), during breastfeeding up to 18 months (postpartum), and after breastfeeding cessation (maternal health). PROMISE 1077FF enrolled women who chose not to breastfeed, and therefore only included antepartum and maternal health components. In the antepartum component, women were randomized to either tenofovir disoproxil fumarate (TDF)/emtricitabine (FTC)/lopinavir/ritonavir (LPV/r), zidovudine/lamivudine/LPV/r, or zidovudine plus single-dose nevirapine at delivery followed by a tenofovir/emtricitabine “tail” (for 6 to 14 days after delivery). In the postpartum component, women were randomized to either maternal TDF/FTC/LPV/r or daily infant nevirapine during breastfeeding. In the maternal health component, women already on ART in the prior component were randomized to stop (consistent with temporal guidelines) or continue ART. Participants who met country-specific guidelines for treatment during any component were started or maintained on ART. In July 2015, in response to the START trial results,³⁴ all participants not currently receiving ART were given the recommendation to initiate ART.

All participants provided informed consent prior to PROMISE study participation. The study was approved by local and collaborating institutional review boards (Supplement, Section I) and other relevant regulatory authorities and was reviewed for safety and efficacy by an independent Data and Safety Monitoring Board.

This analysis includes all women from African sites on ART postpartum (per randomization or meeting country-specific guidelines to continue from pregnancy). Participants attended study visits according to protocol with viral load assessed at least every 6 months with more frequent assessments early in each component. We included viral loads collected after 3 months on ART through 2 years postpartum. Follow-up was censored when new pregnancies occurred (not all potential predictors, e.g., breastfeeding status, were collected in subsequent pregnancies) or if ART was stopped per the PROMISE protocol.

Study outcomes

Our study outcome was viremia (plasma HIV-1 RNA) at thresholds of >50, >400, and >1000 copies/mL. We considered these thresholds for the following reasons: 50 because breastfeeding transmission has been observed with low-level viremia^35–37 and low-level viremia has been associated with subsequent disease progression;³⁸ 400 because this is the lower limit of detection in many assays used in resource-limited settings and was the outcome in PROMISE; 1000 as this is what the WHO defines as virologic failure. We developed models to predict viremic events within six-month intervals (from delivery to six months postpartum, and from 6–12, 12–18, and 18–24 months postpartum), using data collected prior to or at the baseline of each interval as predictors. We assessed both “any” viremia (at least one measure) as well as “confirmed” viremia (2 consecutive measures) during each six-month interval.

Candidate predictors

We considered readily collected demographic predictors including maternal age, education level, employment status, gestational age at first antenatal care visit, and gravidity. Pregnancy-related predictors included time since delivery (modeled as indicators for each 6-month interval postpartum), ART initiation in pregnancy vs. at delivery, breastfeeding status, preterm delivery, twin pregnancy, infant death (neonatal or in follow-up), and infant HIV acquisition. Routine clinical predictors included: nadir CD4 count, hemoglobin in pregnancy/at delivery, recent viremia (>50 copies/mL in the prior 6 months), history of viremia (>6 months prior and >3 months after ART initiation), WHO stage, self-reported ART adherence, missed visit history, and prior hospitalizations. We also considered variables that may not be routinely collected in some settings for potential inclusion in “enhanced” models: food insecurity, alcohol use, HIV status disclosure to male partners or others in the household, absolute neutrophil count, white blood cells, lymphocyte count, and creatinine.

Model development

We first described the unadjusted association between each candidate predictor and viremia (>50 copies, confirmed) using generalized estimating equation logistic models.. All continuous variables were standardized prior to analyses and highly skewed variables were log transformed prior to standardization. Because complete data is required for some machine learning methods,, we handled missing data in model building as follows: for categorical predictors we created a “missing” category; for continuous variables we set missing values to the mean and created a missing indicator variable.

We applied two approaches to model building. First, we used the least absolute shrinkage and selection operator (LASSO) method, which shrinks coefficients to optimize calibration (comparable frequencies of predicted and observed risks), and performs variable selection by shrinking some coefficients to zero.⁴⁰ This approach ensures a parsimonious final model which could be translated into a simple point-based risk calculator. We selected the shrinkage parameter with the smallest cross-validated prediction error. Second, we used SuperLearner⁴¹ which optimizes prediction performance by combining predictions from multiple algorithms (“learners”), weighting each algorithm’s prediction according to its cross-validated prediction error. SuperLearner does not perform variable selection and would require a computer or tablet to calculate predictions. As candidate learners, we included standard logistic regression, LASSO, random forests, and gradient boostingRandom forests and gradient boosting both combine results from a large number of decision trees, each randomly defined based on designated parameters (e.g., the number of trees, the number of variables used in each tree for prediction). In random forests, trees are built in bootstrapped samples; in gradient boosting trees are built sequentially, each fit to the residuals from the prior tree.⁴²

To consider model performance in the context of clinical care, we conducted two secondary analyses. First, we built models following the same approach described above with LASSO and SuperLearner, but excluded participant observations with recent viremia (>50 copies within 6 months prior to the baseline of each interval,after ≥3 months on ART) as women would already be identified as at risk at these time points, not requiring a risk assessment tool. We then assigned a predicted probability of 1 to those excluded observations. Second, we built simple logistic models using recent viremia (>50 copies/mL in the prior 6 months) alone to classify persons as high risk, as a reference point for risk assessment without a novel scoring tool.

Assessment of model performance

We summarized performance in terms of internally cross-validated discrimination (the degree to which the model assigns a high predicted risk to those who actually have the outcome, and low predicted risk to those who do not), quantified as the area under the receiver operating curve (AUC), and calibration (by comparing predicted and observed risks by decile of the predicted risk).⁴³ Cross-validation folds (n=10) were defined by study site. Small sites were included in the same fold of a neighboring site to avoid small numbers in a single fold (two sites represented Gauteng Province, South Africa, two represented KwaZulu-Natal Province, South Africa, and Tanzania and Zambia both had small numbers and were combined). Splitting the data by region, rather than randomly selecting observations, provides a more stringent internal validation.⁴⁴ Moreover, this approach ensures that participants contributing >1 observation have all of their observations in the same fold.This avoids overly optimistic estimates of performance which would occur when evaluating prediction in the same individuals among whom the model was developed. To visualize calibration, we plotted the predicted risks from the model against observed events in the sample. To further assess the clinical utility of the models, we estimated sensitivity, specificity, positive and negative predictive values (PPV, NPV), and report these values when requiring minimum sensitivities of 80% and 90%, and report the proportion classified as “high risk” at these thresholds. We did not specify minimum thresholds across each of these parameters, but estimated them at 80 and 90% sensitivity with an aim to reach the vast majority of women at high risk. Additionally, for appropriate targeting of resource-intensive adherence support that minimizes unnecessary burden for both patients and clinicians, PPVs >50% would be desirable.

Analyses were conducted in STATA (StataCorp, College Station, TX) version 16 and R (R Foundation for Statistical Computing, Vienna, Austria) version 4.0.2.

RESULTS

Among 1486 PROMISE participants on ART postpartum (via randomization or continued from pregnancy), 1321 were included in this analysis; those excluded did not have any viral load data after 3 months on ART, largely due to early termination of ART (Supplement, Section II). The median age of included participants was 26 years (interquartile range [IQR] 23, 30) and 831 (63%) initiated ART in pregnancy (Table 1). In this early treatment initiation trial, nadir CD4 was relatively high (median 526, IQR 433–664) and 96% were classified clinically as WHO Stage 1. Median follow-up time in this analysis (which was censored at 2 years, ART stop, or a new pregnancy) was 23 months (IQR 15–24).

Table 1.

Demographic and clinical characteristics of 1321 postpartum women on ART in the PROMISE 1077BF/FF Study

Characteristics	n (%) or median (IQR) N=1321
Country
Malawi	454 (34.4%)
South Africa	307 (23.2%)
Tanzania	25 (1.9%)
Uganda	212 (16.0%)
Zambia	30 (2.3%)
Zimbabwe	293 (22.2%)
Age	26 (23–30)
Gestational age at first antenatal care visit	23.4 (18.6–28.0)
Gravidity	3 (2–4)
Education
None or some primary	202 (19.8%)
Completed primary but not secondary	461 (45.2%)
Completed secondary	357 (35.0%)
Missing	301
Employment status
Not working	1,066 (83.1%)
Working part-time	63 (4.9%)
Working full-time	154 (12.0%)
Missing	38
Food insecurity
None/mild	1,012 (83.9%)
Moderate	98 (8.1%)
Severe	96 (8.0%)
Missing	115
HIV status not disclosed (to partner/household members)	77 (6.6%)
Still birth or infant death within 7 days	23 (1.7%)
Gestational age at delivery in weeks
< 34	24 (1.8%)
34 – <37	150 (11.5%)
37+	1,130 (86.7%)
WHO clinical classification
Clinical stage I	1,274 (96.4%)
Clinical stage II	46 (3.5%)
Clinical stage III	1 (0.1%)
Nadir CD4+ count	526 (433–664)
Treatment assignment during antepartum component
Enrolled on/after delivery	49 (3.7%)
Triple-drug ART	831 (62.9%)
Zidovudine	441 (33.3%)
Treatment assignment during postpartum component
ART continued from antepartum for maternal health	241 (18.2%)
Maternal ART	1,080 (81.8%)
Treatment assignment during maternal health component
Continue maternal ART	409 (31.0%)
Discontinue triple ART	233 (17.6%)
Not enrolled	679 (51.4%)

Viremia frequency

Between delivery and 24 months postpartum, 42%, 31% and 28% of women experienced at least one viremic episode >50, >400, and >1000 copies/mL, respectively (Table 2). Confirmed viremic episodes (2 consecutive measures) were observed in 26%, 20%, and 17% at thresholds of 50, 400, and 1000 copies/mL, respectively. The median time between a first viremic event (>50 copies/mL) and a confirmatory measure was 56 days (IQR 25–84), with similar times at higher thresholds. The frequency of viremia was similar across 6-monthly intervals, with slightly lower frequencies after 6 months postpartum (e.g., 27% experienced viremia >50 copies/mL at least once between 0–6 months, while 21%, 23%, and 24% did so in each subsequent interval).

Table 2.

Viremia frequency by six-monthly intervals and overall, from 0–24 months postpartum among 1321 women on ART in the PROMISE Study

		0–6 months	6–12 months	12–18 months	18–24 months
Full sample
		N=1,299	N=1,198	N=998	N=682	Ever, 0–24 months, N=1321 participants
Viral load threshold (copies/mL)
50	single	352 (27%)	253 (21%)	226 (23%)	163 (24%)	550 (42%)
	confirmed	211 (16%)	163 (14%)	134 (13%)	79 (12%)	337 (26%)
400	single	228 (18%)	203 (17%)	165 (17%)	131 (19%)	414 (31%)
	confirmed	146 (11%)	126 (11%)	96 (10%)	55 (8%)	261 (20%)
1000	single	199 (15%)	173 (14%)	139 (14%)	113 (17%)	376 (28%)
	confirmed	116 (9%)	101 (8%)	74 (7%)	43 (6%)	220 (17%)

Notes: Total Ns within each interval represent the number of participants with VL assessed after 3 months on ART within the interval

Model building and performance

Consistent with prior work, we observed several strong predictors of viremia in bivariate analysis, for example, younger age (standardized; OR 0.69, 95% CI 0.61–0.79), lower nadir CD4 (standardized, OR 0.83 95% CI 0.73–0.96), and recent viremia (OR 6.78, 95% CI 5.60–8.21; Table 3). In LASSO models developed with routine data, the cross-validated area under the receiver operating curve (AUC) ranged from 0.76 to 0.78 (Table 4). LASSO shrunk very few coefficients to zero; nearly all were retained (Supplemental Table 2). Performance was comparable in SuperLearner (Figure 1) and when incorporating non-routine data (additional lab values, food insecurity, alcohol use) in “enhanced” models (Supplemental Table 3). Across SuperLearner models, standard logistic regression and LASSO predictions were weighted most heavily (Supplemental Table 3).

Table 3.

Bivariate associations between candidate predictors and confirmed viral load >50 copies/mL; all continuous variables are standardized.

Baseline Characterisitics	OR (95% CI)	p-value
Age	0.69 (0.61–0.79)	<0.0001
Education
None or some primary	(ref)	(0.40 overall)
Completed primary but not secondary	1.04 (0.71–1.51)	0.86
Completed secondary	0.83 (0.55–1.24)	0.36
Gravidity	0.90 (0.80–1.01)	0.07
Gestational age at first antenatal care visit	1.10 (0.97–1.25)	0.16
Nadir CD4	0.83 (0.73–0.96)	0.01
Hemoglobin	0.91 (0.81–1.03)	0.12
Initiated ART in pregnancy (vs. at delivery)	1.28 (0.98–1.68)	0.07
Preterm
37 weeks	(ref)	(0.21 overall)
34 up to <37	0.80 (0.54–1.18)	0.26
<34 weeks	1.61 (0.78–3.32)	0.20
Twins (vs. singleton)	3.88 (1.85–8.14)	<0.001
Time-varying characteristics
Time since delivery
0–6 months	(ref)
6–12 months	0.81 (0.69–0.95)	0.01
12–18 months	0.80 (0.66–0.97)	0.03
18–24 months	0.68 (0.52–0.87)	0.003
WHO clinical classification
1	(ref)	(0.19 overall)
2	0.74 (0.42–1.32)	0.31
3/4	2.86 (0.73–11.20)	0.13
Recent viral load >50 (prior 6 months)	6.78 (5.60–8.21)	<0.0001
History of viral load >50 (before the prior 6 months)	2.48 (1.93–3.18)	<0.0001
Stopped breastfeeding prior to current visit	1.08 (0.86–1.37)	0.51
Baby acquired HIV prior to current visit	5.51 (1.96–15.50)	0.001
Infant death any time prior to current visit	1.73 (0.78–3.84)	0.18
Any missed visits in prior interval	1.33 (0.81–2.18)	0.27
Any hospitalizations in prior interval	1.23 (0.62–2.43)	0.56
Any alcohol use*	0.65 (0.33–1.29)	0.22
Absolute neutrophil count*	1.11 (1.02–1.20)	0.01
White blood cell count (log transformed)*	1.12 (1.02–1.22)	0.01
Lymphocytes (log transformed)*	0.93 (0.84–1.03)	0.17
Creatinine (log transformed)*	0.90 (0.82–1.00)	0.05
Adherence, any missed in past 4 weeks	1.24 (1.04–1.47)	0.02
Employment status
Not working	(ref)	(0.63 overall)
Working part-time	0.96 (0.63–1.45)	0.83
Working full-time	0.85 (0.62–1.18)	0.33
Food insecurity*
None/mild	(ref)	(0.10 overall)
Moderate	0.91 (0.62–1.36)	0.66
Severe	1.49 (1.02–2.19)	0.04
HIV status not disclosed to partner or household members	0.91 (0.57–1.47)	0.72

^*non-routine data considered in “enhanced” models

Table 4.

Model performance (with 95% confidence intervals) and proportion classified as “high risk” at selected viremia outcomes

	Viremia outcome
	50 copies/mL		400 copies/mL		1000 copies/mL
	single	confirmed	single	confirmed	single	confirmed
Viremia incidence (across 4177 observations)	550 (42%)	337 (26%)	414 (31%)	261 (20%)	376 (28%)	220 (17%)
LASSO models
AUC	0.77 (0.76–0.79)	0.78 (0.76–0.80)	0.78 (0.76–0.80)	0.78 (0.75–0.80)	0.76 (0.74–0.78)	0.76 (0.74–0.79)
Requiring 80% sensitivity
Proportion classified high risk	0.51	0.44	0.48	0.42	0.48	0.43
Specificity	0.59 (0.55–0.63)	0.62 (0.55–0.67)	0.59 (0.54–0.63)	0.62 (0.56–0.68)	0.58 (0.53–0.63)	0.60 (0.53–0.68)
PPV	0.38 (0.36–0.40)	0.26 (0.22–0.29)	0.29 (0.27–0.31)	0.19 (0.17–0.22)	0.25 (0.23–0.27)	0.15 (0.13–0.18)
NPV	0.90 (0.90–0.91)	0.95 (0.94–0.95)	0.93 (0.93–0.94)	0.96 (0.96–0.97)	0.94 (0.94–0.95)	0.97 (0.97–0.98)
Requiring 90% sensitivity
Proportion classified high risk	0.69	0.64	0.68	0.62	0.67	0.61
Specificity	0.38 (0.33–0.44)	0.39 (0.33–0.47)	0.37 (0.31–0.44)	0.41 (0.34–0.48)	0.37 (0.30–0.44)	0.42 (0.36–0.50)
PPV	0.31 (0.29–0.33)	0.20 (0.18–0.22)	0.23 (0.21–0.25)	0.15 (0.13–0.16)	0.20 (0.19–0.22)	0.12 (0.11–0.14)
NPV	0.92 (0.91–0.93)	0.96 (0.95–0.97)	0.95 (0.94–0.95)	0.97 (0.97–0.98)	0.95 (0.95–0.96)	0.98 (0.98–0.98)
Super Learner models AUC	0.76 (0.75–0.78)	0.77 (0.75–0.79)	0.76 (0.74–0.78)	0.76 (0.74–0.78)	0.74 (0.72–0.76)	0.74 (0.72–0.77)
Logistic regression (sole predictor: viremia >50 in prior 6 months)
AUC	0.66 (0.61–0.70)	0.66 (0.62–0.70)	0.68 (0.64–0.72)	0.66 (0.62–0.71)	0.66 (0.62–0.71)	0.65 (0.60–0.70)
Proportion classified high risk	0.15	0.15	0.15	0.15	0.15	0.15
Sensitivity	0.40 (0.37–0.43)	0.44 (0.40–0.48)	0.46 (0.43–0.50)	0.47 (0.42–0.52)	0.46 (0.41–0.50)	0.46 (0.41–0.52)
Specificity	0.93 (0.92–0.94)	0.90 (0.89–0.91)	0.92 (0.91–0.92)	0.89 (0.87–0.90)	0.90 (0.89–0.91)	0.88 (0.87–0.89)
PPV	0.62 (0.52–0.66)	0.40 (0.30–0.44)	0.52 (0.40–0.57)	0.31 (0.22–0.35)	0.44 (0.33–0.48)	0.24 (0.16–0.27)
NPV	0.83 (0.83–0.83)	0.91 (0.90–0.91)	0.89 (0.88–0.89)	0.94 (0.93–0.94)	0.90 (0.90–0.91)	0.95 (0.94–0.95)

AUC = Area under the receiver operating curve; PPV = positive predictive value; NPV = negative predictive value

Figure 1.

Receiver operating characteristic curves for LASSO and SuperLearner models predicting viremia at thresholds of 50, 400, and 1000.

In the LASSO model predicting confirmed viremia >50 copies/mL, with a predicted probability cutoff achieving 90% sensitivity, the PPV was 0.20 (95% CI: 0.18–0.22) and 64% would be classified as high risk. To achieve 80% sensitivity, the PPV was 0.26 (95% CI 0.22–0.29) and 44% would be classified as high risk. Across viremia outcomes, 42%−50% would be classified as high risk to achieve 80% sensitivity, reflecting only moderate gains in efficiency while missing 20% of those at risk.

In secondary analyses, first, we assigned a predicted risk of 1 for all persons with recent viremia (>50 copies/mL in the prior 6 months) and only modeled risk predictions in those without recent viremia, i.e., those who might otherwise be considered clinically stable. Model performance (accounting for both those assumed to be high risk and those with model estimated risk) was nearly identical to the primary results reported in Table 4 (data not shown). When we used recent viremia (>50 copies/mL in the prior 6 months) alone to classify persons as high risk, AUCs ranged from 0.65–0.66. This approach would identify 40% who would go on to have at least one viremic event >50 copies/mL, or 47% who would experience confirmed viremia >400 copies/mL. Notably, only 15% would be classified as high risk. At the same sensitivities achieved with recent viremia alone, the specificities from LASSO and SuperLearner models were nearly the same as those from the simple logistic models (data not shown).

In the model predicting confirmed viremia >50 copies/mL, calibration was suboptimal, with a non-linear association between predicted risk and observed outcomes (Supplemental Figure).

DISCUSSION

In a cohort of over 1,300 women across six countries in sub-Saharan Africa who initiated ART in pregnancy or at delivery with asymptomatic HIV and high CD4 counts, approximately one third experienced at least one episode of postpartum viremia and about one in five had confirmed viremia on two consecutive specimens during two years of follow-up. Using rigorously collected clinical and socioeconomic data and machine learning methods, we developed models to identify women most likely to experience viremia during this critical period of breastfeeding transmission risk and achieved moderate model discrimination (AUC). Our aim was to develop a simple tool that would enable efficient delivery of enhanced ART adherence interventions to those who stand to gain the most benefit.. However, to achieve high sensitivity (90%), these models would require targeting support to roughly two-thirds of all women, fewer than a third of whom who would go on to experience viremia. Still, in contrast to targeting support based on recent viremia alone, through which we cannot achieve 80–90% sensitivity, the multivariable setting allows for the selection of risk thresholds across a range of higher levels of sensitivity, despite declining specificity, reflected in the higher estimated AUCs in our machine learning models. These results suggest that modeling has the potential to improve risk classification with the incorporation of additional, perhaps non-routine, data.

Prior risk scores to predict virologic failure have been developed in the context of limited access to viral load assays as a potential strategy to inform selective monitoring.^25–30 Some of these models performed moderately better than ours; most were developed with logistic regression while one was developed with SuperLearner²⁹ Aside from being developed in non-pregnant/postpartum populations, three factors may explain differences in performance. First, prior studies included late presenters and persons with advanced disease or long histories of ART use. These characteristics, reflecting a broad spectrum of HIV disease, could be strong predictors of poor adherence and viremia. Our study sample included generally asymptomatic young women who recently initiated ART with high CD4 counts who were not yet eligible for treatment according to country guidelines at the time. Thus, our cohort had little variability in disease progression and we were unable to incorporate variables such as opportunistic infections or long duration of ART use. Second, most prior risk scores included changes in CD4 counts over time as predictors of viremia. Because CD4 counts are no longer routinely collected for treatment monitoring in sub-Saharan Africa, these values would not be available to calculate risk in these settings today. For this reason, we only included nadir CD4 in our models, though we acknowledge that not all settings collect even this baseline measure. Third, one study²⁹ achieved an AUC of 0.88 by incorporating electronically captured pill container openings as an objective adherence metric.

Several psychosocial factors associated with the postpartum period may be important drivers of adherence, yet many of these characteristics are not commonly assessed in routine care in resource-limited settings, nor were they systematically collected in the PROMISE study. Some notable examples include depression, which is a risk factor for suboptimal ART adherence⁴⁵ and is common postpartum;⁴⁶ the impact of pregnancy and breastfeeding on relationships and social support networks may influence adherence; stigma and attitudes about ART use, particularly in the context of the perinatal transmission risk, could affect adherence during this time. Additionally, staying with extended family for postpartum support could interrupt care engagement, prompt hiding pills from family members due to fear of stigma, or otherwise impact pill-taking routines.⁴⁷ Many of these more nuanced psychosocial factors are not routinely collected but may be critical predictors of adherence or viremia in postpartum women.

The PROMISE dataset has both strengths and limitations pertinent to this analysis. A major strength is the systematic and rigorous data collection of key predictors and viral loads. In many resource-limited settings, electronic medical records are still in development, and viral load data is frequently missing. This strength also carries two limitations: (1) although we prioritized models built with routine data, in many resource-limited settings even these variables are not always systematically collected due to resource constraints, and (2) the nature of clinical trial participation is inherently different from routine clinical care. Motivations to adhere may differ in clinical trial settings, where follow-up, tracing, and access to health care are generally improved beyond routine care. Nonetheless, we expect that the variables we considered would have a similar relationship with adherence both within and outside of clinical trial settings. Moreover, the high frequency of viremia that we observed is comparable to what has been observed in other cohorts of pregnant and postpartum women.^2,10–15 We also note that PROMISE was conducted prior to the availability of more modern regimens in sub-Saharan Africa, specifically integrase inhibitors. Dolutegravir, which is associated with higher rates of viral suppression compared to other medications,⁴⁸ may be more “forgiving” of suboptimal adherence. Whether dolutegravir is more forgiving in the context of breastfeeding transmission risk is not known, but prediction of plasma viral loads may be more challenging with increasingly rare viremic events.

CONCLUSIONS

In conclusion, a third of postpartum women on ART in sub-Saharan Africa in this study experienced viremia in the first two years following delivery, which carries implications for both perinatal transmission and lifelong maternal health. We used machine learning methods and routine clinical data to develop models to predict postpartum viremia in order to facilitate timely and targeted adherence support to postpartum women most at risk of viremia in sub-Saharan Africa. However, despite achieving moderate model discrimination, prediction performance was insufficient to correctly identify the majority of women at risk without also classifying an even larger number of low-risk women as high-risk. Nonetheless, our moderate model discrimination was substantially better than prediction with recent viremia alone, suggesting that this approach, with the incorporation of additional variables, could improve prediction. Factors such as depression, partner support, HIV-related stigma, or objective metrics of adherence, for example, may be needed to better inform targeted support. Given the high frequency of viremia during breastfeeding in this cohort and others, and the critical importance of virologic suppression for prevention of perinatal HIV transmission, there remains an urgent need to identify and evaluate approaches for early identification of suboptimal adherence.