Skip to main content
PLOS ONE logoLink to PLOS ONE
. 2020 Dec 29;15(12):e0244376. doi: 10.1371/journal.pone.0244376

Annual and durable HIV retention in care and viral suppression among patients of Peter Ho Clinic, 2013-2017

Debbie Y Mohammed 1,2,*, Lisa Marie Koumoulos 1,3, Eugene Martin 4, Jihad Slim 2,5
Editor: Ellen Wiewel6
PMCID: PMC7771864  PMID: 33373385

Abstract

Objectives

To determine rates of annual and durable retention in medical care and viral suppression among patients enrolled in the Peter Ho Clinic, from 2013–2017.

Methods

This is a retrospective review of medical record data in an urban clinic, located in Newark, New Jersey, a high prevalence area of persons living with HIV. Viral load data were electronically downloaded, in rolling 1-year intervals, in two-month increments, from January 1, 2013 to December 31, 2019. Three teams were established, and every two months, they were provided with an updated list of patients with virologic failure. Retention and viral suppression rates were first calculated for each calendar-year. After patients were determined to be retained/suppressed annually, the proportion of patients with durable retention and viral suppression were calculated in two, three, four, five and six-year periods. Descriptive statistics were used to summarize sample characteristics by retention in care, virologic failure and viral suppression with Pearson Chi-square; p-value <0.05 was statistically significant. Multiple logistic regression models identified patient characteristics associated with retention in medical care, virologic failure and suppression.

Results

As of December 31, 2017, 1000 (57%) patients were retained in medical care of whom 870 (87%) were suppressed. Between 2013 and 2016, decreases in annual (85% to 77%) and durable retention in care were noted: two-year (72% to 70%) and three-year (63% to 59%) periods. However, increases were noted for 2017, in annual (89%) and durable retention in the two-year period (79%). In the adjusted model, when compared to current patients, retention in care was less likely among patients reengaging in medical care (adjusted Odds Ratio (aOR): 0.77, 95% CI: 0.61–0.98) but more likely among those newly diagnosed from 2014–2017 (aOR: 1.57, 95% CI: 1.08–2.29), compared to those in care since 2013. A higher proportion of patients re-engaging in medical care had virologic failure than current patients (56% vs. 47%, p < 0.0001). As age decreased, virologic failure was more likely (p<0.0001). Between 2013 and 2017, increases in annual (74% to 87%) and durable viral suppression were noted: two-year (59% to 73%) and three-year (49% to 58%) periods. Viral suppression was more likely among patients retained in medical care up to 2017 versus those who were not (aOR: 5.52, 95% CI: 4.08–7.46). Those less likely to be suppressed were 20–29 vs. 60 years or older (aOR: 0.52, 95% CI: 0.28–0.97), had public vs. private insurance (aOR: 0.29, 95% CI: 0.15–0.55) and public vs. private housing (aOR: 0.59, 95% CI: 0.40–0.87).

Conclusions

Restructuring clinical services at this urban clinic was associated with improved viral suppression. However, concurrent interventions to ensure retention in medical care were not implemented. Both retention in care and viral suppression interventions should be implemented in tandem to achieve an end to the epidemic. Retention in care and viral suppression should be measured longitudinally, instead of cross-sectional yearly evaluations, to capture dynamic changes in these indicators.

Introduction

An estimated 1.04 million persons were living with HIV (PLWH) in the United States (U.S.) in 2018, with a prevalence of 374.6 per 100,000 population [1]. Males accounted for 75%, of whom 35% were Black, 73% were males who had sex with males (MSM), and injection drug use (IDU) was 9%. Among females, 58% were Black of whom 77% reported heterosexual contact or IDU (20%) as their transmission risk. In comparison, prevalence rates were higher in New Jersey and Essex County, 419.7 and 1,194.9 per 100,000 population, respectively [2]. In Essex County, characteristics of PLWH in this urban area reflect differences when compared to national data. Males accounted for a lower proportion of PLWH (61%), of whom a higher proportion were Black (69%), a lower proportion were MSM (35%) and a higher proportion reported IDU (16%) [2]. Among females, a higher proportion were Black (81%), of whom a lower proportion reported heterosexual contact (67%) [2]. The distribution of PLWH in Newark was similar to Essex County [3]. The Peter Ho Clinic (PHC) located in the City of Newark, is in the County of Essex.

Retention in medical care and viral suppression are beneficial to both the individual and wider community. For PLWH, mortality and opportunistic events decrease, even in patients with very advanced infection, with a concurrent increase in life expectancy [46]. Among PLWH with suppressed viral loads, ongoing transmission of infection decreases dramatically [7]. Despite these benefits, there are challenges to retention in medical care and viral suppression. In 2013, at the national level, 71% of PLWH were retained in care, of whom 77% were suppressed [8]. In comparison at the PHC, of 1,229 PLWH in medical care in 2013, 85% were retained for one year of whom 74% were suppressed.

Age, insurance, income, housing, and drug use were previously reported to impact retention in care and viral suppression. In 2013, the highest proportion of PLWH, retained in medical care were aged 45–54 (72.3%) versus (vs.) those 25–34 years old (69.8%) and viral suppression increased with older age (43.7% and 57.5%, 25–34 and 45–54 years old, respectively) [8]. In a survey conducted in a public health HIV care relinkage program, 124 (50%) PLWH reported that not having insurance was a barrier to retention in care [9]. In another study, among women who did not take part in the AIDS Drug Assistance Program, those with Medicaid or no insurance were more likely to experience virologic failure compared to those with private insurance [10]. In addition, poverty and lack of health insurance were noted to be predictors of mortality among Black, heterosexual women [11]. A systemic review concluded that a lack of stable, secure, adequate housing was a significant barrier to consistent medical care, access, and adherence to anti-retroviral therapy (ART), sustained viral suppression, and risk of forward transmission, among PLWH [12]. Among those reporting IDU, retention in medical care and viral suppression results were equivocal. Sustained high viral loads were more likely among with a reported risk of IDU compared to heterosexuals [13]. In another evaluation, disparities in retention and viral suppression among PLWH with and without a history of IDU were eliminated by 2012, compared to 2000 [14].

A national strategy to combat gaps in care was introduced in 2010 and updated in 2015 [15, 16]. The National HIV/AIDS Strategy (NHAS) goals include improving health outcomes among PLWH. Measurable indicators include increasing the proportion of PLWH retained in medical care to at least 90% and viral suppression to at least 80%, by 2020. However, these indicators are based on cross-sectional data from yearly or 24-month periods, according to guidance from Health Resources and Services Administration, HIV/AIDS Bureau [17].

There is a paucity of reports on durable retention and viral suppression. One study conducted in Atlanta, Georgia reported retention in care for a two and three-year period (60% and 49%, respectively), from 2010 to 2013 [18]. In a national study conducted from 2011–2013, durable viral suppression, in a two-year period, was reported to be 61.8% [19]. However, researchers posit that true durability may best be defined after at least three years of viral suppression [20]. Durable retention over a five-year period was reported in two studies previously. In five sites of the HIV Research Network (HIVRN), five-year retention was 39.3%, while a clinical site in Lexington, Kentucky, reported 61% [21, 22]. Durable viral suppression from a clinical practice in Atlanta, Georgia, was 44% and 36%, in the two-year and three-year periods, respectively [18]. In a study evaluating the impact of care coordination in New York City, from 2009–2016, durable viral suppression was 37% in months 13–36 of follow-up [23]. In the U.S. Military HIV Natural History Study, durable viral suppression at the five and ten-year periods were 85% and 54%, respectively [24, 25].

The results of this study, using cross-sectional and longitudinal data, will be used to assess progress of this clinic towards achieving the NHAS 2020 goals [15, 16]. In addition, factors that facilitate or serve as barriers to achieving these goals, will be identified to inform future interventions and provide valuable lessons to other clinics, serving urban populations with similar challenges.

Methods

Design

This is a retrospective review of medical records data evaluating annual and durable retention in medical care and viral suppression, among patients enrolled at PHC, from 2013–2017.

Setting

The PHC is located on the campus of Saint Michael’s Medical Center, an urban academic institution in Newark, New Jersey. Newark is the epicenter of the epidemic in New Jersey and part of the New York metropolitan area that includes New York City. This is the first clinic in the state to provide medical care for PLWH and serves approximately 1,200 persons yearly. Co-located services include HIV testing, access to pre-exposure prophylaxis and linkage to care coordinators. Clinical staff include infectious diseases providers and fellows, nurse practitioners, medical and non-medical case-managers, and a phlebotomist. Specialty co-located practices include gynecology, medication assisted therapy for opioid use and pain management services.

Study population

Patients included in this study were at least 18 years old and alive as of December 31st of the respective year. At least one time in 2013–2017, they saw a medical provider, received a prescription for ART, and had viral load results documented in the electronic medical record. They were included in the study at the time of their first medical visit or viral load in 2013–2017 to six months after the last documented viral load or medical visit. Six patients were excluded from the virologic failure analysis as their viral loads were less than 200 copies/mL at the time of diagnosis (Fig 1). The medical records for PLWH not retained in medical care were evaluated for documentation of moving to another facility or death; if there was no documentation, they were considered lost to care.

Fig 1. Flowchart of patients assessed for retention in care, virologic failure and viral suppression, Peter Ho Clinic, 2013–2017.

Fig 1

Viral suppression intervention

Beginning in 2014, clinical services were restructured for patients with viral loads of at least 200 copies/mL. Three teams were established, each composed of a clinician (either a nurse practitioner or doctor), a case manager, a nurse, and operational staff. Viral load data were electronically downloaded, in rolling 1-year intervals, in two-month increments, from January 1, 2014 to December 31, 2019. Every two months, each team was provided with an updated list of patients with virologic failure. Initially, staff met weekly to discuss challenges associated with complex patients, and as their confidence increased, meetings were decreased to every two weeks and then monthly intervals.

A nurse practitioner reviewed genotype results and the electronic medical record to ensure that patients were on appropriate ART. Case managers and nurses assessed barriers to adherence and medical care and implemented appropriate interventions. Patients with difficulty adhering to ART came in weekly for prefilled medication boxes and counseling. They were transitioned to prefilled pharmacy packages once they became suppressed. The registration staff were key in scheduling and following up with phone reminders for appointments. Substance use and mental health counseling and treatment services were provided on-site. Referrals were also made to off-site locations, as indicated; those with unstable housing were referred to the New Jersey HIV Housing Collaborative for assistance.

Outcomes

The primary outcomes for this study included the proportion of patients: a) retained in medical care annually, b) retained in durable medical care, c) with viral suppression annually, and d) durable viral suppression. Annual retention in medical care included documentation of at least two medical encounters, at least 90 days apart, at PHC in the respective calendar-year (Yes, No). A medical encounter included a medical visit with a provider with prescribing privileges or an HIV viral load test based on the core performance indicator, released in 2019 [17]. Additionally, patients received prescriptions for ART. Data were collected up to December 31st, 2019 to evaluate durable retention and viral suppression. Durable retention in care was defined as having at least two medical encounters and ART, in each calendar-year at least 90 days apart, over a two, three, four, five or six-year period. After patients were determined to be retained in care annually, the proportion of patients retained in care for each time-period was calculated [19]. Virologic failure was defined as a viral load of at least 200 copies/mL, at any time (Yes, No) [26]. Annual viral suppression was defined as all viral loads less than 200 copies/mL, in the respective calendar-year (Yes, No). Durable viral suppression was defined as all viral load values less than 200 copies/mL over a two, three, four, five or six-year period [19]. After the viral suppression status, for each patient, was determined by each year, the proportion who had durable viral suppression in each time-period was calculated. Secondary outcomes included: predictors of retention in medical care, virologic failure and viral suppression.

Covariates

Patient’s status at the time of the study was defined as Current, New and Reengaged. Current patients were those in care in 2013. In the respective calendar-year, new patients were newly diagnosed and those reengaged did not previously receive care at PHC. Demographic variables included age (18–29, 30–49, 50–59 and 60 years), sex at birth (male, female), race/ethnicity (non-Hispanic Black, Hispanic, Other [non-Hispanic White, Asian, American Indian/Alaskan Native, Native Hawaiian, and multiple races), transmission risk was based on the Centers for Disease Control and Prevention hierarchy (male-to-male sex [MSM], IDU, heterosexual) [27], insurance (Medicaid, Medicare, private, none [the medical care of these patients were paid by charity care, grant funding from the state or Ryan White HIV/AIDS Program]), housing (private, public), income based on the federal poverty limit (FPL) for one person in 2017, (>FPL, FPL) [28], history of drug use (heroin, cocaine, both, none) and mental illness. Mental illness (Yes/No) was a composite of patient self-report, routine psychiatric visits, and medications documented in the electronic medical record.

In the regression analyses, categories of the following covariates were collapsed where outcomes were similar: age (<60, 60 years), race (non-Hispanic Black or Hispanic, Other), insurance (Public [included Medicaid, Medicare, None], Private) and drug use (Yes, No).

Data analysis

This study was approved by the Saint Michael’s Medical Center Institutional Review Board. Data collection were completed as of December 31st, 2019. The Statistical Analysis System (SAS, version 9.4) Cary, North Carolina, was used for all analyses. After quality checks, the data were stripped of personal identifiers prior to analyses. Descriptive statistics were used to summarize sample characteristics by retention in medical care, virologic failure and viral suppression with Pearson’s Chi-square. A p-value <0.05 was considered statistically significant. Multiple logistic regression models, using backwards deletion, were developed to identify factors associated with retention in medical care in 2017; virologic failure, 2013–2017; and viral suppression, 2013–2017, for those who experienced virologic failure. The selected variables were based on clinical and epidemiological significance. Collinearity in the category for risk transmission was mitigated by adhering to the hierarchical categorization developed by the Centers for Disease Control and Prevention [27], and categorizing drug use as cocaine only, heroin only and those with a history of using both drugs. Model fit was evaluated using Hosmer and Lemeshow Goodness-of-Fit test.

Results

From 2013 to 2017, a total of 1,759 PLWH received medical care at the PHC (Table 1). At least 60% were 50 years or older, male (64%) or Black (69%). From 2013 to 2017, shifts in the proportion of some groups were noted.

Table 1. Characteristics of patients, Peter Ho Clinic, New Jersey (2013–2017).

Characteristic 2013 2014 2015 2016 2017 Total
N (%) N (%) N (%) N (%) N (%) N (%)
Age
18–29 40 (3) 55 (5) 72 (6) 82 (7) 75 (7) 140 (8)
30–39 103 (9) 116 (10) 130 (11) 127 (11) 112 (11) 215 (12)
40–49 224 (18) 230 (19) 228 (19) 204 (18) 181 (18) 319 (18)
50–59 469 (38) 447 (36) 428 (36) 408 (36) 347 (35) 617 (35)
≥60 393 (32) 372 (30) 344 (28) 319 (28) 285 (29) 468 (27)
Gender
Female 478 (39) 454 (37) 451 (38) 413 (36) 362 (36) 631 (36)
Male 751 (61) 766 (63) 751 (62) 727 (64) 638 (64) 1128 (64)
Race/ethnicity
Black Non-Hispanic 868 (70) 862 (71) 846 (70) 802 (70) 700 (70) 1222 (69)
Hispanic 278 (23) 279 (23) 279 (23) 267 (24) 231 (23) 400 (23)
Othera 83 (7) 79 (6) 77 (7) 71 (6) 69 (7) 137 (8)
Transmission Risk
Male-to-Male sex 218 (18) 251 (20) 267 (22) 266 (23) 232 (23) 413 (23)
Injection Drug Use 242 (20) 215 (18) 193 (16) 171 (15) 137 (14) 295 (17)
Heterosexual sex 769 (62) 754 (62) 742 (62) 703 (62) 631 (63) 1051 (60)
Insurance
Medicaid 639 (52) 633 (52) 610 (51) 571 (50) 486 (48) 897 (51)
Medicare 220 (18) 218 (18) 200 (17) 195 (17) 169 (17) 267 (15)
Private 136 (11) 150 (12) 169 (14) 168 (15) 148 (15) 224 (13)
Noneb 234 (19) 219 (18) 223 (18) 206 (18) 197 (20) 371 (21)
Housing
Private 995 (81) 1012 (83) 1011 (84) 965 (85) 848 (85) 1444 (82)
Public 234 (19) 208 (17) 191 (16) 175 (15) 152 (15) 315 (18)
Income
>FPLc 355 (29) 375 (31) 388 (32) 382 (33) 340 (34) 526 (30)
≤FPLc 874 (71) 845 (69) 814 (68) 758 (67) 660 (66) 1233 (70)
Drug Use
Heroin 70 (6) 60 (5) 52 (4) 51 (5) 32 (3) 91 (5)
Cocaine 154 (12) 153 (13) 142 (12) 137 (12) 116 (12) 221 (13)
Both 341 (28) 330 (27) 287 (24) 252 (22) 214 (21) 437 (25)
None 664 (54) 677 (55) 721 (60) 700 (61) 638 (64) 1011 (57)
Mental Illness
Yes 481 (39) 450 (37) 408 (34) 381 (34) 325 (33) 630 (36)
No 748 (61) 770 (63) 794 (66) 759 (66) 675 (67) 1129 (64)
Total 1229 1220 1202 1140 1000 1759

a Includes Non-Hispanic white, Asian, American Indian, Alaskan Native, Native Hawaiian, and multiple races

b Patient care was paid for by charity care, state funding, or Ryan White HIV/AIDS program

cFPL- Federal Poverty Level for 2017

By patients’ status, this included 1229 (70%) current patients, 157 (9%) newly diagnosed and 373 (21%) who reengaged in medical care (Table 2). Characteristics were similar for current patients and those reengaging in medical care.

Table 2. Characteristics of patients by status, Peter Ho Clinic, New Jersey (2013–2017).

Characteristics Reengaged New Current Total
N (%) N (%) N (%) 1759
373 (21) 157 (9) 1229 (70)
Age p<0.0001
< 60 309 (83) 145 (92) 837 (68) 1291 (73)
60 64 (17) 12 (8) 392 (32) 468 (26)
Gender p<0.0001
Female 126 (33) 29(18) 478 (39) 631 (36)
Male 247 (67) 128 (82) 751 (61) 1126 (64)
Race/ethnicity p<0.0001
Black or Hispanic 339 (91) 138 (87) 1147 (93) 1624 (92)
Othera 34(9) 19(13) 82 (7) 137 (8)
Transmission Risk p<0.0001
Male-to-Male sex 112 (30) 82 (52) 219 (18) 413 (23)
Injection Drug Use 49 (13) 4 (2) 242 (20) 295 (17)
Heterosexual sex 210 (57) 72 (46) 768 (62) 1051 (60)
Insurance p<0.0001
Publicb 320 (86) 122 (77) 1093 (89) 1535 (87)
Private 53 (14) 35(23) 136 (11) 224 (13)
Housing p<0.0001
Public 74 (20) 7(4) 234 (19) 315 (18)
Private 299 (80) 150 (96) 995 (81) 1444 (82)
Income p<0.0001
FPLc 266 (72) 92 (58) 875 (71) 1233 (70)
>FPLc 107 (18) 65 (42) 354 (29) 526 (30)
Drug Use p<0.0001
Yesd 156 (42) 27 (17) 565 (46) 748 (43)
No 217 (58) 130 (83) 664 (54) 1011 (57)
Mental Illness p<0.0001
Yes 128 (35) 21 (13) 481 (39) 630 (36)
No 245 (65) 136 (87) 748 (61) 1129(64)

aOther–includes Non-Hispanic White, Asian, American Indian/Alaskan Native, Native Hawaiian, multiple races

bPublic insurance include Medicare, Medicaid, and No insurance.

cFPL–Federal Poverty Limit for 2017

dDrug Use (Yes) included heroin and cocaine

However, when new patients were compared with those reengaging in care, higher proportions were < 60 years old (92% vs. 83%), male (82% vs. 67%), reported MSM (52% vs. 30%), had private insurance (23% vs. 14%), private housing (96% vs. 80%), or income > FPL (42% vs. 18%), respectively, p<0.0001. Higher proportions of patients reengaging in medical care were Black or Hispanic (91% vs. 87%), reported a history of drug use (42% vs. 17%) and mental illness (35% vs. 13%) compared to new patients, (p<0.0001).

Annual retention in medical care decreased from 2013 to 2016, 85% to 77%, respectively (Fig 2). Higher proportions of patients were retained in the two (82% vs. 72%), three (66% vs. 63%) and four-year (61% vs. 53%) periods for patients from 2015, compared to 2013. In 2017, the proportions for the annual and two-year period were 89% and 79%, respectively.

Fig 2. Retention in medical care and durable retention in medical care, by time-period.

Fig 2

The proportion of patients moving declined from 2013 to 2017, 9% to 2% respectively, however, those lost to care, was almost two times higher in 2016 compared to 2015 (Fig 3).

Fig 3. Patients moved, lost to care or died, by year, 2013–2017.

Fig 3

Overall, 1,000 patients (57%) were alive and retained in medical care as of December 31, 2017 (Table 3). Retention in medical care was higher for newly diagnosed patients (70%) compared to those in care since 2013 (51%) or reengaging in medical care (49%), p<0.0001 (Table 3 and Fig 4).

Table 3. Factors associated with retention in care, Peter Ho Clinic, 2013–2017.

Characteristic Retained in Care P value Odds Ratio 95% CIa Adjusted Odds Ratio 95% CIa
Yes No
N (%) N (%)
1000 (57) 759 (43)
Status <0.0001
New 110 (70) 47 (30) 1.71 (1.19–2.44) 1.57 (1.08–2.29)
Reengaged 185 (49) 188 (51) 0.74 (0.59–0.94) 0.77 (0.61–0.98)
Current 705 (57) 524 (43) 1.00 1.00
Age 0.0391
<60 715 (55) 576 (45) 0.80 (0.64–0.99) 0.71 (0.56–0.90)
≥60 285 (61) 183 (39) 1.00 1.00
Gender 0.7425
Female 362 (57) 269 (43) 1.03 (0.85–1.26) 1.00 (0.79–1.26)
Male 638 (56) 490 (43) 1.00 1.00
Race/ethnicity 0.1105
Black or Hispanic 931 (57) 691 (43) 1.33 (0.94–1.88) 1.34 (0.93–1.92)
Otherb 69 (50) 68 (50) 1.00 1.00
Transmission Risk 0.0002
Male-to-Male sex 232 (56) 181 (44) 0.85 (0.68–1.07) 0.77 (0.58–1.02)
Injection Drug Use 137 (46) 158 (54) 0.58 (0.45–0.75) 0.71 (0.53–0.95)
Heterosexual sex 631 (60) 420 (40) 1.00 1.00
Insurance 0.0029
Public 852 (56) 683 (44) 0.64 (0.48–0.86) 0.85 (0.61–1.19)
Private 148 (66) 76 (34) 1.00 1.00
Housing 0.0007
Private 848 (59) 596 (41) 1.00 1.00
Publicc 152 (48) 163 (52) 0.66 (0.51–0.84) 0.80 (0.62–1.03)
Income <0.0001
>FPLd 340 (64) 186 (36) 1.00 1.00
≤FPLd 660 (54) 573 (47) 0.63 (0.51–0.78) 0.75 (0.59–0.95)
Drug Use <0.0001
Yese 362 (48) 386 (52) 0.55 (0.45–0.66) 0.64 (0.51–0.80)
No 638 (63) 373 (37) 1.00 1.00
Mental Illness 0.0009
Yes 325 (52) 305 (48) 0.72 (0.59–0.87) 1.01 (0.81–1.26)
No 675 (60) 454 (40) 1.00 1.00

aCI–Confidence Interval

bOther–includes Non-Hispanic White, Asian, American Indian/Alaskan Native, Native Hawaiian, multiple races

c Public insurance include Medicare, Medicaid, and No insurance.

dFPL–Federal Poverty Limit for 2017

e Drug Use (Yes) included heroin and cocaine

Fig 4. Retention in medical care, virologic failure and suppression by status, Peter Ho Clinic, 2013–2017.

Fig 4

Note: Retention in medical care: Two medical encounters (medical visit or viral load), three months apart, in 2017, ART prescription and alive as of 12/31/2017. Virologic Failure: Most Recent Viral load > 200 copies/mL. Suppressed: Most Recent Viral load < 200 copies/mL. Status: Current patients were those in care in 2013. In the respective calendar-year, new patients were newly diagnosed and those reengaged did not previously receive care in PHC.

Factors associated with retention in medical care in the adjusted model included status, age, risk, income, and drug use. When compared to current patients, retention in care was less likely among patients reengaging in medical care (aOR: 0.77, 95% CI: 0.61–0.98) but more likely among those who were newly diagnosed from 2014–2017 (aOR:1.57, 95% CI: 1.08–2.29). Those less likely to be retained in medical care were younger than 60 vs. > 60 years old (aOR: 0.71, 95% CI: 0.56–0.90), reported IDU vs. heterosexual contact (aOR: 0.71, 95% CI: 0.53–0.95), had an income < FPL vs. > FPL for 2017 (aOR: 0.75, 95% CI: 0.59–0.95) and reported a history of drug use vs. none (aOR: 0.64, 95% CI: 0.51–0.80).

A viral load of at least 200 copies/mL was present at least one time for 935 (53%) patients, from 2013 to 2017 (Table 4). A higher proportion of patients reengaging in medical care had virologic failure than those in care since 2013, (56% vs. 46%, p < 0.0001) (Fig 4). In the adjusted model age, insurance, income, and drug use were associated with virologic failure.

Table 4. Factors associated with virologic failure, Peter Ho Clinic, 2013–2017.

Characteristic Virologic Failure p-value Odds Ratio 95% CIa Adjusted Odds Ratio 95% CIa
No Yes
N (%) N (%)
818 (47) 935 (53)
Retained in Care 0.8237
Yes 468 (47) 530 (53) 1.00 1.00
No 350 (47) 405 (53) 1.02 (0.85–1.24) 0.88 (0.72–1.08)
Age <0.0001
20–29 28 (20) 110 (80) 5.46 (3.47–8.60) 7.70 (4.70–12.64)
30–39 77 (36) 137 (64) 2.47 (1.77–3.45) 3.38 (2.32–4.92)
40–49 128 (40) 191 (60) 2.07 (1.55–2.77) 2.44 (1.79–3.32)
50–59 314 (51) 302 (49) 1.34 (1.05–1.70) 1.36 (1.05–1.75)
≥60 271 (58) 195 (42) 1.00 1.00
Gender 0.3435
Female 303 (48) 326 (52) 0.91 (0.75–1.11) 0.84 (0.67–1.06)
Male 515 (46) 609 (54) 1.00 1.00
Race/ethnicity 0.2072
Black or Hispanic 747 (46) 869 (54) 1.25 (0.88–1.77) 1.22 (0.84–1.76)
Otherb 71 (52) 66 (48) 1.00 1.00
Transmission Risk 0.0953
Male-to-Male sex 177 (43) 235 (57) 1.17 (0.93–1.47) 0.75 (0.56–1.01)
Injection Drug Use 150 (51) 143 (49) 0.84 (0.65–1.09) 0.84 (0.63–1.13)
Heterosexual sex 491 (47) 557 (53) 1.00 1.00
Insurance 0.0003
Publicc 689 (45) 841 (55) 1.68 (1.26–2.23) 1.62 (1.17–2.26)
Private 129 (58) 94 (42) 1.00 1.00
Housing 0.8996
Public 148 (47) 167 (53) 0.98 (0.77–1.27) 0.95 (0.73–1.23)
Private 670 (47) 768 (53) 1.00 1.00
Income 0.0005
≤FPLd 541 (44) 690 (56) 1.44 (1.17–1.77) 1.35 (1.06–1.71)
>FPLd 277 (53) 245 (47) 1.00 1.00
Drug Use 0.2210
Yese 335 (45) 410 (55) 1.13 (0.93–1.36) 1.38 (1.09–1.76)
No 483 (48) 525 (52) 1.00 1.00
Mental Illness 0.0716
Yes 275 (44) 353 (56) 1.20 (0.98–1.46) 1.15 (0.91–1.44)
No 543 (48) 582 (51) 1.00 1.00

aCI–Confidence Interval

bOther–includes Non-Hispanic White, Asian, American Indian/Alaskan Native, Native Hawaiian, multiple races

c Public insurance include Medicare, Medicaid, and No insurance.

dFPL–Federal Poverty Limit for 2017

e Drug Use (Yes) included heroin and cocaine

Virologic failure was more likely among those 20–29 (aOR 7.70: 95% CI: 4.70–12.64), 30–39 (aOR: 3.38: 95% CI: 2.32–4.92), 40–49 (aOR: 2.44: 95% CI: 1.79–3.32), and 50–59 (aOR:1.36, 95% CI: 1.05–1.75) vs. >60 years old; with public vs. private insurance (aOR: 1.62, 95% CI: 1.17–2.26), income at < FPL vs. > FPL for 2017 (aOR:1.35, 95% CI: 1.06–1.71) and a history of drug use vs. none (aOR:1.38, 95% CI: 1.09–1.76).

Annual viral suppression increased over time: 2013 (74%), 2014 (77%), 2015 (81%), 2016 (86%) and 2017 (87%) (Fig 5). Higher proportions of patients in care since 2013 were virally suppressed, compared to patients reengaging in care (64% vs. 54%, respectively, p<0.0001). Durable viral suppression increased among patients in the two (59% to 73%) and three-year (49% to 58%) periods, from 2013 to 2017. Similar proportions were noted for the four to six-year periods.

Fig 5. Viral suppression and durable viral suppression by time period, 2013–2017.

Fig 5

Among patients with virologic failure, from 2013–2017, 559 (59%) were suppressed at the last reported result (Table 5). They included new patients (76%), those who were retained in care in 2017 (77%) or were at least 60 years old (66%). In the adjusted model, status, retention in medical care in 2017, age, insurance and housing were associated with viral suppression.

Table 5. Factors associated with viral suppression, among patients with virologic failure, Peter Ho Clinic, 2013–2017.

Characteristic Suppressed p-value Odds Ratio 95% CI a Adjusted Odds Ratio 95% CI a
Yes No
N (%) N (%)
559 (59) 376 (41)
Status <0.0001
New 115 (76) 36 (24) 2.15 (1.43–3.22) 1.86 (1.10–3.14)
Reengaged 104 (50) 104(50) 0.69 (0.50–0.95) 0.77 (0.53–1.11)
Current 340 (59) 236 (41) 1.00 1.00
Retention in Medical Care <0.0001
Yes 408 (77) 122 (23) 5.63 (4.23–7.48) 5.32(3.93–7.20)
No 151 (37) 254 (63) 1.00 1.00
Age 0.3563
20–29 63 (57) 47 (43) 0.69(0.42–1.11) 0.41(0.21–0.81)
30–39 82 (60) 55 (40) 0.76 (0.49–1.20) 0.59 (0.33–1.04)
40–49 110 (58) 81 (42) 0.70 (0.46–1.05) 0.73 (0.45–1.19)
50–59 175(58) 127 (42) 0.71 (0.49–1.03) 0.89 (0.58–1.36)
≥60 129(66) 66(34) 1.00 1.00
Gender 0.0053
Female 175 (54) 151(46) 0.68(0.52–0.89) 0.75(0.53–1.06)
Male 384 (63) 225(37) 1.00 1.00
Race/ethnicity 0.2370
Black or Hispanic 515 (59) 354 (41) 0.73 (0.43–1.24) 0.89(0.48–1.63)
Otherb 44(67) 22(33) 1.00 1.00
Transmission Risk 0.3511
Male-to-Male sex 149 (63) 86 (37) 1.26(0.92–1.72) 1.13(0.73–1.76)
Injection Drug Use 87 (61) 56 (39) 1.13(0.77–1.64) 1.52(0.96–2.41)
Heterosexual sex 323 (58) 234 (42) 1.00 1.00
Insurance <0.0001
Publicc 479 (57) 362 (43) 0.23(0.13–0.42) 0.29(0.15–0.57)
Private 80 (85) 14 (15) 1.00 1.00
Housing <0.0001
Public 76 (45) 91 (55) 0.48(0.35–0.69) 0.62(0.42–0.92)
Private 483 (63) 285 (37) 1.00 1.00
Income <0.0001
≤FPLd 382 (55) 308 (45) 0.48(0.35–0.65) 0.77(0.53–1.14)
>FPLd 177 (72) 68 (28) 1.00 1.00
Drug Use 0.0007
Yese 220 (54) 190 (46) 0.64(0.49–0.83) 0.90(0.62–1.30)
No 339 (65) 186 (35) 1.00 1.00
Mental Illness 0.0024
Yes 189 (53) 164 (47) 0.66(0.51–0.86) 0.94(0.66–1.32)
No 370 (64) 212 (36) 1.00 1.00

aCI–Confidence Interval

bOther–includes Non-Hispanic White, Asian, American Indian/Alaskan Native, Native Hawaiian, multiple races

c Public insurance include Medicare, Medicaid, and No insurance.

dFPL–Federal Poverty Limit for 2017

e Drug Use (Yes) included heroin and cocaine

Viral suppression was more likely among new patients vs. current patients (aOR:1.86, 95% CI:1.10–3.14) and patients retained in medical care in 2017 vs. those who were not (aOR: 5.32: 95% CI: 3.93–7.20). Viral suppression was less likely in patients 20–29 vs. >60 years old (aOR: 0.41, 95% CI:0.21–0.81), with public vs. private insurance (aOR: 0.29, 95% CI: 0.15–0.57), or who lived in public vs. private housing (aOR: 0.62, 95% CI: 0.42–0.92).

Discussion

Annual viral suppression at PHC increased by 18% from 2013 to 2017. This increase coincides with restructuring of clinic services for patients with a viral load of at least 200 copies/mL. Despite this attempt to improve viral suppression, a concurrent intervention to improve retention in care was not implemented and annual retention declined by 10% from 2013–2016.

In this study, virologic failure did not vary by retention status. However, patients who were retained in care until 2017 were at least 5 times more likely to be suppressed, at the last reported result than those who were lost to care, moved, or died. This relationship between retention and viral suppression was previously highlighted. Among patients in a Minnesota clinic from 2003–2015, retention in care for every calendar-year was associated with sustained viral suppression [29]. Continuous retention in Atlanta, Georgia was associated with viral suppression at the end of 36 months (adjusted prevalence ratio 3.12; 95% CI, 2.40, 4.07) [18]. A study in Lexington, Kentucky from 2003–2011, followed patients an average of 6.2 years, and reported that individuals optimally retained in continuous care were almost 3 times more likely to maintain a suppressed viral load compared to those not retained (OR: 2.97; 95% CI: 1.65–5.32) [22]. Additionally, among PLWH who received medical care funded by the Ryan White HIV/AIDS Program in 2011, viral suppression was higher among retained clients (77.7%) vs. clients who were not retained (58.3%) [30].

In this study, we used the recently published definition of annual retention in medical care of two medical encounters (visit with a medical provider or viral load results), 90 days apart, as it reflects a practical approach and is supported in previous studies [17]. The HIVRN reported that 10% of patients from eleven HIV care sites who did not meet the definition for retention, completing 2 or more HIV medical visits separated by ≥90 days apart in a calendar year, remained virally suppressed [21]. In another study, the HIVRN combined data from patients who were eligible for Medicaid and pharmacy utilization in four clinical sites [31]. The researchers noted that patients who continued to receive ART during gaps in care had a suppressed viral load closest to the gap. These findings provide support for less frequent medical visits/viral loads among patients with viral suppression compared to those with virologic failure.

In this study we did cross-sectional as well as longitudinal evaluations of retention in medical care and viral suppression. Recent studies highlight the importance of measuring retention and viral suppression in a longitudinal manner as this better reflects the true status of care over time [29, 3234]. Dynamic viral load trajectories are easily overlooked with cross-sectional evaluation of the last value while longitudinal measures provide more granular data [19]. This longitudinal evaluation may provide the information needed in planning future interventions at PHC, as well as other urban clinics.

Age was associated with retention and viral suppression in this evaluation and supported in previous reports. One national study reported a lower proportion of those 25–34 vs. > 55 years-old (52% and 72%, respectively) with durable (two-year) viral suppression [19]. A cross-sectional analysis, among fourteen cohorts, in the U.S. and Canada reported that (1) the older the individual, the greater the probability of viral suppression in both the retained and not retained in care groups; (2) patients who were retained in care had a greater probability of viral suppression than those not retained in care and (3) the association between retention in care and viral suppression was greatest for younger versus older age groups [35]. Another study in the Southeastern U.S., compared those 18–24 and 35–44 years old and reported that younger patients were more likely to not be retained in medical care or have viral suppression [36].

In this study, patients who lived in public housing (including those with unstable housing) compared to private housing were 40% less likely to achieve viral suppression. This is supported in a recent national study among patients accessing care in Ryan White funded sites in 2018; those who lived in unstable compared to stable housing, had lower rates of viral suppression [37]. Previous reports indicate that public housing, when stable, mitigated barriers to viral suppression. A recent retrospective matched cohort study in New York City reported that low income PLWH who received housing services for more than one year, were more likely to be engaged in care and virally suppressed than similar populations who did not receive these services [38]. In another study, PLWH in continuous, stable housing in 2015, were reported to have continuous, stable viral suppression [39]. Conversely, time spent in emergency housing was a predictor of a lack of viral suppression. Identifying patients in unstable housing and transitioning them to stable housing will improve viral suppression rates, even if they live in public housing.

Insurance status was another structural barrier identified in this study. New Jersey expanded Medicaid and increased access to care for low-income residents, by implementing many reforms recommended by the Affordable Care Act starting in 2014 [40]. Despite the availability of public insurance, before being seen in PHC, there were many processes patients had to navigate to make this a reality. These included qualifying for hospital charity care if they did not have insurance [41], enrollment and reenrollment and requiring a primary care provider to issue referrals if they had a managed care plan. This may have contributed to patients not being retained in medical care or achieving viral suppression. Patients who are lost to care, more symptomatic, and/or racial/ethnic minorities may be less likely to overcome these barriers [42]. At least 70% of PLWH at PHC, qualified for federally funded services, based on an income less than the FPL for one person, in 2017, or had either Medicaid or no insurance (Charity Care/Ryan White funding), and were primarily racial and ethnic minorities (>90%). Numerous studies have documented the importance of Ryan White federal funding in maintaining PLWH in care and improved outcomes [30, 4347]. Urban clinics will need to maximize the use of this resource to ensure that patients overcome structural barriers and achieve the goals of the NHAS.

Strengths and limitations

A strength of this study is that we examined retention in care and viral suppression independent of each other. The conditional cascade methodology requires success at upstream stages before measuring success at later stages and underreports performance by up to 20% compared with evaluating each stage separately [17, 48]. A second strength of this study is that we evaluated longitudinal measures of retention in care and viral suppression. This provides a broader, more comprehensive perspective of retention and viral suppression than cross-sectional evaluations alone [29, 3334].

There are limitations to this study. This was a retrospective, observational study, so that we can only discuss associations and not causation. Secondly, this study may lack generalizability to other clinics as the results are applicable to a single, HIV specialty clinic in the Northeastern U.S. Thirdly, there may be other confounders that contributed to the reported associations. Fourthly, there were frequent changes in insurance type after the introduction of the Affordable Care Act that were not accounted for in this study, these likely impacted retention in care and viral suppression [40, 42]. Lastly, data from PHC were not matched to statewide or national HIV surveillance or vital statistics databases. We may have overestimated PLWH who were lost to care and underestimated those who were deceased.

In conclusion, we evaluated annual and durable retention in medical care and viral suppression using cross-sectional as well as longitudinal methods. The PHC is on-track towards achieving the NHAS goals by 2020 [16]. Concurrent interventions for retention in care and viral suppression should be implemented, with a focus on those reengaging in care. As we embark on ending the epidemic in the U.S., it will be useful to evaluate the NHAS in a longitudinal manner [17, 49]. Patients reengaging in medical care, younger than 60 years old, with public insurance or public housing may benefit from access to intensive care coordination to improve durable retention and viral suppression.

Acknowledgments

Patients

Thank you to all the patients enrolled at Peter Ho Clinic, from 2013 to 2019, without whom this study would not be possible.

Staff

Thank you to the staff for outstanding teamwork, and enthusiastic participation in improving viral suppression for this group of complex urban patients.

Data Availability

All relevant data are within the manuscript

Funding Statement

The authors received no specific funding for this work.

References

  • 1.Centers for Disease Control and Prevention. HIV Surveillance Report, 2018 (Updated); vol. 31. Available from: http://www.cdc.gov/hiv/library/reports/hiv-surveillance.html
  • 2.New Jersey Department of Health [Internet]. Newark (NJ): County and Municipal HIV/AIDS Statistics, 2018; [cited 2020 Aug 5]. Available from: https://www.state.nj.us/health/hivstdtb/hiv-aids/statmap.shtml
  • 3.New Jersey Department of Health [Internet]. Newark (NJ): Top ten cities with the highest numbers of cases; [cited 2020 Aug 5]. Available from: https://www.state.nj.us/health/hivstdtb/hiv-aids/cities.shtml
  • 4.Murphy EL, Collier AC, Kalish LA, Assmann SF, Para MF, Flanigan TP, et al. ; Viral Activation Transfusion Study Investigators. Highly active antiretroviral therapy decreases mortality and morbidity in patients with advanced HIV disease. Ann Intern Med. 2001; 135:17–26. 10.7326/0003-4819-135-1-200107030-00005 [DOI] [PubMed] [Google Scholar]
  • 5.Lesko CR, Cole SR, Hall HI, Westreich D, Miller WC, Eron JJ, et al. ; CNICS Investigators. The effect of antiretroviral therapy on all-cause mortality, generalized to persons diagnosed with HIV in the USA, 2009–11. Int J Epidemiol. 2016. February;45(1):140–50. 10.1093/ije/dyv352 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Siddiqi AE, Hall HI, Hu X, Song R. Population-based estimates of life expectancy after HIV diagnosis: United States 2008–2011. J Acquir Immune Defic Syndr. 2016; 72:230–236. 10.1097/QAI.0000000000000960 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Cohen MS, Chen YQ, McCauley M, Gamble T, Hosseinipour MC, Kumarasamy N, et al. ; HPTN 052 Study Team. Antiretroviral therapy for the prevention of HIV-1 transmission. N Engl J Med. 2016; 375(9):830–9. 10.1056/NEJMoa1600693 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Centers for Disease Control and Prevention. Monitoring selected national HIV prevention and care objectives by using HIV surveillance data—United States and 6 dependent areas, 2014. HIV Surveillance Supplemental Report 2016;21(4). Available from: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-supplemental-report-vol-21-4.pdf
  • 9.Dombrowski JC, Simoni JM, Katz DA, Golden MR. Barriers to HIV care and treatment among participants in a public health HIV care relinkage program. AIDS patient care STDS, 2015; 29(5), 279–287. 10.1089/apc.2014.0346 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Ludema C, Cole SR, Eron JJ, Edmonds A, Holmes GM, Anastos K, et al. Impact of health insurance, ADAP, and income on HIV viral suppression among US women in the women's interagency HIV study, 2006–2009. J Acquir Immune Defic Syndr. (1999), 2016;73(3), 307–312. 10.1097/QAI.0000000000001078 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Watson L, Gant Z, Hu X, Johnson AS. Exploring social determinants of health as predictors of mortality during 2012–2016, among black women with diagnosed HIV infection attributed to heterosexual contact, United States. Racial Ethn Health Disparities. 2019;6(5):892–899. 10.1007/s40615-019-00589-6 . [DOI] [PubMed] [Google Scholar]
  • 12.Aidala AA, Wilson MG, Shubert V, Gogolishvili D, Globerman J, Rueda S, et al. Housing status, medical care, and health outcomes among people living with HIV/AIDS: a systematic review. Am J Public Health, 2016; 106 (1) e1-e23. 10.2105/AJPH.2015.302905 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Terzian AS, Bodach SD, Wiewel EW, Sepkowitz K, Bernard MA, Braunstein SL, et al. Novel use of surveillance data to detect HIV-infected persons with sustained high viral load and durable virologic suppression in New York City. PloS one. 2012; 7(1), e29679 10.1371/journal.pone.0029679 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Lesko CR, Tong W, Moore RD, Lau B. Retention, antiretroviral therapy use and viral suppression by history of injection drug use among HIV-infected patients in an urban HIV clinical cohort. AIDS Behav. 2017. April; 21(4):1016–1024. 10.1007/s10461-016-1585-5 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.The White House Office of National AIDS Policy (2010) National HIV/AIDS Strategy for the United States. Available from: https://obamawhitehouse.archives.gov/sites/default/files/uploads/NHAS.pdf
  • 16.The White House Office of National AIDS Policy. National HIV/AIDS strategy: updated to 2020. Available from: https://www.hiv.gov/federal-response/national-hiv-aids-strategy/nhas-update
  • 17.Health Resources and Services Administration: HIV/AIDS Bureau. Performance measure portfolio, core measures, 2019. Available from: https://hab.hrsa.gov/sites/default/files/hab/clinical-quality-management/coremeasures.pdf
  • 18.Colasanti J, Kelly J, Pennisi E, Hu YJ, Root C, Hughes D, et al. Continuous retention and viral suppression provide further insights into the HIV care continuum compared to the cross-sectional HIV care cascade. Clin Infect Dis. 2016; 62(5):648–54. 10.1093/cid/civ941 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Crepaz N, Tang T, Marks G, Mugavero MJ, Espinoza L, Hall HI. Durable viral suppression and transmission risk potential among persons with diagnosed HIV infection: United States, 2012–2013. Clin Infect Dis. 2016;63(7):976–983. 10.1093/cid/ciw418 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Benator DA, Elmi A, Rodriguez MD, Gale HB, Kan VL, Hoffman HJ, et al. True durability: HIV virologic suppression in an urban clinic and implications for timing of intensive adherence efforts and viral load monitoring. AIDS Behav. 2015, 19(4):594–600. 10.1007/s10461-014-0917-6 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Yehia BR, Stephens-Shields AJ, Fleishman JA, Berry SA, Agwu AL, Metlay JP, et al. ; HIV Research Network. The HIV care continuum: changes over time in retention in care and viral suppression. PloS one. 2015; 10(6), e0129376 10.1371/journal.pone.0129376 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Crawford TN, Thornton A. Retention in continuous care and sustained viral suppression. J Int Assoc Provid AIDS Care. 2017; 16(1):42–47. 10.1177/2325957416678929 . [DOI] [PubMed] [Google Scholar]
  • 23.Robertson MM, Penrose K, Irvine MK, Robbins RS, Kulkarni S, Braunstein SL, et al. Impact of an HIV care coordination program on durable viral suppression. J Acquir Immune Defic Syndr. 2019;80(1):46–55. 10.1097/QAI.0000000000001877 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Marconi VC, Grandits GA, Weintrob AC, Chun H, Landrum ML, Ganesan A, et al. Outcomes of highly active antiretroviral therapy in the context of universal access to healthcare: the U.S. Military HIV Natural History Study. AIDS Res Ther. 2010; 7:14 10.1186/1742-6405-7-14 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Bello KJ, Mesner O, O'Bryan TA, Won SH, Lalani T, Ganesan A, et al. Factors associated with 10 years of continuous viral load suppression on HAART. BMC Infect Dis. 2016; 16:351 10.1186/s12879-016-1677-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.United States Department of Health and Human Services. Guidelines for the use of antiretroviral therapy in adults and adolescents; Laboratory Testing, 2019. Available from: https://aidsinfo.nih.gov/guidelines/html/1/adult-and-adolescent-arv/458/plasma-hiv-1-rna—viral-load—and-cd4-count-monitoring
  • 27.Lee L, McKenna MT, Janssen RS. Classification of transmission risk in the National HIV/AIDS Surveillance System. Public Health Reports 2003; 118:400–407. 10.1093/phr/118.5.400 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.United States Department of Health and Human Services. 2017 Poverty Guidelines. Available from: https://aspe.hhs.gov/2017-poverty-guidelines
  • 29.Enns EA, Reilly CS, Horvath KJ, Baker-James K, Henry K. HIV care trajectories as a novel longitudinal assessment of retention in are. AIDS Behav. 2019;23(9):2532–2541. 10.1007/s10461-019-02450-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Doshi RK, Milberg J, Isenberg D, Matthews T, Malitz M, Matosky M, et al. High rates of retention and viral suppression in United States HIV safety net system: HIV care continuum in the Ryan White HIV/AIDS Program, 2011. Clin Infect Dis. 2014; 60:117–25. https://doi.org10.1093/cid/ciu722 . [DOI] [PubMed] [Google Scholar]
  • 31.Monroe AK, Fleishman JA, Voss CC, Keruly JC, Nijhawan AE, Agwu AL, et al. HIV Research Network. Assessing antiretroviral use during gaps in HIV primary care using multisite Medicaid claims and clinical data. J Acquir Immune Defic Syndr. (1999), 2017; 76(1), 82–89. 10.1097/QAI.0000000000001469 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Marks G, Patel U, Stirratt MJ, Mugavero MJ, Mathews WC, Giordano TP, et al. Single viral load measurements overestimate stable viral suppression among HIV patients in care: clinical and public health implications. J Acquir Immune Defic Syndr. 2016;73(2):205–212. 10.1097/QAI.0000000000001036 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Crepaz N, Tang T, Marks G, Hall HI. Changes in viral suppression status among US HIV-infected patients receiving care. AIDS. 2017;31(17):2421–2425. 10.1097/QAD.0000000000001660 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Powers KA, Samoff E, Weaver MA, Sampson LA, Miller WC, Leone PA, et al. Longitudinal HIV care trajectories in North Carolina. J Acquir Immune Defic Syndr. 2017. February;74 Suppl 2(Suppl 2): S88–S95. 10.1097/QAI.0000000000001234 ; PMCID: PMC5289300. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Yehia BR, Rebeiro P, Althoff KN, Agwu AL, Horberg MA, Samji H et al. North American AIDS Cohort Collaboration on Research and Design (NA-ACCORD). Impact of age on retention in care and viral suppression. J Acquir Immune Defic Syndr. 2015;68(4):413–419. 10.1097/QAI.0000000000000489 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Ghiam MK, Rebeiro PF, Turner M, Rogers WB, Bebawy SS, Raffanti SP, et al. Trends in HIV Continuum of Care Outcomes over Ten Years of Follow-Up at a Large HIV Primary Medical Home in the Southeastern United States. AIDS Res Hum Retroviruses. 2017;33(10):1027–1034. 10.1089/AID.2017.0016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Cohen S. HRSA’s Ryan White HIV/AIDS Program: A Critical Component to Ending the HIV Epidemic. In: American Public Health Association Annual Meeting, 2020, Oct 24–28; Virtual. [cited 2020 Nov 1] Available from: https://apha.confex.com/apha/2020/meetingapp.cgi/Session/61686
  • 38.Zhong Y, Beattie CM, Rojas J, Farquhar XP, Brown PA, Wiewel EW. Enrollment length, service category, and HIV health outcomes among low-income HIV-positive persons newly enrolled in a housing program, New York City, 2014–2017. American Journal of Public Health. 2020;110 (7): 1068–1075. 10.2105/AJPH.2020.305660 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Beattie CM, Wiewel EW, Zhong Y, Brown PA, Braunstein SL, Pamela Farquhar X, et al. Multilevel Factors Associated with a Lack of Viral Suppression Among Persons Living with HIV in a Federally Funded Housing Program. AIDS Behav. 2019. March;23(3):784–791. 10.1007/s10461-019-02399-7 . [DOI] [PubMed] [Google Scholar]
  • 40.United States Department of Health and Human Services (2010). Compilation of patient protection and affordable care act. Available from: http://www.hhs.gov/healthcare/about-the-law/read-the-law/index.html Accessed on August 9, 2020
  • 41.New Jersey Department of Health (2020) Charity Care—New Jersey Hospital Care Payment Assistance Program. Available from: https://www.nj.gov/health/charitycare/documents/NJ%20Charity%20Care%20Fact%20Sheet_Feb2018%20(English).pdf
  • 42.Eaton EF, Mugavero MJ. Editorial commentary: Affordable Care Act, Medicaid expansion … or not: Ryan White Care Act remains essential for access and equity. Clin Infect Dis. 2016; 63(3): 404–406. 10.1093/cid/ciw280 [DOI] [PubMed] [Google Scholar]
  • 43.Ojikutu B, Holman J, Kunches L, Landers S, Perlmutter D, Ward M, et al. Interdisciplinary HIV care in a changing healthcare environment in the USA. AIDS Care. 2014;26(6):731–735. 10.1080/09540121.2013.855299 [DOI] [PubMed] [Google Scholar]
  • 44.Bradley H, Viall AH, Wortley PM, Dempsey A, Hauck H, Skarbinski J. Ryan White HIV/AIDS program assistance and HIV treatment outcomes. Clin Infect Dis. 2016;62(1):90–98. 10.1093/cid/civ708 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Mandsager P, Marier A, Cohen S, Fanning M, Hauck H, Cheever LW. Reducing HIV-related health disparities in the Health Resources and Services Administration's Ryan White HIV/AIDS Program. Am J Public Health. 2018;108(S4): S246–S250. 10.2105/AJPH.2018.304689 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.López JD, Shacham E, Brown T. The impact of the Ryan White HIV/AIDS Medical Case Management Program on HIV Clinical Outcomes: a longitudinal study. AIDS Behav 2018; 22, 3091–3099. 10.1007/s10461-018-2124-3 [DOI] [PubMed] [Google Scholar]
  • 47.Kay ES, Westfall AO. Ryan White HIV/AIDS program recipients more likely than non-recipients to be retained in care using six different retention measures. AIDS Care. 2020;32(1):89–92. 10.1080/09540121.2019.1623375 [DOI] [PubMed] [Google Scholar]
  • 48.Horberg MA, Hurley LB, Klein DB, Towner WJ, Kadlecik P, Antoniskis D, et al. The HIV care cascade measured over time and by age, sex, and race in a large national integrated care system. AIDS Patient Care and STDS. 2015; 29(11):582–90. 10.1089/apc.2015.0139 [DOI] [PubMed] [Google Scholar]
  • 49.Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV epidemic: A plan for the United States. JAMA. 2019;321(9):844–845. 10.1001/jama.2019.1343w [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Joseph Donlan

22 Jul 2020

PONE-D-20-02167

Retrospective review of annual and durable retention in care and viral suppression, Peter Ho Clinic, 2013-2017

PLOS ONE

Dear Dr. Mohammed,

Thank you for submitting your manuscript to PLOS ONE. Please accept my personal apologies for the time it has taken to get a decision to you.After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands.

Your manuscript has been evaluated by two external reviewers who have requested some clarifications about some aspects of your methodology, further discussion of key findings and further contextualisation with respect to existing literature (see the full reports below).

Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 05 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Dr Joseph Donlan

Associate Editor

PLOS ONE

Journal Requirements:

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1:Yes

Reviewer #2:Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1:Yes

Reviewer #2:Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1:Yes

Reviewer #2:Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1:Yes

Reviewer #2:Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1:This is a well written and well articulated research work on an important topic in HIV management. However, the authors are invited to consider some clarification to further improve the understanding of the work.

Title: The authors could consider a review of the title to "Retention in care and viral suppression among HIV patients in Peter Ho Clinic"

Abstract: Line 36-37 What are the comparison group?

Method section:

The authors need to provide information on the study setting to help the readers understand the context of the clinic and its operational environment. There is need for more details about the patients management in the hospital and the total number of patients receiving HIV care in the hospital.

line 89-91 : How long had the newly diagnosed patients been on treatment? Were they excluded because they did not experience virologic failure or because they were not eligible for the test as at the the time the analysis were conducted? Could the authors provide more information about how many patients were excluded from the analysis and for what reasons in a flow diagram?

Line 98: Remove "The definition of"

line 98-100: In defining the annual and durable retention, what was the starting point? The start of study or the point the patient commenced on ART? There is need to clarify on this further for ease of reproducibility. It is also important because retention rate vary with the duration on care from commencement of ART. This is same for viral suppression.

line 109: How about 18 and 19 years of age?

line 121: How did you handle those that did not have insurance "none"? The constituted 21% of the total respondents.

line 129-132: Could the authors provide more details about the model development? How did they handle multiple collinearity? How were factors selected for the final model?

Results section:

Line 134: Were these all PLWH? Where they new enrollment within the period, all previously enrolled prior to this period or both new and old enrollments? This could have been explained in the method section for more clarity.

Table 2: Please check the figures, the total number of new in Table 1 is 157 and is 158 in Table 2. Also, the current was 1229 in Table 1 and 1230 in Table 2. Were there additions? If anything the numbers should be less if some died within the period. Kindly cross check the figures.

How was those without insurance handled. One would expect a footnote here explaining the merging of the cells in table 1.

line 160: "patient" status

Table 4:

The authors should consider rephrasing the title of table 4. I understand that these were the initial 935 patients that ever reported virologic failure whose latest results showed some changes probably after an adherence intervention and a follow up test. It should read factors affecting viral suppression among those that reported an earlier virologic failure at PHC 2013-2017

Discussion:

The authors should resist the urge of repeating the results in the discussion section. They should rather discuss the implication of their finding to practice and situated it in the body of knowledge. They can also provide unique context to help the readers understanding the findings and why they may defer from other well known findings.

Line 221:

The authors could do more search on longitudinal retention in care among PLWH in literature, what is described as durable retention here. Here some literature to consider

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5289300/

https://link.springer.com/article/10.1007/s10461-019-02450-7?shared-article-renderer

https://academic.oup.com/cid/article/62/5/648/2462795

Line 271: Was this a cohort study?

Reviewer #2:This paper reports a 14-percentage-point increase in HIV viral suppression over five years in the HIV-focused Peter Ho Clinic in urban New Jersey, without consistent increases to retention in care. These findings are inspiring, but the paper should frame them more compellingly, more clearly describe analytic decisions, and highlight and analyze their suppression initiative more.

Abstract

Please see comment below about conveying your unique angle and relevance throughout the paper, including in the abstract.

Introduction

Lines 52-54: Please make the statistics about viral suppression comparable in both sentences, i.e., both among (or not among) persons retained in care.

Why this set of years and why this clinic? What unique questions of broad relevance were you able to ask and answer? Share things that will help the reader see how this analysis informs not only where this leaves PHC patients and policies but also other populations and health care systems. For example, PHC is the first and largest HIV clinic in NJ, it is set within a large medical center that also provides social services, it accepts public insurance and is supported by federal Ryan White funding and many patients are low-income, NJ and Newark are racially and ethnically diverse and have a relatively high prevalence of HIV, etc. The particular angle and relevance of your paper should shine not only in the introduction but also the abstract and discussion.

Methods

Line 110: “Gender (male, female)” needs to be changed. Please specify if this is gender identity or sex assigned at birth. If the former, the terms “man” and “woman” are more appropriate, and it would also be helpful to state whether and how non-binary and transgender persons were ascertained and classified.

Your analyses used multiple outcomes, denominators, inclusion criteria, and time periods. This section needs to more clearly convey what exactly the analyses were and who was in each one. Places to insert expansions on this may be around lines 88-89 (“Patients included in this study were at least 18 years old and alive as of December 31st of the respective year.”); and lines 82-83 (“This study evaluated annual retention in medical care and viral suppression, durable retention in medical care and viral suppression among PLWH, in the PHC, from 2013 to 2017.”).

Lines 95-97 (“Annual retention in medical care included documentation of at least two medical

96 encounters at PHC in the respective year (Yes, No). A medical encounter was defined as a

97 medical visit with a prescribing provider or an HIV viral load test…”): Guidelines have been shifting to permit once-yearly testing for persons with durable suppression. Especially since you also have a durable retention measure, did you consider looking at retention as being at least once in the year? Or did clinic policy, prescribing practices, or the HIV/AIDS Bureau guidelines sway you to only look at twice-annual? See also my comments on the first paragraph of the Discussion.

Lines 98-100 (“The definition of durable retention in care was defined as having two medical encounters and ART in each calendar-year, over a two, three, four, or five-year period, from 2013 to 2017.”): The durable retention definition seems unclear. How is the length of the period determined? Might be more standard to just pick one and justify it based on the literature, federal guidance, or the interest and practices of your clinic. Later when I go to the figure, I sort of understand what you’ve done – you calculated as many of the durable measures as you could for each person according to their amount of… enrollment or follow-up time? – and it would be helpful to describe that in the methods.

Line 84 (“These results will be used to assess progress of the clinic towards achieving NHAS 2020 goals.”): Yes, every HIV clinic should be doing this internally. But again, what about your clinic, analysis, or findings makes this publishable because it’s of interest to a broader audience? Expand on this to tell the reader how the findings will illuminate broader truths about HIV care and suppression among attendees of an urban US HIV clinic.

Results

Tables 1 and 2: How come the patient status (current / new / reengaged) breakdown of the 1,000 persons retained in care in Table 2 seems inconsistent with that for the 1,000 2017 persons in Table 1? Are 705 persons current, or 908?

Lines 183-184 (“Viral suppression increased yearly: 2013 (73%), 2014 (76%), 2015 (80%), 2016 (81%) and 2017 (87%).”): What a substantial and exciting increase. Congratulations! You are almost at 90%. What changed to make this happen? Was it clinic practice, medication changes, the slight shift in patient load toward persons with fewer financial barriers to adherence? This could be the angle for your entire paper. With a patient population with as many challenges as yours re substance use, mental illness, and poverty, many would want to know how you worked with your patients to achieve an increase to 87% suppression.

Table 4: Are there any other characteristics about the care itself or contact with PHC or the suppression initiative that you’d want to include in a model? Or do you take the retention measure generally as a proxy for contact with the initiative? Could potentially break out retention differently to be more explicit about the nature of the care, e.g., instead of not retained vs. retained, could instead do something like not retained vs. retained but no case manager / nurse contact vs. retained and had a session with case manager / nurse.

Discussion

Lines 207-208 (“Annual retention in medical care was higher among PLWH, in 2017, in PHC compared to national reports, 90% and 71.1%, respectively”): Are these statistics comparable? Seems like the 71% nationally may be among all PLWH, whereas PHC patients are inherently connected with a provider and were in care during at least one point in the last five years.

Lines 211-220: Given your attention to findings about suppression among persons not meeting the HAB criteria for retention in care (i.e., who have <2 visits/year), why not also look at this, potentially with a single-visit measure of retention? As it is, this first paragraph feels a bit like it goes off on a tangent here. Maybe you could explore suppression among persons not retained, at this length, in later paragraphs and, in the first paragraph, highlight several of your most important findings, such as that you’re hitting 90-90-90 targets for retention and suppression, your five-year retention rates aren’t extraordinary, and you experienced substantial increases in suppression in the last five years.

Lines 227-236: Great job with your viral suppression initiative! Seems like a huge success. Why did you hold this key information until the Discussion, and bury it in the third paragraph? And have you published anything else about it that you can cite, too? Again, this is a big part of your story. Let the reader know in the abstract and the intro, methods, and discussion sections.

Line 236: I see that this was a data-driven initiative with multiple partners, which is terrific. Readers will also want to know what those “appropriate interventions” were and ideally a few measures of those interventions. For example, did referrals to drug treatment / mental health services / housing services / adherence counseling / peer support increase from 2013 (pre-initiative) to 2017, and can you enumerate that change? What else was done? Did other PHC characteristics change? I see that the patient load decreased from >1,200 to 1,000. Did funding or staffing change? Did the ratio of case managers to patients increase? All of this is your story. If you don’t tell it fully in this paper, I hope you’ll at least give us a little more info here, and write it up completely elsewhere, because it seems exciting.

Line 280 (“Improved viral suppression from 2013-2017 was possible due to involvement of all staff.”): Certainly. Did patients and/or your CAB also inform your suppression initiative? Do you want to acknowledge that in some way here or earlier?

Figure 4: Beautiful and encouraging. I’m not clear on how you’re able to calculate five-year suppression for 2014 persons – would’ve thought just 2013. Same for four-year suppression for 2015 persons, etc.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1:Yes:Chukwuma Umeokonkwo

Reviewer #2:No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS atfigures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Dec 29;15(12):e0244376. doi: 10.1371/journal.pone.0244376.r002

Author response to Decision Letter 0


13 Sep 2020

Reviewer #1:

Title: The authors could consider a review of the title to "Retention in care and viral suppression among HIV patients in Peter Ho Clinic"

The title of the manuscript was revised as follows –Annual and durable HIV retention in care and viral suppression, Peter Ho Clinic, 2013-2017.

Abstract: Line 36-37 What are the comparison group?

Based on the word limit for the abstract, this sentence was deleted.

Method section:

The authors need to provide information on the study setting to help the readers understand the context of the clinic and its operational environment. There is need for more details about the patient’s management in the hospital and the total number of patients receiving HIV care in the hospital.

The setting of the clinic was described as follows:

Setting. The PHC is located on the campus of Saint Michael’s Medical Center, an urban academic institution in Newark, New Jersey. Newark is the epicenter of the epidemic in New Jersey and lies in close proximity to New York City. This is the first clinic in the state to provide medical care for PLWH and serves approximately 1,200 persons yearly. Co-located services include HIV testing, access to pre-exposure prophylaxis and linkage to care coordinators. Clinical staff include infectious diseases providers and fellows, nurse practitioners, medical and non-medical case-managers, and a phlebotomist. Specialty practices include gynecology, medication assisted therapy for opioid use and pain management services.

line 89-91: How long had the newly diagnosed patients been on treatment? Were they excluded because they did not experience virologic failure or because they were not eligible for the test as at the time the analysis were conducted? Could the authors provide more information about how many patients were excluded from the analysis and for what reasons in a flow diagram?

The sentence was rephrased to include the reason for exclusion: Six patients were excluded from the virologic failure analysis as their viral loads were less than 200 copies/mL at the time of diagnosis (Fig 1).

Line 98: Remove "The definition of"

Done

line 98-100: In defining the annual and durable retention, what was the starting point? The start of study or the point the patient commenced on ART? There is need to clarify on this further for ease of reproducibility. It is also important because retention rate vary with the duration on care from commencement of ART. This is same for viral suppression.

I added the term – calendar year to the definitions of retention and viral suppression, for example, “Annual retention in medical care included documentation of at least two medical encounters, at least 90 days apart, at PHC in the respective calendar-year (Yes, No). “

Also clarified under study population as follows:

Patients included in this study were at least 18 years old and alive as of December 31st of the respective year. At least one time in 2013-2017, they saw a medical provider, received a prescription for ART and had viral load results documented in the electronic medical record. They were included in the study at the time of their first medical visit or viral load to six months after the last documented viral load or medical visit

line 109: How about 18 and 19 years of age?

The age category was 18-29. 18 and 19-year-olds constitute a miniscule proportion of PLWH in PHC

line 121: How did you handle those that did not have insurance "none"? The constituted 21% of the total respondents.

I added the following definition for patients without insurance: (the care of these patients was paid by charity care, state funding or Ryan White funding)

line 129-132: Could the authors provide more details about the model development? How did they handle multiple collinearity? How were factors selected for the final model?

I expanded on the data analysis section as follows: :

Multiple logistic regression models, using backwards deletion, were developed to identify factors associated with retention in medical care in 2017; virologic failure, 2013-2017; and viral suppression, 2013-2017, for those experiencing virologic failure. The selected variables were based on clinical and epidemiological significance. Collinearity in the categories for risk and drug use were mitigated by adhering to the hierarchical categorization developed by the Centers for Disease Control and Prevention [27], and categorizing drug use as cocaine only, heroin only and those with a history of using both drugs. Model fit was evaluated using Hosmer and Lemeshow Goodness-of-Fit test.

Results section:

Line 134: Were these all PLWH? Yes – I deleted the word patients and added PLWH.

Where they new enrollment within the period, all previously enrolled prior to this period or both new and old enrollments? This could have been explained in the method section for more clarity.

We added the following definition under covariates, earlier in the manuscript: Patient’s status was defined as Current, New and Reengaged. Current patients were those in care in 2013. In the respective calendar-year, new patients were newly diagnosed and reengaged were PLWH who did not previously receive care in PHC.

Table 2: Please check the figures, the total number of new in Table 1 is 157 and is 158 in Table 2. Also, the current was 1229 in Table 1 and 1230 in Table 2. Were there additions? If anything the numbers should be less if some died within the period. Kindly cross check the figures.

Thank you for this comment. I reviewed all the tables, corrected the numbers, and included Figure 1.

One would expect a footnote here explaining the merging of the cells in table 1. Done for each table.

line 160: "patient" status This change was made.

Table 4:

The authors should consider rephrasing the title of table 4. I understand that these were the initial 935 patients that ever reported virologic failure whose latest results showed some changes probably after an adherence intervention and a follow up test. It should read factors affecting viral suppression among those that reported an earlier virologic failure at PHC 2013-2017

The title of Table 4 was changed: Table 4: Factors Affecting Viral Suppression among patients with Virologic Failure, Peter Ho Clinic, 2013-2017

Discussion:

The authors should resist the urge of repeating the results in the discussion section. They should rather discuss the implication of their finding to practice and situated it in the body of knowledge. They can also provide unique context to help the readers understanding the findings and why they may defer from other well known findings.

Line 221:

The authors could do more search on longitudinal retention in care among PLWH in literature, what is described as durable retention here. Here some literature to consider

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5289300/

https://link.springer.com/article/10.1007/s10461-019-02450-7?shared-article-renderer

https://academic.oup.com/cid/article/62/5/648/2462795

Thank you for these references, we have reviewed and included them in the paper. The discussion section was revised extensively.

Line 271: Was this a cohort study? Deleted the word “cohort”. Thanks

Reviewer #2: This paper reports a 14-percentage-point increase in HIV viral suppression over five years in the HIV-focused Peter Ho Clinic in urban New Jersey, without consistent increases to retention in care. These findings are inspiring, but the paper should frame them more compellingly, more clearly describe analytic decisions, and highlight and analyze their suppression initiative more.

Abstract

Please see comment below about conveying your unique angle and relevance throughout the paper, including in the abstract.

Introduction

Lines 52-54: Please make the statistics about viral suppression comparable in both sentences, i.e., both among (or not among) persons retained in care.

The sentence was changed: In comparison at the Peter Ho Clinic (PHC), of 1,229 PLWH in medical care in 2013, 84% were retained for one year, of whom 73% were suppressed.

Why this set of years and why this clinic? What unique questions of broad relevance were you able to ask and answer? I highlighted the following in the paper-

This time-period is relevant as it is within the evaluation period of the NHAS, longitudinal evaluations are more relevant as we progress towards ending the epidemic.

Share things that will help the reader see how this analysis informs not only where this leaves PHC patients and policies but also other populations and health care systems. For example, PHC is the first and largest HIV clinic in NJ, it is set within a large medical center that also provides social services, it accepts public insurance and is supported by federal Ryan White funding and many patients are low-income, NJ and Newark are racially and ethnically diverse and have a relatively high prevalence of HIV, etc. The particular angle and relevance of your paper should shine not only in the introduction but also the abstract and discussion.

We added an Introductory paragraph, in addition to expanding on the setting in the Methods section.

An estimated 1.04 million persons were living with HIV (PLWH) in the United States in 2018, with a prevalence of 374.6 per 100,000 population. [1] Males accounted for (75%), of whom 35% were Black, 73% were males who had sex with males (MSM), and injection drug use (IDU) was 9%. Among females, 58% were Black, of whom 77% reported heterosexual contact and 20% IDU as their transmission risk. In comparison, the prevalence rates in New Jersey and Essex County, were 419.7 and 1,194.9 per 100,000 population, respectively. [2] In Essex County, males accounted for 61% of PLWH, of whom 69% were Black, 35% were MSM, and 16% reported IDU. Among females 81% were Black, of whom 19% reported IDU and heterosexuals were 67%. The Peter Ho Clinic (PHC) is located in the City of Newark, in the county of Essex. The distribution of PLWH in Newark is like that of Essex County. [3]

Under methods – I included a description of the setting to contextualize the study.

Setting. The PHC is located on the campus of Saint Michael’s Medical Center, an urban academic institution in Newark, New Jersey. Newark is the epicenter of the epidemic in New Jersey and lies in close proximity to New York City. This is the first clinic in the state to provide medical care for PLWH and serves approximately 1,200 persons yearly. Co-located services include HIV testing, access to pre-exposure prophylaxis and linkage to care coordinators. Clinical staff include infectious diseases providers and fellows, nurse practitioners, medical and non-medical case-managers, and a phlebotomist. Specialty practices include gynecology, medication assisted therapy for opioid use and pain management services.

Methods

Line 110: “Gender (male, female)” needs to be changed. Please specify if this is gender identity or sex assigned at birth. If the former, the terms “man” and “woman” are more appropriate, and it would also be helpful to state whether and how non-binary and transgender persons were ascertained and classified. Restated as Sex at Birth

Your analyses used multiple outcomes, denominators, inclusion criteria, and time periods. This section needs to more clearly convey what exactly the analyses were and who was in each one. Places to insert expansions on this may be around lines 88-89 (“Patients included in this study were at least 18 years old and alive as of December 31st of the respective year.”); and lines 82-83 (“This study evaluated annual retention in medical care and viral suppression, durable retention in medical care and viral suppression among PLWH, in the PHC, from 2013 to 2017.”).

I expanded on both sentences as follows and added Figure 1:

Population. Patients included in this retrospective observational study were at least 18 years of age and alive as of December 31st of the respective year. At least one time in 2013-2017, they saw a medical provider, received a prescription for antiretroviral therapy (ART) and had viral load results in the electronic medical record. Patients were eligible at the time of their first medical visit or viral load until six months after the last documented viral load or medical visit. Six patients were excluded from the virologic failure analysis as they did not have detectable viral loads at the time of diagnosis.

This study evaluated annual and durable retention in medical care and viral suppression, among PLWH, in the PHC, from 2013 to 2017. These results will be used to assess progress of this clinic towards achieving NHAS 2020 goals. In addition, factors that may contribute or serve as barriers to achieving these goals will be identified to inform future interventions.

A flow chart was added to describe the clarify the denominators used in different evaluations.

Lines 95-97 (“Annual retention in medical care included documentation of at least two medical encounters at PHC in the respective year (Yes, No). A medical encounter was defined as a medical visit with a prescribing provider or an HIV viral load test…”): Guidelines have been shifting to permit once-yearly testing for persons with durable suppression. Especially since you also have a durable retention measure, did you consider looking at retention as being at least once in the year? Or did clinic policy, prescribing practices, or the HIV/AIDS Bureau guidelines sway you to only look at twice-annual?

Current practice required by funders is that we have 2 medical visits and 2 viral loads each year. (goal).

I used the HIV/AIDS Bureau guidelines to arrive at the definitions of retention and durable retention as I think that this is more in line with reality which was described as 2 medical encounters – a medical visit or viral load. These are both considered as medical encounters – different than twice annual recommendations.

Lines 98-100 (“The definition of durable retention in care was defined as having two medical encounters and ART in each calendar-year, over a two, three, four, or five-year period, from 2013 to 2017.”): The durable retention definition seems unclear. How is the length of the period determined? Might be more standard to just pick one and justify it based on the literature, federal guidance, or the interest and practices of your clinic. Later when I go to the figure, I sort of understand what you’ve done – you calculated as many of the durable measures as you could for each person according to their amount of… enrollment or follow-up time? – and it would be helpful to describe that in the methods.

I added the term calendar-year which I think would lend clarity- also the inclusion criteria as follows:

Durable retention in care was defined as having at least two medical encounters and ART in each calendar-year, over a two, three, four, or five-year period, from 2013 to 2017.

Patients were eligible from the time of their first medical visit or viral load until six months after the last documented viral load or medical visit.

Line 84 (“These results will be used to assess progress of the clinic towards achieving NHAS 2020 goals.”): Yes, every HIV clinic should be doing this internally. But again, what about your clinic, analysis, or findings makes this publishable because it’s of interest to a broader audience? Expand on this to tell the reader how the findings will illuminate broader truths about HIV care and suppression among attendees of an urban US HIV clinic.

The results of this study will be used to assess progress of this clinic towards achieving The National HIV/AIDS Strategy 2020 goals. In addition, factors that may facilitate or serve as barriers to achieving these goals, will be identified to inform future interventions and provide valuable lessons to other clinics, serving an urban population with similar challenges.

I also highlight the importance of longitudinal measures of retention and viral suppression.

Results

Tables 1 and 2: How come the patient status (current / new / reengaged) breakdown of the 1,000 persons retained in care in Table 2 seems inconsistent with that for the 1,000 2017 persons in Table 1? Are 705 persons current, or 908?

Thank you for this comment. I reviewed the numbers.

Lines 183-184 (“Viral suppression increased yearly: 2013 (73%), 2014 (76%), 2015 (80%), 2016 (81%) and 2017 (87%).”): What a substantial and exciting increase. Congratulations! You are almost at 90%. What changed to make this happen? Was it clinic practice, medication changes, the slight shift in patient load toward persons with fewer financial barriers to adherence? This could be the angle for your entire paper. With a patient population with as many challenges as yours re substance use, mental illness, and poverty, many would want to know how you worked with your patients to achieve an increase to 87% suppression.

We added the viral suppression intervention under Methods.

Table 4: Are there any other characteristics about the care itself or contact with PHC or the suppression initiative that you’d want to include in a model? Or do you take the retention measure generally as a proxy for contact with the initiative? Could potentially break out retention differently to be more explicit about the nature of the care, e.g., instead of not retained vs. retained, could instead do something like not retained vs. retained but no case manager / nurse contact vs. retained and had a session with case manager / nurse.

Added the clinic-wide suppression intervention to the methods section and discussed the impact on increasing the viral suppression rate.

Discussion

Lines 207-208 (“Annual retention in medical care was higher among PLWH, in 2017, in PHC compared to national reports, 90% and 71.1%, respectively”): Are these statistics comparable? Seems like the 71% nationally may be among all PLWH, whereas PHC patients are inherently connected with a provider and were in care during at least one point in the last five years.

Deleted this comment

Lines 211-220: Given your attention to findings about suppression among persons not meeting the HAB criteria for retention in care (i.e., who have <2 visits/year), why not also look at this, potentially with a single-visit measure of retention? As it is, this first paragraph feels a bit like it goes off on a tangent here. Maybe you could explore suppression among persons not retained, at this length, in later paragraphs and, in the first paragraph, highlight several of your most important findings, such as that you’re hitting 90-90-90 targets for retention and suppression, your five-year retention rates aren’t extraordinary, and you experienced substantial increases in suppression in the last five years.

Done in the discussion section

Lines 227-236: Great job with your viral suppression initiative! Seems like a huge success. Why did you hold this key information until the Discussion, and bury it in the third paragraph? And have you published anything else about it that you can cite, too? Again, this is a big part of your story. Let the reader know in the abstract and the intro, methods, and discussion sections.

This was moved to the Methods section

Viral Suppression Intervention. A clinic-wide viral suppression effort was implemented in 2014, involving all staff and targeting PLWH with viral loads of at least 200 copies/ml. Three teams were established, each composed of a clinician (either a nurse practitioner or doctor), a case manager, nurse, and operational staff. Viral load data were electronically downloaded, in rolling 1-year intervals, in two-month increments, from January 1, 2013 to December 31, 2017. Every two months, each team was provided with an updated list of patients with a virologic failure. A nurse practitioner reviewed genotype results and the electronic medical record to ensure that patients were on appropriate ART. Case managers and nurses assessed barriers to adherence and medical care and implemented appropriate interventions.

Line 236: I see that this was a data-driven initiative with multiple partners, which is terrific. Readers will also want to know what those “appropriate interventions” were and ideally a few measures of those interventions. For example, did referrals to drug treatment / mental health services / housing services / adherence counseling / peer support increase from 2013 (pre-initiative) to 2017, and can you enumerate that change?

This would be difficult to enumerate as there were no data collected. Added the following to the Methods section

Patients with adherence issues came in weekly for prefilled medication boxes and were transitioned to prefilled pharmacy packages when they became suppressed. The registration staff were key in scheduling and doing phone reminders for appointments. Referrals for substance use and mental health were done for on and off-site locations. Patients with unstable housing were referred to the New Jersey Housing Collaborative for assistance.

What else was done? Did other PHC characteristics change? I see that the patient load decreased from >1,200 to 1,000. Did funding or staffing change? Did the ratio of case managers to patients increase? All of this is your story. If you don’t tell it fully in this paper, I hope you’ll at least give us a little more info here, and write it up completely elsewhere, because it seems exciting.

None of this happened. I think just the constant follow up by the staff made it happen.

No measurable changes or significant funding increases occurred. By virtue of fluctuating number of patients there may have been subtle, transient, changes in case managers workloads, however no data was collected in this regard. I believe the consistency and diligence of follow up by the staff contributed significantly.

Line 280 (“Improved viral suppression from 2013-2017 was possible due to involvement of all staff.”): Certainly. Did patients and/or your CAB also inform your suppression initiative? Do you want to acknowledge that in some way here or earlier?

The CAB was not involved in developing the viral suppression initiative.

I added acknowledgements to both patients and staff in PHC

Figure 4: Beautiful and encouraging. I’m not clear on how you’re able to calculate five-year suppression for 2014 persons – would’ve thought just 2013. Same for four-year suppression for 2015 persons, etc.

Used data up to December 2019 – added to Methods section

Decision Letter 1

Ellen Wiewel

16 Oct 2020

PONE-D-20-02167R1

Annual and durable HIV retention in care and viral suppression among patients of Peter Ho Clinic, 2013-2017

PLOS ONE

Dear Dr. Mohammed,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Thank you for your substantial revisions to the paper. This looks much better. The descriptions of the clinic, its epidemiology vs the US HIV epidemic, and its initiative for unsuppressed PLWH are more thorough. I have a few additional suggestions, with line numbers from the manuscript version with changes tracked.

Line 180-181, "From 2009-2016, durable viral suppression was 37% in New York City, for the two and three-year period": Please provide two percentages here if reporting suppression over two and three years.

Line 216-218, "They were included in the study at the time of their first medical visit or viral load to six months after the last documented viral load or medical visit": It's the first visit/VL in the calendar years you looked at, rather than the first for the patient after diagnosis or coming to the clinic, right? Would be clearer to say, for example, "... of their first medical visit or viral load in 2013-2017."

Line 224 and throughout: The term "intervention" can suggest a set of actions that are limited in time or scope. Did you basically permanently restructure clinic services for all unsuppressed PLWH, with no comparison group? May want to use a term other than "intervention" or clarify that this intervention basically became the new way you're delivering services.

Line 300, "...categories of the following covariates were collapsed... insurance (Public, Private)": Please specify which group included the many uninsured patients.

Line 337, "Patients’ status included current patients 1229 (70%), 157 (9%) newly diagnosed and 373 (21%) who reengaged in medical care": Please revise so wording is less awkward, e.g., could say "By patient status, this included 1229 current patients (70%), 157 (9%) newly diagnosed patients, and 373 (21%) patients who reengaged in medical care."

Line 501-503, "This increase is attributed to the implementation of a clinic-wide viral suppression effort, involving all staff starting in 2014": Without a comparison to other PLWH in your clinic, Newark/Essex County, or the US, where VS may have also increased over these years, it would be more appropriate to swap "is attributed to" for "coincides with" to avoid suggesting causality.

Line 586-612, "In this study, PLWH in public housing were...": The consideration of housing and insurance is important. Please tie the literature back to your findings more. For example, how do you resolve the findings that PHC's public housing residents did worse but housing in general is known to be helpful? And the connection between insurance challenges and retention is very plausible but would be even stronger if you could provide evidence that PLWH at PHC had challenges reenrolling in Medicaid that affected their retention in care, and if you could square that with the fact that PHC also (according to your tables) sees uninsured persons.

Hopefully all of these can be addressed before publication, butrewording "intervention" is optional.

==============================

Please submit your revised manuscript by Nov 30 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see:http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Ellen Wiewel

Academic Editor

PLOS ONE

Additional Editor Comments (if provided): see above

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS atfigures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Dec 29;15(12):e0244376. doi: 10.1371/journal.pone.0244376.r004

Author response to Decision Letter 1


28 Nov 2020

Dear Editor and Reviewers,

Thank you for your comments.

Please find the revisions outlined below

Line 180-181, "From 2009-2016, durable viral suppression was 37% in New York City, for the two and three-year period": Please provide two percentages here if reporting suppression over two and three years.

This sentence was clarified as follows:

In a study evaluating the impact of a care coordination program, from 2009-2016, durable viral suppression was 37% in New York City, in months 13-36 of follow-up.

Line 216-218, "They were included in the study at the time of their first medical visit or viral load to six months after the last documented viral load or medical visit": It's the first visit/VL in the calendar years you looked at, rather than the first for the patient after diagnosis or coming to the clinic, right? Would be clearer to say, for example, "... of their first medical visit or viral load in 2013-2017."

This sentence was clarified as follows:

They were included in the study at the time of their first medical visit or viral load in 2013-2017 to six months after the last documented viral load or medical visit.

Line 224 and throughout: The term "intervention" can suggest a set of actions that are limited in time or scope. Did you basically permanently restructure clinic services for all unsuppressed PLWH, with no comparison group? May want to use a term other than "intervention" or clarify that this intervention basically became the new way you're delivering services.

This sentence was changed as follows:

Beginning in 2014, services were restructured for patients with viral loads of at least 200 copies/mL.

Line 300, "...categories of the following covariates were collapsed... insurance (Public, Private)": Please specify which group included the many uninsured patients.

The definitions were included as follows:

insurance (Medicaid, Medicare, private, none [the medical care of these patients were paid by charity care, grant funding from the state or Ryan White HIV/AIDS Program])

Further:

In the regression analyses, categories of the following covariates were collapsed where the outcomes were similar: age (<60, > 60 years), race (non-Hispanic Black or Hispanic, Other), insurance (Public[included Medicaid, Medicare, None], Private) and drug use, (Yes, No).

Line 337, "Patients’ status included current patients 1229 (70%), 157 (9%) newly diagnosed and 373 (21%) who reengaged in medical care": Please revise so wording is less awkward, e.g., could say "By patient status, this included 1229 current patients (70%), 157 (9%) newly diagnosed patients, and 373 (21%) patients who reengaged in medical care."

This sentence was revised as suggested.

Line 501-503, "This increase is attributed to the implementation of a clinic-wide viral suppression effort, involving all staff starting in 2014": Without a comparison to other PLWH in your clinic, Newark/Essex County, or the US, where VS may have also increased over these years, it would be more appropriate to swap "is attributed to" for "coincides with" to avoid suggesting causality.

This sentence was modified as follows:

This increase coincides with the restructuring of clinic services for patients with a viral load at least 200 copies/ml, involving all staff starting in 2014.

Line 586-612, "In this study, PLWH in public housing were...": The consideration of housing and insurance is important. Please tie the literature back to your findings more. For example, how do you resolve the findings that PHC's public housing residents did worse but housing in general is known to be helpful?

This paragraph was clarified as follows:

In this study, patients who lived in public compared to private housing were less likely to achieve viral suppression. They included those who were unstably housed. Among patients accessing care in Ryan White funded sites in 2018, those who lived in unstable housing had lower rates of viral suppression compared to those who lived in stable housing (72.4% versus 88.4%), respectively. [37] Previous reports indicate that public housing mitigate barriers related to viral suppression. A recent retrospective matched cohort study in New York City reported that low income PLWH who received housing services for more than one year, were more likely to be engaged in care and virally suppressed than similar populations who did not receive these services [38]. In another New York City study, PLWH in continuous, stable housing in 2015, were reported to have continuous, stable viral suppression. [38]. Conversely, time spent in emergency housing was a predictor of a lack of viral suppression. Identifying patients in unstable housing and transitioning them to a stable environment will improve viral suppression rates in this subset of patients.

I also identified another limitation

Fifth, in this study we evaluated public and private housing only. In future evaluations, also classifying housing as stable or unstable may improve the granularity of the results.

And the connection between insurance challenges and retention is very plausible but would be even stronger if you could provide evidence that PLWH at PHC had challenges reenrolling in Medicaid that affected their retention in care, and if you could square that with the fact that PHC also (according to your tables) sees uninsured persons.

Changes were made to the discussion paragraph re insurance

Insurance status was another structural barrier identified in this study. New Jersey expanded Medicaid and increased access to care for low-income residents, by implementinged in accordance with many reforms recommended by the Affordable Care Act starting in 2014  [4038]. Despite the availability of public insurance, before being seen in PHC, there were many processes patients had to navigate to make this a reality., These includeding qualifying for hospital charity care if they did not have insurance [41], enrollment and reenrollment, havingand  requiring a primary care provider to issue and referrals if they had a managed care plan,  before being seen in PHC. This may have contributed to patients not being retained in medical care or achieving viral suppression. Patients who are lost to care, more symptomatic, and/or racial/ethnic minorities may be less likely to overcome these barriers [4239]. At least 70% of PLWH at PHC, qualified for federally funded services, based on an income less than the FPL, for one person, in 2017, or had either Medicaid or no insurance (Charity Care/Ryan White funding), and were primarily racial and ethnic minorities (>90%). Numerous studies have documented the importance of Ryan White federal funding in maintaining PLWH in care and improved outcomes [30, 40-44-47]. Urban clinics will need to maximize the use of this resource to ensure that patients overcome structural barriers and achieve the goals of the NHAS.

Decision Letter 2

Ellen Wiewel

9 Dec 2020

Annual and durable HIV retention in care and viral suppression among patients of Peter Ho Clinic, 2013-2017

PONE-D-20-02167R2

Dear Dr. Mohammed,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Ellen Wiewel

Guest Editor

PLOS ONE

Additional Editor Comments (optional):

Thank you for your revisions. Congratulations.

Reviewers' comments:

Acceptance letter

Ellen Wiewel

14 Dec 2020

PONE-D-20-02167R2

Annual and durable HIV retention in care and viral suppression among patients of Peter Ho Clinic, 2013-2017

Dear Dr. Mohammed:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Ellen Wiewel

Guest Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Availability Statement

    All relevant data are within the manuscript


    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES