Abstract
We evaluated the feasibility and efficacy of four existing interventions to improve adherence to them in migrants living with HIV (MLWH): directly administered antiretroviral therapy (DAART), group medical appointments (GMA), early detection and treatment of psychological distress, and peer support by trained MLWH. At baseline and after the interventions, socio-demographic characteristics, psychosocial variables, and data on HIV treatment adherence were collected. The two questionnaires were completed by 234/301 (78%) MLWH included at baseline. Detectable HIV RNA decreased (from 10.3 to 6.8%) as did internalized HIV-related stigma (from 15 to 14 points), and self-reported adherence increased (between 5.5 and 8.3%). DAART and GMA were not feasible interventions. Screening of psychological distress was feasible; however, follow-up diagnostic screening and linkage to psychiatric services were not. Peer support for and by MLWH was feasible. Within this small intervention group, results on HIV RNA < 400 copies/mL (decrease of 23.6%) and outpatient clinic attendance (up to 20.4% kept more appointments) were promising.
Keywords: HIV/AIDS, migrants, treatment adherence, interventions
1. Introduction
Since the introduction of combination Antiretroviral Therapy (cART) in 1996, HIV infection has become a manageable chronic disease with a greatly improved life expectancy [1,2]. However, despite advances in access to treatment and treatment regimens with fewer pills and fewer side effects, adherence to cART remains one of the main challenges to achieving optimal HIV treatment outcomes [2].
Several factors have been associated with adherence to cART, including clinical appointment attendance and psychosocial factors such as HIV-related stigma, social support, psychological distress, and quality of life [3,4,5,6]. Migrants living with HIV (MLWH), who, in Europe, comprised 40% of all people diagnosed with HIV between 2008 and 2017 [7], have worse psychosocial and clinical outcomes compared to people living with HIV (PLWH) born in Western Europe [8,9,10,11,12]. MLWH can thus be considered a “key population”. Specific risk factors for non-adherence to cART in MLWH include experiencing low social support, having low educational attainment, and experiencing low treatment adherence self-efficacy [13]. Because risk factors for non-adherence significantly influence treatment outcomes in MLWH, finding interventions that address risk factors for non-adherence to cART in this population is important.
Research on the feasibility and the efficacy of interventions aiming to improve adherence among MLWH is limited. We identified four existing interventions that could potentially improve treatment adherence in MLWH. The first was directly administered antiretroviral therapy (DAART). This intervention is derived from Directly Observed Therapy that has been used in tuberculosis control programs [14]. It usually entails patients taking their medications under the supervision of a healthcare provider multiple times per week alongside counseling on HIV treatment management during visits. The second intervention was group medical appointments (GMAs), in which multiple patients who share a medical condition collectively receive a consultation with their physician [15]. At these appointments, treatment adherence and living with the condition are discussed. Participants share their experiences of living with HIV, support each other, and learn from the experiences of other patients (peers) and the professional expertise of care providers (e.g., a nurse) [16]. GMA has been deployed in various healthcare settings and has been shown to improve knowledge about the condition and management of the condition, which can lead to improved self-management [17]. The third intervention was screening and treatment of psychological distress (i.e., anxiety and depression). Psychological distress has been found to contribute to non-adherence to cART, suboptimal treatment outcomes, and mortality [3,18,19,20], and previous studies have shown prevalence rates of 33% and 12.8–78% for anxiety and depression among PLWH [18,21,22]. Screening for psychological distress is thus important and subsequent treatment of psychological distress may be a key tool for increasing cART adherence. The fourth intervention was peer support by MLWH, whereby individuals have increased access to emotional support, informational support, and appraisal support [23,24]. Peer support has been recommended as a valuable addition to the clinical care of people living with chronic diseases, especially for individuals who are hard to reach due to individual (e.g., psychological distress) or demographic (e.g., ethnic minority) characteristics [25]. In low-income and middle-income countries, support from peers has shown positive effects on HIV treatment adherence and outcomes, and when compared to standard of care, was superior in reaching viral suppression [26,27,28].
The objective of the current study was to evaluate the feasibility and, if feasible, the efficacy of these four interventions. This study is the first to explore DAART, GMAs, screening and treatment of psychological distress, and peer support as interventions to improve HIV treatment adherence in a large cohort of non-Western MLWH in a Western European context.
2. Methods
2.1. Study Design and Context
This study is embedded in the ROtterdam ADherence (ROAD) project, which is a quasi-experimental intervention study in which the primary aim was to increase adherence in first- and second-generation adult immigrant people living with HIV (The Netherlands National Trial Registry Number—NTR4941). In the ROAD project, we prospectively followed a large group of MLWH over a two to three year period. In the study reported here, we explored the feasibility and, if feasible, the efficacy of four interventions that aim to improve adherence to HIV treatment. Our research questions were: (1) “Are DAART, GMA, screening and treatment of psychological distress, and peer support by MLWH feasible interventions in MLWH, living in The Netherlands?” and (2) “Among feasible interventions, to what extent are the interventions efficacious in improving HIV treatment adherence?”. The Medical Ethics Committee of the Erasmus University Medical Center and Maasstad Hospital have evaluated and approved the study on 28 September 2012 (reference number: MEC-2012-399).
2.2. Sampling and Recruitment
Participants were included if they originated from a region outside of Western Europe (first or second generation), were aged 18 years or older, were diagnosed with HIV, and were sufficiently fluent in Dutch, English, French, Spanish, or Portuguese. Between November 2012 and July 2013, a total of 352 PLWH from outpatient clinics of two Rotterdam HIV treatment centers (Erasmus University Medical Center and Maasstad Hospital) participated in the study [13].
2.3. Selection for Intervention(s)
Participants who experienced substantial difficulty living with HIV, had problems with HIV treatment adherence, or were at risk for non-adherence despite adequate access to HIV care were allocated to an intervention by a multidisciplinary HIV care team who determined which intervention (DAART, GMA, or peer support) or combination of interventions would be most appropriate for each given patient. All participants were asked to complete the Hospital Anxiety and Depression Scale (HADS), and allocation to the early detection and treatment of psychological distress intervention was made based on participants’ scores on the HADS [29]. We allowed allocation to multiple interventions because adherence to antiretroviral therapy is influenced by multiple factors, and multiple interventions have been shown to yield better adherence than single interventions [28].
Directly Administered Antiretroviral Therapy: The DAART intervention required participants to attend three to five additional appointments per week (Monday–Friday) with a nurse specialized in HIV care at an outpatient clinic where they would take their cART under supervision and where they could discuss successes and challenges in cART adherence. Other doses were to be self-administered. After six months, participants were expected to attend one to two visits per week and then, later, monthly visits, with a total duration of the intervention being up to 12 months. Patients’ travel costs to the hospital were reimbursed.
Group Medical Appointment: For the GMAs, eligible participants were initially approached by their physician or HIV nurse during their regular consultation or by phone, after which they received a patient information leaflet about the GMA. Prior to participation, patients signed a confidentiality form stating that they would not discuss patient-related information that was discussed during the GMA. We aimed to include a demographically heterogeneous group of PLWH in the GMAs because previous experience had shown that creating a heterogeneous group seemed to be favorable for group processes [30].
Screening and Treatment of Psychological Distress: All MLWH participating in the ROAD project were asked to complete the 14-item Hospital Anxiety and Depression Scale (HADS) [29], which is a widely used screening tool for psychological distress in multiple languages [31]. A HADS score of ≥15 (HADS+) indicates psychological distress [32,33,34], and all participants with a HADS+ were subsequently invited to complete the Composite International Diagnostic Interview (CIDI), which is a questionnaire designed to assess psychological disorders according to the definitions and criteria in the ICD-10 and DSM-IV [35,36,37]. When the CIDI results indicated psychological distress and the patient was not already enrolled in psychiatric care, a consultation with a psychiatrist was offered and the treating HIV physician was informed about the results (care as usual). This process has been described in more detail elsewhere [38].
Peer Support: ROAD participants eligible for peer support were approached by their HIV care provider. When they consented to participate, the HIV care provider referred them to Stichting Mara (informal care organization), who matched them to a trained peer MLWH. Peers were MLWH who had a history of being adherent to cART and/or outpatient clinic appointments. Previous research has shown that peers are best when matched on similar characteristics like demographics (e.g., sex, age, country of origin), behaviors (e.g., substance abuse), and conditions (e.g., living with HIV) [39]. Peers were selected by their HIV care providers and, prior to being matched, received a three-day training in which they were provided with a medical update on HIV and cART and learned how to deal with psychological distress among PLWH, HIV-related stigma, and taboos. They also discussed setting appropriate limits in the peer relationship, when to refer matched patients to formal care, and their own motivations for participation. Peers participated on a voluntary basis and received ongoing support from the counselors from Stichting Mara.
2.4. Data Collection
Prior to data collection, written informed consent was provided by all participants. Methods for baseline data collection of sociodemographic and psychosocial variables have been presented previously [13]. In short, data were collected via medical records and an interviewer-administered questionnaire. Assessment of psychosocial variables occurred via the eight-item Medical Outcomes Study Social Support Survey (mMOS-SS) [40], the six-item Internalized AIDS-Related Stigma Scale [41], the 12-item HIV Treatment Adherence Self-Efficacy Scale (HIV-ASES) [42], the 12-item Short-Form Health Survey (SF-12) [43], and the Hospital Anxiety and Depression Scale (HADS, only at baseline) [29]. Three four-item measures developed from adherence questions used in previous studies were used to measure self-reported adherence to cART [13]. The results of all three developed measures presenting stricter (Measure I) and less strict (Measure II and III) criteria for adherence are presented. Approximately 1–2.5 years after inclusion, participants were again approached during their outpatient clinic appointment to complete this questionnaire (T1) again.
HIV RNA values, CD4 cell counts, and information on cART use were collected from the ATHENA national observational HIV cohort database (the Dutch national HIV registry of HIV treatment centers). When incomplete, data were cross-checked with medical records. HIV RNA and outpatient clinic attendance data on participants in the peer support intervention were collected between one year before the match and up to two years afterward.
Additionally, participants in the GMA were asked to anonymously complete an evaluation form that included questions on how they experienced the GMA and whether they would participate again.
2.5. Data Analyses
Sociodemographic, clinical, and outpatient clinic attendance data were processed in Microsoft Office Excel and SPSS Statistics 24.0 (IBM New York, NY, USA. The Wilcoxon matched-pairs test, paired-samples t-test, McNemar test, and McNemar-Bowker test were used to compare data of the 2 questionnaires of the participants who used cART ≥ 6 months at the time of inclusion.
3. Results
3.1. Participant Characteristics
Characteristics of the included population in the ROAD project have been reported elsewhere [13]. In short, 352 of the 857 eligible MLWH were included at baseline (41.1% of eligible MLWH) (Figure 1). Approximately 58% were men, almost all were first-generation immigrants (94.6%), and the majority originated from Sub-Saharan Africa (40.6%), followed by Latin America (22.7%), the Caribbean (19.6%), and then other regions (17.0%). Of these 352 participants, 301 used cART ≥ 6 months prior to inclusion (cART-experienced) (Figure 1). A total of 234/301 of the cART-experienced participants completed both the first and the second questionnaire (baseline and T1) (response = 78%) (Figure 1). The main reasons for not completing T1 were of a logistical nature (n = 44, 66%) and refusal to participate (n = 17, 25%).
Figure 1.
Patient inclusion ROtterdam ADherence (ROAD) Project. DAART, Directly Administered Antiretroviral Therapy; GMA, group medical appointments; HADS, Hospital Anxiety and Depression Scale.
Table 1 presents characteristics of the 234 cART-experienced participants who completed both questionnaires. Overall, the percentage of participants with detectable HIV RNA (≥50 copies/mL) decreased (10.3% to 6.8%, p = 0.12), internalized HIV-related stigma score decreased (15 to 14, p < 0.01), and the percentage of participants who were adherent increased (between 5.5% and 8.3%, p-values between 0.15 and <0.05).
Table 1.
Sociodemographic characteristics and psychosocial variables.
| Variable | Baseline n = 234 |
T1 n = 234 |
p | |
|---|---|---|---|---|
| HIV-RNA > 50 copies/mL (%) | 24 (10.3) | 16 (6.8) | 0.12 d | |
| Age (mean, SD) | 43.2 (9.9) | |||
| Male gender (%) | 127 (54.3) | |||
| 1st generation immigrant (%) | 225 (96.2) | |||
| Region of origin (%) | ||||
| Sub Saharan Africa | 103 (44.0) | |||
| Caribbean | 42 (17.9) | |||
| Latin America | 45 (19.2) | |||
| Other | 44 (18.8) | |||
| Sexual orientation (%) a | ||||
| Heterosexual | 153 (65.4) | |||
| Gay/Bisexual | 71 (30.3) | |||
| Does not know | 7 (3.0) | |||
| Living situation (%) | 0.23 e | |||
| With family | 89 (38.0) | 80 (34.2) | ||
| Alone | 85 (36.3) | 96 (41.0) | ||
| Single parent | 45 (19.2) | 44 (18.8) | ||
| Other | 15 (6.4) | 14 (6.0) | ||
| Children (%) a | 143 (61.1) | |||
| Educational attainment (%) a | ||||
| No formal education/Primary school | 62 (26.5) | |||
| Secondary school | 69 (29.5) | |||
| Higher vocational school | 54 (23.1) | |||
| University | 47 (20.1) | |||
| Employment status (%) | 0.23 e | |||
| Paid employment | 92 (39.3) | 88 (37.9) | ||
| Unemployed | 67 (28.6) | 62 (26.5) | ||
| On sick leave | 26 (11.1) | 25 (10.7) | ||
| Other | 49 (20.9) | 59 (25.2) | ||
| Alcohol (%) a | ||||
| Alcohol in the past 30 days | 130 (55.6) | 122 (51.1) | 0.37 d | |
| Alcohol use ≥ 3 days/week | 40 (17.1) | 35 (15.0) | 0.42 d | |
| Drugs (%) a | ||||
| Drugs in the past 30 days | 39 (16.7) | 36 (15.34) | 0.72 d | |
| Drugs use ≥ 3 days/week | 24 (10.3) | 23 (9.8) | 1.00 d | |
| Psychosocial variables (median, IQR) a,b | ||||
| Social support | 75 (40.6–90.6) | 68.8 (43.8–87.5) | 0.97 f | |
| Int. HIV-related stigma | 15 (12–19) | 14 (11–18) | <0.01 f | |
| Adherence self-efficacy | 105.5 (91.8–116.0) | 106 (89.8–115.0) | 0.93 f | |
| Quality of life (physical) | 51.3 (40.7–56.2) | 50.4 (41.7–55.6) | 0.36 f | |
| Quality of life (mental) | 48.9 (38.9–57.2) | 48.8 (37.8–56.3) | 0.84 f | |
| Self-reported adherence a,b | ||||
| Measure I c | Adherent | 81 (34.6) | 96 (41.0) | 0.07 d |
| Non-adherent | 151 (64.5) | 135 (57.7) | ||
| Measure II c | Adherent | 108 (46.2) | 121 (51.7) | 0.15 d |
| Non-adherent | 124 (53.0) | 110 (47.0) | ||
| Measure III c | Adherent | 116 (49.6) | 134 (57.3) | <0.05 d |
| Non-adherent | 116 (49.6) | 97 (41.5) | ||
a Missing values baseline questionnaire: sexual orientation (n = 3), children (n = 1), educational attainment (n = 2), alcohol (n = 1), drugs (n = 1), social support (n = 14), internalized HIV-related stigma (n = 11), adherence self-efficacy (n = 36), physical quality of life (n = 5), mental quality of life (n = 5), self-reported adherence (n = 2). b Missing values T1: alcohol (n = 3), drugs (n = 3), social support (n = 4), internalized HIV-related stigma (n = 8), adherence self-efficacy (n = 32), physical quality of life (n = 8), mental quality of life (n = 8), self-reported adherence (n = 3). c Measures present stricter (Measure I) and less strict (Measure II and III) criteria for adherence [13]. d McNemar test; e McNemar–Bowker test. f Paired—Samples t test.
3.2. Feasibility of the Interventions
Table 2 presents a summary of the interventions’ descriptions, feasibility, and limitations.
Table 2.
Summary of interventions’ descriptions, feasibility, and limitations.
| Intervention | In Short | Feasible? | Inclusion Limitations |
|---|---|---|---|
| DAART | Supervised cART intake during additional outpatient clinic appointments | no | • necessary time investment • eligible participants found it intrusive and impractical |
| GMA | Shared medical consultation with peers | no | Fears for: • disclosure of HIV status • third party disclosure • stigmatization |
| HADS and CIDI | Screening and treatment of psychological distress | HADS—yes CIDI—no |
In administering the CIDI: • language barriers • refusal to complete the questionnaire • logistical challenges For following up with psychiatrist: • refusing the appointment • no show |
| Peer support | Support by migrants living with HIV | yes | Fears for third party disclosure |
DAART, Directly Administered Antiretroviral Therapy; GMA, group medical appointments; HADS, Hospital Anxiety and Depression Scale; CIDI, Composite International Diagnostic Interview.
Directly Administered Antiretroviral Therapy (DAART): Unfortunately, no study participants were willing to take part in this intervention. They objected to the necessary investment of time for outpatient clinic visits. Furthermore, multiple study participants indicated to their HIV care provider that they found the intervention intrusive and impractical. Therefore, we ascertained that this intervention is currently not feasible for MLWH in The Netherlands.
Group Medical Appointment: Four GMAs were held, with a total of 27 participants, including one person participating two times. GMAs were open to all PLWH including MLWH. In each GMA, six to nine PLWH participated. HIV care providers found it difficult to find patients willing to participate in a GMA, especially MLWH. HIV care providers indicated that this was because many patients had concerns about revealing their HIV status to others and/or discussing their medical situation within a group. They feared third party disclosure and stigmatization. In addition, a number of eligible MLWH were not able to participate because the GMAs were held in Dutch and they did not speak Dutch. Of the 27 GMA participants, 13 were included in the ROAD project. While the threshold to participate in the GMA seemed to be high, participants were very positive. The mean overall rating given to the GMAs by participants was 8.2 out of 10. Additionally, 67% of the 27 participants would choose a GMA for their next outpatient clinic appointment, 78% felt better informed about HIV compared to an individual consultation, 93% indicated having learned from the questions asked by other PLWH, and 96% would recommend it to others. Unfortunately, we were not able to ascertain satisfaction specifically among MLWH, as the evaluation form was completed anonymously. Nonetheless, in light of the challenges experienced with participant inclusion, we conclude that intervention is not highly feasible in MLWH.
Screening and Treatment of Psychological Distress: In total, 306 of the 352 participants (87%) completed the HADS and 106 (35%) had a HADS+ indicating psychological distress [38]. The HADS thus seems to be feasible for administration in a multilingual population. However, the HADS only provides an indication of psychological distress in the past seven days and therefore should be followed up upon quickly. As previously reported [38], following up on participants with a HADS+ and having them complete the CIDI was challenging. The CIDI was completed by only 60 of the 106 (58%) eligible participants with a HADS+. This is possibly because it was only possible to administer the CIDI in Dutch and English. Follow-up was further hampered by participants refusing to complete the CIDI or not attending the appointment to do the CIDI, as well as other logistical challenges. We therefore conclude that the feasibility of administering the CIDI to a group of MLWH previously screened with the HADS is minimal. Among those who did complete the CIDI, 21 received a diagnosis of depression or anxiety disorder [38], six of whom were already enrolled in psychiatric care. Unfortunately, of the remaining 15 participants, seven either refused an appointment with a psychiatrist or did not attend the scheduled appointment. In the end, only eight participants met with a psychiatrist. Therefore, we were not able to determine the efficacy of this intervention.
Peer Support: Within the ROAD project, 23 MLWH predominantly originating from Sub Saharan Africa, the Caribbean, and Latin America (additional socio-demographic characteristics are presented in Supplementary Table S1) received peer support, and more than 20 peers received training to provide peer support. Not all participants to whom this intervention was offered agreed to participate, mostly due to fear of third party disclosure. Most matched patients found it pleasant to have a peer with a similar background (e.g., gender and/or country of origin and/or language), but some participants requested a peer with a Dutch country of origin, also because they feared third-party disclosure.
3.3. Efficacy of the Interventions
Peer Support: Of the 17 MLWH who were cART-experienced (six were cART-naïve or used cART < 6 months prior to inclusion), eight (47%) had an undetectable HIV RNA (<50 copies/mL) in the year before they were matched with a peer (Table 3). One and two years after the intervention, HIV RNA was undetectable in 10 (59%) (Table 3). The percentage of patients with HIV RNA < 400 copies/mL increased from 53% before the match to 77%, one and two years later. The percentage of patients who attended all outpatient clinic appointments increased from 48% one year before the match to 50% one year later and 68% two years later.
Table 3.
HIV RNA and outpatient clinic attendance.
| Variable | Match − 1 year | Match + 1 year | Match + 2 years |
|---|---|---|---|
| HIV-RNA a | (n = 17) | (n = 17) | (n = 17) |
| All values < 50 copies/mL (%) | 8 (47.1) | 10 (58.8) | 10 (58.8) |
| All values < 400 copies/mL (%) | 9 (52.9) | 13 (76.5) | 13 (76.5) |
| Outpatient clinic attendance | (n = 23) | (n = 22) b | (n = 22) b |
| Attended all appointments (%) | 11 (47.8) | 11 (50.0) | 15 (68.2) |
a On cART < 6 months prior to match. b Lost to follow up, n = 1.
Overall Effectiveness of the Interventions: In all patients who completed both the baseline and T1 questionnaires, detectable viral load (HIV RNA > 50 copies/mL) decreased from 10.2% to 6.8% (p = 0.12) (Table 1). Self-reported adherence percentages increased in the more conservative Measure I (34.6% to 41.0%, p < 0.10) and Measure II (64.2% to 51.7%, p = 0.15), and in the less conservative Measure III (49.6% to 57.3%, p < 0.05). The score for internalized HIV-related stigma decreased significantly (p < 0.05) between baseline and T0.
4. Discussion
In this study, we followed 234 MLWH over a period of approximately 2.5 years and assessed the feasibility, and, if feasible, the efficacy of directly administered antiretroviral therapy (DAART), group medical appointment (GMA), screening and treatment of psychological distress, and peer support for and by MLWH on improving HIV treatment adherence among MLWH. The results demonstrate that DAART and GMA are currently not feasible interventions within an MLWH population living in Western Europe. In addition, while screening of psychological distress with the HADS was feasible in this population, follow-up screening with the CIDI and subsequent linkage to psychiatric care services were not. Peer support for and by MLWH appeared to be a feasible intervention, and the results within a small intervention group on viral load and outpatient clinic attendance were promising.
Our finding that DAART was not feasible is in line with some literature showing poor feasibility for DAART interventions with other key populations of PLWH with a higher risk for non-adherence, such as people engaged in substance use and people experiencing homelessness [44,45]. Although we were aware of existing evidence that DAART had not worked for other key populations, we nonetheless wanted to offer this intervention to MLWH as MLWH may differ from the key populations in which DAART was previously applied. However, in our study, the participants found the intervention intrusive and impractical, which has also been described elsewhere [46]. It is possible that in another context, e.g., the participant’s home, this intervention would be more feasible among MLWH as this would obviate the need for the participant to travel to the clinic. Future studies should investigate the feasibility of this adapted version of the intervention among MLWH.
Our findings also showed that there were challenges in finding PLWH willing to participate in a GMA. At the same time, those who did participate appreciated the intervention. Similar to the findings in our study, Smets and Zantinge et al. both described recruitment challenges for GMAs and subsequent appreciation by those who did participate [30,47]. It thus appears that the feasibility of GMA in a population of MLWH is limited at this time. Fear of HIV-related stigma resulting from being recognized by other GMA participants and fear of third party disclosure appear to be key barriers to participation in GMAs for MLWH. Future research should investigate the role of HIV-related stigma of MLWH’s willingness to participate in GMA interventions. We contend that HIV-related stigma should first be addressed before GMAs can be used for improving treatment adherence among MLWH.
In line with our expectations, our findings show that the HADS is a relatively feasible tool to administer in an outpatient clinical setting with a multicultural and multilingual population. Our findings also show poor feasibility for the administration of the CIDI as there were significant challenges in getting participants with a HADS+ to complete the CIDI. It is possible that CIDI completion rates may have been impeded by the fact that the CIDI is not available in multiple languages. This has been indicated earlier by Schrier et al. [48]. We therefore suggest that when psychological care is indicated, a diagnostic assessment be used that takes cultural differences into account.
Peer support seems to be the most feasible intervention that was tested in this study. Despite the small sample size, the results on HIV RNA and outpatient clinical appointments were promising. Similar results were found in a previously published systematic review and in a network meta-analysis performed in middle- and low-income countries [26,28]. Both Kanters et al. and Mills et al. found positive effects of peer support on HIV treatment adherence. In addition, when compared to standard care, peer support interventions were found to be superior in reaching viral suppression [28]. Mixed results were shown in two peer intervention studies performed in the United States of America [49,50]. This was partially attributed to the relatively short duration of the interventions (less than three months). Additionally, a qualitative study conducted with participants in our peer support project showed that peer support was reported to have increased patients’ knowledge about (living with) HIV, helped patients to feel emotionally supported, reduced perceived HIV-related stigma, and reduced disclosure concerns [51]. Additionally, that study also showed that the peers found the support they provided meaningful and a source of personal growth. Furthermore, the HIV care providers involved in this study saw a need for this intervention as multiple requests were made to match PLWH who were eligible but not enrolled in the ROAD project. We therefore believe that peer support for and by MLWH to improve HIV treatment adherence has potential, particularly for MLWH who originate from collectivist cultures where peer interaction and support provision within communities is highly valued. We recommend that future studies investigate this intervention on a larger scale than was possible in this study to better determine the effect on psychosocial and clinical outcomes.
This study has a number of strengths and limitations. One key strength is that we successfully included and followed up on 234 cART-experienced MLWH, which is a difficult-to-reach key population. To our knowledge, this is the largest sample of MLWH recruited for a study on testing the feasibility and possible efficacy of interventions aiming to improve HIV treatment adherence in Europe to date [28]. A second strength is that, rather than testing a single intervention, we used our resources to simultaneously test the feasibility and possible efficacy of multiple existing interventions in this specific sample. A limitation is that the sample size in the peer support intervention was relatively small, making the analyses possibly quite limited. Future studies should be designed with sufficient power to be able to determine the effectiveness of peer support. A second limitation is that we did not have a control group, e.g., MLWH who received standard care. However, we performed a pilot study aiming to determine the feasibility and possible efficacy of four interventions aimed to improve treatment adherence. In that pilot study, peer support was a feasible intervention. Nonetheless, future studies should use an intervention and control group to test the efficacy of this intervention. Therefore, we were not able to demonstrate that the overall differences found in detectable HIV RNA, experienced HIV-related stigma, and self-reported adherence at baseline and T1 were due to the interventions that were administered. It is possible that these results are, to some extent, the consequence of standard care or, alternatively, the Hawthorne effect [52]. It is possible that the focus on HIV treatment adherence with the administration of the baseline questionnaire for study inclusion made study participants (perhaps unconsciously) more aware of their HIV treatment adherence behavior and made health care providers more keen to encourage treatment adherence. However, we maintain that being offered and participating in several interventions to improve adherence likely changed adherence behavior, particularly in MLWH, many of whom had not or only selectively disclosed their HIV status to others.
In their systematic review, Whembolua et al. [53] found that disclosure of HIV status is related to decreased anxiety and emotional relief, increased social support, decreased high-risk sexual behaviors, increased help- and information-seeking behaviors, and, importantly, increased adherence to cART. A final limitation is that we did not use focus groups with the MLWH population to ascertain the acceptability of the interventions before implementing the interventions. Future studies aiming to test interventions should ideally gauge the acceptance of proposed interventions in the MLWH population beforehand. This can be done by organizing focus groups or by including key figures from the migrants’ communities in the research team, using a community-based participatory research approach [54,55].
In conclusion, our findings provide evidence for the feasibility, or lack thereof, of DAART, GMA, screening and treatment of psychological distress, and peer support for and by MLWH as interventions improving HIV treatment adherence. Of the four interventions, peer support appeared to be a feasible intervention, and the results on HIV RNA and outpatient clinic visit attendance were promising. Future studies should more extensively, and in a larger sample, evaluate peer support to further determine the effect on both psychosocial and clinical outcomes. Additionally, continuing efforts to assess interventions that aim to improve HIV treatment adherence among MLWH are recommended. In doing so, we recommend including key figures from migrant (living with HIV) communities in the development of such interventions to ensure that interventions are culturally sensitive and likely to be accepted by MLWH.
Acknowledgments
The authors thank the participants for their time and willingness to contribute to this study. We also thank Inês Brito (Dutch Association of People Living with HIV and Hello Gorgeous Foundation) and Mireille E.G. Wolfers (GGD Rotterdam) for their contributions and input when this project was conceived and initiated. In addition, we thank the physicians and nurses at Erasmus University Medical Centre and Maasstad Hospital, and the members of the interview team, for their efforts during the inclusion period. Lastly, we thank our colleagues from the ATHENA observational HIV cohort for contributing to the data collection.
Supplementary Materials
The following are available online at https://www.mdpi.com/2075-4418/10/11/980/s1, Table S1: Sociodemographic characteristics of patients participating in the peer support intervention.
Author Contributions
Conceptualization, S.K.B. and A.V.; Data curation, J.S., N.B. and K.P.; Formal analysis, S.K.B., D.A.M.C.v.d.V., S.E.S. and A.V.; Investigation, K.P.; Methodology, S.K.B., D.A.M.C.v.d.V., S.E.S. and A.V.; Supervision, A.V.; Validation, S.K.B.; Writing–original draft, S.K.B. and A.V.; Writing–review & editing, D.A.M.C.v.d.V., J.S., N.B., K.P. and S.E.S. All authors have read and agreed to the published version of the manuscript.
Funding
This study was supported by a grant from the Dutch Aids Fonds (grant number: 2011107).
Conflicts of Interest
The authors declare no conflict of interest. The institution of A.V. and S.K.B. has received an unrestricted scientific grant from the Dutch Aids Fonds to perform the study.
Footnotes
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.May M.T., Gompels M., Delpech V., Porter K., Orkin C., Kegg S., Hay P., Johnson M., Palfreeman A., Gilson R., et al. Impact on life expectancy of HIV-1 positive individuals of CD4+ cell count and viral load response to antiretroviral therapy. AIDS. 2014;28:1193–202. doi: 10.1097/QAD.0000000000000243. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Trickey A., May M.T., Vehreschild J.-J., Obel N., Gill M.J., Crane H.M., Boesecke C., Patterson S., Grabar S., Cazanave C., et al. Survival of HIV-positive patients starting antiretroviral therapy between 1996 and 2013: A collaborative analysis of cohort studies. Lancet HIV. 2017;4:e349–e356. doi: 10.1016/S2352-3018(17)30066-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Langebeek N., Gisolf E.H., Reiss P., Vervoort S.C., Hafsteinsdottir T.B., Richter C., Sprangers M.A.G., Nieuwkerk P.T. Predictors and correlates of adherence to combination antiretroviral therapy (ART) for chronic HIV infection: A meta-analysis. BMC Med. 2014;12:142. doi: 10.1186/s12916-014-0142-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mannheimer S.B., Matts J., Telzak E., Chesney M., Child C., Wu A.W., Friedland G. Quality of life in HIV-infected individuals receiving antiretroviral therapy is related to adherence. AIDS Care. 2005;17:10–22. doi: 10.1080/09540120412331305098. [DOI] [PubMed] [Google Scholar]
- 5.Wasti S.P., Simkhada P., Randall J., Freeman J.V., Van Teijlingen E. Factors influencing adherence to antiretroviral treatment in Nepal: A mixed-methods study. PLoS ONE. 2012;7:e35547. doi: 10.1371/journal.pone.0035547. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Van Andel E., Been S.K., Rokx C., Van der Ende M.E. Risk factors in an HIV-infected population for refraining from specialist care. AIDS Care. 2016;28:1255–1260. doi: 10.1080/09540121.2016.1168914. [DOI] [PubMed] [Google Scholar]
- 7.European Centre for Disease Prevention and Control (ECDC) World Health Organization (WHO) HIV/AIDS Surveillance in Europe 2018. European Centre for Disease Prevention and Control; Copenhagen, Denmark: 2018. [(accessed on 17 January 2019)]. Available online: https://ecdc.europa.eu/sites/portal/files/documents/hiv-aids-surveillance-europe-2018.pdf. [Google Scholar]
- 8.Monge S., Alejos B., Dronda F., Del Romero J., Iribarren J.A., Pulido F., Rubio R., Miró J.M., Gutierrez F., del Amo J., et al. Inequalities in HIV disease management and progression in migrants from Latin America and sub-Saharan Africa living in Spain. HIV Med. 2013;14:273–283. doi: 10.1111/hiv.12001. [DOI] [PubMed] [Google Scholar]
- 9.Nellen J.F.J.B., Nieuwkerk P.T., Burger D.M., Wibaut M., Gras L.A., Prins J.M. Which method of adherence measurement is most suitable for daily use to predict virological failure among immigrant and non-immigrant HIV-1 infected patients? AIDS Care. 2009;21:842–850. doi: 10.1080/09540120802612816. [DOI] [PubMed] [Google Scholar]
- 10.Staehelin C., Keiser O., Calmy A., Weber R., Elzi L., Cavassini M., Schmid P., Bernasconi E., Furrer H., Swiss HIV Cohort Study Longer term clinical and virological outcome of sub-Saharan African participants on antiretroviral treatment in the Swiss HIV Cohort Study. J. Acquir. Immune Defic. Syndr. 2012;59:79–85. doi: 10.1097/QAI.0b013e318236be70. [DOI] [PubMed] [Google Scholar]
- 11.Sumari-de Boer I.M., Sprangers M.A., Prins J.M., Nieuwkerk P.T. HIV stigma and depressive symptoms are related to adherence and virological response to antiretroviral treatment among immigrant and indigenous HIV infected patients. AIDS Behav. 2012;16:1681–1689. doi: 10.1007/s10461-011-0112-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Ross J., Cunningham C.O., Hanna D.B. HIV outcomes among migrants from low-income and middle-income countries living in high-income countries: A review of recent evidence. Curr. Opin. Infect. Dis. 2018;31:25–32. doi: 10.1097/QCO.0000000000000415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Been S.K., van de Vijver D.A., Nieuwkerk P.T., Brito I., Stutterheim S.E., Bos A.E., Wolfers M.E.G., Pogány K., Verbon A. Risk Factors for Non-Adherence to cART in Immigrants with HIV Living in The Netherlands: Results from the ROtterdam ADherence (ROAD) Project. PLoS ONE. 2016;11:e0162800. doi: 10.1371/journal.pone.0162800. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.World Health Organization (WHO) The End TB Strategy Global Strategy and Targets for Tuberculosis Prevention, Care and Control after 2015. [(accessed on 17 January 2019)];2014 Available online: http://apps.who.int/gb/ebwha/pdf_files/EB134/B134_12-en.pdf?ua=1.
- 15.Noffsinger E. Will Drop-In Group Medical Appointments (DIGMAs) Work in Practice? Perm. J. 1999;3:58–67. [Google Scholar]
- 16.Kirsh S.R., Aron D.C., Johnson K.D., Santurri L.E., Stevenson L.D., Jones K.R., Jagosh J. A realist review of shared medical appointments: How, for whom, and under what circumstances do they work? BMC Health Serv. Res. 2017;17:113. doi: 10.1186/s12913-017-2064-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hayhoe B., Verma A., Kumar S. Shared medical appointments. Br. Med. J. 2017;358:j4034. doi: 10.1136/bmj.j4034. [DOI] [PubMed] [Google Scholar]
- 18.Uthman O.A., Magidson J.F., Safren S.A., Nachega J.B. Depression and adherence to antiretroviral therapy in low-, middle- and high-income countries: A systematic review and meta-analysis. Curr. HIV/AIDS Rep. 2014;11:291–307. doi: 10.1007/s11904-014-0220-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Todd J.V., Cole S.R., Pence B.W., Lesko C.R., Bacchetti P., Cohen M.H., Feaster D.J., Gange S., Griswold M.E., Mack W., et al. Effects of Antiretroviral Therapy and Depressive Symptoms on All-Cause Mortality Among HIV-Infected Women. Am. J. Epidemiol. 2017;185:869–878. doi: 10.1093/aje/kww192. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Brandt C., Zvolensky M.J., Woods S.P., Gonzalez A., Safren S.A., O’Cleirigh C.M. Anxiety symptoms and disorders among adults living with HIV and AIDS: A critical review and integrative synthesis of the empirical literature. Clin. Psychol. Rev. 2017;51:164–184. doi: 10.1016/j.cpr.2016.11.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Robertson K., Bayon C., Molina J.M., McNamara P., Resch C., Munoz-Moreno J.A., Kulasegaram R., Schewe K., Burgos-Ramirez A., de Alvaro C., et al. Screening for neurocognitive impairment, depression, and anxiety in HIV-infected patients in Western Europe and Canada. AIDS Care. 2014;26:1555–1561. doi: 10.1080/09540121.2014.936813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.de Wit M.A., Tuinebreijer W.C., Dekker J., Beekman A.J., Gorissen W.H., Schrier A.C., Penninx B.W.J.H., Komproe I.H., Verhoeff A.P. Depressive and anxiety disorders in different ethnic groups: A population based study among native Dutch, and Turkish, Moroccan and Surinamese migrants in Amsterdam. Soc. Psychiatry Psychiatr. Epidemiol. 2008;43:905–912. doi: 10.1007/s00127-008-0382-5. [DOI] [PubMed] [Google Scholar]
- 23.Dennis C. Peer support within a health care context: A concept analysis. Int. J. Nurs. Stud. 2003;40:321–332. doi: 10.1016/S0020-7489(02)00092-5. [DOI] [PubMed] [Google Scholar]
- 24.Simoni J.M., Franks J.C., Lehavot K., Yard S.S. Peer interventions to promote health: Conceptual considerations. Am. J. Orthopsychiatry. 2011;81:351–359. doi: 10.1111/j.1939-0025.2011.01103.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Sokol R., Fisher E. Peer Support for the Hardly Reached: A Systematic Review. Am. J. Public Health. 2016;106:e1–e8. doi: 10.2105/AJPH.2016.303180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Mills E.J., Lester R., Thorlund K., Lorenzi M., Muldoon K., Kanters S., Linnemayr S., Gross R., Calderone Y., Amico K.R., et al. Interventions to promote adherence to antiretroviral therapy in Africa: A network meta-analysis. Lancet HIV. 2014;1:e104–e111. doi: 10.1016/S2352-3018(14)00003-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Igumbor J.O., Scheepers E., Ebrahim R., Jason A., Grimwood A. An evaluation of the impact of a community-based adherence support programme on ART outcomes in selected government HIV treatment sites in South Africa. AIDS Care. 2011;23:231–236. doi: 10.1080/09540121.2010.498909. [DOI] [PubMed] [Google Scholar]
- 28.Kanters S., Park J.J., Chan K., Socias M.E., Ford N., Forrest J.I., Thorlund K., Nachega J.B., Mills E.J. Interventions to improve adherence to antiretroviral therapy: A systematic review and network meta-analysis. Lancet HIV. 2017;4:e31–e40. doi: 10.1016/S2352-3018(16)30206-5. [DOI] [PubMed] [Google Scholar]
- 29.Zigmond A.S., Snaith R.P. The hospital anxiety and depression scale. Acta Psychiatr. Scand. 1983;67:361–370. doi: 10.1111/j.1600-0447.1983.tb09716.x. [DOI] [PubMed] [Google Scholar]
- 30.Smets E. Gooi Het in de Groep: Medisch Contact. [(accessed on 19 January 2019)];2009 Available online: https://www.medischcontact.nl/nieuws/laatste-nieuws/artikel/gooi-het-in-de-groep.htm.
- 31.PROQOLID Hospital Anxiety and Depression Scale (HADS): Mapi Research Trust. [(accessed on 19 January 2019)]; Available online: https://eprovide.mapi-trust.org/instruments/hospital-anxiety-and-depression-scale.
- 32.Ibbotson T., Maguire P., Selby P., Priestman T., Wallace L. Screening for anxiety and depression in cancer patients: The effects of disease and treatment. Eur. J. Cancer. 1994;30A:37–40. doi: 10.1016/S0959-8049(05)80015-2. [DOI] [PubMed] [Google Scholar]
- 33.Kugaya A., Akechi T., Okuyama T., Nakano T., Mikami I., Okamura H., Uchitomi Y. Prevalence, predictive factors, and screening for psychologic distress in patients with newly diagnosed head and neck cancer. Cancer. 2000;88:2817–2823. doi: 10.1002/1097-0142(20000615)88:12<2817::AID-CNCR22>3.0.CO;2-N. [DOI] [PubMed] [Google Scholar]
- 34.Roth A.J., Kornblith A.B., Batel-Copel L., Peabody E., Scher H.I., Holland J.C. Rapid screening for psychologic distress in men with prostate carcinoma: A pilot study. Cancer. 1998;82:1904–1908. doi: 10.1002/(SICI)1097-0142(19980515)82:10<1904::AID-CNCR13>3.0.CO;2-X. [DOI] [PubMed] [Google Scholar]
- 35.Booth B.M., Kirchner J.E., Hamilton G., Harrell R., Smith G.R. Diagnosing depression in the medically ill: Validity of a lay-administered structured diagnostic interview. J. Psychiatr. Res. 1998;32:353–360. doi: 10.1016/S0022-3956(98)00031-4. [DOI] [PubMed] [Google Scholar]
- 36.Robins L.N., Wing J., Wittchen H.U., Helzer J.E., Babor T.F., Burke J., Farmer A., Jablenski A., Pickens R., Regier D.A., et al. The Composite International Diagnostic Interview. An epidemiologic Instrument suitable for use in conjunction with different diagnostic systems and in different cultures. Arch. Gen. Psychiatry. 1988;45:1069–1077. doi: 10.1001/archpsyc.1988.01800360017003. [DOI] [PubMed] [Google Scholar]
- 37.World Health Organization (WHO) The World Health Organization World Mental Health Composite International Diagnostic Interview (WHO WMH-CIDI): World Health Organization. [(accessed on 19 January 2019)];2017 Available online: https://www.hcp.med.harvard.edu/wmhcidi/
- 38.Been S.K., Schade A., Bassant N., Kastelijns M., Pogany K., Verbon A. Anxiety, depression and treatment adherence among HIV-infected migrants. AIDS Care. 2019;31:979–987. doi: 10.1080/09540121.2019.1601676. [DOI] [PubMed] [Google Scholar]
- 39.Peterson J.L., Rintamaki L.S., Brashers D.E., Goldsmith D.J., Neidig J.L. The forms and functions of peer social support for people living with HIV. J. Assoc. Nurses AIDS Care. 2012;23:294–305. doi: 10.1016/j.jana.2011.08.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Moser A., Stuck A.E., Silliman R.A., Ganz P.A., Clough-Gorr K.M. The eight-item modified Medical Outcomes Study Social Support Survey: Psychometric evaluation showed excellent performance. J. Clin. Epidemiol. 2012;65:1107–1116. doi: 10.1016/j.jclinepi.2012.04.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Kalichman S.C., Simbayi L.C., Cloete A., Mthembu P.P., Mkhonta R.N., Ginindza T. Measuring AIDS stigmas in people living with HIV/AIDS: The Internalized AIDS-Related Stigma Scale. AIDS Care. 2009;21:87–93. doi: 10.1080/09540120802032627. [DOI] [PubMed] [Google Scholar]
- 42.Johnson M.O., Neilands T.B., Dilworth S.E., Morin S.F., Remien R.H., Chesney M.A. The role of self-efficacy in HIV treatment adherence: Validation of the HIV Treatment Adherence Self-Efficacy Scale (HIV-ASES) J. Behav. Med. 2007;30:359–370. doi: 10.1007/s10865-007-9118-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Ware J., Jr., Kosinski M., Keller S.D. A 12-Item Short-Form Health Survey: Construction of scales and preliminary tests of reliability and validity. Med. Care. 1996;34:220–233. doi: 10.1097/00005650-199603000-00003. [DOI] [PubMed] [Google Scholar]
- 44.Ford N., Nachega J.B., Engel M.E., Mills E.J. Directly observed antiretroviral therapy: A systematic review and meta-analysis of randomised clinical trials. Lancet. 2009;374:2064–2071. doi: 10.1016/S0140-6736(09)61671-8. [DOI] [PubMed] [Google Scholar]
- 45.Hart J.E., Jeon C.Y., Ivers L.C., Behforouz H.L., Caldas A., Drobac P.C., Shin S.S. Effect of directly observed therapy for highly active antiretroviral therapy on virologic, immunologic, and adherence outcomes: A meta-analysis and systematic review. J. Acquir. Immune Defic. Syndr. 2010;54:167–179. doi: 10.1097/QAI.0b013e3181d9a330. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Santos C.Q., Adeyemi O., Tenorio A.R. Attitudes toward directly administered antiretroviral therapy (DAART) among HIV-positive inpatients in an inner city public hospital. AIDS Care. 2006;18:808–811. doi: 10.1080/09540120500448513. [DOI] [PubMed] [Google Scholar]
- 47.Zantinge E., Seesing F., Tol F., Raats C., Spreeuwenberg P., van Dulmen A. ‘Gezamenlijk Medisch Consult’: Samen naar de dokter—Ervaringen van patiënten en zorgverleners. Nederlands Tijdschrift voor Geneeskunde. 2009;153:A828. [PubMed] [Google Scholar]
- 48.Schrier A.C., de Wit M.A., Rijmen F., Tuinebreijer W.C., Verhoeff A.P., Kupka R.W., Dekker J., Beekman A.T.F. Similarity in depressive symptom profile in a population-based study of migrants in The Netherlands. Soc. Psychiatry Psychiatr. Epidemiol. 2010;45:941–951. doi: 10.1007/s00127-009-0135-0. [DOI] [PubMed] [Google Scholar]
- 49.Simoni J.M., Pantalone D.W., Plummer M.D., Huang B. A randomized controlled trial of a peer support intervention targeting antiretroviral medication adherence and depressive symptomatology in HIV-positive men and women. Health Psychol. 2007;26:488–495. doi: 10.1037/0278-6133.26.4.488. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Giordano T.P., Cully J., Amico K.R., Davila J.A., Kallen M.A., Hartman C., Wear J., Buscher A., Stanley M. A Randomized Trial to Test a Peer Mentor Intervention to Improve Outcomes in Persons Hospitalized with HIV Infection. Clin. Infect. Dis. 2016;63:678–686. doi: 10.1093/cid/ciw322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Been S.K., Aljabouri H., Stutterheim S.E., Van Dorp H., Joziasse H., Smit J., Bassant N., Verbon A. Impact of peer support for and by migrants living with HIV in The Netherlands. 2019. submitted.
- 52.Sedgwick P., Greenwood N. Understanding the Hawthorne effect. BMJ. 2015;351:h4672. doi: 10.1136/bmj.h4672. [DOI] [PubMed] [Google Scholar]
- 53.Whembolua G.L., Conserve D.F., Thomas K., Handler L. A Systematic Review of HIV Serostatus Disclosure Among African Immigrants in Europe. J. Immigr. Minor. Health. 2017;19:947–958. doi: 10.1007/s10903-016-0456-5. [DOI] [PubMed] [Google Scholar]
- 54.Rhodes S.D., Malow R.M., Jolly C. Community-based participatory research: A new and not-so-new approach to HIV/AIDS prevention, care, and treatment. AIDS Educ. Prev. 2010;22:173–183. doi: 10.1521/aeap.2010.22.3.173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Lessard D., Engler K., Toupin I., Team I.S.C., Routy J.P., Lebouche B. Evaluation of a project to engage patients in the development of a patient-reported measure for HIV care (the I-Score Study) Health Expect. 2019;22:209–225. doi: 10.1111/hex.12845. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.