Abstract
With the expansion of telehealth services, there is a need for evidence-based treatment adherence interventions that can be delivered remotely to people living with HIV. Evidence-based behavioral health counseling can be delivered via telephone, as well as in-office services. However, there is limited research on counseling delivery formats and their differential outcomes. The purpose of this study was to conduct a head-to-head comparison of behavioral self-regulation counseling delivered by telephone versus behavioral self-regulation counseling delivered by in-office sessions to improve HIV treatment outcomes. Patients (N = 251) deemed at risk for discontinuing care and treatment failure living in a rural area of the southeastern USA were referred by their care provider. The trial implemented a Wennberg Randomized Preferential Design to rigorously test: (a) patient preference and (b) comparative effects on patient retention in care and treatment adherence. There was a clear patient preference for telephone-delivered counseling (69%) over in-office-delivered counseling (31%) and participants who received telephone counseling completed a greater number of sessions. There were few differences between the two intervention delivery formats on clinical appointment attendance, antiretroviral adherence, and HIV viral load. Overall improvements in health outcomes were not observed across delivery formats. Telephone-delivered counseling did show somewhat greater benefit for improving depression symptoms, whereas in-office services demonstrated greater benefits for reducing alcohol use. These results encourage offering most patients the choice of telephone and in-office behavioral health counseling and suggest that more intensive interventions may be needed to improve clinical outcomes for people living with HIV who may be at risk for discontinuing care or experiencing HIV treatment failure.
Keywords: Patient preference trial, HIV treatment and care, Rural health, Telehealth
Implications.
Practice: Adapting in-office counseling for telephone delivery may improve the reach of clinical services to patients in the greatest need and may come with little cost in terms of impact.
Policy: Funding and reimbursement for telephone-delivered counseling may come with cost savings and expanded reach to patients in rural areas.
Research: Comparative effectiveness trials are needed to determine the optimal delivery formats for various models of behavioral interventions for people living with HIV.
Antiretroviral therapy (ART) effectively treats HIV infection by suppressing viral activity, ultimately extending lives and slowing the spread of HIV. Unfortunately, not all people living with HIV are benefiting from HIV treatment, with more than one in three people receiving ART in clinical care not demonstrating durable HIV suppression [1]. Among the reasons for these missed opportunities are the individual, social, and structural challenges many people living with HIV face in achieving and sustaining treatment engagement and ART adherence [2]. Several factors contribute to treatment dropout and incomplete ART adherence, most of which are amenable to behavioral interventions. For example, alcohol use is among the most robust predictors of nonadherence to ART that can be addressed by behavioral counseling. Studies show temporal and dose–response relationships between alcohol consumption and ART adherence [3], indicating that even limited or occasional use of alcohol impedes ART adherence [4]. Modest reductions in alcohol use may, therefore, have health benefits for people living with HIV.
Behavioral self-regulation counseling offers an intervention model that directly addresses barriers to HIV care, including alcohol use, by drawing on multiple concepts from social, cognitive, motivational, and behavioral interventions [5,6]. Self-regulation counseling builds behavioral skills, problem-solves structural barriers, and provides adherence support [7], including addressing challenges presented by mental health and substance use [8,9]. Randomized trials have demonstrated that behavioral self-regulation counseling significantly improves ART adherence and HIV suppression [10], with several such models widely disseminated [8,9]. These interventions work with patients through a series of skill-building and problem-solving exercises designed to manage individual, social, and structural challenges [11].
Behavioral self-regulation counseling can be formatted for delivery in telephone sessions [12,13] or in-office clinic-based sessions [14]. Both telephone and in-office counseling services have demonstrated relative advantages and disadvantages for multiple health conditions. Telephone services are of particular interest for people living in rural settings, where HIV care is often sparse and inaccessible [15]. Telephone-delivered counseling also reduces patient burden, avoids social barriers to care, including stigma, and can be cost saving [16]. The advent of the COVID-19 pandemic has brought a new sense of urgency for telehealth services [17,18]. Unfortunately, efforts to scale up telehealth services, including telephone-delivered counseling, for people living with HIV in the midst of COVID-19 lacked an evidence basis and infrastructure [19]. On the other hand, in-office counseling has the advantage of being delivered during routine clinical care, as well as within a broader array of essential clinic-based services. Previous research has reported the relative advantages and disadvantages of telephone versus in-office services [20]. However, we are not aware of previous studies that have performed a head-to-head comparison of telephone versus in-office behavioral counseling to improve HIV treatment outcomes.
Here, we report the results of the first comparison of telephone versus in-office behavioral self-regulation counseling to improve HIV treatment outcomes for people living with HIV. The counseling model takes a skills-building approach to resolve barriers and challenges to care retention and ART adherence by integrating behavioral self-regulation counseling [11,12] with motivational interviewing [21]. Previous clinical trials offer evidence for the efficacy of behavioral self-regulation counseling [10,11], with inclusion in the Centers for Disease Control and Prevention’s (CDC) Compendium of Evidence-Based Interventions [22]. Our trial was designed as a head-to-head comparison of telephone versus in-office counseling, with a direct test of patient preferences and patient characteristics related to preferences. The research was conducted in a rural area of the southeastern USA with patients deemed by their providers to be at risk for treatment discontinuation or treatment failure.
The current trial tested the effects of behavioral self-regulation counseling on retention in care (primary outcome), HIV medication adherence and HIV viral load (secondary outcomes), depression, and alcohol use (other outcomes). We had four main hypotheses: (a) when given the choice, participants would be more likely to choose telephone-delivered counseling, and telephone counseling would yield a greater number of counseling sessions completed; (b) both counseling formats would demonstrate improvements in primary and secondary outcomes; (c) participants given the choice of counseling formats would demonstrate greater improvements in outcomes than those not given a choice; and (d) participants who received a greater number of counseling sessions would demonstrate greater improvements in outcomes; because of the greater number of sessions when delivered by telephone, there would be Counseling Delivery Format × Dose of Counseling interactions on outcomes, with participants receiving a greater number of telephone sessions demonstrating more positive outcomes.
METHODS
Participants
Participants were recruited from a publicly funded HIV clinic in central Georgia between October 2015 and February 2017 through targeted convenience sampling. During a scheduled office visit, clinic patients were referred to the study and invited to participate. Specifically, men and women receiving HIV care were referred by their physician or nurse if they were: (a) new to the clinic, (b) newly diagnosed with HIV, or (c) identified as either having unsuppressed HIV or nonadherent to ART. A total of 375 patients were invited to participate, with 251 attending the enrollment appointment, yielding a 67% acceptance rate. The most common reason for declining participation was competing time obligations. After providing written informed consent, participants completed an audio computer-assisted self-interview (ACASI) followed by 12 monthly phone assessments, including unannounced pill counts and periodic text message assessments. Participants were compensated for completing assessments totaling a possible $580. The trial was registered at clinicaltrials.gov (NCT04180280) and was approved by the University of Connecticut and Mercer University School of Medicine institutional review boards.
Trial design
We used a comparative effectiveness framework to evaluate: (a) patient preferences for telephone versus in-office behavioral counseling delivery and (b) comparative outcomes [23]. The Wennberg Randomized Preferential Design was selected for this trial to simultaneously and rigorously test patient preferences and comparative outcomes [24]. The allocation of participants occurred in two steps (see Fig. 1). In Step 1, participants were randomly allocated to either (a) randomized assignment or (b) patient preference (choice). In Step 2, participants who were allocated to the randomized assignment condition were then randomly assigned to either telephone or in-office counseling, whereas participants allocated to the patient preference condition were given the choice of receiving telephone or in-office counseling. The Wennberg Design as implemented in this trial was, therefore, a modified 2 (randomized vs. preference) × 2 (telephone delivery vs. in office) complete factorial.
Fig 1.
Flow of participants through the two-stage Wennberg patient preference design trial. Note: R = randomized, C = Choice.
Behavioral self-regulation counseling
As developed and tested in previous research [5,10–12,25,26], behavioral self-regulation counseling emphasizes skills and strategies for health monitoring, identifying and resolving structural barriers, cognitive appraisal, stress management, and behavioral control [27]. Unlike curriculum-based health education, behavioral self-regulation counseling uses guides to provide semistructured session content driven by patient needs. This approach is conceptually similar to other interventions grounded in client-centered counseling [28]. Treatment engagement and adherence were conceived as being directly influenced by care expectancies, social interactions, and cognitive/affective processes. The model places significant emphasis on developing cognitive-behavioral problem-solving skills to identify personal and social resources, managing social, environmental, and structural barriers to care, and maintaining treatment adherence.
The first session of counseling commenced with an evaluation of care engagement and ART adherence. For participants who were not yet receiving ART, the focus of counseling was on evaluating individual, interpersonal, and structural barriers to care and increasing health care engagement to initiate ART. The counselor conducted a detailed behavioral and environmental evaluation to formulate a personalized care engagement and adherence plan. Participants worked with the counselor to identify barriers to care and taking ART, such as stigma, transportation, side effects, depression, and access to food. An entire component of the first session was dedicated to assessing substance use, particularly alcohol use, and its potential impact on HIV care.
The remaining five counseling sessions were semistructured and designed to meet the participant at their point-of-need identified in Session 1. In each subsequent session, the counselor delivered immediate feedback and probed participants regarding individual, interpersonal, and environment–structural challenges to their HIV treatment. The counseling linked participants to mental health services, substance use treatment, food banks, and other local resources when needed. The final session terminated with an individualized plan to sustain progress made on goals for persistent treatment engagement and ART adherence. A detailed description of the intervention is available from the authors.
Experimental conditions
Participants received either telephone- or in-office-delivered counseling. The number of counseling sessions and intervention content did not differ for telephone and in-office counseling, with only the delivery format altered. All counseling conditions were delivered by three female bachelor’s degree health counselors trained in the intervention protocol. Counselors received weekly group supervision with the project manager to process sessions and retain fidelity.
Randomization and blinding
Following the baseline assessment, participants were randomly assigned to study conditions using a randomization generator (http://www.randomization.com). We generated a two equal size conditions block/group randomization scheme allocating participants to either: preferences for counseling format or random assignment to counseling format. Participants allocated to the preference condition were allowed to choose between telephone or in-office counseling. Participants allocated to the random assignment condition underwent a second-stage block/group randomization scheme that assigned them to either telephone or in-office counseling. Recruitment, screening, and assessment staff were blinded to condition.
Measures
Participants completed informed consent and a baseline ACASI for demographic and health characteristics [29]. All follow-up assessments were conducted using monthly phone interviews and periodic daily interactive text messaging assessments. Retention in care and HIV viral load were extracted from medical records. We also monitored ART adherence services delivered by nurses during the course of their clinical care using a brief postclinical consultation nurses checklist.
Nurse-delivered adherence services
Nurses completed checklists of adherence services delivered during their routine contacts with patients over the course of 18 months of participant recruitment. The checklist included 20 adherence support services that have been identified in past research describing the standard of care in HIV adherence services [30]. Checklists were completed anonymously with an estimated 80% completion rate.
Participant characteristics and baseline interviews
We collected participant demographic and health characteristics, including gender, age, years of education, race, income, and employment status. We assessed depression symptoms with the Centers for Epidemiological Studies Depression (CESD) scale [31], α = .87, and alcohol use with the Alcohol Use Disorders Identification Test (AUDIT), a 10-item scale designed to measure alcohol consumption and identify risks for alcohol abuse and dependence [32]. HIV-related stress was assessed by participants responding as to whether each of 18 life events had occurred in the previous month, with scores representing the number of stressors experienced, α = .65. To assess attitudes toward taking ART, we administered the Beliefs About Medicines Questionnaire [33], consisting of medication necessity beliefs, reflecting the perceived benefits of medications in direct relation to health, α = .80, and medication concerns beliefs, reflecting the potential adverse effects and costs of medications, α = .83. Physical and mental health quality of life was assessed with the SF-36, a widely used measure of functional health and well-being from the patient’s perspective [34]. We also administered an adaptation of the Utilization of Care module of the HIV Cost and Services Utilization Study [35]. ArcGIS software was used to calculate the distances in miles from the clinic to where participants resided.
Primary outcome: engagement in care
To assess engagement in care, we followed the Institute of Medicine and The Health Resources and Services Administration-HIV Bureau definition by assessing clinical appointments scheduled and completed to calculate appointment adherence, visit constancy (defined by consistent appointment keeping), and treatment disengagement (i.e., discontinuing care) [36]. We coded patient records to capture medical appointments over an 18 month period to assure full coverage over the 12 month study period. Scheduled clinical visits were coded as to whether patients did or did not attend the appointment, with nonattendance further coded as a patient canceling/rescheduling or the patient not attending the appointment without canceling/rescheduling. Disengagement from care was defined by the clinic as a patient discontinuing care, typically as a result of not attending at least one scheduled medical appointment in a 6 month period.
Secondary outcomes: ART adherence and HIV viral load
Participants completed monthly telephone assessments that included unannounced pill counts, which have been demonstrated reliable and valid in assessing HIV treatment adherence [37,38]. Unannounced pill counts conducted over the telephone require counting ability but do not require mental calculation. This procedure has been demonstrated valid in people with low health literacy [39]. Following the training in pill counting procedures at the initial enrollment appointment, participants were called each of the next 12 months at an unscheduled time. Pill counts were repeated every 30–35 days to calculate adherence as the ratio of pills counted relative to pills previously counted, accounting for pills prescribed and dispensed.
Clinical lab reports of blood plasma HIV viral load most proximal and within 3 months of the final assessment were abstracted from electronic medical records. In accordance with HIV treatment guidelines [40], we defined suppressed (undetectable) viral load as <200 RNA copies/mL—a threshold that nearly eliminates most errors caused by viral load blips or assay variability [41]. We used the log value of the number of HIV RNA copies as a continuous variable.
Other outcomes: alcohol use and depression
Participants received daily text message surveys delivered to their phones in blocks of 14 consecutive days with 21 days between survey blocks. We collected 10 blocks (140 days) of alcohol assessments over the year of observation. The questions delivered via texting service assessed alcohol use during the previous day. Data were aggregated over days to form an index of alcohol use. We adapted the first item on the AUDIT (described above) to assess the quantity of alcohol consumed on the previous day. Specifically, participants were asked “How many alcohol drinks did you have yesterday? If you did not drink say 0.” Scores had a possible range of 0–140 representing the number of days drinking.
To assess daily depression symptoms, participants responded to five items adapted from the CESD (described above). Depression items also referred to the previous day and asked if participants: felt depressed; felt that everything they did was an effort; felt happy (reversed); felt lonely; and felt sad. For the depression items, text assessments were sent every other day, therefore assessing depression 7 days over each of the ten 14 day blocks for a total of 70 days. All items were responded to as Yes or No. Affirming any single depression symptom indicated experiencing depressed feelings the previous day. We also calculated the intersection of alcohol use and depression symptoms within the same day. Specifically, we coded days drinking when experiencing depression symptoms by taking the days during which participants both used alcohol and reported at least one symptom of depression. Because this variable required depression responses, scores ranged from 0 to 70 representing the number of days using alcohol and experiencing depression symptoms.
Statistical analyses
Descriptive statistics were used for participant characteristics and nursing adherence services. The first hypothesis regarding patient preference for telephone versus in-office counseling was tested using a contingency table chi-square (χ 2) test. We also tested for associations between participant characteristics and patient preference using contingency table χ 2 tests for categorical variables and independent t-tests for continuous variables. We tested the comparative effects of telephone versus in-office counseling for those participants randomized to counseling formats and those given the choice of counseling formats. For engagement in care, we examined the number of clinic appointments attended, the number not attended (no shows), the proportion of appointments attended, and disengagement from care over the 12 month study period. For ART adherence, we created four follow-up periods by taking the average adherence across Months 2–3, Months 4–6, Months 7–9, and Months 10–12. Log values of HIV viral load were analyzed for the follow-up period. Depression symptoms and alcohol use were annualized across blocks. All outcome analyses included the number of sessions completed as a factor, referred to as counseling dose. Generalized linear models tested the 2 counseling formats (telephone/in-office) × 2 preference conditions (choice/randomized) × 7 doses (0–6 sessions completed). Poisson models with robust estimators were conducted for count variables, binary models for dichotomous outcomes, and linear models for continuous data. Data were assumed to be missing at random and we utilized all available data in all analyses [43]. Statistical significance was defined as p < .05.
The sample size was determined using an effect size of 20% improvement in clinic appointments attended and medication adherence, representing clinically meaningful changes [42], based on previous trials testing behavioral self-regulation counseling [10,12]. For the purpose of this power estimate, we assumed proportionality across the cells. Using an alpha level of .05, a sample size of 240 distributed across four conditions (n = 60/cell) would be sensitive to a .20 difference for interaction effects in either direction on engagement in care and ART adherence with a power of .80.
RESULTS
Clinical context and routine adherence services
The 13 counties that compose the health district in which this study was conducted are either partly or entirely primary health care provider shortage areas. The number of active clinic patients at the start of the study was approximately 850 and the clinic was staffed by three primary care infectious disease physicians and five nurses. All nurses delivered primary care services that included HIV treatment adherence consultations. Nurses completed 1,735 adherence service records over an 18 month period. Nurses spent a median of 5 min with patients (M =7.8 min, standard deviation [SD] =7.4). Nearly all (95%) nursing contacts included a discussion of medication management, most frequently an explanation of how medications should be taken (see Supplementary Table). It was common (76%) for nurses to engage patients in strategies to improve adherence, most frequently encouraging patients to plan ahead to assure taking their medications.
Participant characteristics and trial design integrity
Participants enrolled in the trial were 175 men and 76 women receiving HIV care and referred by clinic staff. Table 1 shows the participant characteristics in all four study conditions. The sample included 47 (19%) patients who were newly diagnosed with HIV and new to HIV care and 131 (52%) patients who had fallen out of HIV care and were returning for services. Additionally, physicians and nurses identified 73 (29%) patients who recently experienced viral rebound or were treatment nonadherent. The sample was predominantly male (70%) and African American (82%). One in three participants had no source of employment and more than half had annual incomes under $10,000. On average participants had been living with an HIV diagnosis for more than 10 years. Figure 1 shows the flow of participants through the trial. Overall, 177 (70%) participants were retained over the 12 month study. Among the 74 participants lost to follow-up, 58 had also fallen out of care, 8 had withdrawn their participation, and 8 participants died during the trial by causes unrelated to the study protocol. There were no serious adverse events.
Table 1.
Demographic and health characteristics of participants allocated to counseling conditions
| Randomized telephone counseling N = 67 |
Randomized in-office counseling N = 59 |
Preference for telephone counseling N = 86 |
Preference for in-office counseling N = 39 |
Test for differences between preference groups | |||||
|---|---|---|---|---|---|---|---|---|---|
| Characteristic | n | % | n | % | n | % | n | % | χ 2 |
| Gender | |||||||||
| Male | 43 | 73 | 43 | 64 | 59 | 69 | 30 | 77 | |
| Female | 24 | 27 | 16 | 36 | 27 | 31 | 9 | 23 | 0.9 |
| Transgender | 2 | 3 | 1 | 1 | 5 | 6 | 2 | 5 | 0.1 |
| Employed | 11 | 16 | 13 | 22 | 19 | 22 | 7 | 18 | 3.8 |
| Income less than $10,000 | 47 | 70 | 38 | 65 | 64 | 77 | 27 | 71 | |
| $10,000–$20,000 | 14 | 21 | 12 | 20 | 15 | 18 | 5 | 13 | |
| Greater than $20,000 | 6 | 9 | 9 | 15 | 4 | 6 | 6 | 11 | 4.3 |
| Taking ART | 49 | 73 | 42 | 71 | 59 | 69 | 23 | 59 | 1.1 |
| Has a case manager | 56 | 83 | 46 | 78 | 61 | 71 | 25 | 64 | 0.5 |
| Receiving mental health care | 11 | 16 | 7 | 12 | 11 | 13 | 4 | 10 | 0.1 |
| Receiving substance use care | 3 | 4 | 1 | 1 | 2 | 1 | 1 | 1 | 0.1 |
| Missed clinic appointments | 14 | 21 | 10 | 17 | 22 | 26 | 16 | 41 | 3.0+ |
| M | SD | M | SD | M | SD | M | SD | t | |
| Age | 43.3 | 12.8 | 41.1 | 11.9 | 41.5 | 12.2 | 43.1 | 12.6 | 0.6 |
| Years since HIV positive | 12.0 | 9.3 | 11.5 | 8.1 | 11.7 | 8.8 | 11.1 | 8.3 | 0.3 |
| Education | 6.0 | 1.8 | 6.3 | 1.7 | 5.7 | 1.9 | 5.7 | 1.9 | 0.1 |
| Distance from home to clinic | 10.7 | 11.6 | 13.3 | 11.8 | 11.2 | 12.3 | 11.4 | 10.9 | 0.1 |
| Counseling sessions complete | 4.9 | 1.7 | 4.1 | 1.19 | 5.0 | 1.5 | 4.3 | 1.7 | 2.3* |
| Medication necessity beliefs | 1.8 | 0.9 | 1.7 | 0.9 | 2.0 | 0.9 | 1.6 | 0.6 | 2.2* |
| Medication concerns beliefs | 3.4 | 1.0 | 3.5 | 0.9 | 3.3 | 1.1 | 3.4 | 0.9 | 0.6 |
| HIV-related stress | 3.7 | 2.9 | 3.3 | 2.4 | 3.4 | 2.6 | 4.6 | 1.9 | 2.7** |
| AUDIT—alcohol score | 5.5 | 7.0 | 3.1 | 3.9 | 3.7 | 5.6 | 3.7 | 5.5 | 0.1 |
| CESD score | 21.8 | 11.8 | 21.3 | 9.7 | 21.0 | 9.7 | 20.5 | 9.0 | 0.2 |
| SF-36 Mental Health | 10.5 | 2.4 | 10.2 | 2.5 | 10.7 | 2.7 | 10.6 | 2.7 | 0.3 |
| SF-36 Physical Health | 6.3 | 1.9 | 6.5 | 2.1 | 6.3 | 2.1 | 7.9 | 1.9 | 2.9* |
ART antiretroviral therapy; AUDIT Alcohol Use Disorders Identification Test; CESD Center for Epidemiological Studies Depression.
+p < .10, *p < .05, **p < .01.
Patient preference for telephone versus in-office counseling
Among participants allocated to the choice condition, we found a significant preference for telephone counseling, χ 2 = 6.43, p < .01; telephone counseling was chosen by 66% of men and 75% of women, relative to 34% and 25% choosing in-office counseling, respectively. As shown in the right-side panel of Table 1, there were few characteristics that distinguished participants who chose telephone counseling relative to those who chose in-office counseling. Participants who chose telephone counseling held greater medication necessity beliefs, experienced fewer HIV-related stressors, and had poorer physical health-related quality of life as indicated by the SF-36.
Effects for counseling delivery formats
Table 2 shows the trial outcomes for participants receiving telephone and in-office counseling. For the primary outcome of engagement in care, participants on average attended four medical appointments and had no-showed to three medical appointments over the follow-up period, demonstrating that 65% of appointments scheduled were attended. For retention in care, clinic records showed that 193 (77%) participants were retained in care and 58 (23%) discontinued care over the follow-up period. There were no significant effects of counseling delivery format on engagement and retention in care outcomes.
Table 2.
Clinical outcomes for telephone and in-office delivered counseling conditions
| Telephone counseling N = 153 |
In-office counseling N = 98 |
Effect Wald χ 2 | ||||||
|---|---|---|---|---|---|---|---|---|
| Clinic attendance and retention in care | M | SD | M | SD | Counseling format | Choice/randomize | Dose | Counseling Format × Dose |
| Appointments attended | 4.3 | 2.0 | 4.1 | 1.8 | 1.1 | 3.2+ | 47.6** | 15.9** |
| Appointments no-show | 3.0 | 1.8 | 3.0 | 1.9 | 0.7 | 1.0 | 12.2* | 9.2 |
| % appointments kept | 63.1 | 22.1 | 65.4 | 22.2 | 1.7 | 2.6 | 3.4** | 0.9 |
| n (%) disengaged from care | 35 | 23 | 23 | 23 | 0.1 | 0.2 | 1.3 | 4.6 |
| ART adherence | ||||||||
| Baseline | 78.1 | 26.4 | 77.3 | 23.7 | 3.1 | 0.9 | 5.6 | 4.5 |
| 3 months | 78.6 | 22.5 | 76.3 | |||||
| 6 months | 75.9 | 23.4 | 74.6 | 24.7 | 4.1 | 3.7* | 15.3** | 12.7* |
| 9 months | 72.5 | 26.7 | 75.9 | 26.2 | 4.0 | 0.2 | 13.7** | 13.8** |
| 12 months | 71.9 | 23.4 | 73.3 | 28.3 | 0.9 | 0.3 | 9.97+ | 11.68* |
| Log HIV viral load | 2.2 | 1.6 | 2.5 | 1.6 | 0.1 | 0.4 | 51.5** | 33.1** |
| Depression and alcohol use | ||||||||
| Days felt depressed | 33.3 | 23.2 | 34.7 | 23.5 | 9.5** | 7.6** | 54.4** | 21.6** |
| Days drank alcohol | 9.2 | 20.5 | 4.8 | 11.8 | 4.7* | 6.7** | 1.1 | 8.2 |
| Mean drinks consumed on days drinking | 3.0 | 7.9 | 1.2 | 5.0 | 3.5 | 16.3** | 15.8** | 4.2 |
| Days drank when depressed | 3.6 | 8.8 | 1.6 | 3.2 | 3.1 | 7.2** | 2.4 | 8.3 |
ART antiretroviral therapy.
+p < .10, *p < .05, **p < .01.
For the secondary outcome, ART adherence, telephone and in-office counseling conditions tended to differ in the first 6 months of follow-up, with telephone counseling showing slightly better adherence than in-office counseling (see Table 2). However, the difference was small and transient. For HIV viral load, results did not indicate a main effect for counseling delivery format.
For the other outcomes, depression and alcohol use, the daily text assessments showed that nearly all participants (98%) experienced depression symptoms on at least one assessed day, with a mean of 33.8 (SD = 23.3) days experiencing depression symptoms. Poisson regression modeling for the number of days during which participants reported experiencing depression symptoms showed that those receiving telephone counseling reported significantly fewer days with depression symptoms (see Table 2).
Daily text assessments of alcohol use indicated that participants receiving in-office counseling reported fewer days of drinking and consumed less alcohol on days when drinking than participants who received telephone counseling. Results showed a significant effect of counseling delivery on the number of days with co-occurring alcohol use and depression symptoms; participants receiving in-office counseling reported fewer days of drinking when experiencing depression symptoms relative to those receiving telephone counseling.
Effects for choice/randomization conditions
There were no significant effects of the choice/randomization conditions on engagement in care, retention in care, medication adherence, or HIV viral load (see Table 2). However, the effects of choice/randomization conditions on days experiencing depression symptoms were significant; participants in the choice condition had fewer days with depression symptoms (M = 29.6, SD = 22.1) over the follow-up than those randomized (M = 38.1, SD = 23.8). The effects of the choice/randomization condition on days of drinking were also significant; participants in the choice condition drank on fewer days (M = 5.1, SD = 11.3) over the follow-up than in the randomized condition (M = 9.2, SD = 22.3).
Effects for the number of counseling sessions completed (dose)
Participants in the telephone counseling condition received a total of 735 sessions out of 1,106 sessions scheduled (66%), an average of 5.03 (SD = 1.62) sessions, compared to 385 in-office counseling sessions out of 831 sessions scheduled (46%), an average of 4.23 (SD = 1.85), a significant difference, F = 10.9, p < .001. Telephone counseling resulted in 89/1,106 canceled and 282/1,106 no-show counseling appointments relative to 62/831 canceled and 384/831 no-show in-office counseling sessions. Overall, 76% of participants receiving telephone counseling completed five or more counseling sessions relative to 55% receiving in-office counseling, χ 2 = 11.43, p < .001. There were no differences between choice and randomized conditions for the number of sessions completed.
Table 2 shows the results for the significance of testing dose and Dose × Condition interactions and Table 3 shows the means for primary and secondary outcomes for participants receiving numbers of telephone and in-office counseling sessions. There was a significant effect of counseling dose on clinical appointments attended, indicating that participants with the highest rate of counseling sessions also had higher clinic attendance. There was also a significant interaction between counseling delivery format and dose; participants receiving less than three telephone counseling sessions had the poorest clinical appointment attendance with no difference between telephone and in-office counseling among participants receiving five or six counseling sessions in either condition. There were no effects of counseling dose on retention in care.
Table 3.
Dose effect of telephone and in-office delivered counseling on mean clinic attendance, ART adherence, and HIV viral load over assessment periods.
| Mean percentage ART adherence | |||||||
|---|---|---|---|---|---|---|---|
| Dose | Percent clinic appointments kept | Baseline | 3 months | 6 months | 9 months | 12 months | Log viral load |
| Telephone counseling | |||||||
| 0–1 session | 41.6 | 89.2 | 84.8 | 79.2 | 87.3 | 86.2 | 2.3 |
| 2 sessions | 48.7 | 74.1 | 66.0 | 65.6 | 71.9 | 62.5 | 3.6 |
| 3 sessions | 62.8 | 67.3 | 81.8 | 69.1 | 64.3 | 61.2 | 1.9 |
| 4 sessions | 49.5 | 70.9 | 77.5 | 79.3 | 72.7 | 85.1 | 2.7 |
| 5 sessions | 61.6 | 91.3 | 84.4 | 79.6 | 89.9 | 74.7 | 1.6 |
| 6 sessions | 68.1 | 78.1 | 78.5 | 76.3 | 71.5 | 71.3 | 2.1 |
| In-office counseling | |||||||
| 0–1 session | 63.4 | 80.3 | 67.3 | 69.8 | 76.5 | 58.6 | 2.8 |
| 2 sessions | 61.8 | 61.3 | 63.4 | 61.9 | 45.7 | 46.0 | 3.5 |
| 3 sessions | 56.4 | 74.5 | 70.4 | 71.3 | 89.4 | 89.7 | 1.9 |
| 4 sessions | 56.6 | 64.9 | 76.1 | 46.4 | 76.8 | 77.4 | 2.1 |
| 5 sessions | 68.5 | 75.9 | 76.1 | 67.0 | 80.6 | 74.1 | 2.4 |
| 6 sessions | 69.7 | 84.3 | 82.6 | 83.2 | 77.4 | 76.4 | 2.1 |
ART antiretroviral therapy.
There was a significant effect of counseling dose on ART adherence, with a greater number of sessions indicating higher adherence. However, the Dose × Condition interactions were significant at the 6, 9, and 12 month follow-ups. Telephone counseling participants who received zero or one session demonstrated the highest ART adherence over time, whereas participants receiving two telephone counseling sessions demonstrated lower ART adherence over time. This pattern was not as apparent for participants receiving in-office counseling. Within each time period, including baseline, participants receiving six in-office sessions demonstrated greater ART adherence than participants receiving six telephone sessions. There was no evidence, however, of clinically meaningful improvements in adherence for any condition over time.
Results also showed that participants who received only two counseling sessions had the highest viral loads relative to the other doses of counseling. In addition, participants receiving five sessions of telephone-delivered counseling demonstrated lower viral loads than participants receiving five sessions of in-office counseling.
For depression, there was a dose effect and a Dose × Condition interaction; participants receiving more counseling sessions indicated fewer days of depression symptoms. Participants receiving two or fewer telephone sessions, as well as those receiving six telephone sessions experienced fewer days of depression symptoms than those receiving in-office counseling. There were no dose counseling effects for alcohol use.
DISCUSSION
Telehealth interventions, including behavioral counseling delivered by telephone, afford opportunities for expanding clinical services to patients in remote areas, as well as all patients at times when facility-based services are disrupted, including during periods of staff shortages, natural disasters, and emerging pandemics. However, the possible advantages to telephone-delivered services may be offset by poorer outcomes. This trial was motivated by the need to directly compare telephone and in-office behavioral self-regulation counseling for patients who are at risk of disengaging from care. Our assessment of routine clinical services showed that an array of adherence strategies is delivered to patients, albeit in relatively brief encounters. Within this clinical context, there were notable differences between telephone and in-office counseling.
We confirmed our hypothesis that, when given a choice, patients would prefer telephone counseling over in-office counseling. These results are consistent with previous research showing preferences and satisfaction with telephone-delivered health services [44–46]. Participants who chose to receive telephone counseling held greater medication necessity beliefs, experienced fewer HIV-related stressors, and had poorer physical health-related quality of life. Telephone counseling may, therefore, be preferred by patients who are motivated to take ART and are experiencing greater physical limitations.
Our hypothesis predicting that both telephone and in-office counseling would improve HIV treatment outcomes was not fully supported. Overall clinic appointment attendance and ART adherence were suboptimal across conditions and time points. In fact, we saw adherence decrease over the follow-ups. Our hypothesis that participants given the choice of counseling format would demonstrate more positive outcomes than those not given the choice was not supported. Patients who were given the choice of counseling format did experience fewer days with depressed symptoms than those not given the choice.
The hypothesis that participants counseled by telephone would receive more counseling sessions than participants receiving in-office counseling was confirmed. This finding extends previous research that has compared telephone and in-office counseling for other health conditions [44–46]. There was also an effect of the number of counseling sessions completed (dose) on outcomes. Clinic appointment attendance showed a somewhat typical dose–response association, with participants who had completed five or six counseling sessions, both by telephone and in office, demonstrating the highest rate of clinic appointment keeping. This pattern of results may represent self-selection in that patients with better clinic attendance may have also had better counseling attendance. For ART adherence, the observed pattern was not what we expected and did not represent a typical dose–response relationship. Specifically, participants receiving no counseling or just one counseling session, as well as those receiving five or six sessions, demonstrated the most positive adherence and HIV viral load outcomes, whereas participants who received only two sessions fared the worst, and this pattern was observed for both telephone and in-office counseling. One explanation for this finding is that participants who felt that they were doing well and did not need counseling to improve their health may have discontinued counseling after one session or never attended any sessions. In contrast, participants who made an initial effort to attend two sessions and stopped may have done so out of a sense of futility. These findings suggest that neither telephone nor in-office behavioral self-regulation counseling may be sufficient to significantly improve treatment retention and ART adherence for patients at risk for disengaging from care and treatment failure.
For the other outcomes, we observed differences between counseling delivery formats on depression and alcohol use that were consistent with past research [47]. Telephone counseling demonstrated fewer depression symptoms over the follow-up, and the more counseling sessions participants received, the fewer days of depression symptoms they experienced. On the other hand, across measures and time points, in-office counseling resulted in less alcohol use, including fewer days drinking and lower quantities of alcohol consumed. In-office counseling also resulted in fewer days when alcohol was consumed when experiencing depression symptoms. Promising outcomes were, therefore, observed for telephone counseling improving depression and in-office counseling reducing alcohol use. Improvements in these presumed drivers of disengagement from care and ART nonadherence, however, did not appear to yield HIV treatment benefits.
Results of the current study should be interpreted in light of its methodological limitations. Although we sampled a clinic that serves a broad geographical area within a state with high HIV prevalence, the sample was one of convenience and cannot be considered a representative of people living with HIV in this region. In addition, the study was conducted in just one state in the southeastern USA and is, therefore, geographically constrained. It should also be noted that we did not hypothesize differences between telephone and in-office counseling formats and, yet, our trial was not designed to test for noninferiority. Therefore, we cannot conclude that the counseling format conditions were equivalent. Our measures were also not without limitations, including reliance on the presumed completeness of clinical records and self-report measures of mental health and alcohol use. Finally, our sample was relatively small for the trial design, especially in terms of examining potential moderating effects of participant characteristics, such as gender. All of our results should be considered preliminary and in need of replication. With these limitations in mind, we believe that our results offer implications for implementing behavioral self-regulation counseling in HIV care settings.
The overall pattern of results in this trial suggests that telephone and in-office behavioral self-regulation counseling demonstrate positive effects on mental health and alcohol use, respectively. However, this counseling model did not show evidence for improvements in ART adherence and health outcomes for patients who were marginally engaged in care and at risk for treatment failure. In light of past efficacy trials with positive outcomes from behavioral self-regulation counseling [10,12], the lack of improvement in treatment engagement and ART adherence observed in our study suggests that more intense interventions are needed for patients who are new to care, not fully engaged in care, or at risk for treatment failure. The CDC’s Compendium of Evidence-Based Interventions includes 18 treatment engagement-adherence programs that share common theoretical bases and substantial common components. Those interventions that were originally tested using in-office sessions can likely be adapted for delivery by telephone without evidence for diminished outcomes. Expanding access to telephone counseling will also require restructuring reimbursement for telemedicine services. Our results, therefore, encourage reconfiguring in-office counseling for telephone delivery. Such adaptations may increase access to essential health services, especially at the time of clinical service interruptions, without apparent diminished outcomes.
Supplementary Material
Acknowledgments
Funding: National Institute of Alcohol Abuse and Alcoholism (grant R01-AA023727).
Compliance with Ethical Standards
Conflicts of Interest: The authors have no conflicts of interest.
Author Contributions:
S.C.K., H.K., & L.A.E. conceptualized, designed and oversaw the execution of this trial and prepared the final manuscript. E.B., M.H., & M.O.K. managed the project implementation and operations. S.C.K. conducted the data analyses.
Ethical Approval: All procedures performed involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Informed Consent: All participants gave written informed consent.
References
- 1. Marks G, Patel U, Stirratt MJ, et al. Single viral load measurements overestimate stable viral suppression among HIV patients in care: Clinical and public health implications. J Acquir Immune Defic Syndr. 2016;73(2):205–212. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Walcott M, Kempf MC, Merlin JS, Turan JM. Structural community factors and sub-optimal engagement in HIV care among low-income women in the Deep South of the USA. Cult Health Sex. 2016;18(6):682–694. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Braithwaite RS, McGinnis KA, Conigliaro J, et al. A temporal and dose-response association between alcohol consumption and medication adherence among veterans in care. Alcohol Clin Exp Res. 2005;29(7):1190–1197. [DOI] [PubMed] [Google Scholar]
- 4. Mayston R, Kinyanda E, Chishinga N, Prince M, Patel V. Mental disorder and the outcome of HIV/AIDS in low-income and middle-income countries: a systematic review. AIDS.. 2012;26(suppl 2):S117–S135. [DOI] [PubMed] [Google Scholar]
- 5. Berg CA, Wiebe DJ, Suchy Y, et al. Individual differences and day-to-day fluctuations in perceived self-regulation associated with daily adherence in late adolescents with type 1 diabetes. J Pediatr Psychol. 2014;39(9):1038–1048. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Jones CJ, Smith HE, Frew AJ, Toit GD, Mukhopadhyay S, Llewellyn CD. Explaining adherence to self-care behaviours amongst adolescents with food allergy: A comparison of the health belief model and the common sense self-regulation model. Br J Health Psychol. 2014;19(1):65–82. [DOI] [PubMed] [Google Scholar]
- 7. Campbell JI, Haberer JE. Cell phone-based and adherence device technologies for HIV care and treatment in resource-limited settings: recent advances. Curr HIV/AIDS Rep. 2015;12(4):523–531. [DOI] [PubMed] [Google Scholar]
- 8. Signor L, Pierozan PS, Ferigolo M, et al. Efficacy of the telephone-based brief motivational intervention for alcohol problems in Brazil. Br J Psychiatry. 2013;35(3):254–261. [DOI] [PubMed] [Google Scholar]
- 9. Brown RL, Saunders LA, Bobula JA, Mundt MP, Koch PE. Randomized-controlled trial of a telephone and mail intervention for alcohol use disorders: Three-month drinking outcomes. Alcohol Clin Exp Res. 2007;31(8):1372–1379. [DOI] [PubMed] [Google Scholar]
- 10. Kalichman SC, Kalichman MO, Cherry C, Eaton LA, Cruess D, Schinazi RF. Randomized factorial trial of phone-delivered support counseling and daily text message reminders for HIV treatment adherence. J Acquir Immune Defic Syndr. 2016;73(1):47–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Kalichman SC, Kalichman MO, Cherry C, et al. Brief behavioral self-regulation counseling for HIV treatment adherence delivered by cell phone: An initial test of concept trial. AIDS Patient Care STDS. 2011;25(5):303–310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Reynolds NR, Testa MA, Su M, et al. ; AIDS Clinical Trials Group 731 and 384 Teams . Telephone support to improve antiretroviral medication adherence: A multisite, randomized controlled trial. J Acquir Immune Defic Syndr. 2008;47(1):62–68. [DOI] [PubMed] [Google Scholar]
- 13. Cook PF, McCabe MM, Emiliozzi S, Pointer L. Telephone nurse counseling improves HIV medication adherence: An effectiveness study. J Assoc Nurses AIDS Care. 2009;20(4):316–325. [DOI] [PubMed] [Google Scholar]
- 14. Musayón-Oblitas Y, Cárcamo C, Gimbel S. Counseling for improving adherence to antiretroviral treatment: A systematic review. AIDS Care. 2019;31(1):4–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Heckman BD, Catz SL, Heckman TG, Miller JG, Kalichman SC. Adherence to antiretroviral therapy in rural persons living with HIV disease in the United States. AIDS Care. 2004;16(2):219–230. [DOI] [PubMed] [Google Scholar]
- 16. Graves N, Barnett AG, Halton KA, et al. Cost-effectiveness of a telephone-delivered intervention for physical activity and diet. PLoS One. 2009;4(9):e7135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Linnemayr S, Jennings Mayo-Wilson L, Saya U, et al. HIV care experiences during the COVID-19 pandemic: Mixed-methods telephone interviews with clinic-enrolled HIV-infected adults in Uganda. AIDS Behav. 2020. Epub ahead of print. doi:10.1007/s10461-020-03032-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Beima-Sofie K, Ortblad KF, Swanson F, Graham SM, Stekler JD, Simoni JM. “Keep It Going if You Can”: HIV service provision for priority populations during the COVID-19 pandemic in Seattle, WA. AIDS Behav. 2020;24(10):2760–2763. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Marhefka S, Lockhart E, Turner D. Achieve research continuity during social distancing by rapidly implementing individual and group videoconferencing with participants: Key considerations, best practices, and protocols. AIDS Behav. 2020;24(7):1983–1989. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Buchanan AH, Datta SK, Skinner CS, et al. Randomized trial of telegenetics vs. in-person cancer genetic counseling: Cost, patient satisfaction and attendance. J Genet Couns. 2015;24(6):961–970. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Miller WR, Zweben A, DiClemente C, Rychtarik R. Motivational Enhancement Therapy Manual. Washington, DC: U.S. Government Printing Office; 1992. [Google Scholar]
- 22. Centers for Disease Control and Prevention. Compendium of evidence-based interventions and best practices for HIV prevention. 2017. Available at https://www.cdc.gov/hiv/research/interventionresearch/compendium/. Accessed 7 July 2020.
- 23. Heo M, Meissner P, Litwin AH, et al. Preference option randomized design (PORD) for comparative effectiveness research: Statistical power for testing comparative effect, preference effect, selection effect, intent-to-treat effect, and overall effect. Stat Methods Med Res. 2019;28(2):626–640. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Rogers M Comparative Effectiveness Research. New York, NY: Oxford; 2014. [Google Scholar]
- 25. Reynolds NR. The problem of antiretroviral adherence: A self-regulatory model for intervention. AIDS Care. 2003;15(1):117–124. [DOI] [PubMed] [Google Scholar]
- 26. Bandura A Self-Efficacy: The Exercise of Control. New York, NY: Freeman; 1997. [Google Scholar]
- 27. Centers for Disease Control and Prevention. Compendium of evidence-based interventions. 2019. Compendium of evidence-based interventions and best practices for HIV prevention. Available at https://www.cdc.gov/hiv/research/interventionresearch/compendium/ma/characteristics.html. Accessed 12 July 2020.
- 28. Rotheram-Borus MJ, Swendeman D, Flannery D, Rice E, Adamson DM, Ingram B. Common factors in effective HIV prevention programs. AIDS Behav. 2009;13(3):399–408. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Gribble JN, Miller HG, Cooley PC, Catania JA, Pollack L, Turner CF. The impact of T-ACASI interviewing on reported drug use among men who have sex with men. Subst Use Misuse. 2000;35(6–8):869–890. [DOI] [PubMed] [Google Scholar]
- 30. de Bruin M, Viechtbauer W, Schaalma HP, Kok G, Abraham C, Hospers HJ. Standard care impact on effects of highly active antiretroviral therapy adherence interventions: A meta-analysis of randomized controlled trials. Arch Intern Med. 2010;170(3):240–250. [DOI] [PubMed] [Google Scholar]
- 31. Van Dam NT, Earleywine M. Validation of the Center for Epidemiologic Studies Depression Scale–Revised (CESD-R): Pragmatic depression assessment in the general population. Psychiatry Res. 2011;186(1):128–132. [DOI] [PubMed] [Google Scholar]
- 32. Saunders JB, Aasland OG, Babor TF, DeLaFuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption II. Addictions. 1993;88(6):791–804. [DOI] [PubMed] [Google Scholar]
- 33. Horne R, Cooper V, Gellaitry G, Date HL, Fisher M. Patients’ perceptions of highly active antiretroviral therapy in relation to treatment uptake and adherence: The utility of the necessity-concerns framework. J Acquir Immune Defic Syndr. 2007;45(3):334–341. [DOI] [PubMed] [Google Scholar]
- 34. Brazier J. The SF-36 health survey questionnaire—A tool for economists. Health Econ. 1993;2(3):213–215. [DOI] [PubMed] [Google Scholar]
- 35. Bozzette SA, Berry SH, Duan N, et al. The care of HIV-infected adults in the United States. HIV cost and services utilization study consortium. N Engl J Med. 1998;339(26):1897–1904. [DOI] [PubMed] [Google Scholar]
- 36. Mugavero MJ, Davila JA, Nevin CR, Giordano TP. From access to engagement: measuring retention in outpatient HIV clinical care. AIDS Patient Care STDS. 2010;24(10):607–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Bangsberg DR, Hecht FM, Charlebois ED, Chesney M, Moss A. Comparing objective measures of adherence to HIV antiretroviral therapy: Electronic medication monitors and unannounced pill counts. AIDS Behav. 2001;5(3):275–281. [Google Scholar]
- 38. Kalichman SC, Amaral C, Swetsze C, et al. Monthly unannounced pill counts for monitoring HIV treatment adherence: Tests for self-monitoring and reactivity effects. HIV Clin Trials. 2010;11(6):325–331. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Kalichman SC, Amaral CM, Cherry C, et al. Monitoring medication adherence by unannounced pill counts conducted by telephone: Reliability and criterion-related validity. HIV Clin Trials. 2008;9(5):298–308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. International Advisory Panel on HIVCCO. IAPAC guidelines for optimizing the HIV care continuum for adults and adolescents. J Int Assoc Provid AIDS Care. 2015;14(suppl 1):S3–S34. [DOI] [PubMed] [Google Scholar]
- 41. Crepaz N, Tang T, Marks G, Mugavero MJ, Espinoza L, Hall HI. Durable viral suppression and transmission risk potential among persons with diagnosed HIV infection: United States, 2012–2013. Clin Infect Dis. 2016;63(7):976–983. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Byrd KK, Hou JG, Hazen R, et al. ; Patient-Centered HIV Care Model Team . Antiretroviral adherence level necessary for HIV viral suppression using real-world data. J Acquir Immune Defic Syndr. 2019;82(3):245–251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Harel O, Pellowski J, Kalichman S. Are we missing the importance of missing values in HIV prevention randomized clinical trials? Review and recommendations. AIDS Behav. 2012;16(6):1382–1393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Lohr PA, Aiken ARA, Forsyth T, Trussell J. Telephone or integrated contraception counselling before abortion: Impact on method choice and receipt. BMJ Sex Reprod Health. 2018;44(2):114–121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Hammersley V, Donaghy E, Parker R, et al. Comparing the content and quality of video, telephone, and face-to-face consultations: A non-randomised, quasi-experimental, exploratory study in UK primary care. Br J Gen Pract. 2019;69(686):e595–e604. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. McKinstry B, Hammersley V, Burton C, et al. The quality, safety and content of telephone and face-to-face consultations: A comparative study. Qual Saf Health Care. 2010;19(4):298–303. [DOI] [PubMed] [Google Scholar]
- 47. Himelhoch S, Medoff D, Maxfield J, et al. Telephone based cognitive behavioral therapy targeting major depression among urban dwelling, low income people living with HIV/AIDS: Results of a randomized controlled trial. AIDS Behav. 2013;17(8):2756–2764. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.

