Client, clinician, and administrator factors associated with the successful acceptance of a telehealth comprehensive recovery service: A mixed methods study

Abstract

The coronavirus disease 2019 SARS-CoV-2 (COVID-19) crisis and subsequent social distancing recommendations resulted in increased use of telehealth within recovery-oriented behavioral health services (RS). Populations with serious mental illness (SMI) rely on psychosocial treatment, care coordination, and pharmacotherapy to achieve recovery goals and increase community engagement. This program evaluation of a group-based RS used mixed methods to better understand the multiple factors that contributed to successful telehealth conversion. Clients’ service utilization over an 18-week period was collected to determine acceptance and the client characteristics associated with utilization (n = 72). Clients completed a treatment satisfaction questionnaire that was distributed ten weeks following telehealth conversion. Qualitative interviews explored staff perspectives on factors that impacted conversion, acceptance, and utilization. Initial staff skepticism gave way to acceptance, while the demands of resourcefulness, flexibility, and competency were emphasized. Clients’ treatment utilization remained stable, while the number of missed/cancelled sessions were less frequent over time, especially for clients with a history of psychosis. Clients reported high overall satisfaction, but a preference for in-person treatment. Within this clinic serving middle to high socioeconomic status (SES) clients, clinicians and clients alike found the virtual group-based RS to be feasible and acceptable while in-person treatment was not an option.

1. Introduction

The coronavirus disease 2019 SARS-CoV-2 (COVID-19) crisis has put the feasibility and acceptability of psychiatric telehealth into the spotlight. As social distancing mandates sought to mitigate the risk of COVID-19 infection, some of the most vulnerable populations, such as individuals with serious mental illness (SMI), required special consideration to maintain their necessary mental health services. Large-scale shifts in care delivery sought to bridge the gap as in-person encounters were no longer feasible, even after initial lockdowns. Recovery oriented behavioral health services (RS), which are often used to serve people with SMI (Anthony, 1993), began using synchronous videoconferencing to offer psychological support and symptom management training. An ever-growing number of studies have explored factors associated with successful transition from in-person psychiatric services to “virtual clinics” (Lynch et al., 2020; Miu et al., 2020, Santesteban-Echarri et al., 2020, Yellowlees et al., 2020). However, most successful telehealth conversions emphasize vital continuity of one-on-one treatment; less studied is telehealth conversion of group therapy interventions (Banbury et al., 2018), which are a common component of psychosocial recovery programs.

Clients, administrators, and clinicians are essential to the successful conversion and acceptance of telehealth services. A systematic review examining perceptions of telehealth among individuals receiving mental health treatment was generally positive (Cowen et al., 2019), though challenges were also noted. Some clients cited concerns about relational and interpersonal factors (e.g., rapport, connectedness), technological limitations and interruptions, and maintaining clinical protections (e.g., privacy, safety). Within the population with SMI, defined as a mental, behavioral, or emotional disorder resulting in serious functional impairment (National Institute of Mental Health 2021), those with access to technology evidenced high rates of telehealth acceptance (Medalia et al., 2020; Lynch et al., 2020). One study found individuals with SMI utilized significantly more telehealth visits compared to non-SMI (Miu et al., 2020). However, social, racial and economic disparities in access to technology and in the requisite technological know-how remain essential considerations when considering telehealth in SMI populations (Torous & Keshavan, 2020).

Clinician perceptions of telehealth, either informed by experience or preconceived notions, also play a significant role in telehealth uptake. There is currently a gap in the literature examining clinician perspectives on evidence-based group therapy in telehealth formats. Within one-on-one interventions, multiple studies have reported that clinicians express apprehension about telehealth, including concerns regarding the regulatory and legal standards, virtual safety planning, and overall uncertainty around best practices for efficacious video-format encounters (Whitten & Mackert, 2005; Cowen et al., 2019). Studies conceptualize that clinician reluctance, doubt and caution fortify a “gatekeeper” role, where clinicians may act as a barrier to telehealth development and growth. An analysis of telehealth clinician interviews led Wade et al. (2014) to posit that the success of telehealth services is related to the extent of “clinician acceptance,” which is grounded in positive beliefs about telehealth, access to adequate technology and resources, strong inter-clinician relationships and responsiveness to demand for their services. The authors proposed that increasing clinician acceptance could facilitate uptake, expansion and sustainability of telehealth services.

This concurrent mixed methods study is the first we are aware of to examine client, clinician, and administrator perspectives on the successful conversion of a group-based RS from in-person to telehealth sessions. The study reviews an 18-week period of in-person and telehealth services, reporting rates of telehealth acceptance and utilization in clients with SMI. Having experienced both in-person and telehealth versions of their personalized treatment, clients reported their relative satisfaction via a self-report measure. To further the understanding of clinician and administrator viewpoints on telehealth, qualitative interviews assessed factors that both facilitated and hindered implementation.

2. Methods

This concurrent mixed methods study used quantitative and qualitative approaches for complementarity (QUAN + QUAL; Palinkas et al., 2011). Quantitative data focused on assessing clinic utilization and satisfaction among clients, while qualitative data focused on the context and process of conversion, the implementation of adapted services, and the experience of delivering telehealth from clinician and administrator perspectives.

2.1. Description of the RS

This study took place in a private university-affiliated outpatient psychiatric treatment center (www.lieberclinic.com) that provides comprehensive psychosocial and rehabilitation services to adults over the age of 18. The RS uses a recovery-oriented model to support people with SMI whose primary diagnoses typically include schizophrenia-spectrum disorder, high-functioning autism spectrum disorder (ASD), and mood disorders. Using shared decision-making, RS participants work with care coordinators to craft individualized therapeutic plans that address their recovery goals. The RS provides a wide range of evidence-based services, including intake assessment, care coordination, group psychotherapies, skills training groups, individual skills coaching, vocational/educational supports, family services and recreational activities (Medalia et al., 2020; Lynch et al., 2020). Prior to the COVID-19 pandemic, all groups were offered in-person at the clinic.

2.2. Process of telehealth conversion

Due to local governmental stay-at-home mandates, the RS underwent a rapid telehealth conversion between March 16-19, 2020. Following the conversion, no clients were seen in-person. To facilitate the conversion to telehealth, care coordinators communicated with clients and stakeholders that they would be able to maintain their treatment plans, but in a synchronous video format (Zoom, myConnect). To participate, clients completed additional written consent for telehealth services, noting risks and benefits.

2.3. Samples

   

2.3.1. Client sample

The sample included all RS participants (n=72) who received treatment during the 18-week study timeframe, including participants who were admitted, discharged or opted out (see Table 1 ).

Table 1.

Sample demographics and service utilization over 18-week time period (N = 72).

	Time 1 (in-person) Weeks 1-6 Pre	Time 2 (telehealth) Weeks 7-12 Post1	Time 3 (telehealth) Weeks 13-18 Post2	p
Enrolled in RS services	n = 60	n = 64a	n = 62b
Age
Mean (SD)	28.1 (10)	28.22 (10.7)	28.45 (11.14)	N.S.
Gender n
Male	31	38	37	N.S.
Female	23	20	21	N.S.
Non-Binary	6	6	4	N.S.
Race/ethnicity n
White/Caucasian	55	58	53	N.S.
Black/African American	1	1	1	N.S.
Hispanic, Latinx	2	3	3	N.S.
Asian	2	2	2	N.S.
Primary diagnosis n
Psychotic disorderc	15	16	15	N.S.
Autism spectrum disorder	15	15	16	N.S.
Anxiety disorderd	2	2	2	N.S.
Affective disordere	28	31	29	N.S.
Telehealth acceptance n/N (%)	N/A	56/60 (93)	N/A
Admission (n)	0	8	4	N.S.
Opt out of telehealth (n)	0	4	2	N.S.
Discharge (n)	0	2	2	N.S.
Sessions Attended
mean (SD)	22.58 (14)	22.13 (16)	23.53 (14.89)	N.S.
Sessions Missed
mean (SD)	5.63 (5.7)	4.31 (4.13)	2.37 (2.12)	**

Note: *p < 0.05; **p < 0.01. N.S. = not statistically significant. Kruskal–Wallis test and Chi-square test used with Yates' continuity correction.

^a64 = 60 – 4_optout – 2_discharges + 8_admissions

^b62 = 64 – 4_optout – 2_discharges + 4_admissions

^cschizophrenia, schizoaffective disorder, other affective disorder with psychosis

^dGeneralized anxiety disorder, social anxiety disorder

^eBipolar disorder, major depressive disorder, persistent depressive disorder

2.3.2. Clinician and Leadership/Administrator sample

The study used purposive sampling procedures (Palinkas et al., 2015) to identify key staff members involved in the telehealth conversion that would maximize the diversity of staff perspectives. All staff invited to participate (n=6, out of 9 total) completed an interview. Staff participants represented a range of disciplines/professions (e.g., social work, psychology, counseling), specialties (e.g., life/social skills coaching, CBT/DBT/ACT group leaders, cognitive remediation) and roles (e.g., therapists, clinic leadership, and executive administration). Regardless of their specific role, all staff were practicing clinicians.

2.4. Method of data collection

Quantitative data included client diagnoses, demographics, service utilization, and a satisfaction questionnaire. Qualitative data were provided in staff interviews.

2.4.1. Quantitative Data Collection

2.4.1.1. Demographic and utilization data

As part of program evaluation initiatives telehealth acceptance, intakes, session attendance, diagnoses, and demographics were determined using a comprehensive electronic medical record (EMR) review for all RS participants. Data was extracted by DL and verified by AM, who both serve as clinicians in the RS, with access to the EMR. Identifiable private information, and all possible linkages, were removed in the de-identified database used for this research. The governing Institutional Review Board determined that criteria for human subjects research, under 45 CFR 46, were not met and exempted this study from further review.

Service utilization data were captured over an 18-week time span and categorized in three, consecutive six-week intervals. The first six-weeks (pre; Week 1-6), prior to COVID-19 stay-at-home orders, captured attendance and no show/cancellation frequency of in-person sessions. The subsequent six-week (post1; Week 7-12) interval followed telehealth consented clients participating in their virtual treatment plans. This time segment reflects the attrition of RS participants who “opted out” following the conversion. The third, and final, six-week interval (post2; Week 13-18), continued to follow RS participants’ service utilization. For each time segment, RS participants were determined by active involvement in treatment activities. RS participants who were discharged or opted out were included in the sample during the time interval when they were still active. Relevant demographic and diagnostic information were also collected to explore potential covariates related to utilization. See Figure 1 for timeline of telehealth conversion and data collection.

Figure 1.

Timeline of telehealth conversion and data collection Note: Utilization data collected throughout the 3 phases. Staff interviews and client satisfaction survey were conducted in Post2.

2.4.1.2. Client Satisfaction

The Client Satisfaction Questionnaire (CSQ-8; Attkisson, 1987) was administered on a volunteer/anonymous basis ten weeks following telehealth conversion, disseminated via Qualtrics to all enrolled RS participants. The CSQ-8 is a valid and reliable 8-item self-report questionnaire used to determine respondents’ perception of treatment quality. Respondents selected from four graded responses (1-4 points), that yield a sum score ranging from 8–32, where higher scores denote greater satisfaction (scores >23 indicate satisfaction). To assess client perceptions of telehealth relative to in-person services, two additional questions were added using the CSQ graded responses: 1) “If there were no health risks, would you prefer to have your sessions conducted in-person?” 2) “Telehealth sessions are as good as in-person sessions for receiving the help I want.” Concordance was established by identifying responses that both prefer telehealth sessions and believe that telehealth sessions are “as good as” in-person.

2.4.2. Qualitative interviews

Semi-structured qualitative interviews were conducted virtually (i.e., Zoom) by two experienced qualitative researchers (AS and LJC) with each participant, two to three months after the clinic had converted to telehealth (post2). The interview guide consisted of questions regarding how the clinic adapted practices to support clients and staff during the COVID-19 pandemic, what factors hindered and facilitated these adaptations, and the perceived drawbacks and potential positive aspects of these changes. Sample questions included “How has COVID-19 impacted your ability to support clients?” and “What adjustments have you made to services?” Interviews lasted one-hour, were audio-recorded, and transcribed verbatim.

2.5. Data Analysis

   

2.5.1. Quantitative analysis

All analyses of quantitative data were conducted using native R packages (RStudio Team, 2020) and lme4 (Bates et al., 2015). Longitudinal analyses were performed via a model building approach using generalized linear modeling with a Poisson log-link. Due to the nested structure of the data, all analyses accounted for within-subject variability by using random effects components in the models, Time by Participant (Time | ID). The nested structure accounts for participant variability, allowing for both within-subject and between-subject comparisons. Diagnostic and demographic covariates were added to the conditional model to assess their relationship to the number of missed/canceled sessions.

2.5.2. Qualitative analysis

A content analysis approach was used to label and organize data into meaningful categories and concepts, and to identify patterns within and across codes and stakeholder groups (Bernard, 2002). Authors AS & LJC read all transcripts to develop initial familiarity with the depth and range of content, documenting emerging topics and concepts in memos. These topics were then used to develop a codebook consisting of broad category labels pertaining to interview questions (e.g., challenges, attendance), as well as specific concepts emerging across interviews (e.g., initial skepticism). One author coded all transcripts and developed a thematic matrix that organized and clustered codes. Authors AS & LJC subsequently refined findings through iterative reading and discussion of coded data and the thematic matrix. We used established strategies to enhance rigor including conducting frequent interview and analysis debriefing meetings, using multiple researchers to review and interpret the data, and drafting memos that constituted an audit trail of key analytic decisions and processes (Creswell, 2003).

2.5.3. Integration of methods

Collection and analysis of quantitative and qualitative data occurred simultaneously. Interim quantitative findings further guided the focus of qualitative analyses when identifying key contextual and process factors that contributed to understanding rates of attendance and satisfaction. Interim qualitative findings were used to identify areas for additional quantitative analysis. Findings emerging from each analysis were then further combined during the final interpretation phase, with all authors reviewing results and clustering qualitative findings from the thematic matrix in ways that elaborated and expanded on quantitative results.

3. Results

3.1. The process of clinic conversion to telehealth

3.1.1. Staff perception of the telehealth conversion

All staff expressed initial skepticism with delivering clinic services via telehealth. Concerns focused on whether clients would attend, the potential for loss of non-verbal cues and interpersonal connection, and the little time that staff had to plan and adapt services for virtual delivery. Though staff perceived the shift to telehealth as slightly more challenging for themselves than clients, they emphasized that they “learn[ed] to navigate” the technology and virtual interaction fairly quickly.

“At first, I was really skeptical about how much I'd be able to do clinically through this platform…for my work, I feel like I take so much information in from body language, facial expression and whatnot. I've found I've gotten used to it and it hasn't hindered me as much as I would have thought before." (Therapist)

Staff highlighted that continuity of care had been preserved to a great extent with a swift transformation to virtual services that allowed the full range and structure of supports to still be offered.

“something crazy, like, three days before we were doing virtual….we've been able to replicate and keep in place essentially almost all of the supports that we had before” (Therapist).

Group clinicians also noted the benefit of increased flexibility to better adapt scheduling to client capacity for engagement, for example offering shorter, more frequent breaks, or reducing session duration but increasing frequency.

While the switch to virtual services was viewed largely positively, staff also had to adapt to challenges. The conversion impacted staff communication, particularly since it constrained their ability to have informal, but essential, conversations that had previously been spontaneous and unscheduled. In addition to formal systems that were put in place to ensure consistent communication (e.g., end-of-day e-mail debriefs), staff credited their existing strong relationships, team mentality, and increased support from supervisors. They maintained frequent interactions with each other via different media (e.g., phone, text, email) to facilitate both client care coordination and opportunities for staff to “support each other as individuals.” See Table 4 for a summary of qualitative themes and clinical practice implications.

Table 4.

Summary of qualitative themes and practice implications.

Themes	Clinical Implication
Skepticism about telehealth	Provide technical support & training, provide supervision on adapting material for telehealth, support staff via check-ins & flexibly scheduled forums to discuss concerns.
Client care challenges	Develop workflows and protocols specific to providing care via telehealth. Regularly review with staff best practices for managing client safety via telehealth. Proactively address regulatory changes.
Virtual etiquette	Provide explicit orientation to etiquette expectations with client input. Offer individual coaching as needed to facilitate social competency with telehealth.
Managing group dynamics	Adapt material, session length and structure to be engaging on virtual platforms.
Communication challenges	Encourage end of day clinical debriefs among the care team, check-in with clients after already scheduled encounters, use text & phone for outreach.
ZOOM fatigue	Offer support to staff, wellness guidance, schedule breaks and intersperse other work tasks.

3.1.2. Impact of organizational structure on the conversion process

The “nimble” and “proactive” clinic organizational culture facilitated rapid transition and the ability to maintain continuity of care. The overarching faculty practice organization (FPO) provided resources and expertise that facilitated the development of policies and procedures. Workflows and infrastructure were developed in anticipation of regulatory change, rather than in response.

“we decided, let's assume they will deregulate and be prepared for it…we were all set to go the minute they deregulated” (Leadership).

Clinicians required training on using telehealth platforms for group services and needed a way to access resources (e.g., shared drives) from home. Client safety and privacy were salient concerns, given the group-based format. This required development of new workflows and protocols (e.g., procedures for responding to someone experiencing crisis) and combined top-down FPO recommendations with staff also “tweak[ing] their workflows” to ensure privacy and safety.

3.2. Acceptability of telehealth

3.2.1. Client acceptance of telehealth

Ninety three percent% (n=56) of the 60 clients enrolled at the time of conversion agreed to maintain their specific treatment plans virtually. Four clients (7%) opted out of telehealth services at time of conversion. Over the 12-weeks following telehealth conversion, the RS documented intakes (n_post1=8; n_post2=4), discharges (n_post1=2; n_post2=2), and enrollees who opted out after spending some time in telehealth sessions (n_post2=2).

3.2.2. Client Satisfaction

The survey response te was 31% (n=20 from N=64); however, two respondents’ surveys were discarded because of incompleteness. All respondents who completed the questionnaire (n=18) provided a score >23, suggesting satisfaction with the telehealth services. Scores on the CSQ ranged from 24 to 32 with a median score of 28.5 (Mean=28.5; SD=3.2).

In response to the question, “If there were no health risks, would you prefer to have your sessions conducted in person?”, 78% of respondents denoted that they prefer in-person sessions. In response to, “Telehealth sessions are as good as in-person sessions for receiving the help I want.”, 50% of the respondents reported that they believe that telehealth sessions are “as good” as in-person sessions, while half did not believe them to be “as good”. Concordance between the two prompts found that 22% of the respondents both preferred telehealth sessions and believed that telehealth sessions were as good as in-person. Two clients chose to write in comments about telehealth:

They are the same quality as in person, I would do online because I live far from the clinic. (Client)

Telehealth is okay, but I prefer in-person vastly more. (Client)

3.2.3. Clinician and administration perspective of telehealth acceptance

Staff believed that telehealth was well-received by most clients, noting that some clients appeared more comfortable with telehealth sessions than in-person. They attributed this to clients’ familiarity with technology, routine use of virtual platforms for socializing, and being in their home environment.

“For some of them, they've been sharing more personal or affective issues than they would..[one] person had a hard time sharing emotions when we were in-person. I've gotten more content than I would have before.” (Therapist)

Nevertheless, staff found telehealth more challenging for clients who had technology or gaming addictions, or symptoms associated with attention deficit hyperactivity disorder (ADHD) or ASD.

“There are cues that get lost online…working with folks who have troubles with focusing attention, that's something that's been a real challenge on this platform. Whether they're bringing up other things on their computer…sustaining their attention, staying connected in general..” (Leadership)

Despite high attendance rates, staff also experienced some increased apprehension regarding clients missing sessions or being unresponsive to outreach. Given that many mundane reasons for non-attendance (e.g., transportation) were no longer applicable, clinicians worked to manage their concerns about client safety.

“maybe I'm a little more vigilant about calling someone, following up quickly. …How come they didn't make it…There wasn't a train issue…So something prevents someone from getting on the computer. That's concerning.” (Leadership)

There were also increased feelings of exhaustion, with more staff time required to find or adapt materials, prepare, and plan session curricula, coupled with more energy needed in-session to manage group processes virtually.

“familiarizing myself with all the technology and all the materials that are needed. That [takes] a bit more planning…” (Leadership)

“if I had to say one negative thing, is that at the end of the day, I'm exhausted. I've never been this [much] on a screen as I am right now…it's all day on Zoom, it takes all my energy.” (Therapist)

This “Zoom fatigue” was further exacerbated by difficulties managing boundaries between personal and professional lives. The blurred schedules, after-hours communication, and additional preparation represented a downside to working remotely. Finally, while staff technological competency increased over time, almost all noted that additional team-based instruction on the telehealth platforms would be beneficial.

3.2.4. Clinician perspective on using telehealth to communicate with clients

Like the constraints on spontaneous communication among staff, there was a loss of spontaneous, in-the-moment interactions with clients that had allowed staff to informally check-in and foster rapport. Telehealth required more purposeful, formal, and planned communication with clients.

“you can't just stop someone in the hallway.” (Therapist)

“a lot of the coordination and care management happens on-the-fly in a live clinic… And sometimes that happens in the Zoom group as well, but it's, I think, harder to create that space in that same way of caring and intimacy…they have to happen much more deliberately now….” (Therapist)

Staff attempted to increase communication and individual follow-up across different media, but still found non-responsiveness from clients more challenging because the remote service environment left staff with fewer options for check-ins, other than involving clients’ families, potentially compromising therapeutic alliances.

3.2.5. Clinician perspective on group dynamics

Overall, staff noted that group dynamics in virtual sessions were largely positive and similar to in-person sessions, with clients interacting with one another and not responding solely to the group leader. Virtual sessions also allowed staff to take advantage of multi-media opportunities to diversify service delivery.

“in the group format there are some interesting ways to engage with patients that are not as easy to do in-person..things like showing videos, that were harder [on-site].” (Admin.)

Additionally, relying on virtual interaction allowed for a positive shift in the dynamic between staff and clients, as clients could take on the role of expert/teacher when it came to troubleshooting technology.

“if I have a problem with sharing my screen or doing something, I think it helps them feel good if they're helping me.” (Therapist)

Nevertheless, there were challenges related to increased interruptions and distractions during sessions. The clinic adopted a collaborative approach to help clients navigate “the social graces of engaging in online group therapy.”

“what should your background be? What do you wear? How do you look at the camera? Do you eat while you're doing this? We handled that by asking the people who were most challenged to work with us to come up with…web-etiquette guidelines and they did a super job….” (Leadership)

Managing group dynamics in virtual sessions required more active facilitation, including redirecting inappropriate behaviors, prompting participation from individual clients, and adapting to the constraints of a virtual world where “silence is not a therapeutic tool.” Given that everyone was navigating a new form of group interaction and that clients had limited ability to socialize, clinicians faced dilemmas of how to balance clients’ interpersonal needs with planned evidence-based skills training.

“‘I'm going to give you some information’…and then allowing for them to kind of have a little space to be spontaneous. And sometimes that means go off track a little bit.” (Leadership)

3.3. Service utilization

3.3.1. Client service utilization data

In the six-weeks prior to the telehealth conversion (pre), the clinical sample (n=60) attended an average of 22.58 (SD=14.02) sessions (3.76/week) while missing an average of 5.63 (SD=5.71) sessions (0.94/week). Following telehealth conversion (post1), 56 in-person participants and 8 newly consented individuals accepted telehealth (n=64) attended an average of 22.48 (SD=15.87) sessions (3.75/week), while missing an average of 4.31 (SD=4.13) scheduled sessions (0.72/week). During the subsequent six-week period (post2), telehealth participants (n=62) attended an average of 23.53 (SD=14.89) sessions (3.92/week), while missing an average of 2.37 (SD=2.12) scheduled sessions (0.4/week). During the 18-week study timeframe, there were no documented psychiatric decompensations or referrals to higher levels-of-care.

Session attendance did not significantly differ over time or between in-person and telehealth formats. The mean no show/cancellation rate was 37% less during Time 3 compared to no show/cancellations while sessions were held in-person (B= -.47, p < 0.05). Multilevel model building found that Time 3 and individuals with at least one psychotic episode were associated with fewer missed sessions (AIC= 911.72). See Tables 2 & 3 for model summaries and comparisons.

Table 2.

Generalized linear mixed models predicting no show/missed sessions, with (Time | ID) as random effects (n = 72, 186 observations)

Model	Variables	Estimate	Standard Error	Z score	p value
Time + (Time | ID) Time + Psychotic + (Time | ID)	Intercept Time 2 Time 3 Intercept Time 2 Time 3 Psychotic	1.12 0.02 -0.47 1.26 0.001 -0.50 -0.49	0.17 0.14 0.22 0.18 0.13 0.21 0.20	6.55 0.15 -2.18 7.21 0.004 -2.35 -2.38	*** * *** * *

Note: p < 0.05; **p < 0.01; ***p < 0.001.

Table 3.

Model comparison between nested intercept-only model and generalized linear mixed models predicting no show/cancelled appointments

	AIC	BIC	Deviance	Chi Square	p value
Time | ID	923.53	936.43	-457.76
Time + (Time | ID)	915.37	934.73	-451.69	12.15	**
Time + Psychotic + (Time | ID)	911.72	934.30	-448.86	5.65	*

Note: p < 0.05; p < 0.01: ***p < 0.001.

3.3.2. Clinician and administrator perspective on client service utilization

Staff also observed high attendance and credited clients’ continued engagement with the speed of the transformation, consistency of services, staff responsiveness, growing familiarity with technology platforms, and the clients and staff having a “collaborative spirit.”

“I do see people caring about one another, reaching out and genuinely being interested in one another. I think that helps with the group retention.” (Therapist)

Echoing quantitative findings, staff believed that telehealth was particularly well-received by clients who had a diagnosis involving psychosis. Staff reported a few clients had technology-related concerns, such as “paranoia associated with being recorded, privacy, and allowing us to see them.” However, this was an exception and primarily affected only initial engagement virtually, with attendance generally maintained over time.

“the people who have…a diagnosis of a psychotic disorder, have all stayed. And the feedback from the group leaders…has been that they continue to be highly engaged.” (Therapist)

Not having to come to the clinic meant that the threshold for attendance was reduced, allowing clients who were reluctant to be around others, had difficulty traveling, or lived out-of-state to maintain access to services.

While viewed positively, staff had lingering concerns that, for some clients, long-term telehealth utilization may hinder recovery. The routine and engagement associated with traveling to a clinic may enhance treatment investment and pro-health behaviors. To encourage alternatives for broader community participation, staff increased suggestions for virtual educational, recreational, and social activities.

4. Discussion

The COVID-19 crisis has yielded unprecedented challenges for the delivery of clinical services for people with SMI. This mixed methods study examined client, clinician and administrative perspectives on group-based telehealth acceptability and the factors that impacted rapid conversion and acceptance. This provided insights into the procedural, clinical and administrative flexibility that contributed to successful telehealth conversion.

4.1. Client service utilization & satisfaction

While studies have highlighted the feasibility of telehealth treatment, this is the first known study examining a group-based telehealth RS serving a SMI population using a time-stratified comparison between in-person and telehealth formats. Session attendance remained stable over time, from six-weeks before to twelve weeks after telehealth conversion. When examining within-subject and between-subject factors associated with missed sessions, a diagnosis of psychosis and longer time participating in telehealth services were found to be associated with fewer missed sessions. This aligns with research suggesting the potential “positives” of telehealth for this specific population (Sharp et al., 2011), but may be counterintuitive, given that technology access is limited in the SMI population (Torous and Keshevan, 2020). In this study clients with SMI had access to technology, suggesting when access is not a barrier, people with SMI acclimate to telehealth, and sustain rates of service utilization.

All the clients surveyed endorsed high levels of satisfaction with telehealth services. However, the majority (78%) of respondents indicated a preference for in-person sessions, and when asked to compare telehealth sessions with in-person, about half of the respondents found them “as good.” Only a fifth (20%) of the sample endorsed a preference for telehealth, believing it was “as good” as in-person. This may suggest that individual preferences play an important role in telehealth's future, emphasizing shared decision-making in clinical care (Dixon et al., 2016). However, since satisfaction surveys were completed anonymously, the researchers could not associate satisfaction, service utilization or covariates.

4.2. Clinician & administrator perspectives on group-based telehealth

Paralleling other studies, clinicians endorsed near universal skepticism prior to and in the early stages of telehealth conversion (Brooks et al., 2013; Wade et al., 2014; Markowitz et al., 2020). Mitigating skepticism, stakeholders reported how nimble leadership coupled with niche contingency planning and specific training in workflow adaptations facilitated successful conversion. Prior to the COVID-related stay-at-home orders, administrators anticipated the changing regulatory landscape and prospectively formulated the workflow, technology, and workforce adaptations. Meanwhile, clinical staff leveraged their strong relationships with clients and stakeholders to convert all in-person services to telehealth formats.

Attendance rates were high and remained stable throughout the telehealth timeframe; however, the shift necessitated modifications to group facilitation and care management. While most clients maintained their treatment plans, clinicians adopted more flexible and visually engaging group content. These adjustments were in response to interpersonal and cognitive factors that interfered with virtual group dynamics and process. Clinicians perceived that younger clients had increased comfort with and access to technology, but tended to be more susceptible to distractions. Since most clients utilized their laptop to attend groups, a myriad of distractions (e.g., digital notifications, open web-browsers, homework assignments) introduced new clinical challenges. Additionally, clients with neurocognitive or attentional issues (e.g., ADHD, ASD) required additional instruction and supports. Beyond group content, clinicians considered the structuring and social forces that are missing in a virtual clinic, such as the behavioral activation, routine-building and incidental socialization associated with travel. Clinical follow-up between care coordinators and clients also required an adapted approach. With limited responsiveness to email and phone communication prior to telehealth, these communication gaps became more pronounced post-conversion. Clinicians could no longer approach clients in the clinic to expedite contact or initiate impromptu check-ins. These communication gaps exacerbated clinician concerns about safety, suicide risk and missed sessions. Despite their reported reticence, clinicians were more likely to mobilize parents or stakeholders, stressing the importance of multilayer support.

4.3. Limitations

The focus is on telehealth conversion and acceptance within a single, private clinic that serves middle/high income clients who have access to adequate technology and space, and results may not generalize to other settings and populations. There was minimal racial and ethnic diversity among the clients. Sample sizes, for both clients and clinicians, were relatively small and required caution in data interpretation. Beyond the anonymous satisfaction questionnaire, there were no corresponding client interviews. Additionally, the short follow-up period makes it difficult to determine longer-term trends in service utilization.

4.4. Conclusions and future directions

In the context of the COVID-19 crisis, this group-based RS was able to quickly transition to a completely virtual format that was accepted by the vast majority of clients, who continued to utilize services at rates similar to the in-person format. The study provides evidence that, with a nimble and person-centered approach, telehealth groups are a viable and satisfactory option to support the recovery of clients with SMI who have access to technology. While this study found diagnosis of a psychotic disorder predicted fewer missed sessions, future research is needed to consider the many variables that could potentially impact telehealth utilization in this population. Access to technology is a known concern, but clinical variables like illness or treatment duration and symptom profile could also have a predictive role. Service variables such as individual versus group treatment modality, intervention focus (e.g., medication, CBT, supportive employment), or provider and setting characteristics could potentially impact telehealth acceptance. Research with larger more diverse samples is needed to appreciate the multiple factors that can impact telehealth acceptance.

CRediT authorship contribution statement

David A. Lynch: Investigation, Data curation, Formal analysis, Writing – original draft. Ana Stefancic: Investigation, Formal analysis, Writing – original draft. Leopoldo J. Cabassa: Investigation, Formal analysis, Writing – review & editing. Alice Medalia: Supervision, Writing – review & editing.

Declaration of Competing Interest

None.