Abstract
Background
Interventions with patients with Serious Mental Illness (SMI) had to adapt rapidly to the COVID-19 safety restrictive measures, leading to the widespread use of teletherapy as an alternative.
Objectives
The aims of this study were to compare the use of different intervention modalities with patients with SMI during the first wave of the pandemic and examine their association with emergency hospital visits and hospitalization rates six months later.
Methods
Records of 270 service users of fifteen outpatient mental health services across Spain, were retrospectively assessed. We retrieved clinical data and data on the modality of intervention received (in-person, over the phone, videoconferencing) in three time points (before, during and after the first COVID-19 wave). Also, data were retrieved regarding the frequency of their emergency hospital visits and hospitalization rates, two, four and six months later.
Results
During the first wave of the pandemic, teletherapy (over the phone and videoconferencing) was the modality most widely used, whilst in-person therapy sessions were significantly reduced, though they seemed to return to pre-COVID levels after the first wave had passed. Importantly, patients receiving teletherapy during the first wave seemed to have significantly fewer emergency visits and hospitalization rates four and six months later (χ2 = 13.064; p < .001). Multilevel analyses revealed that patients receiving videoconferencing interventions had fewer hospitalizations six months after the first wave (OR = 0.25; p = .012).
Conclusions
Under challenging circumstances as those created by the COVID pandemic, teletherapy is a useful tool for protection against hospitalizations and can be used as an alternative to in-person therapy, to ensure continuity of care for patients with SMI.
Introduction
The impact of the COVID-19 pandemic [1] on physical and mental health [2] has been widely reported in the literature. Social distancing, uncertainty about the crisis caused by the pandemic, and the consequent economic collapse, may negatively affect mental health, particularly in individuals with pre-existing mental disorders [3, 4]. Under such circumstances, individuals with Serious Mental Illness (SMI) have been shown to experience greater anxiety than people without previously reported psychopathology [5].
Social distancing practices may pose a great negative impact on individuals with psychotic disorders, as they are known to have small and low quality social networks [6], and are under increased risk of suicide due to isolation [7]. Furthermore, their ability to cover their basic needs may also be compromised, as many of them rely on community services (i.e. employment, occupational, and mutual support groups) that are less accessible under the circumstances created by the pandemic [8].
At the same time, the health crisis caused by the COVID-19 pandemic has posed a major challenge to mental health services. At the peaks of the pandemic, in-person interventions were limited, as they were considered a risk regarding coronavirus transmission. However, since continuity of care plays a key role in preventing the exacerbation of symptoms, and in reducing emergency consultations, hospital admissions, and suicide attempts [9], it was considered vital to find alternative ways of maintaining that continuity without exposing people with SMI to a greater risk of infection.
Alternatives to in-person care were in fact implemented with the aid of teletherapy, un umbrella term for what are known as telematic psychological interventions (telepsychotherapy and e-therapy) [10, 11]. Teletherapy is based on information and communications technology (computer-based Internet tools, mobile and land telephone calls, emails, fax, text messages, and videoconferencing consisting of patient-clinician communication through video consultations) [12]. It is a feasible and effective therapy modality that does not inhibit the therapeutic relationship [13–15] and seems to be widely accepted by both patients and professionals [16]. Despite some reported barriers associated with fears regarding loss of confidentiality, privacy, and legal regulations [14, 15], teletherapy seems to be a valid alternative for treating common mental health disorders, such as anxiety, depression, post-traumatic stress, and eating disorders [17, 18]. Videoconferencing, for example, has been found to be efficient with patients suffering from a variety of mental disorders [19] and is considered a useful tool for maintaining key aspects of the therapeutic process, such as therapeutic alliance [20]. Similarly, help lines are useful when patients do not have an adequate social network [21], while over the phone interventions are effective in reducing depressive symptoms [22, 23]. Teletherapy has been successfully used to implement several treatment models, including cognitive behavioral therapy [24], dialectical behavioral therapy [25], interpersonal therapy [22], and psychoanalysis [26], among others.
But despite the reported benefits of its use with individuals suffering from different mental disorders [19, 27], it has been barely explored in individuals with SMI, and even those studies that have been carried out rarely included patients with increased symptom severity [28]. However, the provision of therapy through telephone calls, Internet, and videoconferencing does, however, seem feasible in patients with schizophrenia and related conditions [29]. The results of a recent systematic review [30] comparing the effectiveness of in-person and videoconferencing interventions in patients with psychosis showed similar levels of effectiveness in terms of health, psychiatric symptoms, and functionality. Also, in two of the studies included in this review treatment adherence was greater when videoconferencing was used. Overall, patients displayed high levels of satisfaction with teletherapy, this modality always being preferred to the waiting list, or having to travel a long distance to the hospital.
One of the reasons behind the limited use of teletherapy with patients with SMI and the lack of research in this regard may be that teletherapy is frequently studied as a complement to treatment as usual [31], and thus, its efficacy as the primary therapeutic modality is not clearly recognized [28]. Also, most studies are characterized by methodological shortcomings, such as small sample sizes, absence of control groups, lack of randomized clinical trials, or short follow-up periods [27, 30]. Importantly, there may also be some sample selection bias, with the inclusion of patients and therapists who have more favorable attitudes towards this intervention modality [28]. Moreover, the use of digital technology among individuals with SMI is lower compared to the general population [32].
However, the COVID-19 pandemic has forced a transition from in-person therapy to teletherapy in patients with more severe conditions, even among patients and health professionals whose attitudes that were not so positive towards this intervention modality [17]. To date, no studies have been carried out into teletherapy outcomes with patients with SMI under adverse psychosocial and health circumstances, such as those created by the COVID-19 pandemic. During the SARS epidemic in 2003, smartphones and internet services were not widely available, and online mental health services were scarce [33], so the study of teletherapy applications was not considered relevant.
It is therefore necessary for research into the impact of COVID-19 to focus on the provision of mental health care for the most vulnerable groups, such as the individuals with SMI. Many of these patients experienced interruptions to the continuity of their care, due to the measures taken to reduce infection rates over the first few months of the pandemic, while others managed to maintain contact with their services through teletherapy. To the best of our knowledge, to date no studies have examined how services were provided to such patients, and how the differences may be associated with a risk of hospitalization.
Therefore, the present study aimed to: a) explore the types of care offered to people with SMI during the first COVID-19 wave, the alternatives used when in person interventions were not possible, and the changes in the modality of interventions used over time (before the first wave, during lockdown, and after the first wave); b) examine whether receiving teletherapy, compared to not receiving teletherapy during the lockdown was associated with the frequency of visits to the emergency department and of hospital admissions, two, four, and six months after the lockdown; and c) examine if different teletherapy modalities are associated with hospitalization rates six months after the lockdown.
Materials and methods
The Strobe checklist was used to prepare the manuscript (See S1 Checklist).
Study design
Retrospective multicenter cohort study.
Setting and participants
The Spanish mental healthcare system
Mental healthcare provision in Spain is quite complex. Among other facilities it includes a) the community mental health units, which constitute the first level of care units for people with SMI and provide both outpatient and homecare service; b) the mental health day hospitals, which constitute an intermediate resource between community mental health and hospitalization units, and provide specialized care in an outpatient/midday-stay mode; c) the inpatient units in general hospitals, providing full and partial hospitalization. Emergencies are handled by the general hospital emergency departments (psychiatry section), as well as the outpatient mental health units. For further information see: National Health System [34]. Fifteen community mental health hospitals in Spain were selected using stratified sampling, to achieve generalization in the Spanish population of people with SMI. Serious Mental Illness (SMI) was defined based on the ICD diagnosis, and on the intensity of the required care required [35]. This criterion was implemented by selecting all the patients who, due to the severity of their condition required follow-ups in an outpatient mental health hospital during the time period of the study. As this is a retrospective study based on clinical records, only patients who fulfilled the above criteria were included.
Patients with incomplete data were excluded from the study.
The total sample consisted of 270 people with SMI over 18 years old of age, nearly reaching the minimum required sample size of (N = 272) as calculated using the GPower analysis [36]. To calculate the sample size, we considered a 20% relapse rate for the group that received teletherapy and 30% relapse rate for the group that did not, with a potential sample loss of 15%, a Confidence Interval of 95% and 80% statistical power (see S1 File).
Variables and data sources/measurement
Three two-month observation periods were established for the first wave of COVID-19 (2020): the period before the pandemic (January 16 to March 15), the lockdown period (March 16 to May 15), and the period following the first wave (May 16 to July 15).
The following variables were collected based on the patients’ clinical records:
Sociodemographic variables: sex, age (in years), living status, occupational status, maximum level of education attained.
Clinical variables: Diagnosis based on the ICD-10 classification [37].
Treatment Adherence (yes/partially/no).
Treatment adherence was determined using the Medication Possession Ratio, defined as the proportion of time when medication supply is available [38].
Use of psychological interventions (yes/no): in-person (individual/group), over the phone, videoconferencing (individual/group)
Modality of intervention received (in-person/teletherapy).
Type of teletherapy received (videoconferencing individual/group, over the phone)
Intervention characteristics
In-person: once per week, individual (45–60 minutes)/group (90 minutes)
Over the phone: once per week, individual (30–45 minutes). Sessions were held using the health service’s landlines, following the same procedures as routine calls.
Videoconferencing: once per week, individual (30–45 minutes, one to one basis)/group (90 minutes, six to eight participants). The sessions were held through the health system enterprise video-calls software and were similar across sites.
Outcome variables: assessed two, four, and six months after the end of the first wave.
Percentage of hospitalizations (defined as admissions to a Mental Health Hospitalization Unit) calculated against the study sample
Mean number of visits to a Mental Health Emergency Department.
Procedure
After receiving the approval of the corresponding ethics committees of the participating centers, data were collected retrospectively (October to November 2020) based on the patients’ clinical records. The overall approval was obtained by the Medical Research Ethics Committee of the Andalusian Government stating the following: "This study fulfills the ethical principles required for conducting studies of this type" (REF: 202077133825). Patients were informed verbally and in writing about the project aims, and signed an informed consent. A password-protected database with sound error prevention mechanisms was designed, which granted access only to the study researchers; clinical data were handled without patient identification details. The study was performed in line with the principles of the Declaration of Helsinki. Data privacy requirements were met in accordance with the European Union legislation.
Statistical methods
Statistical analyses were performed using the SPSS software v.21.0 [39], with a statistical significance of p-values < .05. We first examined the sociodemographic and clinical characteristics of our sample with descriptive statistics.
To study the first aim, we used a series of Cochrane’s tests to determine whether there were statistically significant differences in the proportion of patients making use of the different intervention types over the three time points (before, during, and after the lockdown). We also performed pairwise comparisons through a series of McNemar’s tests, to check whether the proportion of patients using the interventions was sustained or varied from one time point to another (before, during and after the lockdown). For these analyses, we applied a Bonferroni correction to the significance levels as follows: p = .005/3 = .016. For the second aim, we used Chi-square tests to compare the hospitalization rates at two, four, and six months after the first wave, and Student’s t-tests were used to compare the mean number of visits to the Mental Health Emergency Department, between the patients who received teletherapy during the lockdown and those that did not. For the third aim, we first used Chi-square tests to compare the hospitalization rates six months after the lockdown between the patients that received each type of teletherapy (over the phone, videoconferencing individual and group) and those that did not. We then performed a mixed effects multilevel logistic regression including random effects for each patient nested in the three time points (before, during and after the first wave), and adjusting for individual characteristics, interdependence effects and confounding variables. We established Level 1 for interventions received and level 2 for the individual characteristics of the patients nested in them. The dependent variable was the frequency of hospitalizations 6 months after the first wave. In the first step the null model resulted in an ICC = 0.36, explained by level 2 variables, confirming the consideration of multilevel models. The first model included level 1 variables: videoconferencing (individual and group were considered together, due to lack of sufficient cases), over the phone, and in-person. For the second model we added the level 2 sociodemographic variables (sex and age) and for the third model we added the level 2 clinical variables (diagnosis and treatment adherence).
Results
Participants and descriptive data
Our sample consisted of 120 men and 150 women (55.6%), aged between 18 and 67 years (M = 39.90 years). Most of the participants had primary (35.8%) or secondary (41.5%) education, and 14.8% had university studies. The majority of the patients lived with their family of origin (28.9%), their own family (28.9%), or alone (17%); 29.3% were retired, 26.3% were unemployed, 20% had short-term disability, and 16.7% were working (Table 1). The most frequent diagnosis was Schizophrenia or other psychosis (30.4%), followed by personality (27.8%), bipolar (10.4%), or major depressive disorders (9.8%). The majority showed treatment adherence (85.9%). No statistically significant differences were found between the patients that received some type of teletherapy and those who did not, in terms of sex (χ2 = 0.208; p = .649), age (t = -0.059; p = .953), occupational status (χ2 = 6.286; p = .279), level of education (χ2 = 2.993; p = .559) diagnosis (χ2 = 3.853; p = .426), or treatment adherence (χ2 = 0.422; p = .810).
Table 1. Sociodemographic and clinical characteristics of the sample.
Total N (%) | Patients who received teletherapy N (%) | Patients who did not receive teletherapy N (%) |
t/χ2 p-value |
|
---|---|---|---|---|
Total Sample N | 270 (100%) | 175 (64.8%) | 95 (35.2%) | |
Age Mean (SD) | 39.90 (11.81) | 39.93 (11.12) | 39.84 (13.06) | t = -0.059 |
p = .953 | ||||
Gender | χ2 = 0.208 | |||
Female | 150 (55.6%) | 99 (56.6%) | 51 (53.7%) | p = .649 |
Male | 120 (44.4%) | 76 (43.4%) | 44 (46.3%) | |
Living status | χ2 = 5.664 | |||
Family of origin (parents w/wo siblings) | 78 (28.9%) | 52 (29.7%) | 26 (27.4%) | p = .462 |
Own family (partner and/or children) | 78 (28.9%) | 54 (30.9%) | 24 (25.3%) | |
With friends or siblings | 16 (5.9%) | 8 (4.6%) | 8 (8.4%) | |
One parent w/wo siblings | 37 (13.7%) | 23 (13.1%) | 14 (14.7%) | |
Other | 7 (2.6%) | 6 (3.4%) | 1 (1.1%) | |
Single household | 46 (17%) | 26 (14.9%) | 20 (21.1%) | |
Supported housing | 8 (3%) | 6 (3.4%) | 2 (2.1%) | |
Occupational status | χ2 = 6.286 | |||
Student | 20 (7.4%) | 13 (7.4%) | 7 (7.4%) | p = .279 |
STD | 54 (20%) | 39 (22.3%) | 15 (15.8%) | |
Retired, pensioner | 79 (29.3%) | 48 (27.4%) | 31 (32.6%) | |
Unemployed | 71 (26.3%) | 50 (28.6%) | 21 (22.1%) | |
Working | 45 (16.7%) | 24 (13.7%) | 21 (22.1%) | |
Volunteer/Mutual support agent | 1 (0.4%) | 1 (0.6%) | 0 (0.0%) | |
Level of Education | χ2 = 2.993 | |||
No schooling | 8 (3%) | 3 (1.7%) | 5 (5.3%) | p = .559 |
Primary (BGE-CSE) | 96 (35.6%) | 61 (34.9%) | 35 (36.8%) | |
Secondary education | 112(41.5%) | 76 (43.4%) | 36 (37.9%) | |
University studies (BSc) | 40 (14.8%) | 26 (14.9%) | 14 (14.7%) | |
Postgraduate studies (MSc-PhD) | 14 (5.2%) | 9 (5.1%) | 5 (5.3%) | |
Diagnosis (ICD) | χ2 = 5.743 | |||
Schizophrenia/Other psychosis | 82 (30.4%) | 50 (28.6%) | 32 (33.7%) | p = .570 |
Bipolar disorder | 28 (10.4%) | 18 (10.3%) | 10 (10.5%) | |
Personality disorders | 75 (27.8%) | 53 (30.3%) | 22 (23.2%) | |
Depressive disorder | 26 (9.6%) | 17 (9.7%) | 9 (9.5%) | |
Anxiety/Other disorders | 59 (21.9%) | 37 (21.1%) | 22 (23.1%) | |
Treatment Adherence | χ2 = 0.422 | |||
Yes | 232(85.9%) | 152(86.9%) | 80 (84.2%) | p = .810 |
Partially | 27 (10%) | 16 (9.1%) | 11 (11.6%) | |
No | 11 (4.1%) | 7 (4%) | 4 (4.2%) |
Outcome data and main results
Comparison between the different types of intervention before the pandemic, during lockdown, and after the first wave
Table 2 displays the percentages of patients who received the different types of interventions over the three time periods. Overall, there were statistically significant differences across the usage of all types of interventions, as indicated by the Cochrane’s Q results (Table 2). In-person individual interventions decreased from 80.4% before the pandemic to 23% during the lockdown (p < .001), and then increased to 73.7% after the first wave (p < .001); there were no significant differences in the use of in-person individual interventions after the first wave, in comparison with before the pandemic (p = .043).
Table 2. Comparison of interventions received before the pandemic, during the lockdown, and after the first wave.
Type of intervention | Before the lockdown (Jan 16 –March 15) % | During the lockdown (March 16 –May 15) % | After the lockdown (May 16 –July 15) % | Cochrane´s Q | χ2 /p values | ||
---|---|---|---|---|---|---|---|
Before vs during the lockdown | Before vs after the lockdown | During vs after the lockdown | |||||
In-person individual | 80.4 | 23 | 73.7 | Q = 224.573 p < .001 |
χ2 = 142.012 p < .001 |
χ2 = 4.129 p = .042 |
χ2 = 125.823 p < .001 |
In-person group | 29.3 | 1.1 | 23.3 | Q = 99.299 p < .01 |
χ2 = 70.313 p< .001 |
χ2 = 4.500 p = .034 |
χ2 = 54.391 p< .001 |
Over the phone | 4.4 | 60 | 21.9 | Q = 214.048 p> .001 |
χ2 = 144.162 p < .001 |
χ2 = 38.473 p < .001 |
χ2 = 85.983 p < .001 |
Videoconferencing individual | 0 | 3.7 | 1.5 | Q = 11.692 p = .003 |
p = .002 | p = .125 | p = .146 |
Videoconferencing group | 0 | 9.3 | 11.1 | Q = 44.286 p < .001 |
p < .001 |
χ2 = 28.033 p< .001 |
p = .302 |
In-person group interventions decreased from 29,3% before the pandemic to 1,1% during the lockdown (p< .001) and then increased to 23,3% after the first wave (p< .001). There were no significant differences in the use of in-person group interventions after the first wave, in comparison with before the lockdown (p = .034).
The percentage of patients who received over the phone interventions increased from 4.4% before lockdown, to 60% during the lockdown (p < .001), and then significantly decreased to 21.9% after the first wave (p < .001). However, this percentage was still significantly higher than before the lockdown (p < .001).
The percentage of patients receiving an intervention through individual videoconferencing before the pandemic increased from 0% to 3.7% during the lockdown (p < .001) and decreased, although not significantly, to 1.5% after the first wave (p = .146). The differences between the period before and after the first wave (p < .125) were also not statistically significant.
Lastly, patients receiving videoconferencing group interventions increased from 0% before the pandemic to 9,3% during the lockdown (p < .001) and to 11,1% after the first wave (p< .302), with a significant increase in the use of such interventions between the period before the pandemic and the period after the first wave (p< .001).
Differences in hospitalization rates and visits to the mental health emergency department in patients who received teletherapy during the lockdown and those who did not
Hospitalization rates two months after the lockdown did not differ between patients who received teletherapy, and those who did not (p = .068). However, hospitalization rates were significantly lower for the first group of patients at four (p = .004) and six months after the lockdown (p < .001).
Patients who received teletherapy made fewer visits to Mental Health Emergency Department, at two months (p = .030), four months (p = .11NS), and six months (p = .034) after lockdown, than those who did not. The results are shown at Table 3.
Table 3. Hospitalizations and emergency visits two, four and six months after the lock down.
Total (%) | Patients who received teletherapy (%) | Patients who did not receive teletherapy (%) | RR | 95%CI | χ2 /p-value | |
---|---|---|---|---|---|---|
Hospitalizations | ||||||
Two months after the lockdown | 6.7 | 4.6 | 10.5 | 0.43 | 0.18 to 1.06 |
χ2 = 3.334 p = .068 |
Four months after the lockdown | 13.7 | 9.1 | 22.1 | 0.41 | 0.23 to 0.75 |
χ2 = 8.350 p = .004 |
Six months after the lockdown | 20.4 | 13.7 | 32.6 | 0.42 | 0.26 to 0.67 |
χ2 = 13.064 p< .001 |
Number of visits to the Mental Health Emergency Department | Mean (SD) | Mean (SD) | MD | t /p-value | ||
Two months after the lockdown | 0.22 (0.80) | 0.55 (1.64) | 0.33 | 0.03–0.62 | t = 2.185 p = .030 |
|
Four months after the lockdown | 0.42(1.28) | 1.18 (3.52) | 0.76 | 0.18–1.35 | t = 2.571 p = .110 |
|
Six months after the lockdown | 0.70 (2.19) | 1.65 (5.14) | 0.95 | 0.72–1.84 | t = 2.129 p = .034 |
Differences in hospitalization rates six months after lockdown based on the type of teletherapy
Patients who received therapy over the phone had less hospitalizations six months later than those who did not (14.2% vs 29.6%; RR = 0.47; χ2 = 9.340; p = .002). Similarly, those who received videoconferencing interventions during the lockdown had less hospitalizations than those who did not (3.4% vs 22.4%; RR = 0.15; χ2 = 7.840; p = .005).
Table 4 shows the three models built from the multilevel logistic regression analyses. As can be seen, the factor associated with the lowest risk of having a hospitalization six months after the lockdown was receiving an intervention through videoconferencing (OR = 0.25; p = .012), followed by receiving an intervention over the phone (OR = 0.50; p = .003). These findings were maintained when including potential confounders, with lower risk for hospitalizations associated with receiving in-person interventions (OR = 0.56; p = .003), being a woman (OR = 1,50; p = .04) and being older (OR = 1.01; p = .049), whereas diagnosis and treatment adherence were not significantly associated with more hospitalizations (model 3).
Table 4. Multilevel logistic regression models predicting hospitalizations 6 months after the lockdown.
Factor | Model 1: Interventions | Model 2: Interventions +Sociodemographic Factors | Model 3: Interventions +Sociodemographic +Clinical Factors | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
B (SE) | β | p | 95% CI | B (SE) | β | p | 95%CI | B (SE) | β | p | 95%CI | Wald | |
Videoconferencing a | -1.58 (0.55) | 0.25 | .012 | 0.09 to 0.74 | -1.45 (0.55) | 0.23 | .008 | -2.5 to -0.37 | -1.32 (0.55) | 0.27 | .002 | 0.09 to 0.79 | -2.40 |
Over the phone b | -0.69 (0.23) | 0.50 | .003 | 0.31 to 0.78 | -0.71 (0.23) | 0.50 | .002 | -1.16 to -0.25 | -0.75 (0.24) | 0.47 | .002 | 0.30 to 0.75 | -3.13 |
In-person c | -0.57 (0.19) | 0.56 | .003 | 0.39 to 0.83 | -0.59 (0.19) | 0.56 | .003 | -0.97 to-0.20 | -0.62 (0.2) | 0.54 | .001 | 0.36 to 0.79 | -3.10 |
Being a woman d | 0.45 (0,18) | 1.56 | .016 | 0.08 to 0.80 | 0.41 (0.20) | 1.50 | .040 | 1.02 to 2.21 | 2.05 | ||||
Age (in years) | 0.01 (0.01) | 1.01 | .83 | -0.002 to 0.003 | 0.02 (0.01) | 1.02 | .049 | 1.00 to 1.03 | 2.00 | ||||
Bipolar Disorder e | 0.04 (0.25) | 1.04 | .868 | 0.64 to 1.69 | |||||||||
Personality Disorder e | 0.51 (0.30) | 1.66 | .092 | 0.92 to 3.00 | |||||||||
Depressive Disorder e | -0.41 (0.38) | 0.27 | .661 | 0.32 to 1.06 | |||||||||
Anxiety/Other Disorders e | -0.48 (0.28) | 0.62 | .081 | 0.36 to 1.06 | |||||||||
Treatment Adherence f | -0.36 (0.24) | 0.69 | .129 | 0.43 to 1.11 |
Reference groups:
a,b,c Not having received this intervention;
d: Being a man;
e: Psychosis;
f: No/Partial Adherence
Discussion
Key results and interpretation
We performed a retrospective analysis to study the impact of the COVID-19 pandemic on the psychotherapeutic care received by patients with SMI, examining the use of teletherapy alternatives applied when in-person interventions were not feasible. We also examined the potential associations between different types of teletherapy and SMI patients’ hospitalization rates and visits to emergency departments in the medium term.
Our findings suggested that individual in-person interventions were significantly reduced during the lockdown, while group therapy nearly disappeared. On the other hand, teletherapy over the phone was the type most frequently used during the lockdown, perhaps because this was the only type of teletherapy intervention used before the pandemic. Teletherapy via videoconferencing, especially in group format, was introduced during the lockdown, but the percentage of patients who received it was lower in comparison with the over the phone modality. The preference for over the phone interventions may be explained by the scarcity of online platforms available within the Spanish public healthcare system [40], and perhaps also by certain limitations associated with videoconferencing, such as the difficulty of its use [41], poor training [42], or the perception that videoconferencing may increase the clinical workload [43]. Also, previous findings indicate that patients with SMI prefer using the telephone rather than videoconferencing, as they believe it reduces the intensity of the sessions [44]. Notably, to date, Spain has no established deontological framework providing guidelines for the teletherapy practice, and such practices are therefore applied in line with the traditional therapy framework principles. This has created major loopholes [45], especially concerning the information confidentiality and privacy [14, 15].
In our study, the use of the in-person modality returned to previous levels after the lockdown, despite a greater risk of viral transmission. This coincided with a significant decrease in the use of over the phone interventions. These findings indicate that when the restrictions were loosened and the situation was perceived as less critical, there was a clear tendency towards resuming routine intervention modes. Although there was no generalized rejection for teletherapy, professionals still seemed to prefer traditional approaches [46], particularly with high-risk patients [28]. Nonetheless, and albeit to a lesser extent, clinicians continued the implementation of group interventions by means of videoconferencing, as this modality may have offered a good alternative for avoiding the physical gathering of various individuals within the same room, and thus preventing the spread of the virus. It therefore seems that the mental health services opted to offer their patients a range of options with which to guarantee the continuity of services, in line with the recommendations of Kopelovich et al. [47] for good practices in mental health services in times of COVID.
Importantly, for patients who received teletherapy during the lockdown, emergency visits and hospitalization rates were lower four and six months after the first wave of COVID-19. Receiving teletherapy during stressful periods, when in-person sessions are not possible, appears to be a protective factor against hospitalization, particularly in the medium-term. This concurs with previous findings suggesting that over the phone [48] and videoconferencing [41, 49] interventions have levels of clinical effectiveness, similar to those of in-person therapy. The interruption of psychotherapy in times of uncertainty and fear, may certainly have triggered a feeling of loss, helplessness, or sense of vulnerability. Patient-family cohabitation in confined spaces for a long period of time and the emotional discomfort associated with the pandemic, may also have led to tensions within the family affecting expressed emotion levels [50], a well-known risk factor for relapse [51]. In this regard, continuity of care through teletherapy may have offset the negative effects of expressed emotions and have prevented crisis [52]. Importantly, patients that received therapy via videoconferencing seemed to have less hospitalizations six months after the first wave of the pandemic. This supports recommendations that videoconferencing should be used as a means of assuring continuity of care during the pandemic [8]. It also concurs with the conclusions of a systematic review of 65 videoconferencing-based psychotherapy studies that suggested videoconferencing as a feasible alternative to in-person interventions, as it is associated with positive outcomes such as treatment adherence, social functioning, and quality of life levels [53].
Limitations and strengths
This study is the first to explore psychotherapeutic intervention alternatives for individuals with SMI, during the COVID-19 pandemic, a topic that has been largely neglected in the literature. The study’s observational nature, together with the participation of multiple healthcare sites allowed us to draw comprehensive conclusions regarding the clinical care provided in Spain, under the specific conditions created by the COVID-19 pandemic. The extraction of the patients’ data from routine clinical practice records also ensured greater reliability of the findings, while the retrospective design with multiple follow-ups provided us with a unique opportunity to observe the associations of a variety of teletherapy types with the patients’ hospitalization rates after the first wave of the pandemic.
However, the following limitations need to be considered. Firstly, as this is an observational study, it is not possible to establish direct causal relationships, as other confounding variables may play a role in the association between the intervention modalities and the hospitalization rates. Secondly, the study design did not allow for the comparison of teletherapy with in-person interventions, since, due to the special conditions created by the COVID-19 crisis in-person therapy was barely used during the studied period. This explains why the in-person modality was considered as a confounding factor, and not as an independent variable in the multilevel analysis. Thirdly, the retrospective nature of the study may have biased the results, despite the inclusion of objective routine data based on clinical records specifically aimed at reducing such bias. Fourthly, patients were grouped under the Serious Mental Illness condition, and we did not perform a separate analysis for each diagnostic category. Diagnoses in mental health are in fact collections of symptoms and could be explained using completely different lenses to the medical lens [54]. Therefore, including all patients with SMI in our study allowed for an overall and realistic perspective of the routine clinical practice during the first wave of the pandemic. It should be noted that, the multilevel findings suggested that the probability of having a hospitalization six months after the first wave did not seem to vary between patients with different diagnoses. Similarly, previous studies have reported associations between sociodemographic and clinical factors such as sex, age, medication adherence, and hospitalizations [55, 56]. We also included those factors as confounding variables in our multilevel analyses, to control for their potential contribution. Although it lay beyond the scope of this study to focus on such associations, future research should conduct a more in-depth examination of each diagnostic category and of the role played by sociodemographic factors in the context of teletherapy. Lastly, information on theoretical approaches of the specific interventions, was not taken into account for our analyses. This was beyond the scope of the present study, which focused exclusively on the use of teletherapy as a valid alternative in the unique restrictive context of the pandemic.
Future research perspectives
Future research into the topic should compare the implementation of teletherapy in different diagnostic categories of SMI, and also examine its impact on psychosocial outcomes, such as functioning and recovery. Extending the follow-up period would also make it possible to observe any potential changes in the use of teletherapy in the long run, while a detailed description of the specific interventions performed within each therapeutic mode would provide useful information regarding the feasibility of their telematic alternatives. Finally, the application of experimental designs such as randomized control trials would throw light on the effectiveness of teletherapy, beyond the context of the pandemic, as current results are inconsistent [57]. It is worth noting that, to date, no official guidelines exist in Spain for the use of telematic means in healthcare, making this a challenging task in this context [58].
Conclusions
In conclusion, the COVID-19 pandemic seems to have brought changes in the practice of psychological interventions, as in-person interventions decreased, and the use of teletherapy increased, providing an opportunity to explore its applicability with individuals with SMI. This study discusses the implications of the pandemic on the provision of care for individuals with SMI, and opens the door to further research on the effectiveness of teletherapy beyond the pandemic context. Our findings suggest that teletherapy can indeed serve as a valid alternative for protecting patients with SMI against hospitalizations, especially under circumstances where in-person interventions are not feasible.
Supporting information
Acknowledgments
The authors wish to acknowledge the collaboration of the day hospitals section of the Spanish Neuropsychiatry Association.
Data Availability
Our data cannot be shared publicly because they contain sensitive patient information, such as sex, age, living and occupational status, and most importantly, diagnosis and records of mental healthcare visits. Such information is based on the clinical records of specific community healthcare centers, where most of the authors of this paper work as clinicians. Therefore, we consider that there is a risk of potential identification of the patients. To avoid such risk, access to the database can be formally requested through contacting the Biomedical Research Ethics Committee of the Government of Andalusia, Spain in the following address portaldeetica.csalud@juntadeandalucia.es.
Funding Statement
AS received funding by the FPS 2020 – Primary Care Regional Hospitals and CHARES R&D Projects, (within the Project AP-0028-2020-C1-F2) and AS, MS, CM and FP received funding by the Outpatient Mental Health Day Hospitals of the Spanish Association of Neuropsychiatry. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
- 1.Cucinotta D, Vanelli M. WHO Declares COVID-19 a Pandemic. Acta Biomed. 2020. Mar 19;91(1):157–160. doi: 10.23750/abm.v91i1.9397 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Brooks SK, Webster RK, Smith LE, et al. The psychological impact of quarantine and how to reduce it: rapid review of the evidence. The Lancet. 2020;395(10227):912–920. doi: 10.1016/s0140-6736(20)30460-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Moreno C, Wykes T, Galderisi S, et al. How mental health care should change as a consequence of the COVID-19 pandemic. The Lancet Psychiatry. 2020;7(9):813–824. doi: 10.1016/S2215-0366(20)30307-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Yao H, Chen J-H, Xu Y-F. Patients with mental health disorders in the COVID-19 epidemic. The Lancet Psychiatry. 2020;7(4). doi: 10.1016/S2215-0366(20)30090-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.González-Blanco L, Dal Santo F, García-Álvarez L, et al. COVID-19 lockdown in people with severe mental disorders in Spain: Do they have a specific psychological reaction compared with other mental disorders and healthy controls? Schizophrenia Research. 2020;223:192–198. doi: 10.1016/j.schres.2020.07.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Degnan A, Berry K, Sweet D, Abel K, Crossley N, Edge D. Social networks and symptomatic and functional outcomes in schizophrenia: a systematic review and meta-analysis. Social Psychiatry and Psychiatric Epidemiology. 2018;53(9):873–888. doi: 10.1007/s00127-018-1552-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bornheimer LA, Li J, Im V, Taylor M, Himle JA. Correction to: The Role of Social Isolation in the Relationships Between Psychosis and Suicidal Ideation. Clinical Social Work Journal. 2020;48(1):63–63. doi: 10.1007/s10615-020-00751-2 [DOI] [Google Scholar]
- 8.Kahl KG, Correll CU. Management of Patients With Severe Mental Illness During the Coronavirus Disease 2019 Pandemic. JAMA Psychiatry. 2020;77(9):977. doi: 10.1001/jamapsychiatry.2020.1701 [DOI] [PubMed] [Google Scholar]
- 9.Kozloff N, Mulsant BH, Stergiopoulos V, Voineskos AN. The COVID-19 Global Pandemic: Implications for People With Schizophrenia and Related Disorders. Schizophrenia Bulletin. Published online April 28, 2020. doi: 10.1093/schbul/sbaa051 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Joint Task Force for the Development of Telepsychology Guidelines for Psychologists. Guidelines for the practice of telepsychology. American Psychologist. 2013;68(9):791–800. doi: 10.1037/a0035001 [DOI] [PubMed] [Google Scholar]
- 11.Ryu S. Telemedicine: Opportunities and Developments in Member States: Report on the Second Global Survey on eHealth 2009 (Global Observatory for eHealth Series, Volume 2). Healthcare Informatics Research. 2012;18(2):153. doi: 10.4258/hir.2012.18.2.153 [DOI] [Google Scholar]
- 12.Cui L-B, Wang X-H, Wang H-N. Challenges facing coronavirus disease 2019: Psychiatric services for patients with mental disorders. Psychiatry and Clinical Neurosciences. Published online April 1, 2020. doi: 10.1111/pcn.13003 [DOI] [PubMed] [Google Scholar]
- 13.Berger T. The therapeutic alliance in internet interventions: A narrative review and suggestions for future research. Psychotherapy Research. 2016;27(5):511–524. doi: 10.1080/10503307.2015.1119908 [DOI] [PubMed] [Google Scholar]
- 14.Cowan KE, McKean AJ, Gentry MT, Hilty DM. Barriers to Use of Telepsychiatry: Clinicians as Gatekeepers. Mayo Clinic Proceedings. 2019;94(12):2510–2523. doi: 10.1016/j.mayocp.2019.04.018 [DOI] [PubMed] [Google Scholar]
- 15.Smith K, Ostinelli E, Macdonald O, Cipriani A. COVID-19 and Telepsychiatry: Development of Evidence-Based Guidance for Clinicians. JMIR Mental Health. 2020;7(8). doi: 10.2196/21108 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hilty DM, Rabinowitz T, McCarron RM, et al. An Update on Telepsychiatry and How It Can Leverage Collaborative, Stepped, and Integrated Services to Primary Care. Psychosomatics. 2018;59(3):227–250. doi: 10.1016/j.psym.2017.12.005 [DOI] [PubMed] [Google Scholar]
- 17.Poletti B, Tagini S, Brugnera A, et al. Telepsychotherapy: a leaflet for psychotherapists in the age of COVID-19. A review of the evidence. Counselling Psychology Quarterly. Published online May 27, 2020:1–16. doi: 10.1080/09515070.2020.1769557 [DOI] [Google Scholar]
- 18.Shingleton RM, Richards LK, Thompson-Brenner H. Using technology within the treatment of eating disorders: A clinical practice review. Psychotherapy. 2013;50(4):576–582. doi: 10.1037/a0031815 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bashshur RL, Shannon GW, Bashshur N, Yellowlees PM. The Empirical Evidence for Telemedicine Interventions in Mental Disorders. Telemedicine Journal and e-Health. 2016;22(2):87–113. doi: 10.1089/tmj.2015.0206 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bolle SR, Trondsen MV, Stensland GØ, Tjora A. Usefulness of videoconferencing in psychiatric emergencies—a qualitative study. Health and Technology. 2017;8(1–2):111–117. doi: 10.1007/s12553-017-0189-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Fiorillo A, Gorwood P. The consequences of the COVID-19 pandemic on mental health and implications for clinical practice. European Psychiatry. 2020;63(1). doi: 10.1192/j.eurpsy.2020.35 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Dennis C-L, Grigoriadis S, Zupancic J, Kiss A, Ravitz P. Telephone-based nurse-delivered interpersonal psychotherapy for postpartum depression: nationwide randomised controlled trial. The British Journal of Psychiatry: The Journal of Mental Science. Published online February 7, 2020:1–8. doi: 10.1192/bjp.2019.275 [DOI] [PubMed] [Google Scholar]
- 23.Heckman TG, Heckman BD, Anderson T, et al. Tele-Interpersonal Psychotherapy Acutely Reduces Depressive Symptoms in Depressed HIV-Infected Rural Persons: A Randomized Clinical Trial. Behavioral Medicine. 2016;43(4):285–295. doi: 10.1080/08964289.2016.1160025 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Etzelmueller A, Radkovsky A, Hannig W, Berking M, Ebert DD. Patient’s experience with blended video- and internet based cognitive behavioural therapy service in routine care. Internet Interventions. 2018;12:165–175. doi: 10.1016/j.invent.2018.01.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Oliveira PN, Rizvi SL. Phone coaching in Dialectical Behavior Therapy: frequency and relationship to client variables. Cognitive Behaviour Therapy. 2018;47(5):383–396. doi: 10.1080/16506073.2018.1437469 [DOI] [PubMed] [Google Scholar]
- 26.Bakalar NL. Transition from in-person psychotherapy to telephone psychoanalysis. In Psychoanalysis Online 2018. Apr 17 (pp. 103–118). New York: Routledge. ISBN 9781138312425 [Google Scholar]
- 27.Chakrabarti S. Usefulness of telepsychiatry: A critical evaluation of videoconferencing-based approaches. World Journal of Psychiatry. 2015;5(3):286. doi: 10.5498/wjp.v5.i3.286 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Markowitz JC, Milrod B, Heckman TG, et al. Psychotherapy at a Distance. American Journal of Psychiatry. Published online September 25, 2020:appi.ajp.2020.2. doi: 10.1176/appi.ajp.2020.20050557 [DOI] [PubMed] [Google Scholar]
- 29.Kasckow J, Felmet K, Appelt C, Thompson R, Rotondi A, Haas G. Telepsychiatry in the Assessment and Treatment of Schizophrenia. Clinical Schizophrenia & Related Psychoses. 2014;8(1):21–27A. doi: 10.3371/CSRP.KAFE.021513 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Santesteban-Echarri O, Piskulic D, Nyman RK, Addington J. Telehealth interventions for schizophrenia-spectrum disorders and clinical high-risk for psychosis individuals: A scoping review. Journal of Telemedicine and Telecare. 2018;26(1–2):14–20. doi: 10.1177/1357633X18794100 [DOI] [PubMed] [Google Scholar]
- 31.Schuster R, Topooco N, Keller A, Radvogin E, Laireiter AR. Advantages and disadvantages of online and blended therapy: Replication and extension of findings on psychotherapists’ appraisals. Internet Interventions. 2020. Sep 1;21:100326. doi: 10.1016/j.invent.2020.100326 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Robotham D, Satkunanathan S, Doughty L, Wykes T. Do We Still Have a Digital Divide in Mental Health? A Five-Year Survey Follow-up. Journal of Medical Internet Research. 2016;18(11):e309. doi: 10.2196/jmir.6511 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Chan SSC, So WKW, Wong DCN, Lee ACK, Tiwari A. Improving older adults’ knowledge and practice of preventive measures through a telephone health education during the SARS epidemic in Hong Kong: A pilot study. International Journal of Nursing Studies. 2007;44(7):1120–1127. doi: 10.1016/j.ijnurstu.2006.04.019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.National Health System. Mental Health Organisation and Facilities. Institute of Health Information. 2007. Accessed 25 February 2022 https://www.sanidad.gob.es/gl/estadEstudios/estadisticas/docs/siap/SALUD_MENTAL_ORGANIZACION_Y_DISPOSITIVOS.pdf
- 35.National Institute of Mental Health. Statistics Mental Illness website. Updated January 2021. Accessed 22 May 2021 https://www.nimh.nih.gov/health/statistics/mental-illness
- 36.Faul F, Erdfelder E, Lang A-G, Buchner A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods. 2007;39(2):175–191. doi: 10.3758/bf03193146 [DOI] [PubMed] [Google Scholar]
- 37.Andrade SE, Kahler KH, Frech F, Chan KA. Methods for evaluation of medication adherence and persistence using automated databases. Pharmacoepidemiology and drug safety. 2006; 15(8), 565–574. doi: 10.1002/pds.1230 [DOI] [PubMed] [Google Scholar]
- 38.World Health Organization. The ICD-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines. World Health Organization; 1992. [Google Scholar]
- 39.IBM. IBM SPSS Statistics for Windows, Version 21.0. 2012; Armonk, NY: IBM Corp.
- 40.Ramos AC, Buceta BB, Silva ÁFD, Lorenzo RB. La eSalud en España: evolución, estado actual y perspectivas de futuro. Saúde e Sociedade. 2020;29(4). doi: 10.1590/s0104-12902020190886 (article in Spanish) [DOI] [Google Scholar]
- 41.Fletcher TL, Hogan JB, Keegan F, Davis ML, Wassef M, Day S, et al. Recent advances in delivering mental health treatment via video to home. Current psychiatry reports. 2018. Aug;20(8):1–9. doi: 10.1007/s11920-018-0922-y [DOI] [PubMed] [Google Scholar]
- 42.Taylor CB, Fitzsimmons‐Craft EE, Graham AK. Digital technology can revolutionize mental health services delivery: The COVID ‐19 crisis as a catalyst for change. International Journal of Eating Disorders. 2020;53(7):1155–1157. doi: 10.1002/eat.23300 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Perry K, Gold S, Shearer EM. Identifying and addressing mental health providers’ perceived barriers to clinical video telehealth utilization. Journal of Clinical Psychology. Published online March 11, 2019. doi: 10.1002/jclp.22770 [DOI] [PubMed] [Google Scholar]
- 44.Medalia A, Lynch DA, Herlands T. Telehealth Conversion of Serious Mental Illness Recovery Services During the COVID-19 Crisis. Psychiatric Services. 2020;71(8):872–872. doi: 10.1176/appi.ps.71705 [DOI] [PubMed] [Google Scholar]
- 45.González-Peña P, Torres R, del Barrio V, Olmedo M. Uso de las nuevas tecnologías por parte de los psicólogos españoles y sus necesidades. Clínica y Salud. 2017;28(2):81–91. doi: 10.1016/j.clysa.2017.01.001 (article in Spanish) [DOI] [Google Scholar]
- 46.Garcia Lizana F, Muñoz Mayorga I, Yanes López V, Serrano Aguilar P. Las tecnologías de la información y comunicación en salud mental (Telepsiquiatría). Madrid: AETS—Instituto de Salud Carlos III (report in Spanish).
- 47.Kopelovich SL, Monroe-DeVita M, Buck BE, et al. Community Mental Health Care Delivery During the COVID-19 Pandemic: Practical Strategies for Improving Care for People with Serious Mental Illness. Community Mental Health Journal. 2021; 57(3), 405–415. doi: 10.1007/s10597-020-00662-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Varker T, Brand RM, Ward J, Terhaag S, Phelps A. Efficacy of synchronous telepsychology interventions for people with anxiety, depression, posttraumatic stress disorder, and adjustment disorder: A rapid evidence assessment. Psychological Services. 2019;16(4):621–635. doi: 10.1037/ser0000239 [DOI] [PubMed] [Google Scholar]
- 49.Barnett P, Goulding L, Casetta C, et al. Implementation of Telemental Health Services Before COVID-19: Rapid Umbrella Review of Systematic Reviews. Journal of Medical Internet Research. 2021;23(7):e26492. doi: 10.2196/26492 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Mousavi SB. Coronavirus disease 2019 pandemic: Do not forget patients with severe mental illness. International Journal of Social Psychiatry. Published online July 7, 2020:002076402093998. doi: 10.1177/0020764020939982 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Weintraub MJ, Hall DL, Carbonella JY, Weisman de Mamani A, Hooley JM. Integrity of Literature on Expressed Emotion and Relapse in Patients with Schizophrenia Verified by a p -Curve Analysis. Family Process. 2016;56(2):436–444. doi: 10.1111/famp.12208 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Appleton R, Williams J, San Juan NV, Needle J, Schlief M, Jordan H, et al. Implementation, adoption and perceptions of telemental health during the COVID-19 pandemic: a systematic review. medRxiv. 2021. Jan 1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Backhaus A, Agha Z, Maglione ML, et al. Videoconferencing psychotherapy: A systematic review. Psychological Services. 2012;9(2):111–131. doi: 10.1037/a0027924 [DOI] [PubMed] [Google Scholar]
- 54.Bhui K, Priebe S. Assessing explanatory models for common mental disorders. The Journal of Clinical Psychiatry, 2006; 67(6), 1441. doi: 10.4088/jcp.v67n0614 [DOI] [PubMed] [Google Scholar]
- 55.Lazzari C, Shoka A, Papanna B, Kulkarni K. Predominant diagnoses, gender, and admission duration in an adult psychiatric inpatient hospital in United Kingdom. Open Journal of Psychiatry & Allied Sciences, 2018; 9(1), 37–40. [Google Scholar]
- 56.Jawad I, Watson S, Haddad PM, Talbot PS, McAllister-Williams RH. Medication nonadherence in bipolar disorder: a narrative review. Therapeutic advances in psychopharmacology 2018; 8(12), 349–363. doi: 10.1177/2045125318804364 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Hennemann S, Farnsteiner S, Sander L. Internet- and mobile-based aftercare and relapse prevention in mental disorders: A systematic review and recommendations for future research. Internet Interventions. 2018;14:1–17. doi: 10.1016/j.invent.2018.09.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Bhaskar S, Bradley S, Chattu VK, et al. Telemedicine as the New Outpatient Clinic Gone Digital: Position Paper From the Pandemic Health System REsilience PROGRAM (REPROGRAM) International Consortium (Part 1). Frontiers in Public Health. 2020;8. doi: 10.3389/fpubh.2020.00410 [DOI] [PMC free article] [PubMed] [Google Scholar]