Skip to main content
PMC home page
Elsevier - PMC COVID-19 Collection logoLink to Elsevier - PMC COVID-19 Collection
. 2022 Sep 30;320:517–524. doi: 10.1016/j.jad.2022.09.141

Implementation context and burnout among Department of Veterans Affairs psychotherapists prior to and during the COVID-19 pandemic

Craig S Rosen a,b,⁎, Adam N Kaplan c,d, David B Nelson c,d, Heidi La Bash a, Kathleen M Chard e,f, Afsoon Eftekhari a, Shannon Kehle-Forbes c,g, Shannon Wiltsey Stirman a,b, Nina A Sayer c,d,h
PMCID: PMC9523596  PMID: 36191645

Abstract

Background

The first goal of this study was to assess longitudinal changes in burnout among psychotherapists prior to (T1) and during the COVID-19 pandemic (T2). The second objective was to assess the effects of job demands, job resources (including organizational support for evidence-based psychotherapies, or EBPs) and pandemic-related stress (T2 only) on burnout.

Method

Psychotherapists providing EBPs for posttraumatic stress disorder in U.S. Department of Veterans Affairs (VA) facilities completed surveys assessing burnout, job resources, and job demands prior to (T1; n = 346) and during (T2; n = 193) the COVID-19 pandemic.

Results

Burnout prevalence increased from 40 % at T1 to 56 % at T2 (p < .001). At T1, stronger implementation climate and implementation leadership (p < .001) and provision of only cognitive processing therapy (rather than use of prolonged exposure therapy or both treatments; p < .05) reduced burnout risk. Risk factors for burnout at T2 included T1 burnout, pandemic-related stress, less control over when and how to deliver EBPs, being female, and being a psychologist rather than social worker (p < .02). Implementation leadership did not reduce risk of burnout at T2.

Limitations

This study involved staff not directly involved in treating COVID-19, in a healthcare system poised to transition to telehealth delivery.

Conclusion

Organizational support for using EBPs reduced burnout risk prior to but not during the pandemic. Pandemic related stress rather than increased work demands contributed to elevated burnout during the pandemic. A comprehensive approach to reducing burnout must address the effects of both work demands and personal stressors.

Keywords: Healthcare professionals, COVID-19, Burnout, Job demands-resources, Implementation climate

1. Introduction

Clinician burnout is widespread, with significant consequences for health systems, clinicians, and patients (National Academies of Sciences, Engineering, and Medicine, 2019). Burnout involves high emotional exhaustion, high depersonalization, and reduced personal accomplishment (Maslach and Jackson, 1981; Maslach and Leiter, 2016). Burnout contributes to absenteeism and presenteeism (i.e., employees working in poor physical or mental health states), both of which reduce productivity, increase the likelihood of mistakes, and strain other employees who must pick up the slack (Homrich et al., 2020). Burnout among mental health providers has been associated with increased staff turnover and with poorer patient outcomes (Yang and Hayes, 2020; Delgadillo et al., 2018). The ongoing coronavirus 2019 (COVID-19) pandemic poses significant challenges that may increase the prevalence of burnout among health care workers (Dehon et al., 2021; Kok et al., 2021; Schindler et al., 2021). These challenges align with factors that are specified in the job demands resources model of burnout (Demerouti et al., 2001).

1.1. Job demands resources model

The job demands resources models posits that risk for burnout is influenced by job demands and job resources (Demerouti et al., 2001). Job demands are work aspects that are physiologically or psychologically taxing. In healthcare, job demands may include work volume, time pressure, time spent on administrative tasks, and emotional demands of the work (Koranne et al., 2022). Greater job demands are expected to predict exhaustion, a core component of burnout (Bakker and Demerouti, 2017).

Job resources help workers achieve work goals, reduce physical or psychological demands, and/or grow professionally. Examples include control over work tasks, supervisor support, appreciation, and rewards (Demerouti et al., 2001; Koranne et al., 2022). Leaders can be a job resource if they provide support or access to resources (Bakker and Demerouti, 2017; Lewis and Cunningham, 2016). Job resources are expected to produce greater engagement and buffer the effects of job demands on exhaustion and burnout (Bakker and Demerouti, 2017; Demerouti et al., 2001). They are especially important for maintaining employee engagement in high demand jobs that challenge employees to learn new things and use new behaviors (Bakker and Demerouti, 2017) and hence may be critical for mitigating burnout during the current pandemic. In addition to job demands and resources, the model also includes personal resources that may protect against burnout, such as optimism, self-efficacy, and effective coping styles (Demerouti et al., 2001).

1.2. Application of the job demands resources model to burnout among mental health professionals during the COVID-19 pandemic

In studies of health care workers during the COVID-19 pandemic, job demands that increased risk for burnout included increased workload, greater exposure to the virus, greater sense of personal vulnerability, and fear of infecting other people (De Brier et al., 2020; Galanis et al., 2021; Gualano et al., 2021). Protective job resources included clear communication and support from the organization, social support from coworkers and supervisors, strong team leadership, and sufficient access to personal protective equipment and material resources (De Brier et al., 2020; Galanis et al., 2021; Gualano et al., 2021). Personal resources, such as greater coping self-efficacy, may also reduce risk for burnout during pandemics (De Brier et al., 2020).

Although the job demands resources model includes both job resources and personal resources, it neglects the impact of personal demands outside of work. The COVID-19 pandemic produced numerous personal stressors, including school closings, economic losses, disruption of social networks, illness of family or friends, and fear of contagion outside of work. Some studies found higher risk for burnout among clinicians who had a relative diagnosed with COVID-19 (Galanis et al., 2021) and those who experienced financial strain related to the pandemic (Gualano et al., 2021). Female health care workers have been especially impacted, with increased caregiving demands, greater conflict between work and family responsibilities, increased depressive symptoms, and elevated burnout relative to their male colleagues (Frank et al., 2021; Dillon et al., 2022). Research on clinician burnout during the COVID-19 pandemic may therefore benefit from accounting for pandemic-related stress in addition to job-related demands.

While much of the recent literature on burnout has focused on physicians and nurses directly treating patients with COVID-19, mental health professionals also face risk of burnout (O'Connor et al., 2018; Yang and Hayes, 2020). Research prior to the pandemic showed that more sessions per week and a fast-paced, high-pressure environment increased risk for burnout, while greater control over one's job, adequate supervision, and support from coworkers reduced risk of burnout (O'Connor et al., 2018; Yang and Hayes, 2020). During the COVID-19 pandemic, longer working hours, feeling pressure to immediately respond to messages from patients, and poor work-life balance were associated with greater emotional exhaustion and depersonalization, two domains of burnout (Kotera et al., 2021). Another study found burnout among community mental health center providers was increased by changes in tasks, work setting, and work team; and burnout was reduced by higher organizational trust and organizational support (Sklar et al., 2021).

One job demand of mental health work is witnessing human suffering and trauma. It has been theorized that this experience increases provider burnout (Elwood et al., 2011; Velasco et al., 2022). Yet, available research suggests that provision of trauma-focused psychotherapy does not increase burnout. Garcia et al., 2016, Garcia et al., 2018 found that burnout among psychotherapists was unrelated to hearing patients' trauma accounts; rather it was predicted by caseload, excessive bureaucratic oversight, lack of control over work conditions, patient malingering, and treating patients with personality disorders. Craig and Sprang (2010) found that a higher proportion of posttraumatic stress disorder (PTSD) cases increased clinicians' risk for burnout, but this risk was fully offset by the protective effects of trauma-specific training and greater use of evidence-based psychotherapies (EBPs). This suggests that job resources that help clinicians provide effective treatment for trauma-related mental health concerns can reduce clinicians' risk for burnout.

1.3. The present study

The present study leveraged longitudinal data from an ongoing study to examine risk and protective factors for burnout among psychotherapists treating PTSD with EPBs prior to and during the COVID-19 pandemic. The context for this study was the Veterans Health Administration (VHA), which has broadly disseminated two EBPs for PTSD – Cognitive Processing Therapy (CPT; Resick et al., 2016) and Prolonged Exposure (PE; Foa et al., 2019) – and implemented templates for tracking CPT and PE delivery (Karlin and Cross, 2014).

Our first goal was to assess changes in the prevalence of burnout and to differentiate the effects of pre-existing job demands from new challenges encountered during the pandemic, such as use of telehealth for EBP delivery. A second goal of the study was to assess whether job resources that facilitate use of EBPs, specifically implementation climate and implementation leadership, reduced therapist burnout. Implementation climate is the “perception that the organization's policies and practices encourage, cultivate, and reward” use of EBPs (Weiner et al., 2011). Implementation leadership is the extent to which supervisors engage in specific behaviors to encourage implementation of EBPs (Ehrhart et al., 2014).

Based on prior research, we hypothesized that prior to the COVID-19 pandemic a positive EBP implementation climate and positive implementation leadership would protect against burnout. We also hypothesized that during the COVID-19 pandemic strong EBP implementation leadership would protect against burnout, while pandemic-related stress would contribute to burnout. Finally, because job resources are theorized to be especially protect against burnout when demands are high (Bakker and Demerouti, 2017), we expected that implementation leadership for EBP delivery would moderate the effect of pandemic-related stress on burnout, such that the association between level of pandemic-related stress and burnout would be weaker when implementation leadership was strong.

2. Method

2.1. Design and procedures

This was an observational study. We stratified the population of 2962 licensed mental health providers who were already delivering individual CPT or PE to at least 2 patients in 2018 into 12 strata, based on the type of EBP(s) they provided (CPT, PE, or both) and US geographic region (West, South, Midwest, Northeast). Targeted proportional sampling was used to randomly draw a sample of therapists within each stratum who had at least 3 patients in 2018. Therapists were recruited through VA email until a sample size of 200 was obtained. Interested therapists completed an online consent and survey between May 2 and October 8, 2019, before the COVID-19 pandemic. A COVID-19 survey was administered among those who completed the initial survey between November 12, 2020, and January 4, 2021. As shown in Fig. 1 , 346 (30.9 %) of those recruited for the parent study completed the pre-pandemic (T1) survey, 207 (59.8 %) of whom were still providing CPT and PE and completed the COVID-19 (T2) survey, and 193 of the T2 survey responders provided data to evaluate burnout. The Minneapolis VA Health Care System Institutional Review Board approved this research and granted a waiver of documentation of informed consent.

Fig. 1.

Fig. 1

Open in a new tab

Flow of participants.

2.2. Measures

2.2.1. Burnout

Burnout at T1 and T2 was assessed using a single item with 83.2 % sensitivity and 87.4 % specificity among VHA clinicians (Dolan et al., 2015) relative to the emotional exhaustion subscale of the Maslach Burnout Inventory (Maslach and Jackson, 1981), the most widely used measure of burnout in healthcare (Bui et al., 2022). Participants are asked, “Overall, based on your definition of burnout, how would you rate your level of burnout? Responses are rated on a 5-point scale where 1 = I enjoy my work. I have no symptoms of burnout and 5 = I feel completely burned out and often wonder if I can go on. I am at the point where I may need some changes or may need to seek some sort of help. Consistent with prior research, we dichotomized burnout with scores ≤2 (no symptoms of burnout) vs. ≥3 (indications of burnout).

2.2.2. Job resources

At T1, the Implementation Leadership Scale (ILS) was used to assess how proactive, knowledgeable, supportive, and perseverant the therapist's clinic leader was regarding EBP implementation. The ILS has demonstrated excellent internal consistency as well as convergent and discriminant validity (Aarons et al., 2014). Participants rated their agreement with 12 statements about clinic leadership on a 5-point Likert scale ranging from “not at all” (0) to “very great extent” (4). We used two versions of the ILS, one for staff to report about their clinic leader, and another for clinic leaders to report about themselves. The term “evidence-based practice” was replaced with “CPT and PE”. Total scores (ranging from 0 to 4) were computed as the mean across all 12 items. Internal consistency was 0.95 for the staff-reported scale and 0.91 for the clinic leader-reported scale.

At T1, the Implementation Climate Scale (ICS) was used to assess staff perceptions of clinic policies, practices, procedures, and behaviors that are rewarded, supported, and expected to facilitate effective EBP implementation. The ICS is a psychometrically validated and reliable measure (Ehrhart et al., 2014; Lyon et al., 2018). Participants rated agreement with 18 statements describing the implementation climate on a 5-point Likert scale ranging from “not at all” (0) to “very great extent” (4). The referent for the ICS was the clinic in which CPT and PE were delivered. ICS total scores, ranging from 0 to 4 were computed by averaging across all 18 items (Williams et al., 2018, Williams et al., 2020). Internal consistency was 0.92 in this sample.

At T2, we modified the Implementation Leadership Scale (ILS) to assess the degree to which the therapists' clinic leader was proactive, knowledgeable, supportive, and perseverant regarding evidence-based practice implementation during the pandemic. We modified the T1 ILS by adding the phrase “during this pandemic” to the end of every sentence. Total scores were computed as described above. Internal consistency in this study was 0.97 for the staff reported ILS and 0.93 for the leader reported ILS. At T2, we assessed degree of control over EBP-specific work tasks by asking therapists to rate perceived autonomy in where and how they provided CPT and/or PE. Responses rated on a five-point scale ranging from “0: not at all” (0) to a “4: very great extent”.

2.2.3. Job demands

We used VA administrative data to during the three-months before each survey to calculate 1) workload (number of encounters per month) and 2) the percent of patients with a PTSD diagnosis in any of their encounters.

Change in job tasks is another type of job demand (Bakker and Demerouti, 2017; Sklar et al., 2021). To assess whether the pandemic necessitated a change in how therapists were delivering CPT or PE, at T2 we asked therapists to report telehealth use before and during the COVID-19 pandemic. Those new to providing CPT and/or PE via telehealth during the pandemic were labeled as experiencing a change in work tasks. We also asked therapists at T2 to indicate whether they experienced any of 12 specific technical/logistical challenges in delivering EBPs via telehealth, such as problems related to poor or unreliable internet access and problems getting handouts and worksheets to patients. The number of challenges endorsed was summed to create a telehealth challenges score.

2.2.4. Pandemic-related stress

At T2, we used a single item to assess pandemic-related stress that asked each therapist, “Overall, how much stress have you experienced because of the COVID-19 pandemic?” Responses were rated on a 5-point ordinal scale that ranged from “no pandemic-related stress” (0) to “extreme pandemic-related stress” (4). Because an individual item with an ordinal response format does not inherently yield a continuous variable, it is best analyzed as a categorical variable (Carifio and Perla, 2007). We grouped responses into three levels: “no/a little”, “some”, and “a lot/extreme”. The lowest two response categories were combined and the highest two response categories were combined because the sample sizes for the lowest (“no”) and highest (“extreme”) responses were too small to yield accurate parameter estimates.

2.2.5. Sociodemographic and administrative data measures

At T1, therapists reported their race and ethnicity, clinic role (clinic leader versus staff), years of VA employment, percent time in direct patient care, and type of EBP provided over the past year (CPT, PE, both). At T2, we reassessed clinic role.

We used administrative data to identify therapists' professional discipline (psychologist, social worker, counselor, or other), and the clinic setting (PTSD specialty care, other mental health, or mixed clinic type) in which the therapist provided a majority (>80 %) of care in the three months prior to each survey.

2.3. Statistical analysis

We used Pearson χ2 for two-way tables and two-sample t-tests assuming differing variances to compare baseline variables between completers of only the T1 survey and those that completed both the T1 and T2 surveys. We further constrained this comparison to therapists who provided data on burnout at both time points. For screening of marginal associations with burnout, marginal (1 predictor) logistic regressions tested whether individual measures affected the odds of burnout at their respective time points, T1 and T2. All T1 measures were also tested for an association with burnout at T2. Measures with p-values ≤ .15 for these marginal associations were included in multiple logistic regression of the respective burnout measure, along with clinic role (leader versus staff).

We tested our hypotheses using a series of 4 multiple logistic regression models that adjusted for the time-specific measures meeting the 0.15 p-value cut-off from the marginal associations. At T2, we evaluated whether the relationship between implementation leadership supporting EBP delivery and burnout varied by different levels of pandemic-related stress by adding an interaction term between pandemic-related stress and implementation leadership to the model examining burnout as a function of implementation leadership.

3. Results

3.1. Therapist characteristics and study measures

Table 1 presents characteristics of the 346 therapists who responded to the T1 survey and the 193 therapists who also responded to the T2 survey and completed burnout measures at T1 and T2. The only baseline difference between T2 responders and non-responders was that T2 non-responders were more likely to provide over 80 % of their psychotherapy outside of PTSD specialty care (χ2(2) = 9.80, p = .007). Most therapists (77 % for CPT, 83 % for PE) had completed training in CPT and/or PE at least two years before the T1 survey and very few (7 % for CPT and 6 % for PE) completed their training within the six months before the T1 survey. Burnout at T1 was not related to time since completion of training in either CPT (χ2 (4) = 2.29, p < .68) or PE (χ2 (4) = 4.77, p < .31).

Table 1.

Characteristics of participants at Time 1 and Time 2.

Pre-pandemic (Time 1) survey responders
(N = 346)
 COVID-19 (Time 2) survey responders
(N = 195)
n (%) or M (SD) n (%) or M (SD)
Demographics and work setting
 Sex
 Female 182 (52.6 %) 105 (54.4 %)
 Male 62 (17.9 %) 28 (14.5 %)
 Unknown 102 (29.5 %) 60 (31.1 %)
 Race
 African American only 17 (4.9 %) 9 (4.7%)
 Asian American only 10 (2.9 %) 7 (3.6 %)
 Other only 3 (0.9 %) 1 (0.5 %)
 White only 288 (83.2 %) 164 (84.9 %)
 Multiracial 8 (2.3 %) 5 (2.6 %)
 Missing 20 (5.8 %) 7 (3.6 %)
 Hispanic ethnicity 13 (3.8 %) 5 (2.6 %)
 Discipline
 Psychologist 192 (55.5 %) 109 (56.5 %)
 Social worker 143 (41.3 %) 78 (40.1 %)
 Counselor 10 (2.9 %) 5 (2.6 %)
 Years VHA employment
 <1 year 24 (6.9 %) 10 (5.2 %)
 1–5 years 197 (56.9 %) 111 (57.5 %)
 6–10 years 81 (23.4 %) 47 (24.4 %)
 11–15 years 34 (9.8 %) 20 (10.4 %)
 16–20 years 7 (2.0 %) 4 (2.1 %)
 >20 years 3 (0.9 %) 1 (0.5 %)
 Clinic role
 Clinic leader 64 (18.5 %) 36 (18.7 %)
 Staff member 271 (78.3 %) 151 (78.2 %)
 Other 11 (3.2 %) 6 (3.1 %)
 Clinical setting
 ≥80 % PTSD clinic 59 (17.1 %) 37 (19.2 %)
 ≥80 % other mental health 197 (56.9 %) 96 (49.7 %)
 Mixed PTSD specialty and other mental health 90 (26.0 %) 60 (31.3 %)
Work demands
 Percent time in direct patient care 73.9 % (20.9 %) 73.2 % (20.3 %)
 Percent of patients diagnosed with PTSD 47.6 % (24.7 %) 48.7 % (24.3 %)
 Therapy type provided
 CPT only 153 (44.2 %) 80 (41.5 %)
 PE only 30 (8.7 %) 18 (9.3 %)
 Both CPT and PE 163 (47.1 %) 95 (49.2 %)
 Number technological problems related to CPT/PE delivery using telehealth NA 3.28 (2.25)
Work resources
 T1 Implementation climate 3.10 (0.87) 2.16 (0.74)
 T1 Implementation leadership 2.31 (1.04) 2.32 (1.07)
 T2 Implementation leadership for EBP delivery during the pandemic NA 2.52 (1.10)
 T2 Control over where and how EBPs are delivered
 Not at all NA 40 (20.7 %)
 Slight extent 31 (16.1 %)
 Moderate extent 28 (14.5 %)
 Great extent 43 (22.3 %)
 Very great extent 44 (22.8 %)
 Missing 7 (3.6 %)
Open in a new tab

At T1, 38 % (n = 130) of therapists met survey criteria for burnout, with 33 % endorsing “burning out”, 4 % reporting “burnout that will not go away,” and 1 % reporting being “completely burned out.” The remaining 62 % (n = 211) of those who completed this item reported occasionally feeling stressed (56 %) or no burnout symptoms at all (6 %). At T2, 56 % (n = 110) of the sample met survey criteria for burnout, with 37 % reporting “burning out”, 16 % reporting “burnout that will not go away” and 3 % reporting being “completely burned out.” Among the 193 therapists who responded to the burnout question at both T1 and T2, the proportion reporting burnout rose from 40 % (n = 78) at T1 to 56 % (n = 108) at T2 (χ2 (1) = 14.42, p < .001).

3.2. Association between EBP implementation climate supporting use of EBPs and burnout prior to the COVID-19 pandemic

Preliminary analyses were conducted to identify bivariate associations (p < .15) between any other variables included in Table 1 with burnout at either time point for inclusion as covariates. The only qualifying covariate was the type of EBP for PTSD that clinicians provided: burnout was lower among therapists who provided only CPT relative to therapists who provided PE or both CPT and PE. Burnout at T1 was unrelated to job demand factors (percent time spent in direct clinical care, total number of encounters, proportion of patients with PTSD), work setting (PTSD specialty clinic, other mental health setting, or mix of specialty/non-specialty settings), or personal characteristics (length of VA career, sex, race, discipline).

Model 1 (n = 343) predicted risk of burnout prior to the COVID-19 pandemic as a function of implementation climate, being a leader (reference = staff), the interaction of implementation climate and clinic role (leader vs. staff), and whether clinicians provided CPT only (rather than PE only or both CPT and PE). Risk of burnout was much lower among leaders than staff (AOR [95 % CI] = 0.085 [0.008–0.714], p < .03). Consistent with our hypothesis, a positive implementation climate strongly reduced the risk of burnout among staff (AOR [95 % CI] = 0.468 [0.313–0.686], p < .001; see Fig. 2 ). However, the effect of implementation climate on burnout differed for leaders and staff (interaction AOR [95 % CI] = 2.752 [1.189–6.686], p = .02). Among leaders, implementation climate had no significant effect on burnout (AOR [95 % CI] = 1.288 [0.605–2.894], p = .52). Psychotherapists who delivered only CPT had lower risk of burnout (AOR [95 % CI] = 0.622 [0.387–0.992], p < .05) than did providers who delivered PE only or both CPT and PE.

Fig. 2.

Fig. 2

Open in a new tab

Pre-pandemic adjusted probability of burnout by implementation climate and by staff vs. clinic role.

Note: Probability of burnout is adjusted for the type of PTSD psychotherapy clinicians delivered (cognitive processing therapy only vs. prolonged exposure only or both prolonged exposure and cognitive processing therapy). Implementation Climate Scale scores range from 0 to 4, with higher values indicating stronger organizational support for using evidence-based psychotherapies.

3.3. Association between implementation leadership supporting use of EBPs and burnout prior to the COVID-19 pandemic

Model 2 (n = 338) examined burnout as a function of T1 implementation leadership supporting use of EBPs, being a leader (reference = staff), the interaction of implementation climate and clinic role, and whether clinicians provided CPT only. Stronger implementation leadership significantly reduced risk of burnout (AOR [95 % CI] = 0.708 [0.542–0.919], p = .01). Neither the main effect of being a leader (AOR [95 % CI] = 0.093 [0.001–4.397], p = .25) nor the interaction between implementation climate and clinic role (AOR [95 % CI] = 2.134 [0.696–7.311], p = .20) were significant. Use of CPT only did not significantly lower burnout in this adjusted model (AOR [95 % CI] = 0.679 [0.424–1.080], p = .10).

3.4. Association between implementation leadership supporting use of EBPs, pandemic-related stress, and burnout during the COVID-19 pandemic

The M (SD) number of total encounters per month was 94.1 (50.4) at T1 and 72.8 (33.9) at T2, indicating a 22 % reduction in workload during the pandemic. At T2, most providers (n = 172, 89.1 %) were using telemedicine for EBP delivery, and 54 (31.4 %) of these therapists had not used telehealth before the pandemic. Regarding pandemic-related stress, 23 % of respondents (n = 42) reported “no” or “a little” stress related to the COVID-19 pandemic, 42 %, (n = 82) reported “some” pandemic-related stress, and 36 % (n = 69) reported “a lot” or “extreme” pandemic-related stress.

Model 3 examined burnout during the pandemic as a function of implementation leadership supporting EBP delivery during the pandemic, leader vs. staff role, the interaction of implementation leadership by clinic role as well as bivariate associations between (p < .15) covariates and burnout during the pandemic. Table 2 presents the results from Model 3. Contrary to our hypothesis, implementation leadership support for use of EBPs did not reduce risk of burnout during the pandemic. Neither the effect of implementation leadership on staff, the effect of self-reported implementation leadership on leaders nor the interaction of implementation leadership by clinic role were significantly associated with burnout.

Table 2.

Risk of burnout during the COVID-19 pandemic by implementation leadership, clinic role, and covariates.

Adjusted odds ratio Confidence interval p value
Role and implementation leadership
 Leader (reference = staff or other role) 4.410 0.040 to 840.83 0.556
 T2 Implementation leadership 1.109 0.691 to 1.788 0.669
 T2 Implementation leadership x clinic role 0.551 0.159 to 1.712 0.323
Job demands
 Prior burnout at T1 7.474 2.931 to 21.169 0.001
 Percent of patients diagnosed with PTSD 2.596 0.298 to 24.105 0.391
 Number of technological/logistical problems in delivering EBPs during pandemic 1.236 0.992 to 1.565 0.066
 Deliver CPT only (reference = deliver PE only or both CPT and PE) 0.980 0.377 to 2.597 0.968
Job resources
 Control over when/how deliver EBPs 0.682 0.49 to 0.932 0.018
Pandemic-related stress (reference = a lot/extreme)
 None/a little pandemic-related stress 0.028 0.007 to 0.099 0.001
 Some pandemic-related stress 0.064 0.017 to 0.201 0.001
Demographic and work setting variables
 Psychologist (reference = social worker or counselor) 7.327 2.771 to 21.743 0.001
 Male (reference - female or unknown) 0.181 0.041 to 0.724 0.019
 Work setting (reference ≥ 80 % other mental health)
 ≥80 % PTSD specialty program 2.377 0.562 to 10.447 0.242
 Mix of PTSD specialty and other settings 2.301 0.799 to 6.897 0.127
Open in a new tab

As displayed in Fig. 3 , the strongest risk factors for burnout at T2 were T1 burnout and pandemic-related stress. Accounting for T1 burnout and other covariates, pandemic-related stress still strongly elevated risk of burnout. Relative to therapists who experienced “a lot” or “extreme” pandemic-related stress, burnout was far lower among therapists who reported “some” or “no/a little” pandemic-related stress.

Fig. 3.

Fig. 3

Open in a new tab

Burnout during the pandemic (Time 2) by pre-pandemic burnout and pandemic-related stress.

Greater control over when and how to deliver EBPs, a job resource factor, reduced risk for burnout. The number of technological and logistical challenges associated with providing EBPs during the pandemic, a job demand factor, only approached significance (p = .066). Some demographic characteristics influenced burnout. Male clinicians had much lower risk of burnout than did their female colleagues or those with missing sex information. Psychologists had higher risk of burnout than did social workers and counselors.

In addition to predicting that strong implementation leadership support would reduce burnout directly, we hypothesized that leadership would buffer the effect of pandemic-related stress on burnout. Model 4 tested this hypothesis by adding the interaction of pandemic-related stress and implementation leadership to Model 3. Our moderation hypothesis was not supported (χ2 (2) = 3.937, p = .14).

4. Discussion

Prior to the COVID-19 pandemic, the prevalence of burnout in our sample of VHA psychotherapists was 38 %, similar to the prevalence of burnout in studies of VHA primary care providers (38 %) and academic psychiatrists (33 %) prior to the pandemic that used the same measure (Dolan et al., 2015; Williams et al., 2020). As expected, both implementation climate and implementation leadership were associated with lower risk for burnout. This accords with other studies showing that EBP training, greater knowledge, and greater confidence in EBPs were associated with reduced burnout among psychotherapists (Craig and Sprang, 2010; Kim et al., 2018). Thus, we conclude that in the absence of a global pandemic, job resources that facilitate EBP delivery are not only beneficial for patients, but also for the psychotherapist delivering EBPs. We find it interesting that a positive EBP implementation climate reduced risk of burnout among staff but not clinic leaders. At the same time, burnout was lower among leaders than among staff at T1, perhaps because leaders have greater control over their work. This finding suggests the possibility that leaders and staff may be impacted differently by the organizational context and may have different resource needs.

The prevalence of burnout in our sample increased dramatically during the pandemic, consistent with the few other longitudinal studies of physicians and nurses (Kok et al., 2021; Schindler et al., 2021). While implementation leadership was protective prior to the COVID-19 pandemic, it was not protective against burnout during the pandemic, neither as a main effect nor as a moderator of stress. This suggests that during a pandemic, job resources that reduce or manage work demands and protect against exhaustion may be more critical than those that optimize EBP delivery. Consistent with this possibility was our finding that greater control over when and how to deliver EBPs protected against burnout. This finding aligns with other research showing that perceived control over one's job protects against burnout (O'Connor et al., 2018; Yang and Hayes, 2020). Allowing greater flexibility and control may become even more important during a pandemic when clinicians are juggling increased demands outside of work (Frank et al., 2021).

Increased job demands did not appear to be the main drivers of increased burnout during the pandemic in our sample of psychotherapists delivering EBPs. They did not experience the same type of work demands as front-line medical staff such as dramatically larger caseloads, increased risk of infection, or direct exposure to patient death (Gualano et al., 2021). In fact, psychotherapists' caseloads declined significantly during the three months preceding to the second survey while the proportion of their caseload dedicated to EBP delivery remained the same. Neither the number of logistical problems that psychotherapists reported nor whether they were new to using telehealth significantly predicted burnout during the pandemic in our adjusted model. It may be that by the time of the second survey, therapists had already adapted to using telehealth and/or that EBP delivery using telemedicine was not particularly stressful for VHA psychotherapists because of VHA's extensive experience with telehealth prior to the pandemic.

Instead, pandemic-related stress was the main driver of burnout during the pandemic, even controlling for T1 burnout. We did not directly assess the specific causes of participants' pandemic-related stress, but participants' optional open-ended comments in the T2 surveys suggest this may reflect additional challenges in their personal lives as much as increased demands at work. One therapist noted, “Pandemic stress is personal (ill family members and kids doing remote learning/online) and thus, not related to implementation of EBPs via telework. In fact, telework has been a stress reliever to some extent.” Another commented, “The pandemic's main impact on me as an employee is continuing to work while supporting my children, who are too young to attend on-line school alone. The tools we need are flexibility in schedule… part time, and job-sharing options.” Increased caregiving demands outside of work may also explain why female psychotherapists in our study, like female physicians in other studies (Dillon et al., 2022; Frank et al., 2021), had greater depressive symptoms or burnout during the pandemic than did males, despite no sex difference prior to the pandemic.

It is unclear why psychologists in our sample had seven times higher odds for burnout during the pandemic than did social workers, despite no difference before the pandemic. This contrasts with some pre-pandemic studies showing lower burnout risk among psychologists than social workers (O'Connor et al., 2018). One possibility is that social workers' more holistic training may have better prepared them to adapt to changing patient needs and evolving work conditions during the pandemic. Whether and how social work training or job responsibilities helps them cope with the demands of the COVID-19 pandemic is an interesting topic for future study. It is also uncertain why prior to the pandemic, psychotherapists who provided CPT only had lower burnout risk than those who provided PE or CPT and PE. PE requires 90-minute while CPT requires 60-minute sessions and thus may be more difficult to schedule. The difference in burnout by type of EBP, however, was not observed during the COVID-19 pandemic. We are not aware of other studies that have examined the association between therapy type and burnout among psychotherapists.

Support for implementing EBPs was not protective against burnout during the pandemic, when therapists faced numerous stressors and demands outside of work. In that context, resources that support personal accomplishment may be less critical than resources that reduce exhaustion. Other research suggests that healthcare worker burnout during this pandemic can be reduced by clear communication and support from the organization, setting realistic workloads, reducing unnecessary tasks, allowing more flexibility in work hours or conditions (e.g., telework), promoting social support, encouraging self-care, and supporting use of personal leave (De Brier et al., 2020). It has also been suggested that training in Stress First Aid might help workgroups destigmatize stress, encourage mutual support, and facilitate self-care (Watson and Westphal, 2020).

This study has several limitations. We relied on a single-item measure of burnout and therefore were not able to evaluate different dimensions of burnout. However, this measure is a well-validated indicator of emotional exhaustion and is widely used in VHA and other health care organizations (Dolan et al., 2015; Zivin et al., 2022). We did not specify a time frame for this question so we assume that therapists considered the entire period of time from the beginning of the pandemic through to the present when responding. We did not assess which specific COVID-19 stressors at work and at home (e.g., changes in work procedures, fear of infection, fear of infecting others, illness, ill family members, caregiving demands for children out of school) contributed to participants' perceived stress. We also did not measure personal stressors at T1. This precluded our examining how demands outside work might contribute to staff burnout under routine conditions, not solely in the context of a pandemic. Future studies should examine changes in pandemic-related stress over time and in respect to specific work and personal stressors. We assessed work climate related to use of EBPs but not other work climate factors such as psychological safety and mutual social support which might impact burnout. Finally, our study involved psychotherapists who worked within a healthcare system that was poised to transition to telehealth delivery (Rosen et al., 2021). Our findings may not generalize to other healthcare contexts or to staff providing direct care for COVID-19.

Despite limitations, this study provides new information that can inform efforts to protect the psychotherapy workforce against burnout. Most prior research on health care burnout has focused on physicians and nurses. However, we observed troubling levels of burnout among psychotherapists before and even more troubling levels of burnout during the COVID-19 pandemic. This underscores the importance of attending to psychotherapists' well-being to maintain enough therapists who can deliver EBPs to the large number of patients who need them. Our findings indicate that under normal circumstances, support for EBP delivery protects against burnout among psychotherapies. Yet in the face of a global pandemic, support for EBP delivery is not enough – psychotherapists need organizational resources, such as work flexibility, that reduce the burden of managing new demands, many of which occur outside of work. A comprehensive model of and interventions to reduce burnout, therefore, needs to consider not only work demands but also personal demands, and adjust work resources accordingly.

CRediT authorship contribution statement

Drs. Sayer, Nelson and Kaplan had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors have reviewed and approved the manuscript submitted for publication.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: Rosen, Sayer, Nelson, Kaplan.

Drafting of the manuscript: Rosen, Sayer, Nelson, Kaplan.

Critical revision of the manuscript for important intellectual content: All authors.

Obtained funding: Sayer.

Conflict of interest

Dr. Rosen reported receiving grants from the U.S. Department of Defense and the U.S. Department of Veterans Affairs.

Dr. Kaplan reported no conflicts of interest at the time of publication.

Dr. Nelson reported receiving grants from the U.S. Department of Veterans Affairs.

Dr. La Bash reported no conflicts of interest at the time of publication.

Dr. Chard reported receiving grants from the U.S. Department of Defense, U.S. Department of Veterans Affairs and the Chris Sullivan Foundation.

Dr. Eftekhari reported no conflicts of interest at the time of publication.

Dr. Kehle-Forbes reported receiving grants from the U.S. Department of Veterans Affairs and the Patient-Centered Outcomes Research Institute.

Dr. Wiltsey Stirman reported receiving grants from the U.S. National Institute of Mental Health and the Canadian Institutes of Health Research.

Dr. Sayer is the PI of the U.S. Department of Veterans Affairs Health Services Research and Development grant that supported this work.

Acknowledgments

Acknowledgements

This work was supported by a U.S. Department of Veterans Affairs, Health Services Research & Development [IIR-17-178]. The opinions presented are those of the authors and do not necessarily represent the position of the U.S. Government or the U.S. Department of Veterans Affairs.

Role of the funding source

The study sponsor, the U.S. Department of Veterans Affairs Health Services Research and Development, had no role in the conduct of the study, interpretation of the findings, and reporting of the results.

References

  1. Aarons G.A., Ehrhart M.G., Farahnak L.R. The implementation leadership scale (ILS: development of a brief measure of unit level implementation leadership. Implement. Sci. 2014;9:45. doi: 10.1186/1748-5908-9-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bakker A.B., Demerouti E. Job demands-resources theory: taking stock and looking forward. J. Occup. Health Psychol. 2017;22:273–285. doi: 10.1037/ocp0000056. [DOI] [PubMed] [Google Scholar]
  3. Bui T., Tran T., Nguyen T., Vu T.C., Ngo X.D., Nguyen T., Do T. Reassessing the most popularly suggested measurement models and measurement invariance of the maslach burnout inventory - human service survey among Vietnamese healthcare professionals. Health Psychol. Behav. Med. 2022;10:104–120. doi: 10.1080/21642850.2021.2019585. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Carifio J., Perla R. Ten common misunderstandings, misconceptions, persistent myths and urban legends about likert scales and likert response formats and their antidotes. J. Soc. Sci. 2007;2:106–116. http://thescipub.com/PDF/jssp.2007.106.116.pdf [Google Scholar]
  5. Craig C.D., Sprang G. Compassion satisfaction, compassion fatigue, and burnout in a national sample of trauma treatment therapists. Anxiety Stress Coping. 2010;23:319–339. doi: 10.1080/10615800903085818. [DOI] [PubMed] [Google Scholar]
  6. De Brier N., Stroobants S., Vandekerckhove P., De Buck E. Factors affecting mental health of health care workers during coronavirus disease outbreaks (SARS, MERS & COVID-19: A rapid systematic review. PLOS One. 2020;15 doi: 10.1371/journal.pone.0244052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Dehon E., Zachrison K.S., Peltzer-Jones J., Tabatabai R.R., Clair E., Puskarich M.A., Fix M.L. Sources of distress and coping strategies among emergency physicians during COVID-19. West. J. Emerg. Med. 2021;22(6):1240. doi: 10.5811/westjem.2021.9.53406. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Delgadillo J., Saxon D., Barkham M. Associations between therapists’ occupational burnout and their patients’ depression and anxiety treatment outcomes. Depress. Anxiety. 2018;35:844–850. doi: 10.1002/da.22766. [DOI] [PubMed] [Google Scholar]
  9. Demerouti E., Bakker A.B., Nachreiner F., Schaufeli W.B. The job demands- resources model of burnout. J. Appl. Psychol. 2001;86:499–512. doi: 10.1037/0021-9010.86.3.499. [DOI] [PubMed] [Google Scholar]
  10. Dillon E.C., Stults C.D., Deng S., Martinez M., Szwerinski N., Koenig P.T., Gregg L., Cobb J.K., Mahler E., Frosch D.L., Le Sieur S., Hanley M., Pertsch S. Women, younger clinicians', and caregivers' experiences of burnout and well-being during COVID-19 in a US healthcare system. J. Gen. Intern. Med. 2022;37:145–153. doi: 10.1007/s11606-021-07134-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Dolan E.D., Mohr D., Lempa M., Joos S., Fihn S.D., Nelson K.M., Helfrich C.D. Using a single item to measure burnout in primary care staff: a psychometric evaluation. J. Gen. Intern. Med. 2015;30:582–587. doi: 10.1007/s11606-014-3112-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Ehrhart M.G., Aarons G.A., Farahnak L.R. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS. Implement. Sci. 2014;9 doi: 10.1186/s13012-014-0157-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Elwood L.S., Mott J.M., Lohr J.M., Galovski T.E. Secondary trauma symptoms in clinicians: a critical review of the construct, specificity, and implications for trauma-focused treatment. Clin. Psychol. Rev. 2011;31(1):25–36. doi: 10.1016/j.cpr.2010.09.004. [DOI] [PubMed] [Google Scholar]
  14. Foa E., Hembree E.A., Rothbaum B.O., Rauch S. second ed. Oxford University Press; New York: 2019. Prolonged Exposure Therapy for PTSD: Emotional Processing of Traumatic Experiences – Therapist Guide. [Google Scholar]
  15. Frank E., Zhao Z., Fang Y., Rotenstein L.S., Sen S., Guille C. Experiences of work-family conflict and mental health symptoms by gender among physician parents during the COVID-19 pandemic. JAMA Netw. Open. 2021;4 doi: 10.1001/jamanetworkopen.2021.34315. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Galanis P., Vraka I., Fragkou D., Bilali A., Kaitelidou D. Nurses' burnout and associated risk factors during the COVID-19 pandemic: a systematic review and meta-analysis. J. Adv. Nurs. 2021;77:3286–3302. doi: 10.1111/jan.14839. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Garcia H.A., McGeary C.A., Finley E.P., McGeary D.D., Ketchum N.S., Peterson A.L. The influence of trauma and patient characteristics on provider burnout in VA post-traumatic stress disorder specialty programmes. Psychol. Psychother. 2016;89:66–81. doi: 10.1111/papt.12057. [DOI] [PubMed] [Google Scholar]
  18. Garcia H.A., Benzer J.K., Haro E., Finley E.P. Occupational burnout among PTSD specialty clinic providers in the Veterans Health Administration: perceptions of bureaucratic and political oversight. Ment Health Prevent. 2018;12:42–49. doi: 10.1016/j.mhp.2018.09.001. [DOI] [Google Scholar]
  19. Gualano M.R., Sinigaglia T., Lo Moro G., Rousset S., Cremona A., Bert F., Siliquini R. The burden of burnout among healthcare professionals of intensive care units and emergency departments during the COVID-19 pandemic: a systematic review. Int. J. Environ. Res. Public Health. 2021;18:8172. doi: 10.3390/ijerph18158172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Homrich P., Dantas-Filho F.F., Martins L.L., Marcon E.R. Presenteeism among health care workers: literature review. Rev. Bras. Med. Trab. 2020;18:97–102. doi: 10.5327/Z1679443520200478. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Karlin B.E., Cross G. From the laboratory to the therapy room: National dissemination and implementation of evidence-based psychotherapies in the U.S. Department of Veterans Affairs Health Care System. Am. Psychol. 2014;69:19–33. doi: 10.1037/a0033888. [DOI] [PubMed] [Google Scholar]
  22. Kim J.J., Brookman-Frazee L., Gellatly R., Stadnick N., Barnett M.L., Lau A.S. Predictors of burnout among community therapists in the sustainment phase of a system-driven implementation of multiple evidence-based practices in children’s mental health. Prof. Psychol. Res. Pract. 2018;49:131–142. doi: 10.1037/pro0000182. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kok N., van Gurp J., Teerenstra S., van der Hoeven H., Fuchs M., Hoedemaekers C., Zegers M. Coronavirus disease 2019 immediately increases burnout symptoms in ICU professionals: a longitudinal cohort study. Crit. Care Med. 2021;49(3):419–427. doi: 10.1097/CCM.0000000000004865. [DOI] [PubMed] [Google Scholar]
  24. Koranne R., Williams E.S., Poplau S., Banks K.M., Sonneborn M., Britt H.R., Linzer M. Reducing burnout and enhancing work engagement among clinicians: the Minnesota experience. Health Care Manag. Rev. 2022;47:49–57. doi: 10.1097/HMR.0000000000000298. [DOI] [PubMed] [Google Scholar]
  25. Kotera Y., Maxwell-Jones R., Edwards A.M., Knutton N. Burnout in professional psychotherapists: relationships with self-compassion, work-life balance, and telepressure. Int. J. Environ. Res. Public Health. 2021;18:5308. doi: 10.3390/ijerph18105308. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Lewis H.S., Cunningham C.J. Linking nurse leadership and work characteristics to nurse burnout and engagement. Nurs. Res. 2016;65:13–23. doi: 10.1097/NNR.0000000000000130. [DOI] [PubMed] [Google Scholar]
  27. Lyon A.R., Cook C.R., Brown E.C., Locke J., Davis C., Ehrhart M., Aarons G.A. Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implement. Sci. 2018;13:1–14. doi: 10.1186/s13012-017-0705-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Maslach C., Jackson S.E. The measurement of experienced burnout. J. Organ. Behav. 1981;2:99–113. doi: 10.1002/job.4030020205. [DOI] [Google Scholar]
  29. Maslach C., Leiter M.P. Understanding the burnout experience: recent research and its implications for psychiatry. World Psychiatry. 2016;15:103–111. doi: 10.1002/wps.20311. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. National Academies of Sciences, Engineering, and Medicine . The National Academies Press; Washington, DC: 2019. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being. [DOI] [PubMed] [Google Scholar]
  31. O'Connor K., Muller Neff D., Pitman S. Burnout in mental health professionals: a systematic review and meta-analysis of prevalence and determinants. Eur. Psychiatry. 2018;53:74–99. doi: 10.1016/j.eurpsy.2018.06.003. [DOI] [PubMed] [Google Scholar]
  32. Resick P.A., Monson C.M., Chard K.M. Guilford Press; New York: 2016. Cognitive Processing Therapy for PTSD: A Comprehensive Manual. [Google Scholar]
  33. Rosen C.S., Morland L.A., Glassman L.H., Marx B.P., Weaver K., Smith C.A., Pollack S., Schnurr P.P. Virtual mental health care in the Veterans Health Administration's immediate response to coronavirus disease-19. Am. Psychol. 2021;76:26–38. doi: 10.1037/amp0000751. [DOI] [PubMed] [Google Scholar]
  34. Schindler A.K., Polujanski S., Rotthoff T. A longitudinal investigation of mental health, perceived learning environment and burdens in a cohort of first-year German medical students' before and during the COVID-19 'new normal'. BMC Med. Educ. 2021;21:413. doi: 10.1186/s12909-021-02798-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Sklar M., Ehrhart M.G., Aarons G.A. COVID-related work changes, burnout, and turnover intentions in mental health providers: a moderated mediation analysis. Psychiatr. Rehabil. J. 2021;44:219–228. doi: 10.1037/prj0000480. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Velasco J., Sanmartín F.J., Gálvez-Lara M., Cuadrado F., Moriana J.A. Psychological effects of professional exposure to trauma and human suffering: systematic review and meta-analysis. Trauma Violence Abuse. 2022 doi: 10.1177/15248380221074314. 15248380221074314. Advance online publication. [DOI] [PubMed] [Google Scholar]
  37. Watson P., Westphal R.J. National Center for PTSD; 2020. Stress first aid for health care workers manual.https://www.ptsd.va.gov/professional/treat/type/SFA/docs/SFA_HCW_Manual_508.pdf Available on: [Google Scholar]
  38. Weiner B.J., Belden C.M., Bergmire D.M., Johnston M. The meaning and measurement of implementation climate. Implement. Sci. 2011;6:78. doi: 10.1186/1748-5908-6-78. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Williams N.J., Ehrhart M.G., Aarons G.A., Marcus S.C., Beidas R.S. Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: cross-sectional and lagged analyses from a 2-year observational study. Implement. Sci. 2018;13:85. doi: 10.1186/s13012-018-0781-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Williams N.J., Wolk C.B., Becker-Haimes E.M., Beidas R.S. Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: a 5-year panel analysis. Implement. Sci. 2020;15:10. doi: 10.1186/s13012-020-0970-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Yang Y., Hayes J.A. Causes and consequences of burnout among mental health professionals: a practice-oriented review of recent empirical literature. Psychotherapy (Chic) 2020;57:426–436. doi: 10.1037/pst0000317. [DOI] [PubMed] [Google Scholar]
  42. Zivin K., Chang M.M., Van T., Osatuke K., Boden M., Sripada R.K., Abraham K.M., Pfeiffer P.N., Kim H.M. Relationships between work-environment characteristics and behavioral health provider burnout in the Veterans Health Administration. Health Serv. Res. 2022 doi: 10.1111/1475-6773.13964. Advance online publication. doi:10.1111/1475-6773.13964. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Journal of Affective Disorders are provided here courtesy of Elsevier

RESOURCES