Key conditions for the successful implementation of evidence-based practice in concurrent disorder nursing care with the ECHO® model: Insights from a mixed-methods study

Abstract

Background

People with concurrent mental health and substance use disorders have complex biopsychosocial problems but risk not having their healthcare needs met. Nurses are positioned to meet these needs but often lack training in concurrent disorder management. Extension for Community Healthcare Outcomes (ECHO®, University of New Mexico Health Sciences Center, 2003) is a promising technology-enabled collaborative learning model used to implement evidence-based practice and build capacity among healthcare professionals in managing complex, chronic, health conditions.

Objective

To understand how an ECHO program for concurrent disorder management impacts nurses’ competency development and clinical practice and uncover key conditions for successful uptake and implementation.

Design

A convergent mixed-methods design comprising a quantitative, uncontrolled before-and-after study and a qualitative study using interpretive description methodology.

Setting and participants

An ECHO program for concurrent disorder management was implemented in 2018 at a quaternary academic hospital centre in metropolitan Western Canada. All 65 nurses who registered in the program between 2018 and 2020 were invited to participate in the study.

Methods

Online surveys completed by the participating nurses (N = 28) were administered at baseline and six and 12 months following entry-to-program to measure changes in nurse-related outcomes. The survey data were analyzed using descriptive statistics and repeated measures analysis. Semi-structured interviews were conducted with a nurse subgroup (n = 10) to explore how they developed and implemented competencies and what factors influenced this process. Interview transcripts were analyzed using inductive thematic analysis. Using the Pillar Integration Process, we analyzed results from both methods to provide a richer understanding of the phenomena.

Results

We identified six interrelated key conditions for successful uptake and implementation of evidence-based practice in concurrent disorder nursing care with ECHO: (1) Practice and validation opportunities; (2) Reciprocal and trusting relationships in an interprofessional education context; (3) Peer-to-peer experience sharing; (4) Collaboration with experts; (5) Reinforcement of positive attitudes towards one's professional role; and (6) Organizational support.

Conclusions

Outcome measures, perspectives, and experiences collected over 12 months indicated that ECHO contributed to nurses’ competency development and, under some conditions, to effective nursing practice changes. Given the challenges in implementing clinical guidelines in concurrent disorder nursing care, our results highlight the importance of understanding the key conditions for successful uptake and implementation. This informs approaches to optimally adapt implementation strategies to the needs and specificities of nurses to obtain impactful and sustainable results.

What is already known about this topic?

• Concurrent disorder management guidelines are translated into clinical nursing practice in a fragmented and inconsistent way, and concurrent disorders remain complex to manage for many primary care nurses.

• ECHO is a continuing education program using videoconference technology to support the implementation of evidence-based practice and build capacity among primary care practitioners in managing complex, chronic, health conditions.

• There is very little research investigating how ECHO impacts nurses’ competency development and clinical practice in concurrent disorder care and what the key conditions associated with successful uptake and implementation are.

Alt-text: Unlabelled box

What this paper adds.

•In our study, we showed that a 12-month participation in an ECHO program for concurrent disorder management supported nurses in developing their competencies and that this participation can, under a range of conditions, result in effective and sustainable clinical practice changes.

•Opportunities for practice and feedback as part of continuing professional education for nurses are key aspects to fostering ongoing learning and renewal of practice.

•Support from healthcare organizations, both structural (allocation of time and funding for continuing education, adequate staffing, healthy workplace culture) and emotional (explicit managerial support and recognition for professional development, encouragements from peers, experts, and mentors), is essential to increase nurses’ motivation and commitment to continuing professional development.

Alt-text: Unlabelled box

1. Introduction

1.1. Importance of concurrent disorders

In the field of mental health and addiction, the term “concurrent disorders” (also called dual diagnosis, co-occurring disorders, or comorbidity) refers to various combinations of substance use and mental health disorders in the same individual simultaneously (Hakobyan et al., 2020). While the reported prevalence rates for concurrent disorders vary (Substance Abuse and Mental Health Services Administration, 2014; Khan, 2017), it is estimated that up to 50% of individuals with psychiatric or other serious mental health disorders will develop a substance use disorder at some point in their lives and vice versa (Fleury et al., 2015). People experiencing concurrent disorders consistently have poorer physical health, greater psychological distress, and less-than-optimal healthcare compared to people with only a single disorder (Urbanoski et al., 2017). They are also more vulnerable to a wide range of psychosocial difficulties, including housing instability and homelessness, stigma, violence and incarceration, and worsening social functioning (Pinderup, 2017). Furthermore, concurrent disorders are frequently associated with premature mortality, due to an increased vulnerability to medical illnesses and infections, and a higher risk of suicide and accidental death (Mueser and Gingerich, 2013).

The complex healthcare needs of the population with concurrent disorders increase symptom severity and complicate recovery, resulting in sub-optimal levels of engagement in treatment, high relapse rates, long hospital stays, and increased healthcare costs (Litz and Leslie, 2017; Becker et al., 2017). Thus, delivering quality healthcare services that address the specific needs of individuals with concurrent disorders constitutes a high priority for many health organizations and healthcare professionals worldwide (World Health Organization, 2021). To address this priority, evidence-based interventions are a crucial first step to enhance the quality of care and improve patient health outcomes (Hakobyan et al., 2020).

1.2. Evidence-based interventions for concurrent disorder management

Integrated care (or treatment) is an approach relying on the concurrent delivery of multiple evidence-based interventions (pharmacotherapy, psychotherapy, motivational interviewing, or relapse prevention) for both mental-health and substance-use disorders by an individual or team of healthcare professionals within the same facility or by separate healthcare teams working in close collaboration (National Institute for Health and Care Excellence, 2016). Despite multiple clinical guidelines promoting the virtues of integrated care as an evidence-based practice (Hakobyan et al., 2020), researchers have shown that only 7.4% of individuals with concurrent disorders receive care for both their mental-health and substance-use needs and that up to 55% receive no treatment at all (Priester et al., 2016). Some researchers suggest concurrent disorder management guidelines are translated into clinical practice in a fragmented and inconsistent way and that concurrent disorders remain complex to manage for many healthcare professionals (Savic et al., 2017; Padwa et al., 2015). The challenges involve, for instance, screening mental health needs in active users, managing persistent and chronic symptoms, coping with negative symptoms adversely affecting social relationships and functioning, managing substance withdrawal and cravings, and dealing with medication noncompliance and treatment dropout.

Contributing to this evidence-practice gap is a lack of sufficient training in concurrent disorder management (Petrakis et al., 2018; Pinderup et al., 2016). An integrative review by Priester et al. (2016) reported that primary healthcare professionals frequently cited having a skill deficit and a lack of preparation in identifying both mental-health and substance-use disorders. In addition, a systematic review of 28 studies indicated that judgmental attitudes towards service users with substance use disorders were common in healthcare professionals and can lead to suboptimal treatment (van Boekel et al., 2013). As regards nurses—the largest group of healthcare professionals in all sectors of healthcare systems around the world (World Health Organization, 2022)— most continuing education programs available do not provide training on concurrent disorders (Garrod et al., 2020), and many mental health nurses do not feel confident discussing substance use issues with service users (Wadell and Skärsäter, 2007; Coombes and Wratten, 2007; Pinderup, 2018). Further, research indicates that most addiction treatment nurses perceive themselves as ill-equipped to screen, assess for, and refer those with mental health disorders (McCabe and Parrish, 2018).

1.3. Promoting the uptake and implementation of evidence-based practice for concurrent disorder management

Continuing professional education is consistently described as a significant implementation strategy for healthcare professionals to transfer guidelines successfully into clinical practice (Babenko et al., 2017; Scott et al., 2012; Wuchner, 2014). A recent Cochrane review of 215 studies showed that educational meetings and workshops have slightly moderate effects on changing professional practice (Forsetlund et al., 2021). Educational strategies targeting healthcare professionals’ practice include technology-enabled collaborative learning and capacity-building models (United States Department of Health and Human Services, 2019). Among these, the Extension for Community Healthcare Outcomes (ECHO®1) model has shown great promise for implementing evidence-based practice and improving patient health outcomes (Holmes et al., 2020; McBain et al., 2019; Zhou et al., 2016). ECHO is a tele-mentoring model that uses videoconferencing technology to actively spread evidence-based practice and build capacity among healthcare professionals to manage complex and chronic health conditions (Arora et al., 2017). Real-time educational sessions pair frontline medical and allied healthcare professionals (“the spokes”), typically in undeserved or remote communities, with an interdisciplinary panel of experts at an academic medical center (“the hub”), using case-based learning, peer learning, and formal didactics to share clinical guideline recommendations and provide rapid access to reliable information (Arora et al., 2007; Arora et al., 2011).

Three systematic reviews have investigated the impact of ECHO and suggest that this continuing education model positively impacts healthcare professionals’ satisfaction, knowledge, clinical confidence, and behavior changes (McBain et al., 2019; Holmes et al., 2020; Zhou et al., 2016). In addition, a few recent randomized and nonrandomized controlled studies show that ECHO significantly improves patient-related outcomes, including care processes and outcomes of care (Blecker et al., 2020; Diaz et al., 2019; Flynn et al., 2020; Hasselberg et al., 2019; Komaromy et al., 2019; Murphy et al., 2019; Gadomski et al., 2020). Despite this breadth of evidence, most of the empirical research on ECHO has focused on programs in hepatitis C treatment, chronic pain management, and geriatric care. Thus, the three above-mentioned systematic reviews have called for more research on this model across diverse fields and contexts. Indeed, environment and organizational context are often cited as factors playing a role in how successful technology-enabled implementation strategies are at securing changes in practice (Greenhalgh et al., 2017). However, the literature reporting on the outcomes of concurrent disorder-focused ECHO programs often lacks details about contextual specificities and provides limited information about the barriers and enablers to a successful implementation (Komaromy et al., 2017; Komaromy et al., 2016; Mehrotra et al., 2018; Sockalingam et al., 2017). Moreover, the ECHO-related literature gives little attention to the factors that influence practice changes among healthcare professionals—or to understanding the relationships between these factors (Faherty et al., 2020). Specifically, there is a dearth of studies focusing on the nursing perspective and on outcomes in the field of concurrent disorder care (Gordon et al., 2016; White et al., 2019). These topics are particularly important because nurses are on the front lines in most healthcare service access models for individuals with concurrent disorders (Canadian Institute for Health Information CIFI, 2022). In addition, nurses often practice as case managers, which involves creating individualized and comprehensive patient care plans, developing therapeutic relationships that help support people with complex needs, and coordinating health care services (Bauer and Bodenheimer, 2017).

This study strove to help fill these gaps. Its overarching aims were to develop a comprehensive understanding of the impact of a Canadian concurrent disorder-focused ECHO program on nurses’ competency development and clinical practice and to uncover the conditions for successful uptake and implementation of concurrent disorder evidence-based practice in nursing care.

2. Methods

2.1. Context and educational intervention

In 2018, an ECHO program for concurrent disorder management (ECHO Concurrent Disorders) was developed and implemented at a large quaternary teaching hospital centre in metropolitan Western Canada. The initiative showed a high level of acceptance in the community; during its first two cycles (2018–2019 and 2019–2020 curricula), ECHO Concurrent Disorders was offered to up to 200 registrant healthcare professionals each year, with an average of 50 to 60 attendees connecting at the same time in any given online educational session.

Congruent with the ECHO model, the structure and delivery of ECHO Concurrent Disorders was informed by behavior change theory (Bandura, 1977), social learning theories (Wenger, 1998; Lave and Wenger, 1991), and adult teaching methods (Arora et al., 2017), and its content was developed in alignment with recent clinical guidelines for concurrent disorder management (National Institute for Health and Care Excellence, 2016; Crockford and Addington, 2017; Hakobyan et al., 2020). Online educational sessions occurred every two weeks over a 10-month period, for a total of 20 sessions per cycle. During a session, healthcare professionals from diverse practice settings across the province (spokes) and the expert team at the quaternary hospital centre (hub) discussed an active, deidentified patient case presented by a healthcare professional (or a team of healthcare professionals) for about an hour, after which hub and spoke participants provided clinical recommendations and individualized feedback. The last 20 min were reserved for an expert didactic presentation about various topics related to concurrent disorder evidence-based practice.

A detailed description of the educational program—including adaptations made for the learners and their local context, and unplanned modifications that took place in delivering the program's 2019–2020 curriculum due to the COVID-19 pandemic—was previously reported (Chicoine et al., 2022a), in accordance with the Guideline for Reporting Evidence-based practice Educational interventions and Teaching (GREET) (Phillips et al., 2016). More information is also publicly available online (https://ruisss.umontreal.ca/cectc/services/echo-troubles-concomitants/).

2.2. Study design

Our approach relied on a social-constructivist-inspired conception viewing quantitative and qualitative methods as two inputs that can enrich each other through a “looping effect” (Chicoine et al., 2021; Hacking, 1999); thereby providing fertile ground to fulfill the overarching aims of this study. Specifically, this study was guided by the following three research questions:

• -What is the evolution in nurse-related outcomes over a 12-month period of participation in ECHO Concurrent Disorders? (quantitative research question)
• -How did the nurses implement, in their clinical practice, the competencies they perceived as having developed through their participation in ECHO Concurrent Disorders, and what factors have influenced this process? (qualitative research question)
• -In what ways is the evolution in nurse-related outcomes over a 12-month participation in ECHO Concurrent Disorders linked with the development and implementation of their competencies in clinical practice? (mixed-methods research question)

To answer these research questions, the study employed a mixed-methods convergent parallel design (Creswell and Plano Clark, 2018), which is recognized as the ideal method for investigating the impact and conditions of knowledge translation strategies (Greenhalgh et al., 2004; Vedel et al., 2018; Palinkas et al., 2011). It involves using the quantitative and qualitative methods concomitantly to collect, analyze, and interpret the data; then both sets of results are compared in the final integration of the study (Pluye and Chicoine, 2022). The rationale for using this mixed-methods approach was complementarity (Fetters et al., 2013), as it was expected that the nurses’ perspectives on their competency development and the factors influencing this process, from the qualitative component of the mixed-methods study, would be complementary to the quantitative component on the impact of ECHO Concurrent Disorders on nurse-related outcomes. A visual representation of the convergent mixed-methods design was previously published in the study protocol, which can be found in a publication elsewhere (Chicoine et al., 2021).

We present the mixed-methods findings emerging from the analysis and integration of quantitative and qualitative results and on how and when this integration occurred. The article is structured in accordance with the guidelines for Good Reporting of A Mixed Methods Study (GRAMMS) (O'Cathain et al., 2008), which outline six criteria to consider.

2.3. Study population and sampling

The study took place during the program's first two years of implementation, from September 2018 to December 2020. The potential study population comprised all 65 nurses who registered in ECHO Concurrent Disorders for the 2018–2019 or 2019–2020 curricula. In the quantitative component of the mixed-methods study, a census approach to sampling was undertaken: all nurses who registered in ECHO Concurrent Disorders and who consented to the research were included and invited to complete self-administered surveys online. For the qualitative component of the mixed-methods study, nurses who participated in at least one online educational session were recruited by email and invited to participate in semi-structured interviews. Table 1 summarizes the sociodemographic data and practice profiles of participants for both the quantitative and qualitative components.

Table 1.

Sociodemographic data and practice profiles of the nurse participants in the quantitative and qualitative components of the mixed-methods study.

Characteristics	QUAN N = 28	QUAL N = 10
Gender—n (%)
Female	27 (96.4)	9 (90.0)
Male	1 (3.6)	1 (10.0)
Age—Mean (SD)	39.1 (6.2)	39.4 (3.3)
Degree earned—n (%)
Undergraduate diploma	2 (7.1)	0 (0)
Bachelor's degree	20 (71.4)	4 (40.0)
Master's degree	6 (21.4)	6 (60.0)
Professional role—n (%)
Nurse	25 (89.3)	8 (80.0)
Clinical nurse specialist	2 (7.1)	2 (20.0)
Auxiliary nurse	1 (3.6)	0 (0)
Years of clinical experience—n (%)
0–5 years	3 (10.7)	0 (0)
6–10 years	5 (17.9)	2 (20.0)
11–15 years	15 (53.6)	7 (70.0)
16–10 years	2 (7.1)	1 (10.0)
21+	3 (10.7)	0 (0)
Work setting—n (%)
Hospital-based healthcare	15 (53.6)	3 (30.0)
Community-based addiction treatment	3 (10.7)	2 (20.0)
Primary mental healthcare services	9 (32.1)	4 (40.0)
Other	1 (3.6)	1 (10.0)
Area of practice—n (%)
Urban/Suburban	17 (60.7)	5 (50.0)
Rural/Remote	9 (32.1)	4 (40.0)
Mixeda	2 (7.1)	1 (10.0)
Registration year in the program—n (%)
2018–2019 curriculum	10 (35.7)	6 (60.0)
2019–2020 curriculum	18 (64.3)	4 (40.0)
Attendance—Mean (SD)b	8.6 (5.9)	9.1 (4.5)

N = total number of participants; n = subgroup; QUAL = qualitative; QUAN = quantitative; SD = standard deviation.

^aMixed area of practice refers to nurse participants who worked in a large area of the province that included both urban and rural settings.

^bDuring first year of participation in the program.

2.4. Quantitative component of the mixed-methods study

An uncontrolled pre- post-test design was used to measure changes in nurse-related outcomes over a 12-month period of participation in ECHO Concurrent Disorders. Based on Moore et al. (2009), self-efficacy in concurrent disorder management was chosen as the primary outcome. Secondary outcomes included knowledge of and attitude toward working with people with concurrent disorders; participation, satisfaction, and acceptability; and perception of clinical performance. A total of 28, 19, and 12 nurses completed self-reported questionnaires through the online tool, LimeSurvey (Copyright© 2006–2023, Lime Survey GmbH) at baseline and at the 6- and 12-month follow-ups, respectively. Detailed methods and results from the quantitative component of the mixed-methods study can be found elsewhere (Chicoine et al., 2022b).

2.5. Qualitative component of the mixed-methods study

An interpretive description methodology (Thorne, 2016) was chosen to explore the experiences and perceptions of the nurses who participated in ECHO Concurrent Disorders about the competencies they developed and implemented in their clinical practice and the factors that influenced this process. All interviews were conducted by the primary author via the Zoom platform (Copyright© 2023, Zoom Video Communications Inc) and lasted between 45 and 90 min. An interview guide was drafted and refined during the data collection. The recruitment took place between May 2020 and July 2020. Ten nurses volunteered to participate in the interviews (see Table 1). Detailed methods and results from the qualitative component of the mixed-methods study have been previously reported (Chicoine et al., 2022a).

2.6. Analysis and integration of quantitative and qualitative results

The mixed-methods analysis involved three steps: (1) Quantitative results: descriptive quantitative analysis, analysis of variance, and subgroup analysis; (2) Qualitative results: thematic inductive analysis and member checking; and (3) Integration of quantitative and qualitative results. As this article focuses on the study's mixed-methods findings, the contributing quantitative (statistics) and qualitative (themes) results—which have been published elsewhere—are presented here as inputs in the integration process (mixed-methods analysis of two interdependent sets of results).

2.6.1. Step 1 – quantitative results: descriptive statistics, repeated measures analysis (ANOVA), and subgroup analysis

Quantitative analyses were conducted using the SAS statistical analysis software V.9.4 (Copyright 2022, SAS Institute Inc). Participants’ characteristics were assessed descriptively at the baseline. Their satisfaction with and acceptance of the program and their perception of their clinical performance were covered in the 6- and 12-month follow-ups. To measure changes in the main participant-related outcomes (self-efficacy, knowledge, and attitude) over the three data collection time points, a repeated measures analysis of variance (ANOVA) was performed. The linear mixed models (Fitzmaurice et al., 2004) included the continuous dependent outcome measured at baseline, 6 months, and 12 months, with the within-participant time effect as a fixed effect and the participant intercept as a random effect (Ferguson, 2009). Similarly, linear mixed models were used to examine subgroups and interactions with the study's main outcomes, stratified by session attendance frequency (low [0–5 sessions] versus high [6–20 sessions]).

Overall, the ANOVA longitudinal analysis revealed that there was no statistically significant improvement in the nurses’ self-efficacy at the 6-month or 12-month follow-ups (Fig. 1), as compared to the baseline (Chicoine et al., 2022b). Nevertheless, the results showed that both knowledge and attitude scores significantly improved from baseline to the 6-month and 12-month follow-ups (Fig. 1). In the subgroup analysis, the results indicated that self-efficacy improved significantly from baseline to the 12-month follow-up among nurses with high attendance (Fig. 2a). As for knowledge (Fig. 2b) and attitude (Fig. 2c), the improvements found in the ANOVA longitudinal analysis remained statistically significant only for nurses with high session attendance, at both the 6-month and 12-month follow-ups.

Fig. 1.

Quantitative results from longitudinal ANOVA analysis for self-efficacy, knowledge and attitude (N = 28).

Fig.1 shows the least-squares mean estimators from linear mixed models with self-efficacy, knowledge, and attitude outcomes. p-value of change from baseline: *** <0.01 ** <0.05 * <0.1.

Fig. 2.

Quantitative results from longitudinal ANOVA analysis, stratified by session attendance frequency (n = 12/28 versus n = 16/28).

Results for the subgroup with low session attendance frequency (0–5 sessions; n = 12/28) are represented with solid lines, and the subgroup with high session attendance frequency, with dotted lines (6–20 sessions; n = 16/28). p-value of change from baseline: *** = <0.01 ** = <0.05 * = <0.1.

For attitude (in blue color): results show a significant decrease in nurses’ mean scores on the Co-Morbidity Problems Perceptions Questionnaire from baseline to 6 and 12 months, representing an improvement in their attitude toward concurrent disorders (a lower score on the Co-Morbidity Problems Perceptions Questionnaire denotes a more positive attitude toward working with concurrent disorders, while a higher score represents a more negative attitude) (Pinderup, 2017).

2.6.2. Step 2 – qualitative results: thematic inductive analysis and member checking

Qualitative analyses were conducted with MAXQDA 2020.1 (Copyright© 1995–2023, MAXQDA – Distribution by VERBI GmbH) using a thematic inductive analysis (Paillé and Mucchielli, 2016). Once completed, the recorded interviews were professionally transcribed and the transcripts, supplemented with the interviewer's notes, were coded and analyzed using an iterative, data-driven approach to code development. The analyst listened to the audio recordings and read the interview transcripts to familiarize herself with the data and to make sure that the transcription was accurate. Then, meaningful units of text within the transcripts were coded and grouped together to generate a list of initial themes that pinpointed patterns in the data. Charting and visualization tools in MAXQDA were used to further explore the data and scrutinize relationships among emerging themes. This took shape as a hierarchy of themes and subthemes that reflected a deeper understanding and interpretation of how the data contributed to answering the qualitative research question.

The analytical process took place through frequent discussions between the analyst and two other research team members to arrive at a common understanding of the nurses’ experiences by refining and renaming the themes and subthemes until a consensus was reached. To enhance trustworthiness, a member checking technique (Birt et al., 2016), also known as participant validation, was used to explore the creditability of the qualitative results. Four main themes and 18 contributing subthemes resulted from the inductive thematic analyses and member checking (see Supplementary Material Table 3). Together, these themes reflected the progress made by the nurses on some clinical nursing competencies and how these competencies were implemented into their clinical practice (theme 1). The themes also illuminated how this learning process took place during ECHO Concurrent Disorders (theme 2), and they depicted the factors that facilitated (theme 3) or limited (theme 4) the nurses’ competency development and practice changes (Chicoine et al., 2022a).

2.6.3. Step 3 – mixed-methods findings: integration of quantitative and qualitative results

Then, the study's quantitative and qualitative results were merged to compare, contrast, and corroborate the emerging results and to gather complementarity insights (Pluye et al., 2018). This was achieved by applying an integration strategy inspired by the Pillar Integration Process (Johnson et al., 2019). The Pillar Integration Process involves a structured, four-stage process to centralize, balance, and interpret the quantitative and qualitative results, visually and technically, within the same analytical framework (Johnson et al., 2019). The Pillar Integration Process was chosen because it focuses on exploring or expanding on the findings and generating new inferences, rather than comparing two different data collection methods focused on the same phenomena, simply for validation purposes.

In this study, the first step in the analysis and integration of quantitative and qualitative results involved juxtaposing the main study results into a single table, usually referred to as a “matrix” or “joint display” in the literature (Hong et al., 2020). Second, once the listing was completed, a “matching process” was conducted, in which two independent coders performed a side-by-side comparison of the quantitative and qualitative results to identify similarities, differences, or discrepancies, and complementary information. Third, for each combination of quantitative and qualitative results, a preliminary mixed-methods interpretation was generated by the first author and then reviewed with another researcher to verify its accuracy with the study's quantitative and qualitative results and to stimulate further insights into the arising mixed-methods interpretations.

In the fourth and final stage, namely, “Pillar Building”, the preliminary mixed-methods interpretations were collated and synthesized in a last analytical effort by searching for commonalities or relationships between them and by narrowing down the preliminary interpretations into broad-based categories that pinpoint the final mixed-methods findings. The resulting mixed-methods findings were refined and renamed until a consensus was reached among the research team members. To illustrate the analytical integration process through which the study's mixed-methods findings emerged, a sample of combinations between quantitative and qualitative results, alongside their corresponding preliminary mixed-methods interpretations, is displayed in Table 2 for each final mixed-methods finding.

Table 2.

Sample of combinations between quantitative and qualitative results with their corresponding preliminary mixed-methods interpretations, for each mixed-methods finding.

In what ways is the evolution in nurse-related outcomes over a 12-months’ participation in ECHO—CD linked with the development and implementation of their competencies in clinical practice? (MM research question)
QUAN results		MM interpretation		QUAL results
MM finding 1: Opportunities for practice and validation to consolidate learning and foster the implementation of new competencies in clinical nursing practice
Self-efficacy in CD management (primary outcome)ANOVA; N = 28: -LS Mean (95% CI): T0: 7.8 (7.4; 8.2) T1: 7.8 (7.4; 8.3) T2: 7.9 (7.3; 8.4) -p-value, ESa: T1-T0: 0.8363b; 0.06 T2-T0: 0.7665b; 0.07	Highlight: Learning based on experiential knowledge therefore seems more helpful in reinforcing self-efficacy. Complementarity between QUAN and QUAL results: The QUAL subthemes 2.1 and 2.2 on learning processes in ECHO—CD help to better understand the reasons why the nurses’ self-efficacy did not improve significantly from baseline to 6 and 12 months (QUAN results), regardless of session attendance frequency (which components of the programs were more conducive to competency development and practice change). Listening to other participants’ experiences through storytelling, and benefiting from experts’ didactic presentations on EBP allowed nurses to reflect on their own knowledge and clinical practice, and for some of them, this took the form of verbal persuasion to change their current behaviors (QUAL results). However, the nurses who presented a clinical situation or who participated in a problem-solving activity had the opportunity to practice what they had learned and then to receive personalized feedback and validation from the group.				Theme 2 – Learning through a shifting lens and transforming clinical practice: -Subtheme 2.1 – Developing one's competencies through peer experience -Subtheme 2.2 – Developing one's competencies by collaborating with CD experts
Self-efficacy in CD management (primary outcome)Subgroup analysis (low vs high attendance; n = 12/28 vs n = 16/28):Low attendance (0–5 sessions; n = 12/28): -LS Mean (95% CI): T0: 8.0 (6.7; 9.4) T1: 7.6 (6.1; 9.2) T2: 7.0 (5.5; 8.6) -p-value, ESa: T1-T0: 0.3534b; −0.55 T2-T0: 0.0626b; −0.90 High attendance (6–20 sessions; n = 16/28): -LS Mean (95% CI): T0: 7.9 (7.3; 8.4) T1: 8.1 (7.6; 8.7) T2: 8.5 (7.8; 9.1) -p-value, ESa: T1-T0: 0.2162b, 0.33 T2-T0: 0.0213c, 0.53 (moderate ES of change)e	Highlight: Prolonged and committed participation in ECHO—CD is essential for nurses to practice what they have learned and then receive feedback from their peers. These learning opportunities allow for self-reflection and help reinforce nurses’ belief in their own capacity to manage CD. Divergence between QUAN (significant improvements for self-efficacy only among nurses who participated in more than 5 online sessions during a curriculum) and QUAL results (nurses viewed that their participation in ECHO—CD contributed to reinforcing their self-confidence in CD management). Possible explanation: It is possible that, with minimal attendance in ECHO—CD (0–5 sessions), nurses were exposed to their peers’ experiences, which provided them with practical solutions on how to manage the complex healthcare needs of the service users encountered. In this case, learning through storytelling allowed them to reinforce their belief in their own potential to manage CDs. However, improvement in nurses’ self-efficacy requires participation that is both prolonged (12+ months) and committed (presenting a clinical situation, interacting with others).				Theme 2 – Learning through a shifting lens and transforming clinical practice: -Subtheme 2.3 – Developing one's competencies by strengthening one's self-confidence
MM finding 2: Reciprocal and trusting relationships in an interprofessional learning environment to strengthen commitment to continuing education and sustain participation
Satisfaction and acceptability toward the programDescriptive statistics:Domain “Perceived interactions and collaboration with other participants”d (mean[SD]): T1 (n = 19): 5.1(0.7) T2 (n = 12): 4.6(0.6) Items: -“The group members asked questions to foster a deeper understanding” (mean[SD]): T1 (n = 19): 5.9(1.0) T2 (n = 12): 5.3(0.9) -“The group members easily explained things to others” (mean[SD]): T1 (n = 19): 5.8(1.0) T2 (n = 12): 5.1(1.1) -“The group members provided positive feedback at the right time” (mean[SD]): T1 (n = 19): 6.1(1.1) T2 (n = 12): 5.2(0.8) -“I felt comfortable sharing my knowledge, expertise and personal experiences with other participants” (mean[SD]): T1 (n = 19): 3.9(1.6) T2 (n = 12): 3.8(1.4)	Highlight: The group modality in ECHO—CD can be either an enabler for, or a barrier to, nurses’ engagement in the program and competency development. It facilitates self-reflection, knowledge sharing, and uptake. It also gives nurses in difficult situations emotional support and prevents professional isolation. However, trust-based relationships and positive interactions are essential in this type of environment, for participants to engage in learning activities and thereby develop their competencies. Similarity between QUAN and QUAL results: Nurses felt positively about group interactions, and the educational environment was experienced as supportive and convivial. The presence of a facilitator throughout the sessions and the positive feedback from the expert team helped participants share their personal experiences during the online educational sessions. The interdisciplinary background of the participants was viewed as an advantage for knowledge sharing. Corroboration between QUAN and QUAL results: Nurses felt uncomfortable about sharing their knowledge, ideas, and personal experience with other participants when they started the program. The large number of attendees and the high-level expertise in CDs of certain participants caused misgivings, fear, and intimidation in some nurses.				Theme 3 – Factors facilitating competency development and practice change: -Subtheme 3.2 – Feeling a sense of belonging to a community -Subtheme 3.3 – Learning in an interprofessional environment +Theme 4 – Factors limiting competency development and practice change: -Subtheme 4.3 – Learning in a group by way of real-time videoconferences
Self-efficacy in CD management (primary outcome)Subgroup analysis (low vs high attendance; n = 12/28 vs 16/28):Low attendance (0–5 sessions; n = 12): -LS Mean (95% CI): T0: 8.0 (6.7; 9.4) T1: 7.6 (6.1; 9.2) T2: 7.0 (5.5; 8.6) -p-value, ESa: T1-T0: 0.3534b; −0.55 T2-T0: 0.0626b; −0.90 High attendance (6–20 sessions; n = 16): -LS Mean (95% CI): T0: 7.9 (7.3; 8.4) T1: 8.1 (7.6; 8.7) T2: 8.5 (7.8; 9.1) -p-value, ESa: T1-T0: 0.2162b, 0.33 T2-T0: 0.0213c, 0.53 (moderate ES of change)e Knowledge of CDsANOVA; N = 28: -LS Mean (95% CI): T0: 63.4 (58.6; 68.2) T1: 71.5 (66.0; 77.1) T2: 74.5 (67.9; 81.1) -p-value; ESa: T1-T0: 0.0045c, 0.72 (moderate ES of change)e T2-T0: 0.0014c; 0.94 (large ES of change)e	Highlight: Active engagement, which involves participants taking action in their own learning journey and being part of the learning community, is essential for continuing professional development. Passive participation, which implies being in listening mode, may be sufficient for nurses to acquire new knowledge, but these gains remain only potential cognitive resources that could eventually be use in clinical practice. Complementarity between QUAN and QUAL results: The QUAN results on nurse-led outcomes (from subgroup analysis) and the QUAL results (factors facilitating competency development and practice change) complement one another and provide an overview of the “ideal” type of participation in ECHO—CD for learning to occur and be reinvested in clinical practice. Beyond the minimum required level of exposure to ECHO—CD (QUAN results; > 5 sessions and over a 12-month period), the QUAL results provide details on the type of participation essential in each session for participants to develop their own competencies and then implement them in their clinical practice.				Theme 3 – Factors facilitating competency development and practice change: -Subtheme 3.2 – Feeling a sense of belonging to a community
MM finding 3: Peer-to-peer sharing of similar experiences and mentoring activities to normalize experience and provide emotional support
Participants’ characteristics (from sociodemographic data) Descriptive statistics (T0; N = 28): Years of clinical experience (n[%]): 0–5: 3(10.7) 6–10: 5(17.9) 11–15: 15(53.6) 16–20: 2(7.1) 21+: 3(10.7)	Highlight: Emotional support and mentoring are critical components in continuing education programs targeting healthcare professionals who provide care to people with chronic and complex healthcare needs, to prevent professional isolation. Complementarity between QUAN and QUAL results: The nurses who registered for ECHO—CD are not novices (QUAN results), making them open to reflexive-based learning methods (QUAL results). Nurses demonstrated a capacity for introspection during peer-to-peer sharing of similar experiences and when receiving feedback from other participants (peers and experts). Although most nurses (71.4%) had more than 11 years of clinical experience (QUAN results), they experienced ECHO—CD as an opportunity to discuss difficult clinical situations with other healthcare professionals sharing common realities (QUAL results). They expressed a sense of comfort in seeing that others were challenged with the same clinical situations or in receiving positive feedback from experts on their capacity to implement new interventions in their practice.				Theme 2: Learning through a shifting lens and transforming clinical practice: -Subtheme 2.1 – Developing one's competencies through peer experience -Subtheme 2.2 – Developing one's competencies by collaborating with CD experts
MM finding 4: Collaboration with experts to facilitate cross-disciplinary knowledge sharing and rapid knowledge uptake
Knowledge of CDsANOVA; N = 28: -LS Mean (95% CI): T0: 63.4 (58.6; 68.2) T1: 71.5 (66.0; 77.1) T2: 74.5 (67.9; 81.1) -p-value; ESa: T1-T0: 0.0045c; 0.72 (moderate ES of change)e T2-T0: 0.0014c; 0.94 (large ES of change)	Highlight: ECHO—CD allows experts from diverse backgrounds to share CD EBP with other participants and let them access reliable information quickly. This interdisciplinary panel of experts is essential for the nurses to acquire knowledge on CDs, which inherently require multidisciplinary care. There is a corroboration between QUAN and QUAL results regarding the acquisition of new knowledge and skills in CD EBP during ECHO—CD. The QUAL results also complemented the QUAN results by providing further insights into what specific areas of knowledge the nurses improved in. Most mental health or psychiatric nurses felt they gained knowledge in substance use disorders (substance withdrawal management, relapse prevention, harm reduction), while addiction treatment nurses perceived they acquired knowledge about mental health disorders (primary versus induced disorders, psychotic and anxiety disorders, psychiatric medications).				Theme 1 – Developing competencies to use in clinical practice when encountering people with CDs: -Subtheme 1.4 – Using new knowledge and skills to deliver evidence-based interventions to people with CDs
Knowledge of CDsANOVA; N = 28: -LS Mean (95% CI): T0: 63.4 (58.6; 68.2) T1: 71.5 (66.0; 77.1) T2: 74.5 (67.9; 81.1) -p-value; ESa: T1-T0: 0.0045c, 0.72 (moderate ES of change)e T2-T0: 0.0014c; 0.94 (large ES of change)	Highlight: Mentoring activities with experts are essential for participants to acquire specialized knowledge, and less-active means of learning (didactic presentations) remain beneficial for sharing EBP with participants. Complementarity between QUAN and QUAL results: The QUAN results showed that nurses improved their knowledge of CDs after 6 months, with a medium ES of change. The QUAL results extended these results by providing details into how this learning process occurred (program components) and about facilitating factors. Nurses felt they acquired knowledge in CD EBP specifically through collaborating with experts in the field of CDs, whom they viewed as mentors. Also, nurses strongly appreciated the didactic presentations on CD EBP by the expert panel, which provided them with reliable, up-to-date information relevant to their clinical practice.				Theme 2 – Learning through a shifting lens and transforming clinical practice: -Subtheme 2.2 – Developing one's competencies by collaborating with CD experts +Theme 3 – Factors facilitating competency development and practice change: -Subtheme 3.1 – Being provided with relevant educational material
MM finding 5: Reinforcement of positive attitudes about performing their professional role in complex and adverse situations to foster ongoing learning and renewed practice
Attitude toward working with people with CDsANOVA; N = 28: -LS Mean (95% CI): T0: 90.5 (83.4; 97.6) T1: 82.2 (74.0; 90.5) T2: 78.0 (68.2; 87.8) -p-value; ESa: T1-T0: 0.0472c, −0.44 (small ES of change) T2-T0: 0.0139c, −0.59 (moderate ES of change)e	Highlight: Participating in ECHO—CD gave nurses an opportunity to share their CD management problems with peers and mentors, thereby allowing nurses to reflect on their own attitudes toward CDs. This stimulated nurses to shift their outlook to a positive one as regards their professional role and the possibility of recovery for people with CDs. This open-mindedness further motivated the nurses to pursue their own professional development in order to offer quality care. Complementarity between QUAN and QUAL results: The QUAL results provided more details about how ECHO—CD contributed to the nurses’ reinforcement of positive attitudes toward working with people with CDs (QUAN), and how these improvements were implemented in their clinical practice. Reflecting on their participation in ECHO—CD, the nurses appreciated listening to peers’ experiences and witnessing how they were able to manage CDs. This helped them cultivate therapeutic optimism with the service users encountered. This sense of hopefulness for the possibility of recovery in people with CDs fostered the nurses’ interest and their motivation to pursue their own professional development (other educational opportunities). There is a corroboration between QUAN and QUAL results regarding nurses’ adoption of a positive attitude toward working with people with CDs. The QUAL results also complemented the QUAN results by providing further insights into how the nurses’ improvement in their attitude toward CDs was used in their clinical practice; that is, adopting nonjudgmental attitudes toward service users’ choices and lifestyle to maintain therapeutic alliance.				Theme 1 – Developing competencies to use in clinical practice when encountering people with CDs: -Subtheme 1.1 – Pursuing its own professional development and further enhancing practices for CDs by using one's learning experience -Subtheme 1.2 – Integrating new interventions while dealing with the complex healthcare needs of people with CDs -Sub-theme 1.8 – Adopting non-judgmental attitudes toward people with CDs to maintain therapeutic alliance
MM finding 6: Learning experiences that are team-based and tailored to the setting specifics and receive organizational support to promote coherency in practices and a culture of change
ParticipationDescriptive statistics: -Session attendance frequency for first year of participation (N = 28; mean[SD]; possible range 0–20): 8.6(5.8) -Number of clinical situations presented in the last 6 month (mean[SD]; median[min; max]): T1 n = 19): 0.3(0.6); 0(0; 2) T2 (n = 12): 0.7(0.9); 0(0; 2)	Highlight: Lack of support from organizations may have hindered the nurses’ willingness to engage in continuing educational programs and reinvest their new knowledge and skills in clinical settings. Complementarity between QUAN and QUAL results: The QUAL results helped clarify why the nurses did not participate in the program optimally, in terms of session attendance frequency and number of patient cases presented over a curriculum (QUAN results). Organizational-level factors negatively impacted nurses’ attendance and engagement in the program. For example, lack of support from employers hindered the nurses’ capacity to engage in the program (lack of time to prepare a presentation of a clinical situation or to participate in a full session) or ability to take full advantage of the program's learning methods (unavailable minimum required technological equipment, such as webcam, good Internet connectivity or firewalls that hindered access to the Zoom platform).				Theme 4 – Factors limiting competency development and practice change: -Subtheme 4.2 – Experiencing lack of support from employer
Perception of clinical performanceDescriptive statistics:Percentage of nurses who said they implemented into their clinical practice either the recommendations they received or the learning they acquired during ECHO—CD, over the last 6 months (mean[SD]): T1 (n = 19/28): 58% T2 (n = 12/28): 58%	Highlight: Team-based educational strategies are essential to enhance coherence between the care team members and to foster the uptake and implementation of CD EBP. Tailoring educational content to the participants’ specifics is a prerequisite for learners to engage in meaningful learning opportunities and foster their capacity to implement EBP in CD care. Complementarity between QUAN and QUAL results: The QUAL results provided insights into the factors that limited the nurses’ capacity to implement what they learned from their participation in ECHO into their clinical practice. Interpersonal factors: Nurses expressed being unable to implement recommendations or new ideas into the service users’ care plan due to their coworkers’ lack of openness to change or their continued work with a punitive approach toward people with CDs. However, this was not the case for nurses who participated in ECHO—CD with their coworkers; Contextual factors: Clinical recommendations proposed during ECHO—CD that did not match the local resources at the nurses’ disposal, lack of organizational support to implement CD EBP and sustain practice change (utilization of new clinical tools) and difficulty in implementing interventions using an integrated care approach due to the pervasiveness of a silo culture of care (treating substance use disorders before addressing mental health needs).				Theme 4 – Factors limiting competency development and practice change: -Subtheme 4.1 – Working with limited resources outside of major urban centers

ANOVA = analysis of variance; CD = concurrent disorder; CI = confidence interval; EBP = evidence-based practice; ECHO = Extension for Healthcare Community Outcomes; ECHO—CD = ECHO program for concurrent disorder management; ES = effect size; LS = least squares; Max = maximum; Min = minimum; MM = mixed methods; N = total number of participants; n = subgroup; SD = standard deviation; T0 = baseline; T1 = 6-month follow-up; T2 = 12-month follow-up.

^aCohen's d effect size was calculated as the estimated means difference divided by the pooled standard deviation (Ferguson, 2009).

^bStatistically not significant.

^cStatistically significant.

^dEach item was rated on a 7-point Likert scale, from 1 (strongly disagree) to 7 (strongly agree).

^eThe magnitude of Cohen's d effect size was interpreted using Cohen's classification (<0.2 = negligible; 0.2–0.49 = small; 0.5–0.8 = moderate; >0.8 = large) (Cohen, 1998).

2.7. Ethical considerations

Ethics approval was obtained from the Research Ethics Committees at one hospital center (#19.295) and one university (#CERSES-20–017 R). All methods were carried out in accordance with relevant guidelines and regulations (Government of Canada, 2022). Written and informed consent to participate in the study was obtained from all surveyed participants before program onset for the quantitative component of the study and then before each interview for the qualitative component. Participation was voluntary and not connected to program registration or performance; interview participants were compensated with C$50.

3. Results

Integration of quantitative and qualitative results produced a total of six new themes, identified as the study's mixed-methods findings 1 to 6 (see Table 2). Together, these themes describe how the evolution in nurse-related outcomes over their participation in ECHO Concurrent Disorders is linked to the development and implementation of their competencies in clinical practice (mixed-methods research question). Using a three-layered representation, the six mixed-methods findings from the integration process are summarized in Fig. 3 to highlight the relationships among them and then are described in greater detail in the following paragraphs.

Fig. 3.

Key conditions for the successful uptake and implementation of concurrent disorder evidence-based practice in nursing care with ECHO.

CDs = concurrent disorders; ECHO = Extension for Healthcare Community Outcomes.

Overall, Fig. 3 illustrates how mixed-methods findings 1 to 6 are interconnect in a comprehensive view of how ECHO Concurrent Disorders impacts nurses’ competency development and what the key conditions are for successful uptake and implementation of concurrent disorder evidence-based practice in nursing care. In the top layer of Fig. 3 (the contextual layer), the educational and practice contexts appear intertwined, highlighting that these two environments are constantly interacting with and feeding off one another. It also points out that it is paramount for the successful uptake and implementation of concurrent disorder evidence-based practice that these two environments (ECHO program and concurrent disorder nursing care) be aligned and form a cohesive whole rather than two separate entities. This interplay between education and practice depicts how each mixed-methods finding influences the others, meaning that each key condition emphasizes this inseparable relationship between education and practice within the context of an implementation strategy through continuing professional education.

Then, the center of Fig. 3 illustrates mixed-methods finding 1, which reflects the experiential learning process that took place as the nurses developed their competencies and further refined their clinical practice. In an educational environment, this process begins with the nurses taking action in their own learning journey and engaging in meaningful learning experiences. According to our results, these meaningful learning experiences were for the nurses to present a clinical situation of a real, anonymized patient case and to actively engage in problem-solving activities with their ECHO peers and mentors. Meanwhile, in the practice setting, the nurses engaged in an ongoing learning and competency-building process, based on practicing what they have learned in their workplaces, to consolidate learning. These opportunities for practice and feedback in authentic settings were catalysts for the implementation of new concurrent disorder competencies in clinical nursing practice and for fostering ongoing learning and renewed practice. Above that, Fig. 3 shows that the nurses’ commitment to and sustained participation in ECHO Concurrent Disorders (linked with mixed-methods finding 2) were key aspects for this dual process of learning-while-practicing process to take place.

The educational and learning processes layer, which is displayed in the middle layer of Fig. 3, relates to both the educational and practice-related conditions that were essential throughout the nurses’ participation in ECHO Concurrent Disorders. This trajectory began with nurses entering the program, continued with their active participation, and then moved to implementing new concurrent disorder competencies in clinical nursing practice and pursuing their own professional development. The key conditions intertwining in this trajectory were:

• •Reciprocal and trusting relationships in an interprofessional learning environment (mixed-methods finding 2)
• •Peer-to-peer sharing of similar experiences and mentoring activities (mixed-methods finding 3)
• •Collaboration with experts (mixed-methods finding 4)
• •Reinforcement of positive attitudes about performing their professional role in complex and adverse clinical situations (mixed-methods finding 5)
• •Learning experiences that are team-based and tailored to the setting specifics and receive organizational support (mixed-methods finding 6).

Finally, the educational and learning-processes layer is linked to the impacts targeted by the conditions in place for the successful uptake and implementation of concurrent disorder evidence-based practice in nursing care, as shown in the bottom layer of Fig. 3 (the impact layer):

• •Normalize experience and provide emotional support (mixed-methods finding 3)
• •Facilitate cross-disciplinary knowledge sharing and rapid knowledge uptake (mixed-methods finding 4)
• •Consolidate learning (mixed-methods finding 1)
• •Promote coherency in practices and a culture of change (mixed-methods finding 6).

4. Discussion

4.1. Main findings

Using a convergent parallel design, this mixed-methods study developed a comprehensive understanding of the impact of a Canadian ECHO program for concurrent disorder management on nurses’ competency development and clinical practice. The quantitative results (nurses’ self-efficacy, knowledge, attitude, participation, satisfaction and acceptability, and perception of clinical performance) were merged with the qualitative results (nurses’ experiences and perceptions about the competencies they developed and implemented in their clinical practice and the factors that influenced this process) to produce greater insights into the conditions linked to the successful uptake and implementation of concurrent disorder evidence-based practice in nursing care. Overall, our findings suggest that participating in ECHO Concurrent Disorders contributed to the nurses’ competency development and that this participation can, under certain conditions, result in effective and sustainable clinical practice changes. To interpret and discuss the main findings of this study, below we pinpoint a number of questions to highlight our findings’ contributions to the advancement of knowledge and their implications for nursing education and practice and future research.

4.1.1. What do the findings of this study add to our current knowledge of the ECHO model and what are their implications for research?

Based on social learning (Lave and Wenger, 1991; Wenger, 1998) and behavior change (Bandura, 1977) theories, both of which build on the premise that social interactions are essential conditions for effective learning and behavior change to occur (Socolovsky et al., 2013), one promising component of the ECHO model is the interprofessional environment that fosters collaborative learning. The ECHO model therefore endorses that sharing professional experiences improves and reinforces learning, while peer support enhances one's motivation to and self-perception of being able to perform new behaviors in clinical practice (Arora et al., 2011). Consistently, our results indicated that peer-to-peer sharing of similar experiences and mentoring activities were essential conditions to normalize the nurses’ experience with concurrent disorder management and provide emotional support in difficult clinical situations. Previous mixed-methods and qualitative research in the field of ECHO have also reported several benefits of the model's interprofessional and collaborative component in terms of sharing evidence-based practice, empowering participants to manage complex clinical situations, and increasing their understanding of the roles played by each profession (Zhao et al., 2020; Hassan et al., 2020; Pagé et al., 2021; Shea et al., 2019; Shimasaki et al., 2019).

Moreover, our study contributes to advancing this evidence on the impact of the ECHO model by highlighting that opportunities for practice and validation—as a “key educational condition” (Cianciolo and Regehr, 2019) of the ECHO model—seem beneficial for fostering nurses’ competency development and practice change. This finding is critical both for the field of nursing science and for future research on the ECHO model regarding how continuing professional education is designed for and tailored to needs of the participants to help them contextualize their new knowledge and improve their confidence in implementing evidence-based interventions in their practice. As the ECHO model can be characterized as a complex educational intervention (multiple interacting components within the intervention and with its context) (Craig et al., 2008; Petticrew, 2011; Thomas et al., 2019), further research is needed to elucidate which educational conditions and learning methods are better suited to foster successful uptake and implementation of concurrent disorder evidence-based practice in nursing care through ECHO. For example, future studies could use a “blending” approach to adult learning theory/frameworks and design components of implementation research (Curran et al., 2012) and, in doing so, assess more specifically the clinical effects of ECHO on relevant outcomes (sustainable practice change and patients’ health), while systematically gathering and documenting the planned and unplanned strategies used during the research (Birken et al., 2020; Hailemariam et al., 2019).

4.1.2. How do the findings of this study help increase our understanding of the ECHO model's impact on nurses' competency development and practice change, and what are their implications for education and practice?

As outlined previously, our results showed that ECHO Concurrent Disorders provided opportunities for nurses to engage in ongoing learning and practice change, a process that includes two main activities; namely, putting new leaning into practice and then receiving validation from peers and experts regarding the changes made in the workplace. This allowed nurses to share with other participants how they effectively—or not—implemented into their clinical practice their new knowledge and skills, which, in turn, helped strengthen their belief in their own ability to suitably manage concurrent disorders or helped them find tangible solutions to deal with complex and adverse clinical situations. This process was an essential steppingstone in the nurses’ competency development in concurrent disorder care and, most importantly, in clinical practice change. Thus, competency development and practice change occurred simultaneously, through both educational and workplace learning situations. These two environments helped the nurses to consolidate their learning in an irreversible and sustainable way, which, according to Tardif (2006), represents a “cognitive reorganization” or a crucial developmental stage.

These findings reinforce the idea that nurses learn and develop their competencies through experience and that, consequently, education and practice must align and work interdependently (Pepin et al., 2017). Indeed, the belief that nurses learn while practicing nursing is based on decades of evidence in nursing (Benner, 1984; MacLeod, 1996; Takase et al., 2015; Benner et al., 2010; Jantzen, 2019) and adult education (Billett, 2001; Eraut, 2000; Eraut, 2004; Billett, 2004). For example, Jantzen (2019) found, from a grounded theory approach, that refining nursing practice includes both formal and informal learning. However, this author also indicated that significant nursing expertise is developed through “puzzling and enquiring”, an active and iterative process described as learning while nursing in the work setting, which requires self-consciousness and autonomy (Jantzen, 2019). In a related fashion, it has been asserted, as conceptualized in a systematic review by Davis et al. (2016), that working and learning must be understood as an integrated experience that enables nurses to implement contemporary, evidence-based, professional practice and continuously improve safe, quality patient care. Following along similar lines, in a recent metasynthesis, researchers showed that contextualizing learning and placing it in close proximity to practice enhanced nurses’ motivation and engagement toward continuing professional development (Mlambo et al., 2021). Likewise, it has been robustly documented that fragmentation in continuing professional development initiatives should be actively avoided and that this would require strong practice/education organizational partnerships and the promotion of learning in the practice setting (Mlambo et al., 2021; Davis et al., 2016).

Another important finding from this mixed-methods study is that ECHO contributed to reinforce nurses’ positive attitudes about performing their professional role in complex and adverse situations, which further stimulated ongoing learning and practice renewal. This is congruent with adult learning theory and previous nursing education research showing that transformative learning helps nurses develop emancipated and responsible attitudes toward their own learning (Tsimane and Downing, 2020; Cooper, 2009). To this end, Hoggan (2014) argues that a strong professional identity can have a powerful influence on behavior change because it prompts nurses to show receptiveness to new learning and openness to self-directed and reflexive learning and to demonstrate accountability for their own professional development. Overall, the findings of this study corroborate the evidence from many existing knowledge syntheses (Babenko et al., 2017; Coventry et al., 2015; Mlambo et al., 2021; Pool et al., 2016; Price and Reichert, 2017) stating that building a culture of lifelong learning in the workplace, one that values continuing professional education and encourages nurses to grow professionally, is a key condition to maintain high standards of care through competent nursing practice (Coventry et al., 2015; Cooper, 2009).

4.1.3. What theoretical insights do the findings of this study raise?

This study adds to the current literature on the ECHO model by shedding new light on the learning and educational processes that contribute to the successful uptake and implementation of concurrent disorder evidence-based practice in nursing care. Indeed, although the ECHO model was built on the theoretical foundations of social learning theories, the findings of this study seem to resonate with transformational learning theory (Mezirow, 2000). Transformative learning is a process that facilitates the transition from a transmissive pedagogical standpoint to a transformative paradigm of learning and interdependence between continuing education and practice (Pepin et al., 2017; Renigere, 2014). It advocates for democratic education for sustainable development, whereby learners are co-creators of their own learning through questioning, critical reflection, and creativity, in order to arrive at viewpoint changes that guide their actions (Mezirow, 2000). Transformative learning is dynamic and interactive, and, thus, it engages nurses to actively participate in their own learning. It enables the construction of contextualized knowledge that is useful for real-life professional setting (Tsimane and Downing, 2020). Transformative learning stimulates nurses to learn how to think independently. It develops their self-confidence and competence to meet the multifaceted healthcare needs of service users and to renew and adapt their practice in constantly shifting contemporary work environments (Frenk et al., 2010).

In this mixed-methods study, we found that ECHO allowed nurses to actively engage in opportunities for practice and validation, which, according to transformative learning, refers to an integrative process of making connections between concepts and experiences so that knowledge and skills can be reinvested into new, multifaceted clinical challenges (Sandra and Jon, 2011). Congruent with our findings, Tsimane and Downing (2020) report from their concept analysis results that the process of transformative learning in nursing education is facilitated through investigative, collaborative, inventive, and interactive learning activities. This can have implications for the future implementation of ECHO-affiliated programs, in terms of how certain elements in the original ECHO model can be adapted to better support nurses in developing emancipated and responsible attitudes toward their professional development. It also has implications for educators or facilitators regarding their readiness to embrace roles as active knowledge translation agents, mentors, and learning coaches, rather than serving as a transmission channel for the passive dissemination of evidence-based practice.

4.1.4. What can be inferred from the findings of this study to improve future implementations of the ECHO model?

In this study, we lay bare a large number of cultural, contextual, and organizational factors that negatively affected the nurses’ consistent participation in ECHO Concurrent Disorders (time constraints, limited access to technology, or lack of contextualized educative content) as well as their capacity to implement the competencies they developed in their clinical practice (limited concurrent disorder-specialized resources, high job turnover, or lack of employer support toward practice changes) (Chicoine et al., 2022a). Such barriers have also been uncovered in other ECHO programs addressing complex and chronic health topics within the constraints of resource-scarce healthcare settings (McBain et al., 2019; Zhou et al., 2016). For example, Pagé et al. (2021) used a qualitative study design to explore the factors influencing healthcare professionals’ uptake of an ECHO program for chronic pain management and found that expert recommendations or feedback were often “lost in translation,” which was mainly associated with insufficient multidisciplinary healthcare resources to offer gold-standard care. The same research group insists that ECHO programs should offer participants evidence-based guidance balanced between acknowledging the optimal therapeutic path for a given service user and what this service user can realistically have access to. This can have implications for the future implementation of ECHO-affiliated programs, as regards the extent that, and in what ways, the original ECHO model can be adapted to the needs of end users. It also has implications for educators or facilitators regarding their readiness to embrace roles as active knowledge translation agents, mentors, and learning coaches, rather than serving as a transmission channel for the passive dissemination of evidence-based practice.

Similar to our findings, researchers have shown that nurses are often reluctant to leave or prevented from leaving the clinical setting to attend continuing professional education due to heavy workloads and a lack of relief coverage, use of personal time to undertake mandatory training, and organizational culture and leadership issues constraining the implementation of learning to benefit service users (Coventry et al., 2015). Instead, as noted by Mlambo et al. (2021), relevant organizational support should prioritize both structural (allocation of time and funding for continuing education, adequate staffing, or healthy workplace culture conducive to practice change) and moral support (explicit managerial support and recognition for professional development or encouragements from peers, experts, and mentors). However, building on the findings of this study, we emphasize that alongside supportive environments, emotional support provided through peer-to-peer sharing and mentoring activities should also be prioritized in the educational setting to normalize nurses’ experience toward the many challenges they face in managing concurrent disorders.

Lastly, our results showed that a key condition for the successful uptake and implementation of concurrent disorder evidence-based practice was for nurses to attend ECHO with their co-workers. This facilitated knowledge sharing and practice changes and fostered a common, patient-centered vision in care team members. In the implementation science literature, a team-based approach has shown promising results for improving the implementation of evidence-based interventions in interprofessional primary healthcare settings and for overcoming barriers, such as a lack of communication and unshared professional values (Lau et al., 2015; Reimschisel et al., 2017). As the ECHO model typically targets primary care healthcare professionals who work in multidisciplinary teams (McBain et al., 2019), future studies should aim to evaluate the impact of ECHO on specific outcomes of care performance ,or care processes such as teamwork and collaborative skills. Research is also needed to deepen our understanding of the ECHO model's influence on relational dynamics at work and to explore how it can be used or improved further to foster interdisciplinarity and create a healthy learning culture in the workplace.

4.2. Strengths and limitations

This mixed-methods study is unique in that it relies on an integrative conceptual framework (Chicoine et al., 2021) developed by taking inspiration from a social-constructivist worldview of science (Hacking, 1999), stipulating that an in-depth inquiry arises from a ‘“looping effect” between quantitative and qualitative evidence that produces a “mixed kind” of evidence. Hence, the major strength of this study is its rigorous, thoughtfully- planned, mixed-methods design. In our mixed-methods convergent parallel design, both quantitative and qualitative methods were used concurrently, bolstering one another, and allowing us to develop a comprehensive understanding of the impact of an ECHO program for concurrent disorder management on nurses’ competency development and clinical practice. In addition, we used a structured and systematic integration process, the Pillar Integration Process (Johnson et al., 2019), to merge, compare, and contrast the quantitative with the qualitative results, which added value to both methods individually. This interwoven approach provided a fertile analytical ground to study the key conditions for successful implementation of evidence-based practice in concurrent disorder nursing care with the ECHO model.

This mixed-methods study also has some limitations. First, the quantitative study relied on a prospective cohort study design without the use of a control group, and the sample size available for analysis was small (N = 28). For this reason, causal inferences regarding exposure to the educational program and nurse-related outcomes could not be made. Second, our study was conducted in only one Canadian province, and the educational program was strongly contextualized to concurrent disorder care in this area. Although this may well reflect specific contextual aspects, it can also make our results difficult to transfer to other settings or regions. We therefore provided a detailed description of the ECHO Concurrent Disorders program in accordance with reporting guidelines for evidence-based practice educational interventions (Phillips et al., 2016; Chicoine et al., 2022a), which will facilitate its adaptation in other contexts. Third, we investigated outcomes and perspectives at the level of individual nurse participants. Outcomes or perspectives at the organizational and patient level warrant further exploration.

Finally, one important issue to consider in mixed-methods convergent designs is the divergences (also called contradictions, discrepancies, dissonances, and differences) between quantitative and qualitative results that can arise during the integration process, constituting a potential threat to the reliability of the mixed-methods findings (Creswell and Plano Clark, 2018). From a conceptual standpoint, however, some authors in the field of mixed-methods research argue that divergences in data/results can also stimulate rich theoretical questioning and shed new light on existing empirical knowledge (Hesse-Biber and Johnson, 2015). In this study, the divergences noticed during the integration process by the first author were systematically discussed with another researcher and then resolved by reviewing the preliminary mixed-methods interpretations and providing possible explanations, where appropriate. In addition, the Pillar Integration Process ensured rigor throughout the integration procedures, discrepancies being systematically addressed and documented, and its use further enhanced transparency in the reporting.

5. Conclusions

Continuing professional education is central to nurses’ lifelong learning and is a vital part of maintaining high standards of nursing care through competent evidence-based practice. Outcome measures and perspectives collected over 12 months indicated that participation in an ECHO program for concurrent disorder management contributed to nurses’ self-efficacy, knowledge, and attitude. The factors moderating competency development and practice change included consistent, active participation in an interprofessional community of learners supported by their organizations. Given the challenges associated with the implementation of recommended clinical guidelines in concurrent disorder nursing care, our findings highlight the importance of furthering our understanding of the key conditions for successful uptake and implementation. As a prerequisite, a better understanding of these conditions can help inform ways to optimize the applicability of the implementation strategy to the needs and specificities of nurses in order to obtain impactful and sustainable results. The findings of this study will provide guidance to nursing implementation researchers, educators, and practitioners in selecting strategies and conditions that facilitate the adoption of concurrent disorder nursing guidelines. Next steps for future research include adapting valid, reliable tools to measure changes in behavior and intention to use and actual use of concurrent disorder evidence-based practice and assessing the effectiveness of ECHO Concurrent Disorders using outcomes at the patient and organizational levels.

Funding sources

This study received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

CRediT authorship contribution statement

Gabrielle Chicoine: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Writing – original draft, Writing – review & editing. José Côté: Conceptualization, Supervision, Formal analysis, Validation, Writing – review & editing. Jacinthe Pepin: Conceptualization, Supervision, Formal analysis, Validation, Writing – review & editing. Pierre Pluye: Conceptualization, Methodology, Writing – review & editing. Didier Jutras-Aswad: Conceptualization, Data curation, Resources, Supervision, Formal analysis, Validation, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

• The data that support the results of this study are available from the corresponding author upon reasonable request.