Strengthening the Health Workforce through the ECHO Stages of Participation: Participants’ Perspectives on Key Facilitators and Barriers

Abstract

Introduction:

Project Extension for Community Health Outcomes (ECHO) was originally developed by the University of New Mexico’s Health Science Center (UNMHSC) to build the capacities of primary-care providers and to increase specialty-care access to rural and underserved populations. ECHO Colorado, a replication site at the University of Colorado Anschutz Medical Campus (CUAMC), was developed with the same purpose and to help build the health workforce of Colorado. The CUAMC and its community-based partners recognized that by reducing unnecessary referrals to the medical campus and building primary-care capacity in communities, both would increase their scope and expand overall capacity. This study examines the key factors that influence participant engagement, how participants value the ECHO experience, and the utility of the ECHO Colorado experience according to participants.

Methods:

This study used a mixed-methods approach including 42 interviews and 34 completed surveys. Transcribed interview recordings were coded in NVivo 11, and codes were queried in NVivo and Excel to identify key themes. Survey responses were analyzed in SPSS. Data were examined between and across four attendance groups and triangulated to assess the reliability of the data and validity of overall findings.

Findings:

Key factors increasing registrant engagement included relevant and practical curriculum content; strong and supportive relationships among learners, ECHO faculty, and workplace colleagues; and innovative learning approaches that included opportunities for active, virtual participation through technology, participant management activities, and ECHO’s unique curriculum design.

Conclusion:

Findings from this study validated many of the important elements of ECHO Colorado that make it unique from other iterations of the model being implemented nationally and internationally and identified participant-driven strategies for further amplifying its impact.

Introduction

Project Extension for Community Health Outcomes (ECHO) was originally developed by the University of New Mexico’s Health Science Center (UNMHSC) to build the capacities of primary-care providers (PCPs).^1-5 Using telehealth technology, the UNMHSC connects health-care providers across the state, especially in rural and underserved areas, to UNMHSC-based specialists to support the delivery of evidence-based care for patients with long-term health conditions. This model of knowledge sharing has been shown to improve access to specialty care and patient outcomes in many community settings across New Mexico, demonstrating improvement in all of the “quadruple aim”⁶ goals of improving access to and the quality and value of health care, while also improving providers’ knowledge and sense of satisfaction in delivering care.²

With knowledge of the New Mexico model and its dissemination to numerous medical centers across the country, leaders from the academic center at the University of Colorado Anschutz Medical Campus (CUAMC)—including faculty from the School of Medicine, School of Dentistry, School of Public Health, School of Nursing, and School of Pharmacy—began the community-based research across the state of Colorado that would help develop the vision for ECHO Colorado. Many stakeholders across the state commented that the lack of access to specialty care in Colorado was a major problem. They emphasized that the excessive number of patients with medical issues that could be managed in primary-care being unnecessarily referred to the CUAMC was preventing primary-care and specialty care from practicing at the top of their professional scope.

ECHO Colorado, a replication site of the ECHO model, was developed with the aim to design a statewide ECHO system that would engage specialists, generalists, and many different organizations across Colorado in a learning collaborative in which specialty care knowledge could be imparted, skills for care of complex/long-term health conditions could be acquired, and the health resources in Colorado could be coordinated to maximize collective impact. The program launched in January 2015 and has since provided numerous learning series, covering topics included in Colorado’s 10 Winnable Battles⁷ such as tobacco, infectious disease prevention, safe food, and mental health and substance abuse. Other areas of focus have included quality improvement, disease prevention and early identification, integrated and complex care, and cancer-related programs. ECHO Colorado’s approach includes the following:

• A program-center approach offering series development services structured to partner with community-based organizations around the state to address their training and support needs;

• Formal collaborations with multiple organizations representing interests in public, community, rural, and urban health;

• A statewide advisory board representing the breadth and diversity of Colorado;

• The use of robust evaluation methods to better identify the potential impacts of the ECHO model;

• Development of a diversified and blended funding model that engages multiple partners for sustainability;

• Establishment and support of peer networks that extend beyond the individual ECHO series and provide potential for a force multiplication of the effort itself;

• Time-bound learning series with designated start and end dates.

In 2016, with the approval of the Expanding Capacity for Health Outcomes Act (ECHO Act) by the US Congress, Project ECHO grew dramatically and was disseminated nationally and internationally with creative adaptations for a wide range of uses.⁸ One emphasis of the ECHO Act was to support expanded evaluations of collaborative continuing education models involving telementoring and case-based learning to disseminate and build capacities for implementing evidence-based practices. Various studies have demonstrated Project ECHO’s effectiveness in achieving treatment outcomes comparable to those obtained through specialist visits and in improving participants’ self-efficacy, knowledge, and behaviors as well as cost-effectiveness.^1,3,9-12 While the ECHO model has been established as a successful way to support PCPs to use best and promising practices to improve health outcomes across numerous health conditions and specialties, most ECHO sites do not retain all of their participants throughout the course of a learning series, as these series require a significant commitment of time and thus commitment from participating organizations to support consistent attendance of their providers. In addition, given the many fixed costs associated with implementing an intervention like ECHO, identifying strategies for increasing participant engagement and retention is critical to ensuring the program achieves its optimal impact.¹³

The purpose of this study was to understand what influenced participant engagement; how participants valued the ECHO experience; and what the utility of the ECHO Colorado experience was for participants. ECHO Colorado uses the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework^14,15 to evaluate its efforts and designed this study to assess the efficacy of ECHO learning series or the Effectiveness of the RE-AIM framework.

Methods

This study used a mixed-methods approach including participant interviews and surveys to examine the facilitators and barriers to registrant retention. Administering qualitative and quantitative approaches minimized the limitations compared to a single method design, allowed for the triangulation of data, and thus increased the validity of the findings.¹⁶ This project was reviewed by the Colorado Multiple Institutional Review Board and was determined as not human subject research.

Participants and data collection

During ECHO Colorado’s first year of implementation, 580 registrants were recruited across 23 learning series. For the interviews, these registrants were stratified by attendance (Table 1) to explore any inherent differences that may have existed between low and high attendees. Quota sampling was used for each attendance group with the aim to complete 8 to 12 interviews for each group. Sampled registrants were first sent an email inviting them to participate in an interview with the opportunity to receive a US$25 gift card as an incentive. Those who did not reply to the email were then contacted as many as two more times via phone. Semi-structured interviews were conducted using two interview guides: a basic guide for those who registered but did not attend any sessions (non-attenders) and an expanded guide for those who attended at least one session (low, medium, and high attenders). Ultimately, 42 interviews were conducted between October and December 2016; interviews using the basic guide ranged from 5 to 10 minutes in length and interviews using the expanded guide ranged from 20 minutes to an hour and a half in length. All of the recruitment and interviews (including recordings) were conducted virtually through telehealth technology by one member of the research team (M.G.).

Table 1.

Stratified samples of interview participants.

	Non-attenders (0% sessions)	Low attenders (1%-49% of sessions)	Medium attenders (50%-79% of sessions)	High attenders (⩾80% of sessions)	Total
Registrants	137 (23.6%)	238 (41.0%)	132 (22.8%)	73 (12.6%)	580
Sampled	22	21	22	14	79
Interviewed	8 (36.4%)	10 (47.6%)	12 (54.5%)	12 (85.7%)	42 (53.2%)

A supplemental survey was developed and administered to the 34 interview participants who participated in an expanded interview. Survey questions primarily assessed two aspects of ECHO participation. The first was participants’ satisfaction with and impact of participating in an ECHO learning series. The second was the extent to which specific program elements such as the offering of continuing education, agency support, access to resources, and training facilitator contributed to the level in which participants engaged. These questions were strongly aligned with the interview guide to assess the reliability of participant responses. An additional question related to preferred social media platforms was added to further explore participants’ interests in continued connections with other participants, a theme that arose from the interviews. The survey was administered and completed by all 34 participants using Research Electronic Data Capture (REDCap)—a secure, web-based application designed to support data capture for research studies.¹⁷ Respondents received a US$25 gift card for their participation.

Data analysis

Once completed, interviews were professionally transcribed and researchers coded and analyzed the data in NVivo 11 using an iterative, data-drive approach to code development.¹⁸ Two research team member (M.G. and E.B.) met regularly to assess discrepancies, and additional codes were permitted to emerge with each meeting and the renaming of codes and expansion of the code book continued until coding consistency was obtained. Ultimately, 21 interviews were double coded. The remaining transcripts were single coded by the researcher who conducted the interviews. Codes were queried in NVivo and Microsoft Excel (2016) to identify key themes among each attendance group and for the overall study. Survey responses were analyzed in SPSS.¹⁹ Comparisons across attendance groups were examined within and across interview findings and survey findings to assess the reliability of the data and validity of overall findings. These findings were then used to develop program recommendations, which underwent expert validation by internal program staff.

Results

Interview participants represented 15 of the 23 series implemented in ECHO Colorado’s first year. Table 2 shows the demographics of those who participated in the study, which were generally reflective of the larger registrant pool for all ECHO Colorado series at this time. County designations for Colorado participants were determined based on the US Census Bureau’s²⁰ identification of Urbanized Areas as those with 50,000 or more people and Rural as those not including urban areas. The Colorado Rural Health Center²¹ further classifies Frontier counties as those with a population density of six or less people per square mile.

Table 2.

Participant demographics (N = 42).

Profile category	N	Percent
Year of participation in ECHO
2015	6	14.3
2016	34	81.0
2017	2	4.9
State
Colorado	36*	85.7
Outside of Colorado	6	14.3
County designation in Colorado*
Frontier	3	8.3
Rural	7	19.4
Urban	26	72.2
Profession
Administrative staff	5	11.9
Behavioral health provider	2	4.8
Clinical staff or health care provider	16	38.1
Doctor of Pharmacy	3	7.1
Health outreach, coordination, and/or education	8	19.0
Law enforcement	0	0.0
Practice management	2	4.8
Public and environmental health	5	11.9
Other profession	1	2.4
Sex
Female	28	69.0
Male	11	28.6
Prefer not to respond	1	2.4

Participants shared their perspectives on key factors that facilitated their movement from registration to utilization in what has been developed as the Stages of Participant Engagement (Figure 1). The key factors described in the model are listed in descending order based on the extent to which they were discussed in the interviews. Although every item was not required for participants to move along the spectrum, the presence of more factors was associated with greater engagement and utilization. Overall, themes shared by participants regarding their general experience (Table 3) as well as their recommendations for improving engagement (Table 4) could be grouped into three overarching categories—curriculum relevance, relationship, and format—that were fairly consistent across all attendance groups.

Figure 1.

Facilitators in the Stages of Participant Engagement.

Table 3.

Supporting quotes from key themes.

Curriculum relevance
Low attender: “I remember being excited about signing up for a class . . . it was something in the marketing that matched information I needed in my new job or information I thought would be helpful in my new job.” High attender: “It was a good opportunity to network with others and to learn what other people are doing with things and what sort of information they’re looking for.” High attender: “I think that the organizers did a great job in recruiting presenters whose areas, topic areas, really were applicable . . . they gave very practical, easy to apply suggestions and the discussion topics were relevant to what I do.”
Relationship
Medium attender: “Definitely the support of my supervisor made it very easy for me to attend.” High attender: “One it was useful hearing that other people were having the same challenges that we have here at my health department. It’s not like misery loves company but it’s kind of reassuring to know that okay this is a pretty typical challenge and it’s pretty normal.” High attender: “I kind of knew [the facilitator] and we talked about some other interests that we had outside of work which was kind of fun . . . I felt comfortable because I knew her and that really helped. If I hadn’t known her, it would’ve been probably easier for me to sit on the sidelines and not say much.”
Format
Medium attender: “The fact that I was able to use Zoom, the app was in my phone. I work outside of the hospital a lot, so the fact that everything is in my phone, it just made a big difference.” High attender: “I live in the southeast part of the state, so we’re three hours from Denver. So the ability to do that in an online format through—I believe they were using Zoom as well or some type of technology like that—made it much easier, obviously, than if I was trying to attend and participate in person. There would’ve been no way that I would’ve been able to do that because of the travel requirement.” High attender: “I think the ECHO—it enables you to be a little bit more of an active learner because you can—it’s smaller, you can ask questions and it’s a little less intimidating, I think, when you’re on the phone and just kind of in your office to speak up.” High attender: “I think the practical case histories of how people are utilizing a certain service or how they’re dealing with a certain problem is just really helpful, and to see how people used a program and then, you know, the barriers and how they made it successful. I think experience based learning is helpful.”

Table 4.

Participant recommendations for further facilitating engagement.

Curriculum relevance	Increase program staff’s understanding of participants’ job functions and their level of knowledge or skill in the topic area Integrate applicable and practical content into the didactic presentations Elevate the content by going deeper into topics Ask participants pointed questions about challenges encountered in their work Create opportunities for people to ask questions and connect directly with peers and experts to learn about shared challenges and promising solutions Provide participants with relevant and useful feedback around cases, questions, and comments shared in the discussions Share resources and tools that can help participants apply new knowledge directly to their work
Relationships	Create more space in sessions for participants to build relationships Create opportunities for participants to discuss challenges they are experiencing in the workplace and collectively generate ideas for how to address them Validate participants’ experiences, especially when there are shared struggles Help build workplace support so people have the backing to participate in ECHO series, make recommended practice changes, and motivate others in the field to participate in ECHO series Share basic contact information and a short biography and/or photo of other participants in a more accessible way Intentionally facilitate connections via email or an online forum between sessions and at the end of a series Create an online discussion forum (not through social media platforms) to aid the development of professional networks
Format	Limit participants to join session alone or with only one other person (i.e., no more than two people per screen) to maintain the benefits of having face-to-face interactions Increase the amount of time for discussions and have more skilled facilitators to encourage and succinctly guide conversations Clearly communicate expectations of participation

Curriculum relevance

Curriculum relevance was consistently described as content that increased participants’ knowledge around a topic area or best and promising practices from the field and created opportunities for participants to gain new skills. This was the predominant theme that emerged from the interviews with all attendance groups and was valued regardless of whether or not the content topic was directly relevant to participants’ current role or responsibilities.

Participants extensively discussed how curriculum relevance improved their engagement across one or more of the Stages of Participant Engagement. In addition to being the primary reason for registering for the learning series, curriculum relevance was also described as increasing participant attendance and engagement, especially when facilitators and presenters demonstrated the applicability of the content to participants’ work contexts. Conversely, all participants with the exception of low attenders reported that finding the curriculum to be irrelevant to their work was the primary reason for not attending sessions and ultimately dropping out of the series.

Low, medium, and high attenders described curriculum relevance as a critical component to being able to apply the new knowledge and skills they gained through the ECHO learning series to their work. Ways in which participants applied their new learnings included modifying their organizational operations (e.g., developing new materials for patients, creating more efficient processes, and developing or joining relevant committees) and sharing new insights with workplace colleagues. Compared to other attendance groups, higher attenders discussed how they had applied their newly gained knowledge or skills more often and more frequently and indicated in the survey that participation in the ECHO series made them better at their jobs. Conversely, participants who found the curriculum to be irrelevant were less able to apply what they had learned to their work, especially when the curriculum lacked practicality or differed from what they understood to be current best practices in the field. Therefore, the extent to which the curriculum of a series was relevant to participants greatly impacted whether or not they moved through the desired Stages of Participant Engagement from registration and attendance to engagement and utility.

Relationships

Three dimensions of relationships were discussed by participants and emerged as another overarching theme: relationships among participants (existing and new), relationships between participants and ECHO faculty, and relationships between participants and their workplace colleagues and managers. A few participants in the medium- and high-attending groups identified their desire to grow individual professional networks as the primary reason for registering. Furthermore, many of the low, medium, and high attenders perceived the social support they gained to one of the most valuable aspects of their ECHO experience and a strong influencer on their decision to engage in sessions and ultimately apply what they learned.

Relationships in the workplace were emphasized as a facilitator to participants being able to attend sessions and apply what they learned. Expectedly, those who attended more sessions than those who attended fewer sessions more frequently reported having such workplace support, and several participants in the low- and medium-attending groups described the lack of workplace support as a barrier to applying their newly gained knowledge and skills. Results from the supplemental survey aligned with these findings; most medium and high attenders perceived their organizations to be supportive of their participation, while many low attenders felt their agency was unsupportive.

Finally, existing relationships among participants and between participants and ECHO faculty influenced registration, attendance, and engagement. More participants in the high-attending group, compared to participants in the low- and medium-attending groups, knew other participants or ECHO faculty before registering. Some participants in low-, medium-, and high-attending groups also mentioned that knowing others made them feel more comfortable in participating in discussions during sessions.

Format

Format was extensively discussed by most participants. Participants described format as curriculum design, opportunities for active participation, innovative approaches to learning, Zoom²² as an easy and secure video communications platform, and participant management activities. These aspects of how individual ECHO series were delivered played a key role in whether or not participants attended, engaged in, and used the ECHO sessions.

Many participants frequently mentioned that Zoom aided in their ability to attend sessions due to the ability to join from their location of choice and its ease of use. Some participants in the medium- and high-attending groups highlighted Zoom as one of the most valuable aspects of the experience because it allowed them to interact face-to-face, which ensured a higher level of accountability and made it easier to engage with others in sessions. Furthermore, the face-to-face interactions, especially during discussions, was described as a facilitating factor in getting relevant feedback on their specific questions and thus making the content more relevant to their work and ultimately leading to greater utility. However, some low-attending participants reported that Zoom posed a challenge to attending because of various technical difficulties and the time it took them to become familiar with the new technology.

Scheduling was another aspect emphasized by participants as greatly impacting their ability to attend sessions. Scheduling challenges due to competing priorities negatively influenced attendance, and most of those in non-attending and low-attending groups identified competing priorities as the primary reason for not attending.

Most across low-, medium-, and high-attending groups felt the curriculum design was one of the most valuable aspects of their experience and was key in keeping them engaged in each session. The strategies related to curriculum design that participants found particularly helpful included short and focused presentations, opportunities to ask questions and hear responses directly from experts, and the ability to interact with peers in real-time.

Other aspects of format described by participants as impacting their engagement included the number of participants in each session, the number of participants per screen, and participant management activities (behind the scenes coordination efforts including but not limited to: communication with new registrants, ensuring participants have the proper equipment to participate virtually, and general support between sessions). Having too many or too few participants in a series influenced participation in a negative way, in particular the ability to engage with one another and comfortably participate in active discussions. In addition, participants felt one to two people per learning site on the screen was ideal for keeping the benefits of having face-to-face interactions and having three or more people per screen disrupted their ability to engage (Figure 2).

Figure 2.

Screen considerations in participant engagement through virtual platforms.

Supplemental survey findings

The survey results aligned with what participants shared in their interviews. Overall, most participants who had higher attendance felt their participation in an ECHO series was worth their time and that participation made them better at their job. Some who attended less frequently felt neutral or disagreed with these statements. Most medium and high attenders felt their agency was supportive of their participation but about half of the low attenders felt their agency was not supportive.

When asked about ways to support the building of professional networks among participants, most of the low, medium, and high attenders reported that they would not use any social media platforms to connect with other participants. However, they believed participating in an online discussion forum would aid in the development of their professional networks.

Discussion

Growing evidence demonstrating Project ECHO’s effectiveness and the recent introduction of the ECHO Act have contributed to the recent growth of ECHO,^{1–5,8–13,23} but limited studies have been conducted on what keeps health and public health professionals engaged throughout the course of an ECHO series despite the time commitment and known competing priorities. ECHO Colorado uses the RE-AIM Framework^14,15 to provide robust assessments of several program dimensions and continuously strives to identify areas for improvements as well as those that are working optimally and should be expanded or replicated in future initiatives. Results of this study validated existing strengths of ECHO Colorado’s approach to engaging participants and identified new opportunities for program improvements to increase participant engagement.

After this study revealed the efficacy of the intervention, known as Effectiveness in the RE-AIM Framework, ECHO Colorado formed three workgroups (evaluation, marketing, and development and implementation) to prioritize, strategize, and implement program-improvement recommendations generated from the findings to enhance the Adoption and Implementation of such strategies.^14,15 Many of these recommendations have already been integrated into ECHO Colorado’s procedures, and increased engagement has been demonstrated through improved attendance rates as well as increased response rates to evaluation activities. Such changes include the following:

• Emphasizing the value of ECHO in marketing materials, including the convenience of Zoom, relevance and utility of the curriculum, and growing professional networks with peers and experts;

• Modifying marketing materials to only include images with one to two participants per screen rather than those where larger groups were participating together;

• Building support and buy-in from statewide leaders and organizational partners as to the uniqueness of the approach and value compared to traditional remote learning models;

• Leveraging past participants in the recruitment of new registrants, including the integration of quotes from evaluation surveys into marketing materials;

• Narrowing and clearly defining the intended audience of each series and developing questions in the registration process to assess the fit of each registrant;

• Guiding curriculum development partners in incorporating practical and relevant elements into series curriculum and engaging participants in discussions that lead to clear connections to participants’ work;

• Engaging participants from the time they register to the end of the series through calendar reminders, emailed communications, discussion forums, and so on;

• Identifying points within and between sessions to encourage networking and the building of relationships among participants, including but not limited to the distribution of contact lists;

• Continuing to partner with statewide organizations in conducting and utilizing needs assessment data to ensure curriculum relevance.

Recommendations for future efforts to continue increasing participant engagement include the following:

• Determining how best to select and train facilitators with the goal of encouraging and succinctly guiding participation in sessions;

• Continuing to review and elevate current strategies in explicitly sharing participation expectations;

• Assessing the ideal number of participants in virtual trainings.

The University of New Mexico and the ECHO Institute have long conveyed their interest in having replication sites demonstrate fidelity to the original ECHO model to create opportunities for across-program evaluation. However, critically examining the model and assessing ongoing quality improvement efforts to identify strategies for increasing participant engagement and implementing such approaches to further motivate participants to stay engaged and use what they learn is critical to increasing the overall impact of ECHO. The findings of this study highlight key elements for ensuring ongoing engagement from participants in ECHO. While these results cannot be generalized to other ECHO programs and further investigation is required to understand the threshold of how many facilitating factors are required for continued engagement, other ECHO sites may also benefit greatly by maximizing facilitating factors to the extent possible to ensure an engaging and useful experience for learners. Everyone is increasingly busy with competing priorities; in any given day, ECHO must compete with these other activities and provide a comparative value. In particular, what this study has revealed as valuable to participants include the following: relevant and practical curriculum content; strong and supportive relationships among learners, ECHO faculty, and workplace colleagues; and innovative learning approaches that included opportunities for active, virtual participation through technology, participant management activities, and ECHO’s unique curriculum design. These factors were demonstrated to motivate participants to stay engaged longer and perceive greater utility of the series.

In addition to identifying many promising practices for magnifying the impacts of ECHO, this study has also raised numerous questions for further study. Future studies should more thoroughly assess how and why the facilitators identified in the Stages of Participant Engagement impact a learner’s experience with ECHO to understand any additional elements that may be noteworthy for continued program improvement. This may be particularly important as ECHO continues to scale and expand into various fields beyond health care and education, where workforce needs and motivating factors could vary. Furthermore, this framework may also have utility beyond the ECHO model to strengthen other virtual continuing education training models and should be explored as an aspect of value-based health-care delivery to potentially increase access to highly specialized health care, especially for those in rural and underserved areas.

Limitations

The methodologies used allowed researchers to gain a contextualized understanding of facilitators and barriers to engagement, but a potential limitation is how representative the sample was of the population. Sampling was random and saturation was achieved with the interviews but was a selected sample. Furthermore, the results draw upon a limited number of registrants per series and exclusively from ECHO Colorado, which may limit the generalizability of the findings. However, through precautions such as having one interviewer, double coding transcripts until coding consistency was achieved, and using mixed methods to validate qualitative findings with survey results, other limitations common with qualitative approaches were likely minimized.

Conclusion

Learners who participated in ECHO Colorado found value in its services and viewed it as a viable way to improve capacity among health workers throughout Colorado. In particular, curriculum relevance, format, and the opportunity to build new and existing relationships were identified as integral aspects of the program that had the greatest impacts on participant engagement as well as participants’ perceptions of ECHO’s value and utility. These findings confirm the importance of the many elements that make ECHO Colorado unique from other iterations of the ECHO model and opportunities for other ECHO programs to expand their impact as well. As mentioned earlier, ECHO Colorado has already started further amplifying its efforts by Adopting and Implementing the recommendations identified through this study to increase Effectiveness and is continuing to engage participants and program partners in assessing its impacts and opportunities for ongoing growth to ensure long-term success in establishing collaborative learning communities to improve health for all.