Project Extension for Community Healthcare Outcomes Intervention Evaluations: A Scoping Review of Research Methods

Abstract

Introduction:

Since its inception in 2003, the Project Extension for Community Healthcare Outcomes (ECHO) tele-education model has reached and improved outcomes for patients, providers, and health centers through interventions in >180 countries. Utilization of this model has recently increased due to the COVID-19 pandemic and a higher demand for remote education. However, limited research has examined the methodologies used to evaluate Project ECHO interventions.

Methods:

We conducted a scoping review to determine the extent and types of research methods used to evaluate outcomes and implementation success of Project ECHO interventions and to identify gaps and opportunities for future investigation. Using Arksey and O’Malley’s scoping review framework and the PRISMA-ScR checklist, we reviewed study designs, temporality, analysis methods, data sources, and levels and types of data in 121 articles evaluating Project ECHO interventions.

Results:

Most interventions addressed substance use disorders (24.8%, n = 30), infectious diseases (24%, n = 29), psychiatric and behavioral health conditions (21.5%, n = 26), and chronic diseases (19%, n = 23). The most frequently reported evaluation methods included cohort studies (86.8%, n = 105), longitudinal designs (74.4%, n = 90), mixed methods analysis (52.1%, n = 63), surveys (61.2%, n = 74), process evaluation measures (98.3%, n = 119), and provider-level outcome measures (84.3%, n = 102). Few evaluations used experimental designs (1.7%, n = 2), randomization (5.8%, n = 7), or comparison groups (14%, n = 17), indicating limited rigor.

Discussion:

This scoping review demonstrates the need for more rigorous evaluation methods to test the effectiveness of the Project ECHO model at improving outcomes and standardized reporting guidelines to enhance the dissemination of evaluation data from future Project ECHO interventions.

Project ECHO

Telehealth, which involves the use of electronic information and telecommunication technologies to support long-distance health care and health-related education, has transformed the medical field over the past decade.¹ Telehealth provides virtual learning opportunities, or tele-education, to health care providers working in rural and other underserved areas.^2,3 Through tele-education, health care providers are able to obtain continuing medical education (CME) and reduce professional isolation by connecting with other providers remotely.³ Tele-education has recently increased in demand due to technological advancements and the need for virtual learning throughout the 2019 coronavirus (COVID-19) pandemic.^2,4,5

One of the most widely adopted tele-education interventions is Project Extension for Community Healthcare Outcomes (ECHO), which was founded and launched in 2003 by Dr. Sanjeev Arora at the University of New Mexico.⁶ Project ECHO uses digital technologies to equip primary care providers with information, resources, and tools to care for patients with chronic conditions who are unable to access specialty care providers.⁶ There are four key mechanisms through which Project ECHO works: (1) traditional tele-education, (2) a disease management framework based on clinical best practices, (3) patient case-based learning, and (4) an outcome monitoring database compliant with the Health Insurance Portability and Accountability Act (HIPAA).⁷ Project ECHO employs a “hub-and-spoke” model through which a core hub team of clinical experts in a particular disease or specialty area, such as endocrinologists, use tele-education to train nonspecialist providers from spoke sites, including primary care practices, community health centers, and federally qualified health centers.⁸

Hub teams and spoke site providers meet regularly in real time for teleECHO videoconferences, which consist of two components: (1) hub team–led didactic presentations focused on specific aspects of care, such as new treatments and screening practices, and (2) discussions about deidentified patient cases submitted by the spoke site providers.⁸ In addition to the teleECHO clinics, spoke sites receive other tele-education resources, including Internet-based assessments, recorded presentations, and ongoing support from the hub team via telephone, fax, email, and other platforms.⁶ The real-time videoconferencing and communication tools provided through Project ECHO offer value beyond other forms of tele-education such as asynchronous learning modules.³ However, the lack of in-person learning and communication is still a limitation of the Project ECHO model.

Initially, Project ECHO was designed to reduce disparities in health care access and outcomes for geographically underserved patients with hepatitis C who were unable to attend in-person visits with infectious disease specialists.⁶ The first Project ECHO intervention was highly effective, as evidenced by evaluation research demonstrating that it increased primary care providers’ knowledge, self-efficacy, and practices pertaining to hepatitis C care as well as improved outcomes for patients with the condition.^9,10 The Project ECHO model has since been applied in interventions addressing a vast array of conditions including but not limited to diabetes, opioid addiction, autism, arthritis, and COVID-19.^11–15 Furthermore, the Project ECHO model has been utilized in more than 180 countries across North America, South America, Europe, Asia, Africa, and Australia.¹⁶ Project ECHO interventions have also been implemented in a multitude of settings, including academic health centers, palliative care and nursing home facilities, the US Veteran’s Health Administration (VHA), and public schools.^7,17–19

Project ECHO Scoping and Systematic Reviews

Given its increased utilization and the growing number of published studies on Project ECHO, there is a crucial need to determine the scope and types of Project ECHO interventions being implemented and evaluated. Scoping review studies, which “map rapidly the key concepts underpinning a research area and the main sources and types of evidence available,”²⁰ are used to determine the number of publications addressing a certain topic, examine the publications’ methods and recurring findings, and inform best practices.²¹ Scoping reviews also determine gaps in existing research and provide a foundation for subsequent systematic reviews, which involve quality assessment of studies.²² Since 2016, four systematic reviews of Project ECHO interventions have been published^23–26: two reviews examined research methods and outcomes from interventions focused on opioid addiction, including 15 and seven studies, respectively^25,26; one examined patient-level and provider-level outcomes described in 39 studies²³; and the other review examined research methods and outcomes described in 52 studies.²⁴ Over the past several years, Project ECHO interventions have scaled dramatically; as such, a more recent and comprehensive review of Project ECHO interventions is needed.

Objectives

Given the expanded utilization of the Project ECHO model, we conducted a scoping review of recently published literature on Project ECHO interventions. Our primary research aim was to identify the types of research methods used in Project ECHO intervention outcome and process evaluations.²⁷ Specifically, we aimed to understand how the effectiveness and efficacy of these interventions have been measured, respectively. Our secondary research aim was to identify the levels and types of outcomes measured in evaluations of Project ECHO interventions. Example levels include the patient level, provider level, and health center level; example outcomes include patient knowledge, provider prescribing habits, and health center expenditures; and example process indicators include the number of program participants and effectiveness of video-conferencing technology.

METHODS

We conducted a scoping review to examine how Project ECHO interventions are evaluated for their implementation success and outcomes for patients, health care providers, and health centers. We followed Arksey and O’Malley’s framework for scoping reviews,²² which includes five steps: (1) identifying a clear research question, (2) identifying relevant studies, (3) selecting studies for inclusion, (4) creating a data table to summarize each included study, and (5) discussing overall findings. We also followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses—Extension for Scoping Reviews (PRISMA-ScR) Checklist²⁸ to ensure comprehensive reporting of our scoping review data. Since we conducted a scoping review, we did not complete bias risk assessments; however, this would be valuable to do in a subsequent systematic review.

Search Strategy

To identify articles for our scoping review, we searched PubMed, Google Scholar, ProQuest Central, and six EBSCO-Host databases: Academic Search Premier, APA PsycInfo, CINAHL, ERIC, Health Source: Nursing/Academic Edition, and Psychology and Behavioral Sciences Collection. All searches were completed on July 3, 2021. As such, this scoping review only includes articles published by that date. The following search terms were used: (“Project ECHO” or “Extension for Community Healthcare Outcomes” or “ECHO model”) AND (“evaluation” or “study” or “method” or “design” or “trial” or “outcome” or “impact”). In the databases, we limited our searches to article titles and abstracts to ensure relevance of the search results. Specifically, ensuring that the words “Project ECHO” or “Extension for Community Healthcare Outcomes” were part of an article’s title and/or abstract assured that the article focused on an intervention utilizing this model. We also used database search limiters, such as articles published in peer-reviewed journals, in English language, and with available full texts (see Appendix, Supplemental Digital Content A, http://links.lww.com/JCEHP/A315). We also hand-searched reference lists of prior systematic reviews focused on Project ECHO interventions to ensure that no relevant articles included in these earlier reviews were missed in our database searches. We also hand-searched Telemedicine and e-Health and Journal of Telemedicine and Telecare, two peer-reviewed journals focused on digital health interventions that published several of the articles identified in our database searches.

Inclusion and Exclusion Criteria

Our inclusion criteria were as follows: peer-reviewed journal articles that were categorized as original research and discussed process and/or outcome evaluation methods used in Project ECHO interventions. We included articles employing all process and outcome evaluation types, including but not limited to observational, experimental, and qualitative methods. Articles were excluded if they discussed Project ECHO evaluation data without describing the evaluation methods used or if they only described future evaluation methods that were not completed yet. For studies that discussed both completed and future evaluation methods, we only reviewed the methods that were completed. This was due to the possibility that any future methods could be modified or not implemented at all, and we felt it pertinent to only review methods that were successfully conducted. We also did not focus on prepilot formative methods used to design Project ECHO interventions. For studies that evaluated Project ECHO interventions and other initiatives, we only reviewed the evaluation methods used in the Project ECHO interventions. Articles that evaluated interventions based on the Project ECHO model but did not utilize the exact model such as adaptations were excluded to ensure our review focused on those with high fidelity. Dissertations, theses, and book chapters were also excluded.

Screening

We downloaded and tracked articles in EndNote (Clarivate), a reference management software. The lead author initially screened all downloaded articles’ titles and abstracts to exclude those that clearly did not align with the inclusion criteria; for example, articles that used the acronym “ECHO” for a topic other than “Extension for Community Healthcare Outcomes.” The lead author along with two additional authors each screened a 15% subset of the remaining articles’ titles and abstracts to reduce bias in the article selection process. The three authors each reviewed the same 15% subset of titles and abstracts. A random number generator was used to select articles for the subset. Interrater reliability (IRR) using Fleiss kappa was calculated in R (RStudio Team) to determine the percent agreement between the three authors’ inclusion and exclusion decisions for the 15% subset. We utilized Landis and Koch²⁹‘s methods for interpreting IRR. After the subset screenings, attaining an adequate IRR, and reviewing all abstracts, the lead author screened the remaining articles’ titles/abstracts that were not included in the 15% subset. The same process was later followed for full-text screenings; the same three authors each reviewed a randomly selected 15% subset of the articles’ full texts, and IRR using Fleiss kappa was calculated to determine percent agreement in their inclusion and exclusion decisions. Once an adequate IRR was attained, the lead author screened the remaining articles’ full texts to determine the final list of included articles.

Data Extraction and Analysis

We employed a multistep, rigorous process for tracking and analyzing evaluation methods described in each included study. Nine investigators with experience managing Project ECHO interventions or expertise in public health research methods reviewed subsets of the included articles and extracted information. Articles were reviewed at least two times, with some reviewed a third time to resolve any uncertainties regarding evaluation methods. We used Research Electronic Data Capture (REDCap), an online data management platform, to extract relevant information from each article. Based on input from Project ECHO program leadership, we created a REDCap data collection form that included an initial list of evaluation methods to track from each article, including temporality, research designs, types of analysis, levels of outcomes measured (patient, provider, or health center), types of outcomes measured, and process measures. We also used the REDCap form to extract general information about the studies, including their authors, years of publication, locations of Project ECHO interventions, intervention names if specified, and intervention focus areas. The REDCap form included multiple-choice and free-text items to enable investigators to categorize evaluation methods, explain methods that did not fit into the predetermined categories, and note any methods that were not clearly stated in the articles. We later revised the REDCap form with updated multiple-choice options and recoded responses based on recurring methods described in the free-text items. For articles in which it was difficult to discern any evaluation method, for example, temporality, we categorized that method as “other” or “not specified.”

In REDCap, we computed descriptive statistics, including frequencies, means, medians, and ranges for the following items: Project ECHO intervention locations, focus areas, study temporality, study designs, use of randomization, use of comparison groups, data sources, types of data analysis, levels and types of outcome measures, and types of process measures.

RESULTS

Article Selection

Our database and hand searches generated 483 articles (database searches: n = 450; hand searches: n = 33). Of the 450 articles downloaded from the databases, 201 were duplicates. We screened the titles and abstracts of the remaining 249 articles downloaded from the databases. Our 15% title/abstract subset screening IRR (k = 0.852) indicated strong agreement between the three authors’ inclusion and exclusion decisions. During the title/abstract screening stage, 114 articles were excluded. In addition, nine were abstract-only and were excluded. The remaining 126 articles downloaded from the databases and 33 articles identified through hand searches proceeded to the full-text screening stage. Our 15% full-text subset screening IRR (k = 0.871) also demonstrated strong agreement between the three authors’ inclusion and exclusion decisions. During the full-text screening stage, 38 articles were excluded. Ultimately, 121 articles were included in our scoping review. Our article selection process is illustrated in Figure 1.

FIGURE 1.

PRISMA diagram—article selection process. Articles were included if they met the following criteria: published in peer-reviewed journals, categorized as original research, discussed process and/or outcome evaluation methods completed in Project ECHO interventions, available in English language, and had the full text available.

Project ECHO Interventions

The 121 articles were published between 2010 and 2021, with a median year of 2019, indicating substantial uptake, implementation, and assessment of interventions using the Project ECHO model over the past decade. Summary data about the studies’ locations and focus areas, evaluation methods, and process and outcome evaluation measures are included in Tables 1–3, respectively.

Table 1.

Project ECHO Intervention Locations and Focus Areas

	n	%	Reference Nos
Intervention location(s)
United States	84	69.4	9–11,13,14,18,19,30–50,51–70,71–99,100–106
Canada	14	11.6	12,107–119
International	6	5	15,60,120–123
India	5	4.1	124–128
Argentina	4	3.3	129–132
Ireland	3	2.5	99,133,134
England	2	1.7	17,135
Australia	1	0.8	136
Kenya	1	0.8	137
Namibia	1	0.8	138
General focus area
Substance use disorders	30	24.8	9,12,35,43,45,48,58,59,62,65,68,69,85–88,91,106,108–110,115,117,119,121,126–128,136,139
Infectious diseases	29	24	9,10,15,30,32,39,40,60,63,64,76,77,79–81,83,84,87,97,104,105,114,116,130–133,136,138
Psychiatric and behavioral health conditions	26	21.5	9,36,42,45,50,52–54,59,65,68,82,86,87,95,98,100,115–118,121,124,126,127,139
Chronic diseases	23	19	11,14,31,32,46,51,55–57,59,66,74,84,89,90,92–94,102,103,121,122,129
Veteran’s health	18	15	18,35,36,39,40,51,59,61,64,67,77,82,84,92,93,100,103,121
Pain management conditions	18	14.9	12,18,30,33,35,36,58,59,87,91,96,107–110,119,134,140
Palliative health	7	5.8	17,99,111,123,125,134,135
Cancers	7	5.8	30,37,75,101,120,123,128
General workforce capacity	7	5.8	19,34,47,49,65,78,137
Autism spectrum disorders	6	5	13,38,44,53,72,73
Neurological diseases	6	5	31,32,56,57,59,134
LGBTQ health	2	1.7	30,61
Women’s health	1	0.8	67

Most included articles evaluated Project ECHO interventions that were implemented in the United States and focused on substance use disorders or infectious diseases.

Table 3.

Project ECHO Intervention Process and Outcome Evaluation Measures

	n	%	Reference Nos
Process evaluation measures
Participation rates and demographics	105	86.8	12–15,17–19,31–41,43–51,53–62,64,66–80, 82, 84–90, 94–102, 104–106,108–124,126–140
Content relevance or effectiveness	81	66.9	9,11–15,17,19,30–37,39,42–45,47–51,53,54,56,57,60–62,64, 66–68, 70–72, 74, 78–80, 82, 84, 86, 88, 90–92, 95, 96, 100,101,104–107,109,110,113–121,123–125,127,133–135,137–140
Number of case presentations	58	47.9	9,12–14,18,19,31–33,37,40–42,45,49,51,52,54,56,61–64,66–68,71,72,74,76–80, 83–85, 87, 94, 96, 99, 100, 102,104,105,109,118,120,121,124,126,127,129,131,133,134,138,140
Motivations, facilitators, or barriers to participation	54	44.6	9,12–14,17,30,31,34–37,39,41,44,50,51,53,54,56,57, 67, 74, 75, 77, 82, 84–88, 90–92,95,98–101,104,107,113,114,119,120,124–126,129, 133–135, 137, 138, 140
Staff time and support	43	35.5	9,14,17,30,34–37,44,47,49,50,53,56,67,74,75,77, 80, 81, 84–88, 90, 92, 95, 101, 106, 107, 113, 119,120,124–126,133–135,137,138,140
Utilization of online resources	40	33.1	13,17,19,30,31,33–37,43,44,46–48,50,53,56,57,61,68,72,74,79,80,84,86,88,90,95,98,100,106,107,113–115,120,125,140
Suggestions for intervention improvement	36	29.8	9,14,17,30,33–35,37,44,45,47,53,56,57,61,67,74,80,86,88,90,95,98–100,107,111,114,119,120,127,133–135,138,139
Hub team or presenter effectiveness	34	28.1	9,12,13,30,34,43,44,47,48,50,54,56,67,72,77,80,82,87,88,90,106,110–115,117,123–125,134,135,140
Videoconferencing or communication technology effectiveness	32	26.4	9,13,14,17,34,36,44,47,53,67,72,74,82,84,85,90,98–100,106,107,113,119,120,124–127,134,137,138,140
Number of CME credits awarded	20	16.5	9,13,14,53–55,59,62,66,68,86,100,104,108–110,121,124,138,140
Outcome evaluation measures
Provider knowledge	84	69.4	9,11,12,14,17,19,32–37,39,41,43–48,50,53–55,57,60–64,66, 67, 69, 72–75, 77, 79, 80, 82, 84, 86, 88–92, 94–102, 104–107,109–115,117,119–124,127,128,131,133–135,138–140
Provider self-efficacy, confidence, or perceived ability	72	59.5	9,11–14,19,31–33,35–37,43–48,50,53,54,56,57,61–63,66,69,72–75, 79, 80, 82, 84, 86, 88, 90, 92, 94, 95, 97, 99, 100, 102, 104–107,109,111–116,119,122–124,126,127,130,131,133–135,137–140
Provider satisfaction with intervention	70	57.9	9,11,13,14,17,19,30–36,43–45,47–50,53,54, 56, 57, 61, 62, 66, 67, 72, 74, 75, 77, 79, 80, 84–86, 88, 90, 92, 95,98–100,106,107,109–116,119–121,123–125,127,131–135,137–140
Provider practices	69	57	11–14,19,30–33,35–39,41,43,45–50,53,54,56,57,61,66,67,69,71–74, 79, 80, 82, 84, 86, 88, 90–92,94,95,97–101,105,107–115,117,119,121,126,129,134,138–140
Provider isolation, community-building, or networking	52	43	9,12–14,17–19,30,31,33,35–37,39,44,46–48,50,53,57,61,62,72,74,79, 80, 84, 86, 88, 90, 91, 98, 99, 104, 107,110–116,119,120,127,131,133–135,138,140
Provider barriers to health care delivery	26	21.5	13–15,34,37,38,43,47,48,66,72–74,77,79,84,85,88,90,99,107,126,129,132,133,140
Provider attitudes regarding health care, patients, or conditions	25	20.7	19,32,33,44,46,48,72,74,75,77,86,88,90,95,97,99,100,105,107,110,116,119,123,130,140
Patient treatment initiation or continuation	23	19	10,18,40,42,50,52,62,65,70,71,76,79,80,83,87,91,93,108,126,129,131,132,136
Provider perceptions of intervention benefits to patients or health centers	21	17.4	9,19,31,35,39,45,48,50,61,66,71,72,84,90,98,100,109,110,133,134,140
Patient health indicators or lab results	19	15.7	10,18,40,42,52,60,70,71,78,79,81,83,91,93,126,129,131,132,136
Provider job satisfaction	13	10.7	9,19,35,36,39,45,66,75,77,88,90,109,138
Provider consulting ability or efforts	10	8.3	9,19,35,79,80,98,104,105,119,131
Health center practices	10	8.3	34,42,52,54,57,58,78,80,92,120
Health center quality improvement metrics	9	7.4	34,52,54,57,58,78,80,92,103
Patient hospitalizations, hospital readmissions, or inpatient visits	7	5.8	18,42,49,50,62,65,126
Patient outpatient visits	6	5	18,50,62,65,126,136
Health center diagnostics	5	4.1	34,52,57,83,103
Patient behaviors	4	3.3	42,80,97,126
Health center expenditures	4	3.3	50,65,78,83

The most frequently reported Project ECHO process measures were program participation rates and content relevance. The most frequently reported outcome measures were provider knowledge and provider self-efficacy.

Most articles evaluated Project ECHO interventions that were based in the United States (69.4%, n = 84). Other countries with multiple Project ECHO intervention evaluations included Canada (11.6%, n = 14), India (4.1%, n = 5), Argentina (3.3%, n = 4), Ireland (2.5%, n = 3), and England (1.7%, n = 2). Five percent of articles (n = 6) evaluated Project ECHO interventions that had international participation.

Regarding general focus area, most Project ECHO interventions in the included articles addressed substance use disorders (24.8%, n = 30), infectious diseases (24%, n = 29), and psychiatric and behavioral health conditions (21.5%, n = 26). The most frequently covered infectious diseases included hepatitis C (14%, n = 17), human immunodeficiency virus ([HIV], 5.8%, n = 7), and COVID-19 (4.1%, n = 5). The most frequently covered chronic diseases included diabetes (4.1%, n = 5), rheumatic and musculoskeletal conditions such as arthritis (1.7%, n = 2), and cardiovascular conditions such as hypertension (1.7%, n = 2). There was overlap across these focus areas in the articles.

Most articles (94.2%, n = 114) evaluated one Project ECHO intervention, while 5.8% of articles (n = 7) evaluated multiple interventions. Nearly three quarters of the articles employed longitudinal evaluations (74.4%, n = 90), and 15.7% (n = 19) used cross-sectional evaluations although the temporality was often not explicitly stated. Almost all articles presented cohort-based outcome evaluations of their Project ECHO interventions (86.8%, n = 105); 12.4% (n = 13) specified retrospective cohort, while 10.5% (n = 11) specified prospective cohort. Other study designs included case studies (4.1%, n = 5), phenomenological studies (2.5%, n = 3), content assessments (1.7%, n = 2), and a model simulation, quasi-experimental design, and randomized controlled trial (RCT) with a partial stepped wedge design (0.8%, n = 1 each). Only 5.8% of articles (n = 7) specified using randomization in any aspect of their evaluation, while the substantial majority of articles (94.2%, n = 114) did not mention using randomization. Similarly, 14% of articles (n = 17) utilized at least one comparison group in their evaluations, whereas 86% (n = 104) did not use comparison groups.

Regarding data collection and analysis, 52.1% of articles (n = 63) reported mixed methods, 34.7% (n = 42) reported only quantitative methods, and 13.2% (n = 16) reported primarily qualitative methods, with some quantitative methods only used for process measures. The four most frequently reported data sources in the articles included surveys (61.2%, n = 74), tele-ECHO clinic participation logs (24%, n = 29), 1-on-1 interviews (19%, n = 23), and tele-ECHO clinic transcripts, case presentations, or consultation questions (19%, n = 23).

Nearly all articles (98.3%, n = 119) discussed Project ECHO intervention process evaluation measures, which emphasized the interventions’ implementation, reach, and opportunities for continued enhancement. TeleECHO attendance and participation rates, as well as demographic information about participating providers, patients, and health centers, were tracked in 86.8% of the articles (n = 105). Three other frequently reported process measures included teleECHO clinic content relevance and effectiveness (66.9%, n = 81), the number of case presentations (47.9%, n = 58), and providers’ motivations, facilitators, and barriers to participation (44.6%, n = 54).

Nearly all articles (98.3%, n = 119) also presented Project ECHO outcome measures. Regarding the level of outcomes, 84.3% of studies (n = 102) examined provider outcomes, 25.6% (n = 31) examined patient outcomes, and 12.4% (n = 15) examined health center outcomes. There was overlap regarding how the studies categorized outcome measures across these three levels. The three most frequently evaluated provider-level outcomes included knowledge (69.4%, n = 84); self-efficacy, confidence, and perceived ability (59.5%%, n = 72); and satisfaction with Project ECHO interventions (57.9%, n = 70). The three most frequently reported patient-level outcomes were medical treatment initiation and continuation (19%, n= 23); health indicators and lab results (15.7%, n= 19); and inpatient care, hospitalizations, and hospital readmissions (5.8%, n = 7). Lastly, the three most frequently evaluated health center outcomes included clinic practices (8.3%, n = 10), general quality improvement metrics (7.4%, n = 9), and clinic diagnostics (4.1%, n = 5).

DISCUSSION

Our scoping review investigated research methods used to evaluate outcomes and implementation success of international Project ECHO interventions. Our scoping review found that a wide array of evaluation methods has been utilized in Project ECHO interventions with the purposes of assessing process measures and outcomes at the patient, provider, and health center levels. We examined various methodological approaches, including study designs, temporality, types of analysis, and data sources, and found that the most frequently utilized methods in these respective categories were cohort studies, longitudinal studies, mixed methods analysis, and surveys. These methods were each reported in more than half of the articles. It was also encouraging to see the growing number of evaluations that incorporated qualitative methods, such as content assessments and phenomenological studies, which provide deeper context about intervention outcomes and implementation. Although we did not conduct full quality or bias assessments, we found that less than 15% of the articles mentioned incorporating comparison groups in their study designs and less than 6% incorporated randomization in any aspect of their evaluation. Furthermore, less than 2% of the articles presented experimental designs.

The two articles presenting experimental designs utilized a RCT with a partial stepped wedge design and a quasi-experimental design. The partial stepped-wedge RCT involved a sequential and staggered rollout of an autism-focused Project ECHO intervention to providers at 10 sites (n = 11–22 providers per site).⁷³ Every 3 months, two sites recruited providers to enroll in the intervention, and they were randomized to startup order after recruitment into the study. Providers completed assessments measuring their autism care knowledge and self-efficacy at four points at 3-month intervals: baseline/preintervention, mid-intervention, postintervention, and postintervention follow-up. The providers’ autism screening and comorbidity management practices were also evaluated. The quasi-experimental study compared provider-level outcomes including knowledge, attitudes, and beliefs regarding pain conditions, opioid prescribing practices, and referrals to behavioral health and subspecialty care providers among providers participating in a pain-focused Project ECHO intervention (n = 10) with a control group of providers that did not partake in the intervention (n = 10).³³ These outcomes were evaluated before and after the intervention. The lack of experimental methods in most articles and the small sample sizes included in the evaluations using experimental methods indicate significant gaps pertaining to scientific rigor in Project ECHO evaluations. This limited rigor poses threats to the evaluations’ internal and external validity. Furthermore, given that only one quarter of included articles assessed patient-level outcomes, more patient-level evaluations are needed to assess health impacts of Project ECHO interventions. Lastly, increased usage of qualitative and mixed methods as part of Project ECHO evaluations would be highly valuable, given that these methods yield a deeper understanding of programmatic impacts.

Several ongoing and planned Project ECHO interventions have employed more rigorous methods and will help address these gaps. These future evaluation methods were discussed in studies excluded from our scoping review since we focused on methods that were successfully executed. However, it is important to note their potential for improving the quality of Project ECHO evaluation research. An ongoing Project ECHO intervention in India focused on mental health and substance use disorders is being evaluated via two RCTs with clustered and simple random sampling.¹⁴¹ Another ongoing intervention in the United States focused on COVID-19 prevention and quality of life in nursing homes is using a RCT with stratified cluster sampling to evaluate the effectiveness of two different Project ECHO programs.¹⁴² In addition, an ongoing evaluation of a diabetes-focused Project ECHO intervention in the United States is utilizing a stepped-wedge design to evaluate patient-, provider-, and center-level outcomes.¹⁴³ Although these studies should yield higher quality research evidence, there is a need for more rigorous evaluations of Project ECHO interventions to examine outcomes for diverse patient populations, providers, and health centers. There is also still much to be learned from pragmatic clinical trial research regarding implications for Project ECHO interventions.

While reviewing the 121 studies included in our scoping review, we found that most were vague in their descriptions of evaluation methods. Many articles did not directly state their study designs, although it was often clear that they focused on a specific cohort of patients, providers, or health centers. As such, we categorized those articles as cohort studies. However, identifying whether the cohorts were evaluated prospectively or retrospectively was not as straightforward. We only categorized studies as prospective or retrospective if those approaches were directly stated. Most articles also did not define the temporality of their outcome evaluations. Although some studies were clearly longitudinal due to administering pre-and-post measures to the same cohort at multiple points over time, others did not explicate whether their participants came from the same cohort. Other articles did not list their data sources; for example, they noted that providers self-reported outcomes; however, it was not apparent if these outcomes were disclosed via surveys, interviews, or other mechanisms.

We initially intended to report summary statistics pertaining to the number of spoke sites, providers, and patients who completed the Project ECHO program evaluations. However, we observed that these numbers were reported inconsistently throughout the 121 articles, with some articles reporting the total number of teleECHO clinic attendees, the total number of registrants who were not necessarily attendees, the number of participants who completed the evaluations, the number of participants who completed only certain components of the evaluations, or other metrics. Furthermore, we observed that the term “spoke” was defined differently throughout the articles, with some articles using the term in reference to large health centers with multiple smaller practices, some referencing only smaller practices, and others referencing individual providers. These discrepancies indicate a need for clear and consistent reporting of Project ECHO intervention evaluation methods and data. In addition, inconsistencies in participation, such as partial or temporary participation, can potentially reduce the impacts of Project ECHO interventions. A resource guide that includes best practices for data reporting, enrolling and maintaining participants, and other purposes would be highly valuable to develop.

Strengths and Limitations

Our scoping review has numerous strengths. First, our review is the most extensive examination of Project ECHO interventions to date. It is important to mention that another scoping review of Project ECHO interventions was published in 2022 while ours was still in progress; however, its focus was on patient- and community-level outcomes and it only included 15 articles published through June 2020.¹⁴⁴ The timing of our scoping review is paramount given the rise of tele-education and telework during the COVID-19 pandemic and the likelihood that these modalities will gain increased traction in the future. Our scoping review included international Project ECHO interventions and different types of evaluations that were excluded from prior reviews, such as content assessments and phenomenological studies, which yield deeper insights into knowledge translation. Specifically, these qualitative methods reveal ontological themes, processes, lived experiences, and meanings of data associated with programs.^145,146 We also examined outcome measures at the patient, provider, and health center levels, providing a comprehensive look at all indicators of Project ECHO intervention effectiveness. Finally, we reviewed process evaluation measures, which are key to determining the implementation success of Project ECHO interventions.

Despite these strengths, our scoping review has limitations. We did not conduct a systematic review, which would have involved critically appraising articles to assess their risk for bias. However, given that the primary aim of a scoping review is to set the stage for a subsequent systematic review, completing a systematic review would be a pertinent next step. We utilized Arksey and O’Malley’s scoping review methods instead of the Joanna Briggs Institute (JBI) methods, which are more recent.^22,147 Another limitation is that we completed our article search during the COVID-19 pandemic, when new studies on tele-education were rapidly being published. It is possible that relevant articles published shortly after we concluded our search on July 3, 2021, were overlooked. We also did not utilize Medical Subject Headings (MeSH terms) in the PubMed search. Limitations are also associated with the lack of clarity in the included articles pertaining to evaluation methods, such as unclear information about study designs, temporality, and data sources. For articles that did not directly state any aspect of their evaluation, or certain methods were not easily discernible, we marked the methods as “other” and, therefore, were unable to thoroughly categorize them. As noted previously, reporting guidelines for Project ECHO interventions would reduce the likelihood that evaluation methods and data are reported vaguely and inconsistently.

CONCLUSIONS

Project ECHO interventions have reached significant numbers of patients affected by a multitude of health conditions, at-risk communities, primary care providers, and health centers around the globe. Our scoping review identified 121 peer-reviewed evaluations of Project ECHO interventions aimed at determining this model’s effectiveness and efficiency at building capacity among health care providers and health centers as well as improving outcomes, such as knowledge, prescribing practices, and cost-savings. Through these evaluations, a diverse array of research methods has been used to assess implementation success and outcomes of Project ECHO interventions. These methods include but are not limited to observational, experimental, and phenomenological studies; longitudinal and cross-sectional designs; and qualitative, quantitative, and mixed method analysis. Multiple data sources are incorporated in these evaluations, including surveys, focus groups, 1-on-1 interviews, existing datasets, EHR queries and chart reviews, teleECHO clinic transcripts, case presentations, consultation questions, and participation records. Despite the expansive scope of Project ECHO intervention evaluations, few have utilized rigorous methods such as RCTs and most published studies provide vague descriptions of their methods. This lack of rigorous methodology indicates that more RCTs should be conducted to test the effectiveness of the Project ECHO model at improving outcomes. Furthermore, Project ECHO evaluation guidelines should be developed to strengthen the reporting of research methods and data.

Supplementary Material

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.jcehp.org).

Table 2.

Project ECHO Intervention Evaluation Methods

	n	%	Reference Nos
Evaluation temporality
Longitudinal	90	74.4	9–14,18,19,33–35,38,40,42,43,46–55,57,58,61,62,65–73,75–78,80–83, 85–87, 89, 91–95, 97, 99–106,108,109,111–115,120–128,130–136,138–140
Cross-sectional	19	15.7	17,30,31,36,37,39,41,44,45,56,84,88,90,98,107,110,119,129,137
Other/not specified	12	9.9	15,32,59,60,63,64,74,79,96,116–118
Evaluation designs
Cohort study	105	86.8	9–14,17–19,30,31,33–58,61–63,65–69, 72, 75–82, 84–90, 92–95,97–115,119–132,134–140
Pilot study	35	28.9	9,11,13,18,31,35,41–43,45,47,48,52,54,56,61,65–67,72,74,82,84,87,90,95,100,105,115,123,127,132,133,137,138
Feasibility study	31	25.6	11,13,31,32,37,43,47,48,55,56,61,66,72,74,80,82,84,86,89,98,105,111–113,124,127,128,133,137–139
Case study	5	4.1	69–71,91,106
Phenomenological study	3	2.5	17,96,117
Content assessment	2	1.7	64,118
Model simulation	1	0.8	83
Randomized controlled trial: partial stepped wedge study	1	0.8	73
Quasi-experimental study	1	0.8	33
Use of randomization
Yes	7	5.8	53,57,73,92,96,118,138
Use of comparison groups
Yes	17	14	10,18,33,40,52,58,68,74,78,81,83,93,102,103,108,131,136
Data analysis methods
Mixed methods	63	52.1	14,15,19,31–35,37,39,41,45–48,50,51,53,56,57,60,61,67,69–72,74, 77, 79, 80, 82, 84–87, 92, 95, 97, 99–101,106,111–116,120,121,124,125,127, 132–135, 137–140
Quantitative methods	42	34.7	9–11,13,18,38,40,42,43,49,52,54,55,58,59,62,63,65,66,68,73, 75, 76, 78, 81, 83, 89, 93, 94, 102, 103, 105,108,109,122,123,126,128–131,136
Qualitative methods (with quantitative only for process data)	16	13.2	12,17,30,36,44,64,88,90,91,96,98,107,110,117–119
Evaluation data sources
Surveys and questionnaires	74	61.2	9,11,13,19,32–35,37,38,43–49,53–55,57,61–63,66,67,69,72–75, 79, 80, 82, 84, 86, 87, 89, 92, 94, 95, 97,99–102,104–106,109,111–116,120–128,130, 131, 134, 135, 137–140
ECHO clinic participation and attendance records	29	24	14,15,19,31,32,37,39,43,48,51,53,60,63,67,82,83,85,86,89,101,108,109,111,115,122,123,127,133,137
One-on-one interviews	23	19	14,17,30,31,34,36,50,56,67,77,85–88,90–92,97,99,101,132,133,138
ECHO clinic transcripts, case presentations, or consultation questions	23	19	14,15,31,41,42,51,60,64,66,68–71,76,79,91,96,104,106,117,118,129,133
Focus groups	22	18.2	12,14,17,30,33,35,44,46–48,57,97–99,107,110,113,127,134,135,138,140
Electronic health record (EHR) or patient lab reports	17	14	18,33,34,38,40,49,51,57,73,77,80,93,100,103,131,132,139
Existing data, eg, Medicaid Claims and public datasets	13	10.7	18,50,52,54,58,60,65,69,77,78,81,83,108
Observation	5	4.1	13,31,37,64,106
Direct inquiries, eg, emails, meetings, or phone calls	5	4.1	71,82,91,101,125
Not specified/vague	4	3.3	31,87,126,127
Field notes	2	1.7	37,67

The most frequently reported Project ECHO evaluation methods included: longitudinal research design, cohort study, mixed methods analysis, and surveys. Most evaluations did not utilize randomization or comparison groups.

Lessons for Practice.

• A vast array of research designs, measurement tools, data sources, and data analysis methods has been used in Project ECHO intervention evaluations.

• More scientifically rigorous evaluations of Project ECHO interventions such as randomized controlled trials are needed to test the Project ECHO model’s efficacy at improving outcomes.

• Guidelines for Project ECHO evaluations should be developed to ensure clear and consistent reporting of research methods and data.