Use of Project ECHO Telementoring Model in Continuing Medical Education for Pediatricians in Vietnam: Preliminary Results

Abstract

The ECHO (Extension for Community Health Outcomes) model has been introduced and implemented in several hospitals and health programs in Vietnam since 2015. In 2018, Vietnam National Children’s Hospital (VNCH) officially implemented the ECHO model to provide continuing medical education (CME) credits on pediatrics topics for medical staff in its satellite hospitals and health centers in the Northern region of Vietnam. This paper presents preliminary results of the ECHO program at VNCH. Methods included pre- and post-program assessments of pediatricians’ clinical knowledge, self-efficacy, and professional satisfaction. The analysis compared the differences between pre/post scores descriptively. Knowledge of participants increased by 22.5% points on average. More than 90% of Project ECHO-Pediatrics participants experienced increased confidence. Overall, there was an improvement in participants’ self-efficacy in the post-training compared to the pre-training (range 14.7%-22.6% difference from pre-training). All participants improved on their results in the clinical test immediately after the training and maintained it after 3 months. The study demonstrated the ability Project ECHO to improve healthcare worker knowledge and satisfaction.

Background

Approximately 90% of current hospitals in Vietnam are public hospitals. All of them fall under the Ministry of Health’s direct management, including medical services, human resources, and finance. Projects “satellite hospitals” from 2016 to 2020 and “National telehealth program” from 2020 to 2025 in which central-level hospitals are responsible for training and technology transfer to provincial-level hospitals, who then will continue to train district and commune-level hospitals in the fields of Internal Medicine, Surgery, Obstetrics, Pediatrics, Cardiology, and Oncology.

The Vietnam National Children’s Hospital (VNCH) is one of the leading pediatric hospitals in Vietnam. As a central-level hospital, it is responsible for providing mentorship and training for 18 provincial-level hospitals in 29 provinces as assigned by the Ministry of Health of Vietnam. The training courses are typically organized at VNCH’s facilities in Hanoi. Health professionals would travel annually to participate in these CME-credited courses, which could last from a few days to a few months, depending on the specialty. Although face-to-face training provides valuable experience and knowledge for the participants, it also incurs costs to the trainees and their respective hospitals; and causes a lack of health staff at the provincial-level hospitals.

The Extension for Community Healthcare Outcomes (Project ECHO) model is based on an innovative framework that leverages technology to provide training and ongoing mentorship to community-based primary care providers (PCPs).¹ Project ECHO was initially developed in 2003 to improve healthcare for patients with hepatitis C by training and mentoring PCPs in rural regions using videoconferencing technology.^1,2 As of 2020, project ECHO has been replicated in 49 countries throughout 6 continents, with 528 hubs and 955 different programs. The programs cover a variety of disease conditions, such as rheumatology, dermatology, musculoskeletal disease, asthma, chronic pain, palliative care, inflammatory bowel disease, women’s health, complex regional pain syndrome, dementia, diabetes, epilepsy, hypertension, liver disease, mental health, and transgender care, HIV, geriatric, COVID-19, Autism, perinatal care, managing pneumoconiosis.^2-23

From January 2020 to June 2020, VNCH organized 3 ECHO programs, providing training and mentorship in best-practice care for 3 specialties: pediatric cardiology, pediatric respiratory, and pediatric neonatology. ECHO clinics for each of these specialties were held for 2.5 hours, 4 to 5 days per week, for 2 consecutive weeks. Eighteen provincial-level hospitals registered to send participants to the 3 programs. During each clinic, all pediatric-care provider participants (“spokes”) were connected with each other and to a team of experts (“hub”) through secure multipoint videoconferencing technology (Zoom web-based platform). Members of the expert panel at the hub had specific expertise in each of the 3 specialties and included pediatric cardiologists, pediatric respirologists, neonatologists, and radiologists.

Each ECHO session also included a de-identified case presented by one of the pediatric-care provider (PCP) participants from the provincial-level hospitals for discussion, creating opportunities for shared learning, peer support, and mentoring. The expert hub team also gave verbal and written feedback and recommendations for each case presented, providing support to PCPs as they maintained responsibility for their patient’s care. Resources, including the de-identified case report and other documents, were shared and stored via email or a web-based platform.^24,25

This study aimed to evaluate the initial results of Project ECHO’s effect on PCPs’ satisfaction, clinical knowledge, and self-efficacy in pediatric care.

Methods

Study setting

The study was conducted at the Vietnam National Children’s Hospital and its satellite hospitals from January 2020 to June 2020.

In order to implement Project ECHO, VNCH has prepared human resources for implementation and set up the implementation process in addition to equipping an online software system. This key resource is called the implementation team from the Training and Direction Center. The implementation team is trained regularly following ECHO Institute agenda training from New Mexico University (US). The team component included: 1 program manager, 1 facilitator, 1 coordinator, 1 IT excutive.²⁵ In VNCH, a full ECHO program included 4 phases including phase I: ECHO recruitment; phase II: Prior to ECHO training course; phase III: at ECHO training course; phase IV: Post ECHO training course (please see Table 1).

Table 1.

Implementation process at the ECHO-VNCH.

Phase I	ECHO recruitment	Monitored by
1	Implement survey for provincial-level hospital’s training needs	Implementation team at hub
2	Recruit medical leads (experts) from VNCH’s human resource	Hospital leadership’s decision at hub
3	Based on needs, devise a plan for training and inform the provinces about the agenda training	Implementation team at hub
4	Provincial hospitals submit a list of participants to the course	Implementation team at spokes
Phase II:	Prior to ECHO training course
1	Make consensus with medical leads on agenda training following curriculum, knowledge test (pre-test, post-test. . .), toolkit, supervision activity. . .	Program manager or/and facilitator
2	Organize a Zoom call to go through all regulations during the course	Coordinator and IT executive
3	Organize an online pre-test: evaluate professional knowledge	Coordinator and IT executive
4	Guide case-based presentation following format: Case-presentation at the beginning if participants are not willing to present cases, the VNCH will take a case at their site to present, then share the format with the provinces. They later will be required to submit a clinical case to get used to the format	Facilitator
Phase III	at ECHO training course
1	Didactic: took 30′-45′ by medical lead	Facilitator
2	Case-presentation: 30′-45′ by participants from hub and spoke assigned	Facilitator
3	Q&A discussion, interaction: 30′-45′ by participants and medical lead	Facilitator
4	Assessment on spoke’s case report presentation by medical lead	Facilitator
Phase IV	Post ECHO training course
1	Post-test: assessment on participants’ knowledge after the course	Coordinator and IT excecutive
2	Toolkit to evaluate the course: surveys (including self-efficacy assessment, participant satisfaction), interview guides, supervision reports. . .	Program manager and/or facilitator and medical lead
3	Surpervision activity at spoke sites: trips to the provincial-level hospitals for supervision by VNCH team	Supervision team (program manager and/or facilitator, medical lead)

The ECHO curriculum focused on understanding the diagnosis, treatment, and management of patients in the selected specialties and providing evidence-based care. The content was based on the currently available evidence and best-practice approaches for pediatric cardiology, respiratory, and neonatology (see Supplemental Appendix 1).

Clinical knowledge related to pediatrics (neonatology, respiratory, cardiology) was assessed through the pre- and post-tests and post-3 month test with multiple-choice questionnaires covering clinical analysis and reasoning, diagnosis, treatment approach, prescription, prognosis and risk management, and patient counseling knowledge. All participants applied their capacity to manage patients at their workplace during and right after the practicum period. Pre- and post-ECHO training surveys of self-efficacy and confidence of professional satisfaction were calculated for each PCP by summing scores for each statement, using Likert scale with 1 = Strongly Disagree; 2 = Disagree; 3 = Neither Agree nor Disagree; 4 = Agree, and 5 = Strongly Agree.

Questionnaires and scales were completed using a web-based data collection system using an online survey form designed using © Google Forms.

In this study, we evaluate the outcome of ECHO program in VNCH from level 2 (participant’s satisfaction) to level 4 (participant’s competence) following Moore’s expanded framework.²⁶ Toolkits including questionnaires, scales designed to evaluate each levels (\ see Table 2).

Table 2.

Outcome of the ECHO-VNCH to fulfill Moore’s Levels of Expanded Framework.

Levels of expanded framework	Description of level	Outcome evaluation
Level 1: participation	Number of health professionals who participated in CME activity	Attendance records are well kept in 2020. 201 clinicians fulfilled ECHO-Pediatrics and received 12 462 education hours, similar to 24 924 credits
Level 2: satisfaction	Degree to which expectations of participants about CME activity were met	Questionnaires are completed by every provincial pediatrician after a CME activity. Surveys were conducted via an online questionnaire to evaluate the levels of satisfaction of PCPs with acquired knowledge, class organization, connectivity during the course using the 5-point Likert scale from “1 = completely disagree” to “5 = completely agree” (in Supplemental Appendix 2)
Level 3: Learning: knowledge	Participant statements about what the CME intended them to know	Knowledge tests included a pre-test and a post-test, as well as a post-3 month test that evaluate how well they change their knowledge of pediatrics through process of education via Tele-ECHO (in Supplemental Appendices 5a-c)
Level 4: Learning: competence	The degree to which participants show in an educational setting how to do what the CME activity intended them to be able to do	During each tele-ECHO clinic, provincial pediatricians presented patient cases selected from their facility. They identified the questions they wanted the specialists (medical lead) and peers to consider about the patient in writing and in their presentation in CBD. After each patient case, a summary of recommendations from the specialists and peers was presented on CBD (in Supplemental Appendix 3) will share to presenter. The participants also provided recommendations and guidance to their peers during the teleECHO session in response to a peer’s patient case or during the discussion that followed the case presentation. Surveys about competence and professional satisfaction were conducted via an online questionnaire using the 5-point Likert scale with 1 = very low and 5 = excellent (in Supplemental Appendices 4a and b).

Sample and study design

All pediatric-care providers who participated in the program during the period from January to June, 2020 were included in the sample.

A method evaluation comprised quantitative study including participant’s satisfaction, tests of clinical knowledge, and survey of self-efficacy on the participant’s ability in professional competency and confidence in job satisfaction through their experience at pre- and post-ECHO training. Participants completed questionnaires including tests and surveys as mentioned above at pre-ECHO training (before attending the first ECHO session) and post-ECHO training (after completing the program). Pre- and post-data were linked using participant-generated ID codes.

Data analysis

Descriptive statistics were used to describe and summarize participant characteristics. Mann-Whitney U tests were used to explore differences in pre- and post-ECHO evaluations and P-values reported to indicate the model’s impact on physicians’ clinical knowledge and self-efficacy. Statistical significance was set at P = .05.

Results

As mentioned in the background section, from January 2020 to June 2020, VNCH conducted 3 ECHO programs, providing training and mentorship in best-practice care for 3 specialties: pediatric cardiology, pediatric respiratory, and pediatric neonatology. Through the questionnaires, we looked into multiple aspects such as participants’ clinical analysis and reasoning, diagnosis, treatment approach, prescription, prognosis and risk management, and patient counseling knowledge. The analyses show statistically-significant changes in almost all these aspects among participants’ pre-and post-training. These analyses are presented below.

Two hundred one participants completed both the pre- and post-clinical program-knowledge assessments. The average age of participants was 34.9 years, and 79.6% of participants were female. The average length of professional experience was 15.2 years (range 0-25 years). Most participants were pediatricians (81.6%) (please see Table 3).

Table 3.

Characteristics of the participants of the VNCH—Project ECHO program.

Characteristics	n	Percent
Profession
General Doctor	37	18.4
Pediatrician	164	81.6
Gender
Male	41	20.4
Female	160	79.6
Work experience
1-5 years	11	5.5
6-10 years	54	26.9
10+ years	136	67.7

Analyses examined the participant’s self-efficacy of clinical practice improvement at pre-and post-training. The score of all 6 domains increased by 22.5% points, averagely (please see Table 4).

Table 4.

Self-efficacy assessment at pre-and post-training.

Criteria	Pre-training n = 201	Post-training n = 201	P
Ability to make clinical analysis and reasoning when facing the clinical cases	3.1 (0.75)	3.8 (0.55)	<.001
Ability to make diagnosis based on assessment of patient condition	3.2 (0.66)	3.9 (0.52)	<.001
Ability to follow treatment approach based on VNCH guideline	3.1 (0.67)	3.8 (0.53)	<.001
Ability to make prescriptions based on VNCH guideline	3.1 (0.67)	3.8 (0.56)	<.001
Ability to make prognosis and manage risk related to patient’s disease conditions	3.1 (0.71)	3.8 (0.54)	<.001
Ability to counsel patient related to patient’s disease conditions	3.1 (0.69)	3.8 (0.55)	<.001
Overall	3.1 (0.65)	3.8 (0.49)	<.001

Analyses of the pre/post program revealed that more than 90% of participants experienced increased confidence in their jobs. Overall, there was an improvement in participant self-efficacy in all domains assessed in the post-training compared to the pre-training (range 14.7%-22.6% difference from pre-training) (please see Table 5).

Table 5.

Self-assessment of confidence to meet job demand at pre- and post-training.

Criteria	Pre-training n = 201	Post-training n = 201	P
Confidence to perform professional competency at workplace	3.2 (0.69)	3.8 (0.58)	<.001
Opportunity to share clinical experience with colleagues via telehealth activities	3.2 (0.75)	3.9 (0.57)	<.001
Being professionally supported by experts at hub via telehealth activites	3.3 (0.8)	3.9 (0.57)	<.001
Ability to meet job satisfaction at workplace	3.4 (0.74)	3.9 (0.6)	<.001
Ability to improve capacity of healthcare service delivery at workplace	3.1 (0.71)	3.8 (0.54)	<.001
Overall	3.4 (0.71)	3.9 (0.58)	<.001

Before the program, participants earned an average of 59 points, and most of them were below the cut-off point for a pass (70 points). Immediately after the training, 185 out of 201 completed the test and got an average of 88 points. One-hundred percent of them passed the test. Three months after the training, 139 participants still maintained their knowledge when taking the test (please see Table 6).

Table 6.

Scores obtained by participants at pre-training, post-training, and at 3 months after completing the course.

Time of assessment	Score	P
Before the training (n = 185)	59 (11.3)
After the training (n = 185)	88 (8.2)	<.001
3 months after the training (n = 139)	89 (10.9)	<.001

Not all participants completed the tests.

96.7% of participants were satisfied with the training program using the Project ECHO approach. All aspects, including course content, trainers’ support, training materials, and trainer-participant interaction, were highly appreciated (please see Table 7).

Table 7.

Participants’ satisfaction with the telehealth program.

Criteria	Mean	SD	Min	Max
The lectures are useful	4.37	0.69	1	5
The contents of the lectures are relevant	4.32	0.72	1	5
The lessons are useful/interesting	4.32	0.70	1	5
The clinical cases are useful	4.33	0.71	1	5
The clinical cases are relevant	4.27	0.72	1	5
The clinical cases are helpful/interesting	4.25	0.73	1	5
Trainers use suitable teaching delivery methods	4.29	0.73	1	5
The trainer is highly knowledgeable of the subject	4.47	0.70	1	5
Trainers have good interactions with learners	4.38	0.72	1	5
The training time is reasonable	4.05	0.88	1	5
The training content is updated and relevant for learners	4.34	0.72	1	5
Guidance for preparing devices for online lectures is sufficient	4.36	0.73	1	5
Guidance for setting up and using online tools is sufficient	4.38	0.72	1	5
Provision of training materials is timely and sufficient	4.33	0.74	1	5
Internet connection during the sessions is good	4.20	0.75	1	5
The quality of the images and sound is good	4.15	0.77	1	5
Interaction capacity is good	4.21	0.73	1	5
Satisfaction with the training outcomes	4.15	0.66	2	5

Discussion

Project ECHO-Pediatrics was first implemented in Vietnam in 2020 and has continued to show its suitability in capacity building for health workers at all levels. The ECHO-Pediatrics model that connects the central-level with provincial-level hospitals, also known as the “hub and spoke” model, to promote medical examination and treatment by PCPs at the provincial level, through which patients benefit indirectly. This model’s advantages are notable because the PCPs at the provincial level hospital continue to leverage their knowledge to the district-level and community-level primary healthcare providers. Our evaluation results show that pediatricians’ clinical knowledge is significantly increased. Our findings are consistent with those from other settings^27-33 and again show the potential of the ECHO model in improving knowledge for medical staffs at different levels.

The participant satisfaction survey show that learning goals were achieved through the ECHO training model that reduced feelings of professional isolation, increased in providers’ confidence to manage any aspect of patient care,^29,34,35 tightened clinical network through learning loops, broadened community of medical practice,^15,36,37 and achieved CME.^14,30,38-42

Some studies show Project ECHO has succeeded in building networks to respond to disaster or pandemic like Zika, COVID-19^20,43-51 that provide rapid multidirectional information exchange, disseminating, and sharing information.

A study of mental health and addictions ECHO program in India show an increase in learning and self-confidence among participants.^52,53 Similarly, an evaluation of an ECHO-Oncology program in Africa nations found improved self-confidence and knowledge for participating providers,⁵⁴ as assessed the hepatitis C Project ECHO in New Mexico, which found improved self-efficacy and confidence in professional satisfaction.^1,55 Participants develop confidence and expertise in using evidence-based strategies to support their community.^{18,53,54,56-60}

Project ECHO-Pediatrics incorporates principles, practices, and policies recommended by the Agency for Education and Healthcare Research (AEHR), Agency for Care and Treatment (MOH), and other government panels and professional organizations that solve the problem of health disparities.⁶¹

The ECHO-Pediatrics model at VNCH can only be achieved through the government’s collaboration, especially the MOH, the private sector, and stakeholders in the health care sector. The role in the units of training and coordinating healthcare at provincial hospitals is significant in this process, as they take the leading parts in coordinating among different sectors, as well as monitoring as team implementation in this program. Until now, the ECHO-Pediatrics model at VNCH has been operated using national funding and served 18 satellite hospitals of VNCH under the “Satellite Hospital Project” from 2019 to 2020 and in partnership with the ECHO Institute in virtual platform since 2018. Thus, all courses are appropriately designed, organized, and evaluated according to Decree 22 by the MOH. This rigorous framework, however, could be difficult to maintain when national funding runs out in the future.

Moreover, there is a workforce shortage in implementation of the ECHO program for lower-level healthcare facilities, especially at the district and community-level. Therefore, tele-mentoring programs at present is mostly deployed between central and provincial-level hospitals, while demand remains high at district and community health facilities as well. A plan should be in place to sustain funding for other tele-ECHO activities and proper human resources to run in the future that enable to expand to more underserved locations. Cost-analysis studies are encouraged to find out the effectiveness on cost between ECHO and traditional training models.^62-66

Intrinsic to the value of this paradigm shift in the training and treatment of Pediatric diseases is to enable VNCH, Pediatric network, and MOH to fulfill the tremendous 3-fold mission of health education, clinical care, and research, as well as recognized obligation to protect and improve Pediatrics’ health in community^61,67,68

Through the pilot the model of ECHO-Pediatrics, we have compiled documents of training and guidelines database on clinical practices, scheduled for an online library of pediatric knowledge that will leverage to all pediatric healthcare providers in Vietnam.

Programs such as Project ECHO are ideal venues for the pursuit of clinical research in the field of epidemiology, healthcare service delivery, education conveying best practices, evidence-based medicine, and healthcare cost-effectiveness, which have real-world applicability.^4,36

To sustain and replicate ECHO-Pediatrics to the wider Vietnam healthcare system, primarily to medically underserved areas, we subsequently propose the MOH to continue the funding for this and integrate this model into the “National telehealth project” that will be implemented widely throughout Vietnam from the central to the community level starting in 2021.⁶⁷ Another indication for sustainability and expansion of Project ECHO-Pediatrics would be stable fundings from pharmaceutical companies, hospital’s workforce development funding, and VNCH co-funding mechanisms.

Limitation

This study is subject to several limitations. First, while the evaluation demonstrated an increase in provider’s knowledge and self-efficacy, it did not assess the impact on patient outcomes. However, the Project ECHO model has been shown its effectiveness in improved patient outcomes in other settings and disease areas.^19,28,69,70 Second, though participants experience increased satisfaction, this evaluation did not assess provider retention. Also, clinical performance assessment was limited because the assessment tools cannot detect detailed knowledge or participants’ specific skills. Small sample sizes also did not allow us to conduct further analyses in participant subgroups. The lack of a control group is another study limitation.

Conclusions

The Project ECHO-Pediatrics demonstrated the feasibility of adapting the Project ECHO model to continuing medical education in low- and middle-income settings. The Project ECHO-Pediatrics model has successfully promoted distance learning through virtual communities of practice and increased physicians’ self-efficacy. By helping achieve the goal of delivering the proper knowledge to the right place at the right time for those who need it most, the Project ECHO-Pediatrics model has proven to be a powerful tool for improving the health workforce’s quality in Vietnam and should be regarded as the innovative training model of CME that can be applied widely in the situation of a global pandemic such as COVID-19.