Integrating quality improvement into the ECHO model to improve care for children and youth with epilepsy

Abstract

Objective

Project ECHO (Extension for Community Healthcare Outcomes), a telementoring program, utilizes lectures, case‐based learning, and an “all teach–all learn” approach to increase primary care provider (PCP) knowledge/confidence in managing chronic health conditions. The American Academy of Pediatrics (AAP) Epilepsy and Comorbidities ECHO incorporated quality improvement (QI) methodology to create meaningful practice change, while increasing PCP knowledge/self‐efficacy in epilepsy management using the ECHO model.

Methods

Monthly ECHO sessions (May 2018 to December 2018) included lectures, case presentations/discussion, and QI review. Pediatric practices were recruited through the AAP. Practices engaged in ECHO sessions and improvement activities including monthly Plan‐Do‐Study‐Act cycles, team huddles, chart reviews, and QI coaching calls to facilitate practice change. They were provided resource toolkits with documentation templates, safety handouts, and medication side effects sheets. QI measures were selected from the American Academy of Neurology Measurement Set for Epilepsy. The AAP Quality Improvement Data Aggregator was used for data entry, run chart development, and tracking outcomes. Participants completed retrospective surveys to assess changes in knowledge and self‐efficacy.

Results

Seven practices participated across five states. Average session attendance was 14 health professionals (range = 13‐17). A total of 479 chart reviews demonstrated improvement in six of seven measures: health care transition (45.3%, P = .005), safety education (41.6%, P = .036), mental/behavioral health screening (32.2% P = .027), tertiary center referral (26.7%, not significant [n.s.]), antiseizure therapy side effects (23%, n.s.), and documenting seizure frequency (7.1%, n.s.); counseling for women of childbearing age decreased by 7.8%.

Significance

This project demonstrated that integrating QI into an ECHO model results in practice change and increases PCP knowledge/confidence/self‐efficacy in managing epilepsy.

Key Points.

• The project team integrated QI methodology into a hub and spoke telementoring model, using epilepsy quality measures from the American Academy of Neurology

• Using a combination of didactic lectures, case‐based learning, and QI methodology, the project team observed improvement in all but one outcome measure

• This model can be used to bring meaningful change in the care of children and youth with epilepsy at a primary care level, in partnership with neurologists

• Integrating QI into a Project ECHO model can provide MOC credits to providers

1. INTRODUCTION

Epilepsy is the most common childhood neurologic condition in the USA, with approximately 470 000 affected children aged birth to 18 years. ¹ Children and youth with epilepsy (CYE) are at higher risk for developmental, intellectual, and mental health comorbidities. ² , ³ , ⁴ , ⁵ Lack of knowledge about epilepsy and comorbidities can also contribute to negative impact on a child's social and psychological function. This is exacerbated in CYE living in rural and medically underserved areas, who lack access to high‐quality coordinated care provided in a medical home. ² , ⁶ In 2016, a child neurology clinical workforce report highlighted an increase in pediatric neurology referrals, particularly for more complex cases. Epilepsy remains one of the most common reasons for referral to a pediatric neurologist. ⁷ Unfortunately, there is an existing deficiency in the number of pediatric neurologists (estimated to be at least 20% below the national need), often resulting in limited access to care for CYE, especially in rural and underserved communities. ⁸ , ⁹

Health care delivery system use of telemedicine technology is one way to improve subspecialty care access. ¹⁰ Project ECHO (Extension for Community Healthcare Outcomes) is a hub‐and‐spoke knowledge‐sharing model, led by expert teams who use multipoint videoconferencing technology to train and support primary care providers (PCPs) through case‐based learning and brief lectures on various health conditions. ¹¹ , ¹² Unlike telemedicine, this model does not establish a provider‐patient relationship, but rather supports PCPs' ability to manage subspecialty conditions in the primary care setting. From 2003 to 2011, the effectiveness of the ECHO model was evaluated by assessing the impact on rural clinicians participating in the Hepatitis C TeleECHO program in New Mexico. The prospective cohort study demonstrated that patients with hepatitis C who were treated via the Project ECHO methodology had as good or better outcomes as those treated at an academic medical center. ¹² Impact measurements included effect on treatment rates, self‐efficacy, and overall professional satisfaction. ¹¹ Data demonstrated the positive impact of the ECHO model on the current health care system in three major areas: (1) access to specialty health care, (2) expansion of delivery of evidence‐based best practice care, and (3) a new paradigm for team‐based interdisciplinary professional development. ¹¹ This success has largely been credited to the impact of patient‐centered, culturally competent care by local providers, who are often a trusted resource to the patient, and more likely to engage in regular communication, thereby enhancing a patient's adherence with treatment. ¹²

Since 2013, the American Academy of Pediatrics (AAP) has been supported by the Health Resources and Services Administration (HRSA)/Maternal Child Health Bureau (MCHB) to serve as the Coordinating Center for Increasing Access to Care for Children and Youth With Epilepsy (Center). The mission of the Center is to establish a multifaceted, community‐based system to ensure that CYE, particularly those in medically underserved and/or rural areas, have access to the medical, social, and other support and services required to achieve optimal health outcomes and improve quality of life. The Center has used the ECHO model to increase access to quality care for CYE by building knowledge and confidence of PCPs, and through program evaluation it was determined that integrating quality improvement into the model may lead to measurable practice change.

In 2012 the Institute of Medicine released Epilepsy Across the Spectrum: Promoting Health and Understanding, which detailed disease prevalence, data collection, and measurement, as well as recommendations for improving quality of life for people with epilepsy through education, family engagement, community resources, and quality improvement (QI). ⁸ The report recommends implementing Maintenance of Certification (MOC) programs to help lead to improved health outcomes for CYE while continuing to engage and incentivize providers to deliver best practice at the point of care. ¹³ MOC is particularly valuable to pediatricians as it ensures that board‐certified pediatricians and pediatric subspecialists have successfully completed accredited training and continue to expand their medical knowledge, improve their practice, and increase patient safety. Improving Professional Practice and Quality Improvement, Part 4 of MOC, is designed to help board‐certified pediatricians and pediatric subspecialists assess and improve the quality of patient care and processes that will lead to improved child health. The Institute for Healthcare Improvement (IHI) Model for Improvement (Model) was integrated into the ECHO model. Offering MOC Part 4 points provides a simple, structured framework to accelerate quality in the health care environment as well as overcome challenges around participant retention, obtaining cases, and demonstrating impact beyond assessing provider knowledge and confidence. The IHI Model encourages interdisciplinary teams to apply three fundamental questions to guide their improvement work: (1) What are we trying to accomplish? (2) How will we know that a change is an improvement? (3) What changes can we make that will result in improvement? ¹⁴ Plan‐Do‐Study‐Act (PDSA) cycles operationalize the Model and provide an opportunity to test small changes in the clinical setting.

2. MATERIALS AND METHODS

2.1. Study design, planning, and collaborative participants

The AAP Epilepsy and Comorbidities ECHO (Epilepsy ECHO) is one of a series of AAP‐led ECHO approaches. Each has integrated feedback from across primary care participants and expert panels to continuously improve process. Epilepsy was the first of its kind to integrate QI processes and training into the ECHO model. The AAP serves as a Project ECHO Superhub—the only Superhub offering pediatric training and technical assistance to partner organizations implementing pediatric ECHOs. The AAP ECHO Superhub recruited multidisciplinary faculty including pediatric epileptologists, primary care pediatricians, behavioral health professionals, health care transition experts, family engagement specialists, and a QI coach. Through a rigorous application process, including questions around previous experience with implementing QI projects and number of CYE seen per month, pediatric practice teams were recruited through the AAP and partners. All seven applicants met project inclusion criteria and participated in the Epilepsy ECHO program, representing states (Florida, Maryland, North Carolina, Pennsylvania, and Texas) with a higher prevalence of pediatric epilepsy from May through December 2018. The patient‐centered faculty developed an eight‐session pediatric epilepsy curriculum (Table 1) using evidence‐based/‐informed clinical practice guidelines on diagnosis and management of epilepsy to model integrated and team‐based care among patients, families, and primary and subspecialty care practitioners in a patient‐/family‐centered medical home. The Epilepsy ECHO program convened monthly eight times. Each session included a brief faculty‐led lecture followed by at least one deidentified case presentation from the participating practice teams incorporating QI concepts. Each practice presented at least one case presentation throughout the duration of the program per MOC requirements. The cases were selected by the practice providers and were based on relevance to the entire network, and did not always relate to the theme for the session. Collectively, the faculty and participants provided recommendations for appropriate screening tools, best practice protocols, and evidence‐based resources to build provider capacity for higher quality, evidence‐based care for pediatric epilepsy patients. ECHOs focus on an interactive community of practice with practices supporting one another as well as support from the expert panel. Discussion between practices was encouraged but not required. Practice participants who completed project requirements were eligible to receive MOC, Continuing Education, and/or Continuing Medical Education (CME) credits at no cost. The Epilepsy ECHO was reviewed and approved as exempt by the AAP Institutional Review Board.

TABLE 1.

Epilepsy ECHO curriculum

Session #	Topic
1	AAP Epilepsy and Comorbidities ECHO Overview
2	Workflow Processes and PDSA Cycles for Management of Epilepsy
3	Comorbidities of Pediatric Epilepsy
4	Seizures: Diagnostic Evaluation
5	Medication Management of Epilepsy
6	Safety and Education for Epilepsy
7	Role of the Primary Care Provider in a Medical Home Setting
8	Health Care Transition Process for CYE

Abbreviations: AAP, American Academy of Pediatrics; CYE, children and youth with epilepsy; PDSA, Plan‐Do‐Study‐Act.

2.2. QI methodology

Driven by the primary care practice priorities, seven outcome measures were selected from the 2014 American Academy of Neurology (AAN) Measurement Set for Epilepsy based on alignment with existing curriculum and used to track performance improvement in practices (Table 2). Performance improvement goals for each outcome measure were individualized based on the input from the ECHO faculty and QI expert as well as taking the short project timeline into consideration. Some measures (seizure frequency documentation, mental health screening, epilepsy center referral) were identified as ones that providers already discuss routinely; for these, lower improvement goals of 5%‐10% were set. On the other hand, measures addressing safety education, side effects of antiseizure medications, and transition readiness, which were viewed as ones not routinely addressed by PCPs and likely to need more education and intervention, had higher improvement goals. The measure around reproductive health was optional; therefore, target improvement was set lower at 5%.

TABLE 2.

Epilepsy ECHO QI measures, definitions, and target improvement

QI measure	Measure definition	Target improvement over baseline, %
Safety education	Percent of all patients with a diagnosis of epilepsy, or their caregivers, who were provided with personalized safety issues and epilepsy education at least once annually	30
Querying and intervention for side effects of antiseizure therapy	Percent of all patients with a diagnosis of epilepsy with active antiseizure therapy side effects for whom an intervention was discussed	20
Health care transitions	Percent of patients who had a neurological transition plan of care	20
Seizure frequency	Percent of all visits for patients with a diagnosis of epilepsy where the seizure frequency of each seizure type was documented	10
Screening for psychiatric or behavioral health disorders	Percent of all visits for patients with a diagnosis of epilepsy where the patient was screened for psychiatric or behavioral disorders	5
Referral to comprehensive epilepsy center	Percent of all patients with a diagnosis of treatment‐resistant (intractable) epilepsy who were referred for consultation to a comprehensive epilepsy center for additional management of epilepsy	5
Counseling for women of childbearing potential with epilepsy (optional)	All female patients of childbearing potential (12‐44 y old) diagnosed with epilepsy who were counseled or referred for counseling for how epilepsy and its treatment may affect contraception or pregnancy at least once per year	5

Abbreviation: QI, quality improvement.

Practice teams tracked the measures through monthly retrospective chart review to assess progress toward improvement goals and to be eligible for MOC credit. Faculty encouraged the development of sustainable processes, embedded in the practice workflow from the project start. A systematic random sample (at least 10 charts per practice) of patients seen in the previous month with a diagnosis of epilepsy (aged 1‐26 years) were reviewed by the participating practice team (lead physician, nurse, or medical assistant) and entered into the AAP Quality Improvement Data Aggregator (QIDA) system. Data entered in QIDA were available to practices for “real‐time” feedback and evaluation through run chart reports. Monthly individual and collaborative‐wide run charts guided improvement goals for subsequent data cycles, which allowed practice teams to identify barriers, observe patterns or variations, and measure progress toward achieving project aims.

2.3. Statistical analysis

Monthly chart review data were collected from the practice teams over the course of eight data cycles, to assess progress toward improvement goals and to be eligible for MOC credit. It was encouraged that the core QI team at the practice participate in chart review; however, this varied from practice to practice depending on capacity. Any patient chart with an International Classification of Diseases, 10th Revision code for epilepsy could be included in the chart review, regardless of primary provider. The chart review required about 1 hour to pull a systematic sampling of charts and enter the data into the AAP QIDA.

Repeated‐measures analysis of variance (ANOVA) was used to review performance of multiple variables within each study group to determine whether significant improvement occurred over time. The independent variable for each repeated‐measures ANOVA test was time (project period). The dependent variables for each group tested separately are listed in Table 4. The null hypothesis of each test was set to no change or difference between the dependent variable within each group over time. The alternative hypothesis of each test was set to change or difference between the dependent variable within each group exists over time. Alpha was set for .05 for all ANOVA tests.

TABLE 4.

Participating practice quotes around practice change

Practice #1	We presented the project to our peers and we asked to have a more comprehensive history on epilepsy patients and to document presence of side effects. I also included the safety handouts in patient instructions because what we have currently is very long and less useful… Our plan for our next PDSA is to improve the template so that it would be used by all the practice, just as a reminder to people to ask all the necessary questions.
Practice #2	I'm thinking about side effects. I'm much better about asking about medication side effects. I could just kind of gloss over it and say, "Well, how's it going? Do you notice anything? You look great to me!" But thinking more about, "Are you having dry mouth? Are you having more sedation? Are you having…" and reporting that back to the neurologist and trying to figure out if this is the best drug for them. I'm much more sensitive to that. Before I was just, "Are you getting your medications filled and are you taking them?"
Practice #3	…the way we see our patients in the office changed and the way we asked about frequency of seizures—"What did you do about it?"—we really have documented and improved our visit for the seizure patients… Our EMR will trigger you that when you have a seizure patient you do the seizure action plan. You're going to remind them about this and "Did you share this?" and "Did you give this to the patient?" So those are things already that help us and trigger us to continue our good care.

Abbreviations: EMR, electronic medical record; PDSA, Plan‐Do‐Study‐Act.

2.4. Program evaluation

In addition to the analysis of the quality measures, the American Academy of CME Outcomes Model guided the evaluation plan for the Epilepsy ECHO program by assessing five of the seven levels: participation, satisfaction, declarative/procedural knowledge, competence, and performance. ¹⁶ The five levels were assessed through four evaluation components: postsession/CME survey, postprogram (retrospective) survey, focus group, and QI measure outcomes data. The Retrospective Survey format was based on 43 ECHO programs that have or are currently being implemented through the AAP Superhub. Participant experience and benefits were evaluated using the American Academy of CME Outcomes model. Evaluation of the didactics and presenters was shared with individual faculty on a regular basis to continuously inform improvement at the implementation level.

Focus groups were conducted with a subset of the participants with structured and open‐ended questions to learn more about practices' experience with the ECHO program.

3. RESULTS

Twenty‐two participants from seven pediatric practices across five states participated in the Epilepsy ECHO program, as shown in Figure 1. The practices recruited were small community practices, with a variable number and mix of providers. The number of providers ranged from 1 to 12 (mean = 4.4), and included physicians and advanced practice providers. Most practices used an electronic health record (EHR), but there was considerable variation in the type of EHR. Practices reported approximately 30% of patients with epilepsy. All practices had representation on each of the eight ECHO sessions, with an average session attendance of 14. The entire core QI team at the practice was encouraged to attend per practice‐level requirements; however, at least one team member was present for each session. All practices submitted chart review for the required eight data cycles. Baseline for each of the measures was determined after the first data collection cycle and is shown in Table 3. Of the participating physicians, eight were eligible for MOC and seven submitted MOC attestations to receive credit.

FIGURE 1.

American Academy of Pediatrics Epilepsy and Comorbidities ECHO practice teams. The flow chart visualizes the demographic information of the seven practices across five states (Florida, Maryland, North Carolina, Pennsylvania, and Texas) that were selected to participate in the Epilepsy ECHO program between April and December 2018. Across the seven practices, 22 individuals participated, representing physicians (MD), registered nurses (RN), physician assistants (PA), a certified registered nurse practitioner (CRNP), a licensed practical nurse (LPN), medical assistants, office managers, a case manager, and a research scientist (PhD)

TABLE 3.

Epilepsy ECHO quality improvement measures at baseline and after completion of eight ECHO sessions

Epilepsy measure (number of charts)	% target change	% baseline	% at ECHO conclusion	% difference	df	F statistic	P
Safety education documentation (479) a	30	30.9	72.5	41.6	42	2.40054532	.04
Documentation of antiseizure therapy side effects (479)	20	44.1	67.5	23.4	42	2.1835047	.06
Implementation of transition readiness plan (199) a	20	33.3	78.6	45.3	42	3.45366474	.005
Documentation of seizure frequency (479)	10	77.9	85	7.1	42	1.95708523	.09
Screening for mental/behavioral health (446) a	5	25.2	67.5	42.3	42	2.55402735	.03
Referral to epilepsy center (156)	5	73.3	100	26.7	42	0.88087066	.53
Counseling women during childbearing age (107)	5	36.4	28.6	−7.8	42	0.71401269	.66

^aMeasures demonstrating statistical significance.

Analyses of retrospective chart reviews conducted for the seven AAN measures revealed improvement in all but one measure, with statistically significant improvement in three measures (Table 3 and Figure 2).

FIGURE 2.

Epilepsy ECHO statistically significant quality improvement measures. The graph reflects statistically significant results (P < .05) for measures assessing safety education, screening for mental/behavioral health, and documentation of health care transition plan

Seizure frequency documentation was predicted to increase by 10%. The ECHO participants collectively achieved a 7.1% increase. Antiepileptic drug side effect intervention was predicted to increase by 5%, and a 23.4% improvement was identified. Safety education was predicted to increase by 30%, and a 41.6% improvement was identified, which was statistically significant. Screening for behavioral and mental health was predicted to increase by 5%, and a 32.2% improvement was identified, which was statistically significant. Counseling women of childbearing age was an optional measure predicted to increase by 5%, and a decline of 7.8% was identified. Referral to a tertiary center was predicted to increase by 5%, and a 26.7% improvement was identified. Documentation of a transition plan was predicted to increase by 20%, and a statistically significant 45.3% improvement was identified. Statistical process charts for each measure helped track changes over the course of the project (Figure 3).

FIGURE 3.

ECHO participant statistical process charts. Each chart represents a single quality measure that has been measured from a sample of patient charts from the practice population. The control charts show the value of the quality measure percent of documentation over time. Each chart contains a center line that represents the mean value for the in‐control process. The process dispersion of each characteristic measured falls within the specified confidence levels, expressed as the upper control limit and the lower control limit. The upper and lower control limit are calculated by µ ± 3 * σ

Focus groups conducted with the subset of ECHO participants (n = 7) provided insights into the specific ways that practices improved the care and management of CYE (Table 4). The focus group moderator used a semistructured interview guide to facilitate discussion. Open‐ended questions addressed three thematic areas—ECHO Experience (25 minutes), Putting ECHO Learnings Into Practice (Usability and Short‐Term Outcomes; 20 minutes), and Planning for Future ECHOs (5‐10 minutes). Data were content analyzed for key themes and concepts. The specific questions and content from the focus groups can be found in Appendix 1.

Although the number of participants in the sessions and cases discussed was small, there were trends to suggest that these sessions had a favorable impact on the participants. Postprogram (retrospective) survey, focus group, and QI measure data addressed levels 4 (Learning: Competence) and 5 (Performance) of the Outcomes Model. Participants reported statistically significant (P ≤ .02) gains in key areas of provider knowledge, skills, and confidence in epilepsy care. Specifically, improvements were demonstrated in areas of assessing comorbidities of CYE, managing medication side effects of CYE, and interpreting and applying QI data in practice.

Along with increased knowledge and confidence, changes were observed at the provider practice level. Overall, at follow‐up, more PCPs reported developing care coordination plans with the family and youth as well as sharing plans for CYE across providers and nonparent members of the care team.

Participants in the postprogram virtual focus groups valued didactic presentations for relevant information on topics of primary importance to providing CYE with high‐quality care. The content of several didactic presentations prompted reflection on current practice and implementation of changes in the clinical setting. Information conveyed through the lectures helped participants identify and address gaps in care for CYE. Focus group members appreciated the opportunity to view cases through professional lenses beyond their own. Interviewees also discussed the utility of presenting challenging and highly complex patients versus “bread and butter” cases more reflective of the typical patient with epilepsy likely to be seen in the primary care setting.

4. DISCUSSION

To improve quality of care for CYE, innovative models of care delivery are needed to address subspecialty physician shortages and improve care coordination in the medical home. Since its inception in 2003, Project ECHO has been employed to increase the knowledge, confidence, and self‐efficacy of PCPs in managing and comanaging subspecialty conditions globally across several medical conditions. ¹⁷ The Epilepsy ECHO replicated these successes, with participants reporting improvement in knowledge, skills, and confidence in caring for CYE. Other telementoring programs using the ECHO framework usually provide a case‐based learning environment for participants and provide CME credits to participants. However, the Epilepsy ECHO was the first of its kind to integrate QI processes and training into the ECHO model with the goal of furthering quality care for CYE in primary care practices.

In recent years, certifying medical boards in the USA have moved to a more comprehensive system, a more rigorous framework for maintaining board certification for physicians. This MOC system is a multicomponent and complex system and has been perceived as burdensome by physicians; the clinical relevance of MOC activities has also been criticized. ¹⁸ In the Epilepsy ECHO, outcome variables were developed using the 2014 AAN Measurement Set for Epilepsy (2017 Quality Measures were not released at the time of project development), making this activity even more meaningful and clinically relevant for participants. Not only were we able to provide CME credits, the project also provided MOC Part 2 and 4 points at no cost for ongoing physician certification to those participants who met requirements. The opportunity to earn MOC credit was seen as a benefit of ECHO participation, a sentiment shared during the focus groups.

This project demonstrated the feasibility of incorporating subspecialty quality measures into practice, with measure implementation reinforced through ECHO sessions. Combining didactic presentations and case‐based discussions, as well as the systematic QI methodology, led to project participants demonstrating improvement in six of seven quality measures. Three of these—seizure safety education, screening for mental or behavioral health, and implementation of transition readiness—showed statistically significant improvements.

In addition to the importance of increasing the knowledge base and confidence of PCPs, there is a practical reality. Since 2009, several studies and surveys of the child neurology workforce have demonstrated shortages of pediatric neurologists. ⁷ , ¹⁹ Therefore, innovative models of health care delivery must have a strong emphasis on real‐time physician education in a coordinated team‐based approach. Integration of QI into such models makes them more meaningful and clinically relevant for physicians and can translate into sustainable practice change over time. Over a short period of 8 months, the Epilepsy ECHO demonstrated significant improvements in three of seven outcome measures. Physicians found these variables feasible to implement. The practice teams also enhanced their QI skills (eg, chart review, PDSA cycles), enduring skills that the practices can utilize to implement quality metrics across all pediatric health conditions.

The framing, execution, and reporting of the CYE ECHO was developed along the principles of the SQUIRE 2.0 guidelines, as the purpose of this initiative was to improve access to quality health care for children and youth with epilepsy in their patient‐centered medical home. ²⁰ The format of this report was modified to better represent the structure of the project.

There were some limitations to this project. Despite the overall positive trend, these are aggregate data and may not reflect the individual practice data. Although the overall trend of the project demonstrated improvement in most of the outcome measures, smaller improvements were shown for outcome measures with higher baselines. The project team recognizes that the project methodology, incorporation of dedicated QI coaching, and rigorous requirements for chart review may have influenced the positive results. It is also possible that processes employed by the practices were better suited for improvement in some measures. The measure on counseling females of childbearing age failed to show improvement. This was an optional measure, and it is possible that PCPs leave this discussion to pediatric neurologists or need more knowledge in this area for improved discussions with their patients.

At the practice level, the project was time and labor intensive, particularly for smaller primary care groups with limited information technology resources and practice personnel. Workflow demands and competing agendas at the practice level made it challenging to dedicate time for QI activities. Differing electronic medical record systems across practices were also a challenge for uniform implementation. The project was not designed for analysis of each individual practice. Despite this, practices were monitored by the QI expert via QIDA and individualized recommendations were provided to bring about improvement. As this was a time‐limited project, the Epilepsy ECHO was not designed to measure sustained practice change after the program conclusion. The duration of the project did not allow for more longitudinal follow‐up, to assess whether the other measures could have also achieved statistical significance over a longer period of time. However, a retrospective survey to assess continued practice change among the participating practices 1 year after the project close is in development.

In conclusion, the Epilepsy ECHO demonstrated the feasibility of using epilepsy‐specific quality measures in primary care practices for meaningful change toward improving care for CYE. As practices embrace these methods to bridge knowledge gaps, it is possible to increase primary provider capacity to manage more aspects of pediatric epilepsy and make pediatricians partners in care coordination, streamlining referrals to specialists and potentially improving subspecialty care access. Combining ECHO, MOC, and QI coaching presents a promising model for advancing best practices and quality metrics across pediatric conditions. The unique combination of education, QI, and clinical care lends itself to sustainability and can act as an incentive for practices to adopt ECHO. Future studies would be needed to assess the impact on patient outcomes and health system efficiencies, including reduction in wait times for subspecialty appointments.

CONFLICT OF INTEREST

None of the authors have any conflict of interest to disclose. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines. Dr. Sachdeva is uncompensated for his role as Chief Scientific Officer for Virtual Pediatric Systems (VPS), LLC; however, his travel expenses for VPS related meetings were paid by VPS, LLC.

AUTHOR CONTRIBUTIONS

S.J. is a pediatric epileptologist and contributed to developing the ECHO curriculum, as well as the conceptualization, drafting, review, and revision of the manuscript. K.G. is a pediatric nurse practitioner and director of quality in distance health and contributed to the development and integration of quality measures and methods within collaborative practices. L.R. is an independent evaluation consultant and contributed to the evaluation of the Epilepsy and Comorbidities ECHO program as well as conceptualizing, drafting, reviewing, and revising the manuscript. A.S. is program manager for Project ECHO at the AAP and contributed to conceptualization, drafting, review, and revision of the manuscript. S.H. is manager of the National Coordinating Center for Epilepsy at the AAP and contributed to development and integration of quality measures and methods as well as conceptualization, drafting, review, and revision of the manuscript. T.C. is senior director of pediatric population health at the AAP and contributed to conceptualization, drafting, review, and revision of the manuscript. A.K. is program coordinator for the National Coordinating Center for Epilepsy at the AAP and contributed to review and revision of the manuscript. R.S. is senior vice president for quality and clinical affairs and a practicing pediatric intensivist and contributed to planning and implementation of Project ECHO and the ECHO Superhub at the AAP, and review and revision of the manuscript. L.B. is a pediatric epileptologist at the Children's Hospital of Philadelphia and contributed to developing the ECHO curriculum as well as review and revision of the manuscript. E.K. is a primary care pediatrician and founder of All About Children Pediatric Partners in Pennsylvania and contributed to developing the ECHO curriculum as well as review and revision of the manuscript. P.W. is an adult and pediatric rheumatologist, is codirector of the HRSA/MCHB cooperative agreement Got Transition, and contributed to developing the ECHO curriculum as well as review and revision of the manuscript. P.M. is president of the National Alliance to Advance Adolescent Health and contributed to developing the ECHO curriculum as well as review and revision of the manuscript. D.W. is a pediatrician and a member of the faculty in the Department of Pediatrics at East Tennessee State University Quillen College of Medicine and contributed to developing the ECHO curriculum as well as review and revision of the manuscript. E.‐L.N. is a psychologist and professor of pediatrics at University of Kansas Medical Center and contributed to developing the ECHO curriculum as well as review and revision of the manuscript. P.A. is a family engagement and training specialist for the children and youth with epilepsy and other seizure disorders program at Maryland Coalition of Families and contributed to developing the ECHO curriculum as well as review and revision of the manuscript.

Appendix 1. ECHO focus group interview guide

ECHO experience (25 minutes)

• 1We are going to spend a few minutes talking about the different components that make up an ECHO clinic. First please think about the brief faculty lectures offered during each clinic.

  1. In what ways have you used what you learn from the brief lectures for direct patient care and/or practice/system change?

  2. What could be improved about the brief lectures?

  3. In general, how relevant—or not—are the lectures to the issues of greatest interest or challenge regarding epilepsy and comorbidities in your clinical setting?

• 2Now please think about the case scenario presentations by clinicians (presentations by you and your peers).

  1. In what ways have you used what you learn from the case scenarios for direct patient care and/or practice/system change?

  2. What could be improved about the case presentations and discussions?

  3. In general, how relevant—or not—are the case scenarios to the issues of greatest interest or challenge regarding epilepsy and comorbidities in your clinical setting?

  4. Across ECHOs, one of the biggest pain points for hubs is getting participants to submit and present cases, so it is helpful for us to understand the barriers and challenges associated with that request.

     1. How was your experience submitting a case(s)? Probe: How did you feel about the MOC requirement for case presentation?

     2. What would make submitting a case easier?

Putting ECHO learning into practice (20 minutes)

• 3In what ways do you use what you learned from this ECHO clinic with your patients or clients?

Probe: To what degree are you able to apply concepts presented by others in the Epilepsy and Comorbidities ECHO sessions to patients with similar problems?

Probe: Please share an anecdote or example of a situation where the care of a child or family was directly impacted by knowledge or skill you acquired as a result of ECHO participation. As a reminder, please do not include any information that might be construed as protected health information in your comments.

• 4What, if any, practice changes related to epilepsy‐related care have you made as a result of your ECHO participation?

Probe: Policy? Workflow? Systems?

• 5What, if any, clinical or practice‐related problem or concern did you experience prior to involvement that was answered or resolved by your participation in this ECHO?
• 6In what ways, if any, do you use the quality improvement methodology you have learned through Epilepsy and Comorbidities ECHO in the practice setting?

Probe regarding experience with monthly huddles.

• 7Much of health care involves a team of caregivers who care for patients. Did others from your school or organization participate in the Epilepsy and Comorbidities ECHO clinic in which you participated?

If yes, probe in what ways that has been helpful.

If no, probe why and whether that would have been helpful.

• 8Who has shared something that they have learned through ECHO participation with a colleague?

Probe: Tell me more about that.

Probe: What facilitates or inhibits sharing information and practices from the Epilepsy and Comorbidities ECHO clinic with colleagues?

Future planning (5‐10 minutes)

• 9In what ways, if any, does the ECHO model provide value over other types of learning opportunities or training (such as face‐to‐face training, workshops, online learning programs)?

Probe: In what ways, if any, has the Epilepsy and Comorbidities ECHO created a sense of community around this topic? Facilitated networking?

What, if any, are the drawbacks or disadvantages of the ECHO model?

• 10What would you change, modify, or add to improve the Epilepsy and Comorbidities ECHO clinics?

Joshi S, Gali K, Radecki L, et al. Integrating quality improvement into the ECHO model to improve care for children and youth with epilepsy. Epilepsia. 2020;61:1999–2009. 10.1111/epi.16625