Development of pediatric acute care education (PACE): An adaptive electronic learning (e-learning) environment for healthcare providers in Tanzania

Abstract

Globally, inadequate healthcare provider (HCP) proficiency with evidence-based guidelines contributes to millions of newborn, infant, and child deaths each year. HCP guideline proficiency would improve patient outcomes. Conventional (in person) HCP in-service education is limited in 4 ways: reach, scalability, adaptability, and the ability to contextualize. Adaptive e-learning environments (AEE), a subdomain of e-learning, incorporate artificial intelligence technology to create a unique cognitive model of each HCP to improve education effectiveness. AEEs that use existing internet access and personal mobile devices may overcome limits of conventional education. This paper provides an overview of the development of our AEE HCP in-service education, Pediatric Acute Care Education (PACE). PACE uses an innovative approach to address HCPs’ proficiency in evidence-based guidelines for care of newborns, infants, and children. PACE is novel in 2 ways: 1) its patient-centric approach using clinical audit data or frontline provider input to determine content and 2) its ability to incorporate refresher learning over time to solidify knowledge gains. We describe PACE's integration into the Pediatric Association of Tanzania's (PAT) Clinical Learning Network (CLN), a multifaceted intervention to improve facility-based care along a single referral chain. Using principles of co-design, stakeholder meetings modified PACE's characteristics and optimized integration with CLN. We plan to use three-phase, mixed-methods, implementation process. Phase I will examine the feasibility of PACE and refine its components and protocol. Lessons gained from this initial phase will guide the design of Phase II proof of concept studies which will generate insights into the appropriate empirical framework for (Phase III) implementation at scale to examine effectiveness.

Purpose: This paper describes an overview of the development of our intervention, Pediatric Acute Care Education (PACE). PACE is an adaptive e-learning education intervention designed to increase healthcare providers’ (HCP) proficiency in national guidelines for facility-based care of newborns, infants, and children in the Mwanza Region of Tanzania. An HCP is any person responsible for providing clinical care at a Tanzanian health facility. This manuscript was structured according to recommended reporting guidelines for both intervention description^1,2 as well as development studies. ³

• (1) PACE Program Rationale. Ineffective in-service provider education limits delivery of evidenced-based care and contributes to avertable mortality. Pneumonia, birth asphyxia, dehydration, malaria, malnutrition, and anemia collectively cause over 4 million deaths in children under age 5 each year, and 50% of those deaths occur in Sub-Saharan Africa.^4,5 HCPs’ inadequate proficiency with evidence-based guidelines contributes significantly to poor care quality and increased child mortality in low and middle-income countries (LMICs).^1–5 The “know-do” gap in getting research discoveries into practice is greater in rural, under-resourced areas. ⁶ A key research gap of the World Health Organization is to identify effective provider education that extends across health systems.^7,8 In Tanzania's Mwanza region, the under-5 mortality rate is 88 per 1000 live births, nearly double the national rate (49 per 1000 live births) and over three times Tanzania's Sustainable Development Goal Target 3.2 of 25 per 1000 births.^5,9 In a recent multi-site clinical audit in Mwanza conducted by the Pediatric Association of Tanzania (PAT), only 58% of admissions (range: 48%–69%) had correctly documented diagnoses and only 57% correctly prescribed treatment (range: 43%–65%).^10,11 Interventions with increased reach that target improving HCP proficiency may improve care quality and lower child mortality.

  • (a) Limitations of conventional education. Conventional (in person) education is limited by reach, scalability, adaptability, and contextualization (Table 1).

    1. Conventional education has limited reach to rural HCPs. Pediatric acute care conventional education programs in Mwanza, Tanzania, Helping Children Survive and Essential Child Health Services, have had limited reach to rural HCPs. Our 184-provider survey across PAT's sites demonstrated that newborn or pediatric in-service provider education reaches only 60–70% of providers, with disproportionately less at community health center compared to zonal/regional hospital providers. This is consistent with our previous work in Botswana, where only 11% of community clinic HCPs and 0–29% of district hospital HCPs had ever received pediatric acute care conventional in-service education compared with 74% prevalence of referral hospital HCPs,^12–14 as well as other HCP conventional education programs in sub-Saharan Africa.^15–17 Possible reasons include 1) limited time for HCPs to attend education due to clinical demands 2) limited financial support to travel to centralized education events and 3) limited instructor, equipment, and space availability to train on site.
    2. Costs to deliver conventional education limit scalability. High HCP turnover seen in many LMIC health systems requires a scalable solution to be effective. ¹⁸ Costs of delivering pediatric acute care conventional education range from $31 to $90 per HCP per day.^19–21 As an example, high HCP turnover limits the effectiveness of the Emergency Triage Assessment and Treatment - Plus (ETAT + ), a 5-day conventional education adaptation of the WHO hospital-based pediatric emergency care course in Kenya. ²²
    3. Conventional education does not adapt to the learner. In Tanzania, there are many different cadres (professional types), with wide range of pre-service education, that provide clinical care to newborns and children. These include specialists and medical officers (5 years of education  +  1 year of internship); advanced degree nursing (i.e., nurse midwife, 3 years  +  1 year internship); nursing officers and clinical officers (3 years), assistant clinical officers, assistant medical officers (2 years), as well as assistant nursing officers, and medical attendants with only weeks or months of pre-clinical education. The bulk of the care is provided by junior medical officers/nurses, who have limited training and experience caring for children with severe illness. In attempting to have a “one size fits all,” programs often have fixed duration and cannot adapt to HCPs’ knowledge gaps and clinical needs, resulting in a “one size fits none.”^23,24 This may lead to either decreased effectiveness for HCPs with existing proficiencies or inadequate effectiveness for those with greatest need.^13,14,25 Additionally, without educational refreshers, proficiency decreases over time.^13,26
    4. Conventional education does not contextualize to the health system. Contextualization of HCP education is required to ensure it engages the learner and can impact care delivery. Facilities that provide acute care for newborns and children in Tanzania range from least resourced outpatient dispensaries, health centers, and district hospitals to greater resourced referral hospitals and zonal/national hospitals. Unfortunately, contextualization of conventional education is often “on the fly” i.e., during delivery, placing the burden of contextualization is on individual instructors or HCPs and often influenced by resource availability, which leads to incomplete and inappropriate education. ²³ Conventional education is not responsive to changes in local epidemiology or care quality issues and requires years or decades for publication cycles to update.^23,27,28 Finally, current conventional education is not responsive to system shocks, such as emerging epidemics, or natural disasters.^28,29

Table 1.

Conventional Education vs Adaptive E-learning Environments.

	Conventional	Adaptive
Reach
Barrier to training in rural facilities
Requires time off from clinical duties	✓
Requires minimum number providers for on-site training	✓
Requires internet and personal smart phones		✓
Scalability
Large fixed per provider costs	✓
Effectiveness and cost dependent on course duration	✓
Dependent on instructor quality	✓
Adaptability (Provider)
Targets:
minimal competency of (50-84% learning objectives achieved)	✓
proficiency (100% learning objectives achieved)		✓
Fixed amount of time (determined by course structure)	✓
Iterative education: continuously builds provider proficiency over time		✓
Adapts to individual provider’s schedule, knowledge base, grit, and metacognition		✓
Contextualization (Environment)
Burden of contextualization upon the learner or instructor	✓
Burden of contextualization on subject matter expert		✓
Updated content can be immediately sent to providers		✓
Content revisions able to be reviewed and approved by local stakeholders and subject matter experts		✓
Can occur continuously		✓
based on current response data		✓
based on current local epidemiology		✓
based on current local care delivery issues (system shock – staff turnover, supply chain, leadership changes)		✓
based as response to new evidence or changes in practice		✓
Effectiveness
Demonstrated improvements in	✓	✓
patient outcomes at 1 year	✓	✓
health professional behaviors at 3-12 months	✓	✓
provider skills at 0-12 weeks	✓	✓
provider knowledge at 0-12 weeks	✓	✓
Evidence of effectiveness in LMICs	✓

These limitations may explain why conventional education is not more effective at increasing provider proficiency. The median effect size of conventional education alone is only 7.3% (IQR: 3.6–17.4). Even under ideal conditions, combined with longitudinal self-study education or peer-to-peer education, the improvement in provider clinical care only improves by 24% and 25%, respectively.^20,26 These effect sizes remain modest in absolute terms given the low baseline clinical performance of 40%–60%.^5,20,25,30

• (b) Adaptive e-learning environments (AEE) in learning and AEE applications in healthcare. E-learning describes teaching and learning strategies based exclusively on the use of electronic media and devices as education, communication, and interaction tools. ³¹ AEEs, a subdomain of e-learning, use computer science and artificial intelligence principles to create a cognitive model to adapt education to each HCP. ³² There are three components to AEEs: the content (modular curriculum, learning resources, formative assessment), aggregate student data, and cognitive modeling (Figure 1). Usage data is used to create a cognitive model for each student and adjusts the sequencing of content and ratio of learning resources to formative assessments. There is indirect evidence to suggest that adaptive e-learning is at least as effective as conventional education in healthcare. No randomized controlled trials (RCTs) have compared adaptive e-learning to conventional education in LMIC health systems. Of 16 RCTs conducted in high-income countries, AEEs were as effective as conventional in-service education in improving HCP proficiencies and patient outcomes. ³³ This may indicate e-learning is a viable alternative where and when conventional education resources are not available.

• (c) Adaptive e-learning environments (AEE) may overcome the limitations of conventional education in LMICs.

  1. Potential to Increase Reach. AEEs may reach HCPs where equipment requirements and other impediments exist for conventional education.^34–36 Eighty-five percent of Tanzania's population is covered with at least a 3G network and 14% of the population has active mobile broadband subscribers. ³⁷ In our initial feasibility studies, all 21 medical and clinical officer interns had smartphones with data plans, and 19 reported a preference to use their mobile phones for e-learning over tablets or computers.^38,39
  2. Course delivery of AEEs encourages scalability. While course development costs are fixed, course delivery costs are minimal compared to conventional education. AEEs can be time-independent, allowing HCPs the needed educational exposure to achieve proficiency. E-learning does not require significant time off from clinical duties or travel, nor the need for minimum participants for classes. Further, web-based course delivery eliminates instructor quality variability.
  3. AEEs adapt to the learner. AEEs are iterative; they assess not only the HCP's correct responses but also their awareness of being correct/incorrect (metacognition), and ability to work through challenging topics (grit). Such software can determine optimal pacing and sequence of content continuously, building HCP proficiency and confidence over time. Moreover, AEEs analyze HCPs’ performance and knowledge gaps to serve up brief refresh packages repeated over time to maintain HCP proficiency. This calculation is individualized, and as HCPs use the technology regularly (e.g., monthly, quarterly), it becomes increasingly precise in determining what material to reinforce and when to support it.
  4. Contextualization is done during course design by content and context experts. Updated content can be released immediately to HCPs, and aggregate response data can be used to refine program content based upon HCP proficiencies and gaps. This creates the potential of a responsive HCP education that can both identify and address proficiency gaps at a local or individual level in near real-time, while maintaining the ability to disseminate new or revised evidence-based guidelines at scale.

• (2) Pediatric Acute Care Education (PACE) Description

  • (a) Preliminary PACE evaluation/data. To inform the development of PACE, we applied the principles of co-design over 18 months. We prioritized relationships and participation through iterative development, conducted several module experiments, and development capability strategy. Through CLN quarterly in-person meetings all stakeholders (PAT executive and continuing professional development committees, CLN external auditor team, CLN mentors and facilitators, and Stanford/Area9's educational design team) shared power to define program scope, roles and responsibilities and identify a strategy to develop capability to integrate into the existing infrastructure. Our co-design process led to three changes: 1) broadening PACE content, 2) creating content supervised by PAT subject matter experts and using national treatment guidelines as primary source materials, and 3) awarding continuing professional development credit.

    1. Broadened pediatric acute care content. Our original content scope was restricted to our Helping Children Survive conventional in-service education learning objectives (severe pneumonia, airway, oxygen therapy, severe dehydration/shock, triage/systematic assessment). PAT and Ministry of Health feedback led to expansion to 283 learning objectives over nine assignments and includes essential newborn care, sick newborn care, severe malaria, severe anemia, and severe malnutrition.
    2. Use PAT as subject matter experts for content development and Tanzanian national treatment guidelines as primary source materials. National guidelines represent the product of a contextualization of international guidelines by local subject matter experts. In Tanzania, the PAT represents the subject matter experts of Newborn and Pediatric Acute Care. Use of non-contextualized materials has been identified as a barrier to effective HCP education. ²³ By incorporating PAT supervision into content development, discrepancies were discovered prior to and reviewed to ensure local contextualization was accurate and appropriate. This alignment of content development with existing expertise infrastructure proved critical for adoption and ownership.
    3. Award continuing professional development (CPD) credit. The Medical Council of Tanganyika (MCT) is the national regulating body for medical, dental, or allied health professionals in Tanzania. Awarding CPD credit utilizes existing infrastructure to maintain provider's registration while fulfilling MCT's mission to “to improve performance, develop, maintain and update knowledge, skills and attitudes in order to provide safe, ethical, legal and effective care to patients, clients and community.” ⁴⁰
  • (b) Content: PACE contains over 283 discrete, measurable learning objectives (Supplemental Data A). It contains 9 assignments: essential newborn care, sick newborn care, severe pneumonia, airway management/oxygen therapy, triage/systematic assessment, severe dehydration/shock, severe malaria, severe acute malnutrition, and severe anemia (Table 2). The content is aligned with the PAT's Essential Child Health Services, ETAT/ETAT + , ²¹ and Helping Children Survive Essentials conventional education (a derivation of our Saving Children's Lives conventional education). ¹³
  • (c) Educational design theory: PACE uses Area9 Lyceum's Rhapsode™ AEE platform. Rhapsode™ is an adaptive e-learning platform that utilizes a combination of artificial intelligence, learning engineering, analytics, and learning technologies. Its theoretical framework is based upon Fadel's Four-Dimensional Learning Model, including knowledge, skills, character, and metacognition. ⁴¹ Content is organized as a learning objective, probe (i.e., assessment question), and learning resource. There is a 1:1 relationship between learning objectives and probes; however, more than 1 learning objective and probe could be associated with a learning resource. Learning objectives are organized using Bloom's taxonomy. Its content design is centered around Mayer's e-learning principles of multimedia, coherence, redundancy, signaling, spatial contiguity, temporal contiguity, modality, personalization, and segmenting. ⁴² Its algorithmic adaptivity is based on a biological model, not expert design, to tailor content navigation through the learning objectives. ⁴³ Content design includes assigning each learning objective as one of four types: 1) core – essential to have a basic understanding of the overall topic; 2) pass – required to adequately meet the learning goals of the overall topic; 3) excel – required to reach the standard of excellence in the overall topic; 4) extra – supplemental material not used in the adaptive algorithm. PACE's adaptivity algorithm continually adjusts the learner path and adjusts the probes (through shuffling the answers and rotating in/out distractors, for example) based on the accuracy of question response and learner confidence for each response. ⁴⁴ Content design also allows input as to what percentage of learning objective proficiency is required for initial learning completion.
  • (d) Educational Design Theory Related Decisions: Given the lack of data regarding provider proficiency in national guidelines in Mwanza, we elected our initial focus to be tier 1 (remembering) and 2 (understanding) learning objectives of Bloom's Taxonomy. ⁴⁵ We assumed minimal adaptive e-learning experience and attempted to minimize cognitive loading through standardizing learning resources layouts and limiting probes to multiple-choice (≥1 correct answers), rank, fill-in-the-blank, categorize, or match. We targeted each module to require between 10 and 30 min to complete, assumed a largely novice user group, which we estimated would take 1.5–2 min per learning objective, created modules with 8–19 learning probes. Only learning objectives that were determined by the subject matter experts as essential were created; as such, all objectives were assigned either core or pass, and module completion required 100% proficiency. Modules were grouped into assignments by topic themes, but there was no required order of modules after completion of the introductory module. Examples illustrating the educational design theory and learning principles are shown in (Supplemental Data B).
  • (e) Adaptivity subdomains. ³² Locally contextualized multimedia (images, audio, video) was used where possible to increase HCP engagement. PACE does not employ different tools/learning strategies for different types of HCPs.
  • (f) Activities: PACE's intended primary users are all HCPs who provide acute care to newborns, infants, and children at all facilities. HCPs are identified for participation, electronically screened for a minimum proficiency in English and active practice in pediatric acute care, and oriented in person by the PACE coordinator. An HCP's mobile device web access is verified, and supplemental data access is provided, if needed. HCPs log in to PACE using a secure connection and select their assigned modules from their learning dashboard. After completing PACE's introductory module, HCPs can choose from any of the 9 assignments listed in Table 2. Each assignment has 1–6 modules. HCPs work toward proficiency with each module and can stop or switch modules without loss of progress. Inactive HCPs are sent automated reminders through email, WhatsApp, and direct communication with the PACE coordinator.
  • (g) Personnel: The PACE coordinator is responsible for registering, providing IT and content support to HCPs, communicating with relevant stakeholders the aggregate or facility-specific progress. Stakeholders in this process include facilitators, mentors, hospital administration, regional and council health management teams, and the Pediatric Association of Tanzania leadership.

Figure 1.

Adaptive e-learning environments.

Table 2.

PACE content.

Domain	Learning Objectives
Essential Newborn Care	47
Sick Newborn Care	64
Severe Pneumonia	14
Airway Management/Oxygen Therapy	10
Triage/Systematic Assessment	48
Severe Dehydration/Shock	34
Severe Malaria	33
Severe Acute Malnutrition	15
Severe Anemia	18

Our PACE learning engineers are responsible for course content development. Learning engineers develop content with input from PAT subject matter experts and Area9 Lyceum education design team to ensure contextualized, high-quality content. Learning engineers are based in Tanzania and complete training in the concepts of adaptive learning and authoring in Rhapsode™ through self-directed learning modules and development of an adaptive e-learning module with remote supervision by Area9 Lyceum education design team.

All PACE program personnel require a medical (MD) or nursing (RN) degree and experience working in the Tanzanian health system. In addition, effective communication skills and, either formal or informal health education and/or IT skills, are required.

• (h) Mode of delivery: PACE is a web-based educational initiative, with its primary method of delivery being a HCP's smartphone, although Rhapsode™ also allows delivery via tablet and computer.

• (i) Integration with the Clinical Learning Network. PACE is integrated within The Pediatric Association of Tanzania's Clinical Learning Network (Figure 2).^10,11 Adapted from the Clinical Information Network in Kenya, ⁴⁶ CLN improves the quality of care for newborns, infants, and young children through three mechanisms: 1) health facilities’ clinical audit, 2) multi-level leadership root cause analysis and solution planning, and 3) solution implementation including effective dissemination and provider education. Dissemination and provider education is delivered through conventional education (Helping Children Survive and Essential Child Health Services). In addition, CLN provides weekly Grand Rounds lectures, and mentor outreach, and facilitator learning sessions. PACE is expected to strengthen CLN in two ways: provide HCP proficiency auditing for analyses and solution planning as well as utilize an adaptive e-learning platform to increase the reach and effectiveness of the Clinical Learning Network feedback dissemination and HCP education.

• (j) Dosing and Duration: PACE is self-paced and can be accessed anywhere internet is available. PACE will require an estimated 2 h/week during the first month, and 10–20 min/week up to 12 months.

• (k) Fidelity and Intervention Monitoring: Monitoring of progress is done at the facility and HCP level. Standardized reports (Supplemental Data C) are sent weekly to local facilitators, mentors hospital administration, regional and council health management teams, and Pediatric Association of Tanzania leadership on aggregate activity, progress, current strengths, and gaps in workforce proficiency. HCP monitor their progress through their individual dashboard in PACE. New learning objectives are identified quarterly through the CLN's clinical audit with revision of ineffective, inappropriate, or incorrect content.

• (3) PACE implementation and future program evaluation. The relevant outcome for PACE is HCP behavior change. To accelerate the development of HCP education implementation strategy that can improve child survival in low-income settings, we derived an empiric framework to define the basic elements of our behavioral intervention (HCP education) and developed a hypothesized pathway derived from the Obesity Related Behavioral Intervention Trials (ORBIT) model (Supplemental Data D). ⁴⁷ Our current approach is to conduct a set of phase I, (define and refine) behavioral intervention studies. Future proof of concept and pilot studies will determine if a clinically significant signal, a large enough change in HCP knowledge and skills, is observed (Phase II) to support moving to educational efficacy (Phase III) and effectiveness in clinical performance and patient outcomes (Phase IV). This framework features a flexible and progressive process, prespecified clinically significant milestones for forward movement, and return to earlier stages for refinement and optimization.

• (4) Limitations: Several barriers may limit program development.

  • (a) Internet connectivity and mobile phone access - The utilization of computers as an educational supplement, particularly self-study in-service education using computers, is recognized as a significant research gap in provider education in low- and middle-income countries due to the increasing accessibility of personal computers and internet-based education programs. ²⁶ Although 85% of Tanzania has at least 3G, and all our HCP pilot subjects own smartphones, the effect of geographical (rural) or functional (cost and accessibility) barriers on web-based education remains uncertain. As the growth of internet is exponential, this barrier is not expected to exist in the future.^48,49 If these issues prove to be significant, alternatives may include creation of a phone application to alleviate the need for continuous connectivity. Providing supplemental data plans to reduce the costs of internet access as a barrier is another alternative. ⁵⁰
  • (b) Learners’ privacy is protected as Rhapsode™ is GDPR compliant including in encryption, data security, and permanent data deletion, nevertheless, there is always the potential for data privacy breach.
  • (c) Inadequate in-person support: COVID-19 prevented the project team from providing direct project support to HCPs from February 2020 to October 2021. Attempts to enroll participants remotely were moderately successful with screening, but implementation and maintenance of PACE were poor.^38,39 Our current configuration attempts to minimize administrative infrastructure, but the optimal HCP cohort size per PACE coordinator is unknown.
  • (d) Currently, the capacity of our PACE content development team is inadequate for utilizing it as a means of ongoing quality improvement. We anticipate this will become feasible with further experience and revisions to our workflow.
  • (e) Lack of integration with skills training. Currently, PACE does not integrate hands-on practical skill training such as airway or fluid resuscitation management. Rhapsode™ Activity can incorporate skills assessments using either direct observation, and video review by an educator or self-review. These skills will continue to be delivered through conventional in-person education. Integration of blended learning may further increase its impact through use of mentor outreach and local facilitation through PACE personnel, CLN mentors, or peers.
  • (f) Lack of integration with provider performance. CLN's initial detailed clinical audit forms were piloted and adopted at the zonal hospital in 2019 and intended to be used for PACE provider proficiency as well as patient outcomes. CLN's clinical audit was discontinued in 2020 and never implemented at other CLN sites in part due to the national transition to an electronic medical record (EMR) which included the elimination of paper forms. The assumption that this data would be found in the EMR has not yet been realized. Currently, Mwanza's EMRs lack the structured format of the standardized pediatric admission forms to ensure pediatric-specific documentation on admission and during hospitalization is completed. Additionally, we are unable to obtain any degree of standardized output from the EMR CLN's existing limited clinical auditing prevents associating an HCP's PACE proficiency with their clinical proficiency. This limits optimal contextualization as well as adaptivity.

Figure 2.

Clinical Learning Network.

Discussion

In highly resourced settings, e-learning has been shown to be as good as conventional education. Non-adaptive eLearning programs have improved patient outcomes when compared to no intervention and are at least as effective as conventional education.^33,51–53 Adaptive e-learning, which builds on this by using response data to create models of individual learner pathways and adjusts sequencing and learning resources to optimize effectiveness, has shown moderate to large positive effects on both knowledge (SMD 0.70; 95% CI−0.08-  +  1.149, Z  =  1.76, p  =  0.08) and skills outcomes (SMD 1.19, 95% CI 0.59–1.79, z  =  3.88 p < 0.001) when compared to other educational methods. ³² Similar results have been seen in provider trainee ³² and non-provider populations.^54–57 Interestingly, using blended education that includes adaptive and face-to-face education in non-provider populations, achieves similar outcomes but 25% faster than adaptive approaches alone. ⁵⁸

In low- and middle-income countries (LMICs), where HCP in-service education resources are limited, adaptive e-learning may overcome this limitation, but studies are needed. The WHO Digital Guideline Group's priority research raised the following question “For professional health workers, does adaptive eLearning, compared to conventional education alone, improve [outcomes].” ⁸ Tuti et al. conducted a multi-center, multi-LMICs, RCT examining “adaptive immediate feedback” vs standard immediate feedback across HCPs who provide clinical care to newborns and showed no significant difference. The adaptivity was significantly limited by its logic model (Bayesian vs more complex logic models incorporating metacognition and response time), functionality (used to determine level of detail of feedback vs. adjusting course content and sequencing), and granularity (feedback of an entire module/scenario vs individual objectives). ⁵⁹ Notably, Tuti had completion rates of 43%, higher than completion rates of massive open online courses (MOOC) of 7%–15%.^60,61

To determine if adaptive e-learning is effective in improving care quality, its implementation strategy needs to be defined and refined. It is crucial to determine which aspects of the implementation strategy are acceptable to stakeholders and which need to be revised to normalize adaptive e-learning by HCPs. Aspects of the intervention design that were acceptable to the stakeholders included the use of principles of co-design, plan to integrate with CLN's clinical auditing and multilevel stakeholder engagement, its simple and easy user interface, and content sizing. The value of co-design principles and multilevel stakeholder engagement have been identified in previous interventional development studies,^62,63 including the Clinical Learning Network. Use of CLN's infrastructure to develop and explicitly describe health system leaders’ and HCPs’ defined roles during the Phase I studies of PACE, obviated the need for an additional external leadership infrastructure, while simultaneously providing actionable work within CLN aligned with its overall goals. PACE modules capitalized on an easy-to-use interface using established Mayer's e-learning principles, right sizing of content, Bloom's Taxonomy, and established local guidelines. These factors combined with minimal infrastructure and disturbance of clinical care, were key acceptability drivers for us and seen previously.^63,64 It is important that e-health interventions have a positive impact on clinical management to be successful, ⁶⁵ and aligns with our use of clinical auditing as long-term measurement goal. Knowledge assessments should be used as an intermediate marker of PACE efficacy when linked to provider proficiency and patient outcomes. By utilizing existing quality improvement metrics as indicators of education effectiveness, the requirement for test development, implementation, and monitoring is eliminated, reducing the need for additional data collection and management.

We used principles of co-design and feedback from stakeholders over time to refine our empiric implementation strategy. Aspects of the intervention design that needed revision prior to pilot testing included our content development workflow, local capacity to develop content, and our support for e-learning. Our initial content development workflow was led and developed externally and was presented for CLN approval. Through serial stakeholder meetings, feedback, and collaboration with PAT, CUHAS and governmental leadership, PACE program transitioned to a facilitator of CLN priorities, supervised by local subject matter experts. This expansion of PACE allowed it to better address local priorities, leading to an increase in stakeholder participation in content development. Approval was also shifted to subject matter experts within PAT. This greater involvement throughout all stages of development led to fewer revisions as content was approved during its creation.

A major component of PACE implementation strategy is local capacity to develop education content. Local capacity is a known bottleneck to e-learning.⁷¹ Local educational design and implementation capacity is an intrinsic component of PACE which allows for optimal contextualization. We initially underestimated the amount of effort and supervision required. Our approach was altered to have PAT assume the responsibility of recruiting, supporting, and supervising learning engineers. Currently, our approach involves utilizing four senior PAT members who volunteer as learning engineers. More work is needed to determine the optimal local capacity framework for learning engineering.

Finally, we identified that in-person support to providers is needed and the predominant communication with our stakeholders is text messaging, not e-mail. Our initial support for HCP was remote and largely email-based. Through early piloting, we identified that remote orientation to PACE was insufficient, and most communication is done via WhatsApp. Consequently, the position of PACE coordinator was established to provide on-site orientation to PACE and follow-up. Additionally, a unique integration between WhatsApp and our REDCap database was developed to monitor provider follow-up.

Implications for future research and sustainability

Adaptive E-learning Environment Enhanced Learning Health Systems hold tremendous promise to address the HCP education gaps that currently limit the provision of high-quality care in LMICs. The potential of AEEs to scale up, increase reach, and adapt to systems may drive this paradigm shift. We are currently in phase I (define and refine) are using a mixed methods approach to determine feasibility and normalization of our current methods. Once refined, Phase II (proof-of-concept) will allow us to determine if there is a clinically significant signal seen through a clinical audit. Phase III and IV studies (efficacy and effectiveness) will enable us to determine if provider education improves care, patient outcomes, and cost-effectiveness. Future studies will clarify important characteristics of education content, the education environment, work environment, and provider characteristics. This will allow optimization of efficiency and effectiveness of in-service provider education for all types of providers in all types of settings.

PACE's current sustainability is dependent on both support for operation and development, as well as research. Long-term funding sources may be influenced by its utility to the HCPs and the health systems it serves. PAT has incorporated PACE permanently into its national Continuing Professional Development offerings. HCPs may provide support for PACE if it proves a reliable mechanism for continuing professional development. Finally, health systems may employ the integrated PACE CLN model to demonstrate commitment to quality and maintain accreditation.

Conclusion

HCP proficiency is critical to improving patient outcomes, and adaptive e-learning environment may overcome intrinsic barriers to conventional education in low- and middle-income countries. We used principles of codesign to develop PACE to improve provider proficiency in newborn, infant, and pediatric care in Mwanza Tanzania. PACE currently includes 283 learning objectives over 9 assignments. Future research and training include the completion of a phase Ib study to refine PACE for strength and efficiency. We will capitalize on our flexible and progressive development framework to develop an adaptive e-learning in-service provider education implementation strategy that improves provider proficiency, patient care quality, and ultimately newborn and child survival.