Evaluation of an innovative model for leadership and management development among national immunization program teams

Abstract

Objectives

Investments in leadership and management are essential to the achievement of Gavi's global immunization goals. The onset of COVID-19 created obstacles for management interventions that relied on travel, convenings, or in-person coaching. Evidence-based adaptations were required to continue strengthening the leadership and management of national immunization programmes in times of rapid change. To understand the feasibility, acceptability, impact, and cost of innovative models of leadership and management development, we sought to evaluate the COVID-adapted Expanded Programme on Immunization Leadership and Management Programme (EPI LAMP).

Study Design

Consistent with the Kirkpatrick model for training program evaluation, we used a mixed-methods longitudinal evaluation.

Methods

We used program administration data, repeated delegate surveys, and exit interviews to describe EPI LAMP at three levels: engagement (participation and satisfaction), learning (change in competencies), and impact (application of new competencies in the workplace). We also assessed the financial and environmental costs of the program.

Results

The adapted programme engaged 11 countries and achieved a 94% graduation rate. 100% of delegates would recommend EPI LAMP to their peers and over 90% of delegates agreed the course met the learning objectives. Delegates demonstrated significant improvements in management and leadership competencies, and breakthrough projects improved immunization programme performance. The adapted model delivered much lower financial cost and environmental impact compared to the traditional model.

Conclusion

The adapted EPI LAMP model may be a superior approach to leadership development in LMIC settings, even as pandemic-related restrictions recede.

1. Introduction

Since 2020, national Expanded Programmes on Immunization (EPI) in low- and middle-income country (LMIC) settings have been challenged to both introduce new COVID vaccines and maintain lifesaving routine immunization services in the context of disruptions associated with a global pandemic [1]. Investments in leadership and management within national EPI programmes has been essential to the achievement of global immunization goals set by GAVI, the Vaccine Alliance, a global public private partnership with the goal of equitable and sustainable use of vaccines [2]. However, COVID-19 created obstacles in the continuation of proven management interventions that relied on travel, convenings, and in-person coaching [[3], [4], [5], [6]]. Rapid, evidence-based adaptations were required.

The Expanded Programme on Immunization Leadership and Management Programme (EPI LAMP) represented a flagship program in Gavi's leadership, management, and coordination (LMC) portfolio [7]. However, EPI LAMP's operating model was untenable at the onset of COVID-19 due to travel restrictions that prevented in-person gatherings despite high demand from Ministries of Health. Prior to COVID-19, the uptake of fully virtual training programs had been limited in Africa, despite the potential of such programs to support geographically isolated health workers and overcome expensive “fly-in/fly-out” models that are common in these settings [8,9]. Innovative virtual programs for medical education were developed during COVID-19 [[10], [11], [12]]. However, despite the role of adaptive leadership in times of rapid change, we found only one empirical evaluation of leadership development interventions delivered in the context of a national or global pandemic (ref in press), and none that offered direct comparison with pre-pandemic models of programme delivery.

Therefore, we sought to evaluate the feasibility, acceptability, and impact of the COVID-adapted EPI LAMP model using a mixed-methods design, with direct comparison to the evaluation results from the pre-COVID program model. The lessons learned from this evaluation of a fully virtual workforce development program in LMIC settings during times of rapid change could have implications for global health workforce development beyond the COVID-19 era.

2. Methods

2.1. Partners

As described previously [7], the EPI LAMP consortium was led by Yale's Global Health Leadership Initiative (Yale) in partnership with PATH, the University of Global Health Equity in Kigali, Rwanda (UGHE) for English-speaking teams, and the University of Yaoundé I (UYI) and the Department of Disease Control, Epidemics and Pandemics in Yaoundé (DLMEP) of the Cameron Ministry of Public Health for French-speaking teams. The programme was funded by a grant to Yale from Gavi, the Vaccine Alliance.

2.2. Participants

Since 2018, EPI LAMP has reached five cohorts (27 national immunization teams, 102 alumni), including two cohorts under the adapted model (11 countries, 46 alumni). The process for identification of countries and participants (called “EPI LAMP delegates”) has been described previously [7].

2.3. Core curriculum

EPI LAMP was designed as a practical programme for working healthcare professionals [7]. The core curriculum included six modules: strategic problem solving, leading effective teams, political engagement, human resource management, financial management, and supply chain management. The traditional model included 60 h of asynchronous learning on the Canvas™ platform, three one-week forums (in Rwanda or Cameroon), and a team-based breakthrough project. The breakthrough project addressed a pressing challenge or problem in the country's EPI programme, aligned with national strategic priorities. The challenge was defined by each country delegation in consultation with EPI program leadership, MoH leadership and Gavi upon recruitment into the program, and refined for clarity as part of the program itself. The coaching model (previously described [7]) included management coaching on data-driven problem solving from Yale faculty who were experts in leadership and management, technical coaching from PATH senior program officers who were experts on real-world issues in vaccine policy, access, and introduction, and support from a Yale learning coach who tracked and supported progress through the program. Management and technical coaches were assigned to a delegation for the duration of the program based on the topic of their breakthrough project. The learning coach served across all delegations. Delegates that completed all program components were awarded a certificate from Yale, PATH, and the regional implementation partner(s).

2.4. Adaptation

To adapt EPI LAMP to COVID-19 travel and convening restrictions, and to ensure that the program was relevant to the COVID-19 response, we integrated information from a literature review and key informant interviews. First, we synthesized published best practices for professional online education by consulting the Yale Poorvu Center for Teaching and Learning to identify common evidence-based adaptations and challenges, followed by a review of the scientific and grey literature related to these topics. Specifically, we adapted an existing framework for online education for executives [13], and worked in consultation with consortium partners to identify multiple ways to build a remote learning community for executives [14].

Second, we conducted a situational assessment that used in-depth interviews with Gavi Senior Country Managers, Ministries of Health, and potential EPI LAMP delegations to understand changes to work life due to the COVID pandemic, use of telecommunications, and workload (see Supplemental File A for the situational assessment interview guide). While responses varied slightly by country context, the situational assessment revealed high levels of participant demand for an adapted EPI LAMP program. EPI teams had been successful in continuing their routine operations either in-person or via online platforms and had adequate flexibility in their schedules to accommodate the EPI LAMP program. EPI teams had access to reliable internet assuming they were provided with stipends for high-speed internet bundles, and we subsequently allocated funds to country teams to cover internet access charges.

Based on the assessment, we maintained the core curricular content, but adapted discussion prompts and case examples to reflect issues related to COVID-19 response. We adapted programme timing and sequencing, breaking the core curriculum into 9-months of weekly cycles that included e-learning assignments, synchronous sessions via Zoom [15] (2-4 h per week), and coaching (see Supplemental File B for a graphic of the traditional and adapted programme). The Zoom sessions required investment in 6 facilitators that moderated breakout rooms to ensure the understanding and promote application of core content. The coaching component was updated to include a remote management coach and learning coach, and where feasible an in-country technical coach. When an in-country coach was not available, a remote technical coach from with specific country experience was assigned. At-least-monthly coaching sessions used Zoom®, WhatsApp®, and in-person meeting (when available).

2.5. Evaluation design

We conducted a mixed-methods, longitudinal evaluation of EPI LAMP. Consistent with the Kirkpatrick model [16] to assess the effectiveness of training programs, we described EPI LAMP at three levels: engagement (participation and satisfaction), learning (change in management and leadership competencies), and short-term impact (application of new competencies in the workplace). We also developed estimates for the financial and the environmental costs of the program. This study describes the results associated with the adapted model and then provides a direct comparison with the results associated with the traditional model (published previously) [7].

2.6. Data collection and analysis

We used data from five sources. First, we used administrative records to monitor completion of course requirements and graduation rates, disaggregated by cohort and gender. Administrative records were also the source of the information required for the cost estimates described below. Second, at the beginning and end of the program, each delegate completed a web-based self-assessment of a set of individual management competencies for global health that were adapted for national EPI programmes [7]. Each competency was measured using a 5-point likert-type scale: no knowledge; some knowledge, but unable to perform; able to perform with assistance; able to perform independently; and able to mentor others. Using a 95% confidence interval, we tested for a significant change in management competencies by domain using paired t-tests. Third, at several milestones in the program, country teams presented on their breakthrough projects to define and address a complex challenge within their home EPI systems and were assessed by coaches and faculty on their work. The presentations were summarized to form a description of the measurable objectives and outcomes of each breakthrough project. Fourth, delegates completed a voluntary, anonymous, web-based exit survey soliciting feedback on the programme (Supplemental File C). Fifth, delegates were invited to provide qualitative feedback on their experiences using a voluntary open-ended survey, Zoom-based interview, or WhatsApp® voice memo using an open-ended guide (Supplemental File D). Interview recordings and voice memos were professionally transcribed, and the transcriptions were checked for accuracy by the research team. We used the constant comparative method [17] to conduct line-by-line review of the resulting qualitative dataset, coding of blocks of text using a code sheet that included a priori codes based on evaluation goals. Coders met regularly to discuss their observations, reconcile discrepancies in coding through negotiated consensus, update the code sheet to include emergent constructs. The resulting code reports were reviewed by the research team to identify emergent themes, with attention to seeking out negative feedback and disconfirming evidence.

Financial costs were estimated based on the actual expenses incurred against a grant awarded from Gavi to Yale for each cohort. This grant covered the direct costs associated with program delivery, including salary support for programme staff, faculty, and coaches; travel costs associated with the in-person convenings; telecommunications and supplies; and indirect costs based on Gavi policies for university grantees. The programme was delivered at no cost to delegates. Expenses associated with breakthrough project implementation were estimated by the delegates themselves, mobilized as part of their training on financial management and resource mobilization, and excluded from this cost analysis.

The carbon footprint was estimated based on flight miles, hotel nights, and ground travel miles as the inputs [18], converted into MT CO₂E using a set of conversion factors. The conversion factor for commercial air travel was the average for short and long flights (0.0002 MT CO₂E per flight mile) as calculated by carbonfund.org [19]. The conversion factor for a hotel stay in Rwanda (0.1115 MT CO₂E per night) and Cameroon (0.0975 MT CO₂E per night) was obtained by using the Greenview Hotel Footprinting Tool [20]. The conversion factor for in-country transportation around the conference venue was obtained using data from carbonindependent.org [21]. The conversion factor for live sessions via zoom was 0.157 MT CO₂E per hour) [22]. Carbon footprints from delegates and programme faculty/staff were calculated separately.

2.7. Ethical considerations

This evaluation was determined to be exempt from continuing review by the Yale University Institutional Review Board.

3. Results

3.1. Engagement and retention

The adapted model engaged 11 countries (Table 1). The 46 alumni were 41% female and 59% male. Despite the anticipated challenges associated with virtual delivery and the competing priorities of the COVID-19 response, retention was 94%, compared to 86% for the traditional model.

Table 1.

Description of the sample for EPI LAMP Cohorts IV and V.

	Cohort IV	Cohort V	Total
Countries	Ghana, Kenya, Malawi, Somalia, Uganda	Central African Republic, Comoros, Djibouti, Madagascar, Mauritania, Togo	-
Language	English	French	-
Dates of delivery	Aug 2020 – May 2021	Sept 2020 – June 2021	-
Enrollment and completion
Enrolled	24	25	49
Graduated	22	24	46
Graduation Rate	92%	96%	94%a
Gender of alumni
Female (%)	11 (50%)	8 (33%)	19 (41%)
Male (%)	11 (50%)	16 (67%)	27 (59%)
Participant evaluation
Feedback survey	20	22	42
Competency assessment	22	24	46
Qualitative exit feedback	14	18	32

^aRetention in the traditional EPI LAMP was 84% compared to 94% in the adapted EPI LAMP model.

3.2. Delegate satisfaction

All delegates reported that they would definitely recommend the adapted model to their peers, consistent with the levels of satisfaction measured in the traditional EPI LAMP model [7]. Additionally, delegates agreed that the course met the outlined learning objectives and provided positive feedback on programme elements (Table 2), expressing greatest satisfaction with the schedule, mentoring and coaching, and live sessions.

Table 2.

Delegate feedback on the program (n = 42).

Progress toward program objectives	Strongly/Agree	Neutral	Strongly/Disagree
The members of our EPI LAMP team have built skills and relationships that will allow us to work effectively together.	91%	7%	2%
I have developed leadership skills that will allow me to create and utilize effective, inclusive teams.	95%	2%	2%
I feel prepared to complete the entirety of the strategic problem-solving process with my team	95%	2%	2%
Satisfaction with program components	Strongly/Satisfied	Neutral	Strongly/Unsatisfied
Program schedule	98%	0%	2%
Program facilitation	93%	5%	2%
Program content (asynchronous material on Canvas)	88%	10%	2%
Program content (live sessions on Zoom)	95%	2%	2%
Mentoring and coaching	98%	0%	2%

At the end of the program, delegates provided positive qualitative feedback on the acceptability, accessibility, and relevance of the adapted model for EPI LAMP. They reported that they were able to build strong relationships with programme faculty, coaches, and other country delegations, backed by feasible approaches to engagement.

What I enjoyed was the direct contact with the facilitators. Here you interact directly with the teacher and it’s like being in a classroom. That’s what makes it special. When I say goodbye, it’s just like you are in a classroom. (Togo)

It was exciting. At first, I thought it would be easy not to actively participate in live sessions, but this was not the case due to how we were getting engaged by the facilitators. And the breakout sessions left us with no choice but to actively participate. (Malawi)

It was easy. It was just a maximum two hours of two hours a week for such an authentic course. People are doing that for years to get that knowledge. But if you’re doing it for a few hours in a week, and you have a team that you’re working with at your work, it’s really very feasible compared to other adult learning platforms. (Uganda)

Delegates also reported that the course content was relevant to the real-world challenges and priorities they were facing, including in the context of COVID response.

This program allows learners to customize the leadership and management skills to their country’s context in the course of learning. (Kenya)

EPI LAMP allowed us to appreciate the complexity of management, the principles of leadership and management, teamwork, and accountability and made this situation [COVID] normal for me … to maintain our sanity when we’re working. The EPI LAMP principle assures you that as long as you engage, you provide information, you coordinate, problem solving is a continuous process. (Uganda)

They were arranging and bringing guest speakers from COVID-19, and of course the use of practical examples of COVID-19 when we were doing some particular (modules). I think that was great. (Malawi)

Beyond the curricular content, delegates described how EPI LAMP helped build essential professional skills to aid in the new ways of working after the introduction of COVID-19.

Remote meetings are really new in our country’s administration. And we did that from start to finish. And today, we been equipped for that. (Togo)

3.3. Change in management competencies

Delegates demonstrated a statistically significant increase in competencies across all eight domains (Fig. 1). The largest improvement was observed in the domains of community and customer assessment and engagement, followed by strategic problem solving and human resource management.

Fig. 1.

Longitudinal change in competencies by domain across two cohorts (n = 46).

A comparative analysis of changes in management competencies between the traditional model (published previously [7]) and the adapted model shows that delegates in the adapted model made greater gains in the domains of human resource management, operations management, and performance management (Fig. 2), and commensurate gains in other domains.

Fig. 2.

Magnitude of longitudinal change in competencies associated with the traditional EPI LAMP model (n = 51) and adapted EPI LAMP model (n = 46).

3.4. Impact on EPI program performance

Delegations' presentations of their breakthrough projects demonstrated that they had defined a problem and set a corresponding objective, completed a root cause analysis that synthesized multiple sources and types of data, and developed a strategy to address the focal root cause. By the end of the programme, all delegations were in the process of implementing their strategy and monitoring process toward their objective. Delegations focused on diverse challenges related to continuation of routine immunization services, especially for populations experiencing poor immunization coverage. Strategies included strengthening team management (i.e., increasing Penta 3 coverage in Ghana's Volta region by 5% through supportive supervision), improving data quality (i.e., reducing the percentage of facilities with data mismatch from 65% to 35% in the Southern Region of Malawi), integrating immunization services into existing health infrastructure (i.e., increasing Penta 3 coverage by 10% among nomadic populations in Somalia through a one health initiative; increasing private clinic immunization services by 10% in Togo through political advocacy), and improving immunization surveillance (i.e., increasing the number of health centers that report AEFI by 30% in Djibouti). Supplemental File E describes each breakthrough project.

3.5. Financial and environmental cost

The biggest difference between the traditional and adapted models was the cost of the program (Table 3). The average cost for the traditional model was $35,800 per alumnus and for the adapted model was $20,200 per alumnus, a cost savings of over 43%. The average carbon footprint per cohort was 160 MT CO₂e for the traditional model and 0.012 MT CO₂e for the adapted model.

Table 3.

Financial and environmental costs for EPI LAMP program models.

	Traditional model (Cohorts I – III)	Adapted model (Cohort IV – V)
Average cost for one EPI LAMP cohort	$645,475	$464,200
Average cost per alumnus	$35,800	$20,200
Average carbon footprint per cohort
Faculty and staff	59 MT CO2e	0.33 MT CO2e
Delegations	101 MT CO2e	0.21 MT CO2e
Total	160 MT CO2e	0.55 MT CO2e
Average carbon footprint per alumnus	3 MT CO2e	0.012 MT CO2e

4. Discussion

We describe results associated with the adapted EPI LAMP model, a virtual leadership and management program which engaged national teams from 11 Gavi-eligible countries in Anglophone and Francophone Africa. Despite the demanding nature of the program and the competing demands associated with responding to a global pandemic, the adapted programme achieved high levels of engagement and completion (94% graduation rate), supported by high levels of delegate satisfaction with the quality and the relevance of the program. These findings demonstrate a strong demand for leadership and management support for national immunization professionals, even in challenging times.

Delegates demonstrated significant improvements in leadership and management competencies and applied these new skills to shift ways of working in their national immunization programs. Gains were greatest in the domains of community and customer assessment and engagement, strategic problem solving, and human resource management. This is consistent with the programmed curriculum, which emphasized the leadership and management skills essential for the response to COVID-19. The adapted model was associated with greater gains in human resource management, operations management, and performance management compared to the traditional programme model. We hypothesize that this is due to the more sustained weekly support provided to delegates in the adapted model, compared to the three 1-week intensives included in the traditional model.

Through breakthrough projects, country delegations demonstrated application of their new competencies to improve immunization programme performance. Delegations described engagement with diverse stakeholders to conduct a root cause analysis, integration of multiple types of data to understand complex challenges, and development of solutions tailored to the specific country context. As delegations implemented their strategies, they demonstrated new ways of engaging in the political landscape, motivating their health workforce, and mobilizing resources, overcoming barriers to program implementation even in the context of a global pandemic. This is consistent with emerging literature showing that fully virtual models of leadership development can be associated with short-term impact on health systems performance (Ref in press).

Overall, the adapted model was associated with higher retention rates, greater improvements in competency, comparable and meaningful progress on breakthrough projects to improve system performance, and far lower financial and environmental cost compared to the traditional model. Additionally, no countries declined participation in the adapted model due to infrastructure concerns, and we reached delegations that had been unable to travel for the traditional model due to political instability and conflict (i.e., Somalia), signaling that the adapted model is advantageous from an equity perspective.

Our results should be interpreted in light of several limitations. First, changes in competencies were based on self-assessments, which are subject to social desirability bias and are not corroborated by the assessment of others in the organization. However, gains were consistent across operating models, countries, and cohorts, and were substantiated by descriptions [[23], [24], [25]] of how delegates had applied their new skills in the workplace through their breakthrough projects [26]. Second, although delegations demonstrated meaningful progress on their breakthrough projects during the project period, a longer period of observation would be required to determine whether implementation was associated with improvements in their outcomes of interest, or whether the changes made have been sustained. However, all teams achieved novel insights into their problem and engaged with common sources of friction associated with complex adaptive challenges, strengthening likelihood of success as compared to traditional quality improvement interventions.

Strengthening leadership and management is critical to achieving global immunization targets and can be significantly improved through participation in a team-based program. Virtual programs, such as the adapted EPI LAMP model, can generate high levels of engagement and retention, improvements in competencies, and impact on immunization system performance. These benefits are delivered at lower financial and environmental cost, increasing overall return on investment. The results of this study may be useful to educators, practitioners, and government officials seeking to strengthen EPI programme performance, as well as policymakers and global partners seeking to invest in evidence-based management and leadership development models.

Contributors

KB directed the delivery and evaluation of EPI LAMP, conceptualized and drafted the manuscript, and integrated coauthor inputs. LI managed the implementation of EPI LAMP; supported data collection, analysis, and synthesis for the evaluation; and drafted sections of the manuscript. AM served as a co-principal investigator for EPI LAMP with essential input into all aspects of program design, delivery, and evaluation. She also provided critical review of the full manuscript. VB and RW served as core faculty for EPI LAMP, providing critical review of the full manuscript. EL served as Principal Investigator for EPI LAMP providing oversight of funding, design, strategy, and evaluation. All authors reviewed and approved the final manuscript.

Ethics statement

This study was reviewed and deemed to be exempt from continuing review by the Yale University Institutional Review Board.

Funding

EPI LAMP, including the embedded evaluation described in this manuscript, was funded by a grant from Gavi, the Vaccine Alliance to Yale University. All authors have attested that they have no competing interests to declare and have approved the final manuscript for submission.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: