The Benin Experience: How Computational Modeling can lead to Major Vaccine Policy Changes in Low and Middle Income Countries

Abstract

While scientific studies can show the need for vaccine policy or operations changes, translating scientific findings to action is a complex process that needs to be executed appropriately for change to occur. Our Benin experience provided key steps and lessons learned to help computational modeling inform and lead to major policy change. The key steps are: engagement of Ministry of Health, identifying in-country “champions,” directed and efficient data collection, defining a finite set of realistic scenarios, making the study methodology transparent, presenting the results in a clear manner, and facilitating decision-making and advocacy. Generating scientific evidence is one component of policy change. Enabling change requires orchestration of a coordinated set of steps that heavily involve key stakeholders, earn their confidence, and provide them with relevant information. Our Benin EVM+CCEM+HERMES Process led to a decision to enact major changes and could serve as a template for similar approaches in other countries.

The Problem

While scientific studies can show the need for vaccine policy or operations changes, translating scientific findings to action is a complex process that needs to be executed appropriately for change to occur. For example, our previously published study utilized a computational simulation model of the Republic of Benin immunization supply chain, generated by our HERMES (Highly Extensible Resource for Modeling Supply Chains) software platform, to demonstrate the potential benefits of redesigning the Benin immunization supply chain (i.e., the series of locations, storage devices, vehicles, personnel, and processes involved in getting vaccines from the Central location in the country to the population) [1]. However, the study did not describe the efforts and processes that enabled the modeling to lead to policy change. What follows are some of the key steps and lessons learned in our Benin experience that helped computational modeling inform and lead to major policy change.

Key Steps

Key Step 1: Engagement of Ministry of Health

Figure 1 shows a timeline of the project, which emerged from the initiation of the LOGIVAC project by the Agence de Médeicine Préventive (AMP) and World Health Organization (WHO) endeavoring to establish a regional training and reference center for health logistics at the Institut Régional de Santé Publique (IRSP) in Ouidah, Benin. A vital component of the LOGIVAC Project was having a substantial in-country presence. Key members of the LOGIVAC Team (e.g., H. Dicko and M. Avella) resided permanently in Cotonou, the capital of Benin. Having an in-country presence enabled the LOGIVAC Team to develop relationships with the Ministry of Health (MoH) and WHO Expanded Program on Immunization (EPI) team in Benin. Part of the LOGIVAC project focused on identifying innovative supply chain solutions that can be scaled up in Benin, which completed an extensive cold chain equipment survey in 2010. Formal engagement with Benin decision-makers began in December 2011 and continued throughout the entire process (Figure 1). Early and frequent engagement with country officials is essential. They must be involved in, and drive, the decision-making process, since they are ultimately the ones that must accept and implement decisions. Country ownership of decision-making and solutions will maximize the probability of acceptance. Potential system modifications and solutions should come from, or have, strong input from the country stakeholders instead of solely from external parties that may not fully understand all ramifications of their implementation plans. The goal of this procedure was not to make decisions for the country, but rather provide the country with tools to facilitate the country’s decision-making.

Figure 1. Overview of the EVM + HERMES implementation in Benin.

1) The EPI department informed AMP of its immediate needs (mainly related to insufficient cold chain storage capacity throughout the country, in particular, at the central level). The Ministry of Health requested financial support to conduct the EVM assessment from the WHO Benin country office. 2) Questionnaires were pilot tested in April 2012 and revised. Prior to data collection, an AMP- and PATH-facilitated training session (2 days for the EVM questionnaire, 1 day for the HERMES questionnaire, 1 day pre-test and 1 day debrief) familiarized data collectors and supervisors with the questionnaires and Excel data entry sheets. Data was collected from selected locations, followed by data entry and review. 3) On 25July 2012, the Benin MoH convened a preliminary workshop to present results and to define the process of the supply chain optimization project. Attendees included delegates from different departments of the MoH (Essential Drugs Supply and Procurement Division, Department for the maintenance of equipment, HIV/Aids program, TB Program, EPI), PATH, VMI, AMP, WHO, UNICEF, USAID and Transaid. 4) On 20–21 September 2012, a second meeting to review the HERMES simulation experiment results ensued. Attendees included representatives from the Benin’s MoH, ANV-SSP, AMP, WHO, the HERMES Team, the Bill and Melinda Gates Foundation, PATH, Transaid, and UNICEF, among others. Following the presentation of the HERMES, CCEM and EVM assessment results, Benin’s MoH, identified strategic orientations for supply chain optimization, including redesign the vaccine supply chain as supported by HERMES simulation experiments. 5) A report is prepared and distributed to Ministry. Begin plans for implementation.

Key Step 2: Identifying In-Country “Champions”

It is also important to identify key individuals with the ability and enthusiasm to serve as in-country “champions” for the procedure and potential change. In Benin, our team was fortunate to have a forward-thinking Minister of Health Dorothée Kinde-Gazard, who understood the problem, process, and potential solutions, welcomed change and new ways of thinking, had time to dedicate to addressing the issue, and had the charisma and influence to enact changes. Champions may not be evident initially. Those who could eventually serve as champions initially may not be familiar with the problem, potential approaches, and possible solutions. In fact, they may not even realize the magnitude or the urgency of the problem. One should not assume that this lack of awareness is due to a deficiency in ability or education. Key stakeholders in any country can be very busy and occupied by other seemingly more urgent matters. The timing needs to be right for potential champions to dedicate their time and energy. Therefore, the process of identifying “champions” may take time and patience.

Key Step 3: Directed and Efficient Data Collection

Data collection purely for scientific inquiry versus data collection to guide policy change can be very different endeavors. Data collection to guide policy change can in some cases be more directed and focused. In the case of Benin, data collection aimed to meet HERMES data needs. The data collection tool was adapted from vaccine supply chain costing data collection tools developed and utilized by Project Optimize, a collaboration between PATH and WHO. The tools included a questionnaire for each level of the supply chain to capture resource usage for the storage and distribution functions, as well as stock movement data. Collected data included information on human resources, cold chain equipment type and specifications, and transport modes, frequency, and routes for delivering vaccines. Data also came from existing tools such as the Effective Vaccine Management (EVM) and Cold Chain Equipment Management (CCEM) assessments, Stock Management Tool (SMT), and the Benin Comprehensive Multiyear Plan (cMYP).

AMP pre-tested the questionnaires at two health centers and one commune to determine the administration time required and ease-of-use. Our project’s steering committee then evaluated the questionnaires. Following the pre-test, but prior to data collection, AMP and PATH facilitated training sessions to familiarize data collectors and supervisors with the EVM tool and adapted Project Optimize questionnaires. Pairs of agents administered the questionnaires among five to six sites over a ten-day period. We collected data from the national warehouse, six department stores, the regional store, 16 commune stores, and a health center from each commune.

Several challenges emerged during data collection. First, developing and translating data collection tools took longer than expected. Standardizing these tools will likely improve the data collection process in the future. Second, when the data collectors visited a site, data managers (e.g., an accountant who had the cost information) were not always present or ready to share information. In the future, providing sites with advance notice may circumvent this problem. Third, the budget did not include enough supervisors for data collectors, leading to procedural errors that later cost time and resources. Budgetary limitations also reduced the number of health facilities sampled from 71 to 40 locations. Fourth, determining some personnel’s wages (for costing purposes) was difficult, as some roles are ill-defined in functionality and title.

Key Step 4: Defining a Finite Set of Realistic Scenarios to Evaluate

In general scientific inquiry, a wide range of possible scenarios may exist, but in a practical policy setting, only a finite set of scenarios may be possible. Existing constraints may limit what changes are actually possible. For example, decision-makers may have to use certain building and transport routes and not have the luxury of choosing any location to build a new building or establish a transport route. The terrain, climate, available labor force, crime, and political considerations can govern what changes are possible. Ignoring such considerations and the opinions of in-country stakeholders with knowledge of these considerations could lead to impractical solutions.

As outlined in Figure 1, the Benin MoH convened a preliminary workshop in late July 2012 to define three re-design scenarios that HERMES should model [2]: (1) consolidating the commune level from 80 commune stores into 34 zone sanitaire stores, (2) removing the entire commune level so that the supply chain has only three levels and (3) removing the entire commune level while increasing the number of department stores from 6 to 12. Details about each modeling scenario and results can be found in a previous publication [1].

Key Step 5: Making the Study Methodology Transparent

Decision-makers in any country may not accept results when they come from an obscure technique, method, or tool (i.e., “a black box”). The process as well as the tools utilized should be relatively transparent. An important part of the workshop in late July 2012 was presenting in detail HERMES, the types of models HERMES generates, and the relative strengths and limitations of HERMES-generated models. In August 2012, we created HERMES specific inputs from the data sources to generate a discrete-event simulation model of the supply chain. This model served as a virtual laboratory for evaluating the potential operational and cost impacts of the various alternative designs on vaccine distribution and storage logistics before and after new vaccine introductions. In September 2012, the MoH convened a second meeting to review the HERMES simulation experiment results. During this meeting, our team again reviewed the structure of the HERMES models and how the results were generated.

Key Step 6: Presenting the Results in a Clear Manner that is Digestible by Key Stakeholders

How one presents simulation results is important. Presenters must minimize technical jargon and instead use terms familiar to stakeholders. Presentations should be concise and efficient to not waste busy country stakeholders’ time. Using visualizations rather than extensive tables and numbers can efficiently transfer knowledge to individuals of all backgrounds. In addition, “live” rather than “static” data is more compelling; when stakeholders can directly use tools, change parameters, and explore resulting effects in real-time, they feel more connected with the results. Figure 2 shows some of the graphical visualizations presented to the MoH during the September 2012 meeting to review the HERMES simulation experiment results and CCEM and EVM assessments.

Figure 2. Graphical visualizations of HERMES Benin vaccine supply chain model.

Graphical visualizations depict (a) maximum capacity utilization and (b) overall vaccine availability across storage locations and transportation routes for the current system after introduction of rotavirus vaccine in Benin. These visualizations were produced for each re-design scenario and presented to the Ministry of Health in September 2012.

Key Step 7: Facilitate Decision-Making and Advocacy

In order to enact change, the MoH must not only be convinced that a change is beneficial but also be armed with information that they can present to other stakeholders that need to help enable the change such as donors and different implementers. Therefore, a major component of our engagement was building a “business case” for change, i.e., demonstrating the economic/financial benefits of change in addition to the health benefits. Part of this “business case” is showing how each stakeholder may be affected in a positive manner. This business case helped convince the MoH to move forward with redesigning the Benin immunization supply chain based on re-design Scenario 1; consolidating the commune level to a zone sanitaire level combined with adding a loop delivery system was the best option, incurring the lowest capital expenditures and operating costs. An implementation plan is in process, to be developed by the MoH, AMP, Vaccine Modeling Initiative (VMI), PATH, Transaid, WHO, and United Nations Children’s Fund (UNICEF).

Conclusions and Next Steps

Generating scientific evidence is one component of policy change. Enabling change requires orchestration of a coordinated set of steps that heavily involve key stakeholders, earn their confidence, and provide them with relevant information. Our Benin EVM+CCEM+HERMES Process led to a decision to enact a major change and could serve as a template for similar approaches in other countries to evaluate and potentially re-design their immunization supply chains.