Abstract
Introduction
Populations and routine childhood vaccine regimens have changed substantially since supply chains were designed in the 1980s, and introducing new vaccines during the “Decade of Vaccine” may exacerbate existing bottlenecks, further inhibiting the flow of all vaccines.
Methods
Working with the Mozambique Ministry of Health, our team implemented a new process that integrated HERMES computational simulation modeling and on-the-ground implementers to evaluate and improve the Mozambique vaccine supply chain using a system-re-design that integrated new supply chain structures, information technology, equipment, personnel, and policies.
Results
The alternative system design raised vaccine availability (from 66% to 93% in Gaza; from 76% to 84% in Cabo Delgado) and reduced the logistics cost per dose administered (from $0.53 to $0.32 in Gaza; from $0.38 to $0.24 in Cabo Delgado) as compared to the multi-tiered system under the current EPI. The alternative system also produced higher availability at lower costs after new vaccine introductions. Since reviewing scenarios modeling deliveries every two months in the north of Gaza, the provincial directorate has decided to pilot this approach diverging from decades of policies dictating monthly deliveries.
Discussion
Re-design improved not only supply chain efficacy but also efficiency, important since resources to deliver vaccines are limited. The Mozambique experience and process can serve as a model for other countries during the Decade of Vaccines. For the Decade of Vaccines, getting vaccines at affordable prices to the market is not enough. Vaccines must reach the population to be successful.
Keywords: Vaccines, Mozambique, Computational modeling, Supply chain
1. Introduction
How can low- and middle-income countries (LMICs) handle the new vaccines arriving during the “Decade of Vaccines” when many LMIC vaccine supply chains are not even able to deliver current vaccine regimens to their populations? A country's vaccine supply chain is the complex system of locations, storage equipment, vehicles, transport routes, and personnel that bring vaccines from a central location to locations where children and expectant mothers receive immunizations. Since a majority of LMIC supply chains were designed in the 1970s and 1980s [1], populations and the standard childhood vaccine regimen have grown substantially. Vaccines in many LMICs stall in storage locations far from the people who need them, resulting in wasted resources, missed opportunities to provide immunization, and preventable disease burden. Sending new vaccines into outdated, inefficient, and ineffective systems could not only result in the new vaccines not reaching their destinations but also disrupt delivery of other vaccines.
Prior to this decade, consideration of LMIC vaccine supply chain design was more limited, as much of the focus was on developing and financing new vaccines. Project Optimize, including partners from PATH and the World Health Organization (WHO), began evaluating vaccine distribution in LMICs from 2007 to 2012. In 2009, the HERMES Logistics Team began developing the HERMES simulation software tailored for vaccine supply chain evaluation and planning and began working with WHO, the Bill & Melinda Gates Foundation (BMGF), UNICEF, PATH, Ministries of Health (MOHs), and other organizations in Niger, Thailand, Vietnam, and Senegal. The HERMES Logistics Team and Project Optimize joined a series of meetings BMGF convened in 2011 and 2012 to discuss the state of LMIC vaccine supply chains and how to improve them. Two major themes were (1) the need for a process to systematically evaluate and improve vaccine supply chains and (2) the question of whether vaccine supply chains could benefit from re-design. From these meetings emerged the idea for a process of evaluating and improving vaccine supply chains that was first refined and piloted in Benin [2] and later implemented to a greater extent in Mozambique. The meetings also catalyzed the formation of a WHO-UNICEF-Gavi hub to address vaccine supply chains and the initiation of the VillageReach and HERMES Logistics Team partnership to conduct the described work in Mozambique.
We explored the question of whether decades-old supply chain designs are in need of re-design in Mozambique, a low-income country in Southern Africa that has experienced challenges delivering vaccines to its population. Communications with the MOH and in-country non-governmental organizations (NGOs) identified supply chain bottlenecks, such as transport and human resource constraints, leading to an ad hoc adaptation of the vaccine delivery system in place, with health workers picking up vaccines by any means possible. Vaccine coverage rates have been well below 100% (e.g., in 2008, only 75% of one-year-old children were fully immunized with the third dose of diphtheria-tetanus-pertussis vaccine) [3]. Working closely with the MOH, UNICEF, and WHO, our team used a new combination of computational simulation modeling, information systems, stakeholder engagement, and training to help evaluate the Mozambique supply chain and develop, evaluate, and ultimately introduce a new supply chain design.
2. Methods
2.1. Overview
Our supply chain evaluation and re-design implementation process consists of the following steps: engage the Ministry of Health; conduct workshops to train decision makers and various stake-holders on using the HERMES (Highly Extensible Resource for Modeling Supply Chains) modeling software to develop a computational simulation model of the country's immunization supply chain, determine the supply chain's vulnerabilities and constraints, and gain practical experience in evaluating the impact of alternative supply chain designs; and begin discussions of how to move the computational modeling to real world implementation of chosen designs. Mozambique served as a pilot country for this process.
2.2. Engaging the MOH
With impending new vaccine introductions, the Mozambique MOH was interested in finding efficiencies in the vaccine supply chain through system optimization across all provinces. Thus, the MOH initiated a pilot modeling activity in two provinces where an alternative system was operating in order to generate evidence for analyzing different approaches to vaccine distribution. Using HERMES, we constructed models of the vaccine supply chains in the provinces of Gaza and Cabo Delgado and simulated scenarios of interest.
2.3. Conducting workshops
A significant part of this pilot was capacity building, i.e., building the skills of a local team of experts to use the models, interpret the results, and move decisions forward for system optimization. In September 2014, the Mozambique MOH, VillageReach, and the HERMES Logistics Team conducted a weeklong workshop in Mozambique. Participants included representatives from the Expanded Program on Immunization (EPI), Provincial Directorates of Health (DPS) from three provinces (including Gaza and Cabo Delgado), UNICEF, WHO, local University of Eduardo Mondlane, and VillageReach.
The workshop provided an introduction to simulation modeling and its applications to vaccine supply chains. After reviewing and discussing preliminary modeling results, participants provided information to refine the models and generated ideas for additional scenarios of interest. Participants then received hands-on training in creating and modifying a vaccine supply chain model, implementing and running scenarios, and analyzing simulation results within the HERMES user interface.
2.4. Developing the models
2.4.1. Original Mozambique vaccine supply chain design
Historically, the Mozambique routine vaccine supply chain has followed a multi-tiered system. Manufacturers ship vaccines to one national depot in the capital city of Maputo, which sends quarterly shipments of vaccines to most provincial stores by plane while nearby provinces pick up vaccines using 4 × 4 trucks. Each provincial store delivers monthly shipments to the district stores by truck. Health centers order vaccines from the districts on a monthly basis, and though national policy dictates that the district stores deliver vaccines to health centers by truck or motorbike, in reality many health centers must send health workers by public transportation to pick up vaccines due to unavailability of vehicles or lack of fuel funds at the districts. Health workers administer age-appropriate vaccinations at health centers and outreach sessions. The ad hoc nature of transport contributes to delays in getting vaccines to the health centers. This multi-tiered system continues to operate in 5 of the 10 provinces.
2.4.2. Alternative distribution system
In 2002, international NGO VillageReach, together with national NGO Foundation for Community Development, launched a five-year pilot project to streamline and improve the vaccine supply chain in Cabo Delgado province. In partnership with the MOH and the EPI, the project created an alternative distribution system. Instead of provinces delivering to the district level, each provincial store delivers monthly shipments directly to health centers using 4 × 4 trucks in transport loops (i.e., a truck visits multiple health centers in a single trip before returning to the provincial store). The pilot resulted in an increase in vaccine coverage, reduced stock-outs at the health center levels, and improved cost-efficiencies when compared to the original multi-tiered system [4,5]. These positive results encouraged VillageReach to expand the alternative system to three additional provinces, including Gaza, generating interest throughout the country.
2.4.3. HERMES simulation models of Gaza and Cabo Delgado supply chains
We constructed models of the routine vaccine supply chains for Gaza and Cabo Delgado provinces using the HERMES software platform. HERMES generates detailed discrete-event simulation models that include virtual representations of all vaccine vials, storage and immunization locations, storage and transport devices, ordering and shipping policies, and logistics costs associated with supply chain operations. A HERMES-generated model tracks each vial as it moves through the system to be shipped, stored, and ultimately used or wasted. Previous publications describe HERMES in detail [6–10].
The models of Gaza and Cabo Delgado provinces incorporated data collected by the MOH and VillageReach in 2012–2014, supplemented with information from the Mozambique comprehensive multi-year strategic plan (cMYP), Effective Vaccine Management (EVM) assessment, and collaboration with in-country experts. Discussions taking place during and after the modeling workshop guided further data collection and refinement of the preliminary models. The 2014 population was 1,388,332 in Gaza and 1,879,255 in Cabo Delgado [11]. Analyses included the current Mozambique EPI schedule vaccines, as well as introductions of rotavirus (RV), human papillomavirus (HPV), inactivated polio (IPV), and a second dose of measles (MSD) vaccines. Table 1 summarizes the characteristics of these vaccines [12].
Table 1.
Vaccine characteristics. Sources: WHO prequalified vaccines database [15] and WHO vaccine volume calculator [16].
| Presentation | Doses per course | Doses per vial | Vaccine packed volume per dose (cm3) | Diluent packed volume per dose | Storage location | |
|---|---|---|---|---|---|---|
| Current EPIa vaccines | ||||||
| Bacille Calmette-Guérin tuberculosis (BCG) | Lyophilized | 1 | 20 | 1.2 | 0.7 cm3 | Refrigerator (2–8 °C) |
| Measles (M) | Lyophilized | 1b | 10 | 3.5 | 4.0 cm3 | Refrigerator (2–8 °C) |
| Oral polio (OPV) | Liquid | 4 | 10 | 2.0 | n/a | Preferably freezer (−15 °C to 0°C) |
| Diphtheria-tetanus-pertussis-haemophilus influenza type B-hepatitis B (pentavalent) | Liquid | 3 | 10 | 2.6 | n/a | Refrigerator (2–8 °C) |
| Tetanus toxoid (TT) | Liquid | 2 | 10 | 3.0 | n/a | Refrigerator (2–8 °C) |
| Pneumococcal conjugate (PCV) | Liquid | 3 | 2 | 4.8 | n/a | Refrigerator (2–8 °C) |
| Introductory vaccines | ||||||
| Rotavirus (RV) | Liquid | 2 | 1 | 17.1 | n/a | Refrigerator (2–8 °C) |
| Human papillomavirus (HPV) | Liquid | 2 | 2 | 2.46 | n/a | Refrigerator (2–8 °C) |
| Inactivated polio (IPV) | Liquid | 1 | 10 | 4.8 | n/a | Refrigerator (2–8 °C) |
EPI is the Expanded Program on Immunization.
A second dose of measles (MSD) is included as a vaccine introduction.
2.5. Utilizing the models: current evaluation and impact of alternative scenarios
We modeled scenarios of interest identified during the workshop, which focused on comparing multiple vaccine supply chain structures for each province: the original multi-tiered structure, the alternative system, and potential alternative implementations of that system. Every system was modeled with the current EPI and subsequently with RV, HPV, IPV, and MSD introductions. Each scenario simulated one year of supply chain operations. Reported costs are in 2014 USD. Table 2 outlines the scenarios modeled.
Table 2.
Vaccine supply chain design scenarios.
| Gaza supply chain designs evaluated | |
| G1. Original multi-tiered distribution system | Fig. 1a depicts the multi-tiered system, which has three levels below the national depot: province, district, and health center. The provincial store attempts to pick up vaccines from the national level quarterly using a 4 × 4 truck but is able to make additional trips as needed. The provincial level delivers vaccines to the 12 district stores monthly. Approximately 60% of health centers receive monthly deliveries from the district level via pickup truck or motorbike while the remaining health centers retrieve vaccines from the districts each month using public transportation |
| G2. Alternative system, current implementation | Recently implemented in Gaza by VillageReach, this system (illustrated in Fig. 1b) utilizes two distribution loops to deliver from the provincial level directly to health centers in the southern region of the province using district personnel for additional supervision. Due to long distances, difficult terrain, and small populations, two additional loops deliver vaccines from the provincial level to the four district stores in the northern region. Vaccines then move to the northern health centers by the same methods as in structure G1 |
| G3. Alternative system, full delivery loop coverage | This system, shown in Fig. 1c, expands the recently implemented alternative system to full coverage of all health centers throughout the province. In addition to the two existing delivery loops in the south, two distribution loops deliver vaccines monthly from the provincial level directly to health centers in the north |
| G4. Alternative system, full coverage, alternate delivery schedule | Alternative system with full delivery loop coverage, delivery to northern health centers every two months |
| G5. Alternative system, full coverage, every 2 months | Alternative system with full delivery loop coverage, delivery to all health centers every two months |
| Cabo Delgado supply chain designs evaluated | |
| C1. Original multi-tiered distribution system | Fig. 2a displays the multi-tiered supply chain in Cabo Delgado which, similar to G1, has three levels below the national depot: province, district, and health center. The provincial store receives vaccine shipments from the national level quarterly by plane but receives additional deliveries as needed. The provincial level delivers vaccines to the 17 district stores monthly. Approximately 60% of health centers receive monthly deliveries from the district level via pickup truck or motorbike while the remaining health centers retrieve vaccines from the districts each month using public transportation |
| C2. Alternative system, current implementation | This system, depicted in Fig. 2b, includes three loops delivering vaccines monthly from the provincial level directly to 52% of the health centers in the province. The remaining health centers are located in hard-to-reach peripheral areas and must retrieve their vaccines from the health centers located at the headquarters in their respective administrative districts, using motorbikes or public transportation. One district is not included in a loop and instead receives monthly deliveries from the provincial store via 4 × 4 truck |
| C3. Alternative system, full delivery loop coverage | This system expands the existing three delivery loops to full coverage of all health centers throughout the province, as shown in Fig. 2c |
| C4. Alternative system, full coverage, alternate delivery schedule | This system alternates between reduced delivery and full delivery loop coverage each month, resulting in monthly deliveries from the provincial level to 52% of health centers. Peripheral health centers receive deliveries from the province every two months. |
| C5. Alternative system, full coverage, every 2 months | Alternative system, full delivery loop coverage, every two months throughout the province |
3. Results
3.1. HERMES modeling workshop
Workshop participants learned fundamental modeling concepts and demonstrated their understanding of how modeling can be applied to vaccine supply chains as they generated ideas for new modeling scenarios. Participants shared their on-the-ground knowledge and experience to improve the preliminary models with updated data and refined assumptions (e.g., more accurate estimates and updated values for transport vehicle capacities, route transit times, and storage equipment inventories).
Participants explored user interface features as they successfully edited the Gaza province model to implement a scenario they proposed during the workshop and analyzed the results. Participants also completed an exercise in which they each created, ran, and analyzed results for a new, simple vaccine supply chain model. Throughout the process, participants provided valuable feedback to guide further user interface development. In a post-workshop survey, respondents reported that the workshop was very useful to them and their work, and 100% of respondents supported further modeling efforts for the entire country supply chain.
3.2. Modeled scenario results
3.2.1. Supply chain performance indicators
Logistics costs for supply chain operations included the sum of all labor (personnel wages for hours worked in supply chain logistics), storage (equipment maintenance, energy, and amortization), transport (vehicle maintenance, fuel, and amortization, as well as driver per diems), and building (infrastructure overhead and amortization) costs that accrued during each simulated year. Formulas for each cost component are detailed in a previous publication [7]. We measured the following indicators of supply chain performance:
3.2.2. Gaza province
Fig. 3 shows results for Gaza province. The current system following the alternative model in Gaza (G2) can achieve 93% vaccine availability (access) at a logistics cost of $0.32 per dose administered. Due to transport constraints, the province must obtain vaccines 13 times per year on average (Nine additional trips as compared to the policy of quarterly trips). No storage bottlenecks exist with the current EPI schedule, though transport constraints exist at health centers in the north. Introducing RV, HPV, IPV, and MSD to the current system reduced vaccine availability by 7% as compared to the current EPI. These vaccine introductions produced storage constraints at the provincial and district levels while creating transport bottlenecks in both loops in the south and health center routes in the north. Introductions also required more coping in transport, as the provincial store required 22 trips per year on average to maintain sufficient vaccine stock.
Fig. 3.
Vaccine availability and logistics cost per dose administered for Gaza scenarios. +RV + HPV + IPV + MSD indicates introductions of rotavirus (RV), human papillomavirus (HPV), inactivated polio (IPV), and a second dose of measles (MSD) vaccines.
This system achieves higher vaccine availability with greater cost efficiency than the original multi-tiered distribution system, which could supply only 66% of the necessary vaccines under the current EPI at $0.53 per dose administered. The alternative system alleviated transport bottlenecks that existed at the district and health center levels under the multi-tiered system. Expanding the alternative system to cover all health centers with monthly deliveries reduces vaccine availability by 1% from the current system, due to an insufficient number of trucks to cover the distribution loops and the many additional trips needed to retrieve vaccines from the national store. Reducing delivery frequencies for distribution loops to northern health centers regains this 1% in availability and slightly reduces the logistics cost per dose administered. These northern health centers need no additional storage capacity, even when receiving vaccines with RV, HPV, IPV, and MSD introductions every two months. However, delivering to all health centers in the province every two months reduces vaccine availability due to transport constraints in the southern distribution loops.
3.2.3. Cabo Delgado province
Fig. 4 displays results for Cabo Delgado province. Currently, the alternative system in Cabo Delgado (C2) can achieve 84% vaccine availability at a logistics cost of $0.24 per dose administered. Transport constraints cause the provincial store to obtain vaccines 6 times per year on average (despite a policy of quarterly shipments). Transport constraints also exist at hard-to-reach health centers not currently reached by the distribution loops. No storage bottlenecks exist under the current EPI schedule, but introducing RV, HPV, IPV, and MSD to the current system created storage constraints at health centers. These introductions also produced transport bottlenecks in one distribution loop and routes leading to hard-to-reach health centers, and the province required 9 annual shipments. These bottlenecks caused a 12% reduction in vaccine availability.
Fig. 4.
Vaccine availability and logistics cost per dose administered for Cabo Delgado scenarios. +RV + HPV + IPV + MSD indicates introductions of rotavirus (RV), human papillomavirus (HPV), inactivated polio (IPV), and a second dose of measles (MSD) vaccines.
As in Gaza province, the alternative system in Cabo Delgado improved vaccine availability when replacing the original multi-tiered system, which was able to achieve 76% availability for the current EPI schedule at $0.38 per dose administered. Further expanding the alternative system to cover all health centers, including hard-to-reach areas, relieved health center transport constraints and raised vaccine availability to 92% while slightly reducing the cost per dose administered. Reducing the frequency of shipments to hard-to-reach health centers reduced vaccine availability, as many health centers had insufficient capacity to store a two-month supply of EPI vaccines. More accessible health centers faced the same issue, as evidenced by the further drop in availability to 80% when reducing the frequency of shipments to all health centers in the province.
3.3. Implementing chosen designs and technologies
Modeling provided much needed support of re-design and evaluated various shipping schedules to aid in further refining the alternative system. Since reviewing the scenarios modeling deliveries every two months in the north of Gaza, the Gaza provincial directorate has decided to pilot this approach diverging from decades of policies dictating monthly deliveries. This pilot is complemented by new remote temperature monitoring technology for the cold chain, also being piloted in that province, to better guarantee the safety of the vaccines. Engaging in-country supply chain experts in modeling will continue to produce more accurate and complete models as well as scenarios informed by on-the-ground knowledge and experience. Capacity building also empowered stakeholders to be able to consider new ideas and strategies for vaccine distribution together with available resources, policies, structures, and established goals. For example, C3 in Cabo Delgado would require three vehicles to reach the entire province with direct delivery to all health facilities; however, these resources are not currently available.
4. Discussion
While the vaccines that have or will be introduced during the Decade of Vaccines could have tremendous impact, the impact could be limited or even negative without first evaluating and improving vaccine supply chains around the world. This work in Mozambique combined VillageReach's experience, expertise, and positioning as an in-country implementing partner with HERMES modeling capabilities to facilitate MOH decision making and result in positive change. The iterative combination of computational modeling to better understand a system and test changes with on-the-ground implementation has helped transform other industries such as air traffic control and manufacturing. A systems approach also brings together perspectives from experts from a wide variety of disciplines (e.g., operations researchers, MOHs, and NGOs).
The experience in Mozambique makes a strong case for redesigning vaccine supply chains. Without computational modeling, testing new designs can be risky and consume considerable time, effort, and resources through trial and error. As shown in Mozambique, new systems designs can entail changes in not only the overall supply chain structure but also equipment, personnel, policies, and new information technology. Simply adding new storage equipment to the original supply chain would have relieved only some system bottlenecks and would have required more new equipment (and thereby greater capital investment and storage equipment operating costs) than in the alternative system, due to the existence of district stores. Additionally, the alternative system addressed the issues of constrained capacity and delays in transport that would not be addressed by adding storage equipment. Such re-design can not only improve effectiveness of vaccine supply chains but also efficiency, reducing costs while improving service. The Mozambique process can serve as a model for other countries to follow to help them anticipate new vaccine introductions and make the necessary changes, including potential system re-designs.
A major aspect of the Mozambique work has incorporated the newly designed user interface for HERMES, which helps empower in-country decision makers. Allowing in-country experts to directly input data, create new plausible scenarios to test, and analyze the results puts the process in the hands of the in-country decision makers. The user interface can provide ownership of the results without reliance on outside persons. The HERMES workshops conducted in Mozambique demonstrated how in-country personnel, who had little previous experience with computational modeling, could quickly learn to use many HERMES features and thereby gain a better understanding of modeling and its applicability to system re-design.
5. Limitations
Models, by their nature, are simplified representations of reality and cannot capture every factor that could affect the delivery of vaccines [13–15]. This study simulated expanding the alternative system and modifying shipping policies, but such changes may have ramifications not reflected in model assumptions and data inputs. Further exploration of potential system changes is warranted.
6. Conclusions
Without evaluating and addressing the constraints of LMIC vaccine supply chains, the Decade of Vaccines could exacerbate bottlenecks and further impede the flow of both new and existing vaccines. Our Mozambique experience could serve as a model to evaluate and improve vaccine supply chains, combining computational modeling to better understand systems and test improvements along with on-the-ground implementation of recommended changes. Our findings suggest that system redesign incorporating new supply chain structures and policies can substantially help to improve the effectiveness and efficiency of supply chains. Ultimately, the Decade of Vaccines is not just about the vaccines but the systems around them.
Fig. 1.
Gaza supply chain structures considered. (a) Original multitiered distribution system, (b) current implementation of alternative distribution system using transport loops in south, (c) potential implementation of alternative distribution system using transport loops throughout province.
Fig. 2.
Cabo Delgado supply chain structures considered. (a) Original multitiered distribution system, (b) current implementation of alternative distribution system using transport loops in easier-to-reach areas, (c) potential implementation of alternative distribution system using transport loops throughout province.
Acknowledgments
We would like to thank the Ministry of Health, specifically Dr. Mbofana, Benigna Maia, and Graça Matsinhe, who contributed significantly in the process, planning, and implementation of this project. UNICEF and WHO in Mozambique were instrumental in generating interest in this activity. We want to thank all participants in the workshop. This project was funded by the Bill & Melinda Gates Foundation via the Final 20 Project and the HERMES grant, the International Society for Infectious Diseases (ISID) and Pfizer via the SIGMA grant, the Agency for Healthcare Research and Quality (AHRQ) via grant R01HS023317, the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Office of Behavioral and Social Sciences Research (OBSSR) and the Global Obesity Prevention Center (GOPC) via grant U54HD070725, and NICHD via grant U01HD086861. The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
Footnotes
Conflict of interest: The authors have no conflicts of interest to declare.
References
- 1.World Health Organization. Immunization supply chain and logistics. Geneva, Switzerland: 2014. [Internet] Available from: < http://www.who.int/immunization/call-to-action_ipac-iscl.pdf>. [Google Scholar]
- 2.Lee BY, Schreiber B, Wateska AR, Connor DL, Dicko HM, Jaillard P, et al. The Benin experience: how computational modeling can assist major vaccine policy changes in low and middle income countries. Vaccine. 2015;33(25):2858–61. doi: 10.1016/j.vaccine.2015.04.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.WHO-UNICEF estimates of DTP3 coverage; WHO UNICEF coverage estimates. WHO World Health Organization: Immunization, Vaccines and Biologicals. Vaccine preventable diseases Vaccines monitoring system 2015 Global Summary. 2015 Reference Time Series: DTP3 [Internet] [cited 2015]. Available from: < http://apps.who.int/immunization_monitoring/globalsummary/timeseries/tswucoveragedtp3.html>.
- 4.Kane M. Evaluation of the project to support PAV (Expanded Program on Immunization) in Northern Mozambique, 2001–2008 an independent review for VillageReach with program and policy recommendations. 2008 Nov; Available from: < http://www.villagereach.org/wp-content/uploads/2009/08/Evaluation-ExecSum-only.pdf>.
- 5.VillageReach. Comparison of costs incurred in dedicated and diffused vaccine logistics systems. 2009 Oct; Available from: http://www.villagereach.org/wp-content/uploads/2010/10/091009-VillageReach-Cost-Study-Report.pdf.
- 6.Lee B, Connor D, Wateska A, Norman B, Rajgopal J, Cakouros B, et al. Landscaping the structures of GAVI country vaccine supply chains and testing the effects of radical redesign. Vaccine. 2015;33(36):4451–8. doi: 10.1016/j.vaccine.2015.07.033. [DOI] [PubMed] [Google Scholar]
- 7.Brown S, Schreiber B, Cakouros B, Wateska A, Dicko H, Connor D, et al. The benefits of redesigning Benin's vaccine supply chain. Vaccine. 2014;32(32):4097–103. doi: 10.1016/j.vaccine.2014.04.090. [DOI] [PubMed] [Google Scholar]
- 8.Haidari L, Connor D, Wateska A, Brown S, Mueller L, Norman B, et al. Augmenting transport versus increasing cold storage to improve vaccine supply chains. PLoS ONE. 2013;8(5):e64303. doi: 10.1371/journal.pone.0064303. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lee B, Cakouros B, Assi T, Connor D, Welling J, Kone S, et al. The impact of making vaccines thermostable in Niger's vaccine supply chain. Vaccine. 2012;30(38):5637–43. doi: 10.1016/j.vaccine.2012.06.087. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Assi T, Brown S, Djibo A, Norman B, Rajgopal J, Welling J, et al. Impact of changing the measles vaccine vial size on Niger's vaccine supply chain: a computational model. BMC Public Health. 2011;11(1):425. doi: 10.1186/1471-2458-11-425. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ine.gov.mz. INE Destaques â€″ Instituto Nacional de Estatistica Mocambique. 2014 [Internet] [cited 2014]. Available from: < http://www.ine.gov.mz/>.
- 12.Lee B. Editorial commentary: digital decision making: computer models and antibiotic prescribing in the Twenty†First Century. Clin Infect Dis. 2008;46(8):1139–41. doi: 10.1086/529441. [DOI] [PubMed] [Google Scholar]
- 13.Lee B, Biggerstaff B. Screening the United States blood supply for West Nile Virus: a question of blood, dollars, and sense. PLoS Med. 2006;3(2):e99. doi: 10.1371/journal.pmed.0030099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lee B, Wiringa A. The 2009 H1N1 influenza pandemic. Human Vacc. 2011;7(1):115–9. doi: 10.4161/hv.7.1.13740. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.World Health Organization. WHO prequalified vaccines. Geneva, Switzerland: World Health Organization; 2014. [Internet] Available from: < http://www.who.int/immunization_standards/vaccine_quality/PQ_vaccine_list_en/en/index.html>. [Google Scholar]
- 16.WHO. Vaccine volume calculator. 2012 [Internet] [cited 2014]. Available from: < http://www.who.int/immunization_delivery/systems_policy/logistics/en/index4.html>.