Skip to main content
PMC home page
Gates Open Research logoLink to Gates Open Research
. 2020 Apr 6;4:31. [Version 1] doi: 10.12688/gatesopenres.13121.1

Promoting equity in immunization coverage through supply chain design in Pakistan

Mariam Zameer 1,a,#, Nora Phillips-White 1,#, Olamide Folorunso 2, Rachel Belt 3, Hamidreza Setayesh 3, Naeem Asghar 4, Arshad Chandio 4
PMCID: PMC7926263  PMID: 33709056

Abstract

To improve equity in immunization coverage, potent immunization products must be available in the communities in which low coverage rates persist. Most supply side investments are focused on replacing or establishing new health facilities to improve access to immunization. However, supply chain design must be improved to ensure that potent vaccines are available at all facilities to promote immunization equity. We used the supply chain design process in Pakistan as an opportunity to conceptualize how supply chains could impact equity outcomes. This paper outlines our approach and key considerations for assessing supply chain design as a contributing factor in achieving equitable delivery of immunization services.

We conducted a supply chain analysis based on sub-national supply chain and immunization coverage at district level. Supply chain metrics included cold chain coverage and distances between vaccination sites and storage locations. Immunization coverage metrics included the third-dose diphtheria- tetanus-pertussis (DTP3) vaccination rate and the disparity in DTP3 coverage between urban and rural areas. All metrics were analyzed at the district level. Despite data limitations, triangulation across these metrics provided useful insights into the potential contributions of supply chain to equitable program performance at the district level within each province. Overall, our analysis identified supply chain gaps, highlighted supply chain contributions to program performance and informed future health system investments to prioritize children unreached by immunization services.

Keywords: equity, immunization, supply chain, cold chain

Introduction

A strategic target of the Global Vaccine Action Plan is to achieve 90% national immunization coverage in all countries and 80% in each district by 2020 1. Although global coverage has improved steadily since 1980, beginning in 2009 progress stagnated and it is unlikely these targets will be met by the end of 2020 2– 4. Furthermore, inequities in immunization, or “avoidable differences in immunization coverage between population groups that arise because barriers to immunization among disadvantaged groups”, appear to be contributing to stagnation in coverage rates 5. Under-immunized children are more likely to be of lower socioeconomic status than fully immunized children and are geographically clustered in marginalized communities, such as urban poor, remote rural, mobile populations and communities affected by conflict, resulting in a higher burden of vaccine-preventable disease in these communities 5– 14. In order to continue making gains in equitable immunization coverage, interventions need to identify and specifically target these communities 5, 6, 9, 13, 15, 16.

As part of the effort to improve equity in immunization coverage, immunization products must be available and of sufficient potency in the communities in which low coverage rates persist 4, 5. To achieve this, immunization supply chains should be tailored to address health system barriers to equity in service delivery such as vaccine stockouts, inadequate storage capacity, non-functional cold chain equipment (CCE), as well as difficult terrain and road conditions 4, 17– 22. However, there is limited research and evidence on how the design of supply chains can be tailored to deliver potent vaccines to all children.

Supply chain designs traditionally focus on cost-effectiveness and efficiency by streamlining distribution and storage to improve performance 23. However, efficiency approaches could miss marginalized and hard-to-reach communities, and not address inequity. This paper describes the process by which equity was considered during an immunization supply chain (iSC) design analysis in Pakistan through a collaboration between the Pakistan Expanded Program on Immunization (EPI) 24, the United Nations Children’s Fund (UNICEF) 25, Gavi, the Vaccine Alliance 26, and VillageReach, an international non-profit that transforms health care delivery to reach everyone 27. The analysis identified disparities between districts by assessing supply chain, and immunization equity. This approach allowed stakeholders to consider placement of CCE, new warehouses or distribution network options to increase equity in immunization coverage.

Methods

Setting

Pakistan had an annual birth cohort of over 6.9 million children in 2017-18 with 6% of the world’s under-immunized children 2, 28. Currently, the Pakistan EPI is focusing on expanding routine immunization services, reaching every child with vaccines, and, as Pakistan’s Gross National Income per capita continues to grow, they will transition from Gavi financial support 24, 29, 30. As part of this strategy, which includes improving supply chain performance, an analysis was completed in 2018 to map the current supply chain structure and recommend changes to optimize Pakistan’s distribution network 31. We used LLamasoft’s Supply Chain Guru © modeling tool 32 to assess alternative supply chain designs defined by the Government of Pakistan, and supported by VillageReach, UNICEF, Gavi, and LLamasoft. Primary data such as vaccine demand, inventory policies, storage capacity, transport capacity, operating costs, and list of sites were collected from federal and provincial EPI. The modeling tool determined the optimal supply chain configuration for provinces to minimize costs while maintaining high levels of product availability. While the tool provides the most efficient solution for a supply chain system overall, it does not consider disparities and inequities within this system, such as between districts. Stakeholders in Pakistan, including the government, were highly interested in risk to vaccines and equity of service distribution, given such a large under-immunized population. From an immunization equity perspective, it is crucial to understand disparities in immunization coverage between and within districts in order to develop targeted strategies for under-immunized communities 13. Hence, VillageReach used Microsoft Excel to assess the data collected from primary and secondary sources to assess equity in the immunization supply chain.

Assessing equity in Pakistan’s immunization supply chain

To provide further insights about the relationship between supply chain and equity in immunization coverage, we analyzed district-level metrics in 114 districts for 6,400 health facilities from the provinces of Balochistan, Khyber Pakhtunkhwa (KP), Punjab, and Sindh, and Islamabad that cover over 90% of Pakistan’s population. Supply chain metrics were aggregated from the collected data and modeling outputs, and we obtained district-level immunization coverage metrics from Pakistan Social and Living Standards Measurement survey (PSLM) 33. The PSLM was the only data source with immunization coverage rates at the district level that was available to us at the time of analysis. Although the PSLM has some methodological limitations, other data sources, including the Pakistan Demographic and Health Survey and the Multiple Indicator Cluster Surveys, were either not available for all provinces at the time of this analysis or were not representative at the district level 34, 35. We consulted with the Pakistan EPI and UNICEF, and determined PSLM was acceptable for the analysis.

Supply chain metrics

We determined the appropriate supply chain metrics to include in our analysis by relying on data available to us and logical or evidence-based link to equity in immunization. Targets were set for each metric to contextualize districts’ supply chain performance, highlighting specific areas for decision-makers to consider during redesign. VillageReach, Gavi, and UNICEF consulted with the Pakistan EPI to develop appropriate targets for each indicator specific to Pakistan’s country context. The following sections describe each metric and our process to determine appropriate targets. Figure 1 summarizes the three supply chain metrics.

Figure 1. Three supply chain metrics to assess equity in supply chain design.

Figure 1.

Open in a new tab

Red - Indicator is outside range of the target; Yellow - Indicator is at the target range; Green - Indicator is well within the target.

Supply chain metric 1: Cold chain capacity per surviving infant at district and below. This metric assesses the cold storage capacity available to store vaccines at 2–8°C. Lack of functional cold chain constrains vaccine availability and can negatively impact vaccine potency, which limits immunization coverage rates 12, 17, 36. For our analysis, we defined cold chain capacity per surviving infant at the district level and below as the sum of the volume of cold chain storage available at the district stores, sub-district stores, and health facilities, divided by the projected number of infants in the district expected to survive to their first birthday in a one-year period, as shown in Figure 1. This metric included existing cold chain equipment at the time of the analysis plus equipment planned for deployment in 2018.

To understand whether cold chain coverage was sufficient, we used the World Health Organization (WHO) Effective Vaccine Management (EVM) tool to establish a target 37. This target represents the maximum volume needed to store all doses of the recommended vaccines required to fully immunize one child in Pakistan (assuming the child has survived to their first birthday and that products are resupplied once a month). Additionally, the target included Pakistan’s recommended “buffer stock” that should be available in case of spoilage, breakage, or unforeseen changes in demand. For Pakistan, this target was estimated at 0.06 liters of cold chain capacity per fully immunized child (FIC).

Supply chain metric 2: Average resupply distance of vaccines to health facilities. This metric examines the distance that vaccines must travel to a health facility from the store resupplying it in the supply chain. Health facilities far from resupply locations may be at risk of low stock availability, and so low immunization coverage, because these distances may make the supply chain less responsive to outbreaks, emergency orders, or adverse weather. Finally, when vaccines must travel further to reach health facilities, they are more likely to be exposed to temperatures that put them at higher risk of spoilage and long distances can be a barrier for vaccinators for picking up supplies, which may lead to stockouts 38.

For each district, we calculated this metric by taking the weighted average of the distance from each health facility in the district to its resupply store, which was based on the estimated number of doses required for each health facility to serve its target population. We set a target of 60 kilometers as a feasible trip for a vaccinator to travel from a health facility to its resupplying store and return in one business day, based on the average travel speeds reported during primary data collection across all provinces. While this target does not account for variability in road conditions, it does serve as a point of comparison to identify districts where many facilities are far from resupply points.

Supply chain metric 3: Inbound resupply distance of vaccines to district store. We developed this metric to understand supply chain configuration at the intermediate levels of the supply chain, upstream from the health facility. When stockouts occur at the district level, service delivery is interrupted in 96% of cases 39. Additionally, longer resupply distances to districts could result in similar risks to vaccine quality and availability mentioned above. To that end, the inbound resupply distance to the district store measured the distance that vaccines must travel to reach the district from either a province or division store, depending on the province in Pakistan. Based on data collected from provinces, we determined that 120 kilometers was a feasible trip from a district store to its resupply point and back in one business day, and set the target accordingly.

Immunization coverage metrics

Low coverage rates result from several factors in the health and social systems including, but not limited to, supply chain challenges 11, 22, 40. To explore the relationship between immunization coverage and supply chain configuration, our analysis focused on two coverage indicators, both collected from the PSLM.

Immunization coverage metric 1: DTP3 coverage. This metric measures the percentage of children vaccinated with the third dose of the diphtheria-tetanus-pertussis (DTP3) vaccine based on a survey of a sample of households in each district 33. DTP3 coverage is widely accepted as a proxy for full immunization coverage 8. To compare districts relative to each other, we organized districts by percentile; the 90 th percentile for this metric corresponds to a DTP3 rate above 91%, and 70 th percentile indicates a DTP3 coverage rate above 72%.

Immunization coverage metric 2: Disparity in DTP3 coverage between urban and rural areas. Although limited data below the district level were available for this analysis, the PSLM also provided an estimate of the difference in DTP3 coverage between the urban and rural areas in each district 33. Districts with large urban-rural gaps in immunization coverage may face disparities in supply chain performance between urban and rural areas, such as long resupply distances to rural areas 14. Again, we organized districts by percentile to facilitate comparisons, with districts in the 90 th percentile, indicating an urban-rural gap of less than 7 percentage points, and the 70 th percentile indicating an urban-rural gap of less than 22 percentage points.

Results

Applying an equity lens for supply chain design decision-making

Stakeholders used the results from our analysis to decide which districts to prioritize for changes to the supply chain, comparing the metrics for the current supply chain to the proposed design. See Extended data for the immunization supply chain equity metrics 47. Although the analysis did not reveal an overarching trend or pattern between the chosen supply chain metrics and DTP3 coverage, the results highlighted districts with high under-immunized populations and with disparities in urban-rural coverage that should be prioritized for supply chain interventions. The lack of clear trends is expected because many variables affect immunization coverage rates, including but not limited to supply chain, which were beyond the scope of our analysis 41. Pakistan is one of the three countries where polio is endemic, and reported 135 wild polio cases in 2019 according to the Global Polio Eradication Initiative. The polio initiative is a key way for Pakistan to identify zero-dose children who have not received immunization and districts with the most under-served populations that need specific supply and demand interventions. To add further context, districts classified as high priority for polio eradication by the Pakistan’s National Emergency Action Plan for Polio Eradication are highlighted throughout the analysis 42.

Comparing cold chain capacity and immunization coverage

Our analysis indicated that the relationship between immunization coverage and cold chain capacity is ambiguous, as there are districts with cold chain capacity above the target but with low immunization coverage. Figure 2 shows several districts in Sindh, KP, and Balochistan, including Quetta, a high-priority polio district, have low immunization coverage rates, but cold chain capacity above the target. This indicates that factors aside from cold chain are likely leading to low coverage rates. We also highlighted districts with high disparities in urban-rural coverage rates, which might be a result of disparities in cold chain distribution within the district. A helpful next step would be to assess cold chain at each health facility, especially certain rural areas, to understand disparities within districts. Still, there are no districts with high immunization coverage and cold chain capacity below the target, reflecting that cold chain is a minimum condition needed to make potent vaccines available. We identified a group of districts with low immunization coverage and cold chain capacity near or below the target ( Figure 2), including Kohistan district in KP and two high-priority polio districts in Balochistan, Killa Abdullah and Pishin. Government stakeholders took note of these findings, and plan to prioritize these districts for receiving new CCE in an upcoming round of cold chain deployment through the Cold Chain Equipment Optimization Platform (CCEOP).

Figure 2. Cold chain capacity per surviving infant against immunization coverage for all provinces.

Figure 2.

Open in a new tab

Comparing weighted average resupply distance to health facilities and immunization coverage

Examining the distance from health facilities to their resupply points, alongside immunization coverage, helped decision-makers identify districts where resupply distances should be reduced, as shown in Figure 3. Many districts in KP and almost all districts in Punjab had resupply distances for health facilities within the target and high coverage rates, illustrating shorter resupply distance may be a contributing factor to high coverage. On the other hand, in most districts in Balochistan resupply distances were also within the target; however, immunization coverage rates were low, indicating that other factors may underpin low coverage. The results for Sindh were varied, with no apparent trends. A helpful step would also be to assess individual resupply distances for health facilities in rural areas, especially where there is large disparity in urban-rural immunization coverage rates.

Figure 3. Weighted average resupply distance to health facilities against third-dose diphtheria-tetanus-pertussis (DTP3) for all provinces.

Figure 3.

Open in a new tab

To improve product availability in health facilities that are far from their resupply points, stakeholders in KP considered changing from the current distribution system in which vaccinators fetch products, to direct delivery of stock to health facilities from districts 23. This strategy consolidates responsibility for logistics at a higher level of the supply chain where a smaller group of workers can receive specialized training to avoid mishandling vaccines during transit. Furthermore, direct delivery allows vaccinators at facilities more time to provide immunization services 43. For example, Figure 4 shows vaccinators in Shangla district, in which less than half of children have received the DTP3 vaccine, could save 2,019 hours annually, which could be spent on providing immunization services. This is especially critical in areas with low rural coverage rates where the resupply distances may be very long.

Comparing inbound resupply distance of vaccines to district stores and immunization coverage

The resupply distance for districts in the analysis ranged from 4 km to 538 km, as shown in Figure 5, but no notable relationships with immunization coverage are apparent. However, results were useful to stakeholders and provided further insights when considering supply chain design changes at the district level. For example, Figure 6 shows that changing a resupply store location in KP would reduce resupply distances to seven districts with very low rates of immunization coverage (Battagram, Kohistan, Shangla, Tor Ghar, Karak, Lakki Marwat, and Tank).

Figure 4. Average hours saved by each vaccinator in a year per health facility from direct delivery in Khyber Pakhtunkhwa province (excluding Tribal Districts).

Figure 4.

Open in a new tab

Figure 5. Inbound resupply distance of vaccines to district stores for all provinces against third-dose diphtheria-tetanus-pertussis (DTP3) for all provinces.

Figure 5.

Open in a new tab

Figure 6. Changing sourcing would reduce resupply distance of district stores in KP (excluding Tribal Districts) improving reliability and responsiveness of supply chain, and contributing to improved immunization coverage.

Figure 6.

Open in a new tab

Discussion

Our approach represents an important first step in using supply chain analysis to drive equitable design. Still, the analysis was constrained by several limitations that should be addressed in future iterations.

Improving data availability to extend analysis to the community level

Due to constraints in the dataset, the district was the lowest level of analysis possible in Pakistan, yet inequities in immunization often exist between communities within districts 13. For example, we know that the distance that a person must travel to receive vaccinations is associated with coverage 12, 13, yet data was not available on the distances between health facilities and the communities they serve. In addition, due to security concerns, there is restriction on collecting geospatial information in Pakistan, which limited our ability to locate facilities and calculate such distances.

Lack of data at lower tiers of the health system is not unique to Pakistan; a recent assessment showed that 74 countries (38% of 194 countries worldwide) do not report sub-national immunization data 2, let alone at the district, health facility or community level. Similarly, high-quality supply chain data is not consistently available for lower levels of the health system 44. For more specific analysis to identify inequities, countries should assess cold chain capacity and resupply distance for each health facility in relation to immunization coverage. Further, supply chain metrics such as stock availability, on-time/in-full deliveries, and number of temperature excursions for CCE should be used for design. In order to improve equity in immunization, it is critical to have visibility into the supply chain challenges that must be overcome to ensure that potent vaccines are available in all communities.

Developing metrics suited to urban areas

In urban areas, distances between points may be quite short but may take long to cover due to traffic, so resupply distance metrics from this analysis may not provide the full picture. However, even in densely populated urban areas, research shows that placement of facilities near low-income communities seems to be an important factor in urban areas to improving coverage. However, an alternative metric may be needed to show proximity of health facility to resupply to ensure availability of potent vaccines, such as time 18, 19. Additionally, lack of data on population in rapidly growing urban centers results in inadequate cold chain capacity to meet demand, leading to stockouts 45. As a result, urban-specific metrics should be developed and measured at the community-level to identify inequities within cities.

At the time of this analysis, detailed data on Pakistan’s urban poor was not available, although data from focus groups from a separate urban poor study by UNICEF highlighted lack of public health facilities in urban poor communities. However, these data did not identify names of nearest health facilities, so we could not generate specific recommendations to improve the supply chains. Similarly, the PSLM, which provided district-level immunization coverage data, aggregated the six districts in the mega-city of Karachi into a single sampling block, so we could not assess disparities in coverage rates in this major urban area. Recently Gavi and Pakistan EPI have been collecting data in Karachi to identify inequities in the distribution of cold chain and other components of the immunization program.

Implementing master facility lists to assess equity across data sources

A master facility list is critical to match data across different databases, which are created and managed by various partners. When health facility names cannot be matched across different sources, the ability to use available data and to assess cold chain capacity at the health facility-level is limited. In this analysis, VillageReach was only able to match names of 25% of the 8,000 public health facilities in Pakistan between multiple databases. To facilitate data analysis at the facility-level and understand inequities between communities, countries should develop a master facility list with unique identifiers.

Conclusion

Often, the design of supply chains is based on cost-efficiency analysis; however, our approach presented in this paper provides guidance on applying an equity lens when making changes to the supply chain design. The analysis enabled decision-makers in Pakistan to consider alternative supply chain configurations in light of potential improvements to equity and coverage. Further, while most previous analysis has focused on inequities between provinces 35 or focused on a small sample of districts 46, this analysis moves towards a full-country analysis of immunization equity at the district level, which is a critical step to identifying districts which need closer analysis. Previous studies often focus on either provincial analysis which hides disparities or on a subset of districts in a country which does not provide the full picture.

Considering equity in addition to other dimensions such as cost and efficiency yields important benefits for wider planning. Many supply-side investments focus on the replacement and improvement of existing facilities, which does not improve the status quo in underserved areas. While the analysis demonstrated that the relationship between supply chain inputs and program performance is not always clear, supply chain redesign could potentially reduce stockouts and vaccine expiry, inform cold chain deployment in low coverage areas, and ultimately contribute to equitable immunization.

In future applications, this analysis must be adapted to the context in which it is being used by selecting appropriate and salient metrics. Many Gavi-eligible countries receive CCE through CCEOP, so metrics such as cold chain capacity per surviving infant can be used to systematically include equity in deployment. Country-specific targets must be calculated for each country. Furthermore, inequities in immunization coverage in urban areas are a priority in many countries, so urban-specific metrics need to be developed. Finally, product availability and potency is only one aspect of the service delivery, and supply chain design should seek appropriate linkages with other supply dimensions (e.g. financing, service delivery, etc.) and demand multi-dimensional interventions to impact program performance. Integrating equity considerations into the supply chain design process can encourage stakeholders to identify areas for holistic supply chain improvements that prioritize underserved and under-immunized populations.

Data availability

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Extended data

Figshare: Data_Pakistan_GatesOpenResearch. https://doi.org/10.6084/m9.figshare.12032106.v1 47.

This file contains the immunization supply chain equity metrics for Pakistan produced in this study.

Extended data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Disclaimer

The findings and conclusions contained in this report are those of the authors and do not necessarily reflect Gavi and UNICEF policies and positions.

Acknowledgments

We thank Dr. Arshad Chandio, for championing the immunization work, especially supply chain design, to ensure equitable availability of vaccines for the children of Pakistan. His engaged interest in the work were critical in ensuring immunizations are improved in Pakistan. Mr. Naeem Asghar, Deputy Director Operations for Federal EPI, was instrumental in supporting VillageReach with data collection. Mr. Eshioramhe Kelobo, Cold Chain Specialist at UNICEF Pakistan, provided key insights throughout the project especially around cold chain equipment optimization platform (CCEOP). Ryan Purcell and Margaret Reinhard from LLamasoft Inc. analyzed the modeling data and supported throughout the project on using equity metrics as well as iterating on the data maps (not included in this paper). An article on considering equity in supply chains was first published at the VillageReach website and included in the Global Immunization Newsletter for January 2019: https://www.villagereach.org/considering-equity-in-vaccine-supply-chains/. Katie Reynolds from VillageReach contributed to review of drafts, and Melissa West from VillageReach provided critical thinking and framing around using the analysis, and supporting presentation of the results to the Government of Pakistan.

Funding Statement

The analysis and paper were funded by UNICEF Pakistan (contract number 43223114) and the Bill & Melinda Gates Foundation (OPP1195513).

[version 1; peer review: 1 approved, 2 approved with reservations, 1 not approved]

References

  • 1. Global Vaccine Action Plan, 2011-2020.Geneva: World Health Organization;2013. Reference Source [Google Scholar]
  • 2. Immunization SAGoEo: 2018 Assessment report of the Global Vaccine Action Plan.Geneva: World Health Organization;2018. Reference Source [Google Scholar]
  • 3. UNICEF-WHO: Progress and Challenges with achieving Universal Immunization Coverage. 2017 WHO/UNICEF Estimates of National immunization Coverage.2018. Reference Source [Google Scholar]
  • 4. Brooks A, Habimana D, Huckerby G: Making the leap into the next generation: A commentary on how Gavi, the Vaccine Alliance is supporting countries' supply chain transformations in 2016-2020. Vaccine. 2017;35(17):2110–4. 10.1016/j.vaccine.2016.12.072 [DOI] [PubMed] [Google Scholar]
  • 5. Boyce T, Gudorf A, de Kat C, et al. : Towards equity in immunisation. Euro Surveill. 2019;24(2). 10.2807/1560-7917.ES.2019.24.2.1800204 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Countdown 2008 Equity Analysis Group, . Boerma JT, Bryce J, et al. : Mind the gap: equity and trends in coverage of maternal, newborn, and child health services in 54 Countdown countries. Lancet. 2008;371(9620):1259–67. 10.1016/S0140-6736(08)60560-7 [DOI] [PubMed] [Google Scholar]
  • 7. Delamonica E: Trends in immunization inequity: evidence, rights, and planning. Lancet Glob Health. 2016;4(9):e582–3. 10.1016/S2214-109X(16)30189-9 [DOI] [PubMed] [Google Scholar]
  • 8. Hosseinpoor AR, Bergen N, Schlotheuber A, et al. : State of inequality in diphtheria-tetanus-pertussis immunization coverage in low-income and middle-income countries: a multicountry study of household health surveys. Lancet Glob Health. 2016;4(9):e617–26. 10.1016/S2214-109X(16)30141-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Clouston S, Kidman R, Palermo T: Social inequalities in vaccination uptake among children aged 0-59 months living in Madagascar: an analysis of Demographic and Health Survey data from 2008 to 2009. Vaccine. 2014;32(28):3533–9. 10.1016/j.vaccine.2014.04.030 [DOI] [PubMed] [Google Scholar]
  • 10. Pande RP, Yazbeck AS: What's in a country average? Wealth, gender, and regional inequalities in immunization in India. Soc Sci Med. 2003;57(11):2075–88. 10.1016/s0277-9536(03)00085-6 [DOI] [PubMed] [Google Scholar]
  • 11. Schoeps A, Ouédraogo N, Kagoné M, et al. : Socio-demographic determinants of timely adherence to BCG, Penta3, measles, and complete vaccination schedule in Burkina Faso. Vaccine. 2013;32(1):96–102. 10.1016/j.vaccine.2013.10.063 [DOI] [PubMed] [Google Scholar]
  • 12. Sodha SV, Dietz V: Strengthening routine immunization systems to improve global vaccination coverage. Br Med Bull. 2015;113(1):5–14. 10.1093/bmb/ldv001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. UNICEF: What does it take to immunize every child: Achieving equity in immunization.New York, NY;2015. Reference Source [Google Scholar]
  • 14. van den Ent MMVX, Yameogo A, Ribaira E, et al. : Equity and immunization supply chain in Madagascar. Vaccine. 2017;35(17):2148–54. 10.1016/j.vaccine.2016.11.099 [DOI] [PubMed] [Google Scholar]
  • 15. Kc A, Nelin V, Raaijmakers H, et al. : Increased immunization coverage addresses the equity gap in Nepal. Bull World Health Organ. 2017;95(4):261–9. 10.2471/BLT.16.178327 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. DeVito E, Parente P, de Waure C, et al. : A review of evidence on equitable delivery, access and utilization of immunization services for migrants and refugees in the WHO European Region.Copenhagen: WHO Regional Office for Europe;2017. Reference Source [PubMed] [Google Scholar]
  • 17. Favin M, Teinglass R, Fields R, et al. : Why children are not vaccinated: a review of the grey literature. Int Health. 2012;4:229–38. 10.1016/j.inhe.2012.07.004 [DOI] [PubMed] [Google Scholar]
  • 18. JSI: Haiti: Strategies for Improving Vaccination Coverage in Urban Poor Settings. Reference Source [Google Scholar]
  • 19. Mutua MK, Kimani-Murage E, Ettarh RR: Childhood vaccination in informal urban settlements in Nairobi, Kenya: who gets vaccinated? BMC Public Health. 2011;11(1):6. 10.1186/1471-2458-11-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Magadzire BP, Joao G, Lochlainn LN, et al. : Reducing missed opportunities for vaccination (MOV) in Mozambique: Findings from a MOV assessment in 2017. Unpublished. [Google Scholar]
  • 21. Ogbuanu IU, Li AJ, Anya BM, et al. : Can vaccination coverage be improved by reducing missed opportunities for vaccination? Findings from assessments in Chad and Malawi using the new WHO methodology. PLoS One. 2019;14(1):e0210648. 10.1371/journal.pone.0210648 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Sridhar S, Maleq N, Guillermet E, et al. : A systematic literature review of missed opportunities for immunization in low- and middle-income countries. Vaccine. 2014;32(51):6870–9. 10.1016/j.vaccine.2014.10.063 [DOI] [PubMed] [Google Scholar]
  • 23. Prosser W, Jaillard P, Assy E, et al. : System redesign of the immunization supply chain: Experiences from Benin and Mozambique. Vaccine. 2017;35(17):2162–6. 10.1016/j.vaccine.2016.09.073 [DOI] [PubMed] [Google Scholar]
  • 24. Pakistan Go: Expanded Program on Immunization.2018. Reference Source [Google Scholar]
  • 25. UNICEF. Reference Source [Google Scholar]
  • 26. Gavi: The Vaccine Alliance. Reference Source [Google Scholar]
  • 27. VillageReach. Reference Source [Google Scholar]
  • 28. Pakistan Go: Calculations, Planning Commission Form 1 (PC-1): Expanded Programme on Immunization (EPI), 2015-16 to 2019-20. Reference Source [Google Scholar]
  • 29. Gavi: Pakistan Joint Appraisal Report 2018. 2018. Reference Source [Google Scholar]
  • 30. Transitioning out of Gavi Support. 2019. Reference Source [Google Scholar]
  • 31. Unicef. WHOa: System design approach to improve the immunization supply chain.2018. Reference Source [Google Scholar]
  • 32. LLamasoft: Supply Chain Guru.2019. Reference Source [Google Scholar]
  • 33. Pakistan Bureau of Statistics: Pakistan Social And Living Standards Measurement Survey (PSLM) 2014-15 Provincial / District.2016. Reference Source [Google Scholar]
  • 34. MICS, Unicef: Surveys.2019. Reference Source [Google Scholar]
  • 35. ICF. NIoPSNPa: Pakistan Demographic and Health Survey 2017-18.Islamabad, Pakistan, and Rockville, Maryland, USA: NIPS and ICF;2019. Reference Source [Google Scholar]
  • 36. Kahn AL, Kristensen D, Rao R: Extending supply chains and improving immunization coverage and equity through controlled temperature chain use of vaccines. Vaccine. 2017;35(17):2214–6. 10.1016/j.vaccine.2016.10.091 [DOI] [PubMed] [Google Scholar]
  • 37. WHO: Vaccine Volume Calculator. [Google Scholar]
  • 38. Samant Y, Lanjewar H, Block L, et al. : Relationship between vaccine vial monitors and cold chain infrastructure in a rural district of India. Rural Remote Health. 2007;7(1):617. [PubMed] [Google Scholar]
  • 39. Lydon P, Schreiber B, Gasca A, et al. : Vaccine stockouts around the world: Are essential vaccines always available when needed? Vaccine. 2017;35(17):2121–6. 10.1016/j.vaccine.2016.12.071 [DOI] [PubMed] [Google Scholar]
  • 40. Haddad S, Bicaba A, Feletto M, et al. : System-level determinants of immunization coverage disparities among health districts in Burkina Faso: a multiple case study. BMC Int Health Hum Rights. 2009;9 Suppl 1:S15. 10.1186/1472-698X-9-S1-S15 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. Cockcroft A, Andersson N, Omer K, et al. : One size does not fit all: Local determinants of measles vaccination in four districts of Pakistan. BMC Int Health Hum Rights. 2009;9 Suppl 1:S4. 10.1186/1472-698X-9-S1-S4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. National Emergency Operations Plan for Polio Eradication 2017/2018. Islamabad, Pakistan: National Emergency Operations Centre;2017. Reference Source [Google Scholar]
  • 43. VillageReach: The Role of System Design for ImprovedImmunization Supply Chains (iSC): Insights from Mozambique.2016. Reference Source [Google Scholar]
  • 44. Privett N, Gonsalvez D: The Top ten global health supply chain issues: Perspective from the field. Operations Research for Health Care. 2014;3(4):226–30. 10.1016/j.orhc.2014.09.002 [DOI] [Google Scholar]
  • 45. Mission INH: Comprehensive UIP Review: A report on urban immunization.2018. [Google Scholar]
  • 46. Mitchell S, Andersson N, Ansari NM, et al. : Equity and vaccine uptake: a cross-sectional study of measles vaccination in Lasbela District, Pakistan. BMC Int Health Hum Rights. 2009;9 Suppl 1:S7. 10.1186/1472-698X-9-S1-S7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Zameer M, Phillips N, Folorunso O, et al. : Data_Pakistan_GatesOpenResearch.xlsx. figshare.Dataset.2020. 10.6084/m9.figshare.12032106.v1 [DOI]
Gates Open Res. 2021 Mar 8. doi: 10.21956/gatesopenres.14300.r30187

Reviewer response for version 1

Shawn T Brown 1,2

The manuscript describes an effort in Pakistan amongst international partners to attempt improve equity in vaccine coverage through supply chain design. The authors have utilized data from the Pakistan supply chain and worked closely with in-country decision-makers to analyze data in an effort to find metrics that are correlated to identify items in the current design that could be manipulated to improve the cost of the supply chain and improve equitable coverage in the country. While I am excited about the work and think that many of the points in the paper are interesting, I think overall the manuscript has several flaws that would need to be addressed significantly before I believe it could be considered a complete contribution to the topic.

Broadly, I highlight these below:

  1. The authors present three metrics where they look for some correlation with DTP coverage. It is arguable that in all three cases, the metrics did not correlate very well coverage, which I would not expect them to in the first place. All three are entirely static metrics and do not holistically speak to the overall challenges that are associated with coverage (or even vaccine availability) at a location. The first metric, the amount of cold capacity per surviving infant at district and below to some extent will indicate that there is enough capacity for storing the vaccines, but just having functional storage capacity is only one piece of the puzzle. How stable is the power supply at the location? How often is the refresh rate of the vaccine delivery? What are the expected wastage rates at these facilities based on vaccine demand? The are just a few questions that arise from storage needs that this metric does not capture. If I am reading correctly, and the point is a bit confusing, the metric is defined as surviving infants per one year period. Please clarify this. Again, is this based on functioning cold chain equipment? 8C or freezer? And wastage will present a huge factor here, especially in rural areas where the number of doses administered may be 1 or two a day, which means that cold chain per surviving infant would have to be higher than in an area where there are multiple patients per day.

  2. Both distance metrics, again, are very static measures and I am not surprised that they do not correlate to coverage. Again, distance is not necessarily a solid indicator of the quality of a supply route. The quality can vary quite a lot given urban vs. rural, traffic patterns, shipping vehicle, and frequency. This metric also does not speak at all to the cold chain capacity given on the vehicles that are transporting. So again, while I can see the metrics value, the overall design of the supply chain is not solely or significantly dependent on the distance.

  3. There is software being used in the study, Llamasoft's Supply Chain Guru, of which I am familiar. It is a very nice package. Other than it being mentioned in the methods, I am not sure at all how the software was utilized in the study. The results presented seem to be from the spreadsheet that is provided as a supplement on FigShare (which I think is great that the authors provided and shared). If there were results or methodologies that this software was used for (which is not open source, and therefore the reviewer would need to purchase to obtain), they should be described.

  4. There are no costs in this analysis. This should be straight forward given that the authors know the route distances, and if they know the route vehicles, it would add tremendously to the analysis. Again, just because a route is longer, doesn't mean it is less cost effective, as it may be a long route, with a vehicle that carries a lot of vaccine. I don't see how using the metric of just "shortening" trips immediately leads to an improved vaccine supply chain.

  5. I am not sure at all how this study addresses equity. I found the one paragraph that speaks to this basically just says "Look at our data, these are places with low immunization coverage." Equity in immunization coverage goes beyond "this is an area of low immunization coverage", and has fundamental components of policy not covered in this analysis.

  6. This reviewer finds the conclusions of the study follow from the analysis, in that they claim that one needs more data to gain a better picture of the supply chain. I agree, although while the lack of solid results in the analysis indicate there is a need for more data, this is not a result that I would say warrants publication.

  7. As this is an open research submission, I would be remiss to talk of availability of data and software. I applaud the authors providing the spreadsheet that was used to create their scatter plots, indeed a lot of the synthesized data is in there. There is much of the data that went is mentioned in the study that is not available openly. I am aware that there are always sensitivities with country data, so I will leave it to the journal to decide as to the necessity of providing it.

Overall, while I am sure that the team that authored this paper is doing great and relevant work, I think the work here reads more as a high-level brief than an actual academic, methods paper. And I think that supply chain design is important, but the static metrics devised here, the lack of economic analysis, and results that merely show that the metrics are not adequate to find correlation to the actual issues of coverage make this manuscript as submitted not very rigorous or useful. It is a good start as a preliminary analysis, but further work should be done to actually provide the government with a real ability to explore and change the system to promote vaccine coverage and equity.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Partly

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

No

Are sufficient details provided to allow replication of the method development and its use by others?

No

Reviewer Expertise:

Vaccine supply chain modeling and redesign.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Gates Open Res. 2021 Mar 2. doi: 10.21956/gatesopenres.14300.r30188

Reviewer response for version 1

Kath O'Reilly 1

It's nice to see something in cold chain logistics as this is an important and sometimes overlooked area. I would regard this analysis as a pilot study, as there wasn't really much that was convincing about the study. Consequently, I think the discussion should be more focused around what the researchers learnt and how researchers can move forward to make progress in this area. Data on vaccine stockouts and some sort of engagement with district level health practitioners would go a long way.

Abstract:

  • “potent immunization products” - replace with vaccines. PIP is a bit of a confusing term, and immunisation coverage is usually just the proportion of target product reached rather than a measure of whether they have an immunological response. Unless there’s a reason why ‘immunisation products’ would be anything other than vaccines, but I really cannot think of one, especially in the current setting. I don’t really understand why ‘potent vaccines’ is used as a term either - why not just vaccines? Your focus is delivery of vaccines, not whether they remain potent - e.g. There is nothing in the paper about cold chain.

Methods:

  • This paper is described as a “method article” - what’s new here?

  • Only looked at DTP3 coverage. To an extent this is okay, but if there is a specific DTP stockout this may bias the findings. It would be preferable to explore multiple antigens (eg. MCV).

  • As well as cold chain ‘capacity’ can you provide details of how this was measured? It a physical thing, so was it measured by a team or was a survey sent out? This isn’t clear to me. Capacity can sometimes be as simple as a cold storage box, but sometimes far bigger. Did you also have information on the stock available in each health facility?

  • For the health facility, what data are you using, e.g. Data provide by the MoH? How do you ascertain it’s functionality? For example is it open 7 days a week or something else? I’m being naive here because in some settings health centres don’t always precisely exist despite being there ‘on paper’. I’d also like clarification on the number of health centres per district and how the weighted average was calculated.

  • Will you provide this valuable information on health centres for other researchers?

  • Have you considered including health centres per 100,000 population or some similar measure - does this explain any of the differences in coverage?

  • What is a district store and how is it different to a resupply point?

Results:

  • Figure 2: It’s clear that the minimal target has been met in a majority of districts, and isn’t a ‘causal’ factor for poor coverage. What I find puzzling is why some districts have capacity that is orders of magnitude higher? I think you should provide reasons for this - I presume you have an idea?

  • I don’t understand what ‘direct delivery’ means, please explain in more detail. If the resupply point is cut out, who and how is it picked from the higher point? It feels like Fig 4 is an over-simplification of a complex process - will there not be unintended consequences?

  • You haven’t convinced me that supply chain is in any way (that you’ve measured) related to DPT coverage. Your caption for Fig 6 suggests that improving these would improve DPT coverage and I don’t think you can make that statement.

Discussion:

  • I’m not convinced that having sub-district data would improve the analysis. I’m also confused how you can calculate distance from resupply to health facility but not health facility to population. Worldpop provides excellent data on population location and surely you can include in analysis? I think it would be better to write that exploring distance of populations to health facility would be an important next step.

  • I agree with your statement about needing to know about stock availability, for me this is a critically missing piece of information in this analysis. I think you should be more explicit about what (important) factors were not included in this analysis, and what steps could be made to include them in the future.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Partly

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Partly

Are sufficient details provided to allow replication of the method development and its use by others?

No

Reviewer Expertise:

Quantitative epidemiology, public health in LMICs, immunisation coverage

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Gates Open Res. 2020 May 6. doi: 10.21956/gatesopenres.14300.r28754

Reviewer response for version 1

Michelle Seidel 1

1) Is the rationale for developing the new method (or application) clearly explained?

  • The Equity Reference Group views equity from 4 lenses gender, conflict, urban poor and remote rural settings. It is not clear how the methodology assessed these 4 settings other than subdividing between rural and urban sites.

2) Is the description of the method technically sound?

  • Slide - The supply chain metrics do not provide the assumptions for the parameters or the targets. The average resupply distance will negatively affect remote rural populations.

  • The paper assumes that vaccine availability and cold chain will lead to immunization. It does not address the demand side factors or other supply side factors such as the availability of human resources, transport availability, etc.

  • SC Metric 1: The analysis at district level for cold chain capacity is not useful seeing that vaccination services are delivered at the health facility level. It is not clear whether the analysis disaggregates the data at this level.

  • SC Metric 2: Does not consider Cold chain equipment and uses averages.

  • SC Metric 3: Does not control for variances in stock policies.

3) Are sufficient details provided to allow replication of the method development and its use by others?

  • Some of the assumptions and targets for the calculations are missing.

4) If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

  • It is not clear how the metrics were derived and how some of the results were calculated.

 

  • Considering equity certainly brings a new dimension to supply chain design. Although I would be cautious to deem the exercise in Pakistan as supply chain design but rather iSC optimization as it did not consider the entire health supply chain, which would further equity goals under UHC.

  • The deductions drawn from the analysis and the lack of data at service delivery level leads to inconclusive results. The use of averages, particularly for distances is problematic when analyzing equity and I would suggest a categorization of distances rather.

  • The analysis of equity in conflict zones is another opportunity this paper could have addressed in Pakistan.

  • With some revisions to the paper, I would be happy to recommend it for indexing.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Partly

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Partly

Are sufficient details provided to allow replication of the method development and its use by others?

Partly

Reviewer Expertise:

Immunization supply chain

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Gates Open Res. 2020 May 28.
Mariam Zameer 1

1) Is the rationale for developing the new method (or application) clearly explained?

  • The Equity Reference Group views equity from 4 lenses gender, conflict, urban poor and remote rural settings. It is not clear how the methodology assessed these 4 settings other than subdividing between rural and urban sites.
    • Thank you for highlighting this. We agree that equity can be looked at from many different perspectives. Urban and rural are geographic dimensions that we focused on, while gender and conflict are socio-economic dimensions. Many people living in urban poor areas can be internally displaced as part of conflict settings.

 

  While socioeconomic and gender barriers are useful equity dimensions, these data points were not consistently available as disaggregated data from the district, sub-district and health facility level, as acknowledged in the paper. The suggestion is well-taken, and we acknowledge the different dimensions on equity in the introduction paragraph 1: “Under-immunized children are more likely to be of lower socioeconomic status than fully immunized children and are geographically clustered in marginalized communities, such as urban poor, remote rural, mobile populations and communities affected by conflict, resulting in a higher burden of vaccine-preventable disease in these communities”. The disaggregated from the census survey as well as urban slum mapping was not available at the time of the analysis and concluded after the system design analysis; these would provide useful data points for further research and analysis that were not covered under the current scope.

2) Is the description of the method technically sound?

  • Slide - The supply chain metrics do not provide the assumptions for the parameters or the targets. The average resupply distance will negatively affect remote rural populations.
    • In the Discussion section, we acknowledge that average resupply distance is a more appropriate metric for remote rural areas. One of our recommendations moving forward is the need to develop specific metrics for urban areas or other equity lens. The assumptions for the targets are outlined in Figure 1 and section on “Supply Chain Metrics”, e.g. for Metric 2, we state that “we set a target of 60 kilometers as a feasible trip for a vaccinator to travel from a health facility to its resupplying store and return in one business day, based on the average travel speeds reported during primary data collection across all provinces.”

 

In the Discussion, we also acknowledged that averaging resupply distances across districts could potentially mask the longer distances by inclusion of a number of relatively shorter distances. However, this potentially doesn’t drastically affect one population than the other as the metric was applied to all Provinces/Areas. With further data made available, Provinces could be further divided into not only urban/rural (based on distance) but also time and road accessibility to health facilities, which will accommodate for additional peculiarities of remote rural/urban populations. This is recommended as a next step for further analysis.

 

  • The paper assumes that vaccine availability and cold chain will lead to immunization. It does not address the demand side factors or other supply side factors such as the availability of human resources, transport availability, etc.
    • Agree with the reviewer that we do not look at the other supply or demand side factors, and this is outlined in the introduction and framing paragraph 2: “As part of the effort to improve equity in immunization coverage, immunization products must be available and of sufficient potency in the communities in which low coverage rates persist. To achieve this, immunization supply chains should be tailored to address health system barriers to equity in service delivery such as vaccine stockouts, inadequate storage capacity, non-functional cold chain equipment (CCE), as well as difficult terrain and road conditions. However, there is limited research and evidence on how the design of supply chains can be tailored to deliver potent vaccines to all children.”

 

This analysis has focused on supply chain design’s contributions to programmatic outcomes. One limitation is that the factors considered in this analysis are only few of the supply and demand factors that contribute to programme performance. Other factors such as HR availability and competence, data availability and use, health facilities opening times, awareness creation etc. have an impact on performance.’

 

  • SC Metric 1: The analysis at district level for cold chain capacity is not useful seeing that vaccination services are delivered at the health facility level. It is not clear whether the analysis disaggregates the data at this level.
    • We acknowledge this as a limitation of the study due to lack of data that was available at the health facility level, and lack of integrated data systems making it impossible to match health facility data across different databases. This is highlighted in the Discussion section. The authors believe that this is not peculiar to Pakistan and that systematic sub-national supply chain and programmatic performance capture and reliability of such data is challenging in similar countries. This study is however an important step in moving the supply chain contribution to programme outcomes discussion forward. Our recommendation is that this analysis should be extended to the health facility and community level where data is available, as outlined in the Discussion section. For Pakistan, as a first step, we think it is still valuable to identify districts and then be able to deep-dive for the specific districts.
  • SC Metric 2: Does not consider Cold chain equipment and uses averages.
    • Request to the reviewer to clarify, as this metric is related to resupply distance and not cold chain storage. No average was used.
  • SC Metric 3: Does not control for variances in stock policies.
    • In Pakistan, there were no variances in stock policies for districts to the best of our knowledge.

3) Are sufficient details provided to allow replication of the method development and its use by others?

  • Some of the assumptions and targets for the calculations are missing.
    • Please let us know the assumptions and targets that are missing, so we add them appropriately. The targets for all metrics and calculations are presented in the Supply Chain Metrics section, and in Figure 1.

4) If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

  • It is not clear how the metrics were derived and how some of the results were calculated.
    • Request to reviewer to provide more details, so we are able to add more specific details. The Supply Chain Metrics section outlines how each of these metrics was derived and validated and presented in Figure 1. The data for each of the metrics is also available with the paper.

 

  • Considering equity certainly brings a new dimension to supply chain design. Although I would be cautious to deem the exercise in Pakistan as supply chain design but rather iSC optimization as it did not consider the entire health supply chain, which would further equity goals under UHC.
    • The approach to supply chain design is based on the direction from the government, in this case the Government of Pakistan. We assessed and recommended changes to supply chain design based on direction from the federal and provincial government. Network optimization for supply chains usually uses specific modeling tools, which were also used in Pakistan. However, the authors went further than looking at the modeling results to look at comprehensive supply chain design.

 

The specific distinction between SC design and iSC optimization is misapplied here. The analysis presented here considers the system’s design and applied an equity lens using three metrics. From experience in multiple countries, System Design analysis could be a country-wide/entire supply chain or a specific scope (e.g. last mile, outreaches, cold chain utilization etc.).

 

  • The deductions drawn from the analysis and the lack of data at service delivery level leads to inconclusive results. The use of averages, particularly for distances is problematic when analyzing equity and I would suggest a categorization of distances rather.
    • Refer to above for clarification of the use of averages. Averages was used as a form of categorizing findings. We see this work as a Methods paper, to further the conversation on equity in this area. As this methodology is repeated in other countries and refined, we would anticipate seeing trends that are helpful.
  • The analysis of equity in conflict zones is another opportunity this paper could have addressed in Pakistan.
    • This was not an ask by the government of Pakistan, and the data on this is not available.
Gates Open Res. 2020 Sep 15.
Mariam Zameer 1

Thank you to the reviewer for the detailed comments. We respond to each comment below to clarify information and limitations that were included and acknowledged in the paper. We request clarifications for the changes and revisions that are requested to recommend the paper for indexing. We hope that our response would alleviate the reviewer's concerns and fully approve the paper, so it is available for other countries to use and to improve our methodology.

1) Is the rationale for developing the new method (or application) clearly explained?

  • The Equity Reference Group views equity from 4 lenses gender, conflict, urban poor and remote rural settings. It is not clear how the methodology assessed these 4 settings other than subdividing between rural and urban sites.
    • Response: Thank you for highlighting this. We agree that equity can be looked at from many different perspectives. Urban and rural are geographic dimensions that we focused on, while gender and conflict are socio-economic dimensions. Many people living in urban poor areas can be internally displaced as part of conflict settings.  While socioeconomic and gender barriers are useful equity dimensions, these data points were not consistently available as disaggregated data from the district, sub-district and health facility level, as acknowledged in the paper. The suggestion is well-taken, and we acknowledge the different dimensions on equity in the introduction paragraph 1: “Under-immunized children are more likely to be of lower socioeconomic status than fully immunized children and are geographically clustered in marginalized communities, such as urban poor, remote rural, mobile populations and communities affected by conflict, resulting in a higher burden of vaccine-preventable disease in these communities”. The disaggregated from the census survey as well as urban slum mapping was not available at the time of the analysis and concluded after the system design analysis; these would provide useful data points for further research and analysis that were not covered under the current scope.

2) Is the description of the method technically sound?

  • Slide - The supply chain metrics do not provide the assumptions for the parameters or the targets. The average resupply distance will negatively affect remote rural populations.
    • Response: In the Discussion section, we acknowledge that average resupply distance is a more appropriate metric for remote rural areas. One of our recommendations moving forward is the need to develop specific metrics for urban areas or other equity lens. The assumptions for the targets are outlined in Figure 1 and section on “Supply Chain Metrics”, e.g. for Metric 2, we state that “we set a target of 60 kilometers as a feasible trip for a vaccinator to travel from a health facility to its resupplying store and return in one business day, based on the average travel speeds reported during primary data collection across all provinces.” In the Discussion, we also acknowledged that averaging resupply distances across districts could potentially mask the longer distances by inclusion of a number of relatively shorter distances. However, this potentially doesn’t drastically affect one population than the other as the metric was applied to all Provinces/Areas. With further data made available, Provinces could be further divided into not only urban/rural (based on distance) but also time and road accessibility to health facilities, which will accommodate for additional peculiarities of remote rural/urban populations. This is recommended as a next step for further analysis.
  • The paper assumes that vaccine availability and cold chain will lead to immunization. It does not address the demand side factors or other supply side factors such as the availability of human resources, transport availability, etc.
    • Response: Agree with the reviewer that we do not look at the other supply or demand side factors, and this is outlined in the introduction and framing paragraph 2: “As part of the effort to improve equity in immunization coverage, immunization products must be available and of sufficient potency in the communities in which low coverage rates persist. To achieve this, immunization supply chains should be tailored to address health system barriers to equity in service delivery such as vaccine stockouts, inadequate storage capacity, non-functional cold chain equipment (CCE), as well as difficult terrain and road conditions. However, there is limited research and evidence on how the design of supply chains can be tailored to deliver potent vaccines to all children.” This analysis has focused on supply chain design’s contributions to programmatic outcomes. One limitation is that the factors considered in this analysis are only few of the supply and demand factors that contribute to programme performance. Other factors such as HR availability and competence, data availability and use, health facilities opening times, awareness creation etc. have an impact on performance.’
  • SC Metric 1: The analysis at district level for cold chain capacity is not useful seeing that vaccination services are delivered at the health facility level. It is not clear whether the analysis disaggregates the data at this level.
    • Response: We acknowledge this as a limitation of the study due to lack of data that was available at the health facility level, and lack of integrated data systems making it impossible to match health facility data across different databases. This is highlighted in the Discussion section. The authors believe that this is not peculiar to Pakistan and that systematic sub-national supply chain and programmatic performance capture and reliability of such data is challenging in similar countries. This study is however an important step in moving the supply chain contribution to programme outcomes discussion forward. Our recommendation is that this analysis should be extended to the health facility and community level where data is available, as outlined in the Discussion section. For Pakistan, as a first step, we think it is still valuable to identify districts and then be able to deep-dive for the specific districts.
  • SC Metric 2: Does not consider Cold chain equipment and uses averages.
    • Response: Request to the reviewer to clarify, as this metric is related to resupply distance and not cold chain storage. No average was used.
  • SC Metric 3: Does not control for variances in stock policies.
    • Response: In Pakistan, there were no variances in stock policies for districts to the best of our knowledge.

3) Are sufficient details provided to allow replication of the method development and its use by others?

  • Some of the assumptions and targets for the calculations are missing.
    • Response: Please let us know the assumptions and targets that are missing, so we add them appropriately. The targets for all metrics and calculations are presented in the Supply Chain Metrics section, and in Figure 1.

4) If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

  • It is not clear how the metrics were derived and how some of the results were calculated.
    • Response: Request to reviewer to provide more details, so we are able to add more specific details. The Supply Chain Metrics section outlines how each of these metrics was derived and validated and presented in Figure 1. The data for each of the metrics is also available with the paper.
  • Considering equity certainly brings a new dimension to supply chain design. Although I would be cautious to deem the exercise in Pakistan as supply chain design but rather iSC optimization as it did not consider the entire health supply chain, which would further equity goals under UHC.
    • Response: The approach to supply chain design is based on the direction from the government, in this case the Government of Pakistan. We assessed and recommended changes to supply chain design based on direction from the federal and provincial government. Network optimization for supply chains usually uses specific modeling tools, which were also used in Pakistan. However, the authors went further than looking at the modeling results to look at comprehensive supply chain design. 
    • The specific distinction between SC design and iSC optimization is misapplied here. The analysis presented here considers the system’s design and applied an equity lens using three metrics. From experience in multiple countries, System Design analysis could be a country-wide/entire supply chain or a specific scope (e.g. last mile, outreaches, cold chain utilization etc.).
  • The deductions drawn from the analysis and the lack of data at service delivery level leads to inconclusive results. The use of averages, particularly for distances is problematic when analyzing equity and I would suggest a categorization of distances rather.
    • Response: Refer to above for clarification of the use of averages. Averages was used as a form of categorizing findings. We see this work as a Methods paper, to further the conversation on equity in this area. As this methodology is repeated in other countries and refined, we would anticipate seeing trends that are helpful.
  • The analysis of equity in conflict zones is another opportunity this paper could have addressed in Pakistan.
    • Response: This was not an ask by the government of Pakistan, and the data on this is not available
Gates Open Res. 2020 Apr 20. doi: 10.21956/gatesopenres.14300.r28753

Reviewer response for version 1

Wendy Prosser 1, Cary Spisak 1

General comments:

This article is very timely as it reflects the global interest in equity in immunization programs and the goal to have more equitable coverage rates across all countries. This is also timely as the article reflects the desire to have more evidence linking supply chain performance to immunization coverage. The two are intuitively linked, as vaccines must be available at service delivery points in order for vaccines to be administered. However, the evidence definitely showing that link is scant.

This article is a considerable first step to move forward the conversation linking supply chain, coverage and equity. The article does a thorough job of highlighting the limitations and challenges with this type of analysis, which may have prevented finding any clear trends or links between supply chain and coverage equity.

The metrics as defined by the methodology are judicious and are an innovative twist on standard supply chain indicators in use. It is disappointing that the results of the analysis were either ambiguous or showed no overarching trend or pattern between the chosen supply chain metrics and coverage. The article dutifully addresses this in the limitations, noting the many influential factors on coverage rates and inequity. Because of this ambiguity in the link between supply chain and equity, it would benefit the article to adjust some language to be less definitive about these ambiguous links for the results of each metric (more details on suggestions to address this are listed below).

The Conclusions don’t necessarily match the results as the analysis showed no notable relationships between the supply chain metrics and immunization coverage. It almost risks narrowing the vision of the article to interpret the results based on expected outcomes. However, as the article notes, the analysis contributed to alternative supply chain configurations that can reduce the distance to resupply points, which does benefit health workers, the supply chain, and, presumably coverage. It is noteworthy to see this thoughtful analytical approach to better supply chain design, and we hope other countries would adapt similar evidence-based approaches to supply chain design.

Despite the limitations, this article forces the necessary conversation linking supply chain and coverage when making decisions about the supply chain design. The article provides examples of decisions related to the design of the supply chain, for example average hours saved if direct delivery was in place, that could increase the hours a healthcare worker has to provide care for children. As the article marginally notes, this is one of many factors that can influence coverage and equity.

A noted limitation was the data availability and not being able to analyze the metrics at the facility level and instead using the district level for analysis. This is particularly hindering for cold chain capacity as it may be masking capacity variability among service delivery points. These reviewers agree with the recommendation that using this methodology at the facility level would provide further insight and potentially stronger demonstration of the influence of supply chain design on coverage and equity.

Questions for clarification:

In the methods (paragraph 1), how do you define “demand” for vaccines as there are many influential factors to true demand? Or is this referring to estimated need?

Please clarify if CCE considered in this analysis is only used for vaccines.

Specific comments:

The last sentence in the abstract states that the analysis “highlighted supply chain contributions to program performance.” This reviewer’s understanding of the results is this contribution was not clearly demonstrated although this methodology clearly informed future health system investments and guided decision making for those investments.

The first sentence of the Conclusion notes that design is often based on cost-efficiency. It is important to note, however, that cost is also an indicator or proxy for different aspects of system performance, such as distance driven. Another notable design influencer is administrative structure of the government and system.

Specific instances of definitive language that could be adjusted to factor in the ambiguity:

  • Last sentence of introduction section: suggest a re-write of this sentence to state this as more of a hypothesis of what this approach aimed to do. As it is currently written, it assumes a connection between design and increasing equity, which the analysis does not definitively determine.

  • Results section, “Comparing cold chain capacity and immunization coverage.” The article notes that the relationship between these metrics is ambiguous. The statement in the paragraph on disparities in urban-rural coverage rates, “which might be a result of disparities in cold chain distribution within the district”, is not necessarily based on the results of the analysis which are ambiguous. Suggest re-writing this sentence to recognize the many influential factors in coverage, which are dutifully noted in the introduction of the article.

  • Results section, “Comparing weighted average resupply distance to health facilities and immunization coverage.” The average resupply distance in Balochistan, KP and Punjab and relatively similar, yet the coverage rates are quite different. As you state for Balochistan, other factors underpin low coverage. Similarly, other factors underpin high coverage in the other two provinces. How the paragraph is written is biasing the analysis to the ambiguous influence of the supply chain. Suggest a re-write of those two related sentences.

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

NA

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Citations

    1. Zameer M, Phillips N, Folorunso O, et al. : Data_Pakistan_GatesOpenResearch.xlsx. figshare.Dataset.2020. 10.6084/m9.figshare.12032106.v1 [DOI]

    Data Availability Statement

    Underlying data

    All data underlying the results are available as part of the article and no additional source data are required.

    Extended data

    Figshare: Data_Pakistan_GatesOpenResearch. https://doi.org/10.6084/m9.figshare.12032106.v1 47.

    This file contains the immunization supply chain equity metrics for Pakistan produced in this study.

    Extended data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

    Articles from Gates Open Research are provided here courtesy of Bill & Melinda Gates Foundation

    RESOURCES