Strengthening the immunization supply chain through EVM assessment: Comparing India’s two successive national assessments

Abstract

Background:

Effective immunization supply chain (iSC) is crucial for safe and timely transport of potent vaccines to the beneficiary. India’s iSC, with a network of ~29,000 cold chain points (CCP), measures its quality standards using the World Health Organization–United Nations International Children’s Emergency Fund (WHO–UNICEF) global tool on effective vaccine management (EVM). The two national EVM assessments (EVMA) were conducted in 2013 and 2018. This study helps to see the impact of policies and practices through EVMA in maintaining an efficient iSC for effective implementation of immunization program.

Materials and Methods:

We conducted a desk analysis using EVMA reports to summarize and compare the findings of the two studies. Cut-off of 80% was considered ideal for each category/criteria score.

Results:

Both EVMA 2013 and 2018 were conducted using Android-based global EVM tool, though across a wider sample of CCP. Maximum and minimum changes in scores were sub-national and lowest distribution (16% each) and national buffer stores (7%), respectively. Maximum and minimum improvements were seen in vaccine management (29%) and MIS and supportive functions (3%), respectively. The improvement was statistically significant for the overall scores (P = 0.02), primary (P = 0.01), subnational (P = 0.02), and lowest distribution stores (P = 0.03). Among the 36 recommendations of 2013 assessment, 78% and 22% were fully and partially implemented (or ongoing), respectively.

Conclusion:

Implementation of EVM recommendations has significantly led to improvement and continues to provide a benchmark for iSC and its processes. Follow-up assessments every 3–5 years can further help to evaluate iSC and ascertain the impact of recommendations.

Introduction

Most public health programs are implemented primarily at the community level, and India’s Universal Immunization Program (UIP) has often provided a gateway for providing primary health services to all communities, irrespective of their geographic location, caste, and religion. A timely delivery of safe and potent vaccines, which relies on an efficient cold chain system, is the course of action for every health facility to effectively implement routine immunization and other health programs. Successful immunization program is built on functional, end-to-end supply chain and logistics system. Its essential role is to ensure effective vaccine storage, handling, and stock management; rigorous temperature control system; and maintenance of adequate logistics management information system.

Immunization supply chains (iSCs) – the network of staff, equipment, vehicles, and data needed to get vaccines safely from the manufacturer to the people who need them – were first developed in the late 1970s with the launch of the Expanded Program on Immunization. Between 2011 and 2015, low- and middle-income countries had requested US$289 million from Gavi to update and strengthen their iSC systems, complemented by government and other sources of funding for supply chain improvements.[1] India’s iSC is a vast network of ~29,000 cold chain points, which cater to a huge cohort of ~26 million infants and ~ 30 million pregnant women for administering ~390 million doses through 9 million sessions annually.[2] In total, 750 (3%) of the cold chain points are located at the district level and above and the remaining 95% are located below the district level, which includes urban health facilities, primary health centers, community health centers, and subcenters. India is also having one of the world’s largest cold chain infrastructure which includes around 76,000 electrical cold chain equipment, around 55,000 cold chain handlers and more than 500 cold chain technician, and 2.5 million health-care workers.[2,3]

An efficient iSC ensures the availability of the right vaccine in the right quantity in the right condition at the right time, right place, and right cost.[4] As a result, an efficient and evenly targeted immunization program and its ability to reduce the burden of vaccine-preventable diseases (VPDs) will greatly contribute to reaching the sustainable development goal (SDG) number 3 target by 2030, that is, reducing the infant mortality rate to 25 per 1000 live births.[2]

With the rising vaccine costs and launch of newer vaccines in a country’s immunization program, countries must make efforts to augment their cold chain system and reduce vaccine wastage, forecast accurate vaccine requirements, and ensure effective and efficient iSC.[5] The World Health Organization–United Nations International Children’s Emergency Fund (WHO–UNICEF) effective vaccine management (EVM) initiative provides materials and tools required to assess the iSC and help countries improve their supply chain performance. It is based on the principle that repeated assessment of iSC against well-established standards of quality management (such as ISO 9000 series of standards) can help guide policymakers and program managers, and hence improve the performance of country’s iSC.[6] EVM assessment tool consists of supplementary guidance material to conduct a structured assessment. EVM principles span across nine criteria and seven categories, each with a minimum target score of 80%, and aim to avoid vaccine damage due to exposure to temperatures outside the recommended range. The nine EVM criteria include the following: vaccine arrival process; temperature during storage and transport; storage capacity; buildings, cold chain equipment and vehicle; maintenance and repair; stock management; vaccine distribution; vaccine management; and management information system (MIS) and supportive functions. The seven EVM categories include buildings, capacity, equipment, management, repair/maintenance, training, and vehicles.

The EVM tool may be used for two different purposes – as an assessment tool for precise analysis of strengths and weaknesses of the iSC and as a supervisory aid to observe and contribute to the long-term progress of individual cold chain points. The EVM scores further contribute to preparation of the EVM Improvement Plan for implementation to strengthen the iSC through good storage and distribution practices.

The EVM assessments can be conducted at national or subnational level. Since the inception of the EVM assessments in India, the Government of India (GoI) has conducted 29 EVM assessments, including two national EVM (NEVM) assessments. India conducted its first NEVM assessment in 2013 and follow-up NEVM assessment in 2018. Following the NEVM assessment 2013, a continuous improvement plan (CIP) was developed and implemented. The follow-up NEVM assessment in 2018 evaluated the progress of India’s iSC across all the nine criteria and assessed the impact of CIP on strengthening of iSC. This study will help us determine the key challenges in maintaining an efficient supply chain system, assess the progress in recent years, and identify areas with scope of further improvement.

Materials and Methods

This study was a cross-sectional desk analysis of findings and data sets from two NEVM assessments conducted in 2013 and 2018, respectively. The data was retrieved from archives of the National Cold Chain and Vaccine Management Resource Center, the National Institute of Health and Family Welfare, New Delhi, India. Reports of both NEVM assessments were used to retrieve information on sampling methodologies, sampling frame, same size, criteria and category scores, and other relevant findings from each study. This study summarizes the methodologies, followed by estimation of comparative assessment of scores for both studies. Any gaps in data or study findings were traced back to the members of the core research team for clarifications, wherever possible. The data was analyzed using Microsoft Excel and Statistical Package for the Social Sciences (SPSS) version 24. Unpaired t-test was applied to determine the P value. P value less than 0.05 was considered statistically significant.

Since there was no primary data collection, the study was exempted from ethics clearance, though it was, however, reviewed by the Program Advisory Committee at the national institute of health and family welfare (NIHFW).

Results

India has adopted four-tier supply in immunization program. It implies that the vaccine manufacturers send the vaccines directly to the primary stores, which further supply the vaccines to lower stores. NEVM assessments give a cross-sectional sneak peek into the iSC for any country. Two NEVM assessments conducted in 2013 and 2018 in India were both conducted by the National cold chain and vaccine management resource centre (NCCVMRC), in collaboration with United Nations International Children’s Emergency Fund (UNICEF) and ministry of health and family welfare (MoHFW). Both assessments were based on the global WHO–UNICEF EVM guidelines, with a minimum target score of 80% for individual criterion/category. Key features of both assessments are summarized in Table 1. EVM captures the structured assessment for the last 1-year review period. After data collection, the data sets were run in the EVM assessment tool to generate an overall score in percentage for the individual criteria at each level of the supply chain assessed.

Table 1.

Summary of NEVM assessments methodology 2013 and 2018, India

Variable	NEVM 2013	NEVM 2018
Study review period (1 year)	From March 2012 to Feb 2013	From May 2017 to April 2018
Sample size selection	WHO site selection tool
Confidence interval	85%	85%
Precision rate	15%	10%
Number of states visited	11	23
Number of assessors	41	74
Total sample	114	145
National buffer stores	4	4
Primary stores	16	40
Subnational stores	14	11
Lowest delivery stores	28	30
Service point stores	52	60
Study tool	WHO EVM version 1.0.5.0	WHO EVM version 1.0.9.0
Data collection	Android-based KoboCollect application

NEVM=national effective vaccine management, WHO=World Health Organizaiton

Overall scores of NEVM assessment in 2013 and 2018 were 53% and 68%, respectively. As depicted in Figure 1, the improvement in overall scores was highest for subnational and lowest distribution stores (16% each), followed by the primary stores and service delivery points (14% each). Minimum improvement was seen for scores at national buffer stores (7%) between the two EVM assessments conducted in 2013 and 2018.

Figure 1.

Progress of overall NEVM scores across different levels of immunization supply chain. NEVM = national effective vaccine management

Criteria scores

Comparative criteria scores across different levels of iSC during NEVM 2013 and 2018 are presented in Table 2. Overall criteria scores across all levels of iSC show improvement in the mean score across all criteria, with the maximum improvement seen in vaccine management (29%), temperature monitoring (18%), vaccine arrival, storage capacity, and distribution (17% each), stock management (14%), building, cold chain equipment (CCE), and vehicle repair and maintenance (7% each), and MIS and supportive functions (3%) between NEVM 2013 and 2018. The difference in the mean scores was statistically significant for the overall scores (P = 0.02), primary (P = 0.01), subnational (P = 0.02), and lowest distribution stores (P = 0.03), thereby rejecting the null hypothesis.

Table 2.

Comparative criteria scores across different levels of supply chain, NEVM assessments 2013 and 2018, India

Year of NEVM assessment	E1: Vaccine arrival	E2: Temperature	E3: Storage capacity	E4: Building, cold chain Equipment & Transport	E5: Maintenance	E6: Stock management	E7: Distribution	E8: Vaccine management	E9: MIS, supportive functions	Mean±SD	P
Target (%)	80	80	80	80	80	80	80	80	80
Overall score (%)
2013	43	54	63	69	57	51	45	46	56	53±8.61	0.02
2018	60	72	80	76	64	65	62	75	59	68±7.73
National buffer store (%)
2013	52	37	71	65	59	57	24	29	50	49±16.13	0.38
2018	80	41	57	66	37	60	76	42	46	56±15.71
Primary store (%)
2013	34	43	66	64	61	56	41	50	65	53±11.83	0.01
2018	58	62	65	69	65	72	70	71	68	66±4.58
Subnational store (%)
2013	NA	46	46	69	59	49	39	35	52	49±10.32	0.02
2018	NA	67	61	66	65	66	65	70	59	64±3.44
Lowest distribution store (%)
2013	NA	71	57	70	58	46	42	47	58	56±10.73	0.03
2018	NA	78	82	81	66	66	68	71	65	72±7.12
Service point store (%)
2013	NA	70	76	75	49	45	77	67	0	57±26.21	0.20
2018	NA	78	94	81	65	60	52	83	52	70±15.58

NA=not applicable, NEVM=national effective vaccine management, SD=standard deviation

Category scores

Progress in category scores was seen across all iSCs and all categories. Comparative category scores across different levels of iSC during NEVM 2013 and 2018 are presented in Table 3. The overall category scores across all levels of iSC show improvement across all the seven categories, with the maximum improvement found in vehicle (21%), training (15%), management (15%), capacity (10%), repairs/maintenance (10%), buildings (8%), and equipment (4%) between NEVM 2013 and 2018. The management, training, and vehicle scores show an increasing trend at all five levels of supply chain from the government medical store depot (GMSDs) to service points. The difference in the mean scores was statistically significant for lowest distribution stores (P ≤ 0.01), thereby rejecting the null hypothesis.

Table 3.

Comparative category scores, NEVM assessments 2013 and 2018, India

Year of NEVM Assessment	Building	Capacity	Equipment	Management	Repair and maintenance	Training	Vehicles	Mean±SD	P
Target (%)	80	80	80	80	80	80	80
Overall score (%)
2013	70	73	71	47	54	73	53	63±11.1	0.05
2018	78	83	75	62	64	88	74	74±9.4
National buffer stores (%)
2013	60	70	71	43	59	57	42	57±11.5	0.74
2018	61	59	67	49	37	82	63	59±14.10
Primary store (%)
2013	69	69	61	45	57	74	56	61±9.9	0.09
2018	74	68	67	64	65	88	68	70±8.32
Subnational store (%)
2013	69	46	70	40	59	69	52	57±12.18	0.08
2018	73	62	61	62	65	83	73	68±8.20
Lowest distribution store (%)
2013	61	61	75	43	57	76	52	60±11.8	0.009
2018	81	83	80	65	66	87	84	78±8.83
Service point store (%)
2013	78	88	71	51	49	75	NA	68±15.5	0.26
2018	80	98	80	60	65	90	NA	78±14.42

NA=not applicable, NEVM=national effective vaccine management, SD=standard deviation

Improvement plan and its recommendations

Following the NEVM assessment 2013, a CIP was prepared through an exhaustive workshop held at the NCCVMRC, with representatives from MoHFW, the state government, and other immunization partners such as the World Health Organization (WHO), UNICEF, the united nations development programme (UNDP), and so on. The CIP included broad recommendations across four priority timelines, encompassing the following categories: management and policy, human resources, infrastructure (building, equipment, transport, and temperature monitoring), planning, documentation and MIS, capacity building, and supportive supervision and improvement in practices. In total, 36 recommendations were suggested in the CIP, among which 28 (78%) were fully implemented and the remaining eight (22%) were ongoing.

Discussion

The EVM is a continuous process that involves four major key activities – assess, plan, implement, and monitor – which allows focused efforts tailored to every country/state. An EVM assessment gives a methodical study of the strengths and weaknesses across the supply chain and is intended to set good vaccine storage and distribution practices. India’s NEVM assessments are one of the largest assessments in terms of its size and geographic scale of representation of their sample. Nearly three-fourths of the CIP recommendations have been fully implemented since NEVM assessment 2013, which shows ensuing improvement in the overall scores of NEVM. Marked improvement was seen in subnational and lowest distribution stores, and in the training, vehicles, vaccine and stock management, and storage capacity.

Gavi, the Vaccine Alliance, has identified functional cold chain equipment as one of the five fundamental areas that need to be improved in next-generation iSC.[7] In many low-income countries, the scale and geographic spread of cold chain system may be insufficient to meet the demands of the program. Insufficient and suboptimal cold chain capacity is known to hamper the availability of safe vaccines.[8] With the expansion of India’s UIP amid population growth and introduction of newer and costlier vaccines, the need of augmentation of cold chain space was imminent for smoother introduction and delivery of vaccines at the right place, at the right time, and in right quantity. Between 2013 and 2018, India’s network of cold chain points expanded from around 27,000 (2013) to around 28,780 (2018) cold chain points, thereby reaching wider geographic area and more underserved population.[3,9] The National Cold Chain Management Information System (NCCMIS) boasts increase in cold chain space as the number of electrical cold chain equipment increased from 50,519 (2013) to 85,889 (2018).[3] Procurement of WHO performance, quality and safety (PQS)-certified cold chain equipment and new insulated and refrigerated vehicles and establishment of National Technical Advisory Body (NTAB) for introduction of new technology are a few steps ahead in strengthening the cold chain system in India. The GoI’s efforts toward integrating and expanding iSC are helping to provide service delivery in remote and rural parts of the country. It allows for strengthening of primary health care service delivery in most underserved and hard-to-reach areas, thereby enforcing the manpower through infrastructure, resources, and capacity building.

India’s iSC is managed by a host of iSC workforce, namely, senior- and mid-level program managers (including the medical officers in-charge at the state, district, and primary health centers (PHC) level), cold chain technicians (CCT), and the vaccine and cold chain handlers (VCCH). As a sequel to the NEVM 2013 recommendations, a training module for VCCH was developed in 2016. Nearly 55,000 VCCH across India have been trained in the VCCH module 2016 through a cascade mode of training. In addition to the VCCH module, customized training modules were developed, such as Training on Vaccine and Cold Chain Management (T-VaCC) for senior- and mid-level program managers and training on Repair and Maintenance of ICE lined refrigerator (ILR), deep freezer (DF) and voltage stabilizer (VS) and walk in cooler (WIC)/walk in freezer (WIF) for the cold chain technician (CCT), to ensure capacity building of other cadres of immunization workforce in India. As on March 15, 2021, 273 program managers and 821 CCT have been trained in T-VaCC and CCT trainings, respectively.[10] Capacity building of immunization workforce has also contributed to improvement in efficiency of iSC, as 79% districts have their cold chain sickness rate below 2%.[3]

Inadequate temperature monitoring and maintenance systems is often a crippling pillar in maintaining efficient iSC in a country. Many cold chain inventories show that 10%–46% of cold chain equipment are exposing vaccines to the risk of sub-zero temperatures. Furthermore, since continuous temperature monitoring device (CTMD) is not commonly found in low- and middle-income countries, exposure to sub-zero temperature may often be missed.[8] The Electronic Vaccine Intelligence Network (eVIN) was launched in India in 2017 and has now been scaled up to the entire country. As of 2021, all districts have digitized the storage temperature and vaccine stock for real-time monitoring across all cold chain points, and more than 50,000 temperature loggers have been installed so far.[11] Utilization of eVIN and recently introduced freeze tags and freeze-free vaccine carriers will further strengthen the immunization services by allowing storage and transportation of vaccines at the recommended temperature. A reliable immunization program can contribute toward equity in immunization and also act as a mode for introducing modern health care in underserved and remote parts of the country.

India launched a customized five-module Management Information System, which has revolutionized the management of iSC in the country. Five modules include the cold chain supply chain repository (NCCMIS), spare parts module, training module, immunization training management information system (iTMIS), supportive supervision, and a noninterventional dashboard (S4i or supervision for immunization). While the NCCMIS is the cold chain repository for more than 76,000 electrical and more than 1.2 lakh nonelectrical cold chain equipment, the development of iTMIS aims toward tracking the training status and training load of nearly 57,000 iSC workforce in the country.[10] The GoI has streamlined the process of installation, maintenance, repair, phase-out, and condemnation of cold chain equipment through the NCCMIS. Spare parts module of MIS is a web-based platform for indenting spare parts, similar to platforms such as eBay or Amazon (supplied by Government of India at free of cost). Supportive supervision hosts a series of checklists and questionnaires used for supportive supervision for cold chain and immunization session sites. S4i, however, is a visual dashboard that compiles and collates all data publicly available to assess the immunization status for all states/districts.[3] The eVIN has helped achieved more than 80% reduction in instances of vaccine stockouts and has ensured improved availability of adequate and potent vaccines across the country.[11] Program managers at the state and district levels can utilize this tool to understand the functionality and efficiency of the iSC (both infrastructure and human resource (HR)) in their respective jurisdiction for timely and better decision-making.[12]

Failure to improve supply chain adversely affects the immunization program by increasing the cost implications and lowering the system performance.[12] Poor documentation, particularly for distribution and vaccine arrival procedures, has remained as one of the potential contributors in limited improvement of EVM scores. Use of standardized stock and distribution registers, temperature monitoring logbooks, and indent or issue vouchers is essential to further strengthen the stock management practices across the country. Training of VCCH and on-the-job training through supportive supervision can help to improve the vaccine and stock management practices.[8] Vaccine forecasting through real-time stock levels in the eVIN can further help to ensure availability of the right amount of right vaccine at the right time and right place.

While implementation of CIP recommendations from 2013 and 2018 has contributed to marked improvement of iSC in India, an efficient and effective cold chain system depends on sustenance of good practices and rectification of weaknesses. The current interventions like eVIN for stock management and temperature monitoring, vaccine distribution, supportive supervision, management information system, training and capacity building initiatives are some of the key interventions which exhibit India’s marathon-like steps toward improving the iSC of the country. Comprehensive evaluation by EVM assessments provides assessment of a country’s supply chain against predefined benchmarks and helps prepare a roadmap.[13] Though EVM assessments can further guide the way, however, strong political commitment and collaboration with focused efforts is essential to ensure success in improving the iSC for any country.[7,14,15]

Conclusions

CIP developed based on the NEVM assessment 2018 findings is currently in various stages of implementation. Furthermore, the scale-up of new vaccines, such as rotavirus vaccine, pneumococcal vaccine, measles–rubella and tetanus–diphtheria vaccine, and introduction of coronavirus disease 2019 (COVID-19) vaccine on a large scale escalate the need to further strengthen the iSC in India. Implementation of an efficient iSC is in the hands of health workforce at grassroots level, which must be strengthened to provide safe and potent vaccines in India.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.