Cost and effectiveness analysis of immunization service delivery support in Andhra Pradesh, India

Abstract

The immunization service delivery support (ISDS) model was initiated in Andhra Pradesh, India, in November 2003 with the aim of strengthening immunization services through supportive supervision. The ISDS model involves a well-established supervision system built upon the existing health infrastructure. The objectives of this approach are to: (1) identify areas of high performance and those that need improvement, (2) assist staff in identifying and correcting wrong practices, (3) improve staff skills, (4) motivate staff, and (5) initiate corrective actions at appropriate levels through information sharing. An evaluation of cost and effectiveness of ISDS in 16 districts that participated in the programme found that the incremental cost associated with three rounds of supportive supervision visits was approximately US$ 110 630 (US$ 36 877 per round). The performance of health centre and immunization sessions was evaluated using 43- and 28-point checklists, respectively, and demonstrated significant improvement during and following the two-year implementation of ISDS. The average percentage change in health centre performance scores from baseline to the fourth round of evaluation was approximately 36%, and immunization session performance scores increased by an average of 9%. The incremental costs per additional per cent increase in average health centre performance score and per additional per cent increase in average immunization session performance score over the evaluation period were estimated to be US$ 3091 and US$ 12 760, respectively. The incremental cost-effectiveness ratios are relatively sensitive to personnel and travel costs. Integration of ISDS into the Andhra Pradesh immunization system is projected to result in a 39% potential cost savings per round of supervision visit.

Introduction

In November 2003, an immunization service delivery support (ISDS) model was launched in Andhra Pradesh, India, with the aim of strengthening immunization services through supportive supervision. Traditionally, supervision of immunization practices in Andhra Pradesh is nonexistent or conducted minimally in an unstructured manner. Recognizing attitude problems as well as lack of coordination and transportation as common barriers to effective supervision, the Andhra Pradesh state government decided to introduce a three-year collaborative initiative with Program for Appropriate Technology in Health (PATH) in all 23 districts covering a population of 76 million.¹

Supportive supervision is one of five essential components of the WHO’s Reaching Every District strategy, an operational approach to reach 90% DPT3 (three doses of vaccine against Diptheria, Pertussis and Tetanus) coverage nationally and at least 80% DPT3 coverage in every district by the year 2005, the “90/80 goal” adopted by the GAVI Alliance.² Supportive supervision involves on-site training by supervisors, which promotes quality at all levels of the health system by strengthening relationships within the system, focuses on the identification and resolution of problems, and helps to optimize the allocation of resources. The approach also promotes continuous improvements to the quality of care by providing leadership and support and by promoting high standards, teamwork and better two-way communication.³

Different approaches to supportive supervision to strengthen health worker performance and health services have been developed and documented. For instance, an integrated health package to guide essential health service delivery was developed by the United Republic of Tanzania Ministry of Health following health sector reform in 1999. This included a plan to bring a team of supervisors to district health facilities to evaluate how services are being delivered, to provide feedback and to conduct on-site training.⁴ Since implementation, supervisory visits occur more frequently and have become an opportunity for health workers to solve problems and learn additional skills. In 1999, EngenderHealth adapted a Client-Oriented, Provider-Efficient Services (COPE) model, a participatory supervision approach that includes self-assessment and performance plans, for use in child health services in selected study sites in Guinea and Kenya.⁵ Results show that supervision was better than before, supervisors were practicing new skills and following up on identified problems, and clients were very satisfied with the overall service received. The Honduran nongovernmental organization ASHONPLAFA introduced a new supervisory system and strengthened supervisory skills while reducing operating costs to meet sustainability goals. Strengthening supervision resulted in improved service quality and increased client satisfaction without increasing cost.^3,4 A study conducted in rural Mexico concluded that supportive supervision and self-assessment of physicians significantly improved doctor performance and doctor-patient communication.⁶

The Andhra Pradesh model

The Andhra Pradesh ISDS model is a well-established immunization supervision system and builds upon the existing health infrastructure. The objectives of implementing this model are to: (1) identify areas of high performance and those that need improvement, (2) assist staff in identifying and correcting wrong practices, (3) improve staff skills, (4) motivate staff, and (5) initiate corrective actions at appropriate levels through information sharing.

With the total combined number of urban health centres and primary health centres ranging from 70 to 80 per district with only one district immunization officer (DIO) in each district, a supervision visit can accommodate only five to ten health centres (HCs). To establish an effective supportive supervision system, the ISDS model initially used both external and government supervisors to monitor immunization practices and to ensure quality immunization service delivery. The model is a combined intervention of the external consultant mode and medical college mode (Fig. 1). Twenty-one districts were included in the programme during the first two years of implementation. The external consultant mode was coordinated by designated persons at PATH and used a team of external consultants and regional coordinating officers (RCOs), retired state or district level health officers, to implement the model in 14 districts. In each supervision round, a team consisting of six to seven members, including PATH staff, external consultants and RCOs, performed four- to five-day site visits to all urban health centres and primary health centres in each district and to regular fixed and outreach immunization sessions. In the other seven districts, supportive supervision was provided by faculty and postgraduate students of medical colleges in coordination with RCOs and DIOs. In both interventions, the supervision team was accompanied by the DIO and other district programme managers or DIO-appointed supervisors to ensure 
capacity strengthening and sustainability. PATH provided on-the-job training, appropriate tools, and financial and technical support to the teams. Over the course of the programme, management responsibilities were gradually handed over to trained RCOs. As assistance from external consultants and PATH staff phased out, seven RCOs were assigned responsibilities for conducting supportive supervision in 14 districts and coordinating with the medical colleges in nine districts and all DIOs (Fig. 1).

Fig. 1.

The supervision support framework

DIO, district immunization officer; RCO, regional coordinating officer.

Objectives

The overall objective of this paper is to assess the cost and effectiveness of the Andhra Pradesh ISDS model over a two-year period. Information on cost and effectiveness of the intervention is critical for decision-making and planning for sustainable integration of supportive supervision into the immunization system and replication in other states of India. Specific objectives include assessing cost and effectiveness of the model during the study period and estimating financial requirements for integration and expansion.

Methods

Two rounds of supervision visits per district were conducted within a 12-month period. The period between rounds provided an opportunity for divisional/district level monthly meetings to identify areas that need to be improved. During each round of supervision, information on activity costs and key indicators of immunization performance were collected. This paper bases its analysis on outcomes at baseline and after three rounds of supervision and costs incurred during three supportive supervision visits over an 18-month period. Baseline outcome data at all HCs in all districts were collected at the same time as the first introduction of supportive supervision activities.

Measurement of cost

All costs related to ISDS activities were based on the provider’s perspective, interviews with programme staff, and labour and financial reports from November 2003 (the start of the programme) to May 2005 (the end of the third round of supervision visits). All financial support from PATH, the associated costs at PATH’s Hyderabad office and of the Andhra Pradesh state government and field costs during the study period to implement the model were included. Daily activity time logs and activity reports kept during programme implementation allowed for tracking of intervention costs based on the actual staff time spent on supportive supervision activities.

Total costs were presented in US dollars at 2000 prices by using a gross domestic product deflator and exchange rate (45.55 rupees = US$ 1) and included wages and benefits of the project personnel involved with supportive supervision activities, printing of checklists, meals and incidental expense allowances, accommodation, transportation and administration. The estimates represent incremental costs of delivering the programme which are additional costs to the government that would not have been incurred otherwise. Salaries and benefits of health staff at district and HC levels and costs of recruiting external consultants were excluded. Fixed costs (such as housekeeping, capital costs and utilities) were not included because they were jointly incurred by other projects, and supervision activities were conducted at HCs. Costs associated with performance evaluations were minimal and not incremental to the programme, so therefore were not included in intervention costs.

Measurement of effectiveness

Performance data were collected using checklists. User-friendly checklists, which include 43 indicators on programme management, immunization sessions, immunization supplies, cold chain maintenance, immunization biowaste disposal and adverse events following immunization, were used in evaluating HC performance. Fixed and outreach immunization sessions were randomly observed and evaluated using a 28-point checklist to assess practices related to vaccine logistics, immunization practices and community mobilization activities (checklists are available upon request from the authors).

Following each session of performance evaluation, gaps were identified and discussed immediately. HC staff were observed and trained on-site, with feedback given to HC medical officers during the visit. Each day, supervisors consolidated forms and recorded all data from checklists into a computer database. At the end of each visit, feedback and HC performance measurements were provided to district officials, and district-wise aggregate data were then made available to state and district offices. Each point on the checklist was scored and weighted according to its importance (weights are available upon request from the authors), with a total possible score of 75 points. During the two-year period, supportive supervision activities occurred in 1724 HCs and 436 immunization sessions in 21 districts.

Cost-effectiveness analysis

Very few publications address the cost-effectiveness of peer training and supervision to enhance immunization services. One study evaluated an immunization training programme in Maluku, Indonesia, and concluded that it significantly improved the practice of vaccination and the immunization coverage rate.^7,8 The average costs were about US$ 0.05 per additional reported dose and US$ 0.50 per additional fully immunized child. Applying guiding principles including monitoring and supervision to the management of primary health care services in Papua New Guinea resulted in increased immunization coverage from a range of 57% to 67% between 1980 and 1982, to 89% to 94% between 1983 and 1984.⁹ A systematic review of literature on cost and effect of a range of interventions for expanding the coverage of immunization services suggests that peer training is one of the least costly interventions when vaccine costs are not included.¹⁰

In this paper, cost effectiveness is defined as the ratio of incremental cost to the incremental effectiveness as measured by the percentage improvement in performance scores at HCs and immunization sessions. The ratio is expressed as US dollars per percentage increase in performance scores. Between each round of supervision, additional HCs were identified and included in the programme, and some HCs were integrated with others. To allow for comparison of HC performance scores measured pre- and post-intervention, the analysis is limited to cost and outcomes associated with HCs and districts that had participated in the programme from the start until the fourth round of performance evaluation. One-way sensitivity analyses were performed to assess the robustness of cost-effectiveness ratios.

Results

Cost

Since supportive supervision activities were conducted in a phased manner, there were 1116 HCs in 16 districts (Adilabad, East Godavari, Mahabubnagar, Nizamabad and Visakhapatnam were excluded from the analysis) with complete data on performance scores available during the study period. Total associated costs of supportive supervision were approximately US$ 110 630. Table 1 presents the cost of supportive supervision by item.

Table 1. Costs for ISDS programme per supervision round, 16 districts, November 2003–May 2005.

Item	Round 1 (US$)a,b	Round 2 (US$)a,b	Round 3 (US$)a,b	Total (US$)a,c	Average per round (US$)a
Wages, benefits and per diem	7 962.07 (36.00%)	30 134.63 (60.39%)	18 095.53 (46.87%)	56 192.23 (50.79%)	18 730.34
Transportation, vehicle and accommodation	12 768.43 (57.73%)	18 823.61 (37.72%)	18 957.92 (49.10%)	50 549.95 (45.69%)	16 849.98
Expenses associated with training and retraining of programme staff	1 319.30 (5.96%)	879.60 (1.76%)	1 203.70 (3.12%)	3 402.60 (3.07%)	1 134.20
Printing	24.9 (0.11%)	23.7 (0.05%)	24.6 (0.06%)	73.20 (0.07%)	24.40
Administration	44.66 (0.20%)	36.89 (0.07%)	330.24 (0.86%)	411.80 (0.37%)	137.27
Total	22 119.35	49 898.43	38 611.99	110 629.77	36 876.59

ISDS, immunization service delivery support.
^a US$ values are year 2000 prices.
^b Values presented in parentheses are percentages of the total for each round.
^c Values presented in parentheses are percentages of the total.

Travel and personnel expenses were cost drivers of activities and added up to 96% of the total costs. Approximately 46% (US$ 50 550) of the total costs were spent on transportation, vehicles and accommodations. Personnel costs (US$ 56 192) including per diem payments accounted for about half of the total cost. The training and retraining sessions conducted by PATH for external consultants, RCOs and medical college staff amounted to an average cost of US$ 1134.20 per round. The design and printing of 20 000 copies of checklists cost US$ 0.07 per set of checklists. The total cost per round of supervision was on average US$ 36 877 and ranged from US$ 22 119 to US$ 49 898. The average programme cost was US$ 6914 per district (US $99 per HC) or US$ 2305 per district per round of supervision (US$ 33 per HC per round of supervision).

Additional external consultants and RCOs were recruited to the supervision team over the course of the programme. As a result, wages and benefits increased dramatically in the second round of supervision. In round three, four districts (Chittoor, Krishna, Kurnool and Guntur) were transferred to the medical colleges mode, resulting in lower costs.

Effectiveness

Of the 1116 participating HCs in 16 districts, the average district-level HC performance score (maximum 75 points) was 46.73 (mode 57; median 48.5) at baseline, 54.97 (mode 63; median 58) after the first supervision visit, 60.10 (mode 70; median 62) after the second supervision visit and 63.45 (mode 70; median 62) after the third supervision visit (Table 2). The Pairwise t-test shows that the intervention produced statistically significant improvements in HC performance scores during the study period at the 99% confidence level. The majority of districts show significant increases in performance scores over time.

Table 2. Average HC performance scores and percentage change between rounds of performance evaluation.

District	No. of HCs		Average HC performance scores					Percentage change between rounds
			Baseline (a)	Round 2 (b)	Round 3 (c)	Round 4 (d)		(a) to (b)	(b) to (c)	(c) to (d)	(a) to (d)
Anantapur	65		32.18	55.06	59.95	65.97		71.08*	8.88*	10.03*	104.97*
Chittoor	91		48.27	56.94	59.93	53.01		17.96*	5.25**	–11.55*	9.81*
Cuddapa	73		49.67	59.01	62.08	64.20		18.81*	5.20**	3.42***	29.26*
Guntur	77		48.61	60.01	58.90	64.51		23.46*	–5.19**	13.37*	32.70*
Karimnagar	63		41.93	50.16	60.44	62.58		19.60*	20.51*	3.54***	49.24*
Khamman	61		44.54	47.83	57.08	64.29		7.40	19.33*	12.64*	44.35*
Krishna	70		53.01	62.13	53.74	62.63		17.19*	–13.50*	16.53*	18.13*
Kurnool	74		44.20	57.34	54.20	54.93		29.71*	–5.47***	1.35	24.27*
Medak	71		42.98	60.60	65.17	66.53		40.99*	7.53*	2.09***	54.78*
Nalgonda	70		44.04	47.18	58.77	62.91		7.13***	24.55*	7.05**	42.85*
Nellore	74		56.08	55.70	65.54	66.28		–0.67	17.66*	1.13	18.19*
Prakasam	81		56.65	64.80	64.25	69.49		14.38*	–0.86	8.17*	22.66*
Srikakulam	63		52.22	55.92	65.22	68.09		7.08**	16.63*	4.40*	30.39*
Vizianagaram	55		53.36	57.00	61.11	66.11		6.81**	7.21**	8.18*	23.88*
Warrangal	45		17.58	16.02	56.47	55.64		–8.85	252.43*	–1.46	216.56*
West Godavari	83		48.99	55.84	59.58	68.00		13.99*	6.69*	14.13*	38.81*
All	1 116		46.73	54.97	60.10	63.45		17.64*	9.33*	5.57*	35.79*

HCs, health centres; * statistically significant at 0.01 level; ** statistically significant at 0.05 level; *** statistically significant at 0.1 level.

The average percentage change in performance scores between baseline and the fourth round of performance evaluation for all 16 districts was 36% (range: 9.81–216.56) and statistically significant at 0.01 level. Fig. 2 depicts average scores of each round of evaluation and percentage change in scores between rounds.

Fig. 2.

Average performance scores and percentage change in scores during four rounds of evaluation in 16 districts

The evaluation of immunization sessions was conducted at random during each round and included only a small number of sessions. The number of sessions visited in each district during each round was therefore inadequate to perform a statistical comparison. At the aggregate level, there was a statistically significant increase in the average performance score of immunization sessions from 58.85 points at baseline to 59.71 at the second round of evaluation (1.46%; not statistically significant) and to 63.96 points at the fourth round of evaluation (8.68%; P < 0.0001), from 59.71 at the second round to 62.57 at the third round (4.79%; P = 0.0005), and from 62.57 in the third round to 63.96 in the fourth round (2.22%; P = 0.08). Fig. 3 shows the overall percentage change in performance scores and average scores.

Fig. 3.

Average performance scores and percentage change in scores during four rounds of evaluation of immunization sessions

Incremental cost-effectiveness

It costs US$ 110 630 to significantly increase the mean performance score of HCs and immunization sessions by 36% and 9%, respectively – from 47 and 59 points at baseline to 63 and 64 points in the fourth evaluation. This results in a programme cost of US$ 3091 (an average of US$ 193 per district) to increase mean HC performance scores by 1%. Similarly, it costs US$ 12 760 (an average of US$ 797 per district) to increase mean immunization session performance scores by 1% during the same period of time. Table 3 calculates incremental cost-effectiveness ratios and reports sensitivity of cost-effectiveness ratios to changes in cost components and discount rates. For each cost item (wages and benefits, transport and accommodation, training) and percentage change in performance scores, the maximum and minimum values from the cost data were selected to proxy the magnitude of potential variations. Costs for wages, benefits, per diem, transportation, vehicles and accommodation had a large impact on cost-effectiveness. A reduction in compensation and travelling costs could significantly improve the incremental cost-effectiveness ratio, however it is relatively insensitive to costs associated with training.

Table 3. Incremental cost-effectiveness ratios (ICERs) of base case and one-way sensitivity analyses (baseline to round 4).

Item	ICER (US$)a,b
Per 1% change in HC mean score
Wages, benefits and per diem
⁃ high (90 403.90)	4 047
⁃ base (56 192.23)	3 091
⁃ low (23 886.20)	2 188
Transportation, vehicle and accommodation
⁃ high (56 873.35)	3 268
⁃ base (50 549.95)	3 091
⁃ low (38 305.28)	2 749
Training
⁃ high (3 957.90)	3 106
⁃ base (3 402.60)	3 091
⁃ low (2 638.80)	3 070
Discount rate
⁃ costs discounted at 5%	3 040
⁃ base at 0%	3 091
⁃ costs discounted at 3%	3 060
Per 1% change in immunization session mean score
Wages, benefits and per diem
⁃ high (90 403.90)	16 706
⁃ base (56 192.23)	12 760
⁃ low (23 886.20)	9 034
Transportation, vehicle and accommodation
⁃ high (56 873.35)	13 489
⁃ base (50 549.95)	12 760
⁃ low (38 305.28)	11 348
Training
⁃ high (3 957.90)	12 824
⁃ base (3 402.60)	12 760
⁃ low (2 638.80)	12 672
Discount rate
⁃ costs discounted at 5%	12 548
⁃ base at 0%	12 760
⁃ costs discounted at 3%	12 630

HC, health centre.
^a US$ values are year 2000 prices.
^b Cost-effectiveness ratios between each round are available from authors upon request.

Financing immunization service delivery support

Table 4 describes the implementation costs associated with service delivery support if the ISDS model were integrated into the immunization system and covered all 23 districts of Andhra Pradesh. The integrated ISDS model would comprise seven RCOs and nine medical institutions. During each supervision round, seven RCOs would form a team of supervisors to visit all HCs in 14 districts twice a year. In the other nine districts, ISDS activities would be performed by medical colleges. The RCOs, medical colleges and DIOs would coordinate and be jointly responsible for implementing ISDS activities upon approval of an action plan by the state office. Continuing and integrating the ISDS programme into the state immunization system in 23 districts would cost US$ 34 896 (US$ 31 370 in 2000 US$) per round of supervision. Travel expenses are the highest proportion of cost associated with ISDS. Overall, the integration represents a potential savings of 38.71%, which results primarily from replacing wages of PATH staff and consultants and PATH-expense allowances for meals and incidental expenses with government wages and allowance rates for meals and incidental expenses.

Table 4. Implementation costs associated with the integrated ISDS model per supervision round, 23 districts.

Item	Integrated ISDSa			Original ISDSb
	Cost per round (current price, US$)	Cost per round (2000 price, US$)		(2000 price, US$)
Wages, benefits and per diem	15 113.06 (43.3%)	13 586.00 (43.3%)		26 925.44
Transportation, vehicle and accommodation	19 743.69 (56.6%)	17 748.73 (56.6%)		24 221.85
Printing	39.01c (0.1%)	35.07 (0.1%)		35.07
Total	34 895.76	31 369.80		51 182.37

ISDS, immunization service delivery support.
^a Assuming no additional training is required and RCOs will be responsible for administration of the programme following the integration.
^b Costs incurred in 16 districts, excluding training and administration costs, were extrapolated to cover 23 districts.
^c Printing cost was estimated from an average printing cost in 16 districts (Table 1).

Discussion and conclusions

This paper offers the first attempt to quantitatively evaluate the supportive supervision framework. Total costs of activities in 16 districts over the evaluation period were US$ 110 630 (US$ 36 877 per round). The statistical significance of results on effectiveness suggests that performance at HCs and immunization sessions have been greatly improved since the ISDS programme was launched. Travel and personnel expenses, specifically wages, meals and incidental expense allowances for appointed PATH staff, appeared to be the major cost drivers. The involvement of PATH staff in training and other contributions in the early stage of the programme suggests that replication in other states or other settings is probable, but would require a plan for technical assistance. A phase-out of PATH support and successful integration of ISDS activities into the immunization system, however, would result in potential savings of 38.71%, which may improve cost-effectiveness provided that performance scores are maintained or improved.

The analysis between each round of supervision visits provides valuable insight into the impact of the Andhra Pradesh ISDS model. Fig. 2 and Fig. 3 illustrate that the model is significantly effective in improving performance at HCs and immunization sessions. The average performance score of HCs consistently improved, and good performance was maintained or improved, as measured at subsequent supervision visits. Statistically significant regressive performance found in Chittoor and Kurnool perhaps resulted from the change of supervisors during the transition to medical college. Although Fig. 3 demonstrates that it is more difficult to improve immunization session scores, the improvement recorded after the second supervision visit is significant. The relatively slow improvement might be due to the use of indicators that were more closely related to the practice of immunization and the behaviour of immunization officers, and the performance score was good at the time the model was launched (average scores at baseline were at 58.85). Our findings underscore that continuation of supportive supervision is critical in strengthening and enhancing performance at HCs and immunization sessions.

The limitations of this study are those attributed to the limited data and scope of the programme and our analysis. The relatively small number of immunization sessions observed may not be representative. Immunization sessions are usually conducted two days per week with fixed timing. Each supervision visit could incorporate observation of a few sessions. Although checklists were introduced in the programme to minimize subjectivity of supervisors in rating performance, this issue was not assessed. Additionally, the analysis does not consider ongoing training, which may be required to improve knowledge and strengthen supervisors’ capacity.

As there are no comparators, the implication of cost-effectiveness results is limited. Our findings thus provide a basis for a comparison to be made across settings and programmes aiming to improve performance of HCs and immunization sessions. Such comparisons will generate information on how cost-effective the intervention is and how it can be further improved. Similar approaches should be further explored and applied to other health services, such as primary care and maternal and child health, where supportive supervision can play a role in improving knowledge and transferring new skills to multiple and remote sites. Appropriate models should be identified and assessed to be applicable to the local context. The analysis on cost and effectiveness can be a useful tool in planning for sustainable intervention. ■