Did PEPFAR investments result in health system strengthening? A retrospective longitudinal study measuring non-HIV health service utilization at the district level

Abstract

Objectives PEPFAR’s initial rapid scale-up approach was largely a vertical effort focused fairly exclusively on AIDS. The purpose of our research was to identify spill-over health system effects, if any, of investments intended to stem the HIV epidemic over a 6-year period with evidence from Uganda. The test of whether there were health system expansions (aside from direct HIV programming) was evidence of increases in utilization of non-HIV services—such as outpatient visits, in-facility births or immunizations—that could be associated with varying levels of PEPFAR investments at the district level.

Methods Uganda’s Health Management Information System article-based records were available from mid-2005 onwards. We visited all 112 District Health offices to collect routine monthly reports (which contain data aggregated from monthly facility reports) and annual reports (which contain data aggregated from annual facility reports). Counts of individuals on anti-retroviral therapy (ART) at year-end served as our primary predictor variable. We grouped district-months into tertiles of high, medium or low PEPFAR investment based on their total reported number of patients on ART at the end of the year. We generated incidence-rate ratios, interpreted as the relative rate of the outcome measure in relation to the lowest investment PEPFAR tertile, holding constant control variables in the model.

Results We found PEPFAR investment overall was associated with small declines in service volumes in several key areas of non-HIV care (outpatient care for young children, TB tests and in-facility deliveries), after adjusting for sanitation, elementary education and HIV prevalence. For example, districts with medium and high ART investment had 11% fewer outpatient visits for children aged 4 and younger compared with low investment districts, incidence rate ratio (IRR) of 0.89 for high investment compared with low (95% CI, 0.85–0.94) and IRR of 0.93 for medium compared with low (0.90–0.96). Similarly, 22% fewer TB sputum tests were performed in high investment districts compared with low investment, [IRR 0.78 (0.72–0.85)] and 13% fewer in medium compared with low, [IRR 0.88 (0.83–0.94)]. Districts with medium and high ART investment had 5% fewer in-facility deliveries compared with low investment districts [IRR 0.95 for high compared with low, (91–1.00) and 0.96 for medium compared with low (0.93–0.99)]. Although not statistically significant, the rate of maternal deaths in high investment district-months was 13% lower than observed in low investment districts.

Conclusions This study sought to understand whether PEPFAR, as a vertical programme, may have had a spill-over effect on the health system generally, as measured by utilization. Our conclusion is that it did not, at least not in Uganda.

Key Messages

• Ugandan districts that enjoyed more United States’ PEPFAR investment saw no meaningful changes in non-HIV service rates, including for outpatient care for young children, TB tests and in-facility deliveries, compared with districts with lower PEPFAR investment.

• PEPFAR did not, as a result of ‘spill-over’ benefits, strengthen the health system in Uganda.

• The emergency nature of PEPFAR confined investments to vertical, HIV-related activities. To sustain gains on HIV/AIDS while responding to emerging infections, strengthening weak government health systems in the developing world is the next ‘emergency’.

Introduction

One of the effects of the 2003 President’s Emergency Plan for AIDS Relief (PEPFAR) was to expose the inability of Africa’s weak health systems to cope with the devastating HIV epidemic, or even to absorb the unprecedented levels of assistance from donors required to achieve results (Samb et al. 2009). In each of the 12 African PEPFAR ‘focus countries’, then, PEPFAR administrators had a choice to make about how to rapidly deliver AIDS care to populations hit by the epidemic. Would they take the time to rebuild health systems and move AIDS treatment through existing publicly run health centres (HCs) and hospitals, risking delay in treating patients? If so, would delay risk overwhelming health systems with a Tsunami of AIDS patients? Alternatively, would they work around the existing, largely government-run, health system, to accelerate access to immediate care while avoiding the task of building sustainable health system capacity? (Yu et al. 2008)

PEPFAR’s initial rapid scale-up approach was largely a vertical effort focused fairly exclusively on AIDS. The programme’s ‘implementing partners’ were mostly American non-governmental organizations (NGOs), with very little direct funding going to government health system operations. The ‘President’ in the PEPFAR title was George W. Bush, the founder of the ambitious programme to slow the AIDS epidemic. The Bush administration had a stated preference for aligning with private, especially faith-based, partners, consistent with the preferences of other donor-directed global health initiatives (McCoy et al. 2005; Oomman 2007; Biesma et al. 2009; Bradley-Springer 2010; Jappah 2013).

Over time, most international donors, including PEPFAR, shifted strategies as they realized strong health systems would be required to sustain a long-term HIV response (Donoghue et al. 2005; Sepulveda 2007; Spicer et al. 2010; Cohn et al. 2011), and began to move in that direction (WHO 2007; Samb et al. 2009). The World Health Organization developed its health systems ‘building blocks’, but those are but one way to portray the components of a health system (van Olmen et al. 2012). Regardless of the model, the important elements of a health system include the capacity to make decisions (leadership, governance, financing), the people to do the work (health personnel) and the necessary materials, supplies and facilities (clinics, hospital wards, drugs, labs, operating theatres and so on). Alongside each of these, a process for generating health system data should infuse timely information. Figure 1[] portrays our conceptualization of important elements of a health system.

Figure 1.

Conceptual Framework for PEPFAR operations in focus countries, including Uganda.

PEPFAR has been acknowledged for achieving its large, but narrow (HIV-specific) mission of reducing mortality and morbidity from AIDS (Bass 2005; Bendavid and Bhattacharya 2009; Institute of Medicine 2013). The question remains, however, whether health systems emerged from the first PEPFAR decade as stronger, weaker or unchanged. Did PEPFAR, as some have claimed, serve to crowd out non-HIV care (Shiffman 2008), distract (Biesma et al. 2009), lure away health workers (Samb et al. 2009; Oomman 2008; Bajunirwe et al. 2013), waste effort on parallel systems for labs and medical records (Marchal et al. 2009) and largely minimize the importance of caring for health problems that are arguably more significant to the nation’s health (Biesma et al. 2009; Grepin 2012)? Or did it infuse much-needed energy, resources, optimism and momentum that had ‘spill-over’ effects to create a stronger health system generally (De Cock et al. 2011), especially in specific areas such as maternal health (Grepin 2012; Kruk et al. 2012), human resources (Riley et al. 2007; Institute of Medicine 2013) or procurement, training, health information and laboratories (Oomman 2007)? As with other global health initiatives, PEPFAR, in its early years, was reported by some to be non-transparent, non-cooperative and uninterested in other health problems (Donoghue et al. 2005; McKinsey and Company 2005; Shiffman 2006). The clear and explicit shift of PEPFAR reauthorization in 2008 to include health systems strengthening activities acknowledged the broad concern that spill-over effects from a vertical approach might not be sufficient.

The volume of PEPFAR money in Uganda comprised 73% of the budget for AIDS care there in by 2006, obscuring the distinction between PEPFAR and other global health initiatives (Oomman 2007). The entire Ministry of Health budget was smaller than the donor budget in the early years of the PEPFAR program (see Table 1), perhaps minimizing the role of the Ministry in managing the national health program (Samb et al. 2009; Oomman 2007). Myriad independent PEPFAR implementing partners set about establishing separate structures, mechanisms and processes to launch an emergency attempt to stem the epidemic, and they reported their data to an independent private US contractor, Social and Scientific Systems, Inc., which maintained its Monitoring and Evaluation of the Emergency Plan Progress (MEEPP) data separately from the government’s health management information system (HMIS) (Makumbi et al. 2010; Porter et al. 2012). MEEPP data were used to routinely monitor PEPFAR services in Uganda during our study observation period, and the data quality has been reported to be high (Kalibala 2010).

Table 1.

Selected Uganda Health Services Measures 2005–2011

General topic	Specific indicator	Estimates 2005 (or earlier)	2006	2007	2008	2009	2010	2011
Facility infrastructure counts (% govt)a	HC IIs and IIIs	3177 (56%)	2963 (71%)				4141 (59%)	4648 (53%)
	HC IVs	165 (92%)	161 (92%)				178 (93%)	190 (87%)
	Hospitals	101 (54%)	113 (52%)				131 (50%)	143 (46%)
National health indicators from UDHS reports	MMR1,b	505	435					438
	IMR2,b	88	76					54
	Fertility rate3,b	6.9	6.7					6.2
Government Allocation to Health Sector4	Government funding ($x 106) c,4	129.236	132.26	164.66	226.46	217.96	258.96	252.36
	Donor Projects and GHIs ($x 106)c,4	268.385	139.235	141.125	253.005	301.805	90.445	206.105
		150.786	75.876	83.756	152.596	148.676	41.526	81.696
	PEPFAR donor contributions ($x 106)g,6	146.9	170.0	236.6	283.6	285.9	286.3	—
	Health Expenditure as % of Total Expenditurec,5	8.9	9.3	9.0	8.3	9.6	8.9	8.3
Population in millions (source: Annual Health Sector Performance Report reports)		26.7	27.6	28.6	29.6	30.7	31.8	32.9

Denominators and other relevant numerical relationships.

¹Maternal mortality rate (MMR): Expressed in terms of maternal deaths per 100 000 live births in the 7-year period preceding the survey.

²IMR: Expressed in terms of deaths per 1000 live births. Mortality rates refer to the 5-year period preceding the survey.

³Fertility rate reflects the average births per woman for the three years preceding the survey.

⁴These are fiscal years (e.g. 2005/2006, 2006/2007).

⁵Uganda shillings as reported in: Uganda’s Annual Health Sector Performance Report 2010/2011, p. 26 Table 15.

⁶US Dollars, in millions, with exchanges as follows when original source was in Shillings.

2005^h: UgSh 1780/$1.

2006^h: 1835/$1.

2007^h: 1685/$1.

2008ⁱ: 1658/$1.

2009ⁱ: 2030/$1.

2010ⁱ: 2178/$1.

2011ⁱ. 2523/$1.

GHIs, Global health initiatives.

Sources:

^aUganda Bureau of Statistics (UBOS) Statistical Abstract Table 2.5.1, p. 29: Health units in Uganda are classified into hospitals, HSs IV, III and II. These counts include government, private not for profit and private for profit. % public is in parentheses.

^bMMR, IMR and fertility rates all come from the UDHS. 2005 estimates come from the UDHS conducted in 2000. For 2006 and 2011, we report the results from UDHS surveys conducted in 2006 and 2011.

^cUganda’s Annual Health Sector Performance Report December 2011, Table 15. AHSPR contains no explanation of what is included in ‘donor’ funding, but PEPFAR numbers do not seem to be included. A search of the AHSPR document turns up no mention of PEPFAR.

^dEstimate from 2000, reported in 2006 UBOS Statistical Abstract.

^e2007 UBOS Statistical Abstract.

^f2012 UBOS Statistical Abstract.

^gUS Government Accountability Office, Report to Congressional Committees, President’s Emergency Plan for AIDS Relief, September 2010 GAO-10-836.

^hUganda exchange rate history (2005–2007) source: http://www.mongabay.com/history/uganda/uganda-currency_and_inflation.html.

ⁱUganda exchange rate history (2008–2010) source: https://www.cia.gov/library/publications/the-world-factbook/geos/ug.html.

Although there is much in the peer-reviewed literature about the science of AIDS care itself, there is less about the science of implementing the programmes. Some PEPFAR efforts to strengthen health systems were reported in the literature (Potter et al. 2008; Yu et al. 2008; Assefa et al. 2009; Price et al. 2009; Rasschaert et al. 2011; Dutta et al. 2012; Palen et al. 2012; Institute of Medicine 2013), while others probably went unrecorded, as the originating legislation did not support ‘research’. Duber et al. (2010) found very little association between PEPFAR investments and improvements in the performance of health systems as measured by health outcomes.

Many previous studies have been content to analyse international data sets, comparing ‘focus countries’ with non-focus countries using annual country-level data (Duber et al. 2010; Bendavid et al. 2012; Cohen et al. 2012). Other studies have been small, trying to get closer to the data, but suffered from small sample sizes (Price et al. 2009; Filler et al. 2011; Matsubayashi et al. 2011; Moon et al. 2011). Many studies of the effects of PEPFAR have been funded by PEPFAR itself (Biesma et al. 2009), as is ours. Overall, results of studies to date have been mixed.

We sought to learn about PEPFAR’s effects on health systems with evidence from Uganda. We chose Uganda because it was an early leader in showing gains in halting the epidemic (Green et al. 2006), because it was a PEPFAR focus country with significant investments (and where PEPFAR investment dominated other global health initiatives), and because it had a relatively strong HMIS (Gladwin et al. 2003; Kintu et al. 2005; Mandelli and Giusti 2005). We acknowledged the WHO health system building blocks (van Olmen et al. 2012), although Figure 1 portrays our unique conceptualization of important elements of a health system.

The purpose of our research was to identify spill-over health system effects, if any, of investments intended to stem the HIV epidemic over a 6-year period. The test of whether there were health system expansions (aside from direct HIV programming) was whether any changes in utilization of non-HIV services—such as outpatient visits, in-facility births or immunizations—could be associated with varying levels of PEPFAR investments at the district level.

Methods

The research was organized through a Cooperative Agreement from the US Centers for Disease Control and Prevention (CDC), with a PEPFAR Public Health Evaluation award to the University of Washington near the end of 2010. The University of Washington sub-contracted with Makerere University in Kampala, Uganda, to provide in-country partnership for leadership, scientific guidance and management. Additional partners included the Ministry of Health’s Resource Center and the Uganda office of CDC.

The setting for this study was the nation of Uganda, a largely Christian nation with dozens of ethnic groups and English as the official language. The population stands at 35 million people, half of whom are 15 and younger, with the third fastest growth rate in the world. Almost two in three people live below the poverty line of $2 per day (Index Mundi. http://wwwindexmundicom/facts/uganda/poverty-headcount-ratio, accessed 11 September 2013). Yoweri Museveni has ruled as President since 1986 (Central Intelligence Agency 2013).

A 6-year time period (mid-2005 to mid-2011) was used for this nationally representative, retrospective longitudinal study, representing the period of PEPFAR scale-up in Uganda. The political decision taken in 2005 to begin dividing districts into smaller sized units (‘district splitting’) created complications. By the end of our study period, the number of districts doubled (Green 2008).

Data collection

Uganda’s HMIS article-based records were available from mid-2005 onwards. We visited all 112 District Health offices to collect routine monthly reports (which contain data aggregated from monthly health care facility reports) and annual reports (which contain data aggregated from annual facility reports). Districts routinely forwarded their reports to the Ministry headquarters throughout the 2005–2011 period. Reports included the number of facilities reporting, but did not specify which facilities neglected to report their data. In other articles from this study, we report results from data collected from >300 health facilities (Makumbi et al. 2015), we provide more details on methods (Stover et al. 2015) and we report results on the views of District Health Officers (DHOs) (Lohman et al. 2015).

PEPFAR investment measures

PEPFAR (through USAID) maintained the previously mentioned MEEPP data base in Uganda, beginning in December 2004. We obtained a data file with the number of individuals enrolled for care through PEPFAR-funded organizations at the District level. MEEPP provided counts of individuals on anti-retroviral therapy (ART) at year-end, which served as our primary predictor variable. We grouped district-months into tertiles of high, medium or low PEPFAR investment based on their total reported number of patients on ART at the end of the year. The Web Appendix to this article describes ART data. The tertiles ranged from 0 to 191 ART patients per district month in the lowest third to between 1437 and 49 594 patients per district month in the highest tertile. The appendix also portrays ART patient enrolment by district by year, and the number of district years in each ART tertile.

Ethical and data ownership considerations

We obtained institutional review board approvals from the Uganda National Council for Science and Technology, the Makerere School of Medicine, the Makerere School of Public Health, the University of Washington and the CDC’s Center for Global Health Office of Science. All partners signed a ‘Data User Agreement’ clarifying that the Ministry of Health owns the data and grants permission to use them. The agreement requires signatories (Makerere University, University of Washington, Ministry of Health Resource Center and CDC) to agree on the final analyses, and specifically provides the Ministry an opportunity to provide comment before articles are submitted for publication.

Data analysis and model configuration

In initial descriptive analysis, we examined volumes of non-HIV care independent of PEPFAR investment, to see how these changed over time. We also plotted raw ART enrolment against services overall, and by district, see Web Appendix. To estimate the rate of change, we used a simple linear regression slope of change in outcome rate per year.

Next, we ran multilevel mixed-effects negative binomial regression models separately for each non-HIV service outcome, all of which were counts, to assess the effects on volumes of services in relation to the level of PEPFAR investment. We adjusted for repeated measurements by using random effects at the district level. This model takes advantage of the count properties of the outcome data (non-negative, integer) while allowing for over-dispersion and making a less restrictive missing-at-random assumption for analysis of the available data.

Several alternative indicator measures of PEPFAR investment were available in the MEEPP data set (see Table 2). These include number of labs built and supported, number of patients enrolled for ART, number enrolled for mother-to-child transmission prevention, number of HIV patients enrolled for TB care and number in palliative care or counselling and testing. We also considered calculating a ‘score’ using combinations of PEPFAR provided services and estimated the cost of services provided by PEPFAR. We ultimately selected the number of people on PEPFAR-supported ART in each district annually as the most representative indicator of PEPFAR investment over time. Other stand-alone measures were rejected because they were less representative and a multiple-measures ‘score’ was rejected because of collinearity and uncertainty in combining measures on incompatible scales (e.g. number of labs and number of patients enrolled for TB). We might have used cost data had it been available, but we also had concerns that costs of interventions changed substantially over time. We created tertiles of ART enrolment data to facilitate analysis and interpretation. ART tertiles were calculated for each year of the study in each district; in other words, tertiles measurements were stable within years but changed between years.

Table 2.

Inputs: PEPFAR support counts by indicator and year.

Component	2005	2006	2007	2008	2009	2010
ART patients	49 638	51 397	83 549	130 837	175 367	207 872
Pregnant women on ART		12 577	25 539	34 660	45 890	33 135
Counselling and Testing	623 178	914 720	1 025 956	1 579 551	1 884 582	2 145 440
Palliative care	248 351	320 108	339 664	357 467	398 622	493 322
TB/HIV patients	14 310	14 583	11 626	12 770	18 632	14 358
Labs supported	133	114	168	541	270	315

Source of data and notes: Uganda MEEPP data as provided by Social and Scientific Systems, Inc., for each of Uganda’s 112 districts, collapsed to the original 56-district level, for the 6-year period 2005–2010. These are selected PEPFAR investments, not totals. Pregnant women on ART services were not reported to MEEPP in 2005. Where >1 implementing partner was operating in a facility, the number of types services provided are counted only once. For example, if two partners support the same lab, the lab is counted only once.

Control variables included year and month of outcome data from HMIS forms, percent of households with a pit latrine (divided into tertiles), primary school enrolment ratio (divided into tertiles) and HIV prevalence (divided into tertiles); these data came from the Uganda Bureau of Statistics and Ministry of Health. All variables were available at the district level except HIV prevalence, which was available only at the 10-region level from the Uganda Demographic and Health Survey (UDHS). Additional control variables include year and month of source data, to account for seasonal variation and a variety of annual factors. See Web Appendix for more details on all control variable cut points.

The number of people in the district for each month was used as an exposure term in most models, with the total population used in each district-month reflecting the total number of sub-districts reporting the particular service outcome that month. The exposure term adjusts the model for the population-at-risk approach for analysis, to provide the appropriate weight for the district-month unit of analysis. Different numbers of people are at risk for each district-month due to population growth, district splitting during the study observation period and missing source data forms or missing data of interest on the collected forms. Some analyses (e.g. maternal death and DPT3 immunizations) substitute the number of deliveries (rather than population) as the exposure term. The unit of analysis is a ‘district-month’.

The conditional expectation of the outcome given the covariates and the overdispersion is:

$$\mathrm{e}\mathrm{x}\mathrm{p}\{x_{\mathrm{i}\mathrm{j}}\beta +\mathrm{l}\mathrm{n}{\delta }_{\mathrm{j} }+ P_{\mathrm{i}\mathrm{j}}\}$$

where $x_{\mathrm{i}\mathrm{j}}\beta$ is the linear combination of all covariates. $\mathrm{l}\mathrm{n}{\delta }_{j\mathrm{ }}$is the ovedispertion term and is equal to the district intercept. $P_{\mathrm{i}\mathrm{j}}\mathrm{ }$is the exposure term and is the district population. All covariates are included as factor variables with a separate term for each level (other than the reference level) included in the model. See the Web Appendix for detailed notation.

We generated incidence rate ratios (IRRs), which can be interpreted as the relative rate of the outcome measure among medium or high investment tertiles in relation to the lowest investment PEPFAR tertile, holding other variables in the model constant. That is, we estimated the number of times more likely the outcome was to occur for the middle or top third number of people on PEPFAR-supported ART in each district district-month, compared with the bottom third of district-months, when all other variables were held constant.

Stata (version 12) software was used for analysis.

Results

We report findings over the 6-year study period in Uganda related to (a) health systems components and their characteristics (Table 1), (b) counts of PEPFAR services (Table 2 and Web Appendix), (c) volumes of services provided without considering the effects of PEPFAR (Tables 3 and 4 and Web Appendix), (d) results of bivariate comparisons of PEPFAR investment in relation to service volume changes (Figures 2a, b and 3) and (e) results of various regression models with control variables that predict service volume changes in relation to PEPFAR investment (Table 5).

Table 3.

Outputs: non-HIV care service trends in Uganda, 2005–2011

Non-HIV service delivery outcome	2005 (6 months of observations)	2006	2007	2008	2009	2010	2011 (6 months of observations)	Annual change ratea
Number of monthly reports (% possible)	291 (69%)	709 (84%)	826 (86%)	887 (92%)	885 (92%)	1074 (91%)	620 (92%)
OPD 4 visitsb (outpatient department visit for children aged 4 and younger)	3 854 469	8 082 409	8 709 920	9 038 353	9 363 831	9 370 105	4 163 533
OPD 4 rate (per 1000 population)	344.92	337.57	335.97	321.06	322.56	313.68	272.10	−9.99
Total OPD five visits (outpatient department visit for persons aged 5 and older)	7 622 398	16 341 349	18 847 627	20 416 409	20 685 628	22 494 418	10 941 968
OPD 5 rate (per 1000 population)	684.00	682.88	727.59	723.27	714.31	753.69	718.36	8.26
Deliveries in health facilities	154 900	333 531	406 564	476 517	491 146	533 154	286 091
Deliveries (per 1000 population)	13.84	13.97	15.96	17.19	17.19	18.17	19.07	0.90
Maternal deaths	545	1019	1018	1269	1212	1237	575
Maternal deaths (per 100 000 deliveries)	355	310	249	264	256	228	202	−22.00
DPT 3 under 1 year	399 956	853 508	980 650	1 062 148	1 042 701	1 134 591	577 344
DPT 3 under 1 year (per deliveries)	2.58	2.58	2.38	2.19	2.09	2.10	1.98	−0.11
Malaria smears	887 644	1 870 112	2 239 549	2 476 814	3 321 862	4 015 267	1 859 881
Malaria smears (/1000 population)	82	78	87	88	115	136	123	9.6
TB sputum tests	54 404	117 237	142 140	181 800	185 993	210 874	99 407
TB sputum tests (/1000 population)	4.82	4.86	5.52	6.48	6.46	7.12	6.61	0.39

Source of data and notes: Based on data from monthly reports.

Uganda HMIS data from Districts from the monthly UG HMIS123 form as collected by the research teams from each of Uganda’s 112 districts, for the 6-year period 2005/2006–2010/2011. As a verification, note our combined estimate of opd4 + opd5 = 1.07 visits per capita for 2010, the last full year of our data, is similar to the AHSPR 2011 report for the same period, which reports 1.0 visits per capita.

^aLinear regression slope of change in outcome rate per year.

^bHIV visits are a component of outpatient visit totals.

Table 4.

Outputs: non-HIV care service trends in Uganda, 2007–2011

Fiscal year of report (July–June)	2007/2008	2008/2009	2009/2010	2010/2011	Annual change ratea
Number of districts reporting vitamin Ab (out of 56)	32 (57%)	42 (75%)	47 (84%)	52 (93%)
Number of vitamin A administered to children 4 and younger (million)	0.79	0.94	1.13	1.42
Rate of vitamin A administered to children 4 and younger (per 1000 population)	55.63	47.95	50.41	61.55	2.02
Number of districts reporting tetanus	44 (79%)	43 (77%)	51 (91%)	53 (95%)
Number of First tetanus dose (million)	0.44	0.43	0.60	0.71
Rate of First tetanus dose (per1000 population)	19.86	19.75	20.84	23.38	1.17
Number of districts reporting admissions	37 (66%)	38 (68%)	49 (88%)	53 (95%)
Number of hospital admissions (million)	0.76	1.00	1.30	1.29
Rate of Hospital admissions (per 1000 population)	52	52	53	50	−0.48
Number of districts reporting operations	37 (66%)	38 (68%)	47 (84%)	53 (95%)
Number of major operations (million)	0.04	0.05	0.05	0.08
Rate of major operations	2.63	2.59	2.09	2.58	−0.07

Source of data and notes:based on data from annual reports.

Uganda HMIS data from Districts from the annual UG HMIS 128 form, as collected by the research teams from each of Uganda’s 112 districts, for the 6-year period 2005/2006–2010/2011.

Some of the increase in numbers across years is due to the improved availability of forms and more complete reporting of data in more recent years. To adjust for this improved reporting over time, comparison of services across years are adjusted for the population providing data by district and year.

The HMIS annual district forms available for 2005/2006 and 2006/2007 were not consistently available with 23 and 24 forms available, respectively, of a possible 56 original districts. Due to the low proportion of data available from these first 2 years of the study for this HMIS 128 form they were not included in this table, regression modelling or graphs.

^aLinear regression slope of change in outcome rate per year.

^bReports obtained as a percent of the total possible.

Figure 2.

(a) Monthly delivery rates (averaged for each year) in health facilities by the annual number of patients on ART for each of Uganda’s 56 (original) districts, for the years 2005–2010. Source of data and notes: Each point represents a delivery per 1000 population in the district for one year derived from the monthly average (n = 3425 district months). For example, a circle with a y-axis value of two indicates two deliveries per 1000 population in that district in a month. So, if the district had a 600 000 population, there would have been 1200 deliveries in that month (we solve for X, with 2/1000 = X/600 000). b Annual average of monthly delivery rates in Uganda’s health facilities by the number of patients on ART in Uganda, year by year, 2005–2010. Source of data and notes: Each point represents the annual average number of monthly deliveries per 1000 population in the district for one month. (b) Annual average of monthly delivery rates in Uganda’s health facilities by the number of patients on anti-retroviral therapy (ART) in Uganda, year by year, 2005–2010.

Table 5.

IRRs and 95% CIs of the medium and high tertiles of patients on ART relative to the lowest ART tertile on district non-HIV care outputs, from district monthly routine HMIS data reports (2005/2006–2010/2011, 6 years)

Non-HIV care output indicator	Medium investment in relation to low investment IRR (95% CI, P-value)	High investment in relation to low investment IRR (95% CI, P-value)	Number of monthly reports with data	Denominator variable for rates (model exposure)
Outpatient visits for children aged 4 and younger	0.93 (0.90–0.96, <0.001)	0.89 (0.85–0.94, <0.001)	3419	Population
In-facility deliveries	0.96 (0.93–0.99, 0.020)	0.95 (0.91–1.00, 0.033)	3425	Population
DPT3 for children younger than 1 year of age	1.00 (0.96–1.03, 0.778)	0.94 (0.90–0.99, 0.017)	3419	Deliveries
TB tests	0.88 (0.83–0.94, <0.001)	0.78 (0.72–0.85, <0.001)	3369	Population
Malaria blood smears conducted	0.99 (0.94–1.03, 0.519)	1.01 (0.94–1.07, 0.835)	3430	Population
Maternal deaths	0.93 (0.81–1.06, 0.292)	0.87 (0.73–1.04, 0.134)	3357	Deliveries

Source of data and notes: Uganda HMIS monthly data from Districts (based on the UgHMIS123 form), as collected by the research teams from each of Uganda’s 112 districts. Control variables in the models include sanitation at the district level (% of population with pit latrines), % of eligible children enrolled in elementary schools at the district level and HIV prevalence at the 10-region level. Additional control variables include year and month of source data, to control for seasonal variation and a variety of annual factors. The unit of analysis is ‘District Month’. IRRs can be interpreted as the relative rate of the outcome measure in relation to the lowest investment PEPFAR tertile when all other variables are held constant (i.e. considering the number of people on PEPFAR-supported ART in each district, how many more times likely is the outcome to occur in the middle or top third district-months of ART investment compared with the bottom third of district-months.) At the 112 district level, 92% (5295 of a possible 5736) of the forms were collected. When collapsed to the 56-district level, there were 3756 district monthly reports for analysis (some missing sub-district forms). Over the 72 months of the study, an average 52.2 reports were available out of a possible 56, with a range of 45–56.

Secular trends in health services and health status in Uganda, 2005–2010

To understand the context in which PEPFAR operated, we examined elements of the health system and some important indicator variables. Table 1 portrays the increase in the number of HCs over the period, especially HC IIs and IIIs. HCs are numbered according to size and capacity, with HC I’s serving as small health posts in rural areas, intended to serve about 1000 people. HC IIs serve about 5000 people in each parish, and HC IIIs are intended to serve 20 000 people at the sub-county level. HC IV’s have overnight care capacity, sometimes described as mini-hospitals(Ministry of Health 2012). The increase in HCs at the II and III level was largely in the for-profit sector (where facilities increased from 830 to 3510 over the period, a 300% increase)(Uganda Bureau of Statistics 2012). After adjusting for population size for each district, most service volumes grew only slowly or even declined. Services that increased the most were outpatient visits for those 5 years and older (8.3%), and malaria smears (9.6%); however, outpatient visits for people aged 4 and younger declined by 10.0% (Table 3).

The maternal mortality ratio was measured as 505 per 100 000 live births in the 2000 Demographic and Health Survey, as 435 in 2006, and 438 in 2011, for a total 13% decline, all gains achieved prior to significant PEPFAR intervention. The infant mortality rate (IMR) declined at a faster rate (39%), with 88 per 1000 births in 2000, 76 in 2006 and 54 in 2011, consistent with international trends in low-income countries. The fertility rate declined by 10% across the period: 6.9 births per woman in 2000, then to 6.7 in 2006 and to 6.2 in 2011 (Table 1).

Data collected and cleaned

Both districts and facilities were often missing their copies of data reports, especially for the early years of the HMIS reporting period. We collected 5295 of a possible 5736 HMIS monthly district reports, including in split districts, or 92% of the total possible district months in the study period (2005–2010). The number of HMIS forms collected was lowest in 2005, with 69% of the possible number of forms, while 92% were collected for 2008, 2009 and 2011 (Table 3). Annual data, however, were available for only 66% of the total possible district-years. Because annual data were more complete for the period 2007 through 2010 (81% available), we restricted our analysis of annual data to that period. A sample of the entered data was directly compared with the source form scans to provide an estimate of the data entry error rate and to verify extreme values. Based on samples of double-entered data, a data entry error rate was estimated to be 2.6% for annual reports, and 3% for the monthly reports.

Following double entry, we generated plots of individual variables by time and district to identify outlier values; these were verified against the scanned source forms. In this way, we corrected 2% of data points. Another 1% of values were set to missing because they were not believable, even though they reflected the numbers entered by district personnel on the source forms. We combined ‘daughter’ districts that had been split from their 56 original parent districts to create consistent units of analysis over time. The number of people in the district for each month was used to adjust the number of services reported for each HMIS monthly report. The population adjusted approach provides the number of services per 1000 population to compensate for the varying numbers of people at risk for each district-month as discussed in the methods section (Table 3).

PEPFAR services

Although only 16 private PEPFAR implementing partner organizations were funded in Uganda, the number of partnership relationships created by those 16 partners increased between 2005 and 2010, with 87 at the start of 2005 and 445 by the end of 2010. The number of locations where PEPFAR ART services was provided also increased from 85 facilities in 2005 to 374 by the end of 2010, a linear increase of 70 locations per year (data not tabled). MEEPP data indicated total national PEPFAR expenditures in Uganda grew from USD $147 to $286 million between 2005 and 2011, in a steady increase. Uganda’s own government funding for health services overall increased from $130 to $252 million over the period (with a levelling off between the years 2010 and 2011) (Table 1).

The MEEPP database in Uganda provides counts of several service types provided by PEPFAR partners by district for each year. The number of individuals on ART supported by PEPFAR in Uganda grew from ∼50 000 individuals in the first year to 208 000 by 2010, with the subset of those who were pregnant women growing from 12 500 in 2006 to 33 000 in 2010. The proportion of pregnant women among total ART patients, while it initially grew from 24% in 2006 to 31% in 2007, subsequently declined over the remaining period, from 26% in 2008 to 16% in 2010. Unfortunately, data on services for pregnant women were not collected by MEEPP for 2005. The number of individuals enrolled for HIV care (without being put on ART), also called ‘palliative care’, doubled across the 6-year period, growing from a quarter million people to a half million people (Table 2).

Trends in non-HIV service utilization over time, 2005-2010

The routine monthly HMIS data we collected allowed us to measure the number of non-HIV services delivered over the 6-year study period. The secular trend for these data, without relation to PEPFAR investment, shows increased counts during this period. However, the population-adjusted annual rate for some services declined. We saw increases in rates of malaria smears conducted (10%), outpatient visits for people aged 5 and older (8%), and in-facility deliveries (1%) over the 6-year period. The steepest rate of decline was for outpatient visits for children aged 4 and younger (10% decline). DPT3 (diphtheria, pertussis and tetanus) immunizations as a proportion of deliveries were flat (0.1% decline), as were TB sputum tests per 1000 population (0.4% increase). Although total raw number of maternal deaths did not decline over the period, the proportion of maternal deaths per 100 000 reported deliveries showed a 22% decline; this was not statistically significant, however. Comparisons of raw service counts over time could be confused by the increase in reporting with time; however, we adjusted service rates for the underlying population at risk, thus more accurately reflecting trends (Table 3).

Data from the annual district reports, adjusted for population, show a modest 2% increase in vitamin A and 1% increase in tetanus immunizations, but small declines in hospital admissions (0.5%) and major surgical operations (0.1%) (Table 4).

Relationship between PEPFAR-supported ART services and non-HIV service utilization

From the routine monthly reports collected in each district, we could have chosen any of ∼50 available non-HIV services. After assessing the variables for plausible relationships to PEPFAR ‘spillover’, (26 source variables) as well as completeness, we chose and cleaned 17 variables for further analysis. Several source variables were combined, e.g. OPD was collected by gender but inconsistencies in reporting required us to collapse male and female data for analysis. Six of these variables had too many missing for analysis. We rejected measles immunization because data from outreach campaigns were not sufficiently captured in district routine data reports. Family planning (two variables—intermittent preventive treatment in pregnancy second dose and antenatal care fourth visit data) will be presented in another particle focusing on maternal health. We settled on six variables to include in our regression model: in-facility deliveries, outpatient visits for children four and younger, DPT3 immunizations, TB tests, malaria smears and maternal deaths.

To illustrate an example of the bivariate relationship between rates of a non-HIV service and PEPFAR investment, we provide Figures 2a and b illustrating the relationship between rates of in-facility delivery of babies and the number of patients on ART for each of Uganda’s 56 (original 2005) health districts, for the years 2005–2010. Figure 2a shows a bivariate relationship between monthly in-facility deliveries in health units and ART care for the entire study period. Figure 2b portrays the same relationship, disaggregated for each annual period across the 6 years of the study. Each circle represents annual average of monthly rates of the non-HIV services per 1000 people for each district plotted against the number of patients receiving PEPFAR-supported ART in that year. Other than a few outlier districts that did well in increasing in-facility deliveries, the scatter of monthly data points are relatively without pattern.

Table 5 portrays the incident rate ratios (IRRs) for high-investment and medium investment district-months compared with low-investment district-months. These are estimated from negative binomial multivariable mixed effect models that associate outpatient visits, deliveries, vaccinations, TB tests, malaria smears and maternal deaths with categories of counts (tertiles) of patients on PEPFAR-supported ART, adjusted for covariates. Only outpatient visits, TB tests and in-facility deliveries show suggestive relationships with PEPFAR-supported ART in both investment-level categories, and all suggest this investment is associated with a reduction in services. DPT3 showed a significant association only for high investment compared with low (the medium investment was insignificant), and that association also seemed to suggest PEPFAR was associated with fewer immunizations. Data were not sufficiently rich to support estimation of relative rates for outpatient visits for people aged 5 and older. The Web Appendix shows deliveries (collapsed across all districts) and PEPFAR investment (ART enrolment) by time, showing that as ART increased each year, deliveries remained flat. The overall graphic represents fairly the typical pattern in each district.

In-facility deliveries increased from 14 per 1000 population in 2005 to 19 in 2011. We estimated that health facilities with medium and high ART investment levels had 4–5% fewer deliveries when compared with low investment [IRR = 0.96, 95% CI (0.934–0.994) for medium investment compared with low, IRR = 0.95, 95% CI (0.909–0.996) for high compared with low], after adjusting for tertiles of sanitation, elementary education and HIV prevalence, along with month and year control variables and an exposure term for population.

TB tests in Uganda increased from five to seven tests per thousand per year overall during our study period; however, PEPFAR investment appears to modestly detract from that increase. The number of TB sputum tests was 12% lower for medium investment compared with low support [IRR (95% CI) = 0.88 (0.83–0.94)], and 22% lower for high investment [IRR (95% CI) = 0.78 (0.72–0.85)] in similarly configured models.

For outpatient visits delivered to children aged four and younger, our adjusted regression shows 7% fewer visits in medium investment districts [IRR (95% CI) = 0.93 (0.896–0.960)], and 11% fewer in high investment districts compared with low investment district-months [IRR (95% CI) = 0.89 (0.848–0.939)]. Our presumption was that an increase in outpatient visits for children would reflect an improvement in the health system’s ability to serve this highly underserved population.

An alternative explanation for fewer outpatient visits among young children could be reduced demand for care because health status was improving. To test this theory, we ran separate models that included under-5 mortality and underweight-for-age data from the UDHS survey at the regional level (results not shown). We compared the IRRs from our original models to the IRRs generated by the models with additional child mortality data and underweight-for-age data. The results showed little difference, suggesting improvements in child health are probably not an underlying cause for the decline in demand for under-five health services. We rejected the theory that outpatient visits among young children declined in response to improving health status.

To portray the sensitivity of results to modelling choices, Figure 3 shows the results of a set of related models associating monthly volumes of diphtheria, tetanus and pertussis vaccine (DPT 3) by age 1 with annual PEPFAR investment in that district. We show the relative rate estimates for the medium and high ART PEPFAR investment tertile, in comparison to the lowest tertile of no or low investment. Also portrayed are the 95% CIs around these estimates. We portray eight models (each with different combinations of control variables) in Figure 3, to illustrate the effects of the various combinations on the relationship between high or medium PEPFAR investment (in relation to little or no investment) for DPT3 immunizations. Although each model includes a different set of adjustment variables and thus shows slightly different results, the overall pattern across models is fairly consistent: PEPFAR investment is associated with somewhat worse outcomes on DPT3 immunization coverage in district-months with the highest ART investment. See Web Appendix for DPT3 in relation to ART enrolment each year, illustrating DPT3 declines in relation to ART.

Figure 3.

Relative rate increase and its 95% confidence interval of DPT-3 immunizations by age one in health facilities by the tertile of patients on ART relative to the lowest ART tertile for each of Uganda’s 56 (original) districts, across the years 2005–2010. As detailed in the notes below, the first pair of estimates is for minimally specified models, the following pairs show sensitivity of the estimates to various adjustment approaches. Source of data and notes: We portray the results of a number of models for diphtheria, tetanus and pertussis vaccine delivery by age one in relation to PEPFAR investment. Each model is reported in a pair of results: first, the estimated risk ratio for medium investment compared with low or zero investment showing the number of times more likely these immunizations are with medium ART investment, and second, the estimated risk ratio for high investment compared with low or zero investment, again showing the number of times more likely immunizations are with high ART investment. At the top of the figure is the ‘sparest’ model, with adjustment for year, month, and number of individuals receiving PEPFAR-supplied ART and including a model exposure for district population. The next model is the same as the first, but ART investment is also adjusted by population size. All models with ART adjusted for district population include ‘ART Pop Adj’ in the label. The third model replaces population size with the proportion of pit-latrine coverage in the district with ART counts, and the fourth model includes education enrolment (but not pit-latrine coverage or population size). The next two models (5 and 6) adjust ART for population along with pit-latrine and education enrolment. The final two models are fully-adjusted with both pit-latrine and education, the first without ART population adjustment, and the last with ART population adjustment. Conclusion: while each model shows slightly different results, we observe that when other variables in the model are held constant PEPFAR investment is associated with somewhat lower rates of DPT3 immunization coverage, particularly in district-months with the highest ART investment.

We found that PEPFAR investment was associated with small declines in service volumes in several key areas of non-HIV care (outpatient care for young children, TB tests and in-facility deliveries), after adjusting for sanitation, elementary education and HIV prevalence.

Discussion

We sought to understand whether PEPFAR generated any spill-over benefits or harms for the health system in the course of its highly focused efforts to stem the HIV epidemic in Uganda. Although there is ample evidence in the academic literature that PEPFAR successfully addressed its HIV-control mission, we found, in aggregate, only small health system utilization changes related to PEPFAR investment, at least in Uganda. In both qualitative and quantitative results (not shown), DHOs reported many benefits to their operations from HIV-related investments (largely understood to mean PEPFAR). The DHOs also noted, however, that these programmes diverted scarce human resources away from other important health problems (Lohman et al. 2015). The extent of this diversion is underscored by a recent cross-sectional survey demonstrating more than half of medical graduates from Uganda’s Mbarara University worked for HIV-related NGO’s in a country where HIV prevalence is no higher than 7% (Bajunirwe et al. 2013).

We conclude, therefore, that the Uganda PEPFAR programme, with its billion-dollar investment, missed a rare opportunity to strengthen the health system while providing AIDS care and treatment. Some would argue that taking time and resources to work on the public sector health system would have slowed the delivery of care in the emergency. Others would counter that with advancements in treatment, HIV is now a chronic disease best cared for in the regular health system; the best way to respond to HIV would be to broadly strengthen the primary health care system (Yu et al. 2008).

Did PEPFAR contribute to health system strengthening?

Global health initiatives, including PEPFAR, the Global Fund to Fight AIDS, TB and Malaria (Global Fund), and the World Bank Multi-Country AIDS Programme have demonstrated it is possible to rapidly scale up the delivery of health services by channelling funds through NGOs (Biesma et al. 2009). However, these efforts have had the unintended effect of distorting governance of existing health delivery systems in the public sector (Frenk and Moon 2013), distracting governments from efforts to strengthen their health systems and establishing vertical planning, management, monitoring and evaluation systems (Bassett et al. 2013). Further, the Global Fund has been subject to corruption; funds were suspended to Uganda after the discovery that >$50 million were allegedly misappropriated (Eaton 2005a, b).

The emergency nature of PEPFAR confined investments to vertical, largely ART-delivery related activities. The quantitative evidence from our study suggests there were small, mostly negative, effects on the health system from PEPFAR investments. Specifically, our results (reported in Table 5 ranged from a 22% reduction in TB tests [statistically significant 95% CI 0.72–0.85), to a 1% improvement in malaria blood smears (95% CI 0.94–1.07)] associated with PEPFAR investment. We believe the evidence supports the ‘work-around’ model depicted in our conceptual framework (Figure 1), suggesting PEPFAR did not strengthen the existing health system.

The law establishing PEPFAR [Public Law 108–25 (27 May 2003, 117 STAT.)] was specific in its intention to address the HIV/AIDS emergency through non-government implementing partners, many of them US based. The legislation read, ‘NGOs, including faith-based organizations, with experience in health care and HIV/AIDS counselling, have proven effective in combating the HIV/AIDS pandemic …(United States 2003) However, the reauthorization of PEPFAR in 2008 also stated the second generation of PEPFAR would engage in ‘strengthening health policies and health systems of partner countries (Lantos and Hyde 2008). Health system strengthening, however, is a very long-term enterprise, in that it entails infrastructure development, health workforce preservice training, and information system investments. Effects of PEPFAR second-generation activities, if any, may not yet have been realized within our study period.

In our conceptual framework, PEPFAR would have had to simultaneously and broadly invest in public sector decision-making processes, workforce training and retention, and facility construction and maintenance to produce large health system results. To some extent, especially under PEPFAR 2, investment increased in these areas (Institute of Medicine 2013). On the other hand, health facility leaders reported systems grew increasingly separate between 2005 and 2011: HIV medical records (52% reported separate systems in 2005, growing to 79% in 2011), HIV pharmacy (13% separate in 2005, rising to 24% in 2011) and HIV lab facilities (5% in 2005, 7% in 2011), from our study of >300 health facilities conducted separately (Makumbi et al. 2015). Indeed, when visiting facilities for this study, we observed multiple examples of separate HIV laboratory and clinical spaces that were clearly better funded, better cared for, and better run than non-HIV activities (Institute of Medicine 2013; Bajunirwe et al. 2013).

Among the limitations of our study, records of the early years of the Uganda HMIS programme (2005–2007) were missing many data points, and districts with better data were probably biased in relation to those with missing data. Therefore, the data points in the earlier years contributed less information to our model. However, our population at risk approach to the analysis corrected for this bias in estimation of effect size and statistical significance. We limited our analysis for this article to health system outcomes (changes in utilization of selected non-HIV services) rather than inputs; another article in our series assesses the role of PEPFAR investments, such as blood safety, laboratory strengthening and infection control practices. Our proxy for PEPFAR investment was the number of individuals on ART; however, efficiencies over time probably accelerated the number of people on ART at a faster rate than funding increased. Individuals on ART were reported annually. We were not able to obtain all the potentially confounding variables we would have liked, such as total Ministry budgets for health services at the district or facility level. Because our main independent and dependent variables were analysed at the district level, using confounder data for HIV prevalence at the regional level (under the assumption that all districts within a region have the same value) may have caused some confounder misspecification. We did not analyse the variety of explanations for reduced services such as DPT3 or in-facility deliveries, aside from their association with PEPFAR investment. For example, others found reduced DPT3 immunization in Tanzanian clinics with high ART utilization, perhaps driven by stigma (Goodson et al. 2013). District splitting led to lost data, as files were moved from one location to another; as a result, any loss of recorded services could sometimes be attributed to District splitting itself, rather than any PEPFAR or other programmatic associations.

Our study contributes to the literature exploring the controversial question of how PEPFAR affected health systems. We conducted a large nationally representative, longitudinal study, collecting 6 years of routine health data at the sites where they were stored in 112 health districts and >300 health facilities. The routine nature of the data is one of the strengths of the study, as it was collected for the Ministry of Health as a matter of regular operations rather than for purposes of demonstrating effects of an investment programme.

To address an emergency epidemic of HIV in Uganda and other PEPFAR countries, donors insisted on fast, measurable results. Implementing partners did the best they could to track myriad data related to the outcomes of greatest interest. PEPFAR can clearly take credit for treating millions of patients with HIV, while keeping the HIV epidemic from overwhelming the health care system.

Nonetheless, as HIV/AIDS care increasingly resembles chronic disease management, and with new emerging infections such as Ebola and Ebola, it is now clear that low-income countries need to scale up health system strengthening. One Ugandan health leader told us,

This (PEPFAR’s vertical approach) was understandable earlier on to get some results quickly and thereby turn the tide of the epidemic. Global Health Initiative principles, however, emphasize country ownership, country leadership and the need to transition these programs into the existing health systems (U\.S\. Global Health Initiative Priniciples 2013). Now, it is important to fix the inherent weaknesses in the system and make it ready to address other, current and future emergencies.

Conclusion

PEPFAR’s goal was to turn the tide of a major international epidemic, with a massive infusion of money, expertise and technical assistance. Its goal, at least until 2008, was ‘not’ to strengthen health systems generally. This study sought to understand whether PEPFAR, as a vertical program, may have had a spill-over effect on the health system generally, as measured by utilization. Our conclusion is that it did not, at least not in Uganda.

If we seek to sustain our gains on HIV/AIDS while addressing emerging infections, the prevalence of weak government health systems in the developing world becomes the next ‘emergency’ (Barnhart and Hagopian 2014). The global health community has adopted universal health coverage as the goal that bundles health system strengthening, the right to health and the social determinants of health (Horton and Lo 2013). Achieving that goal would require unprecedented investments in health. Ample evidence (Kim 2013; Fernandes et al. 2014) exists; however, that such investments would successfully contribute to the ambitious global goal of prosperity for all.

Funding

The research was organized through a Cooperative Agreement from the U.S. CDC, with a PEPFAR Public Health Evaluation award to the University of Washington near the end of 2010. The University of Washington sub-contracted with Makerere University in Kampala, Uganda, to provide in-country partnership for leadership, scientific guidance and management. Additional partners included the Ministry of Health’s Resource Center and the Uganda office of CDC. Research funding was provided by the PEPFAR through the CDC, under the terms of the Public Health Evaluation project entitled ‘Assessment of the Impact of PEPFAR/Global Disease Initiatives on non-HIV Health Services and Systems in Uganda’ (CE.08.0221). Although the CDC cleared this article for publication, the findings and conclusions in this report are those of the authors and do not necessarily represent the official position of CDC.

Conflict of interest statement: Our study of the health systems effects of PEPFAR was supported with PEPFAR funding.