Challenges to estimating vaccine impact using hospitalization data

Abstract

Because the real-world impact of new vaccines cannot be known before they are implemented in national programs, post-implementation studies at the population level are critical. Studies based on analysis of hospitalization rates of vaccine-preventable outcomes are typically used for this purpose. However, estimates of vaccine impact based on hospitalization data are particularly prone to confounding, as hospitalization rates are tightly linked to changes in the quality, access and use of the healthcare system, which often occur simultaneously with introduction of new vaccines. Here we illustrate how changes in healthcare delivery coincident with vaccine introduction can influence estimates of vaccine impact, using as an example reductions in infant pneumonia hospitalizations after introduction of the 10-valent pneumococcal conjugate vaccine (PCV10) in Brazil. To this end, we explore the effect of changes in several metrics of quality and access to public healthcare on trends in hospitalization rates before (2008–09) and after (2011–12) PCV10 introduction in 2010. Changes in infant pneumonia hospitalization rates following vaccine introduction were significantly associated with concomitant changes in hospital capacity and the fraction of the population using public hospitals. Importantly, reduction of pneumonia hospitalization rates after PCV10 were also associated with the expansion of outpatient services in several Brazilian states, falling more sharply where primary care coverage and the number of health units offering basic and emergency care increased more. We show that adjustments for unrelated (non-vaccine) trends commonly employed by impact studies, such as use of single control outcomes, are not always sufficient for accurate impact assessment. We discuss several ways to identify and overcome such biases, including sensitivity analyses using different denominators to calculate hospitalizations rates and methods that track changes in the outpatient setting. Employing these practices can improve the accuracy of vaccine impact estimates, particularly in evolving healthcare settings typical of low- and middle-income countries.

1. Introduction

In recent decades, the use of vaccines against infectious diseases of global importance has grown substantially. Efficacy is always assessed in pre-licensure trials before regulators approve a new vaccine. However, the real-world impact of vaccines can be greater or smaller than efficacy due to indirect factors such as herd immunity and serotype replacement. Moreover, pre-licensure trials are performed under idealized conditions and often exclude certain high-risk individuals. Therefore, studies that assess the impact of newly introduced vaccines on disease rates at the population level are critical.

In real-world populations, however, nothing is static. Estimating vaccine effects with accuracy is always complicated by other changes that occur in the population around the time the vaccine is introduced. Unrelated trends can be particularly pronounced in rapidly developing countries, where the introduction of a new vaccine often occurs concomitant with unrelated improvements in public health.

Because population-based surveillance data are rarely available on a large scale, and laboratory confirmation of the causative pathogen is often not possible, vaccine impact evaluations often rely on proxy measures of disease rates at the community level. Hospital admission data have been widely used for that purpose, often by comparing hospitalization rates for a disease within the same population before and after vaccine introduction [1–3].

Electronic hospitalization databases are effective tools for public health research, surveillance and planning, providing systematic and low-cost information about large populations. However, hospitalization data are also prone to specific biases and confounding that can affect estimates of vaccine impact, as admission rates are closely linked to changes in broad societal trends that affect not only biological susceptibility to disease, but also healthcare delivery itself.

Here we investigate how changes in healthcare delivery influence estimates of vaccine impact by exploring the association between hospitalization rates and healthcare access and quality. As a case study, we focus on changes in infant hospitalization rates for pneumonia after introduction of the 10-valent pneumococcal conjugate vaccines (PCV10) in Brazil in 2010.

Our results indicate that commonly employed adjustments are not always sufficient to control for changes in hospitalization rates unrelated to vaccine introduction, particularly in evolving healthcare settings. We discuss several ways to identify and address these biases.

2. Methods

To examine how changes in healthcare use and delivery affect hospitalization rates, we calculated several metrics of healthcare utilization, quality, and access before and after PCV10 introduction. We then explored how changes in these metrics were associated with simultaneous changes in crude hospitalization rates for pneumonia and comparison outcomes. We focused the analysis on infants <12 months, the primary group targeted for vaccination.

2.1. Data on hospitalizations

In Brazil, access to the public health service is, in principle, universal, and comprehensive data are available on people who receive public care (82% of the population in 2012). We obtained de-identified, age-stratified monthly data on hospitalizations (Jan2003–Dec2013) from the Unified Health System (SIH-SUS, Ministry of Health; [4]), which maintains a nationwide database that records all hospitalizations paid by the government. This database has been shown to record pneumonia hospitalization incidence in infants as reliably as prospectively collected primary hospitalization data [5].

To minimize the number of nosocomial pneumonia cases included in the time series, we excluded hospitalizations associated with “treatment packages”—sets of services, supplies and procedures—unrelated to community-acquired pneumonia (Table S1). We also excluded records in which the “length of stay” field was inconsistent with the discharge and admission dates (representing instances where a patient was immediately readmitted following discharge). We also excluded records from union, university, and self-financed hospitals, as those did not contribute consistently to the database.

2.2. Inpatient healthcare delivery: use of public hospitals and hospital capacity

Although access to public health is universal, some citizens opt for a privately financed tier of care perceived to offer better quality and faster access. To determine the size of the infant population actually using the public system (SUS) and thus represented in the hospitalization database, we subtracted the number of infants enrolled in private insurance plans offering hospitalization coverage [6] from the total infant population (Brazilian Institute of Geography and Statistics).

We considered hospital bed supply (hospital beds available in SUS; [7]) as a measure of hospital capacity.

2.3. Healthcare delivery: outpatient care access and quality

Because many pneumonia hospitalizations are preventable by appropriate care and management, improvements in outpatient services at the time of PCV10 introduction might reduce pneumonia hospitalizations. We focused on three metrics: (i) the number of health units offering basic healthcare services [8]; (ii) the percentage of the population appropriately covered by health teams [9], defined by the government as the number of teams working in an area multiplied by 3000 (based on the recommendation that each team can provide proper care for 3000 people) and divided by the population living in the area, and (iii) the total number of UPAs (Unidades de Pronto Atendimento; [10]), i.e. emergency care units placed in many municipalities starting in 2008; UPAs are designed to reduce the demand for hospital services, and are equipped to treat cases of higher complexity than basic healthcare units.

We also examined potential changes in the quality of outpatient care by analyzing trends in rates of “potentially avoidable hospitalizations” (PAHs), a group of disease outcomes often used as an indicator of primary care quality; we used the list of PAHs adopted by the Brazilian government (Table S2). In theory, improvements in prevention and early disease management at the outpatient setting should lower PAH rates.

2.4. PCV10 uptake

Brazil introduced PCV10 in its National Immunization Program (PNI) in March 2010. To estimate PCV10 coverage, we calculated the percentage of eligible infants (6–23 months) who had at least the 3 recommended routine doses (at 2, 4 and 6 months) using the age-cohort method described in [11]. Data on doses administered by state, month and age was provided by PNI [12]; live birth statistics (SINASC, [13]) were used to estimate rates. Nationally, PCV10 coverage reached 33%, 76% and 89% by December 2010, 2011 and 2012, respectively.

2.5. Data analysis

We first examined the consistency of ICD10 pneumonia coding from 2003 to 2012. We found that the pattern of codes assigned for many disorders changed dramatically in January 2008, coincident with substantial changes in the system used to reimburse hospitals, which relaxed the specificity of ICD10 reporting requirements. For example, the frequency of pathogen-specific codes such as J13 (pneumococcal pneumonia) and J14 (H. influenzae pneumonia) fell sharply, while the number of J18 codes (pneumonia, organism unspecified) rose (Fig. S1). We therefore limited our pre-post analysis to the period starting in January 2008.

We next tested the association between changes in hospitalization rates before and after PCV10 introduction and concomitant changes in the healthcare metrics previously described. Changes in hospitalization rates were estimated as in [14], by calculating the ratio (incidence rate ratio, or IRR) between the average annual incidence rate two years post-PCV (2011–12) and two years pre-PCV10 (2008–09), with their corresponding 95% confidence intervals [15]. The association between IRR and relative changes in healthcare metrics were tested with Pearson and Spearman correlations.

Because our aim was to illustrate the effect of broader changes in healthcare delivery on simultaneous changes in crude rates of hospitalization rather than to estimate vaccine impact itself, we did not attempt to determine the adjusted impact of PCV10 using models or methods other than the calculation of IRRs. IRRs reported here should therefore not be taken as estimates of PCV10 impact; they are only used to illustrate the effect of the potentially confounding trends discussed on hospitalization rates for pneumonia.

Pre-post PCV10 IRR estimates, calculated as described above, were computed for all-cause pneumonia (ICD10 J12–18), respiratory disease (J00–99), infectious diseases other than diarrhea (A10-B99), which was excluded to avoid confounding by rotavirus vaccine introduction in Brazil, and all hospitalizations (all causes). We conducted the analyses both at a national and state level, to take advantage of variability in the timing at which the states introduced changes in the health system.

We also analyzed whether putative associations between improvements in healthcare and reductions in hospitalizations for pneumonia were a by-product of differences in PCV10 coverage among states, given that in states where the health system improved more, PCV10 coverage may have been higher, driving pneumonia hospitalizations down. To this end, we tested the association between PCV10 coverage (at December 2010, 2011 and 2012) and IRRs for all-cause pneumonia (Pearson correlation). We also tested whether PCV10 coverage was associated with changes in healthcare delivery.

Data extraction and processing were performed with SAS 9.3. IRR estimates were calculated in Excel, and all correlations with Minitab v.17.3.1.

3. Results

3.1. Effect of changes in the population using hospitals in the public system

Estimates of vaccine impact depend on how well the denominator used to calculate hospitalization rates represents the population using hospitals. From 2008 to 2012, the infant population dropped by 7% (from 3.1 to 2.8 million infants). Moreover, the percentage of Brazilian infants who participated in the SUS system, and who thus contributed to the data, fell from 85% to 83% (from 2.6 to 2.3 million infants). When using the total Brazil population of infants as the denominator, the IRR for pneumonia was 0.89 (95%CI 0.88–0.90)—an 11.3% drop nationwide, but only 0.92 (95% CI 0.91–0.93)—a 7.8% drop—when the SUS population was used as the denominator (Table 1). That represents a 30.8% difference between estimates. In all regions, denominating pneumonia hospitalizations by the SUS population rather than the total population reduced the magnitude of the drop in hospitalization rates (Table 1); the difference was greatest in the Southeast, with the estimated drop almost halving when using SUS population (from 11.5 to 6.5%).

Table 1.

Effect of population denominator on IRR estimates of pneumonia hospitalizations (J12–18) in 2011–12 vs. 2008–09 for children <12 months of age.

Regions	IRR (95%CI) [% decline]		% difference in estimated decline [(A − B)/A] / 100	% infants using SUS
	Denominator: total population [A]	Denominator SUS population [B]		2008–09	2011–12
National	0.89 (0.88–0.90) [11.3%]	0.92 (0.91–0.93) [7.8%]	30.8%	85.0%	81.7%
North	0.89 (0.87–0.91) [10.9%]	0.90 (0.88–0.92) [10.2%]	5.9%	93.1%	92.4%
Northeast	0.89 (0.88–0.91) [10.7%]	0.92 (0.90–0.93) [8.1%]	24.2%	93.4%	90.7%
Centre-West	0.85 (0.82–0.88) [15.2%]	0.89 (0.86–0.92) [11.0%]	27.7%	91.4%	86.9%
Southeast	0.88 (0.87–0.90) [11.5%]	0.94 (0.92–0.95) [6.5%]	43.9%	74.6%	70.5%
South	0.85 (0.83–0.86) [15.8%]	0.86 (0.84–0.88) [13.8%]	12.5%	84.0%	82.0%

3.2. Effect of changes in inpatient healthcare delivery systems

Public health care (SUS) is provided in Brazil through both nonprofit (government-owned and philanthropic) and for-profit hospitals (under contract to provide beds to the public system). Nationally, the supply of public hospital beds provided through both for-profit and nonprofit hospitals dropped by approximately 5% from 2008 to 2012 (from 2.2 to 2.1/1000 population). For each Brazilian state, we plotted the change in the number of beds available per 1000 infants against IRR. We found that reductions in hospitalization rates for pneumonia were sharper in states where bed loss was more pronounced (Fig. 1). Strong associations between bed loss and IRR were also observed for the other outcomes, particularly all-cause hospitalizations (Fig. 1).

Fig. 1.

Relative change in hospital beds in the public health system. Bed availability per 100,000 population in 2011–12 vs. 2008–09 (IRR_beds) is plotted against change in hospitalization rates in 2011–12 vs. 2008–09 (IRR_hosp). In states where bed availability decreased more (lower IRR_beds), the estimated drop in hospitalization rates was also larger (lower IRR_hosp). Pearson correlation: J12–18 (r = 0.46, p < 0.05); J00–99 (r = 0.55, p < 0.01); A10-B99 (r = 0.47, p < 0.05); all-causes (r = 0.66, p < 0.001).

Changes in public bed supply, however, were different among for-profit and non-profit hospitals. In for-profit hospitals, beds made available to SUS fell 13% from 2008 to 12 (from 0.89 to 0.77 beds/1000), but increased 5% among non profit hospitals, from 1.29 to 1.36 beds/1000. We hypothesized that if the number of available beds affects admissions, declines in hospitalization rates would be greater in for-profit hospitals. We found this to be the case (Table 2), with a greater decline in pneumonia hospitalizations in for-profit (IRR = 0.70; 95%CI 0.68–0.71) than nonprofit facilities (IRR = 0.98; 95%CI 0.97–0.99). We found a similar result for all respiratory outcomes (J00–99; Table 2), suggesting that IRR differences between for-profit and non-profit hospitals were not due to differences in risk factors for pneumonia or PCV coverage between them. There was no significant association between bed loss and PCV10 coverage (p > 0.05).

Table 2.

The effect of using SUS population and all-cause hospitalizations as the denominator when calculating incidence rate ratios (IRR) (2011–12 vs. 2008–09 pneumonia hospitalization rates) in nonprofit and for-profit hospitals providing care to SUS patients.

		Disease outcome	Effect of changes in bed availability
			IRR (95% CI) in nonprofit hospitals in SUS system	IRR (95% CI) in for-profit Hospitals in SUS system
Effect of denominator used to calculate IRR	SUS population as denominator	Pneumonia (J12–18)	0.98 (0.97–0.99)	0.70 (0.68–0.71)
		All Respiratory (J00–99)	1.04 (1.03–1.05)	0.71 (0.70–0.72)
	All-cause hospitalizations as denominator	Pneumonia (J12–18)	0.87 (0.86–0.88)	0.92 (0.90–0.94)
		All Respiratory (J00–99)	0.92 (0.91–0.93)	0.92 (0.90–0.94)

Because all-cause hospitalizations track the total capacity of the hospitalization system, we next calculated IRRs for pneumonia and respiratory disease using the total number of hospitalizations as the denominator. Overall, there were large differences in estimated IRRs between the analysis using SUS population and total hospitalizations as the denominator (Table 2). In the latter case, IRR differences between for-profit and non-profit were substantially reduced or were not significant.

3.3. Effect of changes in outpatient healthcare delivery system

From Jan-2008 to Dec-2012, the proportion of the population covered by appropriate primary care teams (see Methods) increased nationwide from 64% to 67%. Similarly, the number of outpatient units providing basic care services increased from 96,000 to 115,000 units. At the same time, the proportion of hospitalizations classified as potentially avoidable (PAHs, an indicator of primary care quality; see Methods) relative to total hospitalizations fell from 30% to 25%.

In line with the expectation that most pneumonia hospitalizations are potentially avoidable given appropriate primary care, rate reductions for pneumonia were larger in those states where primary care coverage increased more (Fig. 2), and in states that experienced a larger growth in the number of units providing basic healthcare services (Fig. 2). Significant correlations (Pearson; r = −0.34 to −0.49; p < 0.05) in the same direction were also found for respiratory disease (J00–99), and all-cause hospitalizations, but not for infectious diseases (A10-B99). These findings were not driven by differences in PCV10 coverage among states, as PCV10 coverage was not significantly correlated with any of the changes described.

Fig. 2.

Incidence rate ratio (IRR) for pneumonia (J12–18) in infants (post/pre PCV10 introduction) and three measures of change in outpatient healthcare (post- vs. pre-PCV10). Larger drops in pneumonia hospitalization rates (lower IRR) were achieved in those states where improvements in access to primary care were larger. Pearson correlation between IRR for pneumonia and: A (r = −0.57, p < 0.01); B (r = −0.43, p < 0.05); C (r = −0.52, p < 0.01).

We examined whether the increasing availability of UPAs (new emergency outpatient units; see Methods) was associated with a reduction in pneumonia hospitalization rates following PCV10 introduction. We found that in states that created more UPAs up to 2012, rate reductions in pneumonia hospitalizations were also greater (Fig. 2).

4. Discussion

Observational studies of trends in hospitalizations have been widely and productively used to estimate disease burden and the impact of vaccines [16–20]. However, attributing changes in hospitalization rates before and after the start of an intervention specifically to that intervention can be challenging, and may lead to an over- or under-estimation of benefits in countries where the healthcare system is itself evolving. Because such changes often occur rapidly in low- and middle-income countries, one must be particularly cautious in such settings, especially if the expected impact of a vaccine is relatively modest. In Table 3 we summarize prior studies of PCV impact on all-cause pneumonia hospitalizations in Latin America, and their use of different practices that might help to identify or adjust for such biases.

Table 3.

Summary of studies of PCV impact on hospitalization rates for childhood pneumonia in Latin American countries.

Country	Period studied (Pre/Post)	Age (months)	Nature of data	Geographic subdivision	Denominator	Analyzed Secular Trends?	Analyzed Hospital Capacity?	Used control outcomes?	Analyzed outpatient setting?	Reference
Brazil	2005–09/2011	2–24	Prospective	5 Municipalities	Live births	Yes	No	Yesb	No	[20]
	2002–09/20111–12	<48	Administrative	National	Population	Yes	No	Yesc	No	[41]
	2007–09/2011–12	2–35	Prospective	1 Municipality	Population	Yes	Yes	No	No	[38]
	2007–09/2011–13	<12	Administrative	26 Municipalities	Population	No	No	No	No	[32]
Uruguay	2001–04/2009–11	<60	Prospective	4 Hospitals	Population	No	No	No	No	[33]
	2001–04/09–12	<168	Prospective	2 Municipalities	Population	No	No	No	No	[34]
	2003–07/2012	<168	Prospective	1 Hospital	ACHa	No	No	No	No	[35]
	2005–07/2009	<168	Prospective	1 Hospital	ACHa	No	No	Yesd	No	[36]
Panama	2007–08/2008–10	<60	Administrative	1 Hospital	ACHa	No	No	No	No	[37]
Nicaragua	2008–10/2011–12	<24	Administrative	107 Health Units	Population	No	No	Yese	Yes	[31]
Peru	2006–08/ 2011–12	<12	Administrative	National	Population	Yes	No	No	Yes	[39]
Argentina	2003–05/2012–13	<60	Prospective	2 Hospitals	Population	No	No	No	Yes	[40]

^aAll-cause hospitalizations.

^bNon-respiratory disease & bronchiolitis.

^cNon-respiratory disease.

^dAcute gastroenteritis.

^eDiarrhea.

In Brazil, we found a positive association between the supply of hospital beds and rates of hospitalization for pneumonia; others have published similar findings [21,22]. This suggests that hospitals were operating at full capacity, an observation corroborated by a recent report showing that most public hospitals in Brazil cannot meet demand for services [23]. Indeed, the 2.2 beds per 1000 persons in 2008 was less than half the European average of 5.5 beds/1000 that same year [24].

Interpreting trends in disease rates inferred from hospitalization data is difficult when hospitals are running at or over capacity. In such circumstances, higher community-acquired disease rates may not translate into proportional increases in hospitalizations because hospitals would not be able to meet the increased demand. Conversely, lowered disease rates could be partially or even fully masked, as hospitalization databases have no records of potential patients turned away due to lack of available beds.

Using all-cause hospitalizations to denominate hospitalization rates can help control for changes in hospital capacity. Additionally, all-cause hospitalizations can track and adjust for changes in the population using target hospitals – such as those caused by the migration to private insurance. However, if hospitals are operating at full capacity, and the target disease and/or age group studied has a different admission priority compared to other groups (e.g. pneumonia is high-priority in Brazil), denomination by all-cause hospitalizations can also result in biased estimates. For example, if the incidence of community-acquired pneumonia cases is unchanged (i.e. pneumonia cases decline proportionally with decreases in population size), hospital beds freed from pneumonia cases will be used for other conditions. Accordingly, the proportion of pneumonia over all-cause hospitalizations will reduce due to such non-vaccine related changes. Therefore, considering changes in both population size and all-cause hospitalizations when hospitals operate over capacity is critical.

Importantly, denomination by either all-cause hospitalizations or population does not prevent introduction of bias caused by changes in outpatient services [25]. In Brazil, expansion of outpatient care services was associated with reduced hospitalizations for pneumonia and other outcomes, indicating that changes in this setting can be an important source of confounding. For example, in 2006 Brazil introduced a nationwide program called ‘Pact for Health’. One of its goals was to reduce diarrhea and pneumonia deaths in young children. The program created local committees to monitor child mortality and trained medical staff in proper treatment of these diseases. These measures likely added to the effect of immunization programs, such as the introduction of the rotavirus vaccine in the same year.

One way to identify biases resulting from changes in outpatient care is the analysis of outcomes that are affected by factors that also affect the vaccine target, but are themselves unaffected by the vaccine [26]. Since changes in outpatient care can introduce bias, such comparison outcomes should be similarly sensitive to such changes. Because PAHs are by design sensitive to primary care [27–29], it may be possible to use trends in PAH rates, or appropriate subsets, as a proxy for changes in primary care quality. Alternatively, “synthetic controls” has been shown to provide a data-driven solution to control for unrelated trends in disease rates [30]. In this approach, time series of outcomes assumed to be unaffected by the vaccine are weighted and combined into a single composite time series depending on their fit to the time series of interest (e.g., pneumonia) before vaccine introduction. The composite “synthetic control” is then calculated by applying the weights from the pre-vaccine period to the outcome incidences in the post-vaccine period, producing a counterfactual estimate of disease rates had no vaccine been in use. This method does not require a priori qualitative decisions on what constitutes an appropriate control, which can itself introduce bias. Another option is to analyze inpatient and outpatient trends simultaneously (e.g. [31]). If the vaccine works as expected, and disease etiology is not different between these settings, the burden of both hospitalizations and outpatient visits for the vaccine target should fall.

Of the studies summarized in Table 3, three [32–34] did not seek to adjust for secular trends, potential biases and confounding. Seven addressed potential biases arising from changes in inpatient care by using all-cause hospitalizations as a denominator [35–37], analyzing control outcomes possibly with similar admission priority [20,31,35] or directly analyzing hospital capacity [38]. Similarly, five used controls potentially sensitive to primary care changes [20,31,36] or examined outpatient data [31,39,40]. The analysis of secular trends [20,38,39,41] and single control outcomes [20,31,36,41] can help control biases, but may not have been sufficient to control them completely. For example, the potential reduction in pneumonia hospitalizations resulting from the creation of over 500 UPAs in 2010–2012 (just after PCV10 introduction in Brazil) may not be captured by these adjustments. Other unrelated events may also affect hospitalization rates in the pre- (e.g., 2009 influenza pandemic) or post-PCV periods if not properly adjusted.

In conclusion, caution is needed when attributing changing trends in hospitalization rates to vaccines. We suggest that in addition to considering secular trends, impact studies based on hospitalization data present sensitivity analyses showing the effect of using various denominators, adjust for the potential effect of changes in both the inpatient and outpatient settings and use comparison outcomes that also share similar hospital admission priority and sensitivity to primary care changes with the target disease. These practices would strengthen efforts to understand how vaccines translate into reductions of disease burden, especially where cost-effective interventions are most needed.