Reporting of Quality Indicators and Improvement in Hospital Performance: The P.Re.Val.E. Regional Outcome Evaluation Program

Abstract

Objective

To evaluate whether reporting of hospital performance was associated with a change in quality indicators in Italian hospitals.

Data Sources/Study Setting

Nationwide Hospital Information System for 2006–2009.

Study Design

We performed a pre-post evaluation in Lazio (before and after disclosure of the Regional Outcome Evaluation Program P.Re.Val.E.) and a comparative evaluation versus Italian regions without comparable programs. We analyzed risk-adjusted proportions of percutaneous coronary intervention (PCI), hip fractures operated on within 48 hours, and cesarean deliveries.

Data Collection/Extraction Methods

Using standardized ICD-9-CM coding algorithms, we selected 381,053 acute myocardial infarction patients, 250,712 hip fractures, and 1,736,970 women who had given birth.

Principal Findings

In Lazio PCI within 48 hours changed from 22.49 to 29.43 percent following reporting of the P.Re.Val.E results (relative increase, 31 percent; p < .001). In the other regions this proportion increased from 22.48 to 27.09 percent during the same time period (relative increase, 21 percent; p < .001). Hip fractures operated on within 48 hours increased from 11.73 to 15.78 percent (relative increase, 34 percent; p < .001) in Lazio, and not in other regions (29.36 to 28.57 percent). Cesarean deliveries did not decrease in Lazio (34.57–35.30 percent), and only slightly decreased in the other regions (30.49–28.11 percent).

Conclusions

Reporting of performance data may have a positive but limited impact on quality improvement. The evaluation of quality indicators remains paramount for public accountability.

The assessment of health care quality has become increasingly important in many European countries, and worldwide, in response to requests for greater transparency and accountability and for quality improvement (Groene, Skau, and Frolich 2008). However, there is still considerable debate regarding public reporting of performance data and its potential effects. Possible benefits of disclosing performance data to the public include fulfilling the patients' right to know the quality of care offered by various providers, the possibility for patients to make informed choices, and motivating hospitals and clinicians to improve their quality of care (Resnic and Welt 2009). Potential drawbacks include the possibility that inaccurate data and limits of risk adjustment methods may lead to incorrect identification of some providers as poor performers, damaging their reputations and activities. Moreover, focusing attention on some performance indicators may lead to the neglect of other important areas.

Most studies on the effects of public reporting on quality of care are observational studies that mainly focus on cardiac care (Fung et al. 2008). Overall, there is no clear evidence of an association between public reporting and improved quality of care, although some studies suggest that public reporting may motivate quality improvement activities (Jung 2010). One recent randomized controlled trial of cardiac care (Tu et al. 2009) revealed no significant system-wide improvement in process indicators; however, outcome indicators showed some improvements.

A Regional Outcome Evaluation Program (P.Re.Val.E.) was launched in the Lazio region of central Italy starting in 2005 (Fusco et al. 2012). Lazio has ∼5,700,000 residents who are mainly concentrated in the city of Rome (Italian National Institute of Statistics 2010). During the pilot phase in the years 2005–2007 only very limited information on the collected data was occasionally communicated to specific hospitals. This aimed at resolving problems due to missing or incorrect data. No comparative information on health care providers was provided during this time. The P.Re.Val.E. program evolved over the years from including a small number of indicators to comprising 54 indicators of hospital care in various clinical areas, including cardiology, orthopedics, obstetrics, gastroenterology, respiratory, and cerebrovascular diseases.

The dissemination of the P.Re.Val.E. results has taken place in Lazio at the end of 2007, when performance data were communicated to individual hospitals in at least three professional meetings held over a period of 3 months: one with the General Management of each hospital, one with the Clinical Director, and one meeting with clinical staff. The encounters included the presentation of hospital-specific performance and unblinded peer-hospital comparisons aimed to promote a positive and constructive approach to quality improvement, rather than to identify the best or worst performances. Hospital staff was invited to suggest possible interpretations of unexpected findings. After 2007 indicators have been updated and made available every 2 years. Participation in meetings for clinical staff was voluntary. Participation has been very high among cardiologists and cardiothoracic surgeons (∼90 percent), whereas it was <50 percent for maternity care personnel and orthopedic surgeons. However, it increased substantially for orthopedic surgeons during 2008 and 2009 (70 percent). In February 2008 hospital-specific information was also publicly released on an open access website (http://www.epidemiologia.lazio.it/vislazio/vis_index.php) and through meetings with patient and citizen associations. The information for the public was presented in general terms without inter-hospital comparisons and without highlighting the results of specific indicators. The aim was to inform the general public that a performance evaluation initiative was under way. Newspapers and Italian medical journals covered the project. Based on the P.Re.Val.E. experience, a National Program was developed in 2010, comparing health care throughout Italy. However, in all other Italian regions no active communication of the program has taken place between 2007 and 2009.

The objective of the present study was to evaluate whether public reporting of performance data from the P.Re.Val.E. project in Lazio in early 2008 was associated with a change in risk-adjusted quality indicators. In particular, we evaluated the change in hospital performance via a set of well-established quality indicators in Lazio and in other Italian regions. We hypothesized that public reporting of performance data in Lazio was associated with significantly greater improvement in performance than in Italian regions with no reporting of the outcome evaluation.

Methods

Study Design

We conducted a “quasi-experiment” including a pre-post evaluation and a comparative evaluation (Ross and Gross 2009). In particular, we compared hospital performance in Lazio during the years 2006/2007 with the years 2008/2009, before and after public reporting of the P.Re.Val.E. performance data in Lazio starting at the end of 2007. Hospital performance was also evaluated during the same study period in other Italian regions (except Tuscany) using the same set of quality indicators. No public or private reporting of the outcome evaluation program was performed in the other regions. Tuscany was excluded because it is the only other Italian region that has introduced a region-wide performance evaluation with public and private reporting of results (Nuti 2008).

Study Population and Data Source

We obtained the data from the Hospital Information System, a nationwide database that includes discharge abstracts, with demographic and clinical information, for all hospital admissions within the Italian National Health Service. We focused on three patient cohorts representing common conditions. Similar to previous studies (Serumaga et al. 2011), this sought to achieve stable estimates. We included patients with acute myocardial infarction (AMI), patients admitted for hip fracture, and women who had given birth during the study period. We included all patients, residents of Italy,discharged from any hospital within the Italian National Health Service between January 2006 and December 2009 who fulfilled the inclusion criteria described below.

• The AMI inclusion criteria: International Classification of Disease-9th-Revision-Clinical Modification (ICD-9-CM) code 410.xx, age 18–99 years. Main exclusion: hospitalizations lasting fewer than 48 hours and discharge to home (hypothesizing a misdiagnosis), or previous hospital admission for AMI within the last 4 weeks.

• Hip fracture inclusion criteria: ICD-9-CM code 820.0–820.9, age 65–99 years. Main exclusion: death within 48 hours from admission if not operated on (hypothesizing that surgical treatment was not possible due to severe clinical conditions); or admission for multiple trauma (DRG 484–487) or diagnosis of cancer (ICD-9-CM 140.0–208.9 or V10) during the previous 2 years or during the current admission.

• Inclusion criteria for women having delivered during the study period: ICD-9-CM diagnosis and procedure codes specific for pregnancy, delivery, complications of pregnancy and childbirth, age 11–54 years. Main exclusion: prior cesarean delivery in the past 2 years, stillbirth, or intrauterine death.

Quality Indicators

For the present study we examined a limited set of well-established quality of care indicators. The selection of the indicators was based on the following criteria: clear evidence from the international literature of the association between the indicator and improved health outcomes, reliable data available for all Italian regions, and coverage of different areas of hospital care.

We excluded indicators on mortality because of the lack of a mortality register for all Italian regions and intra-hospital mortality was markedly underreported in some regions precluding meaningful inter-regional comparisons. Furthermore, we excluded indicators on hospital admissions for chronic respiratory diseases, diabetes, and other diseases, because they reflect population and primary health care factors more than hospital care (Calderón-Larrañaga et al. 2011) and indicators examining length of hospital stay, because they are more linked to costs and discharge policies and less so to health outcomes.

Thus, the following indicators were included:

• Proportion of patients with AMI treated with percutaneous coronary intervention (PCI) within 48 hours. Timely PCI after AMI is associated with lower mortality (Tu et al. 2008; Van de Werf et al. 2008; Kushner et al. 2009).

• Proportion of older patients with hip fracture operated on within 48 hours of hospital admission. Earlier surgery reduces mortaliy and morbidity (Mattke et al. 2006; Novack et al. 2007; Shiga, Wajima, and Ohe 2008; Simunovic et al. 2010).

• Proportion of women with primary cesarean (no previous cesarean delivery). High cesarean delivery rates are medically unjustified and are associated with maternal and neonatal mortality and morbidity (Veillard et al. 2005; Belizán, Althabe, and Cafferata 2007).

In accordance with previous studies (Curtis et al. 2009), we used the following ICD-9-CM procedure codes to define PCI: 00.66, 36.01, 36.02, 36.05, 36.06, and 36.07. Surgical treatment for hip fracture was defined as total hip replacement (ICD-9-CM code 81.51 or 81.52) or fracture reduction (ICD-9-CM code 79.00, 79.05, 79.10, 79.15, 79.20, 79.25, 79.30, 79.35, 79.40, 79.45, 79.50, or 79.55). Cesarean section was defined on the basis of the following ICD-9-CM codes: diagnosis code 669.7, procedure codes 74.0, 74.1, 74.2, 74.4, and 74.99.

The 48-hour time interval before treatment was measured from the patient's first hospital access (index admission). The Hospital Information System only reports the date and not the exact time when a procedure was performed; therefore, in this study PCI was considered to have been performed within 48 hours if it was performed the same day as hospital access or within the next day. The Hospital Information System does not allow examination of whether PCI was performed within a shorter time interval. Exact time to surgery is available only in Lazio since 2008. Based on this data we conducted a sensitivity analysis comparing time to surgery in days and in hours. Our definitions for timely PCI and timely surgery for hip fractures correspond to κ = 0.92 and κ = 0.83, respectively.

Conceptual Model

Public reporting of performance data has been suggested to improve quality of care through two interconnected pathways (Berwick et al., #b503; Fung et al. 2008): (1) Published performance data allow patients, or general practitioners acting on patients' behalf, to select the better performing providers, who are “rewarded” by greater demands of services and by public recognition. This is expected to motivate providers to improve performance. However, evidence confirming this pathway is scant (Marshall and Romano, 2005). (2) Performance data with peer comparison are presented to providers, who can thus identify weak areas and implement internal improvement projects. Many consider this pathway to play a much greater role in performance improvements, with motivating forces including provider's professional pride, competitiveness, and sensitivity to their reputation among peers (Davies 2001; Marshall and Romano 2005).

The P.Re.Val.E. program focused particularly on direct involvement of health care professionals and managers, providing them with timely, unblinded peer-hospital comparison. The program emphasized the importance of sharing the interpretation of the results and offered the possibility of additional in-depth analyses, if requested by providers. In line with previous studies (Davies 2001; Marshall and Romano, 2005), we considered that the consumers would play a secondary role in influencing quality improvement.

Demographic and Clinical Factors Considered for Risk Adjustment

Information on patient clinical characteristics, risk factors, and comorbidities was obtained from the Hospital Information System on the basis of ICD-9-CM codes registered in the index hospitalization and in hospital admissions occurring during the preceding 2 years. A validated automatic record linkage procedure was employed (Agabiti et al. 2008). For each indicator we performed a literature review to identify potential confounding factors. Among them, we selected the confounders relevant for our study population by a bootstrap stepwise procedure, to limit the number of variables in the final models. Age and gender were included as a priori risk factors. The demographic and clinical variables included in the final models are shown in the Appendix.

Statistical Analysis

Risk-adjustment methods were applied to take into account that patient demographic characteristics and comorbidities could be heterogeneously distributed across regions of residence and study periods. We performed a stepwise analysis with 500 replicated samples of the original data and significance thresholds of 0.10 and 0.05 for input and removal of covariates, respectively. Risk factors selected in at least 50 percent of the runs were included in the final models.

For each indicator, we calculated the adjusted proportions for the two study periods and the two geographic areas, that is, Lazio and the control group, which comprised all the other Italian regions. Direct standardization was used considering as reference the total population registered over the 4 years (from 2006 to 2009). Three multivariate logistic regression models with no intercept, including centered covariates and an interaction term between geographic area and study period, were applied. These models estimate group-specific (geographic areas) log odds of proportions of outcome. Adjusted proportions were obtained for each level of interest by back-transforming parameter estimates with the following formula:

where k is a correction coefficient introduced to take into account the nonlinear nature of the logistic model. K is calculated as follows:

where p_j are the adjusted proportions, n_j is the group size, and m is the number of groups. To compare the proportions of the study outcomes between the two study periods, for each geographic area we calculated relative risks and absolute differences as ratio and difference between adjusted proportions, respectively. The hypothesis of equal proportions between the two study periods in each region was tested by a Z-test, assuming normal distribution for the logarithm of the relative risk. Similarly, the difference between the change in the intervention region (Lazio) and the change in the comparison group was also tested.

Finally, we analyzed the time trends since 2003 for each indicator in Lazio and the comparison group. All the analyses were performed using SAS Version 9.2.

Results

The main characteristics of the study population in Lazio and in the other Italian regions in July 2006 and September 2008 are shown in Table 1. The study included 381,053 patients treated for AMI; we excluded 6,354 patients from the initial AMI cohort, because of missing PCI date and 38,251 patients due to one or more exclusion criteria. Among hip fracture patients we included 250,712 cases, whereas we excluded 1,100 cases with missing surgery date and 140,889 cases due to one or more exclusion criteria. Among women who had given birth, 1,736,970 were included, whereas 282,258 reported one or more exclusion criteria.

Table 1.

Characteristics of the Study Population in the Lazio Region and in the Other Italian Regions in July 2006 and September 2008

	Lazio Region		Other Italian Regions
Main Characteristics*	2006–2007%	2008–2009%	2006–2007%	2008–2009%
Patients with AMI (n)	18,877	17,190	178,059	166,927
Age
Mean (SD), years	70.1 (13.2)	70.0 (13.3)	70.8 (13.2)	71.0 (13.4)
Gender
Women	35.6	34.8	37.2	37.0
Hypertension	52.1	49.5	47.1	46.2
Hyperlipidemia	22.8	20.5	21.8	20.8
Vascular diseases	16.7	15.7	17.4	17.4
Previous myocardial infarction	15.9	13.7	14.6	14.0
Other coronary heart disease	15.3	13.5	15.8	15.1
Chronic cerebrovascular diseases	11.3	10.2	11.2	11.1
Chronic kidney diseases	11.5	11.8	10.8	11.6
Previous PCI	8.6	9.5	10.2	11.8
Cardiac dysrhythmias	8.0	6.7	7.7	7.7
Heart failure	7.4	6.5	8.1	8.1
Malignant neoplasms	7.1	6.8	6.8	7.0
Chronic obstructive pulmonary disease	6.5	5.7	6.3	6.1
Diabetes	5.9	6.7	7.2	7.3
Previous CABG	5.8	5.7	5.4	5.7
Patients with hip fracture (n)	12,585	12,469	113,436	112,222
Age
Mean (SD), years	82.4 (7.3)	82.7 (7.1)	82.4 (7.3)	82.6 (7.2)
Gender
Women	78.6	77.8	78.1	77.8
Hypertension	14.4	14.9	13.5	13.6
Chronic cerebrovascular diseases	11.3	11.8	10.7	10.7
Other coronary heart disease	10.9	10.0	9.1	8.7
Cardiac dysrhythmias	7.6	7.9	7.1	7.2
Dementia including Alzheimer's disease	7.0	7.6	9.2	9.0
Diabetes	6.6	6.6	6.3	6.3
Chronic obstructive pulmonary disease	5.7	5.6	5.4	5.1
Chronic kidney diseases	4.7	5.6	4.7	5.2
Blood disorders	4.3	4.7	4.2	4.4
Heart failure	4.2	4.7	4.8	5.0
Previous myocardial infarction	2.2	2.8	2.3	2.3
Women having delivered during study period (n)	92,218	87,744	796,223	760,785
Age
Mean (SD), years	31.5 (5.3)	31.7 (5.4)	30.8 (5.3)	30.9 (5.5)
Abnormal fetal presentation	8.1	8.1	7.7	8.2
Premature rupture of membranes	5.1	5.4	9.4	10.1
Fetal distress	4.7	4.0	3.1	2.6
Disorders of amniotic fluid	2.5	2.7	3.2	4.0
Disproportionate fetal growth	2.0	2.0	1.9	1.8
Post-term delivery	1.6	1.6	2.7	3.2
Multiple gestation	1.5	1.6	1.5	1.6
Intrauterine growth retardation	1.4	1.4	1.8	1.8

^*Only the most frequently reported clinical characteristics are shown.

AMI, acute myocardial infarction; CABG, coronary artery bypass graft; PCI, percutaneous coronary intervention.

Patients treated in Lazio were similar to patients treated in the other regions regarding the examined characteristics, and patient characteristics were similar in July 2006 and September 2008. The only noteworthy differences were a lower proportion of women with premature rupture of membranes and with post-term delivery in Lazio compared to other regions.

Table 2 reports the hospital performance indicators in July 2006 and September 2008 for Lazio and the other regions. In July 2006 the proportions of AMI patients treated with PCI within 48 hours were similar in Lazio and in the other regions. In Lazio the risk-adjusted proportion of PCI within 48 hours changed from 22.49 to 29.43 percent before and after reporting of the P.Re.Val.E. results, respectively. This corresponds to an absolute increase of 6.94 percent and an increase relative to the baseline value of 31 percent (p < .001). In the other regions the risk-adjusted proportion increased from 22.48 to 27.09 percent during the same time period. This corresponds to an absolute increase of 4.61 percent and an increase of 21 percent relative to the baseline value (p < .001). The increase in Lazio was significantly higher than the increase in the other regions (p < .001). For PCI performed after 48 hours, we detected similar proportions in July 2006 and September 2008: 19.82 percent and 19.53 percent, respectively, in Lazio and 18.86 percent and 19.11 percent, respectively, in the other regions. The proportion of AMI patients treated with PCI within 48 hours since 2003 exhibited a similar increasing trend in Lazio and in the other regions until 2007; subsequently, the improvement in Lazio was greater than that in the other regions (Figure 1).

Table 2.

Change in Hospital Performance Indicators in Lazio and in Other Italian Regions in 2006/2007 and 2008/2009

	2006–2007			2008–2009			Change between 2008–09 and 2006–07
	Crude Proportion	Risk-Adjusted Proportion*	Total N.	Crude Proportion	Risk-Adjusted Proportion*	Total N.	RR	p-value	Absolute Difference
AMI patients with PCI performed within 48 hours
Lazio	22.97	22.49	18,877	29.34	29.43	17,190	1.31	<0.001	6.94
Other regions	22.79	22.48	178,059	26.71	27.09	166,927	1.21	<0.001	4.61
Hip fracture patients operated within 48 hours
Lazio	11.76	11.73	12,585	15.77	15.78	12,469	1.34	<0.001	4.05
Other regions	29.32	29.36	113,436	28.61	28.57	112,222	0.97	<0.001	−0.79
Primary cesarean deliveries
Lazio	33.61	34.58	92,218	33.80	35.30	87,744	1.02	0.012	0.72
Other regions	29.99	30.49	796,223	28.93	28.11	760,785	0.92	<0.001	−2.38

^*Proportion of PCI adjusted for age, gender, vascular diseases, previous AMI, other coronary heart disease, chronic cerebrovascular diseases, chronic kidney diseases, previous PCI, cardiac dysrhythmias, heart failure, malignant neoplasms, chronic obstructive pulmonary disease, diabetes, previous CABG, obesity, blood disorders, other cardiac diseases, and other chronic diseases; Proportion of hip fracture surgeries adjusted for age, gender, hypertension, other coronary heart disease, chronic cerebrovascular diseases, chronic kidney diseases, cardiac dysrhythmias,chronic obstructive pulmonary disease, diabetes, dementia including Alzheimer's disease, other cardiac disorders, other chronic diseases, rheumatoid arthritis, and osteoporosis; Proportion of cesarian deliveries adjusted for age, premature rupture of membranes, fetal distress, disorders of amniotic fluid, disproportionate fetal growth, post-term delivery, multiple gestation, intrauterine growth retardation, and other maternal and fetal diseases.

AMI, acute myocardial infarction; CABG, coronary artery bypass graft; PCI, percutaneous coronary intervention.

Figure 1.

Trends in the Proportion of Patients with Acute Myocardial Infarction Treated with Percutaneous Coronary Intervention within 48 Hours in Lazio and Other Italian Regions

The proportion of hip fracture patients operated on within 48 hours in Lazio was lower than in the other regions in July 2006 and in September 2008 (Table 2). The risk-adjusted proportion of hip fractures operated on within 48 hours in Lazio changed from 11.73 percent to 15.78 percent before and after P.Re.Val.E. reporting, respectively, corresponding to an absolute increase of 4.05 percent and a relative increase of 34 percent (p < .001). The increase in Lazio was significantly greater compared to the change observed in the other regions (p < .001), where the risk-adjusted proportion remained similar in July 2006 and September 2008 (29.36 and 28.57 percent). As shown in Figure 2 the proportions of hip fracture patients operated on within 48 hours did not show any relevant change until 2006; subsequently Lazio showed a clear improving trend, whereas the other regions remained stable.

Figure 2.

Trends in the Proportion of Hip Fracture Patients Operated within 48 Hours in Lazio and Other Italian Regions

The proportion of primary cesarean deliveries in July 2006 in Lazio was somewhat higher than that in the other Italian regions. It did not show any relevant change before and after P.Re.Val.E reporting (34.57 and 35.30 percent, respectively). In the other regions the risk-adjusted proportion decreased slightly from 30.49 to 28.11 percent during the same period, corresponding to an absolute decrease of 2.38 percent. Thus, the other regions showed a significant, albeit small, reduction of cesarean deliveries compared to Lazio (p < .001). Figure 3 also shows that the proportion of cesarean deliveries did not decrease in Lazio, and it decreased only slightly in other regions.

Figure 3.

Trends in the Proportion of Primary Cesarean Deliveries in Lazio and Other Italian Regions

Discussion

Our study has revealed a mixed picture regarding changes in hospital performance indicators during July 2006 and September 2008, as well as a possible association with P.Re.Val.E. reporting. The increase in the proportion of hip fracture patients operated on within 48 hours in Lazio, compared with no relevant change in the other regions, suggests that reporting of the P.Re.Val.E. results may be associated with an improvement in orthopedic hospital care. Improvement in Lazio was observed by 2007, possibly related to the “Hawthorne effect” (Tu et al. 2009); hospitals may have already introduced quality improvement initiatives when they became aware of the P.Re.Val.E. program and before the communication of performance results. The proportion of timely hip fracture operations in Lazio was very low, remaining low even in September 2008. Even though the other regions reported higher proportions in July 2006, a relatively smaller margin for improvement does not justify the lack of any further increase, considering that only approximately one in four older patients with hip fractures received timely treatment. Hip fracture patients should be operated on within 48 hours of admission (Mattke et al. 2006; Novack et al. 2007; Shiga, Wajima, and Ohe 2008), since earlier surgery is associated with lower risk of death and lower rates of postoperative pneumonia and pressure sores (Simunovic et al. 2010).

Other factors may also have positively affected the treatment of hip fractures in Lazio, including a heightened awareness of orthopedic surgeons following relevant guideline release, scientific publications, and conferences. However, these factors are unlikely to only have influenced surgeons in Lazio. Publication of clinical pathways for hip fracture treatment occurred in Lazio in September 2009, potentially exerting an effect on orthopedic care only for the last few months of the study period. No specific quality improvement and public reporting initiatives were introduced in the comparison group as a whole during the study period. Some initiatives might have taken place in specific hospitals of some regions. This might have influenced our results, in terms of a possible dilution of the effect of the P.Re.Val.E. program.

The proportion of AMI patients treated with PCI within 48 hours improved in Lazio after the P.Re.Val.E. reporting, but some improvement occurred also in the other regions. Overall the proportion of PCI within 48 hours remained relatively low, considering that the recommended time frame for PCI is 90 minutes door-to-balloon (Tu et al. 2008) or as soon as possible, as any delay in reperfusion is associated with a continuous, nonlinear increase in mortality (Van de Werf et al. 2008; Kushner et al. 2009). These recommendations, however, are specific to patients with ST-elevation myocardial infarction (STEMI), and their relevance to non-STEMI patients is less clear. Our data did not distinguish between STEMI and non-STEMI patients, possibly partially explaining the relatively low proportion of patients treated with PCI within 48 hours. We lacked information on pharmacological interventions, and it is possible that acute reperfusion was performed with timely fibrinolysis in a substantial proportion of our study population. However, a multinational study including 54 hospitals in 12 countries reported a clear decreasing trend in the use of fibrinolytic therapy between 1999 and 2006 (from 41 to 16 percent) and an increase in PCI (from 15 to 44 percent) (Eagle et al. 2008). Thus, it is unlikely that fibrinolysis increased in our study population. Our findings of a relatively low proportion of timely PCI is in line with international data showing that, overall, 33 percent of potentially eligible patients did not receive reperfusion therapy (Balzi et al. 2008; De Luca et al. 2008; Eagle et al. 2008) and over 40 percent of reperfused patients received it outside the recommended time window (Eagle et al. 2008). The IN-ACT study involving 44 Italian hospitals between 2005 and 2007 showed that 81.1 percent of STEMI patients and 46.4 percent of non-STEMI patients were reperfused either with thrombolysis or with PCI (Seccareccia et al. 2008).

The increase in the proportion of PCI in September 2008 versus July 2006 in our study may have been influenced by initiatives in various Italian regions over the last decade (Seccareccia et al. 2008), including guideline publication and re-organization of hospital services for the treatment of AMI patients. Initiatives varied by health care trust and regarded only some regions and not the comparison group as a whole. We cannot exclude that this might have influenced the study results, with a possible dilution of the effect of the P.Re.Val.E. program.

Our findings regarding the proportion of primary cesarean deliveries were not encouraging, as there was no improvement in Lazio and only a very limited improvement in other regions. The cesarean delivery rate should not exceed 15 or 20 percent according to the World Health Organization (1985) WHO, and the Italian Ministry of Health (Ministry of Health 2011), respectively. Higher rates are considered medically unjustified and may be associated with maternal and neonatal health risks (Belizán, Althabe, and Cafferata 2007). Nevertheless, there has been a relentless increase in cesarean deliveries in many medium- and high-income countries (Villar et al. 2006). Lazio experienced a cesarean delivery rate of 24.3 percent in 1987, the highest rate in Europe at that time (Bertollini et al. 1992), and reached 32.5 percent in 1990–1996 (Cesaroni, Forastiere, and Perucci 2008). Despite repeated concerns expressed by health authorities and by the scientific community, and reporting of P.Re.Val.E data in Lazio, our findings demonstrate that no progress has been made in reducing unnecessary cesarean births in Italy. Targeted and complex interventions are likely necessary to simultaneously address multiple factors and involve all participants in maternity care (Chaillet et al. 2006; Pronovost, Berenholtz, and Morlock 2011).

Overall, previous studies evaluating the effect of public reporting of performance data have been inconclusive (Baker et al. 2002), with the majority of studies focusing on cardiac care (Fung et al. 2008; Jung 2010). Quality improvement after public reporting is often limited to a subset of indicators and may depend on the type of services examined (Werner et al. 2009; Jung 2010). A recent population-based randomized trial revealed no significant improvement in composite process-of-care indicators for AMI after public release of hospital-specific quality indicators (Tu et al. 2009). Only 1 of 12 individual AMI process-of-care indicators improved significantly. Other observational studies on public release of mortality data for cardiac surgery in American hospitals showed improved outcomes (Hannan et al. 1994; Chassin 2002).

The relatively weak impact of P.Re.Val.E. reporting on quality improvement may have several explanations, including a limited time span between reporting of performance and measurement of potential changes. Preliminary data for 2010 show further improvements in Lazio, with 20 percent of hip fracture patients being operated on within 48 hours, possibly linked to the introduction of a pay for performance program described elsewhere (Pinnarelli et al. 2012).

The effect of public reporting might also have been diluted, because hospitals may have introduced quality improvement initiatives when they became aware of the P.Re.Val.E. program, before the dissemination of performance results at the end of 2007.

Moreover, incentives and pressure perceived by health professionals and managers for the need to undertake quality improvement initiatives may have been too weak. Greater involvement of an interdisciplinary team of experts, including health services researchers, clinicians, social scientists, and management experts, is probably necessary to improve teamwork and more effectively translate evidence into practice (Pronovost, Berenholtz, and Morlock 2011).

The introduction in Lazio of a performance evaluation program and the public release of hospital-specific information generated a lively debate among health professionals. After initial skepticism regarding the reliability of the indicators, the majority of professionals accepted the program and actively participated with constructive suggestions for improving and interpreting unexpected results.

The general public was informed about the project; however, the program was not specifically designed for a non-expert audience. Some newspapers published performance results and the involved hospitals feared that this would affect their reputation and patient volumes. However, in line with previous studies (Romano and Zhou, 2004), we did not observe relevant changes in patient volumes of hospitals with extremely bad or good performance (data not shown). We cannot say if this was due to a limited number of patients having heard about the program, or if few relied on it for making health decisions or other reasons. Future studies should investigate these aspects with specific surveys.

To establish whether overall cardiac and orthopedic hospital care improved, also other quality indicators should be considered. Our study included only a limited set of indicators; however, they address important clinical areas, in terms of frequency and health impact, and their validity is supported by vast international literature (Villar et al. 2006; Belizán, Althabe, and Cafferata 2007; Van de Werf et al. 2008; Simunovic et al. 2010). Furthermore, we achieved stable estimates thanks to the inclusion of large cohorts. The analyses of more indicators for each clinical domain could provide a more detailed picture. However, this goes beyond the purpose of our study that aimed at examining how hospital performance in different clinical areas changed after public reporting, compared to all Italian regions where no public reporting has taken place. Detailed information for a large number of indicators, including relevant risk-adjustment variables, is often not available at national or state level. In fact, only a limited number of studies, mainly based in North America and in a few Northern European countries, have provided such data (Groene, Skau, and Frolich 2008).

Our study specifically provided some new insights regarding the possible role that public reporting may play in different clinical areas. The observed improvements in cardiac and orthopedic performance may be linked to the perception that these indicators can be related to technical and professional skills. Motivating forces for quality improvement may act through providers' professional pride and competitiveness among peers. On the other hand, high cesarean section rates did not decrease in our study, probably obstetricians and the general public do not perceive this as indicative of unsatisfactory technical and professional skills. In this case other forces (for example, patients' requests and providers' economic/organizational interests) come into play. Complex interventions are probably necessary to effectively translate evidence into practice.

Potential Limitations

Italy has a publicly funded universal health care system similar to other European countries (Serumaga et al. 2011). Caution is needed in generalizing our findings to settings with a different organization of health care.

Even though the possibility of gaming of the data on PCI and hip fracture treatment in response to the performance evaluation cannot be excluded, previous studies did not find evidence of gaming (Serumaga et al. 2011). Some studies have reported that changes in data accuracy may partially explain quality improvement (Werner et al. 2009). However, we did not find relevant changes in comorbidities in our study population over the years. The prevalence of certain comorbidities was relatively low in our study population, indicating underreporting in the administrative database (Fortuna et al. 2006). However, underreporting was non-differential in the years and regions included in our analysis. A previous Italian study (Fortuna et al. 2006) demonstrated that an administrative database provided similar league tables as a more complex specialized database. Regarding the increase in timely PCI, we cannot exclude a “risk-avoidance creep,” with hospitals selecting lower risk patients and avoiding higher risk ones. However, this potential problem is more relevant if outcomes such as mortality are evaluated.

We could only examine unadjusted trends before 2006, because information on covariates was not available for all Italian regions. However, analyses for the years 2007 and beyond have shown that crude and adjusted estimates were very similar, even after controlling for numerous risk factors. We also found a fairly homogeneous case mix for the examined years and populations, and we expect that demographic and clinical characteristics were stable also during the years 2003–2006. We cannot exclude residual confounding, which is a typical limitation of observational studies, but ideally to deal with this issue a randomized trial would be necessary.

Conclusions

We have shown that reporting of performance data may have a positive but limited impact on quality improvement. To achieve a greater effect, integrated interventions are likely necessary. Depending on the specific clinical area, educational interventions, pay for performance, or other strong incentives may be needed in addition to performance evaluation.

Evaluation of risk-adjusted quality indicators remains paramount for public accountability of health care services.

SUPPORTING INFORMATION

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Table S1: Proportion of Patients with Acute Myocardial Infarction (AMI) Treated with Percutaneous Coronary Intervention (PCI) within 48 Hours, Predictive Model.

Table S2: Proportion of Older Patients (Aged ≥65 Years) with Hip Fracture Operated on within 48 Hours of Hospital Admission, Predictive Model.

Table S3: Proportion of Women with Primary Cesarean, Predictive Model.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.