AHRQ prevention quality indicators to assess the quality of primary care of local providers: a pilot study from Italy

Abstract

Background: Outside the USA, Agency for Healthcare Research and Quality (AHRQ) prevention quality indicators (PQIs) have been used to compare the quality of primary care services only at a national or regional level. However, in several national health systems, primary care is not directly managed by the regions but is in charge of smaller territorial entities. We evaluated whether PQIs might be used to compare the performance of local providers such as Italian local health authorities (LHAs) and health districts. Methods: We analysed the hospital discharge abstracts of 44 LHAs (and 11 health districts) of five Italian regions (including ≈18 million residents) in 2008–10. Age-standardized PQI rates were computed following AHRQ specifications. Potential predictors were investigated using multilevel modelling. Results: We analysed 11 470 722 hospitalizations. The overall rates of preventable hospitalizations (composite PQI 90) were 1012, 889 and 988 (×100 000 inhabitants) in 2008, 2009 and 2010, respectively. Composite PQIs were able to differentiate LHAs and health districts and showed small variation in the performance ranking over years. Conclusion: Although further research is required, our findings support the use of composite PQIs to evaluate the performance of relatively small primary health care providers (50 000–60 000 enrollees) in countries with universal health care coverage. Achieving high precision may be crucial for a structured quality assessment system to align hospitalization rate indicators with measures of other contexts of care (cost, clinical management, satisfaction/experience) that are typically computed at a local level.

Introduction

The assessment of the quality of health care has been a problematic issue for decades and in all settings,^1,2 including primary care.^3–5 Starting from the 1970s,⁶ a number of quality indicators for primary health care has been proposed,⁷ but their validity has often remained questionable.^3,8,9 An important progress was achieved in the early 1990s with the concept of ‘potentially avoidable’ or ‘ambulatory care sensitive’ hospitalizations,¹⁰ which were extensively used to indirectly evaluate the performance of primary health care in several countries.^11–16

Based on the assumption that the hospitalization for several chronic and acute conditions can be prevented with timely and appropriate ambulatory care,¹⁷ the Agency for Healthcare Research and Quality (AHRQ) developed a set of 14 primary care quality measures, named ‘prevention quality indicators’ (PQIs), which were first released in 2000¹,⁸ and which have been validated in the USA to compare the performance of national^19–28 and local (counties)^29–31 providers.

Some of the PQIs were also used in Italy to compare the quality of primary health care across the regions.^32–35 However, in the Italian National Health System (NHS), primary care services are not directly managed by the regions, but they are in charge of both local health authorities (LHAs) and smaller entities within each LHA named health districts (counting ∼50 000–60 000 inhabitants on average).³⁶ Thus, regional-level comparison may provide limited information.

The present analysis of the hospital admissions of five Italian regions for the years 2008–10 was aimed at evaluating for the first time whether PQIs may be used to compare the quality of primary health care of smaller territorial entities such as LHA and health districts within the Italian NHS.

Methods

The list of all PQIs is reported in box 1. We computed all composite and 13 of the 14 individual PQIs: PQI 9 (low birth weight rate) was excluded because we only obtained data of the admissions of people aged ≥18 years. To compute PQIs, we followed strictly AHRQ technical specifications,³⁷ and all details have been reported at the end of the Supplementary Appendix.

Box 1 List of the PQI measures, both individual and composite.

Individual measures:

• PQI 1—Diabetes short-term complications;

• PQI 2—Perforated appendix;

• PQI 3—Diabetes long-term complications;

• PQI 5—Chronic obstructive pulmonary disease or Asthma in older adults;

• PQI 7—Hypertension;

• PQI 8—Congestive heart failure;

• PQI 10—Dehydration;

• PQI 11—Bacterial pneumonia;

• PQI 12—Urinary tract infection;

• PQI 13—Angina without procedure;

• PQI 14—Uncontrolled diabetes;

• PQI 15—Asthma in younger adults;

• PQI 16—Rate of lower-extremity amputation among diabetics.

Composite measures:

• PQI 90—Overall PQI composite (PQIs 1, 3, 5, 7, 8, 10, 11, 12, 15, 16)

• PQI 91—Acute PQI composite (PQIs 10, 11, 12)

• PQI 92—Chronic PQI composite (PQIs 1, 3, 5, 7, 8, 15, 16)

• PQI ‘Trinity’ (PQIs 5, 8, 11)

In brief, with the exception of PQI 2, which is a percentage, each PQI is a ratio between the number of hospital admissions for specific disease, and the total number of residents, aged ≥18 years, in a metro area or county.¹⁷ For our analyses, each LHA (the Italian ASL) was a separate metro area, and every PQI was computed for each LHA. Thus, each PQI was the rate of admission for a specific disease of all the residents of a single LHA (×100 000).

Hospital discharge abstracts were officially provided by the General Directorate for Health Planning of the Italian Ministry of Health, which granted the permission to extract the data of the 44 LHA of four Italian regions (Abruzzo, Emilia-Romagna, Lazio and Lombardia) and the city of Trieste, for the triennium 2008–10.

The number of residents of each LHA, with age and gender strata, was extracted both from publicly available data sets of the National Institute of Statistics (Istat)³⁸ and from other demographic data sets provided by the Italian Ministry of Health. For each PQI, once the crude admission rates were obtained for each PQI, they were age-standardized using an indirect method and 10-year age classes. We also computed the standardized admission rates for the three composite PQI measures PQI 90, PQI 91 and PQI 92.³⁹ PQI 90 represents the overall measure of quality, and it is the sum of the numerators of all the individual indicators (excluding PQIs 2, 13 and 14). PQI 91 and PQI 92 are the sum of the admission rates for acute and chronic conditions, respectively.

We computed the PQIs, as described above, also for each health district (as a metro area) to evaluate whether at least some of the indicators had enough statistical power to be used for comparing smaller territorial entities. Unfortunately, we were able to retrieve the information on patient's health district only for a few LHAs of the Abruzzo region (n = 7) and the city of Trieste (n = 4) because Italian hospital discharge abstracts do not contain such information, which must be obtained, with permission, from single LHAs.

The computation of age- and sex-standardized PQIs by LHA was demanding: for each LHA (n = 44), in each year (n = 3), demographic and admission data need to be extracted and tabulated in several strata (two genders and seven 10-year age classes) for each PQI (n = 16). Thus, the overall number of cells to deal with is 29 568 (44*3*16*2*7 cells). This is even more complex when the unit of the analysis is the health district because there are several districts in each LHA. In our analysis, we initially observed no influence of gender on the estimates of a pilot analysis on five LHAs from all regions, and thus we did not standardize by gender.

As a secondary analysis, we investigated to what extent composite PQIs were correlated with the overall admission rates (once excluded Ambulatory Care Sensitive Conditions (ACSCs)), that is, whether PQIs did represent a meaningful measure or we could simply use the overall admission rate as a proxy of the efficacy/efficiency of the primary health care services. Besides an initial univariate analysis based on Spearman correlation coefficient, we fitted three random-effects linear regression models, in which the dependent variables were the three composite PQIs (PQI 90, 91 and 92) in the triennium 2008–10. For all models, the cluster units were single LHAs (health districts were too few to allow meaningful multivariate analyses) and the independent variables were overall LHA hospitalization rate (excluding ACSCs), LHA total population, year and region. Because of the hierarchical structure of our data, in which regions, LHA and years represent different levels with potential intra-cluster correlations,⁴⁰ the analysis was repeated using a random-effects multilevel model with three levels of cluster (year, region and LHA). Because the results for the remaining covariates (hospitalization rate and total population) were similar, only the results of the simpler regression model were shown to avoid redundancy.

All analyses were carried out using Stata software, version 11 (Stata Corporation, College Station, Texas, USA, 2007). The study protocol was approved by the ethics committee of the Sapienza University of Rome (prot. n. 657/12, 19 July 2012).

Results

Study sample and comparison among regions and the USA

We applied the algorithm to compute PQIs to all the 11 470 722 hospitalizations that took place in the five selected regions (including ∼18 million residents), between 2008 and 2010 (Supplementary table S1). Overall, the prevalence of admissions for ACSC according to AHRQ criteria¹⁷ was 5.7% (n = 648 180), ranging from 4.9 to 9.7% across regions and years.

The hospitalization rates for composite PQIs have been reported in table 1 (overall), Supplementary table S2 (stratified by region), figure 1 and Supplementary figures S1 and S2 (stratified by LHA) and figure 2 and Supplementary figures S3 and S4 (stratified by district). The overall hospitalization rates for ACSC (composite PQI 90) were consistently different across regions during the study period, with Abruzzo and Trieste showing rates that were 15–25% higher, on average, than those from Lazio and Lombardy (Supplementary table S2).

Table 1.

PQIs: hospital admission rates in Italy (2008–10) and the USA (2008–09)

Indicators (admission rates × 100 000)	Italya			Δ %	USA	Comparison USA vs. Italy, %	USA	Comparison USA vs. Italy, %
	2008	2009	2010	2008–10	2008	2008	2009	2009
PQI 1—Diabetes short-term complications	12	10	10	−16.7	62	+416.7	62	+520.0
PQI 2—Perforated appendix (%)	29	30	31	+8.1	28	−3.4	29	−3.3
PQI 3—Diabetes long-term complications	76	57	69	−9.2	129	+69.7	118	+107.0
PQI 5—Chronic obstructive pulmonary disease or asthma in older adults	188	159	154	−18.1	578	+207.4	559	+251.6
PQI 7—Hypertension	36	39	46	+27.8	62	+72.2	63	+61.5
PQI 8—Congestive heart failure	336	301	399	+18.8	400	+19.0	381	+26.6
PQI 10—Dehydration	27	23	28	+3.7	176	+551.9	139	+504.3
PQI 11—Bacterial pneumonia	181	183	205	+13.3	362	+100.0	336	+83.6
PQI 12—Urinary tract infection	59	59	65	+10.2	206	+249.2	197	+233.9
PQI 13—Angina without procedures	114	98	104	−8.8	25	−78.1	23	−76.5
PQI 14—Uncontrolled diabetes	51	41	45	−11.8	23	−54.9	22	−46.3
PQI 15—Asthma in younger adults	48	30	41	−14.6	60	+25.0	63	+110.0
PQI 16—Rate of lower-extremity amputation among diabetics	7	7	7	0.0	18	+157.1	17	+142.9
PQI 90—Overall PQI composite	1012	889	988	−2.4	1825	+80.3	1714	+92.8
PQI 91—Acute PQI composite	250	268	282	+12.2	744	+197.6	672	+150.7
PQI 92—Chronic PQI composite	761	640	705	−7.4	1081	+42.0	1042	+62.8

a: Abruzzo, Lazio, Emilia-Romagna, Lombardy and the city of Trieste.

Figure 1.

PQIs: PQI 90—overall hospitalization rates for ambulatory care sensitive conditions in 44 LHAs from five Italian regions (Abruzzo: LHAs 201–204; Emilia-Romagna: E101–E113; Lazio: 101–112; Lombardy: 301–315; Trieste: T01); years 2008–10. Data of the local health units 101–105 of the Lazio region (City of Rome) for the year 2009 were not available

Figure 2.

PQIs: PQI 90—overall hospitalization rates for ambulatory care sensitive conditions in 11 health districts from three Italian LHAs (LHA of Pescara: Pe01–Pe06; LHA of Lanciano-Vasto-Chieti: district of Francavilla; LHA of Trieste: Tr01-Tr04); years 2008–10

From 2008 to 2010, the overall hospitalization rates for acute (PQI 91) and chronic (PQI 92) conditions showed opposite trends: in all areas, PQI 91 increased (+12.2% overall; ranging from +3.9% in Trieste—in which the rate was already high—to +15.5% in Lazio) while PQI 92 decreased (−7.4% overall; ranging from −4.3% in Emilia-Romagna to −16.5% in Trieste—Supplementary table s2).

When compared with the USA (table 1), the rates of PQIs of the included Italian regions were generally lower or extremely lower, with the exception of the admissions for angina without procedures (PQI 13) and uncontrolled diabetes (PQI 14), which were higher than in the USA. In the years 2008 and 2009 (last available for the USA), the overall rate of potentially preventable hospitalizations (composite PQI 90) of the USA was 80.3 and 92.8% higher than Italian regions, respectively (table 1).

Use of PQIs to compare local-level providers

The overall hospitalization rates for ACSC (composite PQI 90) for each of the 44 included LHAs, in each year, have been reported in figure 1. LHAs showed relevant (and significant) differences among each other: the rate between the highest and lowest LHA value each year ranged from 2.20 to 2.33. A similar scenario was observed for PQIs 91 (acute conditions only) and 92 (chronic conditions only)—Supplementary figures S1 and S2, respectively. Somewhat smaller differences in PQI 90 values were observed across health districts (figure 2), although the ‘best/worst’ admission rate never fell below 1.97 in the triennium. Again, a comparable picture was obtained when analysing PQIs 91 and 92 (Supplementary figures S3 and S4, respectively).

With regard to stability over years, which would be presumable given the complex multidisciplinary nature of PQIs and the time needed to observe the effects of interventions on health services, most of the LHAs and health districts showed little variation in composite PQIs’ values over the triennium (figures 1 and 2 and Supplementary figures S1–S4). Moreover, the rank of most LHAs (69.8–76.7%) or districts (83.8–100%) showed small variation across the 3 years (Supplementary table S3), indicating that the ranking of the performance of LHAs or health districts based on composite PQIs is generally stable.

Adjusting for year, total LHA population and region, all composite PQIs were significantly and positively associated with the rate of all admissions excluding ACSCs (table 2). In specific, the global hospitalization rate for ACSCs (PQI 90) increased by 0.27 admissions × 100.000 inhabitants every 10 general hospitalizations more, suggesting that the two rates are correlated but PQIs are also appraising another construct in addition to health care use.

Table 2.

Potential predictors of the rate of hospitalization for an ambulatory care sensitive condition (PQIs 90, 91 and 92) in the years 2008–10

Variables	PQI 90 (overall)		PQI 91 (acute)		PQI 92 (chronic)
	Coeff.	(P)	Coeff.	(P)	Coeff.	(P)
Rate of all hospitalizations excluding PQI conditions (10 × 1000), 10-unit increase	0.27	(0.020)	0.16	(0.036)	0.15	(0.029)
Total population, 1000-resident increase	−0.10	(0.2)	−0.08	(0.051)	−0.02	(0.8)
Year, 1-year increase	10.1	(0.3)	27.3	(0.001)	−15.0	(0.011)

Coeff. = regression coefficient.

Random-effects linear regression with one level of cluster (LHA) and 124 observations. The analysis was repeated using a random-effects multilevel model with three levels of cluster (year, region and LHA), with similar results for the remaining covariates (hospitalization rate and total population).

The unit of analysis was the LHA.

With purely descriptive purposes, the rates of all individual PQIs of each of the 44 included LHAs and each of the 11 included health districts, in each year, have been reported in the Supplementary tables S4 and S5, respectively.

Discussion

With this reanalysis of >11 million hospital admissions in 3 years, we evaluated whether PQIs might be used among the measures to compare the health care quality of local-level providers within a European country characterized by universal coverage such as Italy, in which primary health care management is in charge of LHAs and health districts. The results of the present study support the use of the indicators proposed by AHRQ: composite PQIs showed a good discriminative ability among LHAs and among districts, a high level of stability during the triennium, and although they were significantly correlated with the overall admission rates, there was evidence that the rates of preventable hospitalization are not simply a reflection of the general hospitalization rates. Interestingly, while the admission rates for preventable conditions declined by 2.4% during the study period (−7.4% among chronic conditions and +12.2% among acute conditions), the rates for non-preventable conditions (all the others⁴¹) declined by 7.3%, which does not seem to indicate an impact on primary health care—at least in the short-term and especially for acute conditions—of the concomitant in-depth Italian health care system restructuring⁴² (which were only recently focused on primary care organization⁴³).

PQIs have already been validated to assess primary health care quality in several studies carried out in the USA^19–29,44 and other countries,⁴⁵ including Italy.^32–35 The comparisons, however, were carried out mostly at a national^19–25,44 or regional level^34,35 or to identify potential predictors of poorer access to primary health care.^{26–28,32,33} Some technical AHRQ reports reported data at a county level, but no attempt at assessing the validity of PQIs was made.^30,31 In Italy, an agency of the Minister of Health (Age.Na.s) computed PQIs for each province, but LHAs do not always correspond to provinces, and data are not publicly available.⁴⁶ In fact, to date, only one study assessed whether PQIs were appropriate to compare small geographical areas, particularly counties from Kentucky in the years 2006–08.²⁹ The results of such analysis were in line with our findings: PQIs showed significant and clinically important variations across counties, but each indicator had stable rates over time within a county. The authors concluded that PQIs are potentially suitable for the implementation of a structured system for primary health care quality assessment. Having the chance to compare the performance of small territorial entities might be crucial to establish a meaningful (and acceptable) system for the evaluation of the quality of primary care. In several countries including Italy, the organizational (and often even the strategic) management of primary care services is not set only at a regional level, and most responsibility is given to local health care providers. Thus, the detection of poor performances and the development of interventions aimed at improving the quality of primary care should necessarily involve also local-level providers.

The construct validity of PQIs can be reasonably assumed: besides AHRQ,¹⁷ the relationship between ACSCs and other primary health care quality was also supported by a recent systematic review including 51 studies, in which the authors concluded that ‘most studies confirmed the expected relationship between indicators of primary health care and hospitalization for ACSCs, showing lower hospitalization rates for ACSCs in areas with greater access to primary health care’.⁴⁷ If further research will confirm PQIs face and predictive validity, the next step might be the addition of PQIs to enhance the content validity of the sets of indicators that are currently used to assess the performance of primary care practices.^48,49

The major strengths of our analysis include the relatively large data set, the inclusion of several regions throughout Italy, the use of a 3-year time span and the adoption of exactly the same AHRQ criteria for the computation of PQIs (thus increasing comparability). However, this study has some limitations that must be considered in interpreting the results. First, as most studies on PQIs, we did not consider several factors that are beyond the control of family physicians (such as disease prevalence, income level, health care access), and that may explain part of the differences in potentially avoidable admission rates existing between small areas.¹⁴ Notably, however, access to care in Italian NHS is granted for all with limited or no additional expense. Also, unfortunately in Italy most of the epidemiological estimates are based on hospital discharge abstracts; thus, it would have been of limited utility to adjust comparisons of hospitalization rates using other (likely related) hospitalization rates. A second limitation derives from our ability to compute PQI admission rates for a small proportion of health districts only, since data were retrieved from 3 LHAs out of 44 involved in the study (because of privacy barriers and data collection inefficiencies). Thus, further studies including a higher number of health districts are strongly recommended and caution is required to interpret the findings on health districts. Third, the coding of admissions abstract could differ among LHA units and this may induce some bias.^50,51 Fourth, it must be acknowledged that substantive structural differences exist between Italian and the US health care systems,³⁶ and the US PQI rates have been shown for merely descriptive purposes: a structured comparison between the countries would require extra data and analyses. Finally, we only assessed the relatively few diseases that were included in PQIs, and it will certainly be useful in the future to add some conditions to the analysis, especially those that have been examined and validated in previous studies on the Italian setting.^51,52

Italy, as well as several other countries, still misses a structured system to evaluate primary health care quality. For these quality assessment systems, it is crucial to achieve a high level of precision, to discriminate the performance of the entities that are responsible for the performance itself, and to align hospitalization rate indicators with measures of other contexts of care (cost, clinical management, satisfaction/experience⁴⁸) that are typically computed at a local level. The results of the present study support the use of composite PQIs, in nations with universal health care coverage, as a valid set of indicators to evaluate the quality of relatively small primary health care providers (up to ∼50 000 enrollees).

Funding

Italian Ministry of Health (prot. DGPROG.I/F.3.a.b, January 20 2012).

Conflicts of interest: None declared.

Key points.

• Outside the USA, AHRQ Prevention Quality Indicators (PQIs) have never been used to compare the quality of primary care services at a local level; however, in several national health systems, primary care is not directly managed by the regions, but is in charge of smaller territorial entities.

• For a primary care quality assessment system, it is crucial to achieve a high level of precision, to discriminate the performance of the entities that are responsible for the performance itself, and to align hospitalization rate indicators with measures of other contexts of care (cost, clinical management, satisfaction/experience) that are typically computed at a local level.

• Composite PQIs were able to differentiate local-level providers such as Italian local health authorities or health districts, which showed small variation in the performance ranking across the years, as it might be expected given that time is needed to observe the effects of changes on health services.

• In nations with universal health care coverage, composite PQIs appeared a valid set of indicators to evaluate the performance of relatively small primary health care providers.