Relationships between technical efficiency and the quality and costs of health care in Italy

Abstract

Objectives

This paper reports the measurement of technical efficiency of Tuscan Local Health Authorities and its relationship with quality and appropriateness of care.

Design

First, a bias-corrected measure of technical efficiency was developed using the bootstrap technique applied to data envelopment analysis. Then, correlation analysis was used to investigate the relationships among technical efficiency, quality and appropriateness of care.

Setting and Participants

These analyses have been applied to the Local Health Authorities of Tuscany Region (Italy), which provide not only hospital inpatient services, but also prevention and primary care. All top managers of Tuscan Local Health Authorities were involved in selection of the inputs and outputs for calculating technical efficiency.

Main Outcome Measures

The main measures used in this study are volume, quality and appropriateness indicators monitored by the multidimensional performance evaluation system developed in the Tuscany Region.

Results

On average, Tuscan Local Health Authorities experienced 14(%) of bias-corrected inefficiency in 2007. Correlation analyses showed a significant negative correlation between per capita costs and overall performance. No correlation was found in 2007 between technical efficiency and overall performance or between technical efficiency and per capita costs.

Conclusions

Technical efficiency cannot be considered as an extensive measure of healthcare performance, but evidence shows that Tuscan Local Health Authorities have room for improvement in productivity levels. Indeed, correlation findings suggest that, to pursue financial sustainability, Local Health Authorities mainly have to improve their performance in terms of quality and appropriateness.

Introduction

Costs of the healthcare sector in most developed countries have increased substantially during the last decades, highlighting the need for measurement of performance and management of efficiency. As the present period of economic recession continues to impose budget constraints, health systems are asked to reduce the level of expenditures and, at the same time, improve both the appropriateness and quality of services, in order to achieve their mission [1].

This paper describes how an efficiency analysis approach has been combined with quality and appropriateness measures already being monitored by the performance evaluation system developed for Local Health Authorities in Tuscany (Italy).

Literature on performance measurement systems and efficiency analysis has grown in the last decades. On the one hand, during the last 20 years, several performance evaluation systems have been developed, proposing models such as those designed by Kaplan [2, 3], and applied to the healthcare sector [4–6]. On the other hand, the efficiency analysis literature has grown in recent years, especially regarding the measurement of efficiency in healthcare institutions around the world. Recent surveys include Hollingsworth [7] and Worthington [8]. In particular, Worthington [8] notes that only 5% of the studies identified are based on teaching hospitals or on Local Health Authorities. Moreover, the reported literature has been largely concerned with the USA, UK and Northern European institutions, while only a few empirical studies have analysed the Italian Healthcare system, and most of them have focused on hospitals [9, 10]. None of the studies in the literature have analysed Local Health Authorities integrating technical efficiency with other managerial tools used by regional health policy-makers. Finally, there is a lack of literature concerning the relationship between balanced scorecard systems and technical efficiency methods. One exception is Banker [11], who analysed the telecommunication industry in the USA.

This paper is based on an empirical study carried out in the Tuscan healthcare system on 2007 data (The Tuscany Region, with 3.6 million inhabitants, spends about 6.1 million Euros on public health care. That accounted for more than 70% of its global regional expenditure in 2007. The regional government works through a network of 12 Local Health Authorities. They are responsible for providing services to their population regarding prevention, primary care and paediatrics, diagnostic, outpatient and hospital services.).

In order to achieve goals of quality, appropriateness, equity and effectiveness, the Tuscany Region has been using a multidimensional system to monitor and assess the Local Health Authorities’ performance since 2005 (for principles, methods and applications, see Nuti [12, 13]). Through using these tool, the Tuscany Region wants to achieve financial sustainability, maximize productivity and provide the best care for its citizens.

The main research questions addressed in this paper are:

1. What efficiency scores can Tuscan Local Health Authorities obtain by using data envelopment analysis?

2. What are the relationships between technical efficiency scores and cost per capita? What are the relationships between technical efficiency scores and the other indicators monitored by the comprehensive Tuscan performance evaluation system, including quality and appropriateness?

Methods

Overview of approach

Two different methods have been used in order to answer the two research questions. First, a non-parametric approach (data envelopment analysis) was used to calculate the technical efficiency scores for the Tuscan Local Health Authorities. Correlation analysis was then used to investigate the relationship between efficiency and other dimensions (such as quality, appropriateness and economic sustainability).

These two methods are described in the following paragraphs.

Data sources

With regard to the calculation of technical efficiency, data sources for inputs and outputs come from the Tuscan Regional information system. In particular, balance sheets were used for input variables while other administrative data sources, such as hospital discharge flow and outpatient discharge flow, were adopted for output variables.

To answer the second research question, data from the 130 indicators of the Tuscan performance evaluation system were used.

Selection and definition of variables

Chief executive officers from the 12 Local Health Authorities in the Tuscany Region worked with the research team to select the variables to be used in this study. The process of identification of inputs and outputs to be considered and discussion of their coherence lasted 3 years, from 2005 to 2007. This process was carried out through several meetings. After the presentation of the non-parametric technique, researchers proposed a list of variables in order to measure all the outputs of the Local Health Authorities. Once inputs were defined, the Chief executive officers discussed which were the best output measures to be adopted. Once the outputs were calculated, the officers discussed the reliability of the measures. For instance, some prevention services were measured in different ways by the Tuscan local health authorities and could not be used for benchmarking processes. Ultimately, technical efficiency was calculated on the basis of all available and shared variables.

The input variable is represented by the total costs sustained to deliver health services to the population. The shared outputs include number of physicians; number of hospital services; pharmaceutical services and number of outpatient services. This last output is the most innovative because it includes services from primary care and prevention services; Table 1 shows the inputs and the outputs used in the analysis.

Table 1.

Variables for the calculation of technical efficiency

	Variables
Input variables	Total costs
Output variables	No. of physicians that work in primary care	No. of general practitioner and paediatricians
		No. of physicians for duty doctor
		No. of physicians of the emergency and ambulance services
	No. of hospitalization services	No. of hospitalizations for resident population in the LHSs corrected by the average weight of DRG
		No. of non-self-sufficient residents cared into residential facilities.
	Pharmaceutical services	No. of the Defined Daily Dose used out of hospital
	No. of outpatient services	No. of outpatient services (outpatient clinics and diagnostics, for resident population in Local Health Authorities)
		No. of visits at home (integrated domiciliary care)
		No. of access at Emergency Department
		No. of vaccines (flu for elderly people, German measles, mumps vaccines)
		No. of screenings (breast, cervix, colon)
		No. of inspections for safety and security at work

A particular case is the number of physicians. It is generally considered as an input variable. However, the Tuscan healthcare managers explicitly requested to consider the number of physicians as an output variable because, in their view, it provides the best proxy for measuring primary care services delivered by general practitioners, paediatricians, duty doctors and ambulance services.

The variables considered for the correlation analyses with technical efficiency are overall performance and weighted per capita cost.

Concerning the first variable (overall performance) it is measured using all the indicators of the performance evaluation system as a percentage of good performances on the total amount of indicators. The only indicators, which are not included in the overall performance, are those of population health status dimensions because they are not a direct consequence of management's actions in the short run.

Per capita costs are the total costs incurred by Local Health Authorities divided by their inhabitants, weighted according to their age.

Analyses

The main aim of efficiency analysis in the reported literature is to estimate an efficient frontier (a kind of frontier of the best practice) that characterizes the multi-input multi-output process of a group of decision-making units and then measure the distance of each decision-making unit from this estimated frontier. Within this literature, the non-parametric approach has received a considerable amount of interest because it is based on few assumptions and it does not require the specification of a functional form for the frontier. Especially, data envelopment analysis [14] is among the most known and applied non-parametric method for measuring efficiency in production and services activities.

Data envelopment analysis constructs a non-parametric envelopment frontier and, being non-parametric, does not require any assumptions on the functional form of the frontier. The activity of a decision-making unit is characterized by a set of inputs used to produce a set of outputs . In order to estimate the efficient frontier it is assumed that the set ψ of technically feasible combinations of (x, y) exists, is free disposable and convex. It is defined as

Data envelopment analysis involves the measurement of efficiency for a given unit (x, y) relative to the boundary of the convex hull of the observed sample of units X {(X_i, Y_i), i = 1, n}:

where are the intensity variables over which the maximization is made. Following Farrell [15], the technical efficiency has been measured in the input direction. Hence, the technical efficiency is defined as the ratio of the minimum (optimal) amount of inputs on the actual input levels of a decision-making unit for a given level of outputs, keeping the input proportions constant. In this paper, we assume that the technology exhibits constant returns to scale at the regional level. For a decision-making unit operating at level (x₀, y₀), the technical efficiency score is obtained, through linear programming, as follows:

A decision-making unit is considered as efficient if it lies on the efficient frontier and its technical efficiency score is equal to one; otherwise it is inefficient if its efficiency score is less than one.

However, it is well known that the data envelopment analysis estimator of technical efficiency described earlier is biased. For that reason, in this application we have used the bootstrap approach proposed by Simar and Wilson [16] to estimate the bias and provide a bias-corrected measure of technical efficiency as well as confidence intervals for the efficiency scores. The bootstrap is a data-based simulation method for statistical inference, useful to approximate the sampling distributions of interest by simulating data-generating processes. Another problem of data envelopment analysis is its curse of dimensionality; that is, it requires a lot of observations to avoid wide confidence intervals and imprecise estimation of the efficiency scores. The curse of dimensionality implies that working in smaller dimensions tends to provide better estimates of the efficient frontier. For that reason, in this application, we have followed the factorial approach described in Daraio and Simar [17] to aggregate the outputs. In this case it is very useful, given the small size of the sample analysed, and feasible, given the high correlation (higher than 97(%)) found among the outputs. The confidence interval bound computed in the analysis was 95(%).

The relationships among efficiency, overall performance and weighted per capita costs were explored using correlation analysis and linear regression. We estimated three linear regression models, one for each couple of variables: Overall performance on the per capita costs; Technical efficiency on overall performance; Technical efficiency on per capita costs. Results have been reported on mapping quadrants. Using mapping quadrants to provide visual representation of results, it is easy to locate clusters of Local Health Authorities.

Results

First research question

The first research question of the paper is to measure and compare the technical efficiency of Tuscan Local Health Authorities.

The average Local Health Authority in Tuscany sustains a total cost (input) of € 503 670.3 thousand to offer, on average, to its community the service of 435 physicians working in primary care (first output), 61 292 hospitalization services (second output), 909 Defined Daily Doses (DDD) used out of hospital (third output) and 4 893 900 outpatient services (fourth output), as shown in Table 2.

Table 2.

Input and output values

Variable	Mean	Stand. dev	Min value	Max value
Input (thousands of Euro)	503.67	272.434	265.693	1,357,171
First output (No. of physicians)	435	206	247	1.081
Second output (No. of hospitalization services)	61.292	34.379	33.461	170.009
Third output (No. of DDD)	909	80	682	1.002
Fourth output (thousands of outpatient services)	4.893	2.572	2.571	12.258

We can also observe that there is a high variability of the services provided by Local Health Authorities. The total cost sustained ranges from a minimum of € 265 693.1 to a maximum of € 1 357 171.8. Only the third output (pharmaceutical care) does not show such a large range between minimum and maximum values.

Table 3 shows the results of the efficiency analysis carried out on the 12 Local Health Authorities in Tuscany, using data for the year 2007.

Table 3.

Technical efficiency results

LHA	Technical efficiency	Tech. Eff. bias corrected	Bias	Bootstrap Std.	CI lower bound	CI upper bound
12	1	0.979	0.021	0.018	0.937	0.999
1	0.953	0.934	0.022	0.019	0.893	0.953
7	0.928	0.909	0.023	0.019	0.869	0.928
2	0.912	0.893	0.023	0.02	0.854	0.911
11	0.904	0.885	0.023	0.02	0.846	0.903
5	0.893	0.875	0.024	0.02	0.837	0.893
3	0.873	0.856	0.024	0.021	0.818	0.873
4	0.851	0.834	0.025	0.021	0.797	0.851
9	0.851	0.834	0.025	0.021	0.797	0.851
8	0.847	0.829	0.025	0.021	0.793	0.846
6	0.794	0.778	0.026	0.023	0.744	0.794
10	0.692	0.678	0.030	0.026	0.648	0.692

According to our investigation, Local Health Authorities in Tuscany have an average bias-corrected efficiency score of 0.86. This implies that the Local Health Authorities could produce the same level of services to their communities by using 14(%) less resources (inputs).

There is variability among the technical efficiency scores of the Tuscan Local Health Authorities: the bias-corrected efficiency score varies from a minimum of 0.68 to a maximum of 0.98. It is worth noting that even the best-performing Local Health Authority, in terms of technical efficiency, may further improve its results by reducing its input usage by 2(%).

Second research question

The second research question of this paper is to investigate the relationships among technical efficiency scores, weighted per capita cost and overall performance.

Results of correlation analyses and linear regressions are reported in Fig. 1. The first mapping quadrant (Fig. 1a) shows the relation between technical efficiency and overall performance; the other mapping quadrants (Fig. 1b and c) show the relation between per capita cost and overall performance and between technical efficiency and per capita cost.

Figure 1.

Mapping quadrants of: technical efficiency (bias corrected) vs. % of overall performances (a); % of overall performances vs. Per capita cost (b); technical efficiency (bias corrected) vs. (weighted) per capita cost (c).

While there is a significant negative correlation (r = −0.7, P < 0.05 in 2007) between weighted per capita cost and the overall performance (Fig. 1b), no correlation has been found in 2007 (and also for 2005 and 2006) between technical efficiency and overall performance (Fig. 1a) or per capita cost (Fig. 1c).

Moreover, the relationship between overall performance and costs has subsequently been explored for 2008 and 2009, confirming the 2007 result that costs appear to be lower where the overall performance is higher.

Discussion and conclusions

Considering the first research question, the results show that there is a large variability in the efficiency scores among Local Health Authorities. Measuring technical efficiency through data envelopment analysis has been important because it sheds light on the possibility of improving the use of resources and thereby putting all local health authorities on the same level for outputs delivered. Because the Tuscan top managers worked with the researchers to select the variables considered in the non-parametric model, they accepted the ranking of efficiency when the analysis was completed.

Although data envelopment analysis has been accepted by the chief executive officers as a valid tool to measure efficiency, it proved to be a complicated tool to manage efficiency. Indeed, the chief executive officers found it problematic to translate the efficiency scores into specific actions to be taken in order to improve their performance.

Implications of this analysis can be useful for other Italian regions and countries. Data envelopment analysis can provide top managers with a valid technique to measure efficiency in benchmarking. However, it is useful only to detect a lack of efficiency, and not to identify the actions that need to be carried out for improvement. Involving top managers in the selection process of inputs and outputs is fundamental in order to make results accepted.

As regards the second research question, although further investigations are needed, it seems that the overall performance (made up mainly by quality, equity, effectiveness and appropriateness performance indicators) is the main determinant of costs, while technical efficiency seems to have little impact on per capita costs or overall performance.

In the reported literature, there is not a common position on the relationship between quality and costs. For instance, Jarman [18] highlighted that there is no correlation between adjusted mortality rates and reimbursement, while other studies, such as Berg [19], asserted that a reduction of medical errors and quality (measured by the hospital readmission rate) led to a reduction of costs.

The results of our research seem to sustain the last position and suggest that, since no correlation has been found between technical efficiency and per capita costs, in order to gain financial sustainability, it is important to manage appropriateness and quality. Results of a recent study on the Tuscan Performance Evaluation System [20] highlighted that working on the improvements of appropriateness and quality could lead to potential savings. The study evidences that at the regional level, 2–7% of healthcare budget can be reallocated if all institutions achieve the regional or the best practice performance of quality and appropriateness measured by the selected indicators. This method allows managers to identify the areas where the institutions can achieve a higher level of efficiency without negative effects on quality of care and instead re-allocate resources toward services with more value for patients.

The results of this empirical analysis should be considered a preliminary study. In particular, a great limitation of the technical efficiency measure consists of the number of decision-making units included in the study. Future analyses can be made by taking into account other Italian local health authorities, such as those of the Piemonte, Liguria, Umbria, Valle D'Aosta, Trentino-Alto Adige, Marche, Bolzano, Basilicata regions, that have adopted the same Tuscan performance evaluation system [21]. Other developments regarding the calculation of technical efficiency include the application of non-parametric efficiency techniques that are more robust to the influence of outliers [17].

In conclusion, this study demonstrates how data envelopment analysis, in conjunction with a performance evaluation system, such as the one being used by Local Health Authority in the Tuscan Region of Italy, can identify variation in efficiency among administrative units. Furthermore, the results of correlation analysis highlight that costs appear lower where the overall performance is higher, indicating that improvements in performance indicators such as quality and appropriateness may be useful in reducing overall healthcare costs. Involvement of chief executive officers and top managers in selection of input and output variables is essential if the findings from this type of analysis are to be accepted and translated into management decision-making at the system level. Use of performance measures is an important tool for managers who want to maintain effective health services despite budget constraints. Further research is warranted to continue the development and application of quantitative performance feedback for health systems management.

Funding

This working was supported by the Department of Health of Tuscany Region. Funding to pay the Open Access publication charges for this article was provided by Laboratorio Management e Sanità, Scuola Superiore Sant'Anna.