Comparative Study of Three Commonly Used Methods for Hospital Efficiency Analysis in Beijing Tertiary Public Hospitals, China

Abstract

Background:

Tertiary hospitals serve as the medical service center within the region and play an important role in the medical and health service system. They are also the key targets of public hospital reform in the new era in China. Through the reform of health system, the public hospital efficiency has changed remarkably. Therefore, this study aimed to provide some advice for efficiency assessment of public hospitals in China by comparing and analyzing the consistency of results obtained by three commonly used methods for examining hospital efficiency, that is, ratio analysis (RA), stochastic frontier analysis (SFA), and data envelopment analysis (DEA).

Methods:

The theoretical basis, operational processes, and the application status of RA, SFA, and DEA were learned through literature analysis. Then, the empirical analysis was conducted based on measured data from 51 tertiary public hospitals in Beijing from 2009 to 2011.

Results:

The average values of hospital efficiency calculated by SFA with index screening and principal component analysis (PCA) results and those calculated by DEA with index screening results were relatively stable. The efficiency of specialized hospitals was higher than that of general hospitals and that of traditional Chinese medicine hospitals. The results obtained by SFA with index screening results and the results obtained by SFA with PCA results showed a relatively high correlation (r-value in 2009, 2010, and 2011 were 0.869, 0.753, and 0.842, respectively, P < 0.01). The correlation between results obtained by DEA with index screening results and PCA results and results obtained by other methods showed statistical significance, but the correlation between results obtained by DEA with index screening results and PCA results was lower than that between results obtained by SFA with index screening results and PCA results.

Conclusions:

RA is not suitable for multi-index evaluation of hospital efficiency. In the given conditions, SFA is a stable efficiency analysis method. In the evaluation of hospital efficiency, DEA combined with PCA should be adopted with caution due to its poor stability.

INTRODUCTION

Hospitals are the main providers of health services. Thus, countries attach great importance to inputs of financial, material, and human resources into hospitals. According to the demands of people for health services and the comprehensive ability of the hospital, hospitals in China can be divided into three tiers: Primary hospitals, secondary hospitals, and tertiary institutions (highest tier). Until March 2013, 56.76% of hospitals in China are public hospitals, which have the best medical and health resources.[1] Tertiary hospitals serve as the medical service center within the region and play an important role in the medical and health service system; they are also the key targets of public hospital reform in the new era in China. In Implementation Plan for Recent Priorities of the Health Care System Reformissued by the Chinese government in 2009, improving hospital efficiency was included in one of the reform priorities, “promoting public hospital reform.”[2] Public hospital efficiency not only reveals its own ability to transform inputs into outputs, but also concerns the overall structure of health care system reform in the new era in China.[3] Therefore, rational evaluation of public hospital efficiency and improvement of resource efficiency has become common concerns to the Chinese government, public hospitals, and society.

At present, the most commonly used methods for evaluating the input–output efficiency of hospitals are ratio analysis (RA), stochastic frontier analysis (SFA), and data envelopment analysis (DEA).[4,5,6,7] A considerable proportion of domestic researchers has adopted those three methods in the evaluation of hospital efficiency. However, certain problems existed in those researches.

For example, when selecting basic indexes, more consideration was given to the availability of indexes, while the universality and representativeness were ignored.[8] In addition, the screening and selection of input and output indexes could not meet scientific standard, for in most literature, there was no strict screening procedure to select input and output indexes after the establishment of basic indexes database. Only a small proportion of articles explicitly adopted certain methods for index screening, such as factor analysis and cluster analysis (CA).[9,10,11,12] Moreover, most researches adopted DEA to evaluate hospital efficiency and failed to choose the most appropriate evaluation method according to the actual situation of public hospitals in China.[13]

In order to have a more comprehensive understanding of public hospital efficiency in China, this paper will conduct empirical analysis based on measured data from 51 tertiary public hospitals in Beijing from 2009 to 2011. The mean values of hospital efficiency in each year calculated with different methods were compared, and the consistency of results obtained with those three methods was further analyzed in an attempt to explore comprehensive evaluation of hospital efficiency.

METHODS

Source of data

In this study, 50 tertiary hospitals in Beijing were selected in 2009, including 24 general hospitals, 6 traditional Chinese medicine hospitals (TCM hospitals), and 20 specialized hospitals. In 2010, another TCM hospital was rated as a tertiary hospital. Thus, the number of research subjects in 2010 and 2011 was 51.

Study index

Indexes in 2009, 2010, and 2011 annual reports of hospitals were selected. The lack of data could cause error that the hospital efficiency would be higher than the actual efficiency if the input index value of was 0 or missing and that the hospital efficiency would be lower than the actual efficiency if the output index value was 0 or missing. Therefore, to avoid extreme cases, the indexes of which the values of over 5% of the total hospitals were 0 or missing, and the hospitals with over 5% of all the index values scored as 0 or missing were removed. Lastly, 22 input indexes and 20 output indexes were included.

Data analysis

First of all, the input and output indexes of all the hospitals in each year were processed with CA and then classified into 2 or 3 types according to statistical analysis results in combination with the literature review. After dimension reduction of data with principal component analysis (PCA), 3–5 input principal components and 3–5 output principal components were obtained to extract the main information of each index.

As for the efficiency calculation and comparison of efficiency values, the efficiency values of all the tertiary public hospitals in each year were calculated by RA, SFA, and DEA with corresponding input and output principal components. Since all those three methods for efficiency research required nonnegative data, PCA results were range-standardized using Microsoft Office Excel 2007 (Microsoft Corporation Inc., USA) and further calculated with RA using Microsoft Office Excel 2007. Using the specialized software Frontier 4.1 (Tim Coelli Inc., University of Queensland, Australia), SFA was conducted with index screening and PCA results to calculate hospital efficiency. In DEA, DEA- CCR (by A. Charnes & W. W. Cooper & E. Rhodes) model and a professional software DEAP 2.1 (Tim Coelli Inc.) was used to calculate hospital efficiency with index screening and PCA results. At last, SPSS 19.0 (SPSS Inc., USA) was used to carry out the descriptive statistics and the Spearman correlation test of all the efficiency values.

RESULTS

Index classification results

Based on CA results, input indexes were divided into three categories: Medical personnel and equipment input (including the number of authorized staff, total staff, health workers, medical practitioners, registered nurses, pharmacists, pharmacists of western medicine, clinical laboratory technicians, imaging technologists, other health workers, other technicians, management staff, logistics personnel, authorized beds, actual beds, equipment worth over 10,000 RMB, and the total value of equipment worth over 10,000 RMB), construction input (including floor space and the area of work place), and financial input (total assets, subsidies, and estimated revenue charges). The output indexes were also divided into three categories: Economic output (business revenue), bed utilization output (actual available bed days, actual using bed days, and bed days of discharged patients), and service output (including the number of outpatient visits, emergency room visits, hospital admissions, hospital discharges, operations for inpatients, cured cases and improved cases, utilization rate of authorized bed, turnover times, authorized bed utilization rate of actual bed, turnover times of actual bed, average available beds, and working days of hospital beds).

Results of principal component analysis

After PCA, only indexes whose eigenvalues exceeded 1 were exacted. The cumulative proportion in analysis of variance (ANOVA) of each principal component was over 70%. From 2009 to 2011, 3–4 input principal components and 3–4 output principal components were exacted [Table 1]. Those principal components were further standardized with range method to obtain nonnegative standardized principal components.

Table 1.

Each index principal component analysis results in 2009–2011

Year	Index type	Variables	Principal components (n)	Cumulative (%)
2009	Input indexes	Medical personnel and equipment input	2	81.4
		Construction input	1	96.9
		Financial input	1	86.3
	Output indexes	Bed utilization output	1	98.2
		Service output	3	87.2
2010	Input indexes	Medical personnel and equipment input	1	71.4
		Construction input	1	97.6
		Financial input	1	90.3
	Output indexes	Bed utilization output	1	76.9
		Financial input	3	87.2
2011	Input indexes	Medical personnel and equipment input	2	76.7
		Construction input	1	95.1
		Financial input	1	80.9
	Output indexes	Bed utilization output	1	96.2
		Financial input	2	80.5

Efficiency analysis of different types of hospitals from 2009 to 2011

Average efficiency values were obtained with RA, SFA, and DEA, respectively, based on the measured data of different types of tertiary hospitals in Beijing in 2009, 2010, and 2011 [Table 2]. Results obtained with RA in 2010 and 2011 showed that the hospital efficiencies of specialized hospitals, general hospitals, and TCM hospitals decreased successively, while in 2009, the efficiency value of TCM hospitals was the highest; meanwhile, within those 3 years, the average efficiency values of all three types of hospitals showed relatively great fluctuation. Average efficiency values of all three types of hospitals from 2009 to 2011 obtained by SFA with index screening results were all around 0.8 and showed slight fluctuation among different years; the efficiency value of TCM hospitals was the lowest in 3 years. The average efficiency values of all 3 types of hospitals from 2009 to 2011 obtained by SFA with principal component results were all around 0.85 and fluctuated slightly among different years. The average efficiency values of all 3 types of hospitals from 2009 to 2011 obtained by DEA with index screening results were all around 0.8 and showed slight fluctuation; the efficiency value of TCM hospitals was lower than the other two. Results obtained by DEA with principal component results in 2009 and 2011 showed that the hospital efficiencies of specialized hospitals, general hospitals, and TCM hospitals decreased successively while results obtained with the same method in 2010 showed that the hospital efficiencies of specialized hospitals, TCM hospitals, and general hospitals decreased successively; meanwhile, within those 3 years, the average efficiency values of all 3 types of hospitals fluctuated greatly.

Table 2.

Efficiencies of different types of hospitals obtained by three methods (2009–2011)

Year	Type of hospital	RA	SFA		DEA
			Index screening	Principal component	Index screening	Principal component
2009	Specialized hospitals	1.560	0.818	0.854	0.874	0.918
	General hospitals	1.558	0.839	0.856	0.881	0.913
	TCM hospitals	2.362	0.797	0.838	0.806	0.854
2010	Specialized hospitals	3.489	0.842	0.850	0.884	0.564
	General hospitals	1.329	0.856	0.851	0.884	0.348
	TCM hospitals	1.518	0.810	0.844	0.841	0.440
2011	Specialized hospitals	2.313	0.825	0.870	0.837	0.910
	General hospitals	1.518	0.845	0.882	0.864	0.902
	TCM hospitals	2.165	0.812	0.897	0.751	0.842

RA: Ratio analysis; SFA: Stochastic frontier analysis; DEA: Data envelopment analysis, TCM hospitals: Traditional Chinese medicine hospitals.

Analysis results of rank correlation

As shown in Tables 3–5, the results obtained by SFA with index screening results and by SFA with PCA results showed relatively high correlation (r-value in 2009, 2010, and 2011 were 0.869, 0.753, and 0.842, respectively, P < 0.05, with statistical significance). The correlation between results obtained by DEA with index screening and PCA results and results obtained by other methods showed statistical significance, but the correlation between results obtained by DEA with index screening and with PCA results was lower than that between results obtained by SFA with index screening and with PCA results.

Table 3.

Coefficient of rank correlation (r) of results of three methods in 2009

Items	RA	SFA		DEA
		Index screening	Principal component	Index screening	Principal component
RA	–	0.069	0.091	0.295*	0.378*
SFA
Index screening	0.069	–	0.869*	0.404*	0.381*
Principal component	0.091	0.869*	–	0.369*	0.490*
DEA
Index screening	0.295*	0.404*	0.369*	–	0.606*
Principal component	0.359*	0.381*	0.490*	0.606*	–

*P < 0.05. RA: Ratio analysis; SFA: Stochastic frontier analysis; DEA: Data envelopment analysis.

Table 5.

Coefficient of rank correlation (r) of results of three methods in 2011

Items	RA	SFA		DEA
		Index screening	Principal component	Index screening	Principal component
RA	–	0.149	0.049	0.553*	0.529*
SFA
Index screening	0.149	–	0.842*	0.490*	0.619*
Principal component	0.049	0.842*	–	0.292*	0.470*
DEA
Index screening	0.553*	0.490*	0.292*	–	0.610*
Principal component	0.529*	0.619*	0.470*	0.610*	–

*P<0.05. RA: Ratio analysis; SFA: Stochastic frontier analysis; DEA: Data envelopment analysis.

Table 4.

Coefficient of rank correlation (r) of results of three methods in 2010

Items	RA	SFA		DEA
		Index screening	Principal component	Index screening	Principal component
RA	–	−0.070	0.038	0.466*	0.545*
SFA
Index screening	−0.070	–	0.753*	0.477*	0.355*
Principal component	0.038	0.753*	–	0.367*	0.286*
DEA
Index screening	0.466*	0.477*	0.367*	–	0.612*
Principal component	0.545*	0.355*	0.286*	0.612*	–

*P<0.05. RA: Ratio analysis; SFA: Stochastic frontier analysis; DEA: Data envelopment analysis.

DISCUSSION

Ratio analysis is not applicable to the evaluation of hospital efficiency with multiple indexes

RA determines the technical efficiency of a hospital according to the ratio between certain outputs and inputs. The RA indexes commonly used are the occupancy rate of beds, adjusted average length of stay, etc. This method is simple and can identify the hospital efficiency directly with the results. However, when the efficiencies of various hospitals are compared using ratios, there are no objective standards to identify inefficient hospitals. In addition, in RA, only limited indexes are considered, making it unable to determine the overall efficiency. Each ratio is calculated with only one input index and one output index, whereas efficiency evaluation of the majority of hospital involves a number of input and output indexes.

The analysis results showed that the average efficiency values obtained by RA fluctuated greatly from 2009 to 2011. When used to compare the general ability of a group of institutions, RA cannot distinguish the most efficient hospitals from inefficient ones indicating its poor reliability.[14,15] Therefore, RA is not recommended to evaluate hospital efficiency with multiple input indexes. However, it is worth noting that if the relative magnitude of hospitals’ efficiencies is cared about, or if the first input principal component and the first output principal component show enough message of the original data, RA can be an alternative to DEA.

Efficiency analysis by data envelopment analysis with principal component results shows poor stability

Efficiency analysis results of all kinds of hospitals from 2009 to 2011 showed that the average efficiency in 2010 obtained by DEA with principal component results fluctuated greatly compared with that in 2009 and in 2011, which might be caused by data error or data missing in the extraction of principal components. Therefore, if the amount of data information obtained by PCA varies over a year, the efficiency analysis results should be read with caution. An alternative is to adopt the index screening method so that the stability of original index information can be guaranteed. In general, the stability of efficiency analysis by DEA with principal component results is poor. Thus, this method should be used with caution in the evaluation of hospital efficiency.

In the given conditions, stochastic frontier analysis is a stable method for efficiency analysis

The rank correlation between the result obtained by SFA with index screening results and the result obtained by SFA with PCA results was relatively high (r-value in 2009, 2010, and 2011 were 0.869, 0.753, and 0.842, respectively, P < 0.01) and higher than that between results obtained by DEA (r-value in 2009, 2010, and 2011 were 0.606, 0.612, and 0.610, respectively). Since the indexes obtained by dimension reduction in PCA are the linear combination of original indexes, the original indexes and dimension reduction indexes have strong collinearity. Thus, when those two kinds of indexes are used to calculate efficiency with the formula of SFA, EFF = E (Y_i | U_i, X_i)/E (Y_i | U_i = 0, X_i), the efficiency values obtained would show strong correlation. While in the DEA, introduction of more indexes will lead to increased data dimensions and further the change of production frontier and results of relative effectiveness.[16,17,18] In addition, it will cause great fluctuation of efficiency values of institutions for evaluation, which is different in SFA using production function of the same dimension.[19,20] At present, we could not prove theoretically the inner link between original indexes and dimension reduction indexes in SFA. However, the results obtained by SFA with screening indexes and with dimension reduction indexes still showed strong stability. Moreover, average efficiency values of all types of hospitals obtained by SFA with screening indexes and with dimension reduction indexes only fluctuated slightly among those 3 years, indicating that the absolute values of efficiency obtained in this method are stable. The challenges of applying SFA lie in the presumption of specific functions. However, if array data with better effect is adopted, the stability and accuracy of efficiency evaluation results can be improved, which can also provide an explanation for the high rank correlation between SFA results from another aspect.

Policy implications

Large public hospitals took the lead to provide medical services for the general public, however they got the patients stuck in a difficult situation of “poor accessibility and affordability” and “insufficient access to large hospitals and improper treatment in small hospitals” after the 1990s. Although this phenomenon was caused by many factors, the root cause was the issues related to medical resource allocation and operation of hospitals.[21,22] If the efficiency of the health system is too low, it is not possible to solve those two issues mentioned above even with gradually increased investment from the government.[23] Therefore, improving the operational efficiency of public hospitals with limited financial resources from the government so as to make the hospitals achieve the best economic and social benefits shall be prioritized by health policy makers and hospital managers. In this regard, government departments can organize researches and scientific analysts to formulate a scientific and feasible system and also to set standards for comprehensive hospital efficiency evaluation based on the actual operation situation of medical institutions while promoting the reform of public hospitals. Although Hospital Management Evaluation Manual (2008) has been issued in China, quantitative standards for some of the indicators included in the manual were absent.[24,25] The current urgent need is to establish a set of guidelines for hospital efficiency evaluation that accord with the situations in China.

In conclusion, index screening and PCA have special strength, and the method to evaluate hospital efficiency should be selected based on actual conditions. RA is not applicable to multi-index evaluation of hospital efficiency. In the given conditions, SFA would be a stable method for efficiency evaluation. DEA would be important for reducing input when seeking methods to improve hospital's efficiency. However, efficiency values obtained by DEA with principal component results show poor stability. Thus, this method should be used with caution in the evaluation of hospital efficiency. In addition, the evaluation of hospital efficiency should be paid attention to in the promotion of the public hospital reform.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.