A New Approach of Measuring Hospital Performance for Low- and Middle-income Countries

Abstract

Efficiency of the hospitals affects the price of health services. Health care payments have equity implications. Evidence on hospital performance can support to design the policy; however, the recent literature on hospital efficiency produced conflicting results. Consequently, policy decisions are uncertain. Even the most of evidence were produced by using data from high income countries. Conflicting results were produced particularly due to differences in methods of measuring performance. Recently a management approach has been developed to measure the hospital performance. This approach to measure the hospital performance is very useful from policy perspective to improve health system from cost-effective way in low and middle income countries. Measuring hospital performance through management approach has some basic characteristics such as scoring management practices through double blind survey, measuring hospital outputs using various indicators, estimating the relationship between management practices and outputs of the hospitals. This approach has been successfully applied to developed countries; however, some revisions are required without violating the fundamental principle of this approach to replicate in low- and middle-income countries. The process has been clearly defined and applied to Nepal. As the results of this, the approach produced expected results. The paper contributes to improve the approach to measure hospital performance.

INTRODUCTION

Policy analysts have a long-standing interest in cost and efficiency of hospitals by adopting the strategy of increasing competition among the hospitals where insurance system is not effective to reduce the continued escalating health care cost, particularly in low- and middle-income countries (LMICs). An extensive literature on measuring efficiency of the hospitals is available for high income countries (1,2,3,4,5). However in LMICs, little evidences exits to support policies of controlling costs through competitive mechanism, which makes the policy uncertain and debatable. Health care quality improvement can help to reduce resource use by eliminating medical errors and unnecessary procedures (6). If the health care market were perfectly competitive, efficiency measurement would be unnecessary (7). Higher levels of friction, less competition, and underdeveloped demand are basic characteristics of the health market in low and middle countries. The supply of efficiency analyses is, therefore, impressive (6). Promoting efficiency and equity is one of the key strategies to move universal health coverage for these countries. Efficiency of health system broadens fiscal space to increase the population and service coverage.

In practice, two general approaches are popular to measure the hospital performance: a nonparametric (or data envelopment analysis: DEA) and a parametric (the stochastic frontier approach: SFA) (8). Each method has certain strengths and weaknesses and measures different aspects of efficiency (9). Nonparametric techniques are sensitive to the influence of outliers. Parametric approach captures random fluctuations and overcomes the shortcoming of the deterministic cost frontier; however, it requires strong assumptions to form the frontier (7). Strong assumptions and several limitations of econometric analysis may produce bias results. Choice of cost function is extremely important (10) because unlike to other small and medium-sized enterprises (SMEs), introduction of new technology in health sector may increase the cost of services. There are huge debates found in the literature while adopting the process of measurements such as choice of cost functions, assumptions of the models, distribution of composed error, inclusion and exclusion of the variables, appropriate quality variable, use of one stage or two stage estimation approach among others (11). Different approaches or methods have been continuously developed and are being used to measure efficiency and productivity of the SMEs (12). Recent research in economics has suggested a new way of measuring and understanding the performance of SMEs using management approach. The quality of health care has a strong positive relation with better hospital management, and it is found that health care quality improvement can reduce resource use by eliminating medical errors and unnecessary procedures (6).

Due to the absence of significant evidence based on management approach in LMICs, the policy of controlling costs through competitive mechanism is uncertain and has become a matter of debate. Most of the evidences are related to high income or developed countries and output indicators of the hospital in high income countries may be different from those of LMICs due to a number of reasons, for example, curative care can significantly reduce the mortality rate in the service area of the hospital in the high income countries; however, in LMICs, preventive services as well as curative care have contributed to reduce mortality rate. Similarly, access to health care services and coverage of health insurance may be significantly different between developed countries and LMICs. If we use mortality rate as an outcome of the hospital, it would be an upward bias and overestimated. An appropriate method is lacking in the literature to measure hospital performance in LMICs. This paper contributes to the literature to provide a useful method in terms of policy relevance to measure the hospital performance in LMICs. The objective of this paper is to explore a new method of measuring hospital performance in LMICs by utilizing the recent evidences.

MATERIALS AND METHODS

Bloom et al. (13) utilizes the concept recently developed to measure the hospital performance through hospital management. However, this method was applied primarily in high income countries. Some modifications are needed to apply this method in LMICs. The paper utilizes some experiences from Nepal where the method was applied to measure the hospital performance for both public and private hospitals. The methodology has two components: scoring of management practices and linkage to the output indicators of the hospitals while measuring hospital performance. Due to specific characteristics of health care market in LMICs, measurement techniques and indicators were revised while measuring the performance of the hospitals. This paper, therefore, contributes to the improvements through two ways: modifying the management survey techniques to score the management performance without violating the basic concept of double blind survey and identifying the appropriate performance indicators of the hospitals in LMICs.

MANAGEMENT PRACTICES: CONCEPT AND APPROACH

Good management is the most important driver of productivity of small and medium enterprises (SMEs). By using this concept, Bloom et al. (14) provides a methodology and survey tool to evaluate the correlation between management decisions and SMEs' economic performance. The method and survey instruments were successfully tested in major sectors of SMEs such as education, health, manufacturing and services delivery for both public and private enterprises. The empirical analysis (14) innovative survey tool and robust methodology confirmed a statistically valid correlation between management practices and SMEs' performance. The method of measuring management practice is adopted by other research teams in health sector (15,16) and in service delivery sector (17).

The analysis focuses on the relationship between the hospital management practices with their financial performance and clinical outcomes. Management practices are systematically measured, and then their role is investigated in explaining differences in performance across hospitals. World Management Survey (WMS) replicated the concept and methods for developed as well as developing countries; however, due to differences in socioeconomic characteristics and behavior among the different countries some weakness was found in measuring management score. For example in India, they corrected exaggerated values using some subjective measures. Again there are some problems in measuring performance indicators particularly related to hospitals. The hospital performance indicators as used in the hospitals of USA or UK may not be appropriate for the developing countries. For example, WMS did not find the output indicators of hospital performance in India. The literature and experience suggest that modification of the survey methods and performance indicators, especially hospital performance indicators, is necessary.

IMPROVEMENT IN MANAGEMENT SURVEY

Bloom et al. (18) developed a new survey methodology to quantify and measure management practices in four broad dimensions: operational management, monitoring, target setting and people management. In the survey tool, operational management is covered by three questions primarily by gradual improvements in the services in order to make them patient oriented and efficient. Similarly, it also involves problem solving and management, and continuous improvement initiatives. The monitoring section focuses on the tracking of performance of individuals, reviewing performance (e.g. through regular performance appraisal and job plans), and consequence management (e.g. making sure that plans are kept, and appropriate sections and rewards are in place). The targets section examines the type of targets (whether goals are simply financial or operational or more holistic), the realm of the targets (stretching, unrealistic or non-binding), the transparency of targets (simple or complex), and range and interconnection of targets. Finally people management includes promotion criteria (e.g. pure tenure based or linked to individual's performance), pay and bonus system, fixing and firing poor performers, where practices are deemed best when the approach gives strong rewards to those with both ability and efforts. The four dimensions are organized into an interview based evaluation tool that defines and scores from 1 ("worst practice") to 5 ("best practice") based on 18 basic management practices. A high score represents a best practice in the sense that a hospital that has adopted the practice will, on average, increase their performance and productivity. The combination of these indicators reflects "good management" as commonly understood, with our main measure of management practices simply the average of these 18 scores. The tools provide some flexibility to the researchers to apply context specific situation. Therefore, the tools have been used extensively both in production and service sectors. In the hospital sector, the tool also provided consistent applications and results. It is therefore of interest to apply the tool in context of developing country like Nepal. However, the researcher should be careful while asking the manager to score the management practices.

Scoring of management practice using a double-blind survey (researcher don't know the performance indicators of the hospital before conducting the survey and respondents are not aware in advance that they are being scored) is a key element while measuring the management practice. However, the respondents from the LMICs might exaggerate the management practices while conducting the double blind survey. Anecdotal evidence suggests that there are some practices of the people not sharing everything "good or bad" of the enterprises to unknown person; particularly to the persons who are not in the face to face interactions. There is a need of improvement in double blind survey technique to avoid this bias. The study improved the double-blind survey technique to measure the management score, which is suitable for LMICs. There are two approaches. Firstly, it is clear that known person or the person who are coming face to face may get the right answer from the respondents. However, there may be bias towards score of the management practices if the person already knows about the performance of the hospital. In this case, the study can modify the method of double blind survey to avoid or to minimize the possible biasness and to improve the quality of survey. For this case, we need at least two researchers to conduct the double blind survey. A researcher has conducted face-to-face survey to collect the required information rather than telephone survey by administrating the same questionnaire and procedures developed by Bloom et al. (18); however, scoring grid should not be mentioned in the questionnaire.

Discussions and conversation between the researcher and the respondents can be recorded; however, the researcher doesn't inform the respondents that they will be scoring based on their answers. The researcher gives all tape-records marked special code without mentioning the name and location of the hospitals, to the second researcher who doesn't have any information about the hospital performance. The code number plays a significant role to make double blind survey. The second researcher follows the questionnaires and recorded material to score the management practice of the hospital. Face to face interactions at the hospital between the researcher and the respondents escape the possible exaggeration of management practices because the researcher may have or observe some information about the hospital during his/her visit. The researcher who may have information about the hospital has not scored to measure the management practices. The respondent may ask the questions to look at this while conducting the face to face interview. At this time, no scoring grid has been mentioned in the questionnaire. This modified method is able to maintain the double-blind scoring methodology and to escape the possible exaggeration of management practice while conducting the survey.

Secondly, the researcher can do the telephone survey while conducting the double blind survey; however, he or she should have good understanding the local contextual issues for example, policy, government guidelines, general behavior and way of thinking and can cross check the answers through probing questions. It means the researchers should have the understanding of how to score the management practices and how to minimize possible exaggeration through probing the issues based on government guideline, general practices, among others.

However, the researchers should be careful to avoid leading questions while conducting the double blind survey. Nepal experienced second method. After pre-testing the double blind survey, we improved the strategies to conduct the telephone survey; but the questionnaires were not changed. The management practice scores are higher in high income countries and lower in low income countries like Nepal. The improvement strategies include: booking appropriate time for telephone survey because sufficient time is needed to do interactions with the managers, starting from general discussion about the hospitals and personal background, developing a good rapport building, understanding the government policy, rules and regulations, cross checking the answers through asking additional questions and validating the answers. In the telephone survey, response rate may be lower than that of other surveys. In Nepal's case, the response rate in the telephone survey was almost 90 percent, some other survey for example; response rate was found more than 95 percent, such as Nepal Living Standard Survey or Nepal Demographic Health Survey (19,20,21).

REDEFINED THE OUTPUT INDICATORS

Measuring hospital performance indicators (18) are based on high income countries; however, these indicators are quite different in developing countries due to at least two distinct reasons: a) in the high income countries where the patients do not pay directly for their care due to provision of social insurance; however, in LMICs where health insurance is not effective, most of the patients have to pay directly for their care to the hospitals; b) it is more difficult to measure the hospital output, therefore, Bloom et al used some proxy indicators to measure the hospital output or quality such as mortality rate for non-elective admissions; however, in LMICs, access to health care is a primary problem for the patients and mortality rate at the hospital may not be the optimal output indicators. This study uses intermediate output indicators such as bed occupancy rate, outpatient visits, per capital cost of or revenue from the services to measure the hospital performance. These indicators measure access of the services as well as performance of the hospitals.

We developed five separate indicators that reflect one or more set of hospital outputs as well as policy goals of providing health services. The output indicators include: total inpatient days, infection prevention (IP) practice score, bed occupancy rate, inpatient days per technical staff, and recurrent expenditure per impatient day. Table 1 compares hospital output indicators used in high income countries and LMICs. It doesn't mean that one indicator substitutes another. It exhibits overall indicators.

Table 1. Hospital performance indicators.

Low and middle income countries	High income countries
Total inpatient days	Mortality rate from emergency heart attack admissions (AMI) after 28 days
Recurrent expenditure per impatient day	Mortality rate from emergency surgery after 30 days
Length of stay in the hospital	Numbers on waiting list
Infection prevention (IP) practice score as per government guideline	Infection rate of methicillin-resistant Staphylococcus aureus (MRSA) per 10,000 bed days
Bed occupancy rate	Operating margin
Inpatient days per technical staff	Staff likelihood of leaving within 12 months (1 = very unlikely, 5 = very likely)
Unit cost outpatient care	Average health care commission rating (1-4 scale)

Sources: 18, 19.

A total inpatient day was selected as absolute measure of performance particularly for at least three reasons. First, inpatient care is the prime objective of all the hospitals. Second, inpatient services represent the bulk of services within the hospital that consume greatest proportion of total available resources in the hospitals. Third, Waheb et al. (22) suggested to use inpatient days based efficiency indicators while evaluating performance of hospitals.

Infection prevention indicator was used to represent the process quality of services. It is based on national standard for quality improvement developed by the government. An observation checklist that includes 48 aspects to be monitored in order to evaluate the standard IP practice in the hospitals needs to be developed to measure the IP indicator. A single index of IP practice can be developed by utilizing principal component analysis based composite index. Third indicator, bed occupancy rate is widely used as performance indicator for the hospitals in national and international literature. This indicator is a prime indicator for evaluating performance and efficiency analysis of hospitals (22). The fourth indicator, inpatient days per technical staff, is also a relative indicator that represents the services against the available human resources. This is very useful indicator among a list of suggested indicators for the performance assessment of hospitals (22). The fifth indicator, recurrent expenditure per inpatient day is used against the annual recurrent expenditure. A gross-costing approach is used to measure the total annual recurrent expenditure from accounting records. This indicator provides the efficiency in use of monitory recurrent resources in order to produce one unit of inpatient day. All five indicators are expected to represent gross performance of hospitals in terms of volume of services, efficiency in use of non-financial capital resources (beds and human resources), quality of services in-terms of IP-practices, and efficiency in use of recurrent financial resources. The summary results of these indicators are presented in the Table 2.

Table 2. Summary results of output indicators.

Output variables	Mean	Std. Dev.	Min	Max
Bed occupancy rate	43.06	25.51	0.61	107.00
Total inpatient days	10,668.36	16,000.50	38.33	91,250.00
IP-practice score	0.82	0.16	0.35	1.00
Inpatient days per technical staff	123.85	103.44	1.32	620.75
Recurrent expenditure per inpatient day	6,060.60	8,881.99	46.26	68,309.28

Source: 19.

General method of data collection from the health facilities are followed as suggested by Lindelow and Wagstaff (23). Most of the data were collected from health information and management system (HMIS). The data collection instruments need to be developed to measure quality of hospital output such as IP indicator (detail questionnaires can be provided upon request). Top priority was given to maintain the internal and external validity of the data. Questionnaires are in a logical order, easy to understand and in local language and are finalized after pre-testing in the field.

The revised methodology has been applied to measure the hospital performance in Nepal. The study adopted following regression model to describe the relation between management score and hospital outputs. In order to achieve it, following regression equation was estimated.

y_i^k=αM_i + β'X_i + u_i.............................(1)

Where y_i^k represents k^th performance indicator for hospital i. M_i represents average management score for i^th hospital. X_i represents a vector of hospital characteristics, community characteristics and noise controls. ui represents error specific to i^th hospital (Table 3).

Table 3. Hospital performance and management practices.

Variables	Bed occupancy rate	Total inpatient days	IP-practice score	Impatient days per technical staff	Recurrent expenditure per impatient day
Estimate	0.319‡	0.210†	0.161†	0.303†	-0.171*
Robust Std. error	0.092	0.075	0.08	0.09	0.108
t-value	3.47	2.79	2.02	3.36	-1.59
R-squared	0.47	0.65	0.57	0.56	0.34

^*10%, ^†5%, ^‡less than 1%.

The results show that better management practices are strongly associated with expected sign with the indicators of performance of the hospitals in terms of total inpatient days; infection prevention (IP) practice score, bed occupancy rate, inpatient days per technical staff and recurrent expenditure per inpatient day. The result indicates that management score is not only associated with total services but also associated with the quality of service delivery. The study demonstrated that there are several factors including structural quality of the hospital and market environment to improve the management performance of both public and private hospitals.

DISCUSSION AND CONCLUSION

Output indicators of the hospitals are quite different in LMICs as compared to high income countries due to at least two distinct reasons: a) in the high income countries where the patients do not pay directly for their care due to provision of health insurance; however in LMICs where health insurance is negligible, the patients have to pay directly for their care to the hospitals; b) mortality rate for non-elective admissions might not be proxy indicators to measure the hospital output or quality in developing countries. Accessing health care is a primary concern for the patients. This study contributes to redefining the hospital output indicators that are quite relevant to LMICs. These indicators measure access of the services as well as performance of the hospital. Measurement of management practices is principally directed at explaining the performance of hospitals. Scoring of management practice using a double-blind survey (researcher don't know the performance indicators of the hospital before conducting the survey and respondents are not aware in advance that they are being scored) is a key element while measuring the management practice.

This method has a greater policy implication in LMICs. The policy makers and researchers are trying to find out appropriate method to improve the hospital performance in LMICs. The number of ways such as result based financing, capitation based payment is practiced to improve the performance of the hospital outputs in these countries. Improvement in management performance to improve the hospital output may be cost effective compared to other mechanisms such result based financing. Result based financing requires to split purchasing and financing role in the health system, but it very difficult task because of a lack of health insurance (separate provision in the system). Therefore, management measuring approach is an innovative method that can improve the hospital performance through a cost-effective way.

In conclusions, the measuring hospital performance through management approach is quite relevant for LMICs compared to conventional methods such as parametric and non-parametric. Measuring hospital performance through management approach is very useful for policy makers because it may be cost effective approach to improve the health system performance compared other approach such as result based financing, capitation payment among others. The method has been developed based on high income countries; however, it is quite relevant for the LMICs if we improve the survey techniques and hospital output indicators.