A Comparison of Three Models of Hospital Performance Assessment Using IPOCC Approach

Abstract

Background

Hospital performance measurement is an essential component of providing feedback on the efficacy and effectiveness of service. The purpose of this study was to compare three models of performance assessment through the IPOCC approach.

Methods

This descriptive-analytical study was conducted in 2018 in Sari educational hospital. The data collection instrument was BSC, EFQM and accreditation questionnaire which was filled out through census. The validity of the BSC questionnaire and EFQM was based on expert opinion, and its reliability was found to be 0.97 and 0.92 using Cronbach's alpha coefficient. The accreditation questionnaire was developed using a checklist of the Ministry of Health. Using the expert panel, the components of the questionnaires were classified into dimensions of input, process, output, control, and context. Data analysis was done applying descriptive statistics and one way ANOVA.

Results

The highest distribution of components and acquired points through the IPOCC approach were found in the BSC in the process dimension (58.8%) and control dimension (3.62 ± 0.56), in the EFQM, in the result dimension (40.2%) and structure dimension (3.25 ± 0.44), and in the accreditation, in the process dimension (64.4%) and control dimension (3.45 ± 0.72), respectively. The results of one-way ANOVA showed that there was a significant difference between different quality models (P <0.001).

Conclusion

The results of the present study showed that in evaluating the hospital through the IPOCC approach, the distribution of components was more in the dimensions of the process. Therefore, having a robust systematic approach was considered to be effective for hospitals.

Introduction

Complexity of hospitals (1), increasing costs of healthcare, and vitality of services and provided cares are among the factors that have encouraged hospitals to evolve in performance evaluation (2, 3). Performance evaluation consists of measuring organization performance through standard criteria, and determining the organizational performance gap with desirable standards (4). Hospital performance measurement is essential for providing feedback on the effectiveness of healthcare services and the extent to which programs are implemented. Indeed, this would lead to identifying strengths and weaknesses (5).

Locating these points helps in planning to achieve the standard and predetermined goals of the organization (4). Ultimately, this coordination leads to promoting health and meeting the needs of people and society (6). Therefore, evaluation is an integral part of any organization, since service improvement is sought through measurable information (7). Avoiding the assessment means abandoning valuable and scarce resources to an unknown fate (5). Where the performance of hospitals is not measured and evaluated, no steps can certainly be taken to improve them (8). In order to evaluate hospital performance, there are new and different methods, such as benchmarking, European Foundation for Quality Management, Balanced Scorecard, and Accreditation (9). BSC provides a diverse range of performance indicators in four perspectives: financial, customer relationship, internal process, and growth and learning, which can be used to evaluate the performance of hospitals and compare their performances accurately (10).

The EFQM is developed based on nine main criteria that are interconnected, and can affect each other in a progressive and cyclic manner. Of these, five criteria are “empowering” in an organization, and four other criteria are “outcome” of the performance and achievement of the organization's activities (11). Hospital accreditation is a process in which an independent organization, using expert and specialist knowledge (12), and based on written and predefined standards, takes measures in evaluating the units of the organization in the area in question, and decides in the case of granting executive authority (13). In this way, hospitals, in accordance with the existing standards and with regard to the safety of individuals, offer their services of high quality in order to improve their health and treatment outcomes (14). The results of a study done by Yang et al. (2005) showed that the implementation of balanced scorecard was a powerful and feasible framework for evaluating a hospital and had positive effects (15). The findings of a study conducted by Van Schoten (2016) and Favaretti et al. (2015) also showed that the use of the EFQM model in hospitals resulted in improved organizational performance over time (16,17). Meanwhile, the findings from Devkaran and Farrell showed that preparation for accreditation had a significant effect on the improvement of hospital performance, so that 74% of the measures had a significant positive improvement after validation (18).

Varied models of hospital performance evaluation often have a limited focus, and provide one-dimensional results of hospital performance. Hence, the interdependence of components is ignored. As a result, attention to indicators that lead to the overall promotion of the organization's performance is very important (9,19). In the meantime, this model is suitable as a simple mapping tool for all-round performance evaluation (20). Based on the Input, Process, Output, Control and Context (IPOCC) pattern, each system has an input that will be achieved by implementing a series of processes on inputs. Some actions in this pattern are in the background of all steps and actions that shadow all parts of the system, and are referred to as “context” (21). A review of the set of components for evaluation models in the form of IPOCC helps to identify, classify, and prioritize the factors that determine performance, which, in addition to affecting the quality of service, will help managers and active agents in the area of health services to make decent decisions to provide better customer service and satisfaction. Therefore, this study aimed to compare the three different models of hospital performance assessment through the IPOCC approach.

Methods

This descriptive-analytical study was conducted to compare the three models of hospital performance assessment with IPOCC approach in Sari educational and therapeutic Center from May to November in 2018. The hospital performance assessment was done according to three models; accreditation, BSC and EFQM. The validity of the BSC questionnaire and EFQM was based on expert opinion, and its reliability using Cronbach's alpha coefficient of the first two models was assumed to be 0.97 and 0.92. The scale of these questionnaires was a Likert spectrum of five options, varying from the lack of adequate (strongly disagree) actions to complete progress in each field (strongly agree).

Due to the low number of target population, sampling was not carried out, and the study was conducted as a census. The target population of these two models consisted of 33 people of the top, middle, and low level managers of the hospital. They were the chairman, managing director, deputy dean for education, treatment, research, and support in the educational center according to organizational charts. The study population also included nursing director, human resources director, financial manager, and head nurse, quality improvement manager, IT director, director of medical equipment, and director of hospital health information. Chief clinical nurses (25 clinical departments), and head of paraclinical sections, including pharmaceutical management, imaging, laboratory, and pathology staff also participated in this study. The BSC consisted of 4 criteria of internal processes, growth and learning, finance, and customers with 69 questions.

The EFQM questionnaire included 214 questions in 9 criteria of leadership, policy and strategy, people, partnership and resources, processes, customer results, people results, society results, and key performance results, and 36 other sub-criteria that overall had 1000 points, of which 500 points belonged to the outcome criteria. The criteria for entry into the study included the top, mid, and low level hospital managers who were willing to participate in the study. Those who were not willing to participate in the study or those who did not fill out the questionnaire were excluded from the study.

Accreditation data collection was done by using the national accreditation standards verification checklist, the third edition of the ministry of health. The Likert scale with 5 options was also used in this study for scoring. Given that the checklist was extracted from the national accreditation standards book of Iran, its validity was approved by the ministry of health and medical education of Iran. Using these checklists, the data needed to assess the compliance rate of accreditation standards in eight dimensions of management and leadership, care and treatment, nursing service management, patient rights, drug and equipment management, information management, para-clinical services, and prevention and health in different sections were collected by a team of medical specialists and nursing experts along with hospital and health managers. Permission was taken from the university authorities and target hospitals and written informed consent was obtained from the participants. Moreover, confidentiality was afforded to participants in current research

A team of 25 healthcare management scholars with sufficient experience was formed. Using the comments in the expert group, the components of the BSC, EFQM, and accreditation were categorized as input, process, output, control, and context components. Then, the hospital's performance in each model was assessed through the IPOCC approach.To analyze the data, descriptive and inferential statistics, such as frequency, mean, standard deviation, and one way ANOVA were used.

Results

The findings of the present study showed that out of a total of 33 people who responded to organizational excellence questionnaire and balanced scorecard, 26 (78.8%) were females while 7 (21.2%) were males. Also, the results showed that most respondents (45.5%) were in the age range of 40 to 50 years. The participants of the current research were in the age group of 30–40 years (30.3%), and 20–30 years (15.2%). The rest were in the age group of 50 years and older (9.1%). Most of the examinees (78.8%) had bachelor's degree and the rest had master's degree or higher (21.2%). Many of the participants (30.3%) had 10 to 15 years of work experience. Hence, 24.2% of them had an experience of over 25 years, and 21.2% had worked for 15 to 20 years, while 12.1% of them had 20 to 25 years of working experience. The rest (12.1%) had less than 5 years of experience.

Based on the results of the research, the studied center obtained the highest score in the evaluation with the accreditation model with the rate of 70.28% while it gained 62.5% and 63.3% points in the evaluation with organizational excellence models and balanced scorecard, respectively. As is shown in Table 1 and based on the findings, the highest percentage and distribution of components of quality models through the IPOCC approach belonged to the process dimension (58.8%) in the BSC model, outcome dimension in EFQM model (39.2%), and process dimension in accreditation model (64.4%).

Table 1.

Distribution of quality evaluation model components through the IPOCC approach

IPOCC Components	Input N (%)	Process N (%)	Outcome N (%)	Control N (%)	Context N (%)
BSC	7(10.3)	40(58.8)	2(2.9)	7(10.3)	12(7.7)
EFQM	17(7.9)	81(37.9)	86(40.2)	0(0)	30(14)
Accreditation	56(21.9)	113(64.4)	13(6.6)	14(7.1)	0(0)

The general status of the distribution of components of quality models with the IPOCC approach is illustrated using radar chart in Figure 1.

Figure 1.

Radar chart distribution status of the quality evaluation elements eomponents with the IPOCC approach.

As is shown in Table 2, in BSC, process dimension with an average of 3.62, and in EFQM, structure dimension with an average of 3.25, and in accreditation, control dimension with an average of 3.45 acquired the highest score of hospital performance evaluation through IPOCC approach. In this study, the results of one-way ANOVA showed that there was a significant difference between the components of different quality models.

Table 2.

Hospital benefits in quality models with IPOCC Approach

IPOCC Components	Input ±SD Mean	Process Mean±SD	Outcome Mean±SD	Control Mean±SD	Context Mean±SD	P-value
BSC	2.9±0.54	3.18±0.61	2.95±1.19	3.62±0.56	3.02±0.88	<0.001
EFQM	3.25±0.44	3.17±0.58	3.11±0.45	-	3.20±0.62	<0.001
Accreditation	2.99±0.51	3.16±0.36	3.02±0.33	3.45±0.72	-	<0.001

Considering the participants' responsiveness to the scales of BSC, EFQM, and accreditation, the general status of the hospital's percentage points based on quality models with the IPOCC approach using the radar chart is shown in Figure 2. In addition, as is shown in Table 3, the results of one-way ANOVA showed that there was a significant difference between different quality models.

Figure 2.

Radar chart of the percentage of hospital points based on quality models with the IPOCC Approach

Table 3.

One-way ANOVA

Overall performance	Sum of Squares	Degree of Freedom	Mean square	F	P-value
Between groups	4.324	2	0.618	5.567	<0.001
In group	6.213	6	0.111
Total	10.537	8

Discussion

The main aim of this study was to compare the three models of performance assessment with the IPOCC approach in Sari medical educational center. The result indicated that the study center obtained the highest score (70.28%) in the evaluation with accreditation model. It also scored 62.5% and 63.3% in the evaluation with organizational excellence models and balanced scorecard, respectively. This was mainly due to the increasing emphasis of accreditation standards on process and structural components.The results of the study further showed that in distributing BSC components with the IPOCC approach, more attention was paid to process components. In its assessment, control, process, context, outcome, and structure components obtained the highest scores.

The results of Behrouzi et al. (2014) showed that “internal processes” were a significant perspective of BSC. Then, structural resources, such as financial control and the importance of financial information flow were highlighted for efficient use of resources (10). Kim and Josey (2008) investigated the implementation of balanced scorecard in an indigenous hospital over a year. Their results indicated a high growth in the indices of internal processes, and consequently income and profitability (22), which was found to be consistent with the findings of the current study. In a study conducted by Nippak et al. (2014), hospital performance was calculated based on a balanced scorecard and according to input, process, output, and outcome indices, and the most important criterion for priority in measuring the performance level was the “outcome” criterion (20). Hence, the findings of their study was not in line with the results of the present study. The results of studies conducted by Wu et al. (2008) (23) and Chung et al. (2005) (24) also indicated that among the views of balanced scorecard, customer perspective was more significant than other views, which differed from the distribution of the components of the BSC model in the IPOCC approach, as well as the score obtained by the current study. This could be due to less attention to outcome indices in the Ministry of Health's issued programs, especially in state hospitals. Meanwhile, the present study was carried out following the reform of the health system of Iran. Contractionary policies of the government in managing health expenditures reduced the allocation of budget to development activities derived from the results of satisfaction survey in addition to suggestions and complaints of clients.

The results of the present study showed that in the distribution of components of the EFQM evaluation model through the IPOCC approach, more attention was paid to the dimensions of outcome and process. To the best of our knowledge, there are no local or international studies similar to the current study. Considering the analysis of the data and the obtained points about the empowerment measures and the results, it can be concluded that the target hospital was in a relatively good condition, which was partly in line with the results of the hospitals of Hasheminejad (Tehran) and Yodin Italy (25, 26).

Based on the results, the studied center obtained the lowest score in terms of outcome and process. This indicates that the hospital has not fully distinguished and understood the needs of customers, employees, and other stakeholders. These findings were in agreement with the findings of Valleje, Sanchez and Nabitz (27–29), while they did not match the findings of Dehnavi, Vernero, Sajadi (25,26,30). The results also indicated that the key processes and supports of the hospital were not properly designed, which was consistent with the findings of Imani-Nasab et al. (31) and inconsistent with the findings of Nabitz (29). Based on the results of the current research, achieving the highest score in the input dimension has shown that the hospital was in a desirable position in the field of resource management (financial and human). These finding was consistent whit the findings of Vernero (28), but did not match the findings of Sanchez (26). This could be due to less attention given to outcome indices in the ministry of health's issued programs.

The results of the study showed that the distribution of components of the evaluation model of accreditation with the IPOCC approach was more focused on the process dimension. Thus, the distribution of accreditation measures was prioritized in terms of structural, control, and outcome components. In the meantime, no component (metric) focused on examining the grounds on which the hospital was providing services. However, in the center's evaluation, the dimensions of structure and the outcome obtained the lowest scores. Mosadeqhrad (2016) stated that in the first edition of the hospital standards, three standards of structure, process, and output (outcome) were considered. However, more weight was given to structural and process standards, which need to be improved so as to improve hospital outcomes (32).

Another study conducted by Mosadeghrad et al. (2017) showed that Iran's accreditation standards emphasized work structures and processes, and the number of outcome measures and its result were very limited (33). Mohebbifar (2017) also obtained a significant negative correlation between the scores of hospital accreditation and patients'outcomes (34). A study on 36,777 patients in 73 hospitals in Germany (2010) showed that accreditation did not lead to patients' satisfaction (35), which was found to be consistent with the distribution of components and evaluation in this study. It should be said that Several studies have proved the positive impact of accreditation on the effectiveness of services and the improvement of hospital outcomes (36), which was different from the results of the current study. The reason for achieving low scores in terms of structure and outcome can be old age of the studied center (37) and the hoteling project conducted in the hospital, the noise of which is generated by the construction, which then can affect the results and satisfaction of the patients.

In conclusion, the results of the present study showed that in evaluating the hospital through the IPOCC approach, the highest distribution of components was in the BSC and accreditation in the process dimension and in the EFQM in the output and process dimensions. Also, the highest hospital scores were in the BSC and accreditation in the control dimension and the EFQM in the input dimension. Although the distribution of components was more in the dimensions of the process and result, the scores obtained were not appropriate in this dimension. Therefore, having a robust systematic approach that focuses on a management based on all the dimensions effective in performance is suggested. These dimensions mainly include outcome, process, and context, along with the dimensions of control and structure. Meanwhile, using the IPOCC approach as a simple mapping tool was recommended to help managers to identify critical steps in the service provision process.