Use of a national collaborative to improve hospital quality in a low-income setting

Abstract

Background

Quality improvement collaboratives are a widely used mechanism to improve hospital performance in high-income settings, but we lack evidence about their effectiveness in low-income settings.

Methods

We conducted cross-sectional and longitudinal analysis of data from the Ethiopian Hospital Alliance for Quality, a national collaborative sponsored by Ethiopia's Federal Ministry of Health. We identified hospital strategies associated with more positive patient satisfaction using linear regression and assessed changes in patient experience over a 3-year period (2012–2014) using matched t-tests.

Results

A total of 68 hospitals (response rate 68/120, 56.7%) were included in cross-sectional analysis. Four practices were significantly associated with more positive patient satisfaction (p<0.05): posting a record of cleaning activity in toilets and in patient wards, distributing leaflets in the local language with each prescription, and sharing ideas about patient experience across the hospital. Among hospitals that had complete data for longitudinal analysis (44/68, 65%), we found a 10% improvement in a 10-point measure of patient satisfaction (7.7 vs 8.4, p<0.01) from the start to the end of the study period.

Conclusions

Quality improvement collaboratives can be useful at scale in low-income settings in sub-Saharan Africa, particularly for hospitals that adopt strategies associated with patient satisfaction.

Introduction

In the US and other high-income settings, large-scale quality improvement collaboratives (QICs) are a widely-used mechanism to improve hospital performance. QICs are multiorganizational learning networks to promote quality improvement and uptake of evidence-based practice. QICs are varied in design, but are recognized by the following general characteristics: pursuit of a defined health outcome or performance target, deliberate application of quality improvement and learning activities, infusion of expert support or scientific evidence, and creation of a peer network for learning and collaboration.¹ Many, but not all QICs have demonstrated positive impact.^1–3 QICs have resulted in improvements in quality of clinical care, including timeliness of treatment for patients with acute myocardial infarction,^4,5 decreased surgical site infections^6,7 and improved outcomes in the neonatal intensive care setting.^8,9

Although substantial research exists on quality collaboratives in high-income settings, we lack evidence about the effectiveness of this approach to improving quality of care in low-income settings.¹⁰ Only a handful of studies have demonstrated success in QICs in low-income settings, including efforts to improve obstetric and newborn care in Niger,¹¹ decrease child mortality in Ghana,¹² scale up HIV treatment in South Africa¹³ and a series of USAID-funded collaboratives to improve quality indicators across 12 countries.¹⁴ Although promising, these efforts were sub-national in scope, covering a subset of districts or health facilities in the country, and the latter set of experiences is not well-documented in the peer-reviewed literature. These successful collaboratives also required major investments in training and mentoring in quality improvement methods by external organizations, with implications for sustainability without continued external support. In contrast, we found no examples of a country-led national campaign designed to promote inter-organizational learning across participant organizations.

Accordingly, we sought to understand the impact of a national quality collaborative implemented in a low-income country over a 3-year period, from 2012 to 2014. Ethiopia provided an ideal setting for this study because the country has been on a path to improved hospital quality for nearly a decade.^15–21 Although individual hospitals in Ethiopia had achieved substantial improvements in performance, wide national variation in hospital quality persisted. In this context, the Ethiopian Federal Ministry of Health (FMOH) embarked in April 2012 on a national quality collaborative to improve patients' experience of hospital care, providing an opportunity to learn from this effort. The findings of this study can be used to inform the design of future country-driven, large-scale collaboratives to promote improved hospital performance in resource-limited settings.

Methods

Setting and intervention

The Ethiopian Hospital Alliance for Quality (EHAQ) was a national quality collaborative created to improve the hospital performance in Ethiopia, with initial focus on patient experience assessed using the inpatient assessment of healthcare (I-PAHC),¹⁵ a tool validated for use in Ethiopia. EHAQ was the first quality alliance in Ethiopia and, to our knowledge, the first national quality alliance or collaborative in Africa. Led by the FMOH, EHAQ was governed by a national steering committee that included members from the FMOH, regional health bureaus (RHBs), universities, non-governmental organizations (NGOs), patient associations and development partners. Initial technical support for EHAQ was provided by the Ethiopian Hospital Management Initiative, a partnership between the FMOH, Clinton Health Access Initiative and Yale University, with funding from the US Centers for Disease Control and Prevention. EHAQ included every government hospital in Ethiopia (n=120 at the EHAQ launch in 2012). For the first cycle of EHAQ, the target outcome was patient experience. Patient experience was selected for focus by the Ministry of Health because of its importance to promoting patient-centered, quality hospital care and because of the availability of a validated measure of patient satisfaction already embedded in the performance management and reporting system of the country.²² In Ethiopia, patient experience is measured using a tool that was validated in Ethiopia and endorsed for national use by the FMOH.¹⁵ Patient satisfaction, a single item on the I-PAHC tool, is measured and reported as part of the 36 key performance indicators (KPIs), a set of national measures of quality reported by government hospitals to the RHBs as part of the country's hospital performance monitoring and improvement system.²²

The collaborative included two levels of membership: 15 LEAD (an acronym for Leadership, Excellence, Action and Dissemination) hospitals and 105 member hospitals at the launch of the collaborative. The selection of LEAD hospitals was transparent and evidence-based. Finalists were identified based on a three-step screening. First, hospitals were ranked based on the consistency of reporting of the 36 KPIs and performance on two prioritized KPIs (the patient satisfaction score used as an outcome in this study and the % of hospital management standards met). The top 30 hospitals were considered finalists. RHBs were asked to comment on each finalist hospital's management capacity, commitment to quality improvement and capacity to mentor other hospitals. The top 20 hospitals based on RHB feedback were visited by the EHAQ auditing committee to validate the submitted data and corroborate RHB impressions of hospital capacity, resulting in a final set of 15 LEAD hospitals. Each LEAD hospital was assigned to support a cluster of 4 to 10 member hospitals.

EHAQ was officially launched at a national meeting of hospital and RHB leadership. At the launch, each hospital received a change package highlighting practical tools and approaches for improving patient experience based on global best practices. During the next 6 months, several locally developed best practices were collected from LEAD hospitals as strategies for inclusion in a second change package of locally developed strategies to improve patient experience. Each change package was produced by the Ministry of Health (approximately 25 pages, available in hard copy and electronically) and included a detailed, practical description of a handful of concrete strategies to improve patient experience, together with copies of forms and templates to assist in the implementation of the strategy. In addition, LEAD hospitals hosted cluster workshops with their member hospitals to share experiences about improving patient experience, and a national meeting was held in December 2013 (about 18 months into the 3-year study period) with all participating hospitals to highlight improvements in patient experience, increases in uptake of recommended practices and successful implementation of cluster activities. At this meeting, the FMOH presented data about hospital and cluster performance in patient experience and presented sizable cash awards to the highest performing LEAD hospital, member hospital and cluster.

EHAQ was led by the FMOH and supported primarily through the FMOH operating budget, indicating strong national ownership and investment. The FMOH engaged an EHAQ steering committee that included development partners working on issues related to quality of health service delivery. Steering committee member organizations provided some financial and in-kind support to the following activities: national meetings, cluster meetings, site visits to LEAD hospitals for technical assistance and data validation, and production of the change packages. Some cluster meetings were funded through the operating budgets of the Regional Health Bureaus and the LEAD hospitals.

Study design and sample

We conducted both cross-sectional and longitudinal analyzes as part of this study. For the cross-sectional analysis, we used data from a survey of hospitals conducted during the December 2013 annual meeting (18 months into the 3-year period) in which we ascertained self-reported use of a set of strategies anticipated to be associated with patient experience, and we examined the correlation between implementation of these strategies and patient experience scores in the last quarter of the 3-year period. A total of 99 out of the 120 government hospitals participating in EHAQ reported valid patient experience scores in the last quarter; of these, 68 completed the survey, yielding a 68.7% response rate.

We were only able to include a subset of the 68 hospitals in longitudinal analysis, as it required valid patient experience scores in the first quarter (April–June 2012) and the last quarter of the 3-year study period. In the first quarter of EHAQ, 48 of the 68 hospitals (71%) reported valid patient experience scores; of these, 44 (92%) hospitals also reported valid patient experience scores in the final quarter. All research procedures were approved by the institutional review board at the Yale School of Medicine.

Data collection and measures

Outcome variable

Patient satisfaction, the primary endpoint of the study, was measured using a single item, which is a global rating of patient experience. The item asks, ‘On a scale of 0–10, 0 being the worst hospital and 10 being the best hospital, how would you rate this hospital?’ The item is part of a broader I-PAHC survey validated for use in Ethiopia, which measures multiple aspects of the patient experience.¹⁵ Each quarter, a data owner in each government hospital is responsible for administering the I-PAHC survey to 50 consecutive admissions and reporting the average score on this item to the Regional Health Bureau as part of Ethiopia's hospital performance monitoring and improvement system.²²

Independent variables

We assessed hospital-reported use of key practices that were anticipated to influence patient experience using a closed-ended questionnaire. In addition, we asked hospitals to rate the engagement with the cluster, based on how much knowledge about how to improve patient experience was shared among the hospitals in the cluster. The paper-based survey was administered in person at the December 2013 EHAQ national meeting to each hospital Chief Executive Officer (CEO) or his/her designee. We also recorded hospital characteristics including EHAQ status (LEAD vs member hospital), EHAQ cluster and region within the country.

Data analysis

We used standard frequency analysis to describe the sample of hospitals and then generated means to describe patient experience scores overall and by hospital characteristics. We tested differences between characteristics and patient experience scores using independent t-tests and ANOVA. We also examined associations between the use of each of the practices surveyed and patient experience scores in the last quarter of the 3-year study using linear regression. Among the subset of hospitals with valid patient experience data from 3 years earlier (n=44), we described changes in patient experience scores over time for the sample overall and for LEAD and member hospitals separately, as well as by region. We used matched t-tests to compare mean hospital patient experience scores in the last versus first quarters of the 3-year study period.

Results

Description of the sample

Among the 68 sample hospitals that had valid patient satisfaction scores and survey responses in the last quarter of the 3-year study, 18% (n=12) were LEAD hospitals (Table 1). A third of hospitals were located in Oromia (n=24) and 7% (n=5) of hospitals were located in Addis Ababa. The remaining hospitals were fairly evenly distributed among the Amhara (13%, n=9), Southern Nations, Nationalities, and Peoples’ (16%, n=11), Tigray (18%, n=12), and emerging (10%, n=7) regions. Two-thirds of the hospitals (n=45) were members of clusters that met to share knowledge about how to improve patient experiences. The average patient satisfaction score was 8.4 (SD=1.00), and patient satisfaction did not vary by LEAD versus member status, region or participation in cluster meetings.

Table 1.

Description of sample and patient satisfaction scores during the last quarter of the 3-year study period (n=68)

	Description of sample n (%)	Patient satisfaction score Mean (SD)
All hospitals	68 (100)	8.4 (1.00)
EHAQ Status
LEAD hospital	12 (18)	8.1 (0.98)
Member hospital	55 (82)	8.4 (0.99)
Region
Addis Ababa	5 (7)	8.3 (1.16)
Amhara	9 (13)	8.4 (0.91)
Oromia	24 (35)	8.5 (0.92)
SNNP	11 (16)	8.4 (0.96)
Tigray	12 (18)	8.4 (1.19)
Emerging regions	7 (10)	8.2 (1.29)
Cluster met to share knowledge
No	22 (33)	8.5 (0.96)
Yes	45 (67)	8.3 (1.02)

Patient satisfaction scores did not vary significantly by EHAQ status, region or whether the cluster had met to share knowledge.

EHAQ: Ethiopian Hospital Alliance for Quality; LEAD: Leadership, Excellence, Action and Dissemination; SNNP: Southern Nations, Nationalities, and Peoples.

Hospital practices associated with more positive patient experience

In cross-sectional analysis, linear regression models indicated that four hospital practices were associated with more positive patient satisfaction (Table 2). Hospitals that reported posting a record of cleaning activity in the toilets and in the patient wards and hospitals that distributed leaflets in the local language with each prescription had significantly higher patient satisfaction scores than hospitals that did not engage in these practices (p<0.05). Additionally, hospitals that strongly agreed that innovative ideas about patient experience were shared widely in the hospital had significantly higher patient experience scores than hospitals that did not strongly agree with this statement (p<0.05).

Table 2.

Hospital strategies associated with patient satisfaction scores in linear regression (n=68, which participated in follow-up survey and reported valid patient satisfaction data in last quarter of 3-year study)

	nb (%)	B (SE)
Staff identificationa
Uniforms for clinical staff	48 (73)	0.34 (0.28)
Uniforms for non-clinical staff	38 (56)	0.43 (0.24)
Posted signsa
Signs requesting a quiet environment are posted	40 (62)	−0.09 (0.26)
Hospital map is posted at hospital entrance	28 (43)	0.26 (0.25)
A record of cleaning activity is posted in the toilets	29 (45)	0.79 (0.23)c
A record of cleaning activity is posted in patient wards	39 (60)	0.92 (0.23)c
Local language leaflets are distributed by pharmacist with each prescription	18 (26)	0.61 (0.27)c
Medications are stored and distributed to in-patients by nursing staff	42 (63)	0.38 (0.25)
Nursing staff are not assigned shifts longer than 8 hours	22 (35)	0.41 (0.25)
Nursing stations do not have beds for nurses to sleep	26 (41)	0.41 (0.25)
Each patient is assigned to a specific nurse	40 (60)	−0.18 (0.25)
Innovative ideas about patient satisfaction are shared widely in my hospital (strongly agree vs does not strongly agree)	24 (36)	0.80 (0.24)c

^a Comparing hospitals that responded ‘always' versus sometimes, rarely or never.

^b Item responses missing no more than 5 respondents for any item.

^c p<0.05.

Changes in patient satisfaction

Among hospitals that had valid patient satisfaction scores during both the first and the last quarter of the 3-year study period (n=44/68, 65%), we found an approximate 10% improvement in patient satisfaction scores overall (7.7 vs 8.4, p<0.01) (Table 3). In stratified analysis, we found that this significant change was apparent among the member hospitals (n=33) (7.4 vs 8.5, p<0.01), but was not significant among the LEAD hospitals (n=11) (8.6 vs 8.3, p=0.26), which had started at higher baseline scores. We also found that hospitals that reported greater knowledge sharing among the cluster hospitals also had significant improvement in patient experience scores (p<0.01).

Table 3.

Longitudinal analysis on hospitals with valid patient satisfaction scores for the first and last quarter of 3-year study period (n=44)

	Description of sample	Patient satisfaction scores
		Baseline	Follow up	p-value
	n (%)	Mean (SD)	Mean (SD)
All hospitals	44 (100)	7.7 (1.20)	8.4 (0.97)	<0.01
EHAQ status
LEAD hospital	11 (25)	8.6 (0.91)	8.3 (1.16)	0.26
Member hospital	33 (75)	7.4 (1.14)	8.5 (0.91)	<0.01
Region
Addis Ababa	5 (11)	7.5 (0.62)	8.1 (1.02)	0.17
Amhara	8 (18)	7.6 (1.59)	8.6 (0.73)	0.18
Oromia	18 (41)	8.0 (1.06)	8.5 (0.97)	0.07
SNNP	4 (9)	6.7 (1.23)	7.6 (1.12)	0.51
Tigray	8 (18)	7.9 (1.31)	8.9 (0.65)	0.03
Emerging regions	1 (2)	6.9 (0)	6.4 (0)	NA
Cluster met to share knowledge
No	14 (32)	7.9 (0.91)	8.4 (0.95)	0.18
Yes	30 (68)	7.6 (1.31)	8.4 (0.99)	<0.01

EHAQ: Ethiopian Hospital Alliance for Quality; LEAD: Leadership, Excellence, Action and Dissemination; NA: not applicable; SNNP: Southern Nations, Nationalities, and Peoples.

Discussion

In this first of its kind study of a national collaborative in a low-income setting, we found significant improvement in patient satisfaction scores, the primary target of the initiative. We also identified key strategies that were significantly associated with better patient satisfaction. These strategies included cleaning practices, using local language in patient instructions and support for innovations. Although we cannot make causal inferences related to these practices, the magnitudes of these associations were notable and suggest further research is warranted.

The EHAQ strategy to organize hospitals into clusters with one LEAD hospital serving as a mentor of its neighboring cluster hospital was novel among national quality campaigns, which have not typically been organized in this way. The arrangement provided the opportunity for hospitals to teach and learn from each other, perhaps a particularly important resource in some low-income countries that may have less access to external resources via the internet or other sources. As might be expected, clusters that reported sharing information about how to improve patient experience among cluster hospitals had significant improvement over time relative to those that reported less sharing of knowledge. The finding supports the belief that organizations can and do learn from each other, and national alliances designed to facilitate such inter-organizational learning in their national context may be more likely to see benefits.

In designing and implementing EHAQ, the AIDED model of diffusion^23,24 was employed to maximize the likelihood of spread. As the model suggests, innovations are more likely to be adopted successfully if 1. they are designed after careful assessment of the environment, 2. they are designed to fit the needs of the adopting organizations, 3. support is developed in the environment for the innovation, and 4. organizations are properly engaged in the initiative. All these characteristics were apparent in the EHAQ example. First, the environment was strongly conducive after 7 years of investment in hospital management and strong public and government interest in improving hospitals' quality nationally. Second, the innovation, EHAQ, was designed to fit within already existing systems, such as existing KPI reporting requirements,²² a validated measure of patient satisfaction,¹⁵ and annual hospital meetings with the newly created hospital CEOs,²⁰ who championed performance improvement efforts. Additionally, the hospitals themselves proposed strategies that they believed worked in their environments to improve patient experience, rather than depending on external sources for strategies. Third, Ethiopia developed enormous political support for the efforts. The Minister of Health required hospitals to participate and financially rewarded those that improved the most. Furthermore, the KPI reporting system using a validated patient experience survey provided the needed measurement already embedded in the Ethiopian government hospital system as a key environmental factor that helped the EHAQ be sustained and its impact monitored. Last, the engagement strategy with hospitals was carefully conceived to use CEOs as boundary spanners, to package the material in the language that hospital staff understood (EHAQ guidelines were translated into local languages and explained by peer hospitals not external bodies), and to integrate EHAQ into the performance measurement system emerging as a requirement of hospitals in the country.

Implementation challenges, nevertheless, were many. First, the geographical distances and difficult travel conditions reduced the number of times hospitals within clusters could realistically meet and discuss best practices. Second, the lack of reliable internet communication also likely slowed the spread of new information and collaboration. Third, in some cases, disagreements about cluster membership arose, particularly among teaching hospitals that perceived their issues were distinct from issues of non-teaching hospitals in the same cluster. Last, collecting data reliably over the 3-year period from a complete set of hospitals was difficult, resulting in a smaller subset in the longitudinal analysis than intended.

Despite these challenges, many hospitals improved patient experience in the first 3 years of EHAQ; moreover, the success of EHAQ is measured by its replication and devolution going forward. Ethiopia has launched a new effort using the EHAQ mechanism: a national campaign to reduce facility-based maternal deaths by improving the quality of hospital labor and delivery practices. Additionally, Ethiopia has added a new cluster, specific to the university hospitals as they share similar structural challenges as they balance teaching and research objectives with clinical objectives.

Although QICs require tailoring to the country context, our findings suggest that the momentum of national efforts can result in improvements in hospital practices and patient experience. Based on the AIDED model of diffusion,^23,24 we believe that several aspects of the EHAQ design may be of relevance to countries seeking to implement a national QIC. First, national QICs that take into account the country's readiness and capacity for performance management and quality improvement at national, regional and facility levels may produce greater impact. Second, valid and easily available measures of quality are a critical input to strong QICs. Third, political endorsement and a highly visible campaign help create a supportive environment for innovation to spread. Last, finding ways to connect peer institutions, despite logistical hurdles, can promote the spread of locally developed innovations. The model of geographic clusters of LEAD and member hospitals may be especially useful in settings in which web-based and mobile connections are limited.

We lack data on why some practices were associated with patient satisfaction and others were not; however, we suspect that practices that were most visible and apparent to patients would be most important to their satisfaction. Cleanliness and printed instructions in local languages, as well as the overall environment of innovation may have been more felt by patients, whereas patients may not have noticed less proximate practices such as nurse staffing patterns and medication management practices. Additional research in this area would be helpful.

Our findings should be interpreted in light of several limitations. First, our sample size is limited, particularly in the longitudinal analysis in which many hospitals had missing data for patient experience either in the first or second time periods. Hospitals that remained in the sample may have been those that were more likely to be engaged and improve; despite issues of non-response, this does remain the largest longitudinal sample of performance in patient experience in a low-income country of which we know. Second, the time elapse, although longer than many studies of hospital improvements, was relatively short with only 3 years of data. A longer follow-up would be helpful to more thoroughly determine the integration of EHAQ as a standard operating routine of the hospital system in Ethiopia. Third, our analysis is based on self-report of patient satisfaction, which may be susceptible to reporting bias; however, the measure we used was developed and validated in Ethiopia; in addition, it is monitored through a relatively robust hospital performance monitoring and improvement system with regional oversight, and we observed a substantial amount of variability in the scores. These factors give us some confidence that the reporting bias is limited. Fourth, the strategies and approaches developed and shared across the EHAQ network may have had a positive impact on aspects of quality beyond patient experience (i.e., staff experience), but our analysis focused on the single outcome of patient satisfaction. Further research on the impact of QICs on other outcomes of interest in resource-limited settings is warranted. Last, we identified strategies significantly associated with performance using cross-sectional analysis and, hence, cannot establish causality, although the measure of practice pre-dated the measure of performance. Nevertheless, longer term follow-up to assess performance would be responsive to changes in the use of strategies within individual hospitals and would be helpful.

Conclusions

In summary, we found significant improvement in patient satisfaction over the first 3 years of the quality alliance implementation, which was more pronounced among hospitals whose cluster reported sharing knowledge among the cluster hospitals about improving patient experience. Although additional studies are needed, the present findings suggest that the quality alliance model popular in the US and Europe can also be useful in low-income settings in Africa. Tailoring such efforts to the context and ensuring government ownership and institutional championship in the form of strong management are likely key ingredients to promote successful national campaigns to improve hospital quality.