Improving health services to displaced persons in Aceh, Indonesia: a balanced scorecard

Abstract

Problem

After the Indian Ocean tsunami in December 2004, the International Organization for Migration constructed temporary health clinics to provide medical services to survivors living in temporary accommodation centres throughout Aceh, Indonesia. Limited resources, inadequate supervision, staff turnover and lack of a health information system made it challenging to provide quality primary health services.

Approach

A balanced scorecard was developed and implemented in collaboration with local health clinic staff and district health officials. Performance targets were identified. Staff collected data from clinics and accommodation centres to develop 30 simple performance measures. These measures were monitored periodically and discussed at meetings with stakeholders to guide the development of health interventions.

Local setting

Two years after the tsunami, 34 000 displaced persons continued to receive services from temporary health clinics in two districts of Aceh province. From March to December 2007, the scorecard was implemented in seven temporary health clinics.

Relevant changes

Interventions stimulated and tracked by the scorecard showed measurable improvements in preventive medicine, child health, capacity building of clinic staff and availability of essential drugs. By enhancing communication, the scorecard also led to qualitative benefits.

Lessons learnt

The balanced scorecard is a practical tool to focus attention and resources to facilitate improvement in disaster rehabilitation settings where health information infrastructure is poor. Introducing a mechanism for rapid improvement fostered communication between nongovernmental organizations, district health officials, clinic health workers and displaced persons.

Background

After the Indian Ocean tsunami in December 2004, there were 530 000 internally displaced persons in Aceh province, Indonesia. Local health care was often unavailable, since health workers were missing and facilities destroyed. The International Organization for Migration (IOM) worked with the Ministry of Health and local district health offices (DHOs) to construct temporary clinics for displaced populations.

Problem

As of November 2006, 34 000 persons still lived in temporary accommodation centres in the Aceh Barat and Nagan Raya districts of Aceh.¹ Local DHOs were occupied rebuilding a permanent health infrastructure, so temporary clinics had little support.

DHOs asked IOM to assist in monitoring temporary clinic operations. Through site visits, we learnt that the clinics lacked health promotion activities and maintained inconsistent stocks of supplies. Often health workers were newly trained, isolated and unsupervised. Unreliable financial arrangements between DHOs and the government agency for tsunami rehabilitation and reconstruction disrupted planning and contributed to high turnover of clinic staff. The absence of a health information system made it difficult to monitor service delivery and outcomes.

IOM proposed a balanced scorecard to document quality issues for each clinic and to serve as a management tool for DHOs and the Ministry of Health in this setting. A balanced scorecard is a set of simple measures used to describe and improve overall performance.² Afghanistan and the Netherlands recently adopted balanced scorecards to improve their national health systems, largely because of the tool’s ability to quickly identify problems and guide action. In these countries, measures for the scorecard were derived from large national databases.^3,4 Such large-scale databases do not exist in Aceh. Instead, we conducted regular onsite surveys to generate information for the balanced scorecard, which then guided rapid cycle improvement efforts.⁵

Balanced scorecard

Development

In September 2006, IOM staff conducted initial assessments of temporary health clinics, concentrating on four domains: (i) health worker training; (ii) facility resources; (iii) community satisfaction and outreach and; (iv) service provision. In each area, we developed simple measures. Through discussions with district health officials and clinic staff, each measure was selected based on its significance, potential for improvement and ease of data collection (Fig. 1). We hoped to develop a sustainable model to be used by DHOs and local staff.

Fig. 1.

Balanced scorecard measuring health services to displaced persons in Aceh, Indonesia, August 2007^a,b

DHO, district health office; Kaders, community health workers; PPH, postpartum haemorrhage preparation.

^a Missing data were represented with “.”

^b Hours of operation per week.

Two local IOM staff members, a nurse and a public health specialist, collected data through interviews, observations and basic assessments such as counting supplies and determining water availability. In the accommodation centres, the team sampled every fourth household. Data were combined to create composite measures using STATA, version 9 (StataCorp, College Station, United States of America).

The scorecard was presented in an easy-to-read table with each measure in a row and each clinic in a column. Measures were highlighted on the scorecard by colour to signify whether target levels were met: green (met target), yellow (partially met target) and red (below target). For example, the “drug” measure was green for clinics with more than 90% of essential drugs in stock, yellow for 90–50% and red for < 50% (Fig. 1). Target levels were developed from focus group meetings with provincial and district health officials using national standards.

Implementation

The balanced scorecard was piloted in February 2007. Beginning in March 2007, it was implemented every 1–2 months in seven clinics and their nine associated temporary accommodation centres. The project was completed in December 2007, coinciding with the relocation of displaced residents to permanent housing and the closure of centres and clinics. Over the 10-month implementation period, a total of seven scorecards were created. During each cycle, data collectors from IOM spent 4 hours per clinic on clinical assessments and affiliated resident interviews.

Findings

Observations

Several improvements in health services were observed after the scorecard was implemented. Clinic staff received training on disposal of used needles, thermometer use and equipment sterilization. To focus on preventive medicine, clinic staff initiated events such as health promotion talks and fairs. Clinic staff also revitalized the community child health programme by conducting monthly visits to temporary accommodation centres, during which they gave immunizations and tracked the growth curves of children aged less than 5 years. Clinic staff began to actively manage an inventory of drugs. Communication improved between management, staff and accommodation centre residents.

Impact

Over the course of 10 months, there was a steady decline in the number of clinics reusing needles. After training, the score for proper thermometer use was excellent across all clinics. Community satisfaction increased from 68% to 84%. The percentage of residents receiving outreach activities increased from 14% to 85%. The number of sampled children aged less than 5 years with up-to-date vaccinations increased from 6% to 40%. The availability of essential drugs increased slightly from 49% to 59%. Equally important were the unmeasured benefits from increased monitoring and improved communications.

Discussion

The balanced scorecard was an effective management tool used every 2 months for rapid improvement in a disaster rehabilitation setting. Each time the scorecard was used, we expected to see improvements in areas that were previously below standard.

Limitations

In selecting the indicators, we balanced their importance with ease of measurement and implementation using existing resources. For example, while disposal of used needles is arguably not the most important indicator, it acted as a proxy for systems indicators such as staff knowledge and training. As clinic staff became more focused on the scorecard’s output, the reported data became less reliable. For example, after training on thermometer use, all patients at one clinic recorded normal temperatures of exactly 37.0 °C. The scorecard could not directly make up for deficiencies in staff training nor did it quickly change established practice patterns, such as over-prescription of antibiotics and steroid medications. It had minimal effect on external factors such as the drug supply chain.

Lessons and accomplishments

By measuring progress frequently and presenting results graphically, the scorecard provided a collaborative, simple, evidence-based tool to focus efforts on improving health services in a low-resource environment. The implementation in Aceh highlighted the following principles:

Evidence

A data set was produced and constantly updated to provide up-to-date information on clinic activities and outcomes. As data accumulated, the scorecards gave clinic coordinators greater capacity to direct changes. At follow-up meetings, based on the most current set of measures, clinic coordinators discussed problems with DHOs and IOM and developed targeted interventions. Clinic staff could view graphs showing data on supplies and staffing levels. IOM could support its recommendations to DHOs with data. DHOs became more aware of clinic-level activities and needs, and responded with more supervisory visits.

Simplicity

The scorecard benefited from efforts to reduce the number of measures, so as not to overwhelm users with the volume and frequency of new information. Initially, monthly data collection occurred faster than the pace of interventions, so we adjusted the frequency of data collection to every 2 months to better align with reasonable expectations for change. Clinic staff and district health officials found it helpful to review a manageably small set of critical indicators. The number of essential measures could be further pared down. Rather than using STATA, the creation of measures could be done through a simpler programme.

Management

Clinic coordinators and district level supervisors used the scorecard to assess quality, prioritize resources and direct change by identifying action points with limited additional investment in resources and data infrastructure. There was a shift of resources to target problem areas and redefine staff roles and accountability. Interviews were conducted during quiet times in the clinic so as not to interfere with clinical operations. The DHO reallocated previously-budgeted but unused time for monitoring clinical operations to this new management tool. This effort triggered discussions on quality in Aceh as there was a large variation between clinics. The scorecard can be used to monitor trends across facilities and highlight excellent clinics as role models to share their knowledge with other clinics.

Capacity building

Basic skills improved among the clinic staff. The data collection process provided a structured forum for IOM to interface with the community and clinic staff, facilitating opportunities for feedback and training beyond the scope of the study. For example, after evaluating thermometer use among clinic staff, IOM workers would immediately discuss basic thermometer use. Reviews of medical records revealed an over-prescription of antibiotics with no correlation with clinical signs. This led to on-site teaching regarding appropriate antibiotic use.

Communication

The scorecard enhanced communications between accommodation centre residents, clinic staff and DHO and IOM management staff. Through the process of data collection, IOM had the opportunity to interact with and teach clinic staff and accommodation centre residents. Regular meetings to discuss the findings of the scorecard brought clinics together and promoted the dissemination of ideas. By identifying priorities, clinic staff could better focus their attention and resources.

Leadership

Leadership was a large, unmeasured factor affecting clinic performance. Clinics with enthusiastic leaders were the ones initiating and advocating for quality improvement. More emphasis could be placed on developing the role of supervisors at the district level and on training clinic leaders.

Next steps

A balanced scorecard would be relevant to primary health-care facilities attempting to implement improvements with limited resources. Although beyond the scope of this project, the lessons learnt from the period of rehabilitation may contribute to the ongoing reconstruction of the permanent health infrastructure in Aceh (Box 1). The use of a balanced scorecard may be adapted to the permanent subdistrict health clinics. These lessons may apply to other primary health-care settings where there is a need for a simple data-driven method to focus priorities and a mechanism to improve communications between management, staff and beneficiaries.

Box 1. Summary of main lessons learnt.

• Reduce the number of indicators measured by the scorecard so as not to overwhelm users with the volume and frequency of new information.

• Collect data every 2 months to keep pace with interventions.

• Hold regular meetings to discuss findings and determine priorities with clinic staff.