Interventions for hiring, retaining and training district health systems managers in low‐ and middle‐income countries

Abstract

Background

District managers are playing an increasingly important role in determining the performance of health systems in low‐ and middle‐income countries as a result of decentralization.

Objectives

To assess the effectiveness of interventions to hire, retain and train district health systems managers in low‐ and middle‐income countries.

Search methods

We searched a wide range of international databases, including the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE and EMBASE. We also searched online resources of international agencies, including the World Bank, to find relevant grey literature. Searches were conducted in December 2011.

Selection criteria

District health systems managers are those persons who are responsible for overseeing the operations of the health system within a defined, subnational geographical area that is designated as a district. Hiring and retention interventions include those that aim to increase the attractiveness of district management positions, as well as those related to hiring and retention processes, such as private contracting. Training interventions include education programs to develop future managers and on‐the‐job training programs for current managers. To be included, studies needed to use one of the following study designs: randomized controlled trial, nonrandomized controlled trial, controlled before‐and‐after study, and interrupted time series analysis.

Data collection and analysis

We report measures of effect in the same way that the primary study authors have reported them. Due to the varied nature of interventions included in this review we could not pool data across studies.

Main results

Two studies met our inclusion criteria. The findings of one study conducted in Cambodia provide low quality evidence that private contracts with international nongovernmental organizations (NGOs) for district health systems management ('contracting‐in') may improve health care access and utilization. Contracting‐in increased use of antenatal care by 28% and use of public facilities by 14%. However, contracting‐in was not found to have an effect on population health outcomes. The findings of the other study provide low quality evidence that intermittent training courses over 18 months may improve district health system managers’ performance. In three countries in Latin America, managers who did not receive the intermittent training courses had between 2.4 and 8.3 times more management deficiencies than managers who received the training courses. No studies that aimed to investigate interventions for retaining district health systems managers met our study selection criteria for inclusion in this review.

Authors' conclusions

There is low quality evidence that contracting‐in may improve health care accessibility and utilization and that intermittent training courses may improve district health systems managers’ performance. More evidence is required before firm conclusions can be drawn regarding the effectiveness of these interventions in diverse settings. Other interventions that might be promising candidates for hiring and retaining (e.g., government regulations, professional support programs) as well as training district health systems managers (e.g., in‐service workshops with on‐site support) have not been adequately investigated.

Plain language summary

Interventions to hire, retain and train district health systems managers

Researchers in the Cochrane Collaboration conducted a review of the effect of interventions to hire, retain and train district health systems managers in low‐ and middle‐income countries. After searching for all relevant studies, they found only two that met their prespecified study selection criteria.

Interventions to hire, retain and train district health systems managers

In many low‐ and middle‐income countries, the responsibility for managing important aspects of the health services are being decentralized to local governing bodies, including district health teams. As a result, district health systems managers are playing an increasingly important role.

A district manager is responsible for overseeing the operations of the health system within a particular subnational geographical area. District health systems managers are often responsible for planning and budgeting, human resources management and service quality monitoring. Poor performance by a district manager can lead to a number of problems, such as lack of drugs and supplies, delayed repair of broken equipment, health worker absenteeism and lack of motivation among health workers.

Different approaches are used to improve the quality of district managers.  Some of these approaches address the way in which managers are hired and retained, for instance by making district management positions more attractive or by giving contracts ('contracting‐in') to private, nongovernmental organizations (NGOs). Other approaches focus on the training and education of managers. All of these approaches aim to improve the quality of the health system and thereby the health of the population.

What happens when efforts are made to hire, retain and train district health systems managers?

It is difficult to draw any conclusions about the effects of these types of interventions as the review only found two relevant studies. In addition, the evidence that the review did identify was of low quality.

Training: The available evidence suggests that in‐service district manager training:

·       may lead to more knowledge about planning processes 

·       may lead to better monitoring and evaluation skills

None of the studies assessed the effects of district manager training on people’s health, on their access to or use of health care, or on the quality or efficiency of care.

Contracting‐in: The available evidence suggests that private contracts with international NGOs for district health systems management:

·       may not affect people’s illness reporting, diarrhea incidence or infant death

·       may increase the likelihood that a health facility is open 24 hours

·       may increase the availability of medical equipment and supplies

·       may increase people’s use of antenatal care and public facilities

None of the studies assessed the effects of contracting‐in district management on the quality or efficiency of health care, on job vacancy rates, or on district manager knowledge and skills.

Background

In many low‐ and middle‐income countries, responsibilities for important aspects of health systems operation are being reassigned from centralized health ministries to decentralized, local governing bodies, including district health teams (Bossert 1998; Bossert 2002; Grundy 2003). As a result, district health systems managers in these countries are playing an increasingly important role in determining the performance of primary healthcare service delivery systems. The term 'district health systems manager' specifically refers to persons who are responsible for overseeing the operations of the health system within a defined, subnational geographical area that is designated as a district. The primary responsibilities of a district health systems manager often include planning and budgeting, human resource management and service quality monitoring. However, the particular scope of these responsibilities and the autonomy given to district managers by health ministries vary depending on the country and province (Bossert 2002).

Description of the condition

Recent evidence suggests that poor performance by district managers may act as a key constraint to scaling up proven health interventions in low‐ and middle‐income countries (Mangham 2010). Indeed, poor district health systems management can pose a multitude of problems for effective health systems functioning, including poor co‐ordination of health service delivery, insufficient budgeting for affordable and effective health programs, and inadequate resources for health workers to provide high‐quality health care. Resource problems include medicine stock outs, delayed repair of broken equipment, and health worker absenteeism. Poor management may have substantial adverse effects on health workers' motivation, and likely contributes to attrition of health workers from the public to the private sector (Kruk 2008). This may reduce the quality of care at local health facilities and thus reduce demand for health services.

The challenges posed by poor district health systems management cannot be easily overcome, and are unlikely to be solved by simply increasing levels of funding (Hanson 2003). In response to inadequate health systems management, international donors have often chosen to bypass the health system entirely, opting rather to construct independent management structures and implement health programs vertically. However, evidence suggests that such vertical programs are often unsustainable, and may even serve to undermine publicly‐funded parts of health systems (Travis 2004). Interventions that increase the capacity of district health systems managers to effectively oversee the operations of the local health system may improve health systems outcomes.

Description of the intervention

The World Health Organization (WHO) has recently outlined a conceptual framework to assist researchers and public health practitioners in thinking about the challenges posed by poor district health systems management (World Health Organization 2007). The framework delineates four dimensions of planning that determine the effectiveness of local health systems managers: (1) there should be an adequate number of trained managers; (2) managers should have appropriate competencies; (3) there should be support systems in place to provide managers with the resources they need to carry out their responsibilities, including systems for planning and budgeting as well as human resources management; and (4) the environment in which the managers function should enable them to carry out their responsibilities. The 'environment' in (4) refers specifically to the district manager's relationship with the health system's central planning body, usually the Ministry of Health, and the decision space and autonomy that define that relationship. The WHO framework provides a useful starting point for thinking about the types of interventions that may strengthen district management capacity.

For this review, we restricted our interest to interventions falling into one of the first two dimensions outlined in the WHO framework, i.e., those related to hiring, retaining and training district health systems managers. We did not address dimensions (3) and (4) of the framework in this review because, to improve outcomes with regard to these dimensions, very different interventions will be required from those needed to improve outcomes related to the first two dimensions. In particular, interventions for achieving (1) or (2) focus on the health managers themselves, while interventions for achieving (3) and (4) focus on the system and the interaction between the system and the manager. Review of interventions for achieving (3) and (4) should be pursued independently. Examples of interventions related to hiring and retaining district managers include providing financial incentives (e.g., salary increases, bonuses) for service, or providing educational scholarships in return for a commitment to work in particular positions after graduation (so‐called conditional scholarships). Interventions aimed at training managers for the health system include university‐based professional education programs for future district health systems managers (McEwan 2001) and in‐service training programs for current managers (Omar 2009).

How the intervention might work

Interventions related to manager hiring practices may be important in determining the number of competent managers working in areas where they are most needed in low‐ and middle‐income countries. Increasing the size of manager salaries may strengthen the incentives for trained managers to work for ministries of health in rural areas. Further, retaining managers likely provides a consistency in health system operations that may have a positive impact on both service quality and population health outcomes.

The supply of competent health managers available to a particular country's health system has a direct impact on the processes for hiring and retaining these managers. As such, there are likely important areas of overlap between hiring interventions and interventions aimed at training managers. However, the interventions related to training are quite different from those related to hiring and retaining, and the manner in which they affect the outcomes of interest are distinct. In‐service training may serve to improve the skills of managers that were already competent or to expose them to new management techniques, which may in turn lead to improvements in operations efficiency and improvements in the health system's ability to meet the population's health needs given limited resources.

Why it is important to do this review

The international community is increasingly recognizing the importance of sufficient numbers of well‐trained health workers both at the front‐line and at the management level to achieving global health goals, including the Millennium Development Goals (Segall 2003; Travis 2004; Anand 2011). However, there have previously been no systematic reviews dedicated to assessing the evidence for interventions that aim to improve the management of district health systems in low‐ and middle‐income countries. Oliveira‐Cruz 2003 included some interventions at the district level in a review of health systems strengthening programs in general, and found some evidence to suggest that programs that provided training to district managers resulted in better planning and integration of programming. Several evaluations of interventions aimed at increasing district health systems management capacity have been conducted recently, though the quality of these studies varies (McEwan 2001; Jones 2009; Okamoto 2009). This systematic review provides, for the first time, a synthesis of the knowledge base on intervention effectiveness for studies of this kind.

There may be some overlap between the topic we investigate here, and other recent reviews. In particular, Grobler 2009 recently reviewed interventions for increasing the proportion of health professionals practising in rural areas. Similarly, Lagarde 2009 recently reviewed the impact of contracting‐out health services in low‐ and middle‐income countries. Finally, Bärnighausen 2009 has previously reviewed the effectiveness of financial incentives aimed at increasing rural service among health workers. However, the studies reviewed by these authors were focused on front‐line health workers rather than district managers. Thus, it is unlikely that the results of these three reviews are directly applicable to the topic we investigate here.

Objectives

The aim of this review is to assess the effectiveness of interventions to hire, retain and train district health systems managers in low‐ and middle‐income countries.

Methods

Criteria for considering studies for this review

Types of studies

• Randomized controlled trials (RCTs).

• Quasi‐randomized controlled trials (QRCTs).

• Controlled before‐and‐after studies (CBAs).

• Interrupted time series analyses (ITS).

For CBA and cluster randomized studies, we included only those studies that had at least two clusters per comparison group. For ITS studies we included only those that had at least three waves of data before and three waves of data after the intervention.

Types of participants

Our review only includes studies that took place in low‐ and middle‐income countries (World Bank 2010). The primary unit in which outcomes are assessed is the district health systems manager. As mentioned above, 'district health systems manager' specifically refers to persons who are responsible for overseeing the operations of the health system within a defined, subnational geographical area that is designated as a district. While managers often work as part of district health teams, we are primarily concerned here with the single individual that heads this team, as he or she is ultimately responsible for the team's performance. Further, district health teams often comprise persons with a highly heterogeneous mix of responsibilities and skill levels, rendering comparison of such teams across settings difficult. Finally, we did not include interventions solely aimed at managers of specific health facilities (e.g., district hospitals) in this review.

Types of interventions

The aforementioned WHO framework for improving district health systems management provides a starting point for considering interventions that are of interest to this review. We restricted this review to interventions that aimed to address one of the first two dimensions of health systems planning outlined by this framework: (1) ensuring that there are adequate numbers of managers through interventions related to hiring and retention, and (2) ensuring that the existing managers have appropriate competencies through interventions related to training.  Examples of such interventions include the following:

(1) Interventions related to hiring and retaining managers:

• Incentives to increase demand for and retain rural district management positions;

• Contracting‐in district management positions to private and nongovernment organizations.

(2) Interventions related to developing district manager competency:

• On‐the‐job training programs for district managers;

• Education programs to develop future district managers.

We included studies that investigated interventions of interest in combination with additional interventions that were not directly related to the research question, provided that the authors presented an independent assessment of the effect of the intervention of interest. We excluded studies of interventions that were aimed at district health systems managers and district health teams, unless the effects of these interventions were assessed independently for the manager. We included studies that compared interventions of interest to other interventions, as well as studies that compared interventions of interest to no intervention.

Types of outcome measures

Primary outcomes

(1) Health systems outcomes, such as:

• population health outcomes;

• access to health care;

• utilization of health care;

• quality of health care;

• efficiency of health care;

• equity of health care.

(2) Operational outcomes, such as:

• district manager job‐posting vacancy rates;

• district managers' skills measured within a practice environment.

Secondary outcomes

(1)  Patient satisfaction with the health system;

(2)  District manager knowledge and skills measured outside of practice.

We included all studies with eligible designs, participants and interventions in this review, regardless of whether primary or secondary outcomes were reported.

Search methods for identification of studies

Electronic searches

We searched the following electronic bibliographic databases in December 2011:

(1)  The Cochrane Central Register of Controlled Trials (CENTRAL)

(2)  Cochrane Effective Practice and Organisation of Care Group (EPOC) Specialized Register

(3)  PubMed (from 1950)

(4)  EMBASE (from 1980)

(5)  Cumulative Index to Nursing & Allied Health Literature (CINAHL) (from 1982)

(6)  LILACS (from 1982)

(7) ISI Science Citation Index

(8) Social Science Citation Index

The search strategy for these databases utilized methodological components of the EPOC search strategy combined with selected MeSH terms and free text terms. We selected the included terms to capture, as exhaustively as possible, all articles related to the topic of interest. We used the search strategy described in Appendix 1 to search PubMed. This strategy was translated into the other databases' vocabulary as appropriate.

Searching other resources

We searched grey literature in order to identify unpublished evaluations of interventions that aimed to improve district management. Other searches included: WHOLIS, World Bank and Google Scholar. Finally, we scanned reference lists of key papers for additional articles.

Data collection and analysis

Selection of studies

We selected studies into the review using a three‐stage process. In the first stage, we examined all records returned by the above search strategy and identified and removed duplicate studies. In the second stage of the review process, we scrutinized titles and abstracts to determine if identified articles met our inclusion criteria. Only those articles that clearly did not meet inclusion criteria based on title and abstract were excluded in this second stage of the review. Finally, in the third stage, we read the full text of all remaining articles to determine if they met all inclusion criteria. Both review authors conducted each of the three stages of the review process. We addressed disagreements through discussion or by consulting a member of the EPOC group. We recorded the reason for study exclusion for all studies that passed the second stage of the review, i.e., those studies for which we read the full text. The Characteristics of excluded studies table provides the reasons for exclusion.

Data extraction and management

Both review authors independently extracted the following elements from each included study:

(1) Study information: name of the first author and date of publication;
 (2) Description of intervention: type of intervention, intervention group, control group, broader context if mentioned;
 (3) Description of study design: study type, units studied, waves of data, cases per control;
 (4) Results: outcomes measured, measures of effect.

Assessment of risk of bias in included studies

Both review authors independently assessed risk of bias for each included study using the standard criteria described by the EPOC Group (EPOC 2008). We report on the risk of bias assessment in the section Characteristics of included studies. We did not exclude studies on the grounds of their risk of bias.

Measures of treatment effect

We report measures of effect in the same way that the investigators have reported them. We did not conduct meta‐analysis or subgroup analysis, because we could not pool study results due to the varied nature of the interventions included in this review.

Results

Description of studies

Results of the search

Our search produced a total of 4576 records. We considered articles for inclusion in the review in three stages. In the first stage of the review, we reviewed titles and abstracts to determine duplicates. In total, 528 articles were found to be duplicates and thus were excluded from the review. In the second stage of the review, we read the remaining 4048 titles and abstracts, to determine if they met our inclusion criteria. Only those articles that clearly did not meet the inclusion criteria based on title and abstract were excluded in this second stage of the review. In total, we excluded 3988 articles from the review in this second stage. In the third stage of the review, we read full‐text copies of the remaining 60 articles. A search of the grey literature identified two additional relevant articles for full‐text review. Finally, we searched the reference lists of these 62 articles, and identified five additional articles as potentially relevant. We read the full text of these additional five articles. Four full‐text articles were translated to English from another language. Of the 67 articles reviewed in full, two met all inclusion criteria and were included in the review.

Included studies

Two studies (Diaz‐Monsalve 2004; Bloom 2006) met all inclusion criteria and were included in the review. Diaz‐Monsalve 2004 investigated a training intervention and employed a CBA design. District managers in the intervention and control groups were surveyed at baseline and then again at follow‐up, after intervention managers took part in an 18‐month training program that included five training courses lasting five days each. The training courses focused on developing managers’ skills related to “needs assessment, local planning, monitoring and evaluation, drug management, risk approach, quality assurance, resource management, program management, information systems, and community participation.” The study population came from three countries: Mexico, Colombia and El Salvador. In total, Diaz‐Monsalve 2004 reported at baseline on 85 managers who received the intervention and 71 control managers. At follow‐up, data were available for 74 managers who had received the intervention and 66 control managers. The primary outcomes reported in the study were managers’ competencies, including: knowledge, job performance, co‐ordination and communication skills, and use of monitoring and evaluation methods. These outcomes were measured outside of practice.

Bloom 2006 investigated a hiring‐related intervention package and employed an RCT design. Twelve districts in Cambodia were randomized to be a control or to receive one of two treatments: contracting‐in or contracting‐out for district management. Management of control districts remained the responsibility of managers employed directly by the Ministry of Health, as was the norm in Cambodia at the time of the study. As part of the contracting‐in treatment, international NGOs were selected through a competitive bidding process to be district health systems managers and were hired through private contracts to work within the Ministry of Health system. Alternatively, as part of the contracting‐out treatment, contractors had complete responsibility for all aspects of health service delivery in the district:

“Contracting‐in districts were expected to work within the existing government system for procurement of drugs, equipment, and supplies. Their operating expenses were financed through the government budget in the same manner as ordinary districts. They were required to use existing Ministry of Health personnel; they could request transfers of personnel but not hire or fire. Contracting‐out district management had pretty much full authority for and responsibility over their districts. They were allowed to hire and fire staff, could bring in health workers from other parts of the country, and were responsible for their own procurement of drugs, supplies, and equipment.”

In this review, we considered only the results of the contracting‐in treatment as relevant to our original research question. This is because the contracting‐in treatment was more closely constrained to hiring of district managers, while the contracting‐out treatment went well beyond hiring, affecting the support systems that managers work in and their autonomy from the Ministry of Health. Indeed, the contracting‐in intervention described in Bloom 2006 is consistent with a growing practice by donors to support the hiring and retention of qualified managers on behalf of governments in developing countries. As the authors note, the contracting‐in treatment districts differed from control districts in a few important ways beyond just manager hiring (Table 3). In particular, public spending in contracting‐in districts was 60% higher than in control districts during the study period. Three districts were successfully randomized to receive the contracting‐in treatment, while four districts served as controls. Bloom 2006 reported on a variety of outcomes measured at the health facility level and at the individual level, pre‐ and post‐treatment. Indicators included health facility staffing and supervision, maternal and child health service use (e.g., immunization, antenatal care), and population health outcomes (e.g., diarrhea incidence). The authors do not distinguish the impacts of the various aspects of the contracting‐in intervention in their analysis. For this reason, we consider the contracting‐in treatment described in Bloom 2006 to be a hiring‐related intervention package rather than a distinct hiring intervention.

1. Characteristics of contracting‐in versus control districts in Bloom (2006).

Characteristic	Contracting‐in districts	Control districts
Private versus public management	Management by international NGOs	Management by the government
Competitive bidding	Competitive bidding based in part on prior experience of contractor and quality of key staff	None
Contracts	Contract targeting improvements; subcontracts implemented	No contracts
Hiring of district managers	Hiring of expatriates as district managers (presumably with higher salaries)	Political allegiance (corruption) played a role, otherwise not described
Incentives	Performance‐based incentives	Not specified
User fees	Formalization and monitoring of user fees; not a statistically significant difference in the amount collected	Informal fees
Public spending	Total public spending 60% higher
Consulting services and training	Not specified	Health care management consulting services and management training
Supplemental funding	Eligible to receive an operating supplement after submitting an acceptable plan

Excluded studies

As described in Results of the search, we assessed 67 studies by reading the full‐text articles. These studies could not be excluded based on title and abstract review, either because they appeared to meet the inclusion criteria or because the title and abstract did not include enough information to make an accurate assessment as to inclusion or exclusion. Of these 67 studies, we excluded 65 from the review based on a reading of the full text. Many of these studies (46%) were excluded because they did not target the Types of participants outlined above. Specifically, many targeted health facility managers or clinical professionals rather than district managers. An additional 46% of these studies were excluded because they did not employ one of the four Types of studies outlined above. Indeed, several studies that would have otherwise warranted inclusion in the review were excluded because they employed case study methodology.

Risk of bias in included studies

In general, the studies reported on in Diaz‐Monsalve 2004 and Bloom 2006 have a high risk of bias. For Diaz‐Monsalve 2004, there are three main sources of high risk of bias: selection bias, attrition bias, and detection bias. Selection bias is a concern because the author does not explicitly state how managers were selected into the study, nor how managers were selected to receive training. Attrition bias is a concern because, in the control group, the managers reported on at baseline were not the same managers reported on at follow‐up. Furthermore, there was a substantial loss to follow‐up in the intervention group in El Salvador. Finally, the main outcome measure was a composite measure that largely relied on self report, so that detection bias may be an issue.

For Bloom 2006, there are two main sources of high risk of bias: selection bias and bias due to a potentially modifying uncontrolled exposure. Selection bias is a concern because the authors reported that, of the four districts randomized to receive the contracting‐in treatment, one was unsuccessful in establishing a working contractual relationship. This district remained under the management of the Ministry of Health. To deal with this, the authors conducted two types of analyses: intention‐to‐treat (ITT) analysis to determine the effect of being randomized to the contracting‐in treatment group, and treatment‐on‐treated (TOT) analysis to determine the local average treatment effect, i.e., the effect of actually receiving the contracting‐in treatment. The authors found similar results in the two analyses, with the effect sizes of the ITT analysis smaller than the effect sizes of the TOT analysis, as expected. However, TOT estimates reported by Bloom 2006 should be interpreted as resulting from a non‐randomized controlled trial, a lower grade of evidence compared to an RCT.

Furthermore, as discussed above, the contracting‐in intervention described by Bloom 2006 should be seen as a hiring‐related intervention package rather than a distinct hiring intervention. In particular, the fact that contracting‐in districts had 60% greater public spending compared to the control districts is likely a source of bias when assessing the independent impact of the hiring aspect of the contracting‐in intervention package. Specifically, the increased spending likely contributed to inflated impact estimates compared to what would have been expected from a simple hiring intervention.

Effects of interventions

See: Table 1; Table 2

for the main comparison.

Intervention: Training
Outcomes	Relative effect (95% CI)	Quality of the evidence (GRADE)	Comments
Population health outcomes	No evidence
Access to health care	No evidence
Utilization of health care	No evidence
Quality of health care	No evidence
Efficiency of health care	No evidence
Equity of health care	No evidence
District manager job‐posting vacancy rates	No evidence
District managers’ knowledge measured within a practice environment	No evidence
District managers’ skills measured within a practice environment	No evidence
District managers’ knowledge measured outside of practice	In one study (CBA), in‐service district manager training significantly increased knowledge of planning processes.	⊕⊕⊝⊝ low	1 CBA; downgraded due to serious risk of bias; upgraded due to large effect.
District managers’ skills measured outside of practice	In one study (CBA), in‐service district manager training significantly increased monitoring and evaluation skills.	⊕⊕⊝⊝ low	1 CBA; downgraded due to serious risk of bias; upgraded due to large effect.
GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate.

2.

Intervention: Hiring
Outcomes	Relative effect (95% CI)	No of Participants (studies)	No of Participants (studies)	Quality of the evidence (GRADE)
Population health outcomes	In one study (RCT), contracting‐in district management did not significantly affect illness reporting, diarrhea incidence or the probability of infant death.	1 RCT	⊕⊕⊝⊝ low	Downgraded due to serious risk of bias; downgraded due to serious indirectness
Access to health care	In one study (RCT), contracting‐in district management increased the probability that a health facility would be open 24‐hours by 83% (95% CI: 61 to 105). Further, contracting‐in district management increased the probability that medical equipment and supplies would be available.	1 RCT	⊕⊕⊝⊝ low	Downgraded due to serious risk of bias; downgraded due to serious indirectness
Utilization of health care	In one study (RCT), contracting‐in district management increased use of antenatal care by 28% (95% CI: 16 to 40) and use of public facilities by 14% (95% CI: 6 to 22).	1 RCT	⊕⊕⊝⊝ low	Downgraded due to serious risk of bias; downgraded due to serious indirectness
Quality of health care	No evidence
Efficiency of health care	No evidence
Equity of health care	No evidence
District manager job‐posting vacancy rates	No evidence
District managers’ knowledge measured within a practice environment	No evidence
District managers’ skills measured within a practice environment	No evidence
GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate.

Hiring

Bloom 2006 found that contracting‐in district management had a statistically significant impact on various outcomes.  In a TOT analysis at the level of the health facility, the authors found that contracting‐in increased the probability that a health facility would be open 24 hours by 83% (95% confidence interval (CI): 61 to 105) and that medical equipment and supplies would be available. In an ITT analysis at the level of the individual patient, they found that contracting‐in increased use of antenatal care by 28% (95% CI: 16 to 40) and use of public facilities by 14% (95% CI: 6 to 22). In a TOT analysis of the same outcomes, the authors found that contracting‐in increased use of antenatal care by 36% (95% CI: 20 to 52) and use of public facilities by 18% (95% CI: 10 to 26). The authors did not find a significant effect of the contracting‐in treatment on population health outcomes. Finally, the authors found no effect of contracting‐in on healthcare spending at the individual level.

Training

Diaz‐Monsalve 2004 found that a manager training program significantly reduced the relative risk of having management deficiencies, such as lack of knowledge of planning processes and insufficient monitoring and evaluation skills. Indeed, in Mexico, managers in the control group had 8.3 (95% CI: 4.7 to 14.6) times more management deficiencies than managers that underwent the training program. In Colombia, managers in the control group had 3.6 (95% CI: 2.5 to 5.1) times more deficiencies, and in El Salvador managers in the control group had 2.4 (95% CI: 1.7 to 3.4) times more deficiencies than managers attending the training program. The managers in the control group did not change in their managerial performance between baseline and follow‐up.

Discussion

Summary of main results

We conducted a systematic review of interventions for hiring, retaining and training district health systems managers in low‐ and middle‐income countries. After reviewing more than 4500 articles, we found two studies that met all inclusion criteria. Bloom 2006 focused on one method of hiring managers, contracting‐in, and employed a randomized controlled trial (RCT) design. The study produced low quality evidence that contracting‐in improves aspects of health facility operation and increased utilization of certain health services, including antenatal care. The study did not find a significant relationship between contacting‐in for district health systems management and population health outcomes. However, the contracting‐in treatment that the authors described included interventions beyond hiring, including increased public spending, and this may have biased the results. Diaz‐Monsalve 2004 employed a controlled before‐and‐after (CBA) design to investigate the effect of a district health systems manager training program on manager competencies. The author presented low quality evidence of significant improvements in manager performance after training compared to before, in Mexico, Colombia, and El Salvador. There were no studies on interventions for retaining district health systems managers that met all inclusion criteria.

Overall completeness and applicability of evidence

The evidence presented in this review is of low quality and is limited in its applicability for policy makers. Bloom 2006 provided information on one method of hiring, contracting‐in. However, more rigorous research is needed to confirm the positive findings that the authors report. As with all field experiments, there may have been unmeasured factors that influenced the outcomes and threaten the external validity of the findings (Harrison 2004). For example, the local organizational capacity in which the study managers worked may have modified the observed effect sizes. There is no direct evidence to suggest that the external validity of Bloom 2006 should be questioned, but additional studies are necessary to make an informed judgment on this point. Further, the authors found no effect of contracting‐in on health outcomes, an ultimate goal of any health system. This may have been because the time frame of the study was not long enough. Additional evidence over a longer period would be valuable. Finally, there were no studies that investigated other methods of hiring district health systems managers and met our inclusion criteria.

There were no studies that investigated interventions for retaining managers. This is somewhat surprising, given that in recent years increased attention has been given to strategies for retaining clinical health workers (Willis‐Shattuck 2008). Given the importance of managers in determining health systems performance, this work should be expanded to consider interventions for retaining district health systems managers.

Diaz‐Monsalve 2004 analyzed the effects of one specific intervention for training district health systems managers. The 18‐month, intermittent training program the study investigated is one of many potential program designs for manager training, and the study provides limited information to indicate how different components of the training may have had differential effects. This limits the ability to predict how slight differences in program design in alternative settings might change effect estimates. Further, Diaz‐Monsalve 2004 reported only on manager competencies measured outside of the work environment. The study did not provide any information on how these competencies translated to the work environment or on how they ultimately determined the other primary outcomes of interest for this review, including quality of health care, utilization patterns and population health outcomes.

The two interventions investigated in the articles that were included in this review represent a small section of the full range of interventions that may be used (and currently are being used) by policy makers in low‐ and middle‐income countries for hiring, retaining and training district health systems managers. Table 4 provides an overview of potential interventions.

2. Potential interventions for future research.

Types of interventions	Primary outcomes			Secondary outcomes
	Population health	Access to and utilization of health care	Quality and efficiency of health care	Patient satisfaction	District manager knowledge and skill
Hiring and retaining
Direct financial incentives (e.g., signing bonuses)	‐	‐	‐	‐	‐
Bonding (e.g., service‐requiring scholarships)	‐	‐	‐	‐	‐
Professional development opportunities (e.g., future program placement priorities)	‐	‐	‐	‐	‐
Improvements to the work environment (e.g., facility infrastructure)	‐	‐	‐	‐	‐
Training
Off‐site in‐service training	‐	‐	1 study Low quality evidence	‐	1 study Low quality evidence
On‐site in‐service training	‐	‐	‐	‐	‐
Pre‐career education	‐	‐	‐	‐	‐
Mid‐career education	‐	‐	‐	‐	‐
Packages
Contracting‐in	1 study Low quality evidence	1 study Low quality evidence	‐	1 study Low quality evidence	‐

Various strategies have been pursued by ministries of health in low‐ and middle‐income countries to hire health workers into service in the public sector. For example, bonding programs, or service‐requirement scholarships and loans, have been used extensively, in countries such as South Africa (Bärnighausen 2009). Furthermore, direct financial incentives (e.g., signing bonuses) have also been used to attract health workers to take up postings at public facilities. However, research on the effectiveness of these strategies has been limited to clinical health workers.

Health worker recruitment and retention in developing countries has received increased attention in recent years. In 2010, WHO published a report with a set of policy recommendations for health worker recruitment and retention (World Health Organization 2010). The report identified 16 specific intervention areas, including strategies for developing education programs that complement recruitment efforts, suggestions for relevant government regulations (e.g., compulsory service requirements), guidelines for financial incentive programs, and examples of personal and professional support systems (e.g., career advancement programs). The report explicitly stated that the recommendations were relevant for health systems managers as well as clinical health workers. However, as evidenced by the results of this review, the vast majority of recent research on the effectiveness of the various recommendations outlined by the WHO report has been largely limited to studies on clinical health workers.

There have been recent research efforts aimed at understanding the effects of district manager training interventions on health systems and population health outcomes. However, most of the relevant studies have employed observational study designs with a high risk of bias, and thus were excluded from this review. Conn 1996 investigated a manager training and support program in The Gambia using a case study design. The author found that in‐service training coupled with continuous on‐site support over the course of 18 months led to improvements in the quality of health team planning and co‐ordination as well as resource management. Similarly, Mansour 2010 conducted a case study of a manager training program in Egypt where participants attended four short off‐site workshops where they learned “simple planning and performance improvement tools to identify and address challenges”. The author concluded that after the training participants became more active in identifying and addressing management challenges and performed better in their planning responsibilities. While the risk of bias in observational studies such as these is likely high (part of the reason for excluding them from this review), the information contained in these and similar articles is valuable, particularly for researchers.

Authors' conclusions

Implications for practice.

We need more evidence before we can recommend specific interventions for hiring, retaining and training district health systems managers. For hiring, contracting‐in may be an effective way to improve health facility performance and increase utilization. However, before pursuing contracting‐in strategies, policy makers should understand how aspects of their local setting may differ from those described in Cambodia by Bloom 2006, and how those differences may influence the outcomes of interest. For retaining district managers, there is no current evidence to suggest a best practice. Finally, training programs may be effective at improving competencies, but more information is needed before a definitive statement can be made regarding how training programs should be structured. Furthermore, we found no strong evidence to suggest that training programs for district health systems managers improve health facility performance or population health outcomes.

Implications for research.

Future research should focus on understanding how contracting‐in for district health systems managers might affect health service utilization and population health outcomes in diverse settings. Furthermore, research is needed to understand how alternative hiring interventions may compare to contracting‐in. Research into interventions for retaining district health systems managers has been absent from recent efforts to understand retention strategies for other health worker cadres in low‐ and middle‐income countries. Health worker retention research should be expanded to include investigations of district manager retention. Finally, research on district manager training has been widespread. However, much of this work has employed case study methodology. More rigorous methods, including randomized controlled trials (RCTs), should be used in future research on this topic. The nature of the interventions investigated here are particularly amenable to investigation with RCT methodology. Future studies should randomize district managers to promising hiring, retention and training interventions, and measure important outcomes of interest, including district manager job posting vacancy rates, in‐service manager performance and measures of population health.

Appendices

Appendix 1. PubMed search strategy

#1            (((Administrative Personnel [Mesh]) OR (Institutional Management Teams [Mesh:NoExp]) OR (Manager [tiab]) OR (Managers [tiab]) OR (Administrator [tiab]) OR (Administrators [tiab]) OR (Management personnel [tiab]) OR (Management staff [tiab]) OR (Management employee [tiab]) OR (Management employees [tiab]) OR (Management position [tiab]) OR (Management positions [tiab]) OR (Management system [tiab]) OR (Management systems [tiab]) OR (Management support [tiab]) OR (Administrative personnel [tiab]) OR (Administrative staff [tiab]) OR (Administrative employee [tiab]) OR (Administrative employees [tiab]) OR (Administrative position [tiab]) OR (Administrative positions [tiab]) OR (Administrative system [tiab]) OR (Administrative systems [tiab]) OR (Administrative support [tiab])) AND ((Provinc* [tiab]) OR (District* [tiab]) OR (Local [tiab]) OR (decentral*[tiab])) OR ((Decentralized management [tiab]) OR (Decentralised management [tiab]) OR (Decentralized health system* [tiab]) OR (Decentralised health system* [tiab]) OR (Decentralized health service* [tiab]) OR (Decentralised health service* [tiab]) OR (Local health service* [tiab]) OR (Local health system* [tiab]) OR (Local management [tiab]) OR (Local level management [tiab]) OR (Local health team* [tiab]) OR (District health officer* [tiab]) OR (District officer* [tiab]) OR (District management [tiab]) OR (District level management [tiab]) OR (District health team* [tiab]) OR (District health system* [tiab]) OR (District level health system* [tiab])) OR (((Hire [tiab]) OR (Hiring [tiab]) OR (Retension [tiab]) OR (Retain [tiab]) OR (Brain drain [tiab]) OR (Contract services [mesh]) OR (Contract services [tiab]) OR (Contracting [tiab]) OR (Management training [tiab]) OR (Staff development [mesh]) OR (Staff development [tiab]) OR ((Performance [tiab]) AND (Pay* [tiab])) OR ((Human resource* [tiab]) AND (manag* [tiab])) OR (Health manpower [mesh]) OR (Health manpower [tiab])) AND ((Organization and administration [mesh]) OR (manag* [tiab]) OR (admin* [tiab])))) AND ((Randomized controlled trial [pt]) OR (Random* [tiab]) OR (Intervention* [tiab]) OR (Control [tiab]) OR (Controlled [tiab]) OR (Evaluat* [tiab]))

 

#2            LMIC Filters 2010, on http://epocoslo.cochrane.org/

 

#3            #1 AND #2

Appendix 2. EMBASE search strategy

#1            (((‘Administrative Personnel’/exp OR ‘Institutional Management Teams’/de OR ‘Manager’:ti,ab OR ‘Managers’:ti,ab OR ‘Administrator’:ti,ab OR ‘Administrators’:ti,ab OR ‘Management personnel’:ti,ab OR ‘Management staff’:ti,ab OR ‘Management employee’:ti,ab OR ‘Management employees’:ti,ab OR ‘Management position’:ti,ab OR ‘Management positions’:ti,ab OR ‘Management system’:ti,ab OR ‘Management systems’:ti,ab OR ‘Management support’:ti,ab OR ‘Administrative personnel’:ti,ab OR ‘Administrative staff’:ti,ab OR ‘Administrative employee’:ti,ab OR ‘Administrative employees’:ti,ab OR ‘Administrative position’:ti,ab OR ‘Administrative positions’:ti,ab OR ‘Administrative system’:ti,ab OR ‘Administrative systems’:ti,ab OR ‘Administrative support’:ti,ab) AND (Provinc*:ti,ab OR District*:ti,ab OR ‘Local’:ti,ab OR decentral*:ti,ab)) OR (‘Decentralized management’:ti,ab OR ‘Decentralised management’:ti,ab OR Decentralized health system*:ti,ab OR Decentralised health system*:ti,ab OR Decentralized health service*:ti,ab OR Decentralised health service*:ti,ab OR Local health service*:ti,ab OR Local health system*:ti,ab OR ‘Local management’:ti,ab OR ‘Local level management’:ti,ab OR Local health team*:ti,ab OR District health officer*:ti,ab OR District officer*:ti,ab OR ‘District management’:ti,ab OR ‘District level management’:ti,ab OR District health team*:ti,ab OR District health system*:ti,ab OR District level health system*:ti,ab) OR ((‘Hire’:ti,ab OR ‘Hiring’:ti,ab OR ‘Retension’:ti,ab OR ‘Retain’:ti,ab OR ‘Brain drain’:ti,ab OR ‘Contract services’/exp OR ‘Contract services’:ti,ab OR ‘Contracting’:ti,ab OR ‘Management training’:ti,ab OR ‘Staff development’/exp OR ‘Staff development’:ti,ab OR (‘Performance’:ti,ab AND Pay*:ti,ab) OR (Human resource*:ti,ab AND manag*:ti,ab) OR ‘Health manpower’/exp OR ‘Health manpower’:ti,ab) AND (‘Organization and administration’/exp OR manag*:ti,ab OR admin*:ti,ab))) AND (‘Randomized controlled trial’:pt OR Random*:ti,ab OR Intervention*:ti,ab OR ‘Control’:ti,ab OR ‘Controlled’:ti,ab OR Evaluat*:ti,ab)

 

#2            LMIC Filters 2010, on http://epocoslo.cochrane.org/

 

#3            #1 AND #2

 

Appendix 3. CINAHL search strategy

#1            (((TI ("manager" OR "managers" OR "administrator" OR "administrators" OR "management personnel" OR "management staff" OR "management employee" OR "management employees" OR “management position” OR “management positions” OR “management system” OR “management systems” OR “management support” OR “administrative personnel” OR “administrative staff” OR “administrative employee” OR “administrative employees” OR “administrative position” OR “administrative positions” OR “administrative system” OR “administrative systems” OR “administrative support”) OR AB ("manager" OR "managers" OR "administrator" OR "administrators" OR "management personnel" OR "management staff" OR "management employee" OR "management employees" OR “management position” OR “management positions” OR “management system” OR “management systems” OR “management support” OR “administrative personnel” OR “administrative staff” OR “administrative employee” OR “administrative employees” OR “administrative position” OR “administrative positions” OR “administrative system” OR “administrative systems” OR “administrative support”)) AND (TI (provinc* OR district* OR “local” OR decentral*) OR AB (provinc* OR district* OR “local” OR decentral*))) OR (TI (“decentralized management” OR “decentralised management” OR decentralized health system* OR decentralised health system* OR decentralized health service* OR decentralised health service* OR local health service* OR local health system* OR “local management” OR “local level management” OR local health team* OR district health officer* OR district officer* OR “district management” OR “district level management” OR district health team* OR district health system* OR district level health system*) OR AB (“decentralized management” OR “decentralised management” OR decentralized health system* OR decentralised health system* OR decentralized health service* OR decentralised health service* OR local health service* OR local health system* OR “local management” OR “local level management” OR local health team* OR district health officer* OR district officer* OR “district management” OR “district level management” OR district health team* OR district health system* OR district level health system*)) OR ((TI (“hire” OR “hiring” OR “retension” OR “retain” OR “brain drain” OR “contract services” OR “contracting” OR “management training” OR “staff development” OR (“Performance” AND pay*) OR (human resource* AND manag*) OR “health manpower”) OR AB (“hire” OR “hiring” OR “retension” OR “retain” OR “brain drain” OR “contract services” OR “contracting” OR “management training” OR “staff development” OR (“Performance” AND pay*) OR (human resource* AND manag*) OR “health manpower”)) AND (manag* OR admin*)))

 

#2            (PT (“randomized controlled trial”) OR TI (random* OR intervention* OR “control” OR “controlled” OR evaluat*) OR AB (random* OR intervention* OR “control” OR “controlled” OR evaluat*))

 

#3            LMIC Filters 2010, on http://epocoslo.cochrane.org/

 

#4            #1 AND #2 AND #3

Appendix 4. Planned methods not used in review

There were several subsections of the Methods presented in the protocol that were not relevant for the review because the included studies did not warrant their use. In this appendix, we reprint the text from those protocol subsections. These methods, though unused in the current review, may provide guidance for future review updates.

Measures of treatment effect 
 We will record and report measures of effect in the same way that the investigators have reported them. If possible, we will standardize measures of effect as mean differences in natural units or using a standardized scale to allow for comparisons across studies where between‐study comparisons are relevant. For ITS analyses, we will report the immediate effect after the specified transition period, at one year and, if reported, after longer periods of follow‐up.

Unit of analysis issues 
 For cluster randomized trials and controlled before‐after studies we will assess whether an appropriate analysis has been done that adjusts for clustering in calculating measures of precision. If this has not been done, we will attempt to extract necessary data (intracluster correlation coefficients) or obtain these from the investigators, and re‐analyze the results. If this is not possible, we will report point estimates, but not the reported measures of precision.

Dealing with missing data 
 We will attempt to obtain missing data from the investigators. If this is not possible we will report the data as missing and will not attempt to impute values. In cases where authors have reported on ITS studies, but have conducted inappropriate analyses on their data, we will re‐analyze these data, provided the necessary values are published.

Assessment of heterogeneity 
 We will group studies with similarities in interventions in accordance with the categories defined in Types of interventions.

Assessment of reporting biases 
 We will assess selective outcome reporting as a risk of bias criterion, as described in the Cochrane Handbook for Systematic Reviews of Interventions. We will assess the risk of publication bias qualitatively based on the characteristics of the included studies. For example, if we identify only small studies that indicate effects in favor of the interventions, this would raise our concern about the risk of publication bias.

Data synthesis 
 We anticipate that the breadth of the interventions included in this review will be great, making it difficult to combine all intervention results in statistical analysis. However, if the results of subsets of studies can be pooled, we will do so in meta‐analyses. We will present all results in a table showing the extracted data. We will present a qualitative assessment based on this table. Due to inconsistencies in the definitions of outcome measures in the health systems field, we anticipate that it will be necessary to employ a certain degree of discretion in grouping study outcome definitions within review outcome categories as defined in Types of outcome measures.We will deal with such discrepancies as they occur, and will consult with a member of the EPOC Group when necessary.

Subgroup analysis and investigation of heterogeneity 
 Within each group of studies that evaluate similar interventions we will compare the impacts that are reported for similar outcome measures. We will do this in a qualitative manner. In addition, wherever possible, we will explore heterogeneity quantitatively in meta‐analysis using the I² statistic. I² indicates the percent variability due to between‐study variability as opposed to within‐study variability.

Sensitivity analysis 
 If included studies allow for a meta‐analysis, we will conduct sensitivity analyses on all pooled effect estimates. In particular, we will investigate the effects of different statistical approaches, e.g. fixed effects versus random effects models, and we will also investigate the effects of including and excluding studies where there is some ambiguity as to whether they meet the inclusion criteria.

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Bloom 2006.

Methods	Randomized Controlled Trial
Participants	Countries: Cambodia Participants: District health systems managers. Sample: Three districts were randomized to receive a contracting‐in treatment while four districts served as controls.
Interventions	In districts receiving the contracting‐in treatment, district managers were hired through private contracts to work within the Ministry of Health system. Management of control districts remained the responsibility of managers employed directly by the Ministry of Health
Outcomes	Health facility staffing and supervision, maternal and child health service use (e.g., immunization, antenatal care), and population health outcomes (e.g., diarrhea incidence)
Notes	None
Risk of bias
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	Randomization was quasi‐stratified by province; the provincial health systems director randomly drew districts to assign them to the three experimental groups.
Allocation concealment (selection bias)	High risk	Allocation performed centrally at start of study. However, one district randomized to receive the contracting‐in treatment was unsuccessful in establishing a working contractual relationship.
Baseline outcome measurements	Low risk	At baseline, one outcome differed between treatment and control (at 5% significance), but that was expected by chance with 22 outcomes.
Baseline characteristic measurements	Unclear risk	Not reported.
Incomplete outcome data (attrition bias) All outcomes	Low risk	No missing data.
Blinding (performance bias and detection bias) All outcomes	Low risk	Primary outcomes were objective.
Contamination of experimental groups	Low risk	Unlikely that control group received the intervention.
Selective reporting (reporting bias)	Low risk	All relevant outcomes in the methods section are reported in the results section.
Other bias	High risk	Effect of contracting‐in may be confounded by the 60% higher public spending for health care in the contracting‐in districts compared to the control districts.

Diaz‐Monsalve 2004.

Methods	Controlled before‐and‐after study
Participants	Countries: Mexico, Colombia, El Salvador. Participants: District health systems managers. Sample at baseline: 85 intervention managers; 71 control managers. Sample at follow‐up: 74 intervention managers; 66 control managers.
Interventions	18‐month manager training program.
Outcomes	Managers’ competencies, including knowledge, job performance, co‐ordination and communication skills, and use of monitoring and evaluation methods. Measured outside of practice.
Notes	None
Risk of bias
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	No randomization for CBA design.
Allocation concealment (selection bias)	High risk	It is not clear how managers were allocated to treatment and control groups.
Baseline outcome measurements	Low risk	Similar competencies between treatment and control groups at baseline.
Baseline characteristic measurements	Low risk	Similar demographic characteristics between treatment and control groups.
Incomplete outcome data (attrition bias) All outcomes	High risk	Control managers interviewed at baseline not the same as control managers interviewed at follow‐up; substantial loss to follow‐up in the intervention group in El Salvador.
Blinding (performance bias and detection bias) All outcomes	High risk	Main outcome measure largely relied on self report.
Contamination of experimental groups	Low risk	Unlikely that control group received the intervention.
Selective reporting (reporting bias)	Low risk	All relevant outcomes in the methods section are reported in the results section.
Other bias	Low risk	None.

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Agyepong 1999	Not an RCT, QRCT, CBA or ITS; a qualitative description.
Ambegaokar 2004	Not an RCT, QRCT, CBA or ITS; a case study.
Ayaya 2007	Not an RCT, QRCT, CBA or ITS; a cross‐sectional survey.
Basri 2009	Inappropriate target population; targeted TB program managers.
Briggs 2010	Not an RCT, QRCT, CBA or ITS; a program description.
Byskov 2009	Not an RCT, QRCT, CBA or ITS; a case study.
Bärnighausen 2009	Not an RCT, QRCT, CBA or ITS; a review paper.
Clark 2001	Not an RCT, QRCT, CBA or ITS; a program description.
Comolet 1997	Not an RCT, QRCT, CBA or ITS; an uncontrolled before‐and‐after study.
Conn 1996	Not an RCT, QRCT, CBA or ITS; a case study.
De Brouwere 1996	Not an RCT, QRCT, CBA or ITS; a program description.
Dieleman 2009	Inappropriate target population; targeted clinical professionals.
Djibuti 2009	Inappropriate target population; targeted immunization program managers.
Egger 2007	Not an RCT, QRCT, CBA or ITS; a case study.
Enkhtuya 2009	Inappropriate target population; targeted immunization program managers.
Espino 2004	Inappropriate target population; targeted clinical professionals.
Fonn 2011	Not an RCT, QRCT, CBA or ITS; a qualitative description.
Grobler 2009	Inappropriate target population; targeted clinical professionals.
Israr 2006	Inappropriate target population; targeted program managers.
Jack 2003	Inappropriate target population; targeted health facility managers.
Jain 1999	Inappropriate target population; targeted family planning officers.
Jones 2008	Inappropriate intervention; building a support system.
Kahindo 2011	Not an RCT, QRCT, CBA or ITS; a case study.
Kalita 2009	Not an RCT, QRCT, CBA or ITS; a program description.
Koehlmoos 2009	Inappropriate target population; targeted health facility managers.
Krishnamurthy 2007	Inappropriate target population; targeted clinical professionals (community health workers).
Lagarde 2009	Inappropriate target population; targeted health facility managers.
Liu 2007	Inappropriate target population; targeted health facility managers.
Liu 2008	Inappropriate target population; targeted health facility managers.
Loevinsohn 2005	Inappropriate target population; targeted health facility managers.
Loevinsohn 2009	Not an RCT, QRCT, CBA or ITS; a case‐control study.
MacFarlane 2006	Inappropriate target population; targeted disaster program managers.
Mansour 2010	Not an RCT, QRCT, CBA or ITS; a case study.
McEwan 2001	Inappropriate target population; targeted management trainers.
McPake 1994	Inappropriate target population; targeted health facility managers.
McPake 1995	Inappropriate target population; targeted health facility managers.
Mills 1998	Inappropriate target population; targeted health facility managers.
Mills 2004	Not an RCT, QRCT, CBA or ITS; a case study.
Mugisha 2009	Inappropriate target population; targeted health facility managers.
Mutabaruka 2010	Inappropriate target population; targeted immunization program managers.
Naimoli 2003	Not an RCT, QRCT, CBA or ITS; a case study.
Nankumbi 2011	Inappropriate intervention; introduction of HIV clinical services.
Newbrander 2011	Not an RCT, QRCT, CBA or ITS; a qualitative description.
Nigenda 2009	Inappropriate target population; targeted health facility managers.
Okamoto 2009	Not an RCT, QRCT, CBA or ITS; a case study.
Omar 2009	Inappropriate outcome; measured participants’ perceptions of training course.
Omaswa 1997	Inappropriate target population; targeted clinical professionals.
Palmer 2006	Not an RCT, QRCT, CBA or ITS; a case study.
Pappaioanou 2003	Not an RCT, QRCT, CBA or ITS; a case study.
Perry 2008	Not an RCT, QRCT, CBA or ITS; a program description.
Sandiford 1994	Not an RCT, QRCT, CBA or ITS; uncontrolled before‐and‐after.
Siddiqi 2006	Not an RCT, QRCT, CBA or ITS; a qualitative description.
Sinha 2007	Inappropriate target population; targeted insurance program managers.
Soeters 2003	Not an RCT, QRCT, CBA or ITS; a case study.
Talbot 2009	Inappropriate target population; targeted clinical professionals.
Tanaka 1999	Not an RCT, QRCT, CBA or ITS; a case study.
Tarimo 1989	Not an RCT, QRCT, CBA or ITS; a qualitative description.
Topçuoğlu 2004	Not an RCT, QRCT, CBA or ITS; a case study.
Trap 2001	Inappropriate target population; targeted clinical professionals (pharmacists).
USAID 2009	Not an RCT, QRCT, CBA or ITS; a case study.
Van den Broucke 2010	Inappropriate intervention; building a support system.
Varpilah 2011	Inappropriate target population; targeted clinical professionals.
Vian 2007	Inappropriate target population; targeted local non‐governmental organizations.
WGIHSPT 1995	Not an RCT, QRCT, CBA or ITS; a program description.
Yaping 2002	Inappropriate outcome; measured participants’ perceptions of a training course.

CBA: controlled before‐and‐after study
 QRCT: quasi‐randomized controlled trial
 ITS: interrupted time series study
 RCT: randomized controlled trial

Differences between protocol and review

The protocol stated that we would search the Health Management Information Consortium (HMIC) database. However, neither the authors nor the EPOC group had access to this database, so it was not searched for this review. Further, the protocol identified incentive programs, such as pay‐for‐performance, as relevant to this review because they may improve manager competencies. However, the authors decided that such incentive programs primarily influence manager motivation to use existing competencies and therefore excluded such interventions from the review. Finally, several aspects of the methods described in the protocol were not implemented in the review because the included studies did not warrant them (see Appendix 4: Planned methods not used in review). In particular, methods for dealing with missing data, data synthesis, subgroup analysis and sensitivity analysis were not used.

Contributions of authors

Peter C. Rockers (PCR) and Till Bärnighausen (TB) conceptualized the review. PCR wrote the first drafts of the protocol and both authors edited the final version. PCR and TB both independently conducted all stages of article review. PCR wrote the first drafts of the review manuscript and both authors edited the final version.

Declarations of interest

None known.

Edited (no change to conclusions)