Abstract
Many countries have developed strategies to attract and retain qualified health workers in underserved areas, but there is only scarce and weak evidence on their successes or failures. It is difficult to compare lessons and measure results from the few evaluations that are available. Evaluation faces several challenges, including the heterogeneity of the terminology, the complexity of the interventions, the difficulty of assessing the influence of contextual factors, the lack of baseline information, and the need for multi-method and multi-disciplinary approaches for monitoring and evaluation. Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluating interventions for human resources for health. This paper proposes a conceptual framework that offers a model for monitoring and evaluation of retention interventions taking into account such challenges. The conceptual framework is based on a systems approach and aims to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its design phase through to its results. It also aims to guide the monitoring of interventions through the routine collection of a set of indicators, applicable to the specific context. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of the interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It promotes the use of a common understanding on how attraction and retention interventions work, using a systems perspective.
ملخص
قامت العديد من البلدان بإعداد استراتيجيات تهدف إلى جذب واستبقاء العاملين الصحيين المؤهلين في المناطق المحرومة من الخدمات، غير أن البينات التي توضح مدى نجاح أو فشل هذه الاستراتيجيات هي بينات نادرة وضعيفة. ومن الصعب مقارنة الدروس المستفادة وقياس النتائج المنبثقة عن التقييمات القليلة المتوافرة. فعملية التقييم تواجه تحديات كثيرة، منها تغايرية المصطلحات، وتعقد المدخلات، وصعوبة تقييم ما للعوامل المقارنة من تأثير، ونقص المعلومات الأساسية، والحاجة إلى أساليب متعددة الطرائق والنظم للرصد والتقييم. وعلاوة على ذلك فإن المحتوى الاجتماعي والسياسي والاقتصادي التي صممت ونفذت من خلاله هذه المدخلات نادراً ما يؤخذ في الاعتبار في العمليات الخاصة برصد وتقييم مدخلات الموارد البشرية العاملة في المجال الصحي. وتقترح هذه الورقة إطار عمل مفاهيمي يقدم نموذجاً للرصد والتقييم من أجل المدخلات المعنية باستبقاء العاملين مع وضع مثل هذه التحديات قيد الدراسة. ويرتكز الإطار المفاهيمي على أسلوب للنظم، كما يهدف إلى توجيه التفكير نحو تقييم المداخلة من أجل زيادة نسبة الوصول إلى العاملين الصحيين في المناطق المحرومة من الخدمات، بدءاً من مرحلة التصميم وحتى مرحلة استخلاص النتائج. علاوة على توجيه رصد المدخلات من خلال الجمع الروتيني لمجموعة من المؤشرات، التي يمكن تطبيقها في إطار محتوى معين. ويوضح الإطار أيضاً الحاجة إلى استخدام الأسلوب الشامل عند تصميم المدخلات، وتنفيذها، ورصدها، وتقييمها. وليس القصد أن يكون إطار العمل متجمداً بل أن يكون مرناً بحيث يطبق في نطاق كل بلد على حدة. وهو يدعو إلى استخدام المفهوم الشائع حول كيفية عمل المدخلات المعنية بالجذب والاستبقاء باستخدام منظور النظم.
Résumé
De nombreux pays ont mis au point des stratégies pour attirer et retenir le personnel médical qualifié dans les zones mal desservies, mais les preuves du succès ou de l'échec de ces stratégies sont rares et faibles. Il est difficile de comparer les enseignements tirés et de mesurer les résultats à partir des quelques évaluations disponibles. L'évaluation de ces stratégies se heurte en effet à plusieurs difficultés, dont l'hétérogénéité de la terminologie, la complexité des interventions, la difficulté d'évaluer l'influence des facteurs contextuels, le manque de données de référence et la nécessité de démarches empruntant à plusieurs méthodologies et à plusieurs disciplines pour le suivi et l'évaluation. En outre, le contexte social, politique et économique dans lequel les interventions sont conçues et mises en œuvre est rarement pris en compte dans le suivi et l'évaluation des interventions en faveur des ressources humaines pour la santé. Le présent article présente un cadre conceptuel proposant un modèle pour le suivi et l'évaluation des interventions visant à retenir la main d'œuvre qui tient compte de ces difficultés. Ce cadre conceptuel repose sur une démarche systématique et a pour objectif de guider la réflexion dans l'évaluation d'une intervention pour améliorer l'accès au personnel de santé dans des zones mal desservies, depuis sa phase de conception jusqu'à ses résultats. Il vise aussi à guider le suivi des interventions par le recueil systématique d'une série d'indicateurs, applicable au contexte considéré. Il introduit l'idée qu'une approche globale est nécessaire pour la conception, la mise en œuvre, le suivi, l'évaluation et le bilan des interventions. Il n'est pas destiné à être prescriptif et s'applique de manière flexible à chaque contexte national. Il favorise l'utilisation d'une interprétation commune des modalités d'action des interventions pour attirer et retenir la main-d'œuvre, en recourant à une perspective systématique.
Resumen
Muchos países han desarrollado estrategias para atraer y conservar a personal sanitario cualificado en zonas insuficientemente atendidas, pero la evidencia sobre el éxito o fracaso de esas iniciativas es escasa y poco robusta. Es difícil comparar las enseñanzas extraídas y cuantificar los resultados de las escasas evaluaciones disponibles. Las medidas de evaluación tropiezan con varios problemas, en particular la heterogeneidad de la terminología, la complejidad de las intervenciones, las dificultades para evaluar la influencia de factores contextuales, la falta de información basal, y la necesidad de aplicar enfoques multimetódicos y multidisciplinarios en la vigilancia y la evaluación. Por otra parte, en la vigilancia y evaluación de las intervenciones relacionadas con los recursos humanos para la salud rara vez se tiene en cuenta el contexto social, político y económico en el que se diseñan y aplican esas intervenciones. En este artículo se propone un marco conceptual que brinda un modelo de vigilancia y evaluación de las intervenciones de promoción de la permanencia que sí tiene en cuenta esos factores. Dicho marco está basado en un enfoque sistémico y pretende servir de orientación a la hora de evaluar una intervención de ampliación del acceso al personal sanitario en las zonas subatendidas, desde su fase de diseño hasta sus resultados. También tiene por objeto orientar la vigilancia de las intervenciones mediante la recopilación sistemática de un conjunto de indicadores, aplicables al contexto específico. Se sugiere que es necesario aplicar un enfoque omnicomprensivo al diseño, ejecución, vigilancia, evaluación y examen de las intervenciones. El marco no pretende ser prescriptivo y se puede aplicar con flexibilidad en el contexto de cada país. Promueve el uso de una perspectiva común sobre el funcionamiento de las intervenciones de fomento de la captación y la permanencia, aplicando un enfoque sistémico.
Introduction
Health workers’ willingness to practise in underserved areas, such as rural, remote or poor areas, is a recognized challenge in achieving equitable access to health services. Many countries have developed strategies to attract and retain qualified health workers in these areas. But the evidence on the successes or failures of such interventions is scarce and weak, and it is difficult to compare lessons and measure results from the few evaluations that are available.1–3
There has been significant progress in generating common understanding and debate on ways to evaluate the impact of development interventions, and on using consistent monitoring and evaluation terminology, such as “outputs”, “outcomes” and “impact”.4–6 However, this has yet to be applied to the evaluation of human resources for health interventions and, specifically, to those seeking to increase access to health workers in underserved areas. There is an urgent need to achieve an agreement and strengthen the evidence that would underpin sound policy recommendations in this area.
Seeking to address this gap, this paper proposes a conceptual framework to guide managers, policy-makers and evaluators in the assessment of interventions to increase access to health workers in underserved areas. The framework aims to support all stages of policy development. It suggests a logical sequence that can be followed when deciding on any intervention that addresses attraction and retention issues and assists in formulating key questions to be answered when designing, monitoring and evaluating such interventions. It includes examples of indicators to inform the process, which can be adapted to a specific context. A comprehensive consideration of indicators to monitor the health workforce in general is offered elsewhere.7
Challenges in evaluation
Policy- and decision-makers need to know whether interventions work or not, why they work and in which context. Therefore it is important to have information about the effects of the interventions, but also about the factors that made the intervention succeed or fail (the questions dealing with “when, why, how, and in what circumstances such interventions work well or fail to work”).8
Evaluating interventions to improve human resources for health is complex for different reasons. First, relating these interventions to health status is very difficult due to broad socioeconomic, cultural, political and health systems factors that influence health. For instance, although improved health outcomes, such as reduction in maternal mortality, are directly correlated with increased availability of health workers, it is difficult to attribute the improvement directly to a certain health workforce intervention.9,10 Illustrative country case studies showed that Afghanistan and Ethiopia have implemented comprehensive health sector strategies with multiple co-existing interventions, like the recruitment of community health workers, which may have promoted access to life-saving services, thus improving health status.11
Another problem is that many evaluations lack a baseline against which to assess the results, particularly in countries with a major health worker deficit,3,10,12 as well as a specific intervention logic that clarifies the expectations of the intervention designers (S Kane, B Gerretsen, R Scherpbier, M Dal Poz, MA Dieleman, unpublished data, 2009).13 Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluation of human resource interventions.14,15
Thus, the main challenges which evaluators face are related to the multidimensional nature of interventions, and the difficulty of assessing the influence of contextual factors. To be able to assess such interventions in complex systems, there is a need for multimethod and multidisciplinary monitoring and evaluation approaches, inclusive of all relevant stakeholders.16 Another challenge is generating lessons across regions and countries to inform the global health agenda and promote a meaningful dialogue at both national and international level. There is clear need for a framework that offers a model for monitoring and evaluation of health worker retention interventions taking into account such challenges.
Conceptual framework
The proposed framework is based on a systems approach and differentiates between “inputs”, “outputs”, “outcomes” and “impact”, with regards to the results of interventions to attract and retain health workers in underserved areas. It builds on the proposed common framework for monitoring performance and evaluating progress in the scale-up for better health,17 and proposes indicators to measure progress in implementing various strategies, allowing users to determine what works or not, and to explore contextual factors influencing their success or failure. The framework has two aims: (i) to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its inception/design phase through to its results, by suggesting key questions about the relevance, efficacy, efficiency, effectiveness and sustainability of the intervention; and (ii) to guide the monitoring of interventions, through a focus on a routine collection of a set of indicators, applicable to the specific context.
We used the following definitions for evaluation and monitoring:4
We define “evaluation” as the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability.
We define “monitoring” as a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.
Later in this paper, we examine the application of the framework to the stages of policy development: design, implementation, monitoring and evaluation (Fig. 1). The sequence in design and implementation phases does not necessarily imply a linear policy process, as monitoring and evaluation may help identify a need for new interventions. They are, in effect, part of an ongoing policy cycle.
Fig. 1.
The conceptual framework for measuring efforts to increase access to health workers in underserved areas
IMR, infant mortality rate; MMR, maternal mortality rate.
Design
The evaluation of the design stage should identify the extent to which the chosen intervention was relevant and adequate to the need, the expectations of the population and the health system context, and the reasons for selecting a particular option.18 By relevance, we mean the extent to which the objectives and the elements of the intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.4 The relevance issue can and should also be addressed retrospectively; and then the question is whether the objectives of an intervention or its design were and are still appropriate given changed circumstances.4
Questions that an evaluator should ask are: did the intervention respond to a documented need, was it appropriate for the given context and how was it selected? Is the choice of the intervention based on evidence or at least on robust arguments?
The assessment of the situation analysis should check if the problem of limited access to health workers has been clearly described. Is it due to an insufficient number of students willing to work in underserved areas, or an insufficient production of health workers, or were there other reasons? The evaluation needs to assess if a health labour market analysis has been conducted, to identify whether the lack of access to health workers in underserved areas was a matter of insufficient demand, insufficient supply, or insufficient remuneration and incentives, and unattractive working conditions.19
The situation analysis should also identify whether the elements in place to support the implementation of the intervention were assessed and if a feasibility analysis preceded the choice of the intervention. Strategies to increase access to health workers in underserved areas require participation of stakeholders from different sectors. Therefore, questions to be asked include: were mechanisms for multi-stakeholder engagement in place and used, were regulatory frameworks analysed before the intervention, were the intervention costs calculated and considered affordable (availability of funds), was the organizational capacity to support its implementation analysed? At this stage, the Human Resources for Health Action Framework is a recognized tool that can be used to guide the situation analysis (Fig. 2).21 It comprises health system dimensions such as policy, finance, education, partnership, leadership and human resources management systems.
Fig. 2.
The adapted Human Resources for Health Action Framework20
HRMS, human resources management systems.
At the end of this phase, a strategy can be proposed to address the perceived need of lack of access to health workers in underserved areas and of maldistribution of health workers. Such a strategy can be in one or more of the following thematic areas: education, regulatory, financial, and/or management and support, described elsewhere.3
Implementation
When the implementation of an intervention is evaluated, questions might include: did the activities take place according to plan, did any changes occur in implementation capacity or in the context that influenced the implementation of the intervention components? Did any changes occur regarding the involvement and commitment of stakeholders? Which ones and why? Was any action undertaken to adapt the intervention to the changes? What was done and why? Or why was no action taken? Monitoring the progress in implementation can be informed by the systematic collection and analysis of a set of key indicators, which are further discussed in this paper.
Results
Intervention results can be evaluated at different levels (Fig. 1): outputs, intermediate outcomes, outcomes and impact. Questions to be asked include: did the intervention achieve its intended results at the level of outputs, outcomes, or impact? Is the intervention sustainable? What were the contextual factors (political, economic, social, or organizational) that have influenced the results?
The level of “outputs” is the direct effect, attributable to the intervention, such as increased attractiveness of the areas/or the profession, improved recruitment and deployment, and increased retention of health workers in those areas.
The level of intermediate “outcomes” is that of improved health workforce performance, the dimensions of which have been described in The world health report 2006 as “availability, competence, responsiveness and productivity”.10 Some attraction and retention interventions can have a direct effect on these dimensions too. For example, changes in rural curricula can improve competencies to address rural health problems and performance-based financial incentives can improve productivity. However, often improved health workforce performance cannot be attributed to retention interventions alone, as there are many other factors that contribute, such as education strategies or health sector management reforms.
The level of “outcomes” is that of increased health systems performance, measured by increased coverage and utilization of services, and improved responsiveness. Improved health workforce performance could contribute to improved health system performance but a host of other factors can contribute as well, such as the availability of drugs, infrastructure and equipment.22
Finally, the level of “impact” is understood as the effect on health outcomes and an improved health status, and other unexpected effects on underserved communities (e.g. attraction/retention of other professionals or services). This cannot be directly attributed to retention interventions, as the simple availability of health workers cannot directly improve health outcomes in the absence of effective components such as drug supply, functioning facilities and good clinical practices.9 But a well performing health workforce is a significant component of a strong health system that, together with other social determinants, can lead to improved health status.22
Evaluations of any attraction and retention intervention need to consider the contextual factors that may explain why and how such interventions have achieved the intended outputs, outcomes and impact, i.e. the social, economic and political context. This includes a stakeholder analysis including their respective power bases and interests, as well as an analysis of the capacity of the health sector to implement. Evaluations should use both qualitative and quantitative methods to measure outputs and impacts, but also to analyse formal and informal policy processes accompanying the design and implementation of interventions. Ideally, to find out if an attraction and retention intervention works or not, a cluster randomized controlled trial with an adequate control group would be needed, but this may not be feasible or ethical in the “real life” policy scenario. Moreover, causal chains are often complex in public health interventions, making results of randomized controlled trials subject to effect modification in different populations.20 Alternative observational methods such as before-after and plausibility designs can also be considered, taking care to document and control the contextual factors appropriately.20,23 Studies with adequacy and plausibility designs can improve both the internal and external validity of randomized controlled trials, and plausibility studies are often the only feasible option for getting valid evidence of impact when attempting the evaluation of large-scale interventions.23
Monitoring
The framework depicted in Fig. 1 proposes a tool to guide the monitoring of interventions to increase access to health workers in underserved areas. At each stage of the implementation process, indicators are proposed to assess whether the implementation is on track.
Indicators to monitoring at the design stage should be those that will allow for a detailed description of the stocks and flows of health workers across sectors, age and sex categories, geographical areas and professions. Detailed description of these indicators, as well as data sources for health workforce monitoring, are described elsewhere.7
When monitoring the implementation of interventions, systematic data collection is required on the planned activities, by constantly reviewing the production capacity, as well as the organization and management capacity. Monitoring implementation also requires tracking changes in context and stakeholders, so as to be able to adapt the interventions when required.
Regarding the monitoring of results of the intervention, at the level of “outputs” it is important to monitor whether the dimensions of “attractiveness”, “engagement/recruitment”, “deployment” and “retention” are achieved. Indicators to monitor progress on these dimensions can include, for example, changes in the intentions of students or health workers to relocate, reside in or leave underserved areas (attractiveness). The available funded positions in underserved areas can be an indicator of effective engagement, whereas the number of health workers recruited and changes in vacancy rates can be indicators of effective deployment and recruitment. Finally, the effects of an intervention on retention can be measured by duration in post, turnover, absenteeism, stability index or even survival rates.24,25
At the level of “outcomes”, it is proposed to monitor the progress on improved health workforce performance and health systems performance, through dimensions and indicators proposed by The world health report 2006 (Table 1).10 As said before, assessing the longer-term impact of human resource interventions on health outcomes should be done within the broader context of factors influencing health status.
Table 1. Dimensions and indicators to assess health workforce performance11.
| Dimension | Indicators |
|---|---|
| Availability | Staff ratios |
| Absence rates | |
| Waiting times | |
| Competence | Individual prescribing practices |
| Institutional level: readmission rates; live births; cross-infections | |
| Responsiveness | Patient satisfaction |
| Assessment of responsiveness | |
| Productivity | Occupied beds |
| Outpatient visits | |
| Interventions delivered per worker or facility |
The effectiveness of attraction and retention interventions can only be measured at the level of outputs and, to a certain extent, at the level of outcomes. In addition, implementing human resource retention interventions can take many years before any results can be seen, for example in the case of creating new schools in rural areas or rolling out financial incentive schemes. Strong human resource information systems, linked to national health information systems, as well as clearly defined and agreed metrics or indicators are required to monitor the progress of such interventions in terms of increased attractiveness, improved recruitment and duration of stay, increased availability, improved competence, productivity and responsiveness. Table 2 makes an attempt towards advancing a set of key questions and indicators to be used when evaluating and monitoring retention interventions for human resources for health.
Table 2. Proposed questions and indicators for the evaluation and monitoring of interventions to increase access to health workers in underserved areas.
| Stage | Questions | Indicators | Methods |
|---|---|---|---|
| Design | • Did the intervention respond to a documented need? • Is the choice of the intervention based on evidence or robust arguments (situation analysis)? • Were the actual reasons for limited access to health workers clearly documented (i.e. insufficient number of students willing to work in rural areas or insufficient number of funded positions? |
• Human resources for health-costed plan, including situation analysis • Stocks and flows of health workers (for more details see reference)8 |
• Labour market analysis • Survey of intentions • Demographic analysis (health workforce stocks and flows) • Stakeholder analysis • Review of policy documents |
| Implementation | • Did the activities take place according to plan? • Did any changes occur during implementation? |
• Process indicators (changes in the programme elements) | • Surveys • Stakeholder analysis • Review of policy documents |
| Results | • Did attractiveness of profession/rural/remote areas improve? • Did recruitment of health workers in underserved areas improve? • Did retention improve? |
• Total health workers recruited • Changes in preferences for rural/remote areas • Reduced turnover • Reduced vacancies • Stability index • Survival rates (or years in post) |
• Surveys • Survival analysis • Analysis of registries data or facility data |
The conceptual framework presented here needs to be extensively piloted and will be refined through further inputs from different sources. There are ongoing plans for applying it in several countries that are in the process of designing and implementing attraction and retention strategies in underserved areas. Efforts will also be made to make sure the evaluation tool is user-friendly. For the sake of argument, a hypothetical example of its potential use is presented in Box 1.
Box 1. A hypothetical use of the conceptual framework.
The framework can be applied to a hypothetical scenario of country X. The evaluation will explore: (i) Does the intervention respond to the problem identified and has the intervention achieved the intended results? (ii) What improvements in intervention and policy design are needed, or are alternative strategies required? (iii) What indicators should be used to measure results?
Step 1: situation analysis
For example, in country X, a situation analysis has identified a lack of health workers in rural areas. A study of intentions of young graduates to practise identified the unwillingness of health workers to go to those areas due to a lack of opportunities for professional development and low salaries. Thus, the main bottlenecks seem to be in initially attracting staff to rural areas.
Step 2: choice of interventions
The intervention that has been implemented sought to “create more schools in rural areas” (education category) and to implement a “rural allowance scheme” (financial category) for some categories of health workers, and improve support and supervision (management). The intended result of this multi-level intervention was to increase attractiveness of rural practice for health professions and improve access to professional training to students with rural backgrounds, as well as an increased duration of stay of the categories of health workers covered by the scheme.
Step 3: measuring results using a selection of country-appropriate indicators
The indicators chosen for monitoring progress and measuring the direct effects (“outputs”) of the intervention would be: “intentions to go to rural areas after graduation”, “number of applicants from rural background to the newly developed schools compared to urban”, “number of graduates choosing to work in rural areas”, “duration of stay in post of the health workers covered by the scheme”. The indicators to measure the indirect effects (“outcomes”) of the intervention would be: “changes in staff ratio within 3 years”, “increase in outpatient visits” before and after, “changes in staff satisfaction”, “user satisfaction” perceptions of community representatives of service responsiveness to need. The outputs and outcomes can be related statistically to take-up of training programmes, to the percentage of staff covered by financial incentive schemes and supported by an experienced manager, to establish correlation and formulate plausible explanations for the effect of the different types of incentives on the way the intervention has performed. Eventually, the final impact of the intervention can be estimated by improved “health outcomes”, bearing in mind that these changes cannot entirely be attributed to the intervention itself.
Step 4: monitoring
Analysis of longitudinal data from indicators to examine how the situation has changed, especially seeking to identify unintended consequences of implementing the intervention. For example, attracting staff in rural areas may have resulted in their recruitment by nongovernmental organizations and private facilities rather than working in the public sector.
The way forward
This framework proposes a common approach to facilitate the evaluation and monitoring of interventions to increase access to health workers in underserved areas. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of such interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It is hoped that it will promote the use of a common understanding/logic on how attraction and retention interventions work, using a systems perspective. It starts with a common set of indicators, which enable comparison between various cases, and facilitates reviews of published studies. As such it can be used to monitor and evaluate interventions, either using a method-based or a theory-based evaluation approach with a specific set of indicators.
Funding:
This work was funded by the World Health Organization, Health Systems and Services, Department of Human Resources for Health, Health Workforce Migration and Retention Unit, Geneva, Switzerland. Additional support was received from the WHO Collaborating Centre on Human Resources for Health at the Royal Tropical Institute (KIT), The Netherlands.
Competing interests:
None declared.
References
- 1.Grobler L, Marais BJ, Mabunda SA, Marindi PN, Reuter H, Volmink J. Interventions for increasing the proportion of health professionals practising in rural and other underserved areas. Cochrane Database Syst Rev. 2009;1:CD005314. doi: 10.1002/14651858.CD005314.pub2. [DOI] [PubMed] [Google Scholar]
- 2.Wilson NW, Couper ID, De Vries E, Reid S, Fish T, Marais BJ. A critical review of interventions to redress the inequitable distribution of healthcare professionals to rural and remote areas. Rural Remote Health. 2009;9:1060. [PubMed] [Google Scholar]
- 3.Dolea C, Stormont L, Braichet J-M. Evaluated strategies to increase attraction and retention of health workers in remote and rural areas. Bull World Health Organ. 2010;88:379–385. doi: 10.2471/BLT.09.070607. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Development Assistance Committee. Glossary of key terms in evaluation and results-based management. Paris: Organisation for Economic Co-operation and Development; 2002. Available from: http://www.oecd.org/dataoecd/29/21/2754804.pdf [accessed on 15 February 2010].
- 5.Leeuw F, Vaessen J. Impact evaluations and development: NONIE guidance on impact evaluation. Washington, DC: The Network of Networks on Impact Evaluation; 2009. Available from: http://siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf [accessed 15 February 2010].
- 6.White H. Theory-based impact evaluation: principles and practice. New Delhi: International Initiative for Impact Evaluation (3ie); 2009.
- 7.Dal Poz M, Gupta N, Quain E, Soucat ALB. Handbook on monitoring and evaluation of human resources for health – with special applications for low- and middle-income countries. Geneva: World Health Organization; 2009. [Google Scholar]
- 8.Mills A, Gilson L, Hanson K, Palmer N, Lagarde M. What do we mean by rigorous health-systems research? Lancet. 2008;372:1527–9. doi: 10.1016/S0140-6736(08)61633-5. [DOI] [PubMed] [Google Scholar]
- 9.Anand S, Bärnighausen T. Human resources and health outcomes: cross-country econometric study. Lancet. 2004;364:1603–9. doi: 10.1016/S0140-6736(04)17313-3. [DOI] [PubMed] [Google Scholar]
- 10.The world health report 2006: working together for health. Geneva: World Health Organization; 2006. [Google Scholar]
- 11.Peters DH, El-Saharty S, Siadat B, Janovsky K, Vujicic M, editors. Improving health service delivery in developing countries: from evidence to action. Washington, DC: The World Bank; 2009. [Google Scholar]
- 12.Terms of reference for an independent evaluation of the impact of Malawi's Emergency Human Resources Programme. Lilongwe: Department for International Development (UK); 2009.
- 13.Dieleman M, Gerretsen B, van der Wilt GJ. Human resource management interventions to improve health workers’ performance in low and middle income countries: a realist review. Health Res Policy Syst. 2009;7:7. doi: 10.1186/1478-4505-7-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Dieleman MA, Kane S, Zwanikken P, Gerretsen B. Identifying patterns in retention intervention studies: revisiting the evidence from a realist perspective. Geneva: World Health Organization; 2009. [Google Scholar]
- 15.Fogarty L, de Vries D, Reavely E, Bunch E. Indicators for evaluating human resources for health capacity-building. In: Evaluation 2009: context and evaluation, American Evaluation Association, Orlando, 11-14November2009 [Google Scholar]
- 16.Bristow G, Farrington J, Shaw J, Richardson T. Developing an evaluation framework for crosscutting policy goals: the Accessibility Policy Assessment Tool. Environ Plan A. 2009;41:48–62. doi: 10.1068/a4092. [DOI] [Google Scholar]
- 17.Monitoring and Evaluation Working Group of the International Health Partnership and related initiatives (IHP+). Monitoring performance and evaluating progress in the scale-up for better health: a proposed common framework Geneva: World Health Organization; 2009. Available from: http://www.internationalhealthpartnership.net//CMS_files/documents/a_proposed_common_framework_EN.pdf [accessed 15 February 2010].
- 18.Codjia L, Jabot F, Dubois H. Evaluation du programme d’appui à la médicalisation des aires de santé rurales au Mali. Geneva: World Health Organization; 2008. In French. [Google Scholar]
- 19.Lemiere C, Herbst C, Jahanshani N, Smith A, Soucat A. Reducing geographical imbalances of health worker distribution: what works, what does not and why? A labour market perspective of urban-rural health disparities, policy solution and experiences in sub-Saharan Africa Washington, DC: The World Bank; 2009. [Google Scholar]
- 20.Habicht J-P, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 1999;28:10–8. doi: 10.1093/ije/28.1.10. [DOI] [PubMed] [Google Scholar]
- 21.Human Resources for Health (HRH) Action Framework [internet site]. Available from: http://www.capacityproject.org/framework/ [accessed 15 February 2010].
- 22.Everybody's business: strengthening health systems to improve health outcomes. Geneva: World Health Organization; 2007. Available from: http://www.who.int/healthsystems/strategy/everybodys_business.pdf [accessed 15 February 2010].
- 23.Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94:400–5. doi: 10.2105/AJPH.94.3.400. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Buchan J. The benefits of AHP workforce stability. Edinburgh: Scottish Executive Health Department; 2005. [Google Scholar]
- 25.Rabinowitz HK, Diamond JJ, Markham FW, Rabinowitz C. Long-term retention of graduates from a program to increase the supply of rural family physicians. Acad Med. 2005;80:728–32. doi: 10.1097/00001888-200508000-00004. [DOI] [PubMed] [Google Scholar]