Abstract
Aims
To systematically identify global research gaps and resource priorities for integrated community case management (iCCM).
Methods
An iCCM Child Health and Nutrition Research Initiative (CHNRI) Advisory Group, in collaboration with the Community Case Management Operational Research Group (CCM ORG) identified experts to participate in a CHNRI research priority setting exercise. These experts generated and systematically ranked research questions for iCCM. Research questions were ranked using a “Research Priority Score” (RPS) and the “Average Expert Agreement” (AEA) was calculated for every question. Our groups of experts were comprised of both individuals working in Ministries of Health or Non Governmental Organizations (NGOs) in low– and middle–income countries (LMICs) and individuals working in high–income countries (HICs) in academia or NGO headquarters. A Spearman’s Rho was calculated to determine the correlation between the two groups’ research questions’ ranks.
Results
The overall RPS ranged from 64.58 to 89.31, with a median score of 81.43. AEA scores ranged from 0.54 to 0.86. Research questions involving increasing the uptake of iCCM services, research questions concerning the motivation, retention, training and supervision of Community Health Workers (CHWs) and concerning adding additional responsibilities including counselling for infant and young child feeding (IYCF) and treatment of severe acute malnutrition (SAM) ranked highly. There was weak to moderate, statistically significant, correlation between scores by representatives of high–income countries and those working in–country or regionally (Spearman’s ρ = 0.35034, P < 0.01).
Conclusions
Operational research to determine optimal training, supervision and modes of motivation and retention for the CHW is vital for improving iCCM, globally, as is research to motivate caregivers to take advantage of iCCM services. Experts working in–country or regionally in LMICs prioritized different research questions than those working in organization headquarters in HICs. Further exploration is needed to determine the nature of this divergence.
Approximately 6.6 million children die before their fifth birthday every year [1,2]. Together, pneumonia, diarrhoea and malaria accounted for approximately one third of these deaths [1,2], and many of these deaths are preventable. Although Millennium Development Goal 4 (MDG4) made reducing child deaths a global priority, calling for a two–thirds reduction of child deaths between 1990 and 2015, many countries are not on track to achieve this goal [2–6].
Diarrhoea and pneumonia, in particular, disproportionately affect impoverished and marginalized children who do not access treatment [1]. Existing interventions to prevent and treat childhood pneumonia, diarrhoea and malaria are efficacious [6]. Integrated community case management (iCCM) is a delivery strategy that utilizes community health workers (CHWs) to diagnose and treat multiple conditions, most commonly pneumonia, diarrhoea and malaria, in children under five. CHWs are based in the communities they serve, working as an easily accessible community–arm of a country’s existing health care system [7,8].
Major donors and non–governmental organisations (NGOs), including the World Health Organization (WHO) and the United Nations Children’s Fund (UNICEF) are promoting iCCM as a key strategy to reduce child mortality. CHWs have been lauded as “the world’s most promising health workforce resource for enabling health systems in resource–constrained settings to reduce the burden of disease from serious, readily preventable or treatable conditions” [5]. In addition to reducing pneumonia–specific mortality by 36% and malaria–specific mortality by 60%, a recent review found that CHWs can also effectively perform nutritional counselling activities [5,6].
Nepal’s iCCM program has contributed to one of the most rapidly declining child mortality rates in the world [5]. Conversely, despite having a national–level, well–funded, community health workers program, Pakistan has not achieved satisfactory reductions in child mortality [9,10]. This divergence of results highlights the need to understand programmatic factors to strengthen programs delivering iCCM [11,12]. In their call for identification of research priorities in iCCM, Hamer and colleagues emphasize the need for the integration of research and program implementation in addition to a focus on long–term outcomes [11].
While research priorities have previously been developed and published by the Global CCM Operations Research Group (CCM ORG), the development of these priorities was constrained because the advisors were global level iCCM experts and the research priorities were not systematically evaluated [11]. Thus, we applied the Child Health and Nutrition Research Initiative’s (CHNRI) method to identify and systematically evaluate research priorities for iCCM. The CHNRI method has identified research gaps and resource priorities in a variety of contexts, including global childhood diarrhoea, birth asphyxia and childhood pneumonia [13–17].To our knowledge, this is the first use of CHNRI to identify research gaps and resource priorities for a delivery strategy rather than an illness. Over seventy–five experts, representing academics, international organizations and Ministries of Health within countries already implementing iCCM participated in at least one of the steps of this exercise.
METHODS
The CHNRI method was designed to assist policy makers and funders in identifying research gaps and resource priorities in a variety of contexts for health research, as well as the strengths and weaknesses of the research gaps identified. In the last decade, the CHNRI method has been widely used to identify research gaps in childhood illnesses, including global childhood diarrhoea, birth asphyxia and childhood pneumonia [13–17]. The exercise is comprised of four stages: (i) the context of the problem and the evaluation criteria are defined; (ii) technical experts generate and rank research questions against the proposed criteria; (iii) weighting of the evaluation criteria is decided through consultation with stakeholders; and, (iv) research priority scores are calculated for each research priority and agreement between experts is analyzed [18].
An iCCM CHNRI Advisory Group was formed to assist in the execution of this exercise. Together with the co–principal investigators (KW, SS, AZ), the Advisory Group attended a two–day meeting in New York where the criteria used in scoring was finalised and the final list of research questions to be scored was selected. A list of the iCCM CHNRI Advisory Group members and a detailed description of the activities during the New York meeting are presented in Box 1.
Box 1. Details of New York Meeting activities.
The members of the iCCM CHNRI Advisory Group and the co–PIs of the CHNRI exercise met in New York on May 1 and 2, 2013. Members of the iCCM CHNRI Advisory Group are: Shamim Qazi, David Marsh, Franco Pagnoni, Mark Young, Kerry Ross, Karin Kallander, Serge Raharison and Troy Jacobs. The co–principal investigators are: Kerri Wazny, Salim Sadruddin and Alvin Zipursky.
The members combined and eliminated duplicate research priorities, both from the original ORG list [11, 12] and from those submitted through the CHNRI exercise, leaving 119 research priorities. These priorities were scored on a scale of 1–5 (1 being highest, 5 being lowest), by all of the meeting participants. The average score for each priority was calculated, and priorities with a score higher than average (≥2.6) were retained for dissemination for scoring by the larger group of CHNRI participants.
The iCCM CHNRI Advisory Group and co–principal investigators also discussed the standard CHNRI criteria and modified CHNRI criteria used in a previous exercise [17], and finalized the criteria to be used in the iCCM CHNRI exercise. The group decided to use 4, rather than 5, criteria and to weight the criteria equally in the final analysis.
Finally, the group agreed to re–invite all those invited to submit research questions, regardless of whether they did, unless the participant expressed that they were unable to participate in the exercise.
1. Context of the problem and evaluation criteria are defined
We modified the CHNRI criteria used in a previous CHNRI exercise [17], yielding criteria more applicable for evaluating research questions for a delivery system. We chose the following four criteria: (i) answerability; (ii) research feasibility; (iii) deliverability; and, (iv) importance/potential impact. Table 1 displays the specific questions used to evaluate the research questions under each criterion.
Table 1.
Criteria for iCCM CHNRI exercise
| Criterion | Sub–questions |
|---|---|
| Answerability |
1. Would you say that the research question is well–framed?
2. Can a single study or a very small number of studies be designed to answer the research question?
3. Do you think that a study needed to answer the proposed research question would obtain ethical approval without major concerns? |
| Research Feasibility* |
1. Is it likely that, in the context of interest, there will be sufficient capacity to carry out this research?
2. Is it feasible to provide the training required for staff to carry out the research in the context of interest?
3. Is the cost and time required for this research reasonable within the context of interest? |
| Deliverability |
1. Taking into the account the level of difficulty with delivery of the potential intervention or delivery strategy (for example, need for change of attitudes and beliefs, supervision, transport infrastructure), would you say that this intervention or delivery strategy will be deliverable within the context of interest?
2. Taking into account the resources available to implement the intervention, would you say that the intervention or delivery strategy would be affordable within the context of interest?
3. Would government capacity and partnership be essential to ensure the intervention or delivery strategy would be sustainable?† |
| Importance/ Potential Impact | 1. Will the results of this research fill an important knowledge gap? 2. Are the results from this research likely to shape future planning and implementation? 3. Will the results from this research be relevant to most countries in the context of interest? |
iCCM – integrated community case management, CHNR – Child Health and Nutrition Research Initiative
*For this criterion, the “context of interest” refers to countries that do, or would benefit from, implementation of iCCM.
†We eliminated this question from the calculation of the scores for Criterion 3, as described in the text.
2. Technical experts generate and rank research questions
We asked for members of the CCM ORG to nominate experts for participation in the exercise. We also included experts who participated in a previous CHNRI exercise who are involved in iCCM implementation or research [17]. Finally, we invited experts who were referred from others we invited to participate. In total, we invited 127 experts in iCCM to generate research questions for our CHNRI exercise. Experts represented international organizations, ministries of health within low– and middle–income countries, academia and non–governmental organizations. All materials, including instructions and research questions, were translated into French by a professional translator to ensure francophone country participation. In total, 75 experts submitted 366 research questions.
We combined the submitted research questions with those previously generated by the CCM ORG and thematically organized and discussed the research questions during the iCCM CHNRI Advisory Group Meeting [11]. The Advisory Group members removed duplicates, combined similar questions, and then rated each question from 1 to 5. We calculated average scores and selected the 61 questions with above average scores for evaluation by the experts.
We invited all experts who were initially approached to submit research questions, aside from those who declined participation, plus any additional experts we identified after our call for research questions. In total, we invited 133 experts to score the 61 questions. To reduce the possibility that the order of research questions would affect scoring, we made eight versions of the scoring sheet, using a random number generator to shuffle the question order. The scoring sheets were otherwise identical.
Each criterion contained three sub–questions. We asked experts to score 1 for yes, 0 for no and 0.5 if undecided. If the experts did not feel sufficiently knowledgeable to answer a particular question, they were instructed to leave the cell blank. Seventy–five experts returned completed scoring sheets.
3. Weighting of criteria is decided
Prior to scoring, we chose to weigh all criteria equally in the analysis, as we felt they were of equal importance.
4. Research Priority Scores and Average Expert Agreement are calculated
All returned scoring sheets were checked for errors and then scores were entered into a master calculation sheet. The Research Priority Score (RPS) and Average Expert Agreement (AEA) were calculated for each research question. The RPS is a mean score given, across criteria and scorers, for a particular research question. The AEA is the proportion of scorers who chose the mode (the most common score) for each research question.
After distributing scoring sheets to the experts, some of the experts raised concerns regarding the third question of criterion 3 (Table 1). This question asks whether government partnership will be necessary to ensure sustainability of the results of the research. A “yes” response indicates a positive answer for all other questions except this one. If government partnership is necessary to sustain research results, then a “yes” is a negative answer. Given the potential confusion over non–parallel construction, we excluded this question. Thus, we took the average scores for each criterion and then averaged those scores, weighting each criterion equally. For criterion 3, we calculated the average of questions 1 and 2 only. Calculating the RPS in this way, rather than taking the mean scores across the 11 sub–questions, allowed for each criterion to be weighted equally in the analysis.
We used the AEA rather than a Fleiss kappa statistic to calculate agreement among experts, which is in line with previous CHNRI exercises. Due to the large number of scorers and few scoring options, it is not possible to rule out chance with the Fleiss Kappa statistic even in cases with complete agreement. Although the AEA does not give an indication of statistical significance, we thought that policy makers and donors would find it more useful than the kappa statistic, as it can give a general idea of the degree of agreement between experts. The average expert agreement (AEA) was calculated as follows:
| where q is a question that experts are being asked to evaluate competing research investment options, ranging from 1 to | 11 |
5. Comparative analysis of scores given by in–country or regional participants and those working in high–income countries or organizational headquarters
We stratified the responses received into those received by participants working in–country or regionally (LMIC Group) and those working in organizational headquarters or high–income countries (HQ/HIC Group). A list of our participants, their organizations and their categorizations can be found in Online Supplementary Document(Online Supplementary Document), Table S1. We calculated the RPS for each research question separately in these groups and used a Spearman’s Rho correlation coefficient to calculate the correlation of research questions’ ranks between these groups. Spearman’s Rho is used to determine the degree of correlation between two ranked sets of results. A correlation coefficient of 1 indicates high, positive correlation between two sets of results; a correlation coefficient of –1 indicates high, negative correlation between two sets of results and a correlation coefficient of 0 indicates no correlation.
RESULTS
We invited 133 experts to score research questions; 75 returned completed scoring sheets. Three experts declined participation at this stage. We received nearly equal responses from participants working in–country or regionally (n = 36) and from those working in organizational headquarters or high–income countries (n = 39).
The range of RPS across the 61 questions was 64.58 to 89.31 (median = 81.43) out of a possible 100. The AEA ranged from 0.54 to 0.86 out of a possible 1.00. The top 20 research questions overall and their corresponding RPS and AEA scores are displayed in Table 2. Online Supplementary Document(Online Supplementary Document), Table S2 contains all the ranked research questions, their scores in each criterion and their RPS and AEA scores.
Table 2.
Overall rank and research priority scores for top 20 research questions
| Overall Rank | Research question | Research Priority Score (RPS) | Average Expert Agreement (AEA) |
|---|---|---|---|
| 1 |
Assess perceptions of beneficiaries and levels of community satisfaction in CHWs capacity to diagnose and treat sick child (with malaria, pneumonia, diarrhoea and severe malnutrition) at the community level. |
89.31 |
0.84 |
| 2 |
Identify and evaluate strategies for retention and motivation of CHWs. |
89.08 |
0.86 |
| 3 |
Identify and evaluate strategies for improving referral between communities and health facilities, including referral compliance. |
88.94 |
0.84 |
| 4 |
Identify determinants of non–use of iCCM services by caretakers and develop strategies to increase the uptake of iCCM. |
88.89 |
0.84 |
| 5 |
Identify and evaluate new diagnostic tools for improved classification of pneumonia (ie, different ARI timers, respiratory counting beads, etc.) at the community level that are most appropriate for various cadres. |
88.83 |
0.85 |
| 6 |
Evaluate the effectiveness of 3–day vs 5–day amoxicillin treatment regimens in Africa. |
88.61 |
0.84 |
| 7 |
Identify and evaluate innovative strategies to improve community engagement and mobilization for CCM. |
87.49 |
0.83 |
| 8 |
Evaluate the feasibility, effectiveness and impact of adding community–based infant and young child feeding (cIYCF) counseling skills to the CHW workload. |
87.26 |
0.82 |
| 9 |
Identify the primary barriers to CHW supervision and develop and evaluate strategies to motivate CHW supervisors to provide continuous support to CHWs. |
87.18 |
0.82 |
| 10 |
What is the impact of pre–referral antibiotics on treatment outcomes of possible serious bacterial infections? |
86.52 |
0.80 |
| 11 |
Assess perceptions, understanding and motivating factors for caregivers on the need for prompt treatment for the sick child. |
86.41 |
0.82 |
| 12 |
What is the impact of iCCM on health facility worker workload, by disease? |
86.37 |
0.81 |
| 13 |
Develop and evaluate strategies (for example, innovative packaging of drugs) to improve compliance and uptake of treatment. |
86.00 |
0.81 |
| 14 |
Identify and evaluate strategies to improve supervision and quality of care using mHealth technology. |
85.85 |
0.81 |
| 15 |
Identify and evaluate effective and feasible strategies for maintaining quality of case management by CHWs. |
85.56 |
0.82 |
| 16 |
Identify and evaluate strategies for, and costs of, supervising the CHW supervisor. |
85.35 |
0.79 |
| 17 |
Develop and evaluate strategies for using mHealth technology to improve drug supply and logistics for the CHWs. |
85.28 |
0.79 |
| 18 |
Evaluate the impact of iCCM on equity in access and use of basic health services. |
85.14 |
0.80 |
| 19 |
Identify and evaluate the effectiveness and cost of various incentive schemes and strategies for CHWs. |
84.81 |
0.79 |
| 20 | Identify and evaluate strategies to improve integration of iCCM logistics (diagnostics and drug supply) to the central procurement and supply system at the community level. | 84.41 | 0.79 |
CHW – Community Health Workers, iCCM – integrated community case management
Several of the top 20 research priorities overall involved increasing uptake of iCCM services, through community motivation and satisfaction (#1, #7), identification of determinants of non–use (#4), motivating factors for care seeking behaviour (#11) and other strategies to improve compliance and uptake (#13).
Strategies to improve motivation, retention, training and supervision of CHWs were a priority (questions #2, 9, 13, 16 and 19). Identifying and evaluating strategies for retention and motivation of CHWs was 2nd overall, scoring highly in importance/potential impact (0.92) and had high agreement between scorers (AEA = 0.86). Two of the top 20 questions (#9 and 14) emphasized the need for supervision and support of CHWs. mHealth technology was proposed as a tool to strengthen drug supply and logistics systems (#17).
Research questions involving adding responsibilities to CHWs’ workload also appeared in the top 25. Questions #8 and 21 address the feasibility, effectiveness and impact of adding counselling for infant and young child feeding (IYCF) and treatment of severe acute malnutrition (SAM), respectively.
Two of the top 10 research questions dealt specifically with pneumonia; question 5 asks to identify and evaluate new diagnostics specific to different cadres of health workers (eg, respiratory counting beads and ARI timers) and question 6 asks to evaluate 3–day vs 5–day amoxicillin treatment in Africa.
Importance/potential impact
Table 3 displays the top 10 questions in the importance/potential impact criterion. Question #15 overall (identify and evaluate feasible and effective strategies for maintaining CHWs’ quality of case management) ranked first, and question #15 (identify and evaluate new diagnostics for different CHW cadres) ranked second. Questions relating to adding newborn care and SAM to the CHWs’ workload, which were ranked 21st and 24th overall, respectively, both received scores of 0.90, but these questions’ scores in the answerability and deliverability criteria brought down their overall scores.
Table 3.
Top 10 research priorities by importance/potential impact criterion
| Importance/ Potential impact rank | Research question | Criterion 1 score | Criterion 2 score | Criterion 3 score | Criterion 4 score | Overall RPS | Overall Rank |
|---|---|---|---|---|---|---|---|
| 1 |
Identify and evaluate effective and feasible strategies for maintaining quality of case management by CHWs. |
0.77 |
0.90 |
0.83 |
0.93 |
85.56 |
15 |
| 2 |
Identify and evaluate new diagnostic tools for improved classification of pneumonia (ie, different ARI timers, respiratory counting beads, etc.) at the community level that are most appropriate for various CHW cadres. |
0.87 |
0.90 |
0.85 |
0.93 |
88.83 |
5 |
| 3 |
Identify and evaluate strategies for retention and motivation of CHWs. |
0.85 |
0.94 |
0.89 |
0.92 |
89.08 |
2 |
| 4 |
Evaluate effectiveness of 3–day vs 5–day oral amoxicillin treatment regimens in Africa. |
0.83 |
0.88 |
0.92 |
0.91 |
88.61 |
6 |
| 5 |
What are the feasibility, impact and costs of adding newborn care (including PNS, home visits, treatment of infection and Caring for the Newborn and Children in the Community) to the iCCM package? |
0.78 |
0.86 |
0.77 |
0.90 |
82.74 |
24 |
| 6 |
Develop safe and effective treatment strategies in settings where referral is not possible. |
0.62 |
0.79 |
0.73 |
0.90 |
75.88 |
52 |
| 7 |
Evaluate the effectiveness and feasibility of delivering treatment for Severe Acute Malnutrition (SAM) through iCCM. |
0.82 |
0.84 |
0.81 |
0.90 |
84.41 |
21 |
| 8 |
Identify the primary barriers to CHW supervisions and develop and evaluate strategies to motivate CHW supervisors to provide continuous support to the CHWs. |
0.81 |
0.93 |
0.86 |
0.90 |
87.18 |
9 |
| 9 |
Identify and evaluate the effectiveness and cost of various incentive schemes and strategies for CHWs. |
0.81 |
0.88 |
0.81 |
0.89 |
84.81 |
10 |
| 10 | Identify and evaluate determinants of quality of CCM services, including characteristics of health systems (and supporting environment) that are most important for delivering high quality iCCM programs at–scale with limited external support. | 0.74 | 0.86 | 0.84 | 0.89 | 83.22 | 22 |
RPS – research priority score, iCCM – integrated community case management
Research feasibility and deliverability
We eliminated research questions that had scores of <0.8 in at least one of the following criteria: answerability, research feasibility and deliverability. We were aiming to investigate whether eliminating any research priorities that would either be difficult to design research studies to answer or to sustain in the post–research stage would change the results. In this analysis, six research priorities were eliminated from the top twenty–five overall. Research questions that ranked 7, 15, 18 and 22 in the overall list were eliminated due to low scores in the answerability category. Questions seventeen and twenty–four overall were eliminated due to a low score in the deliverability criterion and in both the deliverability and answerability criteria, respectively.
In–country or regional participants vs participants from organizational headquarters or high–income countries
HQ/HIC Group scores (median 80.9, range: 63.3–91.1) were slightly lower than their LMIC counterparts (median 83.4, range: 63.5–93.1). The correlation of the research questions’ ranks between the HQ/HIC group and the LMIC group was weak to moderately positive, though statistically significant (Spearman’s ρ = 0.35045, P < 0.01).
While there was a divergence between research questions prioritized by the LMIC group vs the HQ/HIC group, both groups of experts scored the first overall research question highly; otherwise, many of the research questions with the highest level of agreement between groups were ranked in the middle or bottom by both groups.
Research questions and their corresponding ranks by informant are displayed in Online Supplementary Document(Online Supplementary Document), Table S3. Within this table, research questions with the highest level of disagreement between both groups are shaded; blue indicates research priorities that were ranked highly by the HQ/HIC group but not by those in the LMIC group and research priorities shaded in orange indicate the reverse. Table 4 and Table 5 display the top 5 research questions ranked by those working in organizational headquarter/HICs and by experts working in–country or regionally, respectively.
Table 4.
Top 5 research priorities by organization HQ/HIC participants
| HQ Rank | Research Question | Criterion 1 | Criterion 2 | Criterion 3 | Criterion 4 | HQ RPS | Overall Rank |
|---|---|---|---|---|---|---|---|
| 1 |
Identify and evaluate new diagnostic tools for improved classification of pneumonia (ie, different ARI timers, respiratory counting beads, etc.) at the community level that are most appropriate for various CHW cadres. |
0.89 |
0.92 |
0.87 |
0.96 |
91.09 |
5 |
| 2 |
Evaluate the effectiveness of 3–day vs 5–day amoxicillin treatment regimens in Africa. |
0.83 |
0.90 |
0.92 |
0.94 |
89.83 |
6 |
| 3 |
Assess perceptions of beneficiaries and levels of community satisfaction in CHWs’ capacity to diagnose and treat sick children (with malaria, pneumonia, diarrhea and severe malnutrition) at the community level. |
0.86 |
0.96 |
0.94 |
0.71 |
86.90 |
1 |
| 4 |
Identify and evaluate strategies for improving referral between communities and health facilities, including referral compliance. |
0.79 |
0.94 |
0.85 |
0.90 |
86.86 |
3 |
| 5 | Evaluate the effectiveness and feasibility of CHW’s use of pulse oximetry to identify children with severe pneumonia. | 0.90 | 0.89 | 0.79 | 0.89 | 86.57 | 40 |
HQ/HIC – organizational headquarters or high–income countries, RPS – research priority score, iCCM – integrated community case management, CHW – community health worker
Table 5.
Top 5 research priorities by LMIC participants
| Country Rank | Research question | Criterion 1 | Criterion 2 | Criterion 3 | Criterion 4 | Country RPS | Overall Rank |
|---|---|---|---|---|---|---|---|
| 1 |
Identify the primary barriers to CHW supervision and develop and evaluate strategies to motivate CHW supervisors to provide continuous support to CHWs. |
0.89 |
0.98 |
0.92 |
0.94 |
93.10 |
9 |
| 2 |
Identify determinants of non–use of iCCM services by caretakers and develop strategies to increase the uptake of iCCM. |
0.90 |
0.95 |
0.96 |
0.90 |
92.61 |
4 |
| 3 |
Identify and evaluate determinants of quality of CCM services, including characteristics of health systems (and supporting environment) that are most important for delivering high quality iCCM programs at–scale with limited external support. |
0.85 |
0.94 |
0.94 |
0.98 |
92.55 |
22 |
| 4 |
Identify and evaluate strategies for retention and motivation of CHWs. |
0.92 |
0.96 |
0.85 |
0.96 |
92.27 |
2 |
| 5 | Identify and evaluate innovative strategies to improve community engagement and mobilization for CCM. | 0.84 | 0.99 | 0.96 | 0.90 | 92.19 | 7 |
LMIC – low– and middle–income countries, RPS – research priority score, iCCM – integrated community case management, CHW – community health worker
We mapped the research questions that appeared in the top 10 (either overall, in the LMIC list or the HQ/HIC list) to the iCCM evaluation framework [19]. Spread across the life cycle of the project, the framework (Figure 1) has 8 health system components: (i) organization, coordination, policy & advocacy; (ii) human resources; (iii) supply chain management; (iv) service delivery and referral; (v) social and behavioural change; (vi) supervision and performance; (vii) monitoring and evaluation; and, (viii) budgeting, costing and financing. The top 10 research questions in the LMIC and HQ/HIC lists appear in 7 of the 8 components. Listed under each research question in the Figure are the scores of that research question in the overall, HQ/HIC and LMIC lists, respectively. While some of the research questions have similar scores in all lists, the ranks of some of the research questions in the three lists are greatly divergent.
Figure 1.
Evaluation Framework matched to CHNRI research priority “top 10” questions by list ranking: overall, HQ/HIC and LMIC. Number in parentheses: rank overall, HQ/HIC and LMIC, respectively. HQ/HIC – organizational headquarters or high–income countries; LMIC – low– and middle–income countries. Key: blue – top 10 in all questions, green – top 10 overall and in HQ/HIC, red – top 10 overall and in LMIC, orange – top 10 in LMIC only, yellow – top 10 in HQ/HIC only.
DISCUSSION
This CHNRI exercise is the first one, to our knowledge, that uses the methodology to define research gaps and resource priorities for a delivery strategy, rather than a condition. Moreover, our exercise is the first to conduct a comparative analysis of the priorities of different groups of scorers. The results of this exercise will be important in defining the global research agenda for iCCM. Participation in this exercise was not limited to experts in HICs, but included experts based in low–income countries or at regional level, who are implementing or supporting iCCM programs.
Limitations to our exercise include representativeness of sampling and high non–response rate. Although we aimed to include as many experts as possible working in iCCM and issued several calls for names to be nominated, it is possible that we were not able to identify and invite all experts in iCCM. Furthermore, our response rate for the final scores was 56%. Conceivably, those who responded could be systematically different than those who did not. We believe the low response rate to be due to the time consuming nature of completing scoring sheets. However, our response rate was higher than those reported in previous CHNRI exercises [20–22].
Examining community perceptions and satisfactions with CHWs was the highest ranked research priority overall, and this research question had high scores in the deliverability and research feasibility criteria. However, it ranked 46th in the importance/potential impact criteria with a score of 0.78, thus providing a good example of how the CHNRI method can be used to expose the strengths and weaknesses of a particular research question. Additionally, the research question with the highest score for importance/potential impact scored low in the answerability criterion, which indicates a potential for difficulty in designing a study to address it.
Experts working in LMICs prioritized research questions that were mainly operational or delivery–based, including strengthening CHW supervision, motivation and retention, increasing uptake of iCCM services by caretakers, improving community engagement and mobilization and improving the quality of CCM at the health systems level. Experts from organizational HQ/HICs prioritized more technical questions; the two research questions with the highest scores from this group were identifying and evaluating diagnostic tools for different cadres of CHWs and evaluating 3–day vs 5–day amoxicillin treatment in Africa. Again, this finding highlights the importance of the CHNRI exercise in allowing detections of differences in priority perceptions between groups of experts and thus allows the design of studies according to specific priorities.
While many of the high–ranking questions in the CHNRI exercise mirror those proposed by the CCM ORG [11,12], there were some notable differences. Of the top 10 research questions, questions #1, 5, 8, and 9 were generated by the CHNRI exercise and were not present in the CCM ORG’s original list of research questions. The research question that scored the second highest in the importance/potential impact category was also generated through the CHNRI exercise. For the research questions that were present in the CCM ORG’s original list, the use of the CHNRI method to systematically rank the research priorities against pre–set criteria allowed further exploration of the strengths and weaknesses of each proposed research question and lending credibility to the findings.
To our knowledge, our exercise is the first to stratify analysis based on participants’ location. The weak to moderate correlation between participants from HQ/HIC group and the LMIC group reveals that while there is some consensus on priorities, there is significant divergence that requires further examination. As it is often those working in organizational HQs or HICs who are responsible for setting the research agenda, the disconnect between their priorities and those of in–country or regional counterparts requires careful consideration. Although the differences could reflect different characteristics and interests between both groups, it could also indicate a larger problem. While we cannot say which groups’ opinion on research priorities is has more utility, the discordance is important to highlight to donors and researchers when making decisions on which priorities to fund.
In addition to displaying the research priorities in a particular area, a secondary goal of research priority setting exercises is to stimulate interest and funding for research in that area. We are currently aware of two studies are currently being designed to answer research questions from this exercise. The first is a multi–site study that will explore strategies to improve supervision, retention and motivation of CHWs (questions #2 and 14 overall) and the second will study the effects of adding IYCF counselling skills to CHWs’ workload (question #8 overall) (our unpublished data).
iCCM is capable of reducing a substantial number of unnecessary childhood deaths due to pneumonia, diarrhoea and malaria and improving equity in health care access for poor, rural and hard–to–reach communities. We hope that the results of our exercise will continue to direct and assist funders, policy–makers, program managers and researchers to identify research priorities, their potential strengths and weaknesses, and to stimulate interest and in furthering the iCCM research agenda.
Acknowledgements
We would like to acknowledge all the participants, listed in the Online Supplementary Document(Online Supplementary Document), Table S1, who generated and scored research questions for the exercise.
Funding: Our study group received funds from a research consortium (comprised of government and non–government agencies) to conduct this exercise and prepare and submit an article for publication (contract numbers MCH–98 and MCH–145). Funds were distributed through John Snow, Inc.
Authorship declaration: KW designed and coordinated the study and wrote the first draft of the manuscript. SS and AZ provided substantive technical expertise, assistance in designing and coordinating the study and contributed to drafting the manuscript. DH, TJ, KK, FP, DM, SP, SQ, SR, KR and MY were members of the Advisory Group for the study and as such, contributed to selection of the research questions to be scored, provided feedback during the study and contributed towards editing the manuscript.
Competing interests: All authors have completed the Unified Competing Interest form atwww.icmje.org/coi_disclosure.pdf (available on request from the corresponding author). DH is currently the co–chair of the CCM Operational Research group and is a member of the CCM Task Force. DM chaired the CCM Operations Research Group and served on the Steering Committee of the Global CCM Task Force. KR is a member of the CCM Operations Research Group. TJ reports grants and non–financial support from the US Government (USAID) outside the submitted work. kw reports grants from john snow, inc, during the conduct of the study. AZ reports grants from USAID during the conduct of the study. KK, FP, SP, SQ, SS and MY have nothing to disclose.
Additional Material
REFERENCES
- 1.UNICEF. Committing to child survival: A promise renewed. Progress Report 2013. New York, NY: UNICEF, 2013. Available at: http://www.unicef.org/publications/files/APR_Progress_Report_2013_9_Sept_2013.pdf. Accessed: 6 September 2014.
- 2.UN Inter–agency Group for Child Mortality Estimation. Levels and trends in child mortality: Report 2013. New York, NY: UN Inter–agency Group for Child Mortality Estimation, 2013. Available at: http://www.childinfo.org/files/Child_Mortality_Report_2013.pdf. Accessed: 6 September 2014.
- 3.Countdown to 2015. Accountability for maternal, newborn and child survival: The 2013 Update. Geneva: 2013. Available at: http://www.countdown2015mnch.org/documents/2013Report/Countdown_2013–Update_withprofiles.pdf. Accessed: 6 September 2014.
- 4.Liu L, Johnson H, Cousens S, Perin J, Scott S, Lawn J, et al. Global, regional and national causes of child mortality: an updated systematic analysis for 2010 with time trends since 200. Lancet. 2012;379:2151–61. doi: 10.1016/S0140-6736(12)60560-1. [DOI] [PubMed] [Google Scholar]
- 5.Perry H, Zulliger R. How effective are community health workers?: An overview of current evidence with recommendations for strengthening community health worker programs to accelerate progress in achieving the health–related Millennium Development Goals. Baltimore, MD: Johns Hopkins University, 2012 September 2012. Available at: http://www.coregroup.org/storage/Program_Learning/Community_Health_Workers/review%20of%20chw%20effectiveness%20for%20mdgs–sept2012.pdf. Accessed: 6 September 2014.
- 6.WHO. UNICEF. WHO/UNICEF Joint statement on integrated community case management (iCCM): An equity–focused strategy to improve access to essential treatment services for children. New York, NY: 2012 June 2012. Available at: http://www.unicef.org/health/files/iCCM_Joint_Statement_2012.pdf. Accessed: 6 September 2014. [DOI] [PMC free article] [PubMed]
- 7.Community case management essentials: Treating common childhood illnesses in the community. A guide for program managers. Washington, D.C.: CORE Group Save the Children BASICS MCHIP, 2010. Available at: http://www.coregroup.org/storage/documents/CCM/CCMbook–internet2.pdf. Accessed: 6 September 2014.
- 8.Lainez YB, Wittcoff A, Mohamud AI, Amendola P, Perry H, D'Harcourt E. Insights from community case management data in Six Sub–Saharan African countries. Am J Trop Med Hyg. 2012;87(5) Suppl:144–50. doi: 10.4269/ajtmh.2012.12-0106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lady Health Worker Programme. External evaluation of the national programme for family planning and primary health care: Summary of results. Oxford, UK: Oxford Policy Management, 2009. Available at: http://www.opml.co.uk/sites/opml/files/Lady%20Health%20Worker%20Programme%20–%204th%20Evaluation%20–%20Summary%20of%20Results_0.pdf. Accessed: 6 September 2014.
- 10.National Institute of Population Studies. Measure DHS. Pakistan demographic and health survey 2012–13. Islamabad, Pakistan: 2013. Available at: http://dhsprogram.com/pubs/pdf/FR290/FR290.pdf. Accessed: 6 September 2014.
- 11.Hamer DH, Marsh D, Peterson S, Pagnoni F. Integrated community case management: Next steps in addressing the implementation research agenda. Am J Trop Med Hyg. 2012;87(5) Suppl:151–3. doi: 10.4269/ajtmh.2012.12-0505. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Marsh DR, Hamer D, Pagnoni F, Peterson S. Introduction to a Special supplement: Evidence for the implementation, effects, and impact of the Integrated community case management strategy to treat childhood infection. Am J Trop Med Hyg. 2012;87(5) Suppl:2–5. doi: 10.4269/ajtmh.2012.12-0504. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Fontaine O, Kosek M, Bhatnagar S, Boschi–Pinto C, Chan KY, Duggan C, et al. Setting research priorities to reduce global mortality from childhood diarrhoea by 2015. PLoS Med. 2009;6:e41. doi: 10.1371/journal.pmed.1000041. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kosek M, Lanata CF, Black RE, Walker DG, Snyder JD, Salam MA, et al. Directing diarrhoeal disease research towards disease–burden reduction. J Health Popul Nutr. 2009;27:319–31. doi: 10.3329/jhpn.v27i3.3374. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lawn JE, Bahl R, Bergstrom S, Bhutta ZA, Darmstadt GL, Ellis M, et al. Setting research priorities to reduce almost one million deaths from birth asphyxia by 2015. PLoS Med. 2011;8:e1000389. doi: 10.1371/journal.pmed.1000389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Rudan I, El Arifeen S, Bhutta ZA, Black RE, Brooks A, Chan KY, et al. Setting research priorities to reduce global mortality from childhood pneumonia by 2015. PLoS Med. 2011;8:e1001099. doi: 10.1371/journal.pmed.1001099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Wazny K, Zipursky A, Black R, Curtis V, Duggan C, Guerrant R, et al. Setting research priorities to reduce mortality and morbidity of childhood diarrhoeal disease in the next 15 years. PLoS Med. 2013;10:e1001446. doi: 10.1371/journal.pmed.1001446. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Rudan I, Gibson JL, Ameratunga S, El Arifeen S, Bhutta ZA, Black M, et al. Setting priorities in global child health research investments: guidelines for implementation of CHNRI method. Croat Med J. 2008;49:720–33. doi: 10.3325/cmj.2008.49.720. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.McGorman L, Marsh D, Guenther T, Gilroy K, Barat LM, Hammamy D, et al. A health systems approach to integrated community case management of childhood illness: Methods and Tools. Am J Trop Med Hyg. 2012;87(5) Suppl:69–76. doi: 10.4269/ajtmh.2012.11-0758. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Dean S, Rudan I, Althabe R, Girard A, Howson C, Langer A, et al. Setting research priorities for preconception care in low– and middle–income countries: Aiming to reduce maternal and child mortality and morbidity. PLoS Med. 2013;10:e1001508. doi: 10.1371/journal.pmed.1001508. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.George A, Young M, Bang A, Chan KY, Rudan I, Victora CG, et al. Setting implementation research priorities to reduce preterm births and stillbirths at the community Level. PLoS Med. 2011;8:e1000380. doi: 10.1371/journal.pmed.1000380. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Tol WA, Patel V, Tomlinson M, Baingana F, Galappatti A, Panter–Brick C, et al. Research priorities for mental health and psychosocial support in humanitarian settings. PLoS Med. 2011;8:e1001096. doi: 10.1371/journal.pmed.1001096. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.