Setting Priorities in Global Child Health Research Investments: Addressing Values of Stakeholders

Abstract

Aim

To identify main groups of stakeholders in the process of health research priority setting and propose strategies for addressing their systems of values.

Methods

In three separate exercises that took place between March and June 2006 we interviewed three different groups of stakeholders: 1) members of the global research priority setting network; 2) a diverse group of national-level stakeholders from South Africa; and 3) participants at the conference related to international child health held in Washington, DC, USA. Each of the groups was administered different version of the questionnaire in which they were asked to set weights to criteria (and also minimum required thresholds, where applicable) that were a priori defined as relevant to health research priority setting by the consultants of the Child Health and Nutrition Research initiative (CHNRI).

Results

At the global level, the wide and diverse group of respondents placed the greatest importance (weight) to the criterion of maximum potential for disease burden reduction, while the most stringent threshold was placed on the criterion of answerability in an ethical way. Among the stakeholders’ representatives attending the international conference, the criterion of deliverability, answerability, and sustainability of health research results was proposed as the most important one. At the national level in South Africa, the greatest weight was placed on the criterion addressing the predicted impact on equity of the proposed health research.

Conclusions

Involving a large group of stakeholders when setting priorities in health research investments is important because the criteria of relevance to scientists and technical experts, whose knowledge and technical expertise is usually central to the process, may not be appropriate to specific contexts and in accordance with the views and values of those who invest in health research, those who benefit from it, or wider society as a whole.

When decisions on investments in health research are made, the term “stakeholders” refers to all individuals and/or groups who have interest in prioritization of those investments. The stakeholders will therefore comprise a large and highly heterogeneous group. Some apparent examples may include research funding agencies (eg, governmental agencies, private organizations, public-private partnerships, international and regional organizations, and taxpayers of a certain region), direct recipients of the funding (eg, researchers and research institutions), beneficiaries of the research (eg, policy makers and the general population of a country), and any other group with interest in prioritization process (eg, advocacy groups, journalists and media, lawyers, economists, experts in ethics, and many others).

Two fundamental characteristics of any acceptable and successful priority setting process are legitimacy and fairness (1). In order to ensure the legitimacy and fairness of the priority setting decisions in health research investments, involvement of a wide range of stakeholders (and/or eliciting their values) is needed. Unfortunately, health research priorities are presently mainly driven by technical experts (2-5). The results of prioritization are therefore in danger of being mostly influenced by their personal views, with minimal input from representatives from the wider community who also may have interest in the process but lack technical expertise. Since the values and criteria important to scientists and technical experts may vary remarkably from those of other relevant stakeholders (6-8), the relevance of eliciting wider stakeholders’ input is increasingly being acknowledged (9-11). However, the main challenge is to develop a systematic, flexible, and repeatable strategy on how this can be achieved in different contexts.

The literature on priority setting for health interventions identifies two main strategies: 1) stakeholders’ values may impact decisions through procedural processes (by having access to the decisions and the rationales behind the decisions, and by having the authority to deliberate on the decisions and influence the final outcome); 2) stakeholders’ values can be directly elicited using quantitative methods (through surveys where respondents rank, weigh, or rate their values) and qualitative methods (involving individual interviews, Delphi technique, complaints procedures or group discussions, concept mapping, citizen’s jury, and public meetings) (12). The main challenges in those attempts have mainly been the lack of capacity for some stakeholders to engage in meaningful deliberations (13) and how to practically incorporate the elicited stakeholders’ values in decision-making (12,14). This paper presents our suggestions and experiences on how the values and interests of large and diverse group of stakeholders could still be incorporated in decisions on health research investment priorities. We specified thresholds and weights needed to address stakeholders’ values within CHNRI methodology using three different versions of a questionnaire. We aimed to assess stakeholders’ values for priority setting in global health research investments. We tested questionnaires that presented different levels of complexity and detail of the questions that stakeholders’ representatives would be asked. We also aimed to evaluate different strategies of turning their responses into numerical thresholds and weights.

Materials and methods

Systematic methodology for setting priorities in health research investments

Child Health and Nutrition Research Initiative (CHNRI) has recently presented the main concepts underlying systematic methodology for setting priorities in health research investments (9-11). It is a flexible methodology that first defines all criteria relevant to priority setting in a given context. In the second step, technical experts are responsible for systematic listing of research options and for scoring all those options against all the defined criteria using simple and discriminative set of questions. This eventually leads to all research options receiving several different intermediate scores, each addressing one important criterion relevant to priority setting. These intermediate scores could, for example, address the criteria of answerability of research options in ethical way, efficacy and effectiveness of resulting interventions, deliverability, affordability, and sustainability of resulting interventions, maximum potential for disease burden reduction, and predicted impact on equity (9-11). The final research priority score, which describes the overall value of each research option when all criteria are taken into account, is then computed as the average of the intermediate scores. Research priority score then becomes a basis for ranking the competing research options and for combining their value with their proposed cost to achieve an optimal mix of investments within a fixed funding budget. Up to this point, priority setting process is based on profound understanding and knowledge of technical details and scientific literature. Hence, it is not proposed to involve non-technical stakeholders in the computation of these intermediate scores.

Strategies within CHNRI’s methodology for addressing stakeholders’ values

Intermediate scores assigned to each research option during the process described above can still be modified to include the values of non-technical groups of stakeholders before the final research priority score is computed. The wider group of stakeholders may: 1) define minimal threshold for each intermediate score that needs to be achieved to consider any research option a funding priority; 2) allocate weights to intermediate scores so that the final research priority score is not their simple arithmetic mean, but rather a weighted mean that reflects relative values assigned to each criterion. In this way, some intermediate scores (priority setting criteria) would influence the value of the final score more than the others. The concept of introducing thresholds prevents investments into research options that dramatically fail one of the important criteria, regardless how well they fare when scored against other criteria. The concept of weights is graphically presented in Figure 1.

Figure 1.

Graphical presentation of the concept of thresholds and weights (T1-Tn and W1-Wn). A priority-setting tool below addresses n key priority setting criteria (eg, answerability, effectiveness, deliverability, maximum potential for disease burden reduction, and impact on equity); intermediate scores for this particular research options had values IS1-ISn. Input from stakeholders resulted in values W1-Wn, ie, factors by which each criterion’s value (intermediate score) is weighted, and threshold scores (T1-Tn) needed to be met within each criterion in order for the proposed research option to be considered a priority. The final “research priority score” (RPS) for each proposed research avenue is defined as their weighted average: [W1 × (Criterion 1 score) + W2 × (Criterion 2 score) + … + Wn x (Criterion n score)] / (W1+…+Wn)

The concept of pre-defined thresholds on each of the criteria prevents funding of proposed research option if it largely fails one important criterion, regardless how well it complies with all other criteria. In the first exercise presented in the results section, stakeholders’ representatives were asked to draw a horizontal line on a vertical bar ranging between 0 and 100 (as in Figure 1) to define where they would like to see minimum thresholds for compliance with each of the relevant criteria. Their suggestions were then turned into quantitative values and the average of these values was chosen as the threshold. In the second exercise, thresholds were derived post-hoc, as the lowest 10% of the scores on any given criteria.

The concept of weights complements the concept of thresholds. The scoring performed by technical experts, which eventually leads to intermediate scores, is based on the assumption that all criteria are mutually independent and equally important (Figure 1). The concept of weights makes some intermediate scores more important than the others, which is determined by the stakeholders. In this way, their system of values is reflected in assigning different weights to criteria used for priority setting before the final priority score is computed for each research option. Numerical values for these weights can be obtained through survey conducted among the appropriate group of representatives of the stakeholders (termed “larger reference group” – LRG).

Design of questionnaires used to assess stakeholders’ values

We tested three different versions of questionnaires (Table 1). The simplest version had five simple questions that captured the essence of each of the criteria relevant to priority setting that were highlighted by CHNRI’s previous work on conceptual framework for the methodology (9-11). The questions were short and they used layman’s terms. The intermediate version of the questionnaire (Table 1) listed some technical terms that described each of the five criteria relevant to priority setting (9-11). These terms required some understanding of public health concepts and they were therefore used in a smaller group of stakeholders’ representatives. Finally, the most complex version was tested in a relatively small group of highly motivated people with good understanding of the issues related to health research investments, aiming to obtain deeper understanding of the elicited weights.

Table 1.

Three versions of questionnaire that can be presented to larger reference group members, depending on the size and background of larger reference group membership and the desired level of complexity and amount of information that wishes to be captured

Versions
Version 1 (the simplest form)
Please assign relative importance to the following 5 criteria:*
That the new or improved health intervention is likely to indeed be developed through proposed research investment
That, if developed, it is likely to have a real and true effect against the disease that it aims to tackle
That, if developed, it is likely to be delivered to most of those who are in need for it
That, if developed, it has a potential to make substantial share of the disease cases disappear
That, if developed, it is likely to become available to all segments of the society equally
Version 2 (intermediate complexity)
Please assign relative importance to the following 5 criteria:*
Answerability and ethics
Efficacy and effectiveness
Deliverability, affordability and sustainability
Maximum potential for disease burden reduction
Equity in achieved disease burden reduction
Version 3 (the most complex)
Please assign relative importance to the following 5 criteria:*
The proposed research questions will be answerable in an ethical way:
(Please explain further the relative importance of answerability and ethics to each other within this criterion)
Answerability: the research is likely to lead to discovery of new knowledge
Ethics: the research would achieve its aims in an ethical way
The proposed research will create new knowledge that would increase the efficacy and effectiveness of child health and nutrition interventions
(Please explain further the relative importance of efficacy and effectiveness to each other within this criterion)
Efficacy: The research would yield information on impact of the intervention on disease/disability in child health and nutrition under ideal conditions, such as the randomized controlled trials
Effectiveness: The research would yield information on impact of the intervention on disease/disability in child health and nutrition when implemented in the real world context
The proposed research will create new knowledge that would improve deliverability, affordability and sustainability of child health and nutrition interventions in the given context
(Please explain further the relative importance of deliverability, affordability and sustainability to each other within this criterion)
Deliverability: The intervention based on proposed research will be deliverable (infrastructure: basic intervention design, communication and transport infrastructure, need for human resources) in the context
Affordability: The intervention based on proposed research will be affordable (available resources in place to implement the intervention) in the context
Sustainability: The intervention based on proposed research will be sustainable (government capacity and partnership requirements, ease of delivery and usage characteristics) in the given context
The proposed research will lead to new knowledge that would expand the magnitude of impact of child health and nutrition interventions on human potential through the reduction of mortality, disease and promotion of mental, physical and cognitive development
The proposed research will lead to new knowledge that would improve child health and nutrition interventions in a way that they would become more likely to benefit the most vulnerable populations (eg, the poor, visible minorities, children of female headed house holds, orphans) in the given context

*The way this particular question is phrased will depend on the way in which their responses would afterwards be turned into numerical weights; this question could be posed either as asking larger reference group (LRG) members to rank the 5 criteria from first to fifth in order of importance, or to distribute US $100 by those 5 criteria respecting their relative importance.

In the simplest version of the questionnaire, output from stakeholders’ representatives were simple ranks of the five listed criteria in ascending order, with the first indicating the most important criterion (ranks ranged from 1 to 5). An average of the individually suggested rank for each of the five criteria was computed. Then, if all five criteria were valued exactly the same, their average rank would be 3.0. Therefore, any increase from this expected value toward the theoretical maximum of 5.0 indicated lower assigned importance to the criterion, while any decrease toward the theoretical minimum of 1.0 indicated greater assigned importance. Dividing the expected average rank (3.0) by the observed (ie, obtained) average rank gave us numerical weights for the five criteria. These weights can theoretically range between 3.0 (which is the expected average rank of 3.0 divided by theoretically most favorable average rank of 1.0) and 0.6 (which is the expected average rank of 3.0 divided by theoretically the least favorable average rank of 5.0). In this way, intermediate scores for the criteria that were seen as more important by stakeholders’ representatives were assigned greater weight (of up to 3 times), while the weight of the criteria that were considered less important by the LRG could be reduced in importance (up to 40% reduction).

In the other two versions of the questionnaire, stakeholders’ representatives were asked to allocate US $100 (this amount was set arbitrarily because it is intuitive) across the five criteria. The amount allocated to each criterion reflected its relative importance to stakeholders. Eventually, every criterion then had its average “value” expressed in US$. All intermediate scores could then be multiplied with average value of weights (in US$), added up and then divided by US $100. This gave the final research priority score, which again had a value between 0 and 100.

Composition of larger reference group of stakeholders’ representatives

The CHNRI methodology proposes that the process of setting priorities in health research investments should involve two main groups: 1) technical working group (TWG), which consists of technical experts (mainly scientists) who assign a value (the intermediate score) to each proposed health research option by judging its likelihood to address each of the criteria relevant to priority setting; 2) larger reference group (LRG), which should comprise representatives of all other stakeholders not represented in the TWG. The initiators of the priority setting process (eg, funding agencies, national governments, international agencies) should be responsible for gathering this larger reference group of stakeholders bearing in mind the aim of the exercise and general context.

In our article, where the context was defined as global health research and relevant policies were already set by United Nation’s (UN) Millennium Development Goals (15), we used three different groups of stakeholders’ representatives – all of whom were particularly suitable to administration of the respective version of the questionnaire. This composition is presented in detail in the examples given below in the results section and further discussed in the discussion section of this paper.

Results

Exercise 1: Internet-based survey of the affiliates to the Global Research Priority Setting Network

Between March and May 2006, 30 affiliates to the Global research priority setting network agreed to participate in a piloting test and to represent the reference group of stakeholders for global childhood mortality issues (addressing the UN’s Millennium Development Goal 4). Respondents included researchers, policymakers, and health practitioners with interest in the field of priority setting in health care from high, low, and middle income countries. They were sent the simplest form of the questionnaire (Table 1, version 1) and asked to simply rank the criteria from 1st to 5th by the order of their importance and to return their ranks. They were also asked to set thresholds on each of the five criteria (Table 2).

Table 2.

Suggested weights and thresholds by the representatives of the stakeholders

Version of the questionnaire	Mean rank (1-5) investment	Mean suggested weight	Mean suggested threshold
Simple version of the questionnaire:*
question related to answerability and ethics	3.14	0.96	54/100
question related to efficacy and effectiveness	3.50	0.86	47/100
question related to deliverability, affordability and sustainability	3.36	0.89	42/100
question related to potential for disease burden reduction	1.71	1.75	39/100
question related to predicted impact on equity	3.29	0.91	41/100
Intermediate version of the questionnaire:
answerability and ethics	3.72	0.80	bottom 10%
efficacy and effectiveness	2.75	1.09	bottom 10%
deliverability, affordability and sustainability	2.94	1.02	bottom 10%
maximum potential for disease burden reduction	3.28	0.91	bottom 10%
predicted impact on equity	2.31	1.30	bottom 10%
Complex version of the questionnaire:
efficacy and effectiveness	US $19.20	0.192	NA†
deliverability, affordability and sustainability	US $32.89	0.329	NA
maximum potential for disease burden reduction	US $24.21	0.242	NA
predicted impact on equity	US $23.94	0.239	NA

*The simple version of the questionnaire was used to survey the members of the global research priority setting network; the intermediate version for a diverse group of national-level stakeholders from South Africa; and the most complex one to the participants at the conference related to international child health held in Washington, DC, USA

†NA – not applicable.

The respondents placed the greatest weight (1.75) to maximum potential for disease burden reduction, while the weights for the remaining four criteria were similar to each other and smaller than 1.00. The highest threshold was placed on the criterion of answerability in an ethical way (54/100), while the lowest was placed on maximum potential for disease burden reduction (39/100) (Table 2).

Exercise 2: Interview-type survey in identified group of stakeholders in South Africa

This example differs from the first and third one because it refers to the exercise conducted at the national level, rather than global one (11). It is presented here to demonstrate how, when recruiting larger reference group of stakeholders’ representatives, context can be of particular importance. In South Africa, given its apartheid history and the history of health systems organization and management, when identifying participants in the larger reference group it was very important to recognize the ideological environment within which the study was being conducted. While gender and wealth inequities were central, equity in this context also referred to attempts to redress the racial determination of access to health care and services.

In May 2006, LRG members were recruited. Participants were identified using different strategies and included: 1) participants at a local public health conference; 2) academicians (from disciplines ranging from history to psychology and public health) from the three universities in the Western Cape Province and one from the University of Kwazulu-Natal; 3) workers at the Medical Research Council and the Human Sciences Research Council in Cape Town; 4) “lay people,” including child and youth care workers, teachers, social workers, a statistician, a health journalist; and 5) members of the public. Detailed description of this group of stakeholders’ representatives was presented by Tomlinson et al (11). These stakeholders used the version of the questionnaire of intermediate complexity (Table 1, version 2).

Predictably, the greatest weight in this specific context was placed upon the criterion “predicted impact on equity” (1.30), while the weights for the remaining four criteria were similar to each other and close to 1.00 (Table 2). It was decided that thresholds should not be determined in the same way as in the first exercise, but rather by excluding all research options that found themselves in bottom 10% for any of the five criteria (Table 2).

Exercise 3: Interview-type survey at the international conference on child health

In June 2006, the most complex version of the questionnaire (Table 1, version 3) was administered by trained university students to 20 participants at the conference related to international child health held in Washington, DC, USA. Respondents included mostly female program implementers with a couple of students, researchers, and fundraisers. Respondents were also asked to evaluate the process and suggest any other important criteria that may not have been included in the questionnaire. In this exercise, they were asked to allocate US $100 to the presented criteria and sub-criteria deemed as relevant to health research priority setting.

Three respondents allocated 0 dollars to criterion 2 (efficacy and effectiveness), criterion 4 (maximum potential for disease burden reduction) and criterion 5 (predicted impact on equity) (Table 2). One respondent allocated all US $100 to criterion 4. When the average allocations were computed, the highest allocation (US $32.89) went to criterion 3 (deliverability, affordability, sustainability) and the smallest allocation (US $19.20) went to criterion 2 (efficacy and effectiveness). The criteria 4 and 5 received almost the same allocation (US $24.21 and US $23.94, respectively). Respondents proposed some additional criteria including popular/political support, government commitment, and cultural acceptability (which criteria may be included in the criterion of deliverability and sustainability, Table 2).

Discussion

This article presents a strategy of involving a variety of stakeholders who lack technical expertise into decision-making process on investments into health research. The process of research priority setting is clearly complex and multi-dimensional and it should be perceived as legitimate and fair by those affected by the decisions (1). CHNRI recently presented a new systematic methodology for setting priorities in health research investments that highlighted five different broad and mutually independent dimensions (“lenses”) through which competing research options could be viewed as priorities over each other (Figure 1) (8-10). The methodology takes into account all those criteria when assigning a value to each proposed research option, and it can flexibly introduce more criteria or leave some of the proposed ones out. Each research option is scored against each of the criteria to derive intermediate scores that transparently highlight its strengths and weaknesses (8). Those intermediate scores are initially determined by people with technical expertise, but then subjected to thresholds and weights set by a large group of stakeholders. The thresholds and weights reflect stakeholders’ own systems of values and their definition does not require technical expertise.

Delegating the representatives of stakeholders into the larger reference group should involve systematic listing of the different groups of the relevant stakeholders. These may include research funding agencies, direct recipients of research funds, beneficiaries of the research, and any other group with interest in prioritization process (16). When convening this group, there should be clear agreement upon the rationale and the relative importance of each stakeholder group. The relative importance may be reflected in either the group’s sample size or the weight put on their responses relative to the others’. The actual size and composition of the group would depend on the research options to be considered and the available resources including time. Stakeholders can be represented by a reasonably small or very large reference group, depending on the context. For example, the group may comprise of only 10 persons (eg, main individual donors to a private charity who want to participate in decision-making on how their money is invested) or more than a thousand people (eg, in the case of addressing research to avert global child mortality, where everyone from members of public in developing countries, public and private donor foundations, to the UN, World Health Organization (WHO) and UNICEF officials represent the potential stakeholders).

Small reference group of stakeholders is appropriate when several major donors to any health research-funding organization want to influence priority setting process. They can set very specific thresholds and weights for each criterion. Large and diverse reference group of stakeholders is more appropriate for priority setting for health research on problems of regional or global importance. In the latter case, the size and diversity of the representatives of stakeholders may lead to regression of thresholds to the mean, regardless of the criteria. In this case a different approach is proposed, by setting a threshold at bottom 10% or 20% of values of all scored research options for any of the criteria. In this way, stakeholders will set the overall level of tolerance toward underachieving in any single criterion. They may even set such percentile-based thresholds for each criterion individually (Figure 1).

It should be stressed that intermediate scores upon which weights and thresholds are placed do not exactly represent likelihood that the research option would fulfill the criterion in question. They are abstract numbers that measure collective optimism of technical experts about that likelihood. Therefore, when thresholds are being set by larger reference group of stakeholders as specific cut-off points on a scale 0-100, it may be very difficult to predict them intuitively. In all such cases, we advise that those who set thresholds should be made aware of the distribution of the values of intermediate scores, without linking these values to specific research options. In this way, more useful thresholds can be suggested. Otherwise, it is likely that some thresholds may not disqualify any option, and therefore be redundant, or may disqualify unreasonable proportion of scored research options. In our first exercise, it became apparent that thresholds cannot be set in advance intuitively, given the reasons above. In our second exercise we felt that it would be more appropriate to understand the outcome first, and then set these thresholds post hoc, so we applied the statistical approach based on lowest 10%. In the third exercise we dropped the thresholds altogether, although the statistical approach from the second exercise could still be applied.

The first version of the questionnaire tested in this article is highly recommended for use in very large reference groups, involving representatives from different backgrounds and levels of education. The second (intermediate) version may not be suitable for some members of the larger reference group of stakeholders, some of whom may lack the appropriate background or literacy levels to clearly understand the proposed concepts. However, if a survey involves people with the appropriate knowledge, our view is that this questionnaire should be more useful. This is because the questions in the second version correlate more directly to the chosen criteria for priority setting, whereas in the first version they were translated into layman’s terms, but in this simplification there is always a concern that the essence of the criterion was slightly distorted and that the stakeholders are not understanding all the questions in the same way. Finally, the most complex form of the questionnaire should only be used in exceptional cases such as: 1) when the larger reference group is formed by a relatively small number of highly motivated people with a good understanding of the issues related to health research investments, 2) for the purpose of qualitative research on the stakeholders’ values where this form of questionnaire could provide more detailed information, providing a deeper understanding of the elicited weights. In our exercise, the first group was an experimental group and they had no moderator to their exercise, so we needed to keep the questionnaire as simple as possible and presented them with the simplest form. In the second group, members of the public formed a minority of the group (up to 10%) and they were moderated through the exercise, so they were able to join the survey although it was a more demanding one. In the third group, the background of the stakeholder groups justified the use of the most complex form.

When turning the input from the stakeholders into useful quantitative values, the first version of the questionnaire was the simplest and it worked very well in practice. Still, it should be recognized that it has conceptual shortcomings. There is no clear theoretical justification for limiting the weights for intermediate scores to values between 0.6 and 3.0. In view of this limitation, we recommend the second approach (distribution of US $100) wherever possible. This approach places no limits on the values of the weights. In extreme cases, some criteria (intermediate scores) may be assigned an average weight of zero by the larger reference group of stakeholders and thereby be excluded from influencing the final priority score.

Some important lessons were also learned through interviewing stakeholders’ representatives in different contexts. In South Africa, where stakeholders were gathered from within national boundaries, equity was ranked higher than some of the most popular criteria in similar contexts (7,17). This underscores the importance of eliciting local values (18,19). Given the variations in individual values in this context, technical experts who were deriving intermediate scores recommended that the choice of representatives of the stakeholders should be systematic to ensure representation from all relevant stakeholders (11). The feedback from stakeholders’ representatives attending large international health conference in Washington, USA, where the most complex form of the questionnaire was administered, included comments that the interview should be guided by trained research assistants. Suggestions were made to delineate the criteria especially in contexts where the individual criteria may be valued differently. Finally, since the five proposed criteria were not exhaustive, there should always be provision for respondents to give suggestions for criteria that may be relevant to their local context.

In conclusion, this article addressed one of the most pressing issues in health research priority setting globally – the lack of involvement of stakeholders other than the scientists in decision-making process on investments. This article presents how this can be achieved through introducing a large reference group of stakeholders’ representatives that complement the work of technical experts. The representatives of stakeholders are enabled to assign greater value to some of the priority-setting criteria defined by technical experts. This adds a dimension of public opinion to the rational scientific assessment of the research options derived by technical experts. Thresholds and weights set in this way are transparent and can be later challenged or revised based on a feedback. The change may be prompted by the outcomes of priority setting process or because of the changes in dynamic social, political, or economic context. While this paper provides some progress in including stakeholders’ values in decision making, there are still unresolved problems of how the proposals put forward can be operationalized in the real world (20). Future work should focus on these areas.