Table 3.
Summary of the limitations of the commonly used priority-setting approaches
| PS approach | Limitations | |
|---|---|---|
| As identified in the literature | As identified at the workshop, in addition to those in the literature | |
| ENHR | • Process overly based on participant experience, knowledge and views • Identified interventions and research questions are not systematically compiled • Does not clarify which stakeholders to involve and how they can be involved |
Might involve several costs: • Hiring facilitation experts, e.g. reviewers reviewing the proposals, and those with needed technical skills to translate research issues into research questions • Dissemination costs • Evaluation costs • Implementation costs (while a PS process is not responsible for implementation costs, they must be considered during the PS process) • Oversight or monitoring is necessary, along with a consideration of relevant costs and hence might not be easily institutionalised within the Ministry |
| CHNRI | • High risk of bias: the options that are included in the ranking are generated by a small group of experts who may be influenced by their own knowledge and expertise [1, 6] • Does not consider existing government priorities [11] • In most of the cases, the PS process itself was not evaluated [6, 14] • The process is long and complex, which could directly affect response rates [12] |
• Complex methodology • Difficult to obtain the right mix of stakeholders depending on the area to be explored • Challenges inherent in getting people to understand how to participate in a reference group • Cost of the PS process: • Meeting costs – bringing stakeholders together • Program Budgeting Marginal Analysis costs – expert might be required • Costs associated with call • Implementation costs |
| JLA | • The potential inability of patients to respond to surveys and thus registering their perceived treatment uncertainties • Patients may not be equal participants in prioritisation workshops [25] • In some cases, the scope/boundaries of the treatment uncertainty is ill defined and wide ranging [25, 28] • Focus on patients and on disease-specific areas; unclear if the approach is applicable in a broader context • No clear guidance seems to be provided with regards to ranking the treatment uncertainties |
• Very difficult to use virtual means to involve necessary populations • Overly biased to treatment needs (and not, for instance, to system needs) • Assumption that the representatives are able to ‘truly represent’ those they claim to represent • How to scale this PS process up to a higher level, e.g. meso- or macro-levels? Costing of PS process: • Human resource costs – consultant if necessary to facilitate process; expert to design survey to collect data on uncertainties • Costs of convening stakeholders meetings • Dissemination costs |
| CAM | • Lack of information for decision-making in most LICs presents a challenge • It is a difficult method; may be impossible to adequately summarise the wealth of evidence on some topics to a few sentences • Lacks in rigour: the identified priorities are not systematically compiled • Final decision-making performed by a panel of experts who may not be representative • The information needs may necessitate a lot of resources: time and money |
• Difficulty in obtaining required evidence Long-term use (especially if the approach will be used again) requires routine, functional systems that collect data (e.g. morbidity, mortality causes) over time • Might require experts on the framework, oversight and facilitation • Complex and multifaceted processes • Diverse skill sets required (e.g. epidemiology, health systems, policy-making) Implementation costs • Costs of validation • Hardware and software costs • Paying highly specialised people to spend time to sit together to figure out the individual components is a time-intensive process |
| L4D | • Does not provide enough detail on technical issues related to PS process • Requires evidence which may be lacking in some contexts (e.g. the MENA case) • Data collection/analysis did not distinguish between responses given by policy-makers, researchers and representatives of the non-state sector • Purposeful selection of respondents might introduce bias • The lack of criteria creates a question as to how priority issues are identified • Having ‘research experts’ apply seven criteria could introduce bias |
• Time consuming process – time is an important commodity • Requires expertise in identification of stakeholders • Costs of facilitator for group process • Time discounting – if you are developing priorities for 10 years the process could be seen as cost effective • Validation of research themes with stakeholders • Investing in pilot projects |