Table 2.

Framework for evaluating priority-setting

Domains	Parameters of successful priority-setting	Objectively verifiable indicators	Means of verification
Contextual factors	Conducive political, economic, social and cultural context	Relevant contextual factors that may impact priority-setting	Follow-up intermittent interviews with local stakeholders, systematic longitudinal observations, relevant reports, Media
Pre-requisites	Political will	Degree to which the politicians support the set priorities	Follow-up intermittent interviews with local stakeholders, systematic longitudinal observations, relevant reports, Media
	Resources	Budgetary and human resource allocation to the health research	National budget documents
	Legitimate and credible institutions	Degree to which the priority-setting institutions can set priorities, public confidence in the institution	Stakeholder and public interviews
	Incentives	Material and financial incentives	National budget documents
The priority-setting process	Stakeholder participation	Number of stakeholders participating, number of opportunities, each stakeholder gets to express opinion	Observations/minutes at meetings, media reports, special reports
	Use of clear priority-setting process/tool/methods	Documented priority-setting process and/or use of priority-setting framework	Observation/minutes at meetings, media reports, special reports
	Use of explicit relevant priority-setting criteria	Documented/articulated criteria	Observations/minutes at meetings, media reports, special reports
	Use of evidence	Number of times available data is resourced/number of studies commissioned/existing strategies to collect relevant data	Observations/minutes at meetings, media reports, special reports
	Reflection of public values	Number and type of members from the general public represented, how they are selected, number of times they get to express their opinion, proportion of decisions reflecting public values, documented strategy to enlist public values, number of studies commissioned to elicit public values	Observations/minutes at meetings, study reports, meeting minutes and strategic plans
	Publicity of priorities and criteria	Number of times decisions and rationales appear in public documents	Media reports
	Functional mechanisms for appealing the decision	Number of decisions appealed, number of decisions revised	Observations/minutes at meetings, media reports, special reports
	Functional mechanisms for enforcement	Number of cases of failure to adhere to priority-setting process reported	Observations/minutes at meetings, media reports, special reports
	Efficiency of the priority-setting process	Proportion of meeting time spent on priority-setting, number of decisions made on time	Observations/minutes at meetings, annual budget documents, health system reports
Implementation of the set priorities	Decreased dissentions	Number of complaints from stakeholder	Meeting minutes, media reports
	Allocation of research resources according to priorities	Degree of alignment of resource allocation and agreed-upon priorities, times budget is re-allocated from less prioritised to highly prioritised areas, stakeholder satisfaction with the decisions	Annual budget reports, evaluation documents
	Decreased resource wastage/misallocation	Proportion of budget unused or allocated to non- priority research	Budget documents, research and evaluation reports
	Improved internal accountability/reduced corruption	Number of publicised resource allocation decisions	Evaluation reports, stakeholder interviews, media reports
	Increased stakeholder understanding, satisfaction and compliance with the priority-setting process	Number of stakeholders attending meetings, number of complaints from stakeholder, percentage of stakeholder that can articulate the concepts used in research priority-setting and appreciate the need for priority-setting	Observations/minutes at meetings, special reports, stakeholder satisfaction survey, media reports, stakeholder interviews, evaluation reports
	Improved internal accountability/reduced corruption	Number of publicised resource allocation decisions	Evaluation reports, stakeholder interviews, media reports
	Strengthening of the PS institution	Indicators relating to increased efficiency, use of data, quality of decisions and appropriate resource allocation, percentage of stakeholders with the capacity to set priorities	Training reports, evaluation reports, budget documents
	Impact on research institution goals and objectives	Percentage of of research institution objectives met that are attributed to the priority-setting process	Evaluation reports, special studies
Outcome and impact	Impact on health policy and practice	Changes in health policy to reflect identified priorities	Policy documents
	Achievement of health system goals	Research contribution to achievement of health system goals	Ministry of Health documents, Demographic and Health Surveys, commissioned studies
	Improved financial and political accountability	Number of publicised financial resource allocation decisions, number of corruption instances reported, percentage of the public reporting satisfaction with the process	Reports, media reports, interviews with stakeholders
	Increased investment in the health sector and strengthening of the healthcare system	Proportion increase in the health research budget, percentage of the public/researchers reporting satisfaction with the health research system	National budget allocation documents