Table 2.

Overview of substantive criteria, sub-criteria, and decision aspects

Criteria	Sub-criteria	Decision aspects
Health related balance of benefits and harms	General considerations surrounding benefit/effect	Benefits/effect/efficacy/effectiveness/impact [1, 2, 4–10, 12–15, 17–27, 29, 33–36]
		Health related Benefits/effect/efficacy/effectiveness/impact [1, 2, 4, 8, 11–15, 21, 23, 26, 31, 34, 35]
		Uptake of intervention [15, 20, 34, 35]
		Magnitude of benefit/effect/impact [2, 4, 11, 14, 18, 31, 33, 36]
		Additional or indirect effects [2, 6, 33, 34]
	Type and composition of effect/benefit/impact [2]	Impact on mortality, survival, longevity and life expectancy [1, 2, 4, 11, 16, 19, 21, 24–26, 28, 34–36]
		Last chance therapies [23, 24]
		Impact on morbidity and disability [1, 2, 16, 35]
		Potential changes in health consequences [24, 25]
		Impact on (health-related) quality of life [2, 8, 11, 12, 14, 19, 20, 22, 25, 26, 28, 29, 31, 33, 35, 36]
		Impact on patient-reported outcomes [2, 12, 16, 21, 26]
		Valuation of health outcomes by patients and desirability of the effects [2]
		Preventive benefits/effects or preventive approaches [1, 2, 4, 5, 16, 21, 25, 26, 31]
	Character of benefit or effect	Onset of effect and time until benefit [2, 11, 13]
		Duration, sustainability and lasting effect [2, 11, 13, 15, 27, 31]
	Individual and population level of benefit	Clinical benefits/effectiveness/impact [2, 3, 5, 11, 14, 17–22, 24, 25, 28, 31–33, 36]
		Individual level benefit, effectiveness or impact [1, 2, 4, 18, 21, 22, 25, 26, 36]
		Marginal benefits (for every patient) [20, 26]
		Population level benefit, effectiveness or impact [1–5, 9, 12, 16, 18, 19, 24–27, 31, 34, 35]
		Threshold effectiveness on populations (herd immunity) [15, 34]
	Balance of benefits and harms	Balance of (health) benefits and harms [5, 14, 18, 19, 23, 24, 34]
	General considerations surrounding harm/risk	General safety, risk and tolerability of intervention [2, 4, 5, 7–9, 11–17, 19–26, 29, 31, 33–36]
		Magnitude and likelihood of adverse events [26, 33–35]
		Valuation of health outcomes by patients and desirability of the effects regarding harms [34]
		Short and long term risk and safety profile [26]
		Over diagnosis and over treatment [16, 26]
		Stigmatization [2, 26]
		Risk of failure of intervention [15, 34, 35]
		Burden of treatment [2, 26, 33]
		Risk of inappropriate use [8, 16, 25, 26]
		Impact on disease patterns and reduced long-term effectiveness [15, 34]
		Other or additional adverse events [2]
	Health-related need and priority	Health-related needs: in general [1, 2, 11, 14, 22, 23, 24]
		Burden and impact of disease: in general [1, 2, 4, 7–10, 13–16, 21, 23–26, 33, 34, 36]
		Magnitude of the problem [4, 10, 25, 34]
		Burden of disease measured through epidemiological indicators [1, 2, 4, 5, 8, 11, 13–15, 21, 23–26, 28, 33–36]
		Size of affected population and number of potential beneficiaries [1, 2, 4, 5, 8, 10, 11, 13–17, 20–26, 28, 31–34, 36]
		Maximum potential for disease burden reduction [10]
		Severity of disease/condition: in general [1, 2, 4, 5, 9–12, 14, 16–19, 21–26, 28, 29, 31, 33–36]
		Severity of disease/condition: long term outcomes [34, 35]
		Severity of disease/condition: life threatening disease/condition and prognosis without treatment [1, 5, 11, 16, 21, 28, 29, 34]
		Severity of disease/condition: late stage or end-of life status of disease/condition [5, 21, 24, 25, 28, 29, 36]
		Outbreaks and epidemic potential [15, 34]
		Urgency and emergencies [1, 2, 13, 25]
Human and individual rights		Human rights considerations [2, 11, 12, 20, 36]
		Autonomy and informed consent [2, 11, 35]
		Privacy and confidentiality [26, 35]
		Intrusiveness of intervention [22]
Acceptability considerations	Perceived priority of the problem	Public perception of disease burden, disease risk or severity [15, 34, 35]
	Acceptability	Acceptability in general [2, 4, 6, 14, 15, 24, 26, 27, 34–36]
		Acceptability of cost and financial outcomes [2, 14, 25, 26, 34]
	Acceptability by beneficiaries	Acceptability by beneficiaries: in general [2, 6, 8, 11, 15, 16, 26, 35]
		Comfort, convenience and user experience [2, 11, 12, 14, 15, 20, 24–26, 31, 33, 35]
	Acceptability by those providing intervention	Acceptability by those providing intervention [15, 16, 34, 35]
	Social and cultural acceptability	Social and cultural acceptability [2, 4, 8, 9, 11, 15, 18, 21, 22, 25, 26, 31–34, 36]
		Ethical/moral acceptability [2, 6, 11, 12, 15, 18, 22, 25, 26, 32–34, 36]
	Stakeholder demand, interests and pressures	Advocacy and stakeholder (in general) interests and pressures [1, 2, 6, 14, 15, 21, 25, 33, 36]
		Demands, interest and pressures of the public [1, 2, 8, 14, 24–26, 36]
		Demands, interest and pressures by industry [2]
		Pressures, demand and interest of beneficiaries and patient representatives [2, 17, 20, 22, 25, 32, 33]
		Pressures, demand and interest of those providing intervention [2, 17, 20, 25, 26, 32, 33]
		Media attention and coverage [1, 32]
Societal considerations	Societal needs and priority	Social burden of disease (individual/population) [8, 18, 21, 29, 34, 35]
		Social needs [24]
		Economic burden of disease on society (in general) [32]
	Social and societal impact	Social impact or benefits [2, 3, 5, 6, 14, 21, 23–26, 31, 33, 35]
		Impact on non-health outcomes (in general) [15, 18, 26, 34]
		Impact on poverty [1, 2, 4, 14, 25, 26, 36]
		Relevance to social development of the country [10]
		Value of hope [11]
		Raise profile of condition [14]
	Impact on economy	Impact on economy [2, 25, 26]
		Impact on productivity and population in productive age [2, 12, 16, 25]
		Relevance to economic development of the country
		Innovativeness (potential to encourage innovation) [2, 9, 10, 14, 21, 23–25, 31]
	Environmental impact	Environmental and/or ecological impact of intervention (in general) [14, 16, 21, 29–31, 33]
	Impact on future generations	Impact on future generations [19]
Considerations of equity, equality and fairness	Equity and equality	Equity/equality considerations: in general [1–4, 9–11, 14, 15, 18, 19, 22–24, 26, 27, 29, 33–36]
		Fairness considerations: in general [2, 3, 6, 11, 16, 26, 36]
		Impact on (health) (in-)equity/(in-)equality [2, 6, 10, 14, 26, 31, 34, 35]
		Distribution of benefits and harms [10, 26]
	Accessibility	Accessibility in general [2–4, 6, 9, 14–17, 19, 21, 24, 29, 31, 33–35]
		Equity in accessibility [1, 2, 4, 6, 7, 11, 25, 26, 35]
		Physical and spacial accessibility [2, 4, 6, 16, 26]
		Timeliness of access (time spent waiting for treatment) [2, 3, 22, 36]
		Informational accessibility [11]
		Financial accessibility of intervention/Affordability [4, 9, 10, 12, 14, 19, 25, 26, 34–36]
		Affordability: risk of catastrophic health expenditure [11, 16, 25]
		Affordability: cost and Financial impact on beneficiaries [11, 13, 14, 16, 22, 25, 26, 31, 34, 35]
	Availability	Availability/lack of suitable alternatives [1, 2, 5, 11, 12, 14, 17, 19–25, 28, 32, 34–36]
		Limitations of alternative interventions [2, 11, 14, 15, 25, 26, 34, 35]
		Rare diseases/orphan disease [12, 14, 16, 21, 24, 28]
		Unmet needs [11, 12, 16, 21, 24, 29]
	Responsibility	Ability to reduce own health risk and conditions arising from patient behavior [4, 5, 11, 16, 24, 25]
	Non-discrimination	Non-discrimination [11, 23, 24]
	Consideration regarding specific populations	Consideration of high-risk populations [2, 11, 14, 26]
		Consideration of vulnerable populations [1, 2, 4, 11, 21, 25, 26]
		Consideration of Socio-economic status [4, 16, 25, 26, 36]
		Consideration of sex, gender, and/or sexual orientation [2, 6, 16, 25, 26, 34]
		Consideration of race/ethnicity [25, 36]
		Consideration of care giver responsibilities [29]
		Consideration of age-groups [1, 2, 4, 11, 14, 16, 26, 28]
		Consideration of place of living [11, 16, 25]
		Consideration of identity and ideology [25]
		Other group related considerations [11]
Cost and financial considerations	Financial burden of disease	Financial burden of disease or current intervention on health system [2, 15, 16, 25, 32, 34, 35]
	Cost and budget impact of intervention	Cost/budget impact: in general [1, 2, 4–6, 8, 9, 11–19, 21–23, 25–31, 33–36]
		Cost per unit/usage [1, 2, 5, 11, 15, 17, 20, 23–26, 33, 34, 36]
		Cost over time [20, 21, 25, 32, 34]
		Long term cost/budget impact [25]
		Overall cost/budget impact [1, 2, 11, 22, 26, 36]
		Direct cost [11, 25, 32, 34, 35]
		Indirect/additional/hidden cost [11, 34–36]
		Marginal cost [11, 25]
		Opportunity cost [2, 11, 21, 31]
		Impact on other spending/investments [2, 6, 14, 25]
		Investment/start-up cost [2, 22, 30]
		Operating cost [2, 30]
		Lifecycle cost of intervention/technology [30]
		Economic/financial benefits and cost-minimization potential [1, 3, 8, 11, 13, 22, 25, 27, 28, 34–36]
		Cost to/budget impact on government or society [1, 2, 5, 18, 25, 26, 29, 33, 36]
	Relation of cost and benefits	Relation of cost and benefits [1–7, 9–16, 18–26, 28, 31–36]
	Financial context	Appropriateness [2, 14]
	Financial feasibility	Affordability to health system Availability or lack of funds/funding [1–4, 6, 10–12, 14, 15, 21–25, 31, 34, 36]
	Financial sustainability	Financial sustainability of intervention and consistency of funding [4, 11, 15, 31, 34]
Feasibility and health system considerations	Health-system related needs and priority	Burden of disease on health system [2, 25, 26, 31, 33, 34]
		Needs of the health-system
	Feasibility and capacity to implement	General feasibility considerations [2–4, 7, 9, 10, 14–16, 22–27, 31, 33, 34, 36]
		Technical feasibility considerations [6, 26, 33]
		Practical feasibility considerations [4, 6, 26]
		Capacity to implement [1, 2, 6, 14, 21, 25, 29]
	Considerations of management and organization of health system	Availability of, capacity of and need for management and organizational structure [1, 2, 16, 22, 26, 34]
		Impact on management and organizational structure [2, 8, 11, 25]
		Logistical considerations [2, 15, 34]
		Availability of, capacity of and need for monitoring, surveillance and information system [34, 35]
	Resource considerations	Availability of, capacity of and need for resources (in general) [1, 2, 10, 14, 30, 36]
		Impact on resources (in general) [2, 22–24]
		Efficiency of resource use [18, 24, 31]
	Considerations of human resources and their skills	Availability of, capacity of and need for human resources [2, 10, 13, 14, 18, 22, 26, 30]
		Availability of, capacity of and need for skill levels/knowledge of human resources [2, 3, 14, 16, 22, 25, 30, 35]
		Impact on human resources and skill levels [2, 14, 18, 25, 31]
	Considerations of Non-financial physical resources (equipment, infrastructure)	Capacity of, availability of need for of physical resources and infrastructure [16, 18, 22, 25, 26, 34–36]
		Impact on non-financial physical resources and infrastructure [18]
	Interaction with and impact on health system	Impact on performance of health system and impact on other services [1, 2, 4, 6, 8, 10, 11, 14, 16, 18, 20, 22, 26, 31, 33–35]
		Interaction and compatibility with other health system components [1–3, 6, 14, 22, 25, 33–35]
		Ease of use, application and burden of intervention [13, 15, 22, 26, 35]
	Appropriateness within health system	Appropriateness (in general) [2, 3, 11, 21, 25, 26, 31, 36]
		Appropriateness of intervention for specific context [2, 16, 30]
	Legislative and regulatory considerations	Adherence to legal requirements, constrains and implications [2, 8, 10, 15, 16, 18, 25, 26, 31–35]
		Adherence to other directives, standards and requirements [2, 6, 14, 22, 30, 34]
	Political considerations	Political acceptability, interests and pressures [1–4, 6, 10, 11, 14, 18, 21, 25, 26, 33–36]
		Donor and global interests and pressures [2, 3]
		Political impact [2, 25, 33, 35]
		Alignment with priorities [1–4, 6, 13–15, 21, 23, 24, 26, 29, 30, 34, 35]
		Mission, mandate and goals of health system [2, 13, 14, 21, 25]
	Strategic considerations	Strategic planning and considerations [1, 2, 13, 19, 25, 31]
		Existing cooperation [2, 10, 14, 22]
		Decisions and practice of other institutions or stakeholders [2, 11, 17, 24, 29, 34]
		Alignment with recommendations, guidelines and standards [1–3, 7, 11, 14, 16, 21, 24, 26, 29, 34]
		Historical context and past decisions [1–3, 11, 13, 14, 17, 19–22, 25]
		Availability of incentives [1, 2]
		Impact on future decisions [6, 14]
		Keeping promises and commitments [2, 13]
	Characteristic of intervention	(technical) Complexity of intervention [2, 4, 6, 16, 21]
		Scalability of intervention [9, 27]
		Ability to evaluate intervention [2, 33–35]
		Reversibility of intervention [2]
		Flexibility of implementation [2]
		Uniqueness [21]
		Frequency of use and expected level of usage/activity [1, 2, 6, 8, 14, 25, 26, 32]
		Dependence on maintenance [26, 30]
		Sustainability of intervention utilization (e.g. Supply of parts, vaccines) [2, 14, 15, 27, 29, 34]
		Position of intervention in care pathway [5, 12, 20, 24]
		Additional uses of intervention [5, 21, 23–25]

1 Youngkong et al. 2009 [10], 2 Guindo et al. 2012 [6]; 3 Waithaka et al. 2018 [36]; 4 Wiseman et al. 2016 [35], 5 Fischer 2012 [43], 6 Wahlster et al. 2015 [13], 7 Ricciardi et al. 2015 [47], 8 Specchia et al. 2015 [34], 9 Hayati 2018 [42], 10 McGregor et al. 2014 [41], 11 MacLeod et al. 2016 [7], 12 Angelis et al. 2018 [48], 13 Barasa et al. 2015 [57], 14 Cromwell et al. 2015 [5], 15 Burchett et al. 2012 [4], 16 Varela-Lema et al. 2016 [38], 17 Vuorenkoski et al. 2008 [12], 18 Cowles et al. 2017 [50], 19 Golan et al. 2011 [33], 20 Erntoft et al. 2011 [51], 21 Friedmann et al. 2018 [28], 22 Ølholm et al. 2015 [37], 23 Stafinski et al. 2011a [19], 24 Stafinski et al. 2011b [20], 25 Mobinizadeh et al. 2016 [32], 26 Marsh et al. 2014 [27], 27 Rudan et al. 2017 [39], 28 Ghijben et al. 2018 [31], 29 Drake et al. 2017 [23], 30 Diaconu et al. 2017 [52], 31 Polisena et al. 2013 [22], 32 Noorani et al. 2007 [8], 33 Johnson et al. 2009 [53], 34 González-Lorenzo et al. 2015 [54], 35 Piso et al. 2009 [21], 36 Niessen et al. 2012 [11]

The publication by Niessen et al. (Ref. 36) is highlighted in italic, as it is the only study meeting all criteria of our critical appraisal tool