Good Practices for Health Technology Assessment Guideline Development: A Report of the Health Technology Assessment International, HTAsiaLink, and ISPOR Special Task Force

Abstract

Objectives

Health technology assessment (HTA) guidelines are intended to support the successful implementation of HTA by enhancing consistency and transparency in concepts, methods, processes, and use, thereby enhancing the legitimacy of the decision-making process. This report lays out good practices and practical recommendations for developing or updating HTA guidelines to ensure successful implementation.

Methods

The task force was established in 2022 and comprised experts and academics from various geographical regions, each with substantial experience in developing HTA guidelines for national health policy making. Literature reviews and key informant interviews were conducted to inform these good practices. Stakeholder consultations, open peer reviews, and expert opinions validated the recommendations. A series of teleconferences among task force members was held to iteratively refine the report.

Results

The recommendations cover six key aspects throughout the guideline development cycle: (1) setting objectives, scope, and principles of the guideline, (2) building a team for a quality guideline, (3) defining a stakeholder engagement plan, (4) developing content and utilizing available resources, (5) putting in place appropriate institutional arrangements, and (6) monitoring and evaluating guideline success.

Conclusion

This report presents a set of resources and context-appropriate practices for developing or updating HTA guidelines. Across all contexts, the recommendations emphasize transparency, building trust among stakeholders, and fostering a culture of ongoing learning and improvement. The report recommends timing development and revision of guidelines according to the HTA landscape and pace of HTA institutionalization. Because HTA is increasingly used to inform different kinds of decision making in a variety of country contexts, it will be important to continue to monitor lessons learned to ensure the recommendations remain relevant and effective.

Highlights

• This report provides practical guidance for developing or updating health technology assessment (HTA) guidelines, especially in low- and middle-income countries where HTA is increasingly used as a priority-setting tool for healthcare resource allocation and achieving universal health coverage.

• We provide context-dependent recommendations across six domains for HTA guideline development: setting goals, building a team, defining the stakeholder engagement plan, developing content, putting in place institutional arrangements, and evaluating guideline success.

• Recommendations support the development and adoption of high-quality HTA guidelines that strengthen HTA institutionalization across different settings.

Introduction

Health technology assessment (HTA) can support many types of decisions, from establishing priorities for public investment and formulating healthcare reimbursement policies to evaluating social programs and setting health research strategies. Successful implementation of HTA depends mainly on three factors: having a systematic and best-practice approach to the assessment of health technologies over their lifecycle, creating a well-defined process for the appraisal of this assessment, and mapping a pathway for the incorporation of evidence-informed decisions into health policy or practice. HTA guidelines support successful implementation of HTA by creating consistency and transparency in concepts, methods, processes, and use, thereby enhancing the legitimacy of the decision-making process ^{1 , 2} and improving health outcomes through high-quality evidence-informed practices. ³

A HTA guideline is a document that provides recommendations about standards, which may or may not be mandatory, relating to some or all of the following:

• methods for evidence generation, assessment, appraisal, and presentation of results;

• principles for reporting the peer review of evidence (e.g., HTA dossiers);

• steps of a decision-making process;

• criteria and/or rules for appraisal;

• criteria and/or rules for decision making;

• stakeholder roles and responsibilities;

• and/or governance (e.g., mandate and jurisdiction of the policymaking process; mechanisms for stakeholder participation; mechanisms by which to ensure transparency, manage conflicts of interest, or to capacitate stakeholders with an oversight function).

HTA guidelines may be broad or narrow in scope and may be referred to by a variety of names, such as operational or procedural manuals, assessment, or reimbursement guidelines. In certain cases, a law or ordinance might include the components of a HTA guideline. Some countries distinguish between HTA methods and process guidelines. However, there is no standard practice across countries, and definitions of methods and process guidelines differ. ⁴ It is beyond the scope of this report to define what should constitute a HTA guideline. Rather, we set out a general good practice that aims to be relevant across different guideline scopes and contexts.

The existence of HTA guidelines is often used as an indicator of the degree of HTA institutionalization in a country. ^{1 , 5} Many high-income countries have published HTA guidelines, including specific guidelines for economic evaluation, and there is an increasing number of low-income and middle-income countries developing their own HTA guidelines. ^{6 – 8} Studies have reported a variable quality for existing guidelines, and adherence to them is mixed. ^{9 – 12}

To date, there has been no specific guidance for the development and successful adoption of high-quality HTA guidelines, aside from clinical guidelines. ¹³ In response, this report lays out good practices and practical recommendations for developing or updating a HTA guideline. It focuses on HTA guidelines that are developed at the national, subnational, or cross-country level. In particular, the report aims to highlight when different approaches may be appropriate for individual contexts and stages of HTA development. The primary audience for this report is those tasked with coordinating the development of HTA guidelines. However, it may also be relevant to a broader audience, including HTA researchers, private sector, patient groups, and organizations commissioning or providing technical or procedural assistance for guideline development.

Methods

This report was developed under a joint working collaboration between Health Technology Assessment International (HTAi), HTAsiaLink, and ISPOR – the Professional Society for Health Economics and Outcomes Research. To provide strategic advice on the development of this good practice report, a task force consisting of subject-matter experts and academics was established in early 2022. The first meeting of the task force was convened in July 2022. Membership of the task force was expanded in April 2023 to include experts with experience across diverse geographies. Members of the task force are listed in the Acknowledgment.

The task force held five teleconferences to develop the guidance report. A qualitative study, comprising interviews, and a nonsystematic literature review targeting eight countries was conducted to understand the factors influencing the success of HTA guidelines (pending manuscript). These countries – Canada, Indonesia, the Netherlands, South Africa, Tunisia, Vietnam, Brazil, and the Philippines – were purposively selected to represent diverse geographies and levels of HTA institutionalization. Preliminary findings were presented for stakeholder feedback during a side meeting at the eleventh HTAsiaLink Conference in Putrajaya, Malaysia in September 2023. Working groups comprising members of the task force then drafted each section of the guidance report based on findings from the qualitative study and published literature. Where existing evidence from the literature review and interviews was insufficient, recommendations were based on the expert opinion of working group members. The draft guidance report was reviewed and agreed upon by all task force members, with no conflicting opinions. A consultation was held with HTA researchers, policy makers, and academics at the fourteenth Conference of the International Society for Priorities in Health in Bangkok, Thailand during May 2024. An open invitation to review the draft recommendations was shared with ISPOR membership and participants of the International Society for Priorities in Health, alongside invited review by the HTAsiaLink Board Members and the HTAi Scientific Development Capacity Building Committee. The task force then made revisions. All task force members approved the final document. A summary of the key steps is illustrated in Fig. 1.

Figure 1.

A summary of key steps and milestones of the task force. HTAi indicates Health Technology Assessment International; ISPH, International Society for Priorities in Health; ISPOR, ISPOR – The Professional Society for Health Economics and Outcomes Research; SDCB, Scientific Development and Capacity Building committee.

Good practice for HTA guideline development

This section outlines good practices for developing or updating guidelines across six domains. A summary of key recommendations is shown in Table 1 and several case examples can be found in Box 1.

Table 1.

Overview of recommendations for Good Practice in HTA guideline development

Domain	Summary of good practice recommendations
1. Setting objectives, scope, and principles of the guideline	Before starting HTA guideline development, it is recommended that the guideline authors, project managers, and the commissioner/funder agree on the objectives, scope, and principles for the HTA guideline(s). A concise summary of what is agreed can be a point of reference throughout guideline development. HTA guidelines that aim to generally promote good practices for HTA are more likely to be successful in advancing the systematic use of HTA for decision making than those tied to a single-policy process (for example, a guideline for the process to define medicines included in the national benefit package).
2. Building the team for a quality guideline	HTA guideline development is best led by a core team with oversight from a separate advisory group. One of these groups should have expertise in technical content of the guidelines, whereas the other has expert knowledge of the policymaking context.
3. Defining the stakeholder engagement plan	Develop a stakeholder engagement plan adapted to the HTA guideline objectives and HTA context in the country. As a minimum, end users of the guideline and policy makers using the HTA outputs should be consulted. There should be transparent communication of the mechanisms for stakeholder engagement, the feedback received, and how comments were addressed.
4. Developing content and utilizing available resources	Where appropriate, contextualize guidelines from other settings to fit with existing decision-making structures and requirements. Leverage international HTA networks to draw insights from the experience of other countries, particularly when considering new approaches or techniques.
5. Putting in place appropriate institutional arrangements for implementation	Assign an agency or institution with the appropriate level of authority, independence, and technical capacity to oversee guideline implementation.
6. Monitoring and evaluating guideline success	Monitoring and evaluation of HTA guidelines may include (1) the extent to which the guidelines strengthen systematic and legitimate decision-making processes, (2) adherence, and (3) improvement in HTA quality with guideline use. Appropriate success indicators will depend on the specific objectives of the HTA guidelines.

HTA indicates health technology assessment.

Box 1.

Case studies from around the world of good practice for developing or updating guidelines.

Case study 1:

Setting objectives for development of the Health Technology Assessment Methods Guide 2022–7 in South Africa.

The HTA Methods Guide in South Africa was primarily developed to provide guidance on processes and methods to follow when considering inclusion and exclusion of medicines for the South African NEML. ¹⁰⁹ There was also a broader objective to sensitize and align stakeholders across the country on methods to promote comparability across medicine assessments. ¹⁰⁹ The guideline was developed by consultants with expertise in HTA and experience working with policy bodies in South Africa, with overall leadership from the Department of Health Essential Drugs Programme.

The objectives and scope were agreed by the guideline authors, NEMLC secretariat, Department of Health, and Ministerially Appointed Committees at the outset of the process, before the development of the first draft. This aligned expectations between the guideline authors and the policy users within the NEML process, by defining what would be covered by the guide.

Having a clear scope also facilitated the stakeholder engagement process. At the time of guideline development, there was significant stakeholder interest in HTA, following a draft National Health Insurance Bill proposing to use HTA to define the benefit package. A clear definition of the guideline objectives and scope of what the guideline would/would not cover helped to direct discussions with stakeholders during the extensive 1-year consultation process and served as a reference for deciding which comments to incorporate.

Case study 2:

Development team for the 2016 Guideline for Economic Evaluations in Healthcare (The Netherlands).

In the Netherlands, the ZIN is responsible for implementing and revising the economic evaluation methodological guidelines, as part of their role to advise the Ministry of Health (MoH) on the benefit package. ^{110 , 111} The third revision of the guidelines in 2016 aimed to promote comparable and quality economic evaluation, expanding the scope of the guideline to cover all health technologies and not just pharmaceuticals. ¹¹²

During guideline development, there was an authorship team, consisting of ZIN staff members with knowledge of the benefit package recommendation process and researchers, as well as an expert panel of senior academic staff. The expert panel was consulted several times prior to the launch of the guidelines. The resulting guidelines were considered to be at the forefront of economic evaluation, ¹¹³ and there was good adherence. ¹¹⁴ ZIN chose to follow the same model for the 2024 guideline revision.

Case study 3:

Stakeholder engagement plan for developing the Tunisian methodological guidelines for clinical data, pharmacoeconomic analysis, and budget impact analysis (2021–2).

The 2021–2 guidelines were developed by the INEAS, ¹¹⁵ which is a national scientific authority, under the auspices of the Ministry of Health. ¹¹⁶ At the time of guideline development, the scope of services provided under public insurance schemes was defined by the CNAM, under the Ministry of Social Affairs, ^{117 , 118} and no formal link has been established yet to use HTA to define the benefit package. However, HTA has been defined in the 2030 national health policy as the essential tool to support decision making in terms of inclusion and exclusion of new technologies and interventions in the benefit package.

During the guideline development process, INEAS adopted a strategic approach to stakeholder engagement. Draft guidelines were disseminated for public consultation, to build stakeholder awareness of HTA and the role of INEAS. The guideline development team also undertook targeted discussions with CNAM, pharmaceutical manufacturers, and other stakeholders, to move toward structured use of HTA to inform pricing and coverage decisions. Extensive staff time was dedicated to consultations and addressing stakeholder comments transparently, including publishing responses to all comments. ¹¹⁹ This engagement successfully built trust with stakeholders, and HTA is increasingly being used for value-based pricing, reimbursement, and other type of decisions related to novel technologies. ¹¹⁸

Case study 4:

Leveraging international networks to develop the 2020 Philippine HTA process guide and methods guide.

Following the 2019 Universal Health Care Coverage Act in the Philippines, the HTA Division (then the HTA Unit) dedicated time and resources for the development of a process and methods guide to support HTA institutionalization. ¹²⁰ The draft guidelines were based on international literature reviews, which were contextualized and adapted based on staff knowledge and exposure to some extent of the prior Philippine National Formulary processes. Beyond document review, the HTA Division also consulted on a regular basis with guideline developers and implementers in other countries, to learn from their challenges and learnings. These insights helped the team to develop practical guidelines that have been successfully launched with positive feedback from stakeholders. ¹²¹ The Philippines shall soon release the revised HTA implementing guidelines and the second edition of its annexes to incorporate improvements based on actual experiences, stakeholder feedback, and global best practices.

Case study 5:

Oversight function of the HEWG in Thailand.

In Thailand, HTA guidelines are not legally mandated but are required and endorsed by the Subcommittee for the Development of the NLEM. Compliance to the guidelines is overseen by a HEWG, ¹²² first appointed in 2007 by the NLEM Subcommittee. Members of the HEWG include clinical experts, public payers, and researchers with various expertise (e.g., epidemiology, health economics). ¹²³ Both the Subcommittee and HEWG for NLEM operate on a statutory basis with 3-year cycle terms. All appointed members of the subcommittee and HEWG must declare conflicts of interest and cannot be affiliated with political parties. ¹²⁴

The HEWG commissions domestic research teams from the NLEM research network to conduct relevant studies. ¹²⁵ Research teams throughout Thailand can apply to become a part of the NLEM research network and the application is reviewed and approved by the HEWG. During the process of conducting research studies, at least two rounds of stakeholder consultations must be held (one for the proposal and one for the preliminary findings), and at least one member representing the HEWG must be included in all consultations. Upon study completion, quality and compliance with the guidelines are assessed by at least two peer reviewers: one from the HEWG and one external reviewer. ¹²⁵ The reviewers and HEWG may request revisions from the commissioned research team if deviations from the guidelines are not well-justified.

CNAM indicates National Health Insurance Agency; HEWG, Health Economic Working Group; HTA, health technology assessment; INEAS, Tunisian HTA agency; NEML, National Essential Medicines List; NEMLC, National Essential Medicines List Committee; NLEM, National List of Essential Medicines; ZIN, National Health Care Institute.

Setting objectives, scope, and principles of the guideline

Agreement on the objectives, scope, and principles at the start of HTA guideline development sets mutual expectations between relevant stakeholders. It determines the subsequent stages of the guideline development process, informing decisions around the target users and appropriate language, structure, and content of guidance, and the stakeholder engagement plan. At a minimum, the relevant stakeholders to be involved in this step are guideline authors, project managers, and the institution commissioning or funding HTA guideline development. Depending on the local setting, other stakeholders may also be important to include (for example, members of policy committees or healthcare consumers).

The objectives define what the HTA guidelines aim to achieve. Our analysis found that HTA guidelines are more likely to be successful if they aim to set out best practices for HTA beyond the policy process in question (pending manuscript). For example, the 2009 National Policy on Health Technology Management and accompanying methodological guidelines in Brazil were developed not only to inform processes for federal level decision making but also to set out principles for HTA institutionalization at all levels of the health system, including state, municipal, and hospital levels. ¹⁴ After guideline publication, a network was established of HTA researchers and practitioners, which continues to play a prominent role in HTA institutionalization, ^{15 , 16} and there was a significant increase in the number of economic evaluation studies published, with quality improving over time. ¹⁷

At different stages of HTA institutionalization, HTA guidelines may have different objectives:

• For settings in which policies are not yet made through a systematic process, the objective may be to raise awareness of HTA and its relevance for policy making, to align stakeholders on how evidence-informed decisions should be made, and to instigate discussions around change to current practice.

• If a systematic process for policy making has been developed but is implemented on an ad hoc basis, the objective may be to improve consistency in the application of HTA and to uphold standards across decisions, while also promoting transparency in the process.

• Once HTA has been adopted as the basis for decision making, the objectives are often framed as improving responsiveness to specific decision-maker needs through a transparent and accountable process.

The scope determines the content that will be covered by the guidelines, whereas the principles lay out a set of guiding rules for how the guidelines will be written. Box 2 outlines specific considerations that may be relevant when defining the scope and principles.

Box 2.

Questions to consider when setting the scope and principles for HTA guidelines (note that this list is intended as a starting point and is not exhaustive).

Scope

• Which type of decision questions will the guideline address?

  For example: recommendations for medical devices or procedures to include in the benefit package, price negotiation for high-cost medicines on the national essential medicines list.

• Which components of the decision-making process will be covered by the guideline?

  For example: methods for evidence assessment, topic selection process, governance structures, and conflict of interest management.

• Who is the primary end-user of the guideline? What is their level of HTA experience and competency?

• Which resources are available to support guideline development (in terms of budget, staff time, and access to subject-matter experts)? What is the timeline?

Principles

• Which existing legal frameworks may influence the guideline development process?

  For example: regulations around engagement with private sector, legally defined roles of institutions, delegation of authority to local level.

• What level of detail and explanation will be included in the guidelines?

  1. If the concepts are relatively new to the target audience, more explanation of the theory or an annotated bibliography may be preferred.

  2. Limited institutional memory and high staff turnover may warrant a detailed description of each step, whereas settings with strong institutional memory and high level of staff expertise may provide greater autonomy to staff and instead concisely outline required or recommended steps.

• Will the guidelines prioritize best practices, feasibility for researchers, and/or approaches that are understandable to policy makers?

  1. For settings with difficulties accessing data or low technical expertise, the guidelines may include two layers of recommendations: best practices and approaches that can be used when the best practices cannot be followed.

  2. For policy processes in which a technical committee makes a recommendation to the decision maker, the guidelines may emphasize approaches that are understandable to the decision maker.

• To what extent will the guidelines leverage existing materials and processes from HTA and from the broader health decision ecosystem? ¹⁰⁸

• How will the guideline developers decide which parts of the guideline are mandatory, preferred, or optional?

HTA indicates health technology assessment.

Good practices

• Before starting HTA guideline development, it is recommended that the guideline authors, project managers, and the commissioner/funder agree on the objectives, scope, and principles for the HTA guideline(s). A concise summary of what is agreed upon can be a point of reference throughout guideline development.

• HTA guidelines that aim to generally promote good practices for HTA are more likely to be successful in advancing the systematic use of HTA for decision making compared with those tied to a single-policy process (e.g., a guideline for the process to define medicines included in the national benefit package).

Building the team for a quality guideline

Developing a high-quality guideline normally requires both a core team responsible for developing the guidelines (namely, guideline authors and project managers) and an independent group with an advisory oversight function. Our analysis of factors contributing to high-quality guidelines (those that promote conducting HTA in adherence to international standards) suggests that having a dedicated team focused on developing and/or revising the guideline is critical. The team should ideally have project management skills and the capacity to manage stakeholders, as well as technical expertise in various aspects of HTA (e.g., clinical or epidemiological assessments, economic evaluation, governance, and stakeholder deliberation). ^{18 , 19}

When revising or updating a guideline, if the core team has prior experience implementing guidelines within the target policy process, this may assist with the incorporation of lessons learned, improve quality, and support successful implementation. Particularly in settings where HTA is used on an ad hoc basis to inform decision making, it is important that the team leading the guideline development is perceived as legitimate and credible by a wide range of stakeholders. In many settings, this may require the team to have some level of independence so that they are not perceived as solely representing the interests of the payer (e.g., the insurance agency or government office financing the universal health coverage program). It is beneficial to separate the technical writing team from an oversight group advising on guideline development. Although this may be challenging in resource-limited settings, readers are encouraged to follow practical governance principles (e.g., Greer et al ²⁰).

Depending on the capacity and expertise available, one of the two models could be used:

1. the writing team should be subject-matter experts with oversight from stakeholders linked to the policy process, or

2. the writing team is linked to the policy process, and an advisory board of subject-matter experts provides oversight.

Good practices

• HTA guideline development is best led by a core team with oversight from a separate advisory group. One of these groups should have expertise in the technical content of the guidelines, whereas the other has expert knowledge of the policymaking context.

Defining the stakeholder engagement plan

A stakeholder engagement plan seeks to balance available time and resources for guideline development with the level of participation required for stakeholders to view the guideline development process (and consequently the resulting guidelines) as legitimate. A good stakeholder engagement plan can not only improve guideline quality and adherence but also strengthen understanding of HTA and its use in policy making. This is especially important for initial versions of HTA guidelines that tend to reshape decision-making norms and rules significantly.

Identifying which stakeholders to engage

Relevant stakeholders may vary, depending on the health system and the way in which HTA is implemented in each country. In general, principles for stakeholder engagement should provide a fair opportunity for participation while mitigating against conflicts of interest and accounting for available resources. ²⁰ When deciding the appropriate stakeholders to engage, it may help to consider the following four types of stakeholders.

1. Direct users of the HTA guidelines: These may include researchers or analysts undertaking the assessment of health technologies, private sector preparing HTA dossiers for submission, or the secretariat of policy processes applying HTA.

2. Users of HTA outputs: These may include policy makers, government agencies, appraisal committees, payers, or insurance agencies.

3. Stakeholders affected by the policy process(es) covered by the guidelines: These may include clinicians and healthcare professionals, technology developers, patients and health consumers, marginalized or vulnerable groups, civil society, or the public.

4. International experts, engaged for peer review to uphold standards or for sharing experience on guideline development and implementation.

In settings at an early stage of HTA development or where no HTA guideline currently exists, there may be better guideline implementation and adherence with extensive stakeholder engagement, in which all types of stakeholders are consulted (although there may be greater depth of engagement with the first two groups). Although extensive stakeholder engagement can be resource intensive, it facilitates transitions in the norms and rules for decision making to move toward structured, HTA-informed policymaking processes. This is important even in settings with high-level endorsement or laws for HTA. For such transitions to be successful, it is important that the engagement process is open, transparent, and receptive to change.

Fragmented and overlapping decision-making systems or political tension around policymaking processes may impede stakeholder collaboration and implementation of HTA guidelines. In these cases, HTA guidelines that have been successful in overcoming these barriers followed one of the two approaches.

Approach 1 involves bringing stakeholders together to establish areas of common agreement. The scope of the HTA guidelines covers only those areas of consensus and does not provide guidance on areas of contention. Stakeholders additionally agree on a guideline owner, who is responsible for overseeing implementation and making updates to the guideline. An example of this approach was the development of the first pan-Canadian pharmacoeconomic guidelines. ^{21 , 22}

Approach 2 involves creating an advisory board comprising influential stakeholders from other decision-making processes in the country (such as the Chair of the Essential Medicines List committee, National Immunization Technical Advisory Group, or provincial benefit package policy bodies, for example). This approach encourages learning and adaptation from existing policy processes, as well as building collaboration and mutual trust. An example of this approach was the development of the procedures for the Canada Drug Review. ²³

For resilience to political turnover, it may be important to involve independent influential stakeholders, such as professional bodies, in both of the above approaches. In certain cases, countries may have to consider statutory provisions for setting out the authority of the guideline owner (approach 1) or advisory body (approach 2).

Finally, in settings where HTA is fully accepted, updates to the guidelines may require consultation only with direct users of the guidelines, provided that revisions are not expected to be extensive. However, it can be beneficial to maintain channels for communication and feedback with other stakeholder groups to inform guideline development and periodic revisions. It will provide opportunities for these groups to contribute so that they do not feel excluded.

Mechanisms for stakeholder engagement and communication

Stakeholder engagement may usefully occur at all stages of the guideline development process. During the initial stages of guideline development, input from direct users of the guidelines and users of HTA outputs can help to identify appropriate objective(s) and scope, as well as communicating to end users that the guidelines are being developed or updated. Surveys and questionnaires can solicit input from a wide range of stakeholders, whereas targeted meetings with key stakeholders can provide greater depth.

Draft versions of the guidelines may be circulated to different stakeholder groups, ensuring that stakeholders have ample time to respond. Depending on the objectives of guideline development and the context of HTA, this may be to promote learning and discussion across stakeholders around the role of HTA in decision making, to improve quality of the guidelines, and/or to encourage implementation and adherence. If guidelines are circulated to nontechnical audiences, it is important to consider how to present the information in an understandable and accessible way that allows them to engage. For further discussion on effective stakeholder engagement, the reader may refer to existing literature. ^{24 – 29}

Once the HTA guidelines have been finalized, transparent communication of where they will be published, held, and archived can improve accessibility. Dissemination activities, such as workshops and seminars, may engage stakeholders directly and facilitate discussions on implementation. Other dissemination activities, such as publication of the guidelines in peer-reviewed journals or presentation during conferences can demonstrate that guidelines meet international standards, in addition to building awareness. Establishing feedback mechanisms, such as online surveys or feedback forms, allows stakeholders to provide further input on the guidelines, facilitating continuous improvement and adaptation.

Throughout all stages of guideline development, capacity-building activities, including training workshops, educational programs, mentorship, or creation of knowledge repositories, can support stakeholders to adhere to the guidelines. These activities not only help upskill individuals but can also serve as a platform for building HTA literacy, enabling stakeholders to contribute more effectively to the process and present an opportunity to collect feedback on the guidelines for future updates. Evaluation may be conducted to improve the effectiveness of stakeholder engagement, for example, using the PANELVIEW instrument. ³⁰

Building trust and legitimacy of the process

To establish trust and legitimacy, it is important to have strong leadership from the HTA agency or equivalent authority to promote transparency and accountability of the process. Certain stakeholders may be perceived to have a strong conflict of interest. The team will have to decide how to best represent diverse stakeholder views while mitigating bias. ^{20 , 31} Ideally, this leadership should be reinforced by support from high-level policy makers and other influential parties, to promote a culture of transparency and openness to change that can improve the legitimacy of HTA.

Good practices

• Develop a stakeholder engagement plan adapted to the HTA guideline objective(s) and HTA context. As a minimum, end users of the guideline and policy makers using the HTA outputs should be consulted.

• There should be transparent communication of the mechanisms for stakeholder engagement, the feedback received, and how comments were addressed.

Developing content and utilizing available resources

Many resources are available to facilitate the development of a good-quality guideline. When leveraging these resources, it is important to align them with the objective(s), scope, and principles of the proposed HTA guideline, as well as the prevailing decision-making context. To the greatest extent possible, the content of a HTA guideline should harmonize with existing decision-making procedures and/or previously established guidelines. This not only facilitates compatibility and acceptability among stakeholders but also improves implementation because the guidelines’ standards build upon familiar operating procedures and techniques. ³²

Table 2 ^{4 , 6 , 33 – 38} provides a nonexhaustive list of references that either provide links to existing country guidelines or compare HTA guidelines across countries. Table 3 ^{3 , 18 , 29 , 31 , 39 – 102} refers to international best practices and resources relevant for HTA guideline development. While using these resources, content should be adapted to local governance requirements, such as mandatory timelines for evidence submission or processes for stakeholder engagement, as well as the technical capacity for implementation of the local team.

Table 2.

Resources for accessing HTA guidelines from other countries or cross-country comparisons (resources may be general or domain-specific, and they are not exhaustive)

Key resource	Feature/focus	References
Repository for national HTA guidelines and resources
Guide to Economic Analysis and Research (GEAR) online resource	GEAR offers several useful features, including mind mapping for methodological solutions, comparative analysis of health economics guidelines (which involves comparing recommendations extracted from various health economics guidelines), and “Ask an Expert” function for personalized advice from specialists.	4,6
HTA tools & resources from the International Network of Agencies for Health Technology Assessment (INAHTA)	The site includes updated tools and resources from partners and agencies that are members of INAHTA. These resources cover a variety of materials, including practical handbooks, guidelines for best practices, and reports among others. The INAHTA international database also has over 23,380 HTA reports (as of August 2024).	33
The Regional Database of Health Technology Assessment Reports of the Americas (BRISA) from the Health Technology Assessment Network of the Americas (RedETSA)	Launched in November 2017, the database is overseen by PAHO and RedETSA. It contains over 3,200 HTA reports (as of 13 June 2024) gathered From RedETSA institutions, significantly enhancing their visibility and accessibility.	34,35
Synthesis of HTA guidelines (or components of HTA guidelines) from multiple countries
National Healthcare Economic Evaluation Guidelines: A Cross-Country Comparison	Comparison of economic evaluation guidelines across countries	36
Topic selection process in health technology assessment agencies around the world: a systematic review	Comparison of topic selection processes	37
Australian Government Health Technology Assessment Policy and Methods Review – Research and analysis papers	Scoping reviews and targeted consultation with INAHTA members were conducted in 27 countries for HTA policies, pathways, clinical evaluation methods, economic evaluation methods, and stakeholder engagement. The papers on this site provide information from 2023 on HTA processes and methods in countries with established HTA, with future plans for a living document that is periodically updated.	38

Table 3.

Checklists and guidance for different types of evidence, including reference materials related to procedural and governance aspects of HTA

Topic/theme	Available resources	References
Checklists and guidance for various types of evidence
Clinical assessment/systematic review	The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Handbook	39
	Cochrane Handbook for Systematic Reviews of Interventions	40
	Joanna Briggs Institute (JBI)‘s critical appraisal tools	41
	Methodological Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons	42
	Critical Appraisal of Systematic Reviews With Costs and Cost-Effectiveness Outcomes: An ISPOR Good Practices Task Force Report	43
Risk of bias assessment	The Cochrane tool for assessing Risk of Bias in randomized trials (the RoB tool)	44
	The Cochrane tool for assessing Risk of Bias In Non-Randomized Studies of Interventions (the ROBINS-I tool)	45
	A quality assessment tool for diagnostic accuracy studies (QUADAS–2 tool)	46
Reporting guidelines	Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist	47,48
	The EQUATOR Network offers a comprehensive array of reporting guidelines for different types of studies, including CONSORT for randomized controlled trials, STROBE for observational studies, and PRISMA for systematic reviews, as well as training materials and international events.	49
	Reporting Conflicts of Interest and Funding in Health Care Guidelines: The RIGHT-COI&F Checklist	50
Quality checklist	The Criteria for Health Economic Quality Evaluation (CHEQUE)	51
Modelling	Reports of the ISPOR-SMDM Modeling Good Research Practices Task Force	52–58
Budget Impact Analysis (BIA)	Principles of Good Practice: Report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force	59
	Principles of Good Practice for Budget Impact Analysis: Report of the ISPOR Task Force on Good Research Practices – Budget Impact Analysis	60
Epidemiological assessment	Global Burden of Disease (GBD)	61–63
	Guidelines and recommendations for ensuring Good Epidemiological Practice (GEP)	64
Uncertainty	The CHEERS-VOI checklist (included all CHEERS items with additional seven items for reporting value of information analysis)	65
	Uncertainty identification and management in regulatory and health technology assessment decision-making on drugs: guidance of the HTAi-DIA Working Group	66
	Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: A practical guide	67
Validation	EUnetHTA Practical guideline on validity of clinical studies	68
	Model Transparency and Validation: A Report of the ISPOR-SMDM Modelling Good Research Practices Task Force	54
Measurement of utility data	International Regulations and Recommendations for Utility Data for Health Technology Assessment	69
	Health utility estimation in children and adolescents: a review of health technology assessments	70
	Multi-attribute utility instruments recommended for use in cost-utility analysis. A review of national health technology assessment (HTA) guidelines	71
Costing and discounting	A general framework for classifying costing methods for economic evaluation of health care	72
	Discounting in Economic Evaluations	73
	A systematic review of discounting in national health economic evaluation guidelines: healthcare value implications	74
	Reference case for estimating the costs of global health services and interventions	75
Conduct of HTA for complex technologies	The INTEGRATE-HTA model: a stepwise approach to conduct HTA for complex technologies. It includes guidance for effectiveness, economic aspects, socioethical and legal aspects, patients’ preferences, context, and implementation	76
	A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance	77
Assessment of diagnostic tests and investigational technologies	Guidelines for preparing assessments for the Medical Services Advisory Committee. It provides guidance on assessing the clinical utility of diagnostic tests through use of the Linked Evidence Approach and logic models.	78
Equity, Legal, and Social implication (ELSI), as well as feasibility	The VALIDATE handbook: An approach on the integration of values in doing assessments of health technologies	79
	Ethical, Legal, and Social Issues (ELSI) checklist by the European Network for HTA (EUnetHTA)	80
	Methodological guidance documents for evaluation of ethical considerations in health technology assessment: a systematic review	81
	Steps toward improving ethical evaluation in health technology assessment: a proposed framework	82
	Integrating ethics in health technology assessment: many ways to Rome	83
Synthesis of qualitative evidence	Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions	84
	Appraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools	85
	The GRADE-CERQual approach: a method for assessing the confidence of evidence from reviews of qualitative research	86
	Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs, and relevant methods	87
HTA transferability	Considerations for transferability of health technology assessments: a scoping review of tools, methods, and practices	88
Reference materials related to procedural and governance aspects of HTA
Extensive collection of HTA resources	A Newcomer’s Guide to HTA, a collection of resources for early career professionals from HTAi: HTA 101 (Introduction); Research Protocol Writing; Ethical Evaluation; HTA in Hospitals; Patient and Social Engagement; HTA Publication, dissemination, and implementation support; Health professionals’ involvement; and Economic evaluation in health care	89
HTA institutionalization	WHO Institutionalizing health technology assessment mechanisms: a how to guide	3
	iDSI Health Technology Assessment Toolkit	18
	HTA Core Model by EUnetHTA	80
	Norwegian Institute of Public Health on technical guidance for HTA in LMICs	90
Deliberative processes	Designing and implementing deliberative processes (HTAi/ISPOR)	29
	Practical Guide on Evidence-Informed Deliberative Processes	91
	Multicriteria decision analysis (MCDA) to support health technology assessment agencies: benefits, limitations, and the way forward.	92
	GRADE Evidence to Decision (EtD) framework for different types of decisions (including coverage, health system and public health decisions)	93,94
Stakeholder engagement	Key issues for stakeholder engagement in the development of health and healthcare guidelines.	31
	Voice, agency, empowerment – handbook on social participation for universal health coverage	95
	Patient and citizen involvement	96
Conflict of interest management	Guidelines international network: principles for disclosure of interests and management of conflicts in guidelines	97
	A framework for defining, categorizing, and assessing conflicts of interest in health research	98
Expert opinion elicitation	Using expert opinion in health technology Assessment: A guideline review	99
	Reference case for expert elicitation	100
Data for research	The protection of personal data in health information systems – principles and processes for public health	101
	Data Privacy, Ethics and Protection: Guidance note on Big Data from United Nations Sustainable Development Group	102

Note: these are examples of key resources; the list is not exhaustive.

HTAi indicates Health Technology Assessment International; LMIC, Low- or Middle-Income Country.

Conducting learning discussions with agencies that have already developed HTA guidelines, both nationally and internationally, can be valuable in identifying and addressing challenges. This is particularly relevant when integrating new techniques or applying practices from other settings. Networks such as HTAsiaLink, The Health Technology Assessment Network of the Americas(RedETSA), International Network of Agencies for Health Technology Assessment, HTAi, and ISPOR serve as platforms connecting focal points from various organizations and countries.

Good practices

• Where appropriate, contextualize guidelines from other settings to fit with existing decision-making structures and requirements.

• Leverage international HTA networks to draw insights from the experience of other countries, regional initiatives, and institutions, particularly when considering new approaches or techniques.

Putting in place appropriate institutional arrangements

Effective use of HTA guidelines and good adherence require an authoritative body with technical capabilities and appropriate resourcing to oversee implementation. One of the main barriers to the implementation of HTA guidelines, particularly in settings with ad hoc decision making, is that guideline developers have limited ability to influence change in decision-making processes or low capacity to uphold standards set out in the guidelines (pending manuscript). If an appropriate body for overseeing guideline implementation and revisions does not exist at the outset, a guideline owner should be identified, with agreement on their role with relevant stakeholders. Such a body may be new or an existing body, such as a clinical licensing institution, with additional roles. Support for this body from relevant professional associations and stakeholder groups, together with a degree of independence from the ruling political party, can help to build HTA as an institution that is both resilient to election cycles and political turnover and more likely to retain professional and broader stakeholder support.

In resource-constrained settings, it may be challenging to identify an owner with the necessary expertise or one that is insulated from political influence. To overcome capacity constraints, an advisory group of experts (e.g., academics) may be set up to conduct a peer review of evidence, whereas good governance mechanisms can provide protection from undue external influence. ^{3 , 20} An explicit legal framework for the use of HTA in policy making is not mandatory for the successful implementation of HTA guidelines but having governance arrangements that are agreed upon by stakeholders, with clearly defined roles and responsibilities, can enhance effectiveness and foster compliance. Although many HTA functions may sit within government, it is recommended to establish appropriate oversight structures and mechanisms to manage conflicts of interest effectively, so as not to undermine guideline adherence and the overall effectiveness of the HTA system.

Good practices

• Assign an agency or institution with the appropriate level of authority, independence, and technical capacity to oversee guideline implementation.

Monitoring and evaluating guideline success

Guideline evaluation can inform future guideline revisions and increase impact. Evaluation may take the form of pilot testing before guideline release, continuous feedback mechanisms after guideline release, or impact evaluation 1 to 5 years after guideline publication. It may either be conducted by the guideline owner or by an external third party, depending on the purpose of the evaluation: Assessment by a third party can ensure objectivity and transparency, whereas internal assessment may play an important role in learning and capacity building of the guideline development team.

The success of HTA guidelines can be judged in a number of ways. One criterion is the extent to which they support the institutionalization of HTA within a country’s health system. For example, case studies might assess their impact on establishing or strengthening decision-making processes that are recognized by stakeholders for their legitimacy and effective evidence utilization. In 2016, an assessment of the Common Drug Review Process in Canada was carried out through document analysis and expert interviews. ²³ It revealed that, although recommendations followed a rigorous and transparent process, the Common Drug Review Process had only partly contributed to greater harmonization of recommendations at the provincial level, and difficulties remained covering pricing and common utilization of evidence for drug listing. For HTA guidelines, the emphasis of the analysis can be the impact of the guidelines on changes to HTA practice or its systematic use.

Another criterion for success is compliance, which involves assessing whether the guidelines are being followed. Possibilities include studies comparing evaluations before and after guideline implementation, reviewing HTA reports to evaluate adherence to the prescribed process, and interviews or observational studies conducted by independent groups. This evaluation can focus on a specific policy process or gauge wider impact of the guidelines by evaluating adherence across multiple policy processes or all published HTA studies in the country. For instance, a review conducted in Indonesia to assess the current methods, reporting practices, and the quality of evidence sources revealed that the methods guideline did not influence the standards applied in the eighty-four studies. ¹¹ Similarly, an analysis of reports from Brazil’s National Health Technology Assessment Commission highlighted that only a subset of the required criteria for decision making were discussed in Brazil’s National Health Technology Assessment Commission reports. ¹⁰³

For guidelines on evidence generation, a third criterion is the level of impact that guidelines have on the quality and validity of HTA studies after the guideline implementation. Unlike the point above, this criterion does not measure adherence to the guidelines themselves, but instead evaluates studies according to internationally recognized standards of quality, such as Criteria for Health Economic Quality Evaluation for economic evaluation studies ⁵¹ and Appraisal of Guidelines for Research and Evaluation (AGREE-II) for clinical practice guidelines. ¹⁰⁴ A review of national economic evaluation guidelines, for example, assessed quality using Drummond’s checklist. ¹²

Ultimately, success should align with the stated objectives of the guidelines. In settings without systematic decision-making processes in place, for example, the influence of guidelines on HTA institutionalization may be more important than the adherence of HTA studies to international quality standards. Guideline objectives should be explicit, to ensure a relevant evaluation of their actual impact on healthcare decision-making processes and quality of evidence-based practices.

Good practices

• Monitoring and evaluation of HTA guidelines may include (1) the extent to which the guidelines strengthen systematic and legitimate decision-making processes, (2) adherence, and (3) improvement in HTA quality with guideline use. Appropriate success indicators will depend on the specific objective(s) of the HTA guidelines.

Discussion

In this report, we lay out a set of resources and context-dependent recommended practices for developing or updating HTA guidelines. These recommendations seek to overcome barriers toward developing a HTA guideline that successfully promotes the use of HTA in decision making, including lack of a clearly defined scope, poor stakeholder buy-in, inappropriate institutional arrangements, or ineffective governance mechanisms. ^{3 , 19} A detailed checklist or prescriptive guide was not intended because international experience suggests that there is no one-size-fits-all solution. We have, however, provided an inventory of best-practice resources that may be used to inform the development or update of a HTA guideline.

Timing of the development and update of HTA guidelines will depend on the local HTA landscape and political judgments regarding the pace of change for further HTA institutionalization. Our recommendations highlight the importance of adapting the content of HTA guidelines dynamically as the HTA system evolves, as well as the need to align measurements of guideline success with the objectives of guideline development, which will vary across jurisdictions.

Regardless of specific context, some good practices should almost always be adopted. The report emphasizes the importance of mechanisms for transparency, building trust among stakeholders, and fostering a culture of ongoing learning and improvement. This may be achieved, for example, through separating the functions of the writing team and an oversight group, transparently soliciting and addressing feedback on the guidelines (even after publication), and assigning authority for guideline implementation and revisions to a mutually trusted body that has an interest to uphold best practices.

Although the report aims to be broadly relevant to HTA guidelines developed for different purposes, most of the evidence and expert opinion underlying the recommendations comes from experience developing HTA guidelines for national policymaking processes around the services and technologies to provide in the healthcare system. Given the growing interest in harmonizing HTA across countries ^{105 , 106} and expanding the remit of HTA in many jurisdictions, ¹⁰⁷ we recommend continuing to evaluate and collect good practices for HTA guideline development in these areas.

Beyond the recommendations and resources laid out here, HTA networks can play a key role in facilitating cross-country learning and sharing of good practices. Moving forward, such forums may provide a platform to monitor the use of these recommendations and collect experiences of HTA guideline development across different contexts, to inform periodic revision of these recommendations, ensuring their ongoing relevance.

Author contribution

Siobhan Botwright and Manit Sittimart contributed equally.

Concept and design: Botwright, Sittimart, Merlin, Surgey, Espinoza, Culyer, Oortwijn, Teerawattananon.

Acquisition of data: Botwright, Sittimart, Chavarina, Bayani.

Analysis and interpretations of data: Botwright, Sittimart, Chavarina, Bayani, Merlin, Surgey, Suharlim, Culyer.

Drafting of the manuscript: Botwright, Sittimart, Chavarina, Bayani, Surgey, Espinoza, Culyer, Oortwijn.

Critical revision of the paper for important intellectual content: Botwright, Sittimart, Chavarina, Merlin, Suharlim, Culyer, Oortwijn, Teerawattananon.

Provision of study materials or patients: Chavarina, Suharlim, Oortwijn.

Obtaining funding: Sittimart, Teerawattananon.

Administrative, technical, or logistic support: Botwright, Sittimart, Chavarina, Suharlim.

Supervision: Botwright, Espinoza, Teerawattananon.

Funding/support

This research has received funding support from the National Science, Research and Innovation Fund (NSRF) via the Program Management Unit for Human Resources and Institutional Development, Research and Innovation (grant number: B41G670025). The staff of HTAi and ISPOR supported ongoing meetings of the task force, as well as the 2-round peer-review process.

Role of the funder

The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Author disclosure

Author disclosure forms can be accessed below in the Supplemental Material section. None to declare.