The role of stakeholder involvement in the evolving EU HTA process: Insights generated through the European Access Academy’s multi-stakeholder pre-convention questionnaire

ABSTRACT

Involvement of all relevant stakeholders will be of utmost importance for the success of the developing EU HTA harmonization process.

A multi-step procedure was applied to develop a survey across stakeholders/collaborators within the EU HTA framework to assess their current level of involvement, determine their suggested future role, identify challenges to contribution, and highlight efficient ways to fulfilling their role. The ‘key’ stakeholder groups identified and covered by this research included: patients‘, clinicians‘, regulatory, and Health Technology Developer representatives. The survey was circulated to a wide expert audience including all relevant stakeholder groups in order to determine self-perception by the ‘key’ stakeholders regarding involvement in the HTA process (self-rating), and in a second, slightly modified version of the questionnaire, to determine the perception of ‘key’ stakeholder involvement by HTA bodies, payers, and policymakers (external rating). Predefined analyses were conducted on the submitted responses.

Fifty-four responses were received (patients 9; clinicians: 8; regulators: 4; HTDs 14; HTA bodies: 7; Payers: 5; policymakers 3; others 4). The mean self-perceived involvement score was consistently lower for each of the ‘key’ stakeholder groups than the respective external ratings. Based on the qualitative insights generated in the survey, a RACI Chart (Responsible/Accountable/Consulted/Informed) was developed for each of the stakeholder groups to determine their roles and involvement in the current EU HTA process.

Our findings suggest extensive effort and a distinct research agenda are required to ensure adequate involvement of the key stakeholder groups in the evolving EU HTA process.

Introduction

In December 2021, the European Regulation on Health Technology Assessment (EU HTA R (EU)2021/2282), a fundamental pillar of the EU Pharmaceutical Strategy, was adopted by the Council and the European Parliament and is effective as of January 2022. The regulation aims to harmonize methodological standards for HTA and foster collaboration among the European health technology assessment (HTA) bodies [1–3]. It provides a unique opportunity to consolidate the various national HTA approaches and shape processes and methods to strengthen the European Health Union [3,4]. The harmonized procedure covers the clinical aspects of the assessment of a new technology while responsibility for conclusions on the added value (appraisal) and for decisions on pricing and reimbursement remains within the member states’ remit. Since January 2022, preparatory work has been initiated, including establishing a member states’ coordination group and respective subgroups, establishing a stakeholder network, and drafting guidance documents [1,3,4]. The European HTA regulation will be adopted in a stepwise approach. Commencing from January 2025, all cancer medicines and/or advanced therapy medicinal products (ATMPs) will be jointly assessed, followed by orphan drugs from January 2028 and all other centrally approved medicines from 2030 onward. (Article 7.2) [3]. Invasive or implantable, high risk medical devices with CE marking can also be assessed jointly, as of January 2025.

A service contract was entered with the European network for Health Technology Assessment (EUnetHTA) 21 joint consortium to support the stepwise implementation of the EU HTA regulation. This joint consortium, led by the ‘Zorginstituut’ (ZIN, The Netherlands), includes 13 European HTA bodies [4–6]. The EUnetHTA 21 work agenda covers various deliverables, including developing methodological and process guidance and conducting joint clinical assessments (JCA) and joint scientific consultations (JSC) [4,7].

Parallel to the activities conducted in preparation for implementing the EU HTA framework by the European Commission and the EUnetHTA 21 joint consortium, the ‘European Access Academy’ (EAA) was founded as a multi-stakeholder initiative. The EAA’s aim is to facilitate the implementation of the EU HTA R through the development of a joint European value framework for the assessment of innovative health technologies. Thereby, the EAA supports the regulation’s vision of achieving high-quality joint clinical assessments, inclusiveness, and transparency in the HTA process and predictability both for authorities and industry with the ultimate goal of facilitating patient access to innovative medicines and some medical devices at EU level [1,8].

Many challenges remain for the implementation of the EU HTA regulation and might represent a hurdle to the Commission’s medium-term goal of improving patient access to innovative medicines [2,4]. For the joint European HTA to provide an ‘additional benefit’ over the multitude of currently existing national assessments and to achieve a balanced, improved outcome for all stakeholders, key methodological and process issues need to be addressed [4,9]. During the inaugural convention of the EAA in May 2022, experts from all relevant stakeholder groups developed a research agenda focusing on how to approach these challenges [4,10]. Discussions of the EAA Working Group at the convention further crystallized that comprehensive stakeholder/collaborator involvement (such as patients‘, clinicians‘, regulatory and Health Technology Developer (HTD) representatives) throughout the EU HTA process will be fundamental for successfully implementing the EU HTA regulation and achieving its aim, as stated in §3 [1,3,4,11,12]. This focus on stakeholder involvement is corroborated by the EU HTA regulation verbiage as Article 3.7(j) of the regulation specifies that the Coordination Group shall ‘ensure appropriate involvement of stakeholder organizations and experts in its work’. Furthermore, Article 29 suggests the establishment of a ‘stakeholder network’ that ‘shall support the work of the Coordination Group and its subgroups upon request’ [3].

The joint European Medicines Agency (EMA) and EUnetHTA21 work plan may be an example for involvement of collaborators and stakeholders in the implementation activities [4]. This work plan continues the close collaboration between EMA and HTA bodies across Europe and aims to improve efficiency and quality of processes, whilst respecting the remits for different decision makers, and ensure mutual understanding and dialogue on evidence needs, to facilitate access to medicines for patients in the European Union [13,14].

However, successful implementation of the new regulation will require the involvement of a wide variety of stakeholders and collaborators, such as patient associations, medical societies, regulators, and Health Technology Developers (HTDs), to achieve assessments beyond mere technical discussions with real value for health-care providers and ultimately for patients. Considering the numerous overlapping issues and challenges, an integrated and coordinated strategy that includes all relevant stakeholders is needed [1,4].

Therefore, the aim of this study is to conduct a multi-stakeholder survey to provide insights on the current involvement of stakeholders/collaborators in the national HTA processes as well as on the role definitions and challenges for stakeholder involvement in the developing EU HTA procedure.

Methods

A five-step procedure was applied to design and execute the survey. An overview of the five steps is presented in Figure 1.

Figure 1.

A five-step procedure utilizing a modified Delphi approach was applied to develop and execute the survey.

Step 1: Identification of stakeholder groups

The ‘key’ stakeholder groups whose involvement is required for implementing the European HTA regulation were identified based on article 29 and recital 44 of the EU HTA regulation, which states that ‘the Coordination Group should engage and consult widely with stakeholder organizations with an interest in Union cooperation on HTA, including patient organizations, healthcare professional organizations, clinical and learned societies, health technology developer associations, consumer organizations and other relevant non-governmental organizations in the field of health’. [3]. In addition, for the purpose of this research, ‘regulators’ were included in the ‘key’ stakeholders group. While regulatory decision-making paves the way for any subsequent HTA-related discussion, regulators are not considered ‘stakeholders’ in the EU HTA process. Consequently, regulators are not included in the stakeholder definition of the EU HTA regulation. Regulatory assessment on one side and HTA assessment and appraisal on the other side are two related but clearly differentiated processes. Even if regulators were referenced as ‘stakeholders’ for the purpose of this research focusing on the implementation, ‘collaborator’ rather than ‘stakeholder’ would better reflect the interaction of regulators with the HTA process.

Further, consumer organizations and non-governmental organizations are also relevant stakeholders but were not considered ‘key’ stakeholders for the health technology assessment procedure for the purpose of this research.

Step 2: Design of the semiquantitative questionnaire

The questionnaire was drafted containing a combination of ranking items and free-text questions to collect insights into the involvement of the different stakeholders in the current national and the future EU HTA process. The involvement in the current national HTA system was scored on a Likert scale from 1 to 5 (where a rating of 1 was interpreted as low, a rating of 5 as high). Then, through free-text questions, respondents were asked for details on their involvement in the current HTA process and expected involvement in the future EU HTA process. The design of the item pool of the semi-quantitative questionnaire was based on previous EAA practice and conducted by two of the authors (EJ/JR) [1].

Step 3: Validation and further optimization of the structure of the online questionnaire

In a next step the draft questionnaire was shared and reviewed by the EAA Faculty. Two iterative cycles utilizing a modified Delphi method not including formal ranking and scoring of the panel’s responses were applied [15]. The proposed initial item pool and structure were refined after the first review cycle and subsequently, a second round of review was conducted. During the review cycle, a discussion arose within the Delphi panel of experts how to approach the current Market Access decision makers within the various EU healthcare systems, i.e., HTA bodies, payers, and health policymakers. They were obviously not part of the EU HTA regulation list of relevant stakeholders but still perceived to be a crucial driving force behind the evolving EU HTA process. Therefore, a decision was made to separate two distinct versions of the questionnaire (see supplementary files EAA Questionnaire_1 and EAA Questionnaire_2). While self-rating items were applied for the ‘key’ stakeholder groups (patients, clinicians, regulators, and HTDs), HTA bodies, payers, and health policymakers were asked to reflect their perception of the involvement of these ‘key’ stakeholder groups. In addition, during the Delphi review the questions were optimized to reduce potential cognitive burden which might lead to bias. After the second Delphi cycle, the electronic version of the questionnaire was developed.

Step 4: Circulation of the questionnaire

The multi-stakeholder survey was conducted prior to the Fall convention of the EAA (28/10/2022). The final online questionnaire was circulated to a wide expert audience, including all relevant stakeholder groups, via various channels such as LinkedIn or Twitter and direct e-mailing to the EAA network. The questionnaire was also promoted on different platforms, such as the EAA website and the EU Health Policy Platform [16]. Reminder e-mails were sent to the recipients, and various reminders were posted on the different channels. Several stakeholders attending the EAA convention in Brussels were supplied with a hard copy version to fill in prior to the start of the convention.

Step 5: Data handling and analysis of the questionnaire

Responses to the questionnaire were collected continuously from 16 July 2022 up to the day of the convention (28/10/2022). All responses received were pseudonymized prior to any analysis. Data were stored on a password-protected separate file and transferred into a predefined Excel file to conduct the analysis. As the rating item within the questionnaire was programmed to be mandatory, no missing data approach was required. Among the qualitative questions, those about the role and challenges of the different stakeholder groups in the future HTA system were mandatory. There was no imputation on missing data for the other qualitative questions as no statistical analyses were performed on these items. A preliminary analysis of the responses was carried out by EJ and JR to present interim results during the EAA convention. Predefined descriptive analyses were conducted on the quantitative items of the questionnaire and graphical presentation of the analysis concerning the participation and the involvement rating was prepared. Due to the exploratory nature of this research inferential statistics were not conducted. Framework method analysis was applied to the qualitative responses that were received [17]. Response analysis after the convention was carried out by LVH, including review of the free-text answers, dividing them into different tables, summarizing them and applying minor editorial adjustments for readability. Accuracy was confirmed through the four-eye principle by EJ. Complete free-text responses are available in supplementary tables s1–s4.

Results

Key stakeholder/collaborator groups

The four identified ‘key’ stakeholder groups for this research included i) patients and patients’ representatives; ii) clinicians, healthcare practitioners, and medical associations (clinicians’ representatives); iii) regulatory representatives; and iv) industry associations and health technology developers (HTD, industry representatives) as among the stakeholders mentioned in the EU HTA regulation these have the closest involvement in an HTA procedure and are affected most by the outcomes and implications. In addition, national HTA bodies, payer representatives, and policymakers that participated in the development of the EU HTA R were included in the survey. Finally, the questionnaire included an icon on ‘other’ stakeholders in order to capture respondents that did not identify with any of the pre-specified categories.

Questionnaire design

A key design modification that resulted from the Delphi cycle was the separation of two distinct versions of the questionnaire: the first version covering the self-perception of the four ‘key’ stakeholder groups (patients, clinicians, regulators, and HTDs) and a second version addressing the perspective of the ‘owners’ of the current HTA and Market Access Schemes i.e., the national HTA bodies, payers, policymakers (or any additional stakeholders) on the involvement of the ‘key’ stakeholder groups (‘external rating’ and ‘external perspective).’ An overview of the structure of the two questionnaire versions is presented in Figure 2, the complete questionnaires are available as supplementary materials.

Figure 2.

The final questionnaire was developed in two distinct versions: version 1 for self-rating by the ‘key’ stakeholder groups (patients’, clinicians’, regulatory and HTD representatives) and version 2 reflecting the external perception of those 4 ‘key’ stakeholder groups by HTA body representatives, payers, and policy makers. Any other stakeholders that participated in the survey were also provided with version 2.

Respondents

A total of 54 responses (n = 54) were received. Responses from the ‘key’ stakeholder groups included: patients’ representatives: n = 9 (France: 3, Germany: 1, Italy: 1, Spain: 1, not provided: 3); clinicians’ representatives: n = 8 (EU: 2, France: 1, Italy: 1, Spain: 1, not provided: 3); regulatory representatives: n = 4 (Netherlands: 2, Sweden: 1, not provided: 1); and HTDs: n = 14 (EU: 1, Global: 1, Croatia: 1, France: 2, Spain: 7, UK: 1, not provided: 1). Responses for the second version of the questionnaire: national HTA bodies: n = 7 (Italy: 1, Lithuania, 1: Norway: 1, Spain: 1, UK: 2, not provided: 1); payers: n = 5 (Germany: 3, Italy: 1, Spain: 1, not provided: 1); and policymakers: n = 3 (Cyprus: 1, Lithuania: 2). ‘Other Stakeholders’ that responded to the survey included academia: n = 4 (Belgium: 1, Bulgaria: 1, Germany: 2) (Figure 3a). Overall, n = 13 countries (Belgium, Bulgaria, Croatia, Cyprus, France, Germany, Italy, Lithuania, the Netherlands, Norway, Spain, Sweden, and United Kingdom) were represented in the responses. Interestingly, the highest number of responses was received from Spain (n = 11), followed by France and Germany with n = 6 submissions each. Several respondents specified to be representing a global (n = 1) or EU-wide (n = 3) organisation. Nine respondents did not provide any country information (Figure 3b). Due to the limited number of respondents per country, we did not analyze the received responses on a national level.

Figure 3.

a-b: Background information on submitted questionnaire responses. (a: Number of respondents per stakeholder/collaborator group; b: Number of respondents per country or region, where provided (9 respondents did not provide a country)).

Stakeholder/Collaborator involvement

An overview of self-rating vs. external rating of the four ‘key’ stakeholder groups is provided in Figures 4a-d. Self-ratings consistently scored lower compared to the respective external perceptions provided by HTA bodies, payers, and policymakers.

Figure 4.

a-d: Box Plots (Mean [x]; Medium; Max; Min; Upper and Lower Quartile) of stakeholder involvement self-rating versus external perception of respective stakeholder/collaborator involvement as rated by HTA bodies; payers; and health policy makers. Scale ranging from 1 to 5, per stakeholder group (a: patients’ representatives; b: clinicians’ representatives; c: regulatory representatives; d: HTD representatives).

A summary of qualitative responses covering the four ‘key’ stakeholder groups is included in Table 1, complete responses are available in supplementary tables s1–s4.

Table 1.

Key Insights generated within the qualitative part of the questionnaire regarding the content and key challenges for the involvement of the four ‘key’ stakeholder groups.

Focus of Involvement	Patients	Clinicians	Regulatory	HTDs
Content	Definition of unmet need Relevance of innovation for patients Definition of patient relevant endpoints Assessment of patient reported outcomes Acceptability of adverse events	Definition of PICO Scheme i.e., definition of patient population; comparator choice; clinical trial endpoints etc. Input on applied methodology Application of standards of ‘evidence-based medicine’ Involvement in early multistakeholder dialogues on HTA processes (with patients, regulators, and HTDs)	Approach to unmet medical need Early determination of evidence requirements for both benefit-risk evaluation and comparative benefit Joint scientific consultations including feedback on key elements of PICO scheme and study design that reflect both regulatory and HTA requirements	Design of PICO scheme i.e., definition of patient population; comparator choice; clinical trial endpoints etc. Design of applicable comparative evidence package
Key Challenges	Lack of standardized procedure for involvementLack of capacities & resourcesAlignment of EU umbrella organizations with national organizationsManagement of conflict of interestNo consistent recognition of the value of patient input	Lack of standardized procedure for involvement Requirement of additional resources Alignment of EU umbrella organizations with national organizations e.g., regarding guidelines Management of conflict of interest Heterogeneity in market access and clinical standards across Europe (e.g., eastern vs. Western Europe)	Lack of standardized procedure for involvement Timing of EU HTA Process and EMA regulatory process will overlap and might incur time conflicts Tailoring of EMA documents (e.g., EPAR) for utilization in the subsequent HTA processes	Lack of standardized procedure for involvement Time constraints of EU HTA process Limited opportunities of direct involvement before (e.g., in joint consultations) or after submission of a dossier Concern that EU HTA might result in many diverging PICO schemes

Patients’ representatives

An overview of patient self-rating and external rating is displayed in Figure 4a. The mean score of patient involvement ranged from lowest to highest: a mean score of 2.0 for self-rating by patients’ representatives, and 2.67 for external rating by HTA body representatives, 3.0 (by payer representatives, and 3.0 by policymakers. In addition, respondents were asked for their evaluation of patient involvement in the current and future HTA process using free-text questions. Qualitative points that were raised repeatedly were the need for systematic and structured involvement of patient representatives for insights on the medical context including the unmet need (in conjunction with clinicians’ representatives) and relevance of the innovation, on symptoms, Quality of Life measurements and outcomes, and on the acceptability of adverse events. Many respondents perceived clear differences in the level and quality of patient involvement across the various national HTA systems while there has been valuable involvement in Joint Scientific Consultations during the EUnetHTA Joint Action 3 Phase. Key challenges that need to be addressed to allow efficient, targeted, and sustainable involvement of patient representatives in a centralized EU HTA procedure included lack of capacity/resources, of expertise and training, alignment of national with EU umbrella organizations of patient associations, pragmatic management of conflict of interest, and a shared view and valuation of the role and benefits of patient input by all stakeholders in the process.

Clinicians’ representatives

An overview of clinician’s self-rating and external rating is displayed in Figure 4b. The mean score of clinician involvement ranged from lowest to highest: a mean score of 2.50 for self-rating by clinicians’ representatives, and 3.0 for external rating by HTA body representatives, 4.20 by payer representatives, and 4.33 by policymakers. Qualitative, free-text points raised for the involvement of clinical representatives were in large part highly similar to those for patient involvement, with a need for early structured and systematic multi-stakeholder involvement for input on medical questions such as standard treatments and unmet needs, relevance, and acceptability of Real-World Data, on the Patient/Intervention/Comparator/Outcomes (PICO) scheme as well as on appropriate methodology and specific criteria for evaluation in a given clinical context. Capacity constraints, methodological expertise, geographical coverage of the EU rather than a single country and conflict of interest were the main challenges, also comparable to those of patient representation. As positive example that should be utilized to learn from, the working model of collaboration of the European Medicines Agency (EMA) with clinicians was mentioned by several respondents.

Regulatory representatives

An overview of regulatory experts' self-rating and external rating is displayed in Figure 4c. The mean score of regulator involvement ranged from lowest to highest: a mean score of 2.0 for self-rating by regulatory representatives, 2.86 for external rating by HTA body representatives, 4.0 by payer representatives, and 4.33 by policymakers. In addition, respondents were asked for their evaluation of regulator involvement in the current and future HTA process using free-text questions. The collaboration between regulators and HTA organizations and hence involvement of regulators in HTA procedures was seen as very limited on a national level. While it was acknowledged that the remit of the benefit risk assessment done by regulators differs from that of the (additional) benefit assessment done by HTA, an increased exchange and collaboration e.g., on evidence requirements and how to meet them or on the concept and definition of unmet need would improve outcomes and efficiency. To this end, a systematic approach of joint scientific consultations, also in collaboration with regulators, should be implemented, despite capacity constraints, and the intent should be to generate outputs (e.g., assessment reports) that can also inform down-stream decision-making.

HTD representatives

An overview of HTD self-rating and external rating is displayed in Figure 4d. The rating ranged from 1 (low involvement) to 5 (high involvement). The mean score of HTD involvement ranged from lowest to highest: a mean score of 2.29 for self-rating by HTD representatives, and 2.83 for external rating by HTA body representatives, 3.33 by policymakers, and 4.0 by payer representatives. In addition, respondents were asked for their evaluation of HTD involvement in the current and future HTA process using free-text questions. HTD involvement in the current national HTA systems was seen as highly divergent in terms of procedures and requirements as well as on the level of involvement and transparency on outcomes, partly due to strict conflict of interest regulations hindering communication between HTDs and authorities and partly due to a lack of positive attitude towards each other. The centralized procedure as laid out in the EU HTA regulation was seen by many respondents as challenging for HTDs as they are asked to provide information on the innovation to be assessed in the submitted dossier, but there is limited to no opportunity for alignment on requirements before submission (due to the lack of joint scientific consultation capacities) and no opportunity to provide input on critical points such as PICO after dossier submission. In addition, it was noted that the involvement of HTDs, in particular on novel technologies e.g., for medical devices, was seen as a risk for patient access as only the manufacturers are experts in those technologies in most cases and assessors might not have the specific expertise to interpret and judge the provided evidence in all aspects.

Other Stakeholders

Responses were also received from four academic stakeholders. From an academia point of view, the mean score of perceived involvement for patient representatives was 2.0; for clinical representatives 3.67; for regulatory representatives 3.33; and for HTD representatives 4.25.

Discussion

Already during the initial stages of preparing for a centralized European Health Technology Assessment, in Work Package 6 (WP6) of the EUnetHTA project, stakeholder involvement was found to be necessary to ensure transparency of interests and processes, legitimacy, and utilization of EUnetHTA and its products [18]. Now, more than 10 years later and after finalization of the EU HTA regulation, comprehensive stakeholder/collaborator involvement is still considered a key success factor for the evolving EU HTA process, with respective wording anchored in the EU HTA R [3]. In line with those intentions, the European Commission is currently establishing the EU HTA Stakeholder Network [19]. The Fall Convention 2022 of the European Access Academy explored the various facets of stakeholder/collaborator involvement [20]. The insights and data presented in this article reflect the multi-stakeholder pre-convention questionnaire including i) the identification of relevant stakeholder groups; ii) the assessment of their current level of involvement; and iii) the determination of their suggested role in EU HTA and their specific level of involvement.

Identification of relevant stakeholders

The approach for the selection of stakeholders to be covered and definition of ‘key’ stakeholders in this research was developed in alignment with the Delphi panel. For this, the wording of the EU HTA regulation with its list of stakeholders formed the basis. However, additional stakeholders were included, considering their involvement in the process and impact of the outcomes, while others were excluded. This approach was developed to be stringently aligned with the EU HTA process as outlined in the regulation but to also ensure inclusion of all relevant affected parties.

Interestingly, four responses within the survey category ‘other’ were received from academic stakeholders, a stakeholder group that is neither listed in the EU HTA R nor directly addressed by any EU HTA implementation activity or public consultation to date. Therefore, academia was not included as a separate stakeholder group in this research. The responses received indicate academia as an important stakeholder, hence it should be included in the further prospects of the EU HTA R.

Assessment of current stakeholder/collaborator involvement

A key question that was discussed during the design phase of the questionnaire was the scope of the quantitative item used to determine the level of ‘key’ stakeholder involvement. To be applicable as a baseline metric, it was decided to focus on the ‘current’ status of involvement that e.g., does not yet reflect the EU HTA Stakeholder Network initiative that was announced after the conduct of the questionnaire.

Strikingly, self-rating for involvement was very low across the four ‘key’ stakeholder groups (patients, clinicians, regulators, and HTDs). With a mean of 2.5 clinicians showed the highest score, the mean score for HTDs and regulators was only 2.0. In contrast, external rating by HTA bodies, payers, and health policymakers revealed consistently higher values than the respective self-ratings. Due to a rather low and imbalanced number of responses received between stakeholder groups, as well as the overlap within the box plots, these results may be considered indicative only. However, they do suggest a clear trend and the authors consider this trend meaningful. Differences in self-perceived vs. external ratings are well known – patient–reported vs. clinician-reported outcome measures frequently differ considerably [21]. In the stakeholder management field, much is discussed and published on the value of stakeholder involvement in different contexts and on how to approach it, however details on how the involved parties experience and evaluate the stakeholder journey are scarce and there appears to be no general rule on the perception of involvement [22]. Hence, it is an important finding of this survey that key stakeholders/collaborators consider themselves being involved on a low level, and lower when compared to external views by HTA bodies, payers, and health policymakers. Bearing in mind that in the EU HTA process no public hearing is foreseen for stakeholders to provide their input during the process, this finding requires urgent action to support the claim that the EU HTA R ensures comprehensive stakeholder involvement.

Suggested role of the stakeholders/collaborators in the evolving EU HTA process

A key element of the survey included qualitative questions covering the suggested role of the ‘key’ stakeholder groups in the evolving EU HTA process. Qualitative responses were manifold and are summarized in Table 1. Based on the generated insights, a RACI chart covering responsibility (R), accountability (A), consultation (C), and information status (I) was developed by the Delphi panel, summarizing the suggested role of the various stakeholders/collaborators as per current details in the EU HTA regulation (Table 2) [3,23]. The EU Cooperation on HTA will, however, learn empirically and consequently the roles and contributions of different stakeholders are likely to change as the system evolves. Hence, the definitions provided in Table 2 are an indication of the current status but should be considered open and likely to be adjusted. In addition, the role of the various stakeholder/collaborator groups changes over the course of an assessment procedure due to different scopes throughout early vs late technology assessment stages.

Table 2.

RACI (Responsible/Accountable/Consulted/Informed) Chart for the role definitions of the relevant stakeholder/collaborator groups for the PICO (Population/Intervention/Comparator/Outcomes) categories in the EU HTA procedure as defined in the EU HTA Regulation.

	Patients	Clinicians	Regulators§	HTDs	EU – HTA
Population	C	C (important)	C (optional)	I (evt C)	R*
Intervention	I (optional)	I	n/a	A/R	I
Comparator	C	C (important)	C (optional)	I (evt C)	R*
Outcomes	C (Important)	C (important)	C (optional)	I (evt C)	R*

Note: *: HTA bodies are consulted (C) during the development phase; accountability (A) resides with national HTA bodies. §: For the regulatory approval process, regulators are A/R throughout the process which is creating the frame for EU HTA to operate. evt: eventually.

While ultimate accountability for the HTA appraisals remains within the remit of the national HTA bodies, the technology assessment will be jointly conducted at the EU level. Within the centralized HTA procedure, some involvement of stakeholders/collaborators is planned during the initial stages, but no interaction after the PICO survey and notably no public hearing before adoption of the assessment report. Leveraging the well-established PICO scheme, it became obvious, however, that clear processes and pathways should be in place for patients‘, clinicians‘, and HTD representatives to provide their input regarding e.g., the target population, comparator choice, and patient relevant outcomes. Importantly, it should be ensured that this input will indeed be utilized. It was considered a risk that submissions, e.g., during the PICO survey, might solely be collected to satisfy the requirements of the regulation, but that this input would not actually be considered for the decisions being made. In the current JCA process, involvement of patients and clinicians is already foreseen in the current JCA process. On the other hand, a key perception within the HTD group was such that based on the currently suggested procedural framework they would be informed rather than consulted. Thus, a key opportunity to provide feedback (e.g., via a written statement or in an oral hearing after submission of the dossier) beyond a mere fact-check throughout the process would be missed.

Limitations and further research agenda

While this research provides timely and relevant insights into the important question of appropriate stakeholder/collaborator involvement in the EU HTA process its overall scope might still be considered exploratory. Although information regarding the questionnaire was widely shared via social media and expert networks, the number of respondents was limited to n = 54, and a potential imbalance of views between the ‘key’ stakeholder groups cannot be excluded, with the numbers of responses ranging from n = 4 (regulatory representatives) to n = 14 (HTD representatives). However, this was not unexpected due to the field being highly specialized and as individuals are thought to rather represent organizational views – which need to be developed and approved in formalized internal processes – than their own perspective. Based on high level of active participation in other EAA activities, e.g., the biannual conventions, the relatively low number of responses is not interpreted by the authors as low level of engagement or low interest in the topic.

By sharing the invite to the questionnaire by email, via social media and online platforms as well as offering the opportunity to submit hardcopy responses at the convention, we aimed to minimize selection bias for respondents utilizing online channels as compared to those favoring hardcopies. In addition, stakeholders involved in the topic of EU HTA are likely to utilize online resources and tools as information, stakeholder meetings, webinars etc. are all provided online (e.g., by the European Commission or by the EUnetHTA 21 project). However, as the majority of submissions were provided online, it cannot be entirely excluded that there might be a selection bias. Further, there were submissions both from national representatives and from EU-wide or global representatives, which may have impacted the feedback obtained. Also, the fact that the countries and therefore specific national experiences represented in each of the stakeholder groups differed might have included bias in the outcome. Nevertheless, geographical coverage as well as stakeholder coverage revealed the inclusion of a wide variety of perspectives. Lastly, the selection of stakeholders, definition of ‘key’ stakeholders, ‘stakeholders’ at all – as compared to collaborators – and of ‘other’ involved stakeholders used for this work might differ from that used in other research and could introduce a potential source of bias. Earlier work by Vella Bonanno et al. identified the member states’ authorities as important stakeholders who need to be involved and prepared to collaborate in a joint European HTA process [9].

Despite the limitations and potential sources of bias included in the sample and analysis as detailed above, the authors believe that the results, while indicative, provide important insights and the basis for the development of a work agenda to approach the identified challenges for each stakeholder’s role.

Further research on optimizing stakeholder/collaborator involvement is required to determine the most relevant activities of each of the identified groups to fulfill their respective roles within the developing EU HTA process. In particular, more research is required to further define the roles, challenges, and activities to overcome those challenges for each stakeholder/collaborator group in the EU HTA system, e.g., capacity building, training on HTA, etc., and by learning from similar new process implementation in the EU (e.g., the European Union Medical Device Regulation, Regulation (EU) 2017/745 (EU MDR)) or beyond (e.g., the Health Technology Assessment Policy and Methods Review in Australia) [24,25]. In addition, it remains to be seen if the low ratings of stakeholders’ self-perceived involvement in the EU HTA process improve over time, e.g., by including the same question in upcoming research surveys targeting EU HTA stakeholders/collaborators. In order to address the limitations as discussed above, future research might e.g., analyze feedback from a larger sample of respondents, ensure equal representation of countries and stakeholder/collaborator groups while excluding or separately analyzing multi-national representatives, and compare different operationalizations of the term ‘stakeholder’.

In conclusion, our findings suggest that extensive effort and a distinct research agenda are required to facilitate adequate involvement of the key stakeholder/collaborator groups in the evolving EU HTA process. The ultimate goal is ensuring that those technically and ethically highly complex and challenging HTA procedures are based on a broad societal consensus across EU countries.

Funding Statement

JR & EJ received an unrestricted grant from Abbvie, AstraZeneca, Novartis, Roche, Sanofi, and Seagen that partially funded this research. None of the other authors received any funding for their participation in preparing this manuscript.Financial Disclosures: JR & EJ received an unrestricted grant from Abbvie, AstraZeneca, Novartis, Roche, Sanofi, and Seagen that partially funded this research. LVH: no CoI, TD: no CoI, WVD: no CoI, SS: no CoI, IH: no CoI, RG: no CoI, MT: no CoI, CD: as strategic and legal consultant regularly receives honoraria for consulting from numerous health technology developers, JD: as strategic and legal consultant regularly receives honoraria for consulting from numerous health technology developers, AC: no CoI, FH: no CoI, MP: no CoI, MB: no CoI, PM: no CoI, AS: no CoI, WG: no CoI, FG: no CoI, SC: no CoI, JRy: employed by AstraZeneca, PD: employed by Abbvie, OSM: no CoI.

Disclosure statement

The views expressed in this article are the personal views of the author(s) and may not be understood or quoted as being made on behalf of or reflecting the position of the regulatory agency/agencies or organizations with which the author(s) is/are employed/affiliated.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/20016689.2023.2217543