Evaluation of Ethical Analyses in Seven Reports from the European Network for Health Technology Assessment

Abstract

Objectives.

Ethics has been considered among the core domains of health technology assessment (HTA), but there are still disputes regarding ethical analysis. This study aimed to examine full final reports of the European Network for Health Technology Assessment (EUnetHTA) in terms of their compliance with the ethical methodology and ethical perspective of the HTA Core Model®.

Methods.

The study examines seven full final HTA reports of EUnetHTA written based on the methodology proposed in the HTA Core Model®. The reports were analyzed using the following parameters: competency of the person/group who conducted ethical analysis, assessment elements, and the methodology of ethical analysis.

Results.

The results show that, although the HTA Core Model® helped to standardize the final reports of the assessment, there are still concerns regarding the competency of the ethical analysis team, the perspectives on the purpose of ethical analysis, data sources and viewpoints of various stakeholders, use of ethical analysis methodology, and the evaluation of the ethical appropriateness of the entire HTA process.

Conclusions.

The HTA Core Model® helped to standardize the final reports on the HTA; however, not all issues with the content and outcomes were solved. The lack of expertise in ethics and insufficiency of the teams regarding ethical analysis are other existing problems. This study also demonstrated that stakeholder viewpoints in general and patient perspectives, in particular, have been overlooked in the HTA process.

Significant technological advances, such as in vitro fertilization, intrauterine gender determination, anti-aging interventions, organ transplantations, genetic treatments and interventions, life-sustaining interventions, and stem cells have created the possibility of dramatically changing the practice of medicine. Today, an intervention used to promote health, or prevent, diagnose, treat, or rehabilitate diseases or organize healthcare delivery, is considered a health technology (HT) (1;2). The improvements in medical technology have enhanced the role of health technology assessment (HTA).

Ethics has been considered an essential element of HTA. The ethical considerations of new technology emerge mainly from the unpredicted, unknown, unintended, or unwanted consequences of using that technology (3). Heitman argued (4) that ethical issues in HTA “…could be grouped into broad categories of normative concepts, diagnosis, prevention and therapy, research and the advancement of knowledge, and allocation of resources, … evaluated in terms of the integrity of the project’s goals, procedures, and effects, and evaluators’ open and self-critical acknowledgment of their purposes”.

Although ethics is considered among the core domains of HTA, there are still disputes regarding ethical analysis (EA) in HTA. Until recently, most HTA reports either did not involve EA or did not mention the ethical implications of the HT (5–7).

EUnetHTA and the HTA Core Model®

The European Network for Health Technology Assessment (EUnetHTA), the facilitator of HTA collaboration in Europe, is structured to support efficient production and use of HTA and to provide a science-based platform upon which HTA agencies can exchange and develop HTA information and methodology, and share guidelines for HTA teams (3). The latest version of the methodology guideline by EUnetHTA is the HTA Core Model® version 3.0 (the HTA Core Model®). Since the realization of the HTA Core Model® in 2016, it has been used in many HTA reports.

The HTA Core Model® contains a standard set of questions which aim to define research questions in the HTA within a standard structure. The HTA Core Model® states that the involvement of ethics is not limited to the EA of the proposed HT, and emphasizes that the ethical aspects should be addressed in a broad sense to cover the inherent values and interests that inform the decision to perform the HTA on that particular technology over other options and lists the ethical issues to be considered before starting the assessment, during recruitment of the assessment team, and when writing the final report (3).

The topics in the EA domain are as follows: benefit-harm balance, autonomy, respect for persons, justice and equity, and ethical consequences of HTA. Table 1 shows the topics and the ethical issues they contain (Table 1).

Table 1.

Topics and Issues for the Ethical Analyses Domain of the HTA Core Model®

Topics	Issues
Benefit-harm balance	What are the symptoms and the burden of disease or the health condition for the patient?
	What are the known and estimated benefits and harms for patients when implementing or not implementing the technology?
	What are the benefits and harms of the technology for relatives, other patients, organizations, commercial entities, society, etc.?
	Are there any other hidden or unintended consequences of the technology and its applications for patients/users, relatives, other patients, organizations, commercial entities, society etc.?
	Are there any ethical obstacles to evidence generation regarding the benefits and harms of the intervention?
Autonomy	Is the technology used for individuals that are especially vulnerable?
	Does the implementation or use of the technology affect the patient’s capability and possibility to exercise autonomy?
	Is there a need for any specific interventions or supportive actions concerning information to respect patient autonomy when the technology is used?
	Does the implementation or withdrawal of the technology challenge or change professional values, ethics, or traditional roles?
Respect for persons	Does the implementation or use of the technology affect human dignity?
	Does the implementation or use of the technology affect the patient’s moral, religious, or cultural integrity?
	Does the technology invade the sphere of privacy of the patient/user?
Justice and equity	How does implementation or withdrawal of the technology affect the distribution of health care resources?
	How are technologies with similar ethical issues treated in the health care system?
	Are there factors that could prevent a group or person from gaining access to the technology?
Legislation	Does the implementation or use of the technology affect the realization of basic human rights?
	Can the use of the technology pose ethical challenges that have not been considered in the existing legislations and regulations?
Ethical consequences of the HTA	What are the ethical consequences of the choice of endpoints, cut-off values, and comparators/controls in the assessment?
	Are there any ethical problems related to the data or the assumptions in the economic evaluation?
	What are the ethical consequences of conducting the technology assessment at this point of time?

HTA, health technology assessment.

In the HTA Core Model®, the designation of one person to facilitate and report the EA is recommended; it is preferable that this person is an ethics expert. However, it is stated that most importantly, scientific and clinical experts should be included in EA (3).

Methodology

This study examined the full HTA reports from EUnetHTA. Although there are several HTA reports published by various agencies worldwide, only the reports from EUnetHTA are included in this study. There are three reasons for this: (i) the scope of the agency: EUnetHTA is an umbrella agency that pursues quality, transparency, transferability, and objectivity in HTA, and hence, the reports produced by EUnetHTA have been prepared with a high level of attainable standards. In addition, EUnetHTA is the network for HTA across Europe, which collaborates with several European HTA organizations, and indicates that their work embraces diverse areas of expertise and has a prevalent impact area. Therefore, evaluating these reports would provide a general understanding related to how EA is perceived and conducted by experts across the EU. On this basis, this study examines the final published reports of EUnetHTA in their entirety and evaluates the compliance of their EA sections using the ethical methodology and perspective of the HTA Core Model®. (ii) EUnetHTA reports are systematically published online, which makes it possible to access the necessary data for the study. (iii) EUnetHTA reports have EA sections that have been written using the methodology proposed in the HTA Core Model®, and they reflect its practical use.

Considering the lack of consensus on methodology for EA in HTA, the ethical dimensions of these reports provide good examples of how a methodological framework could work for EA in HTA. For this study, these variables comprised the primary rationale for choosing the full final reports of EUnetHTA. In the “assessment” section of the official website of EUnetHTA, there are assessment reports of EUnetHTA JA3 (2016–20), EUnetHTA JA2 (2012–15), EUnetHTA JA1 (2010–12), and the EUnetHTA Project (2006–08). Among these, seven are full HTA reports with final reports (8–15). Two of the assessments were conducted during the EUnetHTA 2006–08 period, two during the EUnetHTA JA1 2010–12 period, and three during the EUnetHTA JA2 2012–15 period (Table 2). All rapid assessments were excluded from the study regardless of the year in which they were produced.

Table 2.

Full HTAs Included in the Study

EUnetHTA program year	Name of the HTA	Publication year	Intended use of technology	Target population condition, sex, age, and breadth
EUnetHTA 2006–2008	1. Core HTA on multislice computerized tomography angiography	2008	Diagnosis	Not specified
	2. Core HTA on Drug Eluting Stents	2008	Treatment	Not specified
EUnetHTA JA1 2010–2012	3. Abdominal Aorta Aneurysm Screening	2013	Screening	Elderly Any sex
	4. Prognostic tests for breast cancer recurrence	2013	Diagnosis and treatment	Any age Females
EUnetHTA JA2 Full/Comprehensive assessments 2012–2015	5. Fecal Immunochemical Test versus guaiac-based fecal occult blood test for colorectal cancer screening. (Core HTA 1)	2014	Screening	Adults and elderly Any sex
	6. Use of Intravenous immunoglobulins for Alzheimer’s disease including Mild Cognitive Impairment. (Core HTA 2)	2015	Treatment	Elderly Any sex
	7. Structured telephone support for adult patients with chronic heart failure. (Core HTA 3)	2015	Prevention	Adults and elderly Any sex

EUnetHTA, European Network for Health Technology Assessment; HTA, health technology assessment.

These seven reports were included in the study and were read and analyzed in accordance with the following parameters:

1. Competency of the person/group who conducted EA: The competency of the person/group conducting EA is evaluated based on their training and expertise in ethics. The curricula vitae of the analysts were examined to determine if they had the particular training and experience to enable them to perform the EA task. The analysts were considered competent if they had a master’s or doctoral degree in ethics or if their curriculum vitae provided evidence of training or job experience that demonstrated their familiarity with EA and the relevant methodology. The team was considered competent if one person on the team had the defined qualifications.

2. Focus or aim of EA: Although the HTA Core Model® does not require the analysts to specify the focus or the aim of the EA section, most reports contain this element. It was considered that assessing the reports for this criterion would reveal important results regarding the EA team’s perspectives on their task, which would have an impact on the scope and content of the EA they performed. This assessment criterion was applied by reading through the reports to determine whether a specific focus or aim was declared for the EA section. If such a declaration emerged, it was further evaluated for its compliance with the HTA Core Model®’s perspectives on EA in HTA.

3. Assessment elements: An evaluation sheet with a matrix showing the correspondence between ethical domains and assessment elements was developed based on the HTA Core Model®. The HTA reports published after 2016 were compatible with the matrix because they used the same assessment elements and ethical domains, and were assessed based on the matrix. However, the two reports preceding the initiation of the HTA Core Model® revealed significant variations in terms of assessment elements and ethical domains, which could becloud the analysis. In this regard, the questions related to the assessment element for these two reports, which did not provide assessment element tables compatible with the HTA Core Model®, were examined carefully. Each question was checked against the standard questions of the HTA Core Model® in terms of their context and implications and was categorized under the appropriate assessment element and ethical domain.

4. Methodology of EA: The HTA Core Model® suggests using one of eight different analysis methodologies to perform EA: casuistry, coherence analysis, interactive participatory HTA approach, principlism, social shaping of technology, wide reflective equilibrium, a triangular model based on the human-person centered approach, or an axiological approach (3). Each HTA report was assessed in terms of which methodology was used. Also, the methodology section of each EA was scanned to extract the data sources used in the application of the chosen methodology. The data sources and methodologies were evaluated for EA based on their compliance with the HTA Core Model®’s perspective.

Results

1. Competency of the person/group who conducted EA: Competency in EA was present in three reports. The four reports which were published in 2013, 2014, and 2015 did not include any ethical experts on their EA team.

2. Focus or aim of EA: The goal of the EA section of the HTA was not specified in any of the HTA reports because it is not a particular requirement of the HTA Core Model®. The reports that included statements about the focus or aim of EA displayed several implications. The report on the use of intravenous immunoglobulins for Alzheimer’s disease including mild cognitive impairment stated that the aim of EA was to define the framework for the EA and provide criteria for the application of this framework (11). According to this report, the HTA Core Model® should be considered “a general framework to guide experts doing HTA at a local level.”

   Another report stated that the EA section was aimed at “providing a balance between norms and values through the consideration of social, political, cultural, legal, religious, and economic aspects arising from the opposition to the generally accepted environmental values, healthcare system goals, and the application of new technologies (11), while a different report declared that the focus of the EA section was “to present ethical arguments related to the autonomy and benefits for the patient as well as possible complications and limitations pertaining to the implementation of the HT discussed, without aiming to give a definite answer or ethical prescription (12). Another aim that was specified in the reports was “to gather experiences from a novel way of preparing HTA work, rather than prepare a valid assessment on the particular HT” (13).

3. Assessment elements: Benefit-harm balance and autonomy are the only two topics that were included in all evaluated HTA reports. The topic of justice and equity was evaluated in six reports, whereas legislation, questions about effectiveness and accuracy, and principle questions about the ethical aspects of technology were addressed in four reports. Respect for persons as a topic was addressed in only two reports. Human dignity and human integrity were used in three reports. It should be considered that the respect for persons topic includes issues that are similar to those within the human dignity and human integrity topics. Hence, the HTA reports lacking the respect for persons topic have covered the same issues under the topics of human dignity and human integrity. The topic “ethical consequences of the HTA” was covered in only three HTA reports. This constitutes the lowest coverage among the topics labeled as core and deemed critically important in the HTA core Model®.

   The reason for the inclusion of these topics and exclusion of others might be the evaluators’ inclination to report only on the domains where they find ethical problems. However, this is an assumption in need of proof. Moreover, the information in the reports lacks any confirmation of systematically going through all domains.

   Regarding specific issues, the section below outlines each ethical issue and provides the percentage of reports that addressed each ethical issue. Issues are coded as F00##.

   • F0006: “Is there a need for any specific interventions or supportive actions concerning information to respect patient autonomy when the technology is used?” was addressed in 100 percent (n = 7) of reports under the autonomy topic.

   • F0005: “Is the technology used for individuals that are especially vulnerable?” and F0007: “Does the implementation or withdrawal of the technology challenge or change professional values, ethics, or traditional roles?” was addressed in 86 percent (n = 6) of reports under the autonomy topic.

   • F0009: “Does the implementation or use of the technology affect the patient’s moral, religious or cultural integrity?” was also addressed in 86 percent (n = 6) of reports either under the respect for persons (n = 3), human integrity (n = 2), or human dignity (n = 1) topic.

   • F0012: “How does implementation or withdrawal of the technology affect the distribution of health care resources?” was addressed in 86 percent (n = 6) of reports under two topics: justice and equity (n = 5), and ethical consequences of HTA (n = 1).

   Some issues were addressed in more than one topic: For example, F0003: “Are there any other hidden or unintended consequences of the technology and its applications for patients, relatives, other patients, organizations, commercial entities, society etc.?” and F0001: “Is the technology a new, innovative mode of care, an add-on to or modification of a standard mode of care, or replacement of a standard mode of care?” were addressed under benefit-harm balance, as well as in the principle questions regarding the ethical aspects of technology topics. F0012: “How does implementation or withdrawal of the technology affect the distribution of health care resources?” was addressed under the ethical consequences of HTA, as well as justice and equity topics. The issues addressed under more than one topic and their frequency is shown in Table 3, with the overall frequency of topics across the reports shown in Table 4.

4. Methodology, data sources, and quality assessment criteria of EA: 43 percent (n = 3) of evaluated HTA reports used principlism for EA. Coherence analysis was used in one HTA report. In three reports, the methodology for EA was not specified; however, data sources were explained in more detail. A literature review was the most commonly used methodology while search strategies were not specified.

   Ironically, in the reports in which EA methodologies were specified, there was very little or no information regarding the data sources. In one report, the data sources were specified as the results of the other domains and literature review. No details about the search strategy for the literature review were provided. The other reports with specified methodologies (principlism) provided very limited information about the literature review or gave no information about the data sources. Among the HTA reports with a specified methodology for EA, only one mentioned quality assessment, stating that “quality assessment is not needed, because the goal at the level of the core HTA is only to define the framework for EA.”

   None of the reports used data derived from expert opinions of ethicists, the judgments of one or more experts drawing on their perceptions of scientific evidence, patient/service user opinions, or the views of organizational stakeholders, while one report extracted core data from Wikipedia (12).

Table 3.

Issues Addressed Under More Than One Topic, with Frequency Counts

F0003: Are there any other hidden or unintended consequences of the technology and its applications for patients/users, relatives, other patients, organizations, commercial entities, society, etc.?	Beneficence/non-maleficence	1
	Principle questions about the ethical aspects of technology	2
F0001: Is the technology a new, innovative mode of care, an add-on to, or modification of a standard mode of care, or a replacement of a standard mode of care?	Beneficence/non-maleficence	1
	Principle questions about the ethical aspects of technology	3
F0008: Does the implementation or use of the technology affect human dignity?	Human dignity	2
	Respect for persons	1
F0009: Does the implementation or use of the technology affect the patient’s moral, religious, or cultural integrity?	Human dignity	1
	Human integrity	2
	Respect for persons	3
F0014: Does the implementation or use of the technology affect the realization of basic human rights?	Justice and equity	1
	Legislation	1
	Rights	2
F0012: How does implementation or withdrawal of the technology affect the distribution of health care resources?	Ethical consequences of the HTA	1
	Justice and equity	5
F0017: What are the ethical consequences of the choice of endpoints, cut-off values, and comparators/controls in the assessment?	Ethical consequences of the HTA	1
	Questions about effectiveness and accuracy	4

HTA, health technology assessment.

Table 4.

Overall Frequency of Topics Involved in HTA Reports

Topic	Number of HTA reports that address the topic (N = 7)
Benefit harm balance (beneficence/non-maleficence)	7
Autonomy	7
Respect for persons	2
Justice and equity	6
Ethical consequences of the HTA	3
Legislation	4
Questions about effectiveness and accuracy	4
Principal questions regarding the ethical aspects of technology	4
Human integrity	3
Rights	2
Human dignity	3

HTA, health technology assessment.

Discussion

The HTA Core Model® states that apart from the EA domain, ethics has a broader application within the HTA process. This broad sense of ethics in HTA has also been emphasized by various groups (16–19). According to this perspective, evaluating the ethical implications of the HT in the EA section of HTA reports is not enough to ensure that the HT and HTA procedure itself is subject to systematic EA. The HTA Core Model® provides a comprehensive list of ethical concerns that ensure the ethical appropriateness of the HTA process in the broad sense. However, because these concerns are not answered in the ethics domain section, they remain as good intentions and do not have any impact on the EA section or the HTA as a whole. Accordingly, the reports that were included in this study lacked this broad sense of ethics and did not address any of the ethical concerns listed above. Instead, ethics were limited to only the EA sections, and to finding the answers to the issues listed in the topics. Hence, it is reasonable to claim that the evaluated HTA reports failed to address the ethical compliance of the HTA procedure.

The inclusion of benefit and harm balance and autonomy topics in all the evaluated reports reveals that the assessors held these two topics as the most important ethical issues in which HTs might encounter risk. This finding is in line with the recent systematic review done by Bellemare et al. in 2018 (7). This study stated that the values that are embedded in principlism were the most commonly mentioned in EA of HTA (7). It is obvious that applying the HTA Core Model® has a positive effect on the standardization of reports and avoids the exclusion of core topics that are considered to be solely part of the ethical domain. The flexibility of the HTA Core Model® helps the assessors adapt the methodology to the requirements of the HT in question. However, the phrasing of issues and their analyses prompt a consequentialist approach, which prioritizes the possible impacts of the HT rather than the ethical implications of the HT itself. Moreover, the analyses of most of the ethical issues generally lacks theoretical framework and instead reveals an eclectic approach, which makes it more difficult for the reader to follow the ethical reasoning and justification of the report.

The type of expertise that is required to perform EA has been debated. Some authors oppose the possibility of ethical expertise based on several arguments (9;19–23). These objections are supported by additional arguments for excluding ethics from HTA by proving the dissonance between ethics and other domains of HTA. Some of the objections to excluding ethics include: (a) the aim, methodology, and models that are related to the rationality of HTA and ethics are categorically different; (b) there is no agreement on the methodology of EA; (c) other domains, such as law, economics, and sociology, cover ethical issues; (d) ethics is not as involved in HTA as is commonly thought (24).

Bellemare et al. contributed to this discussion by highlighting the issues that challenge the integration of ethics into HTA (7). They identify the lack of a shared standard model for EA and the absence of consensus on the role of theory and ethical expertise as the main problems to including ethics in HTA (7). The results of this study are in agreement with the conclusions of Bellemare et al. (7) and Saarni et al. (16), which argue that the significant lack of familiarity with complex philosophical theories and ethical reasoning, as well as the lack of expertise in understanding ethical justification methods, are some of the most important barriers to including comprehensive EA in HTA (7;25;26).

Conversely, an international survey revealed that 68 percent of HTA professionals surveyed thought that EA was important and 60.8 percent thought that at least one of the HTA experts should have formal training in ethics. In cases where no such ethics expert was available, 78.4 percent of respondents thought that a professional ethicist should perform the EA as an external consultant (19). The HTA Core Model® seems to agree with the redundancy of ethics experts in HTA by stating that the role of an ethics expert is to facilitate and report the EA rather than perform it and that the involvement of scientific and clinical experts in the EA is more important (3).

The overall lack of involvement of ethical experts on the HTA teams and the reports that were evaluated in this study agrees with the overall perspective of ethics in the HTA Core Model®. The results of the study showed that ethical expertise is disregarded because only three of the seven evaluated reports included ethics experts on their team for the EA domain. It is also interesting to note that, in two of the three HTA reports, only one person performed the EA and that person lacked ethical expertise. The lack of expertise had some clear indicators. In some reports, the answers to questions were not relevant to the ethical concerns related to the issue, or they were very abstract and too simple to provide a comprehensive perspective.

The eclectic style of the answers suggests that no systematic ethical reasoning was applied. Instead, data from questionable literature reviews were used by EA teams to support their ideas, or the results of other domains were cited. In some reports, several of the very fundamental ethical issues, such as the risks to basic human rights and human integrity, were suggested for assessment in the legal domain. A common attitude of the EA sections in the evaluated HTA reports was to address ethical questions regarding the consequences of implementing or not implementing an HTA, instead of evaluating the ethics of the technology itself.

The focus or aim of EA in an HTA is as follows: (a) to increase the efficiency of the HTA by addressing the moral and normative issues that are crucial to disseminating the HT; (b) to discuss and reveal the morally relevant consequences of the HT by integrating perspectives of various stakeholders, and most importantly, those of patients (6;27); (c) to highlight the challenges to basic moral principles which are not specific to the HT in question, but are made topical by the development of the technology in general. For example, while performing EA for an HT developed for colon cancer screening, the EA should provide the ethical justification for public screening techniques in general (5); (d) to discuss those values constituting the framework of the issues that the HT aims to solve, and the solutions it suggests, with particular attention to socially interfering implications (5).

However, as established in the results section, the purposes that were stated in the HTA reports that were evaluated were different from the focuses of EA in an HTA. The incongruities in the focus or aim of the reports might stem from a lack of knowledge about what EA is and how it is performed. As seen in the reports, the EA teams did not have clarity on the aim of their task. Hence, very divergent targets were set: some endeavored to provide a balance between norms and values through discussion on social, political, cultural, legal, religious, and economic issues arising from the opposition to the generally accepted societal values, healthcare system goals, and the application of new technologies, while others aimed to provide information on only which questions are to be answered and propose how this might be done in the local context.

The latter approach is, in fact, the purpose specified in the HTA Core Model® and the task of the EA team is to perform the EA, not provide additional frameworks. This problem was addressed in a systematic review by Bellemare et al., which stated that the identification and analysis of ethical issues are considered to be part of the HTA process by some evaluators, while others comprehended that EA is an evaluative process that is required for formulating recommendations (7).

Similar problems were seen in the data resources. The EA sections of the evaluated reports gave general ideas about the HT in question and the data for these general ideas were either obtained from other domains of the assessment, or by literature review, which is mostly limited to PubMed. The search strategies that were used were unspecified, which suggests that the data used for justification were not based on a comprehensive and valid systematic review. In the HTA Core Model®, it is suggested that when conducting a literature review for EA, the ethical implications of similar technologies should be considered and the various viewpoints of stakeholders should be acknowledged (3). These features were lacking in most of the evaluated HTA reports. which may confound the data entered for EA. For a strong ethical justification, the premises should be based on valid data. It was very surprising that one of the HTA reports extracted target population data from Wikipedia, which raises serious concerns about the responsible and scientific conduct of the data collection(12).

The methodologies that are suggested by the HTA Core Model® require competency in ethics. Epistemic expertise in ethics is defined as “the capacity to provide strong justifications for claims in an ethics domain” (9). Developing strong ethical justifications require knowledge about systematic ethical reasoning. In ethical discourse, the term “justification” means “to establish one’s case by presenting sufficient grounds,” without making logical mistakes such as asserting reasons that do not support the conclusion, reasoning that is developed by relying on data that are not valid, or faulty rationale for reaching the suggested conclusion.

Moreover, the ethical analyst should have the knowledge and applicable skills that are related to several models of justification. These models are as follows: (a) a top–down perspective, meaning the capacity to operate justification deductively and develop a claim from a set of premises; (b) bottom–up models, which depend on inductively proceeding the justification starting from paradigm cases known as casuistry; (c) integrated models, such as coherence analysis or reflective equilibrium; or (d) common-morality theory of principlism (28).

It is difficult to believe that a team of experts without epistemic and performative expertise in ethical justification methodologies can perform sufficient and meaningful EA for an HT and identify and evaluate the ethical issues inherent to the whole HTA process, all while systematically addressing competing ethical considerations. The results of this study support this statement. When we examined the methodology sections of the evaluated EAs, we saw that either the methodology was unspecified, or the term of the methodology itself was misused and set as synonymous to data resources. The methodology of EA, if specified, is frequently that of principlism. The lack of details related to the implementation of the methodologies raises suspicions regarding the appropriateness of their operations.

Some methodologies suggested in the HTACore Model®, such as the social shaping of technology or a triangular model based on the human-person centered approach, require deeper philosophical knowledge regarding the intellectual dilemmas in the field. The social shaping of technology concentrates on the content of the technology to facilitate a broader ethical perspective by integrating natural and social science concerns. The essence of the social shaping of technology depends on a discussion of the invalidity of technological determinism (29–32). The triangular model based on the human–person centered approach evaluates HT through a cycle of interviews with all relevant stakeholders to reveal their concerns about the HT; with the aim of identifying the issues where the agreement and disagreement are explicit, to ease the decision-making process for the authorities. It is certain that these methodologies require more expertise and resources than analytical methods such as principlism and might bring a broader and more divergent perspective to HTA. However, none of these methodologies are used in the reports included in this study.

The results show that there are deficiencies in the EA of EUnetHTA reports included in this study. Some of these deficiencies might emerge from the insufficiency of the methodology or a failed implementation of the HTA Core Model®. However, the results also suggest that the construction and content of the HTA Core Model® might not be the sole reason for the deficiencies in the EA. The results also support a perspective that would be more significant for an international audience: ethics should have a broad application within the HTA process and any methodology oblivious to this perspective would be insufficient in terms of producing high-quality EA reports. Without considering the essential ethical issues, such as the expertise of the EA team, morally relevant reasons for performing or not performing HTA on the particular topic, the interests of the content expert groups, the morally relevant issues related to the selection of meta-analysis, and the stakeholder viewpoints, and patient perspectives, the overall EA would be lacking.

The main limitation of this study is the number of reports analyzed. This limitation emerges from the fact that only seven full HTA reports have been realized and made available by EUnetHTA. Although all these reports were included in the study, it is reasonable to assume that we would be able to draw more general conclusions by analyzing future reports produced in accordance with the HTA Core Model®.

In conclusion, although the HTA Core Model® was helpful in standardizing the final reports of the HTA, there were issues with the content and outcomes. Additional research is required to determine the pitfalls and to further advance the HTA Core Model®. A lack of expertise in ethics, and insufficiencies in the teams performing EA contribute to the overall insufficiencies of EA in HTA.