Organizational aspect in healthcare decision-making: a literature review

ABSTRACT

Background

Organizational aspect is rarely considered in healthcare. However, it is gradually seen as one of the key aspects of the decision-making process as well as clinical and economic dimensions. Our primary objective was to identify criteria already used to assess the organizational impact of medical innovations. Our secondary objective was to structure them into an inventory to support decision-makers to select the relevant criteria for their complex decision-making issues.

Materials and methods

A search using the Medline database was conducted in June 2019. The records published between January, 1990 and December, 2018 were identified. The publications cited by the authors of the included articles and the websites of health technology assessment agencies, units or learned societies identified during the search were also consulted. The identified criteria were structured in an inventory.

Results

We selected 107 records of a wide range of evidence mostly published after the 2000s. We identified 636 criteria that we classified into five categories: people, task, structure, technology, and surroundings.

Conclusion

Criteria selection is a crucial step in any multi-criteria decision analysis (MCDA). This work is the first step in the development of a validated MCDA method to assess the organizational impact of medical innovations.

Introduction

Nowadays, health technology assessment (HTA) is mainly conducted based on clinical effectiveness and safety studies and medico-economic studies [1]. The description of other aspects such as organizational, ethic or strategic aspect is rarely considered. Nevertheless, complex decision-making requires the consideration of all relevant aspects. The assessment of innovative drugs, medical devices, medical or surgical procedures and organizations like care pathways currently established does not take into account all the aspects, which characterize an organization in health. This pitfall results in a lack of rationality in decision-making as well as a sub-optimal use of resources. According to Leavitt’s model, an organization is made up of four interdependent entities: people, task, structure and technology [2]. The modified Leavitt’s organizational model includes interactions with surroundings. Evaluating the organizational impact of an innovation includes the study of the expected results on one or more of these entities but also of the changes induced on other entities and surroundings. Its evaluation is all the more important for disruptive innovations, those that create or replace than for incremental innovations, those that improve [3]. In addition, healthcare organizations are characterized by a high level of complexity and a very dynamic environment. That is why, the assessment of organizational impact is gradually seen as one of the key aspects of the decision-making process as well as clinical and economic dimensions [4].

In view of this observation, there is a need for effective decision-making tools enabling a systemic approach of decision-making issues. Multi-criteria decision analysis (MCDA) methods aim to facilitate the identification of the best possible solution to a given problem that requires considering a set of aspects or criteria, which are often heterogeneous. These methods seem to meet this need [5]. Developed in the 1970s and widely used in non-medical domains such as farming, energy or marketing [6–8], they are booming in healthcare since the 2000s [9]. Core components of any MCDA method are the alternatives in competition with one another, the criteria by which alternatives are assessed, the level of performance of each alternative for each criterion and the relative weight of each criterion in relation to the other [5]. The selection of relevant criteria is one of the most crucial steps of any MCDA method.

To the best of our knowledge, no specific tool exists to help decision-makers assess the organizational impact of medical innovations. The primary objective of our study was to identify all the criteria already used to evaluate it. The secondary objective of our study was to build a structured inventory of these criteria to support decision-makers select the relevant criteria for assessing medical innovations considering organizational aspect. This inventory was intended to be used regardless of the type of innovation being evaluated (drugs, medical devices, medical or surgical procedures or organizations) and regardless of the point of view adopted (from a care unit perspective to a national perspective).

Materials and methods

Literature review

We conducted a review of the published and gray literature to identify as many criteria as possible that have already been used to assess medical innovations regarding organizational aspect. All publications dealing with assessment of the organizational impact of health products or organizations, regardless of the decision-making perimeter (health care unit, institution, area, country) were included. An HTA or MCDA methodology had to be implemented and the criteria for assessing the organizational aspect had to be detailed. In order to identify as many criteria in as many different contexts as possible, no exclusion criteria regarding the type of publication were applied. Publications were excluded when their full text was not available. Due to limited language skills, only publications written in English or French were included.

First, a literature search using the Medline database (Pubmed, USA National Library of Medicine, the USA) was conducted in June 2019. Records published between January, 1990 and December, 2018 were retrieved. Indeed, HTA emerged in the 1990s. The following keywords were used to develop two search equations: “decision support tool “, “ decision support model “, “ decision support technique “, “ decision-making “ (Medical Subject Headings (MeSH) term and key word), “ health technology assessment “, “ HTA “, “ criteria “ and “ systematic review “ (Table 1). Second, the publications cited by the authors of the included articles were reviewed. Third, the websites of international, national or regional HTA agencies or units and HTA learned societies identified during the research were consulted.

Table 1.

Search equations used for the literature review on the Medline database.

Search equations
‘Criteria’	((decision support tool[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (criteria[Title/Abstract])) OR ((decision support model[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (criteria[Title/Abstract])) OR ((decision making[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (criteria[Title/Abstract])) OR ((decision making[MeSH Terms]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (criteria[Title/Abstract])) OR ((decision support techniques[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (criteria[Title/Abstract]))
‘Systematic review’	((decision support tool[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (systematic review[Title/Abstract])) OR ((decision support model[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (systematic review[Title/Abstract])) OR ((decision making[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (systematic review[Title/Abstract])) OR ((decision making[MeSH Terms]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (systematic review[Title/Abstract])) OR ((decision support techniques[Title/Abstract]) AND ((HTA[Title/Abstract]) OR (health technology assessment[Title/Abstract])) AND (systematic review[Title/Abstract]))

For each selected document, the following elements were collected: date of publication, name of the first author or institution, origin (country or continent), type of document (article, guide book, thesis, poster, website), content of the document (HTA report or model, MCDA tool, study, round table report), level (local, regional, national, international), criteria and their definition when available.

Structuring the criteria inventory

First, a standardized term or expression was associated with each criterion to facilitate their grouping by theme. We counted the number of similar criteria that were grouped within the same theme. Second, criteria were structured into five categories: people, task, structure and technology and surroundings. These are the five categories of the modified Leavitt’s organization model. We chose this model because it is widely known and for its simplicity. We used the definitions of each category proposed by the Danish Center for Evaluation and Health Technology Assessment (DACEHTA) in its handbook to classify the criteria [2].

Results

After removal of duplicates and exclusion of articles whose full text was not available or not written in English or French, consulting the Medline database allowed us to identify 245 potential publications. After review of the abstract and/or full-text, 22 of these publications, that met inclusion criteria, were selected. References cited by the authors of these 22 publications as well as the websites of HTA agencies or units and HTA learned societies identified during the research were consulted to complete the data collection. Through this research strategy, 107 records were included (Figure 1). The origin, the type of publication, the content of the publication, and the level if the publication presented an HTA model or MCDA tool were collected (Table 2).

Figure 1.

Research strategy flowchart.

Table 2.

Characteristics of the selected records.

Selected records	n = 107	References
Origin
Europe	61	[2,10–12,14,16–18,23,24,26–76]
North America	31	[19,77–106]
Oceania	5	[107–111]
Asia	4	[112–115]
South America	3	[116–118]
Africa	2	[119,120]
International	1	[121]
Publication type
Article	71	(14, 16-19, 23, 26, 27, 29-32, 34, 35, 39-46, 48, 50, 51, 53, 56, 58, 64-69, 71-79, 81, 82, 84-86, 88, 90, 92, 96, 98, 101-106, 108, 111-121)
Guidebook	11	(2, 10-12, 33, 36, 57, 59, 80, 87, 95)
Website	9	(49, 54, 55, 60, 62, 83, 91, 107, 109)
HTA Report	7	(28, 37, 38, 52, 89, 97, 100)
Communication	5	(47, 93, 94, 99, 110)
Thesis	3	(24, 61, 63)
Poster	1	(70)
Content of publications
Survey	29	(24, 27, 29, 32, 34, 39-41, 44-46, 50, 63, 70, 72, 78, 81, 82, 92, 96, 98, 102, 105, 111-113, 117, 119, 120)
HTA model	28	(2, 10-12, 28, 33, 36, 38, 51, 54-57, 59, 60, 77, 80, 83, 87, 91, 97, 99, 100, 103, 106, 107, 109, 114)
MCDA tool	26	(16-18, 26, 31, 35, 42, 43, 47-49, 53, 61, 67, 69, 71, 84, 88, 90, 93-95, 108, 110, 116, 121)
Literature review	15	(14, 19, 23, 30, 52, 58, 64-66, 73, 75, 85, 86, 115, 118)
HTA report	6	(37, 62, 74, 76, 89, 104)
Round table report	3	(68, 79, 101)
Level of HTA models and MCDA tools
HTA models	28
Regional	13	(28, 33, 38, 54, 56, 77, 80, 91, 97, 99, 103, 107, 109)
National	10	(2, 12, 36, 55, 57, 59, 60, 87, 100, 106)
Local	3	(51, 83, 114)
International	2	(10, 11)
MCDA tools	26
Local	12	(17, 18, 31, 35, 42, 43, 47, 48, 61, 67, 88, 108)
Regional	6	(16, 26, 84, 90, 93, 94)
International	5	(49, 69, 95, 110, 121)
National	3	(53, 71, 116)

The great majority of selected publications (n = 103) were published after the 2000s. This is consistent with the growth of HTA and MCDA methods in healthcare, previously reported.

More than half of the selected publications were European (n = 61). For example, HTA core model® from the European Network for Health Technology Assessment (EunetHTA) project and Hospital-based HTA core model® from the Adopting Hospital based Health Technology Assessment (AdHopHTA) project, as well as the health technology assessment handbook and the introduction to mini-HTA published by DACEHTA were selected [2,10–12]. A quarter of the selected documents were published in North America (n = 31), the vast majority of which were published in Canada (n = 28). HTA units, like the one established by the McGill University Health Centre (MUHC) are widely developed in this country [13]. Five selected publications came from Oceania, including four publications from Australia. The remaining documents were from Asia (n = 4), South America (n = 3) and Africa (n = 2). The poor number of publications from these regions is probably due to the recent growth of HTA in developing countries [14]. Finally, a publication was written by an international organization.

Among the 107 records selected there were 71 articles, 11 guidebooks, 9 websites of HTA agencies, 7 HTA agency reports, 5 communication papers, 3 theses and one poster. Half of these publications introduced an HTA model or an MCDA tool. Among the HTA models (n = 28), two were international, ten were national, thirteen were regional and three were local. Regarding MCDA tools (n = 26), five were international, three were national, six were regional and twelve were local. For example, the Evidence and Values: Impact of Decision Making (EVIDEM) and the Valutazione delle tecnologie sanitarie frameworks as well as Matrix4Value® and Innovative Device Assessment (IDA) tools [15–18] were selected. Several of the selected articles (n = 44) were literature reviews or surveys (interviews, Delphi method) regarding the use of HTA or MCDA methods in healthcare. For example, a study carried out in 2012 by Guindo and al. which focused on healthcare decision criteria was included [19]. Moreover, HTA reports of national or regional agencies were identified (n = 6). This was the case of reports published by the Committee for Evaluation and Dissemination of Innovative Technologies (CEDIT), established by the Greater Paris University Hospitals (Assistance Publique des Hôpitaux de Paris, AP-HP) [20].

A total of 636 criteria were identified, with an average of 5.9 criteria per record. A standardized term or expression was associated with each criterion to facilitate their grouping by theme and to classify them more easily into the five categories previously defined (‘people’, ‘task’, ‘structure’, ‘technology’ and ‘surroundings’) (Figure 2). This process resulted in the creation of a structured inventory of all the criteria collected (Table 3).

Figure 2.

Flowchart of the criteria structuring process.

Table 3.

Inventory of criteria for assessing the organizational impact of medical innovations. List of references used in Table 3: (2, 10-12, 14, 16-19, 23, 24, 26-121).

DOMAINS (occurrences)
Criteria (occurrences)	Example(s)
Sub-criteria (occurrences)
SURROUNDINGS (n=169)
Legislation (n=31)	Approval need; Innovation compatible with current legislation; Procedural complexity; Legislative and regulatory requirements; Authorization and safety requirements; Ownership and liability; Responsibility; Professional insurance need; Changes in risk of damage suits.
Cooperation (n=79)	Changes in the cooperation with other actors/sectors; New partnership development; Well suited to joint ventures; Changes in the way medical staff work together (knowledge sharing); Impact on partnership and collaboration; Impact on innovation research; Capacity to stimulate research.
(De)centralization (n=25)	Changes in accessibility; Changes in market regulation; Changes in equity of health care; Ability to reach whole target region/population; Number of operational site within a given territory; Distribution within a given territory (number of sites per area, number of patients per area); Average access time/distance.
Coordination (n=21)	Consequences on organization involved; Consequences on partnership activity; Consequences on whole health care system; Opportunity to harmonize the practices.
Communication (n=16)	Changes in mode of communication; Changes in frequency of communication; Changes to the content; Changes to the communication medium.
Information (n=16)	Changes in mode of information; Changes to the content; Changes to the information medium.
Vigilance (n=8)	Changes in mode of bottom-up/top-down reports; Changes in vigilance process; Changes in surveillance requirements.
Environment (n=7)	Environment impact; Environment impact of production; Environment impact of use; Environment impact of implementation.
TASK (n=148)
Workflow (n=118)
Performance (n=24)	Number of procedures performed; Number of patients treated; Number of beds dedicated; Hospital bed occupancy; Possible impact on operating room productivity.
Intervention characteristics (n=15)	Operating time; Number of different steps.
Hospital stay (n=8)	Average length of stay; hospital stay; Number of rehospitalization; Number of days in intensive care; Number of days in resuscitation unit.
Associated activities (n=6)	Number of associated physical examination; Number of medical appointment; Number of lifestyle and dietary measures; Patient autonomy and privacy.
Time period management (n=6)	Time a patient has to attend; Number of patients on the waitlist.
Activity profile (n=5)	Part of activity dedicated to innovation.
Implementation (n=79)
Planning (n=18)	Delay of implementation; Planning period (duration); Number of planning meetings; Transition period (duration).
Method (n=13)	Number of implementation steps; Flexibility of implementation; Possibility to implement a trial test; Possibility to reproduce a clinical trial method.
Success (n=4)	Use of performance indicators is possible.
Process of care (n=19)
Inappropriate use (n=8)	Risk of inappropriate use.
Quality of care (n=6)	Change in quality of care; Change in continuity of care; Changes in out of hours medical services.
Patient recruitment (n=5)	Change in personal empowered to recruit; Change in decision-making tree; Change in process to remember patient to attend intervention; Changes in process to ensure patient attend intervention.
Quality control (n=16)
Control (n=9)	Changes in frequency of monitoring; new audit needed.
Indicators (n=4)	Number of indicators; Change in type of indicators; Changes in complexity of data collection.
Risks management (n=3)	Changes in risk mapping (type of risk, frequency, severity, control); Changes in procedures of reporting events (sentinel/adverse); Changes in number of reporting events (sentinel/adverse).
Pathways (n=14)
Area (n=9)	Change in area/sector involvement; Task shifting between health professionals; Task shifting on patient/caregiver; Change in social/professional roles and identity.
Chronology (n=5)	Change in pathway steps chronology; Number of modified/added/deleted pathway steps.
PEOPLE (n=122)
Training (n=33)	Learning curve; Continuing professional development; Duration of training; Training needed; Number of training courses; Number of trainees; Support staff training.
Human resources (n=32)	Changes in staff requirements (working hours); Additional staff requirement; Dedicated staff needed; Changes in medical staff/patient ratio; Reduction in staff/redeployment of staff on other activities; Availability of workforce
Knowledge/skills (n=17)	Availability of expertise; Skills requirements; Number of practitioners authorized.
Working environment/conditions (n=15)	Impact on work environment; Comfort of caregivers/medical staff.
Health and safety at work (n=3)	Protective measures needed; Changes in level of risk (accidental blood exposure, chemical risk, infectious risk, radiation exposure, carrying of heavy loads, repetitive gestures, risk of musculoskeletal conditions); Changes in occupational medicine monitoring (examinations, frequency).
TECHNOLOGY (n=100)
Resources (n=85)
Material resources (n=119)	Compatible with existing equipment; Compatible with existing software; Purchase of equipment or consumables needed; Enough available resources; Operational requirements; Additional/new equipment requirements; Resources availability.
Financial resources (n=21)	Changes in part of budget allocation; Changes in payment arrangement; Financial resources requirements; Wage modifications.
Infrastructure (n=25)
Premises (n=13)	Investment in additional areas requirements; Dedicated area requirements; Proximity to other area requirements.
Work (n=12)	Work requirements; Work period.
Innovation characteristics (n=23)
Complexity (n=12)	Enlightened patient decision; Science popularization.
Use (n=6)	Changes in maintenance; Life expectancy.
Managing (n=5)	Changes in purchase process; Changes in supply process; Changes in supplier; Product manageability; Changes in conditions of storage.
STRUCTURE (n=97)
Acceptability (n=29)	Patients/relatives/patient groups acceptability; General population acceptability; Caregiver acceptability; Stakeholders acceptability; Controversial nature; Innovation requested by patient groups; Conflict of interest; Barriers to uptake; Stakeholders pressures; Social influences (support from, trust/respect, past experience).
Culture, values, missions (n=17)	Alignment with mandate/scope/mission/culture/values of health care system; cultural acceptability; Congruence; Precedence; Institutional limitations.
Strategy (n=16)	Priority status; Political priorities; Changes in attractiveness; Changes in market shares; Alignment with objectives of health care system.
Management (n=15)	Changes in hierarchy; Changes in professional liability; Changes in health care structure management; Changes in administration of healthcare system; management problems and opportunities.
Procedures (n=13)	Changes in document system; Changes in quality assurance manual; Organizational changes requirements; Complexity of change management.

This work aimed to help healthcare professionals select relevant criteria to assess the organizational impact of a medical innovation, whether it is a drug, a medical device, a medical or surgical procedure or an organization such as a care pathway. Decision-makers should keep these four requirements in mind when selecting criteria [15,21]:

• Completeness: all relevant criteria are selected.

• Absence of redundancy.

• Mutual independence: the level of performance of each criterion is independent of the level of performance of the other criteria.

• Operationality: the data needed to assess performance are available.

Considering the number of criteria per category, we recommended that eight criteria be selected as follows: two criteria of the ‘Surroundings’ category, two criteria of the ‘Tasks’ category, two criteria of the ‘People’ category, one criterion of the ‘Technology’ category and one criterion of the ‘Structure’ category.

The ‘Surroundings’ category included aspects related to legislation such as the applicability of existing legislation or the risk of conflicts as well as aspects related to cooperation with other organizations. It also described aspects linked to de-centralization (the distribution of the supply of care on the territory), coordination between care providers, communication, information, vigilance system and impact on the environment. Depending on the point of view adopted, interactions were studied between different services, institutions or territories. In addition, interactions between all stakeholders including patients and health authorities could be appreciated. One of the selected models considered criteria related to the border context.

The ‘Task’ category referred to aspects related to workflow, implementation of innovation, the care process, quality assurance and health pathways. The most frequently identified criterion in the selected documents was the workflow. Six sub-criteria were identified, i.e. the characteristics of the intervention, hospital stay, associated activities, time management, activity profile and, most importantly, performance. The second criterion in terms of occurrence was the implementation of innovation. It was subdivided into three sub-criteria relating to planning, method of deployment and evaluation of its success. The process of care criterion referred to risk of misuse, quality of care and patient recruitment. The quality assurance criterion included three sub-criteria relating to controls, indicators and risk management. Finally, the criterion pathways assessed the sectors/actors involved and the chronology of care.

The category ‘Personnel’ referred to aspects related to staff training, human resources and in particular staff resources and skills management. It also dealt with the working environment as well as health and safety at work.

The ‘Technology’ category referred to the resources, infrastructure and characteristics of innovation. The ‘resources’ criterion was subdivided into two sub-criteria concerning hardware and software including their compatibility with the innovation and budget. Financial resources were discussed in a qualitative manner, in contrast to the assessment of the economic aspect.

The ‘Structure’ category referred to both formal and informal structure of an organization. The most identified criterion in the selected documents was acceptability of innovation by health professionals and/or patients. One criterion assessed if the innovation fit with the organization and/or individuals culture, missions and values. Another concerned the congruence with the organization strategic plan. The last two criteria related to management and procedures.

Discussion

The construction of the search equation was complex due to the difficulty in finding the appropriate keywords and MeSH terms. Indeed, the keywords relating to the organizational aspect did not make it possible to identify a satisfactory number of publications and to retrieve the articles identified during the preliminary searches. One of the main reasons could be the low number of publications dealing with the assessment of the organizational aspect during health care decision-making. This is why we have used more general keywords such as ‘criteria’. The completeness of the data collection can be discussed. This data collection was limited to publications that were referenced on a single database. In addition, only publications written in English or French were selected and only one reviewer made the screening. We chose to select documents without any restrictions on the level of evidence to make an inventory as complete as possible of the criteria that have already been used to assess medical innovations regarding organizational aspect. We considered that the number of documents was sufficient and that the criteria identified were representative. The distribution of the criteria into the five categories of the modified Leavitt’s model was facilitated by the use of standardized terms. The definitions of the categories in the HTA handbook published by DACEHTA also assisted us in the allocation. During this step, interpretation and/or translation errors may have led to inaccuracy in wording.

In parallel with this work, an MCDA method was developed. To this end, a group of seven experts has been brought together for a day of experimentation. It is recommended that a multi-disciplinary group carried out the selection of criteria. If a patient could not be part of this group, one of the experts was a citizen. Indeed, the analysis of patient preferences benefits from an increasing interest in healthcare decision-making [22]. On this occasion, a complex decision-making issue was addressed and the inventory of criteria was used for the selection of relevant criteria. During the selection process, the experts identified criteria whose meaning was unclear. As a result, the wording of these criteria has been changed. Some criteria were flagged as implicitly positive or negative, and their wording was also changed. Regarding the number of criteria selected, a literature review of MCDA methods found a mean number of 8.2 criteria used to evaluate interventions (range 3 to 19) [23]. However, there does not seem to be a consensus on the optimal number of criteria to be selected. The number of criteria is directly related to the complexity of the innovation being assessed. However, care must be taken, on the one hand, not to omit any relevant criterion and, on the other hand, to comply with the four requirements mentioned above. When selecting criteria, if redundancy is identified, the creation of a composite criterion is a common solution used in MCDA methods to solve this problem [16,24]. This solution is more appropriate than adding or removing criteria [24]. In addition, it is recommended that each member of the group first performs an individual screening. Differences of opinion reflect the diversity of individual perceptions of participants. These differences are not a limitation, they help to identify the criteria that need to be discussed, and they encourage exchanges with the aim of reaching consensus. A transparent display of the method of selection and the criteria selected is a ‘reasonableness’ approach to decision-making [24,25]. Training of professionals in the MCDA method is a prerequisite.

Conclusion

Taking into account the organizational aspect is a major challenge for the evaluation of medical innovations, especially for disruptive innovations. A review of the published and gray literature was conducted to collect and classify in a structured inventory all the criteria that have already been considered to assess the organizational impact of medical innovations, whether they concern drugs, medical devices, medical or surgical procedures or organizations. The selection of relevant criteria is one of the crucial steps in any MCDA method. The inventory helps decision-makers to select the relevant criteria for their decision-making issue. This review was carried out in parallel with the development of an MCDA method. This work is the first step in the development of a criteria selection tool integrated into a validated MCDA method for assessing the organizational aspect in healthcare decision-making.

Funding Statement

This research received no specific grant from any funding agency, commercial or not-for-profit sectors.

Disclosure statement

No potential conflict of interest was reported by the authors.