ELICIT: An alternative imprecise weight elicitation technique for use in multi-criteria decision analysis for healthcare

Abstract

Objective

In this paper, the readers are introduced to ELICIT, an imprecise weight elicitation technique for multicriteria decision analysis for healthcare.

Methods

The application of ELICIT consists of two steps: the rank ordering of evaluation criteria based on decision-makers’ (DMs) preferences using the principal component analysis; and the estimation of criteria weights and their descriptive statistics using the variable interdependent analysis and the Monte Carlo method. The application of ELICIT is illustrated with a hypothetical case study involving the elicitation of weights for five criteria used to select the best device for eye surgery.

Results

The criteria were ranked from 1–5, based on a strict preference relationship established by the DMs. For each criterion, the deterministic weight was estimated as well as the standard deviation and 95% credibility interval.

Conclusions

ELICIT is appropriate in situations where only ordinal DMs’ preferences are available to elicit decision criteria weights.

Healthcare decision-making under uncertainty and limited resources occurs ubiquitously across healthcare systems worldwide. The multifaceted nature and complexity of healthcare decision-making has led to the development and adoption of decision-making support tools. In recent years, multi-criteria decision analysis (MCDA), defined as both an approach and a set of methods that permit the simultaneous consideration and prioritization of different factors that may conflict during the decision-making process [1–3], has increasingly been used to support healthcare decision-making [3,4].

MCDA was initially applied in the operations research field and has been successfully deployed in engineering [5–7], management [8–10] and environmental sciences [11–13].

In their implementation, most MCDA methods require the elicitation of preferences from decision-makers (DMs) in regard to the decision-making criteria. According to Riabacke et al. (2012) [14], elicitation techniques use the notion of compensation, meaning that preference statements from DMs represent how many units of unfavorable features on one criterion they are willing to give up in order to gain favorable features on another one. These techniques can generally be categorized in: ratio weight procedures and imprecise weight elicitation techniques. Ratio weight techniques are purported to preserve scale properties of the DMs’ preference statements as part of the elicitation procedure. Methods adopting this approach include rating procedures such as point allocation and direct rating methods [14]. These techniques require the DMs to assign precise/accurate numerical values to the decision-making criteria, which may pose a number of implementation issues in practice. In fact, the action of judging and expressing precise values for criteria may be challenging and subject to response error [15], not to mention the cognitive burden associated with this type of procedure [14]. To overcome these issues, imprecise weight elicitation techniques have been developed and used as part of MCDA.

In this article, a quick and easy-to-implement alternative weight elicitation technique called ELICIT, in the context of imprecision and uncertainty, is proposed. The approach builds upon the principal component analysis (PCA) [16], the variable interdependent parameters (VIP) [17] and the Monte Carlo method [18]. A case study is used to illustrate the steps described in the proposed method.

Methods

The application of ELICIT consists of two steps: the rank ordering of evaluation criteria based on DMs’ preferences using the PCA [16]; and the estimation of the criteria weights and their respective standard deviations and 95% credibility intervals using the VIP analysis [17] and the Monte Carlo method (also known as Monte Carlo simulation) [18]. The latter was used in a hypothetical case study involving the elicitation of DMs’ preferences for five criteria, cost, sensor size, zoom, weight and optical image stabilizer, used to select the best device for eye surgery.

Overview of the technique

The implementation of ELICIT requires formulating a number of assumptions (H_i), which deserve highlighting. Let C_i be the criteria guiding the choices of a group of DMs, W_i the weights associated with the criteria C_i, W_i positive or null with i = (1,…, n).

• H₁: The criteria are all comparable, reflexive and transitive.

• H₂: The DMs are all capable of making a choice. Choice is defined as a strict preference of one criterion over another or indifference between criteria.

• H₃: The preference independence of criteria (C_i) should prevail, that is, that the weight given to any criterion should be independent of the performance on other criteria.

• H₄: The utility describing the preference of the DMs over a good is decomposable according to its characteristics or criteria (C_i).

• H₅: Per definition, utility is bounded between 0 and 1.

The following sections describe the previously mentioned steps.

Rank ordering of decision criteria according to DMs’ preferences

Each DM is asked to rank the criteria (C_i) in the order of preference. Then, the individual ranks assigned to the criteria are aggregated into group ranking. The aggregation process is accomplished through the application of descriptive multidimensional statistical methods. More specifically, the PCA is used, in this case, since only quantitative variables (positions) are used to order the criteria. The PCA offers the advantage of exploring the correlation structure of data. Additionally, this method is more robust when applied to ranks than when applied to values representing the variables (decision criteria) investigated.

Technically, the PCA is based on the spectral decomposition of a correlation matrix (CM) [16]. It consists of finding the best linear combination of variables (i.e., the best weighting set) that explains the largest part of the data dispersion (variance). At the same time, the ordinal consistency of the linear combination of variables should be maintained. The implementation of the PCA consists of three steps: centering and reducing DMs’ rankings for the computation of the CM; finding the largest eigenvalue and its associated eigenvectors with the power iteration method [19]; and computing the scores using the eigenvector coordinates (considered as weights) with respect to each DMs’ ranking. Hence, more preferred positions (ranks) would score higher on the scale while less preferred positions would score lower.

In case, the DMs at the group level cannot establish a strict preference relationship among the criteria, these criteria would be assigned the same weights by default.

Estimation of the criteria weights (W_i) & their respective standard deviations & 95% credibility intervals

As mentioned in the first paragraph of the methods section, the estimation of criteria weights builds upon the principles of the VIP [17] analysis and the Monte Carlo method [18].

In the VIP analysis, the weights associated with the decision criteria are treated as interdependent variables capable of taking on multiple combinations of values subject to defined constraints. For illustration purposes, let us consider a decision-making process based on four criteria, with the following rank ordering of criteria: C₁ ≥ C₂ ≥ C₃ ≥ C₄.

The generalization of the above inequalities is given as follows:

$$C_1\ge C_2\ge \cdots \ge C_{n-1}\ge {\mathrm{C}}_n\text{with}{C}_n\text{being}\text{the}\text{n-th}\text{criterion}.$$ (1)

According to the principles of the VIP analysis, equation (1) implies that if a DM prefers the criterion C₁ over C₂, C₂ over C₃ and C₃ over C₄, then the contribution of each criterion (W_i) in the decision-making process will comply with the order of preference established by (1). Thus, the W_i will be arranged as follows:

$$W_1\ge W_2\ge W_3\ge W_4$$

The generalization of the previous inequalities is given as follows:

$$W_1\ge W_2\ge \cdots \ge W_{n-1}\ge W_n$$ (2)

Following the hypothesis H₅, all the W_i must add up to 1 in order to maximize the utility associated with the corresponding choice. Based on the previous example, we have:

$$W_1+W_2+W_3+W_4=1$$

The generalization of the above is given as follows:

$${\displaystyle {\sum }_{i=1}^n{W}_i=1,\text{with}i}=\{1,2,\dots ,n\}$$ (3)

The inequality (2) and the formula (3) are used as constraints as part of the assignment of weights (W_i) to decision criteria. The latter is done through the use of the Monte Carlo simulation.

The Monte Carlo simulation is a computerized mathematical technique usually used to conduct uncertainty analysis. It consists of substituting point estimates of parameters with inherent uncertainty in a model by random values sampled without replacement from predefined probability functions. This process is repeated a number of times (iterations), with the outcome of the substitution being recorded. As part of the weight elicitation technique, the Monte Carlo simulation is used to generate 1000 iterations of each W_i. The probability distribution used for the simulation of the W_i is a function uniformly distributed on the interval [0,1]. The use of this simulation function is done under the constraints (2) and (3) previously mentioned. The simulation of the W_i follows the law of large numbers. Once the different samples of W_i are generated, the estimation of the W_i for each criterion and their respective descriptive statistics can be done. Each sample of values for W_i is averaged and used as a point estimate for W_i. The standard deviation and the 95 % credibility interval are then calculated. The 95% credibility intervals obtained may be used in deterministic sensitivity analyses on W_i. The means and standard deviations may be used in stochastic or probabilistic sensitivity analyses (PSA) on W_i.

The next sections provide a step-by-step illustration of the use of ELICIT through a hypothetical case study.

Application of ELICIT to a hypothetical case study

A teaching hospital is interested in acquiring a new medical device (surgical microscope) to improve the success rate of eye surgery. An evaluation team, composed of five DMs (a surgeon, a health economist, a hospital administrator, a nurse and a patient advocate) is mandated to make a recommendation about the best option from among a large set of new-generation medical devices for the disease. The team was able to narrow down this set to four medical devices A, B, C and D. The evaluation criteria in this decision-making process are cost, sensor size, zoom, weight and optical image stabilizer. The rationale underpinning the use of these criteria is provided below.

The criterion ‘cost’ is selected as an evaluation criterion since the DMs have a limited budget. Therefore, they have to take this criterion into account to ensure they can afford the desired medical device. In other words, they want to obtain the best value for money, which is consistent with the economic theory stipulating that consumers want to maximize their profit while minimizing their costs. This criterion is measured in hundreds of dollars, and this measurement scale is expected to be minimized. The criterion ‘sensor size’ is chosen as a criterion as it allows gaining more information to produce accurate images (resolution), which in surgical procedures is considered a ‘must’. It is generally measured in megapixels, with ‘the bigger size, the better’. Thus, the measurement scale of this criterion is to maximize. The criterion ‘zoom’ is another important criterion identified through the discussion with the DMs. In fact, this feature permits magnifying the pixels of the image. This criterion is measured on a quantitative scale and is to be maximized. The criterion ‘weight’ of the surgical device appears to be important to the DMs provided that it allows surgeons to work conveniently, particularly when moving. This criterion is measured on a quantitative scale (kilograms) and is expected to be minimized. The optical image stabilizer is a relevant criterion in the decision-making process regarding the purchase of a surgical microscope. This owes to the fact this feature helps in compensating for ‘bad moves’ of the surgeon when taking pictures of the eyes (e.g., retina, back of the eye). This criterion is evaluated on a dichotomous scale ‘yes’ or ‘no’. This presence of the feature is preferred.

Since the team members want to make an informed recommendation about the best medical device for eye surgery, they request the assistance of an expert in clinical decision-making. To help the DMs in their recommendation process, the expert has to elicit the preferences of the DMs in regard to the evaluation criteria before he can help address the selection problem. ELICIT is used to attach weights to the decision criteria presented earlier. The application of the method was facilitated by the use of a package developed by the first and last authors. The package was used to automate the steps 1 and 2 that form the pillars of the weight elicitation technique. That being said, the mathematical operations (PCA, VIP and Monte Carlo simulations) can independently be conducted in well-known available packages including Excel, Matlab or SAS.

The evaluation criteria cost, sensor size, zoom, weight and optical image stabilizer are labeled C_co, C_se, C_zo, C_we and C_op, respectively. W_co, W_se, W_zo, W_we and W_op are the respective weights of the evaluation criteria. The first step in estimating the criteria weights consists in rank ordering the criteria according to DMs’ preferences. For the purpose of this case study, imagine that the DMs have indicated the following order of preferences (ranking) regarding the criteria. These preferences are summarized in TABLE 1.

Table 1.

Rank ordering of decision-making criteria by the decision-makers

	Cco	Cse	Czo	Cwe	Cop
Dmsurg	1	2	4	3	5
DMheco	1	2	3	4	4
DMhadm	2	1	3	4	5
DMnurs	1	2	3	3	5
DMpadv	1	3	2	4	5

Dm_surg: Decision-maker surgeon; DM_heco: Decision-maker health economist; DM_hadm: Decision-Maker hospital administrator; DM_nurs: Decision-maker nurse; DM_padv: Decision-maker patient advocate; C_co: Criterion cost; C_se: Criterion sensor size; C_zo: Criterion zoom; C_we: Criterion weight C_op: Criterion optical image stabilizer.

TABLE 1 presents the rank ordering of preferences for decision-making criteria. This matrix is composed of columns and lines, the columns represent the decision criteria while the lines represent the different DMs. The numbers ranging from 1–5 represent the ranks assigned to each criterion by each DM, with the lowest rank the better. The individual rankings of the DMs are aggregated based on the PCA, with the highest score associated with the best rank. As mentioned in the methods section of the paper, the first step of the implementation of the PCA consists of centering and reducing DMs’ rankings for the computation of the CM. To do so, we first compute the means and variances for the ranks given by DMs (TABLE 2). The goal is to standardize the provided ranks following the formula below:

$${\mathit{\text{Rank}}}_{\mathit{\text{Standardized}}}=\frac{{\mathit{\text{Rank}}}_{\mathit{\text{provided}}}-\mathit{\text{Mean}}}{\sqrt{\mathit{\text{Variance}}}}$$

Table 2.

Means and variances of the ranks provided by the decision-makers.

	Cco	Cse	Czo	Cwe	Cop	Mean	Variance
Dmsurg	1	2	4	3	5	3	2
DMheco	1	2	3	4	4	2.8	1.36
DMhadm	2	1	3	4	5	3	2
DMnurs	1	2	3	3	5	2.8	1.76
DMpadv	1	3	2	4	5	3	2

Dm_surg: Decision-maker surgeon; DM_heco : Decision-maker health economist; DM_hadm: Decision-maker hospital administrator; DM_nurs: Decision-maker nurse; DM_padv: Decision-maker patient advocate; C_co: Criterion cost; C_se: Criterion sensor size; C_zo: Criterion zoom; C_we: Criterion weight; C_op: Criterion optical image stabilizer.

Let X denotes the standardized ranks matrix. Once the standardization is done, we compute the CM following the formula below:

$$CM=\frac{1}{5}X{X}^T$$

where X^T denotes the transpose of matrix X and 5 is the number of DMs.

The CM (for our case study) is presented in TABLE 3.

Table 3.

Correlation matrix.

1	0.8489	0.8	0.9594	0.7
0.8489	1	0.8489	0.879	0.8489
0.8	0.8489	1	0.8528	0.7
0.9594	0.879	0.8528	1	0.8528
0.7	0.8489	0.7	0.8528	1

PCA consists of spectral decomposition of the CM. Many approaches are available for numerical computation of the spectral decomposition. Nonetheless, since we are interested in finding of the largest eigenvalue (λ) and its associated eigenvector (μ), we use the power iteration method [19] (second step of the implementation of the PCA). Then, the computed eigenvector is normalized as follows:

$${\mu }_{\mathit{\text{normalized}}}=\frac{\mu }{\Vert \mu \Vert }$$

where ∥μ∥ denotes the norm of μ.

In this case study, the computed largest eigenvalue is λ = 4.3221 while the normalized eigenvector is:

$${\mu }_{\mathit{\text{normalized}}}=\left(\begin{array}{c}-0.447\\ -0.4582\\ -0.4349\\ -0.4707\\ -0.4237\end{array}\right)$$

Finally, PCA scores for the criteria are computed following the formula below:

$$\mathit{\text{Score}}=X^T{\mu }_{\mathit{\text{normalized}}}$$

For our case study, scores for the criteria are respectively:

$${\mu }_{\mathit{\text{normalized}}}=\left(\begin{array}{c}2.8848\\ 1.5292\\ -0.166\\ -1.1495\\ -3.0985\end{array}\right)$$

TABLE 4 summarizes the input and output of the implementation of the PCA.

Table 4.

Results of the principal component analysis.

	Cco	Cse	Czo	Cwe	Cop
DMsurg	1	2	4	3	5
DMheco	1	2	3	4	4
DMhadm	2	1	3	4	5
DMnurs	1	2	3	3	5
DMpadv	1	3	2	4	5
PCA scoring	2.89	1.53	−0.17	−1.15	−3.10

DM_surg: Decision-maker surgeon; DM_heco: Decision-maker health economist; DM_hadm: Decision-maker hospital administrator; DM_nurs: Decision-maker nurse; DM_padv: Decision-maker patient advocate; C_co: Criterion cost; C_se: Criterion sensor size; C_zo: Criterion zoom; C_we: Criterion weight; C_op: Criterion optical image stabilizer.

Based on the results of the PCA, the rank ordering of the criteria is given as follows: C_co > C_se > C_zo > C_we > C_op. This strict inequality implies that if the DMs prefer the criterion C_co over C_se, C_se over C_zo, C_zo over C_we, and C_we over C_op, then the contribution of each criterion (W_i) in the decision-making process will comply with the order of preference established by C_co > C_se > C_zo > C_we > C_op according to the principles of the VIP. Thus, the W_i will be arranged as follows: W_co > W_se > W_zo > W_we > W_op.

The second step consists of simulating five samples of 1000 random values for the criteria weights (W_co, W_se, W_zo, W_we and W_op) using a function randomly distributed on the interval 0–1. The generation of these values is subject to the following constraints: W_co > W_se > W_zo > W_we > W_op and W_co + W_se + W_zo + W_we + W_op = 1.

Then, the mean value for each criterion weight as well as the respective standard deviation and 95% credibility interval are estimated. The results of our simulation are presented in TABLE 5.

Table 5.

Results of the simulation of decision criteria weights.

		Wco	Wse	Wzo	Wwe	Wop
Mean		0.57166	0.24061	0.11079	0.0525	0.02443
Standard deviation		0.0314643	0.01246119	0.00591821	0.00188714	0.00076074
95% CI	L Bound U Bound	0.56970982 0.57361018	0.23983765 0.24138235	0.11042319 0.11115681	0.05238303 0.05261697	0.02438285 0.02447715

W_co: Weight for the criterion cost; W_se: Weight for the criterion sensor size; W_zo: Weight for the criterion zoom; W_we: Weight for the criterion weight; W_op: Weight for the criterion optical image stabilizer; CI: Confidence Interval.

The respective means presented in TABLE 5 will constitute the deterministic weight for the criteria cost, sensor size, zoom, weight and optical image stabilizer, which the DMs may use as part of their multi-criteria analysis. In addition, the standard deviations and 95% credibility intervals may be used to conduct sensitivity analyses.

Discussion

In this article, we introduced an alternative imprecise weight elicitation technique (called ELICIT) for use in MCDA for healthcare. ELICIT offers a simple way to operationalize a rank ordering of criteria by the DMs. The final weights totally depend on the rank ordering provided and are one of many sets of weights that can fit that ranking. The application of ELICIT consists of two steps: the rank ordering of evaluation criteria based on DMs’ preferences using the PCA; and the estimation of the criteria weights and their respective standard deviations and 95% credibility intervals using the VIP analysis and Monte Carlo method. The application of this approach was illustrated using a case study.

Several rank-order approaches have been proposed as imprecise weight elicitation techniques [14]. The main difference among these approaches lies in how ordinal values are transformed into surrogate cardinal weights [14]. The conversion techniques used to achieve this goal include the rank sum method, rank reciprocal method, rank exponent method [20] and the rank-order centroid method [21]. A rank-order method for weight elicitation that is similar to ELICIT is the ordinal criteria in stochastic multi-criteria acceptability analysis [22]. Stochastic multi-criteria acceptability analysis allows for the use of ordinal preferences (ranks) by DMs to estimate criteria weights using Monte Carlo simulation. The main difference with ELICIT lies in the way ordinal preferences are aggregated. To our knowledge, the preference aggregation in the stochastic multi-criteria acceptability analysis method is based on the Dempster–Shafer theory of evidence [23], which involves the use of subjective probabilities. On the contrary, ELICIT uses a preference aggregation procedure based on the PCA. The advantage of using PCA as a preference aggregation tool is further discussed.

There are a number of advantages to using ELICIT. As an imprecise weight elicitation technique, it requires few preference information from DMs to estimate criteria weights (parsimonious technique) compared to trade-off methods such as the DCE [24]. This feature would be appealing to DMs who experience challenges in fixing precise values to the scaling coefficients of decision criteria. ELICIT is appropriate for individual or group decision-making, for one time and recurrent decision-making, especially when the number of criteria or the ranking of existing criteria changes (an advantage over techniques such as the DCE). It can be used in sensitivity analysis to complement an MCDA process that utilizes a cardinal or ratio weight elicitation technique (e.g., DCE). ELICIT is quick, easy to understand and implement. It is grounded in economic theory. In fact, a number of assumptions used in our approach are based on the utility theory and mathematical principles. One unique feature of this technique is the use of the PCA as an aggregation tool for the group criteria ranking. As mentioned earlier, the PCA allows for the exploration of the correlation structure of data. When applied to ranks, this statistical method is robust to outliers. The use of PCA in ELICIT makes it similar to the analytical hierarchy process. The main difference between these two approaches is the extraction of the preference information. The extraction phase in ELICIT consists of only the rank ordering of the criteria. The representation of the data is done through the PCA. As for the analytical hierarchy process (AHP), the extraction of preference information is done through a series of pairwise comparison questions, using a 9-point verbal/semantic scale. These pairwise comparisons between criteria are compiled in reciprocal comparison matrices or matrix algebra. It is noteworthy that the exactness of the conversion of preference data from the semantic scale to the numeric scale in the AHP has been criticized in the literature [25]. Potential inconsistencies in the responses obtained from the pairwise comparisons have led to the identification of rank reversals when using the AHP [25].

Similar to any other methods, there exist limitations to the use of ELICIT. With the Monte Carlo simulations, by use of a uniform distribution, the weights are based on the average values. The standard errors that are obtained are only the standard errors of the mean values. They do not account for the variability among DMs. That being said, this limitation is addressed by the use of PCA at the ranking step. Additionally, the standard errors are highly dependent on the number of iterations.

The application of MCDA in healthcare has gained momentum. One of the growing topics in bridging MCDA and healthcare decision-making is the identification of the most appropriate weight elicitation technique. It is the authors’ belief that there is not a single best elicitation technique (‘panacea’) to estimating scaling coefficients, since that process is dependent upon a number of parameters such as the ability of DMs to provide full preference versus incomplete preference information about the decision-making criteria, the frequency of decision-making (one time vs routine) and the time required to implement the elicitation technique. In this regard, the proposed technique offers a number of features that some categories of DMs would find attractive, while complementing existing weight elicitation techniques available in the literature.

Key issues.

• A critical step in the application of most multi-criteria decision analysis methods is the elicitation of the decision-makers’ (DMs) preferences in regard to the evaluation criteria.

• Assigning precise values to scaling coefficients (criteria weights) through the use of cardinal and/or ratio weight elicitation techniques can be challenging and burdensome for some DMs.

• There is a need to develop imprecise weight elicitation techniques to facilitate the application of multi-criteria decision analysis methods.

• ELICIT is an imprecise weight elicitation technique that can prove useful to DMs who experience difficulties when asked to fix precise values to the scaling coefficients of predefined decision criteria.