A New Coefficient of Rankings Similarity in Decision-Making Problems

Abstract

Multi-criteria decision-making methods are tools that facilitate and help to make better and more responsible decisions. Their main objective is usually to establish a ranking of alternatives, where the best solution is in the first place and the worst in the last place. However, using different techniques to solve the same decisional problem may result in rankings that are not the same. How can we test their similarity? For this purpose, scientists most often use different correlation measures, which unfortunately do not fully meet their objective.

In this paper, we identify the shortcomings of currently used coefficients to measure the similarity of two rankings in decision-making problems. Afterward, we present a new coefficient that is much better suited to compare the reference ranking and the tested rankings. In our proposal, positions at the top of the ranking have a more significant impact on the similarity than those further away, which is right in the decision-making domain. Finally, we show a set of numerical examples, where this new coefficient is presented as an efficient tool to compare rankings in the decision-making field.

Introduction

Decision making is an integral part of human life. Every day, every person is faced with different kinds of decision-making problems, which can affect both professional and private life. An example of a decision-making problem can be a change of legal regulations in the state, choice of university, purchase of a new car, determination of the amount of personal income tax, selection of a suitable location for the construction of a nuclear power plant, adoption of a plan of research, or the sale or purchase of stock exchange shares.

In the majority of cases, decision-making problems are based on many, often contradictory, decision-making criteria. Therefore multi-criteria decision-analysis (MCDA) methods and decision support systems enjoy deep interest both in the world of business and science. Almost in every case, a reliable decision requires the analysis of many alternatives. Each of them should be assessed from the perspective of all the criteria characterizing its acceptability. As the complexity of the problem increases, it becomes more and more challenging to make the optimal decision. An additional complication is that there is no complete mathematical form depending on the criteria and the expected consequences. In particularly important problems, the role of the decision-maker is entrusted to an expert in a given field or to a group of experts who will help to identify the best solution. We talk about individual or group decision making, respectively. Even then, it can often be problematic for an individual expert as well as for collegiate bodies to determine the right decision. In this case, MCDA methods can be helpful.

MCDA methods are great tools to support the decision-maker in the decision-making process. We can identify two main groups of MCDA methods, i.e., American and European schools [33]. Methods of the American school of decision support are based on the utility or value function [5, 16]. The most important methods belonging to this family are: analytic hierarchy process (AHP) [34], analytic network process (ANP) [35], utility theory additive (UTA) [21], simple multi-attribute rating technique (SMART) [28], technique for order preference by similarity to ideal solution (TOPSIS) [3, 32], or measuring attractiveness by a categorical based evaluation technique (MACBETH) [2]. Methods of European school of decision support use outranking relation in the preference aggregation process, where the most popular are ELECTRE family [1, 36] and PROMETHEE methods [8, 15]. Additionally, we can indicate the set of techniques based strictly on the rules of decision making. These methods use the fuzzy sets theory (COMET) [12, 25–27] and the rough set theory (DRSA) [29].

Generally, MCDA methods help to create a ranking of decision variants where the most preferred alternative comes first [4]. The problem arises when we use more than one MCDA method, and the rankings obtained are not identical. Then the question arises on how to compare the received rankings? Currently, the most popular approach is the analysis based on the correlation between the two or more rankings [7, 12, 19, 24]. However, we are going to show that this analysis is insufficient in the decision support domain. An appropriate approach should ensure that a better ranking in terms of the order can be identified. Then, with a proper benchmark, it would be possible to assess the correctness of the MCDA methods in terms of the rankings generated [22].

In this paper, we identify the shortcomings of currently used coefficients to measure the similarity of two rankings. The most significant contribution is the WS coefficient, which depends strictly on the position on which the difference in the ranking occurred. Afterward, three linguistic terms are identified by using trapezoidal fuzzy numbers, i.e., low, medium, and high similarity. We compare the proposed coefficient with Spearman, Kendall, and Goodman-Kruskal coefficients, which are commonly used to measure rankings similarity in MCDA problems [9, 12, 17, 18, 23]. In addition, the proposed approach is compared with the similar coefficients presented in [6, 10, 13]. For this purpose, numerical experiments are discussed.

The rest of the paper is organized as follows: In Sect. 2, some basic preliminary concepts are discussed. Section 3 introduces a new coefficient of rankings similarity in the decision-making problems. In Sect. 4, the practical feasibility study of the WS coefficient is discussed. In Sect. 5, we present the summary and conclusions.

Preliminaries

An important issue is how to compare the correctness of the order of the two rankings. The simplest method is to check whether the rankings are consistent or inconsistent. Such an approach is not sufficient and can be used almost exclusively for 2 or 3 elementary rankings [27]. The much more common approach is to use one of the coefficients of monotonous dependence of two variables, where the obtained rankings for a set of considered alternatives are our variables. The most commonly used symmetrical coefficient of such dependence is the Spearman’s coefficient [9, 17, 18, 23], which is expressed by the following formula (1):

1

where is defined as the difference between the ranks and n is the number of elements in the ranking. The Spearman’s coefficient is interpreted as a percentage of the rank variance of one variable, which is explained by the other variable [31].

The most frequently used asymmetrical monotonous coefficients of two variables are Kendall [12, 20] and Goodman-Kruskal coefficients [12, 14]. They are expressed in formulas (2) and (3) respectively:

2

3

where is the number of compatible pairs, is the number of non-compliant pairs, and n is the number of all pairs. The Kendall and Goodman-Kruskal coefficients, unlike Spearman, are interpreted in terms of probability. They represent the difference between the probability that the compared variables will be in the same order for both variables and the probability that they will be in the opposite order.

The presented coefficients are the most frequently used measures of the analysis of the rankings similarity in decision-making problems [9, 12, 17, 18, 23]. However, we want to indicate a significant shortcoming, which is related to the place of difference occurrence. The idea of measuring the rankings similarity is not new and has been the subject of many works [11, 30]. Particularly interesting in the context of the presented approach are works related to Blest’s measure of rank correlation v and the weighted rank measure of correlation [6, 10, 13]. They are expressed in formulas (4) and (5) respectively:

4

5

The presented coefficients (1-3) are regardless of whether the error occurs at the top or bottom; the values of the factors will be identical. In Table 1, the simple example shows five rankings, including one reference () and four test rankings (). The test rankings were created by a change in the correct ranking of the two adjacent alternatives. We want to remind that the rankings are determined to choose the best possible solution, and the value of the preferences decreases with each position in the ranking. The difference at the top should be more significant than an error at the bottom of the ranking. The exchange of alternative locations from the first and second position is a more considerable error than the swap of the second and third position. However, the values of the coefficients indicate that similarity of the test rankings to the reference ranking is the same for all test sets.

Table 1.

Summary of the test with reference ranking (Rx) and four test rankings () with the calculated correlation factors and proposed WS coefficient for the set of five alternatives (), each having a different position in the ranking.

	1		1	1	1
	2			2	2
	3	3			3
	4	4	4
	5	5	5	5
Coefficients		0.9000	0.9000	0.9000	0.9000
		0.8000	0.8000	0.8000	0.8000
	G	0.8000	0.8000	0.8000	0.8000
		0.8500	0.8833	0.9167	0.9500
	v	0.8500	0.8833	0.9167	0.9500

WS Coefficient of Rankings Similarity

The new ranking similarity factor should be resistant to the situation described in the previous section, and at the same time, should be sensitive to significant changes in the ranking. Besides, this factor should be easy to interpret, and its values should be limited to a specific interval.

We assumed that the new indicator should be strongly related to the difference between two rankings on particular positions. An additional assumption is that the top has a more significant influence on similarity than the bottom of the ranking. Based on these assumptions, a new indicator was developed, which can be presented as (6):

6

where WS is a value of similarity coefficient, N is a length of ranking, and mean the place in the ranking for element in respectively ranking x and ranking y.

The proof of convergence for the WS factor is quite simple. The formula (6) can be divided into two main components. The first one (7) is responsible for making the WS value dependent on the position in the reference ranking ().

7

We are dealing with a geometric series which is convergent. As proof, we can calculate a trivial limit.

8

The second component (9) determines to what extent the difference in rankings affects the similarity of rankings. This value can be obtained from zero (the positions are identical) to one.

9

If we multiply the (7) by (9) then this series cannot be higher than one. Therefore, it is clear that the WS coefficient can only take values from zero to one. We can compare all coefficients for a simple example in Table 1. The WS, , and v coefficients take into account the position of the error occurrence, and the rest of them remain the same regardless of where the error occurs. In the next section, other tests comparing the performance of the indicators will be presented and discussed.

Results and Discussion

Analysis of Five-Element Rankings

The first experiment here presents tied ranks, i.e., the same values in the ranking. It happens when two alternatives get the same place. For example, if two decision variants receive the first place together, the ranking will contain a value of 1.5 for both (the average of their positions). Table 2 shows the results of calculations for the five-element ranking, where the different location of tied pairs is considered.

Table 2.

Summary of the test with reference ranking (Rx) and four test rankings () with the calculated correlation factors and proposed WS coefficient for the set of five alternatives (), where one pair has the same position in the ranking.

	1		1	1	1
	2			2	2
	3	3			3
	4	4	4
	5	5	5	5
Coefficients		0.9747	0.9747	0.9747	0.9747
		0.9487	0.9487	0.9487	0.9487
	G	1.0000	1.0000	1.0000	1.0000
		0.9625	0.9708	0.9792	0.9875
	v	0.9250	0.9417	0.9583	0.9750

One more again, WS, , and v coefficients show the change of value together with the change of position on which there are the tied pairs. It is a property that was identified as a significant drawback of the currently used methods, i.e., Spearman, Kendall, and Goodman-Kruskal. Ranking is more similar than because full correctness occurs on the first three positions and not on the last three.

Another simple experiment consists in creating test rankings, where successive rankings differ from the base ranking by the alternative indicated as the best. The results for the five-element ranking are shown in Table 3. Replacing the best option with the worst in all coefficients results in a negative value result (except WS). It means a negative correlation, which is not trivial to interpret in decision-making problems. Besides, interesting is the case of the ranking because it obtained a total lack of correlation (for and G coefficients). It means that the order of the base and second rankings is utterly unrelated to each other. It is a confirmation that these classical rank coefficients do not examine the similarity of the two rankings thoroughly. In general, all coefficients assess test rankings against the base ranking in a somewhat similar way. The rationale for this is that the three positions in the ranking have been indicated flawlessly.

Table 3.

Summary of the test with reference ranking (Rx) and four test rankings () with the calculated correlation factors and proposed WS coefficient for the set of five alternatives (), where each ranking has a different position error.

	1
	2		2	2	2
	3	3		3	3
	4	4	4		4
	5	5	5	5
Coefficients		0.9000	0.6000	0.1000	−0.6000
		0.8000	0.4000	0.0000	−0.4000
	G	0.8000	0.4000	0.0000	−0.4000
		0.8500	0.4667	−0.0500	−0.6000
	v	0.8500	0.4667	−0.0500	−0.6000

The last example in this subsection examines the coefficients in two cases, i.e., for test rankings and ). Once again, the ranking is used as a reference point. The detailed results are presented in Table 4.

Table 4.

Summary of the test with reference ranking (Rx) and four test rankings () with the calculated correlation factors and proposed WS coefficient for the set of five alternatives (), where the change of coefficients is investigated

	1	2	5	2	4
	2	3	1	3	2.5
	3	4	2	5	1
	4	5	3	4	5
	5	1	4	1	2.5
Coefficients		0.0000	0.0000	−0.1000	−0.0513
		0.2000	0.2000	0.0000	−0.1054
	G	0.2000	0.2000	0.0000	−0.1111
		0.0000	0.0000	−0.0667	−0.0667
	v	0.1667	−0.1667	0.0833	−0.1083

Rankings and have equal values for most of the coefficients, where only WS and v are exceptions. Ranking is significantly better than ranking . Even though the alternative has been identified as the best in . The rest of this ranking has been correctly identified according to the right order. However, in the , the best alternative was wrongly rated as the worst. Therefore, the best alternative () has a chance to be chosen in the first case and not in the second. These rankings cannot be evaluated as being the same. This shows the superiority of WS and v coefficients in decision-making ranks analysis. Rankings and show greater variability of coefficients, i.e., the ranking has a coefficient value less, equal to or greater than the ranking . It all depends on which coefficient is taken into account, but WS and v again point to the superiority of the ranking .

Influence of a Ranking Size on Coefficients

In this subsection, we want to indicate the impact of the ranking size on the achieved value of the indicator. Figure 1 shows comparisons of WS, , , and v coefficients. Only alternatives on the first and second positions have been replaced. It is a consequence of the conclusion drawn from Table 1. We take into account the rankings with the number from 5 to 50 elements. We can observe that with the increased ranking size, the similarity with the assumed assumptions increases. The Ws coefficient is characterized by the greatest variability, depending on the size of the ranking. Figure 2 shows the changes in the WS value when replacing the best elements with the second, third, fourth, and fifth ones (the position of one adjacent pair is swapped). As we can see, the WS values decrease accordingly, as the quality of the rankings decreases as the best solution moves away from the top of the ranking.

Fig. 1.

The value of the coefficients depending on the length of the ranking (n), where occurs one error (change of the first and second position in the ranking) and the converted positions in the ranking.

Fig. 2.

The value of the WS coefficient depending on the length of the ranking (n) and the converted positions in the ranking.

Distribution of Coefficients Values

In the next step, we attempted to visualize the distributions for three indicators. Figure 3 presents the distribution of the Kendall factor for all possible permutations of the sets of five, six, seven, eight, nine, and ten elements. Figures 4 and 5 show the distribution of the Spearman coefficient and the WS coefficient, respectively. The shape of the Spearman values is smoother than the Kendall. Both indicators have a symmetrical distribution, unlike the WS coefficient. The problem may be the interpretation of the WS value, because it is a new approach. However, the question arises when the similarity of the WS coefficient is low, medium, and high. A statistical analysis of the distribution of the WS should be carried out to define three appropriate linguistic terms and answer on this research question.

Fig. 3.

Sorted distribution of all values of the Kednall coefficient in relation to the length of the ranking (n).

Fig. 4.

Sorted distribution of all values of the Spearman coefficient in relation to the length of the ranking (n).

Fig. 5.

Sorted distribution of all values of the WS coefficient in relation to the length of the ranking (n).

Definition of Rankings Similarity

All possible permutations and values of the WS coefficient are determined for ranks of the size from 3 to 10 elements. Based on the obtained values, we calculated basic statistics, which are presented in Table 5. For larger rankings, statistics are based on random samples of 100,000 rankings. Both population and random sampling data are used. Note the convergence of the arithmetic mean, standard deviation, and typical value ranges. The biggest differences concern the arithmetic mean, and it is equal to 0.207 (for a ranking of 10 and 1000 elements). Based on the analysis of typical values, i.e. interval of [; ], we identified the linguistic terms low, medium and high similarity of rankings.

Table 5.

A summary of the basic statistics of the WS coefficient for all possible permutations, where n length of the ranking.

n
3	0.5208	0.2869	0.2339	0.8077	0.1875	1.0000
4	0.5313	0.2164	0.3149	0.7477	0.2083	1.0000
5	0.5135	0.1938	0.3197	0.7073	0.1510	1.0000
6	0.5195	0.1817	0.3378	0.7012	0.1656	1.0000
7	0.5164	0.1757	0.3407	0.6921	0.1383	1.0000
8	0.5197	0.1721	0.3476	0.6918	0.1314	1.0000
9	0.5193	0.1700	0.3493	0.6893	0.1252	1.0000
10	0.5208	0.1688	0.3520	0.6896	0.1144	1.0000

It can indeed be said that if the WS is less than 0.234, then the similarity is low. If the value is higher than 0.808, then the similarity is high. The medium of likeness, which corresponds to a typical value, belongs to the range from 0.352 to 0.689. The remaining values are values where we can talk about a partial belonging to linguistic concepts according to the theory of fuzzy sets, or just low/medium and medium/high concept can be used. Detailed definitions are presented in Fig. 6. Linguistic values are important because they can be used to evaluate the adjustment of the reference and test rankings.

Fig. 6.

The definitions of three linguistic terms, i.e., low, medium, and high similarity of rankings by using trapezoidal fuzzy numbers.

Conclusions

The main contribution of the paper is a proposal of the new coefficient of the rankings similarity. For this purpose, the short analysis of classical factors are presented, and some of their shortcomings are emphasized. The most critical is the equality of the values of the classical coefficients in case the ranking error concerns the replacement of a pair of adjacent alternatives (Table 1). The paper presents a theoretical foundation of proposed WS coefficient, which ensures that a new factor is free of identified shortcomings.

The results of numerical experiments compare all analyzed coefficients and their correctness, i.e., Spearman, Kendall, G Goodman-Kruskal, and WS coefficients. Then, the distributions of Kendall, Spearman, and WS coefficients were compared. WS values can be used to measure similarity of rankings.

Finally, three linguistic concepts were formulated for the low, medium, and high similarity of the two rankings. The properties of the WS coefficient indicate that it is a useful tool for comparing the similarity of rankings and is better suited for this purpose than the currently used correlation coefficients.

During the research, some improvement areas have been identified. The future work directions should concentrate on:

• further comparing between existing coefficients and the proposed WS;

• testing the use of the WS coefficient in real-life examples;

• detection and correction of WS coefficient shortcomings;

• adaptation of the proposed coefficient to uncertain (fuzzy) rankings.