An integrated multi-criteria decision making approach with linguistic hesitant fuzzy sets for E-learning website evaluation and selection

Abstract

Network teaching has been widely developed under the influence of COVID-19 pandemic to guarantee the implementation of teaching plans and protect the learning rights of students. Selecting a particular website for network teaching can directly affects end users’ performance and promote network teaching quality. Normally, e-learning website selection can be considered as a complex multi-criteria decision making (MCDM) problem, and experts’ evaluations over the performance of e-learning websites are often imprecise and fuzzy due to the subjective nature of human thinking. In this article, we propose a new integrated MCDM approach on the basis of linguistic hesitant fuzzy sets (LHFSs) and the TODIM (an acronym in Portuguese of interactive and multi-criteria decision making) method to evaluate and select the best e-learning website for network teaching. This introduced method deals with the linguistic assessments of experts based on the LHFSs, determines the weights of evaluation criteria with the best–worst method (BWM), and acquires the ranking of e-learning websites utilizing an extended TODIM method. The applicability and superiority of the presented linguistic hesitant fuzzy TODIM (LHF-TODIM) approach are demonstrated through a realistic e-learning website selection example. Results show that the LHF-TODIM model being proposed is more practical and effective for solving the e-learning website selection problem under vague and uncertain linguistic environment.

Graphical abstract

1. Introduction

The outbreak of COVID-19 pandemic in late 2019 poses a serious challenge to world public health systems [1]. People’s lives and habits have been greatly affected by the COVID-19 pandemic [2]. The Chinese government has taken prompt actions to reduce the risk of community transmission and cluster infections in schools. In response to the rapid development of the COVID-19 epidemic, all schools across the country have postponed the start of school in accordance with the requirements of the Ministry of Education [1]. Network teaching has been widely developed under the influence of COVID-19 pandemic to protect students’ learning rights and guarantee the implementation of teaching plans [3], [4], [5].

The characteristics of the Internet make network teaching very different from traditional teaching. In network teaching, teachers and students have a lively interaction even though they may come from all over the country [6]. In addition, network teaching resources provide strong support for students in terms of both quantity and sharing. The advantages of network teaching over traditional teaching include time saving, cost reduction, better interaction and increased flexibility. These advantages have made network teaching more and more popular and led to an increase in the number of e-learning websites [7]. The quality of e-learning websites has received great attention from students and website developers [8], [9], [10]. In response, quality assessment of e-learning websites should be studied in more detail from the perspective of users [11], [12]. Choosing a specific e-learning website will directly affect the end users’ performance and promote the network teaching quality. Hence, it is necessary to develop reasonable and effective methods for the e-learning website selection.

In the process of e-learning website selection, various evaluation criteria need to be considered, and thus selecting an e-learning website with the best performance for online education can be regarded as a multi-criteria decision making (MCDM) problem [7], [13], [14]. As an efficient and pragmatic MCDM method, the TODIM (an acronym in Portuguese of interactive and multi-criteria decision making) was developed by Gomes and Lima [15] to choose suitable alternatives based on different criteria [16], [17]. The TODIM method is on the basis of cumulative prospect theory [18], which is an effective method to reflect the psychological behaviors of decision makers in a decision analysis process [19], [20], [21]. In addition, this method can help decision makers adjust the corresponding parameters and make the decision results more in line with their preferences [22], [23], [24]. For these merits, the TODIM method and its extended forms have been widely utilized to handle different sorts of MCDM problems, such as stock investment selection [25], multi-project cash flow evaluation [26], carbon storage technology selection [27], urban photovoltaic charging station location [28], and renewable energy investment risk assessment [29]. Therefore, this paper aims to utilize an extended TODIM method for the e-learning website selection.

On the other hand, it is often difficult for experts to use crisp numbers to accurately evaluate the performance of e-learning websites. Usually, linguistic terms such as “good” and “poor” are more suitable for them to express their opinions on the performance of e-learning websites. The linguistic hesitant fuzzy sets (LHFSs) introduced by Meng et al. [30] are a powerful and effective fuzzy information representation tool. The LHFSs combine hesitant fuzzy sets with linguistic fuzzy sets to externalize the ambiguity of human cognition and the complexity of uncertain environment. This method allows several possible linguistic values to denote the membership degree of an element to a set, and is effective for expressing fuzzy and uncertain information of decision makers [31], [32], [33]. Given above advantages, the LHFSs have been used to describe the vagueness and uncertainty of various decision-making problems, which include renewable energy selection [34], university performance management [35], seawater pumped hydro storage project risk assessment [36], biomass power generation fuel procurement [37], and surrounding rock stability analysis [38]. Therefore, the LHFSs are of great value in dealing with evaluation information in the process of e-learning website selection.

Based on the above discussions, this paper attempts to extend the TODIM method in linguistic hesitant fuzzy environment to develop a new approach, called linguistic hesitant fuzzy TODIM (LHF-TODIM), for e-learning website evaluation and selection. In summary, this research makes the following important contributions: (1) The LHFSs are employed to express the uncertain and complicated assessment information of experts on e-learning websites; (2) an extended best–worst method (BWM) is presented to compute the weights of evaluation criteria on the basis of a constrained optimization model; and (3) a modified TODIM approach is put forward for ranking the considered e-learning websites and determining the best one for network teaching. Finally, a real instance is provided to demonstrate the LHF-TODIM model and make a comparative analysis to further illustrate the benefits of the presented e-learning website evaluation approach.

The structure of this article is arranged as follows. In Section 2, a literature review of current e-learning website evaluation and selection methods is provided. In Section 3, some definitions and operational laws of the LHFSs are briefly reviewed. In Section 4, an extended TODIM method is proposed for e-learning website selection under linguistic hesitant fuzzy environment. In Section 5, a practical e-learning website selection case is provided to demonstrate the proposed LHF-TODIM model. Finally, conclusions and recommendations for further research are summarized in Section 6.

2. Literature review

Over the years, the e-learning website evaluation and selection has become an increasingly important research field and a lot of e-learning website selection methods have been put forward in the literature. For instance, Garg et al. [7] proposed a fuzzy complex proportional assessment (COPRAS) method for evaluating and selecting programming language e-learning websites. Garg [13] employed a matrix method for solving e-learning website evaluation problems. Büyüközkan et al. [8] employed an axiomatic design (AD)-based approach for evaluating the performance of e-learning websites. Büyüközkan et al. [39] presented a quality evaluation framework based on fuzzy VIKOR (VIseKriterijumska Optimizacija I KOmpromisno Resenje) to measure the performance of e-learning websites. Garg [40] developed a computational and quantitative model based on weighted Euclidean distance approximation and complex ratio evaluation for selecting e-learning websites. Kazançoǧlu and Aksoy [41] employed fuzzy logic-based quality function deployment (QFD) to choose the most suitable e-learning service provider. Khan et al. [42] presented an application of proximity indexed value (PIV) method for the selection of the best E-learning website. Jain et al. [43] proposed a weighted distance-based approximation (WDBA) method for e-learning website evaluation and selection.

In addition, some evaluation and selection approaches have been reported for specific websites. For example, Perçin [44] developed a model based on fuzzy decision-making trial and evaluation laboratory (DEMATEL) method and a generalized Choquet fuzzy integral for hospital website quality evaluation. Akincilar and Dagdeviren [45] presented a hybrid MCDM model based on analytic hierarchy process (AHP) and PROMETHEE method for evaluating hotel websites. Özkan et al. [46] established a model on the basis of technique for order of preference by similarity to ideal solution (TOPSIS) to evaluate the websites of industrial engineering departments under hesitant fuzzy linguistic context. Pamučar et al. [47] proposed an evaluation framework based on interval rough numbers and multi-attributive border approximation area comparison (MABAC) method for the selection of university websites. In [48], VIKOR was combined with TODIM for evaluating the internet banking website quality within Pythagorean fuzzy environment. In [49], AHP and fuzzy TOPSIS were integrated to implement an evaluation of websites with specialized cultural content. In [50], an integrated decision system consisting of single-valued trapezoidal neutrosophic sets and DEMATEL method was suggested for the evaluation of e-commerce websites. The DEMATEL, analytic network process (ANP), and VIKOR methods were employed by Tsai et al. [51] for the improvement analysis of national park websites.

As reviewed previously, researchers have made great efforts in evaluating and improving e-learning websites. On the one hand, some studies tackled e-learning website selection problems based on the fuzzy set theory. However, fuzzy sets can only represent fuzziness through membership degree and cannot reflect the inconsistency, hesitancy and uncertainty of decision makers. In addition, the reviewed website evaluation methods rarely consider psychological behaviors of decision makers in the website ranking process. To address these limitations, we develop a new decision-making framework that combines LHFSs and a modified TODIM method for the e-learning website evaluation and selection.

3. Preliminaries

In this section, some basic concepts of the LHFSs, which will be used in the proposed e-learning website evaluation model, are presented.

Definition 1 [30] —

Let $S=\left\{s_1,s_2,\dots ,s_{2t+1}\right\}$ be a linguistic term set. An LHFS in S is a set that when applied to the linguistic terms of S it returns a subset of S with several values in [0, 1]. It is defined as

$$LH=\left\{\left(h_i,l{h}_i\right)\left|h_i\in S,i=1,2,\dots ,l_{LH}\right)\right\},$$ (1)

where $l{h}_i=\left\{r_1,r_2,\dots ,r_{\left|l{h}_i\right|}\right\}$ is a set with $\left|l{h}_i\right|$ values in [0, 1], denoting the possible membership degrees of the element $h_i\in S$ to the set LH; ${\mathit{l}}_{LH}$ is the number of linguistic terms in LH.

Definition 2 [30] —

Let $L{H}_i=\left\{\left(h_i,l{h}_i\right)\left|h_i\in S,i=1,2,\dots ,l_{L{H}_i}\right)\right\}$ and $L{H}_j=\left\{\left(h_j,l{h}_j\right)\left|h_j\in S,i=1,2,\dots ,l_{L{H}_j}\right)\right\}$ be any two LHFSs with $l_{L{H}_i}=l_{L{H}_j}$, and $f\left(s_i\right)=i$ be a subscription function. Then the operation rules of LHFSs are shown as follows:

• (1)$L{H}_i\oplus L{H}_j={\cup }_{\left(s_i,lh\left(s_i\right)\right)\in L{H}_i,\left(s_j,lh\left(s_j\right)\right)\in L{H}_j}\times \left\{\left(s_{f\left(h_i\right)+f\left(h_j\right)},\left({\cup }_{r_p^i\in l{h}_i,r_q^j\in l{h}_j}1-\left(1-r_p^i\right)\left(1-r_q^j\right)\right)\right)\right\}$,
• (2)$L{H}_i\otimes L{H}_j={\cup }_{\left(h_i,l{h}_i\right)\in L{H}_i,\left(h_j,l{h}_j\right)\in L{H}_j}\times \left\{\left(s_{f\left(h_i\right)+f\left(h_j\right)},{\cup }_{r_p^i\in l{h}_i,r_q^j\in l{h}_j}{r}_p^i{r}_q^j\right)x\right\}$,
• (3)$\lambda L{H}_i={\cup }_{\left(h_i,l{h}_i\right)\in L{H}_i}\left\{\left(s_{\lambda f\left(h_i\right),}\right)\left(\left({\cup }_{r_p^i\in l{h}_i}1-{\left(1-r_p^i\right)}^{\lambda }\right)\right\}\right),\lambda \in \left[0,1\right]$,

where $r_p^i\in l{h}_i\left(p=1,2,\dots ,\left|l{h}_i\right|\right)$ and $r_q^j\in l{h}_j\left(q=1,2,\dots ,\left|l{h}_j\right|\right)$ denote the pth and the qth linguistic term possible membership degrees in ${\mathit{lh}}_i$ and ${\mathit{lh}}_j$, respectively.

Definition 3 [30] —

Let $LH=\left\{\left(h_i,l{h}_i\right)\left|h_i\in S,i=1,2,\dots ,l_{LH}\right)\right\}$ be an LHFS and $f\left(s_i\right)=i$ be a subscription function. Then the expectation function and the variance function of an LHFS are defined as:

$$E\left(LH\right)=\frac{1}{l_{LH}}\sum _{i=1}^{l_{LH}}\left[\frac{f\left(h_i\right)}{\left|l{h}_i\right|}\sum _{r_i\in l{h}_i}{r}_i\right],$$ (2)

$$V\left(LH\right)\frac{1}{l_{LH}}\sum _{i=1}^{l_{LH}}{\left[\frac{f\left(h_i\right)}{\left|l{h}_i\right|}\sum _{r_i\in l{h}_i}{r}_i-E\left(LH\right)\right]}^2.$$ (3)

Definition 4 [30] —

Let ${\mathit{LH}}_1$ and ${\mathit{LH}}_2$ be any two LHFSs. Then, their comparison rules are defined as follows:

• (1)If $E\left(L{H}_1\right)>E\left(L{H}_2\right)$, then $L{H}_1>L{H}_2$;
• (2)
  If $E\left(L{H}_1\right)=E\left(L{H}_2\right)$, then

  • (a)
    If $V\left(L{H}_1\right)<V\left(L{H}_2\right)$, then $L{H}_1>L{H}_2$;
  • (b)
    If $V\left(L{H}_1\right)=V\left(L{H}_2\right)$, then $L{H}_1\sim L{H}_2$.

Definition 5 [30] —

Let $L{H}_i\left(i=1,2,\dots ,n\right)$ be a collection of LHFSs and $f\left(s_i\right)=i$ be a subscription function. Then, the linguistic hesitant fuzzy weighted averaging (LHFWA) operator is defined by

$$\begin{array}{c}\text{LHFWA}\left(L{H}_1,L{H}_2,\dots ,L{H}_n\right)={\cup }_{\left(h_1,l{h}_1\right)\in L{H}_1,\dots ,\left(h_n,l{h}_n\right)\in L{H}_n}\hfill \\ \times \left(s_{\sum _{j=1}^n{w}_{j}f\left(h_j\right)},{\cup }_{r_1\in l{h}_1,\dots ,r_n\in l{h}_n}\left(1-\prod _{j=1}^n{\left(1-r_j\right)}^{w_j}\right)\right),\hfill \end{array}$$ (4)

where $w_j$ is the weight of $L{H}_i$, with $w={\left(w_1,w_2,\dots ,w_n\right)}^T$ being a weight vector satisfying $w_j\ge 0$ and ${\sum }_{j=1}^n{w}_j=1$.

Definition 6 [52] —

Let $S=\left\{s_1,s_2,\dots ,s_{2t+1}\right\}$ be a linguistic term set, $L{H}_i=\left\{\left(h_i,l{h}_i\right)\left|h_i\in S,i=1,2,\dots ,l_{L{H}_i}\right\}\right)$ and $L{H}_j=\left\{\left(h_j,l{h}_j\right)\right)\left|h_j\in S,\left(j=1,2,\dots ,l_{L{H}_j}\right\}\right)$ be two LHFSs, and $f^{\prime }\left(s_i\right)=\frac{i}{2t+1}$ be an extended scale function. Then, the distance between ${\mathit{LH}}_i$ and ${\mathit{LH}}_j$ can be computed by

$$\begin{array}{c}d\left(L{H}_i,L{H}_j\right)=\frac{1}{2}\left[\frac{1}{l_{L{H}_i}}\sum _{\left(h_i,l{h}_i\right)\in L{H}_i}\underset{\left(h_j,l{h}_j\right)\in L{H}_j}{min}\right)\hfill \\ \phantom{d\left(L{H}_i,L{H}_j\right)=}\times |\frac{\frac{f^{\prime }\left(h_i\right)}{\left|l{h}_i\right|}\left(\sum _{p=1}^{\left|l{h}_i\right|}p{r}_p^i\right)}{\sum _{p=1}^{\left|l{h}_i\right|}p}-\frac{\frac{f^{\prime }\left(h_j\right)}{\left|l{h}_j\right|}\left(\sum _{q=1}^{\left|l{h}_j\right|}q{r}_q^j\right)}{\sum _{q=1}^{\left|l{h}_j\right|}q}|\hfill \\ \phantom{d\left(L{H}_i,L{H}_j\right)=}+\frac{1}{l_{L{H}_j}}\sum _{\left(h_j,l{h}_j\right)\in L{H}_j}\underset{\left(h_i,l{h}_i\right)\in L{H}_i}{min}\hfill \\ \phantom{d\left(L{H}_i,L{H}_j\right)=}\times \left(|\frac{\frac{f^{\prime }\left(h_j\right)}{\left|l{h}_j\right|}\left(\sum _{q=1}^{\left|l{h}_j\right|}q{r}_q^j\right)}{\sum _{q=1}^{\left|l{h}_j\right|}q}-\frac{\frac{f^{\prime }\left(h_i\right)}{\left|l{h}_i\right|}\left(\sum _{p=1}^{\left|l{h}_i\right|}p{r}_p^i\right)}{\sum _{p=1}^{\left|l{h}_i\right|}p}|\right],\hfill \end{array}$$ (5)

where $r_p^i\in l{h}_i\left(p=1,2,\dots ,\left|l{h}_i\right|\right)$ and $r_q^j\in l{h}_j\left(q=1,2,\dots ,\left|l{h}_j\right|\right)$ are the pth and the qth linguistic term possible membership degrees in ${\mathit{lh}}_i$ and ${\mathit{lh}}_j$, respectively.

4. The proposed method for e-learning website selection

This section presents a new approach called LHF-TODIM for e-learning website selection. This methodology includes two stages: Calculating the weights of evaluation criteria by a modified BWM, and determining the ranking of e-learning websites through an extended TODIM method.

For an e-learning website selection problem, suppose that there are l experts $E_k\left(k=1,2,\dots ,l\right)$ in an expert team responsible for the assessment of m e-learning websites $W_i\left(i=1,2,\dots ,m\right)$ with respect to n evaluation criteria $C_j\left(j=1,2,\dots ,n\right)$. Each expert ${\mathit{E}}_k$ is given a weight ${\lambda }_k>0\left(k=1,2,\dots ,l\right)$ satisfying ${\sum }_{k=1}^l{\lambda }_k=1$ to reflect his/her relative importance in the e-learning website selection process. Let $H_k={\left(L{H}_{ij}^k\right)}_{m\times n}\left(k=1,2,\dots ,l\right)$ be the LHF assessment matrix of the kth expert based on the linguistic term set $S=\left\{s_1,s_2,\dots ,s_{2t+1}\right\}$, where $L{H}_{ij}^k=\left\{\left(h_{ij}^k,l{h}_{ij}^k\right)\right\}$ is an LHFS for the assessment of e-learning website ${\mathit{W}}_i$ with respect to criterion ${\mathit{C}}_j$. Then, the procedure for the developed LHF-TODIM model is explained in the following subsections.

4.1. Evaluation criteria weighting

The BWM originally put forward by Rezaei [53] is a new weighting method based on pairwise comparison in addressing MCDM problems. This method uses two vectors that are compared in pairs to compute the weights of criteria. In this stage, we extend the BWM by LHFSs and propose the LHF-BWM to obtain the weights of evaluation criteria. Its specific steps are explained below.

Step 1: Identify the best and the worst evaluation criteria.

In this step, each expert ${\mathit{E}}_k$ is responsible for determining the best (most important) criterion $C_B^k$ and the worst (least important) criterion $C_W^k$ from the n evaluation criteria on the basis of their understanding of the target problem.

Step 2: Determine the LHF best-to-others vectors.

The l experts give their importance preferences of the best criterion over each of the others utilizing the linguistic term set $S^{\prime }=\left\{s_1^{\prime },s_2^{\prime },\dots ,s_{2t+1}^{\prime }\right\}$. The obtained LHF best-to-others vector determined by ${\mathit{E}}_k$ can be represented by

$$V_B^k=\left(v_{B1}^k,v_{B2}^k,\dots ,v_{Bn}^k\right),k=1,2,\dots ,l,$$ (6)

where $v_{Bj}^k=\left\{\left(h_{Bj}^k,r_{Bj}^k\right)\right\}$ denotes the kth expert’s LHF importance preference of $C_B^k$ over criterion $C_j^k$ for $j=1,2,\dots ,n$.

Step 3: Determine the LHF others-to-worst vectors.

By utilizing the same linguistic term set, the LHF importance preference of each criterion against the worst criterion can be determined by experts. The obtained LHF others-to-worst vector can be represented by

$$V_W^k=\left(v_{1W}^k,v_{2W}^k,\dots ,v_{nW}^k\right),k=1,2,\dots ,l,$$ (7)

where $v_{jW}^k=\left\{\left(h_{jW}^k,r_{jW}^k\right)\right\}$ indicates the kth expert’s LHF importance preference of the criterion $C_j^k$ over $C_W^k$ for $j=1,2,\dots ,n$.

Step 4: Calculate the LHF criteria weights.

Suppose $w_B^k=\left\{\left(h_B^k,r_B^k\right)\right\}$, $w_j^k=\left\{\left(h_j^k,r_j^k\right)\right\}$, and $w_W^k=\left\{\left(h_W^k,r_W^k\right)\right\}$. For the expert $E_k$, the distances $d\left(w_B^k,v_{Bj}^k{w}_j^k\right)$ and $d\left(w_j^k,v_{jW}^k{w}_W^k\right)$ need to be minimized to acquire the LHF criteria weights. Thus, we construct the following constrained optimization model to calculate criteria weights:

$$\begin{array}{c}min{\xi }^k\hfill \\ s.t.\left\{\begin{array}{c}|f^{\prime }\left(h_B^k\right)r_B^k-f^{\prime }\left(s_{f\left(h_{Bj}^k\right)f\left(h_j^k\right)}\right)r_{Bj}^k{r}_j^k|\le {\xi }^k,\mathrm{for}\mathrm{all}j\hfill \\ |f^{\prime }\left(h_j^k\right)r_j^k-f^{\prime }\left(s_{f\left(h_{jW}^k\right)f\left(h_W^k\right)}\right)r_{jW}^k{r}_W^k|\le {\xi }^k,\mathrm{for}\mathrm{all}j\hfill \\ \sum _{j=1}^{n}E\left(w_j^k\right)=1,E\left(w_j^k\right)\ge 0,\mathrm{for}\mathrm{all}j.\hfill \end{array}\right)\hfill \end{array}$$ (8)

According to the definition of absolute values [54], model (8) is equivalent to the following form

$$\begin{array}{c}min{\xi }^k\hfill \\ s.t.\left\{\begin{array}{c}{f}^{\prime }\left(h_B^k\right)r_B^k-{\xi }^k\le f^{\prime }\left(s_{f\left(h_{Bj}^k\right)f\left(h_j^k\right)}\right)r_{Bj}^k{r}_j^k,\mathrm{for}\mathrm{all}j\hfill \\ f^{\prime }\left(h_B^k\right)r_B^k+{\xi }^k\le f^{\prime }\left(s_{f\left(h_{Bj}^k\right)f\left(h_j^k\right)}\right)r_{Bj}^k{r}_j^k,\mathrm{for}\mathrm{all}j\hfill \\ f^{\prime }\left(h_j^k\right)r_j^k-{\xi }^k\le f^{\prime }\left(s_{f\left(h_{jW}^k\right)f\left(h_W^k\right)}\right)r_{jW}^k{r}_W^k,\mathrm{for}\mathrm{all}j\hfill \\ f^{\prime }\left(h_j^k\right)r_j^k+{\xi }^k\le f^{\prime }\left(s_{f\left(h_{jW}^k\right)f\left(h_W^k\right)}\right)r_{jW}^k{r}_W^k,\mathrm{for}\mathrm{all}j\hfill \\ \sum _{j=1}^{n}E\left(w_j^k\right)=1,E\left(w_j^k\right)\ge 0,\mathrm{for}\mathrm{all}j.\hfill \end{array}\right)\hfill \end{array}$$ (9)

The above model is linear. Thus, the LHF weight vector $w^k={\left(w_1^k,w_2^k,\dots ,w_n^k\right)}^T$ determined by ${\mathit{E}}_k$ can be acquired by solving model (9).

Step 5: Calculate the overall weights of criteria.

After acquiring the LHF weight vector $w^k\left(k=1,2,\dots ,l\right)$, the overall LHF weight vector $w={\left(w_1,w_2,\dots ,w_n\right)}^T$ is obtained by the LHFWA operator as:

$$\begin{array}{c}{w}_j=\text{LHFWA}\left(w_j^1,w_j^2,\dots ,w_j^l\right)={\cup }_{\left(h_j^1,l{h}_j^1\right)\in w_j^1,\dots ,\left(h_j^l,l{h}_j^l\right)\in w_j^l}\hfill \\ \times \left(s_{\sum _{k=1}^l{\lambda }_{k}f\left(h_j^k\right)},{\cup }_{r_j^1\in l{h}_j^1,\dots ,r_j^l\in l{h}_j^l}\left(1-\prod _{k=1}^l{\left(1-r_j^k\right)}^{{\lambda }_k}\right)\right).\hfill \end{array}$$ (10)

Table 1.

Information about the e-learning websites.

Website	Developer	Web address
${\mathit{W}}_1$	Shanghai Zhuoyue New Digital Technology Co., LTD.	www.zhihuishu.com
${\mathit{W}}_2$	Tencent Technology Co., LTD.	www.ke.qq.com
${\mathit{W}}_3$	NetEase and higher education society	www.icourse163.org
${\mathit{W}}_4$	Beijing Century Superstar Information Technology Development Co., LTD.	www.i.chaoxing.com
${\mathit{W}}_5$	Beijing Zhiqi Lanmo Information Technology Co., LTD.	www.mosoteach.cn

4.2. E-learning website ranking

The TODIM describes the advantages of each alternative over other alternatives by establishing a multi-criteria value function based on prospect theory [24], [55]. In this stage, the normal TODIM method is extended on the basis of LHFSs for ranking alternative e-learning websites. The LHF-TODIM approach comprises the following steps.

Step 6: Aggregate the individual assessments of experts.

According to the LHFWA operator, the individual LHF assessment matrixes of e-learning websites $H_k\left(k=1,2,\dots ,l\right)$ are aggregated to obtain the overall LHF assessment matrix $H={\left(L{H}_{ij}\right)}_{m\times n}$. That is,

$$\begin{array}{c}L{H}_{ij}=\text{LHFWA}\left(L{H}_{ij}^1,L{H}_{ij}^2,\dots ,L{H}_{ij}^l\right)={\cup }_{\left(h_1,l{h}_1\right)\in L{H}_{ij}^1,\dots ,\left(h_k,l{h}_k\right)\in L{H}_{ij}^l}\hfill \\ \times \left(s_{\sum _{k=1}^l{\lambda }_{k}f\left(h_k\right)},{\cup }_{r_1\in l{h}_1,\dots ,r_k\in l{h}_k}\left(1-\prod _{k=1}^l{\left(1-r_k\right)}^{{\lambda }_k}\right)\right).\hfill \end{array}$$ (11)

Step 7: Compute the relative weights of criteria.

The relative weight of criterion ${\mathit{C}}_j$ to the reference criterion ${\mathit{C}}_r$, ${\mathit{w}}_{jr}$, is calculated by

$$w_{jr}=\frac{E\left(w_j\right)}{E\left(w_r\right)},j=1,2,\dots ,n,$$ (12)

where $w_r=max\left\{\left(w_j\right|j=1,2,\dots ,n\right\}$.

Step 8: Compute the dominances between e-learning websites.

The dominance of the e-learning website ${\mathit{W}}_i$ over e-learning website ${\mathit{W}}_t$ under the criterion $C_j$ can be computed using the following formula:

$${\varphi }_j\left(W_i,W_t\right)=\left\{\begin{array}{cc}\sqrt{\frac{w_{jr}d\left(r_{ij},r_{tj}\right)}{\sum _{j=1}^n{w}_{jr}}},\hfill & \mathrm{if}{r}_{ij}>r_{tj}\hfill \\ 0,\hfill & \mathrm{if}{r}_{ij}=r_{tj}\hfill \\ \frac{-1}{\theta }\sqrt{\frac{\sum _{j=1}^n{w}_{jr}d\left(r_{ij},r_{tj}\right)}{w_{jr}}},\hfill & \mathrm{if}{r}_{ij}<r_{tj}\hfill \end{array}\right),$$ (13)

where the parameter $\theta$ represents the attenuation factor of the losses.

Step 9: Calculate the overall dominances among e-learning websites.

The overall dominance of e-learning website ${\mathit{W}}_i$ over e-learning website ${\mathit{W}}_t$ is calculated by

$$\delta \left(W_i,W_t\right)=\sum _{j=1}^n{\varphi }_j\left(W_i,W_t\right),i,t=1,2,\dots ,m.$$ (14)

Step 10: Compute the global evaluation values of e-learning websites.

The global evaluation value of each e-learning website is computed by

$$G{E}_i=\frac{\sum _{t=1}^m\delta \left(W_i,W_t\right)-\underset{i=1}{\overset{m}{min}}\left[\sum _{t=1}^m\delta \left(W_i,W_t\right)\right]}{\underset{i=1}{\overset{m}{max}}\left[\sum _{t=1}^m\delta \left(W_i,W_t\right)\right]-\underset{i=1}{\overset{m}{min}}\left[\sum _{t=1}^m\delta \left(W_i,W_t\right)\right]},i=1,2,\dots ,m.$$ (15)

For the e-learning website selection problem, the larger the global evaluation value of $G{E}_i$, the better the e-learning website ${\mathit{W}}_i$ will be. Consequently, all the m e-learning websites can be ranked according to the decreasing order of $G{E}_i\left(i=1,2,\dots ,m\right)$ values. The best website corresponding to the alternative with the maximum $G{E}_i$ value can be selected for e-learning.

5. Illustrative example

In this section, a practical example of e-learning website selection is provided to illustrate the feasibility and superiority of the presented LHF-TODIM framework.

5.1. Implementation

During the COVID-19 pandemic, a university needs to set up network teaching to protect students’ learning rights and guarantee the implementation of teaching plans. The target of this case study is to help a case university to seek the optimal e-learning website for network teaching. In this case example, we consider five e-learning websites, which are Wisdom Tree (${\mathit{W}}_1$), Tencent Classroom (${\mathit{W}}_2$), Massive Open Online Course (${\mathit{W}}_3$), Superstar Learning (${\mathit{W}}_4$), and Cloud Class (${\mathit{W}}_5$). These five e-learning websites all offer university courses in a variety of fields and support sharing of courses across universities.

Specifically, Wisdom Tree is a large global credit course operation service platform, which has nearly 3000 member schools. It helps member universities to realize inter-school course sharing and credit mutual recognition. Tencent Classroom is a professional online education platform, connecting users with learning needs at one end and educational institutions or teachers with good content at the other end. It integrates vocational education courses, design and creation, interest in life, language study and other fields to help students improve their vocational and employment skills. Massive Open Online Course provides courses of famous universities in China to the public. On this platform, higher education is free for everyone who wants to improve themselves. Each course is taught regularly, and the whole learning process consists of watching videos, participating in discussions, submitting assignments, interspersed with questions from the course and final exams. Superstar Learning is a professional mobile learning platform, providing users with a convenient network course learning platform. It contains professional course information to help users to learn, but also provides the function of network course for the general students. Cloud Class is an intelligent teaching assistant for teachers. Based on the mobile internet environment, it realizes the instant interaction between teachers and students. The perfect incentive and evaluation system stimulate students’ interest in autonomous learning. Cloud Class provides teachers with high-quality big data for teaching research, and realizes the personalized teaching function based on artificial intelligence technology. Other information regarding the selected e-learning websites is given in Table 1.

To implement the e-learning website evaluation, four experts (${\mathit{E}}_1$, ${\mathit{E}}_2$, ${\mathit{E}}_3$, and ${\mathit{E}}_4$) are invited to form an expert team. These experts are from knowledgeable e-learning teachers involved in educational design and online interface implementation. The detailed information of the four experts is displayed in Table 2. Because of their different backgrounds and experience, the weights assigned to these four experts are 0.25, 0.20, 0.30 and 0.25, respectively, by using the AHP method.

Table 2.

The expert background information.

Expert	Age	Gender	Department	Experience
${\mathit{E}}_1$	42	Male	Education Research Centre	11 years
${\mathit{E}}_2$	37	Female	Peking University	8 years
${\mathit{E}}_3$	45	Male	China Education Design Alliance	15 years
${\mathit{E}}_4$	39	Male	Fudan University	10 years

Based on a literature review [5], seven assessment criteria are taken into account for the e-learning website selection, which include user interface (${\mathit{C}}_1$), personalization (${\mathit{C}}_2$), interactivity (${\mathit{C}}_3$), security (${\mathit{C}}_4$), complete content (${\mathit{C}}_5$), navigation (${\mathit{C}}_6$), and right and understandable content (${\mathit{C}}_7$). By using the linguistic term set S $=$ {${\mathit{s}}_1$ $=$ Very poor, ${\mathit{s}}_2$ $=$ Poor, ${\mathit{s}}_3$ $=$ Medium poor, ${\mathit{s}}_4$ $=$ Medium, ${\mathit{s}}_5$ $=$ Medium good, ${\mathit{s}}_6$ $=$ Good, ${\mathit{s}}_7$ $=$ Very good}, the LHF assessment matrixes of the four experts are obtained as $H_k={\left(L{H}_{ij}^k\right)}_{5\times 7}\left(k=1,2,3,4\right)$. Take the first expert as an example; the LHF assessment matrix $H_1$ is displayed in Table 3.

Table 3.

LHF assessment matrix H${}_1$ of e-learning websites.

Criteria	${\mathit{W}}_1$	${\mathit{W}}_2$	${\mathit{W}}_3$	${\mathit{W}}_4$	${\mathit{W}}_5$
${\mathit{C}}_1$	{(${\mathit{s}}_4$,0.5)}	{(${\mathit{s}}_5$,0.3,0.5)}	{(${\mathit{s}}_6$,0.4)}	{(${\mathit{s}}_4$,0.2,0.5)}	{(${\mathit{s}}_5$,0.6),(${\mathit{s}}_6$,0.3)}
${\mathit{C}}_2$	{(${\mathit{s}}_4$,0.7)}	{(${\mathit{s}}_3$,0.3),(${\mathit{s}}_4$,0.6)}	{(${\mathit{s}}_5$,0.4)}	{(${\mathit{s}}_6$,0.1,0.3)}	{(${\mathit{s}}_5$,0.6)}
${\mathit{C}}_3$	{(${\mathit{s}}_5$,0.1),(${\mathit{s}}_6$,0.2)}	{(${\mathit{s}}_3$,0.3)}	{(${\mathit{s}}_6$,0.5)}	{(${\mathit{s}}_6$,0.3)}	{(${\mathit{s}}_4$,0.7)}
${\mathit{C}}_4$	{(${\mathit{s}}_5$,0.2,0.4)}	{(${\mathit{s}}_6$,0.5)}	{(${\mathit{s}}_5$,0.3),(${\mathit{s}}_6$,0.4)}	{(${\mathit{s}}_5$,0.3,0.6)}	{(${\mathit{s}}_6$,0.2),(${\mathit{s}}_7$,0.4)}
${\mathit{C}}_5$	{(${\mathit{s}}_3$,0.3)}	{(${\mathit{s}}_4$,0.2),(${\mathit{s}}_5$,0.4)}	{(${\mathit{s}}_4$,0.6)}	{(${\mathit{s}}_4$,0.2),(${\mathit{s}}_5$,0.4)}	{(${\mathit{s}}_5$,0.1)}
${\mathit{C}}_6$	{(${\mathit{s}}_5$,0.3,0.6)}	{(${\mathit{s}}_4$,0.4)}	{(${\mathit{s}}_5$,0.3),(${\mathit{s}}_6$,0.4)}	{(${\mathit{s}}_6$,0.2)}	{(${\mathit{s}}_5$,0.2,0.3)}
${\mathit{C}}_7$	{(${\mathit{s}}_5$,0.2,0.6)}	{(${\mathit{s}}_4$,0.2),(${\mathit{s}}_5$,0.4)}	{(${\mathit{s}}_5$,0.3),(${\mathit{s}}_6$,0.7)}	{(${\mathit{s}}_3$,0.6)}	{(${\mathit{s}}_5$,0.3)}

Next, we adopt the proposed LHF-TODIM approach to rank the performance of the five e-learning websites.

Step 1: The four experts determine the best and the worst criteria from their respective perspectives, and the results are shown in Table 4.

Table 4.

The best and the worst criteria identified.

Experts	The best criteria	The worst criteria
${\mathit{E}}_1$	Interactivity (${\mathit{C}}_3$)	Personalization (${\mathit{C}}_2$)
${\mathit{E}}_2$	Navigation (${\mathit{C}}_6$)	User interface (${\mathit{C}}_1$)
${\mathit{E}}_3$	Security (${\mathit{C}}_4$)	User interface (${\mathit{C}}_1$)
${\mathit{E}}_4$	Interactivity (${\mathit{C}}_3$)	Complete content (${\mathit{C}}_5$)

Step 2: By utilizing the linguistic term set $S^{\prime }$ $=$ {$s_1^{\prime }=$ Equally important, $s_2^{\prime }=$ Weakly important, $s_3^{\prime }=$ Strongly important, $s_4^{\prime }=$ Very important and $s_5^{\prime }=$ Absolutely important}, the LHF best-to-others vectors determined by the experts $V_B^k\left(k=1,2,3,4\right)$ are displayed in Table 5.

Table 5.

LHF best-to-others vectors.

Experts	Best criteria	Other criteria
		${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$
${\mathit{E}}_1$	${\mathit{C}}_3$	{($s_3^{\prime }$,0.2)}	{($s_5^{\prime }$,0.4)}	{($s_1^{\prime }$,1.0)}	{($s_2^{\prime }$,0.2)}	{($s_3^{\prime }$,0.8)}	{($s_2^{\prime }$,0.1)}	{($s_3^{\prime }$,0.7)}
${\mathit{E}}_2$	${\mathit{C}}_6$	{($s_4^{\prime }$,0.5)}	{($s_3^{\prime }$,0.3)}	{($s_2^{\prime }$,0.5)}	{($s_1^{\prime }$,0.3)}	{($s_4^{\prime }$,0.2)}	{($s_1^{\prime }$,1.0)}	{($s_2^{\prime }$,0.4)}
${\mathit{E}}_3$	${\mathit{C}}_4$	{($s_5^{\prime }$0.6)}	{($s_4^{\prime }$,0.2)}	{($s_1^{\prime }$,0.3)}	{($s_1^{\prime }$,1.0)}	{($s_4^{\prime }$,0.1)}	{($s_3^{\prime }$,0.2)}	{($s_1^{\prime }$,0.1)}
${\mathit{E}}_4$	${\mathit{C}}_3$	{($s_4^{\prime }$,0.3)}	{($s_3^{\prime }$,0.7)}	{($s_1^{\prime }$,1.0)}	{($s_3^{\prime }$,0.3)}	{($s_3^{\prime }$,0.4)}	{($s_1^{\prime }$,0.8)}	{($s_3^{\prime }$,0.6)}

Step 3: Similarly, the LHF others-to-worst vectors $V_W^k\left(k=1,2,3,4\right)$ are acquired as shown in Table 6.

Table 6.

LHF others-to-worst vectors.

Experts	Worst criteria	Other criteria
		${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$
${\mathit{E}}_1$	${\mathit{C}}_2$	{($s_2^{\prime }$,0.3)}	{($s_1^{\prime }$,1.0)}	{($s_3^{\prime }$,0.1)}	{($s_4^{\prime }$,0.4)}	{($s_2^{\prime }$,0.6)}	{($s_3^{\prime }$,0.1)}	{($s_3^{\prime }$,0.6)}
${\mathit{E}}_2$	${\mathit{C}}_1$	{($s_1^{\prime }$,1.0)}	{($s_3^{\prime }$,0.5)}	{($s_5^{\prime }$,0.2)}	{($s_4^{\prime }$,0.1)}	{($s_2^{\prime }$,0.5)}	{($s_4^{\prime }$,0.5)}	{($s_2^{\prime }$,0.2)}
${\mathit{E}}_3$	${\mathit{C}}_1$	{($s_1^{\prime }$,1.0)}	{($s_2^{\prime }$,0.3)}	{($s_4^{\prime }$,0.6)}	{($s_3^{\prime }$,0.5)}	{($s_2^{\prime }$,0.7)}	{($s_3^{\prime }$,0.2)}	{($s_4^{\prime }$,0.4)}
${\mathit{E}}_4$	${\mathit{C}}_5$	{($s_2^{\prime }$,0.4)}	{($s_2^{\prime }$,0.8)}	{($s_3^{\prime }$,0.7)}	{($s_3^{\prime }$,0.4)}	{($s_1^{\prime }$,1.0)}	{($s_5^{\prime }$,0.3)}	{($s_3^{\prime }$,0.5)}

Step 4: Based on the LHF assessments of experts, four optimization models can be established to obtain the weights of the seven criteria. For instance, the constrained optimization model for the expert ${\mathit{E}}_1$ is constructed as:

$$\begin{array}{c}min{\xi }^1\hfill \\ s.t.\left\{\begin{array}{c}|f^{\prime }\left(h_3^1\right)r_3^1-f^{\prime }\left(s_{f\left(h_{3j}^1\right)f\left(h_j^1\right)}\right)r_{3j}^1{r}_j^1|\le {\xi }^1,j=1,2,\dots ,7.\hfill \\ |f^{\prime }\left(h_j^1\right)r_j^1-f^{\prime }\left(s_{f\left(h_{j2}^1\right)f\left(h_2^1\right)}\right)r_{j2}^1{r}_2^1|\le {\xi }^1,j=1,3,\dots ,7.\hfill \\ f\left(h_1^1\right)r_1^1+f\left(h_2^1\right)r_2^1+f\left(h_3^1\right)r_3^1+f\left(h_4^1\right)r_4^1\hfill \\ +f\left(h_5^1\right)r_5^1+f\left(h_6^1\right)r_6^1+f\left(h_7^1\right)r_7^1=1,\hfill \\ f\left(h_j^1\right)r_j^1\ge 0,j=1,2,\dots ,7.\hfill \end{array}\right)\hfill \end{array}$$

By solving the above model, the LHF weight vector of the first expert is determined as:

$$\begin{array}{c}{w}^1=\left(\left\{\left(s_{0.236}^{\prime },0.439\right)\right\},\left\{\left(s_{0.585}^{\prime },0.264\right)\right\},\left\{\left(s_{0.137}^{\prime },0.572\right)\right\},\right)\hfill \\ \left(\left\{\left(s_{0.382}^{\prime },0.337\right)\right\},\left\{\left(s_{0.264}^{\prime },0.643\right)\right\},\left\{\left(s_{0.318}^{\prime },0.425\right)\right\},\right)\hfill \\ \left(\left\{\left(s_{0.465}^{\prime },0.389\right)\right\}\right).\hfill \end{array}$$

Step 5: Via Eq. (10), the individual criteria weights $w_j^k\left(k=1,2,3,4\right)$ are aggregated to obtain the overall LHF weight vector as:

$$\begin{array}{c}w=\left(\left\{\left(s_{0.401}^{\prime },0.157\right)\right\},\left\{\left(s_{0.348}^{\prime },0.221\right)\right\},\left\{\left(s_{0.293}^{\prime },0.863\right)\right\},\right)\hfill \\ \left\{\left(s_{0.427}^{\prime },0.365\right)\right\},\hfill \\ \left(\left\{\left(s_{0.260}^{\prime },0.265\right)\right\},\left\{\left(s_{0.489}^{\prime },0.472\right)\right\},\left\{\left(s_{0.379}^{\prime },0.398\right)\right\}\right).\hfill \end{array}$$

Step 6: By using Eq. (11), the individual LHF assessments provided by experts are aggregated to acquire the overall LHF assessment matrix $H={\left(L{H}_{ij}\right)}_{5\times 7}$ as shown in Table 7.

Table 7.

The overall LHF assessment matrix.

E-learning websites	Criteria
	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$
${\mathit{W}}_1$	{(${\mathit{s}}_{\mathrm{4.45}}$,0.57,0.58),(${\mathit{s}}_{\mathrm{4.7}}$,0.48,0.50)}	{(${\mathit{s}}_{\mathrm{3.3}}$,0.46,0.49,0.59,0.61)}	{(${\mathit{s}}_{\mathrm{4.3}}$,0.31,0.37),(${\mathit{s}}_{\mathrm{4.55}}$,0.33,0.38)}	{(${\mathit{s}}_{\mathrm{5.45}}$,0.15,0.21),(${\mathit{s}}_{\mathrm{5.7}}$,0.20,0.25)}
${\mathit{W}}_2$	{(${\mathit{s}}_{\mathrm{4.35}}$,0.59,0.62),(${\mathit{s}}_{\mathrm{4.55}}$,0.50,0.54)}	{(${\mathit{s}}_{\mathrm{4.25}}$,0.30,0.35),(${\mathit{s}}_{\mathrm{4.5}}$,0.39,0.44)}	{(${\mathit{s}}_{\mathrm{4.2}}$,0.44,0.49),(${\mathit{s}}_{\mathrm{4.4}}$,0.43,0.48)}	{(${\mathit{s}}_{\mathrm{5.25}}$,0.40,0.49),(${\mathit{s}}_{\mathrm{5.5}}$,0.31,0.42)}
${\mathit{W}}_3$	{(${\mathit{s}}_{\mathrm{5.55}}$,0.67),(${\mathit{s}}_{\mathrm{5.8}}$,0.63)}	{(${\mathit{s}}_{\mathrm{5.2}}$,0.59),(${\mathit{s}}_{\mathrm{5.5}}$,0.44)}	{(${\mathit{s}}_{\mathrm{4.4}}$,0.55,0.57)}	{(${\mathit{s}}_{\mathrm{5.05}}$,0.35,0.46),(${\mathit{s}}_{\mathrm{5.3}}$,0.37,0.48)}
${\mathit{W}}_4$	{(${\mathit{s}}_{\mathrm{4.05}}$,0.64,0.68),(${\mathit{s}}_{\mathrm{4.35}}$,0.55,0.60)}	{(${\mathit{s}}_5$,0.45,0.52,0.48,0.55)}	{(${\mathit{s}}_{\mathrm{4.6}}$,0.48,0.50)}	{(${\mathit{s}}_{\mathrm{5.8}}$,0.35,0.44)}
${\mathit{W}}_5$	{(${\mathit{s}}_{\mathrm{4.55}}$,0.64,0.68),(${\mathit{s}}_{\mathrm{4.35}}$,0.46,0.54)}	{(${\mathit{s}}_{\mathrm{4.7}}$,0.39,0.41)}	{(${\mathit{s}}_{\mathrm{4.15}}$,0.70,0.72),(${\mathit{s}}_{\mathrm{4.4}}$,0.69,0.71)}	{(${\mathit{s}}_{\mathrm{5.5}}$,0.55,0.65),(${\mathit{s}}_{\mathrm{5.75}}$,0.58,0.68)}
E-learning websites	${\mathit{C}}_5$	${\mathit{C}}_6$	${\mathit{C}}_7$
${\mathit{W}}_1$	{(${\mathit{s}}_{\mathrm{4.25}}$,0.42,0.47),(${\mathit{s}}_{\mathrm{4.5}}$,0.31,0.37)}	{(${\mathit{s}}_{\mathrm{5.85}}$,0.33,0.42),(${\mathit{s}}_{\mathrm{6.05}}$,0.35,0.43)}	{(${\mathit{s}}_5$,0.37,0.47),(${\mathit{s}}_{\mathrm{5.3}}$,0.39,0.49)}
${\mathit{W}}_2$	{(${\mathit{s}}_{\mathrm{4.15}}$,0.30,0.32),(${\mathit{s}}_{\mathrm{4.4}}$,0.34,0.37)}	{(${\mathit{s}}_{\mathrm{5.05}}$,0.28,0.41)}	{(${\mathit{s}}_{\mathrm{5.25}}$,0.35,0.39),(${\mathit{s}}_{\mathrm{5.5}}$,0.39,0.43)}
${\mathit{W}}_3$	{(${\mathit{s}}_{\mathrm{4.45}}$,0.60,0.63),(${\mathit{s}}_{\mathrm{4.75}}$,0.50,0.54)}	{(${\mathit{s}}_{\mathrm{5.7}}$,0.38,0.40),(${\mathit{s}}_{\mathrm{5.95}}$,0.40,0.43)}	{(${\mathit{s}}_{\mathrm{5.4}}$,0.47,0.49),(${\mathit{s}}_{\mathrm{5.65}}$,0.57,0.59)}
${\mathit{W}}_4$	{(${\mathit{s}}_{\mathrm{4.25}}$,0.38,0.47),(${\mathit{s}}_{\mathrm{4.5}}$,0.43,0.51)}	{(${\mathit{s}}_{\mathrm{5.85}}$,0.30,0.34)}	{(${\mathit{s}}_{\mathrm{3.95}}$,0.52),(${\mathit{s}}_{\mathrm{4.25}}$,0.54)}
${\mathit{W}}_5$	{(${\mathit{s}}_{\mathrm{4.35}}$,0.34,0.39),(${\mathit{s}}_{\mathrm{4.6}}$,0.28,0.33)}	{(${\mathit{s}}_{\mathrm{4.8}}$,0.47,0.49),(${\mathit{s}}_{\mathrm{5.1}}$,0.47,0.61)}	{(${\mathit{s}}_{\mathrm{4.4}}$,0.27,0.27),(${\mathit{s}}_{\mathrm{4.65}}$,0.20,0.22)}

Step 7: Using Eq. (15), the relative weight of each criterion to the reference criterion ${\mathit{C}}_r$ is calculated as

$$w_{13}=0.249,w_{23}=0.304,w_{33}=1.000,w_{43}=0.617,$$

$$w_{53}=0.273,w_{63}=0.913,w_{73}=0.597.$$

Step 8: By applying Eq. (13) with the sensitive coefficient $\theta =1$, the dominances between the five e-learning websites with respect to each criterion are computed as follows:

$${\varphi }_2=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill 0.057\hfill & \hfill -1.714\hfill & \hfill -0.782\hfill & \hfill -1.082\hfill \\ \hfill -0.739\hfill & \hfill 0\hfill & \hfill -1.168\hfill & \hfill -0.302\hfill & \hfill -0.102\hfill \\ \hfill 0.132\hfill & \hfill 0.090\hfill & \hfill 0\hfill & \hfill 0.121\hfill & \hfill 0.200\hfill \\ \hfill 0.060\hfill & \hfill 0.023\hfill & \hfill -1.576\hfill & \hfill 0\hfill & \hfill 0.058\hfill \\ \hfill 0.083\hfill & \hfill 0.008\hfill & \hfill -1.433\hfill & \hfill -0.748\hfill & \hfill 0\hfill \end{array}\right)$$

$${\varphi }_3=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill -0.678\hfill & \hfill -0.418\hfill & \hfill -0.306\hfill & \hfill -0.429\hfill \\ \hfill 0.055\hfill & \hfill 0\hfill & \hfill -0.305\hfill & \hfill -0.221\hfill & \hfill -0.307\hfill \\ \hfill 0.106\hfill & \hfill 0.077\hfill & \hfill 0\hfill & \hfill 0.061\hfill & \hfill -0.319\hfill \\ \hfill 0.091\hfill & \hfill 0.056\hfill & \hfill -0.243\hfill & \hfill 0\hfill & \hfill -0.038\hfill \\ \hfill 0.109\hfill & \hfill 0.091\hfill & \hfill 0.081\hfill & \hfill 0.097\hfill & \hfill 0\hfill \end{array}\right)$$

$${\varphi }_3=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill -0.678\hfill & \hfill -0.418\hfill & \hfill -0.306\hfill & \hfill -0.429\hfill \\ \hfill 0.055\hfill & \hfill 0\hfill & \hfill -0.305\hfill & \hfill -0.221\hfill & \hfill -0.307\hfill \\ \hfill 0.106\hfill & \hfill 0.077\hfill & \hfill 0\hfill & \hfill 0.061\hfill & \hfill -0.319\hfill \\ \hfill 0.091\hfill & \hfill 0.056\hfill & \hfill -0.243\hfill & \hfill 0\hfill & \hfill -0.038\hfill \\ \hfill 0.109\hfill & \hfill 0.091\hfill & \hfill 0.081\hfill & \hfill 0.097\hfill & \hfill 0\hfill \end{array}\right)$$

$${\varphi }_4=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill -0.422\hfill & \hfill -0.431\hfill & \hfill -0.596\hfill & \hfill -0.337\hfill \\ \hfill 0.066\hfill & \hfill 0\hfill & \hfill -0.014\hfill & \hfill -0.297\hfill & \hfill -0.472\hfill \\ \hfill 0.067\hfill & \hfill -0.088\hfill & \hfill 0\hfill & \hfill -0.076\hfill & \hfill -0.488\hfill \\ \hfill 0.093\hfill & \hfill 0.046\hfill & \hfill 0.012\hfill & \hfill 0\hfill & \hfill -0.592\hfill \\ \hfill 0.107\hfill & \hfill 0.074\hfill & \hfill 0.076\hfill & \hfill 0.092\hfill & \hfill 0\hfill \end{array}\right)$$

$${\varphi }_5=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill 0.013\hfill & \hfill -0.545\hfill & \hfill -0.170\hfill & \hfill 0.035\hfill \\ \hfill -0.190\hfill & \hfill 0\hfill & \hfill -0.690\hfill & \hfill -0.093\hfill & \hfill -0.147\hfill \\ \hfill 0.038\hfill & \hfill 0.048\hfill & \hfill 0\hfill & \hfill 0.011\hfill & \hfill 0.047\hfill \\ \hfill 0.012\hfill & \hfill 0.006\hfill & \hfill -0.412\hfill & \hfill 0\hfill & \hfill 0.025\hfill \\ \hfill -0.159\hfill & \hfill 0.010\hfill & \hfill -0.674\hfill & \hfill -0.357\hfill & \hfill 0\hfill \end{array}\right)$$

$${\varphi }_6=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill 0.232\hfill & \hfill 0.019\hfill & \hfill 0.068\hfill & \hfill -0.081\hfill \\ \hfill -0.317\hfill & \hfill 0\hfill & \hfill -0.302\hfill & \hfill -0.140\hfill & \hfill -0.332\hfill \\ \hfill -0.081\hfill & \hfill 0.070\hfill & \hfill 0\hfill & \hfill 0.064\hfill & \hfill -0.114\hfill \\ \hfill -0.293\hfill & \hfill 0.032\hfill & \hfill -0.276\hfill & \hfill 0\hfill & \hfill -0.310\hfill \\ \hfill 0.019\hfill & \hfill 0.077\hfill & \hfill 0.026\hfill & \hfill 0.071\hfill & \hfill 0\hfill \end{array}\right)$$

$${\varphi }_7=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill 0.009\hfill & \hfill -0.264\hfill & \hfill 0.098\hfill & \hfill 0.073\hfill \\ \hfill -0.058\hfill & \hfill 0\hfill & \hfill -0.270\hfill & \hfill -0.654\hfill & \hfill 0.065\hfill \\ \hfill 0.040\hfill & \hfill 0.041\hfill & \hfill 0\hfill & \hfill 0.066\hfill & \hfill 0.087\hfill \\ \hfill -1.675\hfill & \hfill 0.099\hfill & \hfill -0.438\hfill & \hfill 0\hfill & \hfill 0.125\hfill \\ \hfill -0.485\hfill & \hfill -0.431\hfill & \hfill -0.578\hfill & \hfill -0.830\hfill & \hfill 0\hfill \end{array}\right)$$

Step 9: By using Eq. (14), the overall dominances among the five e-learning websites are determined as:

$$\delta =\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill -1.238\hfill & \hfill -4.993\hfill & \hfill -1.814\hfill & \hfill -2.039\hfill \\ \hfill -1.170\hfill & \hfill 0\hfill & \hfill -4.372\hfill & \hfill -1.816\hfill & \hfill -1.339\hfill \\ \hfill 0.338\hfill & \hfill 0.340\hfill & \hfill 0\hfill & \hfill 0.349\hfill & \hfill -0.489\hfill \\ \hfill -1.704\hfill & \hfill 0.272\hfill & \hfill -4.549\hfill & \hfill 0\hfill & \hfill -1.140\hfill \\ \hfill -0.312\hfill & \hfill -0.152\hfill & \hfill -3.720\hfill & \hfill -1.649\hfill & \hfill 0\hfill \end{array}\right)$$

Step 10: Based on Eq. (15), the global evaluation values for the five e-learning websites are calculated as:

$$G{E}_1=0.000,G{E}_2=0.126,G{E}_3=1.000,G{E}_4=0.279,$$

$$G{E}_5=0.400$$

According to the decreasing order of the global evaluation values $G{E}_i\left(i=1,2,\dots ,5\right)$, the final ranking of the five e-learning websites is determined as ${\mathit{W}}_3$ $>$ ${\mathit{W}}_5$ $>$ ${\mathit{W}}_4$ $>$ ${\mathit{W}}_2$ $>$ ${\mathit{W}}_1$. So, the best e-learning website for this case study is ${\mathit{W}}_3$, which can be selected for online teaching in the considered university.

5.2. Sensitivity analysis

A sensitivity analysis by changing the weights of experts is performed in this part according to the information given in Table 8. For example, Case 0 shows the original weight values of experts considered in the above case while the other cases show different weight values for possible situations. The ranking results of the five e-learning websites for the considered cases are represented in Fig. 1.

Table 8.

Expert weights regarding the considered cases.

Experts	Case 0	Case 1	Case 2	Case 3	Case 4
${\mathit{E}}_1$	0.25	0.70	0.10	0.10	0.10
${\mathit{E}}_2$	0.20	0.10	0.70	0.10	0.10
${\mathit{E}}_3$	0.30	0.10	0.10	0.70	0.10
${\mathit{E}}_4$	0.25	0.10	0.10	0.10	0.70

Fig. 1.

Sensitivity analysis with different expert weights.

It can be clearly seen from Fig. 2 that the ranking orders of e-learning websites are distinctly changed as the weights of experts are varied although the best e-learning website is not influenced. For example, ${\mathit{W}}_1$ is the third when the weight of ${\mathit{E}}_1$ is relatively high. The performance ranking of ${\mathit{W}}_1$ is getting rise to second place when the importance of ${\mathit{E}}_2$ is increased to 0.70. As the weights of ${\mathit{E}}_3$ and ${\mathit{E}}_4$ are relatively high, the performance ranking of ${\mathit{W}}_1$ is turned into the fifth. Hence, proper determination of relative weights of experts plays an essential role in the process of e-learning website evaluation. In general, the weights of experts can be determined by using point allocation, direct rating, AHP, or Delphi method together with experts’ domain knowledge. If there is no sufficient reason or evidence to show the difference among experts in their judgment qualities, the experts should be assigned an equal weight.

Fig. 2.

Ranking comparison by different methods.

5.3. Comparative analysis

In this section, a comparative analysis with other e-learning website selection methods is conducted to show the effectiveness and advantages of the presented LHF-TODIM model. The above illustrative example is solved by the fuzzy VIKOR [39], the fuzzy AD [8], and the fuzzy COPRAS [7] methods. Fig. 2 displays the ranking orders of the five e-learning websites derived by these approaches. From the figure, it can be observed that the best e-learning website (i.e., ${\mathit{W}}_3$) determined by all the four methods are exactly the same; the fuzzy VIKOR, the fuzzy AD and the proposed LHF-TODIM give the lowest rank to e-learning website ${\mathit{W}}_1$. These verify the effectiveness of our proposed e-learning website selection framework.

On the other hand, there are some differences between the ranking results obtained by the proposed method and those with the fuzzy VIKOR (for ${\mathit{W}}_2$ and ${\mathit{W}}_4$), the fuzzy AD (for ${\mathit{W}}_4$ and ${\mathit{W}}_5$) and the fuzzy COPRAS (for ${\mathit{W}}_1$ and ${\mathit{W}}_2$). The main reasons for these differences can be explained as follows: First, the three comparison methods use fuzzy set theory to handle the ambiguity evaluation information of e-learning websites. However, the fuzzy sets, using only one linguistic term, cannot deal with the qualitative situations in which people hesitate about several possible linguistic terms. Thus, the compared methods are not able to reflect the hesitancy and inconsistency of experts, and may cause the loss of expert’s evaluation information. Second, the algorithms adopted to determine the priority ranking of e-learning websites in the four listed approaches are different, which are VIKOR, AD, COPRAS and TODIM, respectively. All the three compared methods are based on the assumption that experts are completely rational in the e-learning website evaluation process, which may produce biased ranking results in real applications.

From the above comparative analysis, it can be concluded that the ranking orders of the e-learning websites derived through the presented approach are reasonable and credible although providing little more computational complexity. To further verify the proposed LHF-TODIM model, we gathered managers in educational design and the university to check the results determined in this study. According to the domain experts, the proposed integrated approach is highly suitable for the considered e-learning website selection problem and can efficiently yield the best website for network teaching. Compared with the existing e-learning website selection methods, the LHF-TODIM model proposed in this paper has the following advantages: (1) By using the LHFSs, the proposed approach can represent experts’ qualitative judgments and reflect their hesitancy and inconsistency in the e-learning website evaluation process. This allows experts to express their qualitative information more flexibly. (2) The presented model extends the BWM for computing the weights of evaluation criteria, which requires less experts’ judgments and yields more consistency of comparisons. (3) Based on the TODIM algorithm, the proposed approach can consider the decision maker’s bounded rationality during e-learning website selection process. As a result, more suitable ranking result of e-learning websites can be obtained according to a decision maker’s actual needs and behavior preference.

6. Conclusions

This research developed a new LHF-TODIM model to evaluate, rank, and select e-learning websites for network teaching. To express experts’ qualitative preferences and reflect their hesitancy and uncertainty, the LHFSs are used for dealing with experts’ evaluation information on the candidate e-learning websites. An extension of the BWM is introduced to calculate the weights of evaluation criteria. A modified TODIM approach is put forward to rank e-learning websites and choose the most appropriate one for providing services. Finally, an actual example of e-learning website selection is implemented to demonstrate the effectiveness and superiority of the presented LHF-TODIM approach. The results displayed that the new framework can not only better reflect the hesitancy and fuzziness of experts’ evaluations, but also obtain promising ranking results of alternative e-learning websites.

In future studies, the following research directions are recommended. First, the proposed model is limited to a small-scale expert group. In the future, it is suggested to introduce a new approach to address e-learning website selection problems in the large group environment. Second, the relative weights of experts are assumed to be known in the proposed model and given directly in the case study. So, future research can explore a method to objectively get expert weights on the basis of e-learning website evaluation information. In addition, although the proposed model based on LHFSs and TODIM method can obtain the optimal e-learning website effectively, it increases computational complexity. Therefore, a computer-based program system can be developed for further work to facilitate the implementation of the proposed approach for e-learning website selection.

CRediT authorship contribution statement

Jia-Wei Gong: Data curation, Writing - original draft preparation. Hu-Chen Liu: Visualization, Supervision. Xiao-Yue You: Conceptualization, Methodology. Linsen Yin: Writing - review and editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.