Doctors ranking through heterogeneous information: The new score functions considering patients’ emotional intensity

Abstract

With the popularity of the Internet and the growing complexity of COVID-19, more and more patients tend to consult doctors online. With the difficulty of doctor selection caused by a massive amount of information, this study proposes a hybrid multi-criteria decision-making framework, which can model patients’ emotional intensity through heterogeneous information and rank doctors. Firstly, online reviews (ORs) are transformed into probabilistic linguistic term sets through sentiment analysis. Then, new score functions are proposed considering the nonlinear influence of doctors’ information and the patients’ negative bias toward ORs. Next, a method of weight determination combining the Term Frequency Inverse Document Frequency and the Decision-making Trial and Evaluation Laboratory method is proposed. Finally, the proposed score functions are applied to the Combined Compromise Solution (CoCoSo) method to aggregate information and rank doctors. The proposed method is verified in a case study on haodf.com. The results show that considering the emotional intensity of heterogeneous information will make the recommendations more realistic. Comparative analysis and sensitivity analysis are further performed to illustrate the availability and effectiveness of the proposed method.

1. Introduction

With the rapid development of information technology and people’s pursuit of healthy lives, Internet medicine has ushered in the spring (Arora et al., 2022, Lu, Wu, Liu, Li, & Zhang, 2017, Xiao et al., 2014). As an important online medical model in the Internet medical platform, online consultation has become an important way of communication between patients and doctors. More and more patients are choosing to consult online through medical websites to save time and eliminate geographical restrictions. Online consultations through medical websites are different from traditional offline consultations (Ba & Wang, 2013). Firstly, patients can get detailed information about doctors through websites, such as doctors’ service attitudes, doctors’ skills and so on, not just the hospitals’ standing and the physicians’ positions (Cao et al., 2017; Zhang et al., 2017). Secondly, there are a lot of online reviews (ORs) on medical websites which have an important impact on patients’ choices towards doctors (Hong et al., 2019).

At present, there are two forms of online consultation. First, the platform assigns doctors to patients after they describe their health conditions. Second, patients choose doctors themselves. Both of these forms have certain defects. In the first form, patients may not be satisfied with the doctor they are assigned because their needs are not considered. Also, platform allocation of doctors needs to spend a lot of labor costs and time costs. In the second form, due to a large number of doctors on the platform, patients cannot select suitable doctors in a short time. The doctor information involved in the Internet medical platform is multi-source and heterogeneous, which leads to a variety of data types: real values, symbolic values, interval values and semantic values. For example, the number of online service patients is the real value. The professional title is the symbolic value. The price is the interval value. And ORs are the semantic values. As more patients opt for online consultations, the amount of information is increasing. It’s hard for patients to select a doctor through overloaded information (Frias et al., 2008). Besides, patients cannot focus on all the criteria due to their bounded rational behavior (Simon, 1947). It has become necessary to provide a way to assist patients in analyzing heterogeneous information and improving their decisions on doctors’ selection.

There are several ways to help users get important information from online information quickly and make decisions, such as extracting the ORs that are considered most useful, ranking the products or services through online information, identifying user preferences and providing similar online information, etc. Among these, the use of online information to rank products or services is regarded as a comprehensive decision-making method by many scholars. This method starts by extracting features in ORs and determining the weights of the features. Sentiment in the ORs is then analyzed using sentiment analysis (SA). Finally, the product or service is ranked using the available information. Thus, multiple criteria are involved in feature extraction and the process of balancing the conflicts between criteria in product or service ranking to obtain the optimal ranking can be considered a multi-criteria decision-making (MCDM) problem.

Nowadays, the doctor recommendation model has become an increasingly heated research topic. Some scholars have applied the above methods to recommend doctors (e.g., (Hu et al., 2018; Li et al., 2020)). However, existing studies of the doctor recommendation based on ORs still have some limitations as follows.

(1) When considering the influence of information on patients, it is often assumed that it is linear. But the influence of changes in the information on patients should be considered non-linear.

(2) When identifying the sentimental orientation of ORs through SA, only the sentimental orientation is considered without considering the negative bias of patients towards ORs, that is, patients will have stronger emotional intensity when reading negative ORs.

(3) When determining the weights of criteria, the interactions among criteria are ignored. There are usually overlapping meanings among criteria, and ignoring interactions will lead to results that deviate from reality.

This paper endeavors to propose a method to address the above limitations. The main objective of this research is to construct a hybrid decision model based on the heterogeneous information from Internet medical platforms to rank and recommend suitable online doctors to patients by using a combination of SA and MCDM techniques. The proposed method considers patients’ concerns, the interactions among criteria and the emotional intensity of patients towards information.

The remainder of this article is structured as follows. Section 2 briefly reviews related literature about MCDM methods based on OR. In Section 3, some of the main concepts are reviewed. In Section 4, the step-by-step framework of the doctor ranking method is described. Section 5 describes a case study for the ranking of six doctors on haodf.com and the comparative analysis and sensitivity analysis are performed. Section 6 discusses the advantages, limitations and managerial implications of the proposed model. Finally, some conclusions of the research are drawn in Section 7.

2. Related works

In this section, the relevant studies of products and services ranking based on online information are briefly introduced. Also, the research gaps of previous studies and the main contributions of this study are listed below.

The development of the information age has changed people’s traditional decision-making mode(Kohli et al., 2004). People will check relevant information, such as details, historical reviews, etc., before choosing products or services. There are a lot of studies on the ranking based on online information such as ORs. Many methods of opinion mining and SA have been proposed to extract useful information from doctors’ information and ORs, especially to extract features and opinions. To describe the results analyzed from ORs, researchers investigated the description of ORs. And intuitionistic fuzzy sets (IFSs), hesitant fuzzy sets (HFSs) and probabilistic linguistic term sets (PLTSs) are most frequently used to describe ORs (e.g., (Hu et al., 2018; Li et al., 2020; Peng et al., 2022)). Ranking based on ORs needs to be evaluated from multiple criteria, so the ranking problems can be defined as the MCDM problem (Peng et al., 2022). Based on ORs, data-driven MCDM becomes an important research topic (Fan et al., 2020). The steps to rank doctors combining the MCDM and SA method are as follows: (1) Identifying and extracting the evaluation criteria. (2) Identifying the sentimental orientation of ORs through the SA method. (3) Determining the weights of criteria. (4) Ranking the doctors. Table 1 shows some studies using ORs in conjunction with MCDM methods for the ranking. As can be seen from Table 1, although these studies have a significant impact on ranking based on ORs, the research gaps are as follows:

Table 1.

Review of studies using MCDM methods through ORs.

Research field	Authors’ names and year	Aggregated form	Ranking method	Features extraction method	Criteria weighting method	Interactions among criteria	The nonlinear influence of information	The negative bias towards ORs
Healthcare	(Hu et al., 2018)	IFSs	VIKOR	Term frequency	TF-IDF	Not considered	Not considered	Not considered
	(Li et al., 2020)	PLTSs	MULTIMOORA	Term frequency	Projection pursuit method	Not considered	Not considered	Not considered
	(Chen et al., 2022)	PLTSs	TOPSIS	Term frequency	TF-IDF	Not considered	Not considered	Not considered
	The proposed method*	Heterogeneous information	CoCoSo	Term frequency and semantic similarity	TF-IDF and DEMATEL	Considered	Considered	Considered
Tourism	(Çalı & Balaman, 2019)	IFSs	Combination of IF-ELECTRE and VIKOR	Term frequency	Entropy method	Not considered	Not considered	Not considered
	(Yu et al., 2018)	Linguistic distribution assessments	VIKOR	Specific criteria	Based on the idea of a prioritized average operator	Not considered	Not considered	Not considered
Products	(Zhang et al., 2020)	HFSs	Extended TODIM	Determined by experts	Using attention-degree	Considered	Not considered	Not considered
	(Wu & Liao, 2021)	PLTSs	Based on their standardized utilities	Specific criteria	Same importance	Not considered	Considered	Not considered
	(Dahooie et al., 2021)	IFSs	IF-MULTIMOORA	High Adjective Count algorithm	IF-IDOCRIW	Not considered	Not considered	Not considered

(1) The influence of information such as price is always considered to be linear, but this is a static, mechanical understanding of the influence. This information can be regarded as the stimulus, which will stimulate users to produce emotional intensity. For example, suppose that the price of online consultation with a doctor is between $5-$20. When the price is between $5-$10, patients will think it is cheap. When the price is between $10-$20, patients will think it is expensive. It’s easy to see that emotional intensity is not linear. Considering psychophysics, there is a functional relationship between the users’ emotional intensity and the information (Baird, 1997; Ehrenstein & Ehrenstein, 1999). To make more accurate results of the ranking, this nonlinear influence of information should take into account.

(2) The users’ negative bias toward ORs is ignored. Sentimental orientation in ORs is mostly calculated by SA (He et al., 2022). But this approach ignores the bounded rationality and the users’ emotional intensity toward different ORs. Because users cannot obtain complete information, and also cannot judge the authenticity of the information (Frias et al., 2008; Purnawirawan et al., 2012; Song et al., 2022), they are more likely to recognize the authenticity of negative ORs (Chen & Lurie, 2013). Online reviews are asymmetric(Sen & Lerman, 2007) and positive ORs have less influence on users’ purchase intention than negative ORs (Cui et al., 2012; Wang et al., 2014). Users will become risk aversion so they will have negative bias toward comments (Mudambi & Schuff, 2010). That is, compared with positive comments, users think negative comments are more valuable and they will have greater emotional intensity when facing negative comments. However, the negative bias towards ORs has received little attention.

(3) Most studies assume that the criteria are independent of each other (Gölcük & Baykasoğlu, 2016). However, complementarity, redundancy and independence exist among the criteria (Tao & You, 2022). For instance, considering a potential patient who wants to select a doctor for online consultation, mainly based on leechcraft, attitude, general response speed and price. The doctor's price will be affected by the other three criteria (Chiu et al., 2021). In other words, doctors with better leechcraft, attitude and general response speed always have a higher price. Therefore, it is necessary to take the interactions among criteria into account. However, the weights of criteria in the above research are usually according to term frequency or objective weighting methods. Although these weight determination methods are mature, the interactions among criteria are not considered.

Therefore, this paper intends to solve the above gaps and proposes a novel and effective doctor ranking model driven by heterogeneous information. A useful MCDM method (Yazdani et al., 2019)) is used to construct a research framework that can rank doctors. To sum up, the main contributions of this study can be summarized as described below.

(1) Considering the nonlinear influence of the information, the quantification method of the nonlinear influence of information is proposed based on Stevens’ law.

(2) Considering patients’ negative bias towards ORs, a new score function is proposed based on the logarithmic proportional law of emotional intensity.

(3) A comprehensive weight determination method is proposed. This method considers not only the importance of criteria but also the interactions among criteria and overcomes the limitations of the existing weight determination methods.

(4) A novel doctor ranking model considering patients’ emotional intensity through heterogeneous information is proposed. Comparative analysis and sensitivity analysis are made to illustrate the availability and effectiveness of the proposed model.

The doctor ranking method proposed in this paper intersects decision theory and psychophysics to solve the above-mentioned gaps based on a comprehensive weight determination method and two newly proposed score functions.

3. Preliminaries

In this section, several preliminary concepts are introduced for subsequent studies, which are PLTS, SA and the weights determined methods.

3.1. Probabilistic linguistic term set

In the real world, many decisions are accompanied by uncertainty because they are made based on the implicit knowledge of the decision makers (Bellman & Zadeh, 1970). Therefore, Zadeh (1996) proposed the theory of fuzzy sets. Pang et al. (2016) put forward a novel concept called PLTS by extending the theory of fuzzy sets. PLTS can comprehensively depict sentimental information, uncertainty and fuzziness in detail (Song et al., 2020). Besides, PLTS is excellent in aggregating different persons’ ideas denoted by linguistic terms (Liao et al., 2020). Therefore, PLTSs can be denoted as follows:

Definition. 1. (Pang et al., 2016). Let $S{\text{= { s}}_{-\tau }{\text{,s}}_{-\left(\tau -1\right)}\text{,}...{\text{,s}}_0{\text{,s}}_1\text{,}...{\text{,s}}_{\tau }\text{}}$ be a linguistic term set (LTS), then the PLTS can be defined as:

$$L\left(p\right)=\left\{L^{\left(k\right)}\left(p^{\left(k\right)}\right)\right|L^{\left(k\right)}\in S,p^{\left(k\right)}\ge 0,\text{k}=1,2,...,\#L\left(p\right),\sum _{k=1}^{\#L\left(p\right)}{p}^{\left(k\right)}\le 1\}$$

where $L^{\left(k\right)}\left(p^{\left(k\right)}\right)$ is the k _th linguistic term $L^{\left(k\right)}$ associated with probability $p^{\left(k\right)}$, and $\#L\left(p\right)$ is the number of all different linguistic terms in L(p).

Remark. 1. (Pang et al., 2016) The normalizing process of PLTSs can be shown in the following two steps:

• (1)Normalization of the probability. If ${\sum }_{k=1}^{\#L\left(p\right)}{p}^{\left(k\right)}<1$, then the normalized PLTS(NPLTS) can be denoted as $\overline{L}\left(p\right)=\left\{L^{\left(k\right)}\left({\overline{p}}^{\left(k\right)}\right)\right|L^{\left(k\right)}\in S,k=1,2,...,\#L\left(p\right),{\overline{p}}^{\left(k\right)}>0,{\sum }_{k=1}^{\#L\left(p\right)}{\overline{p}}^{\left(k\right)}=1\}$, where ${\overline{p}}^{\left(k\right)}=\frac{p^{\left(k\right)}}{{\sum }_{k=1}^{\#L\left(p\right)}{p}^{\left(k\right)}}$ ($k=1,2,...,\#L\left(p\right)$).

(2) Normalization of the granularity. Suppose $L_1\left(p\right)$ and $L_2\left(p\right)$ are two PLTSs, $\#L_1\left(p\right)$ and $\#L_2\left(p\right)$ are the corresponding numbers of linguistic terms. If $\#L_1\left(p\right)\ne \#L_2\left(p\right)$, then add $\left|\#L_1\left(p\right)-\#L_2\left(p\right)\right|$ linguistic terms to $L_i\left(p\right)$, where $\#L_i\left(p\right)=\min (\#L_1\left(p\right),\#L_2\left(p\right))$, ($i=1,2$). The added linguistic terms are the smallest ones with zero probability in $L_i\left(p\right)$.

3.2. Sentiment analysis (SA)

The problem of information overload related to ORs on Internet medical platforms has increased the need for automated systems to extract and organize information. The development and maturity of SA gradually solved this problem (Çalı & Balaman, 2019). Three main classification levels can be found in SA, which are document-level, sentence-level and aspect-level SA (Medhat et al., 2014). SA techniques can be distributed into two general categories, namely, machine-learning-based SA techniques and lexicon-based SA techniques (Madhoushi et al., 2015). The classification level in this paper is sentence-level, and the SA technique is based on the lexicon. The details of the aforementioned methods are not intended to describe in this study to save the space of this research. Details can be found in the aforementioned references.

3.3. Weight determination method

Weight determination is important in the MCDM process, so various methods have been proposed. The following are two methods for calculating weights.

3.3.1. TF-IDF algorithm

The TF-IDF algorithm is most widely used in solving the problem of calculating the weights of criteria for automatic text classification (Liu et al., 2018; Wu et al., 2008). TF-IDF is a combination of Term Frequency (TF) and Inverse Document Frequency (IDF). TF is used to measure how many times a term is present in a document (Hakim et al., 2014). IDF assigns lower weight to frequent words and assigns greater weight to infrequent words. TF-IDF is the multiplication of TF and IDF. The value of TF-IDF will be low when the word appears infrequently in a text or most of the texts. In conclusion, the TF-IDF algorithm is a convenient tool to measure the importance of a word-to-text in a large set of texts. Details about TF-IDF can be found in the aforementioned references. Suppose there are n criteria and their corresponding TF-IDF values can be denoted as $\mathit{tf}-id{f}_j(j=1,2,...,n)$. Then the weight of $j_{\mathit{th}}$ criterion can be calculated by

$${\omega }_j=\frac{\mathit{tf}-id{f}_j}{{\sum }_{j=1}^n\mathit{tf}-id{f}_j}$$ (1)

3.3.2. DEMATEL method

The DEMATEL method was proposed to capture complex relationships and interactions among various criteria (Gabus & Fontela, 1972). This method does not need voluminous information and can easily propose the most important criterion which affects other criteria (Govindan et al., 2015). The steps for calculating the weights of criteria through the DEMATEL method are as follows:

Step 1: Develop a matrix A of direct relations. Experts generate a direct relations matrix $A_{n\times n}$ (the number of criteria is n) by making pairwise comparisons between criteria. Each element $a_{\mathit{ij}}$ in the matrix $A_{n\times n}$ indicates the direct effect of criterion i to j. Influence degree is measured on a 4-level scale ranging from 0 to 3. 0 indicates no influence, 1 indicates moderate influence, 2 indicates strong influence, and 3 indicates very strong influence.

Step 2: Normalize matrix A. The normalized matrix $X_{n\times n}$ is achieved by

$$X=kA$$ (2)

where $k=\frac{1}{\underset{1\le i\le n}{\max }{\sum }_{j=1}^n{a}_{\mathit{ij}}}$. After normalization, each element in the matrix $X_{n\times n}$ ranges from 0 to 1.

Step 3: Compute the total relation matrix T. The total relation matrix can be obtained by

$$T=X{(I-X)}^{-1}$$ (3)

where I denotes the identity matrix. Each element $t_{\mathit{ij}}$ in the matrix $T_{n\times n}$ indicates the indirect influences of criterion i to j.

Step 4: Determine the sums of rows and columns of matrix T. In the matrix $T_{n\times n}$, the sum of rows and sum of columns are represented by vectors R and C, which can be computed by

$$R={\left[\sum _{j=1}^n{t}_{\mathit{ij}}\right]}_{n\times 1}={\left[t_i\right]}_{n\times 1},i=1,2,...,n;$$ (4)

$$C={\left[\sum _{i=1}^n{t}_{\mathit{ij}}\right]}_{1\times n}={\left[t_j\right]}_{1\times n},j=1,2,...,n.$$ (5)

Step 5: Calculate the weights of the criteria.$R_j$ represents the $j_{\mathit{th}}$ element of the vector R, same as $C_j$. The degree of importance of the $j_{\mathit{th}}$ criterion can be calculated by

$${\eta }_j=\sqrt{{(R_j+C_j)}^2+{(R_j-C_j)}^2},j=1,2,...,n.$$ (6)

Then the weight of the $j_{\mathit{th}}$ criterion can be calculated by

$${\omega }_j=\frac{{\eta }_j}{{\sum }_{j=1}^n{\eta }_j},j=1,2,...,n.$$ (7)

4. Doctor ranking model considering patients’ emotional intensity

4.1. Decision support framework

In this section, a decision support framework of doctors ranking based on SA, PLTS and MCDM is introduced as follows.

Suppose a patient intends to select the most suitable doctor from $\mathit{\text{m}}$ doctors on the Internet medical platforms. The $\mathit{\text{m}}$ doctors denoted as $\left\{D_i\right|i=1,2,...,m\}$. Due to a large number of heterogeneous information and the limitation of the patient’s professional knowledge, it is difficult for this patient to select the most suitable doctor.

In our study, SA, PLTSs and the CoCoSo method are used to construct a research framework that can rank doctors (Fig. 1 ). It uses heterogeneous information to recommend doctors. The research framework consists of four phases and the phases of the decision-making method are as follows:

Fig. 1.

Procedure of proposed framework.

Phase 1. Online information analysis. In the first phase, heterogeneous information of selected doctors is crawled and processed. Then, the evaluation criteria are determined by high-frequency words in ORs and key information of doctors. Next, a mixed decision matrix is constructed as decision information for subsequent phases.

Phase 2. Weights determination. In the second phase, considering the importance of decision information is different, comprehensive weights are given to criteria determined in the previous phase.

Phase 3. Scores computation. In the third phase, considering the nonlinear influence of doctors’ information and patients’ negative bias towards ORs, the scores under each criterion are calculated.

Phase 4. MCDM method usage. In the fourth phase, the modified CoCoSo method is applied to rank doctors based on the above information and provide patients with the doctor ranking results.

The steps of each phase are described in detail below.

4.2. Phase 1: Online information analysis

In this phase, online information is analyzed and a decision matrix is constructed composed of heterogeneous information.

4.2.1. Doctors’ information and ORs crawling

This section is for information collection. First of all, an Internet medical platform is selected and Octopus collector is used to crawl the information of m doctors under a certain disease. Octopus collector can expeditiously complete large-scale collection and help reduce the cost of information acquisition. When selecting a doctor, patients will consider the doctor’s service quality (including general response speed and the number of online service patients), price and professional title (Cao et al., 2017; Wu et al., 2021). Therefore, except for ORs, this study also focuses on the following aspects: the general response speed, professional title, price and the number of online service patients. Assume that I is a set of different types of information, which I can be presented as I = Iⁱ ∪ I^s ∪ I^f ∪ I^r. Iⁱ, ${\mathit{\text{I}}}^{\mathit{\text{s}}}$, I^f and I^r represent the set of interval values, semantic values, symbolic values and real values, respectively.

4.2.2. Data preprocessing

After crawling the online information from selected doctors, this section aims to preprocess each data type in different ways.

For doctors’ information, some of them are represented by numbers, which can be represented as real values or interval values. And some of them are expressed in the form of words, such as the professional title and the general response speed. To facilitate the calculation, this research transforms this information into symbolic values. The processing method of the professional title is introduced first. In China, medical institutions have established a professional title promotion evaluation system to classify doctors into residents, attending physicians, associate chief physicians, and chief physicians. Considering that patients seldom select lower-ranking doctors, this study only considers doctors with professional titles of chief physician and associate chief physician. For example, if the professional title of doctor A is chief physician, it will be transformed into 2. And if the professional title of doctor B is associate chief physician, it will be transformed into 1. Then the processing method of the general response speed is introduced. The general response speed of doctors can be divided into very fast, fast, medium, slow and very slow. Considering that patients seldom select low-speed doctors, this study only considers doctors whose general response speed is very fast and fast. For example, if the general response speed of doctor C is very fast, it will be transformed into 2. And if the general response speed of doctor D is fast, it will be transformed into 1.

For ORs, which can be represented as semantic values, the following steps are used to preprocess them in our study. To begin with, ORs that are not concerned with any features should be removed, such as figures, emojis, etc. After that, remove the stop words. Stop words are the words that frequently appear in the text but have insignificant information (Jiang et al., 2009). The next step is tokenization and POS (part of speech) tagging. Tokenization can divide the sentence into words or terms (Klein et al., 2017). POS tagging tags words based on the grammatical context of the word in the sentence and splits the words into nouns, verbs, adjectives, etc (Ratnaparkhi, 1996). Note that, this paper uses the Language Technology Platform (LTP) to perform tokenization and POS tagging.

4.2.3. Evaluation criteria determination

In this section, this study aims to determine the evaluation criteria from the crawled information. The criteria are divided into two parts, one part of the criteria is extracted from ORs, and the other part is extracted from the doctors’ information. The pseudocode of the algorithm for extracting the criteria keywords set using textual analysis of online reviews is shown in Algorithm 1.

Algorithm 1: Extract the criteria keywords set using textual analysis of online reviews.
Input: The online reviews set O = {O1, O2,…, Or} Output: The criteria keywords set Ci = {word1, word 2,…, word I}
1	foreachreview in Odo
2	text preprocess to review
3	returnwords
	end foreach
4	Keywords ← Obtained from TF-IDF algorithm based on words
5	forword in Keywordsdo
6	apply the Baidu similarity API
7	return The similarity set THi = {THi1, THi2,…, THip}
8	$T{H}_w$ ← The threshold of semantic similarity
9	ifsimilarity > $T{H}_w$then
10	classify word into Ci
	end if
	end for
11	returnCi = {word1, word 2,…, word I}

The first part of the criteria is the integration of high-frequency words extracted from ORs. Firstly, the TF-IDF algorithm is used to extract high-frequency words. Then the high-frequency words with similar meanings are integrated into a group to form an evaluation criterion. To ensure that the meanings of high-frequency words under an evaluation criterion are similar, the semantic similarity between two high-frequency words under the same criterion can be calculated through Baidu AI Platform. Set the threshold of semantic similarity as TH_w and the semantic similarity between two high-frequency words i and j as TH_ij. If TH_ij ≥ TH_w, it indicates the meaning of the words i and j are similar, which can be integrated into an evaluation criterion. If TH_ij＜TH_w, it indicates the meaning of the words i and j are not similar and cannot be integrated into an evaluation criterion. Assume that TH_w is equal to 0.1. To be specific, Example 1 is illustrated as follows:

Example 1

Assume that the three high-frequency words extracted from ORs are responsible, responsibility and clear. Their semantic similarity is shown inTable 2. It is easy to find that the semantic similarity between responsible and responsibility exceeds the threshold. So, they can be grouped to form an evaluation criterion. The similarity between clear and the other two words does not exceed the threshold, so they cannot be integrated into an evaluation criterion.

Table 2.

Semantic similarities among three high-frequency words.

semantic similarity	clear	responsible	responsibility
clear	1	0.080	0.091
responsible	\	1	0.432
responsibility	\	\	1

The other part of the criteria is extracted from doctors’ information. According to Subsection 4.2.1, some major information that patients focus on is crawled. Since patients will browse this information when selecting doctors, this information can be used as criteria to evaluate doctors.

Suppose there are n criteria determined after the above procedures. Suppose there are I criteria extracted from ORs, denoted as $\left\{c_j\right|j=1,2,...,I\}$. And there are n-I criteria extracted from doctors’ information, denoted as $\left\{c_j\right|j=I+1,I+2,...,n\}$.

4.2.4. Information transformation

The ORs and the information of doctors on the Internet medical platforms can well reflect the doctors’ multicriteria performance. The PLTS can be a reasonable expression tool due to its advantages in handling linguistic information (Pang et al., 2016). In this section, PLTSs are introduced to describe semantic values.

ORs are transformed into PLTSs through SA. Firstly, the sentences with high-frequency words under each criterion are selected. Then, lexicon-based SA techniques are used. Finally, the emotional scores of the sentences with high-frequency words under each criterion are integrated and converted into PLTSs. For example, screen out the sentences that contain high-frequency words under the criterion “leechcraft” from a doctor’s ORs. 100 sentences contain these high-frequency words. According to SA results of these 100 sentences, 5 sentences scored −1, 80 sentences scored 1, and 15 sentenced scored 2. Then $\{(s_{-1},0.05),(s_1,0.8),(s_2,0.15)\}$ can be used to evaluate the performance of this doctor under the criterion of leechcraft.

The PLTS $L_s^{\mathit{ij}}\left(p\right)=\{L^{\left(k\right)}\left(p^{\left(k\right)}\right),k=1,2,...,\#L\left(p\right),{\sum }_{k=1}^{\#L\left(p\right)}{p}^{\left(k\right)}\le 1\}$ represents the performance of the doctor $D_i$ under the criterion $c_j,j=1,2,...,I$. And $L_s^{\mathit{ij}}\left(p\right)\in I^s$. Let ${\mathit{\text{v}}}_{\mathit{\text{i,j}}}$ be the value of the doctor $D_i$ under the criterion$c_j,j=I+1,I+2,...,n$. And $v_{i,j}\in I^r\cup I^f\cup I^i$. Then the mixed decision matrix $X_{m\times n}$ can be established as follows:

$$X=\left(\begin{array}{ccc}\begin{array}{c}{L}_s^{11}\left(p\right)\\ L_s^{21}\left(p\right)\end{array}& \begin{array}{c}\cdots \\ \dots \end{array}& \begin{array}{c}{L}_s^{1I}\left(p\right)\\ L_s^{2I}\left(p\right)\end{array}\\ ⋮& \ddots & ⋮\\ L_s^{m1}\left(p\right)& \cdots & L_s^{\mathit{mI}}\left(p\right)\end{array}\begin{array}{ccc}\begin{array}{c}{v}_{1,I+1}\\ v_{2,I+1}\end{array}& \begin{array}{c}\dots \\ \cdots \end{array}& \begin{array}{c}{v}_{1,n}\\ v_{2,n}\end{array}\\ ⋮& \ddots & ⋮\\ v_{m,I+1}& \cdots & v_{m,n}\end{array}\right)$$

4.3. Phase 2: Weights determination

According to the evaluation system obtained in the previous phase, the weights determination method is used to obtain the comprehensive weights of criteria in this phase. The weights ${\omega }^1={({\omega }_1^1,{\omega }_2^1,...,{\omega }_n^1)}^T$ are firstly calculated by the attention of patients through ORs. Then the weights ${\omega }^2={({\omega }_1^2,{\omega }_2^2,...,{\omega }_n^2)}^T$ are calculated by considering the interactions among criteria. Finally, the comprehensive weights $\omega ={({\omega }_1,{\omega }_2,...,{\omega }_n)}^T$ are obtained by the multiplication operator. By considering patients’ attention and the interactions among criteria, the final ranking is more consistent with reality.

4.3.1. Weights obtainment by TF-IDF

In this section, the weights of the criteria are obtained by TF-IDF. This study considers two parts of the criteria, which are extracted from ORs and doctors’ information. However, the high-frequency words extracted from ORs can only reflect the attention of patients towards ORs. Therefore, this study assumes that patients think ORs and doctors’ information are equally important (i.e., the sum of weights in two parts are both 0.5). For the criteria extracted from doctors’ information, it is also assumed that patients pay the same attention to them (i.e., the weight of each criterion in this part is equal). According to the above, there are I criteria extracted from ORs, marked as i. Suppose there are J high-frequency words under each evaluation criterion (the number of high-frequency words under each evaluation criterion does not need to be equal), marked as j. The TF-IDF value of high-frequency word j under evaluation criterion i is denoted as $\mathit{tf}-id{f}_j^i$, then the weight of criterion i is:

$${\omega }_i=\frac{{\sum }_{j=1}^J\mathit{tf}-id{f}_j^i}{{\sum }_{i=1}^I{\sum }_{j=1}^J\mathit{tf}-id{f}_j^i},i=1,2,...,I$$ (8)

4.3.2. Weights obtainment by DEMATEL

In this section, the weights of the criteria are obtained by DEMATEL. Select a certain number of experts to score the degree of influence among the criteria. Then weights ${\omega }^2$ can be calculated by the interactions among criteria. The specific calculation steps are shown in Subsection 3.3.2.

4.3.3. Determination of comprehensive weights

The TF-IDF method and the DEMATEL method are combined to calculate the weights. The patients’ attention and interactions among criteria are of the same importance in this research framework. The weights $\omega$ can be fused by the multiplication operator:

$${\omega }_i=\frac{{\prod }_{t=1}^T{\omega }_i^t}{{\sum }_{i=1}^n{\prod }_{t=1}^T{\omega }_i^t},i=1,2,...,n$$ (9)

where ${\omega }_i$ is the combined weight of criterion i. T is the number of weighting methods, $T=2$ in this research.

4.4. Phase 3: Scores computation

In this phase, considering the nonlinear influence of doctors’ information and patients’ negative bias towards ORs, the scores under each criterion are calculated.

Internet medical platforms contain a lot of doctors’ information, as shown in Fig. 2 . As can be seen from the figure, the information is heterogeneous and can be divided into real values (The number of online service patients), symbolic values (Professional title and general response speed), interval values (Price), and semantic values (ORs). According to the existing psychophysical theory, this information can stimulate the patient and then produces emotional intensity. For generality and feasibility, this study looks for patterns of how four types of information approach emotional intensity.

Fig. 2.

The information of one doctor.

In the decision-making process, the psychology of decision makers is one of the important factors that affect the decision results. Many studies try to incorporate risk preferences, expected utility, etc. into decision-making models and have achieved certain results in simulated decision-making. However, they neglect to consider the emotional intensity of decision makers based on the value of objective things. In this section, the novel score functions for different types of information are proposed to consider emotional intensity according to psychophysics. The score functions of real values, symbolic values and interval values are proposed based on the nonlinear influence of information. And the score function of semantic values is proposed based on patients’ negative bias toward ORs.

4.4.1. Score computation based on the nonlinear influence of information

In this section, a novel score function is proposed to reflect the nonlinear influence of information. Through psychophysical experiments, Weber and Fechner (1860) proved that “psychological intensity” is a logarithmic function of “stimulus numbers”, known as Weber-Fechner’s law. Specifically, when the “stimulus numbers” increase in a geometric progression, the “psychological intensity” should increase in an arithmetic progression. In the 1950s, the American psychologist Stevens (1957) proposed that “psychological intensity” does not increase with the logarithm of “stimulus numbers”, known as Stevens’ law. He proposed that “psychological intensity” varies with the power function of “stimulus numbers”, that is, “psychological intensity” is a power function of “stimulus numbers”. The formula is as follows:

$$S=K{I}^n$$ (10)

In the formula, S is “psychological intensity”, K is a constant, I is “stimulus numbers”, and the exponent n varies with different senses. Stevens’ law is widely applied and distributed, such as in science, society, psychology and so on (Adam et al., 1999, Green, 1962; Pauli et al., 2022; Takahashi et al., 2008; Yamanishi et al., 2008). Lootsma (1993) pointed out that Weber-Fechner’s law can be used to drive criteria in MCDM, where the target values of criteria are regarded as “stimulus numbers”. By analogy, Stevens’ law can drive criteria in MCDM. Therefore, the doctors’ information (i.e., the target value of criteria) is regarded as “stimulus numbers”. Using Stevens’ law to calculate the patient’s “psychological intensity” as the patient’s emotional intensity towards doctors’ information. And the emotional intensity can be regarded as the score under the criterion extracted from doctors’ information.

Suppose $r_{i,j}\in I^r$, $\left[r_{i,j}^{-},r_{i,j}^{+}\right]\in I^i$ and $s_{i,j}\in I^f$, then the extended score functions of three types of data are defined as follows:

Definition. 2. Let $K=1$, then the score function of real value $r_{i,j}$ is

$$S\left(r_{i,j}\right)={\left(r_{i,j}\right)}^n$$ (11)

where n can vary with different senses and can reflect the sensitivity of patients to information changes. Also, n is always greater than 0. The different values of n will change the speed of emotional intensity changing with the information of doctors, as shown in Fig. 3 . The function S(r_i,j) is concave when n∈(0,1), which represents diminishing marginal emotional intensity, the function S(r_i,j) is linear when n = 1, which represents unchanging marginal emotional intensity, the function S(r_i,j) is convex when n greater than 1, which represents increasing marginal emotional intensity.

Fig. 3.

The change of emotional intensity at different values of n.

Definition. 3. Let $K=1$, then the score function of interval value $\left[r_{i,j}^{-},r_{i,j}^{+}\right]$ is

$$S\left(r_{i,j}^{\text{int}erval}\right)={\left(\frac{r_{i,j}^{-}+r_{i,j}^{+}}{2}\right)}^n$$ (12)

where n varies with different senses and can reflect the sensitivity of patients to information changes. Also, n is always greater than 0. The midpoint of the interval value is used to represent the entire interval. The change in emotion intensity caused by interval values is similar to Fig. 3.

Definition. 4. Let $K=1$, then the score function of symbolic value $f_{i,j}$ is

$$S\left(f_{i,j}\right)={\left(f_{i,j}\right)}^n$$ (13)

where n varies with different senses and can reflect the sensitivity of patients to information changes. Also, n is always greater than 0. The score function of symbolic values is similar to the score function of real values. The difference is that the symbolic value is limited while the real value is arbitrary. The change in emotion intensity caused by symbolic values is similar to Fig. 3.

After calculating these scores, the normalization of values is accomplished based on the compromise normalization equation.

$$s_{i,j}=\frac{S_{i,j}-\underset{i}{\min }{S}_{i,j}}{\underset{i}{\max }{S}_{i,j}-\underset{i}{\min }{S}_{i,j}};forbenefitcriterion,$$ (14)

$$s_{i,j}=\frac{\underset{i}{\max }{S}_{i,j}-S_{i,j}}{\underset{i}{\max }{S}_{i,j}-\underset{i}{\min }{S}_{i,j}};for\cos tcriterion.$$ (15)

4.4.2. Score computation based on patients’ negative bias towards ORs

In this section, a novel score function is proposed to reflect patients’ negative bias towards ORs. There are differences between the ORs of doctors on Internet medical platforms and the ORs of products on shopping websites. Three doctors and three products are randomly selected from haodf.com and jd.com (Fig. 4 ). The specific differences are as follows:

• •The number of ORs of doctors on Internet medical platforms is low. The average number of ORs of doctors is 53. And the number of ORs of products is 1068.
• •Doctors rarely have negative ORs. It can be seen that the selected doctors have almost no negative ORs, but the selected products have a lot of negative ORs.

Fig. 4.

Distribution of ORs on haodf.com and jd.com.

It can be seen that Internet medical platforms have the characteristics of fewer ORs and fewer negative ORs than general products or services. Negative ORs have a stronger impact than positive ones on patients (Pauli et al., 2022). Considering that patients think negative ORs are more valuable than positive ones, the score function of ORs is improved from the perspective of patients’ negative bias towards ORs.

Emotion is a special kind of subjective response. The generation of emotion is the physiological process of the human brain feeling the value of things. Words (i.e., ORs) have different values (Pavlov, 1952). According to Pavlov’s studies, it can be concluded that when patients read ORs, they can feel the value of the reviews and will produce emotional intensity. Also, positive and negative ORs have different values and stimulate the human brain differently (Fiske, 1980). The generation process of emotional intensity is a special physiological process, so the relationship between emotional intensity and the difference in the value rate of things follows Fechner’s law (Dehaene, 2003). The logarithmic proportional law of emotional intensity can be obtained as follow:

$$\mu =K_m\log (1+\mathrm{\Delta }P)$$ (16)

In the formula, $K_m$ is the intensity coefficient, $\mathrm{\Delta }P$ is the difference between the value rate and the median value rate, and $\mu$ is the emotional intensity.

Patients think the values of ORs are different. Also, emotional intensity follows the logarithmic proportional law. In this study, ORs are converted into PLTSs, and the target values represented by PLTSs under the criteria can be regarded as the “value” to drive the criteria of MCDM. Then according to the logarithmic proportional law of emotional intensity, the function of the emotional intensity caused by ORs can be proposed as follows:

Definition. 5. Let $S{\text{= { s}}_{-\tau }{\text{,s}}_{-\left(\tau -1\right)}\text{,}...{\text{,s}}_0{\text{,s}}_1\text{,}...{\text{,s}}_{\tau }\text{}}$ be an LTS determined by ORs and $s_{r^{\left(k\right)}}$ represents the value for a linguistic variable. Then the emotional intensity of $s_{r^{\left(k\right)}}$ is

$$E{I}_{r^{\left(k\right)}}=\log (1+\frac{\tau -r^{\left(k\right)}}{2\tau })$$ (17)

Suppose $\tau =3$, the emotional intensity of LTS is shown in Fig. 5 . It can be seen that negative ORs produce more emotional intensity in patients than positive ORs. Also, this research defines $\tau$ and $-\tau$ as a critical value to distinguish completely satisfied values and unsatisfied values from general values.

Fig. 5.

Emotional intensity under different subscript.

The emotional intensity of patients stimulated by doctors’ ORs can be regarded as the score of ORs. The score function of PLTSs improved based on the logarithmic proportional law of emotional intensity is defined as follows:

Definition. 6. Let $L\left(p\right)=\left\{L^{\left(k\right)}\left(p^{\left(k\right)}\right)\right|L^{\left(k\right)}\in S,p^{\left(k\right)}\ge 0,k=1,2,...,\#L\left(p\right),{\sum }_{k=1}^{\#L\left(p\right)}{p}^{\left(k\right)}=1\}$ be a normalized PLTS transformed from ORs, and $r^{\left(k\right)}$ be the subscript of LT $L^{\left(k\right)}$. Then the improved score function of PLTSs is

$$S\left(L\left(P\right)\right)=\sum _{k=1}^{\#L\left(P\right)}{p}^{\left(k\right)}\log (1+\frac{\tau -r^{\left(k\right)}}{2\tau })$$ (18)

After calculating the scores by the above functions, construct the score matrix S = [s_ij]_{m × n}. s_ij is the score of i_th doctor under j_th criterion.

4.5. Phase 4: MCDM method usage

In this section, the MCDM method is applied to rank the doctors. The CoCoSo method is a combined compromise decision-making algorithm proposed by Yazdani et al. (2019). This method is based on an integrated simple additive weighting and exponentially weighted product model. The modified CoCoSo method is as follows:

Step 1: The total of the weighted comparability sequence S_i and the total of the exponentially weighted comparability sequences $P_i$ can be calculated by

$$S_i=\sum _{j=1}^n{\omega }_j{s}_{\mathit{ij}},i=1,2,...,m;$$ (19)

$$P_i=\sum _{j=1}^n{\left(s_{\mathit{ij}}\right)}^{{\omega }_j},i=1,2,...,m.$$ (20)

Step 2: Then three appraisal score strategies are used to calculate the relative weight of alternatives. The first appraisal score strategy is the arithmetic mean of sums of weighted sum method (WSM) and weighted product method (WPM) scores, which is shown as follows:

$$Q_{\mathit{ia}}=\frac{P_i+S_i}{{\sum }_{i=1}^m(P_i+S_i)}$$ (21)

The second appraisal score strategy is a sum of relative scores of WSM and WPM compared to the best, which is shown as follows:

$$Q_{\mathit{ib}}=\frac{S_i}{\underset{i}{\min }{S}_i}+\frac{P_i}{\underset{i}{\min }{P}_i}$$ (22)

The third appraisal score strategy is the balanced compromise of WSM and WPM models scores, which is shown as follows:

$$Q_{\mathit{ic}}=\frac{\lambda \left(S_i\right)+(1-\lambda )\left(P_i\right)}{(\lambda \underset{i}{\max }{S}_i+(1-\lambda )\underset{i}{\max }{P}_i)},0\le \lambda \le 1.$$ (23)

where $\lambda$ (usually $\lambda$ = 0.5) is chosen by decision-makers.

The final ranking of the doctors is determined as follows (in descending order):

$$Q_i={\left(Q_{\mathit{ia}}{Q}_{\mathit{ib}}{Q}_{\mathit{ic}}\right)}^{\frac{1}{3}}+\frac{1}{3}(Q_{\mathit{ia}}+Q_{\mathit{ib}}+Q_{\mathit{ic}}),i=1,2,...,m.$$ (24)

4.6. Procedure of the proposed decision-making method considering patients’ emotional intensity

To rank doctors based on heterogeneous online information, this research proposed the MCDM method considering patients’ emotional intensity and the interactions among criteria. The steps of the method are as follows:

Step 1: Doctors’ information and ORs crawling. Data crawling software is used to obtain heterogeneous information from Internet medical platforms, including interval values, semantic values, symbolic values and real values.

Step 2: Data preprocessing. The heterogeneous information is preprocessed according to the data type by the method proposed in Section 4.2.2.

Step 3: Evaluation criteria determination. The TF-IDF method and similarity calculation are used to generate criteria and criteria word sets by extracting keywords from ORs, as can be seen in Algorithm 1.

Step 4: Information transformation. The ORs are transformed into PLTSs through SA by the method proposed in Section 4.2.4.

Step 5: Mixed decision matrix construction. The mixed decision matrix can be established by the transformed information from Steps 2 and 4.

Step 6: Weights obtainment by TF-IDF. The weights of the criteria are obtained by Eq. (8).

Step 7: Weights obtainment by DEMATEL. The weights of the criteria are obtained by Eqs. (2–7).

Step 8: Determination of comprehensive weights. The comprehensive weights of criteria are obtained by Eq. (9).

Step 9: Score computation based on the nonlinear influence of information. The nonlinear influence of real value can be obtained by Eq. (11). The nonlinear influence of interval value can be obtained by Eq. (12). The nonlinear influence of symbolic value can be obtained by Eq. (13). After calculating these scores, the values can be normalized by Eqs. (14–15).

Step 10: Score computation based on patients’ negative bias towards ORs. The patients’ negative bias towards ORs can be obtained by Eq. (18).

Step 11: Modified CoCoSo method application. The modified CoCoSo method is applied according to Eqs. (19–23).

Step 12: Doctor ranking. The doctors are ranked according to the results calculated by Eq. (24).

To show the algorithm process intuitively, the flow chart of the doctor ranking algorithm is shown in Fig. 6 .

Fig. 6.

Flow chart of the doctor ranking algorithm.

5. Case study

Haodf.com is one of the leading Internet medical websites for online consultation in China. When patients encounter health problems, they can consult doctors through this platform (Chen et al., 2020). Haodf.com is the first neutral and objective website to release medical experiences in China. Patients can comment on doctors and the publication rules of reviews are strict. Therefore, the quality and authenticity of information mining from haodf.com can be guaranteed. To evaluate the performance of the improved method, this study selected six doctors from the diabetes department. Only six doctors were selected because patients rarely pay attention to low-ranking doctors. The improved method was used to rank six doctors and the most suitable doctor is recommended to the patient. To protect the selected doctors’ privacy, this study uses $D=\{D_1,D_2,...,D_6\}$ to denote them. In addition, sensitivity analysis and comparison analysis were conducted to demonstrate the validity and robustness of the proposed method.

5.1. Phase 1: Online information analysis

5.1.1. Doctors’ information and ORs crawling

To truly reflect the concentration of patients, this study obtained 1215 ORs from 66 doctors instead of only six selected doctors. And then the information of the selected six doctors was crawled, including the general response speed, professional title, price and the number of online service patients. The information of selected doctors is listed in Table 3 .

Table 3.

Information of selected doctors.

Doctors	General response speed	Professional title	Price (yuan/ 2 days)	Number of online service patients
$D_1$	Fast	Chief physician	[60,160]	15,353
$D_2$	Fast	Chief physician	[200,310]	6924
$D_3$	Very fast	Chief physician	[30,100]	12,868
$D_4$	Very fast	Associate chief physician	[25,85]	10,032
$D_5$	Fast	Chief physician	[25,100]	6748
$D_6$	Very fast	Chief physician	[260,300]	12,903

5.1.2. Data preprocessing

In this section, doctors’ information and ORs were preprocessed respectively. For the information of doctors, it is necessary to preprocess the professional title and the general response speed to facilitate the construction of the hybrid decision matrix. They were described numerically according to Subsection 4.2.2. For ORs, stop words and some irrelevant information were removed. LTP is used for tokenization and POS tagging. Due to limited space, part of the preprocessing results is shown in Table 4 .

Table 4.

Part of the preprocessing results.

Data preprocessing process	Example
Original commentary	Doctor is very professional ∼∼∼ careful… Be amiable!!
Delete stop words and irrelevant information	doctor professional answer careful amiable
Tokenization	doctor professional answer careful amiable
POS tagging	doctor /n professional /a answer /v careful /a amiable /a

5.1.3. Evaluation criteria determination

In this section, evaluation criteria are determined from the crawled information. High-frequency words were extracted from ORs and integrated into the criteria. To better visualize the distribution of high-frequency words, the word cloud for high-frequency words was generated, as shown in Fig. 7 .

Fig. 7.

The word cloud for high-frequency words from ORs.

These high-frequency words can be classified into four evaluation criteria, which are leechcraft (C₁), attitude (C ₂), online communication skill (C₃) and recovery (C₄), respectively. The part of the evaluation system is shown in Table 5 .

Table 5.

Part of the evaluation system.

Part of criteria	High-frequency words
Leechcraft (C1)	Professional, accurate, experience, exquisite
Attitude (C2)	Patient, attentive, kind, perfunctory, responsibility, amiable, careful, mild, responsible, conscientious
Online communication skill (C3)	Perspicuous, timely, clear, irrelevant answer, detailed, clearness
Recovery (C4)	Recover, eliminate, cure, trust

The semantic similarity between every two high-frequency words under the four criteria was calculated through Baidu AI Platform. Due to limited space, the semantic similarity under criteria C₁ and C₄ is shown in Table 6 .

Table 6.

Part of semantic similarity under criteria.

Leechcraft (C1)	semantic similarity	Professional	Experience	Exquisite	Accurate
	Professional	1	0.183	0.248	0.221
	Experience	/	1	0.188	0.155
	Exquisite	/	/	1	0.375
	Accurate	/	/	/	1
Recovery (C4)	semantic similarity	Recover	Eliminate	Cure	Trust
	Recover	1	0.301	0.137	0.141
	Eliminate	/	1	0.238	0.116
	Cure	/	/	1	0.191
	Trust	/	/	/	1

As the semantic similarities of high-frequency words under the four criteria all exceed the $T{H}_w$, the classification of high-frequency words in this study is reasonable.

According to Subsection 4.2.1, four criteria were extracted from doctors’ information, which are general response speed (C₅), the professional title (C₆), the price (C₇), and the number of online service patients (C₈). Therefore, the evaluation criteria of this study are shown in Table 7 .

Table 7.

Evaluation system.

Criteria from ORs	Criteria from doctors’ information
Leechcraft (C1)	General response speed (C5)
Attitude (C2)	Professional title (C6)
Online communication skill (C3)	Price (C7)
Recovery (C4)	The number of online service patients (C8)

5.1.4. Information transformation

In this section, the evaluation under criteria C₁ - C₄ is transformed to PLTSs according to the method described in 4.2.4. Then combined some information in Table 3. The resulting mixed matrix is shown in Table 8 .

Table 8.

Mixed decision matrix for doctor recommendation.

	$C_1$	$C_2$	$C_3$	$C_4$	$C_5$	$C_6$	$C_7$	$C_8$
$D_1$	{(${\text{s}}_{\text{1}}$, 0.57), (${\text{s}}_{\text{2}}$, 0.43)}	{(${\text{s}}_{\text{-1}}$, 0.02),(${\text{s}}_{\text{1}}$, 0.04), (${\text{s}}_{\text{2}}$, 0.94)}	{(${\text{s}}_{\text{1}}$, 0.06),(${\text{s}}_{\text{2}}$, 0.94)}	{(${\text{s}}_{\text{1}}$, 0.92), (${\text{s}}_{\text{2}}$, 0.08)}	1	2	[60,160]	15,353
$D_2$	{(${\text{s}}_{\text{1}}$, 0.54), (${\text{s}}_{\text{2}}$, 0.46)}	{(${\text{s}}_{\text{-1}}$, 0.01),(${\text{s}}_{\text{1}}$, 0.07), (${\text{s}}_{\text{2}}$, 0.9),(s3,0.02)}	{(${\text{s}}_{\text{1}}$, 0.08),(${\text{s}}_{\text{2}}$, 0.92)}	{(${\text{s}}_{\text{1}}$, 0.98), (${\text{s}}_{\text{2}}$, 0.02)}	1	2	[200,310]	6924
$D_3$	{(${\text{s}}_{\text{1}}$, 0.48), (${\text{s}}_{\text{2}}$, 0.52)}	{(${\text{s}}_{\text{1}}$, 0.03),(${\text{s}}_{\text{2}}$, 0.97)}	{(${\text{s}}_{\text{2}}$, 1)}	{(${\text{s}}_{\text{1}}$, 1)}	2	2	[30,100]	12,868
$D_4$	{(${\text{s}}_{\text{1}}$, 0.56), (${\text{s}}_{\text{2}}$, 0.44)}	{(${\text{s}}_{\text{-1}}$, 0.01),(${\text{s}}_{\text{1}}$, 0.1), (${\text{s}}_{\text{2}}$, 0.89)}	{(${\text{s}}_{\text{-1}}$, 0.01),(${\text{s}}_{\text{1}}$, 0.1), (${\text{s}}_{\text{2}}$, 0.89)}	{(${\text{s}}_{\text{1}}$, 0.98), (${\text{s}}_{\text{2}}$, 0.02)}	2	1	[25,85]	10,032
$D_5$	{(${\text{s}}_{\text{1}}$, 0.58), (${\text{s}}_{\text{2}}$, 0.42)}	{(${\text{s}}_{\text{-1}}$, 0.08),(${\text{s}}_{\text{1}}$, 0.1), (${\text{s}}_{\text{2}}$, 0.82)}	{(${\text{s}}_{\text{1}}$, 0.06),(${\text{s}}_{\text{2}}$, 0.94)}	{(${\text{s}}_{\text{-2}}$, 0.1), (${\text{s}}_{\text{1}}$, 0.9)}	1	2	[25,100]	6748
$D_6$	{(${\text{s}}_{\text{1}}$, 0.53), (${\text{s}}_{\text{2}}$, 0.47)}	{(${\text{s}}_{\text{1}}$, 0.1),(${\text{s}}_{\text{2}}$, 0.9)}	{(${\text{s}}_{\text{1}}$, 0.11),(${\text{s}}_{\text{2}}$, 0.89)}	{(${\text{s}}_{\text{-2}}$, 0.02),(${\text{s}}_{\text{1}}$, 0.98)}	2	2	[260,300]	12,903

5.2. Phase 2: Weights determination

In this section, the comprehensive weights are determined by the weights determination method.

5.2.1. Weights obtainment by TF-IDF

By utilizing the TF-IDF algorithm in Python, the TF-IDF values of high-frequency words can be determined. The values are shown in Table 9 .

Table 9.

TF-IDF value of high-frequency words.

Part of criteria	High-frequency words	$\mathit{tf}-id{f}_j^i$
$C_1$	professional	0.3762
	accurate	0.0017
	experience	0.3354
	exquisite	0.0050
$C_2$	patient	0.4599
	attentive	0.0014
	kind	0.4411
	perfunctory	0.0076
	responsibility	0.0017
	amiable	0.0035
	careful	0.0078
	mild	0.0013
	responsible	0.0029
	conscientious	0.0017
$C_3$	perspicuous	0.3852
	timely	0.3517
	clear	0.0023
	irrelevant answer	0.0034
	detailed	0.0037
	clearness	0.0018
$C_4$	recover	0.0009
	eliminate	0.3864
	cure	0.0043
	trust	0.0014

Then the weights of the four criteria can be calculated by using Eq. (8). The whole weights of the criteria are shown as ${\omega }^1={(0.129,0.167,0.134,0.070,0.125,0.125,0.125,0.125)}^T$.

5.2.2. Weights obtainment by DEMATEL

To make full use of the rich experience and professional knowledge of experts, this study invited two groups of people as experts to score the interactions between the criteria. The first group of experts is doctors engaged in the Internet medical industry with more than 2 years of working experience. The second group of experts is patients who have consulted online before. Ten experts were invited to this study. According to Eqs. (2–5), calculate the sum of influences given and received among the criteria, which is shown in Table 10 .

Table 10.

Relationship among criteria.

Criterion	R	C	R + C	R-C
$C_1$	8.11	6.15	14.26	1.96
$C_2$	8.17	6.67	14.84	1.50
$C_3$	6.42	7.01	13.43	−0.60
$C_4$	7.80	7.35	15.16	0.45
$C_5$	6.77	6.98	13.76	−0.21
$C_6$	7.65	6.52	14.16	1.13
$C_7$	5.61	7.73	13.34	−2.12
$C_8$	7.35	9.48	16.83	−2.12

According to Eqs. (6–7), the weights of criteria are shown as${\omega }^2={(0.124,0.128,0.116,0.130,0.118,0.122,0.116,0.146)}^T$.

5.2.3. Determination of comprehensive weights

According to Eq. (9), the comprehensive weights of criteria are shown as$\omega ={(0.128,0.172,0.125,0.073,0.118,0.122,0.116,0.146)}^T$.

5.3. Phase 3: Scores computation

In this phase, considering the nonlinear influence of doctors’ information and patients’ negative bias towards ORs, the scores under each criterion are calculated. The scores of doctors’ information and ORs are calculated according to Eqs. (11–13) and Eq. (18). Suppose the index n in Eqs. (11–13) is equal to 0.5. After calculating the scores, normalize the values based on Eqs. (14–15). The calculated results are shown in Table 11 .

Table 11.

Calculation results of scores.

	$C_1$	$C_2$	$C_3$	$C_4$	$C_5$	$C_6$	$C_7$	$C_8$
$D_1$	0.100	0.072	0.070	0.120	1	1.414	10.488	123.907
$D_2$	0.098	0.248	0.0716	0.124	1	1.414	15.969	83.211
$D_3$	0.0948	0.069	0.067	0.125	1.414	1.414	8.062	113.437
$D_4$	0.099	0.074	0.074	0.124	1.414	1	7.416	100.160
$D_5$	0.101	0.070	0.070	0.139	1	1.414	7.906	82.146
$D_6$	0.098	0.073	0.073	0.128	1.414	1.414	16.733	113.591

5.4. Phase 4: MCDM method usage

In this phase, doctors can be ranked using the CoCoSo method with the above scores. Firstly, according to Eqs. (19–20), this study can calculate the total of the weighted comparability sequence and the whole of the power weight of comparability sequences for each alternative sum of the weighted comparability sequence and also an amount of the power weight of comparability sequences for each doctor as S_i and $P_i$, respectively. The calculation results are shown in Table 12 .

Table 12.

Calculation results of S_i and P_i.

	Si	Pi
D1	0.389	5.910
D2	0.208	5.441
D3	0.499	6.893
D4	0.341	5.849
D5	0.276	4.956
D6	0.393	5.921

Then, using Eqs. (21–23), three appraisal score strategies are used to generate relative weights of other options. λ is set to 0.5 in Eq. (23). The results are shown in Table 13 .

Table 13.

Calculation results of Q_ia, Q_ib and Q_ic.

	Qia	Qib	Qic
D1	0.170	3.057	0.852
D2	0.152	2.098	0.764
D3	0.199	3.784	1
D4	0.167	2.815	0.837
D5	0.141	2.323	0.708
D6	0.170	3.082	0.854

Calculate Q_i according to Eq. (24). The final ranking of doctors is based on descending order. The values of Q_i and the ranking of doctors are shown in Table 14 . Fig. 8 shows the performance of the 6 doctors under different criteria.

Table 14.

Values of Q_i and the ranking result of doctors.

Doctors	Qi	Ranking
D1	2.122	3
D2	1.630	6
D3	2.571	1
D4	2.006	4
D5	1.672	5
D6	2.134	2

Fig. 8.

The performance of 6 doctors under different criteria.

It can be found that: In terms of the performance under C ₂ (attitude), D₂ performs best. The other 5 doctors have little difference in environmental performance. Although the weight of C ₂ is the largest than other criteria, D₂ ranks last in a comprehensive performance. Due to C₇ is a cost criterion, D₂ and D₆ have poor performances. In terms of the performance under C₈, D₁ performs best, followed by D₆ and D₃. In summary, the comprehensive performances of doctors are mainly reflected by the online service attitude, experience and skill of the doctor. Therefore, training for doctors on online services can be carried out.

5.5. Sensitivity analysis

Concerning sensitivity analysis, it is a crucial step to assess the model’s robustness, which is defined as the characteristic of the model to be insensitive to small changes in the input parameters (Muccillo et al., 2015).

Due to the lack of relevant psychophysical experiments, the index n of the score function of doctors’ information in this study is uncertain. This study assumes that the index n is equal to 0.5 in the above case study, which means patients changed from sensitive to insensitive to changes in the information. And sensitivity of patients can be expressed as 0.5. A change in index n may affect the doctor’s ranking. As the usefulness of any MCDM model relies on the reliability of its results, a more robust MCDM model is desired to demonstrate a higher level of tolerance to the change of index n. The index n is set differently, the amount of stimulation that the value brings to the person is different, and the score is also different. Let $n=(0.25,0.5,0.75,1,1.25,1.5,1.75)$. Then recalculate the scores and rank the doctors according to the scores. The ranking results are shown in Table 15 .

Table 15.

Ranking results of the doctors.

n	Doctor ranking
n = 0.25	$D_3\succ D_6\succ D_1\succ D_4\succ D_5\succ D_2$
n = 0.5	$D_3\succ D_6\succ D_1\succ D_4\succ D_5\succ D_2$
n = 0.75	$D_3\succ D_1\succ D_6\succ D_4\succ D_5\succ D_2$
n = 1	$D_3\succ D_1\succ D_6\succ D_4\succ D_5\succ D_2$
n = 1.25	$D_3\succ D_1\succ D_6\succ D_4\succ D_5\succ D_2$
n = 1.5	$D_3\succ D_1\succ D_6\succ D_4\succ D_5\succ D_2$
n = 1.75	$D_3\succ D_1\succ D_6\succ D_4\succ D_5\succ D_2$

It can be seen that the change of n does not change the ranking significantly. For different values of n, D₁ and D₆ are ranked differently. Given that the “Q_i” values of D₁ and D₆ are similar but both doctors differ in price and general response speed. When patients are more sensitive to heterogeneous information, they will perceive D₁ as superior to D₆. Therefore, the change of n can reflect the different emotional intensity toward information and the robustness of the proposed method can be proved by sensitivity analysis.

5.6. Comparative analysis

To evaluate the validity of the proposed method, four existing methods and the original ranking on haodf.com are used to compare with the result above. These four methods are commonly used MCDM methods, which are VIKOR, TOPSIS, MULTIMOORA and DNMA respectively. VIKOR and TOPSIS methods are reference-level methods (Herrera-Viedma et al., 2021). Reference level methods establish several reference points and apply the distance measure methods to determine the relative position of each alternative for these reference points, to determine the most appropriate alternative (Opricovic & Tzeng, 2004). The MULTIMOORA method is a mixed approach that combines the strengths of various approaches (Shang et al., 2022). DNMA can solve the MCDM problems with both PLTSs and numerical numbers (Liao & Wu, 2020). The comparative results are shown in Table 16 and Fig. 9 .

Table 16.

Rank results corresponding to VIKOR, TOPSIS, MULTIMOORA, DNMA and the original ranking on haodf.com.

Alternative	VIKOR	Rank	TOPSIS	Rank	MULTIMOORA	Rank	DNMA	Rank	Haodf.com_Rank
D1	0.524	2	0.680	1	0.037	3	0.437	2	3
D2	0.777	5	0.433	4	−0.004	6	0.106	6	2
D3	0	1	0.663	2	0.072	1	0.607	1	1
D4	0.668	4	0.634	3	0.047	2	0.386	3	4
D5	1	6	0.360	6	0.021	5	0.320	4	5
D6	0.544	3	0.394	5	0.029	4	0.157	5	6

Fig. 9.

Ranking results of the doctors of different methods.

As shown in Table 16 and Fig. 9, it can be intuitionally found that the general ranking trend of our proposed method follows some of the methods and the original ranking on haodf.com. This outcome is acceptable because these results are obtained based on different theories. D₃ is considered the most recommended doctor by all methods except the TOPSIS method. The doctor D₃ has few negative ORs, rich experience and a reasonable price. D₂ ranks low among several methods because of a higher price and fewer online service patients than other doctors. Therefore, this method is not only affected by positive and negative ORs, but also by some doctors’ information that patients are concerned about. Some differences can be found in the ranking orders due to the consideration of the patients’ emotional intensity towards ORs and the nonlinear influence of the doctors’ information. Besides, compared with other models, the proposed model ranking results should have greater differences between alternatives, that is, the proposed model has a better sample discrimination.

Further, to ensure the reliability of the proposed method, the similarity of the results is analyzed through the Spearman rank correlation coefficient. Fig. 10 provides the Spearman rank correlation between the results of different methods. As can be seen in Fig. 10, the Spearman rank correlation coefficient values between some of the methods are large, indicating a high degree of dependence and correlation between the ranking results. Therefore, the model can maintain the relative stability of the ranking, which has strong robustness and feasibility. The results further verified the reasonability and effectiveness of our proposed method in recommending the most appropriate doctor.

Fig. 10.

Spearman rank correlations of different methods.

The comparative methods differ from the method proposed in this paper in the following two aspects.

First, the methods take the emotional intensity of the patient in processing the information into account. The psychological behavior of the decision maker is one of the most important factors influencing the decision results and should be taken into account in the decision application. However, the traditional decision-making methods all ignore the psychological behavior of the decision maker. The proposed method takes into account the psychological behavior of the patient, i.e., the emotional intensity when processing information and applies it to the decision-making problem of the doctors ranking.

Second, the methods differ in terms of decision objectives. The TOPSIS method considers the proximity of the object to the idealized goal. The VIKOR method considers the subjective preferences of the decision maker, and its main feature is the compromise between maximizing “group benefits” and minimizing “individual regrets”. The MULTIMOORA method ranks the alternatives in a combination of three dimensions: ratio system, reference points, and full multiplicative form. The DNMA method consists of three dependent models that rank the options based on these two goal-based normalization methods and three aggregation techniques; Haodf.com ranks doctors based on the number of patients’ likes and their historical consultations. The CoCoSo method is a decision-making method based on a portfolio perspective and a trade-off perspective with the ability to avoid decision compensatory problems and achieve an internal equilibrium in the final utility. It offers more flexibility in ranking alternatives than other multi-criteria decision methods proposed so far. Also, this method is computationally simple and time-consuming. The flexibility and applicability of the method can be reflected more significantly as the number of options or attributes increases.

The comparison results show that the proposed method can be successfully applied to the decision-making method of doctor ranking, and compared with other decision-making methods, it can more accurately and effectively recommend the best doctor in line with the need of patients.

5.7. Recommendation’s accuracy and usability evaluation

This section describes the experiments conducted to explore the accuracy and usability of the recommendations. To assess the accuracy and usability of the method’s recommendations, a questionnaire was administered to 32 users who had used the online consultation and 18 users who had not used the online consultation. Each user was asked to simulate a diabetic patient and choose a doctor by consulting the online information of six doctors in the case study. In addition, each user was asked to score the method’s recommendation results from the usability of the recommendation ranking results by Good, Average and Bad.

The simulation results of the doctor selection are shown in Table 17 . The recommended doctor for the case is D₃, so the Hits Ratio (HR) for the method is calculated as follows. It can be seen from the results that the recommendation results of doctors in this paper meet the needs of users, which once again verifies the rationality and effectiveness of the method in this paper.

$$\mathit{HR}=\frac{1}{N}\sum _{i=1}^N\mathit{hits}\left(i\right)=\frac{43}{50}=86\%$$

Table 17.

Simulation results of questionnaire survey.

Alternative	Number of doctors selected
D1	2
D2	1
D3	43
D4	1
D5	0
D6	3

As shown in Fig. 11 , these users indicated a degree of Good (92.00 %), while only 11.99 % of the total responses belonged to the Medium (6.00 %) and Bad (2.00 %). These results reflect users’ great satisfaction regarding the usability of the recommendation.

Fig. 11.

The scores of the recommendation results’ usability.

5.8. Method complexity analysis

The proposed method can be divided into two parts. The first part is data processing, which takes a lot of time and some of the data needs to be preprocessed manually. This step is the groundwork for the ranking of the alternatives, which is the second part of the proposed method. In the second part, the time complexity of the proposed method in this paper is analyzed from the perspective of the ranking of the alternatives. The complexity of the function in this part is O(m ∗ n), where m is the number of alternatives and n is the number of criteria. In the case study, m = 6 and n = 8. The computing time for this part is about 0.01s. The proposed method was executed on a personal laptop with Windows 11 home Chinese Edition with 16.0 GB RAM and the CPU is 3.20 GHz i5-11320H.

6. Discussion

This study proposed a doctor ranking model based on SA, PLTSs and CoCoSo method. The heterogeneous data were transformed into PLTSs, real values, interval values and symbolic values. A comprehensive weight determination method was investigated to consider patients’ concentration and the interactions among criteria. Considering patients’ emotional intensity, four novel score functions of heterogeneous information were proposed based on the nonlinear influence of information and patients’ negative bias towards ORs. The emotional intensity of patients was used as a score for information. Extended CoCoSo was applied to rank alternative doctors. From the results of the case study, it can be seen that it is necessary and reasonable to consider the emotional intensity of patients when recommending doctors. To verify the robustness and effectiveness of the proposed model, sensitivity analysis and comparative analysis were provided.

The proposed method is advantageous because of two reasons. First, this study models patients’ emotional intensity towards information. In the existing literature on doctor ranking, the patients’ emotions are ignored. As a consequence, it will lead to deviation from reality. Henceforth, personalized cognition should be modeled by considering patients’ emotional intensity. Throughout the proposed method, a more reasonable doctor can be recommended. Second, more comprehensive criteria are determined considering the information concerned by patients. Therefore, the recommended doctor meets the needs and preferences of patients. Also, these criteria can be used for other doctor-ranking studies and can provide a reference basis for their criteria determination.

Compared with doctor recommendation based on the MCDM method without considering ORs, the proposed method understands patients' preferences from various aspects through ORs and extract the value in ORs. Also, this textual information can enrich the evaluation, which can help to improve the reliability of the recommendation. Similarly, compared with doctor recommendations based on the MCDM method considering ORs, the nonlinear influence of doctors’ information on patients and the negative bias of patients towards comments are considered. At the same time, the interactions among criteria and heterogeneous data are fully considered. After full consideration of these, the recommended results are more realistic than other methods and meet the needs of patients.

Three main managerial implications are introduced as follows. Firstly, this study can help doctors improve their performance on criteria that patients are concerned about. When patients have satisfactory medical experiences, they may recommend doctors to similar patients. This will not only increase the popularity of doctors and hospitals but also increase the revenue of hospitals. At the same time, those doctors who are not recommended will actively improve themselves, and provide better services. Also, this will stimulate the development of the Internet medical industry. Secondly, Internet medical platform managers can help doctors strengthen their weaknesses, reduce the number of negative ORs and build harmonious doctor-patient relationships. Also, Internet medical platform managers need to comprehensively analyze the needs of patients, advantages and goals of Internet medical care and determine the direction of future strategic development. At last, Internet medical platforms can recommend doctors from the perspective of patients. Considering patients’ emotional intensity toward information, the platforms can recommend more appropriate doctors for patients. This can not only enhance patients’ satisfaction with doctors but also improve patients' satisfaction with Internet medical platforms.

However, the method proposed in this paper still has the following limitations. First, the method values the patients’ emotional intensity brought by negative ORs but leads to results that can be influenced by malicious negative ORs. And how to screen out and exclude malicious negative ORs is something this method cannot achieve at present. If the website itself is unable to screen malicious negative ORs, the method in this paper will have bias. Secondly, part of the criteria weight is given by experts, i.e., subjective, and experts can control the weight to guide the results. Therefore, to prevent this situation, the selected experts should be screened to ensure that they have no interest. Third, the proposed method removed emojis in data preprocessing which will lead to bias in sentiment analysis. Finally, the proposed method only crawled explicit features in ORs and ignored implicit features. This may overlook key features and key opinions in ORs, such as (price) too expensive, (service attitude) bad, etc.

Although this paper takes the doctor as an example for research, the proposed model can be applied to many research fields. In particular, the score functions of considering emotional intensity can be applied to any ORs-based study.

7. Conclusion

During the global COVID-19 pandemic, Internet-based medical care has seen a boom, with more and more patients opting for online consultations. There needs to be a way to sift through the avalanche of online information to find the key information and recommend the best doctor for patients. The previous methods of ranking doctors based on ORs still have shortcomings. Therefore, this study proposes a hybrid MCDM framework that processes heterogeneous information. Also, our proposed method considers patients’ emotional intensity. The proposed approach can be helpful for patients, which can recommend the most appropriate doctor for patients.

Three main contributions are briefly introduced as follows. Firstly, the proposed doctor recommendation model focus on heterogeneous information, which reduces the loss in the process of information transformation. Therefore, the recommendation results are more in line with reality. Secondly, this study proposed two new score functions considering patients’ emotional intensity which can reflect patients’ actual feelings about data. Finally, this study proposed a weight determination method that is more realistic and comprehensive. By combining the TF-IDF algorithm and the DEMATEL method, the weights can be more precise in reflecting the patient’s concerns and the interactions among criteria.

It should be noted that some interesting questions could be further investigated in future studies. First of all, this study only considers the explicit features of ORs and ignores the implicit features, such as (price) too expensive, (service attitude) bad, etc. In the future, natural language processing can be used to mine implicit features and fully explore the value of ORs. Secondly, emojis were removed during data preprocessing, but emojis contain emotions and their removal may lead to bias in the sentiment analysis. Thirdly, the individual needs of patients can be considered to make personalized recommendations. Finally, this study assumes that the heterogeneous information is real, but there may be false or biased information in it. Therefore, the MCDM framework of this study can be extended to identify false or biased information.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The data is from the Haodf.com, which is accessible to everyone.