Comparing performances of intelligent classifier algorithms for predicting type of pain in patients with spinal cord injury

Abstract

Background and aim

In this study, performances of classification techniques were compared in order to predict type of pain in patients with spinal cord injury. Pain is one of the main problems in people with spinal cord injury. Identifying the optimal classification technique will help improve decision support systems in clinical settings.

Methods

A descriptive retrospective analysis was performed in 253 patients. We compared performances of “Bayesian Networks”, “Decision Tree”, neural networks: “Multi-Layer Perceptron” (MLP), and “Support Vector Machines” (SVM). Predictor variables were collected in data set in SCI patients referred to Shefa Neuroscience Research Center, Tehran, Iran from 2010 through 2016. Performances of classification techniques were compared using “Accuracy”, “Sensitivity or True Positive Rate” (TPR), “Specificity or True Negative Rate” (SPC), “Positive Predictive Value” (PPV), “Negative Predictive Value” (NPV).

Results

MLP with Boosting technique was found to have the best accuracy (91%), best sensitivity (89%), best specificity (95%) best PPV (91%), and best NPV (96%) to predict spinal cord injury in this data set, given its good classificatory performance.

Conclusion

Computer-aided decision support systems (CAD) are dependent on a wide range of classification methods such as statistical methods, Bayesian methods, deductive classifiers based on the state or case, decision-making trees and neural networks: Multi-Layer Perceptron. Neural network classifiers especially, are very popular choices for medical decision-making, with proven effectiveness in the clinical field.

1. Introduction

Pain is an undesirable feeling, and a sensory experience accompanied by tissue damage (1). Pain is the most common reason for consulting a physician in the US (2). The prevalence of pain in adults in Tehran is reported as approximately 24% (3). In many medical conditions, pain is a significant symptom, highly relevant to the quality of life of a person and his general performance (4). Pain medicine is a branch of medical science that is related to a variety of interdisciplinary fields tending to reducing pain and improving the quality of life of the suffering patients (5). Pain is a general issue in medicine; and a source of personal and family suffering (6). In developed countries, nearly 2.8 percent of the gross domestic income is spent on direct or indirect treatment of pain (7). Pain is one of the main problems in people with spinal cord injury (SCI) (8), and is one of the most common reasons of reduced quality of life after SCI (9). In many medical conditions, pain is a major symptom that is significantly associated with reduced quality of an individual’s life and his/her overall performance (10). On the one hand, it must be noted that evaluating the type of pain in patients is one of the most important tasks in the processes associated with knowledge discovery and data mining. On the other hand, artificial intelligence helps physicians to diagnose disease or choose treatment (11). Intelligent systems generally consist of five parts: 1. Choosing a classifier system, 2. Choosing effective features in classification, 3. Training 4. Validation, 5. Assessment (12). Intelligent techniques are steadily changed to assist in accurate and rapid diagnosis (9). Data classification and grouping, play a key role in exploring hidden patterns, data mining, and generally data analysis, which necessitates creating classification patterns to put a new sample with a set of features into a particular class. In artificial intelligence and machine learning methods, it is essential to create an appropriate classification pattern with the best correction factor. Several methods have been proposed to solve this problem, such as artificial neural network, logistic regression, decision tree, rules, Bayesian networks, and so on (13). Some applications of neural networks in medicine were performed and widely used in medical decision support systems (14). William G. Baxt, in 1991, used an artificial neural network to identify myocardial infarction in patients presenting to an emergency department with anterior chest pain (15). Jari J. Forsström, in 1995, used artificial neural networks for decision support in clinical medicine (16). Few works have been published on the comparison of classification techniques in different areas. Imran Kurt, in 2008, compared performances of logistic regression, classification and regression tree, and neural networks for predicting coronary artery disease (17). Delen, Walker, and Kadam, in 2005, compared logistic regression (LR), decision tree (C5) and artificial neural networks for predicting the survival of breast cancer cases (18). Mahdieh Khalilinezhad, in 2015, for prediction of healthy blood with data mining classification used decision tree, naive Bayesian and SVM approaches (19). Stark and Pfeiffer in 1999 compared LR, classification tree algorithms and artificial neural networks to solve classification problems in complex data sets in veterinary epidemiology (20). Support vector machines (SVM), Bayesian networks, decision tree, and neural networks algorithms (MLP) are often used as classification techniques in order to predict type of pain. Neural networks have been used to model medical and functional outcomes of dangerous diseases. They have become a popular tool for classification, as they are very flexible, not assuming any parametric form for distinguishing between categories (21). Neural network is a simple mathematical model of neuron performance in the human brain, capable of learning and extensibility. For this reason, neural network is a part of artificial intelligence (AI) that is vastly used in studies, and is able to model complex systems with unidentified relations between variables in a non-linear manner (22, 23). Bayesian network, or the Bauer network, or the Bauer-Bayesian network is a directed acyclic graph that indicates a set of random variables and their independent relations. For instance, a Bayesian network can show the relations between diseases’ causes and themselves. So, having the causes makes it possible to detect a specific disease in a patient (24). The Bayesian network is a relatively new tool for identifying the probable relations for preventing or evaluating the membership class. In short, it could be said that the Bayesian network is the presentation of unidentified relations between the parameters in an area. The Bayesian network is a directed acyclic graph of nodes capable of showing random variables and curves for the purpose of displaying probable relations between variables (25). A decision tree is a decision support tool that uses trees for modeling. The decision tree is usually used for investigation in operations, specifically in decision analysis, in order to determine the strategy with the highest probability for achieving the objective. The other use of decision trees is for describing the calculations of conditional probabilities. In decision analysis, a decision tree is used as a tool for illustrating and analyzing the decision, where the expected values of competitions are successively calculated (26). Support vector machine is a method for supervised learning, used for classification and regression (27). This method is among the relatively new ones displaying good performance compared to older methods used for classification including perceptron neural networks. The function basis of the SVM classifier is linear data classification. In linear classification, we attempt to select the line with a higher reliability margin (28). The purpose of this study was to compare performances of classification techniques in order to predict type of pain in patients with SCI in a pain clinic. We have created models using support vector machines (SVM), Bayesian networks, decision trees, and neural networks algorithms (MLP) which are often used for classification.

2. Material and Methods

A retrospective analysis was performed in 253 records. Type of pain in patients with SCI was defined and grouped in 4 categories: Nociceptive, neuropathic, psychosomatic, and other pain. All patients’ records were assessed by two specialist physicians not participating in the study. All patients with SCI were seen in the Pain Clinic of Shefa Neuroscience Research Center in Tehran between January 2005 and February 2015. Independent variables were analyzed using the Principal Component Analysis (PCA) technique. Data collection was conducted by using archives of patient’s medical records hospitalized in the Shefa Neuroscience Researches Center, Tehran, and marking the files belonging to patients afflicted by spinal damage pain for future recording in the dataset. Within 7 months, the data belonging to 253 patients were collected and recorded. We used Matlab R2014a, Microsoft Visual Studio Ultimate 2013, and SQL server 2013. Data collection was conducted in accordance with the McGill pain questionnaire (29, 30). MPQ is one of the tools designed for pain measurement, used in hundreds of studies. Since all of the collected data are labeled, this research used the categorization technique, which is one of the supervised methods. For sampling in this study, we used patients with SCI who were seen in the pain clinic between January 2005 and February 2015, and have a complete record and medical diagnosis of pain which has been documented. Before building models, the data set were randomly split into two subsets, 70% (n=177) of the data for training set (for neural network, Bayesian, SVM and decision tree), 30% (n=76) of data for test set. All collected data were labeled, and in this study, the categorization technique which is among the supervised methods, has been implemented. Also, this research used various models (support vector machines, Bayesian networks, decision tree, and neural networks algorithms) to obtain the best model with the highest accuracy at the train stage. The main stages of data collection and preparation for this research are summarized as follows:

• Attribute selection: at this stage, all attributes pertaining to the pain questionnaire of Shefa Neuroscience Centre of Tehran were collected and recorded in the system. At this stage, 386 attributes were collected for each sample.

• Database verification: for accuracy control and achieving certainty about the completion of recording, an analogical evaluation between the electronic data and the paper files was conducted. Finally, after complete updating, and Excel output was obtained from the collection system, and saved in the CSV (MS-DOS) format.

• Dimensionality reduction: reducing the number of attributes (dimensionality reduction) was undertaken in 2 stages, by using feature combination and Expert Physician Opinion.

Performances of classification techniques were compared using accuracy, sensitivity or true positive rate (TPR), specificity or true negative rate (SPC), positive predictive value (PPV), negative predictive value (NPV). Sensitivity in this context is also referred to as the true positive rate or recall, and precision is also referred to as positive predictive value (PPV); other related measures used in classification include true negative rate and accuracy (29). True negative rate is also called specificity. Sensitivity is also called recall, hit rate, or true positive rate (TPR).

$$\begin{array}{c}\text{Accuracy}:(\text{TP}+\text{TN})/(\text{TP}+\text{TN}+\text{FP}+\text{FN})\\ \text{Sensitivity\hspace{0.17em}or\hspace{0.17em}true\hspace{0.17em}positive\hspace{0.17em}rate\hspace{0.17em}}(\text{TPR}):\text{TP}/(\text{TP}+\text{FN})\\ \text{Specificity\hspace{0.17em}or\hspace{0.17em}true\hspace{0.17em}negative\hspace{0.17em}rate\hspace{0.17em}}(\text{SPC}):\text{TN}/(\text{TN}+\text{FP})\\ \text{Positive\hspace{0.17em}Predictive\hspace{0.17em}Value\hspace{0.17em}}(\text{PPV}):\text{TP}/(\text{TP}+\text{FP})\\ \text{Negative\hspace{0.17em}Predictive\hspace{0.17em}Value\hspace{0.17em}}(\text{NPV}):\text{TN}/(\text{TN}+\text{FN})\end{array}$$

3. Results

The characteristics of the study population are shown: Percent of war veterans was 83%, participants with nociceptive pain 24%, participants with neuropathic pain 52%, participants with psychosomatic pain 20%, and participants with other types of pain or unknown 4%. The average age of participants was 55years, the lowest age of participants was 13 years and the highest age of participants was 86 years. The average of BMI was 26 kg/m2. The longest history of pain in the participants was 34 years, and the longest duration of pain attacks was 6 days. Then for all models, the precision, sensitivity, attributes, positive reported values and negative reported values were calculated for comparison. The patients were classified into four diagnosis groups implying the types of their pain. In this project, different models of intelligent classifiers were generated and at the end, through accurate analogy between the networks, the model producing the highest accuracy and least errors was selected. Table 1 shows results of comparison model. The most important attributes, ordered by using the Likert Scale: Sex, Age, Veteran, BMI (body mass index), HADS Anxiety, HADS Depression, PDI, Pain History, History of surgery, Duration of Attack Pain, Number of areas of body that have Pain, Burning pain, Throbbing pain, Needling pain, Sudden onset, Dull pain, Walking-Increase Pain, Sit-Increase Pain, Sleep-Increase Pain, Stand-Increase Pain, Lay down-Increase Pain, Welter in bed- Increase Pain, Anger-Increase Pain, Heat-Increase Pain, Cold-Increase Pain, Walking-Decrease Pain, Sit-Decrease Pain, Sleep-Decrease Pain, Stand-Decrease Pain, Lay down -Decrease Pain, Heat-Decrease Pain, Cold-Decrease Pain. The best accuracy among models is obtained via the MLP neural network model, using the “boosting” optimization technique. The accuracy of diagnosing the type of pain via this model has a compatibility index of 91% with the physician’s diagnosis. MLP with boosting technique was found to have the best sensitivity (89%), best specificity (95%) best PPV (91%), and best NPV (96%) to predict spinal cord injury in this data set, given its good classificatory performance. In addition, the highest measure of sensitivity also belongs to the MLP neural network model, using the “boosting” optimization technique, achieving a sensitivity of 89%.

Table 1.

Comparative Models

	Sensitivity	Specificity	Positive predictive value (PPV)	Negative predictive value (NPV)	Accuracy
MLP (Standard)	0.37	0.69	0.50	0.76	0.54
MLP (Boosting)	0.89	0.95	0.91	0.96	0.91
MLP (Bagging)	0.79	0.90	0.85	0.92	0.84
Bayesian Net	0.70	0.85	0.71	0.85	0.73
Bayesian Net (Likelihood)	0.70	0.85	0.71	0.85	0.73
Bayesian Net (Chi Square)	0.70	0.85	0.71	0.85	0.73
Decision Tree (Boosting)	0.72	0.86	0.74	0.87	0.75
Decision Tree (Bagging)	0.57	0.79	0.69	0.84	0.68
SVM (RBF)	0.74	0.87	0.79	0.89	0.79
SVM (Liner)	0.50	0.75	0.58	0.77	0.60

4. Discussion

In this study, we reviewed some methods to develop several prediction models for SCI pain, especially BN, decision tree, SVM machine and multi-layer perceptron neural network algorithms. Furthermore, we evaluated performance of models according to accuracy, SEN, SPE, NPV, and PPV. Spinal cord damages are among the most common causes for chronic pain in the world, imposing great expense on the health system of any country; while the patients suffer for long periods. The most important recent epidemiological investigation on world population has shown that the incidence of pain due to spinal cord injury is tremendous. In general, there are many causes for chronic pain after a spinal cord injury (SCI). Studies have revealed that there are many characteristics for determining the type of pain that can be different based on the type of therapy specialties according to the classification methods for various types of pain related to SCI. Such a difference in characteristics is due to the fact that a pain management team typically consists of different specialists and is considered a multi-specialty team. The models used in the present study may result in a new approach to the applications in clinical guidelines. In addition, healthcare planners and policy makers can consider them, especially for the pain related to spinal cord injuries and the management of such diseases. After generating models, we illustrated some diagrams for their evaluation. Performing various categorization methods on differing data shows that these methods do not have similar performance, with each having relevant performance for a specific set of data. We implemented the “bagging” and “boosting” performance improvement techniques in order to improve the values needed by the models. It is worth considering that the least accuracy and sensitivity belonged to the MLP (standard) model. Using the “boosting” and “bagging” techniques, that are among the optimization techniques for classifier models, is highly determinant. When the data is not labeled, supervised learning is not possible, and there is a need for a non-supervised learning approach. This task is undertaken for finding the main clusters of the data and allocating new data to each of the clusters thereafter (28). Using the boosting method completely eliminates bias and deviations in the classification of samples. This is a common method in artificial intelligence and machine learning, as well as one of the most important algorithms used for network training to improve network performance. There is also another major algorithm used for boosting network performance and network training called bootstrap aggregating technique. It is used to increase both the stability and accuracy of statistical classification methods, especially logistic regression. It can also be used in other classification methods (33). We used boosting and bagging to improve the classification by combining classifications of randomly generated training sets. Neural networks are powerful tools for helping the physicians. The tools can process a high number of data and minimize errors in patients’ information ignorance. They also reduce the time spent on diagnosis. Neural network strength has been proven for satisfactory diagnosis of various diseases. In addition, they bring about a more reliable diagnosis which increases patients’ satisfaction. However, neural network is a tool provided to help physicians and the physician is the one who should make the final decision (9). Computer-aided decision support systems (CAD) are dependent on a wide range of classification methods such as statistical methods, Bayesian methods, deductive classifiers based on the state or case, decision-making trees and neural networks. Neural network classifiers especially, are very popular choices for medical decision-making, with proven effectiveness in the clinical field (34).

5. Conclusions

In summary, the study findings indicated that accurate and complete medical data and information is of great importance, and could potentially have an impact on the results of similar studies. The present study focused on several common and effective techniques for determining the type of pain in patients, and their accuracy and effectiveness were examined and compared with statistical indices. The practical significance of the findings is the higher effectiveness of artificial neural networks in creating a suitable classification method to predict the type of pain in patients with spinal cord injuries. Other machine learning algorithms can be used to improve training rate, such as boosting and bootstrap aggregating. It is recommended that such data should be analyzed more with intelligent techniques and, in addition, it is very important to refine the data before they are used by models and networks. Future research might include examining and determining the effective characteristics in determining the type of pain in patients with spinal cord injury, especially for injured war veterans, by using methods like machine learning methods, the most important of which is artificial neural network.