A Comparison of Logistic Regression Model and Artificial Neural Networks in Predicting of Student’s Academic Failure

Abstract

Background and objective:

Artificial Neural Networks (ANNs) have recently been applied in situations where an analysis based on the logistic regression (LR) is a standard statistical approach; direct comparisons of the results, however, are seldom attempted. In this study, we compared both logistic regression models and feed-forward neural networks on the academic failure data set.

Methods:

The data for this study included 18 questions about study situation of 275 undergraduate students selected randomly from among nursing and midwifery and paramedic schools of Hormozgan University of Medical Sciences in 2013. Logistic regression with forward method and feed forward Artificial Neural Network with 15 neurons in hidden layer were fitted to the dataset. The accuracy of the models in predicting academic failure was compared by using ROC (Receiver Operating Characteristic) and classification accuracy.

Results:

Among nine ANNs, the ANN with 15 neurons in hidden layer was a better ANN compared with LR. The Area Under Receiver Operating Characteristics (AUROC) of the LR model and ANN with 15 neurons in hidden layers, were estimated as 0.55 and 0.89, respectively and ANN was significantly greater than the LR. The LR and ANN models respectively classified 77.5% and 84.3% of the students correctly.

Conclusion:

Based on this dataset, it seems the classification of the students in two groups with and without academic failure by using ANN with 15 neurons in the hidden layer is better than the LR model.

1. INTRODUCTION

Academic failure is one of the main problems of the universities so that it not only wastes the time and resources, but also cause other problems such as have psychological, family and social problems for the university students (1). UNESCO defines it as repeating an educational grade, early dropout or Reduced quality of education (2).

Every year these problems are increasing so that many of students cannot handle their academic courses and eventually leave the university (3).

The results of studies in developing and developed countries showed a many factors internal and external the educational systems effect on the success or failure of students (4).

There are many factors that may be effected on educational performance of students such as high school final grade, matriculation examination, age of admission, gender, economic problems, parental education and etc. (5, 6). Another study has pointed to issues (factors) such as socioeconomic status of the family, personality characteristics of student (4).

The ability to classify the student based on influential factors is very important to universities or educational institutions because strategic programs can be planned on improving or maintaining the students’ performance during their studies in the university period (6).

For classificating/predicting of binary outcome variable (academic failure), some methods are available such as discriminant analysis, regression techniques, genetic algorithms, different data mining methods, decision tree and artificial neural network models. The general structure of artificial neural network was inspired by neurobiology of the human brain (7).

In theoretical works and more published reports of studies found that ANNs approach compared to traditional statistical methods such as regression analysis, discriminant analysis have better performance in predicting binary outcomes, especially when the relationship between the dependent and independent variables is complex (8, 9).

The results of a meta-analysis with 28 studies showed that in 36% of them, ANN, in 14% logistic regression method, performed better and in other studies (50% of cases) both modes had a similar performance (10). The number of studies compared ANN and logistic regression and it can be seen that both models perform on about the same level more often than not, with the more flexible neural networks generally outperforming logistic regression in the remaining cases (11).

In this study, we applied logistic regression model and ANN to predict the academic failure based on Effective factors and then compared the ability of each of these models to classify academic failure among students of Medical Sciences.

2. MATERIAL AND METHODS

2.1. Study population

In a cross-sectional study, data collected using a stratified random sampling from 275 undergraduate students in schools of nursing & midwifery and paramedic schools of Hormozgan University of Medical Sciences (HUMS) in the first semester of 2013. Bandar Abbas is the capltal of Hormozgan province. This city is located in the south of Iran (north of the Persian Gulf) and it is one of the largest commercial ports in Iran. It is hot and humid in Bandar Abbas.

The data collection tool was a researcher made questionnaire which contained questions related to the factors effecting student’s academic failure as is shown in table 1. These questions by reviewing the valid literature and discussion with experts in the Medical Education Development Center (MEDC) of HUMS were identified.

Table 1.

Student’s Characteristics Used for predicting academic failure

In this study Grade Point Average (GPA) of the previous semester was the output or response variable which represented the performance of a student and other variable as shown in table 1 were introduced as input or independent variables. According to the directive of the Ministry of Health and Medical Education of Iran in 2011, GPA of the previous semester less than 14 is considered as academic failure.

2.2. Artificial Neural Network

Artificial neural networks are one of the most popular models which can used for prediction and classification a outcome variable. The structure of this model is inspired from neural networks of the human brain.

An ANN consists of a number of computational units called neurons and they linked with each other by connections. ANN is consists of three components or layers: a layer of “input” units is connected to a layer of “hidden” units, which is connected to a layer of “output” units (Fig 1) (12).

Figure 1.

Typical a feed forward multilayer perceptron

Currently the several neural networks have been introduced that each application their specific. One of the most common is Multilayer Perceptron (MLP) which widely used in pattern recognition, prediction and no linear relationship. It is a feed forward neural network with an input layer and output layer and a number of hidden layers (Fig 1). One hidden layer is generally sufficient for classifying most data sets and determine the number of hidden layer neurons is an important issue. In fact, network optimization depends on the number of neurons of the hidden layer. However, there is not a rule to select the number of hidden layer neurons, but to prevent of overloading, it better not a large number (13).

A feed-forward network, the neurons in each layer only connect with the neurons in the next layer. These connections are unidirectional, which means signals or information being processed can only pass through the network in a single direction, from the input layer, through the hidden layer (s) (14).

The MLP takes a vector of real values (independent variables) and computes a linear combination of input data. If the calculated value is greater than a threshold value, the output of the MLP equal to -1, otherwise it will be equal to+1 (15).

In the MLP with back error propagation, transfer function in the hidden layer neurons is a nonlinear function such as hardlimit, linear or sigmoid which able to associate training patterns with outputs. But for simple derivative and related derivatives with function, mostly used sigmoid function (14).

2.3. Statistical analysis

To fit ANN to data, the first step, dataset randomly divided into two sets; one set of 70% (193) cases for training and 30% (82) cases for testing the model. Then, a standard feed-forward, back-propagation neural network with three layers: an input layer, a hidden layer and an output layer. The input layer consisted of 15 neurons; in order to prevent overfitting to training data, the hidden layer contained a different number of neurons such as: 10, 15, 20, 25, 30, 35, 40, 45 and 50 and the output layer contained two neurons. We used the sigmoid function in both hidden layer and output layer for activation. The learning rate and momentum for network training were set respectively 0.25 and 0.9 and the models were run until a minimum average squared error < 0.063 was obtained.

Multivariate logistic regression (with the backward stepwise method) used to perform which variable outcome or dependent variable was academic failure (Score, < 14 and >= 14) and other variable were independent variables. Also Hosmer - Lemeshow goodness-of-fit test was used to compare the observed and predicted values of academic failure.

The Area Under the Receiver Operating Characteristic (AUROC), the indicators of sensitivity, specificity and kappa coefficient used to compare ANNs with different neurons and also for comparing the best ANN with logistic regression curves to evaluate the predictive accuracy of two models. The higher ROC areas indicated a better performance of the models.

The neural network development software used in this study was MATLAB 2010 and the statistical analyses, including descriptive statistics, logistic regression analyses and kappa statistic were performed by the IBM SPSS, version 22.0.

3. RESULTS

From 275 students, 77 (28%) suffered academic failure (GPA<14); 183 cases (66.5%) were female, 183 (66.5%) were less than 20 years old, and 216 (75.5%) were single. For 176 (64%) students education of the father and for 219 (79.6%) education of mothers were under high school diploma; 174 (63.3%) studied in paramedical school, 198 (72%) were unemployment, and 216 (78.5%) students resided in dormitory. 196 (76.3%) of them the GPA of diploma high school was greater than 16, 229 (82.3%) students were studying between second to fifth semester, 13 (4.7%) were guest student from other universities, and 8 (2.9%) were transferred from other universities. Also, 252 (91.6%) were interested in their field of study in moderate to very much, 94 (34.2%) had extracurricular activities in their college, 102 (37%) at least once fail of a course.

To select the most appropriate ANN, the nine perceptron models with16 neurons in the input layer, one neuron in output layer and different numbers of neurons in the hidden layer were used (Table 2). Comparison of indicators such as train accuracy, test accuracy, incorrect prediction, AUROC, and kappa statistics showed that the ANN with 15 neurons in the hidden layer, compared with other neural networks, had a better performance. Therefore, the desirable neural network that should be compared with the logistic regression model was a neural network with15neurons in the hidden layer.

Table 2.

Accuracy, incorrect prediction, Roc Curve Area and kappa for comparison ANNs models. *Az =AUROC and SD=Standard Deviation

The UAROC for ANN (15) and logistic models separately were compared with the reference AUROC (Table 3). The results showed that ANN (15) model was significantly more than logistic regression (p<0.001).

Table 3.

Test UAROC for Logistic regression and ANN (15), *P<0.0001

The LR and ANN models classified 77. 5% and 84.3% of students respectively, with and without academic failure correctly. The LR have sensitivity and specificity were 67. 3% and 79.3%, respectively, and the ANN had a sensitivity and specificity of 76.6% and 86.7%, respectively. Also, the kappa statistics for LR was 0.452, showing that the emerged classification may be due to chance (P=0.215) and this statistic for the ANN was 0.60 which was significant; showing that the emerged classification was not due to chance (p<0.001) (Table 4).

Table 4.

Classification of students based on logistic regression and ANN (15) models

Females were two times more than men, married students 3.3 more than singles, students with grade point (GPA) of diploma greater than or equal to sixteen 1.4 times more than others, students how were employed 2.3 times more than the unemployed subjects, students of nursing and midwifery 5.7 times more than paramedics school and students with at least once failed of a courses 2 times more than students without it was at risk of academic failure (Table 5). According to neural network results, five variables, i.e. grade point average of diploma, number of crash course (repeated), age, father’s education, and sex had the greatest effect on how the network classifies the students.

Table 5.

Odds ratios and coefficients of binary logistic regression analysis of factors associated with academic failure

Figure 2.

Factors affecting on academic failure in the order of their importance in ANN (15)

Results of ANN (15) showed that respectively at least once failed of a course, GPA of high school diploma, type of school (Nursing & Midwifery or paramedical) Father’s education were five effective factors on academic failure.

4. DISCUSSION

In this study, after changes in the number of hidden layer neurons, an MLP with 15 neurons in the hidden layer was the best structure among nine ANN models and then it was chosen to compare with LR model. But the result showed that by increasing the number of neurons in hidden layer cannot certainly effected on the performance of the neural network.

Significantly, the AUROC for ANN (15) model (89%) was considerably higher than LR model (55%). Also sensitivity for two models showed that the classification accuracy of academic failure in ANN (76.6%) was more accurate than the LR model (67.3%). So far, many studies in order to compare ANN and LR models in various fields such as medicine, economics, agriculture, etc. have been conducted but there are a few studies in the educational field. In a study almost the same, results showed that the correct classification in predicting graduate students for the ANN (93.3%) was higher than the discriminant analysis (81.5%) (16). Also in other studies showed that ANN model is far better than the LR model (7, 11, 17-20). Although most studies indicated ANNs are a technique alternative to conventional statistical methods for predicting, but there were a few studies which showed LR model have the same or better performance (10, 11). In addition the results of a meta-analysis study showed that in 36% of cases ANN and 14% LR performed better and in others both models functioned well (10).

In general ANN methods as semi-parametric methods have many advantages such as, allow a large number of variables in the model, no need to assumptions such as normality and…, finding the models despite missing data, detection of complex and nonlinear relationship between independent and dependent variables (21).

Although in theory and practical studies have been hinted ANN have better performance than statistical methods, but it has some disadvantage such as, the accuracy of the results depends largely on the size of the training set, requires the initialization and adjustment of many individual parameters to optimize the classification performance, standardized coefficients and odds ratios corresponding to each independent variable cannot be easily calculated, weights are generated in a neural network analysis, but their interpretation is difficult, the weights may be influenced by the program used to generate them (22).

Logistic regression model is mainly influenced by the sample size, number of independent variables, potential multicollinearity and missing (23). Unfortunately, in this study the number of variables relative to the sample size is large and missing data observed, In this case we used ANN beside LR for academic failure data set. The ANN model indicated respectively five factors, at least fail of a course, GPA of high school diploma, type of school (nursing & midwifery or paramedical), father’s education and interest in the field had the greatest impact on academic failure. Only GPA of high school diploma and school were significant variables common in ANN and LR models. Also, some studies noted the effect of these factors on academic failure (4) but other indicators such as school related, family related, student related and community related were the most important factors effect on academic failure (24) which in this study it was not possible to collect them and this was one of the limitations of our study.

Although the results of this study and previous studies showed that the ANNs has better performance, but it seems to perform a meta-analysis is a useful way to identify model with better performance for binary data.

5. CONCLUSION

In general, the results of this study showed that among 9 ANNs, an ANN with 15 neurons in the hidden layers had better performance. In comparison with the conventional LR model, the ANN model in the study was more accurate in predicting academic failure and had higher overall performance indices. Therefore, based on the results of other and academic failure data, it seems that for classification of a dichotomous dependent variable, artificial neural network methods are appropriate to be used.