Logistic random forest boosting technique for Alzheimer’s diagnosis

Abstract

Alzheimer's disease (AD) is a common and well-known neurodegenerative condition that causes cognitive impairment. In the field of medicine, it is the "nervous system" disorder that has received the most attention. Despite this extensive research, there is no treatment or strategy to slow or stop its spread. Nevertheless, there are a variety of options (medication and non-medication alternatives) that may aid in the treatment of AD symptoms at their various phases, thereby enhancing the patient's quality of life. As AD advances over time, it is necessary to treat patients at their various stages appropriately. As a result, detecting and classifying AD phases prior to symptom treatment can be beneficial. Approximately twenty years ago, the rate of progress in the field of machine learning (ML) accelerated dramatically. Using ML methods, this study focuses on early AD identification. The "Alzheimer's Disease Neuroimaging Initiative" (ADNI) dataset was subjected to exhaustive testing for AD identification. The purpose was to classify the dataset into three groups: AD, "Cognitive Normal" (CN), and "Late Mild Cognitive Impairment" (LMCI). In this paper, we present the ensemble model Logistic Random Forest Boosting (LRFB), representing the ensemble of “Logistic Regression” (LR), “Random Forest” (RF), and “Gradient Boost” (GB). The proposed LRFB outperformed LR, RF, GB, “k-Nearest Neighbour” (k-NN), “Multi-Layer Perceptron” (MLP), “Support Vector Machine” (SVM), “AdaBoost” (AB), “Naïve Bayes” (NB), “XGBoost” (XGB), “Decision Tree” (DT), and other ensemble ML models with respect to the performance metrics “Accuracy” (Acc), “Recall” (Rec), “Precision” (Prec), and “F1-Score” (FS).

Introduction

AD is among the common causes of dementia. In 2018, approximately 50 million individuals were affected by this condition, a number that is expected to triple by 2050. Typically, the signs of AD become apparent when a person reaches the age of 60. Nonetheless, specific forms of AD may manifest earlier (30–50 years) in individuals with "gene" mutations. Long-term AD causes anatomical and functional changes in the brain [1]. Numerous decades might elapse between a person's last healthy condition and the onset of AD in the event of AD patients. The progression from moderate cognitive decline (MCI) to AD is gradual. Nevertheless, not all MCI patients progress to AD [2]. In the year 2020, the entire planet was infected with a highly contagious coronavirus, and the only way to prevent its transmission was through lockdown and adequate house seclusion. Individuals shut themselves in their houses and most offline services went online, including education, employment, grocery shopping, and doctor consultations, to limit human contact. But there wasn't a good online platform for predicting AD that a user could use from home instead of having to go to a hospital.

With these considerations in mind, an ML-based framework for the accurate prediction of AD has been established in this study. The experimental dataset was the ADNI data [4] with the parameters Age, Gender (PTG), Years-of-Education (PTEDUCAT), Ethnicity (PTETHCAT), Race (PTRACCAT), APOE4 Genotype, MMSE Score on Cognitive Test (MMSE), Imputed Genotype (IG), and APOE ALLELE 1 & ALLELE 2 genotypes (APOE Genotype), which are described in detail in Sect. 3. This endeavour consists of two steps. Data pre-processing consisting of exploratory data analysis (EDA), feature selection using RF, One-Hot encoding, and data normalisation (Min–Max Scaling) was performed in the first stage. An EDA was conducted to determine the distribution of numerical and categorical records. Feature selection was carried out by computing the significance of the features using the RF method. One-Hot encoding was used to transform categorical characteristics to binary features to facilitate the deployment of machine learning. The dataset was divided into "training data" and "test data," with 80% "training data" and 20% "test data." Using Min–Max scaling, the training and test data were normalised between 0 and 1.

In the second phase, categorization was undertaken. Multiple ML models, including LR, RF, GB, k-NN, MLP, SVM, AB, NB, XGB, and DT, were trained individually. The top performing single ML models were chosen by altering their parameters and calculating their accuracy using "k-fold cross validation" (kcv) with k = 5. The highest performing single ML models were then utilised to construct ensemble ML models. The suggested LRFB model outperformed LR, RF, GB, k-NN, MLP, SVM, AB, NB, XGB, DT, and other ensemble ML models for the early identification of AD by classifying the ADNI dataset into three classes: AD, cognitively normal (CN), and LMCI.

The contribution of this work is highlighted below:

• Developed an ensemble model of LR, RF, and GB named LRFB for the prediction of AD

• Feature selection is performed using the feature importance technique of the RF method.

• Evaluated the ML models using multiple performance metrics such as Acc, Rec, Prec, and FS.

• Multiclass classification was performed by predicting 3 classes, namely AD, CN, and LMCI.

The remainder of the article is organised as shown below. Section 2 explores relevant research in AD prediction utilising ML approaches. In Sect. 3, the proposed framework is outlined. In Sect. 4, the test results are discussed. In Sect. 5, the conclusion and future scope are presented.

Related work

This section gives an overview of some of the most important research that has been done to find AD using ML techniques.

In [5], DT, RF, SVM, GB, and voting classifiers were used to determine the optimal parameters for AD prediction on the Open Access Series of Imaging Studies (OASIS) data. For ML models, performance was tested using parameters such as Prec, Rec, Acc, and FS. On the dataset, ensemble approaches were not evaluated. No feature selection was performed. The work [6] has been completed to predict a patient's future "cognitive state" based on past and present medical data. Thirteen "biomarkers" were analysed by the ML models (LR, SVM, GB, RF, Multilayer perceptron, and Recurrent neural network). Considering such many biomarkers improved the predictability of a patient's state over a particular length of time. The demographic characteristics that are essential for AD classification were ignored. In addition, ensemble methods were not created. There was no feature selection performed.

The primary objective of the research [7] was to apply ML approaches to the "ADNI" dataset to classify the different phases of AD. Based on analysis, the participants were divided into five distinct groups: "CN", "Early Mild Cognitive Impairment" (EMCI), "Late Mild Cognitive Impairment" (LMCI), "Subjective Memory Complaint" (SMC), and "Alzheimer's Disease" (AD). In [7], six machine learning (ML) techniques, including k-NN, DT, rule induction, NB, and generalised linear model, were applied to the ADNI dataset to categorise the five stages of AD. The most distinguishing attribute for each stage of AD was determined. Existing ML models were compared without evaluating the ensemble models. The investigation [8] assessed the accuracy of national population-centred organisational medical data in predicting the potential prevalence of AD. Using three ML techniques, including RF, SVM, and LR, they accurately predicted the possible onset of AD. Without testing further stand-alone and ensemble ML models, just three ML approaches were evaluated. One performance metric was used to evaluate performance.

The authors of [9] compared the ML algorithms J48, NB, RF, and MLP for the detection of "dementia." They saw that J48 performed more effectively. The attribute reduction was carried out using the CFSSubsetEval method. The only performance indicator used to evaluate the ML models was Acc. Ensemble models were not established. Demographic and racial characteristics were not included in the classification of AD. There were insufficient records in the dataset to draw any significant conclusions. The authors of [10] conducted tests on the OASIS dataset. They discovered that SVM outperformed LR, DT, and RF. Multiple performance indicators were used to evaluate the models. Several crucial age, demographic, and ethnicity-related characteristics were overlooked. The ensemble methods were not evaluated. Moreover, the dataset examined was small. In addition, no feature selection was conducted.

The research [11] compared LR, NB, DT, and RF models with the ADNI dataset. Based on the performance criteria Acc, Prec, Sens, and FS, the LR method was determined to be superior. There were no ensemble models considered. In [12], researchers predicted AD using k-NN, AB, SVM, LR, DT, and RF classifiers. On the OASIS dataset, they noticed that the RF classifier performed better. In their work, however, no unique methodology was proposed. Hybrid / Ensemble models were not developed. The models were evaluated using just one performance indicator, namely accuracy.

Based on psychological characteristics, [13] compared SVM and DT for AD prediction using SVM and DT. With a 2% improvement, they discovered that SVM outperformed DT. Only two ML models were evaluated. There was only one accuracy metric used to evaluate the models. Ensemble models were not established. There was no feature selection performed. The researchers [14] conducted studies using 3D Magnetic Resonance (MR) images of the brain to detect AD. Utilizing the technique of principal component analysis (PCA), feature extraction was performed. Using LR, NB, SVM, and AB models, AD and healthy controls were classified using a binary classification scheme. No ensemble model was developed, though. The performance of the models was evaluated using only the Acc metric. There was no multiclass categorization done.

Table 1 is a summary of the work done in the field of AD detection using ML approaches.

Table 1.

Summary of works for AD classification using ML techniques

Related work	ML Algorithms used	Dataset	Observations
[5]	DT, RF, SVM, GB	OASIS	Ensemble methods not used Feature selection not performed Fewer records (436) Multiclass classification was not performed
[6]	LR, SVM, GB, RF, Multilayer perceptron, Recurrent neural network	ADNI	Demographic features not considered Ensemble methods not used Feature selection not performed
[7]	k-NN, DT, rule induction, NB, generalized linear model	ADNI	Ensemble methods not used Feature selection not performed
[8]	RF, SVM, LR	Korean National Health Insurance Service database	Only 3 ML methods used Ensemble methods not developed Only Acc metric used for performance evaluation Benchmark dataset not used for experimentation
[9]	J48, NB, RF and Multilayer Perceptron	OASIS	Only Acc metric used for model assessment Demographic and ethnicity features not considered Small size of dataset (436) Ensemble models were not proposed Multiclass classification was not performed
[10]	SVM, LR, DT, and RF	OASIS	Age, demography, ethnicity features not considered Feature selection not performed Ensemble methods not developed Only 4 ML methods tested Multiclass classification was not performed
[11]	LR, NB, DT and RF	ADNI	Only 4 ML models used Feature selection not performed Ensemble methods not developed
[12]	k-NN, AB, SVM, LR, DT, and RF	OASIS	No novel methodology proposed Hybrid / ensemble models not developed Only Acc metric used for model assessment Small sized dataset
[13]	SVM and DT	ADNI	Only 2 ML models compared Only Acc metric used for model evaluation Ensemble models were not tested Feature selection not performed
[14]	LR, NB, SVM, AB	OASIS	No ensemble method proposed Only Acc metric used Multiclass classification was not performed
[15]	BayesNet, linear discriminant classifier, quadratic Bayes normal classifier and Kernel Support vector machine	ADNI	Feature selection not performed
[16]	DT, k-NN, NB, SVM, Artificial Neural Network	SKN Medical College and General Hospital	Benchmark dataset not used for experimentation Less data used (150 records) Multiclass classification was not performed Ensemble method is not proposed
[17]	k-NN	Smt. Kashibai Navale General Hospital and Research Centre, Pune (India)	Only 1 ML model is used No comparison performed No ensemble model proposed No benchmark / standard dataset used for trials
[18]	No ML model used. Wander detection technique is used	Bengali language	Smart phone app developed No analytics performed

The following points highlight the limitations in the related work and how they have been overcome in our work.

• In most of the works, ensemble ML techniques have not been proposed. In our work, ensemble techniques have been developed that perform superior to other techniques. Specifically, the proposed ensemble method named the LRFB model performed better than other ML models.

• In most of the efforts, feature selection was not performed. In our work, we used the RF method to choose features by figuring out how important each one was.

• In several works, fewer input features were considered to be sufficient to draw any meaningful conclusions. In our work, nine important features were considered for AD classification.

• In a few efforts, binary classification (AD/healthy) was performed. In our work, multiclass classification (AD, LMCI, and CN) was performed.

• Multiple performance metrics were not employed in many studies. In this work, multiple performance metrics such as Acc, Rec, Prec, and FS have been used to evaluate the ML models.

Proposed framework and system design

This section looks at the system's framework and architecture, which were used to build the final solution, which is ML-based AD prediction. The design of the proposed framework is depicted in Fig. 1.

Fig.1.

Proposed framework for AD prediction using ML algorithms

As shown in Fig. 1, the proposed structure consists of 2 stages as explained in (i) and (ii) below.a) 1. Stage-1: Data Pre-processing:

The input to the framework was the unprocessed ADNI data, comprising of 18 input features and a target attribute. The target attribute consisted of 3 classes, namely LMCI, AD, and CN. In stage-1, pre-processing of the input was performed using EDA, feature selection, 1-hot encoding, and normalization. Each of these is explained in sections (a), (b), (c), and (d) below.a) EDA:

The dataset was subjected to EDA to examine the distribution of records. The dataset was discovered to contain three numerical characteristics (AGE, MMSE, and PTEDUCAT) and fifteen categorical features. There were more men in the data set. The characteristics (AGE, MMSE, and PTEDUCAT) were slightly right skewed. 304 samples for the target attribute belonged to the LMCI class. The distribution of the categorical characteristics "imputed genotype" (IG), "APOE Genotype" (APOE), "PTGender" (PTG), "APOE4", "PTETHCAT," and "PTRACCAT" was analysed. It was determined that ethnicity and race were distinct characteristics. The distribution of Hispanics, African Americans, whites, and Asians is not identical to the ethnic and racial composition of the United States as a whole. The APOE Genotype with the number 3 in the first position appears to be the overwhelming majority in 3,3 and 3,4. There were more men in the study than women. The three quantitative characteristics (AGE, MMSE, and PTEDUCAT) were slightly right skewed. The maximum number of entries (304) for the target feature corresponded to the LMCI target feature. This could be because the people being tested weren't just any old people; they were people with known cognitive problems.b) Feature Selection:

The feature selection phase's goal was to select significant characteristics by evaluating the relevance of each feature using RF. In RF, the "Gini importance" method was used to select the most relevant characteristics. The Gini importance (or mean decrease impurity) is obtained from the RF structure. The RF is composed of DTs. Each DT consists of a collection of "internal nodes" and "leaves." Within each "internal node," the selected characteristic determines how the data set is partitioned into two unique groups of related replies. The properties for the "internal nodes" are selected based on a standard, which for categorization purposes may be "Gini impurity" or "information gain." The decrease in the separated impurity can be evaluated, and the attribute with the greatest decrease is chosen as the "internal node." The average decrease in impurity can be calculated for each attribute. The average of all the trees in a forest is the statistic for "feature significance." Here, the relative importance of the crucial calculation is evaluated. The primary advantage of this technique is its faster calculation speed, as all the essential significant values are computed during RF training [19]. Table 2 [6] gives a full look at the selected attributes and the feature priority levels for each one.c) Conversion of categorical attributes utilizing “One-Hot Encoding”:

Table 2.

Description of selected features with importance values

Selected Feature	Feature Importance Value
MMSE	0.299198
AGE	0.299198
PTEDUCAT	0.149885
GENDER (PTG)	0.018829
APOE Genotype	0.017209
RACE (PTRACCAT)	0.014797
Ethnicity (PTETHCAT)	0.013801
APOE4 Genotype (APOE4)	0.012114
Imputed Genotype (IG)	0.010787

MMSE and AGE have the highest important values, followed by PTEDUCAT, PTG, APOE Genotype, RACE, Ethnicity, APOE4, and IG. The following features ("Subject", "RID", "Image Data ID", "Modality", "Visit", "Acquired Date", "DX.bl", and "Exam Date") were excluded from the dataset due to low attribute significance values. The three classes that make up the DX Codes for Submission target attribute are AD, LMCI, and CN

Following the selection of important qualities, the transformation of "categorical" attributes utilizing "One-Hot Encoding" was the next step. The "categorical characteristics" (IG, APOE, PTG, APOE4, PTETHCAT, and PTRACCAT) were encoded with a single hot encoding to be compatible with ML methods. In this procedure, attributes of a category were converted into binary values.d) Data Split:

Using the 80–20 rule, the ADNI dataset was divided into "training" and "testing" datasets (80% training data and 20% test data). The training records consisted of 502 records (representing 80% of the overall data), whereas the test records comprised 126 records (representing the remaining 20% of the data). k-fold cross validation was utilised to further divide the training data when training ML algorithms. The k-fold cross validation was used to improve the predictive accuracy of the ML models on the small dataset.e) Normalization using Min–Max Scaling:

In this method, the characteristics are standardised between 0 and 1. This was done to prevent larger-valued features from dominating smaller-valued characteristics. If features have varying values or scales, then greater values may dominate smaller ones, and the model may not fit well. This results in a bias. In the past, "feature-wise normalization" methods like "Min–Max" scaling were used to solve this problem [20]. Equation (1) shows the formula for normalization of data using Min–Max scaling.

$$f_s=\frac{f-\min (f)}{\max \left(f\right)-\min (f)}$$ 1

where, ‘$f$’ represents the original feature value, $f_s$ represents the scaled feature value; ‘$\min$’ represents minimum; and ‘$\max$’ represents maximum.ii) Stage-2: Classification:

After calculating the significance of each attribute, the information was separated into "training" and "test" data. The AD data was used to train the ML models. The trained ML models were then put to the test on the testing data. The results were evaluated utilizing test data, and the ML model with the best performance was selected. After experimenting with the characteristics, we were able to find the 9 attributes with the best accuracy. The experiment included the execution of LR, RF, k-NN, AdaBoost, NB, MLP, SVM, GB, XGB, and DT in addition to additional ensemble approaches. With all possible combinations of LR, RF, k-NN, AB, NB, MLP, SVM, GB, XGB, and DT, 1024 ensemble models were created. The optimal solution was LR + RF + GB, often known as Logistic Random Forest Boosting (LRFB). The LRFB model is described in depth in subsection (A) below.

A) Proposed LRFB for AD classification

The LRFB ensemble model, which was proposed earlier, is discussed in this section. This model incorporates the LR, RF, and GB classifiers. The proposed LRFB classification model for detecting AD is shown in Fig. 2.

Fig.2.

Proposed LRFB Ensemble Model for AD classification

As depicted in Fig. 2, the designed LRFB ensemble model consists of LR, RF, and GB models. Several ML algorithms and ensemble algorithms were tested. The best outcome was achieved using the proposed LRFB model. The ideal LR model was combined with the optimum RF and GB models using this method. The optimal LR model had a cost of 10, the optimal RF model had a B value of 14, and the optimal GB model had a WC value of 100 and a learning rate of 2. The LR, RF, and GB models used the ADNI training dataset as input. The input ADNI characteristics (× 1 to × 9) were first allocated random weights in the LR model (w1 to w9). Using a cost function and weights, a trained LR model was developed [21]. This trained LR model was then given the ADNI test dataset. The projected result fell into one of three classes: AD, LMCI, or CN. After sampling and bagging, each of the ADNI training data was provided to the decision trees in the RF model (DT1 to DTB). Utilizing majority voting and multiple decision trees, the trained RF model was created [22]. The ADNI test results were then fed into the RF model that had been trained. The trained model for the GB was created utilising weak classifiers ranging from 1 to B and majority voting [23]. The final output of the ensemble model was determined using either hard or soft voting. With "hard voting," the projected class is the class with the most votes, i.e., the class with the maximum likelihood of being predicted by every classifier. In "soft voting," the predicted class corresponds to the prediction based on the mean probability of the class [24]. The best result was achieved by hard voting.

Hard Voting classifies input data based on the average of the predictions produced by numerous classifiers. Hard Voting classifies input data based on the average of the predictions generated by multiple classifiers. The evaluation of "majority voting" varies based on whether the weights connected with the several classifiers are comparable. When equal-weighted majority voting is utilised, the expected mode of the class is decided. Most classes predicted by LR, RF, and GB were selected as the final predicted classes by hard voting. For example, if the LR and RF models predicted AD class and the GB model predicted LMCI class, AD was chosen as the final output of LRFB since the majority of classifiers (LR and RF) predicted AD class. We utilised majority voting with equal weights in this work.

Equation (2) shows the formula for predicting the final output for the proposed LRFB method

$$\widehat{y}={\mathit{MaxArg}}_i\sum _{j=1}^n{\mathit{Wgt}}_j{x}_j({\mathit{LR}}_j\left(x\right)\cup {\mathit{RF}}_j\left(x\right)\cup {\mathit{GB}}_j\left(x\right)=i)$$ 2

where:

• $\widehat{y}$ predicted class (AD/MCI/CN) of the ensemble LRFB method

• $n$ Number of training records of ADNI dataset

• ${\mathit{Wgt}}_j$ Weight of the j^th classifier

• $x_j$ Input feature j

Since we have used majority voting based on equal weights, the term ${\mathit{Wgt}}_j$ was taken equally. The Eq. (2) basically represents that the final predicted output ($\widehat{y}$) represents the majority voting result of LR, RF, and GB models.

The proposed LRFB technique for AD prediction is shown in Algorithm 1.

The ADNI LONI dataset is read via the Read-data () function on line 3. The data is pre-processed from steps 4 through 7. An EDA was performed on the data 'D' at line 4 to analyse the feature distribution. Line 5 used RF to calculate feature importance. Line 6 converts categorical variables to binary values using "one hot encoding." The data was normalised using Min–Max scaling at line 7. The data was divided into a "training set" and a "test set" at line 8, and the best LR model was built from lines 10 to 19. The loop at line 11 cycles through cost function C values ranging from 0.001 to 100. The number of iterations was limited to 1000. At line 12, multiple LR models were built with varying cost function values (0.001 to 100) and optimization strategies ('newton-cg,"sag,"saga,' and 'lbfgs'). They were explored because these three optimizer techniques were suitable for multiclass issues. "newton-cg" produced better results. Since the ADNI dataset was limited, kcv with k = 5 was applied to the training data at line 13 with "accuracy" as the metric. The average cross validation precision was determined at line 14. Lines 16 and 17 saved the best score (accuracy) and cost parameter (best-param) as LR parameters. At line 20, the cost parameter from the training dataset was used to build the best LR model.

The optimal RF model was developed at lines 22 through 37. The outermost for loop at line 22 iterates through 'M' trees, the second for loop at line 23 iterates through the number of features 'd' (1–9) and the innermost for loop at line 24 iterates through the tree's maximum depth 'm' (1–9). Multiple RF models were created at line 25 using different parameter values for M, d, and m. At line 26, kcv with k = 5 and "accuracy" as the metric was applied to the training data. At line 27, the mean accuracy of cross validation was determined. The best values of accuracy (score), trees (M), features (d), and depth (m) were stored from lines 29 to 32. At line 37, the optimal parameters (M, d, and m) and training data were used to build the best RF model. From lines 39 through 51, the superior GB model was constructed. The for loop at line 39 iterates through 1–100 weak classifiers (WC). The inner for loop at line 40 cycled through learning rate (L_r) values ranging from 0.001 to 2. At line 41, many GB models were constructed with varying values of W_C and L_r. At line 42, k-fold cross validation (k = 5) was performed using Acc. The average cross validation accuracy was computed at line 43. The best scores, number of poor classifiers, and learning rate are listed on lines 45 to 47. The optimal GB model was then built at line 51, utilising the optimal parameters (number of weak classifiers and learning rate).

The ensemble model LRFB is built at line 54 using the optimal models and voting scheme. We used both hard and soft voting to assess the ensemble model. Through majority voting, the best result was obtained. At line 56, the test data (TstX) were applied to the trained LRFB model (LRFB), and the predicted class was saved in the variable Pred-Class.At line 56, the trained LRFB model (LRFB) was applied to test data (TstX), and the predicted class was stored in the variable Pred-Class. The performance metrics Acc, Rec, Prec, and FS were calculated for the test set from lines 57–60. The Prec, Rec, and FS were calculated using several average approaches, including "micro," "macro," and "weighted." In "micro," the metrics were estimated by computing the total number of "true positives," "false negatives," and "false positives" (FP). In "macro," each class's metric is computed and their "unweighted" average is determined. In "weighted," metrics are generated for each class, and their weighted average is calculated using "support" (the number of "true instances" for each class) [25]. The best result was obtained for "macro" in this research.

Experimental setup and results

This section provides a summary of the research undertaken and the results gained for AD prediction using the ADNI dataset. The software developed was done with Python 3.7, NumPy, pandas, pycaret, and sklearn modules on the Spyder Anaconda Platform. NVIDIA Tesla K80 hardware was used in the configurations (any CUDA compatible GPU with a minimum compute capability of 3.0 can also be used). In this study, trials were conducted utilising the various ML algorithms. Table 3 illustrates the ML classification models utilised for AD classification along with the associated parameters. Each model was assessed by varying the parameters listed in Table 3.

Table 3.

ML models with parameter tuning for AD prediction

Models	Tested Parameters with description	Parameters yielding best result
LR	kcv technique with k = 5 was utilized to select the best parameters cost (C) value from 0.001 to 100 Solvers such as “newton-sg”, ‘sag’, ‘saga’ and ‘lbfgs’ since they are suited for multiclass problems	C = 10 Solver = “newton-sg”
RF	kcv method was applied for selecting the parameters (number of trees—M, and features) M varied from 2 to 15 Features varied from 1 to 9	M = 14 Features = 9
k-NN	Number of nearest neighbors (‘k’) varied from 1 to 100 After k = 4, accuracy started decreasing	k = 4
MLP	Optimizer functions tested were “adam”, “lbfgs”, and “sgd” “lbfgs” represents an optimizer belonging to the category of “quasi-Newton” techniques. “sgd” means “stochastic gradient descent”. “adam” implies “stochastic gradient-based optimizer” [25] Different activation functions such as ‘identity’, ‘logistic’, and ‘tanh’, ‘relu’ [26] were tested out Different “hidden layers” with different “neurons” in every layer was tested	Optimizer = ” adam” Activation Function = ” tanh” 2 hidden layers with 5 neurons in 1st layer and 2 neurons in 2nd layer
SVM	Grid search was performed to select the best parameters Kernel functions tested with radial basis function (RBF), polynomial (poly), linear, and sigmoid functions hyperparameter ‘C’ which is used to control the error [27] was varied from 0.001 to 50 parameter ‘γ’ (Gamma) is used only in case of Gaussian RBF kernel. It decides how much curvature is required in a decision boundary [28]. It was varied from 0.001 to 2	Kernel = ” RBF” C = 10 γ = 0.001
AB	kcv technique [29] with k = 5 was employed for selecting the best parameters number of trees ‘M’ was varied from 2 to 15 learning rate (Lr) from 0.0001 to 1	M = 6 Lr = 1
NB	Gaussian NB [30] was employed since it is suitable for continuous attributes and the ADNI dataset was numeric in nature	Gaussian NB
GB	Number of weak classifiers (WC) [31] was varied from 2 to 100 Learning rate was varied from 0.001 to 10	Weak Classifiers = 100 Learning Rate = 2
XGB	The booster parameter [32] was varied using “gbtree”, “gblinear”, and “dart” The maximum depth was varied from 1 to 10 The sampling method parameter was tested with “uniform”, and “gradient based”	Booster = “gbtree” Maximum depth = 6 Sampling method = “uniform”
DT	The kcv with k = 5 was utilized to determine the best parameters Tree depth was varied from 1 to 10 Impurity measures “Entropy”, “Gini-Index”, “Information Gain” were used	Tree Depth = 2 Impurity Measure = ” Gini-Index”

The best ML models from Table 3 were ensembled and tested. All the combinations of the above ML models were tested, i.e., 210 = 1024 ensemble models were experimented with. Of them, the LRFB gave the superior result. Since this work involved dataset having 3 classes, multiclass accuracy metric shown in Eq. (3) was used.

$$Acc=\frac{{\sum }_{i=1}^{\mathit{Cls}}\frac{{\mathit{tp}}_i+{\mathit{tn}}_i}{{\mathit{tp}}_i+{\mathit{tn}}_i+{\mathit{fp}}_i+{\mathit{fn}}_i}}{\mathit{Cls}}$$ 3

where, $\mathit{Cls}$ is the number of classes, ${\mathit{tp}}_i$ is the true positive of ith class, ${\mathit{tn}}_i$ is the true negative of ith class, ${\mathit{fp}}_i$ is the false positive of ith class, ${\mathit{fn}}_i$ is the false negative of ith class. It gives the the “average per-class effectiveness” of the classifier. Figure 3 shows the performance of the ML models with respect to the Acc metric.

Fig. 3.

Performance of proposed LRFB with respect to Accuracy

The proposed LRFB model performed with accuracy improvements ranging from 2.39 to 55.18% over other single ML models and ensemble models. Likewise, the proposed LRFB model performed with recall improvements ranging from 0.44 to 52.88%, precision improvements from 0.24 to 50.54%, and F1-score improvements from 2.7 to 60.06% over other stand-alone ML and ensemble models. Table 4 shows the confusion matrix for the LRFB model on the test data.

Table 4.

Confusion Matrix of the proposed LRFB model

		Predicted class
	Classes	AD	LMCI	CN
Actual class	AD	12	0	8
	LMCI	0	30	13
	CN	2	10	51

From Table 4, it can be determined that the proposed LRFB approach accurately identified 12 occurrences as AD, 30 instances as LMCI, and 51 instances as CN. But 8 records from the AD class were wrongly predicted to be from the CN class, 13 records from the LMCI class were wrongly predicted to be from the CN class, two records from the CN class were wrongly classified as being from the AD class, and 10 records from the CN class were wrongly classified as being from the LMCI class.

Table 5 shows the detailed findings of the ML models with respect to the performance metrics.

Table 5.

Performance comparison of the proposed LRFB with other models

ML models	Acc	Rec	Prec	FS
NB	21.01	21.01	28.62	15.01
SVM + NB	21.01	37.69	46.94	17.74
LR + NB	21.65	37.75	47.07	19.58
SVM + NB + GB	40.12	38.78	47.56	19.73
SVM + AB + NB	43.31	45.25	47.83	37.29
RF + AB + NB	44.58	49.96	51.74	39.75
SVM + NB + DT	50.95	52.86	54.3	40.81
SVM + NB + XGB	51.59	53.5	54.57	45.99
RF + k-NN + NB	52.29	54.41	54.59	48.72
AB	52.86	54.77	54.85	49.96
SVM + GB	54.14	54.85	55.39	50.34
RF + SVM + NB	54.14	56.5	55.59	50.8
GB	54.77	57.12	55.93	51.27
LR + SVM + NB	54.77	57.56	56.12	52.76
MLP + GB	54.77	57.77	57.73	53.08
SVM + AB	56.05	58.78	58.06	54.16
LR + k-NN + NB	56.05	58.94	58.16	56.72
SVM	56.67	59.35	58.41	56.8
RF + GB	57.32	59.78	58.88	57.7
LR + GB	57.96	60.63	59	57.85
SVM + XGB	59.23	61.28	59.55	58.31
LR + RF + NB	59.23	61.76	59.76	58.57
RF + SVM	60.5	61.78	60.14	59.13
LR + k-NN + SVM	60.5	61.9	60.25	59.25
SVM + AB + GB	60.5	62.95	60.41	59.8
RF + KNN + SVM	61.14	63	60.51	59.84
XGB	61.78	63.19	60.72	59.89
LR + SVM	61.78	63.56	60.87	60.72
LR + SVM + XGB	61.78	63.59	60.91	60.96
SVM + AB + XGB	61.78	63.67	61.11	61.16
SVM + DT	62.42	63.98	61.13	61.52
LR + RF + SVM	62.42	64.27	61.29	61.74
LR + SVM + AB	62.42	64.33	61.38	61.83
RF + SVM + GB	62.42	64.45	61.53	61.88
RF + SVM + XGB	62.42	64.51	61.85	61.99
k-NN	63	64.54	62.04	62.17
LR + AB	63.05	64.67	62.14	62.3
LR + XGB	63.05	64.96	62.43	62.31
RF + XGB	63.05	65.01	62.5	62.34
LR + SVM + GB	63.05	65.18	62.59	62.75
RF + SVM + AB	63.05	65.25	62.64	63.02
RF + AB + XGB	63.6	65.31	62.89	63.15
RF + NB + XGB	63.69	65.32	62.96	63.16
SVM + AB + DT	63.69	65.35	63.06	63.19
MLP + AB	63.69	65.38	63.31	63.32
RF + AB	64.33	65.48	64.25	63.57
LR + RF + XGB	64.33	65.53	64.4	63.77
RF	64.96	66.16	64.41	63.93
LR + RF + AB	64.96	66.27	64.51	64.05
MLP + k-NN	65.6	66.31	64.73	64.35
LR + k-NN + AB	66.24	66.44	64.99	64.39
LR + kNN + XGB	66.24	66.49	65.02	64.46
RF + k-NN + AB	66.24	66.52	65.02	64.63
RF + k-NN + GB	66.24	66.52	65.1	64.86
RF + k-NN + XGB	66.24	66.69	65.2	65
RF + SVM + DT	66.24	66.71	65.26	65.07
RF + XGB + DT	66.24	66.81	65.27	65.15
LR + RF + k-NN	66.87	67.01	65.5	65.27
LR + RF	67.51	67.01	65.8	65.47
LR + DT	67.51	67.5	65.83	65.7
RF + k-NN	67.51	67.53	65.85	65.7
RF + DT	67.51	67.8	65.87	66
LR + k-NN + GB	67.51	68.28	66.15	66.27
LR + SVM + DT	67.51	68.38	66.21	66.6
RF + AB + GB	67.51	68.53	66.41	66.6
RF + NB + GB	67.51	68.6	66.59	66.65
LR	68.15	68.92	66.64	66.81
RF + AB + DT	68.15	69.03	66.81	67.09
RF + NB + DT	68.15	69.21	66.81	67.5
MLP	68.78	69.25	66.81	67.55
LR + k-NN	69.42	69.93	67.77	67.61
RF + k-NN + DT	69.42	69.96	68.42	68.51
MLP + RF	69.42	70	69.14	68.71
MLP + LR	70.06	70.61	69.22	69.87
LR + RF + DT	70.7	70.7	69.42	69.91
RF + GB + XGB	70.7	71.02	71.11	69.99
DT	71.33	71.33	71.38	70.08
LR + k-NN + DT	71.33	71.88	72.5	71
RF + GB + DT	71.97	72.2	73.85	71.25
MLP + SVM	73.8	73.45	78.92	72.91
LRFB	76.19	73.89	79.16	75.61

Discussion

This section discusses the benefits of the designed LRFB model. When compared to stand-alone ML approaches, the LRFB was able to make more accurate predictions and generate superior results. The LRFB was able to restrict the variance of "model" predictions and "performance". It generated superior predictive outcomes than other models because adding a "load" decreased the "component variation of the "variance error." It increased the average "performance" of the model's durability or consistency. The following criteria emphasise the reasons for the LRFB model's higher performance.

• Improved Robustness:

In predictive modelling, models are frequently evaluated, and the best model is selected. The approach is trained on available data and then used to estimate unseen data. The average "accuracy" or "error" of the "model" provides an overview of the expected result. Since the model may perform well on some data subsets and badly on others, the mean is used to balance the model's results. Only observing the worst and best model outcomes may provide an overview of "worst" and "best" performance, but this may be insufficient for many applications [30]. The LRFB was executed numerous times on multiple subsets of "training" data, and then the prediction results were merged using summary "statistics" such as the mean. This assisted in reducing the scattering of the results produced by the technique. The mean performance was approximately similar, and the "worst" and "best" findings were closer to the mean performance. This led to the smoothing of the method’s estimated performance. The proposed LRFB model reduced the spread of errors since average performance was considered using the kcv technique.

• “Deviation” and “variance”:

Deviation and variance are the two characteristics that characterise ML model errors. The "deviation" measure describes how well the model captures the relationships between "input" and "output" data. It analyses the model's rigidity, which represents the assumption of the functional structure of "input" and "output" data that the method requires. "Variance" is the number of variances in a model's performance with respect to varied "training" data. Consideration is given to the impact of data variability. The performance "variation" and "deviation" of a model are interrelated. By raising the "variance," the "deviation" can be decreased. Similarly, variance reduction can be accomplished merely by increasing "deviation" [31, 32]. The LRFB approach was able to reduce "predictive error" variance, resulting in improved predictive performance. The LRFB model was able to reduce variance and bias, thus improving the performance. It was able to create a deeper understanding of the underlying data patterns.

Conclusion and future scope

AD is a critical neurological disorder whose early diagnosis is difficult. AD causes loss of memory, among other things. No cure exists for this disease. However, its early diagnosis may assist in the effective treatment of AD. This work focused on the development of an ensemble ML framework for the early diagnosis of AD. The ADNI dataset was used as the experimental data for this work. The work consisted of two stages. Initially, in stage 1, data pre-processing was performed, which consisted of EDA, Feature Selection, one-hot encoding, and Normalization. The EDA was performed to analyse the distribution of features in the dataset. The conversion of categorical to binary attributes was performed using the “one-hot encoding” method. “Feature selection” was performed by computing the feature importance using the RF method. Nine features out of 18 input features were selected using this method. These nine features were then subjected to normalization using Min–Max scaling. The second stage was the classification, where the ML model classified the ADNI data into three classes: AD, LMCI, and CN. Several ML models such as LR, RF, k-NN, MLP, SVM, AB, NB, GB, XGB, DT, along with ensemble models, were tested on this dataset using multiple performance metrics. By combining the LR, RF, and GB models, the LRFB ensemble model was created. With respect to accuracy, recall, precision, and F1-score, the proposed LRFB model outperformed other ML models by 55.18%, 52.88%, 50.54%, and 60.06%, respectively.

This work can be improved in the future by blending "brain MRI scans" with the "psychological parameters" to forecast AD with greater "accuracy" using DL techniques. This merging of parameters and the use of DL techniques may not only lead to superior "accuracy" but also facilitate the detection of AD during its early stages more effectively. Sophisticated DL methods may be employed on these datasets. The latest image processing techniques may be employed on image datasets.

Data availability statement

The data set is available on https://adni.loni.usc.edu/.