A brief review and scientometric analysis on ensemble learning methods for handling COVID-19

Abstract

Numerous efforts and research have been conducted worldwide to combat the coronavirus disease 2019 (COVID-19) pandemic. In this regard, some researchers have focused on deep and machine-learning approaches to discover more about this disease. There have been many articles on using ensemble learning methods for COVID-19 detection. Still, there seems to be no scientometric analysis or a brief review of these researches. Hence, a combined method of scientometric analysis and brief review was used to study the published articles that employed an ensemble learning approach to detect COVID-19. This research used both methods to overcome their limitations, leading to enhanced and reliable outcomes. The related articles were retrieved from the Scopus database. Then a two-step procedure was employed. A concise review of the collected articles was conducted. Then they underwent scientometric and bibliometric analyses. The findings revealed that convolutional neural network (CNN) is the mostly employed algorithm, while support vector machine (SVM), random forest, Resnet, DenseNet, and visual geometry group (VGG) were also frequently used. Additionally, China has had a significant presence in the numerous top-ranking categories of this field of research. Both study phases yielded valuable results and rankings.

1. Introduction

Coronavirus disease 2019 (COVID-19) has been a great challenge for the modern human society. Using smart methods to fight this disease is remarkable. Machine learning approaches have been widely explored to aid in COVID-19 detection, leveraging patterns and features in various types of data, including medical imaging, and clinical and molecular data. So far, there has been a lot of work to detect COVID-19 using machine learning methods, and some review articles or surveys have been published focusing on using machine learning for detecting COVID-19 [[1], [2], [3], [4], [5], [6], [7]]. Also, there are some scientometric studies in this field [[8], [9], [10], [11], [12]].

Detecting COVID-19 has several challenges for the current machine learning approaches, which include limited, imbalanced, noisy, and incomplete data, complex and evolving patterns, overfitting and generalization. In addition, the current machine learning approaches have insufficiencies such as low accuracy, high false-positive rates, and high inter-reader variability [13,14]. Ensemble learning is a powerful approach that combines multiple machine learning models to make predictions or classifications. It has gained significant popularity and has been successfully applied to various domains, including medical diagnosis such as COVID-19 detection. Ensemble learning methods seem to be more effective than the other machine learning methods in detecting COVID-19 [9].

Ensemble methods can leverage the strengths of individual models while mitigating their weaknesses by combining multiple models, each trained on different subsets of data or using different algorithms. This leads to improved accuracy, robustness, and generalization performance [13,15]. Ensemble methods also provide mechanisms for uncertainty estimation, which is crucial in their medical applications such as detecting COVID-19, where the consequences of false positives or false negatives can be severe. These advantages make ensemble methods a valuable tool in the fight against the pandemic, enabling more reliable and effective diagnosis of COVID-19 cases.

Until now, many articles have investigated the application of ensemble learning for detecting COVID-19; however, it seems that none of them have undergone a scientometric analysis and have not been included in a brief review. Hence, we did this research with a combined method, i.e., scientometric analysis and brief review, thereby overcoming the constraints of each method, resulting in improved and more reliable outcomes.

This paper has two sections based on the proposed combined method. The first section presents a brief review of the related published works, summarizing their research methods, datasets, and best performances. The second section uses scientometric methods to provide further insights into the authors and research networks, keyword co-occurrence rates, plus top rated publications, authors, countries, and funding organizations. By examining both the summary of previous works and scientometric analyses, we can gain a deeper understanding of this field.

The research questions were:

• ⬥What useful information can be extracted from reviewing and analyzing the articles that used the ensemble learning method to learn more about COVID-19?
• ⬥Are certain algorithms more widely used in ensemble learning for detecting COVID-19 than others? Which algorithms are more common?
• ⬥Who are the most influential authors, and what are the leading countries, universities, and major funding entities in this field?
• ⬥Do the results obtained from the scientometric method for determining the most widely used algorithms correspond with those obtained from the brief review?

This research can provide these contributions:

• ⬥It can provide valuable information and reveal interesting patterns among the studies carried out in this research area.
• ⬥Usage of co-word analysis aids researchers in quickly grasping the most common methods for COVID-19 analysis using ensemble learning, corresponding datasets, and the highest achieved efficiency.
• ⬥The scientometric analysis uncovers the leading influential authors and countries, key sponsors, and highlights the top effective publishers and journals that have publications in this area of research.
• ⬥Usage of co-word analysis on abstracts and review section results, the hybrid approach can provide significant insight into the techniques and algorithms employed in this field.

The rest of this text is structured as follows: a) section 2 explains the research method, b) section 3 presents a brief overview of the related works, c) section 4 gives a scientometric-bibliometric analysis of the extracted metadata, d) section 5 presents a discussion, e) section 6 includes conclusion and f) section 7 has suggestions for future research.

2. Research method

This study aimed to provide insight into the current published research that employed ensemble learning approaches for detecting COVID-19. This research had two phases wherein the related works were briefly reviewed and compared, followed by a scientometric and bibliometric analysis.

It is widely recognized that Scopus is one of the most comprehensive databases available [16]. Compared to other databases, Scopus seems to provide the widest coverage of documents [17,18]. This database contains widest the coverage of documents, peer-reviewed journals and most recent publications [[19], [20], [21], [22]].

While Scopus is a valuable source for conducting literature reviews and obtaining research insights, it is important to consider its limitations and potential biases, particularly when it comes to the coverage and quality of articles in specific domains such as COVID-19 research. Some of the limitations and potential biases of the Scopus database include incomplete coverage, time lag, language bias, quality and peer review variations, publication bias, and access restriction [23,24]. While all databases have their limitations, Scopus boasts the most extensive coverage among them. As such, it has been selected as the preferred database for this research.

To conduct our scientometric and review study, we ran the following queries on keywords and abstracts in the Scopus database: (“ensemble learning” + COVID-19) OR (“ensemble deep learning” + COVID-19) OR (“ensemble model” + COVID-19).

As mentioned, in addition to using it on the keywords, this query was also applied to the abstracts to obtain a greater degree of coverage. Since the validity of the results of papers published in indexed journals is generally higher than those published in conference papers, the above query was applied only to journal papers to have a greater level of confidence. A total of 142 articles matching these characteristics were retrieved on August 1, 2022.

3. Comparing related works

Based on an initial scientometric review in the Scopus database, the peer-reviewed journals with the greatest number of articles published in this research field were identified. Then, journals that had published more than five articles were selected amongst them. Next, the articles that used an ensemble learning method to detect COVID-19 were selected. The best performance, base method, and dataset were extracted from each article. Since presenting all the information in one table would make it hard to read, they were presented in several tables.

Throughout the tables, there was a brief description of the type of ensemble learning method used. In the best performance column, the highest value obtained from a performance criterion was presented. In most of the articles, accuracy was the criterion of performance, but in a few articles, it was also the area under the curve (AOC) or error rate. The dataset field contained a summary of the datasets used in each article. Table 1 shows the related works published by Springer Science and Business Media and Springer. Among the various applied ensemble methods, ResNet, DenseNet, MobileNet, visual geometry group (VGG), and convolutional neural network (CNN) stood out.

Table 1.

Related published research in Springer.

#	Title	Best Performance	Ensemble Method	Dataset
1	Analysis of origin, risk factors influencing COVID-19a cases in India and its prediction using ensemble learning [25]	The highest accuracy: 84.37%	Naive Bayes, Decision tree, SVMb, KNNc, Neural Network	Kaggle, including 5000 samples from the first and another 5000 samples from the second peaks of COVID-19 from India
2	Automatic COVID-19 detection from X-ray images using ensemble learning with convolutional neural network [26]	The highest accuracy: 91.62%	CNNd - DenseNet201 Resnet50V2 Inceptionv3	Two open sources, 1004 chest X-ray images of patients in Europe
3	COVID_SCREENET: COVID-19 Screening in Chest Radiography Images Using Deep Transfer Stacking [27]	The highest accuracy: 100%	model by transfer stacking approach	7725 chest X-ray images from three hospitals in India. Sources: Kaggle, Mendeley, and sirm.com
4	Design ensemble deep learning model for pneumonia disease classification [28]	The highest accuracy: 95.05% F-measure: 94.84%	InceptionResNet_V2 ResNet50 MobileNet_V2	Two datasets from the University of San Diego, California: a CTe scan dataset with 5856 images and a COVID-19 chest X-ray dataset with 231 images
5	Novel deep transfer learning model for COVID-19 patient detection using X-ray chest images [29]	The highest accuracy for multiclass: 99.21% for two-class: 98.95%	EfficientNet GoogLeNet XceptionNet	2869 chest X-ray images from Kaggle, Mporas, and Naronglerdrit
6	COVIDScreen: explainable deep learning framework for differential diagnosis of COVID-19 using chest X-rays [30]	The highest accuracy: 98.67% F1: 100%, 98%, and 98% for COVID-19, normal, and pneumonia, respectively.	VGGf-19, VGG-16 ResNet-50 DenseNet161 DenseNet-169	Normal and pneumonia samples extracted from the open-source NIHg chest X-ray dataset used in the RSNAh pneumonia detection challenge on Kaggle
7	Stacking Deep Learning for Early COVID-19 Vision Diagnosis [31]	The highest accuracy: 98.6%	MobileNet, InceptionResNetV2 ResNet50	500 chest X-ray images (unknown source)
8	Real-time internet of medical things framework for early detection of Covid-19 [32]	The highest accuracy: 95.3%	Random Forest (RF), Gradient Boosted Tree (GBT)	278,848 records in two categories of COVID-19 patients and healthy individuals in Israel
9	Inverted bell-curve-based ensemble of deep learning models for detection of COVID-19 from chest X-rays [33]	The highest accuracy: 99.54%	DenseNet-161, ResNet-8, VGG-16	Two public datasets: 1) the COVID-19 radiography dataset and 2) the chest X-ray images from IEEE
10	Decision and feature level fusion of deep features extracted from public COVID-19 datasets [34]	The highest accuracy: 90.84%	MobileNetV2, VGG16, ResNet50 and ResNet101, NasNet, InceptionV3, Xception	A database called DB1 containing 125 chest X-ray images of COVID-19 cases and 1000 images of non-COVID-19 cases. 353 new COVID-19 scans were added to the DB1 database (named as DB2 database). DB3 database contained 113 COVID-19 scans in addition to DB2 database
11	Adaptive UNet-based Lung Segmentation and Ensemble Learning with CNN-based Deep Features for Automated COVID-19 Diagnosis [35]	The highest accuracy: 97.09%	SVM, Naïve Bayes Autoencoder	Chest X-ray images dataset including 481 and 183 records from the IEEE8023 and GitHub repositories
12	A multichannel EfficientNet deep learning-based stacking ensemble approach for lung disease detection using chest X-ray images [36]	The highest accuracy: 98%	Random forest, SVM, logistic regression	Mendeley-data-V3 dataset, including 4676 records of healthy individuals and 2004 records of people infected with COVID-19
13	A deep learning algorithm using CT images to screen for Coronavirus disease (COVID-19) [37]	The highest accuracy: 89.5% (internal validation) 79.3% (external validation)	Using transfer learning (unknown models)	1065 CT scan images collected from hospitals in China
14	Automated detection of COVID-19 using ensemble of transfer learning with deep convolutional neural network based on CT scans [38]	The highest accuracy: 85.2%	EfficientNetB0 EfficientNetB3 EfficientNetB5 nception_resnet_v2 Xception	349 CT scan images of COVID-19 positive cases and 397 COVID-19 negative cases from Tongji Hospital, Wuhan, China
15	Densely connected convolutional networks-based COVID-19 screening model [39]	The highest accuracy: 98.83%	Densely connected convolutional networks, ResNet152V2, VGG16	Chest X-ray images including 2373 COVID-19 cases in Wuhan, 2890 cases of pneumonia, 3193 cases of pneumonia from North America, and 3038 cases of healthy individuals
16	Deep-LSTM ensemble framework to forecast Covid-19: an insight to the global pandemic [40]	The highest accuracy: 97.59%	Convolutional LSTMi, bi-directional LSTM	Unclear dataset from India
17	Internet of Medical Things-Based COVID-19 Detection in CT Images Fused with Fuzzy Ensemble and Transfer Learning Models [41]	The highest accuracy: 99.15%	MobileNetV2, Sugeno fuzzy integral	CT scan images of 650 cases of pneumonia, COVID-19, and healthy individuals from the National Center for Biological Information, China
18	Automatic detection of COVID-19 from chest CT scan and chest X-Rays images using deep learning, transfer learning and stacking [42]	The highest accuracy: 99.75%	DenseNet169, VGG19	Five datasets: a) 746 COVID-19 CT scan images from Github b) 579 chest X-ray images from Github c) 12058 CT scan images from Github d) 2541 chest X-ray images from Kaggle e) 2482 CT scan images from Kaggle
19	Classifying chest CT images as COVID-19 positive/negative using a convolutional neural network ensemble model and uniform experimental design method [43]	Highest accuracy: 96.7%	A CNN ensemble model including VGG-19, ResNet-101 and DenseNet201 models, and Inception-v3 and Inception-ResNet-v2	612 CT scan images, 309 COVID-19 and 303 healthy cases

^aCoronavirus Disease 2019.

^bSupport Vector Machine.

^cK-Nearest Neighbors.

^dConvolutional Neural Network.

^eComputed Tomography.

^fVisual Geometry Group.

^gNational Institute of Health.

^hRadiological Society of North America.

ⁱLong Short-Term Memory.

Ensemble learning approaches have advantages over individual machine learning algorithms in detecting COVID-19 [44]. They can improve accuracy, handle uncertainty, enhance robustness, and capture diverse patterns. Individual algorithms, on the other hand, maybe computationally efficient, easier to interpret, and provide a clearer understanding of the underlying patterns in certain cases. However, they may lack the ability to capture complex relationships or generalize well to new data, especially in the presence of limited or noisy data.

Table 2 shows the related works published by Elsevier. ResNet, DenseNet, VGG, CNN, and support vector machine (SVM) have been at the forefront of the various machine learning methods used.

Table 2.

Related published research in Elsevier.

#	Title	Best Performance	Ensemble Method	Dataset
1	CoVNet-19: A Deep Learning model for the detection and analysis of COVID-19 patients [45]	Highest accuracy: For three-classification 98.2% Binary classification 99.71%	Two-stage stack ensemble model, which uses VGG19 and DenseNet121 in the first stage, and classification by SVM in the second stage.	Five datasets: a) COVID-19 radiography database from Kaggle b) Chest X-ray images (pneumonia) from Kaggle c) From the GitHub repository: COVID-chestxray-dataset d) From GitHub, the COVID-19 chest X-ray dataset e) COVID-19 X-ray dataset from Kaggle
2	Hybrid ensemble model for differential diagnosis between COVID-19 and common viral pneumonia by chest X-ray radiograph [46]	Highest accuracy: 98.64%	Three-stage dual ensemble model: AlexNet for feature extraction, Relief algorithm for feature selection, and SVM for final classification	1743 CXRa images. The normal CXRs and viral pneumonia CXRs were obtained from the NIH Chest X-ray database, and the Covid-19 CXRs were collected from GitHub
3	The ensemble deep learning model for novel COVID-19 on CT images [47]	Highest accuracy: 99.5%	The model is called EDL-Covid consists of three models AlexNet, GoogleNet, ResNet	2933 chest X-ray images from public databases
4	Ensemble learning model for diagnosing COVID-19 from routine blood tests [48]	Highest accuracy: 99.8%	two-stage stack model: extra trees, RF, and LR were used in the first stage, and the XGBoost class was used in the second stage	56,444 records (blood tests) obtained from the Albert Einstein Hospital in Brazil, including 559 positive results for COVID-19
5	TSRNet: Diagnosis of COVID-19 based on self-supervised learning and hybrid ensemble model [49]	Highest accuracy: 99.8%	A hybrid ensemble model (TSRNet), new pre-training method based on transfer learning with self-supervised learning, a new CNN based on attention mechanism and deep residual network (RANet) for feature extraction	Four datasets: a) ImageNet b) COVID-19 dataset (including 1252 COVID-19 and 1229 lung CT scan images from healthy individuals) c) Use of lung nodule analysis (LUNA) as a source of unlabeled CT scan images, including 1000 images d) Transfer learning to evaluate the effect of the difference between the source and target domains
6	Complex features extraction with deep learning model for the detection of COVID19 from CT scan images using ensemble based machine learning approach [50]	Highest accuracy: 99.37%	CLAHEb in pre-processing step to increase the quality of the images, a new CNN: Gaussian Naïve Bayes (GNB), SVM, DTc, LRd and RFe	Collection of CT scan data for SARS-CoV-2f from Kaggle, original dataset is CT scan from São Paulo Hospitals
7	A multi model ensemble based deep convolution neural network structure for detection of COVID19 [51]	Highest accuracy: 88.98%	deep CNN, namely VGGNet, GoogleNet, DenseNet, and NASNet	Kaggle's chest X-ray images from Indore Hospital
8	An efficient hardware architecture based on an ensemble of deep learning models for COVID -19 prediction [52]	Highest accuracy: 98%	Five deep learning models namely ResNet, Fitness, IRCNNg and Primary Recurrent Convolutional Neural Network	A database of chest X-ray images for COVID-19 positive cases along with images of normal and viral pneumonia

^aChest X-ray.

^bContrast Limited Histogram Equalization.

^cDecision Tree.

^dLogistic Regression.

^eRandom Forest.

^fSevere Acute Respiratory Syndrome Coronavirus 2.

^gInception-Recurrent Convolutional Neural Network.

Table 3 shows the related works published by Nature. It seems that CNN remains at the forefront of the various ensemble learning approaches.

Table 3.

Related published research in Nature.

#	Title	Best Performance	Ensemble Method	Dataset
1	Spatio-temporal prediction of the COVID-19 pandemic in US counties: modeling with a deep LSTM neural network [53]	Root mean square error (RMSE) improvement	LSTM	COVID-19 cases, deaths, and foot traffic at the county level in 33 weeks obtained from the Center for Systems Science and Engineering at Johns Hopkins University and SafeGraph's Places Schema dataset
2	Fuzzy rank-based fusion of CNN models using Gompertz function for screening COVID-19 CT-scans [54]	Highest accuracy: 99.2%	Three transfer learning-based CNN models were used, namely VGG-11, Wide ResNet-50-2, and Inception v3	SARS-COV-2 dataset, Harvard Dataverse chest CT scan dataset
3	Machine learning based early warning system enables accurate mortality risk prediction for COVID-19 [55]	Area under curve of 96.21%	Four techniques: Logistic Regression, SVM, Gradient Boosted Decision Tree, and Neural Network	2520 COVID-19 patients from two hospitals of Tongji Medical College, Huazhong University of Science and Technology, China
4	EpistoNet: an ensemble of Epistocracy-optimized mixture of experts for detecting COVID-19 on chest X-ray images [56]	Highest accuracy: 95%	A decision tree-based ensemble model consisting of two distinct expert combinations called EpistoNet, from the Epistocrac algorithm. Several deep CNNs in each cluster	2500 X-ray images including 1250 COVID-19 cases and 1250 non-COVID-19 cases
5	An ensemble learning approach to digital corona virus preliminary screening from cough sounds [57]	curve = 0.77, precision = 0.80, recall = 0.71, F1 measure = 0.75, Kappa = 0.53	Cough sound samples by splitting/separating the cough sound. CNN for modelling. Shallow machine learning, CNN, and pre-trained CNN models	Crowdsourced respiratory sounds collected to detect COVID-19. The authors used breathing and cough to distinguish COVID-19 sounds from asthma patients or healthy individuals

Table 4 shows the related works published by IEEE. Among the employed ensemble learning approaches, ResNet, DenseNet, VGG, CNN, and SVM were more common.

Table 4.

Related published research in IEEE.

#	Title	Best Performance	Ensemble Method	Dataset
1	EDL-COVID: Ensemble deep learning for COVID-19 case detection from chest X-ray images [58]	Highest accuracy: 96.4%	Deep CNN	The latest COVIDx dataset by Wang et al. containing 15,477 CXR images from 13,870 cases, including 6053 pneumonia cases, 8851 normal cases, and 573 COVID-19 cases. The COVIDx datasets were extracted from GitHub and Kaggle
2	Ensemble learning‐based COVID‐19 detection by feature boosting in chest X‐ray images [59]	Highest accuracy: 99.8%	VGG-16 (base) + logistic regression (meta)	5863 CXR images of normal and pneumonia cases obtained from Kaggle
3	Iteratively pruned deep learning ensembles for COVID-19 detection in chest X-rays [60]	Highest accuracy: 97%	A custom CNN and a set of pre-trained ImageNet patient-level models were trained and evaluated. Knowledge learned was transferred and adjusted to improve performance. The following models were used: VGG-16 VGG-19, Inception-V3	Four datasets: a) The data collected from Guangzhou Women and Children's Medical Center in Guangzhou, China, the anteriorposterior CXRs of children from one to five years old, showing normal lungs, bacterial pneumonia, and non-COVID-19 viral pneumonia b) normal CXRs and abnormal images with non-pneumonia and pneumonia-like opacities from the National Institute of Health CXR-14 dataset c) Twitter COVID-19 CXR dataset d) MONTREAL COVID-19 CXR dataset from GitHub
4	Iteratively pruned deep learning ensembles for COVID-19 detection in chest X-rays [61]	Highest accuracy: 98.33% for binary 92.36% for multi-classes	Several classifications such as decision tree, KNN, SVM, autoencoder, Boltzmann machine, CNN, Inceptionv3, DenseNet121, Xception, Inception, ResNetv2	Chest X-ray image. COVID-19 and non-COVID-19 datasets obtained from different sources plus the pneumonia dataset from Kaggle. This dataset consists of 10,000 CXR images, of which 2022 were for pneumonia, 2161 for COVID-19, and 5863 were for non-COVID-19
5	Choquet integral and coalition game-based ensemble of deep learning models for covid-19 screening from chest x-ray images [62]	highest accuracy: 97%	A hybrid model based on lambda fuzzy from DCNN, VGG16, Xception, InceptionV3 architectures, using Choquet Integral	A new dataset of CXR images by combining three publicly available datasets from Kaggle and GitHub
6	Covid‐19 detection from radiographs by feature‐reinforced ensemble learning [63]	Highest accuracy: 98.47%	Combination of SVM, linear discriminant analysis, KNN new bayes, decision tree, ResCNN, local binary patterns, histogram of oriented gradients, majority voting	5228 chest X-ray images extracted from Kaggle, a subsidiary of Google LLC. The images were categorized into three categories: natural, pneumonia, and COVID-19
7	COVID-19 detection using integration of deep learning classifiers and contrast-enhanced canny edge detected X-ray images [64]	Highest accuracy: 97.9%	VGG16, InceptionV3	588 cases of positive COVID-19 and positive pneumonia obtained from GitHub, Radiopedia, and SIRM

Table 5 shows the related works published by open-access publications such as MDPI, NLM, JAMIR, and Hindawi. Among the various ensemble methods, ResNet, DenseNet, VGG, CNN, and SVM have been at the forefront.

Table 5.

Related published research by open access publishers.

#	Title	Best Performance	Ensemble Method	Dataset
1	Development of Machine-Learning Model to Predict COVID-19 Mortality: Application of Ensemble Model and Regarding Feature Impacts [65]	The highest accuracy: 85%	Combination of deep and machine learning: MLP with SVM, XGBoosta and Random Forest	203 patients with severe and moderate levels of COVID-19 in South Korea
2	An Improved Machine-Learning Approach for COVID-19 Prediction Using Harris Hawks Optimization and Feature Analysis Using SHAP [66]	The highest accuracy: 92.38%	Applying the Harris Hawkes optimization algorithm to XGBoost, light gradient boosting, classification boosting, RF and SVM classifiers	Data collected by the Nexoid research team in London consisting information collected from 1,023,426 individuals, 98.80% of whom had negative COVID-19 results and 1.20% had positive results
3	A Novel βSA Ensemble Model for Forecasting the Number of Confirmed COVID-19 Cases in the US [67]	Not clear (reducing error rate)	Combination of α-Sutte indicator and ARIMAb with error-based dynamic weighting method	Collected by Centers for Disease Control and Prevention in the United States in July 2021
4	An Ensemble Learning Model for COVID-19 Detection from Blood Test Samples [68]	Highest accuracy: 99.4%	CNN as the first stage classifier was combined with 15 supervised machine learning algorithms: SVM, Bayes, DT, RF, MLPc, Addabost, LR, LDAd/QDAe, and stochastic gradient descent	Data from 279 blood tests taken at the San Raffaele Hospital in Milan, Italy
5	Forecasting the Potential Number of Influenza-like Illness Cases by Fusing Internet Public Opinion [69]	Reducing error rate from 6.48% to 2.68%	Combination of XGBoost, RF and SVRf	The data collected at the Yuan Environmental Protection Center and the Statistics Office of the Ministry of the Interior of Taiwan
6	Deep Ensemble Learning-Based Models for Diagnosis of COVID-19 from Chest CT Images [70]	Best accuracy: 99.2%	Transfer learning by pre-trained CNN, stacking and Weighted Aged Ensemble to combine the performance of three base classifier VGG19, ResNet50 and DenseNet201	2482 CT scan images obtained from 120 patients in So Paulo, Brazil. 1252 COVID-19 positive cases and 1230 COVID-19 negative cases
7	Deep Ensemble Model for COVID-19 Diagnosis and Classification Using Chest CT Images [71]	Highest accuracy: 98.59%	Preprocessing based on Gaussian filter to remove noise. Using a shark optimization algorithm and improved bat algorithm with multi-class SVM	A collection of CT chest images uploaded to GitHub, including 349 images from 216 patients in several hospitals in China
8	Automated diagnosis of childhood pneumonia in chest radiographs using modified densely residual bottleneck-layer features [72]	Highest accuracy: 99.6%	Adaboost	5232 CXR images for children aged one to five years old from Kaggle, of which 3883 were infected and 1349 were normal
9	Multi-channel transfer learning of chest x-ray images for screening of COVID-19 [73]	Highest accuracy: 94% Recall: 100%	Three ResNet-based models for one-to-all classification	Chest X-ray images including 1579 normal, 4245 pneumonia and 184 COVID-19 cases
10	Ensemble learning for poor prognosis predictions: A case study on SARS-CoV-2 [74]	Highest accuracy: 95%	Seven prediction models defined in China named Dong, Shi, Gong, Lu, Yan, Xie, and Levy	5394 cases in two hospitals in China, London King's College Hospital and University Hospitals Birmingham
11	Using Automated Machine Learning to Predict the Mortality of Patients With COVID-19: Prediction Model Development Study [75]	Highest accuracy for stacking model: 79.1%	Using a real-time method and building 20 different machine learning models through auto ML. Model interpretation through Shapley's additive explanation and dependency graphs for extracting 10 influential variables - using a binary classifier	4313 cases in Albert Einstein College of Pharmaceutical Sciences in New York
12	COVID-19 epidemic: analysis and prediction [76]	Highest accuracy: 99.94%	Using linear regression, polynomial regression and SVM	14,654 Indian patients' data from Johns Hopkins University Center for Science and Engineering
13	Deep ensemble model for classification of novel coronavirus in chest X-ray images [77]		Deep CNN model namely MobileNet, ResNet50 and InceptionV3.	Four classes of chest X-rays images (natural, bacterial, viral, and COVID-19), each containing 1050 images obtained from Kaggle
14	Ensemble deep learning and internet of things-based automated COVID-19 diagnosis framework [78]	Highest accuracy: 99.12%	CNN, transfer learning and three types of deep learning: ResNet152V2, DenseNet201 and InceptionResNetV2 (IRNV2).	CT scan datasets for CNN, deep learning, and transfer learning from a variety of sources
15	Artificial intelligence and medical internet of things framework for diagnosis of coronavirus suspected cases [79]	Highest accuracy: 99.2%	ResNet152V2, InceptionResNetV2, VGG16, DenseNet201	1663 COVID-19 positive patients, 401 cases of pneumonia (viral and bacterial), 394 tuberculosis patients, and 2039 images of healthy people
16	On the detection of covid-19 from chest x-ray images using cnn-based transfer learning [80]	Highest accuracy for ResNet50V2	Using five models: VGG16, ResNet50V2, Xception, MobileNet, DenseNet121	X-ray images of 678 patients with or without COVID-19 obtained from GitHub
17	COVID-19 cases prediction in Saudi Arabia using tree-based ensemble Models [81]	Highest accuracy: For XGBoost MAE: 4.41, RMSE: 7.11 MAPE: 0.95%	Four ensemble methods based on tree: Gradient Tree Boosting Random Forest Extreme Gradient Boosting Voting Regressor	The data by Our World in Data (OWID) in Saudi Arabia. This dataset contains 59 parameters, updated daily, including confirmed cases, deaths, and testing
18	FLANNEL: Focal Loss Based Neural Network Ensemble for COVID-19 Detection [82]	Highest accuracy: Macro-F1 91.7%	FLANNEL model consists of two stages; in the first stage, Inception v3, VGG 19-bn, ResNetXt101, Resnet152, and Densenet161. in the second stage, Neural Weight Module	The COVID and Kaggle chest X-ray images (pneumonia) dataset containing 5508 images from 2874 patients in four categories

^aExtreme Gradient Boosting.

^bAutoRegressive Integrated Moving Average.

^cMultilayer Perceptron.

^dLinear Discriminant Analysis.

^eQuadratic Discriminant Analysis.

^fSupport Vector Regression.

4. Scientometric and bibliometric analysis

Bibliometric search retrieves data for required documents that have an academic structure [83,84]. VOSviewer [85] and BibExcel [86] softwares are used for doing the scientometric-bibliometric analysis. These two softwares have been widely used in recent scientometric-bibliometric studies [[87], [88], [89], [90], [91]].

As mentioned earlier, a total of 142 journal papers (records) were extracted from the Scopus database based on our criteria including ensemble learning and COVID-19 keywords. This section attempts to capture an overall picture of ensemble learning and COVID-19. The top 10 most influential countries have been India, China, and the United States which have published the greatest number of journal papers (Fig. 1).

Fig. 1.

Top influential countries in this field of research.

Fig. 2 shows the top 10 most influential publishers that have published more than five articles in this research field. Elsevier, MDPI, and Springer have published the greatest number of papers. Although the number of articles published by Elsevier is relatively higher than MDPI, MDPI has published more open-access articles than Elsevier in this field of research.

Fig. 2.

Number of published articles by each of the studied publishers.

Fig. 3 shows the top 10 influential journals based on the number of published papers. Applied Soft Computing, Scientific Report, Journal of Medical Internet Research and Computers, Materials and Continua are the top journals in this field of research.

Fig. 3.

Number of documents per journal.

Table 6 shows the 10 top-cited papers. The most cited paper was [92] authored by Ribeiro et al. with 230 citations. The second place belonged to Ref. [60] by Rajaraman with 129 citations. The next most cited paper was [55] by Gao with 96 citations. The average number of citations from January 2020 to August 1, 2022 was 10.78.

Table 6.

Ten top-cited papers.

Title	Citations	Reference No.
Short-term forecasting COVID-19 cumulative confirmed cases: perspectives for Brazil	230	[92]
Iteratively pruned deep learning ensembles for COVID-19 detection in chest X-rays	129	[60]
Machine learning based early warning system enables accurate mortality risk prediction for COVID-19	96	[93]
The ensemble deep learning model for novel COVID-19 on CT images	83	[47]
Assessment of lockdown effect in some states and overall India: A predictive mathematical study on COVID-19 outbreak	77	[94]
Densely connected convolutional networks-based COVID-19 screening model	52	[39]
Lies Kill, Facts Save: Detecting COVID-19 Misinformation in Twitter	47	[95]
Trends in US pediatric hospital admissions in 2020 compared with the decade before the COVID-19 pandemic	47	[96]
Ensemble learning model for diagnosing COVID-19 from routine blood tests	37	[97]
Full-coverage mapping and spatiotemporal variations of ground-level ozone (O3) pollution from 2013 to 2020 across China	37	[98]

Fig. 4 shows the number of papers funded by each funding entity. The National Natural Science Foundation of China dedicated the largest sum of research funds during our study period. Meanwhile, the investment of Saudi Arabia was also impressive. Fig. 1, Fig. 4 demonstrate that countries with more investments on researching in this area have published more scientific articles, including China and Saudi Arabia.

Fig. 4.

Top funding entities in this field of research.

An investigation of affiliations reveal that the Ministry of Education of China and King Saud University in Saudi Arabia have had the most contributions (Fig. 5). There was a significant relationship between the top 10 counties, funding entities, and affiliations.

Fig. 5.

Number of documents based on affiliation of the published research.

The most effective authors are shown in Fig. 6. These authors have published more than three papers in this research area. In addition, it is necessary to examine the co-authorship map (Fig. 7). They have formed a network of collaboration among themselves, as shown in the diagram below.

Fig. 6.

Top five influential authors in this research field.

Fig. 7.

Co-authorship map for the top five influential authors.

Fig. 7 shows the co-authorship map between the top authors. This figure shows that there are strong cooperation networks between the top influential authors.

Fig. 8 presents a word occurrence map that was extracted from the abstracts. Besides general keywords such as COVID-19, ensemble learning, and models, some of the specialized keywords were particularly noteworthy. In the red cluster, for example, CNN, transfer learning, and Resnet appear to be important. LSTM is prominent in the green cluster while the decision tree and random forest are prominent in the blue cluster. Fig. 9 shows the top algorithms used in the article. This figure is drawn based on the number of frequent words in the abstract of the articles. CNN, random forest, neural network, SVM, and LSTM were among the top algorithms/techniques.

Fig. 8.

Word occurrence in abstracts.

Fig. 9.

Top 10 used methods/algorithms.

5. Discussion

This research was conducted in two phases. The first stage involved a general review and extraction of data from each article in tables, including the general research method, best performance, and dataset.

Then in the brief review section, an attempt was made to provide an overview of the related works. It is beneficial since it allows researchers to understand the types of research already done in the form of a table more readily and easily. A major objective of this research was to obtain the most common algorithms n ensemble learning methods. CNN was the most frequently used algorithm in all the studied publications. SVM was also applied often. Interestingly, other methods such as Resnet, Densenet, and VGG were also widely used (Table 7).

Table 7.

Most frequent algorithms/techniques extracted from the literature review.

Most frequent techniques	Publisher
ResNet, DenseNet, MobileNet, VGG and CNN	Springer
ResNet, DenseNet, VGG, CNN and SVM	Elsevier
CNN	Nature
ResNet, DenseNet, VGG, CNN and SVM	IEEE
ResNet, DenseNet, VGG, CNN and SVM	Open Access Publications

CNN has been frequently used in related works for the following reasons:

• ●Feature extraction: CNNs are designed to automatically learn hierarchical representations of data. They excel at extracting relevant features from raw input data, especially in the case of images. In detecting COVID-19 infection, CNNs can effectively identify distinctive patterns or features from chest X-ray or computed tomography scan images.
• ●Spatial hierarchies: Spatial hierarchies can be captured in data with CNNs. In images, they can learn to recognize low-level features such as edges, textures, and shapes, and gradually build up to higher-level representations. It is beneficial to extract hierarchical features for detecting COVID-19, since certain visual abnormalities associated with the disease may be present at different spatial scales.
• ●Deep learning capabilities: As a type of deep learning model, CNNs are capable of learning complex relationships and representations from large amounts of data. By automating feature learning, they eliminate the need for manual feature engineering, which is time-consuming and may not capture all the information needed. Due to the availability of large datasets of COVID-19 cases, CNNs can effectively take advantage of this depth to learn discriminative patterns.
• ●Ensemble diversity: Ensemble learning benefits from combining diverse models. CNNs can contribute to ensemble diversity by being trained with different architectures, hyperparameters, or subsets of data. This diversity helps mitigate overfitting and improves the generalization ability of the ensemble, making it more robust in COVID-19 detection tasks.
• ●State-of-the-art performance: CNNs have demonstrated excellent performance in various image classification tasks, surpassing human-level performance in some cases. Their ability to learn complex representations and generalize well from large datasets has made them a popular choice for COVID-19 detection, where an accurate and reliable diagnosis is crucial [99,100].

The scientometric analysis section also shows similar results by extracting keywords from the abstracts (Fig. 9). Its findings have many similarities with those of the brief review. Still, there are frequently used algorithms that were revealed by the brief review method, but we were unable to uncover them using the scientometric method. Our findings indicate that scientometric methods and the evaluation of abstracts and keywords cannot provide a complete picture of the content of articles. Yet, they can provide useful information as a complementary method.

A topic that emerged in our study was the frequent appearance of certain countries in the top rankings. Although India has the highest number of publications, China appears in most of the top lists. In terms of the number of published articles, China ranks second. China is the top sponsor for research in this area. Furthermore, some of the top affiliations also belong to China. In evaluating the top influential authors, we found that most are Chinese, but they also have strong co-authorship and cooperation networks. The review of the used datasets reveal that an extensive number of them were prepared in China. It is also necessary to highlight that Saudi Arabia has had the most investments for research in this field after China and the highest level of affiliation was with Saudi universities. The United States was also listed among the top countries after China and Saudi Arabia. Considering the number of published articles, the United States ranks third. American universities are present in the top 10 list of affiliations.

6. Conclusion

We did a brief review and a scientometric-bibliometric analysis of the published research on detecting COVID-19 with ensemble learning approaches. The data were extracted from the articles published in journals that are indexed in the Scopus database using relevant queries. According to the preliminary bibliographic analysis, China and Saudia Arabia had allocated the greatest number of research funds and had the most affiliations. India, China, and Saudi Arabia have had the greatest number of published articles in this field of research. It was found that four authors had significant co-authorship relationships. Among the most commonly used algorithms in ensemble learning approaches, CNN was the most frequently used in the studied published articles. In addition, SVM and random forest method were among the most commonly used methods. A review evaluation also revealed that Resnet, DenseNet, and VGG methods were frequently employed.

7. Challenges and future works

Ensemble learning can be challenging to implement in real-world settings due to the following reasons [14]:

• ⬥Ensemble models are computationally expensive and require more resources than single models.
• ⬥Ensemble models are more complex and difficult to interpret than single models.
• ⬥Ensemble models require more data than single models.

To address these challenges, researchers have proposed several solutions such as using pre-training specific to chest X-ray images in transferring and fine-tuning the learned knowledge toward improving COVID-19 detection performance; using ensembles of the fine-tuned models to further improve performance over individual constituent models; performing feature selection to reduce the dimensionality of the data; and using transfer learning [101].

Further research is needed in several areas based on our findings, including:

• ⬥Standardization and benchmarking: There is a need for standardized datasets, evaluation metrics, and benchmarking protocols to facilitate fair comparisons and reproducibility of ensemble learning models for COVID-19 detection.
• ⬥Interpretability and transparency: Research should focus on improving the interpretability and transparency of ensemble models to gain insights into their decision-making processes, feature importance, and potential biases.
• ⬥Real-world deployment: Investigating the feasibility of deploying ensemble learning models in real-world healthcare settings, considering computational requirements, integration with existing systems, and validation in diverse clinical environments.
• ⬥Addressing data limitations: Addressing data limitations, including imbalanced datasets, lack of diversity in demographic representation, and potential biases, to enhance the robustness and generalizability of ensemble models.

Funding

The author did not receive any financial support.

Data availability

Data will be available upon request.

CRediT authorship contribution statement

Mohammad Javad Shayegan: Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Resources, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.