Optimization of SVR and CatBoost models using metaheuristic algorithms to assess landslide susceptibility

Abstract

In this study, a landslide susceptibility assessment is performed by combining two machine learning regression algorithms (MLRA), such as support vector regression (SVR) and categorical boosting (CatBoost), with two population-based optimization algorithms, such as grey wolf optimizer (GWO) and particle swarm optimization (PSO), to evaluate the potential of a relatively new algorithm and the impact that optimization algorithms can have on the performance of regression models. The Kerala state in India has been chosen as the test site due to the large number of recorded incidents in the recent past. The study started with 18 potential predisposing factors, which were reduced to 14 after a multi-approach feature selection technique. Six susceptibility models were implemented and compared using the machine learning algorithms alone and combining each of them with the two optimization algorithms: SVR, CatBoost, SVR-PSO, CatBoost-PSO, SVR-GWO, and CatBoost-GWO. The resulting maps were validated with an independent dataset. The performance rankings, based on the area under the receiver operating characteristic curve (AUC) metric, are as follows: CatBoost-GWO (AUC = 0.910) had the highest performance, followed by CatBoost-PSO (AUC = 0.909), CatBoost (AUC = 0.899), SVR-GWO (AUC = 0.868), SVR-PSO (AUC = 0.858), and SVR (AUC = 0.840). Other validation statistics corroborated these outcomes, and the Friedman and Wilcoxon-signed rank tests verified the statistical significance of the models. Our case study showed that CatBoost outperformed SVR both in case the models were optimized or not; the introduction of optimization algorithms significantly improves the results of machine learning models, with GWO being slightly more effective than PSO. However, optimization cannot drastically alter the results of the model, highlighting the importance of setting up of a rigorous susceptibility model since the early steps of any research.

Introduction

Landslides are among the most devastating geological disasters occurring in mountainous terrain around the world and may be triggered by rainfall^1,2, seismic activity^3,4, volcanic activity⁵, variations in groundwater level⁶, toe erosion^7,8, and anthropogenic activities that alter the stability of the terrain, such as mining or blasting⁹, hill slope cutting for road widening or constructing new roads⁷, and unscientific construction of structures^7,10,11. These triggering (external) factors, depending on the intrinsic or predisposing factors (topography, soil, geology, and land use) of the area, can result in slope failures or landslides^12,13. An investigation by Froude and Petley¹⁴ listed 55,997 deaths worldwide due to 4862 rainfall-induced landslides from 2004 and 2016. Their investigation found that around 75% of the incidences occurred in Asia alone, with India being the hotspot (highest incidences). A recent study by Gómez et al. ¹⁵ reported 185,753 deaths worldwide due to 37,946 landslides between 1903 and 2020, with the highest fatalities recorded in Asia. As per the study¹⁵, India ranks third in terms of the highest number of small (1–10 deaths) and moderate (11–100 deaths) landslide disasters and second in terms of the number of large (100 + deaths) landslide disasters. Moreover, according to Wang et al.¹⁶, India stands third, after China and Afghanistan, in the list of the top 10 nations where landslides pose the highest risk of casualties. Around 12.6% (approximately 0.42 million sq. km) of the total area of India is prone to landslides¹⁷, including the Himalayas (Northeast and Northwest), the Western Ghats (WG), the Konkan Hills, and the Eastern Ghats¹⁸. The Western Ghats are a mountain range that runs parallel to the west coast of India and encompasses Kerala, Tamil Nadu, Karnataka, Goa, Maharashtra, and Gujarat states of India^19,20. Kerala is an ideal site to study landslide susceptibility, as it is a landslide-prone area with old and recent landslide disasters. According to Hao et al.²¹, 4728 landslide incidences have been reported in the WG region in Kerala alone during the 2018 southwest monsoon.

Landslide susceptibility assessment can be defined as the spatial probability of landslide occurrence and is regarded as a basic and crucial step for landslide risk assessment and management^22,23. Although landslide susceptibility studies date back to the 80 s of the last century and several approaches have been proposed since then, several recent literature reviews highlighted that, at present, Machine Learning (ML) algorithms are consolidated and cutting-edge methods to implement into landslide susceptibility models^24–28. ML models such as random forest^29,30, support vector machine (SVM)³¹, decision trees³², Naïve Bayes³³, k-nearest neighbours³⁴, gradient boosting algorithms³⁵, support vector regression³⁶, k-means clustering³⁷, classification and regression tree³⁸, multi-layer perceptron³⁴, and neural networks^39–42 have been extensively and effectively utilized by researchers for identifying landslide susceptibility. However, the development of newer and more powerful algorithms is constantly progressing, with the results of proposing ML algorithms that must be tested and evaluated in real case study applications. Support vector regression (SVR) and Categorical Boosting (CatBoost) can be included among recently developed algorithms. Support vector regression (SVR) has several advantages over other ML models: its computational complexity remains constant regardless of the dimensions of the input space, it exhibits exceptional generalization potential, and achieves high prediction accuracy⁴³. The Categorical Boosting (CatBoost) regressor is an ensemble model based on the Gradient Boosting algorithm⁴⁴. Boosting is an ensemble strategy that increases the accuracy of any given machine learning algorithm for both regression and classification tasks^45,46. CatBoost can avoid overfitting, facilitates quick learning, and is well-suited for ML applications with categorical or heterogeneous data^44,47. Being an ensemble model, CatBoost integrates various models to improve prediction accuracy and robustness, reduce uncertainties, and address the weak generalization of standalone ML models^44,48. However, complex ML methods are sensitive to hyperparameters, which regulate the behaviour of ML algorithms and, ultimately, affect the final results. For instance, if the default values proposed by computing software are used, often optimal results are not obtained^49,50. One effective solution is to utilize an optimization algorithm to adjust these hyperparameters and improve the models’ accuracy, speed, and scalability^51–53.

Optimization is the iterative process of enhancing an ML model’s accuracy by reducing the error level. Optimization algorithms are of two types: individual-based and population-based^54,55. Individual-based algorithms use a single randomly produced search agent to initiate the optimization process to find and locate the global optimum of the optimization issue⁵⁵. Individual-based algorithms begin the process in the search space at a random starting point. Typically, they can only find the best solution in the immediate vicinity of that random point and cannot leave that area to look for other possible high-fitness regions inside the issue domain. Premature convergence that appears in two kinds of problems, known as uni-modal and multi-modal functions, is the primary drawback of the individual-based algorithm⁵⁵. Premature convergence in uni-modal functions typically happens when the optimization process proceeds too slowly⁵⁵. On the other hand, local optima entrapment is the primary reason behind premature convergence in multi-modal functions⁵⁵. Population-based optimization algorithms offer several advantages, including the ability to search the solution space through multiple points (individuals or solutions) at the same time, the ability to share information and facilitate interactive learning between individuals with different search behaviours, and the fact that they are stochastic due to the inherent incorporation of randomness into search behaviours⁵⁴.

Researchers integrated optimization algorithms such as bat algorithm⁵⁶, cuckoo optimization algorithm⁵⁶, grey wolf optimizer⁵⁶, invasive weed optimization⁵⁷, particle swarm optimization⁵⁸, satin bowerbird optimizer⁵⁹, teaching–learning-based optimization⁵⁹, and whale optimization algorithm⁶⁰ to different ML models for landslide susceptibility modelling. Particle swarm optimization (PSO) and grey wolf optimizer (GWO) are two such population-based optimization algorithms⁶¹. PSO’s fundamental concept is to mimic a swarm of fish, birds, or bees in search of food⁶². These species communicate with one another when searching for food to find their targets separately and stochastically and eventually advance towards the goal from different directions⁶². Finding the global optimum solution is more likely since a larger portion of the design space is examined, and the search pathways differ for various particles⁶². PSO offers notable benefits, such as greater precision and rapid convergence⁶³. GWO mimics a pack of grey wolves’ hunting strategy and leadership hierarchy and can be applied to network optimization⁵⁴. To replicate the leadership hierarchy, four different sorts of grey wolves—alpha, beta, delta, and omega—are used ⁶⁴. Several notable benefits of the GWO algorithm include its minimal number of algorithm parameters, robust global optimization capability, simplicity of use, and superior convergence^64,65.

This investigation sought to evaluate the landslide susceptibility of Kerala state in India by applying two ML regression algorithms such as SVR and CatBoost, integrating population-based metaheuristic optimization algorithms such as PSO and GWO, verifying whether the integration of these two metaheuristics improves the prediction capability of the ML regression models, comparing the predictive capability of all six models, and identifying the important predisposing factors. This study is relevant as ML and optimization algorithms are continuously evolving, and their joint use needs to be corroborated by applications in appropriate test sites, like Western Ghats. Though many researchers^58,66–71 employed ML regression algorithms (MLRA) to assess the landslide susceptibility of different areas within the WG region, only Saha et al.⁵⁸ integrated an optimization algorithm to enhance the prediction accuracy. Moreover, the novelty of this work lies also in the pioneering combination of different ML and optimization algorithms: to the best of our knowledge, it is the first time that PSO and GWO have been combined with ML-based standalone (SVR) and ensemble (CatBoost) regression models to assess landslide susceptibility. Furthermore, the relevance of the final results is enhanced by the fact that the existing susceptibility map (produced in the year 2010) used by the Kerala State Disaster Management Authority⁷² needs to be updated; hence, the methodology adopted in this modelling is of utmost importance.

Materials and methods

Study area

Kerala is situated in the southwestern region of peninsular India, with the Arabian Sea to the west and the Western Ghats (WG) to the east. The rock types include hard crystalline metamorphic rocks (charnockites, gneisses, and khondalites) of the Achaean age in the WG⁷³. The mean annual precipitation is approximately 300 cm, which was brought about by two monsoon seasons: the southwest (June–September) and the northeast (October–November)^21,74. In addition to this climatic condition, land use practices, deforestation, and soil erosion accelerate the weathering and, thereby, the occurrence of landslides⁷⁵. The predominant lateritic soils result from chemical weathering of the underlying charnockites^73,75. Administratively, the state (with 38,863 km² of area) shares its boundary with the Tamil Nadu and Karnataka states (Fig. 1). Some of the notable landslide disasters that occurred in Kerala are displayed in Fig. 1 and Table 1.

Fig. 1.

Location of the study area and the devastating landslide disasters. Figure created with ArcGIS Pro software (Version 3.1; ESRI, Redlands, California, https://www.esri.com/en-us/arcgis/products/arcgis-pro/overview).

Table 1.

Some of the notable landslide disasters occurred in Kerala.

Sl. No	Landslide disaster	District	Date	Fatalities	Source
1	Amboori landslide	Thiruvananthapuram	November 9th, 2001	38	Naidu et al.76
2	Kavalappara landslide	Malappuram	August 8th, 2019	59	Vasudevan et al.77
3	Puthumala landslide	Wayanad	August 8th, 2019	17	Vasudevan et al.77
4	Pettimudi landslide	Idukki	August 6th, 2020	70	Sajinkumar and Oommen78
5	Kokkayar landslide	Idukki	October 16th, 2021	7	Ajin et al.79
6	Plappally and Kavali landslides	Kottayam	October 16th, 2021	10	Ajin et al.79
7	Kudayathoor landslide	Idukki	August 29th, 2022	5	George et al.80
8	Wayanad landslide	Wayanad	July 30th, 2024	452 deaths and 67 missing (As on September 9th, 2024)	District Emergency Operations Centre (DEOC), Wayanad

Methodological framework

This modelling applied two-stage feature selection techniques, followed by two MLRAs, such as CatBoost and SVR, and integrated two population-based optimization algorithms, such as GWO and PSO, to assess the landslide susceptibility, different validation techniques, and finally assessed the factor importance. The flowchart of the susceptibility modelling is displayed in Fig. 2.

Fig. 2.

Flowchart of the susceptibility modelling.

Landslide inventory

The landslide inventory was extracted from two sources: the Bhukosh portal of the Geological Survey of India and the incidence data for the year 2018, published by Hao et al.²¹. The dataset of Hao et al.²¹ comprises 4728 incidence locations. This is a reconstructed inventory that utilized the incidence data of the year 2018 compiled by the National Remote Sensing Centre (NRSC)⁸¹ and derived based on the object-based image analysis (OBIA) algorithm⁸². The incidence data for 2018 prepared by the GSI was based on a detailed survey²¹. The inventory derived from the Bhukosh portal recorded 3164 landslide incidences in Kerala. This study combined these two datasets after removing 117 duplications. Thus, the final landslide inventory employed in this modelling comprises 7775 events (Fig. 3), mainly slides and flows⁶⁸. This data was randomly separated into 70% (5443 landslides) and 30% (2332 landslides) as training and validation datasets, respectively⁸³, with an equal number of non-landslide points. While comparing different data-splitting approaches, previous studies^84,85 identified the 70:30 ratio as the best-performing split. At the same time, the 50:50 landslide and non-landslide sampling strategy is widely employed to maintain an equilibrium between positive and negative samples in the modelling^83,86. A field survey was conducted to validate the accuracy of the landslide incidence locations, especially the ones identified by the GSI. This on-the-ground verification process provided first-hand information to confirm the presence of landslides and ensure the reliability of the inventory (Figs. 3a-c).

Fig. 3.

Landslide inventory – training and validation datasets – a., b., and c. Landslide sites. Figure created with ArcGIS Pro software (Version 3.1; ESRI, Redlands, California, https://www.esri.com/en-us/arcgis/products/arcgis-pro/overview).

Predisposing factors

As a starting point, the modelling considered 18 potential predisposing factors, selected based on existing literature^68,87–91. Data were extracted from diverse sources (DEM, data portals, and published maps), as reported in Table 2. Predisposing factors such as slope angle, slope aspect, elevation, curvature, plan and profile curvatures, terrain ruggedness index (TRI), and topographic wetness index (TWI) were derived from the ALOS PALSAR DEM utilizing ArcGIS Pro spatial analyst and map algebra tools. Equations (1)⁹² and (2)^93,94 were utilized to compute the TRI and TWI, respectively. Factors such as lineaments, lithology, and geomorphology were extracted from the Bhukosh portal, whereas soil bulk density and soil clay content were collected from the Soil Grids portal. The latest road networks were derived from the Planet OSM portal, and built-up area data was downloaded from the GHSL portal. The land use/land cover data (updated on the year 2022) was extracted from the Esri Land Cover portal. The soil texture and stream channels were derived from the soil map (NBSS&LUP) and topographic map (SoI), respectively⁹⁵. The Euclidean distance tool in ArcGIS Pro was used to compute the distance from the lineaments, road networks, and stream channel raster layers.

$$TRI=Y{\left[\sum ,{\left(x_{\mathit{ij}},-,x_{00}\right)}^2\right]}^{1/2}$$ 1

where $x_{\mathit{ij}}$ = elevation of each neighbour cell to cell (0,0)

$$TWI=\text{ln}\left(\frac{\alpha }{\text{tan}\beta }\right)$$ 2

where α = specific catchment area, β = local slope.

Table 2.

The data source of the predisposing factors (categorical factors are in bold).

Data	Source	Data format	Potential predisposing factors	Spatial resolution/Scale
ALOS PALSAR DEM	https://search.asf.alaska.edu/	Raster	Slope Elevation Aspect Curvature Plan curvature Profile curvature Terrain ruggedness index (TRI) Topographic wetness index (TWI)	12.5 m
Bhukosh Portal	https://bhukosh.gsi.gov.in/Bhukosh/Public	Vector	Distance from the fault Geomorphic units Lithology	1:250,000
Topographic map	Survey of India (SoI)	Vector	Distance from the stream channel	1:50,000
Soil map	National Bureau of Soil Survey & Land Use Planning (NBSS & LUP)	Vector	Soil texture	1:250,000
Soil Grids portal	https://soilgrids.org/	Raster	Soil bulk density Soil clay content	250 m
Planet OpenStreetMap (OSM) portal	https://planet.openstreetmap.org/	Vector	Distance from the road (2023)	Multiscale
European Commission’s Global Human Settlement Layer (GHSL) portal	https://ghsl.jrc.ec.europa.eu/datasets.php	Raster	Built-up area (2023)	100 m
Esri Land Cover portal	https://livingatlas.arcgis.com/landcover/	Raster	Land use/land cover (2022)	10 m

Feature selection

Feature selection eliminates irrelevant or redundant predisposing factors, especially useful in case of a large number of input predisposing factors^96,97. The presence of redundant factors will negatively influence the performance and accuracy of the algorithms^29,98,99. In this study, we used a two-step procedure based on multicollinearity analysis and Information gain (IG) method to identify and discard the less important predisposing factors and to objectively define a selection of the optimal parameters for the susceptibility assessment.

Multicollinearity arises when a predisposing factor is very similar (highly correlated) to another^100,101: this is a problem since it undermines the significance of a predisposing factor and adds unnecessary complexity to the model parameterization, which in turn may lead to overfitting issues. Similarly, the existence of multicollinearity will reduce the stability and performance of regression models¹⁰². Hence, similar factors should be excluded or redefined to distinguish them from others¹⁰¹. The most common techniques to evaluate multicollinearity are tolerance (TOL) and variance inflation factor (VIF), which were computed employing Eqs. (3) and (4)^68,88. A predisposing factor has a multicollinearity problem when its VIF value exceeds 10, and its TOL value is lower than 0.1⁸⁸.

$${\mathit{VIF}}_j=\frac{1}{1-R_j^2}$$ 3

$$TOL=1-R_j^2$$ 4

where $R_j^2$ = coefficient of determination of the j^th predisposing factor.

IG is a metric that evaluates how much information a particular feature provides ¹⁰³. The information entropy of sample set R, expressed as H(R), for random variables R and S, where R = {$r_1$, $r_2$, $r_3$, …, $r_N$} and S = {$s_1$, $s_2$, $s_3$, …,$s_N$}, will be computed by applying Eq. (5)¹⁰⁴. H(R) is the prevailing metric for quantifying the purity of a set of samples¹⁰³. As the value of $H(R)$ decreases, the purity of R increases¹⁰³.

$$H\left(R\right)=-\sum _{i=1}^{N}p(r_i)\text{log}(p(r_i))$$ 5

where $p\left(r_i\right)=$$\frac{Numberofinstantswithvalue{r}_i}{TotalnumberofinstantsofR}$

The conditional entropy, denoted as $H\left(S,|,R\right)$ , and joint entropy, denoted as $H(S,R)$ , were computed utilizing equations (6) and (7)¹⁰⁴.

$$H\left(S,|,R\right)=-{\sum }_{r_i\in R}{\sum }_{s_j\in S}p\left(s_j,,,r_i\right)\text{log}(p\left(s_j,|,r_i\right))$$ 6

$$H(S,R)=-{\sum }_{r_i\in R}{\sum }_{s_j\in S}p\left(s_j,,,r_i\right)\text{log}(p(s_j,r_i))$$ 7

The IG for class C and attribute A was computed using Eq. (8)^104,105.

$$IG\left(C,,,A\right)=H\left(C\right)-H\left(C,|,A\right)$$ 8

In this modelling, the threshold is set to be 0.05¹⁰⁵, and any predisposing factor with an IG score below 0.05 will be discarded.

Machine learning regression algorithms

Support Vector Regression (SVR)

SVR is an SVM-based regression tool ¹⁰⁶. SVR possesses a notable benefit in its ability to effectively handle nonlinear processes by utilizing a kernel function, which facilitates the projection of the original data into a linearly separable space of higher dimensionality ¹⁰⁷. SVR utilizes a symmetrical loss function throughout the training process, which equally penalizes both overestimation and underestimation errors ⁴³. For a training dataset $T=\left\{\left(x_i,,,y_i\right),,,i,=,1,\cdots ,.,l\right\},$ where $x_i\in R^N$, $y_i\in R$, SVR seeks to identify a regression function capable of accurately fitting all training samples employing Eq. (9) ¹⁰⁷.

$$fx=w^T\varphi x+b$$ 9

where w = coefficient vector, Φ(x) = kernel function, and b = intercept

Categorical Boosting (CatBoost)

CatBoost is an ML ensemble algorithm based on Gradient Boosting Decision Trees (GBDT) and is particularly suitable for heterogeneous and categorical data^47,108. The CatBoost algorithm inherently incorporates a mechanism to efficiently convert non-numerical data values into numerical ones without the need for parametric tuning and yields good results in a single execution¹⁰⁹. Like other basic gradient lifting algorithms, the CatBoost algorithm creates a new tree and can improve the overfitting issues other traditional algorithms face¹¹⁰. The CatBoost algorithm employs a random sorting technique to arrange the data and subsequently assigns a numerical value to each attribute inside the categorical variables¹¹⁰. The utilization of priority factors and weight coefficients restricts the impact of low-frequency and noise data¹¹⁰. Equation (10)¹¹⁰ was employed for the training dataset D.

$$D=\left(X_j,,,Y_j\right)$$ 10

where j = number of samples (1,2, …n), $X_j$ = j^th target value of X ($x_j^1,x_j^2,\cdots x_j^i)$, and $Y_j$ = j^th target value of Y.

Equation (11)^109,110 was utilized for implementing the model.

$$x_j^i=\frac{{\sum }_{k=1}^n\phi \left(x_k^i,=,x_j^i\right)Y_j+\alpha p}{{\sum }_{k=1}^n\phi \left(x_k^i,=,x_j^i\right)+\alpha }$$ 11

where $\phi$ = indicator function, α = initial weight, and p = initial value.

CatBoost was also utilized for assessing the important predisposing factors. The feature importance was assessed using Eq. (12)^111,112.

$${\widehat{I^2}}_{\mathit{ih}}=\frac{1}{M}{\sum }_{n=1}^N{\widehat{I^2}}_i(T_{\mathit{hn}})$$ 12

where ${\widehat{I}}_{\mathit{ih}}$ = relevance of predictor variable $x_i$ in separating class h from the other classes, and $T_{\mathit{hn}}$ = tree induced for the h^th class at iteration n.

Population-based optimization algorithms

Optimization algorithms were introduced to reduce the likelihood of errors or losses arising from these predictions while enhancing the accuracy of the model⁵².

Particle Swarm Optimization (PSO)

The swarm behaviour of birds in flocks in nature served as the model for the PSO¹¹³. The particles (possible solutions) in PSO move by following the current optimum particles¹¹⁴. PSO requires less memory and is computationally fast¹¹⁴. The two main steps of PSO are as follows¹¹⁵:

Step 1: Velocity and position formulation: A swarm of $n$ particles can find a high-quality solution by updating their positions and velocities as they search through the feasible space. Each particle’s position indicates a potential solution for the optimisation issue. The position and velocity of particle $i$ are shown by m-dimensional vectors $x_i=(x_{i1},x_{i2},\cdots ,x_{\mathit{im}})$ and $v_i=(v_{i1},v_{i2},\cdots ,v_{\mathit{im}})$, considering the decision contains $m$ variables. To regulate the velocities and direct the swarm, PSO defines two positions: the personal-best position, depicted as $p_{best,i}=(p_{best,i1},p_{best,i2},\cdots ,p_{best,im})$ and the global-best position, depicted as $g_{\mathit{best}}=(g_{best,1},g_{best,2},\cdots ,g_{best,m})$. The velocity and position were computed employing Eqs. (13) and (14)¹¹⁵.

$$v_i^{t+1}={\omega v}_i^t+c_1{r}_1\left(p_{best,i}^t,-,x_i^t\right)+c_2{r}_2(g_{\mathit{best}}^t-x_i^t)$$ 13

$$x_i^{t+1}=x_i^t+v_i^{t+1}$$ 14

where $c_1$= cognitive factor, $c_2$ = social factor, $\omega$ = inertia weight, $r_1,r_2$ = real numbers, and t = current generation.

Step 2: Linearly decreasing inertia weight: The careful choice of ω can effectively attain a balance between global search and local exploitation during the evolutionary process. A high ω value can enhance the global search, whilst a low ω value can facilitate local exploitation. Linearly decreasing ω can enhance the performance and be computed by applying Eq. (15)¹¹⁵.

$$\omega ={\omega }_{\mathit{max}}-({\omega }_{\mathit{max}}-{\omega }_{\mathit{min}})\frac{t}{T}$$ 15

where T = maximal generation, and ${\omega }_{\mathit{max}},and{\omega }_{\mathit{min}}$ = upper and lower limits.

Grey Wolf Optimizer (GWO)

GWO is a metaheuristic algorithm that draws inspiration from the behaviour of grey wolves^54,65 and emulates the hierarchical structure of leadership and the hunting mechanism observed in grey wolves⁵⁴. Furthermore, the implementation of the hunting process involves three primary stages, including the search for prey, the encirclement of prey, and the subsequent attack on prey^54,116. The four types of wolves—alpha (α), beta (β), delta (δ), and omega (ω) —are meant to resemble the internal leadership hierarchy of wolves¹¹⁷. The top, second, and third-best wolves are recorded as α, β, and δ, while the others are classified as ω¹¹⁷. The α, β, and δ wolves exhibit a higher degree of proximity to their prey, while the ω wolves assume a secondary role by accompanying and assisting those above three in the tasks of searching, tracking, and surrounding their prey¹¹⁸. Once the circumference of the surrounding circle reaches a sufficiently reduced size, the predators initiate their attack and successfully capture the prey¹¹⁸. The three stages of GWO are as follows¹¹⁸:

Stage 1. Search for prey: The process of hunting commences with the act of searching for potential prey, and their behavioural patterns can be succinctly characterised by Eqs. (16) and (17)¹¹⁸.

$$D=\left|C,.,X_p,\left(t\right),-,X,(t)\right|$$ 16

$$X\left(t,+,1\right)=X_p\left(t\right)-A.D$$ 17

where D = distance between the wolf and the prey, t = current iteration, $X_p\left(t\right)$  = current position of the prey,  $X(t)$= position vector,  $X\left(t,+,1\right)$= updated position vector, and A, C = coefficient vectors

A was computed by applying Eq. (18), while C was computed by employing Eq. (19)¹¹⁸.

$$A={2ar}_1-a$$ 18

$$C={2r}_2$$ 19

where a =$2(1-\frac{t}{T})$ , and $r_1,r_2$  = random vectors

Stage 2. Encirclement of prey: The prey will be encircled under the leadership of α, β, and δ wolves. The distance between these three wolves was computed employing Eqs. (20) to (21), and Eq. (26) was applied to assess how the individual wolf approaches the prey ¹¹⁸.

$$D_{\alpha }=\left|C_1,.,X_{\alpha },\left(t\right),-,X,(t)\right|$$ 20

$$D_{\beta }=\left|C_2,.,X_{\beta },\left(t\right),-,X,(t)\right|$$ 21

$$D_{\delta }=\left|C_3,.,X_{\delta },\left(t\right),-,X,(t)\right|$$ 22

$$X_1=X_{\alpha }-A_1.D_{\alpha }$$ 23

$$X_2=X_{\beta }-A_2.D_{\beta }$$ 24

$$X_3=X_{\delta }-A_3.D_{\delta }$$ 25

$$X(t+1)=\frac{X_1+X_2+X_3}{3}$$ 26

where $X_{\alpha },X_{\beta },and{X}_{\delta }$ are location of α, β, and δ wolves, respectively, and $C_1,C_2,and{C}_3$ are random vectors.

Stage 3. Attack on prey: Upon the cessation of the prey’s motion, the pack of wolves initiates an attack on the prey. The individual wolf moves away from the prey and conducts a global search when |A|> 1. The grey wolf begins attacking its victim when |A|≤ 1.

Implementation of the models

The ArcGIS tool used the landslide and non-landslide locations to sample the thematic layers representing the predisposing factors and to create the training and validation datasets fed into the ML algorithms. The Python programming language was utilized in the Google Colab (https://colab.research.google.com/) to implement the models. Two metaheuristic algorithms (PSO and GWO) were utilized to fine-tune the hyperparameters of the SVR and CatBoost algorithms, and the optimized hyperparameters are displayed in Tables 3 and 4.

Table 3.

Optimized hyperparameters of the SVR algorithm.

Algorithms	Kernel	Degree	Tolerance	Regularization parameter	Epsilon	shrinking	Size of the kernel cache	Fitness function (MAE)
PSO	rbf	10	1	3	0	True	350	0.154
GWO	rbf	3	1	8	0	False	100	0.146

Table 4.

Optimized hyperparameters of the CatBoost algorithm.

Algorithms	Iterations	Learning rate	Depth	Fitness function (MAE)
PSO	71	0.0899	8	0.0665
GWO	97	0.0699	8	0.0656

Validation techniques

Receiver operating characteristic (ROC) curve

The ROC curve is a two-dimensional graph which plots 1-specificity (false positive rate) on the abscissa and sensitivity (true positive rate) on the ordinate to determine the performance of a classification ^119,120. Area under the ROC curve (AUC) is a scalar value that ranges between 0.5 and 1.0, depicting the overall performance of a model ¹²¹. AUC values between 0.5 and 0.6, 0.6 and 0.7, 0.7 and 0.8, 0.8 and 0.9, and 0.9 and 1.0 depict failure, poor, acceptable, excellent, and outstanding performance, respectively ¹²². AUC is the most employed performance metric to quantitatively determine the performance of landslide susceptibility models ¹²³ since it is easy to calculate and interpret ¹²⁴. The AUC provides a good scalar performance measure without relying on a specific threshold ¹²⁵. AUC is useful for imprecise settings because it operates independently of class distributions and misclassification costs ^124,125.

MAE, MSE, and RMSE

MAE, MSE, and RMSE can be effectively employed to validate the models by assessing the errors ¹²⁶. MAE, MSE, and RMSE values can range from 0 to + ∞, and the model’s accuracy is higher when values are close to 0 ¹²⁷. MAE was computed utilizing Eq. (27) ¹²⁸, while MSE and RMSE were computed utilizing Eqs. (28) and (29) ^128,129, respectively.

$$MAE=\frac{1}{n}\sum _{i=1}^n\left|Y_i,-,{\tilde{Y}}_i\right|$$ 27

$$MSE=\frac{1}{n}{\sum }_{i=1}^n{\left(Y_i,-,{\tilde{Y}}_i\right)}^2$$ 28

$$RMSE=\sqrt{\frac{1}{n}}\sum _{i=1}^n{(Y_i-{\tilde{Y}}_i)}^2$$ 29

where $Y_i$ = actual value, ${\tilde{Y}}_i$= predicted value, and n = number of observations.

R-squared

The coefficient of determination (R-squared or R²) is the ratio of the explained variation to the total variation ¹³⁰. The R² value varies from 0 to 1 ¹³¹, with 1 recommending perfect correlation and 0 specifying no correlation ¹²⁷. Jierula et al. ¹²⁷ listed five degrees of correlation for R² values: 0 to 0.2 (very weak), 0.2 to 0.4 (weak), 0.4 to 0.6 (moderate), 0.6 to 0.8 (strong), and 0.8 to 1 (very strong). R² was computed by applying Eq. (30) ¹²⁷.

$$R^2=1-\frac{{\sum }_1^n{\left(D_{\mathit{act}},-,D_{\mathit{pre}}\right)}^2}{{\sum }_1^n{\left(D_{\mathit{act}},-,{\overline{D}}_{\mathit{act}}\right)}^2}$$ 30

where $D_{\mathit{act}}$ = actual value, $D_{\mathit{pre}}$ = predicted value, ${\overline{D}}_{\mathit{act}}$ = mean, and n = sample size.

Friedman test

The Friedman non-parametric test assessed the significance employing Eq. (31) ^132,133. A significance level (often denoted by α) of 0.05 (5% risk) is set as the threshold ^134,135, and to determine the significance, α should be compared with the probability value (often denoted by P). When the P-value ≤ α, the null hypothesis (i.e., no difference between variables) can be rejected, and if the P-value > α, the null hypothesis cannot be rejected due to lack of evidence ¹³⁶. In simple words, a P-value < 0.05 is considered significant (factors are significantly different), and a P-value > 0.05 is considered insignificant ¹³⁷.

$$F_r=\frac{12}{nk(k+1)}\sum _{i=1}^k{R}_i^2-3\left(k,+,1\right)n$$ 31

where n = number of density fractions, and k = number of samples.

Wilcoxon signed-rank test

Wilcoxon signed-rank, another non-parametric test utilized to compare two related or matched samples ¹³⁸, can be employed to assess the statistical significance of the models ^134,135. The Wilcoxon test computes the difference between the two values (d) and the absolute value of this difference ($\left|d\right|$), sorts the pair based on the $\left|d\right|$, and assigns numerical ranking (Rank 1 to the pair with the lowest $\left|d\right|)$¹³⁹. The ranks will be assigned + ve and –ve sign based on the deviation ¹⁴⁰. Samples with $\left|d\right|$ values zero will not assigned ranks and excluded from the analysis ¹³⁹. Finally, the $T_{-}$ and $T_{+}$ values will be computed as the sum of ranks assigned to samples with d < 0 and d > 0, respectively ¹³⁹. The smaller $T_{-}$ and $T_{+}$ values will be the final z-statistic (T) ¹⁴⁰ and were computed employing Eq. (32) ¹⁴¹. The performance of models varies significantly when the values are higher than + 1.96 or -1.96 ^134,135.

$$T=\sum _{i=1}^{N}sgn(X_i)R_i$$ 32

where $X_i$ = sample, and $R_i$ = rank.

Taylor diagram

The Taylor diagram can be effectively utilized to determine the efficacy of multiple models ¹⁴². Taylor diagram is a graphical depiction of the bias, correlation, and variance of the predictions with respect to the actual ones ¹⁴³. The Taylor diagram provides a concise representation of three often employed metrics: the correlation coefficient, standard deviation, and RMSE ¹⁴⁴. The plot consists of three axes: radial (correlation coefficient), angular (standard deviation), and concentric circles (RMSE) ¹⁴³. In its standard configuration, the Taylor diagram displays the centred root mean square error (CRMSE) values ¹⁴⁵.

The correlation coefficient (R) assesses the correlation between actual and predicted values and was computed using Eq. (33) ¹²⁷. The value of R varies from -1 to + 1, with -1 specifying perfect inverse correlation and + 1 recommending perfect positive correlation ¹²⁷. The R-value between 0 and 0.2 (or 0 and -0.2), 0.2 and 0.4 (-0.2 and -0.4), 0.4 and 0.6 (-0.4 and -0.6), 0.6 and 0.8 (-0.6 and -0.8), and 0.8 and 1 (-0.8 and -1) depicting very weak, weak, moderate, strong, and very strong correlations, respectively ¹²⁷.

$$R=\frac{\frac{1}{n}{\sum }_1^n(D_{\mathit{act}}-{\overline{D}}_{\mathit{act}})(D_{\mathit{pre}}-{\overline{D}}_{\mathit{pre}})}{\sqrt{\frac{1}{n}}{\sum }_1^n{\left(D_{\mathit{act}},-,{\overline{D}}_{\mathit{act}}\right)}^2\sqrt{\frac{1}{n}{\sum }_1^n{\left(D_{\mathit{pre}},-,{\overline{D}}_{\mathit{pre}}\right)}^2}}$$ 33

where $D_{\mathit{act}}$ = actual value, $D_{\mathit{pre}}$ = predicted value, ${\overline{D}}_{\mathit{act}}$ = mean, ${\overline{D}}_{\mathit{pre}}$ = mean, and n = sample size.

CRMSE will be computed by employing Eq. (34) ¹⁴⁶. A model with the lowest CRMSE value will be better ¹⁴⁷.

$$CRMSE=\sqrt{\frac{1}{N}}\sum _{i=1}^N{\left[\left(m_i,-,\overline{m}\right),-,\left(o_i,-,\overline{o}\right)\right]}^2$$ 34

where $m_i$ = actual value, $o_i$ = observed value,$\overline{m}$ = mean of actual values, and $\overline{o}$ = mean of observed values.

Standard deviation (σ) computes the variability or dispersion of a dataset, compared to the mean value, and varies between 0 and ∞ ¹⁴⁸. A high σ signifies that the observations deviate significantly from the mean, while a minimal σ suggests that they are closely packed around the mean ¹⁴⁸. Standard deviation will be computed utilizing Eq. (35) ¹⁴⁹.

$$\sigma =\sqrt{\frac{1}{(n-1)}}\sum _{i=1}^n{\left(x_i,-,\overline{x}\right)}^2$$ 35

where $x_i$ = sample value,$\overline{x}$ = mean of sample values, and n = sample size.

Equation (36) ¹⁴⁵ depicts the relationship between R, CRMSE, and σ. The geometric relationship between these three metrics highlights the performance of each model relative to a reference model, which can be depicted in the Taylor diagram ¹⁵⁰. It is clear how the models and the reference model are spatially correlated when looking at a symbol’s azimuthal position in the Taylor diagram ¹⁵⁰. The CRMSE is highlighted by the distance (along the concentric circle) each symbol has from the reference model ¹⁵⁰. The symbol denoting the top-performing models will be placed closest to the reference point ¹⁵⁰. The Taylor diagram was produced using MATLAB 2017b software.

$$E\prime ={\sigma }_o^2+{\sigma }_s^2-2{\sigma }_o{\sigma }_s\rho$$ 36

where $E\prime$ = CRMSE, ${\sigma }_o$ = σ of observed values, ${\sigma }_s$ = σ of simulated values, and $\rho$ = correlation coefficient.

Results

Feature selection

The multicollinearity test showed collinearity issues for the following predisposing factors: curvature, plan curvature, and profile curvature since the VIF values are above ten and TOL values are below 0.1 (Table 5). Hence, these three factors were excluded from the modelling. Among the other predisposing factors, lithology has the lowest VIF (1.104) and highest TOL (0.905) values, whereas soil clay content has the highest VIF (2.761) and lowest TOL (0.362) values.

Table 5.

Multicollinearity test results of the predisposing factors.

Predisposing factors	VIF	TOL
Elevation	2.424	0.412
Aspect	1.183	0.845
Built-up area	1.136	0.880
Curvature	42.569	0.023
Fault	1.140	0.877
Geomorphic units	1.625	0.615
LULC	1.167	0.856
Lithology	1.104	0.905
Plan curvature	17.828	0.056
Profile curvature	14.828	0.067
Road	1.149	0.870
Slope	2.334	0.428
Soil texture	1.476	0.677
Soil bulk density	2.649	0.377
Soil clay content	2.761	0.362
Stream	1.364	0.733
TRI	2.072	0.482
TWI	2.463	0.406

The information gain (IG) analysis led to the exclusion of another predisposing factor, namely the ‘built-up area’, which has an IG score of 0.035 (i.e., below the threshold of 0.05). IG metric defined that elevation (IG = 0.521), geomorphic units (IG = 0.480), slope (IG = 0.421), soil bulk density (IG = 0.356), and TWI (IG = 0.264) are the most important predisposing factors (Table 6). Factors such as distance from the road (0.065), distance from the faults (0.066), and soil clay content (0.099) have the lowest IG scores (i.e., below 0.1).

Table 6.

Information gain (IG) scores of the predisposing factors.

Predisposing factors	IG values
Elevation	0.521
Aspect	0.224
Built-up area	0.035
Fault	0.066
Geomorphic units	0.480
LULC	0.174
Lithology	0.157
Road	0.065
Slope	0.421
Soil texture	0.255
Soil bulk density	0.356
Soil clay content	0.099
Stream	0.133
TRI	0.159
TWI	0.264

After this double-step feature selection procedure, the 14 predisposing factors selected for the modelling are slope, elevation, aspect, terrain ruggedness index, topographic wetness index, distance from the stream channel, distance from the road, LULC, distance from the fault, soil texture, soil clay content, soil bulk density, lithology, and geomorphic units (see Supplementary Figure S1). The discarded factors are displayed in Supplementary Figure S2.

Landslide susceptibility maps

The landslide susceptibility maps created utilizing the six models are displayed in Fig. 4. The AUC scores affirmed that all the models are effective, with CatBoost, SVR, SVR-PSO, and SVR-GWO models having excellent performance (AUC score above 0.80). In contrast, CatBoost-PSO and CatBoost-GWO models have outstanding performance (AUC scores above 0.90), according to Hosmer and Lemeshow¹²². The AUC scores also ascertained that the CatBoost model is more effective than the SVR model. Also, integrating the optimization models improved the performance of both the CatBoost and SVR models (Table 7). Among the two optimization algorithms, GWO is slightly better than the PSO (Fig. 5). The MAE, MSE, RMSE, and R² scores for both the training and validation datasets confirmed the above statement. (Tables 8 and 9).

Fig. 4.

Landslide susceptibility maps. Figure created with ArcGIS Pro software (Version 3.1; ESRI, Redlands, California, https://www.esri.com/en-us/arcgis/products/arcgis-pro/overview).

Table 7.

AUC scores of the models.

Models	AUC	Standard error	95% CI
CatBoost	0.899	0.00459	0.890 to 0.907
CatBoost-PSO	0.909	0.00432	0.900 to 0.917
CatBoost-GWO	0.910	0.00426	0.901 to 0.918
SVR	0.840	0.00606	0.829 to 0.851
SVR-PSO	0.858	0.00562	0.848 to 0.868
SVR-GWO	0.868	0.00541	0.858 to 0.878

Fig. 5.

ROC curve and AUC scores of the models.

Table 8.

Validation scores (training dataset).

Models	MAE	MSE	RMSE	R2
CatBoost	0.065	0.027	0.165	0.890
CatBoost-PSO	0.055	0.020	0.142	0.919
CatBoost-GWO	0.054	0.019	0.140	0.921
SVR	0.179	0.063	0.251	0.746
SVR-PSO	0.147	0.044	0.211	0.821
SVR-GWO	0.138	0.042	0.205	0.830

Table 9.

Validation scores (validation dataset).

Models	MAE	MSE	RMSE	R2
CatBoost	0.071	0.032	0.180	0.869
CatBoost-PSO	0.066	0.029	0.171	0.881
CatBoost-GWO	0.065	0.029	0.170	0.883
SVR	0.181	0.063	0.252	0.745
SVR-PSO	0.154	0.050	0.224	0.798
SVR-GWO	0.146	0.047	0.219	0.808

According to the Friedman test, models are considered significantly different when the P-value < 0.05 and insignificantly different when the P-value > 0.05. Since the P-value is below 0.05 for all the models, the null hypothesis (i.e., all models provide similar results) can be rejected (Table 10). This implies a statistically significant difference in performance among the models. The Wilcoxon signed-rank test was applied to assess the performance of paired models. In the case of the Wilcoxon signed-rank test, a statistically significant difference between models exists when the z-statistic is between − 1.96 and + 1.96. The Wilcoxon signed-rank test shows that the CatBoost-PSO and CatBoost-GWO models do not exhibit a statistically significant difference (i.e., z-statistic = 1.017). However, all other model comparisons demonstrate significant differences (i.e., z-statistic > 1.96) (Table 11). A statistically significant difference between the results of the two models indicates that the observed difference in their performances is unlikely to result from random chance. In other words, it suggests that the models provide significantly different maps. From the Taylor diagram (Fig. 6), it is affirmed that the CatBoost-GWO model with a higher correlation coefficient and lower standard deviation scores has higher accuracy than other models.

Table 10.

Friedman test results.

Models	Mean rank	Chi-square	P-value
CatBoost	3.43	61.31	0.0001
CatBoost-PSO	3.42
CatBoost-GWO	3.47
SVR	3.18
SVR-PSO	3.74
SVR-GWO	3.73

Table 11.

Wilcoxon signed-rank test results.

Models	z-statistic	Significance level (α = 0.05)
CatBoost ~ CatBoost-PSO	6.04	Yes
CatBoost ~ CatBoost-GWO	6.39	Yes
CatBoost ~ SVR	14.40	Yes
CatBoost ~ SVR-PSO	10.51	Yes
CatBoost ~ SVR-GWO	8.74	Yes
CatBoost-PSO ~ CatBoost-GWO	1.017	No
CatBoost-PSO ~ SVR	17.17	Yes
CatBoost-PSO ~ SVR-PSO CatBoost-PSO ~ SVR-GWO	13.17 12.12	Yes Yes
CatBoost-GWO ~ SVR CatBoost-GWO ~ SVR-PSO CatBoost-GWO ~ SVR-GWO	17.32 13.14 12.63	Yes Yes Yes
SVR ~ SVR-PSO SVR ~ SVR-GWO SVR-PSO ~ SVR-GWO	4.70 7.85 2.52	Yes Yes Yes

Fig. 6.

The Taylor diagram.

Discussion

The two-stage feature selection approach discarded four predisposing factors (curvature, plan curvature, profile curvature, and built-up area). The exclusion of all curvature related parameters can have manifold reasons. Even if theoretically each curvature parameter provides unique information about terrain features, they are all sensitive to changes in terrain slope and shape and in areas with steep slopes or complex terrain features, curvature factors tend to be highly correlated due to their shared sensitivity to terrain morphology¹⁵¹. Moreover, the spatial resolution of the DEM used to compute the curvature factors probably influences their effectiveness as explanatory variables. This can be ascertained from the study by Catani et al.²⁹, where the curvature factors are found to be weak predictors when the spatial resolution is 10 m, which is close to the 12.5 m resolution considered in this study. It should also be considered that curvature is commonly linked to hydrological processes, surface and subsurface water distribution and soil moisture^29,152. In the study area, and at this spatial resolution of the DEM, other factors may better describe similar processes, such as the TWI or the distance from the streams. Also, the effect of thick vegetative cover may overshadow the influence of curvature on landslide occurrence, especially in areas with dense vegetation, like the WG region in Kerala.

Discarding the built-up area by the second step of the feature selection process may be misleading, as this factor is commonly strongly related to the occurrence of landslides and higher landslide hazard levels^83,153–155. The outcomes of the feature selection are not intended to deny this relationship; on the contrary, we acknowledge a relevant connection between landslide locations and road network and LULC (which identifies among its classes also urbanized areas). Our interpretation is that built-up area is identified as an uninfluential predisposing factor because LULC (including urbanized areas) and road networks are already included in the optimal feature ensemble and already describe the effect of the urban environment on landslide activation without the need to account for a third factor.

Predisposing factors such as elevation, slope, geomorphic units, distance from the road, soil bulk density, and soil texture have higher (scores ≥ 5.00) importance, whereas soil clay content, TRI, lithology, and distance from the stream channel have moderate (scores ≥ 2.00) importance (Fig. 7). Factors such as distance from the fault, aspect, LULC, and TWI have lower (scores < 2.00) importance. Elevation (39.28), slope (15.13), geomorphic units (11.54), distance from the road (6.92), and soil bulk density (5.44) are the top five important predisposing factors. On the other hand, predisposing factors such as TWI (1.40), LULC (1.43), aspect (1.51), and distance from the fault (1.58) are the least important. Previous landslide susceptibility studies conducted in different areas in the WG region by Abraham et al.⁶⁷ found slope, rainfall, elevation, distance from the road, and TWI as the top five important factors, while Jennifer¹⁵⁶ identified rainfall, elevation, slope, displacement, and distance from the road as the top five factors. Similar to the findings of this modelling, factors such as slope, elevation, and distance from the road are also included in the list of the top five factors of these previous studies.

Fig. 7.

Importance of predisposing factors.

The higher influence of elevation can be attributed to higher precipitation and resulting soil moisture in the WG region. The WG is one of the rainiest regions in the tropics and receives heavy rainfall with a seasonal mean of more than 25 mm/day^157,158. The studies by Abraham et al.¹⁵⁹ and Yunus et al.¹⁶⁰ reported the short-duration, intense rainfall in the highly elevated terrain of the WG as one of the predominant causes of landslide initiation. The high ranges of Kerala have a slope as steep as 83.24°, and this is the most important landslide predisposing factor^161–163 because a less intense hydrological trigger is required at steeper slopes to trigger landslides ¹⁶⁴. The landslides in the study area are predominant in the geomorphic units such as denudational hills and valleys and structural hills and valleys. Denudational hills and valleys result from weathering and erosional processes, whereas structural hills and valleys are the combined effect of denudation and tectonism¹⁶⁵. Gneiss and charnockite are the predominant rock types in these areas. Though these are Precambrian crystalline rocks, the extent of chemical weathering and erosional processes and the presence of fractures and joints can facilitate landslides^68,166. Moreover, a significant level of chemical weathering has been reported in these rock types due to high temperature and abundant rainfall, resulting in thick soil covers, a widely consolidated predisposing factor for shallow landslides^7,75,167.

Both the construction of roads by cutting the mountain slopes and the widening of existing roads can affect the stability of the slopes. Yunus et al.¹⁶⁰ reported cutting mountain slopes for road construction as one of the major causes of sliding in the WG region. The importance of road networks is in contrast with other studies⁸³. However, it is supported by the studies of Ajin et al.⁶⁸, Anchima et al.¹⁶⁸, and Thomas et al.¹⁶⁹ in different parts of the WG region in Kerala, which reported road cuttings as one of the crucial landslide inducing factors. The lack of drainage facilities along the mountain roads intensifies the slope failure in the WG region of Kerala⁶⁸. Kanungo et al.⁷ reported that unscientific road construction, i.e., unprotected road cuttings and roads without drainage channels to divert excess water, is common in the WG region of Kerala. Soil bulk density indicates soil compaction and negatively correlates with landslide susceptibility⁹¹. This is because soil bulk density is roughly inversely proportional to soil moisture, porosity, and hydraulic conductivity^170,171. A major portion of the WG region in Kerala is dominated by clayey soil. However, the soil bulk density varies along the clayey soil. This can be attributed to land use practices and the extent of chemical weathering. In the study area, the landslide incidences are concentrated in the WG region, which has a comparatively lower soil bulk density. Clay is the dominant soil type in the high ranges of Kerala. The lateritic soil in the WG region, which is rich in clay content, will retain water and enhance pore water pressure ⁷. Sajinkumar et al.⁷⁵ reported a significant association between laterite soil and landslide occurrences in the WG. Also, clay acts as a potential slip zone, resulting in landslides^172,173. Therefore, regions with higher clay content are more susceptible to slope failures. The high ranges of Kerala are rugged. Ruggedness implies the variation of slope and the undulation of relief¹⁷⁴. The probability of sliding is high in rugged terrain since the shear stress will be greater than a smoother surface. Toe cutting (erosion) by streams and subsequent slope failures is common in the WG region of Kerala during the monsoon season⁷. All 44 major rivers in Kerala originate from the WG region, and the perennial streams with high erosive power are one of the prime factors for landslide initiation in the WG region¹⁶⁰. Faults have a lesser importance since this predisposing factor does not play a relevant role in the case of rainfall-induced landslides. Aspect influences moisture retention and vegetative cover; however, this factor has a lesser influence due to the thick vegetative cover in the WG region. LULC is also less important since the incidences are more in vegetated areas. The study conducted by Hao et al.¹⁷⁵ in the WG region of Kerala also reported that more than 50% of landslides happened in densely vegetated terrain, and, thus, LULC had a lesser influence on the landslides reported in 2018. TWI has the least importance since the wetness is focused on the study area’s lower portion; hence, the impact on the highly elevated WG region is negligible.

This study affirmed that the CatBoost regression algorithm is superior to the SVR algorithm in identifying the landslide susceptibility of the terrain. The AUC score of the CatBoost model is 0.059 more than that of the SVR model, whereas MAE, MSE, and RMSE scores (validation dataset) are respectively 0.110, 0.031, and 0.072 less, and R² score (validation dataset) is 0.124 higher than the SVR model. This is because CatBoost is an ensemble model integrated with a boosting algorithm, which will reduce bias and variance and thereby enhance performance. The studies by Gupta et al.¹⁷⁶ and Li et al.¹⁷⁷ also ascertained that the CatBoost model is more effective than the SVR model. This modelling underlined that the integration of optimization algorithms improved the performance of both ML regression models, and among the two optimization algorithms, GWO is more efficient. In the case of optimization algorithms, the integration of the PSO algorithm improved the AUC score of the SVR model from 0.840 to 0.858 (a 0.018 increase) and the AUC score of the CatBoost model from 0.899 to 0.909 (a 0.010 increase). For the GWO algorithm, there is an increase of 0.028 for the SVR model and 0.011 for the CatBoost model. Jia et al.¹⁷⁸ also found GWO to be the better algorithm when compared to the PSO algorithm. This results from the fact that the GWO algorithm has better exploration and exploitation performance, convergence performance, and optimization accuracy than the PSO algorithm^179,180. However, it is important to stress that this study shows that an optimization algorithm cannot enhance the performance of a low-performing model to the level of a high-performance model. Indeed, in our study, the optimized SVR still had worse validation scores than the unoptimized CatBoost, demonstrating that selecting a good modelling algorithm and designing a proper model implementation scheme is paramount in guiding good susceptibility assessment. However, optimization algorithms can undoubtedly help to refine the results.

To thoroughly compare the results of the models, we followed the procedure proposed by Xiao et al.¹⁸¹, which includes the subtraction of the susceptibility raster values obtained by the different models from a benchmark susceptibility map (the one defined by the model with the highest AUC metric) and inspecting the factor classes responsible for the overestimation and underestimation. The CatBoost-GWO, being the model with the highest AUC metric, is the benchmark model. Hence, the susceptibility raster layers of the other five models were subtracted from the CatBoost-GWO model susceptibility raster layer. The differences in the mapped susceptibility values were contained in the case of CatBoost-GWO and CatBoost-PSO (-0.12 to 0.12) and higher for the other pairs of maps: CatBoost-GWO – CatBoost (-0.35 to 0.35), CatBoost-GWO – SVR-GWO (-0.43 to 0.45), CatBoost-GWO – SVR-PSO (-0.41 to 0.48), and CatBoost-GWO – SVR (-0.38 to 0.77) (Fig. 8). The mean value of the “difference maps” (Fig. 8) was calculated to assess if, in some cases, an overestimation or an underestimation systematically prevails. From Fig. 8, it is affirmed that the difference is larger for the CatBoost-GWO – SVR map, while the difference is smaller for the CatBoost-GWO – CatBoost-PSO, which agrees well with the AUC scores and proves that optimization algorithms can help the models converge towards optimal results.

Fig. 8.

The comparison maps (with uniform legend ranging from -1 to + 1). Figure created with ArcGIS Pro software (Version 3.1; ESRI, Redlands, California, https://www.esri.com/en-us/arcgis/products/arcgis-pro/overview).

Finally, it is important to discuss the main limitations of the present work. It is widely acknowledged that the performance of ML algorithms depends on the quantity and quality of data^182,183. Therefore, our susceptibility assessment can be ameliorated by improving the training and validation dataset (e.g., by timely updates or surveys with different techniques). The inventory of GSI used in this study was often generated by surveying the landslides that damaged roads, buildings, and other infrastructure²¹. Hence, many landslides in remote and rugged terrain may have gone unnoticed. The inventory for 2018 created by Hao et al.²¹ was verified by employing satellite images and field surveys. Hence, detailed verification of the GSIs landslide inventory with remote sensing techniques will help to derive accurate inventories with a high degree of completeness, at least concerning the landslides large and recent enough to be still distinguishable despite vegetation growth, human intervention on the hillsides and geomorphic process that could conceal the landslide morphology.

Another limitation of the study is the combined use of a landslide inventory with a point-like geometry (a point represents each landslide) and a pixel-based approach for the susceptibility assessment. In this case, the approximation of the landslide location could lead to approximations in sampling predisposing factors over a not-very-representative pixel. Ultimately, that could bring uncertainties and errors in the results. For this reason, further research developments in this study area could consider a slope unit-based approach, which may yield better results when working with point-geometry landslide inventories^162,184.

Conclusion

This study identified the WG region of Kerala (India) as a test site to perform a landslide susceptibility assessment testing two novel ML algorithms (CatBoost and SVR) and the integration of two optimization algorithms. A multi-stage feature selection procedure identified the most important predisposing factors: elevation, slope, geomorphic units, distance from the road, and soil bulk density, whereas TWI, LULC, aspect, and distance from the fault had lesser importance. Concerning the susceptibility maps, the study found that CatBoost and SVR alone can provide good results (with the former outperforming the latter). However, the integration of optimization algorithms such as PSO and GWO improved the overall results of the ML algorithms, with GWO having slightly higher performance. Among the six models, CatBoost-GWO had the best performance as defined by several validation metrics, including an AUC score equal to 0.910. It is important to note that optimized SVR models did not outperform non-optimized CatBoost. This study ascertained the relevance of optimization algorithms in enhancing the performance of ML algorithms and producing better susceptibility maps; however, at the same time, the study pointed out that the role of optimization algorithms is refining the results, helping the original model to converge towards optimal predictions; it cannot radically change the predictions of a low-performing model, thus highlighting the importance of setting up a rigorous susceptibility model since the early steps of any research. From a practical standpoint, this work proposes an updated and cutting-edge susceptibility for an Indian state that, despite being very prone to landslides, is covered only by a former work dating back to 2010. The study output will aid the decision-makers in Kerala and other regions with similar geo-environmental conditions in land-use planning, infrastructure development, emergency response planning, community resilience building, insurance and risk management, and long-term planning and adaptation. In light of climate change’s impact on precipitation patterns and extreme weather events, landslide susceptibility studies are essential for informing long-term planning and adaptation strategies, especially in the Global South countries where the impacts are expected to be particularly severe. By integrating landslide susceptibility information into various planning and management initiatives, disaster managers can effectively mitigate the detrimental impacts of landslides and safeguard the well-being of individuals, assets, and the environment.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

RSA performed the analysis and prepared Figs. 1–8 and supplementary data, RSA and SS wrote the manuscript, SS and RF revised the manuscript, and SS and RF supervised the study.

Funding

This study was carried out within the RETURN Extended Partnership and received funding from the European Union Next-GenerationEU (National Recovery and Resilience Plan - NRRP, Mission 4, Component 2, Investment 1.3 - D.D. 1243 2/8/2022, PE0000005).

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-72663-x.