Skip to main content
NCBI home page
Ecology and Evolution logoLink to Ecology and Evolution
. 2020 Oct 16;10(21):12307–12317. doi: 10.1002/ece3.6859

On the importance of predictor choice, modelling technique, and number of pseudo‐absences for bioclimatic envelope model performance

Mirza Čengić 1,✉, Jasmijn Rost 2, Daniela Remenska 3, Jan H Janse 2, Mark A J Huijbregts 1, Aafke M Schipper 1,2
PMCID: PMC7663074  PMID: 33209289

Abstract

Bioclimatic envelope models are commonly used to assess the influence of climate change on species' distributions and biodiversity patterns. Understanding how methodological choices influence these models is critical for a comprehensive evaluation of the estimated impacts. Here we systematically assess the performance of bioclimatic envelope models in relation to the selection of predictors, modeling technique, and pseudo‐absences. We considered (a) five different predictor sets, (b) seven commonly used modeling techniques and an ensemble model, and (c) three sets of pseudo‐absences (1,000 pseudo‐absences, 10,000 pseudo‐absences, and the same as the number of presences). For each combination of predictor set, modeling technique, and pseudo‐absence set, we fitted bioclimatic envelope models for 300 species of mammals, amphibians, and freshwater fish, and evaluated the predictive performance of the models using the true skill statistic (TSS), based on a spatially independent test set as well as cross‐validation. On average across the species, model performance was mostly influenced by the choice of predictor set, followed by the choice of modeling technique. The number of the pseudo‐absences did not have a strong effect on the model performance. Based on spatially independent testing, ensemble models based on species‐specific nonredundant predictor sets revealed the highest predictive performance. In contrast, the Random Forest technique yielded the highest model performance in cross‐validation but had the largest decrease in model performance when transferred to a different spatial context, thus highlighting the need for spatially independent model evaluation. We recommend building bioclimatic envelope models according to an ensemble modeling approach based on a nonredundant set of bioclimatic predictors, preferably selected for each modeled species.

Keywords: biomod2, climate change, ecological niche modeling, model performance, model transferability, species distribution modeling (SDM)

Bioclimatic envelope models are commonly used to assess the influence of climate change on species' distributions and biodiversity patterns. We systematically evaluated the performance of bioclimatic envelope models in relation to the selection of predictors, modelling technique, and pseudo‐absences. Model performance was mostly influenced by the choice of predictor set, followed by the choice of modelling technique, and based on our results, we recommend building bioclimatic envelope models according to an ensemble modelling approach based on a nonredundant set of bioclimatic predictors, preferably selected for each modelled species.

graphic file with name ECE3-10-12307-g004.jpg

1. INTRODUCTION

In the face of ongoing climate change it is crucial to quantify and understand its impacts on biodiversity. Threats of future climate change to biodiversity are commonly quantified with bioclimatic envelope models, that is, species distribution models (SDMs) that link species presence records with climatic variables to project the future distribution of species in response to climate change (Elith & Leathwick, 2009; Guisan & Thuiller, 2005; Pacifici et al., 2015; Pearson & Dawson, 2003; Thomas et al., 2004). Methodological choices made during the development of SDMs vary widely and may have considerable influence on the model output (Araújo et al., 2005, 2019; Araújo & Peterson, 2012; Barry & Elith, 2006; Pearson et al., 2006; Thuiller et al., 2019). This points at a clear need to understand how methodological choices affect the performance of SDMs in general and bioclimatic envelope models in particular.

Key methodological aspects that influence SDMs include the selection of predictor variables, selection of modeling technique, and selection of pseudo‐absence data (Araújo et al., 2019; Brun et al., 2019; Merow et al., 2014; Thuiller et al., 2019; Warren et al., 2019). Although there is consensus that predictor variables should be ecologically relevant to the species of concern, often this is challenging due to data limitations or because the ecological requirements of the species are not sufficiently known (Araújo et al., 2019). Many studies therefore rely on a (semi‐)automated selection of complementary (i.e., not too highly correlated) predictors from a broader set of variables that are expected to be relevant (Barbet‐Massin & Jetz, 2014; Bradie & Leung, 2017; Dormann et al., 2013; Petitpierre et al., 2017).

When it comes to the choice of modeling technique, there is a lack of consensus as to which technique is most suited for which SDM purpose (Araújo et al., 2019; Benito et al., 2013). Some studies build models based on machine‐learning approaches such as Random Forest or MaxEnt (Bahn & McGill, 2013; Elith et al., 2011; Guisan et al., 2007), while others apply regression‐based approaches (Guisan et al., 2002; Guisan & Zimmermann, 2000; Li & Wang, 2013) or prefer an ensemble of several modeling techniques (Araújo & New, 2007; Marmion et al., 2009; Rapacciuolo et al., 2012). There is evidence, however, that ensemble modeling is the preferred choice if the SDMs are to be used for projections, which are key in future climate threat assessments (Araújo et al., 2005).

The selection of pseudo‐absence or background data is a third key consideration in SDM building, given that the majority of species observations concern presence‐only data (Ponder et al., 2001). Relevant aspects include the number of pseudo‐absences as well as their spatial distribution (i.e., extent and geographic stratification), which may affect model performance as well as the relative importance of predictor variables (Barbet‐Massin et al., 2012; Phillips et al., 2009; VanDerWal et al., 2009).

The impact of predictor set, modeling technique, and pseudo‐absence selection on the performance of bioclimatic envelope models has been studied in isolation (Barbet‐Massin et al., 2012; Beaumont et al., 2005; Moreno‐Amat et al., 2015; Pearson et al., 2006; Pliscoff et al., 2014) or for two of the three factors (Brun et al., 2019; Bucklin et al., 2015; Dormann et al., 2008; Jarnevich et al., 2017; Petitpierre et al., 2017; Verbruggen et al., 2013; Warren et al., 2019). However, to our knowledge, no study so far has systematically evaluated the combined effect of predictor set, modeling technique, and pseudo‐absence selection on the performance of bioclimatic envelope models. Here we seek to fill that gap by systematically varying the choice of predictor set, modeling technique and number of pseudo‐absences and evaluating the performance of the resulting bioclimatic envelope models.

We evaluated five sets of bioclimatic predictors, seven modeling techniques plus a consensus ensemble model, and three sets of pseudo‐absences. We selected a sample of 100 species from each of three major taxonomic groups, that is, mammals, amphibians, and freshwater fish, as the performance of bioclimatic envelope models has proven to differ among species groups (Heikkinen et al., 2012). For each species, we fitted a bioclimatic envelope model for each combination of predictor set, modeling technique, and pseudo‐absence set. We evaluated the models with cross‐validation (random split sample) and based on their transferability to a spatially independent region, made up by different biomes (for mammals and amphibians) or different catchments (for fish). Thus, we test both model accuracy (the ability of the model to predict well within the space and time covered by the input data) and model generality (the ability of the model to predict well in another region or time), whereby the latter is more critical for bioclimatic envelope models given their aim to forecast changes (Araújo et al., 2019).

We asked the following research questions:

  1. How sensitive is the performance of bioclimatic envelope models to the choice of predictor set, modeling technique, and number of pseudo‐absences?

  2. Which combination of predictor set, modeling technique, and pseudo‐absence set yields the highest model performance?

  3. To what extent are the outcomes contingent on the evaluation method, that is, independent testing as opposed to cross‐validation?

2. MATERIALS AND METHODS

2.1. Species data

We retrieved species presence data from expert range maps provided by the IUCN (IUCN, 2018), similar to other recent studies that developed and applied bioclimatic envelope models (Hof et al., 2018; Visconti et al., 2016). The IUCN Red List assesses the conservation status of individual (sub)species and provides range maps that characterize the global distribution of species of several taxonomic groups. We preferred using range maps over global point records databases (e.g., GBIF), as range maps are expected to be less spatially biased (Fourcade, 2016; Merow et al., 2016). We included not only areas where the species are extant but also areas where the species are currently extinct, assuming that extinction was driven by factors other than climate change (Faurby & Svenning, 2015). We obtained data for each species by rasterizing its IUCN range map to a resolution of 5 arc‐minutes and taking each raster cell that falls inside of the range of the species as a presence record.

We retrieved pseudo‐absences from an area outside the range maps, but potentially within reach of the species (Elith & Leathwick, 2009; Phillips et al., 2009). For mammals and amphibians, we used the terrestrial biomes encompassing the species' ranges to sample pseudo‐absences (Vences & Köhler, 2008). For freshwater fish, we used the catchments encompassing the ranges (Berra, 2001). The terrestrial biomes map was derived from the Terrestrial Ecoregions of the World map (Olson et al., 2001). The catchments map was constructed by combining the hydrography of Hydrosheds and Hydro1k at 30 arc‐seconds and derived using the “Basin” function in ArcGIS, with a total of 152,739 catchments (Barbarossa et al., 2018). We rasterized the biome and catchment maps at a 5 arc‐minutes resolution to match the spatial resolution of the rasterized species ranges. For each species, we then randomly selected an initial set of maximum 100,000 pseudo‐absence raster cells outside the range of that species but within the surrounding biomes or catchments.

Next, we split the species data into training and test data. We created two test datasets for each species, that is, a cross‐validation set and a spatially independent set. For the cross‐validation set, we randomly took 80% of the data for model training and used the remaining 20% for model testing, without considering the spatial structure of the data. To create the spatially independent test set, we sorted the biomes (for mammals and amphibians) or catchments (for fish) based on the number of presence and pseudo‐absence points in the area. The records in the biomes or catchments were then alternately assigned to the training or test set, starting with the training set to ensure that the training set would have the largest number of records, resulting in at least half of the species data assigned to the training data set. We then selected species for which both the training and test set included at least 100 points in the species range and 10,000 pseudo‐absences. From this list of species, we randomly selected 100 species from each taxonomic group for our analysis. A list of species is provided in Table S1.

2.2. Predictors, modeling techniques, and pseudo‐absence sets

2.2.1. Predictors

We retrieved the climate data for fitting the bioclimatic envelope models from the global 5 arc‐minutes resolution Worldclim 1.4 set (Hijmans et al., 2005). The Worldclim dataset contains 19 bioclimatic variables that represent aspects of temperature and precipitation that are considered particularly relevant for species and ecosystems. From this dataset we compiled five different sets of predictors. The larger the set of predictor variables, the more likely that it includes ecologically meaningful variables for the species of concern, yet at the expense of an increased risk of model overfitting and spurious relationships (Barbet‐Massin & Jetz, 2014; Merow et al., 2014). We defined four generic predictor sets consisting of an increasing number of variables and a fifth species‐specific set of nonredundant variables specific to each species, as follows:

  1. A predictor set consisting of two variables representing mean climate, that is, annual mean temperature and annual precipitation (bio1 and bio12).

  2. A predictor set consisting of four variables, representing mean climate and seasonality, that is, annual mean temperature and annual precipitation and the seasonality of temperature and precipitation (bio1, bio4, bio12 and bio15).

  3. A set of nonredundant bioclimatic variables identified based on multicollinearity within the entire dataset (i.e., global extent). To that end, we calculated variance inflation factors (VIFs) using the “vifstep” function in R package “usdm” (Naimi et al., 2014). We excluded a variable if its VIF value was above 10, starting with the variable with the highest value (Duque‐Lazo et al., 2016; Zuur et al., 2010). This resulted in the following 10 remaining predictors: bio2, bio3, bio4, bio8, bio9, bio13, bio14, bio15, bio18 and bio19.

  4. A predictor set consisting of all 19 bioclimatic predictors, reflecting the strategy of including a broad set of possible bioclimatic predictor variables and relying on the modeling technique to identify relevant variables or penalize models with collinear ones (Brun et al., 2019; Dormann et al., 2013).

  5. A species‐specific set of nonredundant variables, identified by excluding collinear variables based on a variance inflation factor (VIF) larger than 10 calculated based on the input data specific to each individual species. We calculated the VIF values according to the approach as explained for predictor set 3. The number of remaining variables for each species is given in Table S1.

2.2.2. Modeling techniques

We used seven modeling techniques to fit the bioclimatic models, including one classification method (Classification Tree Analysis, CTA), three regression techniques (Generalized Linear Model, GLM; Generalized Additive Model, GAM; Multivariate Adaptive Regression Splines, MARS) and three machine‐learning techniques (Generalized Boosted Model, GBM; Random Forest, RF; Maximum Entropy, MaxEnt). These techniques are commonly used in bioclimatic envelope modelling (Araújo et al., 2005; Bahn & McGill, 2013; Barbet‐Massin & Jetz, 2014; Beaumont et al., 2016; Jeschke & Strayer, 2008). Although various studies already indicated that some of these techniques perform better than others (Araújo et al., 2005; Bahn & McGill, 2013; Benito et al., 2013; Guisan et al., 2007; Heikkinen et al., 2012; Rapacciuolo et al., 2012; Ruiz‐Navarro et al., 2016), this was typically tested in cross‐validation, and their transferability in space has not yet been extensively tested. Besides using the individual techniques, we evaluated also an ensemble model consisting of the average of the fitted models across the seven techniques (Araújo & New, 2007). We selected individual models only if their true skill statistic (TSS) value was higher than 0.7, and we weighted the models according to their TSS value (Marmion et al., 2009).

2.2.3. Pseudo‐absences

We selected three sets of pseudo‐absences from the full set of pseudo‐absences available in the training data set. The first pseudo‐absence set consisted of 10,000 randomly selected pseudo‐absences. For the second set, 1,000 pseudo‐absences were randomly selected. For the last set, the number of selected pseudo‐absences was equal to the number of presences available for model fitting (i.e., in the training dataset). These sets were selected based on recommendations regarding the number of pseudo‐absences to be used for different modeling techniques (Barbet‐Massin et al., 2012). We applied equal weighting of presences and pseudo‐absences in model fitting, as recommended by Barbet‐Massin et al. (2012).

2.2.4. Model fitting and evaluation

We fitted the models using the “biomod2” package in R (Thuiller et al., 2016), which comprises the most commonly used techniques in species distribution modeling. We kept the default values of tuning and fitting parameters for each modeling technique, as given in the biomod2 package. The combination of five predictor sets, seven modeling techniques and one ensemble, and three pseudo‐absence sets resulted in 120 fitted models for each species, hence a total of 36,000 models for all 300 species. We evaluated the bioclimatic envelope models based on their transferability to the spatially independent test set. For comparison, we also performed cross‐validation within the training dataset that includes all biomes and catchments. We quantified model performance based on the TSS value (Allouche et al., 2006).

To assess the sensitivity of model performance to the number of predictors, modeling technique and pseudo‐absences selection, we fitted linear mixed effects models relating TSS values to these modeling choices using the “lmer” function from the “lme4” package in R (Bates et al., 2015). We fitted a separate model per taxonomic group and per evaluation method (spatially independent testing or cross‐validation) relating the TSS values to predictor set (factor with five levels), modeling technique (factor with eight levels), and pseudo‐absences set (factor with three levels), including their interactions. We included species as a random effect (intercept) in order to account for having repeated measures within species (i.e., the 120 combinations of predictor set, modeling techniques and pseudo‐absences set). Because model performance and transferability may depend on the degree of similarity in environmental conditions between training and extrapolation (Qiao et al., 2019), we added a measure of environmental niche overlap between the training and testing data as an additional fixed effect. To that end, we calculated the overlap in the environmental space between the training and testing datasets based on the predictor set with four bioclimatic variables, measured by the Jaccard similarity index (Jost, 2006) with the R package hypervolume (Blonder et al., 2014). As the overlap estimate can be inflated with the addition of more dimension (Blonder et al., 2014), we used the overlap values derived from the predictor set containing four variables with normalized values. Per taxonomic group and validation method (spatially independent versus cross‐validation), we selected the best model based on the lowest Akaike information criterion (AIC). To quantify the contribution of each random and fixed effect to the variability in model performance, we retrieved the percentage of explained variance using the “r2beta” function with the standardized generalized variance approach from the “r2glmm” package (Jaeger et al., 2017).

3. RESULTS

3.1. Variation in model performance

TSS values derived with spatially independent data ranged from 0.35 to 0.81 for mammals, from 0.44 to 0.92 for amphibians, and from 0.11 to 0.47 for fish (Figure 1). TSS values obtained in spatially independent testing were consistently lower than those obtained in cross‐validation, which ranged from 0.68 to 1 for mammals, from 0.69 to 1 for amphibians, and from 0.29 to 0.97 for fish. We found the lowest average TSS values for fish, while amphibians and mammals had higher values. In general, a species‐specific set of nonredundant variables resulted in the highest model performance. In addition, model performance generally increased with the number of predictors, whereby the largest increase in model performance occurred between the sets with two and four predictors. Models based on the full set of 19 bioclimatic predictors performed no or only marginally better than models based on a nonredundant set of 10 predictors. On average, the ensemble model performed best in the spatially independent testing, while in the cross‐validation the Random Forest technique resulted in the highest predictive power (Figures 1 and 2). The MaxEnt models had the lowest performance on average both in the spatially independent testing and the cross‐validation (Figure 2), which was most evident for the spatially independent evaluation of the mammal species models (Figure 1). There were no evident differences in model performance between the three pseudo‐absence sets (Figures S1 and S2).

Figure 1.

Figure 1

Open in a new tab

Mean TSS values for each combination of predictor set and modeling technique for spatially independent testing (panels a–c) and cross‐validation (d–f). The results are given for all three pseudo‐absence (PA) datasets, including 1,000 PA (panels a and d), 10,000 PA (panels b and e), and a number equal to the number presences (P = PA, panels c and f). Predictor sets include two variables (p2), four variables (p4), a nonredundant set of 10 variables (p10), all 19 bioclimatic variables (p19), and species‐specific nonredundant sets (pSp). CTA = Classification Tree Analysis; GAM = Generalized Additive Model; GBM = Generalized Boosted Model; GLM = Generalized Linear Model; MARS = Multivariate Adaptive Regression Splines; MaxEnt = Maximum Entropy; RF = Random Forest. Mean TSS values and corresponding 5 and 95 percentiles are given in Tables S1 and S3

Figure 2.

Figure 2

Open in a new tab

Modeling technique ranking based on TSS values, shown as distribution of ranks across the species, pseudo‐absence sets and predictor sets. Modeling technique rankings are given for Mammals (panels a and d), Amphibians (panels a and e), and Fish (panels c and f), both in spatially independent testing (panels a‐c) and cross‐validation (panels d‐f). Model ranking of 1 is the highest ranked model (highest TSS value), while 8 is the lowest ranked model. Boxplots show median rank values, quartiles, and 1.5 times of the interquartile distance of the ranks; diamonds represent means. CTA = Classification Tree Analysis; GAM = Generalized Additive Model; GBM = Generalized Boosted Model; GLM = Generalized Linear Model; MARS = Multivariate Adaptive Regression Splines; MaxEnt = Maximum Entropy; RF = Random Forest

3.2. Sensitivity to predictor, modeling technique, and pseudo‐absences selection

Model performance was mostly related to the choice of predictors set, both for spatially independent validation and cross‐validation (Figure 3). The only exception was the performance of the fish models in spatially independent validation, which was mostly related to the overlap between the training and testing data (niche overlap). The choice of modeling technique was the second most important factor explaining variation in model performance, in interaction with the predictor set, or on its own. The number of pseudo‐absences explained only a very small part of the variation in model performance. This ranking did not change when we excluded the smallest and largest predictor sets (Figure S3).

Figure 3.

Figure 3

Open in a new tab

Proportions of variance in model performance explained by predictor set, modeling technique, number of pseudo‐absences and niche overlap. Proportions of explained variance are given for Mammals (panels a and d), Amphibians (panels b and e), and Fish (panels c and f), both in spatially independent testing (panels a–c) and cross‐validation (panels d–f). Proportions of variance explained were rescaled to sum to 100% across the factors. Underlying data is in Table S4

4. DISCUSSION

In this study, we evaluated the sensitivity of bioclimatic envelope model performance to choices in predictor set, modeling technique, and number of pseudo‐absences. Based on spatially independent testing, that is, the preferred testing strategy for bioclimatic envelope models (Araújo et al., 2005; Bahn & McGill, 2013; Roberts et al., 2017), we found that the highest model performance is obtained if the selection of bioclimatic predictors is tailored to the species of concern (Figure 1). In addition, we found that model performance generally improves by increasing the number of predictors (Figure 1). In part, this may reflect a mere effect of chance, as each predictor (even a random one) may explain a bit of additional variation just by chance (Fourcade et al., 2018). However, we also found that the relative increase in model performance became smaller or even declined slightly when the full set with 19 predictors was used rather than more parsimonious predictor sets with 10 variables. Our findings thus indicate that the set of 19 predictors can be profitably reduced to a (preferably species‐specific) nonredundant set based on an evaluation of multicollinearity. This would reduce the risk of erroneous model inference or projections arising from collinearity issues, due to the inflated regression parameters or models being projected to variables with different levels of collinearity (Dormann et al., 2013).

Our results further indicate that an ensemble model is the preferred option for bioclimatic envelope modeling, as the ensemble generally performed best based on the spatially independent testing. Although individual modeling techniques may outperform the ensemble model in some cases, overall the ensemble model demonstrated a higher and a more consistent performance compared to the individual techniques (Figure 2). This is in line with other studies that demonstrated that model ensembles may outperform or give more robust estimates compared to the individual modeling techniques that are used to build a consensus model (Araújo & New, 2007; Buisson et al., 2010; Crimmins et al., 2013; Marmion et al., 2009).

The Random Forest models generally performed best in cross‐validation but showed a clear drop in performance when evaluated based on spatially independent data, both in absolute terms and compared to other modeling techniques (Figure 2, Table S2). High performance of RF and other more complex modeling techniques in cross‐validation is commonly observed, as more complex techniques tend to fit the relationship closely to the data (Merow et al., 2014; Randin et al., 2006), but perform poorly when models are transferred into novel spatial or temporal contexts. Similar results for RF were shown in a study that compared random data partitioning with using temporally independent data for model evaluation, where RF performed well when assessed on internal cross‐validation, yet poorly when assessed with an independent dataset (Crimmins et al., 2013).

We found that MaxEnt models often performed relatively poorly (Figures 1 and 2), despite the fact that MaxEnt is a widely used modeling technique for SDMs (Warren & Seifert, 2011). Applying background data (i.e., sampled from the entire study area) rather than pseudo‐absences (i.e., sampled outside of the species' presence ranges), as recommended for MaxEnt (Phillips et al., 2006), did not result in an improvement in the performance of the MaxEnt models (Figure S4). Our findings are in line with previous studies indicating that MaxEnt is prone to overfitting and does not perform well when a model is transferred to a different spatial or temporal context (Peterson et al., 2007; Radosavljevic & Anderson, 2014). However, the performance of the MaxEnt models clearly increased when fitted based on the species‐specific input datasets (Figure 1), underlining the need to tune MaxEnt models to the species of concern (Elith et al., 2011; Radosavljevic & Anderson, 2014).

We found that the number of pseudo‐absences did not have a strong effect on model performance, in particular when compared with the effect of the predictor set. Although it has been recommended to tailor the number of pseudo‐absences to the technique of choice (Barbet‐Massin et al., 2012), our results indicate only a weak interactive effect of the number of pseudo‐absences and the modeling technique for mammals (Figure 3). Only for the amphibian models, there was a small effect of the number of pseudo‐absence on its own. Our results thus suggest that selecting a fixed number of pseudo‐absences for all species is a defensible choice (Bucklin et al., 2015; Petitpierre et al., 2017; Richmond et al., 2010).

Differences in model performance among the taxonomic groups could result from differences in the background regions from which the pseudo‐absences were selected. For mammals and amphibians the background region consisted of biomes encompassing the species' range, whereas for fish the background region was restricted to the catchments in which the species occurs. The smaller spatial extent of catchments and resulting smaller background area from which values are sampled might explain the lower mean TSS for fish, as a more restricted background region (i.e., smaller environmental gradients) typically reduces model performance (Leroy et al., 2018; Lobo et al., 2008; VanDerWal et al., 2009). The smaller background region for fish species may also explain the larger importance of environmental niche overlap between training and testing data in explaining the variation in TSS (Figure 3). Smaller spatial units to delineate spatially independent training and testing datasets may result in a larger variation in niche overlap between different species (Figure S5) hence a larger influence on variation in model performance.

The relatively low model performance for freshwater fish may also reflect that we fitted the models based on bioclimatic predictors representing air temperature and precipitation, whereas the distributions of freshwater fish is also influenced by other factors (Barbarossa et al., 2020; Knouft & Ficklin, 2017; McGarvey et al., 2018). However, two key factors underlying the occurrence of freshwater fish (i.e., water temperature and streamflow) depend on air temperature and precipitation (Barbarossa et al., 2018; Knouft & Ficklin, 2017), and it has indeed been demonstrated that bioclimatic variables can be effectively used as proxies for macro‐scale modeling of the distribution of freshwater species (Domisch et al., 2015; Frederico et al., 2014; McGarvey et al., 2018).

In conclusion, our study shows a clear impact of predictor and modeling technique selection on the performance of bioclimatic envelope models. Our findings indicate that bioclimatic envelope models are preferably built based on a species‐specific, nonredundant set of predictor variables, and an ensemble modeling approach. A pragmatic choice can be made for the number of pseudo‐absences, for example, based on runtime, as the number of pseudo‐absences appeared to be less influential on model performance. In contrast to the results of the spatially independent testing, cross‐validation pointed toward Random Forest as the preferred modeling technique. This finding highlights that cross‐validation does not necessarily identify the best combination of modeling technique and predictor set to establish a predictive model, as displayed by a drop in performance when comparing cross‐validation and spatially independent validation.

CONFLICT OF INTERESTS

None declared.

AUTHOR CONTRIBUTIONS

Mirza Čengić: Data curation (lead); formal analysis (lead); investigation (lead); methodology (lead); software (lead); validation (lead); visualization (lead); writing–original draft (equal); writing–review and editing (equal). Jasmijn Rost: Conceptualization (equal); formal analysis (equal); investigation (equal); methodology (equal); software (equal); visualization (equal); writing–original draft (lead). Daniela Remenska: Data curation (equal); software (equal). Jan H. Janse: Writing–review and editing (equal). Mark A. J. Huijbregts: Conceptualization (equal); methodology (equal); supervision (equal); writing–review and editing (equal). Aafke M. Schipper: Conceptualization (lead); methodology (equal); supervision (lead); writing–review and editing (lead).

Supporting information

Appendix S1

Click here for additional data file. (3MB, docx)

ACKNOWLEDGEMENTS

This study was funded by PBL Netherlands Environmental Assessment Agency as part of the GLOBIO project (www.globio.info). We thank Sido Mylius for assisting us with coding and arranging sufficient computing power. We thank Valerio Barbarossa for our discussions and for providing the catchments map, and we thank Luca Santini and Ana Benítez‐López for useful discussions and comments.

Čengić M, Rost J, Remenska D, Janse JH, Huijbregts MAJ, Schipper AM. On the importance of predictor choice, modeling technique, and number of pseudo‐absences for bioclimatic envelope model performance. Ecol. Evol. 2020;10:12307–12317. 10.1002/ece3.6859

Mirza Čengić and Jasmijn Rost contributed equally to this manuscript

DATA AVAILABILITY STATEMENT

The species data used in this study are publicly available on the IUCN Red List website (https://www.iucnredlist.org/resources/spatial‐data‐download). Climate data retrieved from the WorldClim v1.4 database are publicly available on the WorldClim website (https://worldclim.org/data/v1.4/worldclim14.html). The code of the analysis is archived at https://doi.org/10.5281/zenodo.4022773 as the GitHub repository. With this code and the input data specified above, results in this paper can be reproduced.

REFERENCES

  1. Allouche, O. , Tsoar, A. , & Kadmon, R. (2006). Assessing the accuracy of species distribution models: Prevalence, kappa and the true skill statistic (TSS). Journal of Applied Ecology, 43, 1223–1232. 10.1111/j.1365-2664.2006.01214.x [DOI] [Google Scholar]
  2. Araújo, M. B. , Anderson, R. P. , Barbosa, A. M. , Beale, C. M. , Dormann, C. F. , Early, R. , Garcia, R. A. , Guisan, A. , Maiorano, L. , Naimi, B. , O’Hara, R. B. , Zimmermann, N. E. , & Rahbek, C. (2019). Standards for distribution models in biodiversity assessments. Science Advances, 5, eaat4858 10.1126/sciadv.aat4858 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Araújo, M. B. , & New, M. (2007). Ensemble forecasting of species distributions. Trends in Ecology & Evolution, 22, 42–47. 10.1016/j.tree.2006.09.010 [DOI] [PubMed] [Google Scholar]
  4. Araújo, M. B. , Pearson, R. G. , Thuiller, W. , & Erhard, M. (2005). Validation of species–climate impact models under climate change. Global Change Biology, 11, 1504–1513. 10.1111/j.1365-2486.2005.01000.x [DOI] [Google Scholar]
  5. Araújo, M. B. , & Peterson, A. T. (2012). Uses and misuses of bioclimatic envelope modeling. Ecology, 93, 1527–1539. 10.1890/11-1930.1 [DOI] [PubMed] [Google Scholar]
  6. Bahn, V. , & McGill, B. J. (2013). Testing the predictive performance of distribution models. Oikos, 122, 321–331. 10.1111/j.1600-0706.2012.00299.x [DOI] [Google Scholar]
  7. Barbarossa, V. , Huijbregts, M. A. J. , Beusen, A. H. W. , Beck, H. E. , King, H. , & Schipper, A. M. (2018). FLO1K, global maps of mean, maximum and minimum annual streamflow at 1 km resolution from 1960 through 2015. Scientific Data, 5, 180052 10.1038/sdata.2018.52 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Barbarossa, V. , Schmitt, R. J. P. , Huijbregts, M. A. J. , Zarfl, C. , King, H. , & Schipper, A. M. (2020). Impacts of current and future large dams on the geographic range connectivity of freshwater fish worldwide. Proceedings of the National Academy of Sciences, 117, 3648–3655. 10.1073/pnas.1912776117 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Barbet‐Massin, M. , & Jetz, W. (2014). A 40‐year, continent‐wide, multispecies assessment of relevant climate predictors for species distribution modelling. Diversity and Distributions, 20, 1285–1295. 10.1111/ddi.12229 [DOI] [Google Scholar]
  10. Barbet‐Massin, M. , Jiguet, F. , Albert, C. H. , & Thuiller, W. (2012). Selecting pseudo‐absences for species distribution models: How, where and how many?. Methods in Ecology and Evolution, 3, 327–338. 10.1111/j.2041-210X.2011.00172.x [DOI] [Google Scholar]
  11. Barry, S. , & Elith, J. (2006). Error and uncertainty in habitat models. Journal of Applied Ecology, 43, 413–423. 10.1111/j.1365-2664.2006.01136.x [DOI] [Google Scholar]
  12. Bates, D. , Mächler, M. , Bolker, B. , & Walker, S. (2015). Fitting linear mixed‐effects models using lme4. Journal of Statistical Software, 67, 1–48. 10.18637/jss.v067.i01 [DOI] [Google Scholar]
  13. Beaumont, L. J. , Graham, E. , Duursma, D. E. , Wilson, P. D. , Cabrelli, A. , Baumgartner, J. B. , Hallgren, W. , Esperón‐Rodríguez, M. , Nipperess, D. A. , Warren, D. L. , Laffan, S. W. , & VanDerWal, J. (2016). Which species distribution models are more (or less) likely to project broad‐scale, climate‐induced shifts in species ranges? Ecological Modelling, 342, 135–146. 10.1016/j.ecolmodel.2016.10.004 [DOI] [Google Scholar]
  14. Beaumont, L. J. , Hughes, L. , & Poulsen, M. (2005). Predicting species distributions: Use of climatic parameters in BIOCLIM and its impact on predictions of species’ current and future distributions. Ecological Modelling, 186, 251–270. 10.1016/j.ecolmodel.2005.01.030 [DOI] [Google Scholar]
  15. Benito, B. M. , Cayuela, L. , & Albuquerque, F. S. (2013). The impact of modelling choices in the predictive performance of richness maps derived from species‐distribution models: Guidelines to build better diversity models. Methods in Ecology and Evolution, 4, 327–335. 10.1111/2041-210x.12022 [DOI] [Google Scholar]
  16. Berra, T. (2001). Freshwater fish distribution. Academic Press. [Google Scholar]
  17. Blonder, B. , Lamanna, C. , Violle, C. , & Enquist, B. J. (2014). The n‐dimensional hypervolume. Global Ecology and Biogeography, 23, 595–609. 10.1111/geb.12146 [DOI] [Google Scholar]
  18. Bradie, J. , & Leung, B. (2017). A quantitative synthesis of the importance of variables used in MaxEnt species distribution models. Journal of Biogeography, 44, 1344–1361. 10.1111/jbi.12894 [DOI] [Google Scholar]
  19. Brun, P. , Thuiller, W. , Chauvier, Y. , Pellissier, L. , Wüest, R. O. , Wang, Z. , & Zimmermann, N. E. (2019). Model complexity affects species distribution projections under climate change. Journal of Biogeography, 47, 130–142. 10.1111/jbi.13734 [DOI] [Google Scholar]
  20. Bucklin, D. N. , Basille, M. , Benscoter, A. M. , Brandt, L. A. , Mazzotti, F. J. , Romañach, S. S. , Speroterra, C. , & Watling, J. I. (2015). Comparing species distribution models constructed with different subsets of environmental predictors. Diversity and Distributions, 21, 23–35. 10.1111/ddi.12247 [DOI] [Google Scholar]
  21. Buisson, L. , Thuiller, W. , Casajus, N. , Lek, S. , & Grenouillet, G. (2010). Uncertainty in ensemble forecasting of species distribution. Global Change Biology, 16, 1145–1157. 10.1111/j.1365-2486.2009.02000.x [DOI] [Google Scholar]
  22. Crimmins, S. M. , Dobrowski, S. Z. , & Mynsberge, A. R. (2013). Evaluating ensemble forecasts of plant species distributions under climate change. Ecological Modelling, 266, 126–130. 10.1016/j.ecolmodel.2013.07.006 [DOI] [Google Scholar]
  23. Domisch, S. , Jähnig, S. C. , Simaika, J. P. , Kuemmerlen, M. , & Stoll, S. (2015). Application of species distribution models in stream ecosystems: The challenges of spatial and temporal scale, environmental predictors and species occurrence data. Fundamental and Applied Limnology/Archiv für Hydrobiologie, 186, 45–61. 10.1127/fal/2015/0627 [DOI] [Google Scholar]
  24. Dormann, C. F. , Elith, J. , Bacher, S. , Buchmann, C. , Carl, G. , Carré, G. , Marquéz, J. R. G. , Gruber, B. , Lafourcade, B. , Leitão, P. J. , Münkemüller, T. , McClean, C. , Osborne, P. E. , Reineking, B. , Schröder, B. , Skidmore, A. K. , Zurell, D. , & Lautenbach, S. (2013). Collinearity: A review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36, 27–46. 10.1111/j.1600-0587.2012.07348.x [DOI] [Google Scholar]
  25. Dormann, C. F. , Purschke, O. , Márquez, J. R. G. , Lautenbach, S. , & Schröder, B. (2008). Components of uncertainty in species distribution analysis: A case study of the great grey shrike. Ecology, 89, 3371–3386. 10.1890/07-1772.1 [DOI] [PubMed] [Google Scholar]
  26. Duque‐Lazo, J. , van Gils, H. , Groen, T. A. , & Navarro‐Cerrillo, R. M. (2016). Transferability of species distribution models: The case of Phytophthora cinnamomi in Southwest Spain and Southwest Australia. Ecological Modelling, 320, 62–70. 10.1016/j.ecolmodel.2015.09.019 [DOI] [Google Scholar]
  27. Elith, J. , & Leathwick, J. R. (2009). Species distribution models: Ecological explanation and prediction across space and time. Annual Review of Ecology Evolution and Systematics, 40, 677–697. 10.1146/annurev.ecolsys.110308.120159 [DOI] [Google Scholar]
  28. Elith, J. , Phillips, S. J. , Hastie, T. , Dudík, M. , Chee, Y. E. , & Yates, C. J. (2011). A statistical explanation of MaxEnt for ecologists. Diversity and Distributions, 17, 43–57. 10.1111/j.1472-4642.2010.00725.x [DOI] [Google Scholar]
  29. Faurby, S. , & Svenning, J.‐C. (2015). Historic and prehistoric human‐driven extinctions have reshaped global mammal diversity patterns. Diversity and Distributions, 21, 1155–1166. 10.1111/ddi.12369 [DOI] [Google Scholar]
  30. Fourcade, Y. (2016). Comparing species distributions modelled from occurrence data and from expert‐based range maps. Implication for predicting range shifts with climate change. Ecological Informatics, 36, 8–14. 10.1016/j.ecoinf.2016.09.002 [DOI] [Google Scholar]
  31. Fourcade, Y. , Besnard, A. G. , & Secondi, J. (2018). Paintings predict the distribution of species, or the challenge of selecting environmental predictors and evaluation statistics. Global Ecology and Biogeography, 27, 245–256. 10.1111/geb.12684 [DOI] [Google Scholar]
  32. Frederico, R. G. , Marco, P. D. , & Zuanon, J. (2014). Evaluating the use of macroscale variables as proxies for local aquatic variables and to model stream fish distributions. Freshwater Biology, 59, 2303–2314. 10.1111/fwb.12432 [DOI] [Google Scholar]
  33. Guisan, A. , Edwards, T. C. , & Hastie, T. (2002). Generalized linear and generalized additive models in studies of species distributions: Setting the scene. Ecological Modelling, 157, 89–100. 10.1016/S0304-3800(02)00204-1 [DOI] [Google Scholar]
  34. Guisan, A. , Graham, C. H. , Elith, J. , & Huettmann, F. (2007). Sensitivity of predictive species distribution models to change in grain size. Diversity and Distributions, 13, 332–340. 10.1111/j.1472-4642.2007.00342.x [DOI] [Google Scholar]
  35. Guisan, A. , & Thuiller, W. (2005). Predicting species distribution: Offering more than simple habitat models. Ecology Letters, 8, 993–1009. 10.1111/j.1461-0248.2005.00792.x [DOI] [PubMed] [Google Scholar]
  36. Guisan, A. , & Zimmermann, N. E. (2000). Predictive habitat distribution models in ecology. Ecological Modelling, 135, 147–186. 10.1016/S0304-3800(00)00354-9 [DOI] [Google Scholar]
  37. Heikkinen, R. K. , Marmion, M. , & Luoto, M. (2012). Does the interpolation accuracy of species distribution models come at the expense of transferability? Ecography, 35, 276–288. 10.1111/j.1600-0587.2011.06999.x [DOI] [Google Scholar]
  38. Hijmans, R. J. , Cameron, S. E. , Parra, J. L. , Jones, P. G. , & Jarvis, A. (2005). Very high resolution interpolated climate surfaces for global land areas. International Journal of Climatology, 25, 1965–1978. 10.1002/joc.1276 [DOI] [Google Scholar]
  39. Hof, C. , Voskamp, A. , Biber, M. F. , Böhning‐Gaese, K. , Engelhardt, E. K. , Niamir, A. , Willis, S. G. , & Hickler, T. (2018). Bioenergy cropland expansion may offset positive effects of climate change mitigation for global vertebrate diversity. Proceedings of the National Academy of Sciences, 115, 13294–13299. 10.1073/pnas.1807745115 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. IUCN (2018). The IUCN Red List of Threatened Species. Version 2018.1. https://www.iucnredlist.org [Google Scholar]
  41. Jaeger, B. C. , Edwards, L. J. , Das, K. , & Sen, P. K. (2017). An R2 statistic for fixed effects in the generalized linear mixed model. Journal of Applied Statistics, 44, 1086–1105. 10.1080/02664763.2016.1193725 [DOI] [Google Scholar]
  42. Jarnevich, C. S. , Talbert, M. , Morisette, J. , Aldridge, C. , Brown, C. S. , Kumar, S. , Manier, D. , Talbert, C. , & Holcombe, T. (2017). Minimizing effects of methodological decisions on interpretation and prediction in species distribution studies: An example with background selection. Ecological Modelling, 363, 48–56. 10.1016/j.ecolmodel.2017.08.017 [DOI] [Google Scholar]
  43. Jeschke, J. M. , & Strayer, D. L. (2008). Usefulness of bioclimatic models for studying climate change and invasive species. Annals of the New York Academy of Sciences, 1134, 1–24. 10.1196/annals.1439.002 [DOI] [PubMed] [Google Scholar]
  44. Jost, L. (2006). Entropy and diversity. Oikos, 113, 363–375. 10.1111/j.2006.0030-1299.14714.x [DOI] [Google Scholar]
  45. Knouft, J. H. , & Ficklin, D. L. (2017). The potential impacts of climate change on biodiversity in flowing freshwater systems. Annual Review of Ecology Evolution and Systematics, 48, 111–133. 10.1146/annurev-ecolsys-110316-022803 [DOI] [Google Scholar]
  46. Leroy, B. , Delsol, R. , Hugueny, B. , Meynard, C. N. , Barhoumi, C. , Barbet‐Massin, M. , & Bellard, C. (2018). Without quality presence–absence data, discrimination metrics such as TSS can be misleading measures of model performance. Journal of Biogeography, 45, 1994–2002. 10.1111/jbi.13402 [DOI] [Google Scholar]
  47. Li, X. , & Wang, Y. (2013). Applying various algorithms for species distribution modelling. Integrative Zoology, 8, 124–135. 10.1111/1749-4877.12000 [DOI] [PubMed] [Google Scholar]
  48. Lobo, J. M. , Jiménez‐Valverde, A. , & Real, R. (2008). AUC: A misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography, 17, 145–151. 10.1111/j.1466-8238.2007.00358.x [DOI] [Google Scholar]
  49. Marmion, M. , Parviainen, M. , Luoto, M. , Heikkinen, R. K. , & Thuiller, W. (2009). Evaluation of consensus methods in predictive species distribution modelling. Diversity and Distributions, 15, 59–69. 10.1111/j.1472-4642.2008.00491.x [DOI] [Google Scholar]
  50. McGarvey, D. J. , Menon, M. , Woods, T. , Tassone, S. , Reese, J. , Vergamini, M. , & Kellogg, E. (2018). On the use of climate covariates in aquatic species distribution models: Are we at risk of throwing out the baby with the bath water? Ecography, 41, 695–712. 10.1111/ecog.03134 [DOI] [Google Scholar]
  51. Merow, C. , Allen, J. M. , Aiello‐Lammens, M. , & Silander, J. A. (2016). Improving niche and range estimates with Maxent and point process models by integrating spatially explicit information. Global Ecology and Biogeography, 25, 1022–1036. 10.1111/geb.12453 [DOI] [Google Scholar]
  52. Merow, C. , Smith, M. J. , Edwards, T. C. , Guisan, A. , McMahon, S. M. , Normand, S. , Thuiller, W. , Wüest, R. O. , Zimmermann, N. E. , & Elith, J. (2014). What do we gain from simplicity versus complexity in species distribution models? Ecography, 37, 1267–1281. 10.1111/ecog.00845 [DOI] [Google Scholar]
  53. Moreno‐Amat, E. , Mateo, R. G. , Nieto‐Lugilde, D. , Morueta‐Holme, N. , Svenning, J.‐C. , & García‐Amorena, I. (2015). Impact of model complexity on cross‐temporal transferability in Maxent species distribution models: An assessment using paleobotanical data. Ecological Modelling, 312, 308–317. 10.1016/j.ecolmodel.2015.05.035 [DOI] [Google Scholar]
  54. Naimi, B. , Hamm, N. A. S. , Groen, T. A. , Skidmore, A. K. , & Toxopeus, A. G. (2014). Where is positional uncertainty a problem for species distribution modelling? Ecography, 37, 191–203. 10.1111/j.1600-0587.2013.00205.x [DOI] [Google Scholar]
  55. Olson, D. M. , Dinerstein, E. , Wikramanayake, E. D. , Burgess, N. D. , Powell, G. V. N. , Underwood, E. C. , D’amico, J.A. , Itoua, I. , Strand, H.E. , Morrison, J.C. , Loucks, C.J. , Allnutt, T.F. , Ricketts, T.H. , Kura, Y. , Lamoreux, J.F. , Wettengel, W.W. , Hedao, P. , & Kassem, K.R. , (2001). Terrestrial ecoregions of the world: A new map of life on earth. BioScience, 51, 933–938. 10.1641/0006-3568(2001)051[0933:TEOTWA]2.0.CO;2 [DOI] [Google Scholar]
  56. Pacifici, M. , Foden, W. B. , Visconti, P. , Watson, J. E. M. , Butchart, S. H. M. , Kovacs, K. M. , Scheffers, B. R. , Hole, D. G. , Martin, T. G. , Akçakaya, H. R. , Corlett, R. T. , Huntley, B. , Bickford, D. , Carr, J. A. , Hoffmann, A. A. , Midgley, G. F. , Pearce‐Kelly, P. , Pearson, R. G. , Williams, S. E. , … Rondinini, C. (2015). Assessing species vulnerability to climate change. Nature Climate Change, 5, 215–224. 10.1038/nclimate2448 [DOI] [Google Scholar]
  57. Pearson, R. G. , & Dawson, T. P. (2003). Predicting the impacts of climate change on the distribution of species: Are bioclimate envelope models useful? Global Ecology and Biogeography, 12, 361–371. 10.1046/j.1466-822X.2003.00042.x [DOI] [Google Scholar]
  58. Pearson, R. G. , Thuiller, W. , Araújo, M. B. , Martinez‐Meyer, E. , Brotons, L. , McClean, C. , Miles, L. , Segurado, P. , Dawson, T. P. , & Lees, D. C. (2006). Model‐based uncertainty in species range prediction. Journal of Biogeography, 33, 1704–1711. 10.1111/j.1365-2699.2006.01460.x [DOI] [Google Scholar]
  59. Peterson, A. T. , Papeş, M. , & Eaton, M. (2007). Transferability and model evaluation in ecological niche modeling: A comparison of GARP and Maxent. Ecography, 30, 550–560. 10.1111/j.0906-7590.2007.05102.x [DOI] [Google Scholar]
  60. Petitpierre, B. , Broennimann, O. , Kueffer, C. , Daehler, C. , & Guisan, A. (2017). Selecting predictors to maximize the transferability of species distribution models: Lessons from cross‐continental plant invasions. Global Ecology and Biogeography, 26, 275–287. 10.1111/geb.12530 [DOI] [Google Scholar]
  61. Phillips, S. J. , Anderson, R. P. , & Schapire, R. E. (2006). Maximum entropy modeling of species geographic distributions. Ecological Modelling, 190, 231–259. 10.1016/j.ecolmodel.2005.03.026 [DOI] [Google Scholar]
  62. Phillips, S. J. , Dudík, M. , Elith, J. , Graham, C. H. , Lehmann, A. , Leathwick, J. , & Ferrier, S. (2009). Sample selection bias and presence‐only distribution models: Implications for background and pseudo‐absence data. Ecological Applications, 19, 181–197. 10.1890/07-2153.1 [DOI] [PubMed] [Google Scholar]
  63. Pliscoff, P. , Luebert, F. , Hilger, H. H. , & Guisan, A. (2014). Effects of alternative sets of climatic predictors on species distribution models and associated estimates of extinction risk: A test with plants in an arid environment. Ecological Modelling, 288, 166–177. 10.1016/j.ecolmodel.2014.06.003 [DOI] [Google Scholar]
  64. Ponder, W. F. , Carter, G. A. , Flemons, P. , & Chapman, R. R. (2001). Evaluation of museum collection data for use in biodiversity assessment. Conservation Biology, 15, 648–657. 10.1046/j.1523-1739.2001.015003648.x [DOI] [Google Scholar]
  65. Qiao, H. , Feng, X. , Escobar, L. E. , Peterson, A. T. , Soberón, J. , Zhu, G. , & Papeş, M. (2019). An evaluation of transferability of ecological niche models. Ecography, 42, 521–534. 10.1111/ecog.03986 [DOI] [Google Scholar]
  66. Radosavljevic, A. , & Anderson, R. P. (2014). Making better Maxent models of species distributions: Complexity, overfitting and evaluation. Journal of Biogeography, 41, 629–643. 10.1111/jbi.12227 [DOI] [Google Scholar]
  67. Randin, C. F. , Dirnböck, T. , Dullinger, S. , Zimmermann, N. E. , Zappa, M. , & Guisan, A. (2006). Are niche‐based species distribution models transferable in space? Journal of Biogeography, 33, 1689–1703. 10.1111/j.1365-2699.2006.01466.x [DOI] [Google Scholar]
  68. Rapacciuolo, G. , Roy, D. B. , Gillings, S. , Fox, R. , Walker, K. , & Purvis, A. (2012). Climatic associations of british species distributions show good transferability in time but low predictive accuracy for range change. PLoS One, 7, e40212 10.1371/journal.pone.0040212 [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Richmond, O. M. W. , McEntee, J. P. , Hijmans, R. J. , & Brashares, J. S. (2010). Is the climate right for pleistocene rewilding? Using species distribution models to extrapolate climatic suitability for mammals across continents. PLoS One, 5, e12899 10.1371/journal.pone.0012899 [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Roberts, D. R. , Bahn, V. , Ciuti, S. , Boyce, M. S. , Elith, J. , Guillera‐Arroita, G. , Hauenstein, S. , Lahoz‐Monfort, J. J. , Schröder, B. , Thuiller, W. , Warton, D. I. , Wintle, B. A. , Hartig, F. , & Dormann, C. F. (2017). Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40, 913–929. 10.1111/ecog.02881 [DOI] [Google Scholar]
  71. Ruiz‐Navarro, A. , Gillingham, P. K. , & Britton, J. R. (2016). Predicting shifts in the climate space of freshwater fishes in Great Britain due to climate change. Biological Conservation, 203, 33–42. 10.1016/j.biocon.2016.08.021 [DOI] [Google Scholar]
  72. Thomas, C. D. , Cameron, A. , Green, R. E. , Bakkenes, M. , Beaumont, L. J. , Collingham, Y. C. , Erasmus, B. F. N. , de Siqueira, M. F. , Grainger, A. , Hannah, L. , Hughes, L. , Huntley, B. , van Jaarsveld, A. S. , Midgley, G. F. , Miles, L. , Ortega‐Huerta, M. A. , Townsend Peterson, A. , Phillips, O. L. , & Williams, S. E. (2004). Extinction risk from climate change. Nature, 427, 145–148. 10.1038/nature02121 [DOI] [PubMed] [Google Scholar]
  73. Thuiller, W. , Georges, D. , Engler, R. , & Breiner, F. (2016). biomod2: Ensemble platform for species distribution modeling. https://CRAN.R‐project.org/package=biomod2. [Google Scholar]
  74. Thuiller, W. , Guéguen, M. , Renaud, J. , Karger, D. N. , & Zimmermann, N. E. (2019). Uncertainty in ensembles of global biodiversity scenarios. Nature Communications, 10, 1–9. 10.1038/s41467-019-09519-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. VanDerWal, J. , Shoo, L. P. , Graham, C. , & Williams, S. E. (2009). Selecting pseudo‐absence data for presence‐only distribution modeling: How far should you stray from what you know? Ecological Modelling, 220, 589–594. 10.1016/j.ecolmodel.2008.11.010 [DOI] [Google Scholar]
  76. Vences, M. , & Köhler, J. (2008). Global diversity of amphibians (Amphibia) in freshwater. Hydrobiologia, 595, 569–580. 10.1007/s10750-007-9032-2 [DOI] [Google Scholar]
  77. Verbruggen, H. , Tyberghein, L. , Belton, G. S. , Mineur, F. , Jueterbock, A. , Hoarau, G. , Gurgel, C. F. D. , & Clerck, O. D. (2013). Improving transferability of introduced species’ distribution models: New tools to forecast the spread of a highly invasive seaweed. PLoS One, 8, e68337 10.1371/journal.pone.0068337 [DOI] [PMC free article] [PubMed] [Google Scholar]
  78. Visconti, P. , Bakkenes, M. , Baisero, D. , Brooks, T. , Butchart, S. H. M. , Joppa, L. , Alkemade, R. , Marco, M. D. , Santini, L. , Hoffmann, M. , Maiorano, L. , Pressey, R. L. , Arponen, A. , Boitani, L. , Reside, A. E. , van Vuuren, D. P. , & Rondinini, C. (2016). Projecting global biodiversity indicators under future development scenarios. Conservation Letters, 9, 5–13. 10.1111/conl.12159 [DOI] [Google Scholar]
  79. Warren, D. L. , Matzke, N. J. , & Iglesias, T. L. (2019). Evaluating presence‐only species distribution models with discrimination accuracy is uninformative for many applications. Journal of Biogeography, 47, 167–180. 10.1111/jbi.13705 [DOI] [Google Scholar]
  80. Warren, D. L. , & Seifert, S. N. (2011). Ecological niche modeling in Maxent: The importance of model complexity and the performance of model selection criteria. Ecological Applications, 21, 335–342. 10.1890/10-1171.1 [DOI] [PubMed] [Google Scholar]
  81. Zuur, A. F. , Ieno, E. N. , Elphick, C. S. (2010). A protocol for data exploration to avoid common statistical problems. Methods in Ecology and Evolution, 1(1), 3–14. 10.1111/j.2041-210x.2009.00001.x [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix S1

Click here for additional data file. (3MB, docx)

Data Availability Statement

The species data used in this study are publicly available on the IUCN Red List website (https://www.iucnredlist.org/resources/spatial‐data‐download). Climate data retrieved from the WorldClim v1.4 database are publicly available on the WorldClim website (https://worldclim.org/data/v1.4/worldclim14.html). The code of the analysis is archived at https://doi.org/10.5281/zenodo.4022773 as the GitHub repository. With this code and the input data specified above, results in this paper can be reproduced.

Articles from Ecology and Evolution are provided here courtesy of Wiley

RESOURCES