Estimating species commonness and prevalence through unsupervised methods

Abstract

The prevalence of a species in a given area is crucial for estimating the environmental conditions associated with its subsistence within ecological niche models (ENMs). Prevalence is defined as the proportion of presences relative to the total number of sampled sites, reflecting prior expectation on species commonness or rarity. However, reliable estimation often faces challenges due to limited or biased occurrence data, particularly for rare or poorly monitored species. This work presents a data-driven, multi-species methodology to estimate species prevalence for use in ENMs. It leverages species occurrence records from the Global Biodiversity Information Facility and is entirely unsupervised. It utilises two clustering methods, one deep-learning model, and an ensemble model, plus statistical analysis to classify species commonness and transform classifications into prevalence probabilities. A case study is presented for 161 species living in the Massaciuccoli Lake basin (Tuscany, Italy), a wetland of high biodiversity value and ecological sensitivity. The models classified the species’ prevalence based on observations from other Italian wetland sites, and were evaluated against expert-based assessments. All models achieved high accuracy, with the deep-learning model achieving the highest (~ 81–90%). The proposed methodology is scalable and reproducible and can inform ENMs with objective, robust prevalence estimates.

Introduction

Ecological Niche Models (ENMs) estimate the array of resources and environmental conditions that facilitate the persistence and proliferation of a species within a specific area, commonly referred to as the species’ ecological niche^1–7. These models can operationally define a species’ suitable habitat as the locations within the study area that possess suitable environmental conditions corresponding to the species’ ecological niche. Consequently, ENMs are also capable of predicting the potential presence or absence of a species in an area different from its native area (the potential ecological niche)^8,9. From a mathematical perspective, a species’ ecological niche can be conceptualised as a region in a vector space (a hypervolume) formed by the environmental variables relevant to the species’ survival.

The efficacy of ENMs is contingent upon the accurate identification of a comprehensive set of environmental variables that constitute the dimensions of this vector space, as well as the availability of prior information regarding the species’ presence in the area^1,10,11. This prior information (species prevalence) is particularly crucial for driving an ENM towards the correct estimation of the ecological niche, as it defines the boundary conditions and starting assumptions for the model’s fitting to the data. Generally, species prevalence is pivotal in both the calibration and interpretation of ENMs^12,13. Formally, it is defined as the proportion of presence data relative to the total number of sites examined while surveying, considering both presences and absences (or pseudo-absences)^14,15. Therefore, it also indicates the extent of the landscape that is occupied or deemed suitable for the species under consideration, before fitting an ENM to the species observation data.

The relevance of species prevalence for ENMs is evident in two primary aspects. First, it directly affects the calibration of model outputs, particularly when algorithms generate continuous suitability scores that can be transformed into probabilities of occurrence^15–18. For instance, in Bayesian models, such as Maximum Entropy (MaxEnt)^19–21, assumptions regarding species prevalence influence the conversion of raw suitability scores into logistic probabilities of species presence. Within these frameworks, prevalence acts as a parameter that introduces prior information into the models concerning the anticipated species presence in the focus area. Species prevalence also corresponds to the prior probability of the species’ presence in the area, indicating the expectation of whether the species is rare or common in that area prior to observation^22–24. Variations in prevalence settings could therefore alter the modelled species habitat distributions.

Second, species prevalence has a considerable impact on the process of thresholding continuous predictions into binary presence-absence maps^10,25,26, where the selection of optimal thresholds often depends on the equilibrium between sensitivity (true positive fraction) and specificity (true negative fraction). This balance is inherently influenced by species prevalence. Many metrics used for evaluating ENM performance (e.g., Kappa, True Skill Statistic) indeed display increased sensitivity to imbalances between presences and absences^25,27–29. This sensitivity is particularly relevant for rare species characterised by low prevalence, as data scarcity may hinder predictive performance and obscure niche boundaries. Prevalence indeed embodies ecological characteristics indicative of species rarity or commonness^30–33. Consequently, a careful consideration of prevalence is paramount for generating robust, interpretable, and ecologically meaningful niche models.

Given the importance of prevalence for ENMs, it is paramount that its estimate is as objective as possible, unaffected by individual conjectures that might be subject to biases. This paper presents a data-driven, multi-species methodology for objectively estimating species prevalence (Fig. 1). The methodology builds upon species occurrence records from the publicly accessible Global Biodiversity Information Facility (GBIF³⁴), while addressing issues related to the possible under-representation of species presence in the analysed area. It uses three different unsupervised machine learning techniques, comprising two cluster analysis methods (Multi K-means and X-means) and one deep learning model (a Variational Autoencoder, VAE). These models process aggregative features for each species under analysis, defined on the species occurrence records. These features highlight the spatio-temporal coverage of the occurrences, the frequency of observation, and the number of diverse data collections that reported presence in the area under analysis. The model outputs are statistically analysed to classify the commonness level of each species, which is subsequently transformed into species prevalence probability. The cluster analysis methods yield discretised values corresponding to classes of species commonness, while the deep learning model provides a continuous score that is eventually discretised for classification.

Fig. 1.

Overview of the methodological workflow.

To evaluate and illustrate the effectiveness of the methodology, a case study centred on the Massaciuccoli Lake basin in Tuscany, Italy, is presented (Fig. 2). This basin is a wetland and a significant tourist attraction, distinguished by its rich biodiversity. The hydrogeological, chemical, and environmental characteristics of the basin are critical in determining the habitat suitability for the presence and persistence of both native species (e.g., black bullhead, eel, tench, largemouth bass, carp, heron, and kingfisher) and non-native species (e.g., sheatfish and red swamp crayfish). Additionally, the basin functions as a crucial stopover point for numerous bird migrations, thus garnering the interest of citizens, tourists, and scientists^35,36. Here, as generally in wetland ecosystems, birds are particularly vulnerable to changes in biogeochemical and hydrological conditions within the current climate change dynamics^37–40. Species prevalence plays a pivotal role in predicting biodiversity patterns within this region over short and long time scales.

Fig. 2.

Overview of the Massaciuccoli Lake basin (left image) and the Italian wetlands from the Ramsar Convention on Wetlands (red areas in the right image). Maps were created with QGIS v.3.40 (URL: www.qgis.org). The Italy map was the OSM standard map of the QuickMapService plugin. The Tuscany map was taken from the Tuscany Regional Government online GIS service (URL: www502.regione.toscana.it/geoscopio/cartoteca.html). The Ramsar wetland areas were taken from the Repertorio Nazionale dei Dati Territoriali (URL: geodati.gov.it/resource/id/m_amte:299FN3:99622692-1e7a-452b-86d7-3ae4fe96c385).

The three models for species commonness classification were applied to numeric features calculated from the occurrence records of 161 species of the Massaciuccoli Lake basin over the past decade (2015-2025). Model training utilised GBIF data from the Italian wetlands catalogued by the Ramsar Convention on Wetlands⁴¹ (Fig. 2-right panel). Alternative species-commonness assessments were conducted by two local experts and used as references for evaluating model classification performance. Additionally, a sensitivity analysis of the models on the features was conducted using a leave-one-out procedure to highlight those with the highest importance for classification.

Overview

The term common species denotes a species that exhibits high abundance in a specific area, has a widespread distribution, and is considered at low risk of extinction^42–44. In contrast, rare species are characterised by lower abundance, unsuitable habitat, or potential threats to their survival. Automatic detection and monitoring of common and rare species is crucial for understanding the implications of environmental change on ecosystem functioning. A species’ abundance serves as an indicator of its ecological role and is influenced by the relative abundances of other species^45–47. Rare species often contribute significantly to biodiversity due to their unique functional traits^48–52, while common species underpin ecosystem function through their biomass dominance^42,53–59.

In ENMs, species commonness can be used to estimate species prevalence^16,18,60,61. Traditional ENMs, like MaxEnt, require prevalence to transform relative occurrence rates into posterior probabilities of presence^21,62. Bayesian approaches also utilise prevalence as an uncertain parameter, enhancing the reliability of species range predictions^63–65. These analyses also highlight the impacts of environmental changes on relative species abundances^66–68.

Monitoring biodiversity changes requires large-scale, multi-species ecological analyses^69–72. Meta-analyses across studies can assess the commonness and prevalence of multiple species within specified timeframes^73–75. Researchers can leverage extensive biodiversity databases, such as the United Kingdom’s National Biodiversity Network⁷⁶, the Global Biodiversity Information Facility, or the Ocean Biogeographic Information System (OBIS)⁷⁷, which provide millions of records detailing species distributions over time. However, extracting robust estimates from these heterogeneous data sources poses significant challenges^78–81.

A primary concern is the difficulty of distinguishing the actual prevalence of a species from the noise introduced by varying sampling efforts that are temporally, spatially, and taxonomically focused⁸². A species may appear abundant during a specific decade in a large dataset due to intensive sampling efforts focused on that species. A subsequent decline in apparent abundance may simply reflect the end of that programme, lacking ecological significance. The presence of a species’ occurrence records in large collections is influenced by various factors, including (i) sampling frequency, (ii) ease of observation, (iii) whether the species is targeted in surveys, and (iv) the temporal extent of those surveys. Such biases may significantly impact assessments made by models or experts regarding species prevalence. Consequently, the information in large datasets generally represents a partial reality.

To mitigate these challenges, several methodologies integrate species observation records with supplementary information from expert documents or other databases^83–88. While these approaches address biases, they may only be generalizable to species for which supplementary data exist, potentially excluding underrepresented species. Some approaches have introduced analytical definitions of species commonness based on subjective expert knowledge^22,89, yet a universally satisfactory definition remains elusive. Therefore, assessing species commonness often necessitates the identification of limited commonness classes, and since species prevalence relates closely to commonness, prevalence probability is also discretised. This does not necessarily diminish the accuracy of information conveyed to multi-species ENMs. For instance, cluster analysis has proven effective for detecting multi-species presence patterns^33,90–94. Furthermore, accuracy in prevalence discretisation can be enhanced through re-analysis of presence-only ENMs, expert assessments, or independent surveys^15,95–97. Discretised prevalence ranges are also useful for estimating statistical uncertainty in habitat suitability probabilities estimated by ENMs^27,98–100.

Despite the importance of species prevalence for ecological niche modelling, current approaches face limitations when applied to large, diverse species sets. Many methods rely on species-specific modelling, external ecological traits, or expert-defined thresholds to address incomplete occurrence data, leading to subjectivity and limiting scalability. Additionally, prevalence is often discretised into fixed categories without a consistent rationale. These challenges highlight the need for a framework that can jointly assess prevalence across multiple species, operate under incomplete data, and minimise expert reliance.

The unsupervised methodology proposed in this paper addresses this gap by utilising aggregated, ecologically meaningful features from occurrence data, enabling direct comparison across species without assuming completeness. Clustering and anomaly-detection techniques serve to identify dominant prevalence patterns and detect deviations in rare or undersampled species. This design enables prevalence to emerge from the data structure while preserving flexibility in discretisation and ensuring reproducibility across taxa.

In summary, the methodology presented in this paper incorporates effective strategies suggested by other works to offer a novel approach that incorporates the following features: (i) it generates objective multi-species prevalence assessments; (ii) accounts for the knowledge gaps inherent in extensive collections of species occurrence records; and (iii) employs prevalence discretisation with adjustable granularity.

Results

Classification performance

The VAE demonstrated the highest level of concordance with all expert assessments, both in the binary and three-category evaluations (Table 1). It achieved the greatest accuracy when compared to the expert ensemble, recording values of 90.06% (146 species assessed correctly) and 85.71% (138 species assessed correctly) in the two evaluations, respectively. Notably, a substantial agreement (following Fleiss’ interpretation of Kappa¹⁰¹) was measured in these cases, with Kappa of 0.80 and 0.76, respectively. The VAE classified 71 species as very common (69 correctly), 51 as fairly common (51 correctly), and 39 as rare (18 correctly). A higher agreement with the “expert ensemble” was observed across all models, with an average relative increase of 10% accuracy compared to the lowest agreement. The minimum level of agreement was consistently associated with Expert 1, with whom the VAE reached a moderate/fair agreement, achieving accuracies of 81.37% and 70.81% and Kappa of 0.63 and 0.54 in the two evaluations, respectively. Expert 1 generally tended to assign higher commonness levels than the models. The agreement with Expert 2 was intermediate between the others, with the VAE achieving accuracies of 89.44% and 85.09% in the two evaluations and a substantial agreement (Kappa = 0.79 and 0.75, respectively).

Table 1.

Performance comparison (accuracy and Cohen’s Kappa) in species commonness classification between the proposed models, two experts, and an ensemble expert assessment. The comparisons are reported for binary classification (very common vs. less common species) and three-category classification (very common, fairly common, or rare).

	Two categories
	Multi K-means		X-means		VAE		Ensemble
	Acc. (%)	Kappa	Acc. (%)	Kappa	Acc. (%)	Kappa	Acc. (%)	Kappa
Expert 1	78.26	0.57	77.64	0.55	81.37	0.63	78.88	0.59
Expert 2	85.71	0.71	87.58	0.75	89.44	0.79	86.96	0.74
Expert ensemble	86.96	0.73	88.20	0.76	90.06	0.80	87.58	0.75

	Three categories
	Multi K-means		X-means		VAE		Ensemble
	Acc. (%)	Kappa	Acc. (%)	Kappa	Acc. (%)	Kappa	Acc. (%)	Kappa
Expert 1	68.32	0.52	69.57	0.50	70.81	0.54	65.22	0.47
Expert 2	81.37	0.70	83.85	0.73	85.09	0.75	80.75	0.68
Expert ensemble	82.61	0.72	84.47	0.73	85.71	0.76	81.37	0.69

The high concordance with the “expert ensemble” can be attributed to the ensemble methodology; when there was significant disagreement between the experts regarding a species commonness classification, the ensemble opted for a moderate classification (“fairly common”). This approach effectively harmonised the differing perspectives of the experts. Overall, expert disagreement occurred for only a lesser fraction of the species (20%). The uncertain cases mainly involved species whose prevalence and observational frequency were on the cusp of being classified as either very common or fairly common. The results revealed a tendency for all models to categorise species as “fairly common” in cases where the experts expressed uncertainty. However, the predominant classification overall was that of very common species.

The fact that the VAE achieved the highest overall accuracy underscores the correlation between anomalies in the feature values and species rarity, thereby validating the hypothesis explained in “Methods” (section “Classification”): the characterisation of rare species is more varied than that of common species.

Table 1 also indicates that Multi K-means (with =5) yielded a lower average accuracy compared to X-means (with =7) in both binary (83.6% vs. 84.5%, on average) and three-category (77.43% vs. 79.30%, on average) classifications. Agreement with the experts ranged from moderate to substantial. These findings suggest that the slightly higher fragmentation of the feature space by the X-means model, as determined by the BIC, was more effective in highlighting species-commonness characteristics within each cluster. Generally, the cluster analysis models achieved high agreement with the experts, although they were less accurate than the VAE.

The ensemble model achieved an average accuracy of 84.5% in binary classification and of 75.8% in three-category classification, exhibiting moderate to substantial agreement with the experts. This model was predominantly influenced by the cluster analysis models. The VAE provided particular support in binary classification, shifting decisions towards the X-means assessments in cases of disagreement between the cluster analysis models. As a result, the ensemble assessment generally resembled an enhanced variant of Multi K-means.

As an additional validation, prevalence scores derived from X-Means were utilised within a multi-species ENM to assess biodiversity trends in the Massaciuccoli basin (described in⁶⁰). This model exhibited considerable robustness, with respect to evidence, in delineating the current species richness scenario (2016–2024) compared to both a remote past (1950–1980) and a more recent temporal context (1981–2015). Furthermore, it successfully projected biodiversity scenarios for both short-term (2025–2050) and long-term (2051–2100) future perspectives. These findings underscored the effectiveness of reliable prevalence discretisation, particularly in the context of big data-driven multi-species modelling.

Sensitivity analysis

The leave-one-out analysis explained in “Methods” (section “Evaluation methodology”) demonstrated that each feature significantly contributed to both accuracy and agreement (Table 2). The removal of any single feature invariably led to a reduction in both metrics. The Kappa reductions provided a clearer delineation of the variable rankings, from the feature with the most substantial information content to that with the least influence. The accuracy losses ranged from -5.56% to -3.17% for binary classification and from -8.18% to -3.64% for the three-category classification. In both classification scenarios, the most important variable was the average number of occurrence records per dataset (IntraDs). Species frequently documented across multiple wetlands were likely to be classified as “very” or “fairly” common. Conversely, the mere fraction of datasets containing species observations (InterDs) was less critical compared to the other features, despite its positive contribution to performance.

Table 2.

Average percentage reduction of accuracy and Cohen’s Kappa for the ensemble model when excluding one feature at a time (leave-one-out process). The expert ensemble assessments were used as the references.

	Relative decrease
Excluded feature	Accuracy (%)	Kappa
Two categories
IntraDs	− 5.56%	− 17.10%
A	− 5.56%	− 16.23%
E	− 3.97%	− 13.61%
F	− 3.97%	− 13.14%
InterDs	− 3.97%	− 11.88%
HF	− 3.17%	− 10.53%

Three categories
IntraDs	− 8.18%	− 14.57%
HF	− 8.18%	− 14.02%
A	− 7.27%	− 13.91%
F	− 5.45%	− 8.99%
E	− 4.55%	− 8.97%
InterDs	− 3.64%	− 8.41%

The average species abundance per occurrence record (A) was another important parameter, since very abundant species are often common species. The extent of species presence in the areas (E) was another important feature for binary classification, although it was less important for three-category classification. A species occupying most of the area was likely to be associated with a “very common” species, but was less crucial to distinguish between “fairly common” and “rare” species.

The importance of observation frequencies (F and HF) was different when transitioning from binary to three-category classification. In binary classifications, the roles of these variables were marginal. In contrast, they became pivotal in distinguishing between “fairly common” and “rare” species within the three-category classification. In particular, the thresholded observation frequency (HF) exhibited a substantial shift in importance ranking, rising from the 6th to the 2nd position. The contribution of F to accuracy was also greater in the three-category classification (-5.45% decrease) than in the binary classification (-3.97% decrease). Overall, maintaining a consistently high frequency of observations over the years proved to be crucial in accurately classifying “fairly common” species.

Analysis of the misclassified species

Four species exhibited a consistent divergence between expert assessments and model predictions, with the experts evaluating them as “very common” species and the models indicating them as “rare” species (Fig. 3):

1. One species was Acrocephalus melanopogon (Temminck, 1823), the moustached warbler (forapaglie castagnolo, in Italian) (Fig. 3a). This bird primarily breeds in southern Europe and southern-temperate Asia. GBIF has collected 82 records of this species over the past decade within the Massaciuccoli basin, with a significant concentration of records in the lake area. Additionally, approximately 40 records were available from the other Italian wetlands. Although A. melanopogon is considered elusive within the Massaciuccoli basin, it encounters a particularly suitable habitat there, more than in other wetlands¹⁰². Observations documented in GBIF were predominantly concentrated in the months of April and May. The observations suggested that the species was abundant around the lake, but the average frequency of observations was relatively low (only five observations per year). The experts classified the species as “very common” due to its well-established presence in the area, whereas the models deduced rarity based on year-round presence and data from other wetlands. This case highlights that data derived from other wetlands do not always provide reliable insights regarding the specific wetland under analysis.

2. Another species was Cyprinus carpio (Linnaeus, 1758), the common carp (carpa comune, in Italian) (Fig. 3b). Across Europe, this fish is prevalent in eutrophic freshwater habitats, including lakes and large rivers¹⁰³. Often classified as invasive and detrimental in regions where it has been introduced, C. carpio presents a contrasting scenario in the Massaciuccoli basin, where it is not perceived as a problematic species. Its presence is well-documented, and the population appears to be stable¹⁰⁴. GBIF has recorded merely 12 observations over the last decade, and a similar number across the other Italian wetlands. Thus, the experts’ assessments were influenced by the general classification of the species as invasive, while the models relied on the data.

3. The third species was Halyomorpha halys (Stål, 1855), the brown marmorated stink bug (cimice asiatica, in Italian) (Fig. 3c). This insect is indigenous to various regions in Asia, particularly China, Japan, and Korea, and is recognised for its highly invasive nature in non-native habitats, where it poses a threat to agricultural crops¹⁰⁵. For the Massaciuccoli basin, GBIF contained 19 observations from the last decade, primarily concentrated along the shoreline and in rural towns. Although the presence of this species in the basin has been documented through scientific surveys and is believed to result from accidental introductions¹⁰⁶, there is currently no indication of an established invasion. Comparable numbers of observations and distribution patterns were available across the other Italian wetlands as well. Consequently, in this instance, the expert assessments were predominantly influenced by the known invasive behaviour of the species.

4. The fourth species was Tarentola mauritanica (Linnaeus, 1758), the common wall gecko (geco comune, in Italian) (Fig. 3d). This nocturnal reptile is frequently observed inhabiting walls within urban settings in temperate coastal regions. In Italy, similarly to other southern European nations, it has developed a longstanding association with humans, primarily as a predator of insects¹⁰⁷. GBIF contained a total of 40 observations for the Massaciuccoli basin, which were notably sparse and predominantly located in urbanised areas. An average of 15 observations was available for each of the other Italian wetlands. The frequency of observations on a yearly basis remained low, with a significant concentration of reports in the past four years. For this species, the assessments conducted by the experts were based on the species’ widespread distribution in Tuscany and throughout Italy, which was not reflected in the data.

A lower level of divergence in species commonness classification between the models and the experts was noted for a few other species. These cases encompassed borderline species positioned at the threshold between two classification categories. One group included species regarded by experts as “very common” but classified as “fairly common” by the models, which were Acrocephalus arundinaceus (the great reed warbler/cannareccione) and Acrocephalus schoenobaenus (the sedge warbler/forapaglie). Another group included borderline-abundant or elusive species that the experts classified as “rare”, whereas the models categorised as “fairly common”, which were Ichthyaetus melanocephalus (the Mediterranean gull/gabbiano corallino), Haematopus ostralegus (the Eurasian oystercatcher/beccaccia di mare), and Plegadis falcinellus (the glossy ibis/mignattaio). In these instances, the significant subjectivity inherent in the assessment process was the principal reason for the observed discrepancies.

Fig. 3.

Species for which maximum disagreement between the experts and the models was observed: the experts indicated the species as very common, whereas the models indicated rarity. The red dots illustrate the spatial distributions of the species occurrence records in GBIF (URL: www.gbif.org). Maps were created with QGIS v.3.40 (URL: www.qgis.org). The Tuscany background map was taken from the Tuscany Regional Government online GIS service (URL: www502.regione.toscana.it/geoscopio/cartoteca.html - Carta Topografica). The species’ images were taken from Wikipedia (URL: co.wikipedia.org/wiki/Acrocephalus_melanopogon#/media/File:Moustached_Warbler_(Acrocephalus_melanopogon)_(33906770030).jpg, URL: it.wikipedia.org/wiki/Cyprinus_carpio#/media/File:Common_carp.jpg, URL: it.wikipedia.org/wiki/File:Halyomorpha_halys_lab.jpg, and URL: it.wikipedia.org/wiki/Tarentola_mauritanica#/media/File:Gekon_murowy_(Tarentola_mauritanica).jpg).

Discussion

This study has presented a novel, data-driven, multi-species methodology for classifying species commonness and subsequently estimating species prevalence to serve as prior probability in ENMs, particularly in contexts where the only data available are species occurrence records from large collections. The methodology utilises occurrence records from GBIF, regarding ecologically similar regions or, alternatively, multiple surveys and data collections. It employs unsupervised methods for processing these data, including cluster analysis (Multi K-means, X-means) and deep learning (Variational Autoencoders, VAE). Moreover, it uses statistical analysis to classify species commonness and translate these classifications into prevalence probabilities. Being completely data-driven, the methodology ensures that the assessments of species commonness and prevalence are not subjective. Consequently, the presented approach ensures the provision of objective and neutral prior prevalence information over multiple species for ENM and biodiversity assessments, thereby contributing to the reproducibility and transparency of habitat distribution estimates. The methodology is designed for direct reusability, enabling quick prior estimates for numerous species. The utilisation of the publicly accessible GBIF data, coupled with standardised feature extraction, ensures that this methodology can be applied across a wide array of ecosystems and taxa, thereby enhancing its scalability and reusability, and potentially achieving global coverage.

Validation of the methodology was presented via a case study conducted in the biodiversity-rich wetland of the Massaciuccoli Lake basin (Tuscany, Italy). The models were trained on GBIF data sourced from other Italian wetland sites and subsequently evaluated against expert-based assessments of species commonness. The VAE demonstrated superior accuracy, achieving up to 90.06% in binary classification (“very common” vs. “less common”) and 85.71% in three-category classification (‘very common”, “fairly common”, or “rare”), along with substantial agreement with expert consensus. This confirmed the model’s effectiveness in capturing patterns of species rarity and commonness. Notably, the correspondence between anomalous features and species rarity suggested a more varied characterisation of rare species compared to common species.

The results highlighted the Multi K-means model’s tendency to assess lower commonness levels, as the () optimisation function, internally used by this model, fostered the formation of large clusters that included a wider range of feature values. The ensemble model exhibited a similar tendency but did not yield a statistically significant improvement in performance. On the contrary, X-means fragmented the feature space into slightly smaller clusters, making the classification more reliable. It also showed a tendency towards assessing higher commonness levels. The VAE yielded higher classification performance, but an important requirement emerged for its proper application to other scenarios: the species list must be sufficiently extended and varied so that anomalies can enable the detection of rare species. Consequently, X-means would be preferable in data-poor and limited-species scenarios. Finally, the sensitivity analysis underscored the relative importance of various features, identifying intra-dataset record abundance, species abundance per record, and observation frequency as key indicators of species commonness.

Although the analysis was demonstrated on this case study, the proposed framework is designed to be transferable to other ecological contexts, provided that comparable occurrence-derived features can be computed and that the species set is sufficiently large and heterogeneous. Generality is thus claimed at the level of the methodological approach rather than at the level of numerical results or ecological thresholds, which are expected to vary across systems.

One critical aspect of our methodology concerns the spatio-temporal aggregation of the observations, which can impact classification outcomes. Aggregating data on an annual basis results in the loss of differentiation between permanent and migratory species. Furthermore, employing a decadal time frame for occurrence retrieval penalises the classification of short-lived species. Additionally, conducting aggregations at a broad spatial resolution restricts the ability to identify species that may be prevalent at sub-area scales. Therefore, optimising spatio-temporal aggregation is critical for enhancing the granularity of assessments and facilitating more thorough investigations into specific species.

An intrinsic limitation of the proposed methodology is the unavoidable presence of spatial, temporal, and taxonomic sampling biases. Although the implementation of aggregative features, in conjunction with multi-clustering and anomaly analyses, alleviates this issue, these biases still influence prevalence estimates, particularly for elusive or underreported species. Additional misclassification can occur for species with borderline commonness status. The reliance on potentially biased information necessitated a compromise on the accuracy of the classifications and prevalence estimations. However, the methodology allows for increasing the resolution of the classes (up to 12), thereby enabling the exploration of sub-classes and more accurate prevalence values. Nevertheless, the basic three commonness classes used in the case study have already exhibited considerable informativeness for ENMs in data-limited scenarios, as evidenced by numerous studies demonstrating their capacity to estimate reliable habitat distributions^{3,16,22,60,108–112}. Deeper classification is even less necessary in multi-species models and macroscopic trend analyses^32,113–117. In data-rich scenarios, instead, the impact of diminished accuracy in prevalence estimation is mitigated by robust model adaptation to the data or by adjusting the estimates through external data.

Future research aimed at refining the methodology will focus on integrating GBIF data with other repositories, citizen science platforms, and field surveys, thereby assessing the necessary efforts to alleviate data biases and enhance feature robustness. Furthermore, the methodology will be expanded to investigate temporal dynamics in species prevalence, supporting ENMs in accounting for ecological changes and anthropogenic impacts over time. Moreover, the impact on commonness classification of a finer temporal granularity in the feature calculation will be investigated for seasonally active species. Lastly, the methodology will undergo testing across diverse ecosystems and taxa to evaluate its generalizability beyond the presented case study and evaluate its applicability in global biodiversity assessments. This testing will be an integral part of our next project proposals.

Methods

This section outlines our multi-species approach for species commonness classification and prevalence probability estimation (graphically illustrated in Fig. 1). Building upon the lessons learned from different methodologies in species commonness classification^{22,33,118–120}, the models presented herein are designed to classify species commonness and subsequently allocate species prevalence probability to each classification. The framework adheres to the strategies for species prevalence discretisation detailed in the “Overview”, which enhance the reliability of prevalence assessments at the cost of reduced granularity.

The methodology principally aims to categorise multiple species into three main commonness classes — very common, fairly common, or rare — based on statistical features derived from species occurrences in the GBIF large data collection. The methodology also permits an increased number of classes, facilitating more nuanced characterisations of species commonness. The basic classes are linked to prevalence probability values of 0.2, 0.5, and 0.8, which serve as multiplicative weights for the likelihood estimated by Bayesian ENMs, informed by species observations and environmental variables. Within this framework, the medium-commonness (fairly common species) class denotes uninformative prior information. Conversely, the low-commonness class (rare species) implies that an ENM should adopt a more conservative stance in evaluating species habitat, with a recommendation against assessing locations distant from the observation points. This is reflected in the multiplication of the likelihood by 0.2, effectively lowering habitat suitability assessments in uncertain or low-probability areas. In contrast, the high-commonness class (very common species) encourages species presence evaluations in areas with fewer observations, introducing a 0.8 multiplicative factor. Notably, setting the high value to 0.8, instead of 1, simulates an intrinsic uncertainty in the assessment (prevalence in the [0.8,1] interval). A more detailed classification is also possible by adding more classes with scores that fall between 0.2 and 0.8 (section “Classification”). This methodology enables quick assessments that yield reliable prior estimates of species prevalence for numerous species.

Our methodology comprises the following principal steps (Fig. 1): feature extraction (section “Feature extraction”), modelling (section “Modelling”), and classification (section “Classification”). During feature extraction, the GBIF observation records pertinent to each species under investigation are sourced. In the modelling step, three models are employed in parallel to analyse the data, organising the features into similarity groups through clustering or anomaly scoring. Finally, in the classification step, a commonness classification is allocated to the groups, thereby distinguishing between variations of species commonness and associating them with respective species prevalence probability values. Given the complementary approaches of the three models, the classification also includes an ensemble model that combines the individual-model classifications.

The following sections provide a more detailed explanation of these steps. Moreover, section “Case study” presents the Massaciuccoli Lake basin case study, utilised to assess classification performance across 161 species inhabiting this wetland. Lastly, section “Evaluation methodology” presents the evaluation metrics used for agreement and sensitivity analyses.

Feature extraction

Among the extensive global-scale data collections of species observations mentioned in the “Overview”, the Global Biodiversity Information Facility (GBIF)³⁴ currently encompasses over 3 billion records sourced from more than 116,000 data providers and surveys (commonly referred to as datasets) and includes representations of over 8 million species spanning multiple decades. GBIF is widely recognised for its utility in ecological and ecosystem modelling^121,122, as it satisfies the principles of findability, accessibility, interoperability, and reusability (FAIRness) through the adoption of domain-specific standards, such as the Darwin Core¹²³, alongside large-scale e-Infrastructures that offer complementary high-availability access services^124–127. Furthermore, GBIF incorporates scientific collections and personal archives contributed by researchers, as well as data from citizen science initiatives like iNaturalist¹²⁸ and eBird¹²⁹. Generally, GBIF is an invaluable repository for species observation data. However, it is essential to account for the sampling biases discussed in section “Overview” when inferring species distributions and prevalence from this collection, given that variations in sampling efforts and pathways over time and space may lead to either an overestimation or underestimation of species presence.

Our methodology accesses GBIF through its publicly available R package¹³⁰ to calculate prevalence-related features pertinent to the species under investigation. The definitions of these features are constructed as numerical indicators reflecting the commonness of a species within a defined geographical area and temporal framework. Although a formal definition of species commonness remains elusive, several studies have proposed various features linked to abundance, distribution extent, and the temporal frequency of species observations, which are closely associated with the concepts of rarity and commonness^{22,42,131–138}. In these frameworks, a species characterised by high abundance and frequent reporting by multiple data providers (i.e., across various datasets) is likely to be commonly encountered within the studied area. The frequency of observation within the specified temporal reference frame serves as an indicator of a species’ persistent presence in the area, potentially correlating with its endemism.

From these studies, it emerges that the input parameters necessary to extract the features required for species commonness characterisation should include (Fig. 1-step 1): (i) a polygon delineating the area of interest, (ii) a temporal framework for the analysis (e.g., from today to 10 years ago), (iii) the temporal granularity of the analysis (time-unit), defined as the minimum time interval during which species presence is to be evaluated (e.g., monthly or annually), (iv) a spatial aggregation resolution for the species observations aimed at quantifying species widespreadness in the area occupied, and (v) a list of species for which the features are to be calculated.

Building on previous research regarding the definition of features derived from extensive data collections^22,139, the formulae in Table 3 were defined. These features originate from possible aggregations of the GBIF data, measuring, for each species, (i) the average number of individuals recorded per observation (A), (ii) the total number of observations within the datasets relevant to the study area (IntraDs), (iii) the relative number of datasets reporting observations of the species (InterDs), (iv) the average widespreadness of the species within the designated area (E), (v) the average observation frequency throughout the overall analysis timeframe per specified time-unit (F), and (vi) the frequency at which the species appears with “many” observations within a time-unit (HF). Notably, these definitions contain parameters that should be adjusted for different geographical regions, i.e., the time frame of the analysis, the time unit, the spatial aggregation for the observations, and the threshold for determining that a number of observations is “relatively high”. Default settings for these parameters can be: a past-decade timeframe, a one-year time unit, a spatial aggregation of (~1 km), and a threshold of 5 observations for HF.

Table 3.

Numeric features extracted from GBIF by our methodology. The term datasets refers to the sub-collections available in GBIF.

Feature name	Description	Analytic definition
Species abundance	Average species abundance per occurrence record
Intra-dataset abundance	Average number of occurrence records per dataset
Inter-dataset abundance	Fraction of datasets containing species observations
Extent	Fraction of occurrence cells at the input spatial resolution with occurrence records associated
Observation Frequency	Average presence in the analysis time frame over the input time-units
High-observation Frequency	Observation frequency with at least input-threshold (Thr) observations

It is important to note that within the definitions, the concept of “dataset” can be effectively substituted with “similar areas”, i.e., regions that exhibit analogous ecosystem characteristics to the area under investigation¹⁴⁰. For instance, this concept could refer to other wetland regions if the focus is on a specific wetland, or adjacent/similar marine zones in the case of marine ecosystems. Generally, comparable ecosystems may serve as substitutes for datasets in the feature calculations, thereby eliminating the need for diverse, independent surveys conducted for the species in the analysed area.

A notable advantage of the proposed features is that they can be retrieved from various data repositories, in addition to GBIF, such as OBIS⁷⁷ and the Integrated Taxonomic Information System¹⁴¹, among others. The only prerequisite is the possibility to access records that include coordinates, the number of individuals, timestamps, and the belonging dataset. Another advantage of using these features is that they are aggregative; therefore, they reduce the bias introduced by individual-record misreporting and wrong localisations through cross-dataset averages and spatio-temporal aggregations. The additional aggregation across multiple species, conducted by the classification (section “Classification”), further contributes to reducing this bias.

In the methodology presented, the occurrence records from GBIF are first pre-processed to remove duplicates (equal coordinates and timestamps). Then, the six features are calculated, for each species under analysis, based on the occurrence records falling within the study area and the input-specified reference time frame (Fig. 1-step 1). These features are then standardised to make their ranges comparable, and compiled into a structured table formatted as a comma-separated-values (CSV) file (species-data table). Each row in this table corresponds to one species, while the columns denote the species’ scientific name alongside the values of A, IntraDs, InterDs, E, F, and HF. This table is subsequently utilised as input for each model described in the following section.

Modelling

This section describes the models used to identify aggregational groups for the features extracted in the previous phase. The models reported herein are derived from a selection of clustering and machine learning methodologies that were rigorously experimented in prior ecological-modelling studies^{3,22,117,142–144}. Specifically, these models incorporate complementary yet effective approaches that have demonstrated considerable efficacy in species prevalence classification (e.g., Multi K-means and X-means) as well as in ecosystem risk assessment (e.g., Variational Autoencoders). Other methodologies, such as traditional Artificial Neural Networks and alternative clustering techniques, were excluded from consideration due to their lack of complementary features relative to the models selected or insufficient performance reported for species prevalence estimation. Artificial Neural Networks, in particular, were evaluated but did not provide complementary results that enriched the methodology compared to the selected models. The following sections outline the technical specifications of the models selected (illustrated in Fig. 1-step 2).

Multi K-means

Cluster analysis is a pivotal data mining methodology aimed at categorising numerical vectors based on their inherent similarities, which are quantified through metrics such as Euclidean distance or density. The Multi K-means method extends the traditional K-means algorithm, enabling clustering across a range of values (the number of clusters). The K-means algorithm utilises Euclidean distance to allocate data points to clusters. K-means processes the data until convergence is attained. Multi K-means repeatedly applies K-means and eventually estimates the optimal number of clusters, denoted as , within a predetermined range. Our methodology uses a range between 3 and number of species/2, to guarantee a minimum distinction between rare, fairly-common, and very-common species, while also exploring a wider array of potential groupings. In each iteration of K-means, initial centroids are randomly chosen from the feature vectors, thereby increasing the probability that the resultant centroids represent realistic conditions.

To evaluate the quality of each clustering and identify the optimal , our Multi K-means employs an optimisation function (), which comes from ecosystem risk assessment¹¹⁷. This function balances model complexity (i.e., the number of clusters) and the quality of the clustering, checking the uniformity of the distribution of vectors across the clusters. It compares this distribution with a theoretical uniform distribution in which the vectors are evenly distributed among the clusters. A Chi-squared test quantitatively assesses the similarity between the population of each cluster and the theoretical distribution. Ultimately, Multi K-means selects the clustering configuration with the best fit, thereby favouring the most homogeneous distribution. Additionally, incorporates rules to automatically exclude clustering configurations that exhibit empty or sparsely populated clusters (detailed in¹¹⁷).

Multi K-means is designed to minimise the risk of over-segmentation and penalise the emergence of sparsely populated clusters. This strategy ensures that the identified clusters are both representative and homogeneous, while preserving the simplicity of the model. A significant outcome of this process is the integration of smaller clusters into larger clusters characterised by similar traits. This merging can potentially yield clusters that are larger than those generated by alternative criteria, such as the Bayesian Information Criterion (BIC)¹⁴⁵. Different from , the BIC assumes that the optimal clustering is achieved when the vectors are Gaussianly distributed around their respective centroids¹⁴⁶. In contrast, does not impose predefined shapes on the clusters.

The output of this model is the association of each species’ feature vector to one cluster among the identified. This association is materialised as cluster indexes reported in a “Multi K-means” column attached to the species-data table.

X-means

Similarly to Multi K-means, X-means¹⁴⁷ represents an advancement of K-means that determines the optimal number of clusters through a systematic investigation of a predefined range of . In our methodology, this range is established between 3 and number of species/2 for consistency with Multi K-means. The algorithm performs K-means clustering for each value of within this specified range, ultimately selecting as the clusterisation with the minimum BIC value. The minimum BIC balances the goodness of model fit to the data — within the assumption of normal distributions of vectors around the cluster centroids — against the complexity of the model, applying a penalty for excessively complex models to reduce the likelihood of overfitting. X-means was included in our methodology to underscore the impact of utilising BIC as an optimisation criterion as opposed to , because BIC overall tends to favour smaller clusters than .

Similarly to Multi K-means, the model output corresponds to the association of each species’ feature vector with one of the clusters of the optimal clusterisation . This association is manifested as an additional “X-means” column in the species-data table.

Variational autoencoder

A Variational Autoencoder (VAE)¹⁴⁸ is an unsupervised, deep, probabilistic Artificial Neural Network (ANN) designed to approximate the data distribution of a multi-dimensional variable . To this aim, it approximates the distribution of latent variables that underlie the generation of the observed data. The architecture of a VAE comprises two principal components: the Encoder and the Decoder. The Encoder is an ANN that maps the input data to the parameters of a probability distribution over the latent variables . In doing so, it outputs the parameters of a multivariate Gaussian distribution over characterised by mean and standard deviation . Through this Gaussian distribution, the Encoder approximates the posterior distribution . Conversely, the Decoder is an ANN with the objective of reconstructing the input data from samples drawn from the latent space (). It also simulates the likelihood , representing the reconstruction probability of given the latent variable . VAEs have found extensive applications in anomaly detection due to their generative capacity and ability to estimate the reconstruction probability of a feature vector^149–151.

In the methodology presented here, a VAE is utilised as an alternative, independent method to cluster analysis. During model training, the VAE adapts the ANNs to the input feature vectors. This adaptation is accomplished by optimizing a loss function composed of (i) the Reconstruction Loss, which assesses the effectiveness of the Decoder in reconstructing the input data (usually measured through mean squared error or cross-entropy); and (ii) the Kullback-Leibler Divergence, which encourages the learned posterior distribution to be close to a multivariate Gaussian distribution with and .

In our application, the optimal architecture of the VAE is determined automatically during training via the “growing” strategy of ANNs, which aims to identify the configuration that yields the highest overall reconstruction probability across the input vectors, while adding nodes to hidden layers incrementally. The dimensionality of the latent variable is also treated as a tunable hyperparameter and optimised during model training. Specifically, latent dimensions ranging from 1 to the feature dimensionality () are systematically evaluated, and the dimensionality yielding the highest reconstruction probability is selected. Notably, searching for a dimensionality lower or equal to that of forces the model to search for compact, disentangled representations of the data that should reduce noise, possibly merge correlated features, and extract dominant generative factors¹⁴⁸.

The optimal VAE architecture identified for the presented case study (section “Case study”) consisted of 5 neurons in the first hidden layer and 2 neurons in the second. The Decoder mirrored this architecture, comprising 2 and 5 neurons in the two hidden layers, respectively, and a final output layer with 6 neurons. The optimal configuration for corresponded to the latent dimensionality matching the feature dimensionality (). Lower-dimensional latent spaces resulted in a systematic increase in reconstruction error for both common and rare species, reducing the contrast required for anomaly-based discrimination. This parametrisation of aligned with those found by ecosystem risk assessment models¹¹⁷.

A threshold on the reconstruction probability across the training dataset can be employed to distinguish anomalous data (which exhibit lower reconstruction probabilities) from non-anomalous data (which exhibit higher reconstruction probabilities).

In our methodology, the model self-trains on all species-commonness feature vectors and estimates reconstruction probabilities for each vector. The output of the VAE is manifested as a continuous value (reconstruction probability) in an additional “VAE” column within the species-data table.

Classification

To classify species commonness and assign a prevalence score, our methodology employs statistical analysis on the outputs generated by the clustering and VAE models (Fig. 1-step 3).

For the clustering models, the analysis begins by calculating the cluster centroids (mean values per feature in the cluster). Subsequently, it determines the quartiles of the statistical distributions of each feature across the entire dataset. The 25th and 75th percentiles of each distribution are utilized to assign labels to each value of a centroid vector: the label “H” (high) is assigned if the value value exceeds the 75th percentile, while the label “L” (low) is attributed if the value falls below the 25th percentile; otherwise, the label “M” (medium) is assigned. This labelling arises from the understanding that a variable whose average value within a cluster exceeds the 75th percentile (or, equivalently, falls below the 25th percentile) of its distribution across the entire dataset can be considered “high-valued” (or, equivalently, “low-valued”) within that cluster.

The purpose of this labelling is to establish an overall commonness level associated with each cluster. For every cluster centroid, the analysis counts the features categorised as H, M, and L separately. If the H labels dominate, the cluster is categorised as a “high species-commonness” cluster; conversely, if the L labels predominate, it is classified as a “low species-commonness” cluster. All other clusters are designated as corresponding to “medium species-commonness”. The species associated with feature vectors residing within high-commonness clusters are labelled as “very common”, whereas those in the low-commonness clusters are categorised as “rare” and those in the medium-commonness clusters as “fairly common”.

In summary, this classification methodology employs a consensus approach based on the statistical classifications of individual features. A cluster that exhibits high values for the majority of the features is regarded as associated with “high species commonness”, due to the simultaneous occurrence of features with elevated levels. This classification mechanism is also capable of yielding assessments with empty commonness classes; for instance, if no cluster contains a majority of “L” values, the classification will not include “rare” species.

In the final computational step, the classification process assigns a prevalence probability value of 0.2 (rare), 0.5 (fairly common), and 0.8 (very common) to each species’ classification, to be used as a multiplicative weight in Bayesian ENMs.

The consensus mechanism explained also allows for the introduction of more nuanced classifications when the models detect more than three clusters. Specifically, by counting the number of high-valued features (up to a maximum of 6), a 12-level gradient can be created between low and high commonness classifications. This scale encompasses six levels between a full-medium-commonness classification (characterised by all “M” values) and a full-high-commonness classification (comprising entirely “H” values), and other six levels between a full-low-commonness classification (containing all “L” values) and the full-medium-commonness classification. These 12 levels correspond to evenly spaced species prevalence scores ranging from 0.2 to 0.8 (in increments of 0.05). As indicated in the introduction of “Methods”, the necessity for an increased granularity in prevalence assessment should be cautiously evaluated against the sensitivity of the ENMs within the final objectives of the assessment.

A similar classification process is also used for the VAE. Depending on the desired number of species-commonness classes to identify (e.g., 3=low/medium/high commonness), the distribution of the VAE reconstruction-probability across the species is divided into a corresponding number of quartiles. The centroids of the feature vectors residing within each quartile are then calculated, and the same classification procedure described for the clustering models is subsequently executed. This approach stems from the consideration that, provided the species list is sufficiently rich and balanced between rare and common species, anomalous feature vectors are often associated with “rare” species⁶⁰. In quantitative terms, this condition requires (i) a sufficiently large number of species relative to the feature dimensionality, so that the VAE can learn a stable latent representation (as a rule of thumb, at least one order of magnitude more species than features^152,153), and (ii) a moderate class imbalance, where rare species represent a minority of the dataset (typically below 30–40%). Under these conditions, common species define the dominant structure of the feature space, while rare species populate the lower-density regions of the learned manifold and exhibit lower reconstruction probabilities. The rarity of a species may manifest as particularly low values in one or more features (e.g., low abundance, sparsity, or infrequent observation)²², leading to heterogeneous and poorly reconstructed vectors. Consequently, rare species typically form small, dispersed groups that deviate from the reference patterns learned by the VAE. Conversely, when the species list is insufficiently large, highly imbalanced, or lacks feature heterogeneity, the distinction between common and rare species in the reconstruction-probability distribution becomes less pronounced. In such statistically underdetermined conditions, automatic thresholding based solely on quantiles may be unreliable. In these cases, a limited set of expert-identified reference species can be used to calibrate the reconstruction-probability thresholds, assuming that rare species generally exhibit lower reconstruction probabilities than very common species. Importantly, this calibration step does not subvert the data-driven nature of the classification: expert knowledge is used only to anchor the thresholds in scenarios where the available data do not support a stable statistical separation, and the calibrated thresholds are then applied uniformly to all species in the list.

As a concluding step, our methodology creates an ensemble assessment that integrates the evaluations provided by the three models (Fig. 1-step 4). This is realised as a consensus model that assigns a commonness classification to each species, corresponding to the classification that receives the majority of assessments from the three models. In instances where no classification achieves a majority, the model defaults to assigning the “fairly common” classification.

In summary, the classification phase assigns four species-commonness labels to each species from the analyses of the Multi K-means, X-means, VAE, and ensemble models, respectively. These labels are then translated into species prevalence estimations (Fig. 1-step 5) and attached to the species-data table. This is the final output of the presented methodology.

Case study

As a case study area, the basin of the Massaciuccoli Lake in Tuscany, Italy, was selected (Fig. 2). This area spans 114 , bordered to the North by the Camaiore River, to the East by the Oltre Serchio Mountains, to the South by the Serchio River, and to the West by the Tyrrhenian Sea. The lake itself covers an area of 13 and has a depth ranging from 1.0 to 2.5 m, with a total water volume of approximately 14 ^35,154–156. The primary source of water for the lake is the pluvial regime supplied by the eastern mountains and adjacent agricultural soils. Human activities surrounding the basin have a significant impact on water resources. Agriculture accounts for 40% of these activities, which overall include: farming, cereal and industrial production, horticulture, olive orchards, and railway transportation¹⁵⁷. Additionally, two wastewater treatment plants operate within the basin, discharging effluents directly into the lake.

The ecological significance of this region is closely linked to its impact on the rich biodiversity, particularly the concentration of bird populations around the lake and its role as a critical hub for migratory routes, which in turn affects related ecosystem services^35,36. However, the biodiversity of birds residing in or stopping over this area during migration is likely to be adversely affected by climate change^37–40. Models predict that habitat conditions will become increasingly unsuitable for the majority of bird species by 2100 due to anticipated rises in temperature and aridity⁶⁰. Shifts in the distributions of both terrestrial and aquatic species are expected, driven by alterations in habitat suitability. Overall, climate change presents significant risks to the vulnerability of species inhabiting the Massaciuccoli basin, consequently impacting the ecosystem services and socio-economic values associated with these species^{37,40,156,158,159}. In light of these challenges, policy-making authorities are investigating various actions aimed at mitigating these risks¹⁶⁰, with a principal focus on ensuring the sustained presence of water in the lake. However, these initiatives necessitate robust support from the scientific community to effectively evaluate the timing of ecosystem changes and identify the species most threatened by these alterations. Such assessments can play a critical role in management planning.

To date, scientific investigations concerning the Massaciuccoli Lake basin have primarily focused on statistically analysing the basin’s aggregated species composition and trophic chains^38,39, alongside specific analyses of vegetation species relevant to the lake’s restoration from eutrophication³⁷. Studies on the effects of climate change have examined the long-term impacts of fire and vegetation alterations on current biodiversity^40,161–163, as well as probing the basin for ongoing geochemical, hydrological, and environmental changes. Notably, one study⁶⁰ has developed future habitat distribution assessments for 180 species utilising ENMs. This research has also underscored the robustness and reliability of such models for aggregated assessments of biodiversity trends, while emphasising the necessity for accurate prevalence assessments to mitigate uncertainty in trend projections and enhance model accuracy.

At the time of this study, GBIF hosts approximately 9,000 records spanning the last decade (2015-2025) for the fauna (Kingdom Animalia) residing in the Massaciuccoli basin. These records encompass approximately 400 species from a total of 11 datasets. Among these, 180 species are characterised by non-occasional records and are also present in other Italian wetlands; thus, they can be taken as a reliable reference for animals living in the basin.

Two local experts involved in the present study, who signed informed consent to participate in this research, with decadal experience in identifying the Massaciuccoli species, were able to assess the prevalence of 161 of these species (listed in “Data availability”), thereby enabling an evaluation of our methodology. Of the assessed species, 130 (~ 83%) were birds, 24 (~ 13%) insects, 3 (2%) amphibians and terrestrial species, and 4 (~ 2%) fishes and crustaceans. Figure 4 shows four examples of common species from the different groups. The next section explains the utilisation of these data in the assessment of our methodology.

Fig. 4.

Examples of common species, from different taxonomic groups, living in the Massaciuccoli Lake basin. The red dots illustrate the spatial distributions of the species occurrence records in GBIF (URL: www.gbif.org). Maps were created with QGIS v.3.40 (URL: www.qgis.org). The Tuscany background map was taken from the Tuscany Regional Government online GIS service (URL: www502.regione.toscana.it/geoscopio/cartoteca.html - Carta Topografica). The species’ images were taken from Wikipedia (URL: it.wikipedia.org/wiki/File:Western_Marsh_Harrier-_Bangalore,_India.jpg, cy.wikipedia.org/wiki/Mursen_dinlas_gyffredin#/media/Delwedd:Blue-tailed_damselfly_(Ischnura_elegans)_male_adult.jpg, it.wikipedia.org/wiki/Procambarus_clarkii#/media/File:Procambarus_clarkii.jpg, and it.wikipedia.org/wiki/File:Podarcis_muralis_on_branch.jpg).

Evaluation methodology

To evaluate our methodology, a common challenge in Ecological Niche Modelling was simulated, that is, the prior assessment of a species’ prevalence in an area before the observation of the species in that area^15,164–168. Within this challenge, the goal is to assess the presence of the species in an area where surveys have not yet been conducted. This is crucial for models tasked with estimating the potential ecological niche of a species, i.e., the species’ habitat within an area different from its native area. In this context, estimating reliable prior information regarding species presence is paramount^{26,63,169–172}, because the environmental variables in the native habitat may not fully represent the conditions of the target area due to slight differences in the variables’ ranges^173–176. Therefore, the results derived from these ENMs should be interpreted with caution and are strongly dependent upon prior probabilities^{173,177–180}. If a species is anticipated to be rare in the new habitat, the multiplicative prevalence probability will lead the model to generate more precautionary (stricter) assessments. Conversely, if the species is expected to be common, the model will be inclined to assess a greater suitability.

To simulate this scenario for the 161 selected species of the Massaciuccoli basin, observations from the Massaciuccoli area itself were excluded from the data collection. Instead, observations from analogous habitats were utilised. Specifically, observations from Italian wetlands, as defined by the Ramsar Convention on Wetlands¹⁸¹, were collected. These wetlands, represented as a geospatial vector feature file (Fig. 2-right panel), encompassed 57 polygonal areas that were utilised as datasets in the feature definitions (section “Feature extraction”). GBIF included a total of 6027 observations in these areas for the 161 species analysed. The threshold for the minimum observations in the HF feature was set to 5, corresponding to the 25th percentile of the distribution of the number of species records across the Ramsar wetlands. The time unit for F and HF was one year. The spatial resolution for E was 0.01° (~ 1 km).

The target was to classify species as “very common”, “fairly common”, or “rare” through the four models explained in “Classification” (Multi K-means, X-means, VAE, and ensemble assessment). Additionally, a binary assessment was conducted to distinguish between “very common” and “less common” species, merging the “rare” and “fairly common” categories. These assessments were compared against the evaluations made by the two experts on the same categories, who classified the species based on their expertise regarding the basin’s biodiversity. The experts were permitted to consult relevant literature as well as the GBIF occurrence distributions within the Massaciuccoli basin, but they did not have access to the automatic assessments. Furthermore, an additional expert assessment was simulated via a consensus model built upon the evaluations of the two experts. This “expert ensemble” was constructed using the same consensus modelling approach as the “ensemble assessment” method (section “Classification”).

For evaluation metrics, accuracy was calculated as the fraction of categories jointly assessed by one method in comparison to each expert’s assessment. In addition, the agreement relative to chance was quantified through Cohen’s Kappa¹⁸².

As an additional evaluation, a sensitivity analysis was performed using a leave-one-out approach, wherein one feature at a time was omitted from the modelling, and the agreement calculation between the “ensemble assessment” and the “expert ensemble” was repeated. This process ultimately resulted in a ranking of the features based on agreement loss, thereby ordering the features according to their contribution to classification performance.

Finally, the species that were misclassified by all methods were analysed to investigate potential causes of errors.

Overall, the Massaciuccoli basin was used as a representative and ecologically meaningful case study to demonstrate the behaviour and interpretability of the proposed framework. The specific prevalence distributions and class boundaries observed in this system were therefore not intended to be universally applicable, but rather to illustrate how the method operates when applied to a real-world, heterogeneous species assemblage.

Design choices and alternatives

The proposed methodology relies on several explicit modelling assumptions that warrant discussion. First, it assumes that species prevalence can be meaningfully inferred from a small set of aggregated, ecologically motivated features derived from occurrence records, rather than from species-specific environmental response models. Second, the use of clustering and anomaly-detection techniques assumes that common species share dominant patterns in this feature space, while rare or undersampled species manifest as deviations from these patterns. These assumptions motivate, but do not uniquely determine, the chosen modelling components.

Alternative design choices are possible. For example, different clustering algorithms, distance metrics, or latent representations could be employed, and prevalence thresholds could be defined using various quantile schemes or continuous scoring methods²². We expect such alternatives to affect more the exact boundaries between prevalence classes than the overall ordering of species along the common–rare gradient, provided that the feature space remains sufficiently informative and the species set is large enough. The methodology presented here, therefore, represents one principled and reproducible instance within a broader class of unsupervised prevalence assessment approaches, rather than a uniquely determined solution.

Author contributions

PB: Conceptualisation, Methodology, Software, Writing. AB: Data Collection, Validation. GC: Conceptualisation, Methodology, Software, Validation, Supervision, Writing.

Funding

The publication has been funded by EU—Next Generation EU Mission 4, Component 2—CUP B53C22002150006—Project IR0000032—ITINERIS—Italian Integrated Environmental Research Infrastructures System.

Data availability

All input and output data, and the software are available on GitHub at https://github.com/cybprojects65/AutomaticSpeciesPrevalenceAssessment . The list of species used in the case study is available within the same repository, at https://github.com/cybprojects65/AutomaticSpeciesPrevalenceAssessment/blob/main/Input/Species_and_groups.csv

Declarations

Competing interests

The authors declare no competing interests.

Ethical approval

Our research did not include invasive research on human or animal subjects. The experts involved signed an informed consent before conducting their assessments.