A global dataset and map of soil phosphorus retention / sorption for assessing phosphorus use efficiency and loss to water

Abstract

Soil phosphorus (P) retention can inform both P use efficiency and the potential for dissolved P loss from soil to water. We compiled and harmonised global measurements of topsoil (0–20 cm) P retention to produce a consistent database and predictive global map. Data from international soil surveys, national repositories, and published studies were screened for coordinate accuracy, methodological consistency, sampling depth, and duplication. After filtering, 6,478 observations from 68 countries were paired with climatic, land-use, soil, and ecological variables at 10-km² resolution. Several modelling approaches were assessed, with quantile random forests selected to predict the median, 20^th, and 80^th percentiles of global P retention. Predictions were generated on a 10-km² grid consistent and resampled to finer resolution for visualisation purposes without increasing the effective spatial detail. The resulting maps show broad spatial patterns and associated uncertainties that reflect major controls such as soil weathering intensity and hydrology, as well as soil physical properties indirectly related to texture. The dataset provides observation-based, quantitative estimates of topsoil P retention that improve upon earlier qualitative global assessments derived from soil classification. These data are intended for application at catchment to regional scales to support assessments of P use efficiency and dissolved P loss risk, rather than fine-scale management decisions.

Background & Summary

Soil phosphorus (P) retention is a measure of the soil’s capacity to remove and retain P from the soil solution. The original method was developed in 1965 in New Zealand to distinguish different soil types¹, and subsequently adopted in the United States to identify andic soil properties in USDA Soil Taxonomy². Phosphorus retention is now also used either in its original or modified form to explain crop response to P fertiliser additions and to predict the potential for dissolved P loss from soil to water either via surface runoff or leachate^3,4. The original method involved shaking soil with a buffered solution at pH 4.6 containing 1000 mg L⁻¹ P. This high concentration of P effectively saturated the soil, overcoming any prior P additions in most agricultural soils, and resulted in repeatable data across soil types whose P sorption was characterised by a broad concentration range of aluminium (Al) and iron (Fe) oxides. Subsequent methods have used lower P concentrations or removed the buffer to better differentiate soil types owing to much stronger Al and Fe oxide concentrations (e.g., adapted to Oxisols in Brazil⁵) or a greater influence of calcium (e.g., phosphate buffering index used in Australia^6–8).

The International Soil Reference and Information Centre developed a global map of P retention using data from the World Inventory of Soil Emission database (WISE) in 2011. The map classified P retention (0–100%) for a soil profile into 16 classes (very low to very high) based off an understanding of the retention of P in response to soil pH, cation exchange capacity, and soil mineralogy as mapped by FAO soil name (at 1:5,000,000 scale)⁹. However, the authors noted that the data have large (but undefined) uncertainties associated and recommend that the map only be used in an exploratory manner until more detailed information is available at a regional level. Furthermore, the data were derived for the entire soil profile, whereas using P retention to estimate P loss from soil to water or crop response is best correlated with topsoil (0–20 cm) characteristics^10,11.

Despite the important of P retention for both agronomic and environmental applications, globally consistent, observation-based datasets of topsoil P retention remain limited. Over time, P retention data have been collected and stored by a range of jurisdictions and made publicly available^12,13, but the datasets are are heterogeneous in terms of analytical methods, sampling depth, and reporting standards. This heterogeneity limits their direct use for global assessments without careful harmonisation. To address this gap, we collated and standardised global measurements of topsoil P retention, aligning them with consistent soil depth definitions and environmental covariates. This harmonised dataset provides a foundation for developing spatially explicit estimates of P retention that are directly linked to measured observations rather than inferred soil classes.

In contrast to earlier approaches based primarily on soil classification, the present study combines this harmonised dataset (n = 6,478) with climatic, soil, land-use and environmental predictors to generate quantitative, spatially continuous estimates of topsoil P retention. This approach enables the representation of global patterns in P retention while also providing associated uncertainty estimates derived from the quantile random forest model.

We envisage that the resulting dataset and maps will be appropriate to inform assessments at the catchment or regional scale of phosphorus use efficiency and the risk of dissolved P from soil to water. The data are not intended for field- or farm-level decision-making, where local variability in soil properties may not be greater than that captured by our model.

Methods

Data were sourced from a combination of global or regional databases and international publications. Each source was checked for:

1. Point data with correct geographic coordinates (decimal latitude and longitude) and not incorrectly input (e.g., the letter O was input instead of the number zero, or latitudes were recorded without negative sign to a denote Southern Hemisphere location).

2. The detection limit was ≥1%, removing any values below this limit.

3. Data were derived after 1980 allowing sufficient time for laboratories to correctly use the method of Saunders¹.

4. Duplicated points. Although these data are likely to come from separate locations, they could not be separated at the resolution of co-ordinates extracted (e.g., through changes in soil types in an agronomic trial). We therefore took the mean of those data with different values.

5. Samples taken at various depths of the topsoil to no deeper than 25 cm of agricultural soils - i.e., the plough layer.

Finally, we converted other measures of P sorption to P retention via regression relationships (see Technical Validation). The unfiltered dataset comprised 48,188 data points from around the world (Table 1), but after checking and filtering, this reduced to 6,478 points from 68 countries (Figs. 1, 2) of consistent methods and depth (mean sampling depth = 0–17 cm; Table 2)¹⁴. The reduction in sample size (approximately one in eight observations retained) reflects inconsistencies among source datasets. Although this filtering substantially reduced the number of observations, it improved the consistency and comparability of the dataset used for modelling. Retaining inconsistent or poorly constrained data would likely introduce greater uncertainty and bias into the predictions than reducing sample size. Nevertheless, the filtering process may have reduced representation in some regions or soil types, particularly where data quality or reporting standards were lower.

Table 1.

Sources of P retention, their geographic spread and the number of samples before and after filtering steps.

Database	Geographic extent	Unfiltered observations	Filtered observations	References#
World Soil Information Service, United States National Cooperative Soil Survey	Global	32,372	2,503	12,41
Manaaki Whenua - Landcare Research	Cook Islands, Fiji, New Zealand, Niue, Indonesia, Tonga, Vanuatu	4,712	305	13
Australian Soil Resource Information System	Australia	516	505	42
Published studies	Global (74 countries)	11108	3165 (2770)*	5,32,33,43–53
Sum		48,708	6,478

*Converted from P sorption maximum, P buffering index or oxalate extractable Al to P retention, see Technical Validation.

^#URLs to each database are provided in the references and in the.xlsx file in the Figshare repository¹⁴.

Fig. 1.

Location of samples with filtered P retention data. The base map used data sourced from OpenStreetMap contributors available under an Open Database License (https://www.openstreetmap.org/copyright).

Fig. 2.

Box plots of observed soil P retention (%) for countries with >20 observations to ensure stable estimation of distributional statistics like median and percentiles. Boxes contain the median value bounded by the 5^th, 25^th, 75^th and 95^th percentile of data. Countries with fewer observations were combined into “All others” to ensure robust estimation of descriptive statistics.

Table 2.

Count and mean P retention, year sampled and depth of sampling by continent.

Continent	Count	Mean P retention (%)	Mean year sampled	Mean sampling depth (cm)
Africa	1,724	16	1983	21
Asia	853	27	1986	15
Europe	354	30	2003	18
North America	1,920	35	1997	16
Oceania	909	31	1985	13
South America	625	45	2019	14
Global	6,385	30	1994	17

Geographically, the continent with the most data was North America, followed by Oceania and Africa (Table 2; Fig. 1). After inspecting the data, we found that four biomes suffered from a paucity of data (n < 15): boreal forests/Taiga, Mangroves, tropical and subtropical coniferous forests, and tundra. These biomes represented 12.6%, <1%, ~2% and 8%, respectively, of the global land area, but apart from the small area of tropical and subtropical coniferous forests, are all largely unproductive.

Modelling

The filtered data were paired with predictor variables obtained from a variety of sources (Table 3). These predictor variables were chosen due to their high likelihood of influencing soil P retention and included climate parameters, land use, soil chemical and physical parameters and ecological classifications^15,16. Additional geographic and socio-economic descriptors (e.g., country and income class) were included to capture regional differences in land management intensity and environmental conditions that may indirectly influence soil development and P dynamics. We extracted data for each predictor variable from the sources outlined in Table 3 at a resolution of 10 km², filtering out polar regions in permafrost, deserts and mountainous regions that are unlikely to be used for primary production¹⁷ leaving 12,453,854 filtered points. Some of the filtered predictor data did not have consistent coverage with all of the predictor datasets and were removed before being modelled - leaving 5,730 points that corresponded to soil P retention points.

Table 3.

Climatic, biophysical and geographic variables and the units, years and sources of the variables used to predict the global P retention.

Variable	Unit	Year parameters was measured	Source
Country	Name	2015
Mean annual temperature	°C	2015	54
Mean annual precipitation	mm	2015	54
Elevation	m above sea level	2010	55
Slope	%	2010	55
Soil bulk density (0–15 cm)	g cm−3	2016	56
Sand (0–15 cm)	%	2016	56
Silt (0–15 cm)	%	2016	56
Clay (0–15 cm)	%	2016	56
Soil pH (water, 0–15 cm)	—	2016	56
Soil organic carbon (0–15 cm)	g kg−1	2016	56
Soil total nitrogen (0–15 cm)	%	2016	56
Non-anthropogenic total P (0–20 cm)	mg kg−1	2015	31
Soil volumetric water holding capacity (0–15 cm)	g cm−3 at −33 kpa	2016	56
Soil cation exchange capacity (0–15 cm)	meq. 100 g−1	2016	56
Enhanced vegetation index (EVI, monthly)	EVI at a 1 km2 resolution	2010	57
Normalised difference vegetation index (NDVI, monthly)	NVDI at a 1 km2 resolution	2010	57
Mean annual runoff	mm	2014	58
Population density	Persons/km2	2015	59
Cropland	Class (yes/no)	2000	60
Percentage of forest	%	2015	61
Percentage of bare land	%	2015	61
Percentage of grassland or shrubland if not forestland, cropland or bare land	%	2015	60–62
World Reference Base Soil Group	Class	2016	56
Soil parent material	Class	2010	63
Income class	Class (low - high)		64
Catchment area	km2	2010	65
Biome	Class	2016	66

Soil P retention data were confirmed as approximately normally distributed with the Shapiro-Wilk test. A range of models (classification and regression trees¹⁸, multivariate adaptive regression splines¹⁹, gradient boosted regression trees²⁰, random forests regression²¹, and quantile random forests regression^22,23) were trialled to predict P retention. Although not strictly necessary to ensure the good performance of these models, predictor variables were either natural log-transformed, or arcsine transformed (for values between 0–100%) to generate predictors with equal variance and maximise the chance that predictors had an unbiased chance of being used by the model. During modelling we used 70% of the data to train the models, while the remaining 30% was reserved to evaluate model performance. To minimise geographic bias and reduce the potential influence of spatial autocorrelation on model evaluation, training and validation datasets were constructed to include comparable proportions of monitoring sites from each continent. This approach ensured that model performance reflected global rather than regionally dominated patterns.

After selecting quantile random forest regression as the best model (Table 4), we assessed the risk of overfitting due to spatial autocorrelation using cross-validation²⁴. The final model was retrained using the full dataset to leverage all available information for spatial prediction^25,26. This approach improves predictive accuracy once the model’s structure, parameters and performance have been validated^27,28. Performance metrics for the final predictions were not calculated, as no independent data remained for testing. Model performance was instead evaluated using held-out validation data and cross-validation prior to final model fitting. Predictor importance was evaluated using permutation importance, expressed as the percentage increase in mean squared error when each predictor was randomly permuted in the fitted model.

Table 4.

Model performance for the prediction of P retention.

Model approach	Coefficient of determination	Root mean square error	Mean absolute error
Classification and regression trees	0.54	18.2	13.1
Multivariate adaptive regression splines	0.55	18.0	13.4
Gradient boosted regression trees	0.67	15.3	10.6
Quantile random forest regression	0.67	13.9	8.8

The best performing model (quantile Random Forest) used 26 predictors each contributing to predictive performance to varying degrees (Fig. 2). In contrast to parametric regression models, predictors in random forest models are not retained based on statistical significance but on their contribution to predictive accuracy and interactions with other variables. Consequently, predictors with relatively low individual importance may still improve model performance when included with other variables. The relative importance values therefore reflect the contribution of predictors to model performance rather than direct causal relationships.

The most important variables were annual runoff, soil water holding capacity, soil bulk density and annual precipitation, whereas soil group and the presence (or absence) of cropland were among the least influential. The importance of runoff and rainfall may reflect strongly leached soils in high rainfall environments^1,5 (Fig. 3). Indeed, predicted P retention was noted to increase above an annual rainfall of 1100 mm and runoff of 150 mm (Fig. 3).

Fig. 3.

Relative importance of predictor for the estimation of soil P retention. The output is for the fit of the quantile random forests model to all data.

The partial dependence plots (Fig. 4) provide further insight into these relationships. The increase in P retention with runoff and rainfall is consistent with enhanced soil weathering and the accumulation of aluminium and iron oxides, which provide reactive surfaces for P sorption. Similarly, higher soil water holding capacity, often associated with finer-textured soils, reflects greater surface area and sorption capacity. In contrast, bulk density may influence P retention indirectly through its effects on soil structure and porosity, which regulate water movement and contact time between soil particles and dissolved P. Together, these patterns are consistent with established controls on soil P retention, where both pedogenic processes and physical soil properties govern sorption capacity and P transport.

Fig. 4.

Partial dependence plot from the model for the three most importance variables (runoff, volumetric water holding capacity, and bulk density) from the fit of a quantile random forest model to predict P retention. These relationships reflect key controls on P retention, including soil weathering and oxide formation (runoff), surface area and sorption capacity (water holding capacity), and soil structure and water movement (bulk density).

These predicted spatial patterns are broadly consistent with earlier qualitative assessments of global P retention potential derived from soil classification⁹, but the present dataset provides quantitative estimates of topsoil P retention derived from measured observations and environmental predictors, enabling improved spatial representation and uncertainty estimation. Land use characteristics and soil classifications were of low importance, reflecting their likely capture in more important climatic variables and ability to farm almost any soil type with appropriate management. Factors affecting soil P sorption such as soil textural composition²⁹, soil organic carbon³⁰ and native or non-anthropogenic soil P³¹ were of modest importance.

To assess whether model performance was driven primarily by a small number of predictors, we re-ran the model using only the five most important variables. This reduced the coefficient of determination from R² = 0.68 using all 26 predictors to R² = 0.62, indicating that while the most influential predictors explain much of the variation in soil P retention, the remaining variables collectively provide additional predictive information.

Predictions of soil P retention in topsoil were projected globally in Q-GIS to a grid at a spatial resolution of 10-km², which corresponds to the coarsest grid cells associated with the input data as listed in Table 3. For visualisation purposes, these predictions were resampled to 30 arc-seconds (~1 km² near the equator) using nearest-neighbour interpolation, which does not introduce additional spatial information but allows the results to be displayed at a resolution commonly used for global environmental datasets. Both the predicted 1-km² grids (EPSG8857) and geotiffs in a WGS84 (EPSG:4326) projection are included in the Figshare repository¹⁴. Once projected globally, the mean predicted median value for P retention was 20%. The corresponding mean of the 20^th and 80^th percentiles were 10 and 34%, respectively indicating a relatively tight range of P retention values with only a few areas exhibiting high P retention percentages. As per previous attempts to measure and map P retention^9,32, high P retention values tended to be greatest in the tropics and the least towards the temperate regions. The exceptions were for soils derived from volcanic activity in, for example, The Pacific Ring of Fire linking parts of Japan, northwest USA, Chile and New Zealand (Fig. 5).

Fig. 5.

Map showing the 20^th (top), 50^th (middle) and 80^th (bottom) percentile of estimated P retention derived from the quantile random forest model. Predictions were generated from a database containing 6,478 observations paired with environmental predictors and initially produced on a 10 km² grid, corresponding to the coarsest spatial resolution of the predictor datasets. For visualisation, predictions were resampled to 1 km² using nearest-neighbour assignment, which does not introduce additional spatial information but allows compatibility with commonly used global environmental datasets. The base map used data sourced from OpenStreetMap contributors available under an Open Database License (https://www.openstreetmap.org/copyright).

Uncertainties are presented in Fig. 5 as the estimated 20^th and 80^th percentiles of P retention, and in Fig. 6 as the residuals from the quantile random forest model fitted to point data. To assess the spatial consistency of model performance and identify potential regional biases, we summarised observed and predicted P retention and associated residuals by continent (Table 5). This regional aggregation provides a coarse but informative assessment of model performance across major geographic domains, where differences in climate, soil development and data availability may influence predictive accuracy. Geographically, residuals varied by continent, with largest values observed in Europe and the smallest in North America, suggesting some regional variation in model performance. Although the residuals are relatively small, on average, there is still scope for prediction errors (mean coefficient of variation = 23% globally; Table 5). Given this variation, the data are better suited for classifying P retention levels rather than for assigning precise absolute values.

Fig. 6.

Map showing the residuals as the difference between observed and estimated percentage P retention values. The base map used data sourced from OpenStreetMap contributors available under an Open Database License (https://www.openstreetmap.org/copyright).

Table 5.

Mean of P retention observations, the mean of predicted P retention observations, residuals and the coefficient of variation by continent and worldwide.

Continent	Observed mean P retention (%)	Predicted mean P retention (%)	P retention residuals (observed - predicted; %)	Coefficient of variation for residuals (%)
Africa	16.0	16.2	−0.2	22.2
Asia	24.4	24.8	−0.5	20.8
Europe	30.0	31.0	−1.0	35.5
North America	31.5	31.6	−0.1	18.3
Oceania	29.3	29.0	0.2	26.1
South America	44.5	44.2	0.3	19.1
Worldwide	26.9	27.0	−0.1	22.7

All values are given as percentages. Residuals are calculated as the difference between the observed and predicted values.

Data Records

All code (“Pret_For_Paper.r”), data and fits (“Pretention_data_Residuals.csv” and “Final_Data_Fits.xlsx”) for the quantile Random Forest model used to generate the global grid at 10 km² (“Global10k2_P_Retention_Predictions_QGIS.zip”) and GeoTIFF maps of the median (50^th Pret.zip”) and 20^th (“20^th Pret.zip”) and 80^th (80^th Pret.zip”) percentile of predicted global P retention in the Figshare repository¹⁴ (https://doi.org/10.6084/m9.figshare.28612607). The spreadsheet contains the final filtered data from each source (Table 1).

Technical Validation

A small (33) and large (2770) number of sites needed P retention to be estimated from other measures of P sorption (phosphate buffer index [PBI] or the P sorption maximum [P_max]) or oxalate-extractable aluminium (Al_ox). To convert data from PBI or P_max into P retention, data were extracted from a study of 87 samples of topsoil (air-dried, sieved <2 mm, sampled to a mean depth of 15 cm in 2001), that had been analysed for P retention, PBI^6–8,33 and P_max as calculated from the fit of a Langmuir isotherm to a gradient of P solutions (from 0–300 mg P L⁻¹). The soils were from a diverse array of locations across Australia (including samples from all states) and exhibited a wide range in chemical characteristics similar to our filtered dataset for all variables but silt and Olsen P (Table 6). These samples allowed P retention to be estimated from relationships to PBI or P_max with good confidence (Fig. 6).

Table 6.

Descriptive statistics for samples used to derive a relationship between P retention and the P buffer index (PBI), P sorption maximum (P_max) or oxalate-extractable aluminium (Al_ox).

Dataset	pH	Clay (%)	Organic C (g kg−1)
PBI and Pmax
Mean	5.8	18	32
Standard error	0.8	14.6	20.4
Interquartile range	0.7	20.9	24.0
Alox
Mean	5.5*	24	87
Standard error	0.7	17.1	78.4
Interquartile range	0.7	18.5	55.0
P retention data
Mean	6.1	23	51
Standard error	1.0	11.1	41.7
Interquartile range	1.3	16.0	43.5

The asterisk refers to significant difference between the mean of P retention data and those calculated from PBI, P_max or Al_ox (P < 0.05, t-test on log-transformed data).

To convert data from Al_ox into P retention, we collated data from Ireland (27 soils)³⁴, Japan (81 soils)³⁵ and New Zealand (9 soils)¹ and one database of 54 historical soil samples from New Zealand³⁶. These 174 soils had similar chemical and physical characteristics to P retention dataset. We used the relationship between Al_ox and P retention (Fig. 7) to convert data on topsoil Al_ox as part of two global metanalyses of the influence of Al and Fe oxides on soil carbon^37,38 to yield an additional 2770 estimates of P retention.

Fig. 7.

Regression relationships to calculate P retention. Relationship between P retention and the phosphate buffer index (top), the Langmuir P sorption maximum (middle) and oxalate-extractable aluminium (bottom).

Usage Notes

The dataset is not intended for use at fine spatial scales such as individual farms. The effective resolution of the predictions is constrained by the coarsest input data (~10 km²), and local variability in soil properties may not be captured. However, the data may be used for catchment (>100 ha) or regional assessments about improving the utilisation of added P to different soil types and minimising the loss of dissolved P from the soil.

Curves between crop yield and soil test P are used for P fertilizer recommendations³⁹. However, these response curves vary by soil P retention. For example, New Zealand data for the response of pasture to soil Olsen P indicates an optimal yield at 20 and 35 mg kg⁻¹ for two soils with a P retention of about 30–50% and 60–80%, respectively (Fig. 8)⁴⁰. Over time, the extra fertilizer P required to maintain soil Olsen P at higher concentrations may be very expensive. Hence, a good estimate of P retention, even at a regional scale, will save many farmers money.

Fig. 8.

Relationship between soil Olsen P and the percentage relative pasture yield and P lost in surface runoff. The distance in Olsen P between the point at which 95% relative yield is obtained and for a break point in Olsen P indicating an increase in P loss in runoff in the 30% P retention soil is about 25 mg kg⁻¹ and 40 mg kg⁻¹ in the 70% P retention soil. Data for relative yield were sourced from New Zealand trials from 1960–1995 summarised in Edmeades et al.⁴⁰. Data for P loss in surface runoff in soils of 30 and 70% P retention were sourced from Morton et al.⁶⁷.

Phosphorus retention is an analogue for P sorption capacity. When combined as the denominator with soil test P as the numerator, the quotient can act as an indication of how much sorption capacity has been occupied by P additions. Moreover, as a single indicator, P retention will help identify break points in soil test P that can be modelled by piecewise regression to show where P loses in surface runoff accelerate³. Conceptually, the balance between production potential and environmental loss can be seen in Fig. 8 as the different in soil Olsen P concentrations between the point at which the P loss in surface runoff and 95% relative pasture yields are obtained for two soils of different P retention. This information allows land managers to adjust P additions cognizant of their production potential and environmental loss. In this example, there is little justification for enriching Olsen P beyond the 95% relative yield, but doing so would pose a greater risk to surface water quality in the soil with lower P retention.

Author contributions

R.W.M. conceived the idea and wrote the manuscript and conducted the geographic analysis. C.D. conducted the geographic analysis. A.N. and R.W.M. developed and performed the statistical analysis. R.W.M., L.B. and P.S.P. sourced and collated the raw data.

Code availability

No bespoke code was produced during the analysis. However, the data generated during each processing step, including the quantile Random Forest model (“PRet_For_Paper.r”), are available in the updated Figshare repository¹⁴.

Competing interests

The authors declare no competing interests.