Estimating soil moisture content under grassland with hyperspectral data using radiative transfer modelling and machine learning

Abstract

The monitoring of soil moisture content (SMC) at very high spatial resolution (<10m) using unmanned aerial systems (UAS) is of high interest for precision agriculture and the validation of large scale SMC products. Data-driven approaches are the most common method to retrieve SMC with UAS-borne data at water limited sites over non-disturbed agricultural crops. A major disadvantage of data-driven algorithms is the limited transferability in space and time and the need of a high number of ground reference samples. Physically-based approaches are less dependent on the amount of samples and are transferable in space and time. This study explores the potential of (1) a hybrid method targeting the soil brightness factor of the PROSAIL model using a variational heteroscedastic Gaussian Processes regression (VHGPR) algorithm, and (2) a data-driven method employing VHGPR for the retrieval of SMC over three grassland sites based on UAS-borne VIS-NIR (399-1001 nm) hyperspectral data. The sites were managed by mowing (Fendt), grazing (Grosses Bruch) and irrigation (Marquardt). With these distinct local pre-conditions we aimed to identify factors that favor and limit the retrieval of SMC.

The hybrid approach presented encouraging results in Marquardt (RMSE = 1.5 Vol_%, R² = 0.2). At the permanent grassland sites (Fendt, Grosses Bruch) the thatch layer jeopardized the application of the hybrid model. We identified the complex canopy structure of grassland as the main factor impacting the hybrid SMC retrieval. The data-driven approach showed high accuracy for Fendt (R² = 0.84, RMSE = 8.66) and Marquardt (R² = 0.4, RMSE = 10.52). All data-driven models build on the LAI-SMC relationship. However, this relationship was hampered by mowing (Fendt), leading to a lack of transferability in time. The alteration of plant traits by grazing prevents finding a relationship with SMC in Grosses Bruch. In Marquardt, we identified the timelag between changes in SMC and plant response as the main reason of decrease in model accuracy. Yet, the model performance is accurate in undisturbed and water-limited areas (Marquardt). The analysis points to challenges that need to be tackled in future research and opens the discussion for the development of robust models to retrieve high resolution SMC from UAS-borne remote sensing observations.

1. Introduction

Soil moisture content (SMC) controls land carbon uptake (Humphrey et al., 2021) and ecosystem productivity (Liu et al., 2020a). It is one of the key parameters of land–atmosphere-interactions (Ravindra Kumar Rekwar et al., 2022). Therefore, monitoring SMC at different spatial scales is of interest for a diversity of applications, such as crop yield estimation (Holzman et al., 2018), irrigation control (Li et al., 2022) or precision agriculture (Bhakta et al., 2019).

In addition to common local ground measurement techniques, a multitude of physically-based (Ma and Li, 2020; Li et al., 2021), semi-physical (Vergopolan et al., 2021; Sadeghi et al., 2017; Li et al., 2021) and data-driven (Döpper et al., 2022; Holtgrave et al., 2018; Liu et al., 2020c, 2021; Pasolli et al., 2014) remote sensing approaches have proven their ability to monitor SMC at different spatial resolutions over larger areas. However, the spatial resolution of space-borne remote sensing observations comprises a significant scale gap compared the local (point) ground reference. This unavoidably leads to inaccuracies when comparing the ground reference data to the remote sensing based SMC products or SMC relevant remote sensing signals (Gruber et al., 2020; Crow et al., 2005).

The high spatial resolution of unmanned aerial systems (UAS) (<1m x 1m) more closely corresponds to the soil medium represented by the local point measurements. Hence, UAS are a useful tool to bridge the scale gap between point measurements and satellite products by retrieving SMC at very high resolution. The application of UAS for SMC retrieval is thus not only relevant for precision agriculture, but also for hydrological analysis targeting SMC patterns at different spatial scales, as well as for validation and downscaling of space-borne remote sensing SMC products.

Reflecting the relevance of UAS-based remote sensing of SMC, an emerging body of literature demonstrates the ability of retrieving SMC using light-weight thermal infrared (TIR) (Paruta et al., 2021), RGB, multispectral (MS)(Cheng et al., 2022), hyperspectral (HS) or LiDAR (Wang et al., 2018; Ge et al., 2021) sensors mounted on UAS. Within this literature, different semi-empirical approaches such as the temperature-vegetation triangle (TDVI) (Wigmore et al., 2019; Wang et al., 2018), its modification towards the OPTRAM (Optical Trapezoid Model) (Babaeian et al., 2019) as well as thermal inertia (Paruta et al., 2021) were successfully applied.

Apart from the high variety of semi-physical approaches, the majority of analysis addressed the SMC retrieval via data-driven, mostly machine learning approaches (e.g. Babaeian et al., 2021; Hassan-Esfahani et al., 2017; Araya et al., 2020; Cheng et al., 2022). Data-driven approaches learn the complex interactions of the plant-soil-atmosphere continuum by building relationships between visible and near-infrared (VIS–NIR) spectral signals and plant characteristics. Thereby, a set of measured predictors, for example canopy reflectance, is directly linked to SMC as target variable via a regression model. In this regard, the SMC at the zone of major water uptake can be mapped (Mahyar Aboutalebi et al., 2019; Hassan-Esfahani et al., 2017; Babaeian et al., 2021). Data-driven algorithms are flexible in identifying relevant indicators of SMC. They can therefore use changes in soil reflectance as SMC-relevant indicators in addition to plant characteristics.

Most of the data-driven approaches have been applied at water-limited sites (e.g. Cheng et al., 2022; Babaeian et al., 2021) that are in some cases irrigated (Hassan-Esfahani et al., 2017; Mahyar Aboutalebi et al., 2019; Cheng et al., 2022). In addition, the algorithms were only tested and trained at one site (e.g. Hassan-Esfahani et al., 2017; Ge et al., 2021; Cheng et al., 2022) and at the same time (Ge et al., 2021; Seo et al., 2021). Lendzioch et al. (2021) targeted UAS-based SMC retrieval at a moist peat area with multitemporal data. They found in general lower model fits compared to analysis over agricultural sites (i.e. Cheng et al., 2022) and lowest model fits during the wettest hydrologic conditions.

The majority of analysis map SMC under agricultural crops (e.g. Cheng et al., 2022; Ge et al., 2021; Seo et al., 2021). Agricultural crops are often more susceptible to changes in water content than natural land cover types such as grassland (Levia et al., 2020). Grasslands occupy 40.5% of the global non-ice covered land area (White et al., 2000) and provide a large range of ecosystem services such as fodder production for livestock (Schucknecht et al., 2020) or soil formation (Zhao et al., 2020).

Most temperate grasslands of Germany are intensively managed by mowing, grazing and fertilization (Gilhaus et al., 2017). This management strongly alters plant physical properties (Benot et al., 2014; Opdekamp et al., 2012), productivity (Li et al., 2019) and species composition (Gilhaus et al., 2017). The potential of data-driven algorithms on grassland areas that are intensively managed is still not addressed.

Even if data-driven-approaches have shown their potential in retrieving SMC and they easily include additional SMC relevant predictors, they are not robust enough for a transfer in space and time (Rocha et al., 2019). Hassan-Esfahani et al. (2017) stressed the need to cover a multiyear phenology of each species in the training data set. This emphasizes the necessity of a high amount of training samples.

Radiative transfer models (RTM), in turn, physically describe the relationship between reflectance and soil properties and are therefore applicable at different times and sites (Bayat et al., 2020; Berger et al., 2018). They additionally reduce the need for calibration data (Bayat et al., 2020). Compared to data-driven modeling, which uses both vegetation and soil reflectance, the physically-based approach disentangles canopy and soil reflectance. The latter can then be used for SMC retrieval.

Soil reflectance is mainly determined by its chemical properties and physical structure as well as observation and illumination geometry. Temporal and spatial variability under a given homogeneous soil property are mainly caused by SMC and roughness (Weidong et al., 2002). Thus changes in soil reflectance are linked to SMC in the very first centimeters of the soil surface (Richter et al., 2012).

So far, an UAS application with RTM inversion targeting SMC has only been performed by Eon and Bachmann (2021). They applied the MARMIT (multilayer radiative transfer model of soil reflectance) (Bablet et al., 2018) at a bare dune site with good model performance. However, a coupling of MARMIT to leaf and canopy RTMs is still missing, which restricts the use of the model to bare-soil applications.

Richter et al. (2012) successfully derived SMC with an inverse application of the coupled leaf-canopy RTM PROSAIL for different agricultural crops using airborne hyperspectral VIS–NIR-SWIR (shortwave-infrared) data. The PROSAIL model requires the input of measured soil spectra whose brightness can be scaled via a dimensionless soil factor (psoil), corresponding to changes caused by different SMC (Atzberger et al., 2003). To the best of our knowledge, PROSAIL-based retrieval of SMC was neither performed using UAS-borne VIS–NIR hyperspectral data nor exploring grassland areas. Given the high potential of physically-based approaches due to their generality and low need of calibration data, its potential for different grassland sites should be explored.

Therefore, the main goals of this study were: (1) to explore the potential of data-driven and physically-based approaches to derive SMC at different test sites and times, (2) to understand how the anthropogenic influence on the site-conditions, namely mowing, grazing, and irrigation, affect the precision of the SMC retrieval, and (3) to define suitable data and framework conditions to derive SMC over grassland sites. Using VIS–NIR range hyperspectral data only, we apply the common data-driven as well as the novel physically-based approach in a hybrid retrieval strategy, exploring specific potentials and limitations on three differently managed grassland sites within Germany.

2. Materials and methods

To understand the potential of both SMC-retrieval approaches over a large range of hydrological and land-use conditions of grasslands in central Europe, we selected three sites in Germany (see Fig. 1) with different agricultural practises and distinct spatio-temporal SMC variability. At all sites one or several UAS-based hyperspectral images were acquired along with accompanying SMC and plant trait measurements.

Fig. 1.

False color images (957nm, 620nm, 421nm) and SMC timeseries at 5 or 10cm depth for Fendt (a,b,c), Grosses Bruch (e,f) and Marquardt(g,h,i,j,k). SMC timeseries are provided by Fersch et al. (2020), Till Francke and Maik Heistermann (UP) and Corrina Rebmann (UFZ).

2.1. Data and sites

2.1.1. Sensor and equipment

The hyperspectral imagery was acquired using a Nano-Hyperspec sensor (Headwall Photonics, Inc., USA) mounted on a DJI Matrice 600 Pro (DJI, China). The Nano-Hyperspec sensor captures 271 bands within the wavelength range from 399 to 1000nm and a spectral resolution of appox. 6nm. The angular field of view accounts for 15.3°. With a given flight altitude of 100 m, the acquisition and lens setting leads to a spatial resolution of approximately 4 cm x 4 cm. The imagery was acquired within a two hour range around solar noon under clear sky conditions.

Radiance and reflectance conversion was performed using the Headwall SpectralView software. We geo-referenced the reflectance data with the Georeferencer tool of ArcGIS (Esri Inc., USA). High resolution multi-spectral UAS data served as a basemap. The high resolution multi-spectral UAS data was orthomosaiced with Agisoft Metashape (Agisoft LLC, Russia). The geo-referencing of the multispectral orthomosaics was performed using ground control points. The location of the ground control points was defined using the Leica Zeno GG04 (Leica Geosystems AG, Switzerland) DGPS antenna with subpixel accuracy.

The geo-referenced hyperspectral products were spectrally corrected for spikes and drops and finally filtered by applying the Savitzky-Golay filter (Savitzky and Golay, 1964) as implemented in the Python SciPy module (Virtanen et al., 2020) with a window width of 7 and a second order polynomial smoothing. The dates of UAS image acquistion in the context of hydrological condition of each site are summarized in Fig. 1.

2.1.2. Fendt (FE)

Fendt is a temperate pre-alpine grassland site in the south of Germany. The soil texture comprises mainly silty and loamy sediments with peaty compositions towards the draining rivulet (Fersch et al., 2018). Typical clay, silt, and sand fractions are 32%, 41%, and 27%, respectively (Kiese et al., 2018).

The site is permanently and intensively used as grassland, leading to different mowing patterns (see Fig. 1 a, b). The hydrological condition at both dates of image acquisition differs markedly: Before 4 June 2019 heavy precipitation led to extremely high SMC values. This data acquisition captures a starting drying of the soils, leading to a high range of SMC. The second data acquisition for this site (27 June 2019), in turn, took place after a minor precipitation event 4 days before and captures the hydrological situation after significant drydown of the soils.

A SoilNet consisting of 55 nodes delivered the SMC ground reference data. Each node measures SMC at 5 cm depth every 15 min at two slightly displaced sensors. We took the mean of corresponding SoilNet nodes for each day. Due to malfunction of some nodes, only 79 SMC measurements were used as ground reference for the algorithms in Fendt. Further information on the sensors of the SoilNet and their calibration can be found in Fersch et al. (2020).

We sampled vegetation traits within 180 1x1 m plots during the days of hyperspectral image acquisition. Ten random leaf chlorophyll measurements within each plot where taken using a SPAD-502 Leaf Chlorophyll Meter (MINOLTA, Inc.). Those measurements where converted to chlorophyll measurements using the calibration curve of Si et al. (2012). The leaf area index (LAI) was assessed using the AccuPAR LP-80 (METER Group, Inc, USA). We took 5 above and 5 below measurements within each plot.

2.1.3. Grosses Bruch (GB)

Grosses Bruch is a pasture within a flat wetland area (Wollschläger et al., 2016). It is located in the Central German Lowland and susceptible to drought due to negative climatic water balance (Hermanns et al., 2021). The soil texture is defined as highly clayey silt (BGR, 2007). During the date of acquisition, the entire area was used for cattle grazing. We consider the hydrological status at the site as water limited, since Hermanns et al. (2021) detected drought stress of the vegetation at the site at even higher SMC than those measured in our data acqusition (Hermanns et al., 2021).

After the overflight, we took 51 SMC measurements with an FDR probe (ML3 Theta Probe, Delta-T Devices Ltd, Great Britain). For each sample, five measurements were taken in a cross-shaped arrangement within a 25 cm radius and then averaged. We recorded the GPS location with the Leica Zeno GG04 DGPS of the central measurement (Fig. 1e). The measurements were calibrated with a sensor specific calibration curve (Francke, 2020).

2.1.4. Marquardt (MAR)

Different to Fendt and Grosses Bruch, Marquardt is a non-permanent grassland site. The field is part of the Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB) and used as grass ley for the vegetation period of 2020. The present soil texture ranges from 4–14% in clay, 8–15% in silt and 71–88% in sand. The data acquisition was part of an irrigation experiment. Using an automatic irrigator, different parts of the field were ’flooded’. Thereby, approximately 40 to 50$\frac{1}{m^2}$ of water were applied within two consecutive days on the 5th and 6th of August 2020 and all at once on 11 August 2020 on different parts of the field. In order to cover the wettest SMC as well as ongoing drying of the soil after the irrigation, we acquired hyperspectral data on four dates with distinct time-gaps to the last applied irrigation (see Fig. 1k).

The experiment took place during dry conditions with no rainfall during the whole period. The ongoing drying of the soils is also reflected by the decreasing SMC values measured at the non-irrigated areas of the field. In total, we took 72 SMC samples. The sampling device and strategy as well as the device calibration is the same as in Section 2.1.3.

At Marquardt, we collected LAI samples with a Licor 2200 (LI-COR Biosciences GmbH, Germany) during dawn to keep the scattering effects of direct sunlight to a minimum. As in Fendt, each sample comprised 5 above and 5 below measurements.

2.2. Methods

A major focus of this analysis is the exploration of a data-driven and a hybrid method involving a physically-based approach in retrieving SMC over grassland areas that differ in land-use as well as spatio-temporal SMC characteristics (see Fig. 2). We implemented the hybrid approach of RTM inversion, which combines the efficiency of machine learning regression algorithms with the generality of the physically-based approaches (i.e., RTM) (Verrelst et al., 2019).

Fig. 2. Workflow of hybrid and data-driven SMC retrieval strategies.

Generally, a hybrid retrieval approach is considered to be more transferable, especially in time, compared to data-driven approaches assuming that all vegetation - soil - reflectance combinations were included in the simulated training data set. Nevertheless, site specific soil spectra are crucial for successful parameter retrieval. Since the soil factor of the PROSAIL model is dimensionless, SMC samples are necessary to enable a transfer of psoil to Vol_%. Yet, the number of required samples is expected to be lower than for the data-driven approach, which needs ground reference data for each site and date.

2.2.1. Hybrid SMC and LAI retrieval

A variational heteroscedastic Gaussian Processes regression (VHGPR) algorithm (Lázaro-Gredilla and Titsias, 2011) was trained to learn the physical connection between the target parameters (psoil and LAI) and the simulated spectra, and the final SMC retrieval model was subsequently applied to the measured spectra. Since machine learning algorithms are susceptible to collinearity of hyperspectral data, a reduction of the predicting features (bands) is necessary to provide confident results (Verrelst et al., 2016). In addition, active learning optimizes the samples of the simulated training data set and has also shown to increase the estimation accuracy through reducing redundancies within the samples (Berger et al., 2021; Verrelst et al., 2021).

We considered these enhancements of the hybrid inversion scheme as follows (see Fig. 2):

1. Training data set creation We created a 10.000-sample training data set using the PROSPECT-D (Féret et al., 2017) and SAIL (Verhoef, 1984) model and the Nano-Hyperspec sensor settings. The model parameters are listed in Appendix Table 1. The parameters are based on own field measurements, measurements of Schucknecht et al. (2020) and other analyses targeting RTM model inversion over grassland (see Appendix Table 1). In order to extract the soil spectra of the sites, we used a spectral angle mapper with soil spectra of the European wide soil spectral database, LUCAS Topsoil 2015 (Jones et al., 2020), as endmembers. Thereby, only spectra within a buffer of 0.25deg around each site were considered as endmembers. We calculated the dry and wet spectral reflectances as mean of the first and 99th percentile of all extracted soil spectra of the site, scaled it in regard to the wettest and driest SMC measured in the long-term time-series and used the resulting wettest and driest soil spectral value as input for the PROSAIL simulation.

2. Band selection The band selection is based on a sequential backward band removal algorithm. It removes the least important band at each iteration based on an assessment of the impact of the inputs on the prediction error in the context or absence of the other predictors (Verrelst et al., 2016). The band selection was employed for the psoil and LAI retrieval separately. Further details on the procedure can be found in Verrelst et al. (2016). Due to the large training data set we applied the faster Kernel Ridge Regression (KRR) (Cristianini et al., 2000) band selection tool (KRR-BAT) with 4% Gaussian noise. The algorithm was set to remove the 200 least important bands after the first iteration and three bands at each subsequent iteration. The validation at each iteration was performed using 70% of the training set as test set. The root mean square error (RMSE) served as accuracy metric. For psoil, we selected 13 bands, whereas for LAI nine bands were necessary. We applied the band selection on Fendt data only, assuming that the selected important bands for psoil and LAI are universal.

3. Active learning: To reduce the amount of samples, we applied the Euclidean distance-based diversity (EBD) algorithm in combination with the fast KRR. For details, see Verrelst et al. (2020, 2021). The initial training data set to start the algorithm comprised 150 samples. The remaining 99.5% of the complete training data set was used as pool data. Non-used pool data served as validation data set. With each iteration, the EBD selected three samples and stopped when there was no sample that further improved the model accuracy. Again, the RMSE served as accuracy metric.

4. VHGPR model: Using the reduced training data set, we trained a VHGPR algorithm which deals with heteroscedastic noise using a marginalized variational approximation. This improves its ability to provide realistic uncertainty estimates (Lázaro-Gredilla et al., 2014). For details on the kernel settings refer to Verrelst et al. (2021). In order to prevent the model from overfitting and allow for a better transfer on non-simulated data (Verrelst et al., 2012a; Brede et al., 2020), we added 15% Gaussian noise on the simulated data. The correlation coefficient (R²) and RMSE of the synthetic VHGPR model was accessed via a nested cross validation. The model was then applied on the measured spectral data and LAI and psoil values were extracted at locations of SMC measurements.

5. LAI threshold and final SMC retrieval: The retrieved psoil parameter can be transferred to SMC using a linear regression (Richter et al., 2012). However, an inversion of PROSAIL towards the psoil parameter is only meaningful when the soil reflectance significantly contributes to the canopy reflectance (Richter et al., 2012). By using the retrieved LAI, we iteratively reduced the samples via a maximum LAI threshold and applied a linear regression between the psoil and measured SMC using the remaining samples. The correlation coefficient (R²) and root mean square error (RMSE) of the different regressions were noted. Since some of the regressions only included seven samples, we derived the regression accuracy using all samples. We selected the optimal threshold and the corresponding intercept and slope parameters based on the best R² and RMSE combination.

Steps 1–4 were implemented within the Automated Radiative Transfer Models Operator (ARTMO) (Verrelst and Romijn, 2012b). The software provides a collection of tools for leaf and canopy RTM applications including forward simulation and hybrid inversion (Verrelst et al., 2019).

2.2.2. Data-driven approach

For the data-driven approach, we applied the selected bands by KRR-BAT and used the VHGPR algorithm for training. We extracted the respective bands at the location of each SMC measurement. As displayed in Fig. 2, the measured spectra were used as a predictor data set for the retrieval of SMC. To achieve this, we trained a VHGPR for each site independently. The accuracy (R² and RMSE) of the VHGPR was accessed via a tenfold cross-validation. For the interpretation of the impacting factors, we also extracted the LAI obtained by the hybrid approach at the location of the SMC samples.

3. Results

We present the results of the hybrid method in Section 3.1 and the corresponding results of the data-driven approach in Section 3.2. The results of all models are discussed with dependency on LAI. Section 3.3 describes the spatial effects of the applied models.

3.1. Performance of the hybrid approach

Fig. 3 displays the results of the retrieved psoil parameters in relation to measured SMC for the different sites and dates of image acquisition. The correlation coefficients between psoil and SMC are expected to be negative, corresponding to higher soil reflectances (i.e. high psoil) with drier soils (i.e. low SMC). The iterative application of LAI thresholds (see Appendix Fig. 6) identified a maximum LAI of 2.5 for Fendt and 0.75 for both, Marquardt and Grosses Bruch. For Fendt, the SMC and psoil were significantly negatively correlated with R² = 0.47 (Fig. 3a). In Fendt, the correlation coefficient was driven by the first image acquisition (R² = 0.55). For the second data acquisition (FE 2019–06-27) no clear correlation between psoil and the SMC was found (R² = 0.00). It is striking that SMC samples with low LAI values from Fendt 2019–06-27 were underestimated, leading to the high RMSE of 16.36Vol_%.

Fig. 3.

Psoil - SMC relationship for a) Fendt,b) Grosses Bruch and c) Marquardt.

The correlation between psoil and SMC in Marquardt was significant but only moderate together with small RMSE (value range: 5.9Vol_%). When extrapolating the regression line for Marquardt towards psoils of 0 it becomes apparent that the relation found will not allow a retrieval of SMC higher than approximately 10Vol_%.

At Grosses Bruch the weak negative correlation was not consistent with decreasing LAI thresholds.

3.2. Performance of the data-driven approach

Compared to the results presented in Section 3.1, the RMSE and correlation coefficients between the predicted and measured SMC reveal superior model fits for the Fendt and Marquardt data overall (Fig. 4). For Grosses Bruch the training of an accurate model again failed.

Fig. 4.

Predicted and observed SMC for a) Fendt (FE), b) Grosses Bruch (GB) and c) Marquardt (MAR).

A closer inspection of the scatter formation of prediction results for Fendt (Fig. 4a) revealed that the LAI as plant trait plays a major role in the SMC retrieval. The predicted SMC formed three distinct clusters along the 1:1 line, with significantly different LAI values. Thereby, lowest LAI values were associated with highest SMC, intermediate LAI with driest SMC and intermediate SMC with highest LAI values. Within these clusters the performance of the algorithm was weak.

The impact of LAI was also evident for the Marquardt site. Here the predicted SMC and LAI values were significantly correlated. The R²= 0.72 showed a superior fit of predicted SMC and LAI than predicted and measured SMC (R²= 0.4).

In Marquardt we also observed the impact of irrigation on the model fit. The model performed best for SMC lower than 10Vol_%. This includes only data of areas where no irrigation took place at least one week before the overflight. The model performed worst for samples of MAR 2020-08-06 (RMSE = 15.39Vol_%), when the area was irrigated on the same day of data acquisition. When the data was acquired at least one day after irrigation (MAR 2020-08-07, MAR 2020-08-12, MAR 2020-08-17) the model accuracy improved to RMSE less than 8 Vol_% (see Fig. 4).

3.3. Spatial application of the models

Lastly, the final SMC-VHGPR models were applied to the images over the three areas (see Fig. 5). By means of these established maps, we can identify the factors impacting the models of both approaches. Considering the LAI threshold identified for the psoil-SMC relation (Section 3.1), only areas with LAI < 2.5 in Fendt and LAI < 0.75 in Marquardt and Grosses Bruch were correctly predicted with the hybrid-approach.

Fig. 5.

Spatial application of LAI, psoil, SMC hybrid, SMC data-driven and an interpolated SMC for Fendt (a-j), Grosses Bruch (k-o) and Marquardt (p-z). An inverse distance-based weighted interpolation with an RMSE - optimized power was applied within gstat of Pebesma (2004).

For Fendt 27 June 2019 was correctly predicted to be dryer than 04 June 2019 by both algorithms. Also the spatial patterns closely matched the interpolated SMC. But we clearly note the impact of overestimation of lower LAI values. This was already indicated in Fig. 3 for the hybrid approach. Also within the applications of the data-driven approach the freshly mown fields in the southern and northern part of the study area were predicted to be more than 20Vol_% more moist than the remaining area.

For Grosses Bruch, we found no meaningful models neither for the transfer of psoil to SMC nor using a data-driven approach only. As these models are not robust enough to capture the SMC variation, the distribution pattern of LAI is again dominant.

In Marquardt, LAI values of the non-irrigated northern-most area decreased markedly due to ongoing drought. SMC also decreased. This trend was correctly covered by the data-driven and, although less pronounced, by the hybrid approach. The wettest areas were only covered by the data-driven approach. In MAR 2020–08-06 the location of the gradient from dry to moist SMC matched closely to the interpolated area. For MAR 2020–08-12 instead, the same strong gradient within the southern area was less pronounced within the data-driven prediction. The high SMC values measured (>35Vol_%) were not covered by the model. Only in a small patch at the lower south-western corner of the field did the estimated SMC reach approximately 35Vol_%.

4. Discussion

We applied a hybrid and a data-driven approach to retrieve SMC at different sites and dates. Overall, for Fendt and Marquardt we found encouraging results using the data-driven approach (see Section 3.2) and moderate correlations by applying the hybrid approach (see Section 3.1). However, well-performing models for Grosses Bruch were lacking. In the following discussion we want to focus on the factors impacting the performance of these models. In Section 4.1 we elaborate on limiting factors of the hybrid approach. In Section 4.2 we discuss the impacting factors of the data-driven approach. Section 4.3 opens the discussion towards future methods to tackle the occurring challenges.

4.1. Impacting factors of the hybrid approach

One of the major benefits of the hybrid approach is the generality of the learned relations. Nevertheless, site specific parameters such as soil reflectance and SMC measurements are still necessary. The main bottleneck is its applicability only for areas where the soil reflectance significantly contributes to the canopy reflectance. We empirically detected an LAI threshold of 2.5 for Fendt and 0.75 for Grosses Bruch and Marquardt. Lower LAI thresholds for GB and MAR are also suggested by a simulation of a single spectrum with varying levels of LAI and psoil (see Appendix Fig. 7). Richter et al. (2012) underlined the dependency of the LAI threshold on plant canopy structure. They identified crop-type-dependent LAI thresholds. For the planophile leaf orientation of maize and potatoes, a LAI threshold of 2.8 was suggested, which equals the chosen threshold at Fendt. For Marquardt, in turn, the LAI threshold of 0.75 indicates an applicability of the hybrid approach only over sparsely vegetated areas. Higher thresholds of LAI apply for longer wavelengths (see Appendix Fig. 7), especially for SWIR (Richter et al., 2012), where the most important water absorption bands are located (Eon and Bachmann, 2021).

The spectral range of our sensor is limited to the VIS–NIR, thus lacking the most important water absorption bands in the SWIR. This may explain the weak performance of the hybrid approach for the Marquardt data set. There is a water absorption feature at 970 nm, which lies within the spectral range of the Nano-Hyperspec sensor. However, this wavelength is at the boundary of the wavelengths that can be detected by the sensor and is therefore increasingly affected by noise. Supporting this, Eon and Bachmann (2021) found lower accuracy for a soil reflectance model inversion (MARMIT) targeting SMC when using the VIS–NIR range water absorption features compared to the SWIR range. Using sensors covering the SWIR range could thus improve our results. However, the study by Wocher et al. (2018) successfully explored the water absorption depth at 970nm for crop water content retrieval with a physically-based method. Although the authors targeted vegetation water content, their findings are promising in view of sensor availability with spectral range limited to VIS–NIR.

Our analysis indicated a superior potential of the hybrid approach to retrieve SMC over non-permanent grassland as opposed to areas with permanent grassland use. The spatial application of the hybrid approach in Marquardt revealed the existing drying of the soil from 6 August 2020 to 12 August 2020. In Fendt, the spatial distribution of SMC followed the patterns of LAI. The psoil retrieval thus rather corresponds to retrieval of LAI, which dominates the reflectance within the VIS–NIR range (Berger et al., 2018). In contrast to the Fendt and Grosses Bruch sites, Marquardt is not permanently used as grassland. Hence, neither a thatch nor a litter layer has been developed and, consequently, the bare soil was visible at areas with low LAI. Within Fendt, areas of low LAI were still covered by the thatch layer and litter. This limits the applicability of LAI as a threshold to define the regression. Additionally, it impedes the hybrid approach for permanent grassland areas when using the psoil parameter as target variable.

The moderate but meaningful results for Marquardt, where the sparse plant distribution exposes the soil, have shown the potential of the hybrid approach especially for agricultural crops. Higher correlations were found by Richter et al. (2012). They stress the strong impact of the canopy structure, especially leaf angle distribution, on the correct retrieval of psoil. For mixed grassland the complex canopy structure has been shown to hamper biochemical plant trait retrieval (Berger et al., 2018; Darvishzadeh et al., 2008). This could also explain the general lack of agreement among our models. For crops with a distinct row structure, such as maize, a higher share of bare soil contributes to the measured signal, bearing a high potential for this retrieval approach. Nevertheless, the site specific soil reflectance shows the need for further analysis to apply the approach on sites with heterogeneous soil types.

4.2. Impacting factors of the data-driven approach

Despite the overall high accuracy of the data-driven models for Fendt and Marquardt, we again notice a strong dependency on canopy structure, especially LAI. In comparison to the hybrid approach, which requires open canopy structure for the bare soil signal, the data-driven approach depends on the relationship between canopy characteristics and SMC. This relationship is impacted by mowing, grazing, and the timelag of vegetation response to irrigation.

For mowing, the LAI decreases sharply, independent of changes in the SMC. During data acquisition in Fendt on 04 June 2019, the mown areas corresponded to moist sites, leading to an inverse relationship between SMC and LAI distribution (Fig. 4). The reduction of LAI can reduce the water demand of the plants, leading to higher SMC in mown areas (Gross et al., 2008). However, the relationship of SMC and LAI is not transferable to other dates, as shown by the moist-prediction of mown areas in Fendt 2019–06-27. Nevertheless, Nakano et al. (2008) found a dependency of grassland regrowth on SMC. To enable a transfer in time of this relationship, our results reveal the need for an extension of the dataset to cover all possible combinations of mowing-regrowth states at multiple SMC conditions..

Mowing also alters other SMC relevant plant traits. The third cluster of Fendt 2019–06-27 does not follow the negative correlation between SMC and LAI within Fendt 2019–06-04. This leads to the assumption that the model finds different SMC relevant plant traits such as photosynthetic rate or chlorophyll content (Nakano et al., 2008; Sarker et al., 1999). However, mowing also has an impact on these plant traits: It increases the amount of scenescent material (Benot et al., 2014; Opdekamp et al., 2012; Li et al., 2019), the photosynthetic rate and chlorophyll content (Li et al., 2019) and decreases plant growth (Benot et al., 2014; Opdekamp et al., 2012; Li et al., 2019). The altering effect of mowing on plant traits explains the low model accuracy within the three clusters of the data-driven model at Fendt.

For grazing, Dusseux et al. (2014) describe lower but more frequent changes in temporal LAI dynamics compared to mowing. Spatially, we observed a high heterogeneity of LAI at small spatial scale due to selective grazing. Similar to mowing, studies prove that grazing strongly alters biomass, plant traits (James M. et al., 2001) and dominant factors controlling plant responses such as light and nutrient availability (Burke et al., 1998). Further complicating a SMC retrieval, the range of the SMC values within the site is minimal with a standard deviation of 2.73Vol_ %. Therefore, the difficulty in mapping SMC in Grosses Bruch using a data-driven approach results from an interplay of uncertainties within ground reference data and spectral data and is magnified by the high impact of grazing on plant traits.

At irrigated sites, LAI has a strong positive relationship with SMC but the timelag between LAI increase and SMC change impacts the model accuracy. The limiting factor of SMC on biomass has been reported in many analysis (Deng et al., 2016; Sherry, 2008; Chu et al., 2019; Nakano et al., 2008). In our case the Marquardt model performs accurately in the northern parts of the area, which were not irrigated, indicating a fast response of LAI to ongoing drought. However, peaks in the SMC distribution are underestimated.

Several studies found significant species (Peng et al., 2014; Zhao et al., 2021), site (Yue et al., 2022; Liu et al., 2016; Zhao et al., 2021) and season (Yue et al., 2022) dependent timelags between changes in SMC and plant traits that are detectable in the VIS–NIR range. These timelags range from 0 to 48 days (Sur et al., 2020; Chu et al., 2019; Liu et al., 2016; Zhao et al., 2021; Peng et al., 2014). Moreover, the response of plants is a reaction to an accumulative effect of climate and SMC conditions (Liu et al., 2020b). Such short-term applications of excessive water as performed in the irrigation experiment in Marquardt are thus not reflected by the plants response and more time is necessary between SMC sampling and image acquisition. This also explains the improved model fit for MAR 2020–08-17 when the system stabilized one week after the irrigation. By then, the SMC distribution reached intermediate values and the LAI distribution adapted to the present SMC at the site.

Apart from mowing, grazing and irrigation, the water limitation of the study site within the gradient of temperature or precipitation driven plant growth also impactss the accuracy of a model. We found only weak model performance of predicted SMC within the clusters at the Fendt site. Fendt is a moist site. Within the week before both data acquisitions in Fendt, rainfalls occurred. Plant traits indicating changes in water availability are expected to be less pronounced within the site. Burke et al. (1998) noted a lower response of temperate grasslands to SMC with decreasing importance of water supply to the plants. This is further supported by higher correlations of plant greenness (Tian et al., 2019) or gross primary production (He et al., 2017) with SMC in water-limited regions. Similar to Lendzioch et al. (2021), we found better model performance when retrieving SMC during the dry season. In Marquardt, the ongoing drought causes a strong response of plants to changes in SMC. Here, the model performance is high within the non-irrigated sites.

4.3. Outlook

UAS-based SMC retrieval is considered a potential tool for the validation of large-scale SMC products. Some studies even state UAS data as alternative to field sampling, although merely for classification tasks (Kattenborn et al., 2019b). However, major challenges exist. Besides the difficulty of creating dense time series due to cloud coverage and the high practical and administrative challenges, the accuracy of the products differs with site and hydrological condition. We found additional challenges when targeting differently managed grassland sites.

In this analysis we applied a data-driven and a hybrid SMC retrieval at different sites using hyperspectral data in the VIS–NIR range. Despite the overall good results, we found a strong dependency of the VHGPR models on LAI. In order to disentangle the relationship between SMC and LAI for capturing short-term SMC changes, additional predictor data sets can improve the data-driven models. The reaction of plants is mainly driven by the water available at the root zone. Several studies have shown higher correlations of bands of the VIS–NIR range with SMC at deeper soil layers (Babaeian et al., 2018; Babaeian et al., 2021). Additionally, soil texture is the major control of plant available water content. Information on field capacity (Hassan-Esfahani et al., 2017), wilting point (Liu et al., 2012) or other soil physical and hydrological parameters (Babaeian et al., 2021) are part of the predictor data set in other successful retrieval studies. However, both of these data sources require high effort and invasive sampling, when using standard point data.

Predictors that are easier to assemble are TIR, LiDAR and meteorological data. Cheng et al. (2022) found strong improvements when adding TIR data to the models. TIR data is suitable to represent the fast changing thermodynamic characteristics of the soil and the plant canopy associated to different SMC (Wang et al., 2018; Paruta et al., 2021). LiDAR data provide information about small-scale surface topography as well as plant canopy structure. Both predictors are relevant parameters of SMC transport and distribution (Rosenbaum et al., 2012; Korres et al., 2010; Luo et al., 2019). Multiple studies also demonstrated improved modeling using metereological information (Araya et al., 2020; Hassan-Esfahani et al., 2015; Li et al., 2022).

Despite the potential to improve the data-driven approaches and despite its good performance at Fendt and Marquardt, a major concern remains: the generality of established models. Our results in Fendt point to the missing generality of SMC estimation for mown areas outside of the SoilNet area. Especially for agricultural sites with changing crops this implies the need for the creation of a training data set for each site and each crop during the whole vegetation period. Hassan-Esfahani et al. (2017) showed high accuracy for such an approach for one test site. Alternatively, a joint effort on fusing the gathered SMC and UAS data collected by different research groups to one super-data-set might lead to a more robust data-driven model that is applicable on different sites and times. Such an approach can lead to a significant decrease of sampling effort and thus enable an applicability for the real world.

Another way to reduce the sampling effort but maintaining generality is the hybrid approach. In this analysis we only assessed the psoil factor describing brightness, thus surface SMC within the RTM PROSAIL. Our analysis reveals the dependency of the hybrid approach on site-specific LAI thresholds. Additionally, the retrieved psoil factor is not per se SMC. When targeting SMC, empirical soil specific conversion functions are necessary. This again hampers the stated generality of the hybrid approach. However, the latter can be improved by enhanced soil reflectance models such as MARMIT (Bablet et al., 2018) or the BSM (Verhoef et al., 2017) model implemented within SCOPE. These models allow a direct retrieval of SMC.

So far, only areas below a certain maximum LAI can be mapped using leaf-canopy RTMs such as PROSAIL. The highly site specific LAI thresholds question the generality of the hybrid approach. At Fendt, we found a strong correlation between the retrieved psoil and LAI, which indicates the dominance of plant traits on measured reflectance in vegetated areas. Instead of using the less dominant soil reflectance, a coupling of existing RTMs with plant hydrological models makes the hybrid approach applicable also for densely vegetated areas by using the physically-described interaction of the dominant plant traits with root zone SMC. SMC relevant plant traits such as plant water or chlorophyll content have been successfully derived using RTM inversion and VIS–NIR and SWIR hyperspectral data for a variety of crops (Lei et al., 2022; Xie et al., 2019; Zhao and Qin, 2019; Zhang et al., 2018). Within SCOPE (Yang et al., 2021), a first coupling of RTM and SVAT (soil–vegetation–atmosphere) models has been implemented. However, the link between RTM-relevant plant traits and the SMC at the root zone is still missing.

5. Conclusion

In this study we tested a data-driven (VHGPR) and a hybrid (PROSAIL & VHGPR) approach for SMC retrieval at three different sites and dates. The selected grassland sites differed in hydrologic conditions and management. By examining these data sets, we aimed to identify factors that impede or favor successful SMC retrieval using these two conceptually distinct modeling approaches.

All data-driven models are dependent on LAI, with a negative correlation between SMC and LAI in Fendt and a positive correlation in Marquardt. Thereby, the found relationships are not generalizable due to several reasons: In Fendt and Grosses Bruch, mowing and grazing alters the plant traits relevant to SMC and drives LAI distribution. In Marquardt, the long timelag of plant response to irrigation is hampers an accurate retrieval within a short time scale. We found good model performance at non-irrigated areas and for periods when SMC stabilized. SMC retrieval using data-driven approaches shows best performance during water-limited periods without impeding factors such as excessive irrigation.

In contrast to the data-driven approach, the hybrid SMC retrieval was only applicable to areas with LAI < 2.5 (Fendt) and < 0.75 (Marquardt, Grosses Bruch). We identified the thatch layer as major impeding factor of the hybrid approach in permanent grassland. For non-permanent grassland the approach shows potential for further enhancement. For both the data-driven and hybrid approach, using the SWIR range and its water absorption features can lead to improved results.

Our findings add to the rapidly growing field of SMC remote sensing with UAS systems an understanding of the relevant factors that favor and limit successful SMC retrieval. We identify the time lag between changes in SMC and crop response, as well as the difficulty of robust model generation at moist and intensively managed sites, as challenges to be addressed in future model development. In this regard we point to the need for a joint effort in UAS and SMC data collection covering a multitude of land-cover types, management practices and hydrological conditions. Such over-arching data sets have the potential to unravel the complex interactions between SMC and plant characteristics and thus ensure data-driven models that are transferable in time and space.

We also made one first step towards a physically-based SMC retrieval. Future enhancements of the RTM can open new possibilities in retrieving SMC over densely vegetated areas and additionally improve the understanding of plant-soil-atmosphere interactions. The combination of data-driven, hybrid and semi-physical approaches within an ensemble learning can finally join their specific advantages and improve general model accuracy. This analysis calls out for the challenging development of spatially and temporally robust SMC retrieval models in order to provide essential information for global and local SMC monitoring, irrigation management, and environmental protection.

Fig. 6.

RMSE and correlation coefficient R of linear regressions for Psoil and smc applying different LAI thresholds for (a) FE, (b) FE 2019–06-04 (c) FE 2019–06-27, (d) GB, (e) MAR 2020–08-06, (f) MAR 2020–08-07, (g) MAR 2020–08-12, (h) MAR 2020–08-17.

Fig. 7.

Interactions between LAI and Psoil in Prosail and related thresholds for soil spectra based on (a) Fendt (FE), (b) Grosses Bruch (GB), (c) Marquardt (MAR).

Fig. 8.

Correlation coefficients for the selected Bands with a) the interpolated LAI and b) the interpolated SMC at Fendt and Marquardt.

Table 1. Parameter ranges of the corresponding models, their unit and data or literature source.

Par	Parameter	Min	Max	Source
LAI	Total Leaf Area Index [m2 mminus2]	0	7	Own measurement
angle	Leaf angle distribution [deg]	20	75	Darvishzadeh 2008
skyl	Diffuse/Direct light	23	23	Berger et al. 2018
hspot	Hot spot	0	1	0.5/LAI (max = 1)
vCover	Vegetation Cover	0	1	site experience
psoil	Soil Coefficient	0	1
tts	Solar Zenith Angle [deg]	25	50	Aphalo (2015)
tto	View zenith Angle [deg]	0	0	Nadir view angle
psi	Relative Azimuth Angle [deg]	0.29	57.38	Aphalo (2015)
N	Leaf Structural Parameter	1.55	1.55	Darvishzadeh et al. 2008
Cab	Chlorophyll AB content (ug.cm-2)	0	60	Own measurement, conversion after Si et al. (2012)
Car	Carotenoids (ug. cmminus2)	0	15	Cab/4
Cbrown	Brown pigments (g. cmminus2)	0.1125	0.1125	Katterborn et al. 2018
Cs	Scenescent material	0	0.8
Anth	Anthocyanin content (ug.cmminus2)	0	5	Hallik et al. 2017
Cw	Equivalent water thickness (g. cmminus2)	0.0	0.08	Schucknecht et al. 2020
Cm	Dry matter content (g.cmminus2)	0.0029	0.0173	Cw/e with e in [3.2,4]; Kattenborn et al. (2019a)