Water quality predictions through linear regression - A brute force algorithm approach

Abstract

Linear regression is one of the oldest statistical modeling approaches. Still, it is a valuable tool, particularly when it is necessary to create forecast models with low sample sizes. When researchers use this method and have numerous potential regressors, choosing the group of regressors for a model that fulfills all the required assumptions can be challenging. In this sense, the authors developed an open-source Python script that automatically tests all the combinations of regressors under a brute-force approach. The output displays the best linear regression models, regarding the thresholds set by users for the required assumptions: statistical significance of the estimations, multicollinearity, error normality, and homoscedasticity. Further, the script allows the selection of linear regressions with regression coefficients according to the user’s expectations. This script was tested with an environmental dataset to predict surface water quality parameters based on landscape metrics and contaminant loads. Among millions of possible combinations, less than 0.1 % of the regressor combinations fulfilled the requirements. The resulting combinations were also tested in geographically weighted regression, with similar results to linear regression. The model's performance was higher for pH and total nitrate and lower for total alkalinity and electrical conductivity.

• •A Python script was developed to find the best linear regressions within a dataset.
• •Output regressions are automatically selected based on regression coefficient expectations set by the user and the linear regression assumptions.
• •The algorithm was successfully validated through an environmental dataset.

Graphical abstract

Specifications table

Subject area:	Environmental Science
More specific subject area:	Water Quality
Name of your method:	Automatic selection of robust linear regression models
Name and reference of original method:	Stanton, J.M., 2001. Galton, Pearson, and the Peas: A Brief History of Linear Regression for Statistics Instructors. J. Stat. Educ. 9. 10.1080/10691898.2001.11910537
	Gang Su, X., 2009. Linear regression analysis: Theory and computing. World Scientific Publishing Co. Pte. Ltd. 10.1142/6986
Resource availability:	Software:
	-Python (https://www.Python.org/downloads/ or https://www.anaconda.com).
	Software for dataset preparation:
	-Microsoft Excel (https://www.microsoft.com/pt-pt/microsoft-365/excel);
	-ArcGIS desktop (https://www.esri.com/);
	-ZonalMetrics-Toolbox (https://www.arcgis.com/home/item.html?id=96c3ffc7439f4972a08f4edbc51d89be).
	Data:
	-Water quality (https://snirh.apambiente.pt/);
	-Land use map (https://dados.gov.pt/pt/datasets/carta-de-uso-e-ocupacao-do-solo-2015/);
	-Pollution sources (https://sniamb.apambiente.pt/content/geo-visualizador);

Method details

Introduction

To formally address water quality concerns, it is necessary to resort to models that expose the problem or test possible solutions [1,2]. Typically, models are classified as either physical or statistical [3]. In physical models, the interactions are already determined by biological, physical, and chemical processes, so they are widely used for prediction purposes [4]. There are a plethora of possible water quality models such as Water Quality Analysis Simulation Program (WASP) [5], and the Hydrological Simulation Program-FORTRAN (HSPF) [6], but the most used in research is Soil and Water Assessment Tool (SWAT) [7]. Statistical models have the advantage that can be used to identify cause-effect interactions revealed by the dataset [8], but can also be used for prediction purposes [9]. Different statistical methods can be used for prediction, such as regression to classification analysis [10], or even more complex methodologies based on machine learning [11]. Linear regression is widely used in different science fields [12], [13], [14], [15], with the advantage of being easily implemented, interpreted, and also because the concerns of meeting the required assumptions have been studied for a long time [16,17]. When it is intended to create a linear regression model, the first step is to prepare a dataset with candidate variables. Then, different combinations of independent variables are tested in order to predict the dependent variable. In addition, statistical tests must be performed to check if the linear regression assumptions are met for each tested combination of candidate variables. This process can be done under a trial-error approach but is not an efficient option, particularly when there are many candidate variables. Multiple regression tools ease this process by calculating the resulting linear regression for each possible regressor combinations. Some tools can be used for that purpose in software, such as ArcMap [18], Microsoft Excel toolboxes [19] or JMP [20]. Though these tools are helpful, they do not provide enough statistical tests to confirm if the linear regression assumptions are met [21]. Stepwise regression is commonly used when datasets have many variables [22], by adding and removing variables based on their statistical significance through an iterative process that does not test all possible combinations [22]. Besides this process reaches models whose regression coefficients are statistically significant and also with high determination coefficients [23], it is also necessary to confirm if the linear regression assumptions are met. The resulting outputs of stepwise algorithms are reasonably fast but do not guarantee that the output selects the best model [24]. To find the best model within a dataset, the possible combinations must be tested, which can be done through brute force. Besides brute force feasibility is relied on computing speed and dataset size [25], brute force guarantees that the best model is found within the dataset, since it tests all the possible combinations [26].

With the advent of geographic information systems (GIS), a spatial linear regression method was developed, called Geographically Weighted Regression (GWR) [27]. Instead of having fixed regression coefficients, these change along in space, which can achieve better predictions for spatial data than ordinary least-squares (OLS) linear regression [28]. To correctly perform a GWR model, it is recommended first to test the OLS linear regression, and if the assumptions are met [29]. Since GWR is a spatial analysis, it is also necessary to test if the residuals distribution is not spatially autocorrelated [30]. Then, the used predictors in OLS model can be used to build a GWR [31,32].

The main purpose of the present study is to develop a Python script to test all the possible combinations of variables to predict a selected variable, regarding the linear regression assumptions resorting to an automatic algorithm using open-source software. The second objective is to test the script to an environmental dataset and predict the surface water quality in all sub-catchments. The study was conducted in a Portuguese River Basin (Fig. 1). The Ave River Basin (ARB), located in the northwest region of Portugal, has poor water quality, which has been a recurring concern for the past decades [33], [34], [35]. The authors' intention for the method application is to build robust linear regression models to predict water quality parameters in each catchment. Among different surface water parameters (SWP), one model was chosen for each parameter. Then, the chosen regressor combination was applied in GWR, to test if this achieves better predictions than linear regression [36,37].

Fig. 1.

Study area - (A) Portugal, (B) portrayed Ave River Basin, and (C) Ave River Basin land uses.

Methodology

The authors developed a script to find linear ordinary least squares models within a dataset, that fulfill the linear regression assumptions, resorting to open source coding language Python. The developed script tests if these assumptions are met, and also, it has a particular feature which is the selection of models based on the expected signs of regression coefficients. Models whose regression coefficients do not meet the operator's expectations are excluded from the output. For environmental purposes, this particularity is crucial to select models. However, the script can be applied to any science field.

The user defines statistical thresholds, such as the maximum variance inflation factor and probability of rejecting the null hypothesis for multiple tests. According to these input parameters, all the models that meet the assumptions are reported. Still, it is up to the user to choose one model, among all the ones reported in the output.

This algorithm was applied to a water quality case study to find a group of regressors that can predict surface water parameters in Ave River Basin (ARB) sub-catchments and then apply a Geographically Weighted Regression (GWR) using the same predictors. The methodological workflow (Fig. 2) is subdivided into three steps: input preparation, operation of the Python script, and the application of the selected linear regression for the case study.

Fig. 2.

Methodological workflow. GWR: Geographically Weighted Regression.

Dataset preparation

To run the Python script, it is required to create three input tables in the comma-separated values (CSV) format. The first table contains the dataset to create the linear regression models, including the predicted variable and possible predictors. The second table contains samples to calculate predictions for each output model, columns are the regression variables and the lines are the samples to predict. The final table contains the expected outcome of each candidate variable. With this table, the algorithm only selects linear regressions whose regressors have a sign according to the established expectations.

An environmental dataset was used to test this algorithm by predicting surface water quality parameters (SWP) based on pollution indicators: landscape metrics, point source pressures, diffuse emissions, and population density. To prepare the dataset, it was necessary to resort to multiple databases. Measurements of surface water parameters for the selected river basin, ARB, were downloaded from the National Hydrological Information System (the Portuguese abbreviation is SNIRH) [38], including the average concentration of the SWP and also the location of the sampling sites. The digital elevation model was downloaded from the European Environmental Agency [39], and was used to delineate the watershed, drainage lines, drainage areas of each sampling site, and sub-catchments (presented in Fig. 1). These procedures were performed in ArcMap [40] using ArcHYDRO functionalities [41]. The land use map of 2015 was downloaded from the Portuguese Territory Planning website [42], to calculate landscape metrics (LSM) for each drainage area and sub-catchments by using a Python toolbox developed by other authors [43]. In total, nine metrics were calculated and are described in Table 1. These metrics were calculated for the generic land use classes, agriculture (AGR), artificial surfaces (ART), forested areas (FOR), water bodies (WAB), and two metrics, Shannon's diversity index (shdi) and edge length of land use (ed) were calculated for all land use classes simultaneously (ALL). The population density data was downloaded for each census subsection, from the National Statistics Institute website [44]. The Portuguese Environmental Agency (APA) provided the point source and diffuse pressures datasets, which can be accessed in the National Environmental Information System [45]. For diffuse emissions, the dataset contains the total loads of nitrogen and phosphorous (N and P) from livestock, agriculture, and forest, in kg/year/km². For point source pressures, the dataset contains the total loads of biological and chemical oxygen demands (BOD and COD), nitrogen, and phosphorous (N and P) from all effluent discharge sites in kg/year.

Table 1.

Calculated landscape metrics. AGR- agriculture, ART- artificial surfaces, FOR- forested areas, and WAB- Water bodies.

A more detailed description of all regressors is available in a CSV file in the supplementary material, “Variables_with_signs.csv”. In total, it was collected 73 possible predictors and it was applied four transformations (x²,x^0.5, log₁₀(x+1), and 1/(1+x)), so the number of “variables” increased to 365. Since this is a high number of predictors to test in a brute force algorithm, only variables whose Pearson correlation with the predicted variable that had an expected sign was selected. For each variable, it was manually chosen the transformation that resulted in the highest Pearson correlation coefficient with the dependent variable in order to reduce the number of input variables. Further, for point source and diffuse emissions, one variable was chosen for each since these are strongly correlated, even with transformations. For each SWP, a CSV file was created, containing the selected predictors and the selected SWP (the predicted variable). These CSV files are presented in the supplementary material, named according to the specific SWP: “Electrical_conductivity”, “pH”, “Total_alkalinity” and ” Total_nitrate”.

As previously mentioned, the algorithm only exports models in which the regression coefficient signs are according to the user's expectation. For the case study application, the file “Variables_with_signs.csv”, presented in the supplementary material, contains this information. In the column “effect”, for each variable it must be set one of three values: 1, -1, or 0. For variables that is expected a positive regression coefficient, the selected “effect” should be 1. For variables in which is expected a negative regression coefficient, the selected “effect” should be -1. Also, if users do not expect any effect for a variable, the attributed “effect” should be 0. In this sense, the algorithm exports models in which the regression coefficient of the respective variable is positive or negative.

The analysis of the expected signs was made according to the expected effect (i.e., point source pressures do not increase water quality (WQ). For variables that theoretically increase contamination, it was selected an expected positive sign for the respective regression coefficient. For metrics related to forest, for example, pz_(FOR), percentage of forested areas, it was attributed a negative sign since forests positively increase WQ and decrease the SWP. But for some variables, in which there is no expectation, it was not attributed an expected effect. For example, for lpi_(ART), since it is not known if having most of the urban areas aggregated or dispersed improves or decreases water quality, for this reason, it was attributed an expected effect of 0.

The table with the entire dataset that was used to build the linear regression models is presented in “All_data_for_models.csv”, but a CSV file is available for each SWP containing all the used variables to build the respective models. The file named “Predictions_input.csv” contains all variables for each catchment and drainage area, which is a file that allows the prediction of the SWP in each catchment. These table files are inputs for the Python script, described in the following sections.

Python script

Linear regression

Linear regression (LR) is a statistical approach to modeling the linear relation between the dependent variable with one or more regressors. For the present approach, it was used the ordinary least squares method, in which the linear regression is calculated to have the least possible sum of squares of the difference between the measured and predicted values [46]. LR is one of the oldest statistical modeling approaches that has been widely used since around two hundred years ago [47,48]. The main purpose is to use this method to forecast/predict the response variable in cases where is easier to measure the independent variables rather than the dependent ones [49]. For example, estimating a tree's biomass is easier by measuring its height and diameter rather than weight [50].

Eq. (1) shows the linear regression model with m predictors. In this equation, y represents the dependent variable, x_j is the independent variable (regressor), β₀ is the constant/intercept, β_j is the regression coefficient associated with each regressor, and ε is the random error term.

$$y={\beta }_0+\sum _{j=1}^m\left({\beta }_j{x}_j\right)+\epsilon$$ (1)

When a linear regression model is created, multiple statistical criteria must be analyzed to assess the model's suitability for prediction [51]. The coefficient of determination (R²) is the proportion of the explained variation of the endogenous variable by the exogenous variables [52]. A high R² does not directly mean that the regression is suitable for predictions because the statistical significance of the R² is dependent on the degrees of freedom. As more explanatory variables are added to a model, the R² tends to increase, but the freedom degrees decrease [53]. In this sense, the adjusted R² is a coefficient that brings the same information as the R², but it is adjusted based on the degrees of freedom [54]. Another way to access the overall significance of a regression is through the F statistics. If the calculated F value is higher than the critical F for the selected confidence interval (normally 95%), the null hypothesis is rejected, which states that none of the regressors is useful for explaining the dependent variable [55].

When is necessary to create a regression model, the first question that might emerge is, “which explanatory variables should be used?”. By trying to find a suitable model, it is recurrent to test different combinations of exogenous variables through a trial-error method. Among the tested variables, some might not support the prediction of the exogenous variable [56]. This can be tested for each regression coefficient through t statistics (coefficient divided by its standard error) [57]. If the p-value is below the confidence interval, is rejected the null hypothesis, which states that the regression coefficient is equal to 0 [58]. In multiple linear regression is recommended to include regressors that are not linearly dependent (correlated) with each other [59]. When one or more regressors are strongly correlated, the variance of the coefficient estimates and predictions becomes sensitive to minor changes in the model, making the regression model unstable [60]. To measure the degree of multicollinearity is commonly calculated the regressors variance inflation factor (VIF) [61]. This factor ranges from 1 to infinite (when there is a perfect correlation between regressors) and must be lower as possible. Still, there are different recommendations for the maximum acceptable VIF, 10 [62], 7.5 [63], or 5 [64].

There are three assumptions for the error terms in regression models: constant variance, normal distribution, and independence [58]. Different statistical tests can be performed to confirm whether a regression model meets these assumptions. For those tests, the null hypothesis states that the assumption is met. Consequently, as lower is the p-value, higher becomes the probability of disregarding the assumption. The constant variance of the error terms, called homoscedasticity, means that along with the regression, the error term variance is equal for low and high predicted values [65]. If this assumption is not met, it means that the regression error terms are heteroscedastic. Consequently, observations with larger errors will deviate from the fitted model, so predictions become biased [66]. Residuals should follow a normal distribution because the calculation of confidence interval and consequent variable significance is based on normal distributions [67]. In linear regression is expected that successive values of the error are independent along time [68]. If this assumption is not met, the error term is autocorrelated, so the estimates’ standard error is affected [69]. The presence of autocorrelation is mandatory to be verified when samples represent different time-steps [70]. However, another type of autocorrelation must be verified when samples represent different locations [71]. For regressions in which samples represent points in space, it is expected that error terms are not spatially clustered but dispersed or random. If the error term is clustered, the data presents spatial autocorrelation, deviating estimations in space [72]. So, to evaluate the spatial distribution of the error term is used Moran's I test [73].

Algorithm workflow

The designed Python script (available in the supplementary material) calculates all the possible linear regressions that fulfill the statistical assumptions for any dataset, according to the workflow presented in Fig. 3. Before using the algorithm is necessary to:

• 1.
  Create 3 CSV files, with the user's data. This can be tested with the files presented in the supplementary material for the case study application.

  • a.
    CSV file with the dataset to create the linear regression models, including the dependent variable and all the possible regressors- “DATASET”.
  • b.
    CSV file with the list of independent variables and the expected effect- “signs_table”.
  • c.
    CSV file with a dataset to calculate the predictions of each output model –“Data_for_predictions”.
• 2.
  Edit the script.

  • a.
    Set the data paths for the input csv files.
  • b.
    Select the dependent variable-“ Chosen_Y”
  • c.
    Set the minimum and maximum number of regressors for the output regression models- respectively, “nor_min” and ” nor_max”
  • d.
    Set if the intercept should be equal to 0 or defined by the regression model- “intercept_value”
  • e.
    Define the statistical thresholds.

Fig. 3.

Algorithm workflow.

The first inputs are the data paths, where the user defines the path of the three necessary CSV files. The first must contain the dataset path, used to create the linear regression models, defined in the script as “DATASET”. The data file should be in the CSV format since this is converted into a data frame using the “pandas.read_csv” command. Furthermore, is necessary to set the index of the data table, which for the present study is “Sampling_site”. The table with the expected signs is defined as “signs_table”. For this table, the index column should be set as “variable”, which is the column containing the list of variables. In the column named as “effect”, should be placed the expected signs of regression coefficients: “0” if there is no expectation, “1” if is expected a positive sign, or “-1” if it is expected a negative sign. The “Data_for_predictions” variable in the algorithm is the path for a CSV file containing all the samples to predict, for the case study, the file is “Data_for_predictions.csv”. This CSV file must have the same independent variables as the ones given for the “DATASET” file. For this data frame, it must be defined the index column, which for the presented case study is “Sampling site”. The output table path is defined in “output_table_path”, this is the path for the output excel file containing all the data of the output regression models.

After defining the data frames, other input parameters must be specified. The “Chosen_Y” is the predicted variable, and all the other variables presented in the dataset will be assumed as regressors. The minimum and the maximum number of predictors are the variables “nor_min” and “nor_max”, respectively. If the user requires that the regressions have an intercept equal to 0, the “intercept_value” must be set as “False”, while if different than 0 should be defined as “True”. For statistical thresholds, it must be defined the maximum allowed variance inflation factor. The authors suggest to select a maximum VIF of 5, since this is the stricter value found in the literature [64]. However, the user can choose higher or lower values, in “ MAXIMUM_VIF”. All the regressions with higher values than the selected will be excluded from the output. The probability to reject the null hypothesis is set in 4 variables. The first is the maximum probability to reject the null hypothesis for the R², set on the variable “max_p_r_sq”. The regression coefficients are defined in “max_p_reg_coef”. For heteroskedasticity and error normality, the minimum p-value to reject the null hypothesis is set in “min_p_for_Heteroskedasticity” and “min_p_for_error_normality”, respectively.

The last variables are the number of samples for the Goldfield-Quandt test. Since this test requires the calculation of two auxiliary regressions, the user sets the range of samples to apply in each regression, which is advisable to contain around one-third of the samples in each regression. In “lower_limit_A_GQ” is set the position of the first sample to use in the first regression, and in “upper_limit_A_GQ” is the position of the last sample for the first regression. In “lower_limit_B_GQ”, and “upper_limit_B_GQ” is set the position of the first and last samples, respectively, for the second regression. Since in the present study the regressions were calculated with 29 samples, it was set the lower and upper limit of the first regression as 1 to 9 (9 samples), for the second regression from 21 to 29 (also 9 samples), and letting the samples from 10 to 20 (11 samples) out of this analysis.

After setting all the inputs in the script, it is ready to run. First, the algorithm calculates all the possible combinations of predictors, regarding the minimum and the maximum number of predictors, using the function “combinations”, stored in a data frame “combo_df”. This is used in a cycle that selects each combination of predictors to be tested. If the combination fails in any test, the algorithm goes for the next combination of predictors until no combination is left to be tested.

The first test is for multicollinearity, accessed through the variance inflation factor of each regressor. If for any of the predictors the VIF is higher than the defined, the algorithm starts to calculate the statistical tests for the next set of predictors. Otherwise, it continues for the current combination, calculating the regression. If the R² statistical significance is below the maximum p-value, the algorithm continues and calculates the statistical significance of the predictors. If the maximum p-value among all predictors is lower than the defined, it is tested if the error has a normal distribution through 5 tests, Jarque-Bera [74], Anderson-Darling [75], Shapiro-Wilk [76], Kolmogorov-Smirnov [77] and Omnibus [78]. The heteroscedasticity is verified through 4 tests, Breusch–Pagan [79], Harvey-Collier [80], Glejser [81], and Goldfeld-Quandt [82]. After all tests, the algorithm calculates the number of predictors that do not have the expected signs and also predictions, until all the possible combinations of predictors have been tested. At the end of the algorithm is created an excel file that contains 5 sheets, “Summary_1”, “Summary_2”, “Predicted_values”, “Specific_predictions”, and “Dataset”. The last sheet contains the used dataset. In “Specific_predictions,” the lines are the predictions of each output regression for each sample (in each column) that were defined in the table “Data_for_predictions”. While in “Predicted_values,” are presented the predictions of each regression model for each sample contained in the dataset. Both these sheets allow to compare the measured vs predicted values. Each line of the summary sheets is a regression, and each row represents a statistical regression output. The first 9 rows are the most important, containing the regression data and statistical tests. The first one is the number of predictors used in the linear regression “number_of_predictors”. The second is the maximum variance inflation factor found among predictors “1_vif”. The 3rd, 4th, and 5th columns are relative to the explained variation, “2_r_sq” is the R², “2_r_sq_adj” is the adjusted R², and “2_p_value_of_R_squared” the probability to accept the null hypothesis of the regression F-test. The maximum p-value of the T-test among predictors is expressed in the column “3_p_regression_coef”. In the columns “4_MINIMUM_P_VALUE” and “5_MINIMUM_P_VALUE” is the minimum p-value of the error normality and heteroscedasticity tests, respectively. The column “6_sign_test” contains the number of predictors that do not have the expected sign. The following twelve columns contain the p values of each error normality and heteroscedasticity tests. Moreover, the 22^nd column contains the regression intercept, while the following columns contain the regression coefficients of each possible regressors. Besides “Summary_1” and “Summary_2” have an identical structure, “Summary_1” contains only regressions whose sign, is according to expectations, and all heteroscedasticity and error normality tests have met the statistical thresholds, while “Summary_2” contains regressions that do not have an expected sign, and whose error normality or heteroscedasticity tests results have met the statistical requirements for at least one of the tests in the group.

Case study application

The Python algorithm's purpose is to find linear regressions within the chosen dataset regarding the linear regression assumptions according to the specified statistical thresholds. Among possible hydrological years, the models were created using the hydrological year (HY) of 2014-2015, which is the one that contains more sampling sites, 29 in total. The algorithm was run for each SWP, and then it was selected a combination of variables that resulted in a model with the highest adjusted R². The chosen linear regressions were applied in the Geographically Weighted Regression (GWR) ArcMap tool. Before executing the GWR, the spatial autocorrelation of the standardized error terms was calculated using the “Global Moran's I” tool. If spatial autocorrelation was present, another set of combinations would be selected. Otherwise, it was performed in GWR to predict the SWP in 248 catchments. The predictions of SWP were obtained for all the sampling sites using both regression methods, and it was compared the measured value of the HY between 2012 and 2017 with the predictions, to check the models' performance using the percent bias (PBIAS) [83] and mean absolute percentage error (MAPE) [84] (ratings are presented Table 2).

Table 2.

Rating of water quality prediction models, based on the Percent Bias (PBIAS) and Mean Absolute Percentage Error (MAPE).

Method validation

Case study outputs

The Python algorithm was applied 4 times, one for each of the selected SWP. The selected inputs were: a number of regressors ranging from 1 to 6, a maximum VIF of 5, a probability to reject the null hypothesis for all statistical tests of 0.05, and the first and last 9 samples for the Goldfeld-Quandt test. Table 3 briefly presents the outputs of the Python script. When the algorithm was performed, it took around 1.5 seconds for each set of 100 combinations of regressors, which means that for each SWP the algorithm required several hours to compute all the possible combinations regarding the selected inputs. The output combinations, presented in Table 3, are the number of combinations that meet the selected statistical thresholds for all tests. In the respective output excel file of each SWP, these combinations of variables and the resulting linear regressions are presented in the sheet “Summary_1”. Among all the output combinations, one model was selected to apply for the case study, those with the highest adjusted R². In the “Summary_2” are presented more regressions since in this summary is demonstrated all regressions that have passed at least one test. By looking at the maximum adjusted R² values, the results indicate that the models can explain more of the pH and total nitrate in surface waters since these achieved the highest adjusted R² values, rather than electrical conductivity (EC) and total alkalinity, with adjusted R² of 0.797 and 0.693, respectively.

Table 3.

Summary of calculated regressions for each SWP.

Endogenous variable	Number of possible regressors	Number of regressors	Combinations of regressors		Adjusted R2 range
			Total	Output
EC	44	[1], [2], [3], [4], [5], [6]	8295045	413	[0.126-0.797]
pH	48		14196868	9871	[0.147-0.928]
Total nitrate	39		3930550	213	[0.113-0.917]
Total alkalinity	36		2391495	82	[0.216-0.693]

The selected linear regressions are presented in Table 4, with the respective combination number. In the supplementary material, these results can be found in the individual excel file of each SWP. It is seen that the maximum VIF among all regressors is below 5, the regression p-values are below 0.05, as also the regression coefficients. The minimum p-values found among all the homoscedasticity and error normality tests are above 0.05 since for these tests is necessary to accept the null hypothesis. For these 4 linear regressions, the spatial autocorrelation of the standardized error term was calculated through the Moran's I algorithm in ArcMap, presented in Fig. 4. When spatial autocorrelation is calculated, is computed a z-score that reports the degree of spatial autocorrelation. If this z-score is higher than 1.65, the residuals are clustered, below -1,65 are dispersed, and within this range, the distribution is spatially random [85]. All of the z-scores are inside the random range, meaning that the pattern is random, which is satisfactory for the selected regressions.

Table 4.

Selected regression for each surface water parameter.

Endogenous variable	Combination number	nº of regressors	VIF (maximum)	R2	Adjusted R2	R2 (p-value)	Regression coefficients (maximum p-value)	Homoscedasticity tests (minimum p-value)	Error normality tests (minimum p-value)
EC	1187093	5	1.616	0.812	0.771	1.18E-07	3.34E-02	1.07E-01	3.08E-01
pH	543724	5	1.607	0.926	0.910	2.86E-12	4.89E-04	5.46E-01	7.21E-01
Total nitrate	284906	5	4.129	0.932	0.917	1.23E-12	4.82E-02	3.82E-01	2.90E-01
Total alkalinity	312277	5	1.618	0.713	0.650	1.26E-05	3.22E-02	1.05E-01	1.08E-01

Fig. 4.

Spatial autocorrelation of the regression models, (A) conductivity, (B) pH, (C) total nitrate, and (D) total alkalinity. Z- scores of Moran's I test are presented in the respective image.

Besides the Python algorithm was calculated using a maximum of 6 regressors, it was found satisfactory regressions with only 5 exogenous variables. According to Table 5, all the regressors are statistically significant, including the intercept term. All VIF values are below 5, meaning that multicollinearity is absent from the models. Notably, the used variables in regressions are predominantly landscape metrics. Only the selected regression for pH contains the diffuse emissions from agriculture and forested areas (AF_P), which is transformed into 1 divided by the diffuse emission plus 1. This does not mean that the other variables, such as point source emissions or population density, are not suitable predictors, since these are also related to water quality. Some regressions are demonstrated in the supplementary material that contains variables that are not landscape metrics. Yet, for these SWP, it resulted that the predictors are majorly landscape metrics. Though the regressions are statistically robust, it was tested these models to predict the respective surface water parameters during different HYs. For LR and GWR models, it was compared the measured with the predicted values, for the 2 HY before and after 2014–2015. In Appendix B, is shown the scatter plots between the measured with predicted values for all models for all hydrological years. Noteworthy, for all SWP, the R² value of GWR is slightly higher than LR, the scatter dispersion of measured vs predicted values is visually identical. To formally address the differences between measured and predicted values, in Table 6 is shown the PBIAS and MAPE for GWR and LR. Comparing GWR with LR, the differences are not considerable. Regarding conductivity, the MAPE coefficient shows noticeable differences between GWR and LR models, but for the other SWP, the differences are low since the maximum is 2.5%. The maximum difference is 5.84% for the MAPE coefficient of the conductivity model applied to the HY of 2012–2013, where the MAPE of LR is 38.52% and the MAPE of GWR is 32.68%. In this way, there is no substantial difference between using GWR or LR since the results are practically the same. In general, LR models provide less biased predictions, while GWR has more accurate predictions since the MAPE is generally lower. Ratings of PBIAS and MAPE, are given in Table 7. As it can be seen, conductivity is the parameter with the worst predictions for almost all HY, with only a reasonable rating for the water quality calibration model efficiency, with the exception for the HY of 2015-2016, which has a low-performance rating. For total alkalinity, all predictions are considered reasonable ratings. Total nitrate predictions are reasonable, and “good” rating is achieved for the HYs of 2014–2015 and 2015–2016. For pH, the predictions are entirely accurate for all HYs with the highest rating.

Table 5.

Regression coefficients for each SWP selected regression.

Endogenous variable	Regression exogenous variables	Coefficients	p-value	VIF
EC	shdi_(FOR)2	-5.97E+01	1.04E-03	1.62
	cce_(ART)/(WAB)2	4.37E-01	3.38E-05	1.40
	cce_(WAB)/(AGR)0.5	8.11E+01	8.29E-03	1.62
	edp_(FOR)0.5	-1.31E+02	3.34E-02	1.03
	cedd_(AGR)/(FOR)2	-1.57E+07	2.41E-03	1.38
	intercept	3.15E+02	1.83E-07	-
pH	1/(1+AF_P)	-2.46E+01	1.03E-07	1.53
	cce_(ART)/(WAB)	2.76E-02	8.53E-08	1.61
	edp_(FOR)0.5	-1.01E+00	4.14E-07	1.36
	cedp_(AGR)/(FOR)2	-3.99E+00	1.32E-04	1.47
	cedd_(ART)/(FOR)0.5	-6.49E+00	4.89E-04	1.16
	intercept	8.48E+00	5.43E-24	-
Total nitrate	pz_(AGR)2	1.27E-02	6.45E-11	1.44
	cedd_(AGR)/(FOR)	-1.95E+03	9.32E-04	2.03
	1/(1+cce_(FOR)/(ART))	7.80E+00	4.36E-04	4.13
	lpi_(AGR)2	-8.46E-03	1.57E-05	3.56
	cedd_(ART)/(FOR)2	1.80E+06	4.23E-06	1.97
	intercept	2.50E+00	4.82E-02	-
Total alkalinity	ppc_(FOR)2	-5.05E-03	3.63E-04	1.20
	1/(1+lpi_(ART))	5.99E+01	4.80E-04	1.62
	edp_(FOR)0.5	-3.79E+01	8.03E-04	1.10
	cedd_(ART)/(FOR)0.5	-4.10E+02	2.13E-03	1.13
	cedd_(FOR)/(WAB)2	-1.17E+09	3.22E-02	1.56
	intercept	5.76E+01	3.28E-07	-

Table 6.

Regressions performance, comparing the predicted values with the measured values for 5 hydrological years, and for the linear regression (LR) and geographically weighted regression (GWR). The hydrological year 2014–2015 is marked with *, since the data of this HY was used to build the models. The color of the values is according to the given ratings in Table 2.

Table 7.

Overall rating of linear regression (LR) and geographically weighted regression (GWR) for each hydrological year. The color of the values is according to the given ratings in Table 2.

In Fig. 5. is shown the prediction of the SWPs for each of the 248 catchments in ARB, in a raster color gradient. These predictions show how the concentrations vary along the river basin. These values were compared to the recommended concentration for human consumption, the stricter water quality thresholds, according to the Portuguese legislation (Decree-law no. 236/98). Under these thresholds, the concentration of conductivity must be below 400 µS/cm, pH between 6.5 to 8.5, total nitrate below 25 mg/L, and total alkalinity below 30 mg/L, all the catchments that do not meet the threshold for the respective SWP are marked with stripes. Fig. 5 (E) demonstrates the number of studied SWP that do not meet the requirements, in each sub-catchment. In general, conductivity is low but only 5 sub-catchments exceed the minimum threshold. It turns out that the maximum concentration is very high, 4000 µS/cm, which is probably an unreal prediction, possibly because the conductivity model had the worst rating. For pH, there are multiple catchments with a low pH, predominantly in the upstream region, where primarily forested areas exist. Total nitrates locations with a concentration above the threshold are scattered and few. Total alkalinity results are above the recommended values in some catchments in the upstream area of the ARB. Still, along the river, there is a sequence of sub-catchments that have a high concentration.

Fig. 5.

Predictions of the surface water parameters (SWP), (A) electrical conductivity, (B) pH, (C) total nitrate,(D) total alkalinity, and (E) is the number of SWP with concentrations above the threshold.

Python script

By applying the Python script to the presented case study, it was demonstrated that the developed script is useful for finding linear regressions regarding the statistical thresholds. However, the authors emphasize that the algorithm does not find the single best linear regression but the best linear regressions set. The best model choice depends on an overall appreciation of the explained variation of the dependent variable, multicollinearity, and the residual assumptions, which the authors leave to the operators/researchers to decide. However, the authors recommend choosing the model with lower VIF values, higher statistical significance for regression coefficients, the lowest probability of having a non-normal error distribution or homoscedasticity, and highest adjusted R-squared. It is also pointed out to follow the best practices by applying suitable statistical thresholds [21]. Among possible programming languages, the authors have chosen to use Python since it is open source and one of the most popular programming languages [86,87], which makes it accessible to the majority of users. For those who are not familiar with programming, Python has the advantage of being one of the easiest languages to learn and apply [88]. Also, there is no need to have a deep programming knowledge of Python to use the developed algorithm since it is only necessary to create the CSV tables, edit the paths, and set the statistical thresholds in the first part of the algorithm. Unlike languages such as Matlab, Python is a non-paid language, so the final reason to choose Python is that it is an open-source language, making it accessible to any user. For homoscedasticity and error normality, it was chosen to apply well-established statistical tests because these are more frequently used and known by the scientific community, which gives more confidence to any user to interpret the results. However, more advanced approaches are being developed [89], which can be applied in the script for researchers who feel comfortable with Python programming. The presented script applies more than one test for homoscedasticity and error normality to bring more confidence to the results since these assumptions might be confirmed in one test and not in another. Still, some tests might be preferred over others [90]. For example, when comparing normality tests, Thadewald and Büning, 2007 [91] recommend that for distribution with short tails, Shapiro-Wilk might be better than Jarque Bera, or for different sample sizes, specific tests are valid [92]. For such reasons, the authors designed the script to create two outputs: “Summary_1” only shows the regressions that have passed all tests, and “Summary_2” shows all the regression models that have passed at least one test. This second output is important for authors that do not agree with the application of one test regarding their demands, and so the regressions are not presented in ”Summary_1” for failing in at least one test. As mentioned in the introduction, other software calculate linear regression models among candidate variables. For example, in ArcMap, the tool “Exploratory Regression“[18] does a similar job, with the advantage that calculates the spatial autocorrelation for each output regression. Still, the disadvantages are the necessity of a paid license, only executing one test for the error normality (Jarque-Bera), and does not perform homoscedasticity tests. In the presented Python script, spatial autocorrelation tests were not included to allow autors who do not have spatial data (in shapefile format) to use it. For authors who require to calculate the error spatial autocorrelation this can be done with the “Moran's I algorithm”, calculated separately in ArcMAP [93] and QGIS [94], or this test can also be included in the script using the “Moran” function available in PySAL library [95].

The Python script application was essential to choose a group of regressors to create GWR for each SWP. For the present study, the authors have gathered 73 possible regressors, and by applying four transformations (x²,x^0.5, log₁₀(x+1), and 1/(1+x)), the number of “candidate variables” increased to 365. Applying the 365 variables would require a very long time for the script to complete the calculations. For this reason, the authors have chosen to reduce the number of candidate variables by selecting only the transformation with the strongest Pearson correlation and with an expected sign for each SWP. For this reason, the number of possible regressors is different for each SWP, Table 3.

For users that would like to apply the presented algorithm, if the number of candidate variables is low, the authors consider that is not necessary to apply any preliminary variable selection strategy. This is because the computing speed was in average 1.5 seconds for every 100 combinations, performed in Python (3.8.8), using Anaconda (4.10.1), Spyder (4.2.5), in a desktop with windows 10 64 bits, 16 RAM, and a processor Intel(R) Xeon(R) CPU E5-2620 v3 2.4 GHz. The number of combinations is given by Eq. (2), which is a factorial relation between the possible candidate regressors (n) and the number of regressors (j). In Fig. 6 is presented the predicted time according to the number of combinations. For the present study, the processing time among surface water parameters ranged from 10 to 60 hours. This is indeed a disadvantage when users want to test a high number of regressors with a high range of candidate variables. In the future, this might not persist as a troubling disadvantage since quantum computer systems might become accessible for researchers [96]. Further, the adaptation of the code for a faster computing language, such as c++ [97], can reduce the script run time. However, it was chosen to develop the script in Python to promote ease of access in exchange for having a lower computing speed.

$$\text{number}\text{of}\text{combinations}=\sum _{\mathrm{j}=1}^{\mathrm{m}}\frac{\mathrm{n}!}{\mathrm{j}!\left(\mathrm{n}-\mathrm{j}\right)!}$$ (2)

Fig. 6.

Algorithm processing time vs the number of combinations between regressors.

Conclusion

A Python script was developed to calculate the best linear regressions for all the possible combinations of regressors of the input dataset. The output regressions are based on statistical thresholds defined by the user. The selected tests for this algorithm are: significance of the regressors, multicollinearity tested through VIF, error normality through five tests, Jarque-Bera, Anderson-Darling, Shapiro-Wilk, Kolmogorov-Smirnov and Omnibus, and heteroscedasticity through 4 tests, Breusch–Pagan, Harvey-Collier, Glejser and Goldfeld-Quandt.

The script included a novelty: the selection of regression models, in which the regression coefficients are according to the user's expectations. This detail is relevant for models that portray pollution since the outcome of pollution sources in models should indicate the increase of contamination. In this sense, the script will only report models that fulfilled this restriction, which is optionally set by the user.

The algorithm was tested with an environmental dataset to find linear regressions that could predict water quality parameters. The dataset included landscape metrics, population density, diffuse emissions, and point source pressures among the possible regressors. It was selected one of the output models for each surface water parameter, also tested in GWR. As a result, the GWR models were not very different from LR, but had higher R² values and also a lower MAPE. The predictions of pH were of high accuracy, for total nitrate, from reasonable to good, for total alkalinity, reasonable, and for electrical conductivity, from reasonable to low. Among the possible combinations of regressors of the models, the majority of regressors were landscape metrics, showing that for the case study, the landscape has a strong effect on water quality.

The developed brute force algorithm proved to be effective by testing all possible combinations, guaranteeing that the best models are found. The method demonstrated to be appropriate for datasets with tens of variables, according to the case study application. However, since the application is under brute force, the algorithm might not be feasible for big datasets with hundreds of variables. In this sense, further research is required to improve the method's computational speed.

CRediT authorship contribution statement

A.C. P Fernandes: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. A. R Fonseca: Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing. F.A.L. Pacheco: Conceptualization, Methodology, Resources, Writing – review & editing, Supervision, Project administration, Funding acquisition. L.F. Sanches Fernandes: Resources, Writing – review & editing, Supervision, Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A

Geographically Weighted Regressions Coefficients

Fig. A1

Fig. A1.

Geographically weighted regression coefficients, (A), (E), (I), (M), (Q), (U) for conductivity, (B), (F), (J), (N), (R), (V) for pH, (C), (G), (K), (O), (S), (X) for total nitrate and (D), (H), (L), (P), (T), (Y) for total alkalinity.

Appendix B- Comparison between the linear regression (LR) with geographically weighted regression (GWR) models

Figs. B1–B4

Fig. B2.

Comparison between the LR with GWR model predictions for pH.

Fig. B3.

Comparison between the LR with GWR model predictions for total nitrates.

Fig. B1.

Comparison between the LR with GWR model predictions for conductivity.

Fig. B4.

Comparison between the LR with GWR model predictions for total alkalinity.

Appendix B. Supplementary materials

Supplementary material, submitted along with the manuscript contains 3 folders: -“input_files”- all the necessary tables for the algorithm, regarding the case study and each surface water parameter; -“output_files”- the resulting excel files for each run of the script, regarding the case study application -“Python_script”- containing the used script for each surface water parameter.

Data availability

• Data is provided in the supplementary material