Comparison of conventional mathematical model and machine learning model based on recent advances in mathematical models for predicting diabetic kidney disease

Abstract

Previous research suggests that mathematical models could serve as valuable tools for diagnosing or predicting diseases like diabetic kidney disease, which often necessitate invasive examinations for conclusive diagnosis. In the big-data era, there are several mathematical modeling methods, but generally, two types are recognized: conventional mathematical model and machine learning model. Each modeling method has its advantages and disadvantages, but a thorough comparison of the two models is lacking. In this article, we describe and briefly compare the conventional mathematical model and machine learning model, and provide research prospects in this field.

Introduction

According to the 10th edition of the International Diabetes Federation Diabetes Atlas, ¹ up to 10% of adults worldwide have been exposed to diabetes, which is linked to increasing morbidity. Diabetic kidney disease (DKD) is one of the main complications of diabetes and the main cause of end-stage renal disease.^2,3 Indeed, chronic kidney diseases have been observed in 50% and 30% of patients with Type 1 diabetes and Type 2 diabetes, respectively. ⁴ Although early diagnosis and comprehensive treatment can prevent kidney diseases and delay their progression, ⁵ presently, diagnosis and rational treatment have limited clinical use in DKD patients. ⁶ Furthermore, the gold standard for DKD diagnosis is an invasive biopsy, which is associated with poor patient compliance. Hence, a non-invasive diagnosis of DKD is urgently needed. ⁷

Most studies used conventional modeling methods such as logistic regression, Cox survival regression, and multivariate analysis to statistically analyze basic information, laboratory indicators, and contributing factors to DKD, establish prediction models, and explore independent risk factors linked to DKD.^8–34 Additionally, machine learning methods have been employed to establish risk prediction models for DKD.^35–43 Although different modeling methods have been used in these studies, they can be generally divided into the conventional mathematical models and the machine learning models. Each type of model has its advantages and disadvantages. The lack of comprehensive reliability evaluation prompts researchers to generally prefer conventional modeling methods, which are simpler and more user-friendly compared to machine learning methods. However, a thorough comparison of these models is lacking. Therefore, this study compares the two models to provide the best-performing model in DKD prediction.

Conventional model

In mathematical modeling, a complex problem is constructed into a mathematical problem and solved by exploring its unique internal laws and using appropriate mathematical methods to connect key factors related to the problem directly or indirectly. ⁴⁴ Common mathematical models include statistical regression model, algebraic equation and difference equation model, differential equation model, discrete model, probability model, and stochastic decision analysis method. These models have been categorized as diagnostic models and prediction models according to their application value in diseases.

As previously reported, the prediction model has wider application prospects than the diagnostic model. Therefore, conventional models are established based on statistical methods for constructing prediction models. First, the data are screened, and the data meeting the criteria are selected. Next, values are assigned to the classification variables, and the data is standardized in the training and validation sets to reduce errors during validation. A single-factor analysis is then performed to identify factors with significant differences by applying statistical methods such as t-test, Kruskal-Wallis test, ANOVA, and χ² test. Factors without differences are converted using log transformation, reciprocal conversion, or square conversion, and then analyzed using the aforementioned statistical methods to evaluate differences. The independent variables were screened by LASSO regression, elastic network graph, stepwise regression, optimal subset, and other methods. Subsequently, classical models such as the Logistic regression equation, Cox survival regression equation, unit linear regression equation, and multiple linear regression equation are used to introduce the selected independent variables and construct a diagnostic or prediction model. Finally, the proposed model is evaluated, validated, visualized, and made into application software (Figure 1).

Figure 1.

Conventional model modeling process.

In the conventional model, data can be processed using simple statistical methods, and this may not affect the implementation. Because of its simplicity, conventional modeling can be readily grasped by researchers with some statistical background within a relatively short timeframe. In addition, selecting the corresponding classical model based on the research question will not affect its prediction efficiency. Although the conventional model is easy and has good prediction performance, it also has some shortcomings. It can only perform optimally when the sample size is neither too large nor too small. A larger sample size of the conventional model will lead to a labor-intensive process of selecting data that meets the requirements, with a high risk of error and bias. On the other hand, a smaller sample size of the conventional model may lead to inaccurate results when applying statistical methods. Because the formula of the classical model is fixed, the range of problems that can be studied using this method is limited. For instance, if the problem of interest does not match existing classical models, applying a conventional model may lead to inaccurate results. Generally, the conventional model includes only those influencing factors screened out as significantly different using statistical analysis and discards those without differences. However, some non-differential factors or those with combined effects with other factors may be excluded, which may lead to incomplete results, and even inaccurate or unconvincing model predictions (Table 1).

Table 1.

Advantages and disadvantages of the models.

Model	Advantages	Disadvantages
Conventional model	Simplicity	Time-consuming and laborious
	Convenient	Average generalization ability
	Low equipment requirements	Small sample
	Mature operation process	Possible omissions in factor selection
	Good predictive performance (no need for parameter adjustment)	Prone to deviations, underfitting, or overfitting
		Few types of models
Machine Learning Model	Labor-saving	Complex
	Large-sample	High equipment requirements
	Comprehensive selection of factors	Difficult to operate and difficult to learn
	Reduce the occurrence of deviations, underfitting, and overfitting(require cross-validation, parameter tuning, and other processes)	Require cross-validation, parameter tuning, and other processes to improve model performance
	Excellent predictive performance (require cross-validation, parameter tuning, and other processes)	The operation process is not very mature, there is still room for exploration.
	Excellent generalization ability (require cross-validation, parameter tuning, and other processes)	Establishing a model requires making multiple models and comparing them to select one with good performance
	Multiple types of algorithms	Poor interpretability (the general ranking of ability of interpretation: regression >cluster classification> tree-based model > Neural Network)

Recently, artificial intelligence (AI) has attracted great attention, and some researchers have even attempted to apply AI algorithms to establish better mathematical models than conventional models. Nevertheless, these novel models and conventional models have not been compared in performance.

Machine learning

AI is the hallmark of the progress in computing from “data processing” to “knowledge processing.” With the rapid advances in computer technology, the question arises whether AI can replace humans in intellectual activities. ⁴⁵ AI research seeks to create computers that can emulate the intelligent actions of humans in their environment. Machine learning, as a branch of AI, serves as a fundamental tool for constructing prediction models.

Machine learning is an interdisciplinary field that integrates probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity to study how computers simulate or implement human learning behaviors. Machine learning focuses on harnessing computational power and human knowledge to optimize system performance. In computer systems, “experience” commonly exists as “data.” Hence, machine learning research mainly focuses on creating a “learning algorithm” that could generate “models” from data archived in a computer. A learning algorithm can generate models based on the empirical data provided. In a new situation, the model will provide us with corresponding judgments. If computer science is the study of “algorithms,” then machine learning is the study of “learning algorithms.” ⁴⁶ Machine learning is at the core of AI, and it is fundamental to creating intelligent computers. ⁴⁷ Common models used for data mining and statistical machine learning include linear regression and linear classification algorithms and model evaluation and selection algorithms, such as decision tree and combination method, support vector machine, neural network, and deep learning methods.^48–50 Initially, we conduct data cleaning, noise reduction, missing value processing, feature engineering, normalization, and standardization. Next, a suitable model is selected for training based on data characteristics, model use, experimental purpose, etc. (multiple models can be selected for comparison and the best selection). Once the model is trained, the performance of the model is evaluated and verified. In cases with an insufficient sample size or there is no independent verification set, we perform cross-verification to prevent problems such as overfitting of the model. The commonly used cross-verification methods include: HoldOut cross-validation, K-fold cross-validation, hierarchical K-fold cross-validation, leave P out cross-validation, leave a cross-validation, shuffle-split, rolling cross-validation (time series), etc. Among them, the rolling cross-validation is considered the most effective, but it cannot be used for random samples. The order of the data is important. Cross-validation employs various strategies to split the main dataset into training and validation sets, each with its unique approach. For instance, K-fold cross-validation partitions the data into K-equal subsets, utilizing one-fold for validation and K−1 folds for training. Rolling cross-validation, on the other hand, divides the data into training and validation sets based on temporal order. Finally, to improve the performance of the model and achieve its generalization performance on the verification set or test set (reduce the problem of underfitting or overfitting and error of the model), the parameters of the model should be adjusted (Figure 2).

Figure 2.

Machine learning model modeling process.

There are two types of parameters in machine learning, one is the parameter, and the other is the hyperparameter. Since the parameter is determined using the model selection algorithm, tuning refers to the hyperparameter. Two approaches are used to conduct parameter adjustment, one is manual parameter adjustment, and the other is automatic parameter adjustment (application algorithm parameter adjustment). Manual parameter tuning, being a time-consuming and laborious process, defeats the purpose of employing machine learning models to enhance efficiency. Consequently, automatic parameter adjustment is adopted as a more efficient alternative. Common auto-tuning methods include grid search, random search, Bayesian optimization, ensemble learning, and others. The choice of parameter tuning method depends on the specific machine learning model selected. For instance, Bayesian optimization is employed to adjust models derived from Bayesian inference. Furthermore, cross-validation remains an essential component of the parameter adjustment process. For example, K-fold cross-validation is often applied to reduce the estimation bias and variance caused by the randomness of data division, because the adjustment of hyperparameters also needs to take into account the structural risk and empirical risk of the model. The parameter adjustment is completed when the model evaluation index reaches the optimum. For example, precision, accuracy, recall, F1 score, and area under the curve of the receiver operating characteristic (AUC-ROC) are all the largest or the larger the better; mean square error, root mean square error, mean absolute error, etc. reach the minimum value or the smaller the better. So based on the characteristics of the above indicators, the performance of different models can be compared by comparing these indicators of different models.

Machine learning can solve problems with larger sample sizes and reduce errors in workforce statistics and data screening. However, if the included data deviates significantly from the real problem, there may be a bias. Machine learning provides several learning algorithms and can create a novel learning algorithm to address a specific problem. Therefore, machine learning can explore a much wider range of problems than the conventional model, but this may result in over-fitting, which, like under-fitting, is not suitable for model application. Due to the overlapping capabilities of various learning algorithms, certain problems can be addressed using multiple algorithms, although with varying prediction accuracies. Thus, it is necessary to establish and compare models to select models optimal for a particular problem separately. For data analysis, machine learning does not randomly discard data. However, some factors that are not significant or useful for the problem may be included in the model, thereby reducing the predictive effectiveness. Because of this characteristic, machine learning will perfectly preserve the undifferentiated but related data that are filtered out when applying the conventional model. While machine learning models can suffer from errors like deviations, overfitting, and underfitting, these issues can be significantly mitigated through techniques like cross-validation and parameter tuning. Since the working principles behind machine learning are not very clear, its interpretability is relatively low. The general ranking of the ability of interpretation: regression > cluster classification > tree-based model > neural network. Despite their relative complexity and high operating coefficients, machine learning models remain highly sought-after due to their remarkable predictive capabilities (Table 1).

Discussion

Two common modeling approaches were compared for predicting DKD risk (Table 2): the conventional mathematical model and the machine learning model, including sample size, key features, performance of different algorithms (such as AUC-ROC), and machine learning also described whether parameters were adjusted and whether cross-validation was performed.

Table 2.

Comparison of performance of various models.

Author	Model	Model specifications	Sample size（N）	Key features	Model performances		Cross-validation	Tuning parameters
					AUC-ROC	Others
Tan et al. 10	Conventional	Logistic regression and nomogram	N:102	Duration of T2DM, HbA1c, presence of DR, absence of hematuria, and absence of systemic biomarkers.	88.60%	Positive predictive value (PPV):87.5%, specifcity:81.1%, negative predictive value (NPV):65.2%, sensitivity:75.4%	Internal validation with bootstrap sampling with 200
Lv et al. 11	Conventional	Cox regression model	N:515	Triglyceride glucose (TyG) index	69.00%	Sensitivity:71.3% and specificity:37.2%
Sanchez-Alamo et al. 12	Conventional	Time-to-event curves were based on the Kaplan–Meier analysis and Cox regression model	N:103	Interleukin-6 (IL-6)	87.90%	Cut-off:4.68 pg/mL, sensitivity:100%, specificity:78.72%, PPV: 45.6%, and NPV: 100%
Yang and Jiang 19	Conventional	Logistic regression, nomogram, and forest plots	N:706	Scr, hypertension, HbA1c, BUN, BMI, TG, and DPN	77.30%	C-index:0.773
Hu et al. 25	Conventional	LASSO regression, logistic regression, and nomogram	N:3489	SBP, DBP, FBG, HbA1c, TGs, SCR, BUN, and BMI	74.40%	C-index:0.744
Hosseini Sarkhosh et al. 35	Machine learning	Random forest (RF) and logistic regression (LR)	Training set:1907, verification set:1543, N:3450	diabetes duration, HbA1c, eGFR, ACR, CVD, and hypertension	75.50%	Accuracy:0.7239, precision:0.6908, and F1 score:0.6051	Recursive feature elimination with cross-validation (RFECV)	Grid search technique
Zou et al. 36	Machine learning and conventional	RF and nomogram	N:390	Cys-C, sAlb, Hb, UTP, and eGFR	90%	Accuracy:82.65%, sensitivity:83.33%, and specificity:81.58%	10-fold cross-validation
Zou et al. 36	Machine learning	Gradient boosting machine (GBM)	N:390	Cys-C, sAlb, Hb, UTP, and eGFR	88%	Accuracy:83.67%, sensitivity:95%, and specificity:65.79%	10-fold cross-validation
Zou et al. 36	Machine learning	Support vector machine (SVM)	N:390	Cys-C, sAlb, Hb, UTP, and eGFR	88%	Accuracy:83.67%, sensitivity:86.67%, and specificity:78.95%	10-fold cross-validation
Zou et al. 36	Machine learning	LR	N:390	Cys-C, sAlb, Hb, UTP, and eGFR	83%	Accuracy:79.59%, sensitivity:78.33%, and specificity:81.58%	10-fold cross-validation
Zhang et al. 37	Machine learning	RF	N:929	DR, DM course, Hb, PP, sCr, ALB, TC, sudden onset of heavy proteinuria, hematuria, and family history of DM	95.30%	Accuracy:88%, sensitivity:84.4%, specificity:89.9%, and balanced accuracy:87.1%	Five-fold cross-validation: 1. AUC-ROC:94.6%, 2. AUC-ROC:94.6%, 3. AUC-ROC:97.4%, 4. AUC-ROC:93.8%, 5. AUC-ROC:96%
Zhang et al. 37	Machine learning	SVM	N:930	DR, DM course, Hb, PP, sCr, ALB, TC, sudden onset of heavy proteinuria, hematuria, family history of DM	94.70%	Accuracy:88.3%, sensitivity:84.2%, specificity:90.6%, and balanced accuracy:87.4%	Five-fold cross-validation: 1. AUC-ROC:94.8%, 2. AUC-ROC:92.8%, 3. AUC-ROC:97.2%, 4. AUC-ROC:94%, 5. AUC-ROC:94.7%
Allen et al. 38	Machine learning	RF/gradient boosted trees (XGB)	N:111046		82.3%/82.5%	Sensitivity: 75%/75%, Specificity:73.9%/74.2%,		Yes
Liu et al. 40	Machine learning	Bayesian network (BN)	N:1485		83.10%		10-fold cross-validation	Tabu search

In general, the sample size of machine learning is larger than that of the traditional model. Based on AUC-ROC, the performance of the machine learning model was higher than that of the traditional model. Most machine learning models use cross-validation. Some studies have applied tuning parameters, such as Zhang et al., ³⁵ Allen et al., ³⁸ and Liu et al. ⁴⁰ In part, based on the AUC-ROC values including those reported by Tanet al., ¹⁰ Sanchez-Alamo et al., ¹² Zhang et al., ³⁵ and Allenet al., ³⁸ the performance of the traditional model is not necessarily lower than that of machine learning. Through the adoption of model optimization processes such as cross-validation and parameter adjustment, the performance of machine-learning models can be greatly improved. A pending question is, can these methods of mathematical model optimization improve the performance of traditional models if they are applied to traditional models?

In this article, we only made superficial comparisons between traditional models and machine learning models based on previous findings. While both models possess their own unique strengths and limitations, further comprehensive research is needed to directly compare their performances under identical conditions. Consequently, definitive conclusions regarding the superiority of either model remain elusive. In the era of big data, machine learning models may have more development prospects and application value. However, whether conventional mathematical models should be completely abandoned is a question that cannot be definitively answered.

The use of prediction models has become increasingly prevalent and influential in the medical field. However, current prediction models show different test power. To integrate research and clinical medicine to develop a reliable mathematical model with high inspection efficiency, we should compare the performance of tools for model establishment under different conditions.

Several research directions are proposed. First, under the premise of unified standards, the same research questions, data set, quantity, and influencing factors should be used. In addition, appropriate adjustments should be made, including selecting multiple types of questions, heterogeneous of population, time factors (e.g. chronology, survival, mortality, etc.), multiple outcomes, using different sample sizes (small, medium, or large), and exploring more influencing factors matching the research questions. The optimized model or algorithm should be selected for modeling using the two types of modeling methods: conventional modeling and machine learning. Finally, the two methods can be compared to identify the more effective one. Alternatively, according to the research method of Zou et al., ³⁶ the combination of these two methods may build a better prediction model, and even a prediction power of 1 + 1 > 2 may appear. The author has carried out some related experiments and obtained some results, among which the combined model is better than the single algorithm model for the same research problem under the same small sample data set.

Conclusion

In summary, this paper provides a comparison between conventional models and machine learning models, highlighting their respective advantages and disadvantages. However, it is worth noting that the existing research lacks a detailed comparative analysis at an equivalent level. Furthermore, our findings indicate that the fusion model can exhibit superior prediction performance compared to both individual models. However, studies on the fusion model remain limited in number. Consequently, these aspects will be the primary focus of our future research endeavors. Currently, our team has initiated research in these areas and has achieved significant progress.