Prediction of Prostate Cancer From Routine Laboratory Markers With Automated Machine Learning

ABSTRACT

Background

In this study, we attempted to select the optimum cases for a prostate biopsy based on routine laboratory test results in addition to prostate‐specific antigen (PSA) blood test using H2O automated machine learning (AutoML) software, which includes many common machine learning algorithms.

Methods

The study included 737 patients (46–88 years old). Routine laboratory measurements were used to train machine learning models using H2O AutoML. We created a model that classifies prostate biopsy results as malignant or benign. The performance of the best model was evaluated using the area under the receiver operating characteristic curve (AUC), log‐loss metric, F1 score, positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity. The model's performance was evaluated through the SHapley Additive exPlanations (SHAP) analysis feature‐based interpretation method applied to comprehend the machine learning model.

Results

The gradient boosting machine model was the most successful. The best result was obtained in the model with 11 parameters, including PSA, free PSA, free PSA to PSA, hemoglobin, neutrophils, platelets, neutrophil‐to‐lymphocyte ratio (NLR), glucose, platelet‐to‐lymphocyte ratio (PLR), lymphocytes, and age. The AUC of this model was 0.72, the specificity was 0.84, the PPV was 0.65, the NPV was 0.69, and the accuracy was 0.68.

Conclusion

Our results suggest that adding only routine laboratory parameters to the PSA test and developing machine learning algorithms can help reduce the number of unnecessary prostate biopsies without overlooking the diagnosis of PCa.

This study utilizes routine laboratory data and machine learning (AutoML) to improve prostate biopsy decisions. A Gradient Boosting Machine model with 11 features achieved an AUC of 0.73 and a specificity of 0.84. SHAP analysis identified total PSA as the most significant predictor, highlighting its potential to reduce unnecessary biopsies.

1. Introduction

Prostate cancer (PCa) is one of the most common malignancies among men [1]. Serum total prostate‐specific antigen (PSA) is an organ‐specific biomarker that often increases in patients with PCa. It has high tissue specificity; however, various benign diseases of the prostate can also lead to elevated serum PSA levels [2]. In addition, up to 25% of patients with normal PSA values may also have underlying PCa [3]. In many patients whose biopsy decisions are based solely on PSA levels, cancer is not detected by a histopathological evaluation [3, 4]. It is now suggested that, if the PSA level is high, the next step should be to use newly developed biomarkers instead of a direct prostate biopsy [2]. However, these new tests are more expensive and often unavailable in many laboratories.

Several routine laboratory markers, including the neutrophil‐to‐lymphocyte ratio (NLR), platelet‐to‐lymphocyte ratio (PLR), and alanine transaminase‐to‐aspartate aminotransferase ratio (ALT/AST), have been shown to be important indicators for the diagnosis and prognosis of malignancies [5, 6].

Machine learning (ML) techniques, with their ability to analyze complex relationships and variable interactions, can make more accurate predictions than traditional statistical methods [7, 8, 9]. ML methods are frequently used in different areas of medicine, including cancer diagnoses, and provide cost‐effective, faster results [8, 9]. Moreover, automated machine learning (AutoML) is a transformative technology that enables high‐quality, scalable model training and reduces the dependency on human experts [10]. ML, particularly AutoML, may offer a new approach to current challenges in PCa screening and prediction [11].

In this study, we aimed to improve PCa screening and select high‐risk cases for biopsies, according to the results of routine laboratory tests through H2O AutoML.

2. Materials and Methods

This study was approved by the Clinical Research Ethics Committee of the Bursa Yuksek Ihtisas Education and Research Hospital with protocol number 2011‐KAEK‐25 2023/10–05 according to the Declaration of Helsinki. Written informed consent was not obtained because the study was a retrospective reanalysis of the dataset using AutoML techniques and had no influence on patients' biopsy and treatment decisions.

A total of 737 consecutive patients were included who underwent 12 transrectal ultrasound scan (TRUS)‐guided core biopsies at the Urology Clinic of Bursa Yuksek Ihtisas Education and Research Hospital between January 2019 and May 2023. We selected men under 50 years old with a PSA level > 2.5 ng/mL and men over 60 years old with a PSA level > 4.0 ng/mL. The median age of the patients was 65 years (range 46–88). The data on laboratory variables included in this study were available in the patient files prior to biopsy. Hemoglobin, white blood cell (WBC), neutrophil, platelet, lymphocyte, platelet‐to‐lymphocyte ratio (PLR), red cell distribution width (RDW), neutrophil‐to‐lymphocyte ratio (NLR), PSA, free PSA (fPSA), fPSA/PSA ratio, total testosterone, aspartate transaminase (AST), alanine transaminase (ALT), AST/ALT ratio, urine WBC, urine erythrocyte, glucose and HbA1c measurements were used to train machine learning models using H2O AutoML.

In our laboratory, hemoglobin, WBC, neutrophil, platelet, and lymphocyte levels were counted using Mindray BC‐6000 (Mindray Bio‐Medical Electronics Co. Ltd., Shenzhen, China) automatic hematology analyzers and reagents. Serum PSA and fPSA levels were measured using an Architect i2000SR (Abbott Diagnostics, Abbott Park, IL, USA), and AST, ALT, and glucose levels were measured using Architect c16000 and Abbott kits. Urine WBC and urinary erythrocyte levels were counted with the automated system Dirui H‐800/ FUS‐200 (DIRUI Industrial Co. Ltd., China). The HbA1c level was determined by HPLC using a Lifotronic H9 (Lifotronic, Shenzhen, China) analyzer and reagents.

An analysis compared the laboratory results between the PCa and benign groups. Continuous variables in the data were expressed as medians and 25th–75th quartiles or means and standard deviations. An unpaired t‐test or nonparametric Mann–Whitney U test was used for the comparisons. p < 0.05 was considered statistically significant.

Prostate biopsy results were coded as a categorical variable in the dataset. This column was coded 1 for malignant cases and 0 for benign cases. This coding was used to facilitate model classification.

To implement the AutoML analysis, the H2O package was installed on the H2O.ai platform (www.h2o.ai), which is open‐source software that includes many common ML algorithms [12, 13] (Figure 1). The H2O AutoML algorithm fine‐tunes various models simultaneously across multiple categories, including gradient boosting machines (GBMs), XGBoost GBMs, generalized linear models, deep neural networks, extreme randomized trees, and distributed random forests. The H2O package automatically selects applicable algorithms and integrates them into multiple ensemble models. Using the H2O AutoML approach, 20 ML algorithms were trained and tested simultaneously. This process continued until 20 models were completed with 100 s time constraint, with the goal to select the most accurate method [12, 14].

FIGURE 1.

Flowchart of machine learning process. Laboratory routine blood test data were used. The dataset was randomly divided into training and test data into 70% training and 30% testing groups, i.e., 532 patients were in the training group, and 205 patients were in the testing group.

Missing values in the dataset were detected and completed with the mean values of the relevant columns. All numeric variables were scaled to a range of 0–1 using the Min–Max normalization method. This scaling enhances the model's performance by eliminating the biases caused by features with varying scales [15].

In addition, autoML algorithms randomly divide data into 70% training and 30% testing groups. That is, 532 patients were in the training group, and 205 patients were in the testing group [12, 14].

A model was built that classifies prostate biopsy results as malignant or benign. Ten‐fold cross‐validation was used during the model's training [12, 16]. H2O AutoML trained models with pre‐specified hyperparameters and also trained models by optimizing different combinations of hyperparameters using the grid search method [17]. The model's performance was measured on the test dataset.

When building ML models with H2O AutoML, we set a time constraint of 100 s and limited the maximum number of models to 20 to accommodate the 100 repeated iterations. While a 100 s limit may introduce constraints, we tested different time settings and found 100 s sufficient for building efficient ML models. H2O AutoML performed 10‐fold cross‐validation on models over 100 iterations using the training data and summarizing the results in a leaderboard (Table 1). The table shows the performance of the top 10 models with the metrics AUC, AUCPR, log loss, mean error per class, RMSE, and MSE.

TABLE 1.

Top 11 models ranked by AUC using 11 parameters with H2O AutoML.

	AUC	Logloss	AUCPR	Mean_per_class_error	RMSE	MSE
GBM_5_AutoML_35_20241114_184033	0.794	0.556	0.786	0.248	0.430	0.185
StackedEnsemble_BestOfFamily_3_AutoML_37_20241114_184236	0.793	0.554	0.792	0.278	0.431	0.186
StackedEnsemble_AllModels_2_AutoML_35_20241114_184033	0.792	0.552	0.792	0.271	0.431	0.186
XRT_1_AutoML_25_20241114_183014	0.790	0.623	0.800	0.283	0.433	0.187
GBM_4_AutoML_33_20241114_183830	0.787	0.555	0.801	0.295	0.433	0.188
DRF_1_AutoML_89_20241114_193710	0.784	0.562	0.793	0.282	0.434	0.189
GBM_3_AutoML_85_20241114_193253	0.782	0.561	0.785	0.318	0.441	0.19
GBM_2_AutoML_40_20241114_184542	0.782	0.562	0.783	0.287	0.435	0.189
XGBoost_2_AutoML_33_20241114_183830	0.778	0.585	0.782	0.318	0.441	0.195
GBM_grid_1_AutoML_27_20241114_183219_model_1	0.777	0.571	0.777	0.272	0.439	0.193
GLM_1_AutoML_1_20241118_113548	0.710	0.616	0.733	0.369	0.461	0.212

Abbreviations: AUC, area under the curve; AUCPR, precision‐recall area under the curve; DRF, distributed random forest; GBM, gradient boosting machine; GBM, gradient boosting machine; GLM, generalized linear model; MSE, mean squared error; RMSE, root mean squared error; XGB, extreme gradient boosting; XRT, extreme randomized trees.

The model was trained with a random starting point using different seed values in each cycle. In each cycle, an AutoML model was retrained using the H2O AutoML (Automatic Machine Learning) algorithm. The best‐performing model was selected for each cycle, and the performance of this model was evaluated on the independent test set. The 95% confidence intervals of each metric were calculated using the performance measurements on the independent test dataset. These confidence intervals helped evaluate the reliability of the model's performance on the independent data set (Table 2). Since we are dealing with a binary classification problem, we rank models according to their AUC scores, the main metric for evaluating their performance. Based on this ranking, the GBM model achieved the highest AUC score, making it the best‐performing model for distinguishing the classes in our dataset (Table 1). The stacked ensemble model was the second best according to AUC values. Consequently, the GBM model was used for the explainability analysis.

TABLE 2.

Summary of prediction results of gradient boosting machine models built with different parameters.

Metrics	Total PSA	Free PSA	Free/Total PSA	11 Parameters
AUC	0.64 (0.62–0.66)	0.51 (0.49–0.53)	0.68 (0.66–0.69)	0.72 (0.69–0.75)
AUCPR	0.56 (0.52–0.61)	0.45 (0.39–0.48)	0.57 (0.54–0.59)	0.66 (0.62–0.68)
Logloss	0.66 (0.64–0.67)	0.69 (0.68–0.73)	0.71 (0.62–1.09)	0.61 (0.58–0.66)
Accuracy	0.49 (0.43–0.58)	0.42 (0.3951, 0.4785)	0.62 (0.46–0.65)	0.68 (0.65–0.72)
F1	0.58 (0.56–0.61)	0.56 (0.56–0.57)	0.61 (0.60–0.62)	0.64 (0.61–0.67)
Sensitivity	0.81 (0.73–0.86)	0.81 (0.66–1.00)	0.71 (0.58–0.96)	0.45 (0.31–0.59)
Specificity	0.29 (0.15–0.49)	0.17 (0.00–0.36)	0.56 (0.14–0.72)	0.84 (0.72–0.91)
PPV	0.43 (0.4–0.48)	0.39 (0.37–0.41)	0.52 (0.42–0.56)	0.65 (0.58–0.72)
NPV	0.69 (0.63–0.75)	0.55 (0.00–0.70)	0.76 (0.71–0.85)	0.69 (0.66–0.75)

Note: Model using 11 parameters includes total prostate‐specific antigen, free prostate‐specific antigen, the ratio of free to total PSA, hemoglobin, neutrophils, age, platelets, the neutrophil‐to‐lymphocyte ratio, glucose, the platelet‐to‐lymphocyte ratio, and lymphocytes.

Abbreviations: AUC, area under the curve; AUCPR, area under the curve precision‐recall; NPV, negative predictive value; PPV, positive predictive value.

The GBM ensemble technique was applied. This technique constructs multiple decision trees sequentially. Each aims to correct the errors of its predecessors, thus effectively managing complex data [18]. In the first phase of the study, the Boruta algorithm was used to determine the features that had the greatest impact on the model, identifying PSA, fPSA, and fPSA/PSA as the three most critical features in terms of model performance. In the second phase, the stepwise feature addition method was applied. First, the model was trained with only three important features selected by Boruta (PSA, fPSA and fPSA/PSA), and its performance was evaluated. Subsequently, other important features were gradually added to the model, and changes in prediction accuracy were studied.

The performance results of the candidate models are presented in Table 2.

The Shapley Additive Explanations (SHAP) analysis feature‐based interpretation method allows an understanding of the predictions of ML models. This method was applied to interpret the ML model (Figure 1) [19].

3. Results

The baseline characteristics of the participants are listed in Table 3. Patients' PSA levels were between 4 and 920 ng/mL. The median age of patients with malignant biopsy results was 67 years (62–76), whereas the median age of patients with benign biopsy results was 64 years (58–68). PSA and fPSA/PSA levels were significantly different between the positive biopsy and negative biopsy groups; however, other studied laboratory parameters were similar between groups. See Table 3.

TABLE 3.

Main patients' characteristics, categorized according to biopsy findings.

Characteristics	Whole cohort (n = 737)	Negative biopsy (n = 441)	Positive biopsy (n = 296)	Missing values	p‐value
Age (year)	65 (60–69)	64 (58–68)	67 (62–76)	—	< 0.001
tPSA (ng/mL)	7.93 (5.53–12.1)	7.03 (5.2–9.9)	10.25 (6.77–19.72)	—	< 0.001
fPSA (ng/mL)	1.54 (1.08–2.45)	1.5 (1.09–2.23)	1.65 (1.03–2.94)	29	0.064
fPSA/PSA	0.19 (0.14–0.26)	0.22 (0.17–0.28)	0.15 (0.1–0.22)	29	< 0.001
Hemoglobin (g/dL)	14.4 (13.5–15.3)	14.5 (13.7–15.4)	14.4 (13.4–15.1)	—	0.012
RDW‐CV (%)	13.5 (13.1–14.2)	13.5 (13.1–14.2)	13.6 (13.1–14.3)	—	0.169
WBC (109/L)	7.6 (6.5–9.31)	7.5 (6.46–9.4)	7.8 (6.54–9.21)	—	0.658
Neutrophils (109/L)	4.69 (3.73–6.07)	4.61 (3.63–6.13)	4.82 (3.87–6.0)	—	0.215
Lymphocytes (109/L)	2.03 (1.63–2.56)	2.09 (1.66–2.65)	1.94 (1.61–2.43)	—	0.006
Platelets (109/L)	236 (201–279)	237 (201–279)	231 (201–287.25)	—	0.801
AST (U/L)	18 (15–23)	19 (15–23)	18 (15–23)	9	0.225
ALT (U/L)	17 (13–22)	17 (13–22)	16.5 (13–21.25)	9	0.177
AST/ALT	1.1 (0.91–1.38)	1.19 ± 0.48	1.23 ± 0.72	9	0.488
Glucose (mg/dL)	99 (90–115)	99 (89–114)	101 (91–118)	2	0.068
HbA1c (%)	5.79 (5.49–6.49)	5.79 (5.49–6.45)	5.80 (5.5–6.7)	344	0.291
WBC in urine	0 (0–2)	0 (0–2)	0 (0–1)	1
Erythrocyte in urine	0 (0–2)	1 (0–2)	0 (0–2)	1
NLR	2.23 (1.74–3.13)	2.15 (1.7–3.02)	2.41 (1.8–3.35)	—	0.05
PLR	8.8 (6.92–11.45)	8.69 (6.79–11.07)	9.09 (7.11–12.25)	—	0.032

Note: Values are presented as median (25th–75th quartile).

Abbreviations: ALT, alanine transaminase; AST, aspartate aminotransferase; AST/ALT, AST/ALT, AST to ALT ratio; fPSA, free‐PSA; NLR, neutrophil to lymphocyte ratio; PLR, platelet to lymphocyte ratio; PSA, prostate‐specific antigen; RDW‐CV, red cell distribution width—coefficient of variation; WBC, white blood cell.

The sensitivity, specificity, accuracy, AUC, AUCPR, F1 score, log loss, positive predictive value (PPV), and negative predictive value (NPV) of all models were evaluated to predict positive biopsy results (Table 2).

The GBM model, the most successful of the models created with PSA alone, had an AUC of 0.64 (0.62–0.66), a sensitivity of 0.81 (0.73–0.86), a PPV of 0.43 (0.40–0.48), an NPV of 0.69 (0.63–0.75), and an accuracy of 0.49 (0.43–0.58) in the test dataset (Table 2).

In the model created with fPSA/PSA alone, the AUC in the test dataset was 0.68 (0.66–0.69). The PPV was 0.52 (0.42–0.56), and the accuracy was 0.62 (0.46–0.65) (Table 2). Parameters were ranked for positive biopsy predictions with Boruta feature selection, after which disparate parameter combinations were tested.

The best results were obtained using 11 parameters: total PSA, free PSA, fPSA/tPSA, Hb, neutrophil, platelet, NLR, glucose, PLR, lymphocytes, and age. The AUC of the model in the test dataset was 0.72 (0.69–0.75), the specificity was 0.84 (0.72–0.91), the PPV was 0.65 (0.58–0.72), the NPV was 0.69 (0.66–0.75), and the accuracy was 0.68 (0.65–0.72) (Table 2).

The model's F1 score showed a moderate performance indicating 0.64. Both precision and sensitivity are comparatively higher than PSA and fPSA alone.

As presented in Figure 1, we observed that many parameters showed high SHAP values, indicating that these parameters were effective at distinguishing negative biopsy test results.

4. Discussion

Herein, we developed a clinical laboratory dataset coupled with AutoML‐based algorithms to discern patients requiring a prostate biopsy on a single‐center retrospective dataset.

The GBM 11‐parameter model, owing to its superior performance and interpretability, can reduce the number of unnecessary prostate biopsies without compromising the diagnosis of PCa. We differentiated at‐risk patients with the addition of only routine hematological and laboratory parameters that were included in AutoML algorithms for the classification task.

Using PSA alone for predicting positive biopsies, the specificity and accuracy were 0.29 and 0.49, respectively, whereas in the 11‐parameter model, the specificity increased to 0.84 [20].

ML techniques were introduced by Snow, Smith, Catalona in 1994 to detect PCa using the PSA level, a digital rectal examination, and a transrectal ultrasonography [21]. They built a neural network model for PCa using several sequential iterations, including additional variables, such as age, ethnicity, free PSA, family history, pro‐PSA, LUTS score, and prostate volume. They achieved an accuracy of 0.87 in predicting biopsy results [22]. Various ML models were used in their study, and the Random Forest model achieved the best results [8].

Regardless of the parameters used in the models, consistent findings have revealed the better diagnostic utility of ML models compared with static parameters only (such as PSA or free PSA).

In our study, the model performance (AUC, 0.72) was better than the performance of the PSA test alone (AUC, 0.64). This finding corroborates previous studies that reported an increase in the AUC of 0.07 [22]. However, this AUC is not statistically perfect and may lead to diagnostic uncertainty [23].

Much research has used ML techniques to diagnose PCa and PSA, and its derivatives have mainly been studied [8, 9, 11, 21, 23]. Perera et al. demonstrated that using a dense neural ML model including age, PSA, fPSA, and the free/total PSA ratio, achieved an AUC of 0.72 compared to 0.63 (PSA alone), improving the PCa diagnosis in a retrospective cohort [22]. Chiu et al. found the AUC of the PSA for PCa to be 0.71 in an ML technique study performed on 4659 consecutive men, which was shown to prevent unnecessary biopsies by 38.3%–52.2% [8]. Similarly, the decrease in the percentage of unnecessary prostate biopsies was 63.9% in our model.

Studies without a dedicated biomarker algorithm and using common laboratory tests similar to our study have also been conducted [24, 25, 26, 27].

Recently, Nitta et al. reported that the ML method, using input data, including age, prostate volume, PSA, and urinary WBC, performed well in predicting PCa in 512 patients who underwent a prostate biopsy after screening with serum PSA [24]. A urinary WBC was selected to exclude patients with prostatitis. In this study, artificial neural networks, a support vector machine, and Random Forest were applied, and AUCs between 0.63 and 0.69 were revealed as superior to the PSA level [24]. Our 11‐parameter ML model also illustrated better AUC than the PSA level alone.

Recently, Chen et al. used the XGBoost ML model, a variant of the gradient boost machine (GBM), and patients' age, body mass index (BMI), PSA‐related parameters, and serum biochemical parameters as model variables and provided better performance (AUC 0.82) than f/PSA (AUC 0.75), PSA (AUC 0.68), and fPSA (AUC 0.61), respectively [27].

An accurate prostate cancer risk assessment is critical to ensure the rapid diagnosis of prostate cancer while limiting the number of unnecessary prostate biopsies. This new ML model can be used to screen populations to optimize patients who are eligible for further diagnostic tests. This approach does not require additional tissue sampling, costly biochemical tests, or trained clinician assessments to enhance prostate cancer risk assessments. Due to mathematical complexity, ML models can be difficult for clinicians to interpret. SHAP analysis of which features are most effective in prediction can provide valuable information and increase the clinical applicability of the model [28].

ML models are a relatively new technology that is rarely used in clinical practice, and these models may contribute to the limited evidence supporting such technologies in urological practice [29].

This study has several limitations. First, the cohort size in the current series was small, and the number of predictor parameters considered was large [30]. All ML models are affected by sample size, and no sample size can truly be described as adequate. Second, in this study, the TRUS‐guided pathological examination of prostate biopsies was adopted as the gold standard because a cancer cell may be missed due to the lack of magnetic resonance imaging guidance.

In this study, both the training and testing groups retrospectively collected data from the same hospital. Due to privacy issues, there is no external validation outside the institution with an independent dataset. Patient management was not algorithm driven. Because of the retrospective nature of this study, some patients were excluded due to a lack of important information. This raises concerns about the generalizability of the findings to other populations and clinical settings.

Furthermore, since ML is trained to find the most statistically relevant features from copious training data, imperfect health data must be carefully considered. Since the data are not pre‐collected according to the specific requirements of the study, missing data are a common problem in retrospective studies and can significantly affect the final prediction results of ML studies [31]. Further external validation of separate cohort datasets is needed, as there is a potential risk of selection bias regarding the patient population and biopsy indications.

Finally, the aim was to use only routine laboratory data. Correspondingly, additional clinical variables were not used, such as family history, pro‐PSA levels, LUTSs, and prostate volume, which may increase the accuracy of ML techniques.

In conclusion, the reported ML pattern to determine whether a prostate biopsy is necessary can safely optimize the number of correctly classified patients. By building ML models with commonly measured biomarkers, we were able to improve the progress of routine medical laboratories toward clinical diagnostic precision. However, further validation in multi‐ethnic populations and performance optimization for routine clinical use are necessary.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The authors have nothing to report.