Improving cardiovascular risk prediction through machine learning modelling of irregularly repeated electronic health records

Abstract

Aims

Existing electronic health records (EHRs) often consist of abundant but irregular longitudinal measurements of risk factors. In this study, we aim to leverage such data to improve the risk prediction of atherosclerotic cardiovascular disease (ASCVD) by applying machine learning (ML) algorithms, which can allow automatic screening of the population.

Methods and results

A total of 215 744 Chinese adults aged between 40 and 79 without a history of cardiovascular disease were included (6081 cases) from an EHR-based longitudinal cohort study. To allow interpretability of the model, the predictors of demographic characteristics, medication treatment, and repeatedly measured records of lipids, glycaemia, obesity, blood pressure, and renal function were used. The primary outcome was ASCVD, defined as non-fatal acute myocardial infarction, coronary heart disease death, or fatal and non-fatal stroke. The eXtreme Gradient boosting (XGBoost) algorithm and Least Absolute Shrinkage and Selection Operator (LASSO) regression models were derived to predict the 5-year ASCVD risk. In the validation set, compared with the refitted Chinese guideline–recommended Cox model (i.e. the China-PAR), the XGBoost model had a significantly higher C-statistic of 0.792, (the differences in the C-statistics: 0.011, 0.006–0.017, P < 0.001), with similar results reported for LASSO regression (the differences in the C-statistics: 0.008, 0.005–0.011, P < 0.001). The XGBoost model demonstrated the best calibration performance (men: D_x = 0.598, P = 0.75; women: D_x = 1.867, P = 0.08). Moreover, the risk distribution of the ML algorithms differed from that of the conventional model. The net reclassification improvement rates of XGBoost and LASSO over the Cox model were 3.9% (1.4–6.4%) and 2.8% (0.7–4.9%), respectively.

Conclusion

Machine learning algorithms with irregular, repeated real-world data could improve cardiovascular risk prediction. They demonstrated significantly better performance for reclassification to identify the high-risk population correctly.

Graphical Abstract

Graphical Abstract.

See the editorial comment for this article ‘From data to wisdom: harnessing the power of multimodal approach for personalized atherosclerotic cardiovascular risk assessment’, by S. Al-Kindi and K. Nasir, https://doi.org/10.1093/ehjdh/ztad068.

Introduction

Safe and cost-effective treatments can reduce cardiovascular risk significantly. The magnitude of treatment benefit is directly related to the pre-treatment cardiovascular risk of individual patients. To accurately determine this risk, reliable risk prediction equations must be employed. The global cardiovascular guidelines recommend various risk assessment tools to tackle the heavy burden of cardiovascular disease (CVD), such as the Pooled Cohort Equation (PCE),¹ the Systematic COronary Risk Evaluation (SCORE) model,² and the Prediction for Atherosclerotic Cardiovascular Disease Risk in China (China-PAR) model.³ Although current risk prediction models using a number of traditional CVD risk factors have played an important role in CVD prevention, the predictive performance has yet to produce satisfactory results. For example, several external validation studies have demonstrated that the C-statistics of these models range only between 0.65 and 0.74,^4,5 and may incorrectly estimate the absolute cardiovascular risk.^4,6 It is known that traditional CVD risk prediction can be enhanced through additional information gained from either new predictors or repeated measurements. In addition to traditional predictors such as age, smoking, and systolic blood pressure (SBP), new predictors from various aetiological pathways [e.g. lipoprotein (a) and apolipoprotein,^7,8 glucose metabolism,^9,10 and renal function markers¹¹] can also potentially improve prediction accuracy. However, implementing traditional prediction models using all these novel predictors for CVD primary prevention in the entire population is not realistic in real-world clinical practice. Secondly, recent evidence suggests that repeated measurements of CVD risk factors in traditional prediction models can improve performance,^12–14 which may capture the longitudinal information of risk factors and help explain the cardiovascular residual risk.¹⁵ However, the current traditional modelling approaches have limitations in that they are able to consider only a limited number and type of repeated predictors,¹⁶ and they may overlook potential interactions among these predictors.^17,18

In contrast, electronic health records (EHRs) can not only provide a wealth of information with repeated measurements on various predictors¹⁶ but also allow for automated screening if the risk prediction tool is embedded.^19,20 However, the data structure in real-world EHR systems often differs from that in traditional cohort studies. Although new predictors may exist in subgroups of the population, the pattern of available risk factors is often irregular. For example, patients may undergo a series of repeated measurements, especially for traditional CVD risk factors, but the number of repeats varies among subjects. Moreover, it is also quite common that different predictors are measured between individuals, even from the same aetiological pathway. For example, someone having information on obesity will have his or her body mass index measured, whereas others will have their waist-hip ratio measured. Besides, risk factors are generally measured at different time points. Therefore, this EHR-based information remains challenging to be incorporated using conventional risk prediction models.

In this case, machine learning (ML) algorithms can be a valuable alternative to handle such complex data. While evidence shows that the benefits of ML algorithms over traditional models using the same predictors were limited, they can excel in accommodating multiple predictors and handling irregular measurements, making them suitable for leveraging the rich information present in EHRs effectively.^21,22 While ML has been increasingly utilized to leverage information from repeated measurements in certain hospital-based scenarios,^23,24 its application in primary care for cardiovascular risk assessment remains limited.^25,26 Existing studies have demonstrated that ML can enhance risk prediction,^22,27 but they have not fully utilized time-to-event information or comprehensively evaluated predictive performance. Developing fixed-term survival prediction models is crucial for CVD risk assessment, as they align with the recommended risk stratification cut-offs in clinical guidelines.^1–3

Therefore, this study aims to investigate the improvement in CVD risk prediction by incorporating irregularly repeated real-world measurements of multiple predictors using ML models. The predictive performance is then compared against the guideline-recommended traditional Cox regression model.²⁸

Methods

Study design

The concept of the study design is shown in Figure 1A. The population included in this study was taken from the Chinese Electronic Health Records Research in Yinzhou (CHERRY) study, which was an EHR-based cohort study in Yinzhou, Ningbo (a developed area in Eastern China). A detailed description of the CHERRY study has been published elsewhere.²⁹ The inclusion criteria of this study population consist of (i) age ranging between 40 and 79 years at the entry time; (ii) Chinese residents registered in the health information system during the period between 1 January 2010 and 31 December 2016; and (iii) had been living in Yinzhou for at least 6 months. The exclusion criteria of this study are as follows: (i) had no records of serum lipid measurements. Since lipid-related predictors were causally related to atherosclerotic CVD (ASCVD), this may cause the model to be not applicable. (ii) had a history of CVD before being enrolled in the study. A flowchart of the inclusion and exclusion process is shown in Supplementary material online, Figure S1. Finally, 215 744 participants were included in the analysis set, among which a random sample of 80% (∼180 000) was separated as the training set to derive the models, and the remaining participants were left for the validation (shown in Supplementary material online, Figure S2). This study was approved by the Peking University Institutional Review Board (IRB00001052-16011).

Figure 1.

The study design and categories of predictors. (A) The cohort design of the study; (B) predictors of seven aetiological categories included in different approaches.

To maximize the number of repeated measurements collected, the baseline in this study was set as (i) the time when the participants registered in the system, (ii) the time when participants reached 40 years old, (iii) the time when the first serum lipid measurement was recorded, or (iv) 1 January 2015, whichever the latest. The repeated measurements were collected from the past 5 years before the baseline. Participants would be followed up to the time (i) when they experienced their first ASCVD event (further defined in the outcomes section), (ii) they were censored from following up, or (iii) 31 May 2020, whichever was the earliest.

Predictors

Seven common categories of cardiovascular risk factors (shown in Figure 1B) with 25 markers in total were pre-identified as the pool of predictors, including demography (age, sex, education levels, settings, smoke status, and family history), lipid metabolism {total cholesterol [TC], HDL cholesterol [HDL-C], LDL cholesterol [LDL-C], triglycerides [TG], apolipoprotein A [apo A], apolipoprotein B [apo B], and lipoprotein (a) [Lp-(a)]}, obesity (BMI and waist circumference), glucose metabolism [fasting blood glucose (FBG), diabetes at baseline, and haemoglobin A1c (HbA1c)], blood pressure [SBP and diastolic blood pressure (DBP)], renal function [estimated glomerular filtration rate (eGFR) and albumin creatinine ratio (ACR)], and medical treatments (antihypertension, antihyperglycaemic, antihyperlipidemic treatment, and aspirin). We selected these risk factors because they were universally incorporated into cardiovascular risk prediction,^{1,12,28,30–32} had possible causal relationships with ASCVD outcomes,^8,9,33,34 or were closely associated with ASCVD from aetiological perspectives.^{7,10,11,35,36} Measurements of these predictors were collected from multiple sources in the regional health system, including census data, electronic medical records (EMRs), disease surveillance, chronic disease management system, and health check, which are summarized in Supplementary material online, Table S1. These records will be inherently linked to each other according to a unique and encoded identifier. Detailed data collection procedures of various data sources are described in Supplementary material online, Method S1. The exact definitions of each medical treatment are given in Supplementary material online, Table S2. Extreme outliers were removed according to pre-specified normal ranges of key predictors (shown in Supplementary material online, Table S3).

Considering the irregular nature of the predictors’ information available, we used a simple but effective approach to leverage these repeated measurements by summarized statistics.^24,37 Standard deviation (SD), range, and the difference between the last and the first measurements were calculated as derived predictors since many studies have proposed that the variability of predictors is associated with CVD.^17,18,38 The number of measurements was also counted and included in the pool of predictors.³⁹ The mean values of the predictors were also summarized to represent the long-term average of these predictors. All the baseline and derived predictors included in this research are listed in Supplementary material online, Table S4.

Outcomes

The definition of the ASCVD was consistent with the one used in the China-PAR or PCE model, which was defined as the composite outcome of non-fatal or fatal stroke (ICD-10 code: I60, I61, I63, and I64), non-fatal myocardial infarction (I21 and I22), and coronary heart disease death (I20–I25).²⁸ The outcomes in this study were collected from the following sources: disease surveillance, chronic disease management system, death registry, and EMR. Among these sources, the disease surveillance and death registry were recognized as the gold standard. The outcome used the first ASCVD events that occurred after the baseline and before 31 May 2020.

Risk prediction models

Since the China-PAR model was the Chinese guideline–recommended risk assessment tool in primary care, our study selected this model as the reference to be compared. The model was modified by two different approaches in this study to make the comparison fair: (i) the refitted China-PAR model was developed by directly replacing all the coefficients in the original model but preserving all the predefined terms (including all the interaction terms); (ii) the recalibrated China-PAR model was developed by replacing the baseline survivals and means of linear predictors in the original model without altering any predefined terms and their corresponding coefficients.

Two ML approaches were finally adopted in this study, which were eXtreme Gradient Boosting (XGBoost) algorithm and Least Absolute Shrinkage and Selection Operator (LASSO) regression. The choice of algorithms depends on various factors such as the nature of the data, the size of the dataset, the complexity of the problem, and the desired interpretability of the model.¹⁹ For large datasets with high dimensionality (many predictors), algorithms that can efficiently handle such data, such as Random Forest, Gradient Boosting, or Deep Learning models, may be suitable.^40,41 Meanwhile, for CVD risk prediction, model interpretability is crucial. Simpler models such as regression-based models or decision trees are still preferred, as they can provide more transparent and easily interpretable results.¹⁹ Random Forest or Gradient Boosting can also offer feature importance rankings and handle missing values without imputation. Finally, the computational cost of training the model is a consideration, especially for large datasets. Linear models and tree-based models tend to be faster and easier to train than deep learning models. To control the potential overfitting, algorithms with built-in regularization, such as LASSO regression, were considered. After studying the performance and feasibility of the four aforementioned methods, XGBoost and LASSO regression were chosen because they performed better than the others and are relatively easy to be interpreted and implemented in the EHR system. These two algorithms utilize the information of predictors from different perspectives so this will also help to find the more suitable approach to leverage the repeated measurements.^42,43 The importance of predictors was assessed according to the average reduction of information entropy in the XGBoost model and the absolute value of the β coefficients in LASSO regression, which reflected the information gains or the marginal effects of the predictors. Two ML classifiers were first trained in the 126 893 subjects with known outcome information at the end of the fifth year. Then, the two ML models were embedded into a Cox regression model to predict the absolute 5-year risk. Hyperparameter tuning was conducted by maximizing the area under the curve (AUC) in the five-fold cross-validation. Grid search were iterated 100 times to acquire the optimized hyperparameter sequence.^44,45 The ranges of the hyperparameters are given in Supplementary material online, Table S5.

Statistical analysis

Continuous predictors were described using means and SDs, while categorical predictors were described using counts and percentages. The associations between predictors and ASCVD were given according to the hazard ratios of Cox proportional hazard regression adjusted for the variables from the China-PAR models. Proportions of missingness were described for each predictor. The predictors in the China-PAR models were multiple-imputed by chain equations (five imputation sets were created) to compare with the two ML models.^12,46 The performance metrics were measured in each imputation set and then pooled according to Rubin’s rules.⁴⁷ Machine learning models can handle the issue of missingness directly to preserve initial information. Details of data imputation are provided in Supplementary material online, Method S2. The performances of the models were evaluated in terms of the following perspectives: discrimination, calibration, and reclassification. The discrimination was assessed by using Harrell’s C-statistic. Calibration was used to measure the coordination between predicted risk and observed risk, which was evaluated using the Hosmer–Lemeshow ${\chi }^2$ and calibration plots.⁴⁸ The C-statistics of different models were compared using the approach proposed by Kang et al.⁴⁹ The risk distribution by different models was also illustrated. To pool the Hosmer–Lemeshow ${\chi }^2$ given by different imputation sets, a D_x statistic following the F distribution was generated based on the approach proposed by Rubin⁵⁰ and Li et al.⁵¹ We provided the standard reclassification table. Net reclassification improvement (NRI) and integrated discrimination improvement (IDI) were calculated to quantify the reclassification benefits of the ML models over the refitted China-PAR model. The cut-offs of risk groups were selected according to the 2019 Guideline on the assessment and management of cardiovascular risk in China.³ A decision curve analysis (DCA) was also conducted to illustrate the clinical implications of the ML models. Sensitivity analyses were conducted as follows: (i) to further ascertain whether the possible improvement in risk prediction was driven by leveraging the information from the repeated measurements or by simply including more baseline predictors, a Cox regression model including all the baseline measurements of each predictor was constructed, and its performance was compared against the two ML models and the refitted China-PAR model; (ii) the performance of the recalibrated China-PAR was assessed and compared to evaluate how much improvement ML models can achieve compared with the per-guideline approach. All analyses were conducted using R version 4.0.4 with a statistically significant level of P < 0.05. The XGBoost model was constructed by using the xgboost package version 1.4.1.1, and the LASSO regression model was constructed with glmnet package version 4.1-1. The mice package version 3.13.0 was adopted for the multiple imputation by chain equations.

Results

Basic descriptions for participants and predictors

The characteristics of the 215 744 included participants are described in Table 1. Fifty-four per cent of the participants were women, and their mean age was ∼56.7 (SD = 9.6). The means of major risk factors for ASCVD: SBP, TC, and high-density lipoprotein cholesterol were 134.5 mmHg, 4.9, and 1.3 mg/dL, respectively. The average BMI was 23.3 kg/m². Overall, 12.1% of them had diabetes at baseline. During a median of 5.4-year follow-up, 6081 individuals (2.82%) had ASCVD outcomes. The incidence rate of ASCVD was 6178 per million person-years. Only TC and anti-hyperglycaemia treatment significantly differed between derivation and internal validation datasets (shown in Supplementary material online, Table S6). The missing proportions of each predictor are shown in Supplementary material online, Table S7. The number of measurements and time intervals between each measurement of key predictors for each individual is given in Supplementary material online, Table S8. The mean number of measurements of TC, SBP, BMI, and fasting glucose were 3, 2, 1, and 3, respectively. The corresponding median time intervals between these measurements were 269, 136, 267, and 251 days.

Table 1.

Characteristics^a of the study population

	Overall (N = 215 744)	Men (n = 100 078)	Women (n = 115 666)
Demography
Age, years	56.70 (9.59)	57.10 (9.75)	56.35 (9.44)
Rural	65 086 (30.34%)	30 016 (30.15%)	35 070 (30.51%)
Smokers (current or ever)	57 961 (26.87%)	53 861 (53.82%)	4100 (3.54%)
Finished high school	108 120 (50.11%)	55 576 (55.53%)	52 544 (45.43%)
Family history of ASCVD	1318 (0.61%)	701 (0.70%)	617 (0.53%)
Blood pressure
SBP, mmHg	134.45 (16.64)	134.58 (16.37)	134.32 (16.88)
DBP, mmHg	82.63 (9.87)	83.10 (9.90)	82.18 (9.81)
Obesity
Waist circumference, cm	81.76 (7.94)	83.93 (7.61)	79.90 (7.73)
BMI, kg/m2	23.31 (2.87)	23.44 (2.71)	23.21 (3.01)
Lipid metabolism
Total cholesterol, mmol/L	4.90 (0.98)	4.77 (0.97)	5.01 (0.98)
HDL-C, mmol/L	1.30 (0.34)	1.25 (0.34)	1.35 (0.33)
TG, mmol/L	1.61 (1.09)	1.66 (1.20)	1.56 (0.99)
LDL-C, mmol/L	2.84 (0.82)	2.77 (0.81)	2.90 (0.83)
Apo A, mmol/L	1.22 (0.27)	1.18 (0.27)	1.26 (0.27)
Apo B, mmol/L	0.95 (0.25)	0.95 (0.25)	0.95 (0.25)
Lp-(a), mmol/L	4.87 (0.14)	4.60 (0.14)	5.12 (0.15)
Glucose metabolism
FBG, mmol/L	5.67 (1.57)	5.76 (1.72)	5.60 (1.44)
HbA1c, %	6.86 (1.90)	6.99 (1.98)	6.73 (1.82)
Diabetes mellitus	26 090 (12.09%)	12 364 (12.35%)	13 726 (11.87%)
Renal function
eGFR, mL/min/1.73 m2	98.92 (15.30)	97.71 (15.28)	99.94 (15.25)
ACR, mg/g	15.90 (45.36)	16.32 (48.91)	15.57 (42.39)
Medication
Anti-hypertension treatment	75 857 (35.16%)	35 590 (35.56%)	40 267 (34.81%)
Anti-hyperlipidaemia treatment	35 561 (16.48%)	15 662 (15.65%)	19 899 (17.20%)
Anti-hyperglycaemia treatment	22 847 (10.59%)	10 881 (10.87%)	11 966 (10.35%)
Aspirin treatment	19 064 (8.84%)	9100 (9.09%)	9964 (8.61%)
Outcome
ASCVD events	6081 (2.82%)	3272 (3.27%)	2809 (2.43%)
Average follow-up time, years	5.41 (1.36)	5.41 (1.51)	5.41 (1.22)
Incidence rate of ASCVD, per million person-years (95% CI)	6178 (6177–6179)	7242 (7241–7243)	5245 (5244–5246)

^aCategorical variables are presented by counts and percentages; continuous variables are presented by means and SDs. All summarized statistics are given based on the complete sets of each predictor.

The discrimination of the models

In the validation set, the C-statistics with the absolute differences compared with the refitted China-PAR model are given in Figure 2. The C-statistic of the XGBoost model was 0.7918 [95% confidence interval (CI): 0.7776–0.8060] and that of LASSO regression was 0.7883 (0.7737–0.8029). The two ML models performed better than the refitted China-PAR model in the parameter of discrimination (differences in the C-statistic for XGBoost: 0.01134, 0.00567–0.01700, P < 0.001; for LASSO: 0.00784, 0.00453–0.01115, P < 0.001). They also performed better than that of the refitted China-PAR model in the same parameter in both men and women, where the XGBoost model performed the best among men, while LASSO regression performed the best among women. The final hyperparameters in the final ML models are given in Supplementary material online, Table S9. The major structures of the final ML models are given in Supplementary material online, Figure S3 and Table S10.

Figure 2.

The difference in C-statistics scores compared with the refitted China-PAR model. The results are given based on the validation set of 31 544.

The calibration of the models

The XGBoost model showed better calibration than the refitted China-PAR model in both men and women (XGBoost: D_x = 0.598, P = 0.75 in men and D_x = 1.867, P = 0.08 in women; refitted China-PAR: D_x = 2.832 in men, P = 0.004 and D_x = 3.352 in women, P = 0.001), while LASSO regression was recalibrated well in men (D_x = 1.639, P = 0.11) but not in women (D_x = 1.950, P = 0.048). The calibration plots are shown in Figure 3. Although the XGBoost model slightly overestimated the risk in the highest risk group, the coordination of the predicted risks and Kaplan–Meier observed risks was much better than the LASSO model and refitted the China-PAR model, especially among low-risk deciles.

Figure 3.

Calibration plots of different models by sex^a. The results are given based on the validation set of 31 544.

Clinical implications on outcomes

In the validation set, the reclassification table is shown in Table 2. By using the XGBoost model rather than the refitted China-PAR model, additional 3355 subjects out of total 24 247 non-case individuals were classified as lower risk. There were 667 subjects classified as higher risk. A net quantity of 2688 people (11.09%) was reclassified into the correct groups. Among 969 individuals who developed CVD during follow-up, XGBoost and the refitted China-PAR model selected a similar number of high-risk subjects, that is, 550 and 585. After taking the medium-risk group into account, China-PAR correctly selected 70 (7.22%) more case individuals. The overall NRI rate was 3.87% (1.35–6.38%). Similarly, the NRI rate for LASSO regression was 2.78% (0.66–4.89%). A direct comparison of the predicted risk against the refitted China-PAR model showed that the IDIs of the XGBoost model and LASSO regression were 0.0174 (0.0135–0.0212) and 0.0106 (0.0081–0.0131), respectively. The risk distributions predicted by the XGBoost and the refitted China-PAR models are illustrated in Figure 4. The risk predicted by XGBoost tended to centralize in the lower range in non-cases in both men and women, with a larger difference between the risks of cases and non-cases. The DCA demonstrated that all three models, namely XGBoost, LASSO, and the refitted China-PAR model, exhibited favourable performance by deviating from the curves of treating all or treating none within the common cardiovascular risk range of 0–20%. Moreover, the net benefit of the XGBoost model surpassed that of the refitted China-PAR model between the threshold range of 7.5 and 12.5%, while the net benefit of the LASSO regression model was superior within the range of 12.5–17.5% (see Supplementary material online, Figure S4).

Table 2.

Reclassification of machine models against the refitted China-PAR model^a

		XGBoost				NRI (95% CI)	IDI (95% CI)
		<2.5%	2.5–4.9%	≥5%	Total
	Refitted China-PAR
Non-case	<2.5%	14 142	270	74	14 486	0.0386	0.0174
	2.5–4.9%	2119	2506	323	4948	(0.0135, 0.0638)	(0.0135, 0.0212)
	≥5%	53	1183	3577	4813
	Total	16 314	3959	3974	24 247
Case	<2.5%	185	11	8	204
	2.5–4.9%	46	114	20	180
	≥5%	1	62	522	585
	Total	232	187	550	969

		LASSO
		<2.5%	2.5–4.9%	≥5%	Total
	Refitted China-PAR
Non-case	<2.5%	14 147	324	15	14 486	0.0278	0.0106
	2.5–4.9%	1057	3543	348	4948	(0.0066, 0.0489)	(0.0081, 0.0131)
	≥5%	3	826	3984	4813
	Total	15 207	4693	4347	24 247
Case	<2.5%	188	14	2	204
	2.5–4.9%	22	132	26	180
	≥5%	0	41	544	585
	Total	210	187	572	969

^aThe results are given based on the subjects who were not censored (25 216) from the validation set of 31 544.

Figure 4.

Distribution of predicted risk given by the XGBoost model and the refitted China-PAR model in the validation set.

The importance of predictors

The associations between the predictors and ASCVD are presented in Supplementary material online, Table S11, which are adjusted by predictors in the China-PAR model. All predictors were included in the XGBoost model because it used the random subspace sampling technique, while the LASSO regression model selected only 78 of the total 101 predictors (i.e. baseline and summarized statistics of repeat information of 25 markers). The rank of importance is given in Supplementary material online, Figure S5. In general, age, anti-hypertension treatment history, glucose metabolism–related predictors, lipid metabolism–related predictors, blood pressure, eGFR, and a family history of ASCVD were most valued by both ML models. FBG ranked third and fifth in the rank of importance in XGBoost and LASSO regression, respectively. Novel lipid predictors such as apo B ranked eighth and tenth in the two ML models, while classic predictors such as TC ranked only 10th and 17th. The importance of smoking and predictors indicating obesity was relatively low (BMI: 16th in XGBoost and 19th in LASSO; waist circumference: 20th in XGBoost and 16th in LASSO; smoking: 18th in XGBoost and 14th in LASSO).

Sensitivity analysis

The Cox regression model with the baseline measurements of all predictors performed better than the refitted China-PAR model, while it was still worse than the XGBoost model in the whole validation set for the parameter of discrimination (the differences in the C–statistics: 0.00563, 0.00118–0.01009, P = 0.01, Supplementary material online, Table S12). Its discriminative performance was not significantly different from that of LASSO regression (0.00214, −0.00088–0.00515, P = 0.17, Supplementary material online, Table S12). The coordination of predicted and observed risk indicated by the calibration plot of the Cox model with all baseline measurements was insufficient, which was not even better than the refitted China-PAR model (D_x = 2.421, P = 0.01 in men and D_x = 2.216, P = 0.02 in women, Supplementary material online, Figure S6). Both ML models performed significantly better than the recalibrated China-PAR model in the two parameters of discrimination (all P < 0.001, Supplementary material online, Table S13) and calibration (recalibrated China-PAR: D_x = 2.421 in men and D_x = 2.216 in women, both P < 0.001, Supplementary material online, Figure S6).

Discussion

This study used two ML approaches (XGBoost and LASSO) to leverage the existing repeated measurements in EHR data to predict 5-year atherosclerotic cardiovascular risk. Both ML models outperformed the recalibrated and the refitted China-PAR model in the parameters of discrimination, calibration, and reclassification, which is the model recommended by the 2019 Guideline on the assessment and management of cardiovascular risk in China.³

Repeated measurements from EHRs offer valuable contributions to cardiovascular risk prediction. Notably, the QRISK3 model in the United Kingdom was derived from EHR data obtained from general practice computer systems, in which the SD of SBP was included as a predictor.¹² This model stands as the first nationwide-used risk prediction model to incorporate predictors derived from repeated blood pressure measurements. Similarly, Paige et al.¹⁴ leveraged EHR data from the Health Improvement Network, a UK general practice electronic database, and applied a landmark model to utilize information from repeated measurements of smoking status, SBP, TC, and HDL-C, resulting in a significant improvement in C-statistics. Our study aligns with these findings, demonstrating that incorporating repeated measurements of multiple predictors from EHRs enhances predictive performance when compared with the Cox model which uses only baseline measurements. The temporal information present in repeated measurements is of great importance. It is reflected by the time intervals between measurements and trends or patterns observed over time. While the QResearch study and Paige et al. did not explicitly report the time intervals (or density) of measurements, Paige et al. did fit the temporal trend and dependency of repeated measurements using a multivariate linear mixed-effects model.^12,14 In our study, we observed that the average time intervals between measurements of key predictors were generally <1 year (see Supplementary material online, Table S8), signifying the richness of information that can be harnessed from EHRs. The correlated predictors,²² irregularly missing records,⁵² and data with strong interaction in the EHR necessitate applying a novel modelling approach such as ML, which usually utilizes high-dimensional unstructured data to enhance predictive performance.^53,54 However, it is worth noting that currently off-the-shelf ML algorithms lack a comprehensive approach to model secular trends and dependencies in irregularly structured data. This presents an area for further methodological investigation to effectively harness the temporal nature of the data for CVD risk prediction.

Although it is controversial whether ML can improve cardiovascular risk prediction using only baseline measurements of limited predictors,^27,55 several pieces of evidence have demonstrated that predictive performance can be largely improved when predictors derived from repeated measurements are fed into ML models.^25,26 For instance, Li et al. summarized the repeated measurements of blood lipid, blood pressure, and HbA1c from the EMRs of 101 110 people in a US regional healthcare system, into extremum, number of measurements, means, etc. Then, these longitudinal-derived predictors were used in the random forest ML model, causing large increments in the AUC (e.g. 0.823–0.902).²⁵ Compared with those studies, our study demonstrated that: (i) by embedding XGBoost and LASSO regression algorithms into Cox regression to leverage time-to-event information, similar improvement could result in the parameter of discrimination when evaluated by using C-statistic; (ii) besides discrimination capability, our study comprehensively assessed the performance of the model from the perspectives of calibration and reclassification based on survival probabilities; (iii) it is feasible to conduct CVD risk prediction using rich but irregular existing EHR data for risk stratification without extra cost for screening new markers.

Under real-world scenarios, many predictors are not universally screened in the population. However, it is shown that these markers can predict cardiovascular risk. For example, the mean of FBG presents the long-term control of glucose metabolism, which is predictive for CVD independently.¹⁰ Apolipoprotein B and Lp-(a) are also useful biomarkers for ASCVD.⁷ Poor renal function (e.g. impaired eGFR) can result in hypertension, left ventricular hypertrophy, endothelial dysfunction, dyslipidaemia, and low-grade inflammation.⁵⁶ In our study, these predictors were informative in predicting cardiovascular events as reflected by the importance of the predictors (see Supplementary material online, Figure S5) and the structures of the models (see Supplementary material online, Figure S3 and Table S10). Making the best use of these existing biomarkers in EHR data to enhance CVD risk prediction may change the current way of screening high-risk populations in clinical practice. Considering the irregular nature of the data, ML algorithms can be good alternatives. The ML models could accommodate residents with some unmeasured predictors flexibly. Including a predictor or its repeated measurement in the model does not necessitate requiring complete information on the whole population.

The absolute increment of C-statistic in our study was 0.0113. This gain in the parameter of discrimination was meaningful compared with the gains generated by adding established risk factors. For example, in the Emerging Risk Factors Collaboration study, incorporating C reaction protein or HDL-C into the traditional Cox model to predict ASCVD incidents will increase the C-statistics by 0.0039 or 0.0050, respectively.⁵⁷ When SBP was removed from the Reynolds score in the Women’s Health Study, the change in the C-statistic was 0.01.⁵⁸  C-statistic is an insensitive indicator that ranges from 0.5 to 1.0. The larger the C-statistic is, the more challenging it is to improve it.⁵⁹ HDL-C can increase the C-statistics score only by 0.0013 in our cohort. As advised by Cook,⁵⁸ the improvement in risk prediction provided by the two ML models was also evaluated using NRI and IDI in this study. The reclassification table of the XGBoost model indeed indicated significant net benefit. In the validation set with 25 216 subjects, according to the cut-offs defined by the current Chinese guideline, ∼4% more subjects will be allocated to proper risk groups and correspondingly receive more suitable recommendations on intervention. Assuming that statin therapy was recommended to the high-risk population, which helped to reduce the CVD risk by 20%,⁶⁰ such assessments of individuals by the XGBoost and the refitted China-PAR models could assign 4529 (17.9%) and 5398 (21.4%) patients to initiate statin treatment and help prevent 110 and 117 CVD outcomes over 5 years, respectively. Correspondingly, for every 41 and 46 patients treated, 1 CVD outcome was prevented by using the XGBoost and the refitted China-PAR models. This is consistent with the calibration plot, in which the risk predicted by the XGBoost model was more coordinated to the observed risk than the refitted China-PAR model, especially among the low- or intermediate-risk groups. Such consistency indicates that the XGBoost model may gain benefit under the existing risk cut-off values. Considering the large numbers of the low-risk population, great benefits are likely to be achieved when this model is implemented for risk screening. In the DCA, the threshold probability defined the criteria for intervention in individuals. If the estimated risk exceeded the threshold probability, intervention would be recommended. The net benefit of the XGBoost model was higher than that of the refitted China-PAR model within the threshold probability range of 7.5–12.5%. This range aligns with the typical cut-off risk values recommended by the guidelines for initiating critically important interventions such as statin therapy.^1,2 These results suggest the potential net benefit of implementing the XGBoost model based on the existing risk cut-offs. On the other hand, the risk predicted by LASSO regression may be more suitable for use in high-risk individuals, given its larger net benefit across a range of 12.5–17.5%. Finally, we note that this administrative data–based approach can enhance CVD primary prevention by offering a more accurate prediction without any extra cost for screening new markers.

In the present landscape, most risk prediction models have developed their own implementation tools, some of which are integrated into the health information system (e.g. QRISK in the United Kingdom and PREDICT in New Zealand),^6,12 while others are offered independently through websites or applications (e.g. PCE, SCORE2, and the China-PAR model).^1,28,30 Given the nature of utilizing comprehensive information from EHRs, we recommend implementing the ML model by embedding it within the healthcare information system. This approach can also facilitate automatic population screening, enhancing the sustainability of cardiovascular risk prediction. However, unlike the traditional Cox model, the implementation of an already derived ML model is not always straightforward. Our algorithm for 5-year prediction of CVD risk involved a two-stage process. The first stage utilized ML classification algorithms, while the second stage embedded the ML classifier into a Cox regression model to predict absolute 5-year risk. Therefore, we firmly believe that the baseline survival characteristics of local populations remain crucial for accurate absolute risk prediction. As a result, recalibration of the model may still be necessary when applying it to different populations, along with external validation to assess its performance in diverse settings.

Our study also has several limitations. First, although internally validated, the ML risk prediction models derived in our study were not externally and independently validated. Our study aims not to propose and generalize the ML models to other populations but to answer a methodological question by comparing the performance of two ML approaches to the locally refitted China-PAR models. The models’ relative performance was still valid since the performance was measured by the same scale from the same dataset. Secondly, only two ML methods were present in this study, considering the nature and sample size of the data, the complexity of the algorithm, and the desired model interpretability. Advanced ML methods such as neural networks can be adapted to use data in the future.⁶¹ Thirdly, our study is based on regional data, which may not fully represent the diversity of the Chinese population nationwide. Variations in genetic background, culture, socioeconomic levels, climate, geographic features, lifestyle, and dietary patterns among different ethnic groups within the Chinese population could influence the generalizability of our findings. Nevertheless, the primary objective of our study was to demonstrate the cardiovascular predictive value of repeated measurements using ML models. As such, the potential limitations arising from regional data may have a limited impact on the overall conclusions of this research. Additionally, we acknowledge that the analysis set, consisting of 215 744 Chinese participants, is a subset of the original CHERRY study, which included 1.05 million adults. Consequently, while our findings are informative, they may not fully represent the entire population. Nonetheless, this subset reflects the current clinical practice where lipid measurements are commonly requested, even when using traditional guideline-recommended models. Furthermore, it is important to note that the data source for our study primarily relied on EHRs, which are generally collected from individuals seeking medical care. This approach may lead to biased representations of certain health conditions or risk factors that are more likely to be captured in clinical settings. Novel risk factors, such as apolipoproteins or eGFR, may be particularly affected by this bias, as their availability could be associated with specific patient health conditions and outcomes. However, we mitigated this concern by leveraging ML algorithms, which effectively handled missing data and enabled us to capture valuable information for CVD risk prediction, including the association between the availability of specific markers and disease outcomes. Finally, although the use of summarized statistics to utilize repeated measurements is common, it is also important to model the time trend and consider the temporal dependence of the measurements from a single individual.^16,24 Our study reinforces the importance of incorporating repeated measurements from EHRs in CVD risk prediction. The temporal aspect of repeated measurements adds valuable insights, but challenges remain in fully capturing this information using current ML algorithms. Future research efforts should focus on addressing these methodological limitations to unlock the full potential of EHR data for improved CVD risk assessment. While our study has several limitations, as listed above, we believe that our focus on assessing the cardiovascular predictive value of repeated measurements with ML models remains valuable and contributes to the current understanding of CVD risk assessment.

Conclusions

The irregularly repeated measurements in the EHR could be leveraged to improve the current 5-year ASCVD incident risk prediction by adopting XGBoost or LASSO regression algorithms. The XGBoost model displayed the best overall performance in the parameters of discrimination, calibration, and reclassification. A comprehensive consideration of the importance of predictors in both ML models showed that the average levels of blood glucose, renal function, and Apo B had relatively high predictive values. Real-world repeated measurements of risk factors have the potential to provide additive value for current ASCVD risk assessment.

Supplementary material

Supplementary material is available at European Heart Journal – Digital Health.

Funding

This work was supported by the National Key Research and Development Program of China (grant number 2020YFC2003503) and the National Natural Science Foundation of China (NSFC) (grant number: 81973132).

Data availability

Data in this study were not publicly available due to administrative control. For placing any request, please contact the author directly.