Validation of the general Framingham Risk Score (FRS), SCORE2, revised PCE and WHO CVD risk scores in an Asian population

Summary

Background

Cardiovascular risk prediction models incorporate myriad CVD risk factors. Current prediction models are developed from non-Asian populations, and their utility in other parts of the world is unknown. We validated and compared the performance of CVD risk prediction models in an Asian population.

Methods

Four validation groups were extracted from a longitudinal community-based study dataset of 12,573 participants aged ≥18 years to validate the Framingham Risk Score (FRS), Systematic COronary Risk Evaluation 2 (SCORE2), Revised Pooled Cohort Equations (RPCE), and World Health Organization cardiovascular disease (WHO CVD) models. Two measures of validation are examined: discrimination and calibration. Outcome of interest was 10-year risk of CVD events (fatal and non-fatal). SCORE2 and RPCE performances were compared to SCORE and PCE, respectively.

Findings

FRS (AUC = 0.750) and RPCE (AUC = 0.752) showed good discrimination in CVD risk prediction. Although FRS and RPCE have poor calibration, FRS demonstrates smaller discordance for FRS vs. RPCE (298% vs. 733% in men, 146% vs. 391% in women). Other models had reasonable discrimination (AUC = 0.706–0.732). Only SCORE2-Low, -Moderate and -High (aged <50) had good calibration (X² goodness-of-fit, P-value = 0.514, 0.189, 0.129, respectively). SCORE2 and RPCE showed improvements compared to SCORE (AUC = 0.755 vs. 0.747, P-value <0.001) and PCE (AUC = 0.752 vs. 0.546, P-value <0.001), respectively. Almost all risk models overestimated 10-year CVD risk by 3%–1430%.

Interpretation

In Malaysians, RPCE are evaluated be the most clinically useful to predict CVD risk. Additionally, SCORE2 and RPCE outperformed SCORE and PCE, respectively.

Funding

This work was supported by the Malaysian Ministry of Science, Technology, and Innovation (MOSTI) (Grant No: TDF03211036).

Research in context.

Evidence before this study

The use of risk scores in clinical practice is an attempt to predict individuals at risk and prevent events by various interventions. Numerous algorithms derived from predominantly Caucasian population are utilised in other country's settings in the absence of resource to produce their own risk prediction calculator. We aimed to validate the Revised Pooled Cohort Equations, the updated SCORE2 and WHO Cardiovascular Disease risk charts in an Asian population published online on 3 July 2018, 13 July 2021, and 1 Oct 2019, respectively. We used PubMed search engine to identify articles on the validation of any of the mentioned CVD risk models published up to May 2022, using the following keywords: (“Pooled Cohort Equations” OR “revised PCE” OR “SCORE2” OR “World Health Organization cardiovascular disease risk charts”) AND (“validation” OR “validating”) AND (Asia). Only one study from China have examined the new WHO CVD risk score in the Chinese population and one study from Korea assessed the discrimination of the revised Pooled Cohort Equations in a Korean cohort. No study was found validating the performance of SCORE2 in a mixed Asian population. Additionally, no study has statistically compared the model performance between the new risk models and the older versions.

Added value of this study

Lower- and middle-income nations continue to witness an increasing number of deaths due to CVD. Misclassification of risk profile may contribute to this burden. It is therefore imperative to assess the clinical usability of the new CV risk models in a heterogeneous Asian population, to ensure its effectiveness. Our study presented a comprehensive analysis and comparison of the discrimination and calibration of the revised PCE, SCORE2 and WHO cardiovascular disease risk charts in Malaysia. Our findings suggested that the revised PCE is the most clinically useful model for Malaysia given its good discrimination. Additionally, our results showed that despite the improved in discrimination in the newer models, they are still not well calibrated to the Asian population, and they also demonstrate a high level of misestimation.

Implications of all the available evidence

Our current evidence signifies a crucial need to recalibrate existing models to a new population before use. More importantly, there is an urgent need for the development of a CVD risk estimation tool using primary Malaysian population dataset and also employing modern machine learning algorithms that are proven to produce a more superior risk score model with high discrimination and calibration.

Introduction

Cardiovascular diseases (CVD) contribute to 87% of non-communicable disease (NCD) deaths in the Western Pacific region. In a country like Malaysia with a diverse ethnic composition, the probability of death caused by CVD between individuals 30–70 years old is at around 18% and it is projected that Malaysia is unlikely to be able to reduce NCD deaths by one-third by 2030, as envisioned by the Sustainable Development Goals.¹

The ‘Regional Action Framework for Noncommunicable Disease Prevention and Control in the Western Pacific’ highly recommends screening and stratifying individuals at risk of developing CVD, thus tailoring prevention initiatives particularly the high-risk individuals who will benefit the most from therapeutic intervention.¹ However, existing CVD risk estimation tools were developed based on derivation cohorts not originating from the region, especially Asian countries. Studies have shown that using a non-native risk prediction model may result in inaccuracies when classifying Asian patients' risk because Asians have different CV risk factor profiles and CVD event ratios than Western populations used to derive existing models.^2,3

In an effort to address the limitations of existing risk scores, these risk scores have been revised in the last decade, incorporating newer population data and improved methodology. SCORE2, Revised PCE and WHO CVD risk prediction tools were developed to replace SCORE, PCE, and WHO/ISH risk tools, respectively.4, 5, 6

However, before being implemented in a different population, any new risk prediction tool must be externally validated, which is the process of assessing the model's performance in a different dataset independent from the derivation cohort. This entails evaluating discrimination and calibration. The WHO CVD risk score, for example, was examined in a Chinese population and found to have good discriminatory capability for CVD risk while over-estimating CVD events regardless of gender. The study also found that the WHO CVD risk charts are more sensitive to screening high risk individuals at a 10% cut-off point rather than a 20% cut-off point, emphasising the significance of validating and recalibrating non-native risk prediction models.⁷ Aside from that, very few studies have assessed the performance of the other revised risk models in other populations in the region.

Therefore, the study aims to validate the FRS, SCORE2, RPCE, and WHO CVD risk models and evaluate their model performance in a large heterogeneous population. The measures of validation are discrimination and calibration. We evaluated the performance of the new models to the FRS because it is used in many Asian countries.⁸ The study will also compare the discrimination and calibration of SCORE2 and RPCE to their previous model versions.

Methods

Study data

This study used data from the Responding to Increasing Cardiovascular Disease Prevalence (REDISCOVER) Study. REDISCOVER, which began in 2007, is an ongoing, observational, longitudinal community-based study in Malaysia. Malaysia consists of a heterogenous ethnic population of Indo-China, South Asia and East Asia origins. The REDISCOVER study looked at the associations between social, behavioural, genetic, and environmental factors with cardiovascular disease in rural and urban Malaysians. REDISCOVER enrolled 12,573 participants aged 18 and up from 8 different states across Malaysia and who provided written informed consent. Follow-up averaged 12.05 years. The sampling and follow-up procedures had previously been described.⁹ The information on sex was based on the participant's assigned biological sex. Ethnicity was based on the ethnic and cultural group that the participant belongs in.

Table S1 lists the fatal and nonfatal cardiovascular events recorded in the REDISCOVER Study.

At recruitment, 8.8%, 18.9%, and 7.7% of individuals were on oral hypoglycaemic agent (OHA), anti-hypertensive, and lipid-lowering drugs, respectively. 11.8% had diabetes, 11.8% hyperglycemia, and 25.9% hypertension. Data was retrieved from REDISCOVER based on the inclusion and exclusion criteria of each four risk score prediction models (Fig. 1). An individual can be in multiple groups. Each risk score model was validated using the validation groups accordingly. Table S3 details each risk score validation group's baseline characteristics.

Fig. 1.

Description of the study population and each risk score validation groups. REDISCOVER Responding to Increasing Cardiovascular Diseases Prevalence study cohort, FRS Framingham Risk Score, SCORE2 Systematic COronary Risk Evaluation 2, RPCE Revised Pooled Cohort Equations, WHO CVD World Health Organization cardiovascular disease risk charts, CVD cardiovascular disease.

Cardiovascular risk prediction models

In our study, four different risk prediction models were validated: (1) FRS, (2) SCORE2, (3) RPCE, and (4) WHO CVD. Table S2 summarises the risk prediction models, including information on the models’ variables, endpoints and risk thresholds. Gender, age, and total cholesterol are common variables used in all models.

The CVD risk scores using FRS in our cohort were calculated using the provided Framingham equations for general cardiovascular risk online, derived using a Cox proportional hazards model.¹⁰

The updated SCORE2 has been recalibrated into four different ‘risk regions’ in Europe based on the latest WHO age and sex-standardized overall CVD mortality rates per 100,000 populations, which are (1) low, (2) medium, (3) high and (4) very-high risk regions. In this study, the 10-year CVD risk scores of the SCORE2 validation group were calculated using the model coefficients of each ‘risk region,’ which included risk factors, baseline survival, region, and sex-specific recalibration scales.⁵ As a result, everyone in the SCORE2 validation group was analyzed four times to determine which ‘risk region’ model would be a better fit for our population.

The RPCE tool used logistic regression model with elastic net regularization.⁶ The revised method was used to avoid the previous PCE risk model's issues with ‘over-fitting’ of certain subpopulations.⁶ No RPCE equation exists for other subgroups. Grundy et al. (2019) recommended the use of PCE equation for white to calculate risk estimates in Asians.¹¹ Therefore, for ease of comparisons with the older PCE, we will also use RPCE equation for white to calculate the risk estimates of our population.¹¹

The WHO 10-year CVD risk was calculated using the ‘whocvrisk’ prediction algorithm available in Stata program, with the recalibration region variable set to ‘MYS’ that codes for Malaysia.^4,12 This was done using the Stata/BE 17.0 statistical software. Risk estimates are calculated for both lab-based and non-lab based versions of WHO CVD.

Outcomes of interest for model validation

To ensure an accurate validation, the endpoints of each validation group were defined separately according to the risk models’ designed endpoints.

RPCE and WHO CVD risk charts used the same endpoints as their previous models. RPCE and PCE endpoints are non-fatal myocardial infarction, death from coronary heart disease, or fatal or non-fatal stroke over a 10-year period. Meanwhile, WHO CVD and WHO/ISH endpoints are fatal and non-fatal myocardial infarction or coronary heart disease (ICD10-codes I21-23), fatal myocardial or coronary heart disease (ICD10-codes I24-25), and stroke (ICD10-codes 160-69). However, SCORE2 made one major change by including non-fatal CVD events in their endpoints. REDISCOVER collected 10-year data on fatal and nonfatal CVD events. Cardiovascular events are recorded every three years during follow-up and are adjudicated using hospital records and national death certificates. Table S1 shows the CVD event subtypes and the number of events that were observed in the REDISCOVER Study. Table S1 also shows the outcomes from REDISCOVER that were used as endpoints for each risk model validation.

Cardiovascular disease risk stratification

The cardiovascular risk was classified into three levels: low, intermediate, and high. The cohort was categorized based on model-specific risk cut-off values. An individual is grouped into the high cardiovascular risk with a 10-year risk of ≥20% for FRS and WHO CVD, ≥7.5% for SCORE2 (age <50) and RPCE, and ≥10% for SCORE2 (age 50–69). Low risk was defined as <10% for FRS, <2.5% for SCORE2 (age <50) and <5% for SCORE2 (age 50–69), RPCE and WHO CVD. All other risk values were categorized as the intermediate.4, 5, 6^,10

Missing value analysis and data imputation

The missing variables needed to calculate the risk scores varied between 0.1% and 6.6%. The variable with the highest missing rate is smoking status. Missing value analysis was performed to determine whether they were random or not. It was found that the dataset is missing completely at random (MCAR), where the probability of missingness is independent of the observed or missing data. Multiple imputation was performed using predictive mean matching (PMM). PMM estimates and then imputes missing values based on observed data with similar predictive mean. Thus, PMM is a robust technique as it can create imputed data that preserves the distribution of the original data. The final output is a dataset of 12,573 complete cases. Outcome data had zero missing value. Missing data analysis was done using IBM® SPSS® Statistics Version 26, whereas data imputation was done in RStudio 2022.07.0 + 548.

Model validation

Model validation was performed by assessing the discrimination and calibration of the models. The area under the receiver operating curve (AUROC), sensitivity and specificity of each model were calculated for overall, men and women. The sensitivity and specificity were based on each model's high-risk threshold. A model is considered useful when the total sensitivity and specificity value is more than 1.5 (sensitivity + specificity = 150%).¹³

Additionally, SCORE2 models had different high-risk cut-off values for participants aged 50 and 50–69 years old.⁵ The analysis on discrimination and calibration was performed using IBM® SPSS® Statistics Version 26.

Discrimination

Discrimination is the prediction model's ability to distinguish between individuals who have an event and those who do not. Discrimination was measured by AUROC. An AUROC of 0.5 indicates no discrimination, 0.75–0.92 is good, 0.93–0.96 is very good, and 0.97–1 shows excellent discrimination. AUROC of less than 0.75 is reasonable, however the model may lack clinical value in risk predictions.¹⁴

Calibration

Calibration evaluates whether the proportion of 10-year cardiovascular disease events differs significantly from the predicted. The model calibrations were assessed using the Hosmer–Lemeshow Chi-square goodness-of-fit test with a significance level of 0.05. Calibration plots were also made to graphically assess the model agreement for overall, men and women. Calibration plots were built using Microsoft® Excel® for Microsoft 365 MSO (Version 2301 Build 16.0.16026.20002).

Comparison with older models

The discrimination and calibration of SCORE2 and RPCE was compared to the older version SCORE and PCE, respectively. The characteristics of SCORE (TC-based model) and PCE models were included in Table S2. A paired sample T-test, with a significant level of 0.05 was done to statistically compare the AUC values of the models.⁵

To directly compare SCORE and SCORE2, the AUROC values of both models were calculated twice using both endpoint definitions, as was done in the original SCORE2 development paper.⁵ The two endpoint definitions are 1) fatal and non-fatal CVD events (SCORE2 model endpoint), and 2) fatal CVD events (SCORE model endpoint).¹⁵

Meanwhile, RPCE were compared to PCE using one endpoint (a composite of non-fatal MI, CHD death, fatal and non-fatal stroke) because the RPCE model was derived using the same endpoint as in the previous PCE.

Comparison analysis between WHO CVD model and previous WHO/ISH model was not performed because the regression equations of WHO/ISH model were not readily accessible.¹⁶ Therefore, the absolute WHO/ISH risk probabilities of the study cohort cannot be calculated.

Role of the funding source

The funders played no part in the study's design, data collection and analysis, publication decision, or manuscript writing.

Results

There were 12,573 participants in the REDISCOVER study, aged 18–92 years old. The participants had a median age of 52 years old and age range of 74 years old. A total of 10,145, 7423, 9162 and 9265 participants were eligible for validation analysis using the FRS, SCORE2, RPCE and WHO-CVD risk model, respectively.

The risk profiles and characteristics of the individuals in the four validation groups did not differ largely between each other, except for the SCORE2 validation group which has no diabetes.

Across the groups being studied, the majority were female (between 56% and 57%) with mean age between 51.90 and 54.79 years old. Common CVD risk factors include hypertension (between 22.1% and 26.7%), diabetes (between 11.5% and 12.4%), and hyperlipidemia (between 10.3% and 11.7%). Mean BMI was around 26.0 kg/m². Most of the individuals (between 52.2% and 53.6%) lived in urban areas.

Endpoints were also defined separately in each validation group based on endpoints specific to each model. The 10-year event rates varied from 3.4% in the WHO CVD validation group to 4.6% in the RPCE validation group. The mean predicted 10-year CVD risk for men ranged from 5.9% in SCORE2-Low model to 15.1% in the FRS. Meanwhile for women, the mean CVD risk ranged from 2.7% in the RPCE model to 8.9% in the SCORE2-Very High model (Table 1).

Table 1.

Mean CVD risk scores for each risk model.

	Overall		Men		Women
	n	Mean (SD)	n	Mean (SD)	n	Mean (SD)
FRS	10,145	10.9 (13.6)	4318	15.1 (12.8)	5827	7.8 (13.3)
SCORE2-LOW	7423	4.1 (3.1)	3157	5.9 (3.3)	4266	2.8 (2.0)
SCORE2-MODERATE	7423	5.1 (4.2)	3157	7.6 (4.6)	4266	3.3 (2.6)
SCORE2-HIGH	7423	6.1 (5.4)	3157	8.4 (5.8)	4266	4.4 (4.3)
SCORE2-VERY HIGH	7423	11.0 (8.0)	3157	14.0 (8.2)	4266	8.9 (7.2)
RPCE	9162	4.3 (4.7)	3989	6.4 (5.3)	5173	2.7 (3.3)
WHO CVD (LAB)	9265	7.7 (7.6)	4054	10.1 (9.1)	5211	5.9 (5.5)
WHO CVD (NON-LAB)	9265	7.2 (6.8)	4054	9.2 (8.1)	5211	5.5 (5.1)

Cardiovascular diseases risk stratification

Fig. 2 shows the dispersion of different cardiovascular risk categories in the four risk models. Overall, there is some variability in risk stratification across the models. The percent of individuals categorised as low risk ranged from 10.5% in the SCORE2-Very High model to 61.8% in FRS model. On the other hand, the percent of cohort in the high risk group ranged from 5.2% in WHO CVD (non-lab) to 48.7% in SCORE2-Very High. In all the models except SCORE2-Very High, a large proportion of individuals are low risk and the minority of them are high risk.

Fig. 2.

CVD risk classification of the study population by each risk score model. REDISCOVER Responding to Increasing Cardiovascular Diseases Prevalence study cohort, FRS Framingham Risk Score, SCORE2 Systematic COronary Risk Evaluation 2, RPCEs Revised Pooled Cohort Equations, WHO CVD World Health Organization cardiovascular disease risk charts, CVD cardiovascular disease.

The models estimate 9%–69.4% of men into high risk group, while only 1.2%–44.1% into low risk group. Conversely, only 0.7%–33.5% of women are categorised as high risk but 17.4%–83.3% of women as low risk.

In men, all models except SCORE2-Very High and RPCE, showed similar trends in risk stratification. Most of the men have intermediate to high CVD risk. SCORE2-Very High and RPCE models have more than 50% of individuals in the high risk group. However, in women, majority of the individuals are low risk in all models except for SCORE2-Very High.

Model performances

Overall, FRS and RPCE showed good discrimination, with AUC values of 0.750 and 0.752, respectively. The other models only showed reasonable discrimination, with AUC values ranging from 0.706 to 0.732. RPCE had the highest AUC value for overall participants (AUC = 0.752) (Table 2). For men, SCORE2-Low (age <50) and SCORE2-Moderate (age <50) models showed the highest AUC value of 0.712. For women, RPCE and FRS both had the highest AUC value of 0.762. The RPCE was also more sensitive than FRS (Sensitivity = 67.7, 82.9, 43.5 for overall, men and women, respectively). Generally, all the models displayed poor sensitivity and specificity measures, whereby the total values were less than 1.5.

Table 2.

The discrimination and calibration of the FRS, SCORE2, RPCE and WHO CVD models for 10-year cardiovascular events.

Models	Discrimination				Calibration
	Cut-off	Sensitivity	Specificity	AUROC (95% CI)	χ2 goodness-of-fit	P-value
FRS
Overall	≥20%	42.2	86.4	0.750 (0.728, 0.772)	387.739	<0.001∗
Men	≥20%	50.9	76.7	0.702 (0.672, 0.733)	229.782	<0.001∗
Women	≥20%	28.6	93.3	0.762 (0.726, 0.797)	167.419	<0.001∗
SCORE2-LOW
Age <50 years
Overall	≥7.5%	14.8	98.6	0.732 (0.661, 0.803)	8.198	0.514
Men	≥7.5%	21.6	96.3	0.714 (0.625, 0.803)	9.321	0.408
Women	≥7.5%	0.0	99.9	0.610 (0.497, 0.723)	6.659	0.673
Age 50–69 years
Overall	≥10%	30.1	92.5	0.722 (0.688, 0.755)	17.767	0.038∗
Men	≥10%	43.8	84.1	0.691 (0.646, 0.736)	17.580	0.040∗
Women	≥10%	7.1	99.0	0.711 (0.656, 0.767)	9.891	0.359
SCORE2-MODERATE
Age <50 years
Overall	≥7.5%	29.6	95.5	0.732 (0.661, 0.803)	12.442	0.189
Men	≥7.5%	43.2	88.4	0.714 (0.625, 0.803)	12.381	0.193
Women	≥7.5%	0.0	99.8	0.610 (0.497, 0.723)	7.960	0.538
Age 50–69 years
Overall	≥10%	45.9	93.2	0.721 (0.688, 0.754)	29.304	<0.001∗
Men	≥10%	61.8	66.6	0.691 (0.646, 0.736)	22.530	0.008∗
Women	≥10%	18.8	96.2	0.711 (0.656, 0.767)	13.996	0.122
SCORE2-HIGH
Age <50 years
Overall	≥7.5%	29.6	94.2	0.730 (0.659, 0.800)	13.806	0.129
Men	≥7.5%	43.2	86.0	0.714 (0.625, 0.803)	11.630	0.235
Women	≥7.5%	0.0	99.3	0.610 (0.497, 0.723)	9.927	0.356
Age 50–69 years
Overall	≥10%	56.8	72.1	0.722 (0.689, 0.755)	65.530	<0.001∗
Men	≥10%	66.7	58.0	0.691 (0.646, 0.736)	30.962	<0.001∗
Women	≥10%	40.0	93.2	0.711 (0.656, 0.767)	37.121	<0.001∗
SCORE2-VERY HIGH
Age <50 years
Overall	≥7.5%	57.4	77.7	0.727 (0.657, 0.798)	80.201	<0.001∗
Men	≥7.5%	75.7	53.0	0.714 (0.625, 0.803)	40.954	<0.001∗
Women	≥7.5%	17.6	93.1	0.610 (0.497, 0.723)	42.072	<0.001∗
Age 50–69 years
Overall	≥10%	89.1	35.9	0.719 (0.686, 0.752)	280.886	<0.001∗
Men	≥10%	94.4	19.7	0.691 (0.646, 0.736)	124.545	<0.001∗
Women	≥10%	80.0	48.7	0.711 (0.656, 0.767)	161.522	<0.001∗
RPCE
Overall	≥7.5%	67.7	69.7	0.752 (0.730, 0.774)	136.540	<0.001∗
Men	≥7.5%	82.9	48.3	0.712 (0.683, 0.742)	123.574	<0.001∗
Women	≥7.5%	43.5	85.6	0.762 (0.726, 0.797)	24.326	0.004∗
WHO CVD (Lab based chart)
Overall	≥20%	19.0	93.5	0.721 (0.695, 0.747)	234.025	<0.001∗
Men	≥20%	24.4	88.8	0.676 (0.641, 0.711)	121.86	<0.001∗
Women	≥20%	9.6	97.0	0.727 (0.685, 0.770)	121.964	<0.001∗
WHO CVD (Non-lab based chart)
Overall	≥20%	12.9	95.1	0.706 (0.679, 0.732)	234.025	<0.001∗
Men	≥20%	17.3	91.4	0.662 (0.625, 0.698)	121.86	<0.001∗
Women	≥20%	5.3	97.8	0.711 (0.668, 0.755)	121.964	<0.001∗

The P-value marked with an asterisk (∗) indicates that the difference between the proportion of observed and predicted outcome events are statistically significant.

FRS = Framingham Risk Score; SCORE2 = Systematic COronary Risk Evaluation 2; RPCE = Revised Pooled Cohort Equations; WHO CVD = World Health Organization Cardiovascular Disease Risk Charts.

The SCORE2 models were analyzed in two age groups. In the age group <50 years old, SCORE2-Low and SCORE2-Moderate models had the highest AUC for overall participants (AUC = 0.732). All four SCORE2 models had AUC = 0.714 and AUC = 0.610 for men and women, respectively. In the 50–69 age group, SCORE2-Low and SCORE2-High had the highest AUC for overall participants (AUC = 0.722). Also, all four SCORE2 models had AUC = 0.691 and AUC = 0.711 for men and women, respectively.

Regarding WHO CVD models, the AUC of the lab-based version was statistically different from the non-lab-based version (0.721 vs. 0.706, P < 0.001). However, the probability of accurately identifying high risk individuals using the WHO CVD model was very low (Sensitivity = 19.0% and 12.9% for lab-based and non-lab-based, respectively) at 20% high-risk cut-off value. Our additional analysis found that at 10% cut-off, WHO CVD model has a higher sensitivity (51.8% and 44.4% for lab-based and non-lab-based, respectively) but lower specificity (24.1% and 21.1% for lab-based and non-lab-based, respectively). Statistically, the calibration of the SCORE2-Low, SCORE2-Moderate and SCORE2-High models (age <50) were good for both men and women (Table 2). Graphically, only SCORE2-Low (age <50) showed good agreement between predicted and observed CVD events for men and women, but not SCORE2-Moderate and SCORE2-High models (age <50) (Fig. S3; Fig. S4). Statistically and graphically, SCORE2-Low and SCORE2-Moderate models (age 50–69 years) demonstrate good calibration for women only. However, these results have low statistical power due to small number of observed events (less than 100) in the overall age <50 group and women aged 50–69 group (Table 3). Additionally, calibration plot of SCORE2-Moderate (age 50–69) models showed good calibration in men. In RPCE, there was good agreement visually, but poor calibration statistically with χ² goodness-of-fit = 136.540, P < 0.001. The rest of the other models showed poor calibration statistically and graphically, whereby the proportion of predicted events are significantly different from the proportion of observed events.

Table 3.

Predicted and observed events for each risk score.

Models	aPredicted Events, n (%)	Observed Events, n (%)	Signed Absolute Difference	bDiscordance, %
Men
FRS	1082 (25.00)	271 (6.28)	18.72	298
SCORE2-Low (age <50)	47 (4.16)	37 (3.27)	0.89	27
SCORE2-Low (age 50–69)	363 (17.91)	144 (7.10)	10.81	152
SCORE2-Moderate (age <50)	143 (12.65)	37 (3.27)	9.38	286
SCORE2-Moderate (age 50–69)	718 (35.42)	144 (7.10)	28.32	399
SCORE2-High (age <50)	542 (47.96)	37 (3.27)	44.69	1364
SCORE2-High (age 50–69)	887 (43.76)	144 (7.10)	36.66	516
SCORE2-Very High (age <50)	169 (14.96)	37 (3.27)	11.69	357
SCORE2-Very High (age 50–69)	1649 (81.35)	144 (7.10)	74.25	1046
RPCE	2141 (53.67)	257 (6.44)	47.23	733
WHO CVD (Lab based)	480 (11.84)	197 (4.86)	6.98	144
WHO CVD (Non-Lab based)	366 (9.03)	197 (4.86)	4.17	86
Women
FRS	430 (7.38)	175 (3.00)	4.38	146
SCORE2-Low (age <50)	1 (0.06)	17 (0.95)	0.89	−94
SCORE2-Low (age 50–69)	30 (1.21)	85 (3.42)	2.21	−65
SCORE2-Moderate (age <50)	3 (0.17)	17 (0.95)	0.78	−82
SCORE2-Moderate (age 50–69)	107 (4.31)	85 (3.42)	0.89	26
SCORE2-High (age <50)	13 (0.73)	17 (0.95)	0.22	−23
SCORE2-High (age 50–69)	439 (17.67)	85 (3.42)	14.25	417
SCORE2-Very High (age <50)	126 (7.07)	17 (0.95)	6.12	644
SCORE2-Very High (age 50–69)	1301 (52.35)	85 (3.42)	48.93	1430
RPCE	790 (15.27)	161 (3.11)	12.16	391
WHO CVD (Lab based)	162 (3.11)	114 (2.19)	0.92	42
WHO CVD (Non-Lab based)	119 (2.26)	114 (2.19)	0.07	3

FRS = Framingham Risk Score; SCORE2 = Systematic Coronary Risk Evaluation 2; RPCE = Revised Pooled Cohort Equations; WHO CVD = World Health Organization Cardiovascular Disease Risk Charts.

^aCut-off values used to calculate predicted number of events: FRS, ≥20%; SCORE2 (age <50), ≥7.5%; SCORE2 (age 50–69), ≥10%; RPCE, ≥7.5%; WHO CVD, ≥20%.

^bPercentage discordance calculation: ([{predicted percentage—observed percentage}/observed percentage] x 100).

Furthermore, most of the models were found to overestimate the intended outcome in men by 27%–1364%, and in women by 3%–1430%. Only SCORE2-Low (women, both age groups), SCORE2-Moderate (women, age <50) and SCORE2-High (women, age <50) underestimated CVD risks in women. SCORE2-Low (age <50) and WHO CVD (non-lab) were the models with the least discordance in men (27%) and women (3%), respectively (Table 3).

Other than that, Table 4 shows the comparison between the old and new models. It was found that the new SCORE2 model had significantly higher AUC values than previous SCORE models, when analysed using two different endpoint definitions: SCORE2 endpoint (fatal and non-fatal CVD events) (difference in AUC = 0.008, 95% CI 0.007995, 0.008005; P < 0.001) and SCORE endpoint (fatal CVD events only) (difference in AUC = 0.004, 95% CI 0.003993, 0.004007; P < 0.001 and AUC = 0.007, 95% CI 0.006993, 0.007007; P < 0.001). RPCE models also had significantly higher AUC values compared to the previous PCE (difference in AUC = 0.206, 95% CI 0.2059, 0.2061; P < 0.001). Calibration test showed only SCORE-Low (men) and SCORE2-Low (men and women) models had good calibration (Table 5).

Table 4.

Comparison of SCORE-LOW with SCORE2, SCORE-HIGH with SCORE2, and PCE with RPCE discrimination in the REDISCOVER dataset.

	Models		Difference: SCORE2—SCORE	P-value
	SCORE-LOW	SCORE2
Overall. N = 7423
AUROC (95% CI) (SCORE2 endpoint)a	0.747 (0.719, 0.776)	0.755 (0.726, 0.783)	0.008 (0.007995, 0.008005)	<0.001∗
AUROC (95% CI) (SCORE endpoint)b	0.796 (0.761, 0.831)	0.800 (0.765, 0.835)	0.004 (0.003993, 0.004007)	<0.001∗
Men. N = 3157
AUROC (95% CI) (SCORE2 endpoint)a	0.706 (0.668, 0.745)	0.717 (0.679, 0.754)	0.011 (0.01098, 0.01102)	<0.001∗
AUROC (95% CI) (SCORE endpoint)b	0.770 (0.723, 0.818)	0.780 (0.733, 0.828)	0.010 (0.009979, 0.01002)	<0.001∗
Women. N = 4266
AUROC (95% CI) (SCORE2 endpoint)a	0.729 (0.677, 0.782)	0.739 (0.689, 0.789)	0.010 (0.009983, 0.01002)	<0.001∗
AUROC (95% CI) (SCORE endpoint)b	0.787 (0.724, 0.850)	0.793 (0.732, 0.853)	0.006 (0.00598, 0.00602)	<0.001∗
	SCORE-HIGH	SCORE2	Difference: SCORE2—SCORE	P-value
Overall. N = 7423
AUROC (95% CI) (SCORE2 endpoint)a	0.747 (0.719, 0.776)	0.755 (0.726, 0.783)	0.008 (0.007995, 0.008005)	<0.001∗
AUROC (95% CI) (SCORE endpoint)b	0.793 (0.761, 0.831)	0.800 (0.765, 0.835)	0.007 (0.006993, 0.007007)	<0.001∗
Men. N = 3157
AUROC (95% CI) (SCORE2 endpoint)a	0.707 (0.669, 0.745)	0.717 (0.679, 0.754)	0.01 (0.009983, 0.01002)	<0.001∗
AUROC (95% CI) (SCORE endpoint)b	0.770 (0.723, 0.818)	0.780 (0.733, 0.828)	0.01 (0.009979, 0.01002)	<0.001∗
Women. N = 4266
AUROC (95% CI) (SCORE2 endpoint)a	0.730 (0.677, 0.782)	0.739 (0.689, 0.789)	0.009 (0.008983, 0.009017)	<0.001∗
AUROC (95% CI) (SCORE endpoint)b	0.786 (0.724, 0.849)	0.793 (0.732, 0.853)	0.007 (0.00698, 0.00702)	<0.001∗
	PCE	RPCE	Difference: RPCE—PCE	P-value
Overall. N = 9162
AUROC (95% CI) (PCE and RPCE endpoint)c	0.546 (0.516, 0.576)	0.752 (0.730, 0.774)	0.206 (0.2059, 0.2061)	<0.001∗
Men. N = 3989
AUROC (95% CI) (PCE and RPCE endpoint)c	0.674 (0.643, 0.706)	0.712 (0.683 0.742)	0.038 (0.03797, 0.03803)	<0.001∗
Women. N = 5173
AUROC (95% CI) (PCE and RPCE endpoint)c	0.329 (0.286, 0.372)	0.762 (0.726,0.797)	0.433 (0.4329, 0.4331)	<0.001∗

Discrimination analysis using the AUROC at 10-years. The AUROC values of the models are statistically compared using the two-tailed Paired T-test. The P-value denoted by an asterisk (∗) indicates that the difference in AUROC values between the two models in comparison is statistically significant.

^aSCORE2 endpoint: fatal and non-fatal CVD events.

^bSCORE endpoint: fatal CVD events.

^cPCE and RPCE endpoint: non-fatal MI, CHD death, fatal and non-fatal stroke.

Table 5.

The calibration of the SCORE, SCORE2, PCE and RPCE models for 10-year cardiovascular events.

Models	Calibration
	χ2 goodness-of-fit	P value
Overall
SCORE-Low	34.101	<0.001∗
SCORE-High	47.477	<0.001∗
SCORE2-Low	17.691	0.039∗
SCORE2-Moderate	34.857	<0.001∗
SCORE2-High	69.254	<0.001∗
SCORE2-Very High	358.372	<0.001∗
PCE	131951.135	<0.001∗
RPCE	136.54	<0.001∗
Men
SCORE-Low	6.736	0.665
SCORE-High	42.654	<0.001∗
SCORE2-Low	14.291	0.112
SCORE2-Moderate	23.868	0.005∗
SCORE2-High	32.421	<0.001∗
SCORE2-Very High	158.555	<0.001∗
PCE	194.094	<0.001∗
RPCE	123.574	<0.001∗
Women
SCORE-Low	31.184	<0.001∗
SCORE-High	19.549	0.021∗
SCORE2-Low	10.993	0.276
SCORE2-Moderate	17.137	0.047∗
SCORE2-High	42.969	<0.001∗
SCORE2-Very High	203.337	<0.001∗
PCE	335780.400	<0.001∗
RPCE	24.326	0.004∗

The P value marked with an asterisk (∗) indicates that the difference between the proportion of observed and predicted outcome events are statistically significant.

Discussion

The dataset used for this study has several advantages compared to other validation studies done in Asia. REDISCOVER dataset is a countrywide sample community dataset that portrays Southeast Asia's diverse population. It includes 10-year data on fatal and nonfatal cardiovascular events. This allowed us to accurately validate existing risk scores.

According to this study, FRS and RPCE has good discriminating power and can accurately predict CVD risk in the Malaysian population. However, both FRS and RPCE had poor calibration. The SCORE2 and WHO CVD models demonstrated reasonable discrimination and poor calibration, except for SCORE2-Low, SCORE2-Moderate, and SCORE2-High models, which had good calibration for women and for men aged <50. The study's low number of cardiovascular events recorded may have contributed to poor calibration.

RPCE

RPCE was shown to be most clinically useful in predicting CVD risk in Malaysians due to its good discrimination of AUC 0.752. Our comparison analysis revealed that the older PCE from 2013 performed worse than the RPCE (Table 4). Chia et al. (2014) also found that PCE had only modest discrimination (AUC = 0.63) in the Malaysian population.¹⁷ Our findings imply that the RPCE improved the accuracy of predicting ASCVD risk in Asians. The improvement in model performance shown in our study could be attributed to the inclusion of a modern derivation study cohort that better represents other subpopulations.⁶

The present study shows that the revised version (RPCE) statistically improved 10-year ASCVD risk discrimination for Asian men and women. In Korea, RPCE had an AUC of 0.748 for men and 0.808 for women (Table S4). The Korean study also showed that the discrimination of RPCE improved compared to the previous PCE, which is similar to the findings of the study.¹⁸ Besides that our study cohort's RPCE risk distribution trend is similar to that in Haiti. A high proportion of the Haiti cohort (60.4%) is in the low risk group. Meanwhile, only 27.4% of Haitians is at high risk.¹⁹ Despite the high prevalence of CVD risk factors, most of the population is low risk. This may be due to the substantial number of individuals already on OHA and lipid-lowering drugs at baseline.

FRS

The FRS model's performance has been extensively validated in several countries (Table S4). A Malaysian study by Chia et al. (2015) found moderate discrimination of FRS (AUC = 0.63) and good calibration (Hosmer–Lemeshow test χ² = 3.25, P-value = 0.78) in the Malaysian population.²⁰ In China, the AUC of the FRS ranged from 0.72 to 0.79 for men and 0.74–0.79 for women.^21,22 In Korea, FRS has acceptable discrimination but poor calibration as well (Men: AUC = 0.730, χ² = 177.71, P-value = 0.001; Women: AUC = 0.726, χ² = 24.70, P-value = 0.002).²³ The slight variability in AUC values of FRS across different populations is due to population specific differences of the validation cohorts, therefore it is important to review multiple validation studies to determine the model's predictive performance. Hence, in general, FRS is found to perform well in most Asian regions, with AUC ranging from 0.7 to 0.8, implying that FRS is appropriate for Asian population.^24,25

WHO CVD

With both the lab and non-lab versions of the WHO CVD models, moderate discriminative ability was seen in the Malaysian population. The model's low sensitivity can be problematic since it results in many high-risk individuals not receiving proper preventative care. WHO did not specify region-specific high risk thresholds. However, our additional analysis at 10% cut-off value shows an increase in sensitivity of the model: from 19.0% to 51.8% in lab version, and from 12.9% to 44.4% in non-lab version. This finding is similar to previous validation study of WHO CVD model in China, which confirms that WHO CVD model is more sensitive to screening high risk Asians at a lower cut-off point.⁷ This highlights the importance of recalibrating a risk prediction model to a local population data and finding a suitable cut off point to maximise the model's sensitivity and sensitivity before the risk charts can be adapted locally.

According to Selvarajah et al. (2014), the previous WHO/ISH model shows a more modest discrimination (AUC = 0.613) in Malaysian population (Table S4).²⁶ The improved AUC value could be due to the WHO CVD model being derived from actual participant data.⁴

China is the only country where WHO CVD risk models have been validated (Table S4). The WHO CVD risk models also had poor agreement when calibration was done in the Chinese dataset, which was similar to what was found in our population.⁷ The moderate model performance observed here could be due to the WHO CVD derivation data, which included cohorts from high-income nations that were not representative of most Asian population regions.

SCORE2

Regarding SCORE2 validation, this is the first study to assess the SCORE2 model in an Asian region and to compare its performance to the earlier SCORE model. We discovered that the SCORE2-Low, -Moderate, and -High models were unable to appropriately categorise high-risk women under the age of 50 (Sensitivity = 0%), making them the least suited model for Malaysian women. SCORE2-Very High appeared to be slightly more sensitive for both age groups, but the poor calibration indicated that the risk model would need to be recalibrated before it could be employed in a different population. In addition, our study found that the SCORE2 model outperformed SCORE in predicting CVD risk in an Asian population. The improved discrimination power might be due to several reasons: 1) the use of an updated and more contemporary cohort and CVD rates for model development, 2) the inclusion of non-fatal CVD events as the outcomes, and 3) the inclusion of total cholesterol and HDL-cholesterol in deriving the risk score.⁵ Published literature emphasizes that a model with multiple lipid measures may be statistically better and outperform models with only a single lipid index.²⁷

Other study findings

Previous studies have confirmed the poor calibration of the older version of PCE, SCORE and WHO/ISH in Malaysians and other Asian populations.^8,26 Our study is the first to analyse the discrimination and calibration of the new models in an ethnically heterogenous Asian population. Despite the better model discrimination, our analysis further shows that the new models lack good calibration. The CVD risk overestimation might due to the high proportion of individuals already on risk-lowering medicines at baseline, which can reduce the person's chances of experiencing a CVD event during the study. Selection of healthy cohort at baseline could also contribute to poor risk estimation. Additionally, family history, lifestyle and environmental risk factors that could be associated with CVD are not accounted for in the present models, further playing a part in the misestimation in Asians. Although models with good discrimination power may be useful, model adjustments to population-specific baseline hazard should be performed before using them in clinical care.

Moreover, in comparison to the original models, the models in this study generally exhibited poorer performance in Malaysian population. For example, RPCE in its original US cohorts has AUC of 0.780 for men, and 0.850 for women (Table S2). Meanwhile, RPCE in Malaysian population shows AUC of 0.712 for men, and 0.762 for women. The large difference in discrimination is evidence that a model developed using Western cohort would have reduced discrimination power when applied to Asian populations.

Furthermore, our findings also demonstrate sex disparities on model performance, mainly discrimination. In men, SCORE2-Low (age <50) and SCORE2-Moderate (age <50) models display better discrimination, whereas FRS and RPCE perform best for women. The superior model performance of FRS and RPCE in women can be influenced by the strength of the baseline model and the inclusion of risk factor that is more highly associated with CVD events in women compared to men.

Firstly, FRS and RPCE were derived by using participants with ages until 79 years old.^6,10 Published literature has shown that women have a much later onset of CVD, around 65–74 years old.²⁸ In this study, the validation groups for FRS and RPCE includes 21.5%–25.8% of women aged 60–79 years old, compared to other validation groups without participants above aged 70 (Table S3). Therefore, the baseline model of the FRS and RPCE are better at predicting CVD event for women as it can account for the later timing of CVD incidence in women.

Secondly, the inclusion of diabetes as the risk predictor in FRS and RPCE improves model discrimination for women because diabetes is more associated with the risk of coronary heart disease and stroke in women compared to men.²⁹ Thus, diabetes as a model variable improves the CVD risk prediction for women.

Recommendations

Based on the evidence from this study, we would recommend RPCE for CVD primary prevention strategy in Malaysian patients aged 40–79 years old without history of CVD. However, risk estimates should be interpreted with caution as the model tend to overestimate CVD risk in Malaysians, leading to over-testing and overtreatment. The initiation of preventive therapies relies heavily on accurate CVD risk estimation, and it involves the need to balance the risks and benefits of medications such as statin in high-cholesterol management. For example, statin is a drug that despite being able to prevent cardiovascular events, has a risk of causing internal bleeding. The use of a risk score model that overestimates risk can potentially expose some individuals to these dangerous side effects, thus doing more harm than good. Hence, there is an urgent need to recalibrate the model using large local population and, ultimately to develop a local risk prediction model with better accuracy to guide patient CVD risk management.

Study limitations

Our study has several limitations. Firstly, the use of PMM as a data imputation method might be limiting when donor sparseness exists, where there are few available data with a similar predictive mean to the missing data. When incomplete variables have few available real values, the imputed values may be inaccurate. However, our dataset has a small missingness of less than 7% and they are MCAR. Therefore, we can assure that the produced imputed dataset is close the actual dataset.

Secondly, follow-up outcomes were self-reported during follow-ups. Since follow-ups are done every three years, participants may forget or leave out past events. Thus, this may result in inaccurate observed event counts. To reduce recall bias, interviewers would ask detailed questions. Additionally, any events or outcomes reported by the participants would be validated and adjudicated using medical records and national death certificates.

Another limitation is that our dataset may not have enough CVD events to ensure sufficient statistical power in calibration testing. The external validation sample should have at least 100 events and 100 non-events, but above 200 is better.³⁰ The proportion of observed CVD events and non-events in the study are provided in Table 3 and Table S3. All validation groups fulfil the minimum number required, except for SCORE2 (age <50) and SCORE2 (women, age 50–69) models. Future studies will look into acquiring more data for these age groups so further validation analysis can be conducted.

Next, the uncalibrated SCORE risk estimates were not calculated because only the region-specific regression equations are published.¹⁵ Therefore, comparison was done between SCORE2 with SCORE-Low and SCORE-High, separately.

Additionally, the threshold for high risk for the RPCE model was still under much debate. Cut-off value of ≥7.5% was used because it was the more common definition of 10-year CVD risk.

Lastly, a direct head-to-head comparison between the risk models was not done due to the models having slightly different endpoints. Because of this, consistent statistical comparisons are required to avoid outcome selection bias, and this would be beneficial in future analysis.

Conclusion

According to the findings of this validation study, the FRS (AUC = 0.750) and RPCE (AUC = 0.752) reported similar AUC value of 0.750 and above, which considered useful in clinical setting and thus can be applied to predict CVD risk in Asian population. Both FRS and RPCE has good discrimination but poor calibration however FRS demonstrates much smaller discordance for FRS vs. RPCE (298% vs. 733% in men, 146% vs. 391% in women). In terms of sensitivity and specificity, all the risk scores analysed in this study has demonstrated poor sensitivity and specificity value in Asian population. Further studies are required to recalibrate the models to our population and improve their calibration measure. SCORE2 and RPCE performance also improved when compared to SCORE and PCE, respectively. The present study demonstrated that employing an updated model, such as the RPCE and SCORE2, provides a better CVD risk stratification. However, accurate prognosis necessarily requires a prediction model that not only discriminates well between those who have an event and those who do not, but is also well calibrated, ensuring accurate absolute risk estimations. Our study shows that a prediction model could have good discrimination but poor calibration, and vice versa. In these circumstances, good discrimination should take precedent, where the use of a model with poor discrimination should be avoided. This is because a model with good discrimination, but poor calibration can always be recalibrated to the desired population. Therefore, in the absence of a local risk model, RPCE is the most clinically useful tool available. This further emphasises the significance of assessing a risk prediction tool before applying the model to a different demographic.

Contributors

S.S.K. conceived the article content and structure, acquired funding, and supervised the study's planning and execution. S.S.K. and N.I. carried out the literature search. N.M.N. and A.S.R. collected and curated the original study data. N.I. and K.S.I. cleaned and pre-processed the data. N.I. performed the formal analysis, data analysis and visualisation. Data and results interpretation was done by S.S.K., N.I. and S.M.. S.M., M.F.A. and C.S. assisted in data analysis and validated the study results. S.S.K., N.I., S.M. and N.M.N. prepared the original draft. K.S.I., Y.C.C., A.S.R. and K.N. provided feedback, critical revisions, and manuscript edits.

Data sharing statement

The data used as the validation data in this study are obtained from the Centre for Translation Research and Epidemiology (CenTRE), Faculty of Medicine, Universiti Teknologi MARA (UiTM) Sungai Buloh, Malaysia. However, the availability of these data is restricted and therefore are not publicly available. The access to the data may be granted upon request to the corresponding author.

Declaration of interests

S.S.K. received grant from Malaysian Ministry of Science, Technology, and Innovation (MOSTI). N.I. received grant from Malaysian Ministry of Science, Technology, and Innovation (MOSTI). Y.C.C. has received investigator-initiated research grant from Pfizer, unrestricted educational grant from Omron, Viatris, Medtronic (on behalf of MyWASSH (Msian society for World Action on Salt, Sugar and Health)), speaker honorarium from Pfizer, Omron, Viatris, Medtronic, Xepa-Sol, Boehringer-Ingelheim, Bayer, Novartis, MSD, AstraZeneca; sponsorship to conferences from Pfizer, Omron, Viatris, Medtronic, Boehringer-Ingelheim, Bayer, Astra-Zeneca; honorarium for Participation on a Data Safety Monitoring Board or Advisory Board from Novartis; sponsorship to Ambassador meetings from Medtronic since Y.C.C. was on the Ambassador program. Y.C.C. was past president and current council member of Malaysian Society of Hypertension, member of Asia Pacific regional Advisory Group of International Society of Hypertension, member of writers of the Malaysian Clinical Practice Guidelines on Hypertension, senior member of network group of hypertension experts/specialists from Asia in the HOPE-Asia Network, and founding president who set up society in January 2022 to do salt advocacy work in the Malaysian Society for World Action on Salt Sugar and Health. Y.C.C. received BP devices for BP Screening worldwide, an initiative of the International Society of Hypertension May Measurement Month. K.N. received grants from NSW Government, National Heaty Foundation of Australia, and Takeda; honoraria from Daiichi-Sankyo, Ohtsuka Seiyaku, and GE; support for attending meeting from Phillips.