Long-term prediction of major coronary or ischaemic stroke event in a low-incidence Southern European population: model development and evaluation of clinical utility

Abstract

Objective

To develop a long-term prediction model of first major cardiovascular event and to assess its clinical utility in a low-incidence European population.

Setting

Four independent population-based cohorts enrolled between 1986 and 1993 in Northern Italy.

Participants and methods

N=5247 35-year-old to 69-year-old men and women free of cardiovascular disease at baseline. Absolute 20-year risk of first fatal or non-fatal coronary or ischaemic stroke event (monitoring trends and determinants in cardiovascular disease (MONICA) validated) was estimated from gender-specific Cox models.

Main outcome measures

Model discrimination (area under the receiver operating characteristic (ROC)-curve, AUC). ‘High-risk’ subjects were identified based on several threshold values for the 20-year predicted risk. Clinical utility was defined in terms of fraction of missed events (events among those considered at low-risk) and unnecessary treatment (false:true positive ratio). A net benefit curve was also provided.

Results

Kaplan-Meier 20-year risk was 16.1% in men (315 events) and 6.1% in women (123 events). Model discrimination (AUC=0.737 in men, 0.801 in women) did not change significantly as compared to 10-year prediction time interval. In men, with respect to risk stratification based on the number of risk factors, a 20% predicted risk cut-off would miss less events (36% vs 50%) and reduce unnecessary treatment (false:true positive ratio 2.2 vs 3.0); the net benefit was higher over the whole range of threshold values. Similar considerations hold for women.

Conclusions

Long-term prediction has good discrimination ability and is clinically useful for risk stratification in primary prevention. A clinical utility analysis is recommended to identify the optimal stratification according to different public health goals.

Introduction

Current European and American guidelines for the primary prevention of major coronary and stroke events recommend the use of a multivariable risk prediction model to identify high-risk subjects.^{1 2} Several risk scores are available in different US^{3 4} and European⁵ populations of middle-aged adults, including the Italian one,⁶ to estimate the risk of first fatal and non-fatal cardiovascular event over a 10-year time interval. Primary prevention, however, has been recently moved towards the concepts of ‘lifetime’⁷ and ‘long-term’ risks,⁸ motivated also by the increasing life expectancy in western countries. To this extent, 10-year risk prediction models are inadequate to distinguish between those at both low short-term and long-term risks, and those at low short-term but at elevated long-term risk due to the presence of non-optimal risk factors levels.^9–11 In the Framingham Study population, an unfavourable risk factor profile led to an increased 30-year risk of first cardiovascular event, independently on the age at the risk factors assessment.¹⁰ In a representative sample of the Italian population, about 80% of individuals classified at low 10-year risk had increased lifetime risk according to US definition (≥40%), potentially leading to a consistent number of unprevented events that might have been prevented if lifetime risk had been considered.¹¹ This group was largely composed of women and young individuals suggesting that long-term prediction models for risk stratification may be even more beneficial in populations at low incidence of cardiovascular disease (CVD).¹² To this extent, the development of a specific long-term risk prediction should be preferred with respect to recalibration of risk models derived in high-incidence countries.¹³ However, extending the range of risk prediction is not a straightforward operation. Although several studies have shown that a single measurement of risk factor is predictive of future events after 30 plus years,^{10 14} behavioural changes and risk factors modification may affect model discrimination. High-quality follow-up data, with a consistent event definition and validation over time, are also required. Finally, subjects' stratification in risk categories is often based on arbitrary cut-points of absolute risk¹⁵ which may show no benefit in clinical practice.¹⁶ The evaluation of the clinical benefit of long-term prediction by means of some standard measure¹⁷ has not been provided so far and is therefore required.¹⁸

The aim of the present article is to develop a 20-year risk score equation in a European population of men and women considered at low incidence of major cardiovascular events. In addition to standard model calibration and discrimination tools, we evaluate the clinical utility of the model for risk stratification.

Methods

Study population

The Brianza population comprises residents in 173 municipalities in the area between Milan and the Swiss border, Northern Italy. The CAMUNI (CArdiovascular Monitoring Unit in Northern Italy) study includes four independent population surveys carried out between 1986 and 1994 as part of either the WHO MONItoring of trends and determinants in CArdiovascular disease (MONICA) Project (3 surveys; 18) or the Pressioni Arteriose Monitorate E Loro Associazioni (PAMELA) study.¹⁹ Participation rates were 70.1%, 67.2% and 70.8% for the three MONICA surveys, respectively, and 64% for the PAMELA Study, with no differences between men and women. The baseline screening as well as the follow-up for all the surveys were approved by the ethical committee of the Monza Hospital.

Baseline assessment of risk factors

Cardiovascular risk factors were collected at baseline strictly adhering to the standardised procedures and quality standards of the WHO-MONICA Project.²⁰ Height and weight were measured on participants without shoes and wearing light clothing. Trained technicians collected blood pressure at right arm on participants in sitting position and at rest, using a standard mercury sphygmomanometer equipped with two side cuff bladders, for normal and obese participants. Systolic and diastolic blood pressures were assessed twice, at 5 min apart, recording the first and fifth phase of the Korotkoff sounds. The study variable for systolic blood pressure is the average of the two measurements. Venous blood specimens were taken from the antecubital vein on fasting participants (12 h or more). Serum total cholesterol, high-density lipoprotein (HDL)-cholesterol and blood glucose were determined using the enzymatic methods; HDL-cholesterol fraction was separated using the phosphotungstate-Mg⁺⁺ method.²⁰ A standardised interview was administered to participants by trained interviewers. Information on the use of antihypertensive treatment in the last 2 weeks was dichotomised as yes/no; similarly, cigarette smoking habit was dichotomised as current versus past/never smokers. Diabetes mellitus was defined using self-reported diagnoses, information on insulin and oral hypoglycaemic treatments and fasting blood glucose exceeding 7 mmol/L (126 mg/dL). The presence at baseline of a previous history of myocardial infarction (MI), unstable angina pectoris, cardiac revascularisation or stroke was defined based on self-reported information.

Study endpoint and follow-up procedures

The study endpoint is defined as the occurrence of first major coronary event (MI, acute coronary syndrome and coronary revascularisation) as well as for first ischaemic stroke or carotid endarterectomy, fatal and non-fatal.¹³ Data completeness for fatal events was assured through a systematic collection of death certificates provided by local health units; vital status and death certificates were available for 99% of the participants. Suspected out-of-hospital deaths were investigated through interviewing relatives. Suspected hospitalised coronary (International Classification of Diseases 9th edition (ICD-9) discharge code 410 or 411 and ICD-9 Clinical Modification (CM) 36.0–9 for coronary revascularisation) and stroke events (ICD-9 430–432, 434, 436; ICD-9 CM 38.01–39.22 or 39.50–39.52 with at least one 430–438 as discharge code, for carotid endarterectomy) were identified through deterministic and probabilistic record linkages with regional hospital discharge databases, obtaining a satisfactory performance in case finding, as reported.^{18 21} All acute events were investigated and validated according to the MONICA diagnostic criteria²⁰; the ischaemic subtype for stroke was attributed after review of the available clinical information.

Statistical analysis

Our 20-year risk prediction model is based on gender-specific Cox regression models with age, total cholesterol, HDL-cholesterol, systolic blood pressure, antihypertensive treatment, cigarette smoking and diabetes. These predictors are core risk factors included in the CUORE Project^{6 13} as well as in other 10-year risk equations.^{3 4} After a preliminary check on linearity, total cholesterol and HDL-cholesterol were included in the model as categorical variables in four standard classes.^{4 22} The interaction between systolic blood pressure and antihypertensive treatment was not statistically significant (p value=0.84 in men and 0.12 in women, respectively). There was no evidence of any cohort effect in the full model, in men (3df test p value=0.2) nor in women (p value=0.5). Finally, no violations in the proportional hazard assumption were observed using a standard test for time-dependent variables.

Model calibration was assessed through the Grønnesby-Bogan goodness-of-fit test.²³ The area under the receiver operating characteristic (ROC) curve (AUC), as well as sensitivity and specificity in the top and bottom predicted risk quintiles, were computed taking censorship into account.²⁴ Correction for over-optimism and CIs for the AUC were obtained through 1000 bootstrapped samples.²⁵ To assess the hypothesis of a loss in discrimination ability due to a longer prediction period, we estimated the 10-year predicted probability of events in our database, using the same set of risk factors but with shorter follow-up period, that is, up to the end of 2002 for all the participants (number of events 234 in men, 79 in women). We then compared the AUC of the two models by looking at their respective bootstrapped CIs. To assess the clinical utility of the long-term model for risk stratification, we considered two different public health goals. One is to decrease the number of events occurring among those considered at ‘low risk’. If we assume that a subject classified at ‘high risk’ will be targeted for prevention (either lifestyle intervention or treatment), any event occurring outside this category is ‘not-identified’ or ‘missed’ by the prevention strategy. The second strategy aims instead to reduce unnecessary treatment, by decreasing the number of non-events among those considered at ‘high risk’. Under the two scenarios, ‘high-risk’ subjects are defined as those with predicted risk above a certain cut-off value. Clinical utility is defined in terms of: (1) fraction of ‘missed’ events; (2) probability of event among those classified at high risk and (3) false positive/true positive ratio, for several threshold values in the 20-year predicted risk. We also provide a decision curve analysis based on the net benefit—Net benefit=(true positives−w×false positives)/n, where n is the sample size and the weight w represents the ratio between the harm of unnecessary treatment and the harm of missing a case at that given value of predicted risk.¹⁷ All the analyses were conducted using the SAS software V.9.2.

Results

N=5426 (2703 men) participants were enrolled in the age range 35–69 years. N=205 participants (3.8%; n=14 events) had at least one missing data; we considered data imputation (R transcan function²⁶) and excluded only those with missing values in more than four covariates of interest (n=6 men and n=3 women). Finally n=120 men and n=45 women with a positive history of CVD at baseline were also excluded, reducing the sample size to 2574 men and 2673 women.

Baseline characteristics of the study population, by gender, are shown in table 1. During a median follow-up time of 15 years (IQR 12–20), we observed 315 first CVD events in men (233 coronary events) and 123 in women (n=85 coronary events). The Kaplan-Meier estimate for 20-year risk was 16.1% and 6.1% in men and women, respectively.

Table 1.

Baseline characteristics (mean (SD) or %) of the study population and number of incident events, by gender

	Men (n=2574)	Women (n=2673)
Age (years)	50.8 (9.1)	50.3 (9)
Years of schooling	8.5 (4.2)	7.3 (3.4)
Total cholesterol (mmol/L)	5.8 (1.1)	5.8 (1.1)
HDL-cholesterol (mmol/L)	1.3 (0.3)	1.6 (0.4)
Body mass index (kg/m2)	26.2 (3.5)	25.6 (4.7)
Systolic blood pressure (mm Hg)	134.8 (19.3)	131.6 (20.2)
Diastolic blood pressure (mm Hg)	85.9 (10.6)	82.8 (10.8)
Antihypertensive treatment (%)	11.8	16.0
Fasting plasma glucose (mmol/L)	5.4 (1.3)	5.1 (1.2)
Diabetes (%)	6.7	4.0
Current smoker (%)	37.1	19.6
Incident coronary event (n)	233	85
Incident ischaemic strokes (n)	99	43
Incident CVD event (n)	315	123
20-year absolute risk of CVD*	16.1	6.1

Men and women, 35–69 years, CVD-free at baseline.

*Kaplan-Meier estimate.

CVD, cardiovascular disease; HDL, high-density lipoprotein.

Model development

The β-coefficients for the 20-year risk prediction model, as well as the baseline survival term and the calibration slope,²⁵ are provided in the online supplementary table S1. All the risk factors were statistically significant, except for antihypertensive treatment, though its point estimate reflected a 30% increase in hazard in men as well as women; the variable was retained in the model for comparability with the short-term CUORE model.⁶ There were no significant differences in the set of β estimates for the 20-year model as compared to those from the 10-year risk model for the risk factors in the model (data not shown). The model calibration was satisfactory, in men (Grønnesby-Bogan goodness-of-fit χ² 6.7, p value=0.67) and in women (χ² 9.6, p=value 0.38); calibration plots are available as online supplementary figure S1.

We found no statistically significant difference in the overall discrimination ability between long-term and short-term prediction models, in men (0.736 vs 0.731) and in women (0.801 vs 0.816; table 2). Only 5% of 20-year events in men occurred among subjects with a predicted risk below the 20th centile (bottom fifth); the corresponding figure in women is 2%. The relative risk of event for being above the 80th centile versus below the 20th centile of 20-year risk was 9.5 (ie, 35.1/3.7) in men and 22.4 (ie, 20.2/0.9) in women. Finally, the value of the 80th centile for 20-year risk was more than twice as high than the similar percentile for 10-year risk in men (26.8 vs 10.8) and more than three times as high in women (10.1 vs 3.0). A similar consideration holds for the 20th centile of risk or the median value.

Table 2.

Discrimination ability for the 10-year and the 20-year risk prediction models

	Men		Women
	10-year risk	20-year risk	10-year risk	20-year risk
AUC (95% CI)	0.731 (0.702 to 0.761)	0.737 (0.713 to 0.764)	0.814 (0.779 to 0.853)	0.801 (0.771 to 0.833)
Subjects with predicted risk below the 20th centile
20th centile of risk	2.3	6.3	0.3	1.1
Fraction of events* (%)	4.4	5.1	1.4	2.0
Probability of event in the group† (%)	0.8	3.7	0.2	0.9
Subjects with predicted risk above the 80th centile
80th centile of risk	10.8	26.8	3.0	10.1
Sensitivity* (%)	49.9	45.6	68.7	62.0
Specificity (%)	82.4	85.5	81.1	83.1
Probability of event in the group† (%)	19.4	35.1	7.5	20.2

Men and women, 35–69 years, CVD-free at baseline.

AUC was estimated taking censorship into account, and adjusting for over-optimism (n=1000 bootstrap).

*Probability of belonging to the group, given that the person is a case.

†Kaplan-Meier estimate of the probability of event in the group.

AUC, area under the receiver operating characteristic (ROC) curve.

Clinical utility

Tables 3 and 4 describe strategies for the identification of high-risk subjects, based on predicted 20-year risk, in men and women, respectively. A cut-off value of 10% 20-year risk in men would result in 9% of ‘missed’ events (ie, events among those with predicted risk below the cut-point), with a probability of event of 23% and 1 true-positive for every 3.4 false-positives. (table 3). In the second scenario, by choosing the 20% 20-year risk threshold value, the fraction of missed events was 36%. Note that about 30% of events occurred for a predicted 20-year risk between 20% and 30%. Finally, using the number of risk factors to define high-risk would result in a higher fraction of missed events, with no changes in specificity or in the prevalence of subjects at high risk.

Table 3.

Identification of high-risk men based on the 20-year risk prediction model with respect to the number of risk factors, according to strategies aiming to (A) reducing the fraction of missed events; and (B) reducing unnecessary treatment

	Men at high risk		Fraction of missed events (%)	Specificity (%)	Probability of event* (%)	FP/TP ratio
	n	%
Strategy a: reduce the fraction of missed events
All	2574	100.0	0.0	–	16.1	5.2
1+ Major risk factor†	1842	71.6	13.7	32.5	19.5	4.1
20-year absolute risk >10%	1645	63.9	9.1	41.2	22.9	3.4
20-year absolute risk >15%	1169	45.4	22.1	60.9	27.7	2.6
Strategy b: reduce unnecessary treatment
2+ Major risk factors†	828	32.2	50.4	73.6	24.9	3.0
20-year absolute risk >20%	841	32.7	35.7	73.7	31.7	2.2
20-year absolute risk >30%	415	16.1	62.6	88.9	37.4	1.7

Men, 35–69 years, CVD-free at baseline.

‘Missed’ events are events occurring among men not classified at ‘high risk’, that is, with 20-year absolute risk (or a number of risk factors) below the cut-off point.

*Kaplan-Meier estimate of the probability of event in the group (positive predicted value).

†Total cholesterol >240 mg/dL; HDL-cholesterol <40 mg/dL; systolic blood pressure >160 mm  Hg; smoking; diabetes.

CVD, cardiovascular disease; FP, number of false positives; HDL, high-density lipoprotein; TP, number of true positives.

Table 4.

Identification of high-risk women based on the 20-year risk prediction model with respect to the number of risk factors, according to strategies aiming to (A) reducing the fraction of missed events; and (B) reducing unnecessary treatment

	Women at high risk		Fraction of missed events (%)	Specificity (%)	Probability of event* (%)	FP/TP ratio
	n	%
Strategy a: reduce the fraction of missed events
All	2673	100.0	0.0	–	6.1	15.3
1+ Major risk factor†	1654	61.9	17.7	40.1	8.2	11.3
20-year absolute risk >2%	1733	64.8	4.5	37.4	9.0	10.1
20-year absolute risk >5%	1067	39.9	14.7	63.2	13.1	6.6
Strategy b: reduce unnecessary treatment
2+major risk factors†	640	23.9	42.3	79.5	14.8	5.8
20-year absolute risk >8%	698	26.1	22.7	77.1	18.2	4.5
20-year absolute risk >10%	545	20.4	32.1	82.7	20.4	3.9

Women, 35–69 years, CVD-free at baseline.

‘Missed’ events are events occurring among women not classified at ‘high risk’, that is, with 20-year absolute risk (or a number of risk factors) below the cut-off point.

*Kaplan-Meier estimate of the probability of event in the group (positive predicted value).

†Total cholesterol >240 mg/dL; HDL-cholesterol <50 mg/dL; systolic blood pressure >160 mm  Hg; smoking; diabetes. CVD, cardiovascular disease; FP, number of false positives; HDL, high-density lipoprotein; TP, number of true positives.

Among women, a cut-off value of 2% would result in a 5% of missed events, with a probability of event of 9% and a true-positive for every 10.1 false-positive women (table 4). In the second scenario, the probability of event among those with absolute risk greater than 10% was 20.4%, with a true-positive for every 3.9 false-positives. However, the fraction of missed events would be 32%; this number can be reduced by lowering the cut-off value to 8%. By considering at high risk those with two or more risk factor would result in a higher fraction of missed events, with no gain in specificity or in the probability of event in the group. Figure 1 illustrates the decision curve analysis based on the net benefit,¹⁷ for men (left) and women (right). The figure suggests a greater net benefit for the predicted risk with respect to the number of risk factors over the whole range of values, thus generalising the findings from tables 3 and 4.

Figure 1.

Decision curve for the 20-year risk prediction model in the CAMUNI population, Northern Italy. Men (left) and women (right), 35–69 years, free of cardiovascular disease at baseline. Net benefit =(true positives−w×false positives)/n, where w=(absolute risk threshold)/(1−(absolute risk threshold)); n=sample size. Number of risk factors: total cholesterol>240 mg/dL; high-density lipoprotein-cholesterol <40 (men) or <50 (women) mg/dL; systolic blood pressure >160 mm Hg; smoking; diabetes.

Discussion

In this article we present the 20-year prediction model of first major coronary or ischaemic stroke event in a Northern Italian population aged 35–69 years at baseline. To our knowledge, this is the first long-term prediction model in a low-incidence, European population. The discrimination ability of the long-term model did not significantly drop with respect to a 10-year risk prediction model derived on the same population. Risk stratification based on the predicted 20-year risk can be modulated according to different prevention aims, that is, either to reduce the fraction of events potentially unprevented or to avoid unnecessary treatment. Under both scenarios, the predicted 20-year risk showed an overall better net benefit with respect to a risk stratification based on the number of risk factors.

Our data confirmed previous findings on predictiveness of a single measurement of risk factors on long-term CVD risk, in the Italian²⁷ as well as in other populations.^{10 14} Event discrimination for the 20-year risk prediction model did not change significantly from 10-years, although in women it decreased from 0.814 to 0.801. In the Framingham Offspring Study updating the baseline measurement of blood pressure and lipids with a later assessment poorly affected model discrimination and reclassification²⁸ and cardiometabolic risk factors clustering has been found to be quite stable over time.²⁹

As in the Framingham population, in our study the long-term predicted risk was more than simply n-times the short-term risk prediction.¹⁰ In addition in the age range 35–49 years, the long-term predicted risk in subjects with one or more non-optimal or elevated risk factors (defined as in Ref. 7) was three times the short-term risk in men and four times in women (see online supplementary figure S2). This conveys the importance of long-term prediction for early identification of young subjects and women at increased likelihood of the event during their remaining lifespan. We observed in our data a modest net reclassification improvement (computed as in²⁴) for the 20-year risk prediction model over the recalibrated 10-year risk, in men (1.8%) and in women (4.5%). The net reclassification increased when we considered subjects with a low 10-year predicted risk but a cluster of two or more risk factors (5.4% and 7.6% in men and women, respectively; data not shown).

Subjects' stratification is often based on arbitrarily chosen thresholds of predicted risk,¹⁵ which may limit the clinical utility of risk prediction models.¹⁶ We considered two strategies for the identification of ‘high-risk’ subjects with contrasting goals, either to decrease the fraction of missed events or to decrease unnecessary treatment. These can be implemented by choosing threshold values for the predicted risk driven by either sensitivity or by specificity, respectively. Despite the lowering costs of statin treatment with respect to the costs of one unprevented event, the high sensitivity scenario was not cost-effective over a 10-year period.³⁰ These two scenarios might be combined to adopt a more complex risk stratification, as often present in clinical practice.^{1 2 12} For instance, if we consider at ‘low risk’ the 36% of men with 20-year absolute risk less than 10%, the fraction of missed events would be 9%, that is, 31 first events in 20 years. About 31% of men with absolute risk between 10% and 20% could be addressed for lifestyle modification or treatment according to the presence of specific risk factors; this category accounts for about 25% of cases. Finally, 33% of men with predicted risk above the 20% could be targeted with treatment intervention; they account for 65% of events, and of 3.2 treated men, one is a case. A similar stratification can be provided for women, with different threshold values reflecting gender-specific underlying risk as for the cardiovascular age assessment.¹⁵

Among the study strengths and limitations, our sample comprises participants drawn from a representative northern Italian population, with a satisfactory participation rate. The underlying population is characterised by high levels of industrialisation and urbanisation, with one of the highest average incomes in Italy. A major limitation is the lack of an external validation. External validation for long-term prediction models is in general an issue¹⁰; we provide the over-optimism adjusted AUC as well as the calibration slope²⁵ to allow applying our equation to different contexts (see online supplementary material). We also mention high quality of follow-up procedures, including case ascertainment for non-fatal events²¹ and a consistent event validation according to MONICA criteria, resulting in a Standardized Incidence Rate for the study cohorts above 1 over the whole follow-up period.¹⁸ Finally, the study endpoint reflects the clinical need to treat the ‘global’ ischaemic risk of a given patient, and not its separate components.³

In conclusions, we provide a model to predict long-term risk of first major ischaemic cardiovascular event in a low-incidence population. Risk stratification based on long-term risk can be clinically useful, especially for young individuals and women. A clinical utility analysis is required to identify the optimal stratification, according to different public health goals.