Criteria to Assess the Predictive and Clinical Utility of Novel Models, Biomarkers, and Tools for Risk of Cardiovascular Disease: A Scientific Statement From the American Heart Association

Abstract

Risk prediction has been used in the primary prevention of cardiovascular disease for >3 decades. Contemporary cardiovascular risk assessment relies on multivariable models, which integrate established cardiovascular risk factors and have evolved over time from the Framingham Risk Model to the pooled cohort equations to the PREVENT (Predicting Risk of CVD Events) equations. Recent scientific (ie, genomics, proteomics, metabolomics) and methodologic (ie, artificial intelligence) advances have led to a proliferation of novel models, biomarkers, and tools for potential use in risk prediction. In parallel, the growing armamentarium of preventive therapies, some with considerable cost, underscores the need for more accurate and precise risk assessment to prioritize those at highest risk who will derive the greatest absolute benefit. Accompanying the considerable enthusiasm for the potential of newer approaches to improve risk prediction is the need for rigorous evaluation and assessment of their performance (ie, accuracy, precision, incremental performance when added to contemporary multivariable risk models or established risk factors) and clinical utility (ie, actionability, scalability, generalizability) before adoption in clinical practice. Additional considerations in risk tool evaluation include reproducibility, cost–value considerations (including impact on downstream health care costs), and implications for health equity. This scientific statement defines a standardized framework for general considerations in risk prediction, statistical assessment of predictive utility, and critical appraisal of clinical utility and readiness. This scientific statement is intended to support clinicians, researchers, and policymakers in how best to evaluate current and emerging risk prediction tools and ultimately improve the prevention of cardiovascular disease in diverse populations.

Cardiovascular disease (CVD) remains the leading cause of morbidity and mortality in the United States,¹ with estimates that >45 million adults will have CVD by 2050 if current trends continue.^2,3 Effective preventive strategies are needed to address these projected increases in CVD prevalence. A foundational part of the contemporary clinical approach to the primary prevention of CVD is risk assessment with use of multivariable risk prediction models that integrate established cardiovascular risk factors. This risk-based paradigm is widely endorsed across multiple American Heart Association/American College of Cardiology clinical practice guidelines (Table 1). These models have advanced considerably over the past 3 decades, from the development of the Framingham Risk Model to predict coronary heart disease to the pooled cohort equations (PCEs) to predict atherosclerotic CVD (ASCVD; coronary heart disease and stroke) to the PREVENT (Predicting Risk of CVD Events) equations to predict total CVD (ASCVD and heart failure [HF]). Still, great enthusiasm remains to further improve accuracy, precision, and clinical utility of risk prediction models beyond those that rely on established or traditional risk factors alone.^11,12 Moreover, the emergence of novel but costly therapies, such as glucagon-like peptide-1 receptor agonists, sodium-glucose cotransporter 2 inhibitors, nonsteroidal mineralocorticoid receptor antagonists, and proprotein convertase subtilisin/kexin type 9 inhibitors, underscores the need for accurate risk models to consider risk-based prioritization to target those at highest risk who may derive the greatest absolute benefit and maximize the cost-effectiveness of these therapies.

Table 1.

Examples of American Heart Association/American College of Cardiology Guidelines That Incorporate Risk Prediction

Guidelines	Recommendations and supporting text
2025 AHA/ACC/AANP/AAPA/ABC/ACCP/ACPM/AGS/AMA/ASPC/NMA/PCNA/SGIM guideline for the prevention, detection, evaluation and management of high blood pressure in adults3a	Use of BP-lowering medications is recommended for primary prevention of CVD in adults at increased 10-y risk of CVD based on PREVENT ≥7.5% for treatment of stage 1 hypertension (average SBP 130–139 mm Hg or DBP ≥80–89 mm Hg)
2024 AHA/ACC/ACS/ASNC/HRS/SCA/SCCT/SCMR/SVM guideline for perioperative cardiovascular management for noncardiac surgery4	In patients with known CVD being considered for noncardiac surgery, use of a validated risk prediction tool is recommended to estimate perioperative MACE risk
2024 AHA/ACC/AMSSM/HRS/PACES/SCMR guideline for the management of hypertrophic cardiomyopathy5	Comprehensive noninvasive risk assessment for sudden cardiac death in children and adults is recommended; risk stratification should be improved and expanded
2023 ACC/AHA/ACCP/HRS guideline for the diagnosis and management of atrial fibrillation6	Anticoagulation should be guided by yearly thromboembolic event risk estimated using a validated clinical risk score, such as CHA2DS2-VASc (there are >20 risk prediction models for incident AF in the community; the most widely replicated risk prediction model for newly diagnosed AF is CHARGE-AF)
2023 AHA/ACC/ACCP/ASPC/NLA/PCNA guideline for the management of patients with chronic coronary disease7	Development and validation of comprehensive MACE risk scores for patients with CCD that include demographics, medical information, social determinants of health, and test results are recommended
2019 ACC/AHA guideline on the primary prevention of cardiovascular disease8	Routine assessment of traditional cardiovascular risk factors and calculation of 10-y ASCVD risk using PCEs are recommended for cardiovascular risk assessment
2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol9	For adults age 40 to 75 y, 10-y risk of ASCVD should be estimated using PCEs and adults should be categorized as intermediate or high risk at ≥7.5% for primary prevention

AACVPR indicates American Association of Cardiovascular and Pulmonary Rehabilitation; AANP, American Association of Nurse Practitioners; AAPA, American Academy of Physician Associates; ABC, Association of Black Cardiologists; ACC, American College of Cardiology; ACCP, American College of Chest Physicians; ACPM, American College of Preventive Medicine; ACS, American College of Surgeons; ADA, American Diabetes Association; AGS, American Geriatrics Society; AHA, American Heart Association; AMA, American Medical Association; AMSSM, American Medical Society for Sports Medicine; APhA, American Pharmacists Association; ASCVD, atherosclerotic cardiovascular disease; ASNC, American Society of Nuclear Cardiology; ASPC, American Society for Preventive Cardiology; BP, blood pressure; CCD, chronic coronary disease; CHARGE-AF, Cohorts for Heart and Aging Research in Genomic Epidemiology for Atrial Fibrillation; CVD, cardiovascular disease; DBP, diastolic blood pressure; HRS, Heart Rhythm Society; MACE, major adverse cardiovascular event; NLA, National Lipid Association; NMA, National Medical Association; PACES, Pediatric and Congenital Electrophysiology Society; PCE, pooled cohort equation; PCNA, Preventive Cardiovascular Nurses Association; PREVENT, Predicting Risk of CVD Events; SBP, systolic blood pressure; SCA, Society of Cardiovascular Anesthesiologists; SCCT, Society of Cardiovascular Computed Tomography; SCMR, Society for Cardiovascular Magnetic Resonance; SGIM, Society for General Internal Medicine; and SVM, Society for Vascular Medicine.

Scientific, technologic, and methodologic advances have led to a proliferation of novel biomarkers and tools for risk prediction of CVD, with a growing number of scientific statements (Supplemental Table 1) highlighting the importance of innovative approaches to assess risk (eg, noncoding RNAs,¹³ polygenic risk scores [PRS],¹⁴ artificial intelligence [AI]¹⁵). However, published studies often have variable rigor in their reporting of the predictive and clinical utility of such novel approaches to assess CVD risk. In addition, some risk tools may warrant unique considerations in determining their clinical utility and readiness for clinical implementation, such as variation across omics platforms and assays or evaluation of bias with AI tools that rely on data collected for clinical care in electronic health records (EHRs).

This scientific statement provides an updated framework for the critical appraisal of the predictive and clinical utility of novel biomarkers, models, and tools and builds on the American Heart Association scientific statement “Criteria for Evaluation of Novel Markers of Cardiovascular Risk.”¹⁶ Herein, we review general considerations for CVD risk prediction; statistical assessment of accuracy, precision, and incremental value relative to current risk models used in clinical practice; clinical utility; and additional considerations before implementation in practice (Figure 1). Although many of the principles outlined are also applicable to risk prediction of recurrent CVD events (ie, secondary prevention) or non-CVD events, the focus of this scientific statement is evaluation of novel tools for prediction of incident CVD for primary prevention, where risk assessment has the greatest impact on clinical practice and is used in contemporary guidelines to inform recommendations for preventive therapies. The intent is to support clinicians, researchers, and policymakers in evaluating current and emerging risk prediction strategies and ultimately improving CVD prevention in diverse populations.

Figure 1. Key steps for evaluating cardiovascular disease risk prediction models, biomarkers, and tools from development to clinical implementation.

Outline of a 4-step framework for the evaluation and implementation of cardiovascular disease risk prediction models in clinical practice that includes considerations for (1) general model development, (2) statistical assessment, (3) clinical utility, and (4) clinical implementation. Each step represents a necessary phase in the rigorous evaluation to ensure that cardiovascular disease risk prediction tools are accurate, are equitable, and have clinical utility in real-world settings. AI indicates artificial intelligence; CVD, cardiovascular disease; and ML, machine learning.

GENERAL CONSIDERATIONS IN RISK PREDICTION

Development of a risk prediction model or validation of a new biomarker or tool typically proceeds in steps from concept to statistical evaluation to clinical utility (Supplemental Figure 1). A biomarker, as defined by the National Institutes of Health, is “a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacological responses to a therapeutic intervention,” and thus is not limited to blood tests.¹⁷ Before the predictive utility of a new biomarker can be evaluated, the distribution must be demonstrated to differ significantly among individuals who eventually are affected by the outcome of interest compared with those who do not develop the outcome. This is best accomplished with a prospective study design to assure that the individuals in the study were at essentially similar risk of the outcome at baseline and did not already have the disease or outcome of interest. If data are prospectively collected in an observational research cohort, relevant variables are more likely to be systematically collected in every individual, which will minimize missing data and bias in model development.

Key Questions in Model Development

Table 2 outlines the key questions to consider when developing a risk prediction tool, which includes a priori clarifying the clinical question of interest and defining the intended population of interest, as a test may predict well in one clinical setting but not in a different setting, population, or indication.¹⁸ The process used to evaluate differences across groups (eg, race and ethnicity as a social construct, genetic ancestry), which could include assessment of generalizability across different subgroups, should be decided on in advance. The relevant time frame of interest should also be defined in the context of the clinical question. A disease that develops over a short time period can be studied in a few days or weeks, whereas many important outcomes, such as incident ASCVD or HF, usually take years to develop and require longer-term studies to understand the magnitude of association with biomarkers or risk models. The time horizon selected can also inform the approach for considering competing risk of non-CVD events, as adjusting for competing risk has less impact on risk estimation with shorter follow-up compared with longer follow-up times.

Table 2.

General and Specific Considerations for Developing and Evaluating Novel Models, Biomarkers, or Tools to Predict Risk of Incident Cardiovascular Disease

Developing the model	Evaluating predictive and clinical utility	Specific considerations for prediction of incident cardiovascular disease
Defining the inputs for risk prediction tools
Define the clinical question of interest	Does the model address a clinically significant problem? Will the model inform clinical decision-making?	Does the model predict the first CVD event (as a composite) or a specific subtype of CVD (eg, ASCVD or HF)?
Define the target population and data sources	Is the sample from the selected data source representative of the intended use population? Are the inclusion/exclusion criteria consistent with clinical use?	Is the sample contemporary and representative of a primary prevention population without baseline CVD as the intended target for risk assessment?
		Are models stratified for absence of baseline CVD of one type when predicting CVD of another type (eg, prediction of HF among those without baseline CHD)?
		For external validation, is the target population defined similarly (eg, age) as the base model?
Define the predictors	Can the predictors be measured in the data source reliably and accurately? Are the predictors routinely measured in clinical practice? Are the predictors associated with the outcome?	Self-report of novel predictors may be reasonable but of lower quality (eg, self-report for family history of CVD, adverse pregnancy outcomes)
		Some conditions may be poorly captured in health records (eg, postpartum depression)
		When approaches with ML or AI or high-dimensional omics data are used, individual predictors may not be identifiable
		When considering omics data, preanalytic considerations may impair reproducibility (eg, batch effects)
		For external validation, are the predictors defined similarly as the base model?
		Are predictors measured at one point in time or measured repeatedly and integrated to capture cumulative risk factor information?
Define the outcomes	Are the outcomes clinically relevant? Can the outcomes be accurately and reliably identified in the selected data set?	Does the model need to account for competing risk of death based on the incidence of non-CVD death?
		Are the outcome definitions aligned across data sets (eg, chart adjudication, ICD codes, CPT codes, DRG codes)?
		Are the outcomes influenced by local practice patterns or structural differences in treatment receipt (eg, revascularization)?
		For external validation, are the outcomes defined similarly as the base model (eg, restricted to the same ICD codes)?
Define the time horizon for time to event	Is the follow-up period sufficient to capture meaningful event rates? Can the model be applied across different follow-up times?	How variable is follow-up among individuals in the data set?
		For external validation, is the median follow-up similar to the base model?
		Should model development consider or adjust for competing risks over follow-up?
Additional considerations in development of risk prediction models for CVD
Sex stratification	Do the associations between the predictor and outcome differ (eg, effect modification) to warrant stratification?	Given the known sex differences in CVD, stratification is typically warranted
		Evaluate model discrimination and calibration by sex
Race, ethnicity, and ancestry considerations	Are race and ethnicity defined by self-report, EHR, or other imputation algorithms? Are data for ancestry available and which approach is used to define proportions of diverse ancestry groups (eg, self-report, genomic data)?	Because race and ethnicity are social constructs, race or ethnicity should not be included in model development as a predictor or adjustment factor to avoid inferring that race or ethnicity are biological constructs; models may be evaluated to examine whether performance may differ with or without race or ethnicity as a predictor
		Predictors or risk factors should be included in the model that represent risk related to adverse social factors (eg, individual- and area-level social determinants of health) or are on the causal pathway and are also influenced by structural and systemic racism (eg, blood pressure) to accurately capture differences in risk by race and ethnicity
		To ensure generalizability, population samples should strive to be diverse and inclusive in regard to race and ethnicity and model performance should be evaluated by race and ethnicity groups
		For genomic models, evaluate representativeness and model performance across ancestry groups

AI indicates artificial intelligence; ASCVD, atherosclerotic cardiovascular disease; CHD, coronary heart disease; CVD, cardiovascular disease; CPT, Current Procedural Terminology; DRG, Diagnosis-Related Group; EHR, electronic health record; HF, heart failure; ICD, International Classification of Diseases; and ML, machine learning.

Outcome Ascertainment

The accuracy and interpretability of the model relies on the fidelity of outcome ascertainment. For example, defining the incident CVD outcome of interest can be challenging and requires careful consideration depending on the data source. For studies relying on observational research cohorts or clinical trial data, event follow-up and investigator-based adjudication may already be standardized as part of study procedures (eg, clinical end points committee). However, studies relying on EHR outcomes data require different approaches for event ascertainment, and may include use of International Classification of Diseases, Diagnosis-Related Group, and Current Procedural Terminology codes. Variable selection of these may influence event rates related to local differences in coding practices or clinician procedural decision-making. For example, revascularization rates vary greatly across the United States and therefore were not included in the outcome definition for the PCEs or PREVENT equations to allow for a harmonized national approach.^19,20

Defining the Base Prediction Model

Another important early step is to identify the most contemporary or well-accepted risk prediction model (when available) for the CVD outcome of interest. The evidence base to support specific risk models, thresholds, and preventive strategies is most well-established for ASCVD, but other CVD subtypes are included in PREVENT–HF, and there is growing interest to consider other CVD end points, such as atrial fibrillation. When a multivariable prediction model based on traditional risk factors is not available, well-established risk factors that are routinely measured in clinical practice should be defined with continuous measures (eg, cholesterol, blood pressure, body mass index), because including categorical risk factors can underestimate their contribution to risk and also depends on awareness of diagnosis (eg, hyperlipidemia, hypertension, obesity).

STATISTICAL ASSESSMENT OF PREDICTIVE UTILITY

When a putative risk marker is shown to significantly differ in people with or without the outcome, several statistical metrics should be evaluated and reported. Table 3 provides a list of recommended statistical criteria for assessing model performance for accuracy, precision, and incremental utility. Various additional statistical approaches^21–24 may be considered on the basis of comprehensive frameworks and reporting guidelines (eg, TRIPOD [Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis], TRIPOD+AI, TRIPOD+LLM [large language models]) in designing and evaluating prediction studies more broadly for use in clinical medicine beyond preventive cardiology, including with specific guidance for certain risk tools (eg, PRS).^25–29

Table 3.

Statistical Metrics of Association, Predictive Utility, and Clinical Utility When Evaluating Models, Biomarkers, or Tools for Risk of Incident Cardiovascular Disease

Questions	Statistical metric reporting	Comments and considerations
1. Is the marker associated with incident CVD in a prospective cohort?	Continuous risk marker: RR per SD Categorical risk marker: RR of exposed vs unexposed Coefficient of variation	The threshold for which a risk marker is defined as “high” or “elevated” may influence the magnitude of association reported. Therefore, whenever possible, both continuous and categorical reporting should be included for continuous markers.
		Is there an absolute cutoff for high or is a population-based percentile (eg, top 20th percentile) used to determine “elevated?”
		Is there variation in measurement and are there differences across techniques, assays, or platforms?
2. How well does the marker discriminate between those who will and those who will not develop CVD?	C-statistic	Model performance can be highly dependent on the age distribution of the population and event definition
		The C-statistic ranges from 0.5 to 1, with 0.70–0.79 considered acceptable and 0.80–0.89 considered good. However, a C-statistic ≥0.90 may represent overfitting.
3. How closely does the marker approximate the true observed CVD event rate?	Calibration in the large (mean absolute and relative calibration) and calibration slope of observed to predicted risk	Evaluate calibration overall and across key sociodemographic groups to inform generalizability
		Poor calibration may be present even if discrimination is good
		Poor calibration will limit clinical utility when absolute risk thresholds are used
4. Does the marker improve prediction beyond standard, established CVD risk scores or traditional risk factors in both internal and external validation samples?	Change in discrimination or C-statistic Change in calibration Categorical NRI at clinically defined thresholds or at event rate when a clinically defined threshold is not available	Change in discrimination has limitations and small changes in C-statistic can still be clinically meaningful
		Is the change in predicted risk or risk reclassification sufficient to change treatment recommendations (estimated by categorical NRI and should include reporting for reclassification of events and nonevents)?
		Clinically meaningful thresholds for these metrics have not been defined, but CAC has been reported to result in a change in C-statistic of ≈0.10 and categorical NRI of ≈0.20 (driven by reclassification of events)
		Continuous NRI is not a useful statistic to judge clinical utility
		External validation is essential to ensure generalizability
5. If 1–4 support predictive utility in internal and external validation, is the marker practical, cost-effective, and actionable in clinical practice?	Cost-effectiveness Decision analysis Plot and interpret decision curves	Consider ease of implementation, reproducibility, and cost
		Assess whether the new marker or model justifies added complexity or change in clinical practice. Can consider whether it may inform therapeutic allocation given that the increasing armamentarium of costly preventive therapies may benefit from improvement in risk assessment (eg, CAC to prioritize GLP-1 RA therapy).
		What additional considerations regarding preanalytic processing, reproducibility, and cost should be evaluated?
		What steps are needed to implement the model or marker in clinical practice (eg, EHR frameworks for AI models)?

AI indicates artificial intelligence; CAC, coronary artery calcium; CVD, cardiovascular disease; GLP-1 RA, glucagon-like peptide-1 receptor agonist; EHR, electronic health records; NRI, net reclassification improvement; and RR, relative risk.

To determine whether the risk marker is associated with the outcome, the relative risk (RR) is calculated, which is the ratio of the probability of an outcome in the exposed group to the probability in an unexposed group. Continuous approaches better capture the nuance of the association between the biomarker and outcome and may identify nonlinear associations. Categorical approaches may be preferred when clinically relevant thresholds are available but may not fully capture risk. Moreover, there are challenges to the interpretability of the RR when a clinically defined level for “high” is not available. Often a population-based percentile may be used, as is the case with lipoprotein(a) or PRS, where the top 20th percentile is often defined as elevated.^30,31 However, the definition of “high” or “elevated” for a risk marker influences the RR. For example, selecting “high” as the top first percentile compared with the top fifth or the top 20th percentiles will lead to varying magnitudes of RR.^32,33 Therefore, a risk marker should be analyzed both continuously and categorically whenever possible. Furthermore, when comparing the magnitude of association across continuously measured biomarkers, evaluation should be performed per SD change of each biomarker to allow for direct comparisons.

External Validation

Validation of a model should include both internal and external validation.^34,35 Internal validation examines the model’s statistical performance in the same data set as the derivation samples and can be assessed by data-splitting, bootstrapping, or cross-validation. External validation examines the model’s statistical performance (with coefficients as originally specified) in a distinct but related setting where the model might plausibly be applied in clinical practice.³⁵ External validation should be assessed in a different data set of a relevant population to avoid overfitting and ensure generalizability.²¹ To interpret the results of external validation, attempts should be made to align inclusion and exclusion criteria, predictor definition, and outcome ascertainment with the model development. Despite its well-described importance, external validation has frequently not been conducted in risk prediction studies, with a recent large-scale review of the literature demonstrating that more than half (58%) of the 1382 CVD prediction models identified had never been externally validated.³⁶ As baseline absolute risk of the population changes due to secular trends, changes in life span, or introduction of novel therapies or public health policies, risk prediction models should be reassessed.

Discrimination

Standardized assessment of predictive utility should include measures of discrimination in internal and external validation samples. Discrimination evaluates the ability of the marker or model to distinguish or separate study participants into those with or without events. An ideal model (which rarely, if ever, exists) would identify 100% of patients who have an event and exclude everyone without an outcome. Discrimination can be visually displayed by boxplots, histograms, or density plots, with better discriminating models showing less overlap between these groups.³⁷

Discrimination is typically measured using the concordance statistic (C-statistic), which is identical to the area under the receiver operating characteristic curve, defined as a plot of sensitivity versus 1−specificity. The C-statistic estimates the probability that someone who develops the disease will have a higher risk estimate than someone who does not develop the disease. The C-statistic ranges from 0 to 1, with 0.5 being a coin flip, values of 0.70 to 0.79 considered good, and ≥0.80 considered excellent (Supplemental Table 2).^38,39 The values of C-statistics are a function of not only the model itself but also the distribution of the predictors. For example, the C-statistic for the same model that includes age as a predictor will be much lower when applied to a sample with a narrow age range than if applied to one with a wide age range. For this reason, the performance of risk prediction models can be reported conditional on age (eg, breast cancer). For CVD risk models, several alternative approaches have been proposed, such as longer-term risk estimation (eg, 30-year risk estimation) and age- and sex-specific percentiles, because younger individuals with a low short-term absolute risk often have high long-term risk, which may improve interpretation and clinical decision-making.^40,41

Calibration

Standardized assessment of predictive utility should also include measures of calibration in internal and external validation samples. Calibration refers to the agreement between predicted risk and observed outcomes. Calibration can be examined visually by plotting the predicted versus observed risks. In logistic and survival settings, calibration is more complex. The predicted risks are derived by applying the model to the data, and the observed risks are proportions of observed outcomes or events grouped according to quantiles of predicted risk. The agreement between predicted risks and observed outcomes can be quantified using calibration in the large (or mean calibration, comparing the averages of predicted risks and observed outcomes in the full sample or a subset) and the calibration slope (the slope of a logistic regression line of outcome versus predicted risk).⁴² A calibration slope of 1.0 indicates ideal calibration, with a slope <1 indicating overprediction and >1 underprediction. It is recommended to report at least the calibration in the large and calibration slope with a calibration plot. Both are considered weak calibration metrics, and a model may be poorly calibrated even if these metrics suggest good performance.⁴³ Use of the Hosmer-Lemeshow goodness-of-fit test is widely discouraged due to its limited power and poor interpretability.²¹ Particular attention to calibration across demographic subgroups is needed to ensure generalizability in diverse populations. Calibration is particularly important around decision thresholds in prediction of incident CVD when absolute risk thresholds are used to guide clinical decision-making for preventive therapies (eg, statins, initiation of antihypertensive drug therapy for stage 1 hypertension).^8,9,44 After publication of the PCEs, a validation study in REGARDS (Reasons for Geographic and Racial Differences) demonstrated poor calibration in the overall sample but good calibration at the decision threshold for PCEs between 5% and 7.5%.⁴⁵

Risk Reclassification

Once a model is demonstrated to have good discrimination and calibration, the next step is to determine the incremental utility when added to the clinical standard of care for risk assessment. This should be measured with the change in C-statistic and the net reclassification improvement (NRI). For reasonably well-performing prediction models, changes in the C-statistic are not typically large even with the addition of biomarkers that have strong association with the outcome.⁴⁶ C-statistic changes also depend on the performance of the baseline model: for the same marker, the better the performance of the baseline model, the smaller the increase in C-statistics with addition of the new marker.⁴⁷ For the prediction of incident CVD, baseline risk prediction models are well-established on the basis of routinely measured and established risk factors that are the targets of treatment (eg, blood pressure, cholesterol). Markers with excellent discrimination and calibration metrics may still not be clinically useful if they do not significantly improve predictive utility when added to the standard of care model (eg, change in C-statistic when compared with PCEs or PREVENT). The magnitude of change in the C-statistic should be reported with a corresponding CI rather than a P value. For example, the change in C-statistic for coronary artery calcium (CAC) when added to the PCEs was 0.09 with a 95% CI of 0.06 to 0.13 (Supplemental Table 3).^48,49

The NRI is a statistical measure to evaluate the change in clinical risk categories for an individual by adding a new marker to an existing risk prediction model. This can be calculated as the categorical NRI defined at a clinically meaningful threshold (when one is available) or at event rate (when one is not available or for comparison).⁵⁰ The categorical NRI ranges from −1 to +1 and calculates the proportion of individuals with an event who are moved to a higher risk category and the proportion of individuals without an event who are moved to a lower risk category. It is important to report reclassification for both events and nonevents given the different clinical implications of upward or downward reclassification between groups. When summed together, a positive NRI indicates that the addition of the new marker improves overall predictive performance. A useful novel risk marker or model would upclassify risk in individuals who have events when added to or compared with the base model and would downclassify risk in those who do not have events. Categorical NRI can be useful in showing model improvement but may be challenging to interpret when clinical thresholds for decision-making are not widely accepted. In these cases, the event rate over a prespecified time period (eg, years) should be evaluated.⁵⁰ Unlike weighted forms of categorical NRI, the NRI at event rate adheres to principles of decision analysis and can be interpreted as increase in maximum standardized net benefit. This is also a useful comparison when a new model is developed and risk thresholds need to be reexamined and revised.⁵¹ For ASCVD, the categorical NRI has been studied for various biomarkers, including high-sensitivity C-reactive protein, carotid intima–media thickness, ankle brachial index, CAC, and PRS, as summarized in Supplemental Table 3.^48,49,52,53 Of these, the greatest NRI was observed with the addition of CAC, representing an improvement in risk reclassification of ≈20% (driven by reclassification of events), which may be a useful benchmark when evaluating new biomarkers or tools.^48,49 A continuous form of the NRI can also be calculated, which defines reclassification as any change in the predicted probability. Whereas the continuous NRI is often reported, it simply represents the strength of the association of the biomarker and outcome without accounting for the predictive utility of the biomarker. Thus, continuous NRI is not an informative statistical metric for clinical decision-making, and NRI at event rate should be reported when a guideline-recommended threshold is not available.⁵⁰

These measures of discrimination, calibration, and reclassification (or incremental predictive utility) when compared with the standard of care CVD risk model (eg, PCEs, PREVENT) are necessary to indicate the potential of a new marker, model, or tool. A sample template for reporting is provided in Supplemental Table 4. Beyond the required set of measures of discrimination and calibration, other metrics of global fit and metrics specific to machine learning (ML) models can also be considered, and are summarized in Supplemental Table 3. However, a statistically accurate and valid model may still not be useful in the clinical setting. As such, statistical assessment does not equate to clinical utility or readiness, which is the next step for promising risk prediction tools.^54,55

EVALUATION OF CLINICAL UTILITY OF RISK PREDICTION TOOLS

Once predictive utility is suggested based on good to excellent performance in the statistical metrics outlined, the next step is to determine clinical utility. This is more challenging to assess, requires quantitative and qualitative assessment of potential clinical impact, and may include feasibility, complexity, and cost of implementation. Even when a biomarker or model is evaluated extensively and appears to have favorable model performance metrics, if it is not actionable (ie, affecting clinical decision-making, treatment recommendations, or outcomes), it will not be clinically useful. For example, biomarker testing for individuals who are already optimally treated (eg, high-intensity statin at target for lipid levels) is not clinically useful. In addition, use of a risk assessment tool in a population in which it is not relevant or not indicated will not have clinical utility and could result in increased health care costs through excessive downstream testing (eg, CAC scoring in a nonagenarian or in someone presenting with acute chest pain).

Clinical Actionability and Decision Curve Analysis

Typical approaches to evaluate potential for clinical impact include whether the model would result in a change in clinicians’ decision-making, patient behaviors, clinical outcomes, or health care utilization or costs. Such outcomes may require defining an actionable clinical threshold or setting the percentage of highest-risk patients to receive a given intervention. If meaningful thresholds exist, the use of categorical NRI as described earlier may be informative. Additional considerations regarding the potential harms of tests, such as radiation exposure or downstream invasive procedures, as well as costs, should be evaluated and weighed to determine who would benefit before implementation in clinical practice.⁵⁶

A particularly appealing approach that can illustrate clinical implications across the full domain of classification thresholds is decision curve analysis.⁵⁷ Decision curves are created by plotting the net benefit of a prediction model across a range of classification thresholds. Net benefit is estimated as a weighted combination of true positives and false positives, with weights determined by the odds of the classification threshold probability. When multiple models are considered (ie, with and without new biomarkers), the plot allows a quick visual assessment of their value in terms of net benefit across all classification thresholds. As indicated earlier, NRI at event rate quantifies the difference in maximum standardized net benefit. The net benefits of treat all (thresholds of 0) and treat none (thresholds of 1) are also typically plotted for comparison. An example of this to demonstrate the concept is shown in Supplemental Figure 2 from a study evaluating a biomarker-based score for predicting stroke in patients with atrial fibrillation.⁵⁸

Risk Thresholds

The evaluation of the clinical utility of a risk marker or model depends on the defined actionable threshold. As noted previously, the magnitude of an RR can vary greatly depending on how the marker or model is categorized. There are a number of methodologic considerations for defining a threshold to initiate treatment, including evidence from clinical trials of preventive therapies to determine the risk or cutoff at which the potential benefit is likely to outweigh harm. This is discussed in greater detail in a recent scientific statement.⁵⁹ For the PCEs, this was determined to be at a threshold of 5% to 7.5% on the basis of randomized controlled trial data indicating that statins result in meaningful absolute risk reduction (ARR) at this threshold and that the benefits outweigh potential harms (estimated at ≈3% at 10 years for incident diabetes).¹⁹ For the PREVENT model for antihypertensive drug therapy, this was determined to be at a threshold of ≥7.5% on the basis of randomized controlled trial data.^3a When randomized controlled trial data are examined in subgroups, most preventive therapies have been demonstrated to have a similar relative benefit regardless of baseline risk. Thus, when advancing a marker or model for clinical application, careful consideration should be given to threshold selection to optimize clinical utility.

ARR in Treatment Decisions

It is also necessary to understand the ARR for a treatment, which is defined as the difference in disease rates in exposed versus unexposed individuals. A large RR may not have as large a clinical impact at the population level for rare or uncommon diseases, because the absolute risk in those situations can be small. This can be quantified by number needed to screen or number needed to test (Supplemental Table 5). For example, consider a marker that has an RR of 5.0 for a positive test result compared with the rest of the population or sample. Then, the ARR will depend on the absolute rates of the disease in the group that tests positive compared with the group that tests negative. If the rate in the test-positive group is 25% (0.25) and the RR is 5.0, we can assume that the rate in the test-negative group is 5% (0.05). The ARR is therefore 0.25−0.05 (0.20), and the number needed to screen is 1/ARR (1/0.20=5 [ie, only 5 individuals would need to be tested to find 1 with disease]). However, if the rate in the tested-positive group was only 5% (0.05), then the rate in the tested-negative group would be 1% (0.01), and the number needed to screen would be 25 to find 1 with disease. Therefore, both RR and ARR are necessary to understand the clinical utility of a risk prediction approach.

An important extension of this concept is the ability of a risk marker or tool to improve efficiency of treatment and cost-effectiveness. With the growing list of new and high-cost preventive therapies, such as glucagon-like peptide-1 receptor agonists, risk prediction offers the ability to identify those with higher versus lower ARR to prioritize who may derive greater absolute benefit from these therapies.⁶⁰ For example, in one modeling study of individuals potentially eligible for glucagon-like peptide-1 receptor agonists, those with a CAC score >100 had significantly higher ARR of CVD than those with CAC=0. Assuming a uniform RR reduction of CVD with glucagon-like peptide-1 receptor agonists across CAC categories, the ARR (RR×predicted risk) was greatest and thus the number needed to treat to prevent 1 event (number needed to treat=1/ARR) was lowest in those with CAC >100 versus 0 (number needed to treat=79 versus 237).⁶¹ In addition to imaging studies, laboratory-based biomarker studies have demonstrated utility in clinical decision-making and prevention. In the STOP-HF trial (St Vincent’s Screening to Prevent Heart Failure), BNP (B-type natriuretic peptide) screening followed by referral for collaborative care to intensify preventive therapies in patients with elevated levels was effective at reducing the primary end point (new-onset left ventricular dysfunction or HF).^62,63

Causality in Biomarkers for Risk Assessment

Causality or direct involvement in the pathogenesis of CVD can be determined in many ways, including mechanistic studies, Mendelian randomization techniques, and randomized controlled trials testing whether targeting the biomarker alters the clinical disease. It is well-established that not all biomarkers that contribute to risk prediction are causal, modifiable targets. A prime example of this is high-density lipoprotein cholesterol (HDL-C), which can contribute to risk prediction of CVD, but modification of HDL-C levels has not resulted in improved outcomes. The inverse of this is also true; biomarkers that are established causal targets may not be included in risk models or may not improve risk prediction depending on their availability, prevalence, and distribution in the population from which models are derived. Measurement of a causal risk factor may have a population-level impact if treatment modifies CVD risk. For example, adding non–HDL-C to a base model is associated with a change in C-statistic <0.01 and NRI <0.01. However, the population-attributable fraction of non–HDL-C ≥130 mg/dL is 17%.⁶⁴ More recently, it has been reported that ApoB (apolipoprotein B) is a more accurate marker of the benefit of lipid-lowering therapies than low-density lipoprotein cholesterol.^65,66 ApoB is not commonly measured in clinical practice, and most trials have relied primarily on low-density lipoprotein cholesterol or non–HDL-C to identify participants and measure the degree of lipid lowering. As such, low-density lipoprotein cholesterol or non–HDL-C is typically included in risk models rather than ApoB. However, an elevated ApoB level is highlighted as a risk-enhancing factor and may be useful in the consideration of intensifying preventive treatment, particularly in the subset of individuals with discordance between low-density lipoprotein cholesterol and ApoB levels.^8,67

Health Care Costs of Novel Risk Tools

Traditional or established risk factors that are already assessed in routine clinical care, such as hypertension, dyslipidemia, and diabetes,⁶⁸ are appealing to include in risk prediction models as a starting point. Even if a novel biomarker is identified to improve predictive performance, the economic cost and value of a novel blood-based or imaging-based marker should be considered in addition to implications on downstream health care costs.⁶⁹ Formal cost-effectiveness studies for biomarkers should be considered before widespread implementation, but no universally agreed-upon threshold exists for cost-effectiveness for a biomarker. In the United States, thresholds for cost-effectiveness for treatments are commonly cited in the range of $100 000 to $150 000 per quality-adjusted life-year gained.⁷⁰ However, the value of a diagnostic test may be indirect by changing downstream clinical decision-making. For example, natriuretic peptide–based testing followed by echocardiography and collaborative care in people with elevated natriuretic peptides (or pre-HF) was estimated to have a high likelihood of being cost-effective (defined as a willingness to pay threshold of €30 000 in individuals with risk factors for the prevention of HF based on data from the STOP-HF trial conducted in Ireland).⁷¹

ADDITIONAL CONSIDERATIONS BEFORE CLINICAL IMPLEMENTATION

Analytic Assay Considerations

The clinical utility of novel biomarkers relies not only on their statistical performance (ie, accuracy) but also on their analytical variability (ie, precision) across different testing environments, operators, and time points, or quality control, which can be quantified by the coefficient of variation. The coefficient of variation represents a standardized measure of variability, expressed as the SD as a percentage of the mean (SD/mean×100%), which can help evaluate analytical precision, assess biological variability, and inform clinical utility. For example, an analysis from NHANES (National Health and Nutrition Examination Survey) examined the within-person variability of high-sensitivity C-reactive protein in which participants had high-sensitivity C-reactive protein assayed ≈2.5 weeks apart. Results demonstrated high variability, with Spearman rank correlation between visits of 0.65 and intraclass correlation coefficient of 0.77 (95% CI, 0.69–0.84), albeit with greater discordance at higher levels (>1 mg/dL).⁷² Specific platform-related technical considerations include assessing variability and reproducibility (which can vary within a platform across different markers) and use of standards or repeated samples (ie, that can be used for evaluation of coefficients of variation and normalization across batches). Preanalytic considerations at the sample level can also significantly affect technical variability, such as differences in age of samples, sample processing (ie, time from collection to processing and storage), sample transfer conditions, and previous freeze/thaw cycles. Biological variation related to fasting status, time of day of collection affecting circadian variations, hormonal status, medications, and type of sample reflect “true” values but can similarly result in inaccurate findings, lack of validation, and complexities for clinical implementation. Differences in test characteristics and reporting in different concentrations can also create challenges for clinical interpretation and implementation (eg, lipoprotein[a] can be reported in either nmol/L or mg/dL).

Quality Control and Analytic Considerations With Omics

Technological advancements have exponentiated discovery science with assessment of omics in human cohorts. Although there are common issues to consider in evaluating the robustness and clinical utility of all novel risk factor– or biomarker-based models (ie, study design, reproducibility, discrimination, calibration, incremental predictive utility), there are special considerations for omics. Generation of omics data uses a wide variety of profiling platforms for high-throughput measurements of hundreds to millions of individual markers (eg, genomics, proteomics, metabolomics, epigenetics, the microbiome). These include traditional methods such as ELISA and mass spectrometry as well as evolving aptamer-based approaches.

Technical considerations are vital for analytic robustness and clinical translation. In addition to concerns regarding variability, reproducibility, and preanalytic factors relevant for all laboratory-based biomarkers, there are some unique considerations for omics. The diverse omics platforms offer various methods for use but vary with regard to dynamic range (ie, ability to quantify the range of concentration of biomarkers present in a given sample) and for relative versus absolute quantitation. Although useful for discovery science and mechanistic investigations, biomarker studies using relative quantitation platforms can suffer from issues of specificity due to off-target binding, inability to measure posttranslational modifications (eg, glycation), and a need for downstream absolute quantitation of identified markers for clinical translation. Platforms providing absolute quantitation are limited by the number of analytes and biological pathways represented and a need for larger amounts of samples.

Beyond sample-related and technical considerations, statistical analysis and model development of omics measurements also present unique considerations. Careful and consistent preprocessing of data from omics profiling is not only vital for reproducibility and validation but needs to be incorporated into assessments of clinical utility; for example, biomarkers with values frequently below the lower limits of detection may have limited utility depending on the use case, and outlier influence can result in false-positive associations. Lack of normalization can result in batch effects and spurious findings.

In addition, omics discovery, by its nature, measures hundreds to millions of individual—but partially correlated—markers, resulting in an incredibly high volume of data that present unique analytic challenges, including multiple comparisons.^73,74 A priori attention to multiple comparisons testing is necessary to prevent type I error, which may include approaches to control the false discovery rate (eg, Benjamini-Hochberg procedure, Bonferroni), dimensionality reduction techniques (eg, factor analysis) or high-dimensional data regression modeling (eg, elastic net). However, a given study may desire to explicitly model collinearity in determining an overrepresented biological pathway in the discovery study. This consideration is particularly relevant when integrating multiple types of omics (eg, genomics, proteomics, metabolomics) into predictive models.^75,76 As with all prediction models, model discrimination, calibration, and whether identified biomarkers add incremental value over traditional models in internal and external data sets need to be evaluated.

Preanalytic and analytic considerations notwithstanding, unique aspects of clinical utility should also be considered for omics biomarkers.⁷⁷ Technical difficulties related to multiplexed assays or biomarker measurement include cost-prohibitive scaling, the need for large amounts of biospecimen, and complex processing, storage, or transport requirements that are not feasible for routine collection in clinical practice.^78,79 Furthermore, lack of clinician knowledge about omics technologies could hamper uptake and use for patient care. Each of these would limit clinical utility or implementation.

Risk Assessment With Genetic Testing

Precision medicine refers to the goal of personalizing risk assessment and prevention by accounting for the unique interplay of genetic, environmental, and lifestyle factors. Moving toward greater personalization of risk and, therefore, management is appealing, but the evaluation of prediction based on genetic risk (ie, PRS) should adhere to the same principles outlined for determining the predictive and clinical utility of models for incident CVD as for nongenetic biomarkers. Genetic studies also need to consider ancestry in study design and in analysis, as genetic variant frequencies vary by ancestry, with variable performance of PRS across different ancestry groups.⁸⁰ Given the increasing number of PRS developed for coronary heart disease and other CVD subtypes, determining how each PRS differentially classifies the same individual should be evaluated (ie, intraclass coefficient, κ).³⁰ In contrast with population-based approaches, genetic testing for affected individuals (phenotype+) with monogenic or single-gene cardiovascular disorders (eg, familial hyperlipidemia, dilated cardiomyopathy, hypertrophic cardiomyopathy) and first-degree relatives (ie, cascade testing) is recommended to inform further risk stratification.⁸¹

Life Course Perspectives in CVD Risk Prediction

Risk prediction for CVD traditionally has focused on middle-aged or older adults over a 10-year time horizon. However, ample observational evidence supports that risk for CVD begins early in the life course. Therefore, how the prevalence and distribution of biomarkers, and, therefore, variation of their performance across the life course needs to be considered. For example, PRS may have relatively better performance in younger rather than older individuals, with increase in C-statistic when added to PCEs of 0.063 in those <55 years of age versus only 0.029 in those ≥55 years of age in one large cohort study.⁸² Furthermore, guidelines recommend the consideration of CAC when there is uncertainty about initiation of statin therapy for refining risk in those at borderline to intermediate predicted risk of ASCVD.⁹ However, CAC is unlikely to have clinical utility if used widely among young adults (ie, <40 years of age), in whom the prevalence of CAC is low, or among older adults (ie, >75 years of age), in whom it is high.⁸³ Once CAC is present, repeat scanning along the life course will likely have minimal utility, because CAC consistently and predictably increases with age.^84,85

Survivor bias may also affect the performance of biomarkers or models when evaluated in older cohorts, as informative individuals who develop disease may have died and thus are not represented in the data set. Thus, when reporting biomarkers or models, the age group in which they have been evaluated and to which they apply should be clear, and efforts should be made to assess their utility where clinically reasonable. For example, incident CVD is rare in adults <30 years of age, and therefore prediction of CVD events in this age range is unlikely to have utility or be cost-effective for short-term risk assessment ≤10 years given the low absolute risk estimates. However, prediction of intermediate risk factor development may have greater utility in this age range. Most current CVD risk models are constructed using risk factors measured at a single time point. Repeated measures for most risk factors appear to add modestly to risk prediction and add complexity in model validation and clinical application.^86–88 In addition, a risk factor measured once in early adulthood seems to accurately capture the cumulative decades-long exposure through middle age.⁸⁹

AI and ML in CVD Risk Prediction

The complexity of data elements that can be integrated in the development of a risk model has traditionally been constrained by the use of a limited set of predictors, often guided by clearly definable or established CVD risk factors.⁹⁰ Computational limitations have also required that the tools be deployable in clinical settings without infrastructure to automate computation of the risk scores to ensure clinical utility. However, 2 concurrent trends in health care now offer the framework for integration of additional data markers and tools if they meet the predictive and clinical utility criteria outlined previously⁹¹: the broad digitization of health care data from both consumer devices and within the EHR and the growth of computational methods and infrastructure (ML and AI) that can process large and complex data, including from wearables, imaging, and EHRs, and identify patterns to predict risk of incident CVD from both structured and unstructured fields.

Predictive and clinical utility criteria to evaluate emerging predictive models that have leveraged advanced analytics, encompassing ML and AI, should follow the same framework outlined for traditional risk models. These may include those that use discrete tabular data elements similar to those used by traditional statistical methods,^92,93 as well as other, more advanced models that leverage rich unstructured data from ECG, cardiac imaging, or unstructured text in the EHR (Figure 2). Additional considerations may be warranted, such as assessment of image quality, before AI model development. Several examples of AI models have been published recently.^94,95 A recent study demonstrated that a single-lead ECG that can be acquired on a portable, wearable device predicted the risk of incident HF with similar performance as PREVENT-HF in the UK Biobank.⁹⁶

Figure 2. Example of a machine learning—based approach to integrate multimodal data to predict risk of incident cardiovascular disease.

Representative machine learning framework to integrate multimodal structured and unstructured data, including imaging, genomics, and text from clinical encounter documentation for risk prediction. This integrative approach enables the combination of diverse data sources and modalities. Echo indicates echocardiography; and MRI, magnetic resonance imaging.

There are several additional considerations for predictive models that leverage ML and AI. A majority of the models have been developed within health systems, using data opportunistically collected on both exposures and outcomes. This introduces several biases that may then be amplified in these tools, including selection, confounding by indication, and ascertainment, which may propagate bias when these algorithms are deployed. This underscores the importance of fairness frameworks, assessment of algorithmic bias, and external validation of AI models to ensure calibration across subgroups.⁹⁷ Second, when locally derived AI models are used in clinical practice, recurrent local validation becomes increasingly important to regularly assess and monitor performance, particularly calibration.⁹⁸ Third, AI tools can only be clinically useful if they can be deployed automatically in clinical settings, particularly for “black box” tools that leverage complex data streams where clinicians cannot directly access the data or compute risk.⁹⁹

Effectiveness of CVD Risk Communication

For a risk tool to truly be clinically useful in the clinical setting, effective risk communication strategies are needed to support shared decision-making between clinicians and patients, which was the focus of a recent American Heart Association scientific statement.¹⁰⁰ However, risk communication is challenging, and multiple factors affect a patient’s perception of risk, including the method used by clinicians in presenting this information and which metric is used (eg, numeric format). Timeframe (eg, 10-year, lifetime) of risk has also been shown to influence perception of risk severity, health behavior changes, and intention to initiate treatment.¹⁰¹

Optimal communication of CVD risk supports shared decision-making or the partnership between the clinician and patient to share responsibility for the medical decision.¹⁰⁰ Evidence suggests that presentation of statistics in shared decision-making interventions (including communication of CVD risk) should include both qualitative and quantitative formats, using diagrams or pictures (eg, icon arrays).¹⁰² Alternative formats to present and communicate risk may be more appealing or interpretable, including percentiles and “heart age,” which allow individuals to compare their risk with that of their peers.^103–105 Studies should evaluate the most effective means to communicate risk information in a new marker or model to optimize risk perception and adherence to preventive therapies.

CONCLUSIONS

Evaluating the growing body of models, biomarkers, and tools that are available and emerging to quantify risk for incident CVD requires a standardized framework to rigorously assess predictive and clinical utility overall and in subgroups to ensure generalizable and equitable performance. This evaluation should also incorporate consideration of cost and complexity in case of clinical implementation, as well as best practices for risk communication. Due to the growth of high-dimensional data that can be integrated to estimate risk across various disciplines (eg, genomics, proteomics, metabolomics) and new computational approaches (eg, AI, ML), the current scientific statement serves as a timely update and includes sample templates for investigators to assist in study design and reporting (Supplemental Table 4).¹⁶ The examples within highlight that integrating larger volumes of information or more data does not necessarily equate to improved predictive or clinical utility for assessing risk of incident CVD. Whereas advances in our understanding of the pathophysiology of disease and emergence of novel platforms and tools offer potential opportunities, there are also challenges that require a systematic approach to understanding the role, relevance, and utility of novel biomarkers or models in risk assessment. Identifying whether newer approaches generate orthogonal and incremental risk information compared with what is currently assessed in contemporary multivariable risk models based on traditional CVD risk factors is necessary but insufficient. Updating practice paradigms for the primary prevention of CVD to incorporate novel approaches to risk assessment will also require careful assessment of the clinical utility of these approaches when compared with standard approaches that consider cost, scalability, and, ultimately, the ability to improve patient outcomes.

Supplementary Material

Supplemental Material is available at https://www.ahajournals.org/doi/suppl/10.1161/CIR.0000000000001401