Evaluating the American Heart Association/American College of Cardiology Guideline—Recommended and Contemporary Pretest Probability Models in a Mixed Asian Cohort: The Contribution of Coronary Artery Calcium

Abstract

Background

Most pretest probability (PTP) tools for obstructive coronary artery disease (CAD) were Western ‐developed. The most appropriate PTP models and the contribution of coronary artery calcium score (CACS) in Asian populations remain unknown. In a mixed Asian cohort, we compare 5 PTP models: local assessment of the heart (LAH), CAD Consortium (CAD2), risk factor‐weighted clinical likelihood, the American Heart Association/American College of Cardiology and the European Society of Cardiology PTP and 3 extended versions of these models that incorporated CACS: LAH_(CACS), CAD2_(CACS), and the CACS‐clinical likelihood.

Methods and Results

The study cohort included 771 patients referred for stable chest pain. Obstructive CAD prevalence was 27.5%. Calibration, area under the receiver‐operating characteristic curves (AUC) and net reclassification index were evaluated. LAH clinical had the best calibration (χ² 5.8; P=0.12). For CACS models, LAH_(CACS) showed least deviation between observed and expected cases (χ² 37.5; P<0.001). There was no difference in AUCs between the LAH clinical (AUC, 0.73 [95% CI, 0.69–0.77]), CAD2 clinical (AUC, 0.72 [95% CI, 0.68–0.76]), risk factor‐weighted clinical likelihood (AUC, 0.73 [95% CI: 0.69–0.76) and European Society of Cardiology PTP (AUC, 0.71 [95% CI, 0.67–0.75]). CACS improved discrimination and reclassification of the LAH_(CACS) (AUC, 0.88; net reclassification index, 0.46), CAD2_(CACS) (AUC, 0.87; net reclassification index, 0.29) and CACS‐CL (AUC, 0.87; net reclassification index, 0.25).

Conclusions

In a mixed Asian cohort, Asian‐derived LAH models had similar discriminatory performance but better calibration and risk categorization for clinically relevant PTP cutoffs. Incorporating CACS improved discrimination and reclassification. These results support the use of population‐matched, CACS‐inclusive PTP tools for the prediction of obstructive CAD.

Nonstandard Abbreviations and Acronyms

AHA/ACC‐PTP

American Heart Association/American College of Cardiology pretest probability

CACS

coronary artery calcium score

CACS‐CL

CACS‐weighted clinical likelihood

CAD2

Coronary Artery Disease Consortium

ESC‐PTP

European Society of Cardiology pre‐test probability

LAH

local assessment of the heart

PTP

pretest probability

RF‐CL

risk factor‐weighted clinical likelihood

Clinical Perspective.

What Is New?

• This is the first study that uniquely compares the performance of 5 pretest probability (PTP) models: local assessment of the heart, Coronary Artery Disease Consortium, risk factor‐weighted clinical likelihood, the American Heart Association/American College of Cardiology PTP, and the European Society of Cardiology PTP and extended versions of these models that incorporated the coronary artery calcium score (CACS): local assessment of the heart_(CACS), Coronary Artery Disease Consortium_(CACS), and the CACS‐clinical likelihood in a broadly representative and ethnically diverse symptomatic Asian cohort.

• In a mixed Asian cohort, the Asian‐derived local assessment of the heart models had better calibration and risk categorization compared with Western‐based PTP models, despite similar discrimination.

What Are the Clinical Implications?

• This study reinforces the incremental value of incorporating CACS to improve model discrimination and reclassification.

• Discriminative performance is important but alone is insufficient in selecting a PTP model.

• Population‐matched, CACS‐inclusive PTP tools should be selected for the prediction of obstructive CAD to ensure real‐world clinical relevance.

Angina symptoms are common, with a lifetime prevalence up to 40%, resulting in millions of individuals undergoing evaluation for obstructive coronary artery disease (CAD) each year. ¹ , ² Pretest probability (PTP) assessment is part of the evaluation strategy and is guideline recommended. ¹ , ³

Various versions of the updated Diamond‐Forrester PTP tool, which included sex, age, and symptoms, have been used in the prediction of obstructive CAD. Other models incorporating risk factors have shown increased discrimination and reclassification of patients, including the CAD Consortium 2 (CAD2) model. ⁴ , ⁵ , ⁶ , ⁷ Recently, tools such as the risk factor‐weighted clinical likelihood model (RF‐CL) have been shown to appropriately recategorize very low (≤5%) clinical likelihood patients, which would potentially avoid unnecessary testing. ⁸ , ⁹ The 2021 American Heart Association/American College of Cardiology (AHA/ACC)/American Society of Echocardiography/American College of Chest Physicians/Society for Academic Emergency Medicine/Society of Cardiovascular Computed Tomography/Society for Cardiovascular Magnetic Resonance Guideline for the Evaluation and Diagnosis of Chest Pain includes a sex‐ and age‐stratified PTP (AHA/ACC‐PTP). ¹ This is an amalgamated version of the European Society of Cardiology (ESC) guideline PTP tool (ESC‐PTP), itself developed using European and American cohorts. ³ , ¹⁰ Given the clinical utility of the Agatston coronary artery calcium score (CACS) as a gatekeeper to further testing, ¹¹ it has been incorporated in PTP tools such as the CAD2_(CACS) model and the CACS‐weighted clinical likelihood model (CACS‐CL), resulting in its guideline recommendation. ¹ , ⁸ , ¹² , ¹³

The performance of a PTP tool depends on the CAD prevalence in the investigated population. Tools derived and initially well validated in Western cohorts have performed less well in Asian contexts. ¹³ , ¹⁴ , ¹⁵ Singapore has a population comprising 3 major Asian ethnicities (Chinese [74.3%], Malay [13.5%], and Indian [9.0%]) that provide a unique snapshot of the genetic diversity across East, Southeast, and South Asia. ¹⁶ , ¹⁷ There is a paucity of evidence comparing contemporary Western tools to a model in the Asian context. Similarly, the role of CACS in this cohort has not been extensively evaluated.

The recently developed local assessment of the heart (LAH) score is a PTP tool that was calibrated to a Singapore‐based mixed Asian population with the optional incorporation of CACS in estimating obstructive CAD defined by coronary computed tomography angiography (CCTA). ¹³ Although providing superior discrimination compared with CAD2, the validity of the LAH models has not been fully assessed.

In this study, we evaluate the Asian‐developed LAH score in a separate mixed Asian cohort, comparing its performance to counterpart Western‐developed CAD2 models, the contemporary RF‐CL and CACS‐CL scores, and the guideline‐recommended AHA/ACC‐ and ESC‐PTP models. We also evaluate the contribution of CACS in the performance of these PTP tools.

Methods

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Study Population

This retrospective cohort study included all consecutive patients with stable chest pain who underwent cardiac CT at a tertiary cardiac institution in Singapore from July 1, 2015 to October 31, 2017. It did not include any patients used to develop the LAH algorithm. Patients with prior cardiovascular events were excluded. Patients who underwent clinically indicated CT scans for suspected CAD were eligible. Of a total of 2444 patients, 1151 patients with incomplete symptom documentation were removed. A further 522 patients who were used in the prior LAH development were removed and used for the machine learning (ML) model development, leaving a total of 771 final subjects for this study (Figure 1A). A waiver of informed consent was granted and this study was approved by the Central Institutional Review Board. Hypertension, diabetes, hyperlipidemia, smoking, family history of CAD and chest pain status were ascertained using 2‐way verification.

Figure 1. Pretest probability models for coronary artery disease and the study cohort.

A, A total of 771 patients were included in the analysis. Patients with incomplete symptom documentation and those used for derivation were excluded. B, This study included 5 PTP models (LAH clinical, CAD2 clinical, RF‐CL, AHA/ACC‐PTP, and ESC‐PTP) and 3 extended versions of these models that incorporated CACS (LAH_(CACS), CAD2_(CACS), and the CACS‐CL). AHA/ACC‐PTP indicates American Heart Association/American College of Cardiology pretest probability; CACS, coronary artery calcium score; CACS‐CL, CACS‐weighted clinical likelihood; CAD, coronary artery disease; CAD2, coronary artery disease consortium; ESC‐PTP, European Society of Cardiology pretest probability; LAH, local assessment of the heart; ML, machine learning; and RF‐CL, risk factor‐weighted clinical likelihood.

Computed Tomography Acquisition and Interpretation

Scans were performed in accordance with the Society of Cardiovascular Computed Tomography recommendations. ¹⁸ , ¹⁹ CCTA was performed with prospective electrocardiographic gating using the following parameters: gantry rotation time of 350 to 400 ms, tube voltage of 100 to 120 kV, field of view at 57 to 159 mm, and a slice thickness of 0.5 mm. CAC images were acquired using 120 kVp, 300 to 600 mAs, prospective ECG gating, and 3/3 mm reconstructions.

All scans were analyzed by level III or equivalent trained cardiologists or radiologists with extensive experience in cardiac CT analysis according to current guidelines. Each coronary segment with a >2 mm diameter was analyzed for the presence of coronary atherosclerosis, and each lesion was quantified by visual estimation into 3 categories: no disease, nonobstructive disease (1%–50% stenosis), and obstructive disease (>50% stenosis). Obstructive CAD was defined as at least 1 segment with a lesion with >50% stenosis. Obstructive CAD was further characterized as left main, 1‐, 2‐, and 3‐vessel disease. CACS was quantified using the Agatston method. ¹¹

Risk Scoring for Prediction of Obstructive Coronary Artery Disease

The Singapore‐developed LAH clinical model includes the following variables: age, sex, symptom typicality, diabetes, hypertension, hyperlipidemia, and smoking status. The LAH_(CACS) model adds the continuous Agatston score (CACS) to these variables. ¹³ The LAH was selected as it was developed using a local cohort most similar to the current study and derived using CCTA. The CAD2 clinical and CAD2_(CACS) models use the same variables as their LAH counterparts and have previously been externally validated. ⁵ , ⁶ , ⁷ , ²⁰ The RF‐CL and CACS‐CL were selected as contemporary nonlocal comparator models to the LAH and have been shown to improve discriminatory performance of obstructive CAD at ICA compared with the CAD2. Just like the ESC‐PTP and AHA/ACC‐PTP, the CACS‐CL has been included in guidelines. ¹ , ¹⁰ , ²¹ The RF‐CL model uses the ESC‐AHA/ACC model of age, sex, and symptoms and further adds weightage for the number of risk factors (0–1, 2–3, or 4–5) in a logistic regression model. ⁸ The CACS‐CL model additionally adds weightage for these risk factors across strata of increasing CACS (1–9, 10–99, 1–399, 400–999, ≥1000). ⁸ AHA/ACC‐PTP and ESC‐PTP are table‐based PTP models from pooled analysis of 3 large‐scale studies (Figure 1B).

The LAH models were developed to predict the prevalence of 50% diameter stenosis at CCTA whereas the CAD2 and the CL models were originally developed to predict the prevalence of 50% diameter stenosis using invasive coronary angiography. AHA/ACC‐PTP and ESC‐PTP were based on the observed prevalence of obstructive CAD using a mixed reference standard of invasive coronary angiography and CCTA.

Statistical Analysis

Mean±SD or median (interquartile range) were used for continuous data. Categorical data were reported as percentages where appropriate. The unpaired t test for continuous data and the χ² test for categorical data were used for comparison between groups. To assess the calibration of each PTP tool, observed and predicted risk was computed based on categories defined by quintiles of predicted risk. The Hosmer–Lemeshow goodness‐of‐fit chi‐square statistic across each quintile was calculated to measure the agreement between observed and predicted events.

Receiver operating characteristic curves and the corresponding area under the curve (AUC) values were generated to compare the discriminatory power of CACS for predicting obstructive CAD. This was done for the overall cohort, and for subgroups based on age, sex, and ethnicity. Respective AUCs were compared using the Hanley and McNeil method for paired data. ²²

To evaluate the incremental value of CACS as a predictor of obstructive CAD, the AUC values of the CACS‐incorporating models (LAH_(CACS), CAD2_(CACS), and CACS‐CL) were compared with their counterparts (LAH clinical, CAD2 clinical, and RF‐CL respectively). ²³ To assess potential differences in decision‐making, reclassification was evaluated using net reclassification improvement. ²³ , ²⁴ This was assessed in a binary manner to identify low‐risk patients, defined as those not requiring further testing as recommended by contemporary guidelines. This was defined as a very low PTP, using a 5% cutoff for a test or no‐test threshold. In the ESC guidelines, this is the cutoff for deferring testing, whereas in the AHA/ACC guidelines, testing may be considered with PTP ≤15% based on clinical judgment. ¹ , ³ Assessment of net reclassification improvement used the non‐CACS model (eg, LAH clinical) as a base and compared the CACS‐incorporating counterpart model (eg, LAH_(CACS)). Statistical significance was defined as a 2‐tailed P value of <0.05. Statistical analyses including receiver operating characteristic curves and AUC values were generated using R version 3.6.3. To evaluate the role of CACS and other variables in model prediction, one clinical ML model and one CACS‐containing ML model were developed using extreme gradient boosting (XGBoost). Both models were trained on the 522‐patient cohort used for the original LAH model development to maintain consistency in comparison to the other models. Their performance was subsequently assessed on the 771 final subjects. Shapley Additive Explanations, a game‐theory based interpretability method, was used for feature importance analysis. Both ML model development and feature importance analysis were performed using Python 3.8.3 (Data S1).

Results

Clinical Characteristics

Mean age was 57.0±11.4 years; 47.2% were female (Table 1). Ethnicity was 75.0% Chinese, 9.7% Indian, 7.0% Malay, and 8.3% other Asian ethnicities (Bangladeshi, Filipino, Eurasian). More than half of the patients (59.0%; 455/771) had CACS>0 and were more likely to be older, male, and have hypertension, diabetes, or hyperlipidemia. The median PTP using the LAH clinical score was 21.5% (interquartile range: 10.9–39.2) and was significantly higher in the CACS >0 group [33.5% (interquartile range: 18.8–47.8)] compared with the CACS=0 group [11.5% (interquartile range: 6.7–19.8)] (P<0.001). Overall, 36.2% had no CAD, 36.3% had nonobstructive CAD, and 27.5% had obstructive CAD. In the CACS=0 group, the prevalence of no CAD, nonobstructive CAD, and obstructive CAD was 84.2%, 11.4%, and 4.4% respectively, whereas in the CACS>0 group, the prevalence of no CAD, nonobstructive CAD, and obstructive CAD was 2.9%, 53.6%, and 43.5% respectively.

Table 1.

Demographic and Clinical Characteristics of Patients With Zero CACS and Nonzero CACS

Variables	All patients (n=771)	CACS=0 (n=316)	CACS>0 (n=455)	P value
Demographics
Age, y	57.0±11.4	50.8±10.2	61.4±10.1	<0.001
Body mass index, kg/m2	25.9±5.1	25.3±5.0	26.3±5.1	0.008
Female sex	364 (47.2)	185 (58.5)	179 (39.3)	<0.001
Ethnicity				0.011
Chinese	578 (75.0)	219 (69.3)	359 (78.9)
Malay	54 (7.0)	29 (9.2)	25 (5.5)
Indian	75 (9.7)	33 (10.4)	42 (9.2)
Others*	64 (8.3)	35 (11.1)	29 (7.7)
Medical history and risk factors
Hypertension	347 (45.0)	86 (27.2)	261 (57.4)	<0.001
Diabetes	134 (17.4)	36 (11.4)	98 (21.5)	<0.001
Hyperlipidemia	481 (62.4)	157 (49.7)	324 (71.2)	<0.001
Smoking	151 (19.6)	52 (16.5)	99 (21.8)	0.083
Family history of coronary artery disease	298 (38.7)	119 (37.7)	179 (39.3)	0.692
Chest pain				0.011
Typical	58 (7.5)	23 (7.3)	35 (7.7)
Atypical	291 (37.7)	132 (41.8)	159 (34.9)
Nonanginal	159 (20.6)	74 (23.4)	85 (18.7)
Dyspnea	263 (34.1)	87 (27.5)	176 (38.7)
Coronary artery stenosis
Pretest probability†	21.5 [10.9, 39.2]	11.5 [6.7, 19.8]	33.5 [18.8, 47.8]	<0.001
No disease	279 (36.2)	266 (84.2)	13 (2.9)
Nonobstructive disease	280 (36.3)	36 (11.4)	244 (53.6)
Obstructive disease
1‐vessel disease	121 (15.7)	14 (4.4)	107 (23.5)
2‐vessel disease	54 (7.0)	0 (0.0)	54 (11.9)
3‐vessel disease	21 (2.7)	0 (0.0)	21 (4.6)
Left main disease	16 (2.1)	0 (0.0)	16 (3.5)

Values are n (%), mean±SD, or median [interquartile range]. CACS indicates coronary artery calcium score.

^*Others includes Bangladeshi, Filipino, Eurasian.

^† Pretest probability was calculated using local assessment of the heart clinical model.

Model Calibration

Calibration plots for the overall cohort using all 8 models are shown (Figure 2A). Of the non‐CACS models, the LAH clinical fitted best with no significant deviation between observed and expected cases (χ² 5.8; P=0.12). The least well‐fitted was RF‐CL (χ² 436.4; P<0.001), followed by ESC‐PTP (χ² 135.2; P<0.001), CAD2 clinical (χ² 108.2; P<0.001), and AHA/ACC‐PTP (χ² 38.2; P<0.001), respectively. Of the CACS incorporating models, LAH_(CACS) showed the least deviation between observed and expected cases (χ² 37.5; P<0.001), followed by CAD2_(CACS) (χ² 56.6; P<0.001) and CACS‐CL (χ² 295.9; P<0.001). Overall, the LAH clinical and LAH_(CACS) models most closely estimated risk. This was followed by the AHA/ACC‐PTP. All other models underestimated risk, with the RF‐CL and CAC‐CL models underestimating to the largest extent. In general, models incorporating CACS showed a better fit than their non‐CACS counterparts.

Figure 2. Calibration and risk categorization of the 8 models on the study cohort.

A, The calibration plot of the 8 models on the overall study cohort. B, Distribution of patients and obstructive CAD prevalence according to pretest probability value groups for the 5 clinical models. C, Distribution of patients and obstructive CAD prevalence according to pretest probability value groups for the 3 CACS models. AHA/ACC‐PTP indicates American Heart Association/American College of Cardiology pre‐test probability; CACS, coronary artery calcium score; CACS‐CL, CACS‐weighted clinical likelihood; CAD, coronary artery disease; CAD2, coronary artery disease consortium; ESC‐PTP, European Society of Cardiology pretest probability; LAH, local assessment of the heart; and RF‐CL, risk factor‐weighted clinical likelihood.

Distribution of Patients Using Clinical Cutoffs

The LAH clinical, CAD2 clinical, RF‐CL, AHA/ACC‐PTP, and ESC‐PTP categorized 6.1%, 24.4%, 45.1%, 7.3%, and 16.0% of subjects respectively into the ≤5% PTP group (Figure 2B; Table 2). The true prevalence of obstructive CAD at this PTP cutoff was 6.4%, 8.0%, 13.5%, 7.1%, and 10.6% for the LAH clinical, CAD2 clinical, RF‐CL, AHA/ACC‐PTP, and ESC‐PTP scores respectively. At the ≥50% PTP cutoff, the LAH clinical, CAD2 clinical, RF‐CL, AHA/ACC‐PTP, and ESC‐PTP categorized 13.7%, 5.2%, 0%, 2.7%, and 0.3% of subjects respectively. The true prevalence of disease at this PTP cutoff was 50.0%, 62.5%, 42.9%, and 50.0% for the LAH clinical, CAD2 clinical, AHA/ACC‐PTP, and ESC‐PTP scores respectively. The RF‐CL did not categorize any subjects as having a ≥50% PTP.

Table 2.

Distribution of Patients and Obstructive CAD Prevalence According to Pretest Probability Value Groups for the 8 Models

Probability of CAD	≤5%		>5% to ≤15%		>15% to ≤50%		>50%
	Number of patients	Obstructive CAD prevalence	Number of patients	Obstructive CAD prevalence	Number of patients	Obstructive CAD prevalence	Number of patients	Obstructive CAD prevalence
LAH clinical	47 (6.1%)	6.4%	233 (30.2%)	12.0%	385 (49.9%)	33.2%	106 (13.7%)	50.0%
CAD2 clinical	188 (24.4%)	8.0%	276 (35.8%)	21.7%	267 (34.6%)	41.9%	40 (5.2%)	62.5%
Risk factor‐weighted clinical likelihood	348 (45.1%)	13.5%	292 (37.9%)	31.8%	131 (17.0%)	55.0%	0 (0.0%)	N.A.
American Heart Association/American College of Cardiology PTP	56 (7.3%)	7.1%	269 (34.9%)	19.7%	415 (53.8%)	34.9%	21 (2.7%)	42.9%
European Society of Cardiology PTP	123 (16.0%)	10.6%	365 (47.3%)	21.6%	271 (35.1%)	43.5%	2 (0.3%)	50.0%
LAH(CACS)	354 (45.9%)	4.8%	64 (8.3%)	6.2%	128 (16.6%)	29.7%	225 (29.2%)	68.0%
CAD2(CACS)	352 (45.7%)	4.5%	119 (15.4%)	20.2%	189 (24.5%)	45.5%	111 (14.4%)	77.5%
CACS‐weighted clinical likelihood	429 (55.6%)	7.2%	136 (17.6%)	31.6%	183 (23.7%)	63.9%	23 (3.0%)	91.3%

The calibration for the Asian‐based LAH models were better suited across risk categories than the Western models, with better matches of predicted versus observed disease prevalence across the clinically relevant ≤5%, >5% to ≤15%, >15% to ≤50%, and <50% PTP cutoffs. CAD indicates coronary artery disease; CAD2, coronary artery disease consortium; CACS, coronary artery calcium score; LAH, local assessment of the heart; and PTP, pretest probability.

For the CACS models, the LAH_(CACS), CAD2_(CACS), and CACS‐CL categorized 45.9%, 45.7%, and 55.6% of subjects respectively into the ≤5% PTP cutoff (Figure 2C; Table 2). The true prevalence of obstructive CAD at this PTP cutoff was 4.8%, 4.5%, and 7.2% for the LAH_(CACS), CAD2_(CACS), and CACS‐CL respectively. At the ≥50% PTP cutoff, the LAH_(CACS), CAD2_(CACS), and CACS‐CL categorized 29.2%, 14.4%, and 3.0% of subjects respectively. The true prevalence of disease at this PTP cutoff was 68.0%, 77.5%, and 91.3% for the LAH_(CACS), CAD2_(CACS), and CACS‐CL respectively.

Discriminative Performance

Among the non‐CACS models, there was no difference in discriminative performance between the LAH clinical, CAD2 clinical, RF‐CL, and ESC‐PTP for the overall cohort (Figure 3). However, the AHA/ACC‐PTP model performed worse than all others with an AUC of 0.67 (95% CI, 0.63–0.71) (P<0.05 for all). The LAH clinical model obtained an AUC of 0.73 (95% CI, 0.69–0.77), whereas the CAD2 clinical model obtained an AUC of 0.72 (95% CI, 0.68–0.76), the RF‐CL AUC was 0.73 (95% CI, 0.69–0.76), and the ESC‐PTP AUC was 0.71 (95% CI, 0.67–0.75). This pattern remained true across age‐ and sex‐based subgroups (Table 3) (P=NS for all). Discriminatory performance remained consistent for the LAH clinical, CAD2 clinical, RF‐CL, and AHA/ACC‐PTP scores across age‐, sex‐, and ethnicity‐based subgroups (P=NS for all), whereas ESC‐PTP performed worse in females compared with males (Table 3; Table S1). Comparing performance between CACS models, there was no difference between LAH_(CACS), CAD2_(CACS), and CACS‐CL using the overall cohort (Table 3). Among the age ≥55 years group, LAH_(CACS) (AUC, 0.88 [95% CI, 0.85–0.91]) performed better than the CAD2_(CACS) (AUC, 0.85 [95% CI, 0.81–0.88]) and the CACS‐CL (AUC, 0.86 [95% CI, 0.82–0.89]) (P<0.05 for all). There was no further difference between age‐, sex‐, and ethnicity‐based subgroups within any of the CACS models.

Figure 3. The receiver operating characteristic curves of the 8 models on the study cohort.

The incorporation of CACS improved the models' discriminatory performance compared with their clinical counterparts. AHA/ACC‐PTP indicates American Heart Association/American College of Cardiology pretest probability; CACS, coronary artery calcium score; CACS‐CL, CACS‐weighted clinical likelihood; CAD, coronary artery disease; CAD2, coronary artery disease consortium; ESC‐PTP, European Society of Cardiology pretest probability; LAH, local assessment of the heart; and RF‐CL, risk factor‐weighted clinical likelihood.

Table 3.

Discriminatory Performance of the 8 Models on the Study Cohort and Subgroups

	LAH clinical	CAD2 clinical	RF‐CL	AHA/ACC‐PTP	ESC‐PTP	LAH(CACS)	CAD2(CACS)	CACS‐CL
All (n=771)	0.73 (0.69–0.77)	0.72 (0.68–0.76)	0.73 (0.69–0.76)	0.67 (0.63–0.71)	0.71 (0.67–0.75)	0.88 (0.85–0.91)	0.87 (0.84–0.90)	0.87 (0.84–0.90)
By sex
Male (n=407)	0.73 (0.68–0.77)	0.73 (0.68–0.77)	0.73 (0.68–0.78)	0.64 (0.59–0.70)	0.71 (0.66–0.76)	0.88 (0.84–0.92)	0.87 (0.84–0.91)	0.87 (0.84–0.90)
Female (n=364)	0.68 (0.61–0.74)	0.67 (0.60–0.73)	0.66 (0.60–0.73)	0.58 (0.51–0.66)	0.62 (0.55–0.69)	0.86 (0.80–0.91)	0.85 (0.80–0.90)	0.85 (0.80–0.90)
P value	0.231	0.176	0.116	0.188	0.041*	0.484	0.569	0.571
By age, y
<55 (n=318)	0.71 (0.62–0.79)	0.71 (0.63–0.80)	0.71 (0.64–0.79)	0.62 (0.53–0.71)	0.65 (0.56–0.74)	0.81 (0.73–0.90)	0.86 (0.79–0.92)	0.85 (0.78–0.91)
≥55 (n=453)	0.66 (0.61–0.71)	0.66 (0.60–0.71)	0.66 (0.61–0.71)	0.61 (0.56–0.66)	0.64 (0.59–0.69)	0.88 (0.84–0.91)	0.85 (0.81–0.88)	0.86 (0.82–0.89)
P value	0.336	0.249	0.262	0.872	0.870	0.160	0.798	0.779

AHA/ACC‐PTP indicates American Heart Association/American College of Cardiology pretest probability; CACS, coronary artery calcium score; CACS‐CL, CACS‐weighted clinical likelihood; CAD2, coronary artery disease consortium; ESC‐PTP, European Society of Cardiology pretest probability; LAH, local assessment of the heart; and RF‐CL, risk factor‐weighted clinical likelihood.

^* P value <0.05.

Using a 5% cutoff, the LAH clinical and the AHA/ACC‐PTP had the highest sensitivities of 0.99 (95% CI, 0.96–1.00) and 0.98 (95% CI, 0.95–0.99) respectively, with no significant difference between them (P=0.41) (Table S2). This was followed by the CAD2 clinical and ESC‐PTP with sensitivities of 0.93 (95% CI, 0.89–0.96) and 0.93 (95% CI, 0.89–0.96) respectively, and last was RF‐CL, with a sensitivity of 0.78 (95% CI, 0.72–0.83) (P<0.05 for all). The negative predictive values for LAH (0.94 [95% CI, 0.82–0.99]) and CAD 2 clinical (0.92 [95% CI, 0.87–0.95]) models were higher than RF‐CL (0.86 [95% CI, 0.82–0.90]; P<0.05). The reverse trend was observed for specificity and positive predictive value. Otherwise, there was no significant difference between the non‐CACS models. Similar patterns were observed for the CACS models. In all cases, the incorporation of CACS improved the specificity and positive predictive value (P<0.001 for all).

Incorporation of Coronary Artery Calcium Score

Incorporation of the CACS score in the LAH_(CACS) model improved performance when compared with its LAH clinical counterpart amongst the overall cohort (AUC 0.88 versus 0.73). This was also observed when comparing the CAD2_(CACS) model to the CAD2 clinical version (AUC 0.87 versus 0.72) and the CACS‐CL model to the RF‐CL model (AUC 0.87 versus 0.73) (P<0.001 for all) (Figure 3).

All CACS‐incorporating models demonstrated improved reclassification compared with their respective clinical counterparts. At a 5% cutoff, the categorical net reclassification improvement was 0.46 (95% CI, 0.40–0.51) for LAH_(CACS) model; 0.29 (95% CI, 0.23–0.34) for CAD2_(CACS) model; and 0.25 (95% CI, 0.18–0.31) for CACS‐CL. Among 515 patients without obstructive CAD classified as high risk by LAH clinical, the LAH_(CACS) model reclassified a significant proportion (57.5%, n=296) into the low‐risk group (Table 4). ML analysis showed that CAC‐related features had the most contributory role in the prediction of obstructive CAD (Figures S1–S3).

Table 4.

Reclassification Table Comparing LAH Clinical With LAH_(CACS)

	Risk groups by LAH(CACS)		Total
	Low	High
Patients with obstructive CAD
Risk groups by LAH clinical
Low	1	2	3
High	16	193	209
Total	17	195	212
Patients without obstructive CAD
Risk groups by LAH clinical
Low	41	3	44
High	296	219	515
Total	337	222	559

With the incorporation of CACS, LAH_(CACS) model reclassified a significant proportion of patients into the low‐risk group. CACS indicates coronary artery calcium score; and LAH, local assessment of the heart.

Discussion

To determine the dependence of PTP model accuracy and utility on population characteristics, we used a mixed Asian cohort to compare the performance of scores derived in Western populations, RF‐CL, AHA/ACC‐PTP, ESC‐PTP, and CACS‐CL, with an Asian‐derived score, LAH. Despite similar discriminative performance, the more closely calibrated and Asian‐based LAH models demonstrated better categorization of subjects into clinically relevant cutoffs, sensitivity, and negative predictive value. For all models, discrimination, specificity, positive predictive value, and net reclassification improvement were improved by incorporating CACS. ML analysis showed the prioritization of CACS information over symptoms in predicting obstructive CAD.

Guidelines have advocated PTP score use in assessing patients with stable chest pain. ¹ , ³ However, most scores have consistently performed less well when applied to Asian cohorts. ¹³ , ²⁵ , ²⁶ The calibration for the Asian‐based LAH models were better suited across risk categories than the Western models, with better matches of observed versus expected across the clinically relevant ≤5%, >5% to ≤15%, >15% to ≤50%, and <50% PTP cutoffs. This is despite similar discrimination in the current study, as well as comparable discriminative performances in prior respective individual derivation and validation studies. ⁸ , ¹³ , ²⁷

In selecting a PTP model, discriminative performance is important but alone is insufficient, as clinical action for test or no‐test decisions is dependent on specific thresholds. The ESC guidelines recommend deferred testing for <5% PTP, and the AHA/ACC guidelines recommend diagnostic noninvasive testing at the >15% PTP cutoff. ¹ , ³ Matching the PTP tool to risk groups of the population being studied is of pertinence. ²⁸

The Asian‐developed LAH is a better‐suited PTP tool in this study due to the following provisions. First, the prevalence of obstructive CAD in the studied population is 27.5%, comparable to the LAH derivation (24.9%) but markedly different from the observed prevalence of CAD in the RF‐CL (8.8%), AHA/ACC‐ and ESC‐PTP (14.9%), and CAD2 (57%) derivation cohorts. ⁵ , ⁸ , ¹⁰ , ¹³ Second, within the context of PTP scores, the coefficients and weightage given to the variables are dependent on the derivation population. As evidenced by the ML and Shapley Additive Explanations analysis, Asian‐developed LAH and ML clinical models emphasized the nonmodifiable risk among all clinical features. In contrast, Western‐developed CAD2 prioritized symptoms. This partly explains the systemic mismatch of the Western models.

Finally, the current study uses CCTA as a reference standard. Although this is similar to the original LAH derivation and AHA/ACC and ESC derivations, the CAD2, RF‐CL, and CACS‐CL models were developed using invasive coronary angiography as a reference. Compared with these prior non‐Asian studies with obstructive CAD prevalences of 8.8% to 10.1%, our current study has a higher prevalence of 27.5%. ⁸ This is more comparable with another large Asian study with a 27.9% prevalence. ²⁹ Asian ethnic subgroups have consistently been shown to have a higher prevalence of obstructive CAD, even in invasive coronary angiography studies. ³⁰ , ³¹ , ³² Nevertheless, similar to other studies using a single reference modality for CAD diagnosis, there may be limitations to generalizability and further evaluation is warranted.

Incorporating CACS improved PTP discrimination and appropriately downgraded risk in the no obstructive CAD subjects. The incorporation of CACS into PTP models has previously shown improved performance within an Asian context. ²⁹ The improved redistribution of patients across the clinically relevant ≤5% and <50% PTP cutoffs may result in significant reductions in unnecessary testing and costs. CAC supersedes all other clinical variables in prediction of obstructive CAD. ⁸ , ¹² , ¹³ CAC quantification reveals the result of all exposomes toward the pathological process in CAD, rather than indirectly using a limited number of contributary risk factors as done by the non‐CACS PTPs, thus accounting for its superior performance across sex‐ and age‐based cohorts.

The usage of CACS antecedent to downstream diagnostic testing has been well established. This further testing may not obligatorily be CCTA but could instead be any other anatomical or functional test. In recognition of that, the 2021 AHA/ACC/American Society of Echocardiography/American College of Chest Physicians/Society for Academic Emergency Medicine/Society of Cardiovascular Computed Tomography/Society for Cardiovascular Magnetic Resonance Guideline for the Evaluation and Diagnosis of Chest Pain has PTP options with and without CACS. ¹ However, clinical practice may have additional pragmatic considerations beyond guidelines. As such, the current study similarly allows options with and without CACS: (!) a sequential model that initially uses a PTP tool without CACS (LAH clinical) that then guides CAD assessment into further testing options, including CACS, CCTA, and other anatomical or functional testing; and (2) a PTP tool that incorporates CACS from the start (LAH_(CACS)). This is to ensure flexibility of application to varying clinical needs.

Limitations

There are limitations to this study. Patients were referred for cardiac CT and selection bias cannot be fully excluded. However, we sought to examine the performance of PTP tools in a population that uniformly underwent evaluation for CAD with minimal posttest referral bias. Thus, the population studied herein is representative of patients where PTP tools may have optimal utility. The patients recruited for this validation study are from within the same medical system as those used for the previously published derivation cohort and thus may share common demographic features. This may have introduced bias in comparisons with Western models. However, this also emphasizes the need for close calibration between the PTP tool and the population studied. Further validation studies in completely independent Asian cohorts are in progress by this group. This study did not evaluate prognostication for cardiovascular events. However, the incorporation of CACS into risk scores has however been shown to significantly improve cardiovascular event prediction in symptomatic patients in numerous other studies, including those involving Asian cohorts. ²¹ , ²⁹ , ³³ Nevertheless, further studies are needed. Although this study uses a mixed Asian cohort, it does not specifically assess the role of ethnicity as a variable in the prediction of CAD. Current work is underway to assess the contribution of ethnicity and geography in CAD evaluation. Similarly, our data cannot directly address the accuracy of the LAH in Asians living in the United States or Europe.

Conclusions

In this study of a mixed Asian cohort, the Asian‐derived LAH models demonstrated superior performance compared with Western‐based PTP models, despite similar discrimination. The incorporation of CACS improved overall model performance. To ensure real‐world clinical relevance, population‐matched PTP tools should be selected for the prediction of obstructive CAD.

Sources of Funding

This study was supported by the National Medical Research Council (NMRC) of Singapore Centre Grant [Program for Transforming and Evaluating Outcomes in Cardiometabolic disease (PROTECT), Grant number: CG21APR1006], the National Medical Research Council (NMRC) of Singapore Transitional Award Grant (Improving Obstructive Coronary Artery Disease and Cardiovascular Risk Prediction Using Deep Learning Analysis on Coronary Artery Calcium Imaging, Grant number: TA21nov‐0001), and the National Medical Research Council (NMRC) of Singapore Collaborative Centre Grant IMPACT‐2 (Implementing a Partnership for Cardiovascular Trials‐2) Program, Grant number: NMRC/CG2/001a/2021‐NHCS]. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the article; and decision to submit the article for publication.

Disclosures

Dr Michelle C. Williams is supported by the British Heart Foundation and has given talks for Canon Medical Systems, Siemens Healthineers, and Novartis. Dr Ming‐Yen Ng has received educational grants from Circle Cardiovascular Imaging, Bayer, GE, TeraRecon, Arterys, and Lode, as well as speaker's fees from Boehringer Ingelheim and Circle Cardiovascular Imaging. The remaining authors have no disclosures to report.

Supporting information

Data S1

Tables S1–S2

Figures S1–S3