Clinical Risk Scores to Minimize Low Yield CAD Testing: PROMISE or Reality?

Reduction in low-yield testing has been the focus of many efforts in the cardiovascular community including appropriate use criteria and education campaigns (1–4). Unfortunately, translation to clinical practice of these efforts has been challenging and incomplete. In this context, the application of clinical risk scores for initial risk stratification and better targeting of advanced imaging methods is of tremendous interest.

Use of risk prediction rules to identify patients suitable for more aggressive investigation has a long history dating at least as far back as to 1979 with Diamond and Forrester’s seminal risk stratification paper using decision tree methods to classify patients based on using age, sex and chest pain characteristics (5). Since that time, a large number of other cardiovascular risk prediction schemes for both symptomatic and asymptomatic individuals using regression and more recently statistical learning methods have been published. As of 2013, 363 prediction models and 473 external validations had been published (6). Despite concerns over poor calibration in light of changing demographics and risk factor profiles, the Diamond and Forrester model remains one of the most widely used models, likely due to its simplicity (5, 6). Ultimately, the declining prevalence of CAD in testing cohorts (7) results in an inevitable decline in positive predictive value of both clinical decision rules and advanced imaging(8).

Recently, the United Kingdom’s National Institute for Health and Clinical Excellence (NICE) recommended coronary computed tomography angiography (CCTA) for all patients with chest pain who have typical or atypical angina, and those with non-anginal chest pain but concerning electrocardiographic changes based mainly on analyses of the Scottish COmputed Tomography of the HEART (SCOT-HEART) trial and the Prospective Multicenter Imaging Study for Evaluation of Chest Pain (PROMISE) trial (9, 10). Although the excellent negative predictive value of CCTA seen in several prospective trials may be helpful in some clinical scenarios, it is unclear whether it is cost saving (9, 10). Furthermore, full implementation of such a change is extremely difficult and would result in an increase in the annual volume of CCTA in the United Kingdom from 42,340 to 350,000 (or from 66 to 545 per 100 000 population, per year) (11).

Consequently, regardless of the preferred CAD imaging test, optimal patient selection remains critical and great potential utility exists for improved clinical tools. To that end, in this issue of Circulation: Cardiovascular Imaging, Jang et al performed a post-hoc analysis on patients who underwent CCTA in the PROMISE trial (12). Recognizing that among stable patients, it is critical to identify coronary anatomy which could meet indications for coronary artery bypass grafting (13), the investigators set to derive a clinical prediction model of high-risk CAD which was defined as 50% stenosis for 1) left main 2) three-vessel disease or 3) two-vessel disease including the proximal left anterior descending coronary artery (pLAD) (HR-CAD50). A second model was derived to identify 1) >50% left main stenosis 2) >70% stenosis in three vessels or 3) >70% stenosis in two-vessel including the pLAD (HR-CAD70) (12). Out of 4,589 patients, prevalence of HR-CAD50 was 6.6% and HR-CAD70 was 2.4%. This itself is a very important finding. Nearly 19 out of 20 patients who underwent CCTA did not meet the more inclusive definition of disease for which revascularization might alter prognosis and 39 out of 40 did not meet the more stringent criteria. In many ways, the investigators set out to identify a needle in haystack.

Their final risk model for HR-CAD50 included age, gender, diabetes mellitus, family history, type of symptoms (classified as typical, atypical, non-cardiac), estimated glomerular filtration rate, systolic blood pressure, and smoking. The final risk model for HR-CAD70 was similar though smoking which was replaced by sedentary lifestyle. Ultimately all of these factors are generally already being considered by clinicians when prioritizing patients for additional testing in clinical practice and may have less predictive power in this cohort, all of whom were planned for additional testing, than in broader clinical cohorts. Further, these regression equations formalize the process and have the potential to reduce between clinicians’ variability. The final models were well calibrated, meaning the range of predicted risk was broadly in line with observed risk. The models’ risk discrimination (ability to separate high and low risk individuals) compared favorably to previously published models (Table 1). It is important to note that although the investigators used bias correction methods to help offset the bias towards higher performance seen whenever a risk score is fit and evaluated within the same cohort, the new model is likely to perform somewhat less well in other cohorts. Furthermore, the net differences between these new and older risk models such the CONFIRM high-risk model (14) and the Pooled Cohort Equations are small enough that these equations may not be superior in clinical practice. Importantly, the CONFIRM model does not include laboratory variables and may be more easily generalizable as a consequence.

Table 1.

A summary of the performance of the different models presented in the original paper by Jang et al(16).

Model	C-statistic (95% CI)	Bias-corrected C-statistic (95% CI)
HR-CAD50	0.75 (0.72, 0.78)	0.73 (0.71, 0.76)
HR-CAD70	0.76 (0.71, 0.81)	0.73 (0.68, 0.77)
Comparison with other models
Published Model	Published model C-statistic	New model for comparison	Chi square	p-value
Pooled Cohort Equation	0.70	HR-CAD50	18.18	<0.0001
	0.71	HR-CAD70	5.97	0.0145
Modified Diamond-Forrester	0.68	HR-CAD50	26.90	<0.0001
	0.71	HR-CAD70	9.47	0.0021
CONFIRM High-Risk	0.73	HR-CAD70	2.15	0.1422

HR-CAD50: High-risk coronary artery disease 50% model; HR-CAD70: High-risk coronary artery disease 70% model (refer to text for definitions); CI: confidence interval.

So, is there little hope for application of risk equations to help better target advanced imaging methods? We computed sensitivity, specificity, positive and negative predictive value for cut points at each decile of the two risk scores from supplemental tables 2 and 4 (Table 2). As with any model, there were trade-offs between sensitivity and specificity at different cut points. Given a strong desire not to miss high-risk CAD anatomy, we plotted the negative predictive value as compared to the fraction of patients not referred to CCTA based on various thresholds of HR-CAD50 and HR-CAD70 (Figure 1). It is possible to avoid testing in as much as 40% of individuals with the lowest risk based on these risk models and still have a net negative predictive value for HR-CAD50 of 98.1% and for HR-CAD70 of 99.2%.The concept of reducing imaging by 40% is certainly attractive. However, the sensitivity of such a strategy would only by 88.4% for HR-CAD50 and 88.2% for HR-CAD70, implying nearly 1 in 8 patients with high-risk CAD would not undergo testing under such a scheme and would presumably be missed. It is not certain whether clinicians would accept such a compromise.

Table 2.

Sensitivity, specificity, positive predictive value, negative predictive value, fraction of CCTA avoided and fraction of disease missed of Model HR-CAD50 and Model HR-CAD70 according to different deciles presented originally in supplemental Tables 2 and 4, respectively (16).

Decile	Number of subjects	Observed events	Sensitivity	Specificity	Positive predictive value	Negative predictive value	Fraction of CCTA avoided (%)	Fraction of HR diseases missed (%)
Model HR-CAD50
0	443	3	1.0000	0.0000	0.0660	-	0	0.0
1	444	2	0.9898	0.1061	0.0726	0.9932	10	1.0
2	444	14	0.9829	0.2127	0.0811	0.9944	20	1.7
3	444	15	0.9352	0.3164	0.0882	0.9857	30	6.5
4	444	19	0.8840	0.4199	0.0972	0.9808	40	11.6
5	444	26	0.8191	0.5224	0.1081	0.9761	50	18.1
6	444	28	0.7304	0.6233	0.1205	0.9703	60	27.0
7	444	41	0.6348	0.7236	0.1396	0.9656	70	36.5
8	444	60	0.4949	0.8208	0.1633	0.9583	80	50.5
9	444	85	0.2901	0.9134	0.1914	0.9479	90	71.0
10	0	0	0.0000	1.0000	-	0.9340	100	100.0
Model HR-CAD70
0	444	3	1.0000	0.0000	0.0248	-	0	0.0
1	444	5	0.9727	0.1018	0.0268	0.9932	10	2.7
2	445	3	0.9273	0.2031	0.0287	0.9910	20	7.3
3	444	2	0.9000	0.3051	0.0318	0.9917	30	10.0
4	444	5	0.8818	0.4071	0.0364	0.9927	40	11.8
5	445	7	0.8364	0.5084	0.0414	0.9919	50	16.4
6	444	7	0.7727	0.6095	0.0478	0.9906	60	22.7
7	445	15	0.7091	0.7104	0.0585	0.9897	70	29.1
8	444	17	0.5727	0.8096	0.0709	0.9868	80	42.7
9	444	46	0.4182	0.9081	0.1036	0.9840	90	58.2
10	0	0	0.0000	1.0000	-	0.9752	100	100

HR: High risk; CCTA: Coronary computed tomography angiography; HR-CAD50: High-risk coronary artery disease 50% model; HR-CAD70: High-risk coronary artery disease 70% model (refer to text for definitions)

Figure 1.

A plot of the negative predictive value as compared to the fraction of patients not referred for CCTA based on various thresholds of HR-CAD50 and HR-CAD70.

CCTA: Coronary computed tomography angiography; HR-CAD50: High-risk coronary artery disease 50% model; HR-CAD70: High-risk coronary artery disease 70% model (refer to text for definitions).

Stepping back, the exercise of risk prediction and coronary imaging to identify high risk anatomy has two main goals. First, in educating patients and caregivers as to prognosis for adverse CAD outcomes. Second, to separate patients whose prognosis might be improved from revascularization from those in whom a trial of medical therapy prior to further testing would be more reasonable. Patients identified as higher risk by both models experienced more adverse outcomes, including more death and myocardial infarction. Although these events were adjudicated by a blinded committee, local clinicians decided on the process of care in non-randomized fashion, so it remains unclear the extent to which additional imaging may or may not have impacted outcomes (10). Another limitation pertains to the use of CCTA as a surrogate for obstructive coronary artery disease on invasive coronary angiography. In a recent report from the same study, 16% discordance between CCTA interpretations performed at recruiting sites and the core laboratory was noted with 41% fewer patients being reported to have significant CAD by the core laboratory (14% in core laboratory vs 23% in site reported, p<0.001) (15). Importantly, the present study used site reads and results could be different with core lab reads, given the meaningfully lower prevalence of disease in the core lab interpretation.

In summary, risk prediction in CAD remains challenging. Although the current study is a step in the right direction and could form the basis of strategies to reduce low-yield testing, as of yet no clinical cut point has been validated and widely accepted as a threshold beyond which clinicians should feel comfortable in not offering further testing. The search for better performing prediction models should continue, although improvement in performance is likely to be modest without incorporation of additional powerful risk markers such as coronary artery calcium score. Equally critical, however, is to validate the safety of risk based algorithms for patient selection in prospective studies.