Coronary CT in Australia has high positive predictive value unaffected by site volume: An analysis of 510 positive CTCA scans with invasive angiographic correlation

Abstract

Background

It was hypothesized that the accuracy of coronary CT angiography would be affected by case volume of the referring sites.

Methods

The positive predictive value (PPV) of CTCA performed at a tertiary hospital specialising in cardiothoracic medicine and services with lower case-volumes were calculated. The tertiary hospital used as the high case-volume reference centre was The Prince Charles Hospital, which performed >1500 CTCA scans per annum over the study period. The low case-volume services used in the study were suburban radiology services, each with <500 cases per year. The PPV of positive CTCA at the reference site was compared to the pooled PPV of all other sites as a combined cohort, using invasive angiography as the reference standard. 512 scans were included, n = 199 subjects in the reference centre cohort, and n = 311 subjects in the pooled community radiology practice cohort.

Results

The positive predictive value (PPV) of the high case-volume group (n = 199) was 0.7538. The PPV of the pooled low case-volume services (n = 589) was 0.7331, p = 0.604, with no statistically significant difference in positive predictive values.

Conclusions

There was no significant difference in PPV between the two groups. This suggests that high-volume and lower-volume sites both have high PPV in Australia, above the published pooled PPV of four large prospective diagnostic accuracy studies (Miller et al., 2008; Budoff et al., 2008; Meijboom et al., 2008; Achenbach, 2007).

1. Introduction

Coronary artery disease (CAD) is a significant contributor to global mortality and morbidity. In the clinical assessment of acute chest pain, a clinical dilemma is presented by patients with negative serum biomarkers and electrocardiography (ECG), but who go on to have a diagnosis of CAD confirmed on invasive coronary angiography (ICA), the relevant gold-standard investigation [1]. Due to the invasiveness and risk of complications inherent to ICA, there is a clinical requirement for an expeditious, non-invasive, and accurate coronary imaging modality [1].

Due in part to recent technological advances, CTCA has emerged as an accurate modality for the evaluation of clinically suspected CAD. Over this period, a body of literature has formed concerning the accuracy of the aforesaid modality, and a number of factors affecting its accuracy have been identified [[1], [2], [3], [4], [5]]. The literature demonstrates a trend toward improved accuracy with advances in technology and radiographic protocols, including application in acute and chronic chest pain, and the Emergency department setting [6, 7].

Although numerous studies investigating the accuracy of CTCA have been published in recent years [3], there is a paucity of data concerning the effect of case-volume on the accuracy of individual CTCA services. A large number of other factors have been noted to influence the accuracy of CTCA. The patient-related factors which have been described in studies to date include calcium score, heart rate, obesity, and previous coronary artery bypass grafts (CABGs). In addition to these, various technical and protocol-related factors have been delineated, including electrocardiogram-gating, number of CT slices, and the use of axial vs. helical CT apparatuses.

Despite an increasing interest in CTCA, the literature to date does not address the extent of inter-user variability, inter-service variability, or the experience required by a technician to accurately interpret CTCA. In an editorial by Achenbach [5], the concept that CTCA accuracy could be influenced by the clinical setting in which it is performed was raised. However, Achenbach's hypothesis at this time was that differences in patient populations would necessarily have differing pre-test probabilities, thus affecting the investigation's accuracy. As such, the question of whether all CTCA services are equally accurate, and whether case-volume is an independent predictor of accuracy, remains unanswered. In a study by Chow et al. the accuracies of CTCA performed at different centres came to several conclusions about the factors influencing the accuracy of the investigation [6]. However, while alluding to the likelihood of case-volume being a potentially important factor, this was not directly investigated in the study design [6].

The influence of the investigating service on the accuracy of CTCA may derive from a number of confounding variables. For instance, if there are significant differences between the populations referred to different services, the type of CT apparatus used at the site, or the radiographic protocols employed at each service. Indeed, these variables may account for some of the variability observed in the results of early, single-centre studies. However, as has been discussed, there are as yet no data on whether a service's case-volume directly influences the accuracy of CTCA performed by the service. CTCA is an operator-dependent modality in many ways; the large, multi-centre studies which form the basis for much of the literature on its accuracy [[2], [3], [4]] each used a small number of readers experienced in the interpretation of CTCA. As none of the studies to date have investigated interpreting clinician experience as an independent determinant of accuracy, its importance remains unquantified. Furthermore, the external validity of much of the literature is limited by the use of a consensus interpretation of coronary stenosis from groups of interpreters, which represents a departure from clinical practice.

Despite the literature's silence on the issue, the notion of case-volume affecting the effective implementation of a CTCA service is raised in the guidelines for the use of diagnostic CTCA in acute coronary syndrome [1]. The Society of Coronary Computed Tomography (SCCT) guidelines state that CTCA should only be performed at services expecting case-volumes of at least 300 scans per year [1]. However, this figure does not appear to be based upon any empirical data, instead representing, seemingly, the expert opinions of the members of the SCCT. The SCCT also recommends a quality assurance program to ensure that CTCA accuracy remains above 75% for each site, and that all interpreting clinicians have a minimum level of experience [1]. Furthermore, the literature does make mention of the potential unreliability of single-centre trials, citing service-specific biases as explanations for differing accuracies between centres [[2], [3], [4]], suggesting that different centres or sites have different accuracies. Therefore, it appears that there has hitherto been an assumption made that case-volume, and other factors which differ between sites, are correlated with CTCA accuracy; however there have been no formal studies investigating this assumption.

It is clear from the literature that the accuracy of CTCA in the assessment of coronary artery stenosis has been well established, with large, multi-centre trials confirming this in 2008 [[2], [3], [4]]. In the interim, technological advances and improvements in radiographic protocols have improved the safety, and to some degree the accuracy of this modality. Despite this, the question of whether case-volume is an independent predictor of CTCA accuracy remains unanswered, with no literature published to date on the effect of case-volume on the accuracy of CTCA at an individual service. This is an area of continued relevance, as differing accuracies between services could have a significant impact upon clinical decision making.

The purpose of the present study is to investigate a possible relationship between case-volume of CTCA services and their respective accuracies. It is hypothesized that services with higher case-volumes will have a higher positive predictive value (PPV) of CTCA as compared with the gold standard investigation.

2. Methods

The positive predictive value (PPV) of CTCA performed at a tertiary hospital specialising in cardiothoracic medicine, and those performed at services with lower case volumes were calculated. The tertiary hospital used as the high case-volume reference centre was The Prince Charles Hospital (TPCH), which performed >1500 CTCA scans per annum during the study period. The low case-volume services used in the study were suburban radiology services, each with <500 cases per year.

Inclusion criteria were: patients clinically referred for invasive coronary angiography for the criterion of “positive CTCA” (>50% stenosis), as recorded in the Agfa Heartlab database at TPCH. The database included data points on referral diagnosis (positive CTCA), patient age and sex, and the outcome of the angiogram routinely recorded for each subject. As per the inclusion criteria, only positive CTCA scans that had been referred to TPCH for invasive angiography were included in the study. All CTCA were performed on >64-slice CT systems, and requested by a specialist or consultant physician as per Medical Benefits Scheme item 57360.

The PPV of positive CTCA at the reference site was compared to the pooled PPV of all other sites as a combined cohort, using invasive angiography as the reference standard. Invasive coronary angiography (ICA) was used as the gold standard investigation for coronary disease, and all angiograms were reported upon by an interventional cardiologist. Interventionists' reports were interpreted by investigators and classified as positive or negative based on the presence of significant coronary artery stenosis. Positivity was declared on a per-patient, not per-lesion, basis, with disease in any coronary vessel being classified as a positive result in both CTCA and ICA.

Data analysis was performed using Stata Statistical Software Package (StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP), with PPV's of the study arms compared using a Pearson Chi Square test. ICA's with indeterminate results, or without diagnostic information recorded in the database, were excluded from the analysis.

3. Results

A total of 512 scans were included as per the study criteria. The reference centre group contained 199 scans, with 133 of these being on male subjects and a mean age of 62.48. The pooled community group contained 311 scans, with 217 being male and a mean age of 61.43. The sex and age distributions of the two study groups were not significantly different, p = 0.485 and p = 0.597 respectively. A total of 2 scans in the pooled community group were not included in the analysis; one ICA was aborted due to complications, and another did not have a diagnostic outcome reported. All of the scans from the reference centre group were included in the analysis [Table 1].

Table 1.

Demographics of the two study arms. P-values calculated using Pearson Chi Square Test

		Males	Females	Mean Age
External	n=	217	94	61.43
	%	70%	30%
Internal	n=	133	66	62.48
	%	67%	33%

The PPV of the reference centre group was 0.7538, while that of the low case-volume group was 0.7331. There was no statistically significant difference between the PPV of the two groups, with p = 0.604. The overall PPV of both arms was 0.7412 [Table 2 and Fig. 1].

Table 2.

Results of data analysis for primary endpoints for pooled and reference centre cohorts

		False Positive	True Positive	n =	PPV (%)
External	n =	83	228	311	73.31
	%	26.69	73.31
Internal	n =	49	150	199	75.38
	%	24.63	75.38
Total	n =	132	378	510	74.12
	%	25.88	74.12

Fig. 1.

Positive predictive values of pooled community cohort and reference centre CTCA scans. p = 0.604 (not significant).

4. Discussion

The outcome of the data analysis demonstrates that there was no significant difference in PPV between CTCA performed at high and low case-volume services. This suggests that the technological and methodological aspects of CTCA have become sufficiently standardised across services in Australia to guarantee a reliable level of accuracy. This may be in part due to technical aspects of CTCA being linked to remuneration, as per Medicare item number 57360.

It was also noted that the PPV of CTCA observed in the study was relatively high compared with that previously observed in large multi-centre trials [[3], [4], [5]]. This suggests that CTCA performed in Australia may have a greater PPV than was previously suggested by the literature.

The study outcome is significant for a number of reasons. Firstly, the question of whether CTCA performed at suburban radiology practices can be relied upon appears to have been answered; this may have resource allocation applications in the area of public health policy, as well as being of clinical utility. Moreover, the results suggest that the technology and methodology used in modern CTCA scans is reliable and consistent between services.

4.1. Study limitations

The present study was a retrospective analysis of routinely collected clinical data, and as such contained a number of methodological limitations. Inherent to the study design was a reliance upon the accuracy of recording in the Agfa Heart Database, and the reliability of an in-house data mining tool to extract the data. Furthermore, the database did not record variables such as BMI, heart rate, and calcium score; if these variables were not comparable between groups this may have skewed the data. Despite this, it was noted that the two study groups did not significantly differ with regard to the variables of patient age and sex.

An important methodological consideration is that no negative CTCA's were included in the study. The calculation of sensitivity of CTCA in the two study arms was therefore impossible with the current methodology. As CTCA typically has a very high negative predictive value [[3], [4], [5]], very few patients with a negative CTCA would be referred for ICA; moreover the clinical database did not record negative scans performed at the community CTCA services included in the study.

An important limitation of the study was that costing data are not available for a formal cost-utility evaluation. However, the high positive predictive value in both cohorts suggests that CTCA scans are being acquired and reported with good quality and high accuracy compared to downstream cardiac catheterisation.

This study was the first of its kind, insofar as it investigated service case volume and CTCA accuracy as its primary study variables in the Australian healthcare context. Due to the limitations discussed, research is necessary to further characterise the relationship between these two factors. Prospective studies measuring both positive and negative predictive values, with multiple reference centres and larger number of scans are necessary to confirm the results of this analysis. Future studies may better stratify individual CTCA services by case volume, to see if there is an incremental increase in accuracy which was missed in the present study. Furthermore, the recording of variables known to affect CTCA accuracy, such as BMI and heart rate during the scan, would reduce the effect of confounding on the study results. There is room in the literature for investigating whether referral for CTCA, and outcome of CTCA, significantly alters patient management and outcomes.

5. Conclusions

This is the first Australian study in the literature to directly compare case volume and CTCA accuracy. In light of the results, it is reasonable to conclude that the assumptions made by CTCA guidelines regarding case volume of services professing to provide accurate CTCA interpretation are reasonable [1]. The PPV's of CTCA in the present study were above that of a previous large, multi-centre trial [4], suggesting that CTCA use in Australia is yielding studies of good accuracy compared with downstream cardiac catheterisation.