Abstract
Background
Studies conducted decades ago described substantial disagreement and errors in physicians’ angiographic interpretation of coronary stenosis severity. Despite the potential implications of such findings, no large-scale efforts to measure or improve clinical interpretation were subsequently made.
Methods & Results
We compared clinical interpretation of stenosis severity in coronary lesions with an independent assessment using quantitative coronary angiography (QCA) in 175 randomly selected patients undergoing elective percutaneous coronary intervention (PCI) at 7 U.S. hospitals in 2011. To assess agreement, we calculated mean difference in percent diameter stenosis between clinical interpretation and QCA and a Cohen’s weighted kappa statistic. Of 216 treated lesions, median percent diameter stenosis was 80.0% (Q1 and Q3, 80.0 and 90.0%) with 213 (98.6%) assessed as ≥70%. Mean difference in percent diameter stenosis between clinical interpretation and QCA was +8.2 ± 8.4%, reflecting an average higher percent diameter stenosis by clinical interpretation (P<0.001). A weighted kappa of 0.27 (95% CI, 0.18 to 0.36) was found between the 2 measurements. Of 213 lesions considered ≥70% by clinical interpretation, 56 (26.3%) were <70% by QCA though none was <50%. Differences between the 2 measurements were largest for intermediate lesions by QCA (50 to <70%) with variation existing across sites.
Conclusions
Physicians tended to assess coronary lesions treated with PCI as more severe than measurements by QCA. Almost all treated lesions were ≥70% by clinical interpretation, while approximately a quarter were <70% by QCA. These findings suggest opportunities to improve clinical interpretation of coronary angiography.
Keywords: Health policy and outcomes research, Quality improvement, Coronary angiography, Percutaneous coronary intervention, Quantitative coronary angiography
Millions of coronary angiograms are performed annually to obtain information that, when combined with clinical data, guides treatment decisions for patients with coronary artery disease.1 These tests are performed, in large part, to determine the presence and severity of coronary stenoses, which in turn plays a key role in selection of patients for revascularization.2 In clinical practice, stenosis severity is typically determined during or shortly after the procedure, and most commonly relies on visual estimation by physicians. This approach, however, has well-known limitations.3,4 Older studies, conducted a decade or more ago, described interobserver and intraobserver variation in visual estimations of stenosis severity and inaccuracies when compared with computer-assisted techniques, expert panel review, autopsy results, or simulations.5–13 Despite the potential implications of these findings – particularly regarding the consistency and quality of treatment decisions for revascularization13 – no widespread efforts have been undertaken to improve clinical interpretations of coronary angiograms nor has there been further study of the issue.
We lack contemporary information about the quality of clinical interpretations of coronary angiograms. Since studies were last performed in the early 1990s, significant advances in digital technology have transformed angiographic imaging.14 Whether this has led to concomitant improvements in clinical interpretations is largely uncertain, however. Understanding this issue is relevant given that stenosis severity, as assessed by physicians, remains a pivotal variable for framing treatment options – even in the current era where pre- or intra-procedural functional testing of a stenosis is widely available. Moreover, the percent diameter stenosis continues to be used as an entry criterion for clinical trials of revascularization and its reliable measurement is a key assumption of current Appropriateness Use Criteria for revascularization.15 Errors in the clinical interpretation of coronary angiograms therefore have important consequences for treatment decisions, potentially leading to both overuse and underuse of revascularization.
To explore the quality of clinical interpretation in the modern era of interventional cardiology, we designed the Assessing Angiography (A2) project. We randomly selected coronary angiograms from patients undergoing percutaneous coronary intervention (PCI) at seven large US hospitals. The clinical interpretation of stenosis severity among lesions with PCI by physicians was compared with measurements made by an independent, blinded review using state-of-the-art quantitative coronary angiography (QCA) – a computer-assisted technique for measuring stenosis severity employed for decades for quality assurance within clinical trials.16 We purposely selected QCA as a benchmark tool given its high reproducibility and potential freedom from observer influence and bias.17
Methods
Data Sources and Clinical Abstraction
We enrolled seven PCI hospitals participating in the CathPCI Registry® of the National Cardiovascular Data Registry® (NCDR®), sponsored by the American College of Cardiology (ACC) and the Society for Cardiovascular Angiograpy and Intervention (SCAI). We selected sites for this study to ensure diversity in regional location. We only included sites that had digital storage capability and could transfer coronary angiograms digitally for further assessment and interpretation. The Aetna Foundation provided funding for the study. The investigators were responsible for all data collection and analyses, as well as the decision to publish the findings. The study was initially designed to produce information that could be used for a future quality improvement initiative by initiating feedback to participating hospitals on the correlation of their clinical interpretation of coronary angiograms with QCA from a core laboratory. When a decision to publish these findings was made, we obtained approval from the Institutional Review Board of the Saint Luke’s Hospital (Kansas City, Missouri) where analyses were conducted as an exempted study, since all data were de-identified at this point.
From each of the hospitals, we obtained coronary angiograms on patients who underwent PCI during calendar year 2011. Data managers at the NCDR generated a random list of patients at each hospital after excluding patients undergoing PCI for urgent or emergency indications. For each patient, we obtained the clinical report of the coronary angiogram and catheterization laboratory log, after they were stripped of all unique patient identifiers, as well as a de-identified digital copy of the coronary angiogram. Data abstracted directly from the clinical records (not the CathPCI Registry) included information on: catheter size, lesion location, maximal percent diameter stenosis before and after PCI, and use of fractional flow reserve (FFR). In cases where multiple lesions were described and treated, we abstracted data pertaining to each lesion. We obtained supplemental information on the clinical characteristics and presentation of each patient from each site as part of the data that they routinely collected and provided to the CathPCI Registry.
Quantitative Coronary Angiography
The de-identified clinical records and angiograms were managed by ImageCor, LLC (Bradenton, FL) and analyzed by the Yale Cardiovascular Research Group (New Haven, CT), an experienced core laboratory. The trained analysts at the core laboratory were blinded to the clinical records and worked independently of the sites and other investigators. The analysts first subjectively evaluated the overall technical quality of the images and stenosis visualization in multiple views using standardized criteria based on: availability of imaging a calibration catheter, the presence of excessive foreshortening, vessel or side-branch overlap, contrast streaming or streaming artifact, limited ostial bifurcation imaging (excessive overlap or inadequate separation of vessels), and over- or under-exposure.
The core laboratory then used the Cardiovascular Measurement System (QAngio XA 7.2, MEDIS, Leiden, The Netherlands), a PC-based system, for off-line quantitative angiographic analysis. Specific features of the CMS include 2-point user-defined pathline (centerline) identification, arterial contour detection using a minimal cost matrix algorithm, and an “interpolated” reference vessel diameter. The interpolated reference vessel diameter is broadly accepted and a well validated method of measuring reference diameter by QCA; it is obtained at the site of minimal lumen diameter and derived by an iterative linear regression technique that is operator independent and accounts for vessel tapering.18,19 The minimal lesion diameter was used to calculate percent diameter stenosis relative to the interpolated reference vessel diameter of the lesion of interest. The core laboratory assessed the reference and minimal lesion diameters from the single-best, available projection with least foreshortening that best demonstrated the stenosis as selected by the analyst. At the Yale Cardiovascular Research Group core laboratory, repeated QCA analyses of reference and minimal lumen diameters have demonstrated good reproducibility within a range of 3.7% to 5.8% for estimates of percent diameter stenosis (Personal Communication: Alexandra J. Lansky, MD).
Data Analysis
We used univariate statistics to describe the study population. We then used a lesion-specific approach to compare the percent diameter stenosis by the two methods of assessment and this was expressed as the difference between the clinical interpretation and QCA using Student t-tests. Concordance was further analyzed using 2 quantitative methods. First, we evaluated the correlation between clinical interpretation and QCA as continuous variables using Pearson’s correlation coefficient and graphically presented these data, including a simple linear regression analysis. Second, we categorized percent diameter stenosis from two methods according to the following cutoffs: <50%, 50 to <70%, 70 to <90%, 90 to <100%, and 100% (but explored additional cutoffs in sensitivity analyses). We then assessed concordance between clinical interpretation and QCA using Cohen’s weighted kappa statistic,20 a statistical measure of interrater agreement for categorical items. The kappa statistic is generally considered a more robust measure than a simple percent agreement calculation, since it considers agreement occurring by chance. Because the kappa statistic takes the observed categories' frequencies as givens, it may underestimate agreement for a category that is also commonly used. Given this concern, the kappa statistic is considered an overly conservative measure of agreement.21
We also performed subgroup analyses. We first repeated our analyses after excluding patients with lesions thought to be associated with a recent non-ST elevation myocardial infarction or within coronary artery bypass grafts, since the thresholds for revascularization based on percent diameter stenosis may be different in these circumstances. We also examined variation in angiographic interpretation across differences in stenosis severity, lesion location in the coronary vasculature, lesion reference vessel diameter, lesion length, quality of the coronary angiogram determined by the core laboratory, presence of a stress test or FFR, and individual hospital sites. The sample size for this study was difficult to estimate, given the study’s intent to generate basic descriptive information about agreement between clinical interpretations and QCA. We proposed to collect 25 studies from each of eight hospitals, and seven hospitals ultimately participated in the quality improvement initiative. All analyses were conducted with SAS (Version 9.3) and R (Version 2.15.0) software.
Results
Study Population
The study sample included 175 patients who underwent PCI of 228 lesions at the 7 sites. A list of baseline characteristics is displayed in Table 1. The mean age of patients was 66.7 ± 10.7 years with 59 (33.7%) women, and 20 (11.4%) non-white patients. A history of prior PCI was present in 73 (41.7%) patients and prior coronary artery bypass grafting in 42 (24.0%). At the time of PCI, 26 (14.8%) patients were asymptomatic or had symptoms unlikely to be ischemic; 48 (27.4%) patients had stable angina; 87 (49.7%) had unstable angina; and 14 (8.0%) had a NSTEMI. A stress test was performed before PCI in 100 (57.1%) patients.
Table 1.
Baseline characteristics of 175 patients undergoing PCI.
| Total |
|
|---|---|
| n = 175 | |
| Age | |
| Mean ± standard deviation | 66.7 ± 10.7 |
| Median (interquartile range) | 66.0 (58.0, 75.0) |
| Age category | |
| 34 to <50 | 7 ( 4.0%) |
| 50 to <60 | 46 (26.3%) |
| 60 to <70 | 47 (26.9%) |
| 70 to <80 | 51 (29.1%) |
| 80 to 92 | 24 (13.7%) |
| Women | 59 (33.7%) |
| Race | |
| White/Caucasian | 155 (88.6%) |
| Black/African-American | 17 (9.7%) |
| Other | 3 (1.7%) |
| Hispanic or Latino ethnicity | 3 (1.7%) |
| Current/recent smoker (<1y) | 37 (21.1%) |
| Hypertension | 154 (88.0%) |
| Dyslipidemia | 151 (86.3%) |
| Family history of premature coronary artery disease | 54(30.9%) |
| Prior myocardial infarction | 55(31.4%) |
| Prior heart failure | 22 (12.6%) |
| Prior PCI | 73 (41.7%) |
| Prior coronary artery bypass grafting | 42 (24.0%) |
| Body mass index | |
| Mean ± standard deviation | 30.3 ± 6.7 |
| Cerebrovascular disease | 18 (10.3%) |
| Peripheral arterial disease | 23 (13.1%) |
| Diabetes | 53 (30.3%) |
| Clinical presentation | |
| No symptoms, no angina | 20 (11.4%) |
| Symptoms unlikely to be ischemic | 6 (3.4%) |
| Stable angina | 48 (27.4%) |
| Unstable angina | 87 (49.7%) |
| Non-ST elevation myocardial infarction | 14 (8.0%) |
| Angina classification in past 2 wks | |
| No symptoms | 25 (14.3%) |
| CCS I | 9 (5.1%) |
| CCS II | 44 (25.1%) |
| CCS III | 64 (36.6%) |
| CCS IV | 33 (18.9%) |
| Stress or imaging studies performed | 100 (57.1%) |
| Staged PCI | 15 ( 8.6%) |
| Angiogram image quality | |
| Excellent | 25 (14.3%) |
| Good | 121 (69.1%) |
| Satisfactory | 29 (16.6%) |
| SYNTAX Score (for lesion) | |
| Mean ± SD | 9.2 ± 6.0 |
| Median (Q1-Q3) | 8.0 (5.0, 12.0) |
PCI = Percutaneous Coronary Intervention, CCS = Canadian Classification System
Table 2 lists characteristics associated with the 228 lesions that were treated by PCI in the study population. Most treated lesions were in the left anterior descending coronary artery followed by the right coronary artery and left circumflex coronary artery. There were 16 FFR assessments performed, of which 13 were abnormal with values less than or equal to 0.80. In 216 lesions, a clinical interpretation with percent diameter stenosis was available while the remaining 12 lesions were reported in qualitative terms (e.g., “severe” or “critical”) (Table 2). These 12 were excluded from analyses evaluating concordance.
Table 2.
Characteristics of 228 lesions undergoing PCI.
| Total |
|
|---|---|
| n = 228 | |
| Vessel territory | |
| Left circumflex | 65 (28.5%) |
| Left anterior descending | 86 (37.7%) |
| Left main | 2 (0.9%) |
| Ramus intermedius | 2 (0.9%) |
| Right | 73 (32.0%) |
| Pre-PCI stenosis (Clinical Interpretation) | |
| No quantitative report | 12 (5.3%) |
| 50 to <70 | 3 (1.3%) |
| 70 to <90 | 113 (49.6%) |
| 90 to <100 | 87 (38.2%) |
| 100 | 13 (5.7%) |
| Pre-PCI stenosis (QCA) | |
| 50 to <70 | 61 (26.8%) |
| 70 to <90 | 141 (61.8%) |
| 90 to <100 | 11 (4.8%) |
| 100 | 15 (6.6%) |
| FFR Performed | 16 (7.0%) |
| FFR Abnormal (≤0.80) | 13 (81.3%) |
PCI = Percutaneous Coronary Intervention, QCA = Quantitative Coronary Angiography, FFR = Fractional Flow Reserve
Of the 216 lesions treated with PCI where stenosis severity by clinical interpretation was reported, median percent diameter stenosis was 80.0% (first and third quartiles, 80 and 90%) and mean percent diameter stenosis was 84.2% (± 10.1). The most commonly reported percent diameter stenoses were in the range of 70 to <90% followed by 90 to <100%. In only 3 (1.4%) lesions was the percent diameter stenosis reported to be <70% by clinical interpretation; a stress test was documented (although information on specific results is unavailable) and/or an FFR was performed in these 3 patients. No lesion was reported to be <50%.
Comparison of Clinical Interpretation and QCA
QCA was performed in all 228 lesions treated with PCI with a median percent diameter stenosis of 74.6% (first and third quartiles, 69.5 and 82.5%) and mean percent diameter stenosis was 76.1% (± 10.9). Similar to clinical interpretation, the most commonly calculated percent diameter stenosis was in the range of 70 to <90% (Table 2). The next most frequent category of stenosis severity by QCA was 50 to <70% with 61 (26.8%) lesions in this category; of these, 35 (57.4%) had documentation of stress testing or FFR before PCI, and rates of stress testing did not vary across categories of stenosis severity (see Supplementary Material: Table A). No lesion was calculated to be <50% by QCA. There was no significant difference by QCA between the 12 lesions where stenosis severity by clinical interpretation was reported in qualitative terms (e.g., “severe” or “critical”) and others (77.4% versus 76.0%; p=0.66).
The mean difference in percent diameter stenosis between the clinical interpretation and QCA was +8.2% ± 8.4% (n=216), reflecting an average higher percent diameter stenosis by the clinical interpretation (P<0.001). The distribution of this difference across the lesions is shown in Figure 1. Of the 213 lesions considered 70% or greater by clinical assessment, 56 (26.3%) were measured at less than 70% by QCA and 10 (4.7%) were less than 60%.
Figure 1.
Distribution of mean difference in percent diameter stenosis between clinical interpretation and Quantitative Coronary Angiography (QCA).
A scatter plot of the clinical interpretation and quantitative assessment by QCA is displayed in Figure 2, demonstrating a Pearson correlation coefficient of 0.68. Clinical interpretations had a discrete distribution with most of the reported values being divisible by 10% (e.g., 70%, 80%, etc.) while QCA stenoses were continuously distributed. Table 3 lists a comparison between the 2 methods after categorizing each assessment according to clinically meaningful cutoffs and showing agreement. In this analysis, a weighted kappa of 0.27 (95% CI, 0.18 to 0.36) was found between the 2 measurements.
Figure 2.
Comparison of percent diameter stenosis between clinical interpretation and Quantitative Coronary Angiography (QCA). Pearson’s correlation coefficient (r).
Table 3.
Comparison between quantitative assessments across categories of percent diameter stenosis by Clinical Interpretation and QCA. Each cell represents the number of lesions in agreement in that category between the 2 methods.
| Pre-PCI stenosis (%) by QCA | |||||
|---|---|---|---|---|---|
| Pre-PCI stenosis (%) by Clinical Interpretation |
50 to <70 | 70 to <90 | 90 to <100 | 100 | Total |
| 50 to <70 | 2 (0.9) | 1 (0.5) | 0 (0) | 0 (0) | 3 (1.4) |
| 70 to <90 | 50 (23.2) | 63 (29.2) | 0 (0) | 0 (0) | 113 (52.3) |
| 90 to <100 | 6 (2.8) | 70 (32.4) | 9 (4.2) | 2 (0.9) | 87 (40.3) |
| 100 | 0 (0) | 0 (0) | 1 (0.5) | 12 (5.6) | 13 (6.0) |
| Total | 58 (26.9) | 134 (62.0) | 10 (4.6) | 14 (6.5) | 216* (100) |
12 lesions not compared due to missing quantitative information on stenosis severity in the clinical record
PCI = Percutaneous Coronary Intervention, QCA = Quantitative Coronary Angiography
Our findings were essentially unchanged when we repeated our analyses after excluding patients with lesions associated with non-ST elevation myocardial infarction or within coronary artery bypass grafts (see Supplementary Material: Tables B and C; and Figures A and B). Finally, we found the mean difference in percent diameter stenosis between the clinical interpretation and QCA was greatest for lesions between 50 to <70% by QCA, but diminished with higher stenosis severity (Table 4). Less variation in the mean difference in percent diameter stenosis between the 2 methods was noted across several other subgroups (Table 4), with the exception of variation across individual hospital sites that ranged from 5.6% to 11.2% (Figure 3). Using alternative cutoffs to categorize lesions by percent diameter stenosis from two methods did not lead to substantial differences in our results (Supplementary Material: Table D).
Table 4.
Mean difference in stenosis severity (%) between clinical interpretation and QCA across subgroups.
| N | Mean Difference | SD | P-Value | ||
|---|---|---|---|---|---|
| Total Population | 216 | +8.2 | 8.4 | <0.001 | |
| Subgroup Category | |||||
| Percent Diameter Stenosis, QCA | 50 to <70 | 58 | +12.3 | 8.4 | <.0001 |
| 70 to <90 | 134 | +7.7 | 8.0 | ||
| 90 to <100 | 10 | 2.6 | 4.9 | ||
| 100 Left | 14 | −0.1 | 0.4 | ||
| Lesion Location | Left circumflex | 61 | +8.1 | 8.0 | 0.7014 |
| Left anterior descending | 82 | +9.0 | 9.0 | ||
| Left main | 1 | +7.5 | -- | ||
| Ramus intermedius | 2 | +11.6 | 2.1 | ||
| Right | 70 | +7.1 | 8.0 | ||
| Lesion Reference Vessel Diameter, QCA | <2.5mm | 87 | +8.8 | 9.0 | 0.6800 |
| 2.5 to <3mm | 73 | +7.9 | 7.6 | ||
| 3 to <3.5mm | 39 | +8.1 | 9.1 | ||
| >3.5mm | 17 | +6.1 | 6.5 | ||
| Lesion Length, QCA | <10mm | 65 | +9.8 | 8.7 | 0.1003 |
| 10 to <20mm | 98 | +8.2 | 8.5 | ||
| >20mm | 50 | +7.5 | 7.5 | ||
| Angiogram Quality by QCA Core Lab | Excellent | 31 | +7.2 | 8.5 | 0.5898 |
| Good | 148 | +8.1 | 8.5 | ||
| Satisfactory | 37 | +9.3 | 7.9 | ||
| Stress Test or FFR Performed | No | 88 | +9.1 | 8.6 | 0.1836 |
| Yes | 128 | +7.5 | 8.2 | ||
| Hospital Site | 1 | 35 | 5.6 | 7 | 0.0437 |
| 2 | 28 | 5.9 | 8 | ||
| 3 | 30 | 7.4 | 6.6 | ||
| 4 | 29 | 7.8 | 9.4 | ||
| 5 | 30 | 8.2 | 8.8 | ||
| 6 | 28 | 10.6 | 7.8 | ||
| 7 | 36 | 11.2 | 9.4 | ||
QCA = Quantitative Coronary Angiography, FFR = Fractional Flow Reserve
Figure 3.
Variation by hospital in differences in mean percent diameter stenosis and 95% confidence intervals between clinical interpretation and Quantitative Coronary Angiography (QCA).
Discussion
We found significant differences between the percent diameter stenosis of a lesion as assessed by clinical interpretation and QCA in patients undergoing PCI at seven U.S. hospitals. In general, the clinical interpretation by physicians was 70% or greater in most treated lesions, whereas approximately a quarter of the measurements by QCA were below that level. However, the extent of differences was +8% on average and no lesion was less than 50% by QCA. Overall, findings from our study suggest potential opportunities for improving the clinical interpretation of coronary angiograms in routine practice, and thus, optimizing the selection and care of patients considered for revascularization.
The clinical value of any imaging test depends upon several factors, including acquisition and interpretation of the images and incorporation of this information into clinical decision-making. Despite many technical advances that have transformed the ways in which image acquisition now occurs with coronary angiography, little work has been done on developing strategies for improving its interpretation over the years. Indeed, interpretation may be even more challenging today as more decisions about revascularization are made during or just after the procedure is performed, in order to maximize efficiency and minimize costs (i.e., ad hoc PCI).22 This may limit what formerly occurred through collective discussions (e.g., “cath conference”), despite earlier evidence that “group” reads significantly improves the accuracy of interpretations.23,24
Thus, our findings of the inconsistency between the clinical interpretation and an independent measurement by QCA, particularly for lower severity stenoses, raise concerns. Despite its limitations, newer-generation systems of QCA have high reproducibility and precision in quantifying stenosis severity even in complex lesions,25 which has contributed its widespread use in clinical trials of revascularization. Although differences between the clinical interpretation and QCA in an isolated patient should never be considered an automatic “flag” for inappropriate PCI, identifying where inconsistencies exist may provide opportunities for clinicians to understand ways to improve. For example, routine feedback on ‘over-reads’ of coronary angiograms through educational initiatives could enhance clinical decision-making about the need for further testing (e.g., FFR) prior to PCI. In our study, for example, use of FFR was relatively uncommon despite its growing role in the assessment of the physiological significance of angiographic lesions and determinations of revascularization. Expanded use of FFR, as well as techniques like digital calipers and online QCA, may be tools that could improve assessment of stenosis severity by clinical interpretation.1
Providing feedback to hospitals also may be useful for improving clinical interpretation, as we did notice facility-level variation in the mean difference in percent diameter stenosis between the 2 methods despite the small number included in this analysis. In this context, our findings may be particularly important for quality assurance programs. Although earlier efforts have focused on improving the selection and quality of care for PCI patients through clinical registries,26 practical constraints have forced such programs to focus largely on evaluating data obtained via chart abstraction, rather than validating the accuracy of the primary data on which clinical decisions are made – in this case, stenosis severity. Recently, these concerns were exacerbated by high-profile cases in which cardiac surgeons and cardiologists were accused of performing revascularization on patients with coronary artery disease of questionable severity.27,28 Moreover, some of these providers have consistently reported better than expected outcomes,29 since treating mild coronary artery disease is almost always safe for patients, despite providing little benefit. This underscores the limitations of quality assurance tools that focus largely on chart abstraction and assessing complications.
Challenges exist when considering the potential next steps that may result from our findings. New approaches need to be developed for improving clinical interpretation through innovative educational initiatives or quality assurance programs. Given its potential scalability, QCA may offer be an efficient method for achieving these objectives, but this is unknown. In particular, it is necessary to examine how QCA or other methods to improve clinical interpretation may be integrated into the busy workflow of interventional cardiology. This must be done in a critical and rigorous manner, as the addition of such tools does not necessarily result in improvement. For example, data on the clinical value of computer-assisted screening mammography in routine practice have been mixed.30,31
Our study should be interpreted in the context of the following limitations. First, we only examined patients undergoing PCI. We did not perform QCA in lesions that were considered clinically insignificant or managed medically or surgically; our findings are not relevant to those settings, which will require additional investigations. Second, QCA itself has limitations. As such, this study was focused specifically on assessing the quality of the clinical interpretation of the coronary angiogram, not the appropriateness of the clinical decision to intervene. For example, QCA does not include information on the hemodynamic significance of a stenosis. In isolation, it does not account for many factors that should influence clinical decisions on revascularization nor does it alone predict long-term outcomes after treatment.32 Nevertheless, accurate assessment of stenosis severity is essential for physicians and patients, as this remains arguably the most critical factor in practical, day-to-day decisions about revascularization. Even current Appropriateness Use Criteria that emphasize the importance of symptoms and functional testing assume the presence of a “significant” stenosis of “greater than or equal to 70% luminal diameter narrowing, by visual assessment” prior to revascularization. Of course, future work will need to tie findings such as ours directly to clinical decisions and outcomes.
Third, calculating stenosis severity by QCA still requires satisfactory image acquisition and minimal user input to identify imaging frames for analysis, which may introduce variability as well. For this reason, our assessments were performed using analysts blinded to the clinical interpretation and at a core laboratory with broad experience in regulatory studies involving QCA. Fourth, our study was limited to 7 hospitals. These were primarily high-volume and recognized PCI centers, and importantly, each volunteered to participate as part of a pilot quality improvement initiative. Whether our findings are applicable to a more broadly representative group of hospitals is uncertain, though the results may represent a best-case scenario. For example, it may be that coronary angiograms at other hospitals may have technical deficiencies not found here, particularly with more complex lesions. Finally, due to our limited sample size we were unable to examine variability in assessments of stenosis severity by physician. Future work will need to better quantify the effects of both hospital- and physician-level variation on clinical interpretation.
In conclusion, we found that physicians tended to assess lesions treated by PCI as more severe than measurements by QCA. Findings from our study are consistent with older work and suggest possible opportunities to further improve clinical interpretation of coronary angiography and optimize the selection and care of patients undergoing PCI in contemporary practice.
Supplementary Material
Acknowledgements
Funding Source: Supported by Aetna Inc., one of the nation’s leaders in health care, dental, pharmacy, group life, and disability insurance, and employee benefits. The views presented here are those of the author and not necessarily those of Aetna, its directors, officers, or staff. The investigators were responsible for all data collection and analyses, as well as the decision to publish the findings.
We are grateful to the participating physicians, nurses, technicians, and research assistants from the participating hospital sites. We are grateful to Ken Huelskamp, Jim Beachy and Kathleen Hewitt for their assistance at the National Cardiovascular Data Registry® (NCDR®).
Dr. Spertus reports grant funding from the Aetna Foundation. Dr. Cohen reports consulting income from United Healthcare. Dr. Kureshi received support from NIH grant T32HL110837. Dr. Walsh reports consulting income from United Healthcare and Eli Lilly. Dr. Chazal reports being a member of the Scientific Advisory Board of United Healthcare. Dr. Rumsfeld reports being the Chief Science Officer for the NCDR. Dr. Reiber reports being the President and CEO of Medis, a leading provider of software for the quantification of cardiovascular images. Dr. Richard Krumholz reports being the General Manager of ImageCor, a company focused on improving clinical interpretation of medical imaging. Dr. Harlan Krumholz reports equity interest in ImageCor and being chair of the Scientific Advisory Board of United Healthcare.
Footnotes
Disclosures
All other co-authors report no relevant conflicts of interest.
References
- 1.Scanlon PJ, Faxon DP, Audet AM, Carabello B, Dehmer GJ, Eagle KA, Legako RD, Leon DF, Murray JA, Nissen SE, Pepine CJ, Watson RM, Ritchie JL, Gibbons RJ, Cheitlin MD, Gardner TJ, Garson A, Jr, Russell RO, Jr, Ryan TJ, Smith SC., Jr ACC/AHA guidelines for coronary angiography. A report of the American College of Cardiology/American Heart Association Task Force on practice guidelines (Committee on Coronary Angiography). Developed in collaboration with the Society for Cardiac Angiography and Interventions. Circulation. 1999;99:2345–2357. doi: 10.1161/01.cir.99.17.2345. [DOI] [PubMed] [Google Scholar]
- 2.Levine GN, Bates ER, Blankenship JC, Bailey SR, Bittl JA, Cercek B, Chambers CE, Ellis SG, Guyton RA, Hollenberg SM, Khot UN, Lange RA, Mauri L, Mehran R, Moussa ID, Mukherjee D, Nallamothu BK, Ting HH. 2011 ACCF/AHA/SCAI Guideline for Percutaneous Coronary Intervention A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines and the Society for Cardiovascular Angiography and Interventions. Circulation. 2011;124:e574–e651. doi: 10.1161/CIR.0b013e31823ba622. [DOI] [PubMed] [Google Scholar]
- 3.Marcus ML, Skorton DJ, Johnson MR, Collins SM, Harrison DG, Kerber RE. Visual estimates of percent diameter coronary stenosis: “a battered gold standard”. J Am Coll Cardiol. 1988;11:882–885. doi: 10.1016/0735-1097(88)90226-4. [DOI] [PubMed] [Google Scholar]
- 4.Raphael MJ, Donaldson RM. A “significant” stenosis: thirty years on. Lancet. 1989;1(8631):207–209. doi: 10.1016/s0140-6736(89)91214-2. [DOI] [PubMed] [Google Scholar]
- 5.Detre KM, Wright E, Murphy ML, Takaro T. Observer agreement in evaluating coronary angiograms. Circulation. 1975;52:979–986. doi: 10.1161/01.cir.52.6.979. [DOI] [PubMed] [Google Scholar]
- 6.DeRouen TA, Murray JA, Owen W. Variability in the analysis of coronary arteriograms. Circulation. 1977;55:324–328. doi: 10.1161/01.cir.55.2.324. [DOI] [PubMed] [Google Scholar]
- 7.Galbraith JE, Murphy ML, De Soyza N. Coronary angiogram interpretation. Interobserver variability. JAMA. 1978;240:2053–2056. [PubMed] [Google Scholar]
- 8.Fisher LD, Judkins MP, Lesperance J, Cameron A, Swaye P, Ryan T, Maynard C, Bourassa M, Kennedy JW, Gosselin A, Kemp H, Faxon D, Wexler L, Davis KB. Reproducibility of coronary arteriographic reading in the coronary artery surgery study (CASS) Cathet Cardiovasc Diagn. 1982;8:565–575. doi: 10.1002/ccd.1810080605. [DOI] [PubMed] [Google Scholar]
- 9.Goldberg RK, Kleiman NS, Minor ST, Abukhalil J, Raizner AE. Comparison of quantitative coronary angiography to visual estimates of lesion severity pre and post PTCA. Am Heart J. 1990;119:178–184. doi: 10.1016/s0002-8703(05)80098-5. [DOI] [PubMed] [Google Scholar]
- 10.Fleming RM, Kirkeeide RL, Smalling RW, Gould KL. Patterns in visual interpretation ofcoronary arteriograms as detected by quantitative coronary arteriography. J Am Coll Cardiol. 1991;18:945–951. doi: 10.1016/0735-1097(91)90752-u. [DOI] [PubMed] [Google Scholar]
- 11.Desmet W, Willems J, Lierde JV, Piessens J. Discrepancy between visual estimation and computer-assisted measurement of lesion severity before and after coronary angioplasty. Cathet Cardiovasc Diagn. 1994;31:192–198. doi: 10.1002/ccd.1810310306. [DOI] [PubMed] [Google Scholar]
- 12.Folland ED, Vogel RA, Hartigan P, Bates ER, Beauman GJ, Fortin T, Boucher C, Parisi AF. Relation between coronary artery stenosis assessed by visual, caliper, and computer methods and exercise capacity in patients with single-vessel coronary artery disease. The Veterans Affairs ACME Investigators. Circulation. 1994;89:2005–2014. doi: 10.1161/01.cir.89.5.2005. [DOI] [PubMed] [Google Scholar]
- 13.Leape LL, Park RE, Bashore TM, Harrison JK, Davidson CJ, Brook RH. Effect of variability in the interpretation of coronary angiograms on the appropriateness of use of coronary revascularization procedures. Am Heart J. 2000;139:106–113. doi: 10.1016/s0002-8703(00)90316-8. [DOI] [PubMed] [Google Scholar]
- 14.Bashore TM, Balter S, Barac A, Byrne JG, Cavendish JJ, Chambers CE, Hermiller JB, Jr, Kinlay S, Landzberg JS, Laskey WK, McKay CR, Miller JM, Moliterno DJ, Moore JW, Oliver-McNeil SM, Popma JJ, Tommaso CL. 2012 American College of Cardiology Foundation/Society for Cardiovascular Angiography and Interventions expert consensus document on cardiac catheterization laboratory standards update. A report of the American College of Cardiology Foundation Task Force on Expert Consensus documents. J Am Coll Cardiol. 2012;59:2221–2305. doi: 10.1016/j.jacc.2012.02.010. [DOI] [PubMed] [Google Scholar]
- 15.Patel MR, Dehmer GJ, Hirshfeld JW, Smith PK, Spertus JA. ACCF/SCAI/STS/AATS/AHA/ASNC/HFSA/SCCT 2012 Appropriate use criteria for coronary revascularization focused update: a report of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, Society for Cardiovascular Angiography and Interventions, Society of Thoracic Surgeons, American Association for Thoracic Surgery, American Heart Association, American Society of Nuclear Cardiology, and the Society of Cardiovascular Computed Tomography. J Am Col1 Cardiol. 2012;59:857–881. doi: 10.1016/j.jacc.2011.12.001. [DOI] [PubMed] [Google Scholar]
- 16.Reiber JH, Kooijman CJ, Slager CJ, Gerbrands JJ, Schuurbiers JC, Den Boer A, Wijns W, Serruys PW, Hugenholtz PG. Coronary artery dimensions from cineangiograms methodology and validation of a computer-assisted analysis procedure. IEEE Trans Med Imaging. 1984;3:131–141. doi: 10.1109/TMI.1984.4307669. [DOI] [PubMed] [Google Scholar]
- 17.Reiber JHC, Tuinenburg JC, Koning G, Janssen JP, Rareş A, Lansky AJ, Goedhart B. Chapter 2.2: Quantitative Coronary Arteriography. In: Oudkerk M, Reiser MF, editors. Coronary Radiology, 2nd Revised Edition, Oudkerk M, Reiser M.F. (eds), Series: Medical Radiology, Sub-series: Diagnostic Imaging, Baert AL, Knauth M, Sartor K (eds) Berlin-Heidelberg: Springer-Verlag; 2009. pp. 41–65. [Google Scholar]
- 18.Reiber JH, Serruys PW, Kooijman CJ, Wijns W, Slager CJ, Gerbrands JJ, Schuurbiers JC, den Boer A, Hugenholtz PG. Assessment of short-, medium-, and long-term variations in arterial dimensions from computer-assisted quantitation of coronary cineangiograms. Circulation. 1985;71:280–288. doi: 10.1161/01.cir.71.2.280. [DOI] [PubMed] [Google Scholar]
- 19.Reiber JHC, Serruys PW, Slager CJ. Quantitative coronary angiography; methodology and clinical applications. Boston/Dordrecht/Lancaster: Martinus Nijhoff Publishers; 1986. [Google Scholar]
- 20.Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213–220. doi: 10.1037/h0026256. [DOI] [PubMed] [Google Scholar]
- 21.Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37:360–363. [PubMed] [Google Scholar]
- 22.Nallamothu BK, Krumholz HM. Putting ad hoc PCI on pause. JAMA. 2010;304:2059–2060. doi: 10.1001/jama.2010.1509. [DOI] [PubMed] [Google Scholar]
- 23.Beauman GJ, Vogel RA. Accuracy of individual and panel visual interpretations of coronary arteriograms: implications for clinical decisions. J Am Coll Cardiol. 1990;16:108–113. doi: 10.1016/0735-1097(90)90465-2. [DOI] [PubMed] [Google Scholar]
- 24.Kussmaul WG, Popp RL, Norcini J. Accuracy and reproducibility of visual coronary stenosis estimates using information from multiple observers. Clin Cardiol. 1992;15:154–162. doi: 10.1002/clc.4960150305. [DOI] [PubMed] [Google Scholar]
- 25.Lansky A, Tuinenburg J, Costa M, Maeng M, Koning G, Popma J, Cristea E, Gavit L, Costa R, Rares A, Van Es GA, Lefevre T, Reiber H, Louvard Y, Morice MC. Quantitative angiographic methods for bifurcation lesions: a consensus statement from the European Bifurcation Group. Catheter Cardiovasc Interv. 2009;73:258–266. doi: 10.1002/ccd.21814. [DOI] [PubMed] [Google Scholar]
- 26.Dehmer GJ, Weaver D, Roe MT, Milford-Beland S, Fitzgerald S, Hermann A, Messenger J, Moussa I, Garratt K, Rumsfeld J, Brindis RG. A contemporary view of diagnostic cardiac catheterization and percutaneous coronary intervention in the United States: a report from the CathPCI Registry of the National Cardiovascular Data Registry, 2010 through June 2011. J Am Coll Cardiol. 2012;60(20):2017–2031. doi: 10.1016/j.jacc.2012.08.966. [DOI] [PubMed] [Google Scholar]
- 27.Klaidman S. Coronary: A True Story of Medicine Gone Awry. Scribner; 2008. [Google Scholar]
- 28.Devi S. US physicians urge end to unnecessary stent operations. Lancet. 2011;378(9792):651–652. doi: 10.1016/s0140-6736(11)61317-2. [DOI] [PubMed] [Google Scholar]
- 29.Parker JP, Li Z, Damberg CL, Danielsen B, Marcin J, Dai J, Steimle AE. The California Report on Coronary Artery Bypass Graft Surgery 2000–2002 Hospital Data. San Francisco, CA: California Office of Statewide Health Planning and Development and the Pacific Business Group on Health; 2005. Feb, [Google Scholar]
- 30.Fenton JJ, Taplin SH, Carney PA, Cutter G, D'Orsi C, Sickles EA, Fosse J, Abraham L, Taplin SH, Barlow W, Hendrick RE, Elmore JG. Influence of computer-aided detection on performance of screening mammography. N Engl J Med. 2007;356:1399–1409. doi: 10.1056/NEJMoa066099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Gilbert FJ, Astley SM, Gillan MGC, Agbaje OF, Wallis MG, James J, Boggis CR, Duffy SW. Single Reading with Computer-Aided Detection for Screening Mammography. N Engl J Med. 2008;359:1675–1684. doi: 10.1056/NEJMoa0803545. [DOI] [PubMed] [Google Scholar]
- 32.Faxon DP, Vogel R, Yeh W, Holmes DR, Jr, Detre K. Value of visual versus central quantitative measurements of angiographic success after percutaneous transluminal coronary angioplasty. NHLBI PTCA Registry Investigators. Am J Cardiol. 1996;77:1067–1072. doi: 10.1016/s0002-9149(96)00133-6. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.