Evaluation of Carotid Stenosis by Angiography: Potential Bias toward Overestimated Measurements Introduced by Prior Interpretation of Doppler Sonograms

Abstract

BACKGROUND AND PURPOSE: Doppler sonography of the carotid arteries is routinely performed before catheter angiography, and its results may bias the subsequent interpretation of angiograms. We attempt to establish that Doppler sonography may show an exagerrated degree of carotid stenosis, introducing bias to the evaluation of carotid stenosis by subsequent catheter angiography.

METHODS: Angiograms of the carotid arteries obtained to evaluate potential carotid stenosis in patients who had undergone Doppler sonography of the carotid arteries were retrospectively reviewed (from 1993 to 1998). Readers who were blinded to the previous interpretations of the angiography and Doppler sonography results measured carotid stenosis. The results of Doppler sonography of the carotid arteries were not re-evaluated. Based on the original Doppler sonograms, stenoses were categorized as normal/mild (<30%), moderate (30–59%), severe (60–79%), and critical (80–99%). Within these categories, the differences between the original percent stenosis, as determined by angiography, and the blinded measurements were determined.

RESULTS: A total of 106 patients with angiographically measurable stenoses in 128 vessels were identified. The difference between the blinded readers was 3% (±8%), with no category statistically different from the other. The difference between original and remeasured stenoses in carotid arteries in the Doppler categories were as follows: mild stenosis, 2% (±9%); moderate stenosis, 6% (±15%); severe stenosis, 8% (±15%); and critical stenosis, 22% (±12%). A significant overestimation occurred in the severe (P < .05) and critical (P < .0001) stenosis categories. One third of patients with stenoses in the severe or critical Doppler category had significant stenoses on the original angiograms that were less than 60%, according to blinded remeasurement.

CONCLUSION: Doppler sonography of the carotid arteries has the potential to bias the subsequent interpretation of catheter angiography. Care must be taken to measure stenosis accurately, using strict criteria to determine the potential benefit of carotid endarterectomy for the individual patient and to ensure that the criteria for Doppler sonography of the carotid arteries are based on accurate catheter angiography measurements.

Conventional catheter angiography has been the traditional standard of reference for evaluation of carotid bifurcation stenosis. The North American Symptomatic Carotid Endarterectomy Trial (NASCET) showed that carotid endarterectomy can reduce the risk of stroke that is associated with atherosclerotic carotid stenosis if the diameter of the stenosis revealed by angiography is greater than 50% in symptomatic patients (1, 2). More controversially, the Asymptomatic Carotid Atherosclerosis Study (ACAS) showed that carotid endarterectomy can reduce the risk of stroke that is associated with atherosclerotic carotid stenosis in asymptomatic patients with greater than 60% stenosis (3). In ascertaining the validity of the NASCET measurement, a tendency to overestimate percent stenosis was found when the measurements obtained by neuroradiologists from multiple institutions were compared with those obtained by the single investigator in the NASCET (4). At our facility, we noted that catheter angiography routinely indicated a higher percent stenosis than our Doppler sonography studies had suggested. We attempted to determine whether overestimated percent stenosis by angiography was similar to that experienced by the NASCET group, indicating a similar mechanism, or whether the positive results of the Doppler sonography biased the results of the angiography. Our study was designed to evaluate the potential bias created in the minds of the angiographers by the “positive” Doppler results that indicated “severe” or “critical” stenoses.

Methods

The angiograms of the carotid arteries of consecutive patients who were referred to our institution between December 1993 and May 1998 for evaluation of carotid stenosis and who had undergone Doppler sonography of the carotid arteries were retrospectively reviewed. Using the NASCET criteria, two radiologists, who had not been involved in the original Doppler sonography or angiography and who were blinded to the previous results, remeasured the degree of carotid stenosis (5, 6). Four angiographers interpreted the original angiograms, and six radiologists interpreted the Doppler sonograms. Exclusion criteria included an incomplete examination, carotid occlusion, and partial collapse of the distal internal carotid artery (approaching near-occlusion, according to the NASCET criteria) that would preclude accurate calculation of percent stenosis. Based on Doppler sonography of the carotid arteries, stenoses were categorized according to the criteria presented by Bluth et al (7) into normal/mild (<30%), moderate (30–59%), severe (60−79%), and critical (80−99%). The results of the Doppler sonography were not re-evaluated. Vessels with no stenosis revealed by angiography were removed from the normal/mild category, leaving only vessels with measurable stenoses. Within Doppler sonography categories, the differences between the blinded calculations of the angiography percent stenosis and the original dictated percent stenosis were determined. A statistical comparison of the differences between the measurements obtained by the two blinded readers and of the differences between the average blinded calculation and the original reported percent stenosis within the Doppler sonography–derived categories of mild, moderate, severe, and critical was conducted.

Results

A total of 106 patients were identified, and measurable stenoses were shown in 128 vessels. Of the remaining 84 vessels, 47 carotid arteries had no measurable stenosis (carotid bulb larger than distal internal carotid artery), 23 vessels were occluded, five were “approaching near-occlusion,” and the remaining nine had incomplete Doppler sonography or angiography studies.

The difference between the readings of two blinded angiographers for all Doppler categories was 3% (±8%), with no category statistically different from the others. The differences between the original and remeasured angiography percent stenoses in the carotid arteries within the Doppler categories were as follows: mild stenosis (34 vessels), 2% (±9%); moderate stenosis (11 vessels), 6% (±15%); severe stenosis (38 vessels), 8% (±15%); and critical stenosis (45 vessels), 22% (±12%) (Table). An example of each Doppler category is shown in Figures 1 through 4. There was a statistically significant difference in angiographic stenosis in the severe (P < .05) and critical (P < .0001) stenosis categories when the moderate, severe, and critical categories were compared with the mild category. One third of the patients (27 of 83 patients) in the severe or critical stenosis Doppler category were determined to have greater than 60% stenosis according to the original interpretation but less than 60% according to the blinded measurement.

Percent stenosis interpreted from angiography, by carotid Doppler category

fig 1.

Results of Doppler sonography of the carotid arteries were interpreted as mild stenosis with a peak systolic velocity of 104 cm/s. Percent stenosis was measured as 25% according to the original interpretation and as 23% by the blinded readers. Arrow indicates point of maximum stenosis, and arrowhead indicates normal distal internal carotid artery.fig 2. Results of sonography of the carotid arteries were interpreted as moderate stenosis with a peak systolic velocity of 132 cm/s. Percent stenosis was measured as 33% according to the original interpretation and as 28% by the blinded readers. Arrow indicates point of maximum stenosis, and arrowhead indicates normal distal internal carotid artery.fig 3. Results of Doppler sonography of the carotid arteries were interpreted as severe stenosis with a peak systolic velocity of 170 cm/s. Percent stenosis was measured as 70% according to the original interpretation and as 52% by the blinded readers. Arrow indicates point of maximum stenosis, and arrowhead indicates normal distal internal carotid artery.fig 4. Results of Doppler sonography of the carotid arteries were interpreted as critical stenosis with a peak systolic velocity of 450 cm/s. Percent stenosis was measured as 83% according to the original interpretation and as 61% by the blinded readers. Arrow indicates point of maximum stenosis, and arrowhead indicates normal distal internal carotid artery

Discussion

Radiologists require all available pertinent information when interpreting a patient's examination results, but they should be aware that the reported results of previous studies or other elements of a patient's history might significantly bias their interpretations. All examinations are potentially biased by a priori information, but physicians should be cognizant of the biases to decrease their potential harm. Bias in the workup of carotid stenosis warrants particular attention because of the profound societal costs and potential risks of stroke if carotid endarterectomy is performed inappropriately.

The stroke reduction in the NASCET study was based on conventional catheter angiography, with greater stroke reduction associated with higher grades of stenosis. The correlation of Doppler sonography of the carotid arteries and stroke reduction in the NASCET study was poor, without proven stroke reduction if the endarterectomy had been based on the Doppler examination (4). The ACAS included a combination of catheter angiography and Doppler sonography of the carotid arteries, but the Doppler criteria were variable from site to site, with a requirement that the Doppler criteria produce a 95% positive predictive value (3). The ACAS patient population included a mixture of patients with more severe stenoses, who were entered in the study based on Doppler criteria, and patients with less severe stenoses, who were entered in the study based on catheter angiography criteria.

The published criteria for Doppler sonography of the carotid arteries are markedly variable regarding the velocity threshold chosen to indicate a significant stenosis, but the published reports of the sensitivity and specificity of the examination are more consistent. There are several possible reasons for the variability of the published criteria, including machine and technician variation, spectrum of disease in the patient population, and many definitions of a significant stenosis. Area stenosis and diameter stenosis determined using the NASCET or European Carotid Surgery Trial measurement techniques create marked variation in the measured stenosis on angiograms with which Doppler velocity criteria are correlated. The consistency of the published reports of the sensitivity and specificity with differing velocities is possible by varying the threshold velocity. In most studies, this is done to produce a sensitive screening examination for subsequent confirmatory angiography. Because of this reasoning, the threshold velocities do not equal the stated percent stenosis but merely indicate a high probability of finding most patients with significant stenoses. The criteria published and used by many institutions will therefore consistently lead to overestimated measurements of stenosis when compared with measurements determined by conventional angiography. In our patient population, a peak systolic velocity of 225 cm/s produced a sensitivity of 90% and specificity of 96%, similar to those of recent studies, but the average percent stenosis of patients with a peak systolic velocity of 225 cm/s is only 50% (±18%). If surgery were performed based on the peak systolic velocity of 225 cm/s alone, half of the patients would have angiography percent stenosis of less than 50%. We currently use a low peak systolic velocity threshold of 130 cm/s because of its high sensitivity (identifying patients with potential benefit) but require MR angiography or catheter angiography to confirm surgical disease. Both the radiologist and the referring physician must be aware that the positive Doppler result does not signify a severe stenosis but is only a screening test; severity of stenosis should be based on MR angiography or conventional angiography.

In the original NASCET study, a single radiologist measured the percent stenosis on the original angiograms. To verify the method, several angiograms were sent to multiple institutions and the percent stenosis was remeasured (4). The blinded readers in that study systematically overestimated percent stenosis in 6% of carotid arteries with less than 70% stenosis. Presumably, our blinded readers' interpretations yielded similar overestimation, suggesting that the actual overestimation of stenosis in the original dictation may have been even greater. In the verification study presented by the NASCET group, no remeasurement of severe stenosis vessels is reported. It is possible that an even greater overestimation of stenosis may have occurred in those vessels.

Our results indicate that a bias may be created in the minds of angiographers by the positive results of Doppler sonography. Small sample size and the fact that evaluation was conducted at a single institution limited our study. At our facility, Doppler sonography of the carotid arteries is performed and interpreted in the radiology department. This would increase bias if the radiologist performing the catheter angiography were reluctant to disagree with a colleague. A systematic overestimation of higher grades of stenosis by the original angiographers, regardless of the Doppler result, would create a similar judgment, and is an alternative explanation of our findings. Neither of the two blinded radiologists in our study had a known bias that would lead to systematically underestimated measurements in cases of higher-grade stenosis, but if this bias existed, it would yield underestimated percent stenoses.

Doppler criteria based on the NASCET have been published, but they also are designed to create a particular combination of sensitivity and specificity not to equal a percent stenosis. The result may still constitute a bias on the angiographers' part to agree with the Doppler examiners' impression of a higher-grade stenosis, although the new higher velocity criteria may lessen the degree of overestimation.

Conclusion

No significant difference between the original and blinded measurements of carotid stenosis based on angiography was found if Doppler sonography of the carotid arteries was predictive of mild or moderate stenosis. If Doppler sonography of the carotid arteries were predictive of greater than 60% stenosis, the interpretation of original angiograms led to overestimated measurements of stenosis by 16% (P < .05). Care must be taken to measure stenosis accurately by using strict criteria to determine the potential benefit of carotid endarterectomy in the individual patient and to ensure that the criteria for Doppler sonography of the carotid arteries are based on accurate catheter angiography measurements.