Optimization of duplex velocity criteria for diagnosis of internal carotid artery (ICA) stenosis: A report of the Intersocietal Accreditation Commission (IAC) Vascular Testing Division Carotid Diagnostic Criteria Committee

Abstract

Diagnostic criteria to classify severity of internal carotid artery (ICA) stenosis vary across vascular laboratories. Consensus-based criteria, proposed by the Society of Radiologists in Ultrasound in 2003 (SRUCC), have been broadly implemented but have not been adequately validated. We conducted a multicentered, retrospective correlative imaging study of duplex ultrasound versus catheter angiography for evaluation of severity of ICA stenosis. Velocity data were abstracted from bilateral duplex studies performed between 1/1/2009 and 12/31/2015 and studies were interpreted using SRUCC. Percentage ICA stenosis was determined using North American Symptomatic Carotid Endarterectomy Trial (NASCET) methodology. Receiver operating characteristic analysis evaluated the performance of SRUCC parameters compared with angiography. Of 448 ICA sides (from 224 patients), 299 ICA sides (from 167 patients) were included. Agreement between duplex ultrasound and angiography was moderate (κ = 0.42), with overestimation of degree of stenosis for both moderate (50–69%) and severe (⩾ 70%) ICA lesions. The primary SRUCC parameter for ⩾ 50% ICA stenosis of peak-systolic velocity (PSV) of ⩾ 125 cm/sec did not meet prespecified thresholds for adequate sensitivity, specificity, and accuracy (sensitivity 97.8%, specificity 64.2%, accuracy 74.5%). Test performance was improved by raising the PSV threshold to ⩾ 180 cm/sec (sensitivity 93.3%, specificity 81.6%, accuracy 85.2%) or by adding the additional parameter of ICA/common carotid artery (CCA) PSV ratio ⩾ 2.0 (sensitivity 94.3%, specificity 84.3%, accuracy 87.4%). For ⩾ 70% ICA stenosis, analysis was limited by a low number of cases with angiographically severe disease. Interpretation of carotid duplex examinations using SRUCC resulted in significant overestimation of severity of ICA stenosis when compared with angiography; raising the PSV threshold for ⩾ 50% ICA stenosis to ⩾ 180 cm/sec as a single parameter or requiring the ICA/CCA PSV ratio ⩾ 2.0 in addition to PSV of ⩾ 125 cm/sec for laboratories using the SRUCC is recommended to improve the accuracy of carotid duplex examinations.

Introduction

Internal carotid artery (ICA) stenosis is a common manifestation of atherosclerotic vascular disease and an important risk factor for ischemic stroke. ¹ Duplex ultrasound uses color and spectral Doppler evaluation of blood flow combined with grayscale imaging of plaque to determine the presence and severity of ICA stenosis. Since the original University of Washington carotid duplex criteria were published and widely adopted in the 1980s, there have been ongoing efforts to refine carotid diagnostic criteria that have continued to the present time.^2–10 In addition, the definition of stenosis using the gold standard of catheter angiography has evolved, including methodology for determination of percentage ICA stenosis on angiography. ¹¹ This refinement has led to a proliferation of different diagnostic criteria and a lack of standardization across vascular testing facilities, even among those accredited by a single organization.^12,13

In 2003, a set of consensus-based carotid diagnostic criteria were developed and published under the leadership of the Society of Radiologists in Ultrasound (SRU). ⁶ These SRU Consensus Criteria (SRUCC) incorporated different elements of previously validated and published parameters into proposed multiparameter criteria with a goal of wide deployment in vascular laboratories (online supplemental Table 1). At the time of publication, validation of the SRUCC criteria in comparison to an imaging standard was not performed, as the parameters in the SRUCC were based on expert consensus and amalgamation of previously published correlation studies.^4,9,14,15 Of note, in addition to proposing diagnostic criteria, the SRUCC report recommended the use of the North American Symptomatic Carotid Endarterectomy Trial (NASCET) methodology for measurement of correlative carotid angiograms rather than the European Carotid Surgery Trial (ECST) method, which had historically been used due to less variability of measurement of the reference lumen diameter (online supplemental Figure 1).^6,11

Intersocietal Accreditation Commission (IAC) Vascular Testing has offered accreditation for vascular testing facilities performing extracranial carotid duplex ultrasound since 1991. To achieve accreditation, IAC requires facilities to have written protocols for the performance (scanning) of studies and adherence to previously published/referenced or internally validated diagnostic criteria for interpretation of studies, though no specific diagnostic criteria have been mandated for use. IAC Vascular Testing has previously reported on marked variability of carotid diagnostic criteria among its accredited laboratories. ¹³ In 2012, IAC Vascular Testing performed an electronic survey of medical and technical staff of its accredited facilities regarding this issue and reported that more than two-thirds of respondents indicated that there should be only one set of diagnostic criteria and further that IAC Vascular Testing should require the consistent use of one set of researched and validated criteria in its facilities. ¹³ As a result of these data, IAC Vascular Testing commissioned the Carotid Diagnostic Criteria Committee and this research study to facilitate standardized carotid diagnostic criteria in its accredited facilities.

Methods

This study design was a multicentered, retrospective correlative imaging study of duplex ultrasound versus catheter angiography for diagnosis of ICA stenosis. Present and former members of the IAC Vascular Testing Division Board of Directors were invited to provide appropriate cases for the study (Appendix). Each participating site was accredited in extracranial carotid testing by the IAC and agreed to contribute a series of de-identified carotid duplex ultrasound studies with the corresponding catheter angiograms along with limited demographic and clinical information. The study protocol was approved by the Institutional Review Board of each participating site.

Inclusion criteria

• Complete bilateral carotid duplex ultrasound examination performed between 1/1/2009 and 12/31/2015. Complete carotid duplex examinations must have included, at minimum, velocity measurements at: the proximal common carotid artery (CCA), mid and/or distal CCA, proximal ICA, and distal ICA obtained per IAC Vascular Testing standards. ¹⁶

• Bilateral catheter cerebral angiography performed within 3 months of the duplex ultrasound study (duplex performed prior to angiogram). Imaging of each ICA to have included at least two angiographic views.

Exclusion criteria

• Cases derived from patients with prior carotid endarterectomy or stenting.

• Cases of known or suspected non-atherosclerotic disease, including fibromuscular dysplasia, vasculitis, radiation arteritis, or arterial dissection.

Data collection and study interpretation

De-identified images along with a worksheet of demographic and clinical information were sent to the IAC. Each set of images was assigned a unique study identification number.

Duplex ultrasound technical review

De-identified ultrasound images were uploaded into ImageShare PACS system (Vigilant Medical, Baltimore, MD, USA) and individually reviewed by a single Registered Vascular Technologist (RVT) technical reviewer (MSH). Ultrasound studies were evaluated for adherence to the inclusion/exclusion criteria, completeness (including required Doppler evaluation of specific CCA and ICA segments), appropriate formatting of images for upload and review, and overall technical quality. Velocities at each carotid segment were extracted from the duplex images and entered into the electronic database along with additional technical details. Velocities entered into the database for both the right and the left side were as follows: proximal, mid and/or distal CCA peak-systolic velocity (PSV) and end-diastolic velocity (EDV), proximal, mid (if available), and distal ICA PSV and EDV. For velocity measurements, the EDV associated with the highest PSV for a given arterial segment was entered into the database. For purposes of analysis, the ICA/CCA PSV velocity ratio for each side was defined as:

$$ICA/CCAPSVratio=\frac{\begin{array}{l}MaximumrecordedPSV\\ intheproximalormidICA\end{array}}{\begin{array}{l}RecordedPSVin\\ thedistalCCA.\end{array}}$$

Maximum recorded PSV in the proximal or mid ICA was used as the numerator to allow for capture of the highest velocity of flow associated with an atherosclerotic lesion in the origin/proximal segment of the ICA and to distinguish these stenoses from lesions of non-atherosclerotic disease, especially FMD, which generally involves the more distal ICA. PSV in the distal CCA was chosen as the denominator for the ICA/CCA PSV ratio as IAC standards require only one measurement from mid and/or distal CCA (a second being optional) and distal CCA PSV is most commonly reported if only one segment is measured. ¹⁶ Duplex ultrasound studies that met inclusion and exclusion criteria as well as technical requirements were sent for physician interpretation.

Duplex ultrasound physician interpretation

Following technical review, two physician reviewers independently reviewed and interpreted each eligible carotid duplex ultrasound study blinded to the findings of the angiogram. The group of six ultrasound reviewers were readers in accredited vascular laboratories and included cardiology/vascular medicine specialists (HLG, NMH), diagnostic radiologists (JSP, LN), a vascular surgeon (JML), and a stroke neurologist (TR). Each physician reviewer received a manual of procedures that included the table of the SRUCC, standardized definition of plaque, and examples of Doppler manifestations of turbulence and delayed/dampened Doppler waveforms. Reviewers provided an interpretation for the percentage stenosis for each ICA using the SRUCC. In addition, reviewers provided an assessment of the overall quality of the duplex examination (good/fair or compromised/inadequate), identified and characterized atherosclerotic plaque (plaque defined as wall thickness ⩾ 1.5 mm per the Mannheim consensus), and identified the presence of spectral Doppler abnormalities including poststenotic turbulence and delay and/or dampening of the distal ICA waveform. ¹⁷ Reviewers identified studies that did not meet inclusion/exclusion criteria which were not detected by the initial technical review and reported ICAs for which application of SRUCC would not be appropriate, such as tandem lesions. Reviewers also reported the primary and secondary SRUCC parameters upon which their interpretation of percentage stenosis for each ICA was based (i.e., PSV, EDV, ICA/CCA PSV ratio, plaque assessment, or other).

Each ICA side of each duplex ultrasound study was assigned a percentage stenosis category using SRUCC. Sides which were rated as compromised/inadequate by both ultrasound reviewers were excluded from the analysis, though the contralateral side was included if of adequate quality. Normal and < 50% stenosis were coded as < 50% stenosis. If both physician reviewers were in agreement, the category for percentage ICA stenosis was coded final. If the two reviewers had a discrepant interpretation, two additional blinded reviewers reviewed the case. If there was majority agreement (i.e., 3:1) for categorization for percentage stenosis, this was coded into the database as the SRUCC interpretation. If there was no majority agreement (2:2 or worse), cases were discussed by a tie breaker panel of at least three physician reviewers via web-based video teleconference with review of the ultrasound images, and the final determination of the degree of ICA stenosis was obtained by a consensus interpretation using SRUCC.

Catheter angiogram physician interpretation

A panel of physicians reviewed each uploaded catheter angiogram and measured percentage ICA stenosis using electronic calipers with the OsiriX MD DICOM Viewer (Pixmeo SARL, Bernex, Switzerland). The group of five angiogram reviewers included interventional radiologists (JFB, KSR), vascular surgeons (MPL, SAL), and a diagnostic radiologist (ND). Angiogram reviewers were blinded to all ultrasound data. Angiograms were reviewed by consensus panel with at least two and up to five reviewers during live web-based video teleconference. Each angiogram was assessed for adherence to inclusion/exclusion criteria including bilateral studies with multiple views of the ICA. Angiographic quality of the angiogram images was rated as good/fair or compromised/inadequate. Angiographic views of the right and left ICA were measured using NASCET-based methodology (online supplemental Figure 1). ¹⁸ NASCET-based methodology was used given the recommendation of the SRUCC panel and the broad adoption of this method as the current standard, including by quality organizations and governmental payers. ¹⁹ A quantified percentage stenosis was reported for each ICA representing the higher of the NASCET-based measurements of the two views. The angiogram reviewers noted exclusion criteria and/or circumstances which precluded inclusion of each ICA side in the final analysis dataset, including previously undetected exclusion criteria, technically compromised/inadequate imaging or inadequate ICA views (e.g. unilateral study), as well as the presence of tandem arterial lesions in the CCA and ICA, or stenosis in the distal ICA (possible non-atherosclerotic disease). Angiograms demonstrating tandem lesions in which the most severe area of stenosis was within the CCA rather than the ICA were excluded from the primary analysis. ICAs without a visualized normal ICA segment distal to the stenosis for adequate NASCET-based measurement were also excluded.

Statistical analysis

Descriptive statistics for patient and study characteristics and the distribution of duplex ultrasound velocity parameters were calculated. Data from right and left ICA sides were considered independent.

Distribution of category of ICA stenosis by catheter angiography and physician ultrasound interpretation using SRUCC was determined. Agreement in ultrasound interpretation using SRUCC between the first two readers, second two readers, and the tie breaker panel (when needed) was calculated. We compared the agreement in ICA stenosis categories (< 50% stenosis, 50–69% stenosis, ⩾ 70%, near-total occlusion, and total occlusion) by SRUCC ultrasound interpretation and angiography, and calculated the kappa statistic.

Receiver operating characteristic (ROC) curves were generated to compare various categories of ICA stenosis by angiography with velocity parameters of maximum ICA PSV, maximum ICA EDV, ICA/CCA PSV ratio, and selected combinations. ²⁰ Areas under the curve were calculated for the full study population and stratified by multiple parameters. ICA sides with near-total or total occlusion as confirmed by angiography were excluded from ROC analysis, though the contralateral side was allowed to remain in the analysis.

We calculated the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of SRUCC velocity parameters for diagnosis of ⩾ 50% vs < 50% and ⩾ 70% vs < 70% ICA stenosis compared with catheter angiography. Cases of near-total/total occlusion were excluded. We reviewed ultrasound parameter data across the range of PSV, EDV, and ICA/CCA PSV values in relation to angiography cut-points to identify velocity thresholds with improved performance compared with the SRUCC parameters. PSV was selected as the primary parameter for optimization of diagnostic performance. Prespecified minimal requirements for selection of an optimized ultrasound parameter were sensitivity > 90%, specificity > 80%, and accuracy > 80%. After identifying PSV thresholds with improved diagnostic performance compared with SRUCC, combinations of these PSV thresholds with EDV and ICA/CCA PSV ratio parameters were evaluated. All statistical analyses were performed by SAS software, Version 9.4 (SAS Institute Inc., Cary, NC, USA).

Results

A total of 224 cases (448 sides) were submitted for initial technical review. Of these, 299 ICA sides (66.7%) from 167 patients met inclusion/exclusion and imaging review criteria and were included in the final analysis dataset. Information regarding the primary reason for exclusion are shown in Figure 1; some sides may have been marked for exclusion by both ultrasound and angiogram review. If a case study met inclusion/exclusion criteria but one side (right/left) ultrasound or angiogram was excluded due to inadequate/compromised image quality or other angiographic criteria (e.g. missing two imaging views per ICA, tandem lesions/CCA stenosis, inadequate distal ICA for measurement), the eligible contralateral side ultrasound and angiogram (if of adequate quality) was retained in the analysis dataset.

Figure 1.

Case study flow diagram.

CEA, carotid endarterectomy; IAC, Intersocietal Accreditation Commission; ICA, internal carotid artery.

Characteristics of the 167 patients in the final study population are shown in Table 1. Mean patient age was 69.9 ± 10.3 years (range 37–91 years), and 60% of patients were male. There was a high prevalence of atherosclerotic risk factors. Approximately 50% of patients had a history of hemispheric neurological symptoms, and this was the most common indication for the carotid duplex study (46%) followed by surveillance of known carotid artery stenosis (31%).

Table 1.

Characteristics of the study population (n = 167 patients).

Parameter	n (%)
Male	100 (60)
Risk factors a
Hypertension	146 (87)
Diabetes	51 (31)
Hyperlipidemia	131 (78)
History of tobacco use	101 (60)
Hemispheric symptoms b	83 (50)
Ultrasound study indication c
Hemispheric symptoms	77 (46)
Known carotid stenosis	51 (31)
Cervical bruit	22 (13)
Preoperative exam	22 (13)
Atherosclerosis elsewhere	20 (12)
Other	10 (6)

^aHypertension and hyperlipidemia were defined as risk factors requiring pharmacological therapy. Diabetes was defined by requirement for pharmacological therapy or a special diet. History of tobacco smoking was defined as smoking > 100 cigarettes in the lifetime.

^bDefined as any history of transient ischemic attack, stroke, or retinal ischemia up to the date of the ultrasound examination.

^cMore than one study indication possible.

Summary of duplex ultrasound velocity parameters are shown in online supplemental Table 2. Mean (± SD) of the maximum proximal/mid ICA PSV was 193.6 ± 137.0 cm/sec and the mean ICA/CCA PSV ratio was 2.6 ± 2.2. The distribution of interpretation of ICA stenosis by physician ultrasound reviewers using SRUCC and by angiography is shown in online supplemental Table 3. There were fewer ICA lesions meeting criteria for 50–69% and ⩾ 70% stenosis by angiography compared with ultrasound using SRUCC (50–69%: 18.7% vs 29.1%; ⩾ 70%: 11.0% vs 24.4%). Among 33 ICA sides reported with ⩾ 70% stenosis by angiography (excluding nine near-total/total occlusions), only 13 had ⩾ 80% stenosis. Agreement between two physician ultrasound reviewers for category of ICA stenosis was 90.3% (270/299 sides), 6.7% (20/299) achieved consensus with two additional reviewers, and 3.0% (9/299) required consensus panel discussion. As shown in Table 2, there was only moderate agreement of categorization of ICA stenosis by ultrasound interpretation using SRUCC compared with angiography (kappa = 0.42). For ICA lesions of < 50% stenosis by angiography, there was agreement by duplex in 64.2% of ICAs with overestimation of percentage ICA stenosis for 35.9%. For ICA lesions of 50–69% by angiography, there was agreement by duplex in 42.9% and disagreement with upgrading of severity of stenosis to ⩾ 70% stenosis in 53.6% of ICA. Conversely, among ICA lesions interpreted as 50–69% stenosis by SRUCC, 69.0% actually had < 50% stenosis by angiography. There was excellent agreement between duplex ultrasound and angiography for ICA lesions of ⩾ 70% confirmed by angiography (90.9%) but among lesions interpreted as ⩾ 70% by duplex, 41.1% had 50–69% stenosis and 16.4% had < 50% stenosis by angiography.

Table 2.

Agreement of percentage ICA stenosis by catheter angiography (NASCET) versus ultrasound interpretation (SRUCC) (n = 299).

Duplex	Angiography
	< 50%	50–69%	⩾ 70%	Near occlusion	Total occlusion
< 50% a	129 99.2% b 64.2% c	1 0.8 b 1.8% c	0	0	0	130
50–69%	60 69.0% b 29.9% c	24 27.6% b 42.9% c	2 2.3% b 6.1% c	1 1.2% b 33.3% c	0	87
⩾ 70%	12 16.4% b 6.0% c	30 41.1% b 53.6% c	30 41.1% b 90.9% c	0	1 1.4% b 16.7% c	73
Near occlusion	0	1 25.0% b 1.8% c	1 25.0% b 3.0% c	2 50.0% b 66.7% c	0	4
Total occlusion	0	0	0	0	5 100% b 83.3% c	5
	201	56	33	3	6	n = 299

Kappa = 0.42.

Cells in gray demonstrate agreement between angiography and duplex ultrasound for category of percentage ICA stenosis.

^aNormal and < 50% stenosis by SRUCC coded as < 50% stenosis.

^b1st percentage listed in each cell is for the row category of percentage ICA stenosis by duplex.

^c2nd percentage shown in each cell is for the column category of percentage ICA stenosis by angiography.

ICA, internal carotid artery; NASCET, North American Symptomatic Carotid Endarterectomy Trial; SRUCC, Society of Radiologists in Ultrasound Consensus Criteria.

ROC analysis

ROC analysis was performed for individual ultrasound parameters for classification of ICA stenosis as determined by angiography and AUC parameters (Table 3). AUC was high for all velocity parameters for classification of ICA stenosis < 50% vs ⩾ 50%, < 50% vs 50–69%, < 70% vs ⩾ 70%, and < 80% vs ⩾ 80% (all ⩾ 0.89). AUC was moderate (0.63–0.76) for all ultrasound parameters for classification of 50–69% vs ⩾ 70% and 50–69% vs 70–79% stenosis. AUC was poor for classification of 70–79% vs ⩾ 80% stenosis (AUC 0.50–0.59), likely related to the small number of ICA with ⩾ 80% stenosis by angiography. For classification of < 50% vs ⩾ 50% ICA stenosis, addition of the ICA/CCA PSV ratio to the model resulted in minimal improvement in AUC. For classification of < 70% vs ⩾ 70% ICA stenosis, addition of the ICA/CCA PSV radio, EDV, and the ICA/CCA PSV ratio plus EDV to the model had little impact on the AUC beyond PSV alone. There was no impact of presence of contralateral side near-total or total occlusion (n = 6) on AUC for classification of ⩾ 50% or ⩾ 70% ICA stenosis by angiography (online supplemental Table 4). Similarly, analysis stratified by patient sex, left and right ICA separately versus together, and hemispheric symptoms, as the study indication showed little effect on the AUC of ultrasound parameters for classification of ⩾ 50% or ⩾ 70% ICA stenosis by angiography aside from slightly higher AUC for ⩾ 50% vs < 50% and ⩾ 70% vs < 70% ICA stenosis for right versus left ICA sides as well as slightly higher AUC for ⩾ 70% vs < 70% ICA stenosis for female versus male patients and for hemispheric neurological symptoms (online supplemental Table 4).

Table 3.

Area under the curve for ROC analysis of duplex ultrasound velocity parameters for prediction of percentage ICA stenosis by catheter angiography (NASCET). ^a

	Angio < 50% vs ⩾ 50%	Angio < 50% vs 50–69%	Angio < 70% vs ⩾ 70%	Angio 50–69% vs 70–79%	Angio 50–69% vs ⩾ 70%	Angio < 80% vs ⩾ 80%	Angio 70–79% vs ⩾ 80%
Max ICA PSV	0.94	0.93	0.91	0.74	0.74	0.89	0.50
Max ICA EDV	0.93	0.91	0.91	0.71	0.74	0.92	0.55
ICA/CCA PSV ratio b	0.96	0.95	0.90	0.63	0.66	0.89	0.57
PSV + ratio	0.96	0.95	0.91	0.75	0.75	0.90	0.59
PSV + EDV	0.94	0.93	0.91	0.76	0.76	0.90	0.50
PSV + EDV + ratio	0.96	0.96	0.91	0.76	0.76	0.90	0.59

^an = 9 sides with near-total/total ICA occlusion excluded.

^bRatio = ICA/CCA PSV ratio.

CCA, common carotid artery; EDV, end-diastolic velocity; ICA, internal carotid artery; NASCET, North American Symptomatic Carotid Endarterectomy Trial; PSV, peak-systolic velocity; ROC, receiver operating characteristic.

The findings of ROC analysis for specific velocity parameters for diagnosis of ⩾ 50% and ⩾ 70% ICA stenosis are shown in Tables 4 and 5. Parameters which met prespecified criteria for sensitivity (> 90%), specificity (> 80%), and accuracy (> 80%) are highlighted in gray. For predicting ⩾ 50% vs < 50% ICA stenosis (Table 4), the SRUCC PSV threshold of ⩾ 125 cm/sec had excellent sensitivity (97.8%) but inadequate specificity (64.2%) and accuracy (74.5%). The sensitivity, specificity, and accuracy of the ICA/CCA PSV ratio of ⩾ 2 alone (sensitivity 95.4%, specificity 84.2%, accuracy 87.6%) or in combination with PSV ⩾ 125 cm/sec (sensitivity 94.3%, specificity 84.3%, accuracy 87.4%) exceeded the performance of PSV ⩾ 125 cm/sec alone as a single velocity parameter. For the PSV parameter alone, raising the PSV threshold above 125 cm/sec improved performance for predicting ⩾ 50% stenosis with PSV ⩾ 180 cm/sec associated with sensitivity 93.3%, specificity 81.6%, and accuracy of 85.2%. Various combinations of PSV and ICA/CCA PSV ratio thresholds had improved performance for prediction of ⩾ 50% ICA stenosis compared with that of the PSV ⩾ 125 cm/sec threshold alone, including requiring the addition of ICA/CCA PSV ratio ⩾ 2.

Table 4.

ROC analysis – sensitivity, specificity, PPV, NPV, and accuracy of duplex ultrasound velocity parameters at specific values for prediction of ⩾ 50% versus < 50% ICA stenosis by catheter angiography (NASCET). ^a

Velocity parameter threshold	Sensitivity	Specificity	PPV	NPV	Accuracy
From SRUCC
PSV ⩾ 125 cm/sec	0.978	0.642	0.547	0.985	0.745
PSV ratio b ⩾ 2	0.954	0.842	0.728	0.976	0.876
PSV ⩾ 125 cm/sec + ratio ⩾ 2	0.943	0.843	0.726	0.971	0.874
Modified parameters
PSV ⩾ 125 cm/sec	0.978	0.642	0.547	0.985	0.745
PSV ⩾ 140 cm/sec	0.966	0.702	0.589	0.979	0.783
PSV ⩾ 160 cm/sec	0.955	0.771	0.649	0.975	0.828
PSV ⩾ 170 cm/sec	0.944	0.791	0.667	0.970	0.838
PSV ⩾ 180 cm/sec	0.933	0.816	0.692	0.965	0.852
PSV ⩾ 190 cm/sec	0.899	0.836	0.708	0.949	0.855
PSV ⩾ 125 cm/sec + ratio ⩾ 2	0.943	0.843	0.726	0.971	0.874
PSV ⩾ 140 cm/sec + ratio ⩾ 2	0.931	0.855	0.736	0.966	0.878
PSV ⩾ 160 cm/sec + ratio ⩾ 2	0.920	0.860	0.741	0.961	0.878
PSV ⩾ 170 cm/sec + ratio ⩾ 2	0.908	0.865	0.745	0.956	0.878
PSV ⩾ 180 cm/sec + ratio ⩾ 2	0.897	0.870	0.750	0.951	0.878

Shaded parameters met prespecified requirements for > 90% sensitivity, > 80% specificity, and > 80% accuracy.

^an = 9 sides with near-total/total ICA occlusion excluded.

^bRatio = ICA/CCA PSV ratio.

CCA, common carotid artery; ICA, internal carotid artery; NASCET, North American Symptomatic Carotid Endarterectomy Trial; NPV, negative predictive value; PPV, positive predictive value; PSV, peak-systolic velocity; ROC, receiver operating characteristic; SRUCC, Society of Radiologists in Ultrasound Consensus Criteria.

Table 5.

ROC analysis – sensitivity, specificity, PPV, NPV, and accuracy of duplex ultrasound velocity parameters at specific values for prediction of > 70% versus < 70% ICA stenosis by catheter angiography (NASCET). ^a

Velocity parameter	Sensitivity	Specificity	PPV	NPV	Accuracy
From SRUCC
PSV ⩾ 230 cm/sec	0.939	0.782	0.356	0.990	0.800
EDV ⩾ 100 cm/sec	0.758	0.906	0.510	0.967	0.889
PSV ratio b ⩾ 4.0	0.818	0.864	0.443	0.973	0.859
PSV ⩾ 230 cm/sec + EDV ⩾ 100 cm/sec	0.758	0.907	0.510	0.967	0.890
PSV ⩾ 230 cm/sec + ratio ⩾ 4	0.818	0.875	0.458	0.974	0.868
PSV ⩾ 230 cm/sec + ratio ⩾ 4 + EDV ⩾ 100 cm/sec	0.636	0.930	0.539	0.952	0.896
Modified parameters
PSV ⩾ 230 cm/sec	0.939	0.782	0.356	0.990	0.800
PSV ⩾ 240 cm/sec	0.939	0.794	0.369	0.990	0.810
PSV ⩾ 250 cm/sec	0.909	0.802	0.370	0.986	0.814
PSV ⩾ 260 cm/sec	0.909	0.813	0.385	0.986	0.824
PSV ⩾ 270 cm/sec	0.849	0.833	0.394	0.977	0.835
PSV ratio ⩾ 3.3	0.939	0.836	0.431	0.991	0.848
PSV ratio ⩾ 4.0	0.818	0.864	0.443	0.973	0.859
EDV ⩾ 70 cm/sec	0.909	0.816	0.390	0.986	0.827
EDV ⩾ 90 cm/sec	0.788	0.891	0.482	0.970	0.879
EDV ⩾ 100 cm/sec	0.758	0.906	0.510	0.967	0.889
PSV ⩾ 230 cm/sec + ratio ⩾ 3.3	0.939	0.847	0.443	0.991	0.858
PSV ⩾ 250 cm/sec + EDV ⩾ 70 cm/sec	0.849	0.860	0.438	0.978	0.859
PSV ⩾ 250 cm/sec + EDV ⩾ 90 cm/sec	0.788	0.903	0.510	0.971	0.890
PSV ⩾ 250 cm/sec + EDV ⩾ 100 cm/sec	0.758	0.911	0.521	0.967	0.893
PSV ⩾ 250 cm/sec + ratio ⩾ 3.3	0.909	0.855	0.448	0.986	0.861
PSV ⩾ 250 cm/sec + ratio ⩾ 4	0.788	0.878	0.456	0.970	0.868
PSV ⩾ 250 cm/sec + ratio ⩾ 3.3 + EDV ⩾ 70 cm/sec	0.849	0.883	0.483	0.978	0.879
PSV ⩾ 250 cm/sec + ratio ⩾ 3.3 + EDV ⩾ 90 cm/sec	0.788	0.910	0.531	0.971	0.896
PSV ⩾ 250 cm/sec + ratio ⩾ 3.3 + EDV ⩾ 100 cm/sec	0.758	0.918	0.544	0.967	0.900
PSV ⩾ 250 cm/sec + ratio ⩾ 4 + EDV ⩾ 70 cm/sec	0.727	0.902	0.490	0.963	0.882
PSV ⩾ 250 cm/sec + ratio ⩾ 4 + EDV ⩾ 90 cm/sec	0.667	0.926	0.537	0.956	0.896
PSV ⩾ 250 cm/sec + ratio ⩾ 4 + EDV ⩾ 100 cm/sec	0.636	0.934	0.553	0.952	0.900
PSV ⩾ 260 cm/sec + EDV ⩾ 70 cm/sec	0.849	0.860	0.438	0.978	0.859
PSV ⩾ 260 cm/sec + EDV ⩾ 90 cm/sec	0.788	0.903	0.510	0.971	0.890
PSV ⩾ 260 cm/sec + EDV ⩾ 100 cm/sec	0.758	0.911	0.521	0.967	0.893
PSV ⩾ 260 cm/sec + ratio ⩾ 3.3	0.909	0.859	0.455	0.987	0.865
PSV ⩾ 260 cm/sec + ratio ⩾ 4	0.788	0.883	0.464	0.970	0.872
PSV ⩾ 260 cm/sec + ratio ⩾ 3.3 + EDV ⩾ 70 cm/sec	0.849	0.883	0.483	0.978	0.879
PSV ⩾ 260 cm/sec + ratio ⩾ 3.3 + EDV ⩾ 90 cm/sec	0.788	0.910	0.531	0.971	0.896
PSV ⩾ 260 cm/sec + ratio ⩾ 3.3 + EDV ⩾ 100 cm/sec	0.758	0.918	0.544	0.967	0.900
PSV ⩾ 260 cm/sec + ratio ⩾ 4 + EDV ⩾ 70 cm/sec	0.727	0.902	0.490	0.963	0.882
PSV ⩾ 260 cm/sec + ratio ⩾ 4 + EDV ⩾ 90 cm/sec	0.667	0.926	0.537	0.956	0.896
PSV ⩾ 260 cm/sec + ratio ⩾ 4 + EDV ⩾ 100 cm/sec	0.636	0.934	0.553	0.952	0.900

Shaded parameters met prespecified requirements for > 90% sensitivity, > 80% specificity, and > 80% accuracy.

^an = 9 sides with near-total/total ICA occlusion excluded.

^bRatio = ICA/CCA PSV ratio.

CCA, common carotid artery; EDV, end-diastolic velocity; ICA, internal carotid artery; NASCET, North American Symptomatic Carotid Endarterectomy Trial; NPV, negative predictive value; PPV, positive predictive value; PSV, peak-systolic velocity; ROC, receiver operating characteristic; SRUCC, Society of Radiologists in Ultrasound Consensus Criteria.

For predicting ⩾ 70% vs < 70% ICA stenosis (Table 5), the SRUCC PSV threshold of ⩾ 230 cm/sec had adequate sensitivity (93.9%) but inadequate specificity (78.2%) with overall accuracy of 80.0%. For the PSV parameter alone, raising the threshold above 230 cm/sec was associated with improved specificity and overall accuracy, with the best performance at PSV ⩾ 250 or ⩾ 260 cm/sec. The ICA/CCA PSV ratio of ⩾ 4.0 had 81.8% sensitivity, 86.4% specificity, and 85.9% accuracy, while a threshold of ⩾ 3.3 had higher sensitivity (93.9%), as well as adequate specificity (83.6%) and overall accuracy (84.8%). For the EDV parameter, EDV ⩾ 100 cm/sec had inadequate sensitivity (75.8%) but adequate specificity (90.6%) and accuracy (88.9%), and lowering the EDV threshold to ⩾ 70 cm/sec was associated with adequate sensitivity (90.9%), specificity (81.6%), and accuracy (82.7%). Combining SRUCC parameters resulted in improved specificity and accuracy beyond PSV ⩾ 230 cm/sec alone. Combinations of PSV ⩾ 230 cm/sec, ⩾ 250 cm/sec, and 260 cm/sec with ICA/CCA ratio ⩾ 3.3 met criteria for adequate sensitivity, specificity, and accuracy. All parameters and thresholds were observed to have low PPV, partially attributable to the low number of ICA sides with ⩾ 70% stenosis by angiography; thus our findings must be interpreted with caution.

Discussion

In a balanced study population of symptomatic and asymptomatic patients, we report that the SRUCC produced significant overestimation of stenosis for both moderate (50–69%) and severe (⩾ 70%) ICA lesions as determined by catheter angiography using NASCET-based methodology. The primary SRUCC parameter for ⩾ 50% stenosis of PSV ⩾ 125 cm/sec did not meet prespecified requirements for sensitivity, specificity, and accuracy. We report improvement with increasing the PSV velocity threshold, with the best performing PSV threshold of ⩾ 180 cm/sec for ⩾ 50% stenosis or with PSV thresholds ranging from 125 to 170 cm/sec when the ICA/CCA PSV ratio was also ⩾ 2. The performance of the existing SRUCC PSV threshold of 125 cm/sec was significantly improved by adding the requirement of ICA/PSV ratio ⩾ 2. For ⩾ 70% stenosis, the primary SRUCC parameters of PSV ⩾ 230 cm/sec, EDV ⩾ 100 cm/sec, and PSV ratio ⩾ 4 also did not meet prespecified requirements for sensitivity, specificity, and accuracy. There were other parameter thresholds that met these requirements, including higher PSV (250 or 260 cm/sec) and a lower ICA/CCA PSV ratio threshold of 3.3. The combination of PSV ⩾ 230 cm/sec or higher (250 or 260 cm/sec) and PSV ⩾ 3.3 ratio also met these requirements. The analysis was limited by a low number of cases with angiographically confirmed severe ICA stenosis which affected PPV.

Our study findings that the SRUCC PSV threshold for ⩾ 50% ICA stenosis of 125 cm/sec is not adequate to distinguish ICA lesions below and above 50% stenosis is consistent with the findings of a single-center study by AbuRahma and colleagues, which used computed tomography angiography (CTA) as the diagnostic gold standard and proposed a PSV threshold of > 137 cm/sec as an improved threshold for diagnosis of ⩾ 50% ICA stenosis. ²¹ Our findings are also similar to a pooled analysis by Beach and colleagues which used digitized scattergrams of nearly 3000 data points from 19 previously published ultrasound–angiogram correlation studies and identified a PSV of 165 cm/sec as a potential diagnostic threshold for ⩾ 50% ICA stenosis of the native ICA. ²² Our study identified PSV ⩾ 180 cm/sec to be an optimal PSV threshold for ⩾ 50% stenosis, which is higher than previously suggested, likely attributable to the use of catheter angiography in our study compared with CTA in AbuRahma, et al. ²¹ Felbaum and colleagues recently reported that CTA significantly overestimated the degree of ICA stenosis compared with NASCET-based catheter angiography among patients with moderate ICA stenosis. ²³

Data reporting the PSV threshold of ⩾ 125 cm/sec is too low for diagnosis of ⩾ 50% ICA stenosis by NASCET methodology is not surprising. The original PSV threshold of ⩾ 125 cm/sec was based largely on the work of Dr Eugene Strandness and colleagues at the University of Washington which used the original methodology for measurement of percentage ICA stenosis by angiography with the reference diameter being an estimate of lumen diameter at the site of stenosis (‘ECST method’; online supplemental Figure 1) rather than lumen diameter at a distal normal segment of the ICA (‘NASCET method’).^2,9,24 It has been demonstrated that the ECST method produces a higher angiographic percentage stenosis for the same degree of luminal narrowing compared with the NASCET method. ²⁵ As detailed above, the NASCET method has now been recommended as the national standard for angiographic interpretation.^6,19 Defining an accurate threshold for diagnosis of ⩾ 50% ICA stenosis is important as this is the threshold at which carotid revascularization for a patient with hemispheric neurological symptoms may be recommended. ¹

With only 33 ICA lesions of ⩾ 70% stenosis by angiography in our study, our ability to derive meaningful conclusions regarding the performance of the SRUCC for diagnosis of severe ICA stenosis was limited. Data from previous publications enriched for patients with severe ICA stenosis were used to propose the PSV > 230 cm/sec, ICA/CCA PSV ratio > 4, and EDV > 100 components of the SRUCC for diagnosis of ⩾ 70% stenosis.^4,14,15 Given the limitations of our dataset, we were unable to further validate these parameters or confidently propose modification. These parameters were originally derived from studies that incorporated NASCET methodology for angiogram interpretation, and thus may not be as subject to overestimation of severity of ICA stenosis associated with the ECST methodology. In the CTA-based study of AbuRahma and colleagues, no modifications to existing SRUCC parameters for ⩾ 70% stenosis were proposed, though it was noted that raising the PSV threshold to > 252 cm/sec did improve diagnostic performance. ²¹ Beach and colleagues suggested raising the diagnostic threshold for ⩾ 70% ICA stenosis to 280 cm/sec. ²² Our data demonstrated the PSV of 230 was close to achieving our prespecified criteria for diagnostic performance, but also supports these later studies since higher PSV thresholds (250 or 260 cm/sec) and PSV thresholds combined with a lower ICA/CCA PSV ratio of ⩾ 3.3 also met prespecified criteria.

Our study raises concerns regarding the performance of the SRUCC for assessment of ICA stenosis. In recent years, the SRUCC have gained traction among vascular laboratories despite no definitive validation study. In 2010, the SRUCC were reportedly used in approximately one-quarter of IAC accredited facilities and up to half of laboratories accredited since 2005. ²⁶ In a more recent analysis of the variability of carotid ultrasound criteria and diagnostic thresholds for ⩾ 50% and ⩾ 70% stenosis from 338 accredited vascular laboratories participating in the Vascular Quality Initiative, the median PSV for ⩾ 50% stenosis was 125 cm/sec and the median threshold for ⩾ 70% stenosis was 230 cm/sec, suggestive of further uptake of the SRUCC in recent years. ¹²

Study limitations

This was a real-world correlative imaging study with case study materials taken from clinical practice and collected retrospectively, rather than from a prospective research study in which images were obtained in a standardized fashion using a prespecified study protocol. A significant number of ICA sides were excluded from the analysis based upon inadequate angiographic images (e.g. missing two angiographic projections of the ICA) or poor angiographic image quality. Fewer cases were excluded from the analysis due to inadequate ultrasound images, which may reflect more consistent protocols from accredited ultrasound laboratories. High-quality catheter angiography was chosen as the gold standard for comparison to duplex given the history of this modality as the correlative gold standard as well as less variability in equipment and post-processing from facility to facility compared with CTA or magnetic resonance imaging (MRA). While the choice of catheter angiography was a potential strength, it also made collection of case studies challenging because in 21st century practice, complete diagnostic catheter angiograms, and especially those with multiple views of the ICA, are rarely obtained for diagnostic purposes and have been largely supplanted by noninvasive angiographic modalities (CTA, MRA). Catheter angiography may be performed on a more limited fashion for procedural guidance (e.g. carotid stenting). This may have also introduced potential for selection bias as only a minority of carotid duplex studies at a given center had a corresponding correlative catheter angiogram. In addition to the above, the small number of cases with severe ICA stenosis on catheter angiography was the major limitation of our study. As a result, this study had limited sample size for ROC analysis of velocity parameters for determining ⩾ 70% stenosis, resulting in a low PPV for all parameters at all thresholds reported, and any findings with regard to this category must be interpreted and/or applied with caution. Taken together, these changes in practice indicate the need to determine whether catheter angiography can reliably continue as the ‘gold standard’ for future correlative imaging studies.

Finally, our analysis focused on validation of velocity and did not address other duplex ultrasound features that may be incorporated into diagnostic criteria, such as presence, morphological appearance, and severity of visualized plaque on grayscale imaging, presence of poststenotic turbulence on color and/or spectral Doppler analysis, or morphology of Doppler waveforms in the distal ICA beyond areas of suspected stenosis. These adjunctive ultrasound features are not broadly available with other noninvasive imaging modalities (CTA, MRA) and warrant further investigation.

Conclusions

In a real-world correlative imaging study performed with case studies obtained from IAC accredited vascular laboratories, interpretation of carotid duplex ultrasound using SRUCC resulted in significant overestimation of degree of stenosis for both moderate (50–69%) and severe (⩾ 70%) lesions. Owing to the low number of ICA stenoses ⩾ 70% in the final dataset, no definitive conclusions can be made regarding modification of existing SRUCC parameters to further optimize their diagnostic performance for severe ICA lesions. Laboratories currently using SRUCC should consider modification of existing criteria to incorporate more stringent and more accurate parameters for ⩾ 50% ICA stenosis by increasing the PSV threshold to ⩾ 180 cm/sec or requiring the ICA/CCA PSV ratio ⩾ 2.0 in addition to PSV of ⩾ 125 cm/sec.

Appendix: Participating sites and site investigators

• Cleveland Clinic, Cleveland, Ohio (Heather L Gornik, Alia Grattan)

• Novant Health Heart and Vascular Institute, Charlotte, North Carolina (Kelly Hicks)

• Riverside Radiology, Columbus, Ohio (Lucy LaPerna)

• TriHealth, Cincinnati, Ohio (Joann M Lohr)

• University at Buffalo, Buffalo, New York (Adnan Siddiqui)

• University Hospitals and Clinics, Lafayette, Louisiana (Michel Comeaux)

• University of Maryland, Baltimore, Maryland (Michael P Lilly)

• University of Miami, Miami, Florida (Tatjana Rundek)

• University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania (Steven A Leers)

• University of Southern California, Los Angela, California (Susana Robison)

• University of Washington, Seattle, Washington (R Eugene Zierler)