Levels of agreement in the measurements of carotid artery ultrasound across a regional vascular network

Abstract

Introduction

Duplex ultrasound is often the sole imaging modality used in diagnosing carotid artery disease. However, the reproducibility and repeatability of scientists in determining the peak systolic velocity and end diastolic velocity of the internal carotid artery and common carotid artery (CCA) is widely debated.

Study aim

To investigate intra- and inter-operator variability in diagnostic ultrasound of the carotid arteries across a centralised vascular network using a healthy test subject. To identify potential causes of variability and highlight areas for improvement.

Methods

Fifteen vascular scientists across four hospital Trusts in the Bristol, Bath and Weston vascular network measured the peak systolic velocity and end diastolic velocity of the internal carotid artery and common carotid artery in a subject using a single portable ultrasound machine. A double blind assessment of spectral Doppler images was performed by two vascular clinical scientists for optimal caliper placement, spectral gain and angle correction. Results were compared for intra- and inter-operator variability.

Results

Initial quality assessment of the Doppler images revealed that three out of 15 scientists produced suboptimal results. Box plot analysis of the common carotid artery and internal carotid artery for each scientist revealed significant variance (ANOVA p < 0.05). However, a Levene’s test revealed no single operator who consistently produced highly variable results (p = 0.569).

Conclusion

This study highlights the difficulty in obtaining consistent velocity measurements from a subject. Despite the variability in absolute peak systolic velocity and end diastolic velocity, scientists were generally consistent in obtaining an optimal spectral Doppler trace. Some issues with consistency were, however, identified which were subsequently addressed.

Introduction

Quality assurance (QA) testing on ultrasound (US) machines may be performed annually, however, QA testing of clinical staff is rarely routinely performed after completion of training and there are a number of factors which affect carotid duplex assessment.¹ Many US machine manufacturers provide detailed specifications regarding QA tolerance limits and QA criteria, but such criteria are rarely regularly applied to intra- and inter-operator variability.

Carotid artery US

Of particular interest is the grading of carotid artery disease. Stenosis of the carotid artery is known to be an important cause of ischemic stroke. Duplex ultrasound (DUS) is often the sole imaging modality used in diagnosing carotid artery disease and planning surgical intervention.² Diagnostic US for carotid artery disease is undertaken in a variety of hospital settings in the United Kingdom, including radiology/vascular departments, medical physics units, and specialist stroke units. Perkins³ highlights the many sources of variability in UK practice in the diagnosis of carotid artery disease. This study found across 86 NHS Trusts, a median of 450 carotid diagnostic US scans were performed annually. Of these 48% were performed in a vascular department and 34% in the radiology department. Eight different professional disciplines in total performed carotid US all of whom may have undertaken different training pathways. Imaging was most commonly performed by radiologists (34%), vascular technologists/scientists (31%) and medical physicists (28%). It was observed that just 11% of departments always used both angiography and DUS to confirm a stenosis whilst 70% of units only used angiography if DUS was inconclusive (digital subtraction angiography in 60 units and magnetic resonance angiography in 16 units). Only 51% of units had validated the accuracy of DUS with angiography and the study demonstrated velocity and frequency shift criteria showed significant differences across different US machines.³

A defined protocol for using DUS to determine carotid stenosis is based on the North American Symptomatic Carotid Endarterectomy Trial.⁴ This method of grading carotid stenosis has been recommended to all vascular labs in the UK.⁵ This protocol recommends describing the nature/length of a plaque, measuring the peak systolic velocity (PSV) and end diastolic velocity (EDV) in both internal carotid arteries (ICAs) and distal common carotid arteries (CCAs) and measuring all velocities at a Doppler angle of between 45° and 60°.⁵

The reproducibility and repeatability of scientists to determine the PSV of the ICA and CCA has been debated for several years.⁶ There are a number of settings which a scientist can select which can affect these velocity measurements. These include: sample gate position, spectral gain, angle correction and Doppler insonation angle. Although image settings should be optimised in order to obtain the most accurate velocities, each scientist’s scanning technique can vary. This was illustrated by a study which determined the inter-operator variability in grading a 70% to 99% carotid stenosis across two hospital Trusts. A receiver operator curve showed a significant difference (p < 0.05) between hospital Trusts when grading the carotid stenosis using the same DUS model type.⁷ The scientist’s ability to obtain a consistent Doppler and insonation angle is highly important. There are two schools of thought regarding best practice:⁸

1. A fixed angle of 60° is used to ensure only a moderate constant error in velocity estimation occurs.

2. The smallest angle (45° to 60°) is used to reduce error in alignment of the angle correction cursor, thus reducing the error in velocity determination and intrinsic spectral broadening (ISB).

The inconsistent application of these two approaches in determining the same PSV measurement using a flow phantom across several vascular departments has been shown to produce significantly different results.⁹ This illustrates the need for scientists to have a consistent method of determining carotid velocities. In addition, studies have shown stenosis grading variability between scientists can be reduced by regular training.^10,11

Centralisation of services

In recent years, NHS strategy has seen the consolidation of acute services into fewer specialist centres. Many commissioners, clinicians, social services and patients believe, if done well, this process can improve patient outcomes and reduce the demand on acute hospitals. With the financial challenge of 24/7 service provision, which is increasingly accepted as a necessity, the centralisation of services could be considered a must. A 2014 review of data collected during the period from January 2008 to March 2012 examined whether the centralisation of acute stroke services in London and Manchester in 2010 was associated with changes in mortality and length of hospital stay. The study found a significant decline in risk adjusted mortality at 3, 30 and 90 days after admission in London, indicating 168 fewer deaths (95% confidence interval 19 to 316) as well as a significant decline in risk adjusted length of hospital stay in both London and Manchester.¹²

Since 2014 several UK NHS Trusts have centralised vascular surgery with a central ‘hub’ hospital providing all surgical interventions whilst the other ‘spoke’ hospitals continue to provide a diagnostic service. Such a network was formed in October 2014 with Trusts in the Bristol, Bath and Weston-super-Mare region in which North Bristol NHS Trust (NBT) became the surgical hub and the other Trusts (University Hospitals Bristol (UHB), Royal United Hospitals Bath (RUH) and Weston Area Health Trust (WAHT)) became the spoke hospitals. This process of centralisation prompted a review of our current practices in diagnostic vascular US and the creation of working groups, consisting of representatives from each Trust, to review all aspects of the diagnostic US pathway including scanning protocols, disease grading and reporting. This led to the creation of a set of standard operating procedures to be used by all scientists undertaking diagnostic vascular US in the network. With a continued desire to improve our services post centralisation, it was considered important to evaluate variation in services across the network. This study aimed to investigate intra- and inter-operator variability in diagnostic US of the carotid arteries across the four hospital Trusts within the Bristol, Bath and Weston Area Vascular Network, to identify the potential causes of variability, examine its effect on grading a carotid stenosis and highlight areas for improvement.

Methods

Sample population and setting

Fifteen vascular scientists across the four NHS Trusts (UHB, RUH, NBT and WAHT) volunteered for the study. All participating scientists have achieved both clinical and academic competency in carotid US and met the standards described in both the National School of Healthcare Science Scientist Training Program (STP) and the Society for Vascular Technology (SVT) accreditation. All scientists routinely perform carotid US as part of their daily clinical practice in vascular US and maintain continued professional development. Across the network, lengths of individual clinical practice varied between 2 and 20 years.

The assessment was undertaken using a 9 MHz linear probe on a 2013 portable LOGIQ e ultrasound machine (GE Medical Systems) which had undergone weekly and monthly QA testing as per departmental protocol at UHB. The variety of US machine makes and models used within the network necessitated the use of a portable US machine which could be relocated to each site to protect the internal validity of the study.

A healthy male test subject aged 42 years was used for the assessment of the carotid arteries. The subject refrained from cardiac stimuli for 24 hours and a set criteria of blood pressure (120/80 ± 10%) and pulse (77 ± 10%) was established before each carotid US assessment. The subject was positioned supine on an examination couch in a light adjusted and air conditioned room (Figure 1).

Figure 1.

Subject’s carotid artery being scanned by a scientist using the portable GE LOGIQ e demonstrating the experimental setup.

Blood pressure and pulse rate were measured using an Intelli™sense automated monitor (The Boots Company PLC) to ensure these fell within the agreed limits. If the blood pressure and pulse rate fell outside of the allowed range, the test was rescheduled until the desired range was achieved.

Carotid duplex assessment

Each scientist was required to measure the PSV and EDV in the CCA and ICA. Measurements were taken at 2 cm before the carotid bifurcation in the CCA and 2 cm distal to the origin of the ICA.⁵ Seven sets of the measurements described above were obtained in succession from each of the right and left sides of the test subject to ensure the minimum statistical normality was reached. A carotid preset with a pulsed wave central frequency of 5 MHz and a sample volume of 2 mm was set for each scientist. Scientists were allowed to choose either an angle correction of 45–60° to establish the most correct angle of insonnation to the vessel or ‘heel/toe’ technique and adjust for spectral gain.⁵ These details were recorded for each scientist.

Although the auto-trace function was available on the Doppler spectrum, all scientists were asked to record velocity values using manual placement of the measurement calipers to obtain the most accurate results. The scientists were not blinded to the PSV/EDV on-screen measurements however they were blinded to the results of the other scientists. For each measurement taken a triplex image was saved from which the PSV and EDV were recorded in an Excel spreadsheet. Limits for allowable data were set in the spreadsheet to avoid copy errors and recorded entries were checked for suspicious outliers.

Image quality assessment

Each triplex US image was saved, cropped of all scientist identifiers, including date and time and imported onto PowerPoint slides for visual quality inspection. The images were then randomly sorted by an independent third reviewer before each image was visually inspected for quality by two reviewers who were blinded to each other’s assessments. The reviewers were all HCPC registered Clinical Scientists with between 4 and 20 years experience of carotid duplex clinical practice and teaching. The quality assessment included sample gate placement and alignment, measurement cursor placement on the pulsed-wave Doppler trace and appropriate gain setting for which the assessment criteria were pre-agreed. The results of the visual inspections by the two reviewers were subsequently compared by the third reviewer. Each reviewer completed the quality assessment in a single sitting and under identical environmental conditions. Where the assessment of individual images did not agree, the third reviewer, a Consultant Clinical Scientist with over 20 years clinical experience of diagnostic carotid US, had the final decision on the assessment of quality.

Statistical analysis

Statistical analyses were undertaken using Minitab^Inc 17.0 (Norfolk, USA). Inter-operator normality was determined using the Anderson-Darling (AD) test (p > 0.05). A one-way ANOVA and Tukey test were used to analyse inter-operator variability and multiple comparison test of operators respectively. An equal variance test was used to compare the standard deviation of intra-operator’s variability. Conclusions were drawn from Levene’s method rather than the multiple comparison method as observations were less than 20 per operator making the type 1 error rate likely to be greater than the specified significance level. All tests were two tailed and considered significant at p < 0.05.

Results

The majority of scientists used a fixed angle of insonation of 60° with only four out of 15 (27%) scientists choosing to vary the angle between 49° and 60° (Table 1). These four scientists were from different departments, had undergone different training schemes and had varying durations of clinical practice. Ten out of 15 (67%) scientists also chose to use a fixed gain setting whilst five scientists chose a range of gain settings (7–31).

Table 1.

Scientist and selected Doppler settings

Scientist ID	Gain settings CCA/ICA	Angle correction CCA/ICA (°)
1	18/18	60/60
2	16/16	60/53
3	18–20/18–20	60/60
4	18/18	60/60
5	18/18	60/60
6	17/17	60/60
7	18/18	60/60
8	28/28	60/60–49
9	20–31	60/60
10	18/18	60/53
11	18/18	53/53
12	24/24	60/60
13	15–19	60/60
14	11–21	60/60
15	7–26	60/60

CCA: common carotid artery; ICA: internal carotid artery.

Image quality assessment

The results of the visual quality inspection of the individual US images which assessed the scientists ability to optimise angle correction, spectral gain and caliper placement (Figure 2(a)) demonstrated that clinical judgement by scientists met an adequate standard although errors were seen across all three domains on which they were assessed (Table 2). Agreement between the two reviewers was reached in 94% of cases with only 6% of cases requiring adjudication by the third reviewer. There was no preponderance towards the judgement of either reviewer.

Figure 2.

Images of carotid arteries with optimal angle correction, spectral gain and caliper placement (a), suboptimal angle correction in the CCA (b), suboptimal spectral gain in the CCA (c) and suboptimal caliper placement in the ICA (d). CCA: common carotid artery; ICA: internal carotid artery.

Table 2.

Results of the visual quality inspection of individual ultrasound images

Suboptimal angle correction, n (%)	Suboptimal spectral gain, n (%)	Suboptimal caliper placement, n (%)
38 (9%)	14 (3%)	18 (4%)

The most frequent error was suboptimal angle correction with 38 out of 420 images (9%) clinically determined as suboptimal. Scientist 11 consistently chose an angle of 53° but did not ‘heel and toe’ the probe to obtain the correct angle to the vessel wall (Figure 2(b)). This, represented 11 of the 38 images (29%) deemed to have a suboptimal angle of insonation. Scientist 13 used an angle correction of 60° but also failed to ‘heel and toe’ the probe to obtain the correct angle to the vessel wall. This represented 12 of the 38 images (32%). Similarly Scientist 2 failed to ‘heel and toe’ the probe to obtain the correct angle representing 6% of the 38 (16%) images. Of the 14 images with suboptimal spectral gain selection, Scientists 8 (Figure 2(c)) and 9 chose to over-saturate the spectral trace which represented 7 (50%) and 5 (36%) respectively of the 14 suboptimal images. Suboptimal placement of callipers appeared to be not as clearly associated with any particular scientist (Figure 2(d)).

Intra-operator variability

Box plot analysis was used to illustrate the results graphically by depicting groups of numerical data through their quartiles and median. In simple terms, the length of the vertical line through each box depicts the level of agreement between measurements from each individual scientist with longer lines depicting a larger range in measurements and the height placement of the different boxes on the scale depict the differences between scientists. Box plot analysis of the combined left and right sided PSV measurements taken from the CCA and ICA for each scientist revealed significant variance (ANOVA p < 0.05). Scientists 3 and 5 had the greatest ranges in PSV measurements in the CCA and ICA respectively (Figure 3(a) and (b)). This was not, however, associated with gain settings or angle correction. CCA F (14, 190) = 41.59 MSE 55.88, p < 0.0001. ICA F (14, 90) = 20.04 MSE 20.76, p < 0.0001. Similarly, box plot analysis of the combined left and right sided EDV measurements from the CCA and ICA for each scientist also revealed significant variance (ANOVA p < 0.05). Scientists 9 and 14 had the greatest ranges in velocity in the CCA and ICA respectively (Figure 3(c) and (d)). Again, this was not, however, associated with gain settings or angle correction. CCA F (14, 195) = 8.785 MSE 136, p < 0.0001. ICA F (14, 195) = 9.876 MSE 149.6, p < 0.0001.

Figure 3.

A box plot of combined (left and right side) CCA (a), ICA (b), PSV and CCA (c) ICA (d) EDV measurements by scientists. The box plot indicates the range, median and the quartiles. CCA: common carotid artery; EDV: end diastolic velocity; ICA: internal carotid artery; PSV: peak systolic velocity.

Inter-operator variability

Equal variance, multiple comparison Tukey post-hoc test revealed that all scientists shared a 95% confidence interval and equal variance with another scientist. This revealed there was no single scientist who consistently produced highly variable results (Levene’s test p = 0.569) and thus the range of velocities must be due to biological variability which occurred naturally within the subject during the assessment (Figure 4). This was demonstrated by Scientist 4 where the PSV of the right ICA measured 65 cm/s and then 1 minute later measured 79 cm/s at the same position (Figure 5(a) and (b)). A similar range was seen when different scientists measured the PSV in the left ICA (86 and 69 cm/s) (Figure 5(c) and (d)).

Figure 4.

Test for equal variances: mean CCA vs. scientist (a) and mean ICA vs. scientist (b). CCA: common carotid artery; ICA: internal carotid artery.

Figure 5.

Images of the right internal carotid artery demonstrating physiological range of the PSV ((a) 65 cm/s and (b) 79 cm/s) measured by Scientist 4 1 minutes apart. Images of the left internal carotid artery demonstrating physiological range of the PSV ((c) 86 cm/s and (d) 69 cm/s) measured by Scientists 12 and 5 respectively.

Discussion

Natural biological variation

One of the major limiting factors of the study was the natural biological variability in the test subject during the US assessment. Although blood pressure and pulse rate were used to determine mean arterial pressure to establish tolerance limits before the examination to try to minimize this variability,¹³ the suggestion that biological effect had more of an influence on variability than inter-operator variability itself was supported by the Tukey post-hoc test which revealed that all scientists shared a 95% confidence interval and equal variance with another scientist.

The Levene’s test showed the degree of variance was similar throughout operators, however, it must be considered that significant variation in the parameter being measured (i.e. natural biological variation) could undermine this inference as it is not possible to be certain that the PSV or EDV variability was equal for each operator at the time the measurements were taken.

Image optimization and angle correction

The qualitative assessment of the individual US images revealed most scientists chose the most appropriate angle correction, spectral gain and caliper placement. However, it was observed that Scientists 2, 11 and 13 consistently did not correctly angle the probe to either the vessel wall and or the flow and Scientists 8 and 9 consistently oversaturated the spectral Doppler trace. This information was fed back to the scientists/units concerned. A draw back of the study was that the portable GE LOGiQe was not equiped with fine angle steer so scientists were forced to ‘heel and toe’ to obtain the most appropriate angle of insonation to the vessel wall.⁵ Oates et al. recommend using an angle of between 45° and 60° and the correct caliper placement has been illustrated by Joint Recommendations for Reporting Carotid Ultrasound Investigations in the United Kingdom.⁵

Previous studies using a flow phantom^1,9 have shown that gain setting, angle correction and cursor placement have a significant effect on the varibility of Doppler peak velocity (DPV).^1,9 These studies were able to accurately determine the DPV (equivalent to PSV) and demonstrated both inter- and intra-operator variability, similar to the findings obtained in this study.¹ This further supports the case that the subject was the main contributing factor in the variability seen in this study. Although the true PSV of the control subject could not be known, the variability seen does highlight the difficulties in obtaining velocities in a clincal setting compared to using a flow phantom.

This study illustrates the need for vascular networks to establish a consenus on optimal caliper placement, spectral gain and Doppler angle. In particular to agree as a network if scientists should maintain a fixed Doppler angle of 60° or use a range of angles between 45° and 60°. The joint recommendation committee suggested all UK vascular Departments adopt a fix angle of 60° or a range of insonation angles between 45° and 60°⁵ but there is limited data to confirm how widely this is implemented in current UK practice. Selecting the smallest possible angle of insonation will reduce the percentage error when angle correction is applied and this also reduces the inaccuracy caused by ISB.⁵ This error will however differ when different angles must be selected, for example where the ICA dives deeper in the tissue or when an irregular stenotic lesion requires angle adjustment to the flow, thus making a comparison more challenging. This further highlights the challenge of implementing such standardisation within the highly variable reality of clinical practice. Technical judgement is always required to ensure the same angle is selected when measuring the PSV prior to and at a stenosis to ensure the correct grade of stenosis. This is especially important when measuring a borderline stenosis. Ideally, however, a compromise of using a fixed 60° angle gives a percentage inaccuracy of 10–15% and, if all scientists maintain this approach, there is then a constant bias.⁸ By keeping the angle of insonation at 60°, which was observed with most scientists in this study, other factors within the machine, such as ISB,^14,15 are kept constant across the network.

Ongoing training

The results from our study also highlight the important issue of the need for ongoing training/skills update sessions within vascular networks. Other studies have also shown that regular updates on scanning techniques improve reproducibility of stenosis grading.^10,11,16 Although a different field of diagnostic US, Thavendiranathan et al.¹¹ showed that self-directed education alone improved accuracy and precision in the measurement of ejection fraction with an improvement in the baseline inter-operator variability from ± 0.120 to ± 0.097. The misclassification was also significantly reduced from 56% to 47% (p = 0.001). Thus self-directed learning may also have an important role in reducing the variability increasing precision and reducing misclassification.

This study was presented to scientists in the vascular network to help illustrate, through education support, the need to be more consistent in angle correction, spectral gain and caliper placement to maintain comparative duplex reports. This is particularly important when determining 50–69% stenoses in the ICA. A comparative meta-analysis study illustrated the sensitivity and specificity of DUS to be 0.36 (95%Cl 0.25–0.49) and 0.91 (95%Cl 0.87–0.94) respectively whilst the sensitivity and specificity of computed tomographic angiography (CTA) was 0.67 (95%Cl 0.30–0.90) and 0.79 (95%Cl 0.63–0.89) respectively in determining a 50–69% ICA stenosis.¹⁷

This study focussed on using a single machine and a single subject and scientists’ agreement as to the PSV/ICA of the CCA and ICA, however there are other sources of variability that could be considered in future research which would be of relevance to clinical practice such as inter-machine variability.

Conclusion

QA is clearly an important part of any service provision, especially with the centralisation of services. This study has highlighted several key issues in effectively assessing agreement across a network including consistent selection of the most appropriate angle correction, optimisation of spectral gain and correct caliper placement. The study demonstrated a generally acceptable level of agreement in assessing the carotid arteries of a healthy test subject across a centralised vascular network. However, a small number of scientists consistently made errors which highlighted important discussion points around scientist ongoing training and skills maintenance in order to sustain consistency across the vascular network.

The variability seen in the PSV and EDV measurements was most likely due to natural biological variance in the test subject, however, this issue has in itself raised some interesting points relating to the practicalities of implementing QA testing programs in clinical practice. The use of a flow mimicking phantom to test inter-operator variabilty could be useful in establishing a QA programme across the network and this could be assessed in future research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethics approval

In accordance with the revised edition of Governance Arrangements for Research Ethics Committees (GAfREC), National Research Ethics Committee (NRES) approval was not required for the study. Approval for the study was granted by the Service Evaluation departments at UH Bristol, NBT, RUH Bath and WAHT.

Guarantor

TR

Contributors

MM and TR designed the study concept, MM carried out data collection and analysed the data, MB and TR provided methodological and statistical support and MM and MB wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.