Skip to main content
NCBI home page
Heart, Lung and Vessels logoLink to Heart, Lung and Vessels
. 2014;6(2):92–104.

Validation of a novel duplex ultrasound objective structured assessment of technical skills (DUOSATS) for arterial stenosis detection

U Jaffer 1,✉, P Singh 1, V A Pandey 1, M Aslam 1, N J Standfield 1
PMCID: PMC4095836  PMID: 25024991

Abstract

Introduction

Duplex ultrasound facilitates bedside diagnosis and hence timely patient care. Its uptake has been hampered by training and accreditation issues. We have developed an assessment tool for Duplex arterial stenosis measurement for both simulator and patient based training.

Methods

A novel assessment tool: duplex ultrasound assessment of technical skills was developed. A modified duplex ultrasound assessment of technical skills was used for simulator training. Novice, intermediate experience and expert users of duplex ultrasound were invited to participate. Participants viewed an instructional video and were allowed ample time to familiarize with the equipment. Participants’ attempts were recorded and independently assessed by four experts using the modified duplex ultrasound assessment of technical skills. ‘Global’ assessment was also done on a four point Likert scale. Content, construct and concurrent validity as well as reliability were evaluated.

Results

Content and construct validity as well as reliability were demonstrated. The simulator had good satisfaction rating from participants: median 4; range 3-5. Receiver operator characteristic analysis has established a cut point of 22/ 34 and 25/ 40 were most appropriate for simulator and patient based assessment respectively. We have validated a novel assessment tool for duplex arterial stenosis detection. Further work is underway to establish transference validity of simulator training to improved skill in scanning patients.

Conclusions

We have developed and validated duplex ultrasound assessment of technical skills for simulator training.

Keywords: training, ultrasound; Duplex; teaching; assessment

Introduction

Vascular ultrasound training. On both sides of the Atlantic, there is accumulating data regarding excessive patients mortality at weekends where full hospital care provision is not offered [1]. Duplex scanning for vascular patients is currently performed largely by specialists, varyingly known as vascular sonographers/ sonologists/ scientists/ technologists. Again, on both sides of the Atlantic, there is a chronic shortage of vascular sonographers [2] and indeed in the UK is listed as a shortage occupation by the UK Borders agency [3]. It seems very unlikely that a 24 hour, seven day a week service will be practicable or affordable to deal with clinical emergencies.

There has been much debate regarding the dissemination of diagnostic ultrasound into professions with little previous experience in its performance. With the wide availability of portable Duplex ultrasound machines (Figure 1), there is a growing desire for doctors to be able to perform arterial scanning.

Figure 1.

Figure 1

Open in a new tab

Portable Duplex ultrasound machine.

It seems logical to consider training vascular specialists in the practical application of vascular Duplex scanning to cover this ‘out-of-hours’ shortfall. Some work has been done in defining how much practical experience is required to attain competence in ultrasound scanning [4]. Simulation based training in ultrasound is beginning to develop, however there is as yet no development for vascular ultrasound [5].

Skills associated with vascular ultrasound include using B mode as well as colour coded ultrasound. However, quantifying arterial stenosis requires analysis of the Doppler spectrum displayed by selective sampling from the arterial stream by pulsed wave Doppler ultrasound (Figure 2).

Figure 2.

Figure 2

Open in a new tab

Real time Doppler frequency spectrum. Time is along the x axis and Doppler shift frequencies (expressed in units of velocity) are along the y axis. The relative intensities of Doppler shift frequencies are depicted by using varying gray scales. A lighter pixel represents a higher relative intensity at that particular frequency and time.

There is currently no accepted validated and reliable assessment tool for duplex assessment of arterial stenosis. There may be a need to objectify credentialing, as is the case with other vascular skills [6]. The use of standardized tests, perhaps simulation based, may be advantageous over current assessments that are performed on patients.

Arterial stenosis detection. There are a variety of skills that need to be learnt before a trainee is proficient in arterial stenosis detection. The operator needs to understand how surrogate physiological markers correspond to degree of stenosis. Also, the operator needs to optimize the Duplex machine settings for Color coded Doppler and Doppler Spectral analysis. Lastly, direct luminal measurements are used to corroborate findings.

The physiological basis of stenosis assessment using Doppler spectral analysis relates to the physics of blood flow through a stenosis. Quantitatively, the amount of blood flowing through a stenosis (Q) is described by the following equation:

                                             Q = v x A

 

Where v = velocity and A = cross sectional area.

Area reduction in a stenosis is proportionate to velocity increase. This is true until a haemodynamically significant stenosis is reached, following which trickle flow ensues (Figure 3).

Figure 3.

Figure 3

Open in a new tab

Flow and velocity changes expressed as percent maximal in response to percentage stenosis.

Hence, the percentage stenosis can be calculated from the ratio of peak systolic velocities at and adjacent to the site of stenosis (peak systolic velocity ratio; PSVR; Table 1).

Table 1.

Table 1

Open in a new tab

Sensitivity and Specificity for Peak Systolic Velocity Ratio (PSVR) for detection of stenosis. From Ranke at al. (1992).

Previous in-vitro work has correlated angiographic diameter reduction to PSVR [7].

As practitioners become more skilled at using the ultrasound machine, they will become more adept at optimizing the imaging both for colour and pulsed wave modes. For colour coded imaging, Duplex settings should be adjusted to optimise: frame rate, velocity range, colour wall filter, colour persistence, colour gain, colour box width.

For spectral readings, the following setting should be optimised: sample volume, velocity range, Doppler angle and need for beam steering, wall thump filter setting.

An additional method used to confirm the stenosis assessment is the diameter reduction at the stenosis. Here the diameter at the stenosis is expressed as a percentage of the diameter of the disease free artery.

Assessment of procedural skills. Previously, the assessment of technical skills, for instance in arterial stenosis detection, has been limited to the use of logbooks recording the type and numbers of procedures performed and global assessments by trainers through observation [8].

While these are important evidences of training that should be recorded, they lack content validity as they do not provide an assessment of the technical ability of the individual performing the procedure [9, 10]. The concept of more objective approaches to technical skills assessment is gaining greater acceptance [11,12,13].

In an attempt to improve the validity and reliability of assessment, the objective structured assessment of technical skills (OSATS) was developed for surgical tasks [14]. This uses a combination of a checklist and global rating scale to assess trainees performing surgical tasks on models. It has been shown to be reliable and have content and face validity. More recently, OSATS has been used in simulated theatre environments [15].

Procedure-specific global rating scales have also been developed, i.e. for sapheno-femoral ligation, long saphenous stripping and multiple avulsions [16].

A valid, reliable and feasible instrument to assess duplex arterial stenosis detection is required for formative and summative assessment as well as revalidation. The duplex ultrasound objective structured assessment of technical skills (DUOSATS) was developed by two experienced practitioners who routinely clinically practice and train in these skills (MA and UJ).

Content analysis along the principles formulated by Gagné (the conditions of learning 1965) was performed. This informed the design of DUOSATS whose domains are: patient positioning, transducer selection, ultrasound coupling gel usage, acquiring images in B mode, evaluation of stenosis in colour, spectral doppler, doppler angle, assessment of PSVR, calculation of diameter reduction at stenosis, calculation of degree of stenosis, reporting.

In order for the assessment instrument to function as a formative and summative assessment, the scoring for tasks was set to reflect progression from lower order concepts, through to rule learning and finally synthesis of rule learning principles into higher order problem solving (Figure 4).

Figure 4.

Figure 4

Open in a new tab

DUOSATS marking scheme.

DUOSATS = duplex ultrasound objective structured assessment of technical skills.

More complex educational objectives were given proportionately greater weight in the overall score to reflect this.

Pulsatile flow simulator. The Axiom vascular flow simulator (Axiom Medical ltd, London) forces pulses of blood-mimicking solution through a simulated blood vessel, arranged within a tissue ‘phantom’.

The flow can be regulated in both the forward and reverse direction. This allows any arterial waveform to be easily simulated by individually adjusting timings and flow rates. A simulated variable arterial stenosis is adjustable in the simulated vessel. This stenosis can be imaged in B mode, colour and pulsed wave mode ultrasound, allowing stenosis evaluation. This simulator was used to generate a femoral artery waveform in a straight vessel phantom containing a 75 percent stenosis.

This study aimed to establish content, construct and concurrent validity as well as reliability of a novel DUOSAT.

Methods

Medical students, junior doctors of varying degrees of experience with Duplex ultrasound and Vascular Scientists (also known as Vascular sonographers/ technologists) and trainees from Imperial College NHS Trust were invited to participate in this study. Audit of teaching does not require formal ethics committee approval in the UK.

For the purposes of subsequent data analysis, participants were classified into nominal categories on the basis of stenosis detection experience (‘novice’, ‘intermediate’ and ‘experienced’). A ‘novice’ had no practical experience of stenosis detection, the ‘intermediate’ group had performed one to 20 cases of stenosis detection and the ‘experienced’ group had performed over 20 cases.

An instructional video was prepared and played to all participants. The video covered theoretical and practical principles outlined in 1.2 and demonstrated how arterial stenosis detection is performed using PSVR and diameter reduction methods. Sections of the video corresponded to domains of the DUOSATS.

Following viewing of the video, sufficient time was allowed for the participant to familiarize themselves with the functioning of the portable Duplex machine used for the study (Mindray M7, Shenzhen, China). Questions regarding the functioning of the Duplex machine and the simulator were answered. When participants felt they were sufficiently familiar with the machine and the task, they were asked to assess a pre-defined stenosis (70%) in the simulated vessel under femoral artery pulsatile flow conditions (Figure 5).

Figure 5.

Figure 5

Open in a new tab

Participant measuring simulated arterial stenosis. Duplex scanner is displaying both Colour and Spectral Doppler scan of stenosis being measured.

Participants were asked to use both peak systolic velocity ratio as well as diameter reduction to form their conclusions. No restriction to the number of measurements was made, however a single percentage was taken from participants for the purpose of analysis. Video clips of each attempt were recorded for later analysis.

Participants were asked to record their stenosis assessment via PSVR and diameter reduction.

Percentage error for both assessments were calculated against the actual grade of stenosis. The video clips were evaluated independently by four experienced practitioners in vascular ultrasound (>fours years practical experience) and were scored on the DUOSATS tool. For these simulator-based assessments, a modified DUOSATS was used; this did not count the patient positioning and reporting fields (scoring guidance detailed in the Figure 4).

A ‘global assessment’ rating was also made on a four point Likert scale (level 1 representing ‘Unable to perform the procedure’; level 2 representing ‘Able to perform the procedure with prompting’, level 3 representing ‘Able to perform the procedure with minimum prompting’; and level 4 representing ‘Competent to perform the procedure unsupervised’).

A four point scale forces raters to take a position and is associated with clearer decision making [17].

The four point Likert scale (with the addition of 0 signifying not enough data available to make assessment) format has been extensively used recently in the context of procedure-based assessments in the UK Intercollegiate Surgical Curriculum Project. The scale has been reported to have a high degree of acceptability by assessors and trainees [18].

Demographic data and experience with ultrasound and specifically stenosis detection were recorded.

Participants were asked to rate their satisfaction with the pulsatile flow simulator on the five point Likert scale.

Analysis. Inter-observer reliability between the four assessors was determined using Cronbach’s alpha (α). Non-parametric tests were used in all the analyses. Spearman’s Rank (R) was used to correlate continuous variables.

The Kruskal Wallis test was used to identify differences between the subgroups tested. Construct validity was assessed by comparing DUOSATS scores of expert, intermediate and novice participants. Concurrent validity was assessed by comparing percentage error in stenosis detection (PSVR and diameter reduction method) and overall rating to DUOSATS. SPSS 20 (IBM corporation) was used in the statistical analysis p < 0.05 was considered statistically significant.

Results

Demographic data. There were 23 participants in total, of which 15 were male (65%).

Content validity. The DUOSAT was developed in conjunction with two experienced practitioners and trainers in vascular ultrasound and addresses domains considered essential in Duplex arterial stenosis detection according to the principles of content analysis.

The device was rated by all participants on a five point rating scale with one given for a poor representation and five for an excellent representation of vascular ultrasound. The scores ranged from 3-5 (median 4), the higher scores coming from the more experienced participants, however this difference was not significant (Kruskal-Wallis p=0.813).Construct validity. The average scores for both DUOSATS and the ‘global’ assessment were used in all subsequent statistical analyses. Scores for DUOSATS ranged from 18.25 to 33.24 (median 19.6 of a maximum of 45). Calculated PSV ratio’s ranged from 0.87-4.83 (median 2.21). The percentage stenosis using diameter reduction criteria ranged from 46-83% (median 70%). Study participants were assessed according to previous practical ultrasound experience (novice n=9, intermediate n=8 and experienced n=6; Table 2).

Table 2.

Table 2

Open in a new tab

Demographic data for participants. Ultrasound (US) experience (1: Theoretical; 2: 1-20 cases; 3: >20 cases). Stenosis detection experience (1: 1-10 cases; 2: 10-20 cases; 3: >20 cases).

There were no significant differences in the DUOSATS score (Kruskal Wallis; p=0.122) or the ‘global’ assessment (p=0.143) across the three groups. The groups were subsequently analysed according to previous stenosis measurement experience. This analysis identified significant differences with both measures (DUOSATS p=0.004, ‘global’ assessment (p=0.006).

A further analysis of individual DUOSATS domains using stenosis measurement experience was performed to establish construct validity for individual items and the results are summarised in Table 3.

Table 3.

Table 3

Open in a new tab

Breakdown of individual domains within Duplex Ultrasound Objective Structured Assessment of Technical Skill (DUOSATS; * p<0.05). Peak Systolic Velocity Ratio (PSVR).

Statistically significant fields were acquisition of image in B mode, evaluation of stenosis using Spectral Doppler angle, calculation of diameter reduction and calculation of degree of stenosis.Concurrent validity. When the two measures were correlated, there was a significant correlation between DUOSATS score and ‘global’ assessment scores (Figure 6) (Spearman’s rank correlation coefficient; R=0.737, p=0.000092).

Figure 6.

Figure 6

Open in a new tab

Scatter plot of average overall (‘global’) score versus DUOSATS score demonstrating correlation.

DUOSATS = duplex ultrasound objective structured assessment of technical skills.

The DUOSATS score correlated with stenosis measurement using PSVR criteria (R=0.46, p=0.03; Figure 7) but not with stenosis measurement using diameter reduction. criteria (R=0.4, p=0.065; Figure 8).

Figure 7.

Figure 7

Open in a new tab

Scatter plot of average DUOSATS score versus the error score stenosis measurement using PSV ratios demonstrating correlation.

DUOSATS = duplex ultrasound objective structured assessment of technical skills; PSV = peak systolic velocity ratio.

Figure 8.

Figure 8

Open in a new tab

Scatter plot of average DUOSATS score versus error score for stenosis calculation using diameter reduction criteria.

 DUOSATS = duplex ultrasound objective structured assessment of technical skills.

On multivariate analysis, age (contribution is 29.8%, p=0.05) and stenosis detection experience had a significant effect on score (contribution is 43.1%, p=0.008). However, ultrasound experience had no effect on the DUOSATS score (contribution= 1.7% of variance, p=0.861). Reliability. Four independent expert assessors reviewed the videotape footage of each participant performing stenosis measurement. Assessors were blinded to the identity of the study participant. Inter-observer reliability for DUOSATS and ‘global’ assessment scores were high (Cronbach’s alpha=0.971 and 0.957 respectively). Sensitivity and specificity. A Receiver Operator Characteristics curve (ROC; Figure 9) was plotted for the sensitivity and specificity of the test using the scores of those experienced in stenosis detection to determine the cut point in determining competence for this procedure.

Figure 9.

Figure 9

Open in a new tab

ROC curve demonstrating the correlation between true positive versus false positive rates. Using these characteristics, an appropriate cut point for the evaluation were determined.

ROC = receiver operator characteristics curve.

The area under the curve was 0.895. For various cut points the specificity and sensitivity is shown in Table 4.

Table 4.

Table 4

Open in a new tab

Specificities and Sensitivities associated with various cut points for the Duplex Ultrasound Objective Assessment of Technical Skill (DUOSATS) score.

It was decided the cut-point should have a high specificity (i.e. less experienced participants were unlikely to be considered competent) trading off for a slightly reduced sensitivity for this test (more experienced participants may not achieve competence in this test. The cut point was set at 22 (out of a possible 34).

Discussion

Regarding assessment instruments, Cook and Beckman discuss that “validity is not a property of the instrument, but of the instrument’s scores and their interpretations” [19]. Both DUOSATS and the ‘global’ assessment were able to differentiate participants according to their experience of Duplex arterial stenosis detection (DUOSATS: p=0.004, ‘global’ assessment; p=0.006). This suggests good construct validity. Importantly, this discriminating ability did not extend to previous ultrasound experience not specifically related to arterial stenosis detection (DUOSATS: p=0.122 ‘global’ assessment p=0.143) across the three groups, suggesting that DUOSATS is sensitive to specific skills related to the task at hand and has content validity. It may be suggested that ‘global’ assessment is an acceptable alternative to DUOSATS, however the limitation is the lack of content validity and unstructured unsystematic approach. DUOSATS score correlated with stenosis measurement using PSVR criteria (R=0.46, p=0.03) indicating concurrent validity with an objective ‘end product’ style assessment.

A potential criticism of DUOSATS is its development by only two people. However, the DUOSATS domains cover a wide range of skills related to the task and multiple domains show significant discriminating ability to level of experience. The DUOSATS domains best able to discriminate between novice, intermediate and experienced users were found to be: acquisition of image in B mode, evaluation of stenosis using spectral doppler angle, calculation of diameter reduction and, calculation of degree of stenosis. These domains should be most closely followed when using the DUOSATS in formative assessment. Careful feedback on these fields may focus and hasten training.

ROC data suggests that a score of 22 (out of a possible 34) has high sensitivity and specificity, sufficient for high stakes assessment on the pulsatile flow simulator. We believe that it is reasonable to increase the pass mark to 25 (out of a possible 40) for the ‘full’ DUOSATS scoring to be used in patient assessment. This data facilitates standard setting in assessing Duplex arterial stenosis evaluation not subject to the ‘halo’ effect encountered in relative or norm referenced assessment tools. Although this methods of standard setting is itself technically ‘norm referenced’, we feel that introduction of this skill to a new group of practitioners should not be associated with a decline in overall standards. This approach potentially goes some way to satisfy the fit for purpose, supported by research, easy implement standard of assessment [20]. Further validation of the full DUOSATS on patients is underway.

We have demonstrated content and concurrent validity. Inter-rater reliability amongst experienced assessors was high, comparing favorably with other objective structured technical skills assessments [14, 15, 21]. Participants rated the simulator highly in terms of satisfaction. This compares favourably with other technical skills simulators [22,23,24]. Investigators in a number of technical skills simulation disciples have reported transference of skills from bench top simulators to a human cadaver model [25] as well as operating theatre environments [26, 27]. Further work is required, in the form of a trial, to assess transference of skills learnt using this pulsatile flow simulator to scanning on patients. Further development in phantom design may be desirable to provide more challenging anatomy, for instance the obese patient and arterial wall calcification. Validation of the DUOSATS system for simulator training potentially represents an early yet important milestone in the development of training and assessment in Duplex arterial assessment. Validation of the DUOSATS for patient scanning remains to be assessed. We envisage potential difficulties in terms of patient variability, a problem not encountered in ‘standard’ simulated scenarios. This may potentially be overcome by the use of statistical process control techniques and learning curve assessment rather than a single point assessment.

 

Limitations

This study only used a single severity of stenosis for validation of the OSAT. This was chosen to be 70 percent as this degree of stenosis is clinically significant. We have not established validity using different degrees of stenosis. Further work should take this into consideration and also have increased number of participants to compensate for the increased variability of errors when considering different degrees of stenosis. The validation has been performed for a modified DUOSATS for simulator training. Further correlation with the full DUOSATS for actual patient testing to include patient positioning, will need separate assessment.

 

Conclusion

We have developed a novel objective structured assessment of technical skills for duplex assessment of arterial stenosis. The modified DUOSATS for simulator training and assessment was validated.

Acknowledgments

We would like to thank Mary Ellis and Surinder Dhaljal for the participation of their laboratories in this work. We would like to thank Mindray Corporation for supply of the Duplex machine.

Footnotes

Source of Support Nil.

Disclosures Usman Jaffer is a Director of Axiom Medical ltd.

Cite as: Jaffer U, Singh P, Pandey VA, Aslam M, Standfield NJ. Validation of a novel duplex ultrasound objective structured assessment of technical skills (DUOSATS) for arterial stenosis detection. Heart, Lung and Vessels. 2014; 6(2): 92-104.

References

  1. Keogh B. Should the NHS work at weekends as it does in the week? Yes. BMJ. 2013;346:621–621. doi: 10.1136/bmj.f621. [DOI] [PubMed] [Google Scholar]
  2. Williams C D, Maloney E M, McElveny C. Resolving the technologist shortage: A summary of ACR and ASRT efforts to meet the increasing demand for RTs. Journal of the American College of Radiology. 2004;1:842–847. doi: 10.1016/j.jacr.2004.05.020. [DOI] [PubMed] [Google Scholar]
  3. Agency UB. Shortage occupation list 2013 [cited 2013 01/08/2013]. [Available from: http://www.ukba.homeoffice.gov.uk/business-sponsors/points/sponsoringmigrants/employingmigrants/shortageoccupationlist/]
  4. Eiberg J P, Hansen M A, Gronvall Rasmussen J B, Schroeder T V. Minimum training requirement in ultrasound imaging of peripheral arterial disease. Eur J Vasc Endovasc Surg. 2008;36:325–330. doi: 10.1016/j.ejvs.2008.06.006. [DOI] [PubMed] [Google Scholar]
  5. Sidhu H S, Olubaniyi B O, Bhatnagar G, Shuen V, Dubbins P. Role of simulation-based education in ultrasound practice training. J Ultrasound Med. 2012;31:785–791. doi: 10.7863/jum.2012.31.5.785. [DOI] [PubMed] [Google Scholar]
  6. Hobson RW 2nd, Howard V J, Roubin G D, Ferguson R D, Brott T G, Howard G. et al. Credentialing of surgeons as interventionalists for carotid artery stenting: experience from the lead-in phase of CREST. J Vasc Surg. 2004;40:952–957. doi: 10.1016/j.jvs.2004.08.039. [DOI] [PubMed] [Google Scholar]
  7. Ranke C, Creutzig A, Alexander K. Duplex scanning of the peripheral arteries: correlation of the peak velocity ratio with angiographic diameter reduction. Ultrasound Med Biol. 1992;18:433–440. doi: 10.1016/0301-5629(92)90082-l. [DOI] [PubMed] [Google Scholar]
  8. Scott A R. Educational challenges in vascular surgery: an Australasian response. J Vasc Surg. 2008;48:66S–68S. doi: 10.1016/j.jvs.2008.09.002. [DOI] [PubMed] [Google Scholar]
  9. Reznick R K. Teaching and testing technical skills. Am J Surg. 1993;165:358–361. doi: 10.1016/s0002-9610(05)80843-8. [DOI] [PubMed] [Google Scholar]
  10. Cuschieri A, Francis N, Crosby J, Hanna G B. What do master surgeons think of surgical competence and revalidation? Am J Surg. 2001;182:110–116. doi: 10.1016/s0002-9610(01)00667-5. [DOI] [PubMed] [Google Scholar]
  11. van Bockel J H, Bergqvist D, Cairols M, Liapis C D, Benedetti-Valentini F, Pandey V. et al. Education in vascular surgery: critical issues around the globe-training and qualification in vascular surgery in Europe. J Vasc Surg. 2008;48:69–75. doi: 10.1016/j.jvs.2008.08.035. [DOI] [PubMed] [Google Scholar]
  12. Mitchell E L, Arora A, Moneta G L. Ensuring vascular surgical training is on the right track. J Vasc Surg. 2011;53:517–525. doi: 10.1016/j.jvs.2010.08.082. [DOI] [PubMed] [Google Scholar]
  13. Neequaye S K, Aggarwal R, Van Herzeele I, Darzi A, Cheshire N J. Endovascular skills training and assessment. J Vasc Surg. 2007;46:1055–1064. doi: 10.1016/j.jvs.2007.05.041. [DOI] [PubMed] [Google Scholar]
  14. Martin J A, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C. et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273–278. doi: 10.1046/j.1365-2168.1997.02502.x. [DOI] [PubMed] [Google Scholar]
  15. Moorthy K, Munz Y, Adams S, Pandey V, Darzi A. A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg. 2005;242:631–639. doi: 10.1097/01.sla.0000186298.79308.a8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Pandey V A, Wolfe J H, Liapis C D, Bergqvist D. The examination assessment of technical competence in vascular surgery. Br J Surg. 2006;93:1132–1138. doi: 10.1002/bjs.5302. [DOI] [PubMed] [Google Scholar]
  17. Garland R. The mid-point on a rating scale: Is it desirable. Marketing bulletin. Department of Marketing, Massey University. 1991;2:66–66. [Google Scholar]
  18. Eardley I, Bussey M, Woodthorpe A, Munsch C, Beard J. Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme. ANZ Journal of Surgery. 2013;83:448–453. doi: 10.1111/ans.12187. [DOI] [PubMed] [Google Scholar]
  19. Cook D A, Beckman T J. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166–166. doi: 10.1016/j.amjmed.2005.10.036. [DOI] [PubMed] [Google Scholar]
  20. Norcini J. Edinborough: Elsevier Churchill Livingstone. 2005. Standard Setting. [Dent JA, Harden, R.M., editor.] [Google Scholar]
  21. Pandey V A, Wolfe J H, Black S A, Cairols M, Liapis C D, Bergqvist D. Self-assessment of technical skill in surgery: the need for expert feedback. Ann R Coll Surg Engl. 2008;90:286–290. doi: 10.1308/003588408X286008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Black S A, Harrison R H, Horrocks E J, Pandey V A, Wolfe J H. Competence assessment of senior vascular trainees using a carotid endarterectomy bench model. Br J Surg. 2007;94:1226–1231. doi: 10.1002/bjs.5794. [DOI] [PubMed] [Google Scholar]
  23. Frost D W, Cavalcanti R B, Toubassi D. Instruction using a high-fidelity cardiopulmonary simulator improves examination skills and resource allocation in family medicine trainees. Simul Healthc. 2011;6:278–283. doi: 10.1097/SIH.0b013e3182207d1c. [DOI] [PubMed] [Google Scholar]
  24. Singh P B, Saw N K, Mokete M, Martin F L, Matanhelia S S. An integrated laparoscopic simulator (i-Sim) to develop surgical skills outside the operating theatre: a novel means to improve training facilities in the UK. Int J Surg. 2008;6:64–70. doi: 10.1016/j.ijsu.2007.06.001. [DOI] [PubMed] [Google Scholar]
  25. Anastakis D J, Regehr G, Reznick R K, Cusimano M, Murnaghan J, Brown M. et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg. 1999;177:167–170. doi: 10.1016/s0002-9610(98)00327-4. [DOI] [PubMed] [Google Scholar]
  26. Beard J D, Jolly B C, Newble D I, Thomas W E, Donnelly J, Southgate L J. Assessing the technical skills of surgical trainees. Br J Surg. 2005;92:778–782. doi: 10.1002/bjs.4951. [DOI] [PubMed] [Google Scholar]
  27. Datta V, Bann S, Beard J, Mandalia M, Darzi A. . Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg. 2004;199:603–606. doi: 10.1016/j.jamcollsurg.2004.05.269. [DOI] [PubMed] [Google Scholar]

Articles from Heart, Lung and Vessels are provided here courtesy of Edizioni Medico Scientifiche

RESOURCES