Integrating Direct Observation of Procedural Skills as a workplace‑based assessment tool for residents working in an intensive care unit

Abstract

Background

Direct Observation of Procedural Skills (DOPS) is a workplace-based assessment tool where the student is directly observed by the assessor during the procedure in an actual clinical situation, followed by immediate feedback.

Objectives

To introduce DOPS as an assessment tool for ultrasound-guided central venous catheter insertion by residents (registrars) working in an intensive care unit.

Methods

A mixed-methods study design was used for assessment (two sessions) using a pre-validated DOPS checklist for scoring 25 residents. Feedback was obtained from faculty members and residents after the second session using an anonymous questionnaire, and the satisfaction index was calculated for each item.

Results

With the implementation of DOPS, a significant improvement in the second session score in comparison with the first session was observed (p<0.001). This improvement can be attributed to use of the DOPS checklist, deconstruction of steps and provision of immediate constructive feedback, which helped to cover all the aspects of the procedure in a systematic manner. The overall perception of the residents as assessed by the feedback questionnaire was satisfactory, with most of them feeling that the assessment by DOPS was unbiased and that constructive feedback positively impacted on their learning process, making it more engaging and interesting. The satisfaction index for all the items was ≥88%. Faculty members were also satisfied with DOPS as a formative assessment tool.

Conclusion

DOPS promotes active and deep learning with effective assessment methods. It can improve teaching and training if there is commitment from all stakeholders and a supportive organisation.

Study synopsis

What the study adds. This study demonstrates that Direct Observation of Procedural Skills (DOPS) can be effectively implemented as a formative workplace-based assessment (WBPA) tool for residents (registrars) performing ultrasound-guided central venous catheter insertion in an intensive care unit. Use of DOPS with structured feedback using the Pendleton model led to significant improvement in procedural competence and confidence. Both residents and faculty members perceived DOPS as an objective, valid and engaging method of assessment.

Implications of the findings. The study indicates that introducing DOPS as a WPBA tool is an effective and engaging way to encourage residents to adopt correct practices of procedural skills, as reflected by the largely positive responses from both the residents and faculty members. The implications of this project will be the educational and quality assurance impact of DOPS implementation in resident training.

Background

Residents (usually termed registrars in South Africa) working in an intensive care unit (ICU) need to be skilled in performing various bedside procedures. One such procedure is insertion of an ultrasound guided central venous catheter (USG-CVC).^[1] However, there is a lack of uniformity in teaching and assessment of various bedside skills in postgraduate training, one of these being CVC insertion.^[2] The traditional methods of assessment such as an oral examination/viva or long case mainly test the cognitive aspect, while the skills aspect is seldom assessed or is highly subjective (e.g. student logbooks).^[3,4] It has been reported that residents are seldom observed or assessed during the performance of procedural skills (<50%), and very rarely given feedback (<20%).^[2]

Direct Observation of Procedural Skills (DOPS) is a formative assessment tool first introduced by the Royal College of Physicians in the UK in 2005.^[5] It is a workplace-based assessment (WPBA) tool that allows direct observation of the student by the assessor in an actual clinical encounter with a patient, followed by immediate feedback.^[6,7] Based on Miller’s pyramid, DOPS therefore assesses the highest levels of competence (‘Shows how’ and ‘Does’).^[8] DOPS is learner led and helps students to identify their strengths and weaknesses.^[9] It serves the dual purpose of assessment and an opportunity to provide immediate feedback to improve competence.^[9,10] Feedback is given using the Pendleton model, which offers learners the opportunity to evaluate their practice and identify ways of improving.^[11] DOPS is reported to have high reliability, good validity and objectivity.^[12]

The present study was undertaken with the objectives of integrating DOPS as a WPBA tool among residents working in an ICU, and assessing the perceptions of residents and faculty members regarding DOPS as a formative assessment tool.

Methods

Guided by a pragmatic paradigm, this prospective mixed-methods study in health professional education was conducted in the Department of Anaesthesiology, Intensive Care and Pain Medicine at the Government Institute of Medical Sciences (GIMS), Greater Noida, India, over a 3-month period, targeting 2nd- and 3rd-year residents working in the ICU. The study focused on the commonly performed core skill of USG-CVC insertion,^[9,13] which was selected as a procedural skill for using the DOPS assessment tool. The aim was to introduce DOPS as a WPBA tool for evaluating residents’ proficiency in USG-CVC insertion in the ICU setting.

Recruitment for the study started after institutional ethics committee approval (ref. no. GIMS/IEC/HR/2024/32) and clinical trial registration (ref. no. CTRI/2024/04/066477). The participants were faculty members and residents who gave voluntary consent to participate in the study. A core team of faculty members (n=6), who volunteered for the study, was formed. A faculty orientation seminar on DOPS assessment and providing feedback using the Pendleton model was organised in the Department of Anaesthesiology, Intensive Care and Pain Medicine.^[11,14] The residents working in the ICU were also orientated to DOPS as an assessment and learning tool. Thirty one residents who gave informed consent for participation in the study were registered. Only those residents who participated in two DOPS sessions (D1 and D2) and completed the feedback questionnaire were included in the final analysis (n=25).

Assessment of USG-CVC insertion was done using a pre validated structured proforma (DOPS checklist form) for central venous catheter insertion (ultrasound guided) by NHS Mastery Skills Pathway (Annexure 1; also available at NHS Lothian, Medical Education Directorate 2020).^[13] In this checklist, each criterion is further deconstructed for the observers (faculty members in our case) to assess it more easily. The checklist had a total score of 25. A score ≥15 (≥60%) was benchmarked as a pass mark based on the external data from previous research.^[15] As DOPS is learner led, the residents approached the faculty to seek their assessment for both the sessions. Every student was directly observed by a faculty member while performing the procedure in the ICU. Observations were documented on the DOPS checklist, and the final score was noted as the D1 score by the principal investigator on an Excel spreadsheet, 2019 (Microsoft, USA). During the first DOPS assessment, checklist-based constructive feedback was given to residents by the observer to encourage self-directed learning and reflective practice. The Pendleton model^[15] was used for giving feedback to the residents and was documented in the feedback space provided in the DOPS checklist. After ensuring that the patient was stable, feedback was given in the side-room of the ICU to maintain privacy. The resident was first asked what they thought had gone well, followed by the faculty member’s feedback regarding observations on those points or any other positive point. Then the resident was asked what they thought could be improved, which was consolidated by the faculty members following up on the areas that required improvement.

The second DOPS assessment (D2 score) was done 15 days after the first assessment to evaluate the improvement in procedural skills and check retention of knowledge. During this intervening period, the residents were encouraged to carry out self-directed learning and practise skills based on feedback suggestions. Feedback was given in the second DOPS session as well. The time taken for the whole assessment and the feedback process was recorded for both the DOPS sessions. To reduce the effect of confounding variables such as lack of motivation for self-learning on the part of the resident, personality of the resident (e.g. anxiety on observation) or observer bias, multiple DOPS sessions are required; however, owing to the limited time available for the research project, only two sessions could be conducted.

Feedback questionnaires in the form of structured sets of closed ended questions to assess overall perceptions of the faculty members and residents of their experiences with DOPS as a WPBA tool were prepared, reviewed and validated for content by three faculty members of the Medical Education Department and approved by the GIMS ethics committee. The feedback questionnaires comprised eight closed-ended items regarding the perception of participants regarding DOPS as an assessment tool. The rating was done using a five-point Likert scale ranging from 1 to 5 (5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree, 1 = strongly disagree). After completion of both DOPS sessions, the feedback questionnaires were electronically shared with the faculty members and residents as a Google form. The feedback was collected anonymously and confidentially. Fig. 1 is a flowchart of the study process.

Fig. 1.

Flowchart of the study

IEC = institutional ethics committee

DOPS = Direct Observation of Procedural Skills

USG-CVC = ultrasound-guided central venous catheterisation

D1 and D2 = first and second DOPS sessions

ICU = intensive care unit

Data were collected, entered into an Excel spreadsheet (2019), and analysed using the Statistical Package for Social Sciences (SPSS), version 28 (IBM, USA).

The satisfaction index (SI) of each item was then calculated using the following formula^[16]

SI = [(n₁ × 1) + (n₂ × 2) + (n₄ × 4) + (n₅ × 5)] × 20 / (n₁ + n2₂ + n4₄ + n₅)

where n is the total number of participants gaining the score represented by the subscript for that particular item. The scores were rated on a satisfaction index scale of 1 - 100.

The quantitative variables were presented as means and standard deviations. To calculate the statistical difference in the frequency of pass percentages between the D1 and D2 scores, Pearson’s χ² test was used and the p-value was obtained after Yates’s correction. A p-value <0.05 was considered statistically significant. Cronbach’s alpha values were calculated for the survey items to have good internal consistency with values >0.8, suggesting that the items are well correlated and are likely to be measuring the same underlying construct effectively. With eight items in each survey, the value >0.8 was considered to have a robust reliability coefficient, giving confidence in the ability of the survey to assess the targeted construct. However, the small number of participants in the faculty group (n=6) could significantly affect statistical analysis by decreasing power, increasing the risk of type II errors, and leading to less reliable results owing to increased variability and potential bias. It also limits the generalisability of findings and could make it difficult to explore heterogeneity within the data.

Results

Of 31 residents registered, only 25 completed the study, as there were six participant dropouts. The average time taken for each DOPS session, including assessment and feedback, was 25 minutes (range 21 - 28 minutes). In the first DOPS assessment (D1), only 28% of the residents secured a pass percentage (score ≥60%). In the second DOPS assessment (D2), done 15 days later, the residents showed statistically significant improvement in their scores (p<0.001) (Table 1).

Table 1. Comparison of DOPS scores.

D1, n (%)		D2, n (%)
Pass	Fail	Pass	Fail	p‑value*
7 (28)	18 (72)	19 (76)	6 (24)	0.001

DOPS = Direct Observation of Procedural Skills; DI and D2 = first and second DOPS sessions

* Pearson’s χ² test, with Yates’s correction

p<0.05 was considered significant

At the end of the second session, the responses of the participants regarding DOPS as an assessment tool were recorded using a feedback questionnaire. The feedback received through anonymous responses to the questionnaire was positive for DOPS (Table 2). The satisfaction scores were rated on a satisfaction index scale of 1 - 100. The satisfaction index was high (≥88) for all parameters, with the highest (94) corresponding to improvement in resident faculty relationships. DOPS was felt to be a more engaging method and to have a positive impact on learning compared with traditional methods. The residents felt the need for this method to be incorporated as a routine means of formative assessment.

The responses of the faculty members to the feedback questionnaire are shown in Table 3. Analysis also revealed positive feedback on DOPS as an assessment tool. While most of the faculty members were of the opinion that training in DOPS as an assessment tool is required before its implementation, they considered DOPS to be an accurate, uniform and easy method of assessment that could be incorporated into the regular ICU routine. The satisfaction level for DOPS as a tool for assessment was similar among the residents and faculty members. Improvement in the learner-assessor relationship with DOPS was perceived as variable by faculty members. The small faculty population in the study (n=6) limits the generalisability of the results.

Table 2. Responses of the residents to the feedback questionnaire (N=25).

	Response on Likert scale, n (%)
Item	5	4	3	2	1	Median Likert score	SI
DOPS is an unbiased method of assessment	14 (56)	7 (28)	2 (8)	1 (4)	1 (4)	5	88
I find the assessment method more engaging and interesting compared with the traditional method	12 (48)	11 (44)	2 (8)	0	0	4	90
DOPS had a positive impact on my learning	11 (46)	12 (50)	2 (8)	0	0	4	90
DOPS helped me in improving resident-faculty relationship	17 (68)	8 (32)	0	0	0	5	94
DOPS should be incorporated in normal routine	12 (48)	11 (44)	2 (8)	0	0	4	90
I received timely and constructive feedback during the assessment	15 (60)	9 (36)	0	1 (4)	0	5	90
I was allowed to put forward my point of view	12 (50)	11 (44)	2 (8)	0	0	4	91
DOPS should be continued as a method of assessment	13 (52)	10 (40)	2 (8)	0	0	5	91

Response on Likert scale: 5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree, 1 = strongly disagree

SI = satisfaction index (scale 1 - 100)

DOPS = Direct Observation of Procedural Skills

Table 3. Responses of the faculty members to the feedback questionnaire (N=6).

	Response on Likert scale, n (%)
Item	5	4	3	2	1	Median Likert score	SI
DOPS is an accurate way of determining practical skills	3 (50)	1 (17)	2 (33)	0	0	4.5	95
Training of faculty in DOPS is required before assessment	5 (58)	1 (17)	0	0	0	4	97
I was able to assess all the students uniformly using DOPS	3 (50)	2 (33)	1 (17)	0	0	4.5	92
I feel that DOPS as an assessment method is better than the traditional method	2 (33)	3 (50)	1 (17)	0	0	4	88
DOPS should be continued as a method of assessment in the ICU	3 (50)	3 (50)	0	0	0	4	90
DOPS as an assessment method is easy and should be incorporated in normal routine	2 (33)	3 (50)	1 (17)	0	0	4	88
DOPS improves performance scores of residents	4 (67)	2 (33)	0	0	0	5	93
DOPS helps to improve learner-assessor relationship	2 (33)	2 (33)	2 (33)	0	0	4.5	90

Response on Likert scale: 5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree, 1 = strongly disagree

SI = satisfaction index (scale 1 - 100)

DOPS = Direct Observation of Procedural Skills

ICU = intensive care unit.

Discussion

It is essential to stress the importance of achieving expertise in clinical skills.^[3] In the present study, the residents working in an ICU were assessed using DOPS during the insertion of a USG-CVC. This formative assessment used a pre-validated structured proforma (DOPS checklist form) for central venous catheter insertion (ultrasound guided) by NHS Mastery Skills Pathway.^[13]

There was a significant improvement in the D2 DOPS scores in comparison with the D1 scores (p<0.001). The improvement in the scores can be ascribed to use of the DOPS checklist and deconstruction of steps which are integral to the DOPS checklist. The use of the checklist helped to cover all the aspects of the procedure in a systematic manner and encouraged residents to develop self regulated learning and reflective practice. Similar improvement in DOPS scores with the help of a structured checklist for reference has been reported.^[15] A study by Hill et al.^[17] demonstrated that a structured standardised checklist comprising the key components of ultrasound-guided central line insertion can improve performance in most participants. Studies have reported that regular and constructive feedback in a non hostile setting forms an integral component of formative assessment tools such as DOPS, which can lead to effective reinforcement of good practices and promotes self-reflection. ^[11,14] Using the Pendleton method of providing timely feedback in our study may have motivated the residents to work towards their desired outcome and develop correct practices. These findings are consistent with the results obtained by Kamat et al.,^[18] who reported significant improvement in procedural skills performance after DOPS feedback. A study by McLeod et al.^[19] also concluded that DOPS improves overall procedural skills assessment in medical students.

The residents perceived the feedback as helpful, as it was delivered immediately and adequate time was allowed for it. They reported that feedback was constructive and that they were allowed to put forward their point of view. Several studies have shown high satisfaction levels among faculty and students when adequate time (20 - 30 minutes) for feedback is given, with a chance to clarify their muddiest points.^[11,15,20] Some studies have reported dissatisfaction with this method, which can be attributed to insufficient time for feedback (<15 minutes) or DOPS assessment being conducted outside of working hours.^[21]

In the present study, DOPS was perceived by the participants as an effective formative assessment tool compared with the traditional methods (e.g. oral examination/viva or long case), as has been reported by several previous studies comparing DOPS with traditional assessment methods.^[22,23] A systematic review by Erfani et al.^[12] reported that DOPS assessments can motivate learning.

DOPS as an assessment tool was perceived as unbiased by the participants in this study. The faculty members who took part in the study believed that DOPS incorporation into the regular routine of training should be continued as a formative assessment tool to improve outcomes in competency-based education. Similar views were expressed in a study by Amini et al.,^[24] with DOPS considered to be an unbiased and motivating exercise for skills enhancement.

The satisfaction index was high in all the survey items assessed for both faculty and residents. These results agree with several other studies where levels of satisfaction were high when DOPS was used as a formative assessment tool.^[14,24]

A major strength of the present study lies in the effective utilisation of DOPS to provide immediate formative feedback – a feature intrinsic to DOPS and successfully implemented in this context. In addition, the involvement of motivated faculty members who remained committed to the study time frame and enthusiastic residents who were very forthcoming in approaching the faculty for their assessments was a bonus.

This study had a few limitations. The time constraint imposed by the project deadline (3 months) meant that only two DOPS sessions could be planned for each resident. A busy ICU, high-risk patients, and limited training of the participants in using the DOPS tool may have affected the quality of assessment. The study population was small, which limits its generalisabilty. These limitations can be overcome by recruitment of larger numbers of participants, repeated training, and motivation of faculty and residents to adopt DOPS as a formative WPBA tool.

Conclusion

With the effective implementation of DOPS as a WPBA tool, the core procedural skills required in ICU training can be achieved in a clinical setting. The participants perceived immediate and constructive feedback as an effective measure to improve practices and achieve competency in USG-CVC insertion in the ICU. However, successful implementation will only be possible with diligent planning, commitment from the stakeholders, and a robust organisation.

Supplementary Tables

Annexure 1: CENTRAL VENOUS CATHETER INSERTION