Introduction of direct observation of procedural skills (DOPS) as a formative assessment tool during postgraduate training in anaesthesiology: Exploration of perceptions

Abstract

Background and Aims:

Formative assessment of procedural skills of Anaesthesiology postgraduate (PG) students is not conducted conventionally. Direct observation of procedural skills (DOPS) helps to identify gaps in performance and provides structured feedback. The present study was taken to explore perceptions of PG students and faculty about DOPS.

Methods:

This mixed design interventional study was conducted on 12 PGs and 10 faculty members in Department of Anaesthesiology. After conducting DOPS, a pre-validated questionnaire was given to them about perceptions using 3-point Likert's scale along with open ended questions. Statistical analysis was done using descriptive statistics of perception to calculate percentages and themes were identified for qualitative data.

Results:

Responses of students were positive about skill improvement (83%), time provided (75%), feedback (100%), interaction (83%), motivation (83%), satisfaction (83%), effectiveness (83%) and opportunity creation (92%). Faculty responded positively regarding change in attitude (100%), effectiveness (100%), scope (90%), feasible application (90%), ease (90%), opportunity creation (80%), gap identification (100%), satisfaction (80%). However, 60% felt training was required, 50% thought more time and commitment was required. Themes identified were DOPS is comprehensive, interactive, student-friendly, good teaching-learning tool, identifies gaps, provides focus for learning, provides systematic constructive feedback, improves skills, prepares for future practice, requires planning, may not reflect competence, has assessor variability and can be included in PG curriculum.

Conclusion:

DOPS was perceived as an effective assessment and teaching-learning tool by PG students as well as faculty.

INTRODUCTION

Clinical practice of anaesthesia mandates mastery over procedural skills as a basic requirement. Direct observation of procedural skills (DOPS) is a tool for workplace-based assessment (WPBA) designed to observe procedural skills involving real patient encounter and to give feedback based on this observation.[1,2,3] This formative assessment has been a part of the curriculum for specialist training in anaesthesiology in the western world. Despite realising the importance, formal assessment of procedural skills has not been a part of the postgraduate(PG) training programme in India. So far, literature from our country has offered limited evidence on DOPS as a WPBA tool. Being a teaching institute, we wished to introduce DOPS in the Department of Anaesthesiology as this would help in ensuring competency of PG trainees. Thus, we conducted the present study with objectives to explore perceptions of PG students and faculty about DOPS as a formative assessment tool and teaching-learning method. We also intended to explore their perceptions about feasibility and limitations of its implementation.

METHODS

This prospective mixed design study was conducted as project work for advanced course in medical education. It was Kirkpatrick level one study exploring participant reactions to DOPS intervention. Following Institutional Ethics Committee (IEC) clearance and registering with the Clinical Trial Registry of India (CTRI REF/2019/07/027270), 12 PG students and 10 faculty members (senior residents, assistant and associate professors) in the Department of Anaesthesiology were included in the study by convenient sampling. As the study project needed to be completed within a limited time duration of four months, PGs available in operation room (OR), four belonging to each of the three years of training, were included in the study, remaining being in peripheral postings. Orientation to DOPS was conducted by faculty from the Department of Medical Education which involved a theory session followed by demonstration of conduct of DOPS and use of checklist. The faculty members were trained to give high quality feedback, were sensitised and motivated for participation. They showed interest in the project and felt the need for the same following the orientation programme. The 10 faculty members who volunteered were included in the study and informed consent was obtained from all the participants.

DOPS form the Australian and New Zealand College of Anaesthetists (ANZCA),[4] freely available on web, was used as a resource and modified to create checklists for individual procedures. Each criterion in the DOPS checklist was further deconstructed for ease of administration by the assessors [Table 1]. These were reviewed and validated by five senior faculty members of the department. DOPS was performed three times for each procedure for every student at an interval of one month. Each time, the observer conducting DOPS was a different faculty member so as to avoid bias. Difficulty level of the procedures was appropriate for the level of training of students. Students in the first year of training were assessed for the techniques of spinal anaesthesia, insertion of laryngeal mask airway and endotracheal intubation. Those in the second year of training were assessed for lumbar epidural, paediatric and nasotracheal intubation. DOPS for the third year PG students was conducted for central venous catheterisation, arterial cannulation and ultrasound-guided nerve blocks.

Table 1.

Direct observation of procedural skills (DOPS) form

Procedure	Endotracheal intubation
Patient details
Degree of difficulty	Low	Time taken for procedure
	Moderate
	High
Reason for difficulty		Time taken for feedback
Time pressure	Elective	Number of times procedure previously completed by student
	Emergency
Observation
Knowledge	Demonstrates relevant basic science/clinical knowledge and understanding of the procedure including indications, contraindications, anatomy, technique, side effects and complications
	Airway anatomy
	Airway evaluation
	Mallampati classification
	Endotracheal tube
	Indications/contraindications
Consent	Explains procedure to the patient and obtains valid and adequate informed consent
Preparation	Prepares OR appropriately for procedure
	Ensures assisting staff are present
	Checks machine
	Monitors
	Prepares drugs
	Keeps appropriate size laryngoscope/tube/airway/LMA
	Suction
Vigilance	Demonstrates situational awareness through constant clinical and electronic monitoring.
	Maintains focus on the patient and avoids distraction
	Identify SpO2 based on tone modulation
	Look for EtCO2 curve
	Looks for chest expansion
Infection control	Demonstrates aseptic/clean technique
	Standard (universal) precautions
	Use of gloves
Technical ability	Demonstrates manual dexterity and confidence
	Able to perform mask ventilation
	Keeps position of the head correctly
	Introduction of DL scope into the oral cavity/LMA
	Correct procedural sequence with minimal hesitation and unnecessary actions
Patient interaction	Provides reassurance and checks for discomfort, concerns and complications
Insight	Ability to identify difficulty
	Ask for extra aids for intubation (bougie, stylet, McCoy, External laryngeal manipulation)
	Knows when to seek assistance
	Abandon procedure or arrange alternative care to prevent harm to patient
Documentation/post procedure management	Confirm tube position by auscultation/EtCO2
	Fixing tube appropriately
	Start IPPV
	Check ventilator setting
	Start appropriate FGF and inhalational agent
	Documents the episode including problems and complications; arranges and documents plans for post procedural care
Team interaction	Provides clear and concise instructions to assisting staff/paramedics
	Conveys relevant information concerning the patient
	Explains the plans to team members
Overall feedback
What was done well
Areas that need improvement
Trainee reflection
Trainee comments
Trainee action plan
Outcome
DOPS Score (1 to 9)
Competence	Need constant supervision and assistance
	Needs minimal assistance
	Can perform independently
Date of assessment
Trainee name
Trainee email
Assessor name
Assessor email

OR: Operation room, LMA: Laryngeal mask airway, SpO2: Oxygen saturation, EtCO2: End-tidal carbon dioxide, DLscope: Direct laryngoscope, IPPV: Intermittent positive pressure ventilation, FGF: Fresh gas flow

Actual encounter with a patient was observed by the assessor from the time of pre-anaesthetic review till the end of the anaesthetic procedure in OR. Observation was not limited to the technique but included various aspects namely knowledge, consent, preparation, vigilance, infection control, technical ability, patient interaction, insight, documentation and team interaction. On completion of the task, once the patient was stabilised, feedback was given in the side room of OR to maintain privacy and confidentiality, while another PG student would monitor the patient. Specific feedback using the checklist as the basis was provided to students by the observer. This process took around 20-30 minutes.

A questionnaire comprising of closed-ended and open-ended questions exploring participants’ perceptions regarding DOPS was prepared and was reviewed by three faculty members from the Department of Medical Education and approved by IEC of our institute. Once the DOPS interventions were completed, this pre-validated questionnaire was given to students and faculty [Tables 2 and 3]. The questionnaire had a number of statements regarding the use, structure, nature, feedback and potential of DOPS. Participants were asked to rate their responses on a three point Likert scale ranging from agree to not sure to disagree. The questionnaire for PG students had 10 while the one for faculty had 12 closed-ended questions. They were also asked open ended questions about their experience of DOPS, views regarding the role of DOPS as teaching-learning tool, usefulness of feedback, perceived limitations, modifications for improved outcome and feasibility of inclusion into routine work.

Table 2.

PG students’ response to closed ended questions on Likert scale

Closed-ended questionnaire	Participant response number (percentage)
	Agree	Not sure	Disagree
DOPS improved my procedural skills	10 (83%)	2 (17%)
Feedback was constructive	12 (100%)
Time given for feedback was adequate	9 (75%)	3 (25%)
I was given opportunity to put my views	10 (83%)	2 (17%)
The exercise motivated me	10 (83%)	2 (17%)
Observation adversely affected my performance		6 (50%)	6 (50%)
Satisfied with the exercise	10 (83%)	2 (17%)
DOPS is an effective teaching learning tool	10 (83%)	2 (17%)
DOPS creates opportunities for learning	11 (92%)	1 (8%)
DOPS improves student-teacher relationship	8 (67%)	4 (33%)

Table 3.

Faculty response to closed ended questions on Likert scale

Closed-ended questionnaire	Participant response {number (percentage)}
	Agree	Not sure	Disagree
DOPS improved my attitude towards resident training	10 (100%)
DOPS assesses more aspects of procedural skills	9 (90%)		1 (10%)
Prior faculty training is necessary to conduct DOPS	6 (60%)	2 (20%)	2 (20%)
DOPS requires more commitment and time	5 (50%)	1 (10%)	4 (40%)
DOPS is an effective teaching learning tool	10 (100%)
DOPS is easy to carry out	9 (90%)	1 (10%)
DOPS can be included in formative assessment	9 (90%)	1 (10%)
DOPS describes criteria for reference	7 (70%)	3 (30%)
DOPS identifies developmental needs of students	10 (100%)
DOPS creates opportunities for learning	8 (80%)	2 (20%)
DOPS improves student-teacher relationship	5 (50%)	5 (50%)
Satisfied with the exercise	8 (80%)	2 (20%)

The quantitative data were entered into an excel spreadsheet and descriptive statistics of perceptions was used to calculate percentages. Thematic analysis was done for analysing the qualitative data by identifying common themes from the responses of the participants to the open-ended questions.

RESULTS

Most of the perceptions of students were found to be positive. All the PG students were of the opinion that feedback was constructive. 10 students perceived DOPS as an effective teaching learning tool, a motivational exercise which helped to improve their procedural skills and put forth their views, and were satisfied with the same. 11 students felt that DOPS had a potential to create more opportunities for learning. Time for feedback was considered adequate by nine students. Eight students felt that DOPS can improve student-teacher relationship. Six students opined that observation does not affect the performance while the remaining six students were unsure [Table 2 and Figure 1a].

Figure 1.

(a) Perceptions of PG students, (b) Perceptions of faculty

All the participating faculty members agreed that DOPS improved their attitude towards teaching, and perceived it as an effective teaching-learning tool that helps to identify the developmental needs of the PG students. Nine faculty members felt that DOPS can assess more aspects of procedural skills compare to the traditional methods and that it can be a part of formative assessment. They found DOPS easy to carry out. DOPS was perceived by eight faculty members as a satisfactory tool that can create more opportunities for learning. 50% of the faculty members felt that DOPS requires more time and commitment and has a positive impact on student-teacher relationship. Seven faculty agreed that DOPS clearly describes the criteria for reference [Table 3 and Figure 1b].

Thematic analysis of the qualitative data was done and the responses to open ended questions were analysed to interpret the patterns of meaning. The themes identified for faculty and students along with the common responses from which they were derived have been presented [Tables 4 and 5].

Table 4.

Themes identified from student responses to open-ended questions

DOPS improves learning

I was motivated to make a mental check-list of all the steps.

It was useful as all steps were observed and detailed feedback was given.

I could recollect all steps involved in the procedure.

DOPS gave an overall picture of the procedure.

DOPS helps in PG training.

More effective than the traditional method.

Observations made us more careful and alert.

DOPS provides a focus for learning

DOPS helped to identify my strengths and weaknesses.

DOPS makes us aware of our mistakes and deficiencies are pointed out.

Corrective measures were suggested.

Constructive feedback was provided.

Systematic and timely feedback was helpful.

Student-friendly atmosphere and interactive nature of feedback is helpful.

I was given an opportunity to put forth my views.

Assessment was done in a calm student-friendly environment.

During feedback my plan for improvement was discussed.

DOPS prepares PG students for professional practice.

It helped me understand how to work under pressure.

DOPS improved confidence level.

It helped in being more responsible towards patient care.

DOPS gives an overall picture of the procedure

DOPS may not correctly reflect the ability to perform.

Performance depends on the mental state of the student on that

particular day.

I became conscious and had exam fear as I was being assessed.

As I was not oriented to DOPS I could not perform well.

Assessor variability is a limitation of DOPS.

Performance depends on the mental state of the student on that particular day.

I became conscious and had exam fear as I was being assessed.

As I was not oriented to DOPS I could not perform well.

DOPS requires planning and scheduling

Different teachers may stress on different points.

Observer variability exists, same assessor should follow up the future assessments for the same procedure

Table 5.

Themes identified from faculty responses to open-ended questions

DOPS is a comprehensive assessment

DOPS covers all details related to procedural skills which includes pre and post procedural period

What we tend to overlook is brought to notice

DOPS covers multiple aspects

DOPS helps faculty in identifying gaps in performance

DOPS helps to identify lacunae requiring improvement

Detailed checklist helps to detect deficiencies

Helps to analyse mistakes

Helps to review students

DOPS is a good teaching learning method

Better understanding of procedure

Constructive inputs given to students

Clearly defines goals and all the sub steps involved in the procedure

DOPS can help in improving procedural skills of students

Effective and systematic approach is helpful

Active involvement of students during procedure as they are being assessed leads to better performance

It has better impact than traditional methods

Practical difficulties encountered need to be looked into.

DOPS is time consuming, hence cannot be performed in high risk patients and short procedures.

Faculty orientation & training is required to conduct DOPS in standardised manner.

We need to involve all faculties so that DOPS can be conducted easily.

Meticulous planning and scheduling is required

DOPS should be included in PG training programme

DOPS should be conducted at regular intervals.

Periodic feedback should be given to the students.

Student performance log books need to be maintained.

DISCUSSION

There has been a paradigm shift in medical education towards competency-based training supplementing the conventional knowledge-based curriculum. This concept is driven by concerns regarding patient safety and stresses on the importance of achieving expertise in skills.[5,6] We need to ensure that PG students have achieved competence and proficiency during training program for safe clinical practice. Procedural skills mandate integration of knowledge, motor skills and behavioural aspects. Cognition, integration and automation are the stages that a trainee passes through for acquisition of expertise.[5]

Assessment should serve the purpose of providing evidence of competence appropriate for the level of training and determine if the student is fit for professional practice.[3,5,7] However, conventional method of assessment in India focuses on the cognitive domain primarily assessing the acquisition of knowledge and limited importance has been given to assessment of procedural skills. This is a major drawback of the assessment system in our training programme.[3,8]

Traditionally, log books with a list of procedures performed have been used as an indicator of procedural competence. But, merely carrying out various procedures a prescribed number of times cannot ensure that the student is adequately skilled.[9] Moreover, this is self-reported and may be inaccurate. Usually, informal feedback is given by the supervising consultant which mainly focuses on technical skill. Other aspects such as context, decision making and communication skills may not be taken into consideration and feedback may vary between assessors due to an element of subjectivity involved.[3]

Many methods have been described for assessment of procedural skills such as procedure lists, objective structured clinical examination (OSCE) stations with mannequins, cumulative sum analysis, DOPS and global rating scales. OSCE is a feasible option, but its reliability is found to be lower compared to DOPS. Also, it is a time consuming exercise with compartmentalisation of assessment. Simulation offers a safe environment, assessing team work, communication skills and high cognitive skills. However, it is expensive, it still differs from actual patient care and there is no proven benefit.[3,5,7] Newer methods like motion analysis, psychometric and aptitude testing also have been tried.

DOPS, a type of WPBA, formative in nature, has been widely employed in western countries.[10] It was developed by the Royal College of Physicians (Norcini & McKinkley 2007), and now forms a part of specialist training by the Royal College of Anaesthesia. DOPS targets the highest level of Miller's triangular framework for clinical assessment.[5,7,11] It is found to have high reliability, good validity and objectivity. It is easy to use with good feasibility but requires training for optimal reliability.[2,3,5,6]

DOPS is a criterion referenced assessment that evaluates cognitive, psychomotor and affective domains of performing a procedure. The predefined criteria include understanding of indications, obtaining informed consent, appropriate preparation, technical ability, aseptic technique, awareness of complications, post-procedure management, communication skills, overall ability and clinical competence.[5,10,11] Faculty were of the opinion that the checklist clearly defined the goals and the steps brought the attention of the trainees to the essential skills required to perform a procedure. Thus, DOPS was perceived as a comprehensive assessment tool with a holistic approach as explained by Norcini et al.[11]

In our study, DOPS was perceived as a more effective teaching-learning tool compared to conventional methods. Faculty members thought that it improved their attitude towards resident training. Students and faculty felt that it resulted in better understanding of the procedures due to systematic approach. Deconstruction of steps helped to cover all aspects giving a complete picture of the procedure. This in turn, made teaching easier by clealy defining the expected performance by the students. The students were impelled to create a mental checklist that aided in performing a procedure in a standardised manner. Educational impact of DOPS has been explored by several other investigators. Improved training and tutoring has been reported which results in deep learning and reflective practices.[9,12,13] This in turn, promotes self-directed and life-long learning that is essential in the medical field. A significant improvement in performance by students across specialities was observed in multiple studies and was concluded to be an impact of DOPS.[9,13,14,15,16,17]

Formative assessment (FA) is an assessment for learning unlike summative assessment. The goal of FA is to monitor progress, with the aim of improving performance and building a foundation for learning more advanced skills.[3] In our study, faculty members agreed that DOPS can be a tool for formative assessment. In their opinion, DOPS helped to detect the gaps in student performance. It also provided a focus for learning by identifying strengths and weaknesses. Analysis of performance along with constructive feedback helped to formulate a plan for improvement. DOPS was perceived as motivating exercise which can further enhance clinical skills. These opinions are similar to several other studies regarding DOPS.[11,14]

In the current study, the feedback focused on what was done well by the students and areas that needed improvement. The assessors used a structured checklist for reference which helped in giving specific feedback. The trainees appreciated the interactive nature and student-friendly atmosphere which encouraged active involvement. They analysed their own performance, reflected on the experience and shared own views during the discussion. Feedback was perceived to be helpful as it was delivered immediately and adequate time was dedicated for the same. These responses were corroborated by other studies where the opportunity to receive feedback was appreciated as constructive criticism.[6,8,9,11,17]

Faculty were of the opinion that DOPS should be conducted at periodic intervals with maintenance of student performance log. Incorporating DOPS into PG curriculum would be a step towards competency based medical education for a better outcome. The students felt that the DOPS exercise prepared them to work under stress and made them aware of their responsibility which in turn would tune them for professional practice. These views are in agreement with earlier research.[8,10,18]

Being a teaching institute, our students are used to supervision. The only difference was that all the steps were surveyed methodically during DOPS. The mere thought of being assessed made some students conscious and stressed. Probably this made 50% students think that observation had negative impact on performance whereas others felt it made them more alert and careful. It has been a concern for students in other studies also which found observation stressful and artificial.[6,8,13,14,16,19]

Our study reported a high satisfaction rate of 80% by faculty and 83% by students. These results were in agreement with several other studies.[10,17,20,21] The level of satisfaction has been related to the time duration dedicated to DOPS. Bindal et al. reported dissatisfaction by Anaesthesiology residents who perceived DOPS as a tick box exercise.[19] This was attributed to the fact that the time dedicated for feedback was less (<15 min) and occurred beyond working hours due to busy schedule. Although conducted by consultants, they had not received prior training for the same. We feel that adequate time, opportunity to put forth their views and non-threatening atmosphere would have contributed to the satisfaction in our study.

Few limitations of DOPS were highlighted in our study such as time constraints, subjectivity, assessor bias and lack of training, affecting quality of assessment. Busy OT list, high-risk patients, unplanned emergency cases, postings for anaesthesia in remote locations were recognised as some of the contributing factors for difficult implementation. Although easy to carry out, need for orientation and training of assessors was brought out. Similar limitations have been highlighted by other investigators.[6,8,9,10,14,16] We feel that meticulous planning, scheduling and involving all the faculty members are mandatory to make DOPS feasible as a departmental policy.[6]

DOPS provides limited assessement of cognitive domain. For intellectual skills such as decision making, critical thinking and situational awareness, other tools like simulation, case-based discussion, chart review or long case evaluation need to be employed.[1,7] We feel, this was not highlighted in the current study as it focused on procedural skills.

The main limitation of the present study is a small sample size due to less number of PG students available in OR in the stipulated period. Assessment of impact of DOPS on learning needs evaluation of retention of knowledge and skills by students which requires long-term periodic assessments using Global score scale (GSS). Although we had GSS in place, it was not feasible due to time constraints as the study period was 4 months.

As a future recommendation, DOPS can be introduced at the entry level of PG training program for a better outcome. Periodic assessment with a regular follow up will provide an objective evidence of learning. The results of assessment can be utilised to make a portfolio for the students. The process can be made online including scores and feedback given for future reference.

CONCLUSION

Formative assessment of procedural skills during PG training programme is important to ensure competence. DOPS is perceived as an effective assessment and teaching-learning tool by PG students as well as faculty. Its implementation is feasible with prior training, meticulous planning and scheduling. Inclusion of DOPS into PG Anaesthesiology curriculum will serve as a step towards competency-based training.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.