A Global Rating Scale and Checklist Instrument for Pediatric Laceration Repair

Abstract

Introduction

Laceration repair is a core procedural skill in which pediatric residents are expected to attain proficiency per the Accreditation Council for Graduate Medical Education. Restricted trainee work hours have decreased clinical opportunities for laceration repair, and simulation may be a modality to fill that clinical gap. There is a therefore a need for objective measures of pediatric resident competence in laceration repair.

Methods

We created a global rating scale and checklist to assess laceration repair in the pediatric emergency department. We adapted the global rating scale from the Objective Structured Assessment of Technical Skills tool used to evaluate surgical residents' technical skills and adapted the checklist from a mastery training checklist related to infant lumbar puncture. We tested both tools in the pediatric emergency department. Eight supervising physicians used the tools to evaluate 30 residents' technical skills in laceration repair. We performed validation testing of both tools in the simulation environment. Based on formal evaluation, we developed a video to train future evaluators on the use of the global rating scale.

Results

The global rating scale and checklist showed fair concordance across reviewers. Both tools received positive feedback from supervising physicians who used them.

Discussion

We found that the global rating scale and checklist are more applicable to formative, rather than summative, training for resident laceration repair. We recommend using these educational tools with trainees in the simulation environment prior to trainees performing laceration repairs on actual patients.

Educational Objectives

By using these tools, facilitators will be able to evaluate learners in the following areas:

• 1.Preparing for a laceration repair procedure, including assembling necessary and appropriate equipment and properly positioning patients.
• 2.Executing the laceration repair procedure completely and efficiently, demonstrating knowledge of each step in the procedure.
• 3.Exhibiting technical proficiency in the laceration repair procedure.

Introduction

The Accreditation Council for Graduate Medical Education has developed milestones for trainees that include procedural or technical skills under the core competency of patient care.¹ The Pediatric Resident Review Committee recommends that laceration repair is a procedure pediatric residents should receive training in and that progress in competency should be monitored by pediatric residency programs.² In a survey of pediatric residency graduates from a large primary and tertiary care teaching hospital, 92% of graduates responded that they were “adequately trained” to perform suturing for laceration repair. However, 39% of respondents reported they had performed less than one laceration repair with suturing, and 27% reported they had never performed a laceration repair.³ Additionally, procedural experience, the traditional evidence of procedural competence provided through procedure logs, has not been shown to ensure competence.⁴ Thus, there is an increasing need for residency programs to objectively evaluate residents' technical skills in laceration repair.

This mandate to evaluate procedural competency arrives in a climate of decreasing opportunities for trainees to practice skills in the clinical setting. Work hour restrictions and changes in medicine have greatly decreased learners' opportunities to learn by doing.⁵ Surveyed pediatric residency directors believe that many residents fail to achieve competence in nine of 13 “very important” procedures, with 36% of directors stating that not all their residents achieve competence in laceration repair.⁶

This has led to the development of alternative modalities for procedural education, such as simulation training and just-in-time procedural practice. To this end, we developed a novel global rating scale and checklist for laceration repair. The global rating scale is based on the Objective Structured Assessment of Technical Skills (OSATS) tool used to evaluate surgical residents' technical skills. Modified OSATS tools have been designed and validated to evaluate trainee performance in neonatal lumbar punctures and obstetrical procedures.^7–9 The modified OSATS tool evaluating competency at neonatal lumbar puncture was validated for “content, response process, and interrater reliability.”⁷ As residents progressed in their training, the OSATS tool reflected their increased competence in lumbar puncture. Similarly, our global rating scale may be useful for formative assessment of pediatric residents' technical skills in laceration repair, particularly for just-in-time training.

The training checklist is used to determine if the trainee properly prepares for and performs laceration repair by checking off whether the trainee independently and correctly performs specified tasks or not. The global rating scale, on the other hand, allows the observer to evaluate how well, on a scale from 1 to 5, the trainee performs various steps in laceration repair. While some researchers have concluded that using global rating scales is preferable to using a checklist,¹⁰ others have demonstrated the value of using both.^11,12 Because of the binary nature of a checklist, it provides “specific, concrete feedback” to the learner.¹³ We propose the use of the checklist in addition to the global rating scale for formative assessment; however, if time is limited, the global rating scale can be used alone. Procedural checklists have been developed to assess learners' skills in various procedures, but no procedural checklist for laceration repair currently exists in the MedEdPORTAL literature to our knowledge.^14–27

Methods

As pediatric emergency medicine (PEM) providers at an academic institution with multiple visiting residents, we know that it can be difficult to assess trainees' procedural skills. We therefore aimed to create a way for supervising PEM physicians (attendings and fellows) to train and assess trainees' technical skills. To do so, we developed a modified OSATS.

While we adapted the present tools from others developed to evaluate surgical skills, the items on our tools were grounded in skills appropriate for the pediatric emergency department. We applied the framework from previously published OSATS tools to develop our tools for laceration repair. The global rating scale was primarily adapted from a previously published and validated surgical global rating scale.¹⁰ Common concepts between surgical procedures and laceration repair procedures, such as efficiency in motion and procedural knowledge, were retained, while concepts unique to surgery were discarded and replaced with skills specific to laceration repair. We adapted the checklist from a previously validated checklist for pediatric lumbar puncture technique.¹⁰ The two checklists were similar in their detailed descriptions of procedural steps as well as their emphasis on the entirety of the procedure, including field preparation and the discarding of sharps. We based specific content points for the laceration repair procedure, such as how to hold and use equipment, skin entry technique, and suture tail length, on the New England Journal of Medicine's video on basic laceration repair.²⁸ We also consulted other sources on pediatric laceration repair to validate the use of specific techniques and add additional content, such as volume of water for wound irrigation.^29,30 Once the tools had been developed, independent PEM physicians reviewed them for content validity and then piloted them in the clinical setting. We used the resulting feedback to further refine the tools.

As in the surgical OSATS, the two assessment tools serve different, complementary purposes. For the checklist, the emphasis is on performing the correct steps of the procedure in the correct sequence. The global rating scale, on the other hand, emphasizes technique. To achieve a high score, a procedure needs to be performed not only correctly but also adeptly. We adapted both the 5-point scale and many of the descriptors used in each item from the original OSATS tool. We based individual items, particularly pre- and postsuture placement, on the basic laceration repair resources described above.

Prior to implementation, we performed a formal assessment of the validity of both the global rating scale and the checklist through video recordings of residents performing simulated laceration repair procedures. We recruited eight emergency and PEM attendings who had an interest in simulation or medical education and who regularly performed and supervised laceration repair in pediatric patients to participate in the validation study as video reviewers. Between January 2016 and November 2016, we filmed 29 pediatric residents and one family medicine resident at a tertiary care center performing simulated laceration repairs on Simulab Tissue Suture Pads (SKU:TSP-10). The residents were of different training levels, including 14 interns, four second-year residents, and 12 third-year residents. Irrigation supplies, suturing tools, and suture material came from repurposed clinical materials. Each video was evaluated by five of the eight emergency or PEM attending physicians using the checklist and the global rating scale for laceration repair. Evaluators were blinded to the identity of the proceduralist. To estimate agreement between the evaluators, we calculated concordance correlation coefficients (CCCs) for both the global rating scale and checklist scores. We calculated average scores for each resident and compared scores across levels of training and laceration repair experience. For each item, we calculated score ranges for each proceduralist and then calculated the median range for each item.

We emailed instructions to supervising physicians regarding how to use the checklist (Appendix A) and the global rating scale (Appendix B) and encouraged them to contact us if questions arose. A list of necessary materials for practicing laceration repair is included in Appendix C. We sent an evaluation form for the global rating scale (Appendix D) to supervising physicians who used the scale. Instructions for the learner, including a clinical scenario, are included in Appendix E. The expectation is that the supervising physician should have knowledge of laceration repair and be considered a technical expert. While evaluators should be familiar with laceration repair, they may need to be oriented to specific criteria of the checklist and global rating scale used to evaluate procedural competency. This can be done via the training video, which was developed based on evaluations of the global rating scale from supervising physicians who used it (Appendix F).

Results

Eight pediatric emergency department and emergency department supervising physicians (attendings and fellows) used the tools to assess video recordings of 30 resident trainees performing laceration repair procedures in the simulation environment. Fourteen interns, four second-year residents, and 12 third-year residents participated in the study. All participants were pediatric residents, with the exception of one family medicine resident.

Our calculated CCCs showed fair concordance across reviewers for both the checklist (0.55; 95% confidence interval [CI], 0.38–0.69) and the global rating scale (0.53; 95% CI, 0.36–0.67). There was no statistically significant difference in either global rating scale or checklist scores by years of training or procedural experience. However, the effect size of years of training as calculated by Cohen's d was moderate for both the global rating scale and the checklist (Table 1 & Table 2). We also noted that the poorest performers (Remedial tier as per Table 3) tended to have low ratings across reviewers.

Table 1. Mean Resident Score for Each Component of the Modified Objective Structured Assessment of Technical Skills by Level of Training.

Component	M (SD)		p	Cohen's d
	Interns (n = 15)	Seniors (n = 15)
Global rating scale (maximum score = 60)	45.3 (7.2)	48.5 (4.0)	.15	0.56
Checklist (maximum score = 18)	12.8 (2.3)	14.2 (1.9)	.09	0.67

Table 2. Change in Resident Score on Each Component of the Modified Objective Structured Assessment of Technical Skills by Procedural Experience.

Component	Estimated Score Increase	95% Confidence Interval	p
Increase in score for every 10 previous laceration repair procedures performed:
Global rating scale (maximum score = 60)	3.0	−0.1, 6.1	.06
Checklist (maximum score = 18)	0.8	−0.4, 1.9	.20

Table 3. Suggested Tiering of Individual Global Rating Scale Scores.

Cutoff	Range	Tier	Mean Question Score at Cutoff	Comment
55	55+	High Performance	4.58	High performance on most items
49	49–54	Good Performance	4.08	High performance on some items
39	39–48	Fair	3.25	Understands the basics but needs oversight on most items
≤38	12–38	Remedial	N/A	Needs oversight/instruction on most items

Supervising users evaluated the tools using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree), with a response rate of six out of 10 users. Most users found the tools to be useful, with a median score of 5.0 for “The rating scales assessed areas that I typically assess when determining trainee level of proficiency with laceration repair.” The median score for determining trainee proficiency and use of the tools for summative assessment was 4.0. The median Likert scale score was 4.5 in ease of using the tools for formative assessment. In general, most users felt the checklist was useful as a framework but wanted more training on use of the tools. Some specific examples of participant responses to our prompts included the following:

• •
  List/describe one or more ways the rating scales will change how you do your job:

  • ○
    “Liked the order of the OSATS checklist and will help me with my mental checklist when teaching trainees.”
  • ○
    “Helpful to have a summary of steps to assess the trainee.”
  • ○
    “Provides a nice framework for discussing proper technique with trainees.”
  • ○
    “Helpful rubric to have steps for performance and for prep.”
  • ○
    “Nice to break down prep and procedure.”
  • ○
    “Helpful with newer trainees.”
• •
  How we could improve the rating scales:

  • ○
    “OSATS is more user friendly and intuitive.”
  • ○
    “Suggest training the trainer on the scales.”
  • ○
    “Consider including the new EPA approach.”
  • ○
    “Technique of holding instruments may vary with similar outcomes.”
  • ○
    “Few items (can't recall which) seem institution specific.”
  • ○
    “Consider combining the tools into one.”
• •
  Comments:

  • ○
    “Although the 5-point checklist is a little more work to use, it allows for more refined feedback.”
  • ○
    “Overall, I preferred the global ratings scale.”

Discussion

We created a novel resource, including a checklist and a global rating scale, to evaluate trainees' competence in laceration repair in the emergency department setting. We adapted an original OSATS grounded in widely accepted techniques for performing simple laceration repair. We chose the OSATS model because of its use with surgical trainees' technical skills and because similar checklists have been utilized with pediatric lumbar puncture.⁷ Once developed, the tools were refined based on the input of various PEM content experts who used them in clinical practice. Originally, we intended the tools to be used for summative assessment; based on feedback and our validity process, we now recommend them for formative rather than summative assessment.

These tools were piloted in the clinical environment and evaluated in the simulation environment. The advantage of using the global rating scale in the clinical setting is that the proceduralist is truly performing the actual procedure, as opposed to a simulation environment, which merely approximates the procedure. However, there are barriers to using these tools in the clinical environment as it may be awkward to complete a checklist in front of a patient/family. We therefore encourage evaluators to complete the forms after the procedure, outside of the patient room, if using them in the clinical setting. There are a number of disadvantages to completing the tools postprocedure, including inability to give real-time feedback and delays in form completion leading to inaccurate assessment. The benefits of using the assessment in the simulation environment include the ability to complete the tools in real time, the opportunity to utilize formative assessment and give feedback during the procedure, and the opportunity for learners to practice discussed skills.

The global rating scale and checklist are useful for providing a formative assessment of trainees' technical skills for laceration repair. They not only help identify which trainees have not yet achieved competence in laceration repair but also reveal specific areas of laceration repair in which a trainee needs further instruction and experience (e.g., learners who scored in the Remedial tier for one reviewer tended to receive similar scores from other reviewers). After using these assessment tools, supervising physicians can then target different steps in laceration repair to maximize learning for individual trainees.

Limitations

Limitations we found in using these tools included recruitment of supervising physicians to use them, difficulty finding time to engage in just-in-time assessments, and limitations in the validity of the scores as described above. Recruiting supervising physicians may be difficult initially, especially when introducing the new tools. However, we found that discussing them at various administrative meetings was helpful in encouraging their use. It can be difficult to find time to engage in just-in-time formative assessment, especially in a busy emergency department; thus, it may be useful to utilize the tools for formative assessment during an emergency department orientation.

A consistent theme of the feedback we received from evaluators was that they desired more training in tool use. When we tested the tools, we did not formally train our trainers, as we hoped the tools were intuitive enough to use in a stand-alone way. It is unclear if training evaluators would result in greater concordance in scores across raters. Nevertheless, to address these concerns, we developed a training video for the global rating scale. We hope that orientation to the scale using this video will allow for more precise evaluation of proceduralists.

Conclusion

In summary, the present tools have varying levels of validity across multiple domains. We believe the tools have a high level of content validity, as they were developed using established sources for procedural evaluation and technique. The tools were reviewed and subjected to modification based on the feedback of content experts. The tools showed less strength in other domains of validity. While learners understood the presented task (response process), there was only fair concordance between raters in scores on formal assessment. There were at least moderate correlations between levels of training and performance on both the global rating scale and checklist, although we did not find these differences to be statistically significant. For these reasons, we recommend the tools be used only for formative feedback, as a way to structure feedback to learners who wish to improve their procedural skills. For the global rating scale, scores were placed into descriptive tiers based on average scores per item (Table 3). If a trainee performs poorly, the skills to perform well can be learned via formative assessment using the tools as a guide.

Prior to the development of this resource, there existed no validated method to evaluate resident laceration repair performance in the emergency department. This resource represents the first iteration of a process to develop valid tools for summative evaluation of laceration repair techniques. Future attempts at tool refinement should look at more formal training for evaluators and at revising the individual items on both the global rating scale and the checklist to allow for more precise evaluation and more consistent scoring across evaluators. However, we believe this resource is a good starting point for more objective evaluation of laceration repair performance and can serve as a basis for directed formative feedback for trainees.

Appendices

A. Laceration Checklist.docx

B. Global Rating Scale.doc

C. Laceration Supplies Checklist.docx

D. GRS Evaluation Form.docx

E. Clinical Scenario.docx

F. Laceration Repair GRS Training Video.mp4

All appendices are peer reviewed as integral parts of the Original Publication.