Creating and confirming observable professional activities (OPAs): A brief report on the practical approach for OPA design for resident education

Abstract

Context

The transition of graduate medical education to competency-based education systems has resulted in exploration of the efficacy of Entrustable Professional Activities (EPAs) and related Observable Practice Activities (OPAs) as evaluation tools. EPAs were introduced to PM&R in 2017, but no OPAs have been reported for a non-procedurally based EPA. The primary aims of this study were to create and form consensus on OPAs for the Spinal Cord Injury EPA.

Methods

A Modified Delphi panel of seven experts in the field was utilized to gain consensus on ten PM&R OPAs for the Spinal Cord Injury EPA.

Results

After the first round of evaluations, most OPAs were evaluated by experts as requiring modifications (30/70 votes to keep, 34/70 votes to modify) with a majority of comments focusing on the specific content of the OPAs. Edits were made, and after the second round, the OPAs were evaluated and determined to be kept (62/70 votes to keep, 6/70 votes to modify) with most edits being about semantics of the OPAs. Ultimately, there was significant difference in all three categories between round 1 and round 2 (P < 0.0001) and 10 OPAs were finalized for use.

Conclusions

This study created 10 OPAs that can potentially help provide targeted feedback to residents on their competency in caring for patients with spinal cord injury. With regular usage, OPAs are designed to provide residents with insight into how they are progressing towards independent practice. In the future, studies should aim to assess the feasibility and utility of implementing the newly developed OPAs.

Introduction

Despite the broad use of Entrustable Professional Activities (EPAs) throughout graduate medical education (residency training), there are limited investigations into the related Observable Practice Activities (OPAs). As a result, educators lack appropriate methods to provide timely evaluation and feedback to residents using OPAs. Though previous studies have demonstrated the utility of OPAs in procedural PM&R rotations (1, 2), OPAs for non-procedural PM&R rotations have not been reported. Educators require effective evaluation tools to successfully evaluate and provide competency-based feedback for learners.

As competency-based education gained traction in graduate medical education (3, 4), a tool to evaluate this new system was needed. Competencies are qualities (i.e. medical knowledge or interpersonal skills) which are inherently difficult to evaluate. EPAs and related OPAs were introduced as competency-based evaluation tools (5, 6). EPAs are holistic aspects of medical practice (activities) that translate competencies (qualities) into an objective, evaluable unit (7). However, medicine is complex and there are some activities that are too broad to be evaluated with a single activity (i.e. managing a patient with spinal cord injury), so OPAs were created to be a more discrete task that when combined with related OPAs can be used assess the EPA (6). Based on how a trainee performs, the evaluator uses an entrustability scale to determine how much supervision is required for the task (8). This can then be used to determine when and if a resident is ready to progress to more advanced tasks and greater levels of independence. Though there are residency programs domestically and abroad that utilize them in resident evaluation (9–13), EPAs were only introduced to Physical Medicine and Rehabilitation (PM&R) in the United States of America in 2017 as a way to provide formative and summative feedback to residents (2). Subsequently, in 2020, physiatrists created OPAs for the PM&R EPA on electrodiagnostics (EDX) and assessed the feasibility of this tool (1).

In their study, Baer et al mentioned that EDX was selected because procedurally based activities were perceived to be easier to assess with OPAs. Since that time, OPAs for the other PM&R EPAs have not been reported. Though procedurally based OPAs may be inherently easier to assess, a majority of PM&R education is non-procedurally based and there is a paucity of resident competency evaluation tools despite the demonstration of their success in other clinically based specialties (6, 14).

This study aims to take the next step in assessment of PM&R OPA feasibility by (1) developing OPAs for a non-procedural EPA and (2) forming consensus on the content of the OPAs. Specifically, OPAs were designed for the Spinal Cord Injury (SCI) EPA, “Evaluating and managing patients with spinal cord injury and other spinal cord disorders including secondary conditions and complications” (2).

Methods

Developing the OPAs

An initial list of OPAs was created by the authors utilizing the rotation objectives for the SCI Inpatient rotation (Supplemental Appendix 1), ACGME milestones for PM&R (15), previous publications on OPA development in graduate medical education (13, 14, 16), as well as personal expertise in developing resident curriculum.

Content consensus of the OPAs

As utilized in previous studies (1, 2, 9, 12, 13), a modified Delphi method was employed to finalize the OPAs. This system uses a semi-structured format to allow for quantitative assessment by experts to create a final consensus. Our panel consisted of seven board certified spinal cord injury medicine physicians who completed their training at United States based programs. Currently, participants worked at 3 different ACGME-accredited PM&R residency programs (n = 6) and in private practice internationally (n = 1). Investigators selected this panel based on their knowledge of spinal cord injury medicine, familiarity with resident education, and availability to participate in the process.

In this study, the modified Delphi method consisted of two rounds of edits. In the first round, the experts received the suggested OPAs (via email) and were asked to non-anonymously evaluate them by determining if they should “Keep,” “Keep but modify,” or “Remove” the OPA. If they selected “Keep but modify,” they were instructed to include edits and/or leave comments on how to enhance it. After the first round, the authors edited the OPAs before returning them for a second round of evaluation. The panel received the same instructions for the second round. After both rounds of evaluation, the authors finalized the OPAs.

Results

After reviewing the rotation objectives and PM&R milestones, 10 OPAs were developed and distributed to the expert panel. In each round 100% (7/7) of participants submitted feedback on the 10 OPAs for a total of 70 votes per round (7 total votes for each OPA).

Round 1 review

The evaluation forms were completed with the greatest amount of OPA votes falling under the “Keep but modify” selection (34/70 selection, between 3 and 5 votes per OPA), followed by “keep” (30/70, between 1 and 4 votes per OPA), and “remove” (6/70, between 0 and 2 votes per OPA). After this round, the majority of comments focused on expanding the description of the OPAs and what is specifically being assessed (Fig. 1). Additionally, the panel expressed concern that too many things were being evaluated in a single OPA and requested additional subdivisions of OPA 9, “To diagnose and manage other secondary medical conditions and complications including deep vein thrombosis and pulmonary emboli, stress ulcers, heterotopic ossification, musculoskeletal pain, overuse injuries, neuropathic pain, dysfunctional thermoregulation, syringomyelia, spasticity, sexual dysfunction, and reflex sweating.”

Figure 1.

Results of the Delphi panel: Round 1. Figure visualizes the votes (“keep,” “keep but modify,” or “remove”) made by Delphi panel for each OPA after the first round of evaluation.

Round 2 review

The evaluation forms were completed with a majority of OPAs falling under the “keep” selection (62/70, between 5 and 7 votes per OPA), followed by “keep but modify” (6/70, between 0 and 1 votes per OPA), and “remove” (3/70, between 0 and 1 votes per OPA). In the final round, more of the experts reached consensus on what to keep (Fig. 2). Modifications focused on semantics and wording, not on the concept being covered. One similarity between the two rounds was the concern for the complexity of OPA 9 as described in round one.

Figure 2.

Results of the Delphi panel: Round 2. Figure visualizes the votes (“keep,” “keep but modify,” or “remove”) made by Delphi panel for each OPA after the second round of evaluation.

Final list

After 2 rounds of OPA feedback, Fisher’s exact test and Chi Squared were performed to determine if there was a significant difference between how the experts evaluated the two rounds. Both analyses demonstrated significant differences in all three groups (P < 0.001). Though there was no clear association, there was a greater proportion of submissions to “keep” OPAs. Based on this consensus the authors finalized the 10 OPAs (Table 1) with details to provide context for the evaluator and resident (Table 2 and Supplementary Appendix 2).

Table 1.

Final list of OPAs. List of 10 OPAs created based on feedback of the modified Delphi panel.

OPA	OPA title
1	Perform & collect a Spinal Cord Injury focused History and Physical Exam.
2	Perform & interpret the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) Examination.
3	To assess, classify, and manage neurogenic bladder including long-term management to prevent secondary complications.
4	To assess, classify, and manage neurogenic bowel including long-term management to prevent secondary complications.
5	To assess and manage the respiratory status of patients including preventing secondary complications.
6	To diagnose, manage, and implement strategies to prevent and treat autonomic dysreflexia (AD).
7	To diagnose, manage, and implement strategies to prevent and treat orthostatic hypotension.
8	To identify, manage, and implement strategies to prevent and treat pressure injuries.
9	To diagnose and manage other secondary medical conditions and complications including deep vein thrombosis and pulmonary emboli, stress ulcers, heterotopic ossification, musculoskeletal pain, overuse injuries, neuropathic pain, dysfunctional thermoregulation, syringomyelia, spasticity, sexual dysfunction, and reflex sweating.
10	Oversee and manage the rehabilitation strategies and adaptive tools of a patient with a spinal cord injury.

Table 2.

Detailed versions of OPAs 1 and 2. This demonstrates how in-depth descriptions can provide context and guidance for what the evaluater should consider when determining the level of supervision a learner needs. A list with the full descriptions of all of the OPAs can be found in Supplemental Appendix 2.

OPA	OPA title	OPA description
1	Perform & collect a Spinal Cord Injury focused History and Physical Exam.	Within this context, consider the resident’s ability to: Elicit the etiology and circumstances of the injury, comorbidities, prior functional history, psychosocial considerations, as well as living situation and barriers to accessibility; On exam to identify impairments related to the injury and comorbidities that will affect patient outcomes (including weakness, sensory deficits, spasticity, contractures, impaired respiratory function, presenting signs of heterotopic ossification, autonomic dysfunction, cognitive deficits, and presenting signs of syringomyelia).
2	Perform & interpret the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) Examination.	Within this context, consider the resident’s ability to: Assess and grade motor function; Assess and grade sensory function; Perform anorectal exam, Determine the neurologic level and ASIA impairment Scale grade.

Discussion

The transition to competency-based education requires a change in resident assessment. In several specialties, including PM&R, there are ongoing investigations as to the efficacy of EPAs and OPAs. However, to effectively study their feasibility and utility, it is important for experts in the fields to come to a consensus on what is important for evaluation. Though a previous study created and demonstrated the utility of OPAs in a procedurally based PM&R rotation, this has not been applied to create OPAs in a non-procedural rotation. This study created an assessment form that can be utilized in the assessment of resident competency in caring for patients with spinal cord injury.

Despite the achievements of this study, there are limitations. For one, more experts could have been polled in the development of the OPAs. The authors restricted the number of evaluators to ensure full participation. Inclusion of more experts would have resulted in more logistical challenges to ensure the return of the survey. Second, although OPAs are already a subdivision of EPAs, in the process of making these OPAs, it was evident that they could be broken down even further. This is something that the field will have to address situationally. The benefit of creating more subdivisions allows for a more precise evaluation of a resident’s competency. However, more items to evaluate can become tedious and overwhelming for faculty trying to use the tool.

Though this study created a tool for use in resident evaluation, its utility in practice is unproven. It is imperative that future work assess the feasibility of this tool on a spinal cord injury rotation at an academic institution with residents. Doing this will allow researchers to determine the ease of implementation, how attendings and residents receive OPAs, and if there is any impact on the amount of feedback discussed on clinical rotations. The successful implementation of these OPAs would standardize how PM&R programs evaluate residents across the country because the same tools would be used. An efficacious demonstration of non-procedural OPAs can complement the success of procedural OPAs of EDX. Together, this will provide the opportunity for the PM&R educational community to develop OPAs for the other 17 PM&R EPAs developed by Mallow et al. (2). Once there are established OPAs for each EPA, the OPAs can be mapped onto the ACGME milestones to streamline the interpretation of attending evaluations for program milestone committees.

Conclusion

This study created an assessment form of 10 OPAs that can potentially be utilized in the evaluation of resident competency in caring for patients with SCI. As consensus was achieved by providers at multiple institutions, therefore, it should be generalizable at all programs. If put into practice, evaluators would have a tool that could be used to provide targeted feedback efficiently and effectively to residents. With regular usage, OPAs are designed to provide residents with insight into how they are progressing in their goal to becoming independent providers.

Future studies should look to evaluate the utility of this tool to determine if teachers and learners benefit from this tool and if it enhances SCI education.

Disclaimer statements

Contributors None.

Funding None.

Conflicts of interest Authors have no conflict of interests to declare.