The Value of 2 Orthopaedic Learning Platforms from the Learners' and Educators' Point of View

Background:

Online learning platforms are a staple of orthopaedic resident education. These platforms typically address a user's knowledge base, aiming to improve OITE and AAOS Board Examination scores. Orthopaedic residents often use these platforms as their primary educational resource. However, an orthopaedic surgeon is more than an orthopaedic knowledge base; acquisition of clinical acumen is integral as well. We sought to investigate the following: From a learner's and educator's perspective, do Orthobullets (OB) and Clinical Classroom (CC) contribute to both knowledge base and clinical acumen?

Methods:

Thirty residents and 16 attending surgeons at a single institution were assigned to review topics and complete questions on either the OB and CC platform. Participants then filled out surveys regarding the platform they were assigned, switched platforms, and completed a final survey. Independent-samples Student t tests and χ² tests were used to analyze differences in continuous and categorical data.

Results:

Residents and attendings reported a preference for OB for fact acquisition, relevance to the OITE, and explanation of answers. Senior residents (PGY5) and attending surgeons reported that CC had a greater impact on their clinical acumen. Junior residents (PGY1, PGY2, and PGY3) reported the opposite. Participants responded that both platforms expand a learner's knowledge base and clinical acumen.

Conclusions:

Learners and educators felt both platforms addressed knowledge base and clinical acumen. Junior residents reported a preference for OB to CC to advance their knowledge base and clinical acumen, but senior residents and attendings felt the opposite was true. Based on survey responses, these platforms were found to be additive, complementary, and that their value to the learner changes during the course of residency education.

Level of Evidence:

III.

Introduction

Online learning platforms have become an integral part of orthopaedic resident education. An ideal learning platform would target multiple elements that contribute to the education and development of a complete orthopaedic surgeon. Two key domains in the education of an orthopaedic surgeon are knowledge base and clinical acumen. It is unclear whether a single learning platform can adequately address both of those domains.

Learning platforms provide numerous resources, including video lectures, high-yield short-hand topic reviews, quizzes, and clinical vignettes in an effort to prepare learners for knowledge-based examinations and, ultimately, for clinical practice. A 2019 nationwide survey reported that Orthobullets (OB) was the most valued resource for orthopaedic surgery resident education¹. Our orthopaedic residency program adopted an OB-based educational curriculum in 2016 and saw enhancements in knowledge base, evidenced by significant improvements in Orthopaedic In-Training Examination (OITE) and American Academy of Orthopaedic Surgeons (AAOS) Board Examination performance. In 2017, another learning platform, JBJS Clinical Classroom (CC), was released with the goal of expanding both knowledge base and clinical judgment². JBJS CC uses a different approach to knowledge-delivery and retention-evaluation through its adaptive learning technology, which individualizes the learner's experience based on content and skills that require additional revision². Since its launch, JBJS CC has become integrated into the curriculum of numerous orthopaedic residency programs.

The aim of this study was to determine whether the OB and CC online learning platforms, in the opinion of learners and educators, contribute to both the knowledge base and clinical acumen domains and if they do, to what extent does each platform target those domains? This study was undertaken to understand learner and educator preferences and to gauge how the implementation of each online learning platform is experienced from the perspective of the residents and orthopaedic surgeon educators (OSE) using them.

Materials and Methods

Orthopaedic surgery residents and OSE from a single orthopaedic residency program were recruited to participate in this survey study. Thirty residents (6 from each class) and 16 OSE participated in the study. The residents and OSE were randomly assigned to begin with either the OB or CC online learning platforms. Randomizing half of the participants to start with OB and the other half to start with CC was intended to eliminate bias had all participants used one platform first. Participants were then asked to read topics and complete practice questions from the assigned learning platform. On completion of the practice questions, the participants were asked to fill out an initial opinion survey (Appendix 1, http://links.lww.com/JBJSOA/A393). The participants were then asked to switch platforms, review topics, and complete practice questions before filling out a second opinion survey (Appendix 2, http://links.lww.com/JBJSOA/A393).

Data were collected over a 2-week period, from August 14, 2021, to August 28, 2021. Survey responses were summarized for the total population and stratified by year of training. Survey questions used a visual analog scale, using a range of 0 to 10 with 0 being the least and 10 being the most. Resident scores and attending scores were analyzed individually and then as a single group.

Significant differences in continuous data among respondents were calculated using an independent-samples Student t test. Differences in frequency of categorical data among respondents were examined using χ² tests. A 2-tailed p value of 0.05 was considered statistically significant. A power analysis was performed using the Statistical Power and Sample Size Calculator (Tempest Technologies). Using a sample size of 32 responses per group would be necessary to detect a 2-point change with an SD of 2, a significance level of 0.05, and a power of 0.8.

Results

Demographics

A total of 30 orthopaedic surgery residents, from one program (100% program participation), completed the surveys. The 2 groups contained 15 residents, with 3 representatives from each class. Resident Group A comprised 4 female residents (26.7%) and 11 male residents (73.3%) and started with the OB pathway. Resident Group B comprised 3 female residents (20%) and 12 male residents (80%) and started with the CC pathway. Sixteen OSE participated in the study, the 8 in OSE Group A started on the OB pathway and 8 in OSE Group B started on the CC pathway. The majority of the attendings (68.8%) who participated were 0 to 10 years into practice. All the participants completed the appropriate second pathway, Group A's finishing with the CC pathway and Group B's finishing with the OB pathway. All 46 participants completed the final survey.

Survey Responses

Survey responses are displayed in Table I. Residents and attendings reported significantly higher scores for the OB user interface than that of CC (8.17 ± 1.45 vs. 6.74 ± 2.17, p = 0.0003). This was also true when resident and attending data were analyzed alone. When asked about the question quality, both residents and attendings reported that there was no difference between OB and CC (p = 0.053). Residents reported a preference for the quality of answer explanations provided by OB as compared to CC (7.50 ± 1.74 vs. 5.97 ± 2.39, p = 0.0061). OSE did not report a preference for the answer explanations provided by OB (8.25 ± 1.84 vs. 7.25 ± 2.02, p = 0.1536).

TABLE I.

Summary of Resident and Attending Surgeon Survey Responses

Questions	Residents			Attendings			All Respondents
	Orthobullets	Clinical Classroom	p/χ2	Orthobullets	Clinical Classroom	p/χ2	Orthobullets	Clinical Classroom	p/χ2
User Interface	7.97 ± 1.43	6.37 ± 2.34	0.0023	8.56 ± 1.48	7.44 ± 1.67	0.0516	8.17 ± 1.45	6.74 ± 2.17	0.0003
Quality of questions	7.10 ± 1.56	6.23 ± 2.50	0.1128	8.06 ± 1.44	7.38 ± 1.67	0.2212	7.43 ± 1.57	6.63 ± 2.29	0.0529
Quality of answer explanations	7.50 ± 1.74	5.97 ± 2.39	0.0061	8.25 ± 1.84	7.25 ± 2.02	0.1536	7.76 ± 1.79	6.41 ± 2.32	0.0025
Information is aimed at clinical practice and the operating room	7.23 ± 1.81	6.53 ± 2.43	0.2113	7.06 ± 2.17	7.94 ± 1.69	0.2137	7.17 ± 1.92	7.02 ± 2.29	0.7305
Information is aimed at fact acquisition	8.10 ± 1.47	6.67 ± 2.12	0.0035	8.88 ± 1.31	7.69 ± 1.30	0.0153	8.37 ± 1.45	7.02 ± 1.93	0.0003
Impact on clinical acumen	7.07 ± 1.62	5.73 ± 2.74	0.116	7.29 ± 1.25	7.56 ± 1.33	0.6866	7.14 ± 1.49	6.42 ± 2.45	0.24
Impact on knowledge base	8.00 ± 1.41	5.87 ± 2.53	0.0081	7.43 ± 1.90	7.33 ± 1.41	0.91	7.82 ± 1.56	6.42 ± 2.26	0.0196
Do you think you will treat your patients better because of _____?	26/30 Yes	23/30 Yes	1.0019	10/16 Yes	10/16 Yes	0	36/46	33/46	0.5217
Relevance for OITE	8.04 ± 1.47	6.27 ± 2.32	0.0008	8.38 ± 1.67	7.69 ± 2.02	0.3028	8.15 ± 1.53	6.76 ± 2.30	0.001

The values in bold indicate statistically significant findings.

CC has a feature that asks users to rate “how confident they feel in their answer choice.” When asked to rate the usefulness of this feature and the learning algorithm associated with it, residents recorded scores of 6.10 ± 2.66, and OSE recorded scores of 6.50 ± 2.61.

Participants were asked “to what extent each platform targeted teaching to prepare learners for clinical practice and the operating room?” Overall, residents scored OB higher than CC (7.23 ± 1.81 vs. 6.53 ± 2.43, p = 0.2113); however, when answers were stratified by year of training, Postgraduate year- (PGY) Y4 and PGY5 residents as well OSE reported the opposite (7.75 ± 2.05 vs. 6.67 ± 1.91; 7.94 ± 1.69 vs. 7.06 ± 2.17, respectively). When asked about the degree to which each platform's information targeted fact acquisition, residents and OSE scored OB higher than CC (8.37 ± 1.45 vs. 7.02 ± 1.93, p = 0.0003). When asked to rate each platform's impact on clinical acumen, residents indicated a slight preference for OB (7.07 ± 1.62 vs. 5.73 ± 2.74). Although not statistically significant, PGY5 residents scored CC higher than OB (8.00 ± 1.41 vs. 6.33 ± 0.94), when stratified by year of training. OSE agreed with PGY5 residents, as evidenced by a 7.56 ± 1.33 vs. 7.29 ± 1.25 (not statistically significant) score in favor of CC's impact on clinical acumen. When asked to rate each platform's impact on knowledge base, residents reported that OB targeted this area more than CC (residents: 8.00 ± 1.41 vs. 5.87 ± 2.53, p = 0.0081). The OSE agreed (attendings: 7.43 ± 1.90 vs. 7.33 ± 1.41, p = 0.91). Although not statistically significant, PGY5 residents felt the opposite was true and scored CC higher than OB (7.67 ± 1.70 vs. 7.00 ± 1.63), when stratified by year of training.

When asked whether each platform would help them better treat their patients, most residents and OSE voted “yes” to OB and CC, and there was no significant difference between the 2 platforms (Table I). Participants were asked to rate each platform's relevance for the OITE. Residents and OSE reported that OB was more relevant to the OITE than CC (8.15 ± 1.53 vs. 6.76 ± 2.30, p = 0.001), but when stratified by year in training, PGY5 residents felt CC was more relevant to the OITE (7.50 ± 1.80 vs. 7.00 ± 1.83).

When asked who they would recommend each platform to, residents recommended both platforms to orthopaedic surgery residents in all stages of their training, with more emphasis on the more junior residents. PGY1 to PGY3 years received more votes than PGY4 and PGY5 for both platforms (Figs. 1-A and 1-B). OSE recommended both platforms to orthopaedic surgery residents in all stages of their training but emphasized OB for the junior residents and CC for the senior residents (Figs. 1-A and 1-B).

Fig. 1.

Fig. 1-A This graph shows which residency year of training respondents would recommend Orthobullets to. Fig. 1-B This graph shows which residency year of training respondents would recommend Clinical Classroom to.

When participants were asked which platform they liked better, most residents indicated their preference for OB (Fig. 2). The most common reason for this preference was a better user interface. Multiple residents reported that the CC learning platform offered better tools for long-term retention with its enhanced learning algorithm design.

Fig. 2.

This graph shows the platform preference of all respondents stratified by year of training.

Discussion

This study sought to evaluate how the OB and CC learning platforms contribute to acquisition of knowledge base and clinical acumen from both learners' and educators' perspectives. In our study, in general, residents and OSE reported a preference for OB over CC for fact acquisition, relevance for OITE success, answer explanations, and contribution to knowledge base. However, and importantly, when the OSE and senior resident surveys were analyzed separately, CC was rated superior to OB for its impact on clinical practice and the operating room and in the platform's ability to address clinical acumen. Although these findings were not statistically significant, the small sample size and limited power may be a factor. However, one important take away from this study is that a learner's needs are dynamic during the course of residency. It is our supposition that OB may be of greater value to help build knowledge base at the beginning of residency, and as residents begin to apply that knowledge base to clinical practice in the later years of residency, the CC learning platform may be of added benefit because it was reported to target different areas than OB. One important component of the CC platform is the adaptive learning technology. This system may help with a user's retention of learned information. The importance of this technology cannot be understated because the overall impact of the learning may be increased with improved retention of information and the automatic ability to focus on a learner's areas of deficiency.

A recent multi-institutional study found that OB had a significantly higher score improvement than CC when comparing pre-test and post-test outcomes. In that study, all residents reported that they felt OB would contribute to their OITE success, whereas only 67% felt the same about CC³. The residents in our study also felt that OB was more relevant for OITE success than CC. However, preparing residents for the OITE and improving a resident's knowledge base is only one area these platforms would ideally target. Clinical acumen is the other domain which these platforms could possibly help teach.

Interestingly, the senior residents and OSE in this study, who presumably have more extensive knowledge base than junior residents, felt that CC had a much greater focus on clinical acumen than OB. This finding was again true when participants were asked which platform focuses more on clinical practice and the operating room. Although these findings were not statistically significant, they demonstrated that more experienced clinicians saw greater utility for CC in its impact on teaching clinical acumen and practice. Although both platforms were reported to address knowledge base and clinical acumen, the results of this study suggest that OB targets knowledge base more heavily, whereas CC focuses on clinical acumen. This may help us understand why junior residents favored OB, whereas more senior residents and attendings favored CC in certain domains.

There were several limitations to this study. Because the data in this study were collected from a single institution and residency program, it may instill bias and limit the generalizability of the results. Residents at our program have participated in an OB-based curriculum for 5 years, and CC was only recently added to our educational program. This likely introduced familiarity bias into the results. A crossover design was used to limit temporal effects of all participants starting with a single learning platform; however, this does not prevent familiarity bias. It should be noted that the CC platform has recently instituted upgrades and changes to its operating platform. As such, the results of this study are only applicable to the platform version available at the time of data collection. Furthermore, the power of the study was limited, and adding more participants and different institutions would improve the generalizability of the study.

This study aimed to assess learners' and educators' perspectives of the OB and CC learning platforms and how each platform contributes to knowledge base and clinical acumen. Both platforms were found to contribute to both domains from the resident's and attending's perspective. Based on the results of this survey, it is felt at our program that these 2 platforms are truly additive and complementary. Quoting from one of the survey responses, “Orthobullets is an invaluable tool which builds a residents knowledge base and Clinical Classroom helps to apply that knowledge to real practice situations.” We will continue to use these tools as complementary learning platforms to improve resident education and prepare residents for practice in our program.

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A393).