Theoretical Frameworks in Medical Education: Using a Systematic Review of Ophthalmology Education Research to Create a Theory of Change Model

Abstract

Background

Theoretical frameworks provide a lens to examine questions and interpret results; however, they are underutilized in medical education.

Objective

To systematically evaluate the use of theoretical frameworks in ophthalmic medical education and present a theory of change model to guide educational initiatives.

Methods

Six electronic databases were searched for peer-reviewed, English-language studies published between 2016 and 2021 on ophthalmic educational initiatives employing a theoretical framework. Quality of studies was assessed using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach; risk of bias was evaluated using the Medical Education Research Study Quality Instrument (MERSQI) and the Accreditation Council for Graduate Medical Education (ACGME) guidelines for evaluation of assessment methods. Abstracted components of the included studies were used to develop a theory of change model.

Results

The literature search yielded 1661 studies: 666 were duplicates, 834 studies were excluded after abstract review, and 132 after full-text review; 29 studies (19.2%) employing a theoretical framework were included. The theories used most frequently were the Dreyfus model of skill acquisition and Messick's contemporary validity framework. GRADE ratings were predominantly “low,” the average MERSQI score was 10.04, and the ACGME recommendation for all assessment development studies was the lowest recommendation. The theory of change model outlined how educators can select, apply, and evaluate theory-based interventions.

Conclusions

Few ophthalmic medical education studies employed a theoretical framework; their overall rigor was low as assessed by GRADE, MERSQI, and ACGME guidelines. A theory of change model can guide integration of theoretical frameworks into educational initiatives.

Introduction

A theory is a set of logically related propositions that describe relationships among concepts and help explain phenomena.¹ In medical education, theories serve as the basis of theoretical frameworks that provide a lens to explore questions, design initiatives, evaluate outcomes, measure impact, and disseminate findings.² Studies grounded in theory guide best practices and may serve as “clarification” studies that evoke depth of understanding and propel the field forward.^2,3 For example, the Shannon and Weaver Model of Communication has been used to analyze opportunities for error in clinician handoffs,⁴ and Ericsson's deliberate practice theory has been used to design a simulation course to teach advanced life support skills.⁵

However, theoretical frameworks are underutilized in medical education research.^3,6 Many educational initiatives, especially within subspecialty medical education, continue to be developed based on the traditional teacher-apprentice model.^2,7 Lack of theory-based educational initiatives can preclude meaningful interpretation of study methods and results, as theories ground new scholarly work within current literature, allow application of findings to other settings, and provide a framework for adaptation of existing theories or development of new theories.^3,6 Additionally, there is a dearth of studies on the prevalence of theoretical framework usage in subspecialty medical education.⁸

This article has 2 purposes: to systematically review the role of theoretical frameworks in subspecialty medical education, using ophthalmology as an example, and to use the findings to construct a theory of change model⁹ for guiding the development of theory-based graduate medical education curricula. Our primary questions are: What is the prevalence of theoretical framework use in ophthalmic medical education, and how can educators best integrate theory-based educational initiatives? Our findings may benefit educators by highlighting the state of theoretical framework use in subspecialty medical education and by extending these findings into a theory of change model to encourage the more widespread use of theoretical frameworks.

Methods

Search Strategy

A research librarian (L.P.) was consulted to develop a comprehensive search strategy. Following updated Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines¹⁰ on the conduct and reporting of systematic reviews, we searched 6 online databases (PubMed, Embase, Web of Science, CINAHL, PsycINFO, and ERIC) for articles published between January 1, 2016 and January 16, 2021 (Figure 1). We selected a 5-year period prior to the writing of this review to capture current practices in medical education. Our searches included database-specific thesaurus terms, such as medical subject headings (MeSH) and Emtree, as well as keywords relevant to ophthalmic education and theoretical frameworks (online supplementary data).

Figure 1.

PRISMA Flow Diagram

Selection Criteria

Eligibility criteria included peer-reviewed, English-language studies discussing educational initiatives in an ophthalmology setting that employed a theoretical framework at the onset of the initiative. We used the definition of theoretical framework by Varpio et al: “a logically developed and connected set of concepts and premises—developed from one or more theories—that a researcher creates to scaffold a study.”¹ Educational initiatives included development of curricula, learning interventions, training strategies, and evaluation methods (eg, rubrics). We also included studies that referenced initiatives informed by a theoretical framework and studies that assessed learners with clinical evaluation methods, such as rubrics, employing a theoretical framework. We excluded reviews, studies that were not explicitly informed a priori by a theoretical framework, and studies that focused on populations other than medical students, ophthalmology trainees, or ophthalmologists. We also excluded studies that employed best practice models without describing a theoretical framework.

Eligible studies were de-duplicated in EndNote (Clarivate Analytics, Philadelphia, PA) using the method by Bramer et al¹¹ and imported into the systematic review software Covidence (Melbourne, Victoria, Australia) for screening, full-text review, and data extraction. Two reviewers (S.L.S, Z.Z.Y.) conducted abstract screening and full-text review independently and in duplicate, with disagreements arbitrated by the senior author (P.B.G.).

Data Extraction

A data extraction template developed in Covidence was used to extract relevant information, including year of publication, location, study design, characteristics of study participants, sample size, educational initiatives, theoretical frameworks, underlying theories, outcomes, and results. Data extraction was completed independently and in duplicate by 2 reviewers (S.L.S., Z.Z.Y.), with disagreements arbitrated by the senior author (P.B.G.).

Quality Assessment

The Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) guidelines¹² were used to evaluate the overall quality of the studies. The GRADE approach scores quality of evidence based on risk of bias, inconsistency, indirectness, imprecision, and publication bias; studies can be upgraded by demonstrating large effects, plausible confounding, and dose response gradients. The GRADEPro Guideline Development Tool (Evidence Prime, Ontario, Canada) was used to create a GRADE evidence profile for outcomes.

Comprehensive risk of bias (methodological quality) for experimental, quasi-experimental, and observational studies was measured using the Medical Education Research Study Quality Instrument (MERSQI).¹³ MERSQI scores medical education studies on 10 questions across 6 domains for a maximum of 18 points.

Comprehensive risk of bias (methodological quality) for studies that developed clinical assessment methods (eg, rubrics) was determined using guidelines developed by the Accreditation Council for Graduate Medical Education (ACGME).¹⁴ Unlike the GRADE standards which evaluate overall quality of studies based on outcomes, the ACGME guidelines are the only published method to date that evaluates quality of clinical assessment methods.¹⁵ Studies are assigned a letter grade ranging from A to C on 6 domains (reliability, validity, ease of use, resources required, ease of interpretation, and educational impact), an overall level of evidence, and an overall recommendation for uptake into a program's evaluation system. All components of quality assessment and risk of bias analysis were completed independently and in duplicate by 2 co-authors (S.L.S., Z.Z.Y.) with disagreements arbitrated by the senior author (P.B.G.). This study adhered to the tenets of the Declaration of Helsinki.

Developing a Theory of Change Model

Theory of change models are commonly used in large-scale projects^16,17 to delineate the steps and interventions needed to achieve a set of long-term outcomes by backwards mapping the required preconditions, assumptions, rationale, and interventions necessary to achieve these outcomes. We used our findings to construct a theory of change model⁹ depicting the steps and resources required for an educational system to develop theory-based initiatives.

Results

Study Selection

A total of 1661 results were identified: 700 from PubMed and 961 from the other electronic databases (Figure 1). After excluding 666 duplicates, 995 potential studies were identified; 834 were excluded following title and abstract screening. We reviewed 161 articles in full. We excluded 10 articles that were not studies or were not ophthalmology specific. Of the remaining 151 articles discussing educational initiatives in ophthalmology research, 29 (19.2%) were explicitly informed by a theoretical framework and made up the final analytic sample.

Study Quality

According to the GRADE approach for rating certainty of outcomes, 10 outcomes were rated as “very low” certainty, 7 were rated as “low” certainty, 1 as “moderate” certainty, and 1 as “high” certainty; this is consistent with reported ratings for non-randomized studies.¹² The online supplementary data contain a GRADE evidence table for the 7 most important outcomes, rated by 3 authors (S.L.S., Z.Z.Y., P.B.G.) using the GRADE guidelines.

The average MERSQI score for all applicable studies was 10.04 out of 18 points; by comparison, recently published mean MERSQI scores ranged from 9.05 to 12 in other surgical subspecialties.^18,19 The online supplementary data list MERSQI scores for each applicable study.

For studies that developed clinical assessment methods, overall ACGME guideline scores were mixed for reliability, relatively high for validity, high for ease of use, very high for resources, relatively high for ease of interpretation, and unclear for educational impact. In the absence of large-scale studies or randomized trials, the overall recommendation for all applicable studies was judged as “Class 3” (provisional usage as a component of a program's evaluation system), the lowest rating. These scores are consistent with other reviews of clinical skill assessment methods.^15,20 The online supplementary data list ACGME guideline ratings for clinical assessment development studies.

Characteristics of Included Studies and Interventions

The most common study types were prospective cohort^21-29 and cross-sectional.^30-36 Studies were most commonly conducted in the United States,^27,37-42 Denmark,^23,28,36,43 and the United Kingdom.^25,26,44,45 Only 7 studies^{22,26,31,32,35,44,46} had sample sizes over 50 participants (range 52-311). Ten studies^{22,29,34,36,38,44,45,47-49} included attending ophthalmologists, 9 studies^{21,27,30,33,37,39-42} included residents, 6 studies^{24,26,31,32,35,46} included medical students, and 4 studies^23,25,28,43 included a mixed selection of participants. Seven educational intervention studies were conducted in a hospital/clinic,^{21,22,29,32,37,39,40} 6 in an in-person classroom,^{24,30,31,35,41,46} 9 in simulation centers,^{23,25,28,33,36,40,42,44,45} 1 in a virtual classroom,²⁶ and 1 at an academic conference.³⁴ Table 1 contains characteristics of the included studies.

Table 1.

Characteristics of Included Studies

First Author (Year)	Country	Participants; Setting (If Applicable)	Study Type	Theoretical Framework and Methodology (n=Sample Size)	Theory or Theories (Reason for Use, If Reported)
Aslan (2018)21	Turkey	Residents; hospital/clinic	Observational	To evaluate learning curve of non-penetrating glaucoma surgery in a “humanistic” training model informed by humanistic theory (simulation models) (n=1) vs “conventional” training model (books, videos) (n=2)	Humanistic theory (simulation may improve surgical ability)
Atta (2018)31	Saudi Arabia	Medical students; in-person classroom	Cross-sectional	To assess the effectiveness of SDL in comparison PBL (n=60) for an ophthalmology course; performance was compared with that on ENT course	Bloom's taxonomy
Bharucha (2020)22	India	Attendings; hospital/clinic	Prospective cohort	To evaluate skills transfer in a phacoemulsification surgery training program (wet lab training and lectures) by the ICO-OSCAR: Phaco rubric (n=52)	Dreyfus model of skill acquisition
Borboli-Gerogiannis (2019)37	United States	Residents; hospital/clinic	Retrospective cohort	To evaluate the impact of a cataract surgery curriculum on the incidence of intraoperative complications (n=6); compared with residents who graduated prior to intervention (n=16)	Briefing-intraoperative teaching-debriefing model
Dean (2019a)44	United Kingdom	Attendings; simulation center	Assessment development	Developed, tested, and determined whether a surgical competency assessment tool for simulated glaucoma surgery is valid (interobserver reliability: n=4; face validity: n=67; content validity: n=71)	Dreyfus model for skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Dean (2019b)45	United Kingdom	Attendings; simulation center	Assessment development	Developed and tested the validity of a surgical competency assessment tool for simulated small-incision cataract surgery (development: n=8; interrater reliability: n=4; face and content validity: n=12)	Dreyfus model for skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Jacobsen (2020)23	Denmark	Residents, attendings; simulation center	Prospective cohort	To develop and investigate an EyeSi simulator-based test for the more experienced cataract surgeon (n=20)	Messick's contemporary validity framework (gold standard for evaluating validity)
Golnik (2017)38	United States	Attendings	Assessment development	Created a standardized, internationally valid method to help teach and assess vitrectomy (n=unspecified)	Dreyfus model for skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Hassanzadeh (2019)30	Iran	Residents; in-person classroom	Cross-sectional	To assess participant learning styles through Kolb's and VARK questionnaires (n=45)	Kolb's experiential learning theory; Curry's onion ring model of learning (assessing learners' dominant learning styles help educational systems select tailored teaching methods)
Jorgensen (2019)43	Denmark	Medical students, attendings	Assessment development	Developed a theoretical test of proficiency in direct ophthalmoscopy and collected validity evidence using Messick's validity framework (development: n=5; test-takers: n=30)	Messick's contemporary validity framework (to evaluate validity)
Juniat (2018)48	Multinational	Attendings	Assessment development	Developed a rubric for anterior approach ptosis surgery (n=unspecified)	Dreyfus model of skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Kang (2018)39	United States	Residents; hospital/clinic	Prospective case series	To examine the effect of group goal and performance theories on OR efficiency in cataract surgery (n=2); compared to residents pre-intervention (n=2)	Weldon and Weingart group goal and performance theories (to provide a strategy to improve OR efficiency)
Lin (2017)24	China	Medical students; in-person classroom	Prospective cohort	To assess if a flipped classroom model (n=22) is effective compared to a lecture-based classroom (n=22) in ophthalmology	Bloom's taxonomy of cognitive abilities (flipped classroom may allow students to develop higher order cognitive abilities)
McCannel (2017)40	United States	Residents; simulation center and hospital/clinic	Retrospective case series	To assess the specificity of simulation-based ophthalmic cataract surgery training on the EyeSi surgical simulator using the CITC curriculum (n=38)	Motor learning theory (microsurgical motor learning is highly specific with limited skill crossover)
Mishra (2017)42	United States	Residents; simulation center	Randomized controlled trial	To develop a new oculoplastic curriculum that incorporates learning theory of skill acquisition; to develop and evaluate the effectiveness of instructional videos for an oculoplastic surgical wet laboratory (n=16)	Fitts and Posner model for the acquisition and retention of complex motor skills (to guide surgeons through the “cognitive” phase and into the “associative” phase of the model before entering the OR)
Mishra (2018)41	United States	Residents; in-person classroom	Mixed methods	To conduct a needs assessment for communication skills training (cross-sectional) and pilot a communication workshop (prospective cohort) (n=9)	Relational learning framework
Nathoo (2019)32	Canada	Medical students; hospital/clinic	Cross-sectional	To (1) evaluate the undergraduate ophthalmology clerkship experience, and (2) apply educational theories to improve curriculum design (n=311)	Critical realist approach (to emphasize context when evaluating an intervention's effectiveness; demonstrated that varied approaches can still meet clerkship objectives)
Ng (2018)33	Hong Kong	Residents; simulation center	Cross-sectional	To identify residents' perceived barriers to learning phacoemulsification procedures and to evaluate whether virtual reality simulation training changed these perceptions (n=13), compared to residents not taking simulation course (n=6)	Dreyfus model of skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Palis (2021)47	Multinational	Attendings	Assessment development	Revised the OCEX and validated it for international use (n=9)	Dreyfus model of skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Prior Filipe (2020)34	Portugal	Attendings; conference	Cross-sectional	To assess the benefit of a microlearning experience on the improvement of the quality of educators' learning objectives (n=29)	Connectivism; Bloom's taxonomy in intervention
Sahoo (2016)35	Malaysia	Medical students; in-person classroom	Cross-sectional	To find SDL readiness among fourth-year medical students and to analyze the effect of weekly assessment of SDL topics (n=51)	Zimmerman's model of self-regulated learning (to incorporate self-regulated learning through sequential developments)
Sahoo (2018)46	Malaysia	Medical students; in-person classroom	Mixed methods	To analyze the effect of academic writing and journal critiquing in improving critical thinking and collaborative learning (n=188) through qualitative and cross-sectional analysis	Tittle's steps of critical thinking; Bloom's taxonomy (to foster transferable thinking skills in students)
Saleh (2016)25	United Kingdom	Surgical teams; simulation center	Prospective cohort	To explore the feasibility of providing immersive simulation human factors training for ophthalmic surgical teams (n=20 ophthalmology surgical teams: 1 attending ophthalmologist per team)	Gordon's model of systems design and cognitive task analysis; Dickinson and McIntyre model of teamwork (used to evaluate simulation scenarios)
Swaminathan (2016)49	Multinational	Attendings	Assessment development	Developed a standardized, internationally valid tool for teaching and assessing pediatric cataract surgery (n=unspecified)	Dreyfus model of skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Thomsen (2017)36	Denmark	Attendings; simulation center	Cross-sectional	To investigate the correlation in performance of cataract surgery between a virtual reality simulator and real-life surgery using 2 objective assessment tools with evidence of validity (n=11)	Fitts-Posner theory of motor skill acquisition; Messick's contemporary validity framework
Tzoumas (2020)26	United Kingdom	Medical students; virtual classroom	Prospective cohort	To design and implement an e-learning tool to reduce extraneous cognitive overload in undergraduate ophthalmic teaching (n=116)	Cognitive load theory (adherence to cognitive load theory in resource design may improve student satisfaction)
Vagge (2017)27	United States	Residents	Prospective cohort	To investigate the effectiveness of a strabismus surgery training course (n=12)	Dreyfus model of skill acquisition (used ICO-OSCAR, which employed Dreyfus model)
Vergmann (2017)28	Denmark	Medical students, residents, attendings; simulation center	Prospective cohort	To test the validity of the EyeSi surgical simulator as an assessment tool in a virtual reality vitreoretinal training program (n=35)	Messick's contemporary validity framework (to evaluate validity); Gallagher's skills transfer training modality
Yu (2016)29	China	Attendings; hospital/clinic	Prospective cohort	To investigate whether a short-term training program can produce competent cataract surgeons (n=12)	Deliberate practice; Dreyfus model of skill acquisition (used ICO-OSCAR, which employed Dreyfus model)

Abbreviations: SDL, self-directed learning; PBL, problem-based learning; ENT, otolaryngology; ICO-OSCAR, International Council of Ophthalmology–Ophthalmology Surgical Competency Assessment Rubric; ICO-OSCAR: Phaco, International Council of Ophthalmology–Ophthalmology Surgical Competency Assessment Rubric: Phacoemulsification; VARK, visual, aural, read/write, kinesthetic; OR, operating room; CITC, capsulorhexis intensive training curriculum; OCEX, Ophthalmic Clinical Evaluation Exercise.

Theories and Theoretical Frameworks

Studies used a variety of theoretical frameworks (Table 1). The most commonly used theoretical frameworks were the Dreyfus model of skill acquisition,^{22,27,29,33,38,44,45,47-49} Messick's contemporary validity framework,^23,28,36,43 and Bloom's taxonomy of cognitive abilities.^24,31,34,46

Outcomes and Results

Table 2 describes outcomes and study results. Most studies measured components of surgical performance and skill, such as intraoperative complications,^{21,22,29,37,40} surgery performance,^22,27-29,36 aesthetic grade,⁴² surgery completion,²² and surgical efficiency.^39,42 Several studies investigated learning outcomes, such as examination performance,^24,31,43 learning readiness,³⁵ learning style,³⁰ and learning barriers.³³ Nine studies examined components of validity.^{23,28,38,43-45,47-49} Studies also assessed subjective participant evaluation of the initiative; 8 studies^{24,26,27,32,34,41,42,46} surveyed participants, and 1 study surveyed surgical teams.²⁵

Table 2.

Outcome Measures and Study Results

First Author (Year)	Outcome Measures	Results
Aslan (2018)21	Intraoperative complications, intraocular pressure, success rates, trabeculo-descemetic membrane rupture	All outcomes not statistically significant between groups
Atta (2018)31	Performance on multiple choice question examination	Scored significantly higher on PBL ophthalmology vs SDL ophthalmology; significantly lower on SDL ophthalmology vs SDL ENT; significantly lower on SDL ophthalmology vs PBL ENT
Bharucha (2020)22	Pre-training and post-training average mean scores, independent completion rates, complication rates	Complication rate: 12.3%; OSCAR scores correlated significantly with independent completion rates
Borboli-Gerogiannis (2019)37	Intraoperative complications	Significant reduction in posterior capsule tear, vitreous loss, and anterior vitrectomy
Dean (2019a)44	Interobserver reliability, face and content validity, construct validity	Face validity: 4.04/5.00; content validity: 4.00; interobserver reliability >0.60 (Krippendorff's alpha) in 19 of 20 steps
Dean (2019b)45	Face and content validity	Face validity: 4.60/5.00; content validity: 4.5/5.00; interobserver reliability >0.60 (Krippendorff's alpha)
Jacobsen (2020)23	Internal consistency reliability; content, response process, internal, relations, and consequences validity	Internal consistency reliability: Cronbach's alpha=0.63; Differences between groups only significant for the iris expansion ring extraction, level 2
Golnik (2017)38	Face and content validity	ICO-OSCAR: VIT had a degree of face and content validity
Hassanzadeh (2019)30	Learning style	assimilative>convergent>accommodative>divergent (Kolb); auditory>multimodal (VARK)
Jorgensen (2019)43	Content, response, internal, relationship, and consequences validity	High internal consistency for 60 items (Cronbach's alpha=0.95), significantly discriminated groups, correctly established pass/fail score
Juniat (2018)48	Face validity, content validity	The final rubric was internationally agreed upon and demonstrated face and content validity
Kang (2018)39	Room-to-incision, incision-to-close, close-to-exit, room turnover time	Significant differences for room-to-incision time, close-to-exit time, and overall case time
Lin (2017)24	Performance on final examination, survey responses	No statistical differences between 2 groups in either glaucoma scores or ocular trauma scores
McCannel (2017)40	Rate of vitreous loss, retained lens material	Vitreous loss rate and retained lens material similar and not preceded by errant CCC in 86.2% for “CITC done ≥ once,” 57.1% for “no CITC/some EyeSi,” and 48.9% for “none” groups
Mishra (2017)42	Time to completion, overall aesthetic grade	Laceration repair video group had better aesthetic grades than text-alone group. Blepharoplasty video group required more time to complete the task than text-only group
Mishra (2018)41	Questionnaires before and after the workshop	Significant improvement post-workshop in managing emotions during difficult conversations
Nathoo (2019)32	Rotation evaluation	No significant differences between outcomes; inputs and outputs successful but not activities
Ng (2018)33	Perceived difficulty of each step listed by the ICO-OSCAR: Phacoemulsification	Difficulty ranking: nucleus cracking/chopping > capsulorhexis completion > nucleus rotation/manipulation; simulation training significantly associated with lower difficulty scores
Palis (2021)47	Face and content validity	This tool had face and content validity for an international audience
Prior Filipe (2020)34	Microlearning intervention learning survey	Participants agreed that intervention was clear/useful; many intended to use this information
Sahoo (2016)35	SDL readiness scale score	Most students found to be ready for SDL
Sahoo (2018)46	Qualitative analyses of reflections; survey results	Most students found improvement in critical thinking and collaborative learning
Saleh (2016)25	Scores on the NOTSS, NOTECHS, ANTS, OTAS tools for assessing non-technical skills	ANTS had strongest correlation with the other 3 tools; NOTSS provided highest degree of internal consistency; ANTS and NOTSS were best-performing tools
Swaminathan (2016)49	N/A	Behavioral narrative anchors provided objective benchmarks and specific targets for change
Thomsen (2017)36	Simulation-based test score, motion tracking score	Proficiency-based test EyeSi simulator strongly correlated to real-life performance
Tzoumas (2020)26	Module satisfaction	Intervention received improved ratings for quality, utility, reliability, and satisfaction
Vagge (2017)27	Resident perceptions of course; ICO-OSCAR: strabismus scores	Residents felt less anxious after course and found it helpful preparation for surgery; significantly improved in ICO-OSCAR: strabismus mean score after training
Vergmann (2017)28	Module performance scores	Vitreoretinal surgeons had the highest overall median scores in 4 of 6 modules
Yu (2016)29	ICO-OSCAR Phaco score, complication rate	Successfully performed phacoemulsification through wet laboratory exposure, deliberate practice, and frequent formative feedback

Abbreviations: PBL, problem-based learning; SDL, self-directed learning; ENT, otolaryngology; ICO-OSCAR: VIT, International Council of Ophthalmology–Ophthalmology Surgical Competency Assessment Rubric: Vitrectomy; VARK, visual, aural, read/write, kinesthetic; CCC, continuous curvilinear capsulorhexis; CITC, capsulorhexis intensive training curriculum; NOTSS, Non-technical Skills for Surgeons; NOTECHS, Non-technical skills; ANTS, Anesthetists' Non-Technical Skills; OTAS, Observational Teamwork Assessment for Surgery; ICO-OSCAR: Phaco, International Council of Ophthalmology–Ophthalmology Surgical Competency Assessment Rubric: Phacoemulsification.

Theory of Change Model

Our theory of change model (Figure 2) aimed to provide a framework to guide educators in developing, implementing, and evaluating theory-based educational interventions. We abstracted key components of ophthalmic educational initiatives based on theoretical frameworks. Additionally, we analyzed studies that described how theoretical framework usage informed study design to reveal preconditions for designing theory-based initiatives. Given the relative dearth of studies that transparently reported theoretical framework usage, we also referenced literature on theory of change models and the Center for Theory of Change's guidelines⁵⁰ to further inform the development of our model.

Figure 2.

Theory of Change Model

Assumptions that must hold true for developing theory-based interventions successfully include flexibility of the curriculum to accommodate change, educators' willingness to learn about and employ theoretical frameworks, participants' willingness to trial curricular interventions, and administrators' willingness and ability to support educators and participants. Resources include educators, participants, administrators, material resources, educational resources, and data collection systems.

In our hypothetical example of an ophthalmology residency curriculum initiative, an area for curriculum improvement or change must first be identified by analyzing performance trends and summative or formative evaluations or conducting a needs assessment. For example, if 40% of first-year residents in an ophthalmology program scored poorly on their national training examination, educators may be asked to develop an educational intervention to rectify the low scores.

Prior to developing the intervention, educators may undergo training in educational theory to better select and apply theoretical frameworks. Administrators may set aside protected time for learning and make funding available to provide educators with learning resources such as webinars and reading lists. Educators are then better equipped to conduct a literature review and select appropriate theories to inform their intervention. Educators may, for example, select Vygotsky's collaborative learning theory,⁵¹ which suggests that peer-to-peer learning fosters deeper thinking. With administrative support, they may review available resources and plan to dedicate 30 minutes at the end of weekly didactics for resident-led examination practice. After each session, residents may be invited to fill out evaluations on their satisfaction with the initiative within 72 hours.

Intermediate outcomes include satisfaction with the initiative, improved standardized evaluation metrics (eg, proportion of residents who score well on the national examination), and increased number of learning or graduation competencies fulfilled. Long-term outcomes for learners include improved knowledge base and better performance as resident and practicing physicians.⁵² Long-term outcomes for educators include increased use of conceptual frameworks in educational initiatives, which may translate to increased scholarly output and funding.³ Ultimately, achieving these outcomes will support the goal of increasing theory-based educational interventions throughout an educational system.

Finally, the initiative development process is iterative, and performance data may be routinely reviewed to inform future modifications. For example, residents may prefer more timed examination simulations; educators may then reexamine the initiative using another theoretical framework.⁵³

Discussion

The primary purpose of this systematic review was to evaluate theoretical frameworks in subspecialty medical education, using ophthalmology as an example. We found that less than 20% of ophthalmic medical education studies published between 2016 and 2021 were informed by a theoretical framework. When included studies used frameworks, they often named the theory without describing how it framed the research question, informed the methods, or elucidated the results.⁶ Several studies incorporated previously designed theory-based courses or evaluation methods into their medical education initiative but did not further describe the theoretical framework.

Few studies have investigated the prevalence of conceptual frameworks in medicine and surgery. Schwartz et al reviewed the use of conceptual frameworks in the study of duty hours regulations for residents and found that several made contradictory predictions.⁵⁴ Davis et al reviewed the conceptual underpinnings of pediatrics quality-of-life instruments and found that only 7.9% (3 of 38) were based in theory.⁵⁵

Our findings are consistent with other studies investigating use of theoretical frameworks in medical education. A review by Bajpai et al on the use of learning theories in digital health professions education reported that 33.4% (81 of 242) were informed by theory.⁵⁶ Similarly, a review by Hauer et al on behavior change curricula for medical trainees demonstrated that 35.7% (39 of 109) used a theoretical framework.⁵⁷ In addition, a review by van Gaalen et al on gamification in health professions education found that only 15.9% (7 of 44) of studies employed a theoretical framework,⁵⁸ and a review by Leslie et al on faculty development programs in medical education found that only 18.2% (4 of 22) of studies employed a theoretical framework.⁵⁸ Of note, some studies employed different definitions of a theory-based approach,^56,58 and others did not define theory or theoretical framework.^57,59 These discrepancies obscure accurate prevalence data and highlight the need to adhere to a standardized set of definitions.^1,2,53,60-62

The secondary purpose of this systematic review was to use our findings to construct a theory of change model to guide educators in creating theory-based initiatives. Given the complexity and heterogeneity of medical education systems across institutions, theory of change models are excellent tools to map large-scale initiatives, especially those with multiple outcomes.⁵⁰ Our theory of change model illustrated the comprehensive process of selecting, integrating, and evaluating theory-based interventions, including the resources required and the underlying assumptions.

There are several limitations of this study. Our definition of a theoretical framework may differ from that of other studies; there is a need for researchers to adopt standardized definitions of the following terms: theory, theoretical framework, and conceptual framework.¹ We used the definitions by Varpio et al, as they provide the most current model of these terms, informed by literature review, for health professions education research.¹ We also limited our review to studies published between 2016 and 2021 in order to focus our search on current work in ophthalmic medical education⁶; however, this may have masked trends over time. Due to the heterogeneity in study initiatives and outcomes, we were unable to evaluate the efficacy of theoretical framework use and its effects on learner performance. Additionally, given the relatively poor overall quality of included studies and the heterogeneity in reporting use of theoretical frameworks, we were unable to assess the impact of theoretical frameworks on ophthalmic medical education. Moreover, it is possible that effective theoretical frameworks employed in certain study settings (eg, a classroom) may not translate to real-world practical settings (eg, an operating room).

We were also unable to fully determine applicability of many domains of the ACGME guidelines due to ambiguity in their wording; however, reviewers remained consistent in their application of this tool. Other studies have reported similar challenges in evaluating clinical assessment methods, such as evaluation tools for surgical skills, using the ACGME guidelines.^15,20 Further investigation is needed into the efficacy of assessment and evaluation tools for surgical subspecialties. Finally, it is possible that some authors used theoretical frameworks without reporting them or without being consciously aware of using them⁵³; for a study to be included, authors must have reported usage of a theoretical framework or employed a named intervention or methodology based in theory.

Educators interested in designing curricular interventions or longitudinal programs based on theoretical frameworks may benefit from examining questions and results through several “lenses” of theoretical frameworks and using standardized evaluation and assessment systems. In addition, medical educators may consider testing interventions in more than one study setting or institution. Future studies can be improved by transparently reporting theoretical frameworks, including the rationale for selecting a particular framework and how it informed study design and setting.

Conclusions

In summary, theoretical frameworks are underutilized in ophthalmic medical education research, and many studies that employ them do not do so transparently; in the few studies that integrated a theoretical framework, overall study rigor was low as assessed by GRADE, MERSQI, and ACGME guidelines. A theory of change model may guide educators in selecting, applying, and evaluating theory-based initiatives.