Interactive Learning Modules with Visual Feedback Improve Resident Learning in Colposcopy

Abstract

Objective/Purpose:

The Accreditation Council for Graduate Medical Education (ACGME) and the Council on Resident Education in Obstetrics and Gynecology (CREOG) have milestones and/or competencies relating to colposcopy, however the optimal way to reach these objectives are not proscribed and left to individual programs. Here we aim to assess resident skill, confidence levels, perceived level of knowledge and satisfaction with colposcopic training before and after implementation of a new interactive learning module with visual feedback.

Methods:

A new online educational intervention was developed by the author (EN) based on adult learning theory and introduced into our obstetrics and gynecology resident colposcopy curriculum in July 2014. We assessed performance on an objective competency exam administered at baseline and repeated after 6 months of our 24 residents.

Additionally, we assessed resident confidence levels, perceived level of knowledge and satisfaction with training prior and 6 months post intervention.

Results:

Scores on a national online examination improved following the intervention (p= 0.014). Significant improvements on the exam were seen in the sections of medical knowledge (p=0.031) and management (p=0.011). Residents’ perceived knowledge increased significantly after the intervention (P=0.030).

Conclusion:

Learning outcomes improved after introduction of a novel teaching intervention.

Précis:

Introduction of a novel teaching intervention utilizing interactive learning modules and visual feedback results in improvement of learning outcomes for OBGYN residents.

Introduction:

Colposcopy is the most commonly used modality to evaluate abnormal cytology results and triage patients who have a precancerous lesion requiring therapy from those who do not. As a diagnostic tool, colposcopy is important since it serves to guide the biopsy site selection. This histopathologic biopsy result then forms the basis for future treatment recommendations^{1, 2}. Training of residents in colposcopy historically has involved a didactic curriculum (the components of which are variable) and supervised performance of colposcopy. The Accreditation Council for Graduate Medical Education (ACGME) and the Council on Resident Education in Obstetrics and Gynecology (CREOG) have milestones³ and/or competencies⁴ relating to colposcopy, however the optimal way to reach these objectives are not proscribed and left to individual programs.

There are no published studies comparing different teaching modalities for colposcopy so the optimal path to competence is unclear. A curriculum for colposcopy education has been published by the American Society for Colposcopy and Cervical Pathology (ASCCP)⁵ but has not been widely adopted for the purposes of resident education. In a survey of 202 obstetrics and gynecology residency programs, the most common evaluation of resident performance in colposcopy was via direct observation by the supervising faculty (68.8%). A written or visual assessment examination or a basic competency examination was utilized in 35 of 202 obstetrics and gynecology programs (17.3%) and completion of the colposcopy rotation without formal evaluation of colposcopic skills was considered adequate training in 24 of 202 obstetrics and gynecology programs (11.9%)⁶.

In summer 2014, our program designed and implemented a new colposcopy online teaching series to supplement the standard instruction and supervised performance of colposcopy in the clinic. It was hypothesized that these learning tools would improve colposcopic knowledge, confidence and learning outcomes the residents in comparison to the standard clinical training. The objective of this study was to assess the effectiveness of this new online teaching modality to improve resident knowledge of cervical dysplasia and colposcopy.

Methods

Study Design:

A pre/post interventional study design was used to evaluate knowledge, skills, confidence and satisfaction with training before and after a new educational intervention for a cohort of residents.

The primary endpoint for the study was change in knowledge as measured by the RACCE scores from baseline to post-intervention among the entire residency program. Secondary endpoints were 1) change in knowledge measured by RACCE scores from baseline to post-intervention at each level of training; 2) change in knowledge of specific RACCE exam topics; 3) change in confidence in knowledge, colposcopy skills from baseline to post-intervention; 4) satisfaction with the educational intervention.

Development of Educational Intervention:

A novel online teaching series was developed based on Malcolm Knowles’ assumptions on how adults learn (i.e. andragogy- defined as the method and practice of teaching adult learning)^7,8 : (1) Adults are independent and self-directed learners; (2) they value learning that integrates with the demands of their everyday life; (3) they are interested in immediate, problem-centered approaches.

The educational program is comprised of 5 online modules given over 20 weeks. A schedule of opening and closing module dates (4 weeks per module) was provided prior to study initiation and reminders were sent. The modules could be opened and closed as needed to complete learning. Module topics included colposcopic equipment, technique, normal and abnormal colposcopic findings, and treatment modalities. Table 1 illustrates a complete breakdown of module topics. Sample snapshots of online modules are provided in Figure 1 for modules one and two. The modules contained interactive learning platforms including interactive games and puzzles to engage learners, images and examples of colposcopic abnormalities to contextualize new knowledge, and immediate feedback in the form of a short 5question quiz at the end of each module. Quiz scores were based on completion of the module only and were credit/no credit. Completion of the quiz closed the module and assisted in tracking program progress. Due to software design, once a module was completed it was no longer available to the resident. The online educational modules were pilot tested by PGY1 and PGY3 residents, with minor revisions made to the education program prior to program implementation.

Table 1 –

Module Topics

Module 1	Background, equipment and technique
Module 2	Colposcopic findings of the normal transformation zone, variants and low-grade dysplasia findings
Module 3	Colposcopic findings of high-grade dysplasia and malignancy
Module 4	Review of dysplasia treatment modalities
Module 5	Special populations: young, pregnancy or postmenopausal

Figure 1:

Module screenshot samples

Study Participants:

All 24 residents from a single OB/GYN residency program, PGY years 1–4, who received standard colposcopic training program prior to this intervention were included. A convenience sample of residents in the program was used, the residents were not randomized. All residents had access to and used the intervention. The study was reviewed by the University of Texas Health Science Center at San Antonio School of Medicine Institutional Review Board and was designated EXEMPT.

Quantitative Assessment tool:

A national online exam- The American Society for Colposcopy and Cervical Pathology (ASCCP) Resident Assessment of Competency in Colposcopy Examination (RACCE)was selected as the objective assessment tool.⁹ This online examination includes multiple-choice questions, slide identification questions and case history studies. It was designed to assess medical knowledge, diagnostic skills and patient management competencies colposcopy. The questions were developed by expert medical consensus and content is periodically reviewed and updated. The exams are created dynamically, with questions pulled randomly from 16 question pools so each resident receives a slightly different subset of questions which assess the same competencies. This one- hour exam was taken in one sitting by each resident. Scores were collected and reported confidentially as a percentage of questions correct. Each resident received their scores upon completion.

The RACCE online examination was given in the same manner described above at baseline and 6months later after completion of the online educational modules.

Survey assessment:

Anonymous pre- and post- intervention surveys were developed modeled on the works by Skeff¹⁰and colleagues to assess self-perceived confidence, comfort with, and knowledge of skills taught. Our survey asked residents to rate their level of confidence in colposcopy skills, their level of knowledge of colposcopic abnormalities and their level of satisfaction with residency training in colposcopy. The tool was a four- question online survey administered through survey monkey. The first question collected demographic data (PGY year). The remaining three questions asked residents to rate their confidence in colposcopy skills, their perceived level of knowledge and satisfaction with training using a 4- point Likert scale. This survey was administered before and after the educational intervention and results were compared.

Sample size calculation:

The sample size, 24, was a convenience sample based on the number of residents in our obstetrics and gynecology program. One resident did not complete the RACCE exam and their scores were excluded from analysis.

Analysis of scores:

The pre-intervention RACCE scores were matched with the post-invention scores for all residents and then stratified by postgraduate year and four specific exam topics. After checking the distribution of the RACCE scores and three Likert scales for normality, nonparametric tests were selected for the analysis. The paired RACCE scores were analyzed using the Wilcoxon Signed Rank Sum test. A p-value of 0.05 was considered statistically significant and exact probabilities were used. Mann-Whitney tests were used to compare the three individual Likert-scale questions on the pre and post intervention survey. Analysis was performed using STATA/SE version 16 (STATA Statistical Software: Release 16. StatCorp LLC, 2019; College Station, TX).

Role of the Funding source:

There is no funding source for this article. The corresponding author (Erin L. Nelson, MD) had the final responsibility for the decision to submit the manuscript for publication.

Results

Demographics:

The study was conducted between July and December 2014. The resident cohort was composed of 4 males and 20 females. The residents were divided amongst the four years of residency as follows:PGY1 (25%), PGY2 (30%), PGY3 (17.4%) and PGY4 (26%) at the time of study initiation (Fig 2). Completion rate of the RACCE was 100% pre-intervention and 95.8% post-intervention (one resident did not complete the RACCE). The rate of completion for the survey was 100% except for one “no response” for confidence level.

Figure 2:

Demographics of study population

Quantitative assessment scores:

Baseline RACCE scores of our cohort were 51 out of 73.5 possible points for our PGY1 and 2 residents, with the national mean of 49.1. The PGY 3 and 4 residents scored higher than the national average at baseline (59.3 and 59 respectively). Their overall score improvement post-intervention compared to pre-intervention increased significantly (57% vs 62%, P= 0.011) (TABLE 2). There was no significant difference in scores by postgraduate level. Improvements were seen in specific topic areas of medical knowledge (P= 0.031) and management (P=0.011) (TABLE 2). The biopsy placement and colposcopic diagnosis sections had increased knowledge but did not reach statistical significance.

Table 2:

Median Pre/Post RACCE Scores by Topic, 2016

	Median Score (Min, Max)		p*
	Pre-Intervention	Post-Intervention
Overall	57 (32, 73)	62 (43, 78)	0.014
Topic
Medical Knowledge	58 (24, 83)	67 (42, 92)	0.031
Diagnosis	48 (28, 76)	48 (32, 72)	0.256
Biopsy	58 (25, 92)	75 (8, 92)	0.416
Management	60 (24, 88)	68 (44, 88)	0.011

^*p values compare pre and post-intervention scores using Wilcoxon Signed Rank Sum Tests.

Qualitative survey results:

Pre and post intervention survey results were reported for confidence level, knowledge level and satisfaction with training. (Table 3) Prior to the intervention, 26.9% of residents were “not at all” confident in their colposcopy skills”. Post intervention, 4% of residents reported being “not at all” confident. The level of self-assessed knowledge was as “poor” or “fair” for 29% of our residents prior to intervention and 0% post intervention. The level of perceived knowledge increased significantly from 25% residents reporting “fair” pre-intervention vs. 41.7% reporting “above average” post- intervention (p= 0.030). Satisfaction with training showed a trend towards improvement with 38% of residents being “satisfied or very satisfied” prior to the intervention and 63% were “satisfied or very satisfied” post intervention. However, the change in level of satisfaction with training was not statistically significant. (Figure 3)

Table 3:

Institutional Survey and Distribution Item Responses, 2016

Item	Pre-Intervention (%)	Post-Intervention (%)	p*
Confidence in Colposcopic Skill (4-point Likert Scale)
Not at all confident	26.1	4.2	0.210
Somewhat confident	30.4	41.7
Confident	39.1	50.0
Very confident	4.4	4.2
Perceived Knowledge Level (5-point Likert Scale)
Poor	4.2	0.0	0.030
Fair	25.0	0.0
Average	45.8	58.3
Above average	25.0	41.7
Outstanding	0.0	0.0
Satisfaction with Training (4-point Likert Scale)
Dissatisfied	16.7	12.5	0.121
Somewhat dissatisfied	45.8	25.0
Satisfied	33.3	50.0
Very satisfied	4.2	12.5

^*p values compare pre and post-intervention scores using *Mann-Whitney Tests.

Figure 3:

Qualitative survey graphic

DISCUSSION:

Our Results:

Following a novel online educational series, performance on an objective online competency assessment (RACCE) improved in obstetrics and gynecology residents. The highest increase was at the PGY1 level (from a median score of 46.5 preintervention to 65 post intervention). At baseline, our overall resident scores (average 59) were similar or above that of the national average. The national average for this exam was 49.1 out of 73.5 possible points (standard deviation 8.97) the year of the study with similar averages seen the year prior (48.87) and after (49.8) the study. The largest difference in RACCE scores was seen in the PGY1 level, however not significant, following the novel online educational series. It is conceivable that formal colposcopic teaching and learning is most influential in early stages (PGY1) of residency and plateaus thereafter. Given the steep learning curve in the PGY1 year, it is possible these residents are more open and adaptable to feedback. Little improvement was noted in scores for PGY3&4 residents. This may be attributed to high baseline knowledge, as demonstrated by a mean RACCE score >1 standard deviation above the national mean at baseline. Alternatively, the minimal improvements in PGY3&4 residents may suggest current residency training methods are adequately teaching colposcopy skills. Additionally, upper-level residents may be more focused on learning complex surgical skills than colposcopy.

Resident confidence levels and satisfaction with colposcopy training improved following the educational intervention. While changes from pre- to post-intervention were not statistically significant, power was very limited due to the small sample size. The perceived knowledge levels did demonstrate a statistically significant increase. This may reflect the fact the residents were actively engaged during the intervention (which was focused on improving knowledge) and thereby perceived learning had occurred.

Educational Strategy and other literature:

The optimal educational strategy for teaching and mastering colposcopy likely involves a combination of didactic training and hands-on practice with a potential role for a formal assessment at the completion of training to document competency. Though formal colposcopy curricula are available [5] widespread adoption has not occurred, and residency programs decide individually how their residents will learn colposcopy. One similar peer reviewed publication on teaching intervention for colposcopy in resident training was identified. It assessed the effect of a new multidisciplinary curriculum (composed of monthly interdisciplinary conference and weekly small group didactics during ambulatory care block)on resident learning of colposcopy.¹¹ The authors were able to demonstrate high- attendance residents(defined as attending >6 sessions) improved their overall score on the RACCE assessment by 8.7% and management of colposcopy by 11.3%. Level of attendance did not impact scores on the categories of medical knowledge, diagnosis or biopsy placement. The residents in this study had similar baseline scores on the RACCE to our cohort (57–68 versus 59). The overall improvement demonstrated by our cohort is comparable to the 8.7% improvement seen in their high-attendance resident cohort. Like their group, we found no improvement in knowledge related to biopsies or diagnosis, suggesting that clinical experience may be a better mode of education for these skills than didactics.

The online educational intervention created and tested through this study targets learning styles favored by the millennial learner. These include (but are not limited to) team- based learning, flipped classroom, peer- to- peer teaching, sharing via social media (twitter, Facebook, google drive, basecamp), gamification, e-learning and simulation¹². We specifically targeted utilization of an e-learning platform, simulation and gamification to engage our learners. Increasingly in education, we must develop and adapt teaching methods to address the needs of a variety of learning styles.

Strengths and limitations:

This low cost, high yield set of online modules is an easy to use addition to any residency program or to healthcare providers who are new to colposcopy. Additional strengths of this intervention are its interactive nature and online platform making it available 24/7. Additional study strengths include the high level of resident participation in both the intervention and two separate pre- and post-intervention assessments. One assessment tool utilized quantitative (RACCE) data, and the other (institutional survey) qualitative data. Study limitations include the study’s small sample size (based on available number of residents in the program) and single institution. The study design was not randomized, and all residents had access to and used the intervention. We do not have any data on change in RAACE score before and after participation in the standard clinical colposcopy curriculum with which to compare the intervention. The RACCE assessment tool is standardized but not validated and so improvements in performance cannot be correlated with improved clinical outcomes. Based on review of existing resources, however, we felt the RACCE tool to be the best available assessment of colposcopy knowledge. The institutional qualitative assessment tool was designed after previously published work to collect data on confidence levels and satisfaction with training, but it not been validated.

Conclusions and next steps:

While statically significant in several areas, this is a pilot study with a small population, limiting the ability to draw definitive conclusions.

The results suggest a novel web-based education which incorporates visual patterns and features of dysplasia as well as self-assessment and feedback may improve resident learning and knowledge in colposcopy, especially when colposcopic skills are first being developed. To further delineate the intervention’s impact on training and resident perception, it will need to be tested in a larger population to include both programs with and without robust colposcopy curricula.

The next planned steps in our research will be to expand access of the online educational modules to a larger number of training programs to evaluate its performance among a larger sample size with a larger variation in baseline dysplasia and colposcopy knowledge and skills.

List of all Abbreviations and Acronyms

ACGME

Accreditation Council for Graduate Medical Education

ANOVA

Analysis of Variance

ASCCP

American Society for Colposcopy and Cervical Pathology

CREOG

Council on Resident Education in Obstetrics and Gynecology

FIG

figure

IRB

Institutional Review Board

NIH

National Institutes of Health

OB/GYN

Obstetrics and Gynecology

PGY

Postgraduate year (year of residency)

RACCE

Resident Assessment of Competency in Colposcopy Examination