Competency-based Learning

Abstract

Little is known about assessing or improving competency in Papanicolau (Pap) smear sampling among internal medicine residents. We hypothesized that a 3-part targeted resident physician educational program (educational presentation by a knowledgeable instructor, skills workshop, and peer comparison feedback) would be effective in increasing the quality of Pap smears obtained by internal medicine residents. We conducted a randomized, pre-post comparison study over a 16-month period to assess the effect of our educational intervention. We found no difference in baseline adequacy rates. Residents who received the intervention were twice as likely to obtain an adequate Pap smear. Our results suggest that a brief multifaceted intervention designed to improve the frequency with which internal medicine residents obtain endocervical cells while performing Pap smears is effective.

The Papanicolaou (Pap) smear serves as the main screening test for cervical cancer.^1,2 An optimal Pap smear contains endocervical cells as evidence that the smear was taken from the transformation zone. Published reports have shown that an optimal conventional Pap smear has a specificity of 98% and a sensitivity of 51% for detecting cervical cancer and its percursors.^3,4 Studies are limited on the operating characteristics of the new liquid-based techniques; however, published data suggest that this approach improves sensitivity slightly while reducing specificity.^5,6

Some investigators have suggested that the importance of the presence of endocervical cells on a Pap smear has been diminished.^7,8 The 2001 Bethesda System⁸ terminology for reporting results of cervical cytology eliminates the “satisfactory but limited” category of Pap smear specimen adequacy, which was most often used for Pap smears lacking endocervical cells. This term was felt to promote too much repeat testing. However, several studies have found that endocervical cells serve as a reliable indicator of the quality of the Pap smear and their presence improves detection of cervical neoplasm.^2,9,10 The 2002 American Cancer Society's guideline⁶ for early detection of cervical neoplasia and cancer notes that an adequate cervical cytologic specimen involves circumferential sampling of ectocervix adjacent to the transformation zone, the endocervix, and the cervical transformation zone. Additionally, published data suggest that women who develop cervical cancer are more likely to have had Pap smears that were suboptimal, often lacking the endocervical component.^7,9,10 One study estimated that one-half to two-thirds of false-negative results were caused by inadequate specimen collection.¹¹ Therefore, it is critical that proper techniques be used in obtaining Pap smears.

Despite the widespread use and acceptance of Pap smears in cancer prevention, training in proper Pap smear technique in medical school and residency is variable. We found no published curricula at the residency level that teaches adequate Pap smear sampling, although the American Board of Internal Medicine lists obtaining a Pap smear as one of the women's health core competencies that internists should acquire.¹² However, the number of Pap smears obtained by residents may vary markedly between training sites.¹³ Although one survey reported that most academic general internists are confident precepting the pelvic examination and Pap smear,¹⁴ we found no published curricula at the residency level that teach adequate Pap smear sampling.

The specific aim of this study was to determine whether a targeted physician education program would improve the quality of Pap smears obtained by internal medicine residents. Because research in medical education suggests that optimal learning involves the use of different methods of teaching and that a multifaceted approach leads to higher retention of knowledge and skills,^15–17 we hypothesized that a 3-part program (educational presentation by a knowledgeable instructor, skills workshop, and peer comparison feedback) would be effective in increasing the quality of Pap smears obtained by resident physicians.

METHODS

Design and Data Collection

We conducted an Institutional Review Board-approved, randomized, pre-post comparison study over a 16-month period to assess the effect of a targeted resident physician educational program on the quality of Pap smears obtained by residents. A 6-month pre-intervention observation period (November 2001 to May 2002) was followed by the intervention (May 2002: educational presentation and skills workshop; November 2002: peer comparison feedback) and a 10-month evaluation period (June 2002 to March 2003) with a concurrent control group. The intervention was targeted at first- and second-year residents at one university-based internal medicine residency program. Twenty-three residents were randomly assigned to the intervention cohort and 30 residents were assigned to the control cohort. All first- and second-year residents who performed Pap smears during the pre- and postcomparison period were included in the study. We excluded 12 residents who did not complete Pap smears in both periods (6 in the intervention cohort and 6 in the control cohort). Although only one internal medicine department was involved, two practice sites were evaluated: one university practice site (UPS) and one university-affiliated community health center (CHC). Residents are assigned to only one clinic site and different faculty precept at each site. Faculty were refreshed in adequate Pap smear sampling technique, though nurses received no additional training on Pap smear sampling.

To obtain information about Pap smears performed by residents, clinic nurses collected information on the physician performing the Pap smear, the date of collection, the patient's medical record number, and the age of the patient at the time of the examination. Three weeks after the Pap smear was collected, results were obtained from the computerized medical records (Lastword Client for Windows, version 4.1, IDX Systems Corporation, Boston, Mass). These results represented the final cytologic interpretation of the Pap smear.

Both the UPS and the CHC use the liquid-based technique for the procedure. Pap smears obtained at both these sites are sent to the same university cytology laboratory for evaluation. The university cytology department uses the 2001 Bethesda System⁸ to describe the adequacy of Pap smears. Smears are reported as either satisfactory or unsatisfactory for evaluation. A “satisfactory” Pap smear includes the presence of 5,000 squamous cells for liquid-based preparations. A notation is made regarding the presence or absence of an endocervical/transformation zone. The criteria for an “unsatisfactory” smear include lack of patient identification, a broken, unrepairable slide, and/or scant squamous epithelial component. For purposes of this study, we defined an adequate Pap smear as one that was both “satisfactory” as defined above and included endocervical cells, as required for this designation by the American Cancer Society.⁶

Description of the Educational Intervention

Prior to this intervention, we provided no formal training to residents on Pap smear sampling technique. We designed a 3-part intervention that combines 3 of the most effective methods of teaching: an educational presentation, skills workshop, and the availability of data for residents to compare their performance to that of their peers.^15–17 The educational presentation on Pap smear sampling technique included a 20-minute PowerPoint and video¹ presentation. This was followed by a 30-minute hands-on training skills workshop using a life-size gynecological manikin, “Eva.”¹⁸ During these small group sessions (6 to 10 residents), the instructor answered questions about appropriate Pap smear technique and residents had an opportunity to practice Pap smear sampling technique as well as view normal and pathological cervices. Six months later, residents received peer comparison feedback on the quality of their Pap smears. Each resident was sent a graph that depicted their 6-month Pap smear adequacy rate compared to the mean adequacy rate of their peers. Residents were not required to respond to the feedback that they received. Total resources dedicated to the intervention include 2 hours of faculty effort, 10% research assistant effort, and the cost of the manikin ($500).

Curriculum Evaluation

To assess how the residents received the educational presentation and skills workshop, residents were asked to complete a 6-question evaluation form immediately after the educational presentation and skills workshop. Using a 4-point scale set as “yes,”“yes but I’d hoped for more,”“not much,” and “not at all,” we asked residents to rate their preparedness to complete Pap smears. We also asked residents to use a 4-point scale set as “yes,”“somewhat,”“not much,” and “no” to rate the usefulness of the workshop materials.

Statistical Methods

We compared unadjusted Pap smear adequacy rates at baseline and during the evaluation period using the χ² test. A logistic regression model was used to assess which factors were jointly predictive of an adequate Pap smear. The generalized estimating equations (GEE) approach was used to account for the binary response variable (adequate or not adequate) controlling for the variation among multiple Pap smears done by each physician. The GEE model included: physician performing the Pap smear, baseline Pap smear adequacy rates, resident training year, resident gender, resident practice site, and intervention or control group. A backward stepwise algorithm excluded nonsignificant variables from the model. Odds ratios (OR) and 95% confidence intervals (CI) were calculated for predictors based on the GEE estimates of the effects and their standard errors.

RESULTS

All 23 residents in the intervention cohort and 30 residents in the control cohort were followed prospectively. Seventeen residents (74%) in the intervention cohort and 24 residents (80%) in the control cohort completed Pap smears in the pre- and postcomparison periods and were included in the analysis. Intervention and control cohorts were similar in age and percentage assigned to each site. The intervention group was composed of more women and first-year residents. Table 1 reviews descriptive statistics for physicians and patients.

Table 1.

Baseline Characteristics for Physicians and Patients

Characteristic	Intervention Cohort	Control Cohort
Total residents	17	24
Resident gender, n (%)
Male	9 (53)	19 (79)
Resident training year N (%)
First	10 (59)	9 (38)
Second	7 (41)	15 (62)
Resident practice site, (%)
UPS N (%)	9 (53)	11 (46)
CHC N (%)	8 (47)	13 (54)
Patient age
Mean, y (SD)	37.5 (12.4)	38.5 (13.3)

UPS, university practice site; CHC, community health center; SD, standard deviation.

A total of 428 Pap smears were obtained by the 41 internal medicine (IM) residents over a 16-month period. 163 Pap smears were obtained during the 6-month pre-intervention period. There was no difference in baseline adequacy rates between the intervention or control cohort (Table 2).

Table 2.

Univariate Analysis

Characteristic	Pre-intervention Paps	Post-intervention Paps
Intervention cohort
Adequate/total (%)	49/68 (72)	86/103 (84)
Control cohort
Adequate/total (%)	62/95 (65)	113/162 (70)
OR (95% CI)	1.37 (0.69 to 2.7)	2.13 (1.15 to 3.96)
P value	.36	.02

OR, odds ratio; CI, confidence interval.

In the 10-month evaluation period, residents obtained a total of 265 Pap smears. We found significantly higher adequacy rates in the intervention cohort (84% adequate) compared to the control cohort (70% adequate). Residents who had received the educational intervention were twice as likely to obtain an adequate Pap smear (OR, 2.13; 95% CI, 1.15 to 3.96; Table 2).

In the GEE model, patient age, resident gender, training year, and practice site were not significant. The only variable predictive of an adequate Pap smear was having had the educational intervention. Physicians who completed educational training were 1.73 times more likely to obtain an adequate Pap smear than those who had not (adjusted OR, 1.73; 95% CI, 1.07 to 2.80; P = .03).

All residents in the intervention cohort evaluated the educational presentation and skills workshop. We collapsed residents’ responses from the two sites into one for the overall analysis (Table 3). Over 90% of respondents felt that “yes” they had a clear understanding of the steps involved in obtaining a Pap smear and felt able to obtain satisfactory smears. Greater than 74% of residents responded “yes” to the usefulness of the workshop materials, PowerPoint, and video. However, only 52% of residents answered “yes” or “somewhat” when asked if compared to a live model, this workshop was more helpful.

Table 3.

Residents’ Evaluation of Educational Presentation and Skills Workshop

	n (%) of Residents Who Responded
Residents’ Perceptions	Yes	Yes, but I’d hoped for more	Not much	Not at all
Residents’ perceptions of their preparedness to perform Pap smears
I have a clear understanding of the steps involved in obtaining a Pap smear	23 (100)	–	–	–
I feel that I will be able to obtain satisfactory Pap smears	21 (91)	2 (9)	–	–
Residents’ perceptions on the usefulness of the educational presentation and skills workshop	Yes	Somewhat	Not much	No
The workshop materials were useful	21 (91)	1 (4)	–	–
The PowerPoint presentation was helpful	23 (100)	–	–	–
The video recording was helpful	17 (74)	4 (17)	1 (4)	–
Compared to using a live model, this workshop was more helpful	7 (30)	5 (22)	1 (4)	–

DISCUSSION

Our results suggest that not all resident physicians practice competently, at least with regard to Pap smear sampling. We found that instruction which includes an educational presentation by a knowledgeable instructor, skills workshop, and peer comparison feedback resulted in a 21% increase in this skill-based competency in resident physicians. The usual increase in physician performance from similar intensity interventions is between 12% and 15%.^19,20 This striking improvement in adequacy rates could translate into fewer repeated Pap smears, decreased frustration for patient and physician, and improved quality of care to patients. Trainees performed Pap smears infrequently during the study period. This educational intervention allowed a trainee, who does not practice obtaining Pap smears frequently, to perform this skill proficiently.

Although further testing in other settings is needed, our findings are important to medical education because our multifaceted intervention may serve as a model for teaching trainees other clinical skills. In addition, this intervention allows educators to assist trainees in accomplishing two core competencies as outlined by the Accreditation Council on Graduate Medical Education (ACGME): patient care and practice-based learning. Our intervention emphasized patient care that was appropriate and effective for the promotion of health. Through education and feedback, we offered residents the opportunity to investigate, evaluate, and improve their own patient care.

Our study has several limitations. First, our study was conducted at one university training program. Although this training program is similar in size to many other mid-size residency training programs, and our patient population is likely similar to patients seen in urban academic centers, further testing of this intervention in other settings is needed. Second, we do not know how many of the residents had previous training on Pap smear sampling during medical school. We do not think that this had a major impact on the results, as the baseline adequacy rates were similar and residents were randomly assigned to the intervention or cohort group, thus limiting confounders. Third, although cross-contamination was possible, we do not know of any systematic cross-contamination by residents who knew there was a curriculum but were not getting the intervention. Fourth, we focused on only one aspect of a good pelvic exam, obtaining an adequate Pap smear. We did not evaluate other important components of a pelvic exam, such as patient comfort. Fifth, we did not perform a true postcurriculum evaluation. Therefore, we are not able to report on residents’ perceptions on the usefulness of the curriculum over the 10-month period. By having some of the postdata precede the third component of the educational intervention (peer comparison feedback), the relative impact of feedback in improving Pap smear adequacy is unclear.

In summary, this brief multifaceted educational intervention is an effective tool to improve the frequency with which internal medicine residents obtain endocervical cells. Our findings are important given increasing public concerns regarding quality of care in the medical field and new stipulations from the ACGME on the need to implement competency-based curricula. Residency programs should consider incorporating targeted education coupled with peer comparison feedback into in-service training.