Do procedural skills workshops during family practice residency work?

ABSTRACT

OBJECTIVE

To determine if participation in a procedural skills workshop during family practice residency affects future use of these skills in postgraduate clinical practice.

DESIGN

Survey involving self-assessment of procedural skills experience and competence.

SETTING

British Columbia.

PARTICIPANTS

Former University of British Columbia family practice residents who trained in Vancouver, BC, including residents who participated in a procedural skills workshop in 2001 or 2003 and residents graduating in 2000 and 2002 who did not participate in the procedural skills workshop.

MAIN OUTCOME MEASURES

Self-assessed experience and competence in the 6 office-based procedural skills that were taught during the procedural skills workshops in 2001 and 2003.

RESULTS

Participation in a procedural skills workshop had no positive effect on future use of these skills in clinical practice. Participation in the workshop was associated with less reported experience (P = .091) in injection of lateral epicondylitis. As with previous Canadian studies, more women than men reported experience and competence in gynecologic procedures. More women than men reported experience (P = .001) and competence (P = .004) in intrauterine device insertion and experience (P = .091) in endometrial aspiration biopsy. More men than women reported competence (P = .052) in injection of trochanteric bursae. A third year of emergency training was correlated with an increase in reported experience (P = .021) in shoulder injection.

CONCLUSION

Participation in a procedural skills workshop during family practice residency did not produce a significant increase in the performance of these skills on the part of participants once they were in clinical practice. The benefit of a skills workshop might be lost when there is no opportunity to practise and perfect these skills. Sex bias in the case of some procedures might represent a needs-based acquisition of skills on the part of practising physicians. Short procedural skills workshops might be better suited to graduated physicians with more clinical experience.

Research in Canada, Australia, New Zealand, the Netherlands, and the United States has suggested that the procedural skill set expected of new family or general practice physicians is not being adequately taught in residency or registrar programs.^1–4 There are a number of factors that could be contributing to this situation. Problems with loss of skill proficiency on the part of teaching faculty seem to be most acute in urban settings.^5,6 This results in preceptors recognizing certain skills to be important but lacking the expertise required to pass this skill set along to residents. Rural programs are not immune to procedural skills training challenges. In Australia accreditation pressures have resulted in some rural family physicians abandoning skills they have expertise in because of the imposition of procedure volume minimums by some regulatory bodies and health authorities.⁷ Many skills that were once practised routinely by family physicians have been co-opted by other specialists and allied health professional colleagues.⁸ Within the culture of postgraduate training, there has also been a decrease in on-call work on the part of trainees, and this affects the number of meaningful opportunities available to learn many basic skills. Even within the context of remaining on-call hospital experience, many trainees report having to self-teach certain procedures in contrast to the formal and supported procedural skills teaching received by nurses.⁹

In 2001, one of the authors (M.S.M.) looked at the list of procedural skills then listed as “required” and “recommended” by the University of British Columbia (UBC) Family Practice Residency Program. This list was circulated to all family practice teaching faculty in Chilliwack, BC, and faculty were asked to comment on their own level of experience and expertise for each skill. The resulting skill profile was individually uneven but collectively complete. The required procedural skills expertise existed among the Chilliwack local faculty, but the challenge was translating this into a teaching opportunity that could benefit the entire cohort of residents.

A brief but concentrated and interactive procedural skills workshop was designed using local faculty. It included stations for intrauterine device (IUD) insertion, endometrial aspiration biopsy, assessment and injection of lateral epicondylitis, assessment and aspiration or injection of the knee, assessment and injection of trochanteric bursitis, and assessment and injection of the shoulder. Each workshop station was facilitated by a skilled preceptor and, with the exception of the trochanteric bursitis station, involved the use of anatomic models designed for procedural skills training.

The entire workshop took 3 hours and was held in 1 afternoon session. The format of the workshop was similar to workshops that have been shown to make a difference in the skill proficiency of practising family and general practitioners.¹⁰

This workshop was made available to residents in Vancouver, BC, as well as to those in the smaller community of Chilliwack, and it has been repeated every 2 years since 2001. The involvement of Vancouver-based residents has not been consistent over the past 8 years, and this means that there are closely matched cohorts of former Vancouver-based residents who have and have not participated in the workshop.

The workshops have always been well received by residents. Postworkshop evaluations are almost always positive, and there are frequently comments attesting to the readiness of participants to try these procedural skills in the future.

The residents in Chilliwack work in a much smaller hospital and community than those residents in Vancouver, and the literature suggests that this would automatically account for more procedural skills proficiency compared with their urban counterparts.^4,11–14

The primary research question addressed by this study is whether participation by the city-trained family practice residents in a procedural skills workshop led to greater experience and greater competence in procedural skills compared with peers who did not participate in such a workshop. In this respect the study represents the outcome component of a formal program evaluation.

We also examined whether or not there was a sex effect on the acquisition and practice of the procedural skills taught in the procedural skills workshop.

METHODS

Survey instrument

The 2 parts of the survey instrument were developed from an extensive literature review of procedural skills training and from the Canadian National Physician Survey for respondent characteristics; this established content validity of the instrument.¹⁵

The 5-point Likert scale for self-assessment of experience and competence was the same one used by Goertzen in his large study of Canadian family practice residents in 2006.¹¹ For each procedural skill, subjects were asked to rank their experience levels as follows: 1— never observed, 2—never performed, 3—performed with major assistance, 4—performed with minimal assistance, or 5—performed independently. For each procedural skill, subjects were asked to rank their competence levels as follows: 1—not competent, 2—minimally competent, 3—somewhat competent, 4—adequately competent, or 5—very competent.

Face validity was addressed through a pilot test of the survey instrument using first- and second-year residents of the UBC Chilliwack Family Practice Residency Program. Content validity is related to the breadth of inference that can be drawn. Because the procedures being assessed by the survey instrument were those explicitly addressed in the workshops, generalizations to other family practice procedural skills are very appropriate. In the absence of a criterion standard, criterion validity cannot be addressed. Construct validity is addressed implicitly by examining the twin concepts of experience and competence.

Test-retest reliability was not examined here, as only a single retrospective self-assessment of experience and competence can be done. It is assumed that responses are stable in time, given the interval between the workshops and survey administration. Because the primary interest is in individual items (ie, procedures), rather than an overall score, internal consistency is not relevant here.

Participants

Sample size was limited by the total actual enrolment in the workshops in 2001 and 2003 and by the number of nonparticipant residents in 2000 and 2002. Attendance lists for the 2001 and 2003 workshops were reviewed, and residents from Vancouver who had attended the procedural skills workshops were sent an e-mail survey in December 2007.

Program administrators from the St Paul’s Hospital Family Practice Residency Program and the Vancouver City Site Family Practice Residency Program provided lists of their graduating residents from 2000 and 2002 who had not participated in the workshop, and these former residents were sent the same survey.

Responses to the survey were collected through an online survey facility (Survey Monkey) and were coded based on year of graduation from the UBC Family Practice Residency Program and whether or not the subject had participated in the procedural skills workshop.

A $25 dollar gift certificate for a national bookstore was issued to subjects who submitted a completed survey.

Analysis

Five-point Likert scale responses were dichotomized into experienced and not experienced, and competent and not competent.

Never observed, never performed, and performed with major assistance responses were considered to be not experienced. Performed with minimal assistance and performed independently responses were considered to be experienced.

Similarly, not competent, minimally competent, and somewhat competent were considered to be not competent, while adequately competent and very competent were considered to be competent.

Dichotomized results were analyzed using cross tabulations and χ² tests of independence.

Based on available sampling frames of 40 control subjects and 50 participants, margins of error would be approximately ± 15% for each group. Hence confidence intervals for differences of proportions would have margins of error of no more than ± 20%, thus establishing an acceptable level of detectable differences. Achieved sample sizes of 19 and 24 resulted in detectable differences between proportions of about 30%.

Before distribution of the survey, ethical approval for the study was obtained from the Fraser Health Authority Research Ethics Board and the UBC Behavioural Research Ethics Board.

RESULTS

In December 2007, 87 subjects were sent e-mail surveys; 38 of these were control subjects and 49 were participant subjects.

This resulted in 16 responses. Reminder e-mails did not affect this response. In March 2008, a conventional mailing of the survey to remaining nonrespondents yielded 21 more responses.

In June 2008 a second conventional mailing of the survey was sent to remaining nonrespondents accompanied by telephone calls to 10 subjects to request that they fill out the survey. This resulted in 6 more responses. The total number of respondents was 43, representing an overall response rate of 49%. Response rates were 49% for workshop participants and 50% for controls, hence the nonresponse rate was the same for both groups.

Results of this study show that participation in the procedural skills workshop had no positive effect on future use of these skills in clinical practice (Tables 1 to 4).

Table 1.

Proportion of workshop participants and nonparticipants rating themselves as experienced with the procedures studied

PROCEDURE	THOSE RATING THEMSELVES AS EXPERIENCED		P VALUE
	DID WORKSHOP, % (N)	DID NOT DO WORKSHOP, % (N)
IUD insertion	68 (15)	52 (11)	.29
Endometrial aspiration biopsy	55 (12)	43 (9)	.44
Shoulder injection	64 (14)	71 (15)	.59
Injection of trochanteric bursa	59 (13)	62 (13)	.85
Injection of lateral epicondylitis	41 (9)	67 (14)	.091
Knee injection or aspiration	77 (17)	91 (19)	.24

IUD—intrauterine device.

Table 2.

Proportion of workshop participants and nonparticipants rating themselves as competent with the procedures studied

PROCEDURE	THOSE RATING THEMSELVES AS COMPETENT		P VALUE
	DID WORKSHOP, % (N)	DID NOT DO WORKSHOP, % (N)
IUD insertion	50 (11)	48 (10)	.88
Endometrial aspiration biopsy	18 (4)	29 (6)	.42
Shoulder injection	46 (10)	52 (11)	.65
Injection of trochanteric bursa	41 (9)	48 (10)	.66
Injection of lateral epicondylitis	32 (7)	48 (10)	.29
Knee injection or aspiration	59 (13)	71 (15)	.40

IUD—intrauterine device.

Table 3.

Proportion of all participants rating themselves as experienced, by sex

PROCEDURE	THOSE RATING THEMSELVES AS EXPERIENCED			P VALUE
	TOTAL, % (N)	MALE, % (N)	FEMALE, % (N)
IUD insertion	61 (26)	30 (6)	87 (20)	.001
Endometrial aspiration biopsy	49 (21)	35 (7)	61 (14)	.091
Shoulder injection	67 (29)	70 (14)	65 (15)	.74
Injection of trochanteric bursa	61 (26)	70 (14)	52 (12)	.23
Injection of lateral epicondylitis	54 (23)	60 (12)	48 (11)	.43
Knee injection or aspiration	84 (36)	90 (18)	78 (18)	.30

IUD—intrauterine device.

Table 4.

Proportion of all participants rating themselves as competent, by sex

PROCEDURE	THOSE RATING THEMSELVES AS COMPETENT			P VALUE
	TOTAL, % (N)	MALE, % (N)	FEMALE, % (N)
IUD insertion	49 (21)	25 (5)	70 (16)	.004
Endometrial aspiration biopsy	23 (10)	15 (3)	30 (7)	.23
Shoulder injection	49 (21)	60 (12)	39 (9)	.17
Injection of trochanteric bursa	44 (19)	60 (12)	30 (7)	.052
Injection of lateral epicondylitis	40 (17)	50 (10)	30 (7)	.19
Knee injection or aspiration	65 (28)	75 (15)	57 (13)	.21

IUD—intrauterine device.

Workshop participants were less likely to report experience in injection of lateral epicondylitis than non-participants were (P = .091). More women than men reported experience and competence in gynecologic procedures. More women than men reported experience (P = .001) and competence (P = .004) in IUD insertion and experience (P = .091) in endometrial aspiration biopsy. More men than women reported competence (P = .052) in injection of trochanteric bursae.

A total of 12 study subjects (6 who had completed the workshop and 6 who had not) had completed a third year of enhanced emergency medicine training. This extra year of training was not correlated with any increase in the use of these basic procedural skills, with the exception of shoulder injection; subjects with a third year of emergency medicine training reported more experience with this procedure (P = .021) than the rest of the subjects did.

DISCUSSION

Despite historical and ongoing controversy over what constitutes essential skills for family practitioners, there is consensus that procedural skills are an important part of family medicine. The benefits for patients, health service insurers, and physicians themselves have been well documented.^12,16,17

Wetmore and colleagues confirmed that many Canadian family physicians do not feel that they are competent in the skills that they themselves see as being essential for family practice training.^1,5,8 Recognizing the discrepancy between the skills expected of a graduating resident and the ability of faculty to provide adequate training in these skills, previous authors have suggested that “innovative solutions” be found to improve the skills training of family practice residents, especially those in urban programs.^1,5,6

The workshop in this study represented an effort to create this type of innovative solution. The time efficiency and fun factor inherent in short, interactive, hands-on procedural skills workshops make them popular with faculty and residents alike. It would be gratifying if a one-time inoculation with this type of skills training was enough to affect long-term proficiency in a set of basic office skills. Unfortunately this study found that such a workshop, when facilitated during residency, does not affect future use of skills in independent practice. The fact that such workshops have been found to be at least modestly effective as interventions with practising physicians suggests that the success of a brief intensive workshop depends on participants bringing to the process an existing base of clinical experience and confidence. There might be a tipping point at which a critical amount of existing procedural and clinical skill allows the easy acquisition of new skills. This and other studies suggest that this tipping point does not occur during residency.

A New Zealand study of junior doctors found that a skills workshop was only effective if participants were subsequently able to practise those skills during the junior doctor year. As a result, of 6 skills covered in an orientation workshop, the only skill that improved by the end of the junior year was urethral catheterization, which was the procedure cited as being performed most often after the workshop.¹⁸

This is consistent with other studies that have identified the opportunity to practise and perfect skills as a necessary part of even their basic acquisition. However, once most skills have been mastered, maintenance of competence does not require repetition and practice.^1,7,11

A short skills workshop might therefore still be useful during residency, as long as participants have opportunities for practice and reinforcement of these skills during their residencies.

This study found that more women than men performed gynecologic procedures. This is consistent with previous Canadian studies.^7,19 This likely represents a needs-based acquisition of skills, as female family physicians see more female patients than their male colleagues do. The need to acquire a skill in order to serve a patient population has been shown to be a powerful motivator for skills acquisition.^1,11,12 This represents patient-centred care as a driver for skills development.

Previous studies have shown that men perform more nongynecologic procedures than women do.^8,11,13,14 This study confirmed this sex bias in the case of trochanteric bursa injection.

This study also showed that those subjects who completed an extra year of emergency medicine training rated themselves as being more experienced with shoulder injections than the rest of the subjects did. In Canada, the third year of emergency training emphasizes critical care medicine and resuscitative skills. It is therefore not surprising that elective basic gynecological procedures are not overrepresented in this group. It is, however, unclear why participants were more experienced with shoulder injections than with other musculo-skeletal procedures, such as knee injections.

This study showed that workshop participants were less likely to report experience in injection of lateral epicondylitis than nonparticipants were. This was the only procedure with such an association. Each procedure station was facilitated by the same faculty facilitator for both workshops. It could be that the facilitator in this case was less effective than his colleagues.

Limitations

This study is limited by its small sample size and its relatively low response rate of 49%. Response rates were, however, the same for control and participant groups. Validity is strengthened by the fact that sex bias with respect to skills, as demonstrated in previous studies, has been demonstrated in this study.

Another relative limitation of this study relates to the use of self-assessment of experience and competence as opposed to objective observation and measurement. Most previous studies on procedural skill proficiency share this limitation and make use of self-assessment. The validity of this approach has been demonstrated previously in the literature.¹¹ One Australian study showed that self-assessment of procedural skill proficiency closely correlated with supervisor assessment.¹

Conclusion

This procedural skills workshop during family practice residency had no positive effects on future use of these skills on the part of participants. The benefit of a procedural skills workshop is lost unless there is opportunity to subsequently practise and master the skills learned. Sex bias in the case of some procedures might represent a needs-based acquisition of skills. Short procedural skills workshops might be better suited to graduated physicians with more clinical experience.

EDITOR’S KEY POINTS

• Research has suggested that the procedural skill set expected of new family or general practice physicians is not being adequately taught during residency programs. This study examined whether participation by family practice residents at the University of British Columbia in a short, concentrated, interactive procedural skills workshop led to greater experience and greater competence in procedural skills compared with peers who did not participate in such a workshop.

• This procedural skills workshop during family practice residency had no positive effect on future use of these skills on the part of participants. The benefit of a procedural skills workshop is lost unless there is opportunity to subsequently practise and master the skills learned.

POINTS DE REPÈRE DU RÉDACTEUR

• Selon les recherches, les compétences opératoires attendues des nouveaux médecins de famille ou omnipraticiens ne seraient pas enseignées adéquatement durant les programmes de résidence. Cette étude examinait si la participation de résidents en pratique familiale de l’University of British Columbia à un court atelier intensif et interactif sur les aptitudes opératoires avait entraîné une exécution accrue et une plus grande maîtrise de ces interventions opératoires par rapport à leurs pairs qui n’avaient pas participé à un tel atelier.

• Cet atelier sur les aptitudes opératoires durant la résidence en pratique familiale n’a pas eu d’effets positifs sur l’utilisation ultérieure de ces habiletés de la part des participants. Les avantages d’un atelier sur les aptitudes opératoires se perdent s’il n’y a pas de possibilités de mettre en pratique subséquemment et de maîtriser les habiletés apprises.