Assessing Medical Students’ Tobacco Dependence Treatment Skills Using a Detailed Behavioral Checklist

Abstract

Construct

This paper describes the development and implementation of an assessment intended to provide objective scores that would be valid indications of medical students’ abilities to counsel patients about tobacco dependence.

Background

Assessing medical students’ advanced communication skills, particularly in the context of providing tobacco dependence treatment, consistently and accurately is challenging; doing so across multiple medical schools is even more difficult.

Approach

Ten medical schools implemented a tobacco dependence treatment case as part of an Objective Structured Clinical Examination (OSCE) for third year medical students. A 33-item checklist with detailed criteria and examples was developed for scoring students’ performances. Trained coders viewed and coded 660 videotaped encounters; approximately 10% also were coded by the coding supervisor to check accuracy.

Results

Average time required to code an encounter was approximately 31 minutes; accuracy (i.e., agreement with the gold standard coder) was excellent. Overall, students performed an average of 1 in 4 of the 33 behaviors included on the checklist, and only 1 in 10 discussed setting a quit date. Most students (almost 9 in 10) asked how much the patient smoked in a day, and just over 7 in 10 informed the patient that the cough was due to smoking.

Conclusions

The authors developed and implemented a rigorous assessment that will be used to evaluate medical students’ tobacco dependence treatment skills. Operationalizing the specific counseling behaviors, training coders to accurately capture students’ performances using a structured checklist, and conducting the coding all required substantial time commitments, but will provide confidence in the objectivity of the assessment results. Additionally, this assessment can be used to provide formative information on medical students’ tobacco dependence treatment skills, and to tailor ongoing training for medical students in this area.

Introduction

Smoking continues to be a major public health problem, with 443,000 deaths annually attributed to smoking in the United States.¹ Although the prevalence of smoking declined by more than 50% between 1965 and 2010, an estimated 43.8 million, or 19% of adults aged 18 and older continue to smoke.²

Approximately 70% of all smokers have at least one physician visit each year.³ Physician-delivered interventions influence patients to think about quitting, and ultimately to quit.³ However, in 2010 only 51% of smokers reported receiving advice to quit from a healthcare provider.⁴ One explanation for these relatively low rates of advice is that many physicians report limited skills in this area⁵ suggesting that interventions to improve physicians’ tobacco dependence treatment skills, which include both counseling and pharmacologic interventions, are needed. Rigorous methods of assessing medical students’ tobacco dependence treatment skills are needed both to describe students’ current skill levels, and to support curriculum evaluation.

Assessing advanced communication skills such as tobacco dependence treatment is challenging. A variety of approaches have been used, including self-report, preceptor reports, patient reports, multiple choice tests, video-based tests, direct observation of actual encounters, and objective structured clinical examinations (OSCEs). Each of these approaches has strengths and weaknesses, which have been discussed previously by Epstein.⁷ Which approach is optimal will depend on the purpose of the assessment (i.e., the decisions or conclusions that one hopes to make based on the resulting scores) and the context, including the resources available and the practical constraints.

Limitations to these approaches vary and include the following: With regard to self-report of skills, students who are not skilled may be unaware of their deficit, and/or unwilling to report this honestly. Preceptor ratings or patient ratings are difficult to standardize, in part because of differences in the number and type of patients who smoke that medical students encounter, e.g., some students might see smokers who present difficult communication challenges, while others might see patients who are very receptive. In addition, if the intent is to compare cohorts across medical schools, it would be very difficult to obtain comparable ratings from preceptors and patients across schools. Multiple-choice test items, while standardized, are likely to be perceived as artificial, and also are less related to actual performance of these skills. OSCE-based assessments can overcome several of these limitations.

OSCEs are used widely in medical education, in part because of the apparent authenticity or realistic nature of the task, and the focus on skills rather than knowledge alone. A 2010 survey found that all but one US medical school required students to complete an OSCE at some point during the clinical years.⁸ In a typical OSCE, a standardized patient (SP) is trained to portray a patient with a specific history and complaint. The examinee is given brief background information and then meets with the SP, typically conducting a history and physical, and often including some counseling or advice. OSCEs, however, have limitations. Operationalizing communication skills is not trivial, and consistent, accurate scoring of performance is difficult to achieve.^9–12

The purpose of this paper is to describe how we approached the challenge of assessing medical students’ tobacco dependence treatment skills in an OSCE, and how we operationalized the specific skills involved to allow highly standardized assessments across multiple medical schools. Our intent is to provide an example that may be useful to other investigators and educators presented with similar challenges, and a discussion of the generalizable issues and measurement decisions we faced.

Methods

Sample

Ten medical schools around the country participated in this study. Students received traditional training in tobacco dependence treatment skills according to the existing curriculum at their school, i.e., no special training was provided as part of this study. All 10 participating medical schools implemented the tobacco dependence treatment OSCE case described below. Approximately 50% (660/1345) of these encounters were randomly selected for the present study. The videos of those encounters form the sample for the study reported here. All student participants were third year medical students; 353 (53.5%) were male; no other demographic or background information was collected on these participants.

OSCE tobacco dependence treatment case

The tobacco dependence treatment case was based on a case that had previously been developed at a participating school but was not in current use at any of the schools at the time of the study. This case was selected because it provided an opportunity for tobacco dependence treatment in the context of a relatively straightforward presenting complaint (persistent cough) making the case appropriate for use in an Internal Medicine or Family Medicine clerkship curriculum. The case materials were enhanced for the present study by elaborating on the medical history. Case materials included a simulated medical record (available outside of the exam room for the student’s review prior to beginning the encounter), which specified that the patient was a smoker, and that the presenting complaint was a cough. Students were instructed to obtain a focused and relevant history, perform a focused and relevant physical exam, and discuss initial diagnostic impressions, follow-up testing and initial management plans with the patient, including tobacco use counseling.

Training of standardized patients in portrayal

Standardized patients (SPs) at each of the 10 participating medical schools were trained to portray the patient described in the case materials. Central study staff used a train-the-trainer model, training the SP trainers at each participating medical school, who in turn trained their school’s SPs. A trainer manual was developed which described the study, and provided detailed instructions on case implementation and an SP training protocol. An SP training manual provided included detailed instructions for the SP on how to respond to specific questions about smoking and readiness to quit.

SPs used materials created by their local schools (e.g., history checklists, communication rating scales) to record students’ performances; these scores were part of each school’s local assessment but were not provided to investigators at the University of Massachusetts Medical School.

Global ratings of performance

Four items assessing global communication skills were adapted from the Communication Assessment Tool developed by Makoul and colleagues, to measure patient views of physician communication skills.¹³ The items required the coder to rate the extent to which the student displayed the following items: 1) Treated patient with respect; 2) Gave patient opportunity/time to talk without interruptions; 3) Adapted to patient’s level of understanding (e.g. avoided/explained jargon); and 4) Showed care and concern. The response scale ranged from 1 to 5, with 1 = Poor and 5 = Excellent. No explicit criteria for these ratings were provided as the intent was to capture the coders’ subjective impressions of the students’ communication skills, and no training on how to rate these items was provided; inclusion of these global ratings was done to allow post hoc exploratory analyses.

Behavioral checklist of tobacco dependence treatment skills

Because our goal was to provide accurate, consistent scores that represented students’ tobacco dependence treatment skill, we developed a detailed checklist to capture the specific behaviors that students would be expected to perform to provide appropriate tobacco dependence treatment for this case.

The 5As (Ask, Advise, Assess, Assist, and Arrange)¹⁴ provided the initial organizing framework. Within this framework, we identified the specific behaviors that would constitute tobacco dependence treatment for this case, drawing on the team’s clinical experience, clinical guidelines, the tobacco dependence treatment literature, and patient-centered counseling strategies found to be effective in a previous study teaching tobacco dependence treatment skill to residents.¹⁵ Detailed criteria for each behavior and examples of statements or questions meeting those criteria were developed and assembled into a coding manual. Table 1 provides an example of a behavioral checklist item with corresponding scoring criteria and examples. The checklist included 33 behaviors (Table 2). These behaviors were reviewed and confirmed by other local tobacco dependence treatment experts to ensure content validity.

Table 1.

Sample Behavioral Checklist Item

Checklist Item	Criteria and Examples
Asked about or prompted discussion about patient’s readiness to quit	The student asked the SP how ready s/he is, if s/he is ready or what s/he thinks or feels about quitting. Examples: How do you feel about quitting? What are your thoughts about quitting? How ready are you to quit? Are you thinking about quitting? Are you ready to quit?
Discussed medications used in past quit attempts	Student asked specifically about medications used to quit in the past – does not need to mention any specific type of medication. If SP offers information about medication used before being asked the student follows up with question about the medication. Examples: What medications/medicines have you used to help you quit smoking? Have you ever tried anything like the patch or gum to quit? How did you feel when you used the gum?
Summarized treatment plan (behavioral and medication) options in which the patient expressed interest	Towards the end of the session the student summarized at least some of the behavioral and medication strategies that had been discussed. This must be a repeat of items previously discussed and does not include any new information. Not all strategies need to be included. Examples: So we have talked about maybe using straws instead of smoking and you are interested in learning more about some of the other nicotine products. You don’t want to use the gum but you might be interested in the patch or lozenge. You are thinking about ways you can change your routine and you definitely don’t want to use the gum again.

Table 2.

Behavioral Checklist Items and Percentage of Students Performing Each Item^*

5A Category	Behavioral Item	Percent Performing	N Performing
Ask	Asked how soon after waking patient smokes	0.5	3
Assist	Elicited patient’s views of specific behavioral strategies to consider during a new quit attempt	0.8	5
Assist	After making suggestions, asked patient his/her opinion about/reaction to behavioral strategies suggested	1.4	9
Assist	Asked about triggers and situations associated with tobacco use	2.1	14
Assist	Asked why patient started smoking again after quit attempts	2.4	16
Assist	Informed patient that counseling plus medications increases success in quitting	2.7	18
Arrange	Encouraged patient to discuss tobacco use during next visit with regular PCP	2.9	19
Arrange	Provided information about 1-800-QUIT NOW or state quitline	5.0	33
Assess	Asked about/prompted discussion of the patient’s confidence in ability to quit	5.2	34
Assist	Asked patient who could be supportive or helpful for patient to quit	6.7	44
Assist	Elicited patient’s response to the plan	6.8	45
Assess	Invited additional discussion of tobacco use	7.4	49
Assist	Provided suggestions of behavioral strategies to consider	8.9	59
Assess	Asked about the patient’s reasons to consider quitting	9.2	61
Assist	Discussed setting a quit date	9.4	62
Assess	Asked about/prompted discussion about pros of smoking/potential barriers to quitting	9.7	64
Assist	Summarized treatment plan (behavioral and medication) options in which the patient expressed interest	10.6	70
Arrange	Discussed other referral options for support	15.6	103
Arrange	Recommended that the patient return for a follow up visit within a specific time period	16.8	111
Assist	Before making suggestions, elicited patients views of medications to help with quitting	18.9	125
Assist	Asked about success or difficulties during past quit attempts	20.5	135
Assist	Discuss medications or behavioral strategies used in past quit attempts	21.7	143
Assist	After making suggestions, asked patient his/her opinion about/reaction to medication options presented	23.5	155
Assist	Presented information about use of bupropion to help patient quit	25.5	168
Assist	Presented information about use of varenicline to help patient quit	34.2	226
Assist	Asked about past quit attempts	48.9	323
Assess	Praised/congratulated the patient for cutting down/quitting	50.2	331
Advise	Advised patient to quit smoking in a clear manner	56.5	373
Assist	Presented information about nicotine replacement therapy options to help patient quit	57.6	380
Assess	Asked about/prompted discussion about patient’s readiness to quit	66.8	441
Ask	Asked how long patient has been smoking	69.7	460
Advise	Informed the patient that the cough was related to smoking	71.7	473
Ask	Asked how much on average the patient smokes in a day	88.9	587

^*Six hundred sixty SP encounters with students from 10 medical schools were coded between September 2011 and June 2012.

Pre-testing

In September 2010 the case was piloted in the University of Massachusetts Medical School Medicine Clerkship OSCE; 31 third year medical students participated and were video-recorded. A random sample of these student videos (n=6 of 31) and three videos of attending physicians with expertise in tobacco dependence treatment were reviewed by two authors (KM, DJ) in order to check the completeness of the checklist and to pretest the scoring criteria. Minor revisions to the coding criteria were made.

Training in using the checklist

All coder trainings were led by one of the authors (DJ), a tobacco dependence treatment specialist with extensive experience in training clinicians in tobacco dependence treatment skills. Four non-study research personnel were hired as coders. One served as the coding supervisor and gold standard and received additional 1:1 instruction in the checklist prior to participating in the core training activities in which all coders participated. All coders first independently reviewed the coding manual. This was followed by group didactic instruction, with review of each checklist item and coding criteria. Coders then viewed and scored practice videos as a group, discussing coding decisions and discrepancies. Next, each coder viewed and coded a sample of videos (n=16) independently, and met to review and discuss their results. Finally, each coder practiced coding a sample (n=10) of non-study videos. Prior to coding study videos, each coder was required to demonstrate at least 95% accuracy (as determined by comparison of codes to those assigned by the expert trainer [DJ]) on 5 non-study videos. Altogether, total time spent in training activities prior to coding ranged from approximately 24 to 36 hours per coder.

Data collection

All coding of study videos was done independently, with the coder blind to the student’s school. Each video was randomly assigned to one of the three coders. Videos were assigned in blocks, with the constraint that each coder would view encounters from at least 4 different schools within a block. Coders were allowed to replay and review the video as needed to determine whether or not a behavior had occurred.

Assessment of coding accuracy

Approximately 10% of the encounters were randomly selected to be independently coded by the coding supervisor as the gold standard, in addition to the original coder. Discrepancies between the supervisor and coders were identified and recorded, and feedback was provided to the coder on an on-going basis.

Results

A total of 660 videos of student encounters using this tobacco dependence treatment case were collected from the 10 participating schools during the comparison data collection period (August 2010 through October 2011). Each encounter lasted approximately 15 minutes or less. Average time required to code an encounter was approximately 31 minutes. Of the 64 randomly selected encounters double coded to check for accuracy, no coding errors were identified on 61 encounters. On the three encounters where a discrepancy in coding of checklist behaviors was noted, the discrepancy involved only 1 behavioral item (2 video encounters) or 2 behavioral items (1 video encounter).

The average global communication rating (based on the mean of the four global communication items) was 3.16 (SD = .46; range 1.5 to 4).

Overall, students performed an average of approximately 8 (24%) of the 33 behaviors included on the checklist (range: 1 to 18; 3% to 56%). The percentage of students performing each of the tobacco dependence treatment counseling behaviors is provided in Table 2. For 16 of the 33 behaviors on the checklist, fewer than 10 percent performed the behavior.

The correlation between the global communication ratings (averaged across the 4 global communication items) and checklist scores on the smoking checklist was .42; however, the correlation did vary by rater (r = .32, .50 and .43) for the 3 raters each of whom viewed and rated 240, 241 and 179 encounters. These values represent medium to large effect sizes,¹⁶ and all were statistically significant (p <.001).

Discussion

This paper describes the development and implementation of an assessment of students’ tobacco dependence treatment counseling skills using an OSCE case. Our primary goal was to design and implement a rigorous method of consistently and accurately capturing students’ performances in order to provide scores that would be valid indications of their abilities to counsel patients about tobacco cessation. Establishing score validity is a process, not an endpoint.¹⁷ Our first step in this process was to carefully review the relevant literature and clinical guidelines. We involved clinicians in checklist development, and both clinicians and experts participated in reviewing the final checklist content. Thus, we are confident that our checklist items represent the key counseling skills appropriate for counseling the patient described in the case materials, a necessary but not sufficient step to establish score validity.

Reliability or consistency in scoring also is a key aspect of validity; in this study we sought to maximize consistency across coders both through careful operationalization of skills, and extensive coder training. We checked consistency and accuracy by having a second “gold standard” coder independently code approximately 10% of the encounters, and found excellent agreement, giving us confidence that coders were able to use the coding checklist to accurately and consistently capture behaviors. Correlations between raters’ global assessments of students’ communication skills and their checklist-based scores were positive and statistically significant, as expected, suggesting that the behavioral checklist scores reflect specific skills which are related to but distinct from general communication skills.

One of the major challenges in cross-school comparisons is ensuring consistency in scoring across schools. Even if case materials are held constant across sites, it is virtually impossible to standardize scoring completely, and site- or SP- specific errors and differences are likely.^{12, 18} Such errors would reduce the evaluator’s ability to detect real differences between schools. By conducting all coding at a central study site, with highly trained, standardized coders, we were able to minimize such errors.

While describing the level of performance of participating students was not the primary focus of the present study, our checklist and explicit criteria provide a detailed description of these students’ proficiency in tobacco dependence treatment, including relative strengths and weaknesses. For example, while approximately 9 out of 10 students in this study asked how much the patient smoked in a day, only 1 in 10 discussed setting a quit date, and almost none (3 of 660) asked how soon after waking the patient smoked, a useful indicator of nicotine dependence.¹⁹ Overall, students tended to focus on pharmacotherapy (e.g. varenicline, bupropion) rather than behavioral strategies or behavioral problem solving (e.g. arranging for referral support, assisting via problem-solving and discussing barriers to quitting). Although few studies have checked for such differences previously, this finding is consistent with the literature on physicians’ behaviors regarding use of pharmacotherapy versus use of counseling and problem-solving techniques. Various surveys reporting physician data specifically on prescribing, for example, have reported higher rates of prescribing compared to arranging for follow-up.^20–22 Another study showed that a greater proportion of physicians perceive pharmacotherapy to be more effective than behavioral counseling or other group treatment programs.²³ These findings provide additional evidence supporting the validity of the checklist scores, and can serve as useful feedback to those responsible for training medical students in these skills, as well as for the students themselves.

While this approach has substantial strengths in providing an objective assessment of students’ tobacco dependence treatment skills, and resulting in scores that are reliable and valid, it also has important limitations. A major limitation is the amount of resources required. Preparation of the detailed behavioral checklist and coding criteria required a considerable investment of time on the part of the study team; training the coders was very resource intensive for the trainer and the coders. Coding the actual encounters was costly. Coders were allowed to pause and replay portions of the encounters, but while encounters were approximately 15 minutes in length, the time required to code was approximately twice that. Independent coding of a sample of encounters to check accuracy was an additional cost. Most medical schools will not have the resources to support these development, training, and coding activities, and will need to rely on checklists or ratings provided by a single SP who is also portraying the case. While the time and effort described here was required to provide a strong evaluation, such an investment typically would not be feasible in a low stakes examination. In fact, less expensive, less resource intensive approaches may be preferable, depending on the purpose of the assessment.

This study focused on coding and scoring encounters, however, training standardized patients to portray the encounter similarly at 10 medical schools around the country also was challenging. Central study staff provided initial case training to the SP trainers at each site, who were then responsible for training and coaching SPs, and for monitoring portrayal fidelity at the site. Our research study staff noted variability in the sophistication and expertise of the SP programs at the participating schools. In some instances, the need to standardize portrayal for the tobacco case required additional SP training, with associated additional costs.

Perhaps a less obvious limitation of the current study is our use of a single case. As noted above, implementation of this single case was costly and resource intensive. In addition, OSCE “space” is limited, and is typically allotted to cover diverse aspects of the curriculum. In most settings it is not feasible to have multiple tobacco-related cases in an OSCE. Given this, it is important to acknowledge that the use of a single tobacco case limits generalizability – we do not know how students would do in other tobacco dependence treatment situations. Perhaps students would be more (or less) likely to solicit the patient’s views on behavioral strategies for quitting if the case presentation were different.

Finally, the checklist described here was based primarily on the widely used 5As framework, as implemented in the context of a brief encounter. Other models of tobacco dependence counseling– such as multi-session treatment or cognitive behavioral therapy would require other assessment tools.

We have described a behavioral approach to defining, coding, and scoring students’ tobacco dependence treatment skills as performed in a single OSCE case. With similar effort, comparable cases, coding rubrics, and training materials could be developed and implemented for other counseling behaviors (e.g. weight management).

These findings suggest we were able to develop a rigorous assessment tool for objective evaluation of students’ tobacco dependence treatment skills. In addition, the coding tool with its detailed description of tobacco treatment behaviors may be a useful teaching tool for medical students.