Medical Students' Self-Reported Typical Counseling Practices Are Similar to Those Assessed With Standardized Patients

Abstract and Introduction

Abstract

Context/Objectives

We surveyed fourth-year medical students in the Class of 2003 at Emory University School of Medicine, Atlanta, Georgia, about various personal and clinical practices. We were especially interested in the frequency that these seniors reported of talking with patients about nutrition, weight, exercise, alcohol, and cigarette smoking. Because the validity of our findings about these counseling practices was limited by our having only self-reported data from seniors' questionnaires, we developed a standardized patient (SP) examination to test the relationship between what students reported on the questionnaires and how they actually performed with SPs.

Design/Setting/Main Outcome Measures

As part of a lengthy questionnaire, 88 senior medical students answered these 5 separated questions: “With a typical general medicine patient, how often do you actually talk to patients about: (1) nutrition; (2) exercise/physical activity; (3) weight; (4) smoking cessation (among smokers); and (5) alcohol? (never/rarely, sometimes, usually/always).” As part of their internal medicine subinternship final exam, students clinically assessed 4 SP cases with predetermined risk factors (poor diet, exercise, alcohol, and/or cigarette-smoking habits).

Results

For every risk factor, the proportion of SPs actually counseled was higher for those students who self-reported discussing that risk factor more frequently with their patients. Additionally, the odds of counseling an SP for any risk factor were significantly higher (odds ratio = 1.76-2.80, P < .05) when students reported more frequent counseling.

Conclusion

Student self-reports regarding patient counseling may be useful when resources are limited, and the purpose is to grossly and anonymously distinguish between higher and lower performers.

Introduction

We surveyed Emory University School of Medicine (Atlanta, Georgia) medical students in the Class of 2003 on 3 occasions (first-year orientation, entry to wards, and senior year). Among many questionnaire items about students' personal and clinical practices, we were especially interested in the frequency that they reported as seniors of talking to patients about nutrition, exercise, alcohol, and cigarette smoking. However, the validity of our findings about these counseling practices was limited by the self-reported nature of these data.

Prior publications on the relationship between students' self-report and their actual performance with standardized patients (SPs) are limited. The few prior studies that have been conducted correlating physicians'/medical students' self-reports and their clinical performance have been generally (although not strongly) positive, finding that clinician underreporting[1,2] of performance may be more common than overreporting.[3] Given the limitations of previous studies, we developed an SP program to test the relationship between what students reported on the questionnaires and what they actually did with SPs.

Methods

In the academic year 2002-2003, we assessed fourth-year medical students' counseling practices at Emory University (n = 88 completing both the SP exam and the questionnaire). We compared students' self-reports of their usual prevention-counseling practices with their actual practices observed during SP exams.

As part of their internal medicine subinternships, students clinically assessed 4 SP cases. A team of clinician-educators with SP experience created the cases. Cases were reviewed by clinician-educators in preventive medicine and internal medicine as well as pilot-tested and refined 3 times with fourth-year medical students prior to their use for testing. Eight cases with identical precipitating factors were grouped into 4 pairs containing (1) myocardial infarction or diabetes, (2) alcoholic gastritis or headache, (3) overweight or hypertriglyceridemia, or (4) reflux or hypertension. Each pair had identical, predetermined risk factors: poor diet, exercise, alcohol, and/or cigarette-smoking habits. To decrease the likelihood that the students might know about the case subject in advance, 1 case from each of these 4 pairs was selected randomly for each month's examination.

Actors experienced in portraying SPs were assigned 2 paired cases and attended 2 intensive, introductory training sessions, participated in 3 pilot tests, and received 2 more follow-up trainings. During training sessions, SPs watched videotaped previous portrayals to promote consistency and portrayed their cases with clinicians who provided feedback. SP training sessions also reviewed instructions for completing the “Standardized patients' assessment of medical students” instrument to record students' querying and counseling on nutrition, exercise, alcohol, and cigarette smoking. Students were marked as providing counseling on a topic if they expressed or discussed the need for improvement in a related behavior with the SP. The Principal Investigator (EF) reviewed randomly selected videotaped cases, and SPs' completed questionnaires throughout the experiment to ensure that SPs were correctly and consistently portraying the cases and completing the assessments.

As part of a larger study, seniors also completed a lengthy questionnaire that included these 5 self-report items: “With a typical general medicine patient, how often do you actually perform this activity? Talking to patients about: (1) nutrition; (2) exercise/physical activity; (3) weight; (4) smoking cessation (among smokers); and (5) alcohol?” Available responses were “never/rarely,” “sometimes,” or “usually/always.”

For each student, we calculated the percentages of SPs who were counseled on the risk factors appropriate for each case. Students were divided into 2 groups on the basis of their questionnaire responses: those who reported talking “never/rarely/sometimes” and those who reported talking “usually/always” about a risk factor with patients. For the 2 groups, we examined the distribution of the proportion of patients counseled by each student. Because most students saw 3 patients needing counseling for each risk factor, the possible proportions were 0, .33, .67, and 1. Because of tardiness or other, inflexible obligations, a few students only saw 2 patients needing counseling for a given risk factor, accounting for the sparse numbers counseling appropriately half the time. The 2 groups were also compared with logistic regression methods appropriate for clustered data (multiple patients seen by each student), run in SAS PROC GENMOD.[4] Our dichotomous dependent variable measured whether counseling regarding the risk factor was performed, and the predictor was the self-reported frequency (“never-rarely/sometimes” or “usually-always”) of discussion on that risk factor. We checked for possible confounding by sex and intended specialty, but they were not needed in the final models.

Results

Our sample of 88 seniors included 42 women (48%) and 11 underrepresented minorities (13% Hispanic, black, and Native American; data not shown). The proportions of patients who were counseled on each indicated risk factor are shown in (Table 1). For every risk factor, the proportion of SPs actually counseled was higher for those students who self-reported discussing that risk factor more frequently with their patients. Results from the repeated-measures logistic regression analysis were significant at the 5% level for every risk factor, with odds ratios ranging from 1.76 to 2.80 (Table 2).

Table 1.

Relationship Between Students' Actual Counseling on a Topic and Their Self-Reported Frequency

Proportion of SPs Counseled on the Topic
Topic	Self-Reported Frequency	n	0	.33	.5	.66	1.0
Alcohol
	Never/rarely/sometimes	37	41%	27%	0%	22%	11%
	Usually/always	51	16%	31%	4%	37%	12%
Diet-nutrition
	Never/rarely/sometimes	66	3%	14%	0%	39%	44%
	Usually/always	22	0%	0%	5%	32%	64%
Diet-weight
	Never/rarely/sometimes	63	3%	13%	0%	43%	41%
	Usually/always	25	0%	4%	4%	24%	68%
Exercise
	Never/rarely/sometimes	62	6%	16%	0%	52%	26%
	Usually/always	26	0%	8%	8%	19%	65%
Tobacco
	Never/rarely/sometimes	27	19%	22%	0%	48%	11%
	Usually/always	61	5%	21%	0%	49%	25%

SP = standardized patient

Table 2.

Repeated-Measures Logistic Regression Analysis Associating the Frequency of Actually Counseling Patient With Self-Reported Frequency

Topic	OR (95% CI)*
Alcohol	1.83 (1.02, 3.30)
Diet-nutrition	2.17 (1.05, 4.50)
Diet-weight	2.28 (1.05, 4.97)
Exercise	2.80 (1.32, 5.93)
Tobacco	1.76 (1.00, 3.10)†

OR = odds ratio; CI = confidence interval

^*The odds of counseling standardized patients in the self-reported “usually/always” category were compared with the odds in the self-reported “never/rarely, sometimes” categories.

^†P = .049

Discussion

This is one of the first studies to validate the accuracy of a medical student counseling self-report. We found that medical students who said that they usually counseled indeed usually did counsel, and clearly did so more frequently than those who said that they did not usually do so. This finding is very useful for us, as it helps to validate the counseling claims of the medical students in our study. This finding may also be useful for other investigators, as obtaining gold-standard data (from SPs or real clinical encounters) to test whether students indeed counsel is expensive and time-consuming.[5]

When added to the scant prior literature (showing a weakly positive relationship between clinicians' self-reported and actual clinical practices[3,4]), our data suggest that a self-report can be useful for grouping students along a spectrum of clinical behaviors for research purposes, although it is likely unsuitable for assessing individual students. This is probably especially true if students' self-assessments have personal implications (ie, are used for grades), as their self-report of counseling frequency may then tend to be inflated. Although not the primary purpose of our investigation, it is worth noting that counseling frequencies were relatively high, especially for non-alcohol-related counseling.

Our comparison between actual and reported counseling is limited by having tested it only at 1 school and by our small sample size. The generalizability of our data is enhanced by the school's typicality: a school size of 454 students (compared with a national average of 532), 50% female (vs 46% nationally), 13% underrepresented minorities (vs 11% nationally), and a traditional curricular structure.[6,7]

The potential utility of student self-report is encouraging, and may be better than other proxies for actual practice in some circumstances. For example, we found poor correlations between the written clinical vignettes that we piloted and our students' actual counseling practices with SPs. Likewise, there are varying correlations in the literature between actual clinical performance and methods, such as vignettes, chart review, and patient reporting.[5,8] Students' self-assessments may be useful when resources are limited, and the purpose is to grossly and anonymously distinguish between higher and lower performers.