Pharmacy Students' Perceptions of a Teaching Evaluation Process

Abstract

Objective

To assess PharmD students' perceptions of the usefulness of Duquesne University's Teaching Effectiveness Questionnaire (TEQ), the instrument currently employed for student evaluation of teaching.

Methods

Opinions of PharmD students regarding the TEQ were measured using a survey instrument comprised of Likert-type scales eliciting perceptions, behaviors, and self-reported biases.

Results

PharmD students viewed student evaluation of teaching as appropriate and necessary, but conceded that the faculty members receiving the best evaluations were not always the most effective teachers. Most students indicated a willingness to complete the TEQ when given the opportunity but expressed frustration that their feedback did not appear to improve subsequent teaching efforts.

Conclusion

The current TEQ mechanism for student evaluation of teaching is clearly useful but nevertheless imperfect with respect to its ability to improve teaching. Future research may examine other aspects of pharmacy students' roles as evaluators of teaching.

INTRODUCTION

Even after years of study and considerable debate, the value of student evaluations of faculty teaching remains a contentious issue. Opponents of student evaluations cite the well-known “Dr. Fox” study, in which a nonsensical lecture delivered by a persuasive actor was afforded high ratings by a professional audience.¹ Student evaluation proponents point out the many flaws in this study as well as evidence that students are careful discerners of teaching quality.² Overall, students appear to be appropriate, albeit imperfect, evaluators of good teaching, and are better judges of some aspects (eg, creating an environment conducive to learning) than others (eg, expertise of the faculty member).³

A survey of nearly 700 pharmacy faculty members found that this population was neutral regarding whether student ratings were the best mechanism for evaluating their teaching.⁴ A primary criticism of student evaluations of teaching among Duquesne University School of Pharmacy (DUSOP) faculty members is that the process amounts to a teacher “popularity contest.” According to this argument, the highest TEQ scores go to professors who (1) focus on entertaining the students during a lecture; (2) teach courses that students believe are “more important”; and (3) liberally dispense “A” grades. However, the literature on student ratings of teaching (discussed below) suggests that such practices may not necessarily lead to high TEQ scores. Ironically, scores from pharmacy faculty self-evaluation of teaching do not differ significantly from student rating scores of the same teaching effort.⁵

The finding that a charismatic individual posing as an instructor could elicit high teaching ratings¹ implies that the presentation style or personality of the instructor may influence the TEQ score. While students appreciate friendly, humorous instructors, these attributes are minor considerations in student evaluations of teaching. In fact, students criticized even popular instructors if the course was disorganized or unmotivating.⁶ If an instructor employs humor, anecdotes, or other strategies to fill class time chiefly in an effort to be popular or to decrease his or her workload, students may not learn the material necessary to pass board examinations and become competent pharmacists. On the other hand, by employing such strategies judiciously in an “active learning” environment that allows for feedback and discussion during the lecture, the instructor may stimulate learning and inquiry. Expressiveness and enthusiasm on the part of the instructor enhances learning and may mildly increase student rating scores of teaching,⁷ but instructor popularity itself does not assure good scores.⁶

If the level of rigor for the course is higher than average, one might expect students to penalize the instructor via the evaluation. This line of thinking also implies that instructors who teach “low concept” courses or courses that require little student effort outside of class will be rewarded with high TEQ scores. The literature on the subject suggests quite the opposite. Students are reported to give higher rating scores when the course is perceived to be difficult or require hard work.^7-9 Nevertheless, instructors in science courses receive lower student ratings than those teaching non-science courses,^6,10 an inconsistency, assuming that science courses are more difficult. It may follow that instructors of science-based pharmacy courses are rated lower than instructors of non-science pharmacy courses in a pharmacy curriculum; alternatively, students may select pharmacy as a major primarily for their interest in science-based courses and thus may evaluate non-science pharmacy courses lower.

The assumption that students tend to rate instructors higher who give higher grades is questionable at best. From reviews^6,7 of over 60 articles dealing with the subject, the consensus is that there is little or no correlation between higher student ratings of teaching and higher grades for that course. For those studies in which a positive correlation between higher ratings and higher grades was observed, it has been hypothesized that the students who learned more earned higher grades and gave higher ratings.⁷

To the best knowledge of the authors, the pharmacy student's perception of his or her ability to evaluate teaching quality and the evaluation mechanism has not been studied. This is a timely issue in that DUSOP students and faculty members regularly voice their displeasure with the process, and because Duquesne University is in the process of re-evaluating this mechanism.

METHODS

To determine how DUSOP PharmD students evaluate their instructors, a comprehensive survey instrument that addressed some of the aforementioned issues was constructed by the authors. First-, second-, and third-professional year Duquesne University PharmD students were separately asked by a pharmacy school administrator unrelated to the project to complete the survey instrument on Duquesne University's Teaching Effectiveness Questionnaire (TEQ). The questionnaire was completed during the first 10-15 minutes of a required, regularly scheduled class. The survey instrument elicited Likert-type scale responses of “1” to “5” that indicated degrees of agreement, empathy, or importance (Table 1), plus a “check the appropriate blank” section and a section for student comments (the complete survey instrument is available on request). The survey instrument was designed to assess student opinions of the TEQ, which was the approved mechanism used by the University to student-evaluate teaching. The survey instrument also sought to determine whether students valued the TEQ, to understand the rationale behind their answers, and to detect any student bias in the process. Survey items were generated in consultation with the literature along with anecdotal information acquired from faculty members and students, and feedback from a sixth year PharmD student undergoing a clerkship experience on teaching pedagogy.

Table 1.

Pharmacy Students' Responses to Survey Items Regarding the Quality of the Teaching Effectiveness Questionnaire (N = 377*)

^*Since not all students responded to all statements, the number of responses for each item varied from 374-377.

^†Scales ranged from 1 to 5, with 5 indicating greatest degree of agreement or similarlity.

Statistical analysis of survey results was conducted using SPSS 11.0 software. Descriptive statistics were tabulated. An exploratory factor analysis procedure employing principal axis factoring of the 11 items measuring student perceptions was undertaken to reduce the data into workable domains for subsequent analyses and to better understand how students conceptualize the TEQs. Factor scores were subjected to oblique rotation, allowing for correlation among the putative domains, and subsequently saved as dependent variables for use in multivariate analyses of variance (MANOVA) procedures to determine differences in perceptions among students as a function of class and gender.

RESULTS

Quantitative Survey Results

Three hundred seventy-eight first-, second-, and third-professional year (third, fourth, or fifth year) PharmD students participated in, and 369 completed, the TEQ survey (Table 1). First-, second-, and third-professional year students comprised 42%, 31%, and 27% of the respondents, respectively, numbers that correlate well with the size of each class. Of the 93% of respondents that indicated gender, 66% were female, a value slightly higher than the percentage of registered female PharmD students.

The majority of students indicated that evaluation of teaching was necessary (statement 1) and appropriate (statement 8). Most students indicated willingness to complete a TEQ when given the opportunity (statement 12). Students also felt strongly that all faculty should be evaluated (statement 4). Regarding the latter issue, students largely disagreed with the notion that senior faculty should be less frequently evaluated than junior faculty (statement 5). Many students voiced their displeasure that tenured faculty can exempt themselves from evaluation (see “comments” section, below), and the majority felt that the promotion or tenure of a professor should be linked to TEQ results (statement 3). The majority of students indicated they did not believe their professors accept, or even read, the constructive criticism offered by students (statements 2 and 6). Students acknowledged that the personality of the professor affected their TEQ responses (statement 10), and even influenced their decision on whether to complete the TEQ (statement 13). Moreover, the majority of students agreed somewhat with the contention that professors who receive the best TEQ scores are not necessarily the most effective teachers (statement 9).

Table 2 shows the factor loadings of the 11 items eliciting student perceptions about the TEQ. The factor analysis procedure allows for statistical grouping of items based upon patterns of students' responses. The results yielded 3 domains or “factors” that can be said to compartmentalize how students perceive the faculty evaluation process. Factor 1 is comprised of items evaluating the appropriateness of students as judges of teaching quality. Factor 2 deals with the importance and efficacy of the process in improving faculty teaching. Factor 3 deals with relationships between students and faculty resulting from the administration of TEQs. Thus, based upon the items comprising this survey, students' perceptions are governed by perceptions of their appropriateness as judges of teaching quality, the importance they place on the process, and the impact this process has on faculty-student relationships. Item 3 exhibited some cross-loading by loading modestly onto Factor 1 in addition to Factor 2. This indicates that students who view themselves as appropriate judges of teaching quality also place greater importance on the entire teaching evaluation process. Item 5 demonstrated only a modest loading onto Factor 3. Item analysis provided additional evidence that this item should be modified or removed in future iterations of the scale. Cronbach's alpha values for Factors 1 and 2 were estimated at 0.60 and 0.57, respectively, while the correlation between Items 7 and 9 (Factor 3) was subtle (0.06, p = 0.24). These results are indicative of modest internal consistency reliability.¹¹

Table 2.

Factor Loadings^* of Items Measuring Pharmacy Students' Perceptions of the Teaching Effectiveness Questionnaire (TEQ).

^*Factor 1 is comprised of items evaluating the appropriateness of students as judges of teaching quality. Factor 2 deals with the importance and efficacy of the process in improving faculty teaching. Factor 3 deals with relationships between students and faculty resulting from the administration of TEQs.

^†Bold type indicates a strong loading by the item onto the respective factor.

There were significant differences in perceptions on Factor 1 by gender, and on Factors 2 and 3 by class cohort. A post hoc t test revealed that female students indicated higher levels of agreement with items comprising Factor 1 (appropriateness of students as judges of teaching quality). Post hoc one-way analyses of variance revealed that third-professional year students expressed greater agreement than second-professional year students that faculty members currently use the TEQs to improve their teaching. Although respondents as a whole disagreed with the premise that less coursework translated into higher TEQ scores, male students (mean = 2.3) were more likely than female students (mean = 2.0) to indicate that less demanding courses were indeed rewarded with higher scores. Female students (mean = 4.3) indicated a greater propensity than male students (mean = 4.0) to fill out the TEQ when given the opportunity. Female students were apparently less likely (mean = 3.9) than male students (mean = 4.3) to fill out the TEQ when the professor taught especially well or poorly. Of 369 students responding to an item soliciting their behavior in evaluating faculty members by gender, 363 (98.4%) indicated that they provided identical scores to male and female instructors for the same teaching performance; the remaining 6 students (1.6%) were evenly divided in their scoring preference for a male or female instructor. Similarly, 359 students (97.0%) indicated that their scoring was unrelated to an instructor's age. Five respondents (1.4%) indicated a slight preference for younger faculty member; 6 students (1.7%) indicated a slight or definitive preference for older faculty members.

Written Comments

The 190 student responses for this section can be grouped into 3 chief concerns. Two of the concerns centered on students' perception that their comments “fell on deaf ears,” and the third dealt with a perceived lack of accountability on the part of tenured professors.

“The professors don't read our comments, so why bother filling out this part of the TEQ?” Many students indicated that they had offered constructive criticism using the write-in format in the past, but gave up when the professor's subsequent teaching suggested that the advice went unheeded. Approximately half of the comments reflected that the students had no reason to believe that their responses were valued, and several mentioned that they therefore now devoted less thought and effort to the evaluation. Students suggested more than once that they should be given access to the results of the evaluation. Nevertheless, the more senior students were more optimistic that their TEQ feedback could lead to improved teaching.

“How does the Administration allow him or her to keep teaching? They must not pay attention to the TEQ results.” There were many comments to the effect that the student's tuition pays the professor's salary, so the professor should make teaching a higher priority than research or other duties. Other comments indicated that the students expected the professor to face career-affecting consequences for a poor teaching effort, the least of which would be that the professor not be allowed to teach PharmD students if a pattern of substandard teaching is evident.

“The tenured faculty most urgently need these evaluations, yet we never get to fill out TEQs for them.” Some third-professional year students revealed their frustration that they had never been asked to evaluate certain tenured professors even though these professors could have benefited most from the feedback. The majority of student responses opined that as a rule, the more senior, tenured professors devoted less effort to teaching because they no longer had to answer for poor teaching performances. Students typically recognized the difficulty in rectifying the perceived problem, but nevertheless suggested quality control measures. These included occasional, unannounced lecture attendance by the dean or associate dean, and that the written TEQ comments are first seen by the dean, associate dean, and other appropriate university administrators.

Other notable comments in this section of the TEQ do not easily fit into the above 3 categories. A handful of responses requested that school administrators be similarly evaluated. Other students thought that they should be given incentive (eg, “extra credit” in the class) to complete the TEQ, or that the TEQ should be completed online and outside of class time, when the student could more thoughtfully compose a response. Students were divided on when the TEQ should be administered. Some felt that the last day of class, or even after the final examination, was most appropriate, apparently to maximize professor accountability. Others felt that mid-semester was a more useful evaluation point, in that the professor could obtain feedback in time to improve that course.

DISCUSSION

Pharmacy faculty members appear diffident on the subject of student evaluations of teaching, yet over 96% of these faculty members revised their course in some manner in response to the student evaluations.⁴ The finding that pharmacy faculty members' self-ratings are consistent with those of student ratings⁵ suggests that student evaluations of teaching are valid indicators of teaching quality. Wolfgang et al¹² noted that pharmacy faculty members would prefer that more emphasis be placed on teaching for promotion and tenure decisions. Respondents to their survey expressed a preference that administrators use more creative means to assess teaching quality, but still acknowledged the value of student feedback.¹²

Over 85% of all registered students for these 3 years participated in the present survey, a high level of cooperation considering that this percentage does not account for registered students absent from class that day. The survey indicated that PharmD students recognize that rating the teaching of their instructors is appropriate and necessary. Female students may place more faith and concern in the TEQ process, based on analysis of Factor 1 and their greater likelihood of completing the survey. Although the more senior students displayed some faith that their comments were taken to heart, the students as a whole questioned whether the faculty members seriously consider the students' feedback. Interestingly, the senior students also expressed consternation that tenured professors could be shielded from this form of scrutiny. It cannot be determined from the study whether the differences in perceptions among the first-, second- and third-professional year classes are attributable to longitudinal or cohort effects. In keeping with the literature on the subject,^6,7,13,14 over 98% and 97% of the student respondents were also indifferent to the gender and seniority of the professor, respectively, and the remaining 2%-3% were evenly split (data not shown). Some students did admit to bias regarding course workload and faculty personality, as just over 30% and 55% indicated agreement with the midpoint of the scale or higher on statements 11 and 15, respectively. A course that the student perceived as less demanding in content or workload would not be expected to translate into higher TEQ scores for that professor, in agreement with previous studies.^8,9,15 An instructor's winning personality was viewed as a positive feature and appeared to improve the TEQ score, yet was not considered a prerequisite for excellent teaching. The finding that instructor personality could influence TEQ outcome countered the conclusions of several other studies.⁵ Still, these results corroborate evidence that student ratings of teaching rise with the enthusiasm of the professor even when student learning is unaffected.¹⁶

Faculty members, administrators, and students should be mindful of several factors concerning evaluation of teaching. Students appreciate the opportunity to voice their opinion toward improving teaching quality, even if some students are somewhat biased in their judgments or cynical of the entire process. Therefore the college or school must take the opportunity to educate students about the evaluation process. For example, statement 7 (Table 1), designed to assess student understanding of the TEQ process, implies that it is possible that the professor has access to TEQ feedback before he or she administers the final examination. Actually, the Duquesne University professor does not have this feedback even 2 months after the examination. The survey results suggest that most students understood University procedure regarding this issue, but the broad range of responses (data not shown) for, and occasional comments on, this statement indicated significant student confusion. The result suggests that the administration may need to more fully explain the mechanics of the TEQ. Students should also be made aware that their input carries considerable weight in career- and curriculum-affecting decisions, and thus should not be frivolous or vindictive. Specifically, students should be apprised of college or school policy on the frequency of student feedback for tenured and non-tenured faculty members, and to what extent their feedback impacts promotion and tenure decisions. Students must also appreciate that changes in teaching quality occur incrementally; wholesale changes in course design or teaching performance cannot be expected from one iteration of student feedback. As for instructors, they should additionally seek out quantitative and qualitative student feedback using their own feedback mechanisms to improve teaching and rapport during a course. The gesture itself connotes a sincerity that, coupled with readily observable, substantive improvements, should translate into better teaching and better evaluation scores.

Equally important to the student evaluation is adhering to tenets of good item-writing in constructing the university's survey instrument. Most psychometricians and survey methodologists agree that the use of words such as “fair” can have multiple meanings and can emotionally charge the respondent. Using generic terms (eg, “availability”) may either confuse student respondents or yield various interpretations; an instrument with such items is subject to lower reliability and is more prone to elicit student biases.^17,18 Items that elicit student opinions about specific instructors' behaviors will likely yield more reliable results.^19,20 For example, instructor availability can be rated by the student through multiple items, such as, “The instructor promptly returns e-mail communication,” or “The instructor is present during office hours.” Still, even diligent item-writing efforts carry unavoidable potential limitations.

The results of this study suggest avenues for future research. While the goal of determining the “validity” of student evaluation of teaching performance may be elusive, its correlation with peer reviews and other means of assessing teaching quality could be investigated. This may shed light on the specific strengths and limitations of student evaluations of teaching performance. Student evaluation scores for sole instructor versus team-taught courses could be compared for a given instructor, including whether the team-taught scores were influenced by the contributing instructors' pharmacy discipline. While the results of this study have external validity, they reflect the opinions of pharmacy students at only one university. A multi-site study would be particularly useful in identifying student biases specific to the field of pharmacy.

CONCLUSIONS

Duquesne University PharmD students acknowledge the need for evaluating their instructors as a quality control measure of teaching. As a whole, students indicated that they complete the TEQ survey in an unbiased fashion to yield useful feedback for the instructor and administrators that influence promotion and tenure decisions. The students were nevertheless quite cynical about the premises that instructors modify their teaching in accordance with the constructive criticism, and that instructors are held accountable by the University for substandard teaching efforts. Fully educating the students on the evaluative process and administrative uses of the resultant scores might restore their faith in the process. Meanwhile, the instructors could take more initiative in obtaining teaching effectiveness feedback throughout the course for the sake of improving the product. Imposing on tenured instructors more accountability for their teaching performance is a very challenging but necessary remediation; failure to do so will continue to compromise the teaching evaluation process and cost pharmacy faculty and administrators credibility in the eyes of the students.