A 1-Day Course to Improve the Teaching Effectiveness of Health Professions Faculty Members

Abstract

Objective. To determine the effect of a 1-day teaching-methods course for pharmacy and nursing faculty members on teaching proficiency and perceptions of effective teaching.

Design. A 1-day teaching-methods course was created and presented to 12 pharmacy and nursing faculty members. Participants’ teaching sessions were video-recorded pre- and post-course.

Assessment. A panel of educators evaluated pre- and post-course video-recorded teaching presentations to assess teaching effectiveness. Participants completed pre- and post-course self-evaluations and surveys. Panelists’ assessments confirmed significant improvement in 7 of 10 domains. Perceptions of teaching efficacy and proficiency to teach effectively improved after completion of the course.

Conclusion. A 1-day teaching-methods course, requiring minimal resources, is a valuable development resource that can improve teaching effectiveness through enhanced communication and teaching techniques. Because effective teaching is positively correlated with students’ learning, training seminars can be integral to comprehensive quality improvement.

INTRODUCTION

Effective teachers motivate, educate, and exhilarate their students. Unfortunately, not all who teach stimulate the minds of their pupils. Student teachers of primary and secondary education learn through specific didactic curricula as well as practice application during internship training.^1,2 However, US educators with advanced degrees in other fields who chose academia as a profession receive little to no formal training in the actual process of teaching.^1,3-5 According to Knapper, “Many faculty lack a sophisticated conceptual understanding of how learning takes place in college students.Most faculty have been trained as researchers in their discipline but have had no background in educational or pedagogical theory.”⁶ Moreover, instruction in effective teaching techniques is not a standard curricular component of most health-professions education, despite that accreditation bodies require quality educators to deliver the curricula. The current system simply does not teach teachers how to teach.

Although lack of teaching skill is not a new problem, no solution has yet been identified for scholars who choose a career in higher education.^1,5 Graduate education, which often focuses on research design and techniques, does not sufficiently prepare students to teach, despite many pursuing academia upon graduation.⁷ Health-professions educators face the additional challenges of faculty shortages, lack of extrinsic incentives, and career opportunities in clinical practice and industry, which often outweigh the desire to learn how to be a better teacher.⁸ Draugalis and Plaza emphasize the importance of educators receiving appropriate training in their assertion that “proficiency in teaching and service roles will not be achieved if left to osmosis or simple modeling.”⁷ Health-professions education should be invested in effective teaching because of the negative impact poorly trained graduates will have on the future health care of society.⁹

There is a positive correlation between individual teacher skills and student academic performance, such that quality of teaching is thought to contribute significantly to student achievement of learning outcomes.^5,10 Without sufficient training and development, educators potentially lack the knowledge, skills, and abilities to maximize the intended positive impact of teaching. An important point that educators should consider is why the education system tolerates ineffective teaching when the impact on society is so significant.

Historically, evaluation of effective teaching has not been well established.¹¹ Although there have been recent efforts to promulgate the idea of teaching excellence recognition, many institutions of higher education have not yet developed effective systems for assessment.^12-14 There have been efforts to remedy the problem, but there is still considerable disagreement regarding the utility of student evaluations compared with peer- or self-evaluation systems.^15,16 While there are no formal guidelines for evaluating effective teaching, meeting this challenge may lie in establishing definitions of effective teaching.

A common response when asked to describe a skilled educator is, “You know a good teacher when you see one.” An efficient way to increase the number of effective teachers would be to identify consistent attributes and traits and then inculcate these to current and future educators. Unfortunately, these attributes can be difficult to quantify. Bain completed a longitudinal study to identify the characteristics of effective teachers across multiple higher-education disciplines, compiling student input over 15 years.¹⁷ Across all disciplines studied, consistent traditional themes emerged, such as knowledge in subject area, preparation and organization, and treating students with “simple decency.”¹⁷

Although training teachers to be effective is the expected approach, the reality is surprising: accessible teaching development opportunities are lacking in higher education.¹ The Graduate Diploma of Tertiary Education program exists in Australia. In the United Kingdom, accreditation of higher education teacher programs is available.¹⁸ Unfortunately, the United States offers few opportunities. Teaching-certificate programs completed during residency training early in the career of a clinically trained academic are not widely available and include limited curricula with experiential components designed to provide basic skills, such as syllabi development, presentation techniques, and application.^19,20 A handful of faculty-development programs supporting teaching excellence, including Johns Hopkins Bayview Medical Center Faculty Development Program and University of California, San Francisco, School of Medicine Teaching Scholars Program, can be found using an Internet search.^2,21 These programs reflect positive outcomes documented by means of subjective assessment; however, opportunities such as these are neither standard nor ubiquitous. Limitations in resources, lack of release time, and modest capacity result in programs that are offered to few individuals on a one-time basis.^2,21,22

This study considered whether a modest investment of time and resources in training as an alternative to a costly, limited-access certificate/scholar program could improve teaching proficiency. The goal of this study was to determine the effect of a 1-day teaching-methods course on the teaching proficiency of nursing and pharmacy faculty members and to establish whether the course expanded their perceptions of effective teaching.

DESIGN

A 6-hour teaching methods course was designed for faculty members in health-professions education. Teaching effectiveness was defined as the ability to display knowledge of the subject, project effective disposition, and utilize communication techniques.²³ The study was approved by the institutional review boards at A. T. Still University, Pacific University, and Washington State University (WSU), and all participants provided informed consent.

To explore interprofessional aspects, nursing and pharmacy faculty members were recruited as participants. Teaching proficiency of participants was evaluated by a panel of educators using a validated teaching performance tool.²⁴ The panel was composed of exemplary faculty members representing the College of Pharmacy pharmacotherapy department, the College of Nursing, and the College of Education. Panelists were exemplary educators identified as individuals with a minimum of 5 years’ teaching experience who were actively teaching and had received significant teaching expertise recognition (ie, award recognition, multipeer recognition, and administrator recommendation). Other measurements included a self-evaluation of performance and a perceptions survey. Goals for the 1-day teaching workshop included developing a template for effective lecture-presentation skills to better connect with students and incorporating active-learning techniques in presentations to enhance learning. After completing the course, we anticipated that participants would be aware of the value of clear communication, lecture organization, and the integration of active-learning techniques in effective teaching. Each participant's teaching effectiveness was evaluated by the panelists following a short sample presentation completed pre- and post-course.

Participants in the teaching methods course were a cohort of faculty members from Washington State University, a public university of over 25,000 students. The university houses professional programs, including the College of Pharmacy and the College of Nursing on multiple campuses. Recruitment for study participants was completed through program administrators in nursing and pharmacy education to secure balanced representation between the disciplines.

Participants selected the topic of their pre- and post-course presentation approximately 2 weeks prior to the course. Panelists received on-site training to ensure consistent use of the teaching-effectiveness instrument. One day prior to the course, participants completed a precourse video-recorded presentation to provide a sample of their teaching for evaluation. Immediately following the presentation, participants completed a presentation self-evaluation, designed to encourage reflection and awareness of teaching effectiveness. The 6-hour teaching methods course occurred the following day. Participants completed a pre- and post-course teaching effectiveness perceptions survey instrument, designed to evaluate their perceptions of teaching efficacy and active-learning techniques and to measure any change after completion of the course. Following the course, participants completed their video-recorded postcourse presentations and a presentation self-evaluation.

The course was a lead-by-example design, in which many of the skills and techniques integrated in the course objectives were introduced and explained. The investigator led interactive discussions, highlighting effective teaching techniques and applying active learning throughout. Online links, such as recommendations for effective presentations, were emphasized to provide future resources. Participants developed a template for an effective lecture, with emphasis on measurable learning objectives. Participants also practiced active-learning techniques in pairs and in small groups during the course. These skills and activities were then incorporated into their own teaching skills and presentation design. Participants worked on their presentations throughout the course, updating and revising material as well as practicing new techniques.

EVALUATION AND ASSESSMENT

Three instruments were used in the project. Panelists used the formal observation summary instrument, a 10-item validated assessment tool, to objectively evaluate participant presentations. This evaluation tool was selected because of its documented use in evaluating teachers in colleges of education and in primary and secondary educational practice.²⁴ The instrument consisted of 3 domains (planning and preparation, the classroom environment, and instruction) combined for a total of 16 items defining expectations of educators’ capabilities. There were 4 delineations for each level of the scaled evaluation tool: unsatisfactory, basic, proficient, and distinguished.²⁴ Panelists were trained using a prerecorded sample presentation detailing how to apply the instrument. During training, 10 of the 16 measurements were determined to be applicable to the project design, and the panelists extrapolated evaluations for the 10 identified areas. For example, “instructional outcomes” was interpreted as the ability of students to gain understanding from the presentation. Table 1 provides a list of the10 areas of measurement.

Table 1.

Descriptive Statistics for Participants’ Pre- and Post-Course Assessment-Domain Responses

The panelists requested that each participant be asked 4 questions throughout the presentation to simulate audience interaction. The questions were based on “typical student questions” related to clarifying a concept, challenging a statement, and requesting an example or additional background. The participants’ self-evaluation instrument was modeled after the panelists’ tool used by converting it to a 5-point Likert scale with qualifiers from 1 = strongly agree to 5 = strongly disagree.

The teaching effectiveness perceptions survey utilized a similar 5-point Likert scale consisting of a 16-item survey with 5 domains: identification of problem, effective teaching methods, ineffective teaching methods, active-learning techniques, and effective teacher traits. The statements were developed using perceptions accumulated from interviews with exemplary educators in nursing and pharmacy education from multiple universities.²⁵ Negatively phrased questions were interspersed among the positively phrased questions.

The project was designed to include 12 participants equally representing the 2 schools. However, scheduling difficulties resulted in 3 nursing and 9 pharmacy participants. Participant demographics are presented in Table 2.

Table 2.

Demographic Characteristics of Faculty Members Participating in a 1-Day Teaching-Methods Course

Examination of items within pre- and post-course assessment on all measures revealed a significant lack of homogeneity of variance between many study assessment item pairs. Because of this and the relatively small sample size, nonparametric analyses and paired-samples sign-test analyses were conducted to determine changes from pre- to post-course assessment for all individual study questions. These analyses determined significant changes from pre- to post-course assessment by examining the number of individual responses that improved, decreased, or remained the same between assessment periods.

With respect to the summation of scores on the objective performance measure (ie, the formal observation summary instrument used by the panelists), a significant improvement was found between precourse (mean = 23.8, median = 21.9) and post-course (mean = 27.5, median = 26.8) assessments (p [2-tailed] = 0.006). The sign-test procedure was used to determine significant changes from pre- to post-course assessment. The paired-sample analysis indicated significant differences between pre- and post-test assessment (p < 0.05) for 7 of 10 performance descriptions assessed by the panelist evaluators. Scores were determined using a scaled format with a 4-choice rating: 1 = unsatisfactory, 2 = basic, 3 = proficient, and 4 = distinguished.

The analysis included determination of type of change between each participant's pre- and post-course evaluation: “positive change” identified improvement in score, “negative change” identified decline in score, and “no change” indicated the same relative score pre- and post-course. Descriptive statistics for each assessment domain are presented in Table 1, and the number and type of changes from pre- to post-test assessment are presented in Table 3.

Table 3.

Number and Type of Changes Between Pre- and Post-Course Assessment

Interrater agreement analysis of the 3 panelist evaluators was assessed using the intraclass correlation coefficient (ICC) approach. Use of the ICC was deemed more appropriate than the Kappa coefficient, which is used for interrater reliability if there are just two raters. Determination of interrater agreement was based on published standards.²⁶ Standards of agreement were described using a 5-level determination: poor agreement = ICC < 0.20, fair agreement = ICC 0.20 – 0.40, moderate agreement = ICC 0.41 – 0.60, good agreement = ICC 0.61 – 0.80, and very good agreement = ICC 0.81 – 1.00. The data indicated high agreement among raters, including “good” and “very good” pre- and post-course in 13 of 20 (65%) and moderate agreement in 4 of 20 (20%) performance descriptions representing 3 domains. Only 3 of 20 (15%) were considered poor agreement. The areas of poor agreement were pre-course planning and preparation - knowledge of resources; post-course planning and preparation - knowledge of content; and post-course planning and preparation - knowledge of resources.

The pre- and post-course teaching effectiveness perceptions survey, which was designed to evaluate the participants, provided additional information regarding the change in participants’ perceptions of effective teaching pre- and post-course. The sign-test procedure was used again to determine significant changes from pre- to post-course assessment by examining the number of individual responses that improved, decreased, or remained the same between the 2 assessment periods. Using data collected in an earlier study with exemplary educator interviews,²⁵ analysis revealed that some participant perceptions changed after the course and aligned with the current perceptions of effective teaching. On 6 of the 14 (43%) statements related to effective teaching, participants’ perceptions significantly moved from nonalignment to alignment with those of exemplary educators (p < 0.05). Changes from pre- to post-course evaluations are presented in Table 4.

Table 4.

Changes in Participants’ Perceptions of Their Teaching Effectiveness

Data from the pre- and post-course presentation self-evaluations were analyzed to determine the presence of significant differences in the participants’ perception of their teaching capabilities before and after the course. The sign-test procedure was used to determine significant changes from pre- to post-course assessment by examining the number of individual responses that improved, decreased, or remained the same between assessment periods. The presentation self-evaluation, which was adapted from the panelist evaluators’ formal observation summary, contained statements based on the 3 domains and subsections for a total of 19 performance descriptions on a 5-point scale: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree. Significant differences were found in 3 of 19 (16%) performance descriptions in this instrument (Table 5).

Table 5.

Changes in Participants’ Presentation Self-Evaluation Scores

Relationships between teaching proficiency, as measured on the formal observation summary, and presenter characteristics (eg, age and years of teaching experience) were assessed. A single score consisting of the sum of objective pre- and post-course performance measures was created for each participant. A Pearson-correlation procedure indicated a significant positive correlation between age and performance for both the precourse (r [12] = 0.73, p = 0.007) and postcourse (r [12] = 0.65, p = 0.022) evaluations. A greater number of senior teachers were evaluated as performing better on the formal observation summary, particularly for the precourse presentation. A significant positive correlation was also found between years teaching and performance for both the precourse (r [12] = 0.73, p = 0.007) and the postcourse (r [12] = 0.58, p = 0.049) evaluations. Higher scores on the formal observation summary were associated with more reported years of teaching experience.

DISCUSSION

The teaching methods course used in the present study was designed to aid in the examination of a 1-day, minimal-resource workshop. The course provided an opportunity to evaluate the influence of training on improving teacher capabilities while supporting networking among faculty members from 2 colleges within the same university, with minimal financial or time investment.

The findings of this research indicate promising results, as measured by improvement in objectively evaluated presentations, and qualitative findings, as seen in overwhelmingly positive anecdotal input from the participants. The multiperspective evaluation, wherein the expert panelists’ assessments showed significant improvement in 7 of 10 domains of teaching effectiveness, is a strength of this research. Positive comments were documented on postcourse evaluations completed by the panelist evaluators. The quantitative and qualitative results indicate value for improved teaching efficacy for the cohort examined. As expected, the data suggest that participants showed improvement in their perceptions of teaching efficacy as a result of the course and adopted the perceptions of exemplary educators.

When educators apply new teaching approaches, preparation and practice are encouraged to refine performance and display proficiency.^2,21 The 1-day course may not have provided sufficient time for the participants to fully implement, practice, and master improvements in their teaching. The results appear to show improvement immediately following the course, which may indicate influence of the type of participant interested in self-selecting to participate in the project.

A 6-hour course was selected to address the need for presentation of new ideas followed by discussion and application while fitting into a 1-meeting model. Published training courses range from 3 hours/week over a period of 12 months,² 4 hours/day in 1- to 6-week modules over 9 months,²² to 6 hours/day taught over 1 week.²⁷ A lunch was purposefully included to encourage ongoing discussion and networking among participants. Resources for the course educational material included teaching texts, online university learning centers, and personal experiences. Too few details were available for a comparison between the educational content of this course and published teaching programs.

The size of the sample in this project was small, and an equal representation of nursing and pharmacy faculty members was not attained. A cohort of 12 participants was chosen based on other published teaching programs, which focused on small-group training (cohorts of 9 to 25). The cohort represented 2 colleges within a single university. Broadening and increasing the cohort in future studies would allow greater extrapolation of value.

Evaluation of teaching performance by the panelists supported an “expert” version of the peer-assessment model, which has not been extensively studied in the literature. Significant limitations exist in published studies of educator training, which rely solely on student evaluations of teachers (SETs) or faculty self-evaluations. The use of peer evaluations is not without limitations as well and may be influenced by differences in evaluator training and background. Combining objective and subjective presentation evaluation was intentional, as few previous studies have evaluated training programs objectively. Using panelists to evaluate the pre- and post-course presentations provided an opportunity for objective evaluation. Because of their accomplished background and experience, panelists in this project may have been more critical of the participants. However, panelist assessments showed improvements that were significant and interrater variability of the panelists appeared minimal. An analysis of interrater reliability displayed high agreement in more than half of the evaluations, despite representing 3 different fields of study (education, nursing, and pharmacy). To reduce potential bias in evaluation, future studies should consider use of a blinded panel and a control group whose members do not participate in a teaching-methods course.

The use of the formal observation summary instrument provided valuable insight regarding the objective evaluation of teaching by a peer. Incorporation of this tool or a similar version may provide an additional dimension of evaluating teaching efficacy that is currently unavailable. The rubric design provides concrete feedback that should be useful to educators striving for improvement and development.

This research project was designed to measure the difference in teaching proficiency pre- and post-course. It is possible that self-selected project participants were highly motivated to improve their teaching efficacy or had already embarked on trials of new teaching methods. As such, participants may have been primed for success. In future research, a random sampling of participants would provide a potentially less-biased cohort. In this project, it may have been helpful to complete a follow-up evaluation at a point in the future, data from which could be used to evaluate retention and application of the effective teaching techniques introduced in the course.

Participants were not surveyed at the conclusion of the course regarding the course format, what they had learned, what they still did not understand, and suggestions to improve the course in the future. The investigators avoided this technique of evaluation (often referred to as the “minute paper”) to avoid overwhelming the participants with surveys. Evidence of ongoing improvement in teaching may have further elucidated the value of this project. Ideally, a variety of opportunities for teaching training would be readily available to meet the varied needs of current educators.

Participants provided unsolicited anecdotal feedback after the project was completed. Based on e-mails and verbal communication, participants valued the course and received it positively. One stated that the course “resulted in a significant improvement on my teaching reviews this year.” Another participant created a poster for a national meeting, describing how the teaching-methods course improved teaching effectiveness as measured through student surveys and subjective input.

From a more global standpoint, perhaps effective teaching should be broadly interpreted and diverse opinions and evaluations encouraged. According to McKeachie, “There is no 1 best way of teaching.”²⁸ The authors of the present investigation recently completed a pilot study (in process, 2011), which suggests there are profession-centric perceptions about teaching effectiveness in the areas of nursing and pharmacy education. Exemplary educators in nursing and pharmacy were surveyed regarding their perceptions of effective teaching techniques and traits, ineffective teaching, and active-learning techniques. The nursing educators’ perceptions were in general agreement while those of pharmacy educators varied widely, frequently stating exceptions and variations. In this era of interprofessional and interdisciplinary training, the concept of profession-centric perceptions of teaching effectiveness is a topic that warrants greater attention.

Mentoring systems have been explored as a method of developing teaching skills. In this model, senior faculty members who are effective teachers can serve as mentors to others, sharing their experience, skill, and passion. A mentoring process can provide rewards and opportunities for discussing innovative ideas for education.²⁹ Popovich and colleagues at the University of Illinois examined the “networking model” of faculty mentoring focused on improving the teaching efficacy of the mentor-mentee relationship.³⁰ They concluded that “successful mentoring, kept simple and informal, requires only a time and place where a discussion about teaching can occur.” Perhaps ineffective teaching would be less prevalent if faculty members were more frequently encouraged to openly discuss teaching-method successes and failures.

CONCLUSION

A 1-day teaching methods course requiring minimal time and expense is a valuable development resource that can improve teaching effectiveness through enhanced communication and teaching techniques. Because effective teaching is positively correlated with students’ learning, training seminars can be integral to comprehensive quality improvement. The ability to raise awareness of the value of teaching effectiveness also may support the scholarship of teaching. The results of this study show that the Teaching Methods Course was effective in a single university with 12 faculty participants. To determine how this method compares with other training programs, further research is needed to validate the results of this study with an expanded group of participants.