Collaborative Learning Teams to Longitudinally Teach and Assess Teamwork Behaviors and Attitudes

Abstract

Objective. To create and assess the effectiveness of a model of continuous development of teamwork skills (CDTS), which used a longitudinal peer feedback process across multiple courses that incorporated collaborative team learning.

Methods. Pharmacy students participated in collaborative learning teams across the first three years of the doctor of pharmacy (PharmD) curriculum, with team membership changing annually. Self, peer, and team evaluations were completed using the Comprehensive Assessment of Team Member Effectiveness (CATME) Smarter Teamwork system at four time points each year (three formative assessments and one summative assessment). Faculty members used peer and team evaluations to identify when additional coaching on teamwork behaviors, attitudes, and norms was needed.

Results. Self, peer, and team evaluations of 261 unique learning teams were conducted between fall 2015 and spring 2018. The majority of students and teams performed highly on teamwork behaviors and attitudes. Individual students and teams were identified for additional development on teamwork behaviors and attitudes as follows: for the 2015-2016 academic year, 5 (2%) individual students and 8 (20%) teams; for the 2016-2017 academic year, 15 (3%) individual students and 19 (22%) teams; and for the 2017-2018 academic year: 15 (2%) individual students and 24 (18%) teams.

Conclusion. The CDTS model, which incorporates formative and summative assessments, identified individual students and teams that met the teamwork standards established by the college as well as those students and teams that needed additional coaching to achieve the teamwork learning outcome.

INTRODUCTION

Teamwork is common in all types of work environments in society. Salas defined teamwork as “the ability of team members to work together, communicate effectively, predict and anticipate each other’s needs, and inspire confidence leading to a coordinated collective action.”¹ Teamwork and team performance can be developed through training that involves team members in learning about teamwork, practice, and continuous development of teamwork skills.²

Recognizing that high levels of teamwork are important in today’s workplace, curricular learning activities are incorporating collaborative learning approaches such as team-based learning and group projects.^3,4 Some pharmacy educators have assessed teamwork using survey instruments and team evaluation rubrics, but those assessments described approaches used for one or two isolated courses in a curriculum.^5,6 Fete and colleagues validated an instrument for peer evaluation in a team-based learning curriculum with longitudinal collaborative learning teams.⁷ The instrument focused on individual contributions to the team but did not assess team dynamics; neither did the process include a structured approach for students to discuss their teamwork behaviors and attitudes with the entire team. Knowing how to provide and receive feedback is considered a key aspect of being prepared to work in today’s contemporary health care teams, whether interprofessional or intraprofessional.⁸ Although researchers have established that team development involves four stages (forming, storming, norming, and performing), the length of time needed to progress through these stages depends on factors such as the team skills of members.⁹

Our premise was that teamwork skills needed to be developed and evaluated longitudinally across the curriculum rather than within single courses because graduates will encounter the need to be a team member for a sustained period. In addition, teamwork assessment plans should incorporate a process to assess individual contributions and team dynamics, as individuals on a team can perform well, but the team as a whole may struggle and vice versa.

The purpose of this study was to describe a model of continuous development of teamwork skills (CDTS) that incorporated longitudinal peer feedback using the Comprehensive Assessment of Team Member Effectiveness (CATME) Smarter Teamwork system^10-12 and to evaluate the use of CATME for assessing teamwork.

METHODS

The University of Florida College of Pharmacy has defined teamwork as an integral element of leadership, which is a component of the Center for the Advancement of Pharmacy Education 2013 domain 4 outcomes.¹³ The curriculum committee established the following key attributes of the CDTS model: team membership remains consistent across all intraprofessional courses during an academic year; students are randomly placed into teams, with consideration made during the second and third professional years to ensure students are working with an entirely new group of individuals; and formative and summative feedback is provided to both teams and individual students to improve individual and team performance. The CDTS model was used across multiple courses that incorporated collaborative learning, primarily team-based learning during the first three years of the curriculum, with students assigned to the same team for two semesters. A summary of the components of the CDTS model is included in Table 1.

Table 1.

Components of a Model for Continuous Development of Teamwork Skills Used to Longitudinally Teach and Assess Teamwork Behaviors and Attitudes

In fall 2015, the CDTS model was introduced in the first professional year as the new curriculum was implemented.¹⁴ At the start of each academic year, teams developed a team contract that had been modified from Ofstad and colleagues to incorporate elements of the TeamSTEPPS program and the Comprehensive Assessment of Team Member Effectiveness (CATME) Smarter Teamwork.^12,15,16 The team contract is available upon request from the authors. The contract was a written agreement among team members that contained four sections: reflection on past experiences working in teams to identify effective and ineffective teamwork behaviors and attitudes; identification of team norms by selecting behaviors they want to adopt; identification of a process to prevent and resolve conflict; agreement to the terms of the contract by signing the document. The team contract was stored in the electronic learning management system to allow access for all team members for the duration of team membership.

Students participated in three formative and one summative assessments each academic year as detailed in Table 1. The formative assessments were placed at the conclusion of a block of courses that corresponded with middle of fall semester, end of fall semester, and middle of spring semester, with the summative assessment completed at the end of the spring semester.

The formative and summative assessments were implemented using the CATME instrument.^10-12 The instrument assesses individual performance of teamwork in five dimensions: contributing to the teams work, interacting with teammates, keeping the team on track, expecting quality, and having related knowledge, skills, and abilities. Each dimension was rated on a scale of 1 to 5, with 5 being the best rating. It also provided flags for what were described as “exceptional conditions.”¹⁷ For example, a student was considered “under confident” if the team rating for the student was at least 3 and the student’s self-rating was one or more points lower than the team rating for them. The instrument also calculated an adjustment factor (the average of each individual student divided by the average of all members of the same team).¹⁷ Adjustment factor scores ranged from 0 to 1.05, with scores greater than 1 indicating a significant contribution to the team by that individual. Though the adjustment factor could be easily used to compare student teamwork across a cohort, limited reports existed in the literature that established a minimum standard for the adjustment factor to determine whether an individual embodied appropriate teamwork behaviors.

To assist in determining a minimum standard for the adjustment factor during the 2015-2016 academic year (year 1), we collaborated with the Office of Interprofessional Education at the University of Florida Health Science Center as they had previously adopted CATME to assess teamwork.¹⁸ Upon informal review of their existing data, adjustment factors below 0.8 indicated that either the individual student needed additional guidance related to teamwork behaviors and attitudes or there was a significant amount of missing data. Similarly, Simmons used CATME in a large enrollment engineering course and also established a minimum adjustment score of 0.8.¹⁹ Based on these observations, for the first year of implementation at the University of Florida College of Pharmacy, an adjustment factor of 0.8 was established as the minimum standard.

At the end of year 1, the minimum standard established for the adjustment factor was reassessed. We convened a panel of five faculty members who represented various areas in the curriculum where teamwork was vital (eg, director of experiential education, director of skills laboratory, director of personal and professional development curriculum, chair of the curriculum committee, and associate dean for curriculum and assessment). This panel reviewed results of four administrations of CATME and compared them to observed behaviors of students known to have issues with peers, faculty members, and/or staff members. The panel established 0.85 as the new minimum standard for the adjustment factor for the 2016-2017 and 2017-2018 academic years (study years 2 and 3, respectively). This value was used to identify students needing a one-on-one meeting with a faculty member to pinpoint areas for improvement related to teamwork behaviors and attitudes. The director of personal and professional development curriculum was responsible for continuous monitoring and quality improvements of the CDTS model throughout the study.

The CATME instrument also allows for additional domains to be added to assess the team, including, but not limited to, team satisfaction and psychological safety.^20,21 For year 1, we used the team satisfaction domain. Team satisfaction was assessed using a three-item questionnaire with responses based on a five-item Likert scale, with a score of 5 being the best rating. Faculty members used this domain in addition to comments provided by students to identify teams that were struggling to reach peak performance. Because others using CATME have reported significant variability in the ratings provided by each team, we determined that team scores that were two standard deviations below the mean would prompt a meeting facilitated by the faculty member to help the team identify areas for improvement.^18,22 In years 2 and 3, the psychological safety domain was added to the peer evaluations for each assessment period. Psychological safety was assessed using a 7-item Likert scale, with scores ranging from 1 to 7 with 7 being the highest rating. The scale was reversed for three of the items. In years 2 and 3, the faculty members triangulated the psychological safety and team satisfaction data along with written comments to identify teams that needed additional coaching to improve the quality of team interactions.

To encourage students to revisit their peer evaluation results, in years 2 and 3, the CDTS model incorporated three team debriefings that were embedded in a required course. These debriefings were conducted at the conclusion of each formative peer evaluation (Table 1).

The subjects who participated in this evaluation were all first professional year (P1) pharmacy students in 2015-2016 (year 1); all P1 and second professional year (P2) students in 2016-2017 (year 2); and all P1, P2, and third professional year (P3) students in 2017-2018 (year 3). Both formative and summative assessment data were summarized using descriptive statistics. Student performance on the summative assessments were calculated for each dimension in CATME using self- and peer evaluations. Calculations used to determine the adjustment factor did not incorporate student self-evaluation ratings. Team satisfaction and psychological safety scores were calculated by first determining the mean score for each team; those scores were then used to calculate the mean for the cohort. The mean team scores for team satisfaction and psychological safety were used to identify teams that were two standard deviations below the cohort mean. The University of Florida institutional review board designated this study to be exempt.

RESULTS

During year 1, there were 41 student teams (238 students) in the P1 class. In year 2, there were 45 student teams (285 students) in the P1 class and 41 student teams (236 students) in the P2 class. In year 3, there were 47 teams (273 students) in the P1 class, 46 teams (268 students) in the P2 class, and 41 teams (230 students) in the P3 class. Across the three years, 261 unique collaborative learning teams were created, with nine formative assessments and three summative assessments conducted. Completion of the self and peer evaluations were a mandatory course requirement for each assessment time period, resulting in a 100% response rate for all evaluations reported.

The summative assessment results for each cohort related to individual student evaluation of contributing effectively to teams for each dimension in CATME are provided in Table 2. Self and peer evaluation ratings were similar. There were no observations of “over confident” ratings on the summative assessments. However, there were observations of “under confident” ratings (year 1, P1=3%; year 2, P1=3% and P2=5%; year 3, P1=1%, P2=3%, and P3=3%).

Table 2.

Pharmacy Student Performance as Measured by the Comprehensive Assessment of Team Member Effectiveness Dimensions on Summative Assessments

Individual student performance on teamwork behaviors and attitudes based on the CATME adjustment factor (minimum standard score ≥0.85) are provided in Table 3. The mean adjustment factor was similar across all assessment time points. Table 3 also lists the number of students identified at each assessment period who did not meet the minimum standard score. In year 1, five P1 students (2%) were identified as requiring additional coaching related to teamwork behaviors and attitudes. No students had an adjustment score below the minimum standard during more than one assessment period. At the conclusion of year 1, one student (0.4%) was found to have significant teamwork deficiencies on the summative assessment. In year 2, 11 unique P1 students (4%), and four unique P2 students (1.7%) were identified as requiring additional coaching related to teamwork behaviors and attitudes. There were five students who had an adjustment score below the minimum standard on more than one assessment period. At the conclusion of year 2, one P1 student (0.4%) and three P2 students (1.3%) had significant teamwork deficiencies on the summative assessment. In year 3, 11 unique P1 students (4%), one P2 student (0.4%), and three unique P3 (1.3%) students were identified as requiring additional coaching related to teamwork behaviors and attitudes. There were five students who had an adjustment score below the minimum standard on more than one assessment period. At the conclusion of year 3, one P1 student (0.4%), one P2 student (0.4%), and two P3 students (0.9%) were found to have significant teamwork deficiencies on the summative assessment. There was one student who had an adjustment score below the minimum standard on the summative assessment for two consecutive years.

Table 3.

Individual Pharmacy Student Performance on Teamwork Behaviors and Attitudes Using the Adjustment Factor from the Comprehensive Assessment of Team Member Effectiveness Tool on Formative and Summative Assessments

Team performance based on team satisfaction and psychological safety ratings is described in Table 4. Overall, teams rated themselves highly. The number of teams that were identified as requiring a facilitated team meeting with a faculty member to discuss teamwork behaviors, attitudes, and norms was consistent across all years of the program. During year 1, eight unique teams (20%) were identified as requiring a facilitated team meeting with a faculty member to discuss teamwork behaviors, attitudes, and norms. There were four teams with mean scores that were two or more standard deviations below the cohort mean for more than one assessment period. Five teams were able to resolve conflicts resulting in improved ratings on subsequent assessments. There was a decline in psychological safety mean score in the year 2 P1 teams between the first and second formative assessments. However, these ratings increased during the third formative assessment. During year 2, nine unique teams in the P1 class (20%) and ten unique teams in the P2 class (24%) were identified as requiring a facilitated team meeting with a faculty member to discuss teamwork behaviors, attitudes, and norms. There were seven teams with mean scores that were two or more standard deviations below the cohort mean on more than one assessment period. Six teams were able to resolve conflicts resulting in improved ratings on subsequent assessments. Five teams were first identified as having team dynamic difficulties on the summative assessment. During year 3, eight unique teams in the P1 class (17%), seven unique teams in the P2 class (15%), and nine unique teams in the P3 class (22%) were identified as requiring a facilitated team meeting with a faculty member to discuss teamwork behaviors, attitudes, and norms. There were nine teams with mean scores that were two or more standard deviations below the cohort mean on more than one assessment period. Sixteen teams were able to resolve conflicts resulting in improved ratings on subsequent assessments. Five teams were first identified as having team dynamic difficulties on the summative assessment.

Table 4.

Pharmacy Student Collaborative Learning Team Performance on Teamwork Behaviors and Attitudes as Measured by Team Satisfaction and Psychological Safety on Formative and Summative Assessments

DISCUSSION

For the University of Florida PharmD program, the leadership-teamwork education outcome includes preparing students to contribute effectively as team members. The team experience, CATME instrument, team debriefings, and feedback coaching that encompasses the CDTS model provide a system in which students can successfully demonstrate teamwork. The collaborative learning sessions in every course during the first three curricular years provide the “laboratory” where students put their teamwork behaviors and attitudes into action. The assessment process within the CDTS model provides ongoing, formative, feedback during the first three years of the curriculum to individual students, teams of students, and faculty members. Creating a program culture that embraces the value of teamwork is the first step to educating students about appropriate teamwork behaviors and attitudes.

While others have attempted to assess individual students or teams as a whole in individual courses,^5,6 we have described a process that incorporates evaluation of both individuals and teams across a program and can be adopted by other educational programs that use collaborative learning teams across multiple courses. The majority of faculty effort necessary to sustain a CDTS model involves coaching individual students and teams that are identified as having teamwork deficiencies. As noted in Tables 3 and 4, a small number of students require additional coaching from faculty members to assist in improving teamwork behaviors and attitudes, with the majority of these being P1 students. We expected that there would be a higher number of P1 students who would require additional faculty coaching compared to students in other program years, as few students had previously experienced being part of a longitudinal collaborative learning team and teamwork skills require practice. Teams that required faculty coaching in teamwork behaviors, attitudes, and norms were dispersed evenly across all years of the curriculum, indicating that team conflicts can arise at any point and may vary based on team composition. Across the formative and summative assessments that we have completed, the data have shown that the majority of students and teams have met or exceeded the minimum standards we had established.

Students in this cohort accurately rated their own teamwork performance, as scores for the self-evaluation aligned with peer evaluations (Table 2). However, there were a few instances where students rated themselves significantly lower than they rated their peers, which were flagged in CATME as an indication the student is under confident. Students who received these flags were provided with additional information in the report generated by CATME to help prevent the student from undervaluing their contributions to the team in the future.

Assessment data from year 1 drove improvements in the CDTS model for years 2 and 3. During years 2 and 3, the revised minimum score for the adjustment factor appropriately identified more individual students who needed additional coaching related to teamwork behaviors and attitudes. The majority (75%) of the students identified who needed additional development of teamwork behaviors and attitudes were P1 students. This was expected as most of these students had not previously experienced longitudinal collaborative learning teams. Although incorporation of multiple assessments of global team quality (ie, team satisfaction, psychological safety, open-ended comments) did not change the number of teams identified with room for improvement, these results were beneficial to faculty members in guiding the discussion when meeting with each team. Use of three formative assessments followed by team debriefings provides opportunities for students to self-identify areas that need improvement, while also providing opportunity to seek feedback from team members. It also provides teams with dedicated time to discuss the behaviors and attitudes the team members expect of each other.

Our experience within a large enrollment PharmD program demonstrates that the CATME peer and team evaluation system is practical and feasible for both faculty members and students. Combination of both individual and team assessments into one tool improves efficiency in a process that requires timely feedback. However, the CATME tools need to be incorporated into a process, such as the CDTS model, that encourages self-reflection and provides a mechanism for additional development of teamwork behaviors and attitudes.

Limitations of this process include the expectation that students provide honest feedback to each other and the team with the goal of improving team performance. Team members are better positioned than faculty members to provide these evaluations as they are direct observers of each other’s behaviors and attitudes. However, some students fear potential retaliation from team members if they rate each other and the team honestly. This limitation was likely observed based on the number of teams that were first observed to have team dynamic difficulties on the summative assessment. We hypothesize that this was generated from students’ need to feel as though they are “high performing” at all times. Singer and Edmondson describe how learning and performance can conflict with each other.²³ This conflict has also been recognized as a factor that can compromise patient safety in institutions.⁸ It also underpins the concept of deliberate practice, which is important in student development of self-awareness.²⁴ Research related to team learning and performance and psychological safety emphasizes the importance of establishing an environment where students and teams are encouraged and comfortable with being honest in evaluating their teamwork. Individual students, teams, and faculty members must understand that, while the team learning environment may reveal performance shortcomings in the short term, addressing these shortcomings will lead to higher levels of performance in the long term. Inculcating the importance of deliberate practice may assist students in rating themselves, each other, and the team honestly.

CONCLUSION

The CDTS model, which incorporates formative and summative assessment, was successful in identifying individual students and teams that met the teamwork standards established by the college, but more importantly identified students and teams that needed additional coaching to achieve the teamwork learning outcome. This process can be adopted by other pharmacy schools that use collaborative learning techniques. It incorporates multiple measures that are easily triangulated by faculty members to assist with identifying students and teams that require additional coaching so they learn from their teamwork experiences and are better equipped to become collaborative team members.