A Faculty Toolkit for Formative Assessment in Pharmacy Education

Abstract

This paper aims to increase understanding and appreciation of formative assessment and its role in improving student outcomes and the instructional process, while educating faculty on formative techniques readily adaptable to various educational settings. Included are a definition of formative assessment and the distinction between formative and summative assessment. Various formative assessment strategies to evaluate student learning in classroom, laboratory, experiential, and interprofessional education settings are discussed. The role of reflective writing and portfolios, as well as the role of technology in formative assessment, are described. The paper also offers advice for formative assessment of faculty teaching. In conclusion, the authors emphasize the importance of creating a culture of assessment that embraces the concept of 360-degree assessment in both the development of a student’s ability to demonstrate achievement of educational outcomes and a faculty member’s ability to become an effective educator.

INTRODUCTION

The 2013 revision of the Center for Advancement of Pharmacy Education (CAPE) outcomes once again reminds the academy that the goal of its professional programs is to produce pharmacy graduates capable of achieving specific educational outcomes.¹ Current pharmacy education can be described as competency-based, focusing on the achievement of learning outcomes, rather than just completion of requirements. This approach to education addresses what the learners are expected to be able to do at the completion of their education, rather than what they are expected to learn during their education. Whether utilizing the CAPE outcomes or school-specific curricular outcomes, educational goals, in terms of measurable abilities pharmacy graduates should possess, represent the knowledge, attitudes, and behaviors required to successfully perform as a pharmacist. Because competency-based education requires acknowledging that teaching and learning are not synonymous, faculty members are tasked with ensuring an optimal learning environment and measuring student progress toward achieving learning outcomes. To accomplish these tasks, faculty members must utilize a variety of assessment measures in the classroom, including formative and summative evaluations.

In the late 1960s, Michael Scriven first coined the terms “formative” and “summative” in the context of program evaluation.² In 1968, Benjamin Bloom expanded on this to include formative assessment as a component of the teaching-learning process.³ For both Scriven and Bloom, an assessment was only formative if it was used to alter subsequent educational decisions. According to Bloom, the purpose of formative assessment was to provide feedback and allow for correction at any stage in the learning process.⁴ Subsequently, Black and William suggested expansion of the definition of formative assessments to include evidence that student achievement was used by teachers and learners to make decisions pertaining to subsequent steps in instruction that were likely to be better than the decisions they would have taken in the absence of such evidence.⁵ In her 2005 AJPE series of articles on assessment, Anderson provided this definition: “Formative: an assessment which is used for improvement (individual or program) rather than for making final decisions or accountability. The role of formative assessment is to provide information which can be used to make immediate modifications in teaching and learning and in the program.”⁶

The primary differences between formative and summative assessments are when they occur in the teaching-learning process and what is done with the information acquired from each. The summative assessment happens at the end of the teaching-learning process, while formative assessment occurs during that process. Information obtained from formative assessment can be used to immediately modify the instructional experience based on how well students are progressing in their achievement of intended outcomes. Because summative assessments occur at the end of the teaching-learning process, information obtained that might improve the process cannot be applied until the next offering of the course, leaving no opportunity for students currently enrolled to benefit from such changes.

To better clarify the definitions of and stress the roles of formative and summative assessments, many educators opt to use the term “assessment for learning” to describe formative assessment and the term “assessment of learning” to describe summative assessment. With this approach, the assessment of learning (summative) is designed to confirm what students have learned and can do at the end of instruction—particularly if they can demonstrate proficiency related to intended curricular outcomes. Summative assessments are typically “high-stakes,” meaning they determine student progression to the next phase of the curriculum or graduation.

In contrast, formative assessments are “low-stakes.” Assessment for learning (formative) activities are typically instructionally embedded in a class activity and are designed to guide instructional decisions. These activities aim to gather information about what students know and can do, what preconceptions and confusions exist, and what educational gaps exist. Because these activities occur while material is still being taught, they are designed to yield diagnostic information to allow students to self-assess their own achievement, to provide faculty with an understanding of student progress toward achievement of outcomes, and to guide instructional decisions. Faculty can utilize the information to identify the learning needs of students and adapt educational strategies to help students move forward in their learning. Key components of assessment for learning are summarized in Table 1.

Table 1.

Key Components for Assessment for Learning (Formative)

The AACP Assessment Special Interest Group leaders have discussed the importance of further promoting the culture of assessment and increasing the proficiency of various assessment techniques among the members of the academy. Formative assessment was identified as a priority because of its powerful impact on learning outcomes and faculty member development. For this paper, rather than targeting individuals specifically involved in assessment at the programmatic or institutional level, the target audience is all faculty members involved in didactic and experiential education. This paper aims to increase understanding of and appreciation for formative assessment and its role in improving student outcomes and the instructional process, while educating faculty on formative techniques readily adaptable to various educational settings. All faculty members, whether responsible for classroom or experiential teaching, can utilize formative assessment activities described in this paper to ensure the achievement of educational outcomes, rather than only using summative assessments to determine the degree to which students have achieved outcomes at the end of the educational experience. The authors relied on their expertise and knowledge of relevant literature rather than conducting an extensive and comprehensive literature evaluation when preparing this paper.

FORMATIVE ASSESSMENT OF STUDENT LEARNING

Classroom Strategies

A variety of classroom assessment techniques are available that can provide formative feedback. Such assessment techniques can range from a simple, nonlabor intensive tool to a complex process requiring considerable preparation time. The following examples require minimal preparation and analysis yet can provide valuable feedback to students and instructors alike (see Table 2). For a more comprehensive source of classroom formative assessment methods refer to Angelo and Cross.⁷

Table 2.

Summary of Formative Assessment Strategies that Can be Used in the Classroom

Prior Knowledge Assessment: Knowing what your students bring with them to a course or class period is valuable in determining a starting point for a lesson. ⁷ In order to collect information regarding the level of preparedness of students at the beginning of a class, a prior knowledge assessment can be administered. The assessment typically takes the form of a few open-ended questions, short answers, or multiple-choice questions. This method requires preparation in advance of the class and time to review the scores or responses and, possibly, time in class to administer the assessment. Because students realize there are no stakes involved with completing the assessment, they may not feel the need to provide an accurate or complete answer.

Minute Paper/Muddiest Point: Both the minute paper and muddiest point methods provide useful feedback and encourage students to listen and reflect on what they have learned in class.⁷ For the minute paper, it is useful, but not necessary, to focus on a specific topic or concept. For these assessments, time is allotted, usually at the end of class, to allow students to reflect on and write down the most important thing they learned and to acknowledge what questions they have as a result of the class. Along with identifying the most important point(s) from the lecture, students can also explain why they felt it was the most important point. This calls for more thought on the part of the student. With the muddiest point, the instructor simply asks the students to share what they thought was the lecture’s “muddiest” point, that is, what was unclear or confusing.

These methods provide immediate feedback and a manageable amount of information that can be quickly reviewed. Overusing these techniques may cause students to lose interest or not view the exercises as important. Muddiest point feedback and any questions that remain can help instructors identify areas of difficulty for the students and address them in a timely manner, either in class or through the use of technology, such as discussion boards available in learning management systems.

Audience response systems: “clickers:” The use of “clickers” has become common in the classroom.⁸ Clickers are an example of audience response systems, which are used for a variety of purposes such as quizzes, voting, and active learning. Multiple tools and technologies are available that allow the instructor to project or pose questions to the class and gather answers from students. Each student or a group of students use a clicker that connects to a receiver or personal electronic devices (eg, smartphones, tablets, or computers) that transmit answers via the Internet. Some instructors then opt to discuss the results while others have the students discuss the results among themselves and repeat the selection process. Clickers engage all students in the classroom in traditional question-and-answer active-learning activities while providing assessment data. Although this technique is very useful as a formative assessment strategy, it can also be used in a summative way.⁹

Audience response systems positively affect student engagement, active learning, and learning outcomes, and the use of such systems is well received by students.^10-14 This strategy, however, requires moderate preclass preparation on the part of the instructor to prepare the questions and response options in the system and then requires class time to administer, answer, and discuss the questions.

Case studies: Case studies are useful tools that can converge principles, concepts, and knowledge.^15,16 The process involves providing cases representing patients with one or several diseases, symptoms, or conditions. The cases can include as much information (lab values, physical assessment observations, etc.) as necessary for the level and background of the students. Students, working independently or in groups, are asked case-related questions regarding a diagnosis, drugs of choice, monitoring parameters, counseling points, and long-term consequences. After a period of time to prepare responses, students respond to the questions while the facilitator engages the students in discussion. Students participating in this method of assessment learn to develop critical-thinking, problem-solving, and communication skills. The instructor can use this exercise to assess student performance, difficulty of material, and success of instruction. A drawback is the considerable amount of preclass preparation and in-class administration time. However, time outside class, such as a skills laboratory, can be also used to complement and supplement material presented in lecture and still allow for collection of formative assessment data.

Case studies can be particularly helpful in teaching foundational sciences; however, an instructor’s lack of clinical expertise and/or students’ lack of background knowledge of the subject and drug therapy can make this a daunting undertaking. Fortunately, instructors who lack clinical expertise to develop their own case studies can seek the expertise of their colleagues or available resources such as published books of case studies or online case collections. For example, the National Center for Case Study Teaching in Science provides free access to an award-winning collection of more than 450 peer-reviewed case studies in sciences appropriate for high school, undergraduate, and graduate education. ¹⁷

Laboratory and Experiential Settings Strategies

In laboratory or experiential settings, instructors can observe and provide feedback on student performance in all 3 domains of learning: cognitive, affective, and psychomotor.^18,19 Students can be assessed on all levels of the Bloom’s Taxonomy learning pyramid from recalling information to making judgments. When designing an assessment and evaluation strategy in these settings, it is important to begin with identifying specific goals, objectives, and criteria for performance.¹⁸

A wide variety of formative assessment tools and methodologies can be used for in laboratory and experiential settings.^18-21 One of the most commonly used assessment strategies in the laboratory setting is objective structured clinical examinations (OSCEs). This strategy is designed to assess specific competencies in a planned and structured way with particular attention paid to the objectivity of the examination.²² Typically, OSCEs consist of multiple stations with standardized tasks to assess students’ knowledge, skills, and abilities.^22,23 OSCEs are often used as a form of summative and high-stakes assessment; however, they are also useful as a formative assessment strategy to provide feedback for improvement prior to a summative assessment. Developing OSCEs that are valid and reliable is challenging and resource intensive.²⁴

Another common method of assessment in laboratory and experiential settings is an instructor’s direct observation of students. These observations can range from informal and spontaneous as part of practice activities or structured and rubric-guided with predefined criteria for evaluation and feedback. Through direct observation and the feedback that follows, faculty members have the opportunity to help students acquire and improve skills needed for pharmacy practice and patient care. Most institutions and course instructors develop their own evaluation instruments and rubrics, which are based on specific objectives for given activities. The value of the rubric is in both the standardization of criteria for grading and in the definition of an appropriate performance for students.²⁵ The goal is to establish validity and reliability of these instruments and to share them with the academy.

Feedback quality is paramount in further developing our learners. The acronym SMART (Specific, Measurable/Meaningful, Actionable/Accurate, Respectful, Timely), often used for developing goals and objectives,²⁶ can be used as a quick reminder for elements of effective feedback (Table 3). The person directly observing the student should provide the feedback, which should be a 2-way conversation that allows the student to self-assess and reflect on a particular task.¹⁸ Feedback sessions should offer a balance of positive observations and recommendations for improvement and lead to a mutually agreed upon action plan, which includes ongoing observation and feedback. Formative feedback in general should avoid placing a judgment or a score on particular performance.

Table 3.

Elements of Effective Feedback – the SMART System

Formative Assessment in Interprofessional Education

Formative assessment is integral to the learning process of interprofessional education and the development of high-performing teams. Course developers must not only take into account the level and experience of the learner, but also prior assumptions about his or her role and other team members’ scope of practice. Traditionally, simulation has been used to build skill performance within a discipline (eg, surgical techniques). Many schools are now using unfolding cases (cases that evolve over time in a manner that is unpredictable to the student), simulation training, and the debrief session to build competency in areas essential to collaborative practice.²⁷

The debrief session is effective in allowing learners to reflect on and analyze what they have experienced as a team. During debrief sessions, students are given an opportunity to not only reflect upon the clinical decisions made, but to also discuss gaps in their knowledge, possible errors they made, and improvements they can make in their performance. An effective debrief session also allows students to express their feelings about stressful and rapidly changing situations (eg, cardiac arrest codes) and to generate solutions as a team. As cases become more difficult, questions addressing what went well, what the team could have done better, and what they would do differently next time become important. Providing comprehensive feedback to novice learners in a safe and supportive environment contributes to a better understanding of the experience and how the learning can be applied to new situations.

Engaging students in peer assessment as part of simulation exercises enhances constructive criticism, collegiality, and accountability within the team. Learning how to assess the performance of a team brings new insights to individual learners about ways they can improve their own performance on teams. The Performance Assessment of Communication and Teamwork (PACT) Tool Set, developed by the University of Washington Macy Assessment Team, includes readily accessible tools designed for use by either a student or faculty rater.²⁸ The Agency for Health Research and Quality has made available on its website TeamSTEPPS, a tool that provides guidelines for conducting a team debrief in a progressive manner, from exploring the roles and responsibilities of the team, using specific skills, situational awareness, and transitions of care, to exploring strategies for raising concerns about clinical decisions and how to ensure appropriate action is taken.²⁹

Reflective Writing and Portfolios

Given that formative assessment occurs during the learning process and is used for improvement, using reflective exercises in learning is consistent with these principles. Reflective learning requires the ability to self-assess, be aware of one’s own learning (metacognition), and develop lifelong learning skills.³⁰ Learning from education and practical experiences, articulating what has been gained, and considering why it is important are essential to the development of a practitioner with reflective capacity. Schon identified reflective practitioners as those who use reflection to purposefully think about their experiences with the goal of learning from them and using that information for future problem-solving in the clinical setting.³¹ Faculty members cannot possibly teach students all the knowledge and skills they will ever need in their future practice. Instead, the goal is to foster students’ ability to appreciate the uncertainty in knowledge and be able to clinically reason when encountering a problem with no clear solution.

Writing is a tool that can help cultivate thoughts, feelings, and knowledge on a subject.³² Reflective writing promotes self-directed learning and various models may be used. At its most simple, reflective writing may be a description and analysis of a learning experience within a course or clinical experience. It may include a review of what has been learned to a certain point or an analysis of a critical incident. Often, a model of “what/so what/now what” is used to guide the learner’s reflection. Reflective writing can be incorporated throughout the curriculum (didactic, experiential, and service learning) to encourage development of a reflective practitioner.

Reflective portfolios may also be considered as a type of formative assessment as they are used to monitor growth in students’ personal and professional development.³³ As opposed to a showcase portfolio, a reflective portfolio includes artifacts that a student selects over time to show how proficiency has changed, whereas the showcase portfolio is a collection of the student’s best work. For example, a pharmacy student may include a patient counseling video from his or her first professional year and write a reflection on what was learned from the experience and how he or she could improve upon it. For the next patient counseling encounter, the student may review the video, reflection, and instructor feedback. Reflective portfolios turn random assignments with no connection or reflection into a visibly cohesive learning tool for the student. In addition to emphasizing more student accountability for their own learning, portfolios facilitate a deeper learning by involving students in ongoing, reflective analysis of projects and assignments. This type of portfolio allows students to track their growth and development and, with good scaffolding, demonstrates how they can integrate and apply their learning.

The key to an effective reflective portfolio is the combination of documentation and reflection, along with feedback from faculty members. Instructor-student interaction around portfolio development is important to clarify why the student chose specific artifacts and what the artifacts mean in the context of a course or program. The feedback provided by the instructor helps give the student direction for improvement, along with providing a connection between the instructor and student. Ideally, the development of a reflective portfolio with appropriate structure, support, and instructor feedback will also increase the student’s engagement in learning.

Technology’s Role in Formative Assessment

The major goal of formative assessment is to provide feedback in a timely fashion for improvement. The collection, storage, and dissemination of formative data will likely require the use of computerized technology. While many vendors are competing to provide products and services in this arena there are a few principles that should guide the use of assessment data: 1) only data that is useful in improving outcomes should be collected and stored; 2) no amount of statistical manipulation can alter the usefulness of bad data; 3) data must be quality-controlled and accurate; 4) faculty members and students must be trained on the proper interpretation and use of the data—even if the correct data is accurately collected, it will not lead to improvement in curricular outcomes if misused; 5) data must be readily available to those who need it when they need it.

With modern computing power and virtually unlimited data storage capacity, the challenge is no longer in collecting and storing data but in using data to create effective improvements. Faculty members should investigate available technologies adopted by their institutions and seek out training to determine whether these can be used in a meaningful way to facilitate formative assessment in the didactic, laboratory, or experiential settings.

Formative Assessment of Faculty Teaching

Concepts and examples of formative assessment for enhancing student learning and development are also applicable to improvements in teaching effectiveness. Moreover, items that improve student performance such as classroom assessment techniques, student teaching evaluations, portfolios, self-reflections, performance evaluations, and peer assessments create opportunities for faculty members to strengthen their teaching abilities.

During the last several decades, survey research has shown that student teaching evaluations were the primary source institutions of higher education used for evaluating teacher effectiveness.³⁴ Student evaluations normally occur at the end of a class and often serve as summative assessment measures for promotion and tenure decisions. Numerous published articles debate the pros and cons of using summative student teaching evaluations. As a single source to rate teaching, faculty members question the validity and reliability of student perceptions. Although students are able to evaluate certain aspects of teaching, some research points to areas of teaching performance that students are not qualified to assess in terms of the quality (ie, content expertise, teaching methods, knowledge, etc.).³⁵ Other research, however, indicates student ratings are “reliable, valid, relatively unbiased, and useful.”³⁶ The challenges are applying student ratings to determine what actions should occur to improve teaching and getting comprehensive and constructive feedback at the end of the course from students unmotivated to provide it as they are not likely to benefit from any future course improvements.

Elzubeir and Rizk found faculty members paid more attention to written comments than the mean scores when reviewing student teaching evaluations in a medical education environment.³⁷ Thus, formative feedback could be gathered from students midsemester, specifically asking for written comments rather than ratings. Mid-semester feedback could be facilitated by assessment personnel, campus staff dedicated to improvement of teaching and learning, faculty members, or student leaders. Such feedback would allow faculty members to make adjustments in instruction or the course content and immediately benefit the students providing it. Student ratings for decision-making purposes are not sufficient evidence to evaluate teaching effectiveness.³⁵

Several colleges and schools of pharmacy include faculty member peer reviews of classroom teaching.^38-40 used in conjunction with student teaching evaluations, peer reviews often serve as another summative measure for promotion and tenure decision-making. Turpen et al indicated in preliminary findings that even when institutions used both student teaching evaluations and peer reviews, they tended to rely on student teaching evaluations only for faculty performance evaluations.⁴¹ Feedback from peers, however, serves as a powerful tool for formative assessment to enhance the quality of teaching.⁴²

Current research suggests that using multiple sources of feedback is more effective in evaluating teaching and professionalism.^43,44 Berk refers to this process as the “360^o multisource feedback model to evaluate teaching and professionalism.”⁴³ Successful implementation of formative feedback requires an understanding of the teaching environment and agreed-upon best practices for the institution as well as student and faculty member development on how to provide effective constructive feedback.

Faculty members often base the assessment of their own teaching effectiveness on student test performance. Choosing formative assessments that provide feedback on how well students are learning can enhance the quality of teaching in the classroom. For example, in the classroom faculty members may notice students not paying attention or displaying a look of confusion on a particular topic. By implementing classroom assessment techniques, such as the minute paper or clickers, faculty members can gauge understanding, obtain immediate feedback, and allow for teaching modifications. Midsemester and end-of-semester student feedback may help enhance effective communication or organization of the course or lecture. Similar techniques can be used in experiential settings. Peer feedback can also be instrumental in further refining instructional strategies, organization of content, student engagement, and assessment methods. Research-based assessments and student performance on examinations, quizzes, or homework can demonstrate attainment of educational outcomes and thus verify teaching effectiveness. Reflective teaching portfolios could be another valuable method for assessing teaching strengths and weaknesses and determining focused development in these areas. Action research, a disciplined process of inquiry for the purpose of reflection and quality improvement, allows faculty to engage in rigorous examination of their teaching practices.^37,45 Documentation of evidence of successful teaching and student learning, as well as self-reflection should be included in such portfolios.

CONCLUSION

A variety of formative assessment techniques can be applied in didactic or experiential settings and can be used to perfect student knowledge, skills, and attitudes. Formative feedback aimed at instructors develops faculty members and improves the quality of teaching. We recommend that professional pharmacy programs create a culture that encourages “formative failure” in a safe environment as a key element of learning. Formative failure allows the student to make mistakes and learn before being officially graded on the assignment. Formative failure allows teachers to experiment in order to further refine their teaching abilities.

Further, we recommend that, in order for students to achieve the optimal learning outcomes, faculty members and preceptors must learn to effectively integrate a variety of formative assessment strategies into their teaching. Formative assessment of students and faculty members should employ the 360^o multisource feedback approach. Sources of feedback for students include competency benchmarks, instructors, peers, preceptors, advisors, employers, patients, and self-reflection. Teachers should reflect on their performance by triangulating feedback from students and peers and evidence of learning from formative and summative assessments.

The proper use of formative assessment of both student outcomes and teaching skills should liberate the instructor to teach important material, the students to learn from their mistakes, and the culture of the institution to refocus on the development of outcomes. Institutions may want to utilize technology to manage the process of compiling, evaluating, and reporting on student progress effectively to provide timely data for learners and teachers to quickly address learning gaps. A well-structured assessment program that includes a balance of formative and summative assessments will help programs document student achievement of educational outcomes and provide data necessary for the continuous improvement of the curriculum.