Self-Assessment in Pharmacy and Health Science Education and Professional Practice

Abstract

Self-assessment is an important skill necessary for continued development of a health care professional from student pharmacist throughout their professional career. This paper reviews the literature on student and practitioner self-assessment and whether this skill can be improved upon. Although self-assessment appears to be a skill that can be improved, both students and professionals continue to have difficulty with accurate self-assessment. Experts' external assessment of students should remain the primary method of testing skills and knowledge until self-assessment strategies improve. While self-assessment is important to lifelong learning, external assessment is also important for practitioners' continuing professional development.

INTRODUCTION

Self-assessment has been extensively discussed in health science education and professional practice and is often utilized in both settings. The Accreditation Council on Pharmacy Education (ACPE) states in guideline 11.1 that self-assessment should become a part of the pharmacy curriculum, starting at an early point in the student's career and continuing throughout the curriculum. However, its usefulness in advancing and changing overall outcomes of practice by practitioners has not been realized.

This may stem from educators' and practitioners' belief that self-assessment is easily adapted to professional education and practice. However, understanding self-assessment is more complex than it appears. For example, differing terminology may be used to describe a concept or phenomenon in self-assessment while in other cases a particular term may be used to describe several different concepts or phenomenon. For example, different terminology may be used interchangeably to describe the same concept or phenomenon and, in other cases, the same term is used to describe different concepts and phenomenon. Also, evaluation methods and tools, as well as proposed motivation for an individual's assessment, are often the same for students and practitioners. This vagueness in the tools and terminology may contribute to ineffectiveness in the implementation of self-assessment activities as a pharmacist transitions from the educational to the practice setting. Although self-assessment skills are believed to be teachable and will result in better and more competent students and future practitioners, this assumption has not been rigorously tested and validated.

In order to clarify some of the issues associated with self-assessment, this review will define the terms associated with self-assessment, provide a foundation for integrating self-assessment into professional programs, examine the challenges associated with self-assessment, and discuss how to enhance the self-assessment skills of students as future practitioners.

Self-Assessment vs. Self Reflection vs. Self Evaluation

The literature on self-assessment uses the terms self-assessment, self-reflection, and self-evaluation interchangeably. Andrade defines self-assessment as: “…a process of formative assessment during which students reflect on and evaluate the quality of their work and learning, judge the degree to which they reflect explicitly stated goals or criteria, identify strengths or weaknesses in their work and revise accordingly.”¹

According to this definition, self-assessment is a “process” designed to allow a person to collect information about his/her own performance and compare it with the goals and/or the criteria for his/her work. Self-evaluation involves an individual making summative judgments about his/her own performance. Self-reflection, on the other hand, takes a more global view of learning and is generally a reflection on achievements over a certain period of time or on a particular subject matter; this assessment usually is made without an established set of criteria.¹

Self-assessment has been described as an “unguided” and “self-generated” method of assessment to differentiate it from self-reflection. Directed self-assessment commonly involves obtaining information from peers, teachers, and other sources.^2,3 Eva and Regehr characterize directed self-assessment as, “…a process which one takes personal responsibility for looking outward, explicitly seeking feedback and information from external sources, and then using the feedback to direct performance improvements.” Typically, self-assessment refers to the degree of correspondence between the individual and those providing a summative evaluation of the individual's performance compared to self-evaluation which is a summative or outcome evaluation. The important element distinguishing self-assessment from the other terms is that directed self-assessment seeking behaviors are primarily a “process” for formative evaluation and lifelong self-improvement.²

Reflection, as typically defined in the educational literature, is not a universally accepted concept. However, its definition is evolving and is characterized as “…a conscious and deliberate reinvestment of mental energy aimed at exploring and elaborating one's understanding of the problem one has faced (or is facing) rather than aimed at simply trying to solve the problem.”² Using this evolving definition as a framework, reflections are more likely to focus on understanding “why.” The central assumption of this educational strategy is that self-reflection will result in finding a better solution the next time a similar situation is encountered. Unfortunately, without appropriate and structured feedback, self-reflection does not provide a means to confirm knowledge or competence and still suffers from the consequences of “not knowing what one does not know.”

Professional ethics also require that reflections be “morally open”⁴; however, students may not feel free to express their true feelings to faculty members. For example, the most valuable thing a student may have learned from an advanced pharmacy practice experience (APPE) in long-term care was that she does not like working with the elderly. Assessing students' “feelings” is not an adequate assessment of knowledge or competency. While one would hope we are simply beyond assessing students' feelings, anecdotal evidence indicates that we may not be.

Self-Monitoring vs. Self-Assessment

Another set of terms frequently confused are self-monitoring and self-assessment. Self-monitoring is characterized by an “…ability to attend, moment to moment, to one's own actions, curiosity to examine the effects of those actions, and willingness to use those observations to improve behavior and patterns of thinking in the future.”⁵

A cognitive neuroscience basis for differences in self-monitoring ability has been postulated and may explain how data are managed by the brain. These differences in the brain may explain the differences in an individual's ability to self-monitor, including his/her level of motivation, attentiveness, alerting, orienting, executive attention, curiosity, and meta-awareness.⁵

Criteria-Referenced Self-Assessment

Moving closer to what is deemed an adequate self-assessment strategy is “criteria-referenced, directed self-assessment.”^1,6,7 “Criteria-referenced assessment has 4 essential stages”: (1) clear learning targets and criteria, (2) modeling application of the criteria, (3) providing feedback to students on their application of the criteria, and (4) setting new learning goals and strategies for the student. These stages can be applied by anyone from beginning students to experienced practitioners (Figure 1). However, one of the biggest challenges in this approach is getting students to assess their performance based on standards established in the curriculum by the instructor rather than what they experienced during a limited internship or APPE.

Figure 1.

Student to professional self-assessment cycle.

Self-Assessment in Education and Professional Practice

Self-assessment complements other teaching elements to enhance students' knowledge, skills, attitude, and values, and forms the foundation for the development of an independent learner with the necessary life-long, self-directed learning skills. Self-assessment provides professionals with the means to balance their daily practice, establish and set their personal learning goals, and have confidence in their professional activities. Ideally, individuals aware of the areas where they lack professional expertise will realize they need assistance from colleagues to select an appropriate action plan when they encounter negative feedback or obstacles in their professional practice.⁸

One model for effective self-assessment requires students and practitioners to assess the following 4 elements to determine which are already present in their professional expertise and which need to be developed: (1) prerequisite competencies (things I am), (2) a process for practitioners to follow (things I can do), (3) skills and knowledge the practitioner will apply in the process (things I can apply) and (4) various tools that are available to be used with the specific applications (things I can use).⁹ The model also emphasizes the importance of a supportive environment and suggests that professionals incorporate their knowledge of the desired behavior and compare it to their current practice behaviors. A cycle for the professional self-assessment process is presented in Figure 1. This approach reveals the voids in a professional's practice, subsequent learning needs, and how to incorporate reflection into daily practice.⁹

SELF-ASSESSMENT STUDIES

Professionals and Self-assessment

Theoretically, competent professionals pursue lifelong learning to formulate appropriate learning goals to correct perceived and real deficiencies. Despite the accepted theoretical value of self-assessment, the accuracy of self-assessment is poor.^10,11 For example, nearly 65% of studies have found little, no, or an inverse relationship between self- and external assessments in a systematic review of the physician literature.¹⁰ Typically, the worst accuracy is found among those least skilled and most confident.

While the ability to self-assess is necessary to maintain professional competence, an individual's self-assessment skills seem to be related to his/her competence.^10,12 Self-assessments by individuals with the highest levels of competence tended to be over critical of themselves compared to external assessments of their competence. Conversely, individuals with the lowest competence tended to overestimate their abilities and did not effectively recalibrate when provided with external assessment. Individuals in the mid-level of competence were generally most accurate in their self-assessment and remained at the same level when provided with external assessment.¹² Similar trends in the accuracy of self-assessment were found among family practice residents.¹³

Professional Education and Self-Assessment

The development of self-assessment skills is thought to be an important element in enhancing professional education. Videotaped clinical encounters are an effective means of facilitating self-assessment of students' communication skills. Videotaping allowed them to review their own behavior and make specific comments on how they could improve by looking at specific examples.¹⁴ More remarkable was that 30% of these students gained valuable insights into their communication abilities and how they would affect the patient encounter; even without faculty guidance. In another case, reflection-in-action or real time self-assessment, by requiring students to provide a brief explanation and justification of why they rated themselves at a particular level of confidence, was found to positively influence the performance of fourth-year BScPhm students' critical-thinking skills.¹⁵ The ability of students and clinicians to gain insight into their own strengths and weaknesses in the provision of patient care is an important component in professional self-assessment.

METHODOLOGICAL ISSUES ASSOCIATED WITH SELF-ASSESSMENT

Correlational Analysis

An important reason for the poor correspondence between self- and peer or expert assessment may be the methods used to test the association. The most frequently used strategy in the self-assessment literature is correlational methods. With correlational studies, individuals' self-ratings are correlated with experts' ratings to obtain a single numeric value for the group. This single number is based on the groups' responses and interpreted as an individuals' ability to self-assess. If the correlation is high, the conclusion is that students performed self-assessments well; if the correlation is low, students self-assessed poorly. The fallacious assumptions and consequences of using this strategy are described in the organizational and educational literatures as the “ecological fallacy.”^16-18

Proportional Self-Assessment

A related strategy is to report the proportion of self-ratings that correspond with experts' ratings. Here, a self-assessment standard is set and the proportions of agreement and disagreement are noted. Finally, the absolute difference between the self-rating and the peer- or expert-rating is calculated by subtracting the students' self-rating from the peer or expert's rating. Smaller differences indicate a greater correspondence and larger differences indicate lesser correspondence between the 2 assessments. For each strategy, a central assumption is that both groups (ie, those doing the assessing and those being assessed) are homogeneous groups that agree on the level and skills needed to demonstrate competence, which may be an unwarranted assumption.

First, the homogeneity of the groups' definitions of the “gold standard” is suspect. At times, even “experts” have different opinions regarding performance on the gold standard. If experts do not agree, then how can we expect students to agree with the experts or each other? Assessment of competence is based upon the reliability of the raters; however, the reliability of such measures is suspect, even among experts.^11,19 This notion is especially worrisome in situations where peer evaluators are also naïve or inexperienced in the skills needed to demonstrate competency. In these instances, peer evaluators do not assess performance against the standard, but rather they assess their colleagues' performance against their own performance or on what they perceived their peers meant to do rather than what they actually accomplished. This phenomenon is also present in experts' ratings, but to a lesser degree. It is not surprising that the correspondence between the assessors' and students' ratings is variable given that expert raters have variability in their own performances. This situation produces significant measurement error and the correlation is attenuated (eg, biased downward) because the differences between an individual's rating and the highly variable external assessor group is not taken into consideration, especially in peer-rating situations. Thus, the underlying assumption that the external raters have a homogeneous assessment of competence may be untenable. Even so, this proposition does not completely suffice in studies that demonstrate the lack of correspondence between individual students' assessments and a single expert's assessment. In this circumstance there should be a lesser degree of variability because a single assessor's judgments are more likely to be internally consistent—at least theoretically.

Global Competency Assessment Measures

A different but related reason for lack of correspondence between self- and external assessments of competency is the way competency is measured. Generally, measures of competence used in pharmacy are global measures. Global assessment measures likely have similar strengths and weaknesses as other global measures, such as health-related quality of life. In the quality of life literature, multiple domains often underlie global measures and it is not always known which ones are the most important or predominate in the assessment. However, if clear criteria are not given to the raters (eg, when using global assessment measures), then individual expert's and peer's ratings of competency may be based on different domains, even within the same assessment episode. If so, the students are effectively measured against as many implicit criteria as there are raters. Thus, it should be no surprise that the correlations are low. Frequently, the global measure of assessment is too simplistic and defies standardized definitions or weighting. Moreover, global assessments may not be based on the skills necessary to accurately demonstrate competence. The net outcome of this conundrum is that the accuracy of global self-assessments is probably worse than self-assessments of factual knowledge which, ostensibly, are more criteria-based than global assessments.²⁰

Validity/Reliability

Fundamental to the valid and reliable measurement of any concept is the requirement that multiple raters view the performance of a single student with a high degree of agreement. In the pharmacy literature, validity and reliability assessment of the measures have not been carefully and thoroughly conducted. The core assumption is that the external assessors' judgments of student performance are reliable and valid. Moreover, it requires that a single assessor would come to the same conclusion when viewing the same performance by the same individual on different occasions—an assumption that has not been demonstrated. In most cases, studies on the reliability of self-assessment do not ensure that either of these tenets has been satisfied. It is important to validate this assumption, especially in the case of peer review. If peers are naïve to the conduct of the skill under scrutiny, then the quality of the assessment is suspect. By blindly accepting these assumptions, current approaches using peer review often succumb to the problems associated with the group-level correlations discussed earlier (ie, attenuation of the correlation).

One recommendation when using “expert raters” is to limit the assessment to short, structured activities that are relatively simple. We currently ask preceptors to “globally assess” student performance in advanced practice experiences. By their nature, assessments of IPPEs or APPEs are not short, structured, or simple if one considers the vast array of the intended educational outcomes.^21,22

Level and Alignment of Self-Assessment

Probably the most important methodological/pedagogical factor in establishing the success of an assessment program, whether self- or external assessment, is aligning the program with the level and purpose of the assessment. First, is the assessment for evaluating performance (eg, summative or self-evaluation) or for assessment of learning or gaining knowledge (eg, formative self-assessment, self-monitoring). One means of characterizing the differences in the 2 paradigms is to ask whether the assessment is designed to ascertain a practitioner's strengths and weaknesses (formative) via self- or external assessment, or their performance on specific skills based upon mutually accepted criteria.

Each type of self-assessment requires different methods or combinations of methods. For example, personal, unguided reflections on practice simply do not provide sufficient information to guide performance improvement.^2,10 Consequently, summative assessments should be based on a foundation of evaluations by external assessors. Over time, externally defined criteria for success become internalized, and individuals learn to compare and contrast their own performance using a personalized plan.^15,23 Consequently, individuals determine their own abilities in relationship to their own strengths and weaknesses rather than to their peers' abilities.^11,21 Consequently, developing actions using intra-individual versus inter-individual assessments enables pharmacists, student pharmacists, and faculty members to self-assess their own strengths and weaknesses and is basic to developing a personal continuing professional development (CPD) plan.

RECOMMENDATIONS

The literature is replete with recommendations regarding how to improve individual learner's self-assessment skills. Most of the recommendations move away from the “grade yourself” approach discussed earlier in conjunction with correlational analyses as a “generalized, reflective process of generating an unguided, personal, summative assessment of one's own level of ability or performance.”² Steps for improving self-assessment skills include the following:

Emphasize external feedback to inform self-assessment.

Encourage quality external feedback to inform more self-directed assessment. This will reduce professionals' reliance on their own, possibly uninformed, opinions on performance and provide benchmarks for informed self-assessment.³

Improve feedback quality.

Create tools to improve feedback quality, including simulations and Objective Structured Clinical Examination (OSCE).²⁴ The development of these tools will require additional training to effectively utilize the tools.

Recognize theoretical vs. achieved value of reflection.

Develop a reflective thinking process⁴ that brings awareness of knowledge, skills and abilities. This process should improve insights leading to enhancement of professional performance.

Respond to both external and internal motivation.

Adopt an active approach to professional development that capitalizes on professionals' internal motivation, as well as external factors (eg, continuing education requirements). Both internal and external motivations are needed to maintain competencies in base knowledge and skills.⁸

Maintain attentiveness and habits of the mind.

Attentiveness is maintained through continually paying attention to the current situation versus practicing or learning/memorizing on “autopilot” (eg, develop meta-cognition, self-monitoring skills). It is important to be attentive while also maintaining openness and curiosity.⁵ Along with these attributes, one should develop good habits of the mind. These include (1) training oneself to experience information as novel, even in familiar, “autopilot” situations; (2) recognizing that one's senses are sometimes fallible and deceptive, especially in familiar situations; and, (3) seeing a situation from multiple perspectives, even to the point of taking opposing perspectives at the same time.”⁸

TESTING MODELS TO MEET ACPE'S GOALS FOR SELF-ASSESSMENT

There are multiple reasons for the phenomenon behind poor self-assessments of one's own performance including the “above average” interpretations of performance,² over confidence,²⁶ doing assessment for the wrong reasons, and misaligned motivations. To further complicate matters, most people think of quality of self-assessment as a unipolar continuum; from good to bad or from under- to over-estimators. However, it is probably much more complex because people have different abilities and skills based upon the requirements and social context in most aspects of life.

Models are being developed in the educational and health professional literature to capitalize on these different aspects. For example, different characteristics of the assessor are associated with over-, under-, and accurate assessment. Over-estimators were found to be more confident and more goal-oriented and to make more use of feedback.²⁶ However, over-estimators made the greatest skill and knowledge improvements. Many of these psychosocial models contain various aspects shown in the proposed Figure 1, adapted from models presented by Asadoorian and Batty and McMillan and Hearn, and warrant rigorous testing.^6,9,31

Figure 1 highlights the fundamental relationships between the process of the student self-assessment cycle facilitated during health professions education and the essential role of the self-assessment cycle for successful professionals. The key linkages and areas where health professions education could focus their efforts are indicated in the striped arrows between the 2 cycles. The challenge to health care educators will be to provide these linkages in our professional programs. If our students are to be successful practitioners who embrace the concept of professional self-assessment, we as educators must through our educational programming instill the value and importance of this process (left hand stripped arrow). It must be the goal of our programs to strive toward educating our health professions students to see self-assessment skills as fundamental to their professional practice. Likewise, we must provide our health professions students with the skills and knowledge for the process of self-assessment. Our educational programs must provide students with the awareness to evaluate areas for improvement (left hand stripped area) as well as the willingness to accept feedback and identify the tools or experts to advance and improve their professional practice (right hand stripped arrow). These will require health professions educators to step back and think critically on the nature and type of pedagogical approaches for developing self-assessment knowledge, skills, and attitudes in our students.

Pharmacy educators have made a few small strides in this direction.^15,32-34 Self-assessment has laudable goals and is theoretically central to professionals' motivations and success for lifelong learning and continuing professional development. However, given the current state, relying on self-assessment too much is a mistake. External assessments by experienced mentors or peers are still necessary, even with continuing professional development. Even when agreement is high regarding learning domains and skills that one is knowledgeable about, the challenge of “knowing what you don't know” will always require the assistance of an external reviewer or assessor. More importantly, developing more effective, reliable, and valid self-assessment strategies based on existing or evolving models is crucial to patient safety and quality care.

CONCLUSION

If we are to advance and employ self-assessment skills and abilities in our professional and continuing professional development programs, we must develop and validate the appropriate models and methods, provide educators with theoretical background and practical applications, and embrace the culture in our educational programs where self-assessment is an essential element to successful professional practice. Furthermore, this will require a commitment to collaborate between the key stakeholders that regulate professional practice, specifically educational institutions, professional practice organizations, accreditation organizations and state and national licensing bodies in the health professions.