Use of Reflective Portfolios in Health Sciences Education

Abstract

Reflective portfolios are a collection of evidence that through critical reflection on its contents demonstrate achievement as well as personal and professional development. The purpose of this paper is to provide a review of the literature on the use of reflective portfolios and to develop a set of factors to guide schools and colleges of pharmacy as they implement reflective portfolios into their respective curricula as stated in Standards 2007.

INTRODUCTION

The 2007 Accreditation Council for Pharmacy Education (ACPE) Standards state in Guideline 15.1 that a school or college of pharmacy's system of evaluation should include demonstration and documentation of student learning and attainment of desired competencies in a variety of healthcare settings in a student reflective portfolio format.¹ ACPE Guideline 15.4 states that student reflective portfolios should be used, “…to document students’ progressive achievement of the competencies throughout the curriculum and the practice experiences. The portfolios should be standardized and include some aspect of student self-assessment, as well as, faculty and preceptor assessment of the educational outcomes.”¹ The purpose of this paper is to provide a review of the literature on the use of reflective portfolios and to develop a set of factors for schools and colleges of pharmacy to consider as they implement reflective portfolios into their respective curricula.

Reflective portfolios have been defined as the collection of evidence that attests to achievement as well as personal and professional development through a critical analysis and reflection of its contents.² Engaging in introspection is what separates the reflective portfolio from the “showcasing of best work” approach seen in other fields such as architecture. This definition has been used extensively in the nursing and medical literature and will serve as the definition of reflective portfolios used in this paper. Another definition of portfolios often cited in the general higher education literature is that a portfolio is, “…a purposeful collection of student work that exhibits the student's efforts, progress, and achievement in one or more areas. The collection must include student participation in selecting contents, the criteria for selection, the criteria for judging merit and evidence of self-reflection.”³ The constraints of accreditation requirements and the necessary use of standards- or competency-based reflective portfolios often preclude health professions students from participating in setting criteria for judging and exclusive content selection.^4-6 Reflective portfolios have been used for a variety of purposes such as authentic assessment, bridging the “theory-practice divide,” curricular improvement, formative and summative assessment, and taking a deep approach versus a surface approach to learning. Reflective portfolios have been suggested as a more constructivist approach to the assessment of student learning in that learning is constructed by students in selecting and assembling evidence of meeting course and learning objectives and the associated justification of inclusion based on the said objectives.^7-8 Chambers and Glassman discussed portfolio evaluation in general as a form of authentic assessment where faculty members and students have distinct roles.⁹ Faculty members are responsible for identifying and defining competencies and standards, defining what constitutes acceptable evidence of accomplishment of said competencies, and determining timelines and guidelines for evaluation.⁹ Students on the other hand are responsible for collecting, presenting, and defending evidence of the accomplishment of the desired competencies and ultimately accepting responsibility for being deemed competent.

The current uses of portfolios in pharmacy education appears to fall into 3 general categories: (1) in the experiential component of the curriculum either in its entirety or in part associated with specific rotations,^10-16 (2) for specific courses in the didactic portion of the curriculum,^17-19 or (3) as an integrated component throughout the entire curriculum.^20-22 In a descriptive study examining programmatic curricular outcomes assessment at schools and colleges of pharmacy in the United States and Puerto Rico, Kirschenbaum, Brown, and Kalis reported that 27 of 68 institutions indicated that they used student portfolios as 1 component of the curricular assessment process.²³ They noted that the literature on the use of student portfolios was scarce and that it was not possible to discern whether portfolios were being used for programmatic curricular outcomes assessment or if these portfolios were standards or competency based, or reflective in nature. The majority of the available literature on the use of student portfolios in pharmacy education exists as meeting poster abstracts, so it is not possible to get a complete picture of their use including whether they involved reflection.

FACTORS TO CONSIDER

Based on an extensive review of the literature, several themes emerged as critical components or considerations in the use of reflective portfolios. The literature review was conducted using MEDLINE, International Pharmaceutical Abstracts, the Journal website, and the ERIC database in the fall of 2005 and spring of 2006. Search terms included: “portfolio(s),” “higher education,” “reflection,” and “reflective.” The reference lists of articles were further scanned for additional studies that were not picked up on the various searches. Relevant articles were then analyzed for recurring themes in the literature to develop a set of factors to consider in the implementation and use of reflective portfolios. This section reflects a synopsis of the literature on reflective portfolios to develop a set of factors to consider.

Faculty and Student Buy-In

Given the potential for negative student reactions, it is important to get student buy-in into the use of reflective portfolios. The use of reflective portfolios for summative assessment may contribute to their dislike and devaluation as well as decrease their effectiveness as a learning experience by stifling honesty and reflection.²⁴ The lack of student buy-in can hamper both learning and assessment efforts. A key barrier to both student and faculty buy-in is the view of the reflective portfolio as an “add-on” to an already heavy or difficult workload.^4,25-27 Snadden and Thomas examined the use of a reflective portfolio learning in medical education. They suggested that medical students may have found the independent learning required to construct a reflective portfolio threatening given the undergraduate focus on knowledge acquisition.²⁸ Different learning styles and the inability to see the relevance of reflection to the learning process were also suggested as potential causes of negativity among students.²⁸ Lack of faculty understanding of the standards or competencies upon which reflective portfolios are based can negatively influence student buy-in, potentially creating frustration and decreased guidance.^24,26-27

In examining student attitudes towards reflection in a community health nursing program, Smith and Jack found that student learning style affected how useful students found reflective practices, which was similar to the findings by Snadden and Thomas.^28,29 This small exploratory study appeared to support the idea that student buy-in can affect attitudes and the effectiveness of reflective portfolio implementation. Citing the concern that reflective portfolios used for both assessment and learning would affect not only the content but also student experiences and perceptions of the process itself, Mitchell conducted an exploratory study with 23 postregistration student midwives and 8 instructors.⁶ Mitchell also sought to address the issue of the lack of actual examination of student perceptions concerning the use of reflective portfolios compared to the theoretical advantages often cited in the literature. Overall students were found to be “marginally negative” and that the reflective portfolio failed to stimulate them to learn and required a great deal of time that took away from other learning opportunities. Faculty members who were interviewed found the reflective portfolio to be a valuable diagnostic tool; however, their overall positive views of the advantages of the use of reflective portfolios were not mirrored by the students.⁶

Student-Faculty Link

The student-faculty link is an important component in the reflective portfolio process.^{2,4,25,28,30-34} Crandall suggested that in addition to providing valuable formative assessment and discussion, meeting with students on a regular basis helped to prevent the reflective portfolios from becoming a “last ditch effort” at the conclusion of the semester.²⁵ The reflective portfolio can also serve as a platform for discussion between students and faculty members to help to bridge the theory-practice gap.^25,30 Challis suggested that this student-faculty link could take many forms ranging from regularly scheduled formal meetings with a mentor to telephone or e-mail communication as needed.⁴ Challis also proposed small working groups of students serving as peer mentors; however, fellow students might not be able to provide adequate guidance on the portfolio process or an adequate understanding of the competencies themselves.⁴ The student-faculty link could potentially reduce negativity and confusion associated with the use of reflective portfolios, as well as provide valuable formative assessment information to both students and faculty alike.

Reflection on Each Piece of Evidence

Students should reflect upon and explain how the evidence or artifacts in a reflective portfolio relate to each competency along with the rationale for their inclusion in the portfolio.^{2,4,7,9,24-25,27,35-38} The reflective explanation should provide both the rationale for inclusion and a description of the role the evidence played in the learning progression to validate accomplishment of the competency and should be appropriately referenced and logically organized.^4,37 The inclusion of evidence should be both thoughtful and representative of the professional standards and not merely a collection of items bound together in the same folder or binder.^2,25,35,39 While the demonstration of learning through artifacts or evidence is a key feature of reflective portfolios, students have difficulty in assembling and integrating this portion of the reflective portfolio assessment.^{2,24,27,39-40} Jasper and Fulton suggest that not only students but also assessors are confused at times about what constitutes appropriate supporting evidence.³⁷ Schaffer et al found that students had difficulty in identifying appropriate experiences as well as in making connections between evidence and competencies.²⁴ Zou and Pitts et al found that student selection of artifacts were inappropriate and irrelevant to the corresponding professional standards.^27,39 Examination of the evidence provided and the linkages students make to the appropriate standards and competencies is essential to preventing the reflective portfolio from becoming merely a collection of items thrown together. While not examined in the literature, it is not clear whether providing guidance on the potential content of the reflective portfolio affects student reflection on the linkages and rationales provided for inclusion of evidence.

Dual Use of Reflective Portfolios for Learning and Assessment

The use of reflective portfolios for both learning and assessment can create conflict for students when used for both purposes and ultimately may reduce the reliability and validity of the assessment.^24,28,30,41 When reflective portfolios are used for self-reflection and students have true ownership of the contents as with formative assessment there tends to be less conflict with learning.^6,39,42-43 When reflective portfolios are used for summative assessment, the conflict appears to increase. ^39,42-43 The potential conflict between the use of the reflective portfolio for both learning and assessment can affect student buy-in, creating negativity towards the reflective portfolio as seen in the Mitchell study.⁶ The use of reflective portfolios can affect the sense of ownership of the contents, such as less willingness to admit difficulties or mistakes even when they served as learning experiences.^24,28,30,34 Schaffer et al suggested that there appears to be social desirability bias in reflective statements when they are used for assessment purposes.²⁴

Gannon and colleagues considered the dual use of reflective portfolios for learning and assessment in nursing education in the United Kingdom, with particular emphasis on honesty, best practices, and reliability and validity issues.⁴¹ They suggested that, “…there may be an inverse relationship between the use of portfolios in the assessment process and the honesty of the records which the keeper of the portfolio may maintain.” Related to the issue of honesty is credibility as it applies to reliability and validity.⁴¹ Reflective portfolios have been assumed to be credible sources of evidence of competency since it originated as a form of authentic assessment.^42,44-45 However, Gannon et al argued that a form of assessment that is unreliable and thus lacks validity cannot be credible.⁴¹ They suggested the use of qualitative or mixed-methods of both quantitative and qualitative methods to examine the reliability, validity, and credibility of reflective portfolios used for both learning and assessment.

Reliability and Validity Considerations

The most often cited disadvantage or limitation of the use of reflective portfolios is the reliability, or lack there of, with which they can be assessed.^{32,37,39,41-44,46-48} While not sufficient, reliability is necessary for validity; thus, low reliability in assessment methods for reflective portfolios is a serious limitation or disadvantage to their use.⁴⁹ Issues central to the debate over the extent to which reflective portfolios can be assessed reliably include the purpose of the assessment (eg, formative versus summative assessment), holistic versus component grading or assessment, rater training, and the development of the grading rubric. The various issues related to reliability and validity are intertwined, with each consideration affecting the other. Herman and Winters proposed a series of questions along with suggested data to guide the evaluation of reliability and validity in reflective portfolio assessment.⁵⁰ For reliability, questioning if scores are consistent involves examining inter-rater agreement as a traditional conception of reliability. In terms of validity, the first consideration is questioning what the scores mean, examining if they are based on standards, and determining what achievement on the reflective portfolio and its respective tasks mean. The other consideration regarding validity is questioning whether inferences from scores on the reflective portfolio lead to appropriate decisions about students and the program. A key feature of this questioning of validity is an examination of the consequences of reflective portfolio use.^44-45,50 Various methods have been suggested to increase or improve reliability and thus validity; however, the literature is inconclusive on their effectiveness. ^{39,42-43,51-52} LeMahieu et al recommended a shared understanding by raters of the grading rubric in addition to training and calibrating raters using the grading rubric with benchmark samples of student reflective portfolios.⁵³

Qualitative vs. Quantitative Methods

Since the literature is not clear on whether it is possible to increase reliability estimates even with well-trained raters and a clear grading rubric, qualitative methods may be more appropriate given the qualitative nature of reflective portfolios.^{39,42-43,46,48,52} While inter-rater reliability has been suggested as the goal of reliability assessment, the inherent qualitative nature of reflective portfolios may preclude achieving acceptable reliability estimates, especially in regard to summative estimates. Driessen et al suggested that while having stricter assessment criteria and increased structure to the reflective portfolio process would increase inter-rater reliability, it would also negate the proposed benefits of the reflective portfolio itself.⁴⁶

Use of Electronic Reflective Portfolios as an Emerging Consideration

Electronic reflective portfolios have been used primarily in the instructional technology and the library science disciplines where electronic presentation is a natural extension of the type of work done in those fields.^36,38,54 Electronic reflective portfolios have been proposed as a way of addressing some of the disadvantages of traditional portfolios such as issues related to accessibility and dissemination, storage, and ease of revision.^55-56 The use of electronic reflective portfolios has been suggested as a form of authentic assessment that is not confined to a specific class schedule or meeting time.⁵⁵ With the continual advancements of technology and the use of computers, the expansion of electronic reflective portfolios appears to be an emerging consideration as institutions attempt the use of portfolios for learning and assessment. Greenberg cautioned that an electronic portfolio is not a personal homepage with links to examples of work but rather, “…a network application that provides the author with administrative functions for managing and organizing work (files) created with different applications and for controlling who can see the work and who can discuss the work (access).”⁵⁵ The ePortconsortium (http://eportconsortium.org) has developed a white paper on the use of electronic reflective portfolios to help begin the dialogue on issues related to the use of electronic reflective portfolios.

While electronic reflective portfolios do have some potential advantages relative to their traditional paper-based counterparts, the fundamental limitations associated with the use of reflective portfolios themselves, such as issues related to reliability and validity and student and faculty buy-in, remain. For example, McCurdy et al in exploring a web-based approach to outcomes assessment in a pilot study found that students have difficulty associating educational experiences with appropriate educational outcomes, usually attributing an excessive number of educational outcomes to each experience.⁴⁰ The difficulty in making connections between experiences and/or artifacts and competencies has also been identified as a problem with traditional paper-based reflective portfolios.^22,26,39

CONCLUSION

Student reflective portfolios have been used in many settings including nursing and medical school curricula. As schools and colleges of pharmacy work towards implementing student reflective portfolios as stated in Standard No. 15 of Standards 2007, there are several procedural and psychometric considerations that should be taken into account. The factors to consider synthesized from the literature along with associated questions are summarized in Table 1. These questions are offered to help schools and colleges of pharmacy develop and integrate student reflective portfolios into their respective curricula.

Table 1.

Factors and Associated Questions to Consider Based on the Literature