Student Evaluations of the Portfolio Process

Abstract

Objective. To evaluate pharmacy students’ perceived benefits of the portfolio process and to gather suggestions for improving the process.

Methods. A questionnaire was designed and administered to 250 first-, second-, and third-year pharmacy students at the University of Arizona College of Pharmacy.

Results. Although the objectives of the portfolio process were for students to understand the expected outcomes, understand the impact of extracurricular activities on attaining competencies, identify what should be learned, identify their strengths and weaknesses, and modify their approach to learning, overall students perceived the portfolio process as having less than moderate benefit. First-year students wanted more examples of portfolios while second- and third-year students suggested that more time with their advisor would be beneficial.

Conclusions. The portfolio process will continue to be refined and efforts made to improve students’ perceptions of the process as it is intended to develop the self-assessments skills they will need to improve their knowledge and professional skills throughout their pharmacy careers.

INTRODUCTION

The Standards and Guidelines of the Accreditation Council for Pharmacy Education (ACPE) recommend use of student portfolios to evaluate competency achievement.¹ Standard 15 states that every pharmacy college or school is required to assess and evaluate student learning and curriculum success, and Guideline 15.4 states, “Student portfolios should be used to document the students’ progressive achievement of the competencies throughout the curriculum and practice experiences. Portfolios should be standardized and include student self-assessment, as well as faculty and preceptor assessments of the educational outcomes.” Colleges and schools can assess such portfolios and use the knowledge gained to make appropriate changes in their curriculum when needed.

The University of Arizona College of Pharmacy incorporated portfolios into the doctor of pharmacy (PharmD) curriculum in 2002 to accomplish student- and college-oriented goals. Portfolios provide students an opportunity to evaluate and reflect on what they have learned over the past year relative to written competencies expected of them. These reflections are expected to help students develop professionally over the duration of their time in the program and reflect on those experiences while practicing. To help ensure that expected outcomes are achieved, faculty members at the college use students’ comments about the curriculum, specific classes, or subjects taught within a particular class to assess and change the curriculum when needed.

When done correctly, student-oriented portfolios can benefit students, although not all students may perceive the benefit. For example, approximately 31% of undergraduate pharmacy students at the University of Manchester disagreed with the statement, “Building the portfolio gave me a sense of achievement.”² Limited evidence exists to illustrate the perceived or actual benefit of completing portfolios.^3-5

Because portfolios are now strongly encouraged by ACPE, this study was undertaken to determine students’ perceived value of the portfolio process at the University of Arizona College of Pharmacy and to gather suggestions for improvements that might enhance students’ perception of benefit.

METHODS

This descriptive study used survey techniques on a convenience sample of first- to third-year PharmD students. Fourth-year students who were not on campus and could not be easily accessed were not included. The following background information provides context for the questions asked in the survey instrument.

Portfolio Process and Requirements

All PharmD students were required to complete the paper-based portfolio in January of each year of the program. Instructions about the portfolio process were initially given to students during first-year orientation and were provided again annually. Course coordinators provided guidance to students for their reflections by describing in their course syllabi the competency components of the outcomes expected that were covered in the course. Every year, at least 2 months prior to the time the portfolios were due, students also received “Outcomes Expected of Graduates,” a document that outlined the competencies on which they were expected to self-reflect.

In the instruction document, the purpose of the portfolio assignment was described to students as follows:

• To provide a documented overview of the experiences encountered during the student's professional training, co-curricular activities (eg, professional organization activities, such as brown bags, health fairs, attendance at professional meetings), and extracurricular activities (eg, work, other organization activities).

• To connect PharmD program outcomes to the professional training process.

• To provide the student an opportunity for self-assessment during the professional training process.

• To create meaningful opportunities for the student to meet with their college of pharmacy advisor.

• To provide an opportunity for the student to give feedback to the college about the curriculum, specifically related to the outcomes expected, so that continuous improvement can occur.

Students were told the following as part of the assignment overview:

The primary goals of the portfolio are to provide reflection on and documentation of professional growth as a pharmacy student, using the “Outcomes Expected Document” as the underlying structure. A written reflective report will provide a self-assessment of progress using the activities accomplished under each domain as evidence of professional growth. Students are responsible for the selection of the elements to support the experiences related to each domain. Documentation should come from course assignments, co-curricular and extracurricular activities, and internship.

Students had to organize their portfolio into 3 domains: patient care; professionalism and management in health systems; and health improvement, wellness, and disease prevention. Each domain had specific competencies that the students were required to address. The instruction document included the purpose of the assignment, an overview of the portfolio, a listing of required elements (table of contents, curriculum vitae or resume, and the reflective report, which should contain an introduction, reflections on each of 3 domains, and a conclusion), and examples of materials that would support reflection on learning experiences and activities. The latter section was limited to no more than 10 pages. This requirement was intended to prevent students from submitting excessively large binders full of photos, examinations, and certificates of participation in activities incongruous with the purpose of the process, which was to enhance skills of reflection about learning.

The document also suggested course associations and extracurricular and co-curricular associations that might have occurred with the various domains. For example, for domain 2, Professionalism and Management in Health Systems, students were told to consider how specific courses (by review of syllabi), introductory pharmacy practice experiences, work experiences, and co-and extra-curricular activities had helped them develop competencies in the domain.

Students submitted their portfolio work in binders to a central location, and the binders then were distributed to the students’ advisors. Students also were required to upload the reflections section of their portfolios to Turnitin.com. This service checks documents to determine whether there are elements common to previously submitted documents and publications as a check to ensure each student's work is original. Each student's portfolio was read by an advisor, who graded it using a 50-point grading rubric that was available for students to review for guidance prior to completing their portfolios. This grade was incorporated into a designated class as part of the student's grade for that course. A different course was used for this purpose for each academic year except for the first in which portfolios were considered a practice run and, although graded, did not count for course credit.

The advisor met with students individually to discuss their grade as well as thoughts and concerns they had addressed in the portfolio. Students were asked about stressors and courses or types of content that were particularly effective or ineffective.

The advisors collated student responses without attribution to the student, and the college's assessment committee reviewed this information annually. Problem areas then were discussed with individual faculty members and/or department heads in an attempt to identify areas for improvement. Examples of changes resulting from student feedback include movement of courses to a different semester to reduce the stress students reported were associated with a particular semester and better coordination of examination schedules.

All advisors were required to participate in an orientation session on evaluating the portfolios. This step was considered particularly important to create expectations of their role in the process, to inform nonpharmacist faculty members about the expected outcomes, and to help create consensus around the scores. Groups of faculty members met with the administrator in charge of the portfolio process, and each individual reviewed and graded the reflections from 3 portfolios that were selected to represent poor, intermediate, and high-quality performance. The facilitator then asked each faculty member to discuss his/her grading for each item on the scoring rubric and consensus was sought on appropriate scores for the 3 portfolios.

Survey Administration and Analyses

A questionnaire was developed by the authors and evaluated by 3 students and 3 faculty members, and revisions were made based on their suggestions. The authors obtained permission from course instructors to administer the questionnaire in May 2010 during a class session for a required course for each of the 3 pharmacy years. A class for a required course was selected for questionnaire distribution to have the greatest potential for reaching the most students in each class. Because this was a convenience sample, there was no attempt to survey students who were not in attendance on the day the survey was conducted.

Students’ responses from the questionnaires were entered into a database and SPSS Statistics 19 (IBM, Chicago, IL) was used to analyze the data. Written comments were collated and grouped by category for reporting and analyses. Differences between class years were analyzed using 2-way ANOVA. When significant differences were identified, a Bonferroni post hoc test was used to determine where the differences occurred. Demographic variables evaluated were age groupings and sex. The a priori alpha level was 0.05. To identify the aspects of creating a portfolio that contributed most to student perceptions of the benefits of the process, a multiple regression analysis was conducted. The analysis used overall benefit as the outcome variable and all other variables as predictors. Variables with a P value of less than 0.10 were retained in the final model. The University of Arizona Social and Behavioral Science Institutional Review Board exempted this study.

RESULTS

Questionnaires were completed by 250 students: 89 from the first year of the pharmacy program, 86 from the second year, and 75 from the third year. This response represented 85% to 90% of the students in each class. Table 1 describes the demographics of the 3 class years. The age of students tended to increase as the classes matriculated through the program with the exception of the oldest age category. No differences were noted in perceived benefit by age groupings or sex.

Table 1.

Demographics of Pharmacy Students Who Participated in a Survey on Their Perception of a Required Portfolio Process (N = 250)^a

Table 2 shows ratings of the questions related to the impact of the portfolio process, grouped by pharmacy class year. The ratings were not significantly different among the 3 years except in understanding the impact of extracurricular activities on achievement of desired competencies, time spent completing the portfolio, and the belief that more information about creating a portfolio would be helpful. The third-year class believed they spent more time on portfolio preparation than did either the first- or second-year classes, and the perceived amount of time spent increased incrementally from the first to third years. The first-year class thought that the portfolio process helped them to understand the contribution of extracurricular activities to achievement of competencies and to identify what they needed to learn to a greater degree than did third-year students. The first-year class believed more strongly that additional information would have helped them in the portfolio process.

Table 2.

Pharmacy Students’ Perceptions of the Impact of Preparing a Portfolio and Factors that Could Facilitate the Portfolio Process^a

Overall, students thought the portfolio process provided less than a moderate amount of benefit (mean, 2.3). Although still considered of only slightly greater than moderate value, ratings were highest for benefit received from doing their curriculum vitae (CV) or resume as part of the process. Students gave the lowest ratings to “modification of their approach to learning” associated with completing the portfolio (mean, 1.9).

Factors that might be expected to facilitate the portfolio process are also shown in Table 2. All classes rated the various statements about help they received from their advisors as being of moderate benefit to the process. Bivariate correlations among the rating of overall benefit of the portfolio process and advisor support, feedback on the portfolio from the advisor, directions for creating the portfolio, and help from the advisor for improving the CV/resume were all significant (P < 0.001). Perception of time spent preparing the portfolio was not correlated with perceived overall benefit (r = 0.05; P = 0.46).

Items that were predictive of students perceiving higher benefit from the portfolio process are shown in Table 3. In the regression model, there was an increased perception of overall benefit identified among students who thought the portfolio process (1) increased understanding of the expected outcomes of the professional program, (2) helped them examine the impact of extracurricular activities on attaining competencies, (3) helped them identify what should be learned, (4) helped them identify their strengths and weaknesses, and (5) modified their approach to learning. Variables related directly to advisor influence (ie, advisor support, feedback from advisor, or meeting with advisor) were not retained in the final regression model for perceived overall benefit.

Table 3.

Regression Model of Predictors of Benefit from Portfolio Process^a

Fifty-six percent of the students provided suggestions (Table 4). When comments were categorized into classes, 3 thematic areas and 1 “other” category resulted. These included: making the portfolio worth more credit (7%), providing more examples of portfolios (34%), having more time to spend with the advisor (17%), and other (42%). Of the first-year students who responded, 64% suggested more examples of portfolios be provided. Compared with first-year students, a higher percentage of second- and third-year students wanted more time with their advisor. The “other” category consisted of comments that did not have a high frequency of repetition or had no relevance to suggestions on improving the portfolio process, such as “don't make us do it” or “let the students do it in groups.” The other responses also were sufficiently varied so that it would have been conjecture to assume other students thought similarly. Thus, these comments are not reported. However, they do provide information that can be used in follow-up studies.

Table 4.

Pharmacy Students’ Written Comments on a Required Portfolio Process, No. (%)

DISCUSSION

The finding that time spent by students on their portfolios incrementally increased with each pharmacy year is to be expected because knowledge related to the expected outcomes increases with each pharmacy year. Thus, the reflection on pharmacy courses and where knowledge was acquired would be more in depth and require more time to process. Another factor likely impacting this outcome is the feedback received from advisors over the years. The students gained understanding of what the advisors expected from their reflections with each annual meeting.

Overall, the students perceived the portfolio process as having less than moderate benefit. However, ratings for each item had a large standard deviation, indicating that responses from those who found little value in the process were balanced by responses from those who found considerable value in the process. The highest mean was the students’ perceived benefit from completing the CV/resume as part of the process, although this was not a significant predictor of perceived overall benefit. Some other components that were individually associated with more positive perceptions of benefit were not seen as significant in the final regression model (Table 3). It appears to be somewhat important to students that the advisor is willing to meet with them, supports the portfolio process, and provides helpful feedback. Although students rated help from their advisors as being of moderate benefit to the process, there was considerable variability in response to these items, indicating that additional advisor training might be necessary to improve these outcomes.

The students’ lukewarm assessment on the benefit of the portfolio process was somewhat disappointing, considering the effort put forth by students and advisors, but the process definitely improved the role of the advisor. As with most colleges and schools of pharmacy, there is almost no need for academic advising at our college about which courses to take because almost all of the PharmD curriculum is required. Prior to the portfolio assignment, most students rarely ever visited their assigned advisor or saw the Associate Dean for Student Affairs unless they were in academic difficulty. However, based on the opportunity to work together on the portfolios, there is now a fairly robust relationship between advisor and advisee, and some of the results indicate perceived value in this relationship.

The percentage of students suggesting the need to increase the amount of time spent with their advisors increased according to year in school. While third-year students had completed the portfolio process 3 times, first-year students only had completed it once. The results of comment categories in Table 4 suggest that the students began to increasingly understand the importance of and to desire advisor involvement with each portfolio completed. Based on the results of a large meta analysis, the role of a mentor or advisor was important in the perceived benefit of the portfolio process.⁵ Several authors from different studies concluded that lack of support from advisors limited the potential benefit of the portfolio.^6-9 Another study suggested that the advisor should be involved in a discussion of the student's weaknesses and plan for improvement.⁶ The results of this study appear to support these findings to some degree, although variability in the quality of advising about the portfolio may have limited positive perceptions.

Portfolios can enhance students’ understanding of the impact of extracurricular activities, help them identify what they need to learn, identify their strengths and weaknesses, and help them understand how to modify their approach to learning.⁴ The results presented in Table 3 support this enhancement of student understanding and suggest that students would perceive the portfolio process as being more beneficial if advisors emphasized the understanding of expected outcomes of the professional program, the impact of extracurricular activities on attaining competencies, what should be learned, self-identification of the student's strengths and weaknesses, and modification of the student's approach to learning.

The advisor's role is to help students initiate and expand their self-reflection in these areas. Researchers at the University of Toronto stated that although completing a portfolio does not come naturally to many students, faculty members incorrectly assume that pharmacy students know how to complete portfolios and that there is little need for the advisor to explain the process.³ These findings seem to be supported by the responses of some students in this survey who expressed a need for greater explanation of the process to improve their understanding and to increase the benefit of completing the portfolio. First-year students specifically had a greater desire for more examples of portfolios compared with second- and third-year students. This makes sense because students in their second or third year had completed the portfolio process once or twice already and had personal examples and experience to which to refer.

A study performed by researchers at the Albany College of Pharmacy and Health Sciences evaluated the efficacy of electronic portfolios, which are becoming more common at colleges and schools of pharmacy.¹⁰ It would be helpful to know whether students perceive greater benefit from an electronic portfolio compared with a paper-based approach.

Because this study was conducted at a single college, the ability to generalize to the entire pharmacy student population is limited. Assumptions include that students read the questions completely and responded to them truthfully and that they understood the statements. Because the study was conducted 5 months after completion of the last portfolio, some of the associated issues would not be in the students’ most recent memory. Results may have been slightly different depending on when the survey was conducted relative to the portfolio completion date. It would be interesting to determine whether fourth-year students and those who have graduated have a different opinion of the process.

Based on the results of this study, a number of changes will be made to the portfolio process at the University of Arizona. The results will be provided to the faculty of the college so that they can see the potential impact of the advisor on more positive outcomes. Additional development of the advisors is also planned to enhance advisors’ portfolio-related interactions with students. This will include helping advisors guide student understanding of where learning is occurring and how they might enhance learning if they rate themselves as deficient in a particular competency. The Assessment Committee has decided that, in addition to reflecting on how and where they are achieving competencies, students also will be required to rate their level of achievement in each competency on a scale ranging from novice, to intermediate, to proficient. Although student directions for completing the portfolio imply assessment of competency achievement and lack of achievement (ie, students are told to include short- and long-term plans for gaining additional knowledge in the domains), using a rating scale may help students better reflect on their achievement of the competencies and change their approach to learning when they do not feel competent in a given area. Compilation of these scores also will allow the assessment committee to work with the curriculum committee to address any student-identified gaps in learning.

CONCLUSIONS

Self-assessment is a skill that should be refined throughout a pharmacist's career. The portfolio process is intended to be a tool to help students learn how to assess the development of their knowledge and skill competencies so they can continue to grow in areas of potential deficiency once they are practicing pharmacists. Our hope is that improving the portfolio process will help students develop these skills for application throughout their professional life.

The portfolio process also creates a meaningful opportunity for advisors to meet with their students and for students to document their experiences. Even though students in this study perceived the overall benefit of creating a portfolio as marginal, we are hopeful that their perspective on the value of the process will improve as they progress in their careers. Further, we expect the results of this study to serve as a guide to the college in enhancing the portfolio process for both students and their advisors.