A Mixed-Methods Analysis in Assessing Students’ Professional Development by Applying an Assessment for Learning Approach

Abstract

Objective. To describe an approach for assessing the Accreditation Council for Pharmacy Education’s (ACPE) doctor of pharmacy (PharmD) Standard 4.4, which focuses on students’ professional development.

Methods. This investigation used mixed methods with triangulation of qualitative and quantitative data to assess professional development. Qualitative data came from an electronic developmental portfolio of professionalism and ethics, completed by PharmD students during their didactic studies. Quantitative confirmation came from the Defining Issues Test (DIT)—an assessment of pharmacists’ professional development.

Results. Qualitatively, students’ development reflections described growth through this course series. Quantitatively, the 2015 PharmD class’s DIT N2-scores illustrated positive development overall; the lower 50% had a large initial improvement compared to the upper 50%. Subsequently, the 2016 PharmD class confirmed these average initial improvements of students and also showed further substantial development among students thereafter.

Conclusion. Applying an assessment for learning approach, triangulation of qualitative and quantitative assessments confirmed that PharmD students developed professionally during this course series.

INTRODUCTION

Assessment drives learning.^1,2 While well-accepted and established, this dictum comes from experiences of summative assessment within an assessment of learning paradigm. More recently, an assessment of learning paradigm has been juxtaposed with an assessment for learning approach.^3,4 Inherent to this second approach is using formative assessment to help guide learners’ development as opposed to simply having learners react to their summative grades. Formative assessment can include instructor feedback and students’ self-assessments; an evidence-based method for formative assessment involves writing reflections.^5,6 Based on the question posed for students to answer, reflective writing can describe students’ experiences, behaviors, and values from their own pharmacy practice opportunities. Reflections have also shown a post-assessment “backwash” effect, wherein students can learn from their experiences and create or revise their mental framework for growth.^5,7 While the dictum “assessment drives learning” still applies to this formative approach, it may be better stated as “assessment guides learning.”

Professionalism is the soul of any profession,⁸ and understanding its framework is essential.^8,9 Accreditation organizations for health professions education stress deliberate, explicit instruction to students in this foundational domain,^10-14 and not leaving it as a “hidden curriculum.”⁹ Using a programmatic assessment for learning approach, we developed a longitudinal series of professionalism and ethics throughout the pharmacy practice development coursework for PharmD students. This didactic and recitation coursework was focused on introducing: (1) a definition and framework for the six tenets of professionalism (altruism, honor and integrity, duty, excellence, respect for others, and accountability);^15,16 (2) professionalism tenets applied to pharmacy students experiences (altruism, professional presence, professional stewardship, commitment to excellence, honesty and integrity, and respect for others);¹⁷ (3) biomedical ethics principles (beneficence, nonmaleficence, autonomy and justice);¹⁸ (4) situations and issues in professionalism and ethics; (5) interprofessionality;^19,20 and (6) teamwork.²¹ A developmental portfolio was created to compile student self-reflections written on these topics over time and to allow students to revisit previous reflections longitudinally throughout their studies.

In using a programmatic assessment approach, Standard 4 of the 2016 Accreditation Council for Pharmaceutical Education (ACPE) Standards requires that programs develop PharmD students in aspects of professionalism, self-awareness, leadership, and innovation, while focusing on values and behaviors (ie, attitudes and behaviors).¹⁰ We developed an innovative method of assessment for 2016 Standard 4.4, as requested in ACPE’s accompanying Standards 2016 guidance document.²² While the method builds on an assessment for learning paradigm, the adaptation and development of this approach to professional development of pharmacy students is novel.

METHODS

This was a mixed-methods investigative report,²³ using triangulation²⁴ of quantitative data to confirm our qualitative assessment approach using written reflections. This investigation was IRB-approved as exempt by the University of Toledo.

At a 4-year PharmD program, students participated in a longitudinal professionalism and ethics series that spanned their first, second, and third years of mainly didactic instruction. At key points throughout this series, students completed multiple reflective writings. Appendix 1 provides instructional details. These reflections were most often in-class, free-flow writing assignments in response to a question prompted in class (much like a diary entry). Formal instruction on how to write reflections was not provided but students practiced writing reflections during orientation before the first year of this PharmD program (students were asked to write a reflection specific to a provided question based on required orientation readings). Students’ ability may have been further expanded with feedback received from orientation instructors on those initial reflections.

Reflection prompts were designed to encourage student reflection and writing around the core components in a conceptual model on writing reflections proposed by Nguyen et al: (a) focus on self in an attentive, critical, exploratory, and iterative way; (b) deal with each learner’s thoughts and actions; (c) involve learners’ underlying conceptual framework of professionalism and bioethics; and (d) use a view towards changing learners.²⁵ Reflections in this development series were planned to guide students in mindful reflection²⁶ on their progress within the tenets of professionalism.^6,17,27 Over the series of lectures, concepts were introduced and revisited in subsequent semesters as students were asked to reflect on their ongoing progress with a specific tenet of professionalism.

Students were required to upload all reflections to an electronic development portfolio (CoreTrax, Temperance, MI). Within this portfolio were 22 total assignments with eight written reflections from year 1 (P1), six from year 2 (P2), and eight from year 3 (P3), culminating in a summative reflection focused on professional development across the series. An overview of the longitudinal series design, along with many examples of specific reflection questions, is in Appendix 1; specific lecture and reading references are included as well.

Each reflection was reviewed and graded by the instructor, an experienced educational researcher. Each reflection was graded as “acceptable” if the student adequately addressed the questions or was returned to the student to resubmit a second reflection if the questions were not adequately addressed. Since these reflections were based on student experiences, there was no “right” or “wrong” answer, as long as the question posed was answered. The instructor also wrote a short statement of correction, encouragement, or a thought-provoking question on each reflection, for students to reflect on afterwards.

In addition to grading the reflections as “acceptable” or “redo,” the researcher used qualitative research methods to identify themes discussed by students describing their professional development. Qualitative methods allow a deeper analysis of the written text to help explain how development occurs in students, and can provide a method of triangulation for quantitative data collected in parallel.²⁸ Using an a priori method of interpretive coding, the researcher established a coding scheme looking for two identifiers of professional development. The first was to identify if students discussed improvement in their professional development in the summative essay. Each essay was assessed as “yes” or “no” for this item. Examples of terminology describing this improvement include reference to “growth,” “improvement,” or “development.” The second coding category was predetermined based on the professionalism literature of evidence for development in this area. Evidence of change in attitude or change in behavior are key factors in professional development,²⁹ so each reflection was reviewed for mention of these changes. Essays could reference either of the categories or both.

The researcher independently coded all 192 final reflections written by students at the end of their P3 year. Essays submitted throughout the series were not assessed individually; instead, the culmination of students’ development as described in the summative essay was explored using thematic coding. Because each essay was approximately one page in length, coding took about three minutes per essay (approximately 10 hours over two years for 192 essays).

The study followed a longitudinal design, a strength noted by others for studies of development.³⁰ Over four years, PharmD students were required to take critical thinking assessments during their didactic curriculum. The critical thinking assessment results have been reported elsewhere.³¹ The Defining Issues Test (DIT) is a standardized test used in studies of cognitive-moral development and involved tens of thousands of participants.^32,33 Ethics is foundational to professionalism,³⁴ and the DIT is used to measure professionalism in medicine³⁵ and pharmacy³⁶ among other professions.³³ Furthermore, multiple authors recommend the DIT for assessment within pharmacy education.^36-38 Using multiple test administrations over time, the DIT is a quantitative measure of development in professionalism. In this basic parallel mixed analysis,²³ the DIT was used to confirm the qualitative reflective portfolio approach. In a four-year PharmD program, the DIT was administered in fall of the first year, spring of the first year, spring of the second year, and spring of the third year. Students in each class were broken into two groups. Group A took the DIT in the P1 fall semester and the P2 spring semester; they took a different critical thinking test during their P1 and P3 spring semesters. Meanwhile, group B took the DIT in the P1 and P3 spring semesters. They took another critical thinking test during the P1 fall semester and P2 spring semester. As a result, students took only one test during each test administration in an attempt to ease test-taker fatigue during this longitudinal programmatic assessment initiative.

By measuring professionalism, the DIT was used as a quantitative confirmation of our qualitative findings. The DIT is a multiple-choice test that takes about 20-30 minutes to complete, and has only one version for all testing administrations. The latest version (2) has five ethical cases for which the test-taker ranks a sequence of possible solutions, with options based on different levels of moral thinking and ethical values. Because this test does not use knowledge-based ethics content, but instead asks what the test-taker would do, it approximates a behavior-based assessment—similar to situational judgment tests used in admissions and personnel hiring.³⁹

The N2-score from the DIT is recommended for professional and graduate program use—with an N2-score scale between 1-95.⁴⁰ While there are no interpretations for specific score ranges, Table 1 provides norms.⁴¹ Development from initial to subsequent scores is emphasized by test developers.³² Multiple articles using the DIT appear in the Journal, however most reported the older P-score index. While the DIT has remained similar, the newer N2-score index is recommended for professional and graduate program use instead of the prior P-score index.⁴⁰ The N2-score is on the same scale as the older P-score but uses more developmental DIT data than the P-score. We also found the N2-score more psychometrically reliable than P-scores with repeated test administrations.

Table 1.

Defining Issues Test (version 2) N2-Score Norms and Range⁴¹

After administration to students, the DIT was scored remotely. Student scores were collated using Excel, while descriptive and inferential statistics were identified with SPSS, v19 for Mac (IBM, Armonk, NY). For longitudinal, dependent test scores of group A or group B, a paired t test was used for each comparison. Thus, students in group A had their scores from their P1 fall semester compared with their P2 spring scores, while group B student scores from their P1 spring semester were compared to their P3 spring scores. Positive development was defined as an improvement in scores over time.

Similar to Phillips et al⁴² and Duckett et al’s nursing research report that used the DIT,⁴³ we compared development in the upper 50% of the class to that of the lower 50% of the class using a paired t test. The upper and lower half of each class was calculated from assessment scores on the initial occasion (ie, a student with a P1-fall DIT in the upper half of their class, was placed in that half for further analysis).

Cohen’s d was used to provide evidence towards practical significance.^44,45 It was calculated using an online calculator (University of Colorado, Colorado Springs, CO). Effect sizes were compared with standards recommended by Cohen (trivial<0.2, small=0.2, medium=0.5, large=0.8).⁴⁶ For example, with Cohen’s d, an effect size of 0.20-0.49 represents a small effect size—meaning the two groups’ average scores differ by 0.20-0.49 standard deviations; this difference is often inconsequential, even if it is significant statistically.⁴⁵ Medium or larger Cohen’s d effect sizes are often meaningful for educational interventions.^45,47 To avoid erring with too many inferential statistical tests and inflating the experiment-wise alpha, comparison of these effect size interpretations was used instead across independent groups and yearly cohorts.

RESULTS

For the final reflective writing in the professionalism series, 192 P3 students from graduating classes from 2015 and 2016 were reviewed. In this final reflection, students were prompted to describe their overall development with the professionalism tenets (reflection number 22 in Appendix 1). All students discussed improved professional development (192 of 192; 100%). Examples descriptions of this improvement can be found in Table 2. For the second coding category describing how this development occurred, 174 of 192 (91%) described changes in attitudes while 157 of 192 (82%) described changes in behavior. Most students described changes in both attitude and behavior (146 of 192; 76%).

Table 2.

Qualitative Excerpts Describing Professional Development^a

Examples of student descriptions of change in attitude include: “My outlook towards professionalism shifted from my initial impression that was simply a demeanor that I should embrace while in a work environment, to an understanding that professionalism is a lifestyle choice;” “Picking out one tenet at a time and reflecting on it made me look at many different aspects of my personality and how it applies to working in a pharmacy setting;” and “I was surprised to see the difference in my thought process from two years ago.”

Examples of student descriptions evidencing change in behavior include: “I have been practicing the tenets as a student pharmacist and in my everyday life;” “I believe all the tenets have helped me in my personal and professional relationships by teaching me how to properly communicate with others and view others’ opinions;” “Since my P1-year, I have progressively put the patient’s needs first, taken responsibility for my actions, have improved my ethical conduct, and improved the way I present myself to all patients;” and “I feel that I still have some struggles (but have improved) in the tenets: respect for others, professional stewardship, and dedication & commitment to excellence.”

The internal consistency (reliability) from all DIT administrations was ≥0.7 for N2-scores. For the 2015 PharmD class (n=84), group A’s DIT N2-scores were 41.5 (SD=12.1, P1-fall) and 45.4 (14.8, P2-spring). Inferentially, a paired t test was significant (p=0.05); its effect size of 0.3 by Cohen’s d was small (Figure 1). In group A, the upper 50% and lower 50% were different (as in Figure 2): the upper 50% showed no difference between DIT measurements (paired t test, p=0.92; trivial effect size of 0.03), while the lower 50% had a large effect size of 0.8 (paired t test, p<0.01). In Figure 1, group B’s DIT N2-scores were 37.1 (12.2, P1-spring) and 39.9 (16.0, P3-spring); a paired t test was p=0.14; its effect size of 0.2 by Cohen’s d was also small. Differences between the upper 50% and lower 50% of group B were muted and not significant (paired t tests, p=0.20, p=0.34, respectively; small effect sizes of 0.3, 0.3 respectively; Figure 3).

Figure 1.

Professionalism development within PharmD students (using N2-scores from Defining Issues Test, version 2).

Figure 2.

Professionalism development in the upper and lower halves of group A pharmD students (using N2-scores from Defining Issues Test, version 2).

Figure 3.

Professionalism development in the upper lower halves of Group B PharmD students (using N2-scores from Defining Issues Test, version 2).

For the 2016 PharmD class (n=82), group A’s DIT N2-scores were 38.4 (13.7, P1-fall) and 43.2 (14.3, P2-spring). Inferentially, a paired t test was significant (p=0.03); its effect size of 0.3 by Cohen’s d was small (Figure 1). Similar to the 2015 data, the 2016 upper 50% and lower 50% appeared different (Figure 2). The upper 50% showed no difference between DIT measurements (paired t test, p=0.76; trivial effect size of 0.07), while the lower 50% had a large effect size of 0.8 (paired t test, p=0.03). In Figure 1, group B’s DIT N2-scores were 35.8 (15.1, P1-spring) and 45.8 (14.3, P3-spring); a paired t test was p<0.01, and its effect size of 0.7 by Cohen’s d was a medium change. As opposed to the 2015 class results, both the upper 50% and lower 50% of group B improved (paired t tests, p<0.01, p<0.01, respectively; effects sizes of 0.5 [medium], 1.3 [large], respectively; Figure 3).

DISCUSSION

University of Toledo PharmD students developed professionally during the series; both qualitative reflection evidence and quantitative DIT scores showed this development. While reflective portfolios are considered a powerful learning assessment, their lack of reliability is a drawback.⁴⁸ However, this assessment was triangulated with the reliability from the DIT, which showed similar outcomes in student development. When students could reflect on growth over reflections from earlier years, this developmental portfolio became an assessment for learning. Students’ past reflections helped guide them toward thinking about, reflecting on, and monitoring their future growth in professionalism (ie, formative assessment). This guidance towards reflection-in-practice is an important skill for future professionals to construct as the transform their professional identity.⁴⁹

By comparing the upper and lower 50% from each cohort, we illustrated that initial development appeared to primarily come from less professionally developed students. However, later development appeared in both more and less professionally developed students. Over time, all students appeared to benefit from this approach.

It is essential to create learning assessments that foster this type of learning. An instructor can follow guidelines for teaching professionalism, but without the critical step of having students reflect and accrue multiple related reflections over a longitudinal period in a repository to document development, that longitudinal teaching may not “stick” or be appreciated as such by learners. Reflections can help students see the educational strategies that instructors work to implement effectively.^5,7 This assessment for learning is vital to a professional identity formation that is student learner-centered.

Statistical significance does not mean practical significance, though practical significance is not as well-defined as statistical significance. Pursuant to this investigation, there are a few ways we suggested practical significance.⁴⁵ One is using standardized effect-sizes such as Cohen’s d, with generic effect-size interpretations of small, medium, and large.⁴⁶ This study highlights promising results that were replicated in a subsequent PharmD class, albeit from the same learning environment and institution.

A second method for practical significance is to use educational benchmarks. Findings from K-12 education show a similar large initial development followed by less development as students’ education continues.⁵⁰ In higher education, the Wabash study was a noteworthy longitudinal study of cognitive-moral development among 2200 undergraduates at 17 higher education, 4-year institutions (mainly liberal arts colleges).⁵¹ Using the DIT, the Wabash study found a similar trajectory in development—most in the first year (0.38 x standard deviation) and less over the subsequent three years (0.26 x standard deviation). While the 2015 class showed less development statistically, figures 1-3 show similar trends between the 2015 and 2016 PharmD classes.

A further means of identifying practical significance within benchmarking is to compare similar institutions. In a meta-analysis of DIT studies of development in pharmacy education, investigators found negative growth in pharmacy student development; the pooled estimate was negative.⁵² There was substantial heterogeneity, though no groups showed an effect size as large or positive as that shown in this mixed methods analysis. However, in a meta-analysis including health professions students beyond pharmacy, the DIT consistently showed improvement for educational programs except pharmacy.⁵³ For pharmacy education, this investigation showed student development that education programs from other professions have already consistently shown.

Of note in the Wabash study findings, assessment also directly drove learning. That is, after controlling for other issues, students who took the DIT three times in four years showed a greater increase in scores than did students who took it only twice.⁵⁴ This may be a promising use of the DIT in pharmacy education as well, not only using the DIT for assessment of learning during PharmD curricula as has been recommended,^36-38 but also as assessment for learning, with multiple (more than two) administrations driving student learning in moral reasoning and professionalism.

Following their book Teaching Medical Professionalism,⁹ Cruess et al provided this summary regarding teach professionalism: “The features of successful programs devoted to the teaching of professionalism include longitudinal integration of material throughout the educational continuum, a cognitive base outlining the nature of professionalism and professional obligations, and programmed opportunities for reflection on experiences relating to professionalism.”⁵⁵ The assessment at the University of Toledo College of Pharmacy and Pharmaceutical Sciences had included all of these successful program features. As all pharmacy education institutions are challenged to comply with ACPE Standards, this formative assessment for learning approach may help document learning and guide learner development. Programmatic assessment is not confined to only Standard 4.4, but is a paradigm shift away from focusing only on assessment of learning for other standards as well.

We are not advocating that PharmD programs always do qualitative and quantitative assessments of professional development. However, after implementing a qualitative development portfolio for Standard 4.4, it may be helpful to confirm the assessment with quantitative findings—though not essential. We suggest that programs use the DIT periodically to assess professional development of students. As noted previously, a meta-analysis of pharmacy education did not show substantial increases.⁵² Our study appears to be the first to report a substantial positive development. Note that students within other health professions did show development,⁵³ but not pharmacy before this report. We suggest pharmacy programs administer the test only twice (ie, near the beginning of program and near completion of didactic portion) because two scores can be more easily analyzed, and the longer time period between the tests can maximize the period for development to take place. Administration more than twice may help further guide learning,⁵⁴ but must be balanced with poor student motivation^56,57 of this low-stakes testing. In the Wabash study, students received financial remuneration for taking the multiple tests.

In terms of limitations, the association between feedback from qualitative portfolios and results from quantitative DIT scores is not cause and effect. Within this mixed-methods approach for assessment, triangulation was used to converge these validity evidence sources. Using portfolio-based reflections was key to capture the varied experiences of students and help cultivate their professionalism learning. The majority of students’ reflected on experiences from outside of the formal classroom-based curriculum; pharmacists and preceptors in many settings were vital to these students’ reflections. As with any single assessment method, content specificity is a concern;⁵⁸ however, triangulation of evidence was helpful, as was using the DIT assessment, which is similar to reliable situational judgment tests that use multiple cases during each test administration.⁵⁹ Regarding the basic parallel mixed-methods design, we did not categorize reflections within the same group A/B that we used for the students taking the DIT. Students were randomly assigned to group A/B for DIT administration in an attempt to limit test-taking burden for student with low-stakes tests and maximize the quality of test responses.^56,57 While we used a only statistical comparison with longitudinal data for study of development within each group,³⁰ we hoped randomization would allow us to generalize the interpretation of group A or group B results to the entire class. A more complex integrated mixed analysis design²³ could be used in a future study. Finally, this investigation was conducted in a learning environment at one institution and used only one coder.

CONCLUSION

Using triangulation qualitative and quantitative assessments confirmed that PharmD students developed professionally during a professionalism series. Students who were less developed initially improved more than other students did. However, all students ultimately appeared to develop. This formative assessment approach helped document and guide learners through their professional development. This study serves as a PharmD-specific example in assessment for learning.

Appendix 1. Instructional Details of the Professionalism and Ethics Series: Reflection Questions and Readings.^a

P1 Fall semester: (Initial foundational professionalism framework, after lecture^15,16 and reading¹⁷)

• 1) What did you find most difficult or confusing to understand in today’s reading? (If nothing was difficult or confusing, describe what you found most interesting)

• 2) What tenet will you find most challenging for your future as a pharmacy student, and why?

• 3) Describe how a pharmacist (or senior pharmacy student) has embodied your challenging tenet

P1 Spring semester: (with another reading⁶⁰)

• 4) How have you grown with your difficult tenet since your last reflection (#2)

• 5) What did you find most difficult or confusing to understand in today’s reading? (If nothing was difficult or confusing, describe what you found most interesting)

• 6) Did you notice that the author never used the term “customer” when describing a “patient”? Why?

P1 Spring semester: (following a bioethics lecture¹⁸)

• 7) [ethical case on universal healthcare coverage] From a bioethical standpoint, should we (the United States) strive for universal health care coverage to include all Americans?

• 8) [ethical case on refusing to dispense] How can this pharmacist meet bioethical principles without compromising his own belief set? What ethically should be this pharmacist’s next thinking and action?

P2 Fall semester: (no prior reading or lecture)

• 9) Describe your progress after stumbling with a difficult professionalism tenet. How have you stumbled with it (or found it challenging to develop further) this past year? What plans do you have to improve?

• 10) Which professionalism tenet has been most difficult for you while using social networking (like Facebook)? Why?

P2 Spring semester: (with multiple readings^36,61-63)

• 11) How is professionalism impacted by commercialism? What implication does this have for you as a student pharmacist?

• 12) In the article, the four bioethics principles are noted, and the impact of beneficence is explained further. How are the other three principles impacted by a pharmacist’s lack of action when knowing better than the status quo?

• 13) Specific to the six tenets of professionalism, how does this article challenge you with your upcoming IPPEs and APPEs?

• 14) After learning about the psychological perils of accepting gifts, how do you see your view and practice changing for a future situation in which a gift is offered to you?

• 15) How long did Kegan find it takes to progress through one developmental stage? What can YOU do to mitigate barriers to your own development and optimize your progression as best you can?

P3 Fall semester: (no prior reading or lecture)

• 16) Describe the interprofessional (or interdisciplinary) team that your neighbor had participated within or observed. Which of the seven principles of interprofessional teamwork did your neighbor’s team most effectively illustrate through their interactions together?

• 17) Which of the principles do you expect to be most difficult for YOU to do within your next team? Why?

• 18) Looking back at a recent team you have been apart of, which dysfunction was most problematic? How should this have been addressed early? What role should YOU have played in addressing it?

P3 Spring semester: (prior reading^64,65)

• 19) How have you developed your empathy to this point, and how are you planning to continue?

• 20) From your experience, give a brief example about dealing with EACH of the four bioethics principles.

• 21) In second year, you discussed a tenet that you stumbled with and a plan for improvement. How did you do with your prior plan? What was successful and what plan pieces needed revision. How do you think you are doing with development of this difficult tenet now?

• 22) Having completed this professionalism series from first-year to third-year, how would you describe your own OVERALL development with professionalism tenets for this very important PharmD program outcome?

^a An expectation was for a half- to one-page response for each question (depending on time given of 5-10 minutes for writing)