Pharmacy Student Learning During Advanced Pharmacy Practice Experiences in Relation to the CAPE 2013 Outcomes

Abstract

Outcomes from The Center for Advancement of Pharmacy Education (CAPE) are intended to represent the terminal knowledge, skills, and attitudes pharmacy students should possess and have guided delivery of pharmacy education for more than two decades. Advanced pharmacy practice experiences (APPEs) are the endpoint of pharmacy curricula where demonstration and assessment of terminal learning occurs. This review examines published literature in relation to the most recent CAPE outcomes to determine the extent to which they have been addressed during APPEs since 1996. Details related to the APPE focus, intervention(s)/learning setting(s), and assessments are summarized according to the 15 CAPE outcomes. Further, the assessments are categorized according to the level of learning achieved using an available method. Common CAPE outcomes are highlighted, as well as those for which published reports are lacking for APPEs. The range and quality of assessments are discussed and emphasize the need for continuous improvement of scholarly design and assessment.

INTRODUCTION

The Center for Advancement of Pharmacy Education (CAPE) outcomes have guided pharmacy curriculum design, implementation and assessment since 1992, with the most recent iteration released in 2013.¹ Since their inception, the CAPE outcomes have been a key resource for colleges and schools, faculty members, and preceptors when considering terminal knowledge, skills, and attitudes pharmacy graduates must possess. The revision process of the CAPE outcomes seeks to identify and include contemporary and emerging issues in pharmacy education so that the pharmacy profession may meet current and future health care needs. The CAPE outcomes are intended to be achievable by the conclusion of a doctor of pharmacy (PharmD) program, and measureable within practice environments. Since the advanced pharmacy practice experience (APPE) component of the curriculum is the culmination of a professional program, it represents the final phase in which achievement of CAPE outcomes occur. Further, the American Council for Pharmacy Education (ACPE) chose to adopt, without modification, the CAPE 2013 outcomes as the educational outcomes of focus for the latest revision of its Accreditation Standards for Doctor of Pharmacy Programs.^2,3 More emphasis is now placed on assessment of student achievement of CAPE outcomes,² which requires thoughtful design and measurement.

Promoting the scholarship of teaching and learning is important for the advancement of academic missions. Experiential education has been capturing representative terminal assessments of curricular outcomes through publications for nearly two decades since conversion to the entry level PharmD program model. The CAPE outcomes have influenced the focus of experiential education during this time frame; however, it is uncertain if literature related to experiential education addresses all of the CAPE outcomes. Experiential education program design, delivery, and assessment provide an important opportunity for terminal assessment of CAPE outcomes. Understanding how extensively experiential education literature addresses CAPE outcomes may identify gaps and help the academy advance curricular initiatives. Equally important is describing the level at which achievement of educational outcomes has occurred in APPE environments, which can help refine approaches to experiential learning. A method of evaluating the effectiveness of training programs as defined by levels of learning using Kirkpatrick’s hierarchy (KH) has been described.^4,5 This method of evaluating levels of learning has been applied to e-learning in pharmacy education using four general levels [(1) reaction, (2) learning, (3) behavior, and (4) results], and is applicable to experiential education as well.⁶ The primary purposes of this Review are to determine the extent to which published literature involving pharmacy students on APPEs addresses the CAPE outcomes and to describe the level of learning assessments relative to KH.

METHODS

To identify keywords and Medical Subject Heading (MESH) terms related to our primary purpose, the table of contents of the American Journal of Pharmaceutical Education, Currents in Pharmacy Teaching and Learning, and Annals of Pharmacotherapy were reviewed for the calendar year 2012. These journals were selected to represent those likely to publish scholarly articles related to experiential teaching and learning in pharmacy. A Medline search was also performed from 1996 to 2014 using the term “experiential education and pharmacy.” Article titles and abstracts for both introductory search methods were reviewed for relevance, in addition to extracting MESH headings to assist with refining search terminology. By reviewing desirable citations for common search terms, the initial MESH terms of “education, pharmacy” or “students, pharmacy” were grouped with “Experien*.mp” to broadly characterize our initial search strategy. This search strategy was first utilized in OVID for journals indexed in Medline using the time frame of 1996 to December 31, 2014 and subsequently deployed for other search engines including Excerpta Medica Database (EMBASE), Education Resources Information Center (ERIC), and International Pharmaceutical Abstracts (IPA), matching the most closely related search terms within each database for relevant journals. The beginning time frame of 1996 was selected as a time point just prior to implementation of the ACPE entry-level PharmD program standards in 1997, when APPEs were first articulated, to capture early reports that may have addressed terminal outcomes still deemed relevant at the present time. The final databases used were selected to encompass major teaching and learning journals in the health professions.⁷ In addition, the table of contents of the journal Innovations in Pharmacy was reviewed since it was not indexed in any of the available search engines.

Citations retrieved from the search strategy were reviewed for relevance and included if they reported assessment of student learning through APPEs in PharmD programs of US schools of pharmacy. Reports of international APPEs were included only if they originated from a US school of pharmacy. Only noncompensated reports were included, as opposed to those involving paid internship experiences outside the curriculum. For inclusion, an article needed to include data from assessment of learning in relation to the educational activity or intervention. Articles that reported student contributions to patient care such as documentation of clinical interventions but that did not also include assessment of student impact such as acceptance or implementation of interventions were not included. Articles purely descriptive in nature or that reported general findings for competencies or outcomes, but lacked specific assessment information, also were not included in our review.

A minimum of two authors independently reviewed each citation for inclusion and selected up to three CAPE outcomes most closely related to the learning assessed through the respective APPE; discrepancies of categorization were resolved via author consensus. Data extraction for each citation included the type of educational intervention, assessment methods utilized, number/type of students, limitations/other relevant information, and the practice setting (if applicable). Each citation was also reviewed in duplicate for the impact of the educational intervention and assigned one of six levels from KH of learning (Appendix 1).^4-6 Discrepancies of categorization were again resolved via author consensus.

RESULTS

Our review and synthesis included 91 published articles. The CAPE outcomes deemed to be the primary focus of each article and the associated KH assigned are listed in Table 1.^8-98 Overall, the subdomains of patient-centered care (2.1), learner (1.1), and self-awareness (4.1) were the most commonly addressed primary outcomes for included publications, and these areas occurred in a comparatively equal number of reports. The subdomains of patient advocacy (3.3), leadership (4.2), and innovation/entrepreneurship (4.3) were not found to be primary outcomes of focus in published literature to date. With regard to KH level, the approximate distribution was 36% of reports at level 1 (reaction/participation), 30% at level 2 (learning), 7% at level 3 (behavior change), and 27% at level 4 (results).

Table 1.

Categorization of Publications Reporting Student Learning on Advanced Pharmacy Practice Experiences by Primary CAPE Outcome Addressed and Impact of Intervention by Kirkpatrick’s Hierarchy Level

Tables 2-4 represent stratifications of included articles by KH levels 1, 2, and a combination of levels 3 and 4, respectively. Summary details for articles are organized by CAPE outcome subdomain, APPE knowledge/focus, intervention/learning setting, and assessment method(s) within each of the respective KH level designations. Table 2 includes all articles that used assessment methods consistent with KH level 1, such as student or faculty-completed surveys post-APPE, the majority of which were not specified as being validated.^{8-12,24-26,50,53-56,61,65,66,69-71,74,76-88} Approximately three-quarters of the articles used student-completed surveys as the primary assessment method. Two-thirds of the 15 CAPE outcome subdomains were represented by APPE reports using assessment methods consistent with KH level 1, with self-awareness (4.1) being the most commonly represented subdomain for this level.

Table 2.

Advanced Pharmacy Practice Experiences by Primary CAPE Outcome of Focus Categorized at Kirkpatrick’s Hierarchy Level 1 (reaction)

Table 3.

Advanced Pharmacy Practice Experiences by Primary CAPE Outcome of Focus Categorized at Kirkpatrick’s Hierarchy Level 2a (change in attitude/perception) or Level 2b (change in knowledge)

Table 4.

Advanced Pharmacy Practice Experiences by Primary CAPE Outcome of Focus Categorized at Kirkpatrick’s Hierarchy Level 3 (change in behavior), 4a (change in organizational practice) or 4b (improvement in health/well-being of patients)

Table 3 includes all articles that used assessment methods consistent with KH levels 2a (modification of attitudes/perceptions) or 2b (modification of knowledge/skills).^{13-21,27-31,57,58,63,72,73,75,89-95} Survey assessment methods included pre/post-APPE applications and some examples of validated surveys for CAPE subdomain 3.5 (cultural sensitivity). Knowledge and skill assessments included methods such as multiple-choice or written examinations and preceptor-directed performance assessments of students. Only 40% of the 15 CAPE outcome subdomains were represented by APPE reports using assessment methods consistent with KH level 2a or 2b, with learner (1.1) and self-awareness (4.1) being the most commonly represented subdomains for this level.

Table 4 includes all articles that used assessment methods consistent with KH levels 3 (behavior change), 4a (change in organizational practice), or 4b (improvement in patient health/well-being).^{22,23,32-49,51,52,59,60,62,64,67,68,96-98} Nine of 15 CAPE outcomes (60%) were represented at these higher KH levels, and approximately 60% of the reports were comprised of subdomain 2.1 (patient-centered care). Assessment methods used were consistent with this subdomain and included medication reconciliation, identification of drug-related problems, and acceptance of clinical intervention recommendations, as well as other measures promoting quality and/or cost-effectiveness.

Table 5 summarizes the categorization of publications including not only the primary CAPE outcome, but a second and third level of focus, when applicable. This summary shows a broader view of the CAPE outcomes addressed and reveals that the subdomain of communication (3.6) was commonly included at the second or third level of focus, making it the fourth most commonly addressed subdomain behind patient-centered care (2.1), learner (1.1), and self-awareness (4.1). Table 6 organizes the information from Table 5 by date ranges and shows a progressive increase in the number publications per time frame for the majority of the subdomains since 1997, while other subdomains have only been reported more recently.

Table 5.

Categorization of Publications Addressing up to Three CAPE Outcomes for Advanced Pharmacy Practice Experiences

Table 6.

Distribution of CAPE Outcomes Addressed in Advanced Pharmacy Practice Experiences from 1997 to 2014.^a

DISCUSSION

A primary purpose of this review was to determine the extent and level to which published literature involving pharmacy students on APPEs assesses learning relative to the CAPE outcomes. Several observations from the results are noteworthy. First, faculty members and preceptors reported pharmacy student learning and contributions in areas consistent with the majority of the 15 CAPE outcome subdomains. This is encouraging because the APPE year comprises at least 25% of the curriculum and represents terminal learning for professional programs. It is also clear that the primary focus for approximately 47% of the included reports related to the subdomains of either patient-centered care (2.1) or learner (1.1). This finding is not surprising as these two subdomains are inclusive of many fundamental competencies in areas such as knowledge application, literature evaluation, collection/interpretation of evidence, implementation/monitoring of plans, and documentation. Certain CAPE outcome subdomains such as communication (3.6) and educator (3.2) were less likely to be coded as the primary focus but frequently occurred at the second or third level, emphasizing their importance.

While our review provides a useful glimpse at the distribution of CAPE outcomes in published literature, underlying reasons why certain subdomains are underrepresented cannot be specifically determined. Conceivably, assessment methods to capture APPE learning outcomes for some CAPE outcome subdomains (such as leadership) are more challenging or not yet developed, making them less likely to be reported in recent literature. Another consideration is that a general CAPE outcome subdomain such as problem solving (3.1) overlaps with patient-centered care (2.1), making it less likely to be separately identified. The focus of our review was reporting at the APPE level, so some of these outcomes may have been investigated and published at the introductory pharmacy practice experience level and would not have been included. Another important distinction is the evolving nature of the CAPE outcomes and ACPE accreditation standards, which we suspect may influence scholarly reports of student learning outcomes over time (Table 6). For example, 75% of the publications we reported involving the cultural sensitivity subdomain (3.5) occurred in 2008 or later. When reflecting on ACPE accreditation standards, those adopted in 2006 (effective in 2007) included stronger language for cultural competency within standards for curriculum and professional outcome expectations in comparison to the previous ACPE standards. Population-health management was also emphasized in the ACPE 2007 standards and all publications judged to include the related CAPE 2013 subdomain of provider (2.4) occurred in 2010 or later (Table 6). These occurrences may be coincidental, but schools and faculty members may (and should) be utilizing changes in ACPE standards as opportunities for investigation and dissemination within the academy. Notably, the CAPE subdomain of interprofessional collaboration (3.4) was minimally addressed in the findings through 2014, yet represents a significant focus for ACPE Standards 3 and 11 in the 2016 iteration.² To a similar degree, the CAPE outcomes of leadership (4.2) and innovation and entrepreneurship (4.3), reports of which are absent from our review, have essentially been incorporated directly into ACPE standards 2016 within Standard 4 (Personal and Professional Development).² These specific CAPE subdomains represent important areas for schools to explore and share findings through scholarly publications as encouraged in the ACPE 2016 guidance document.³

Equally important to the emphasis on CAPE outcomes addressed by APPEs in published literature is determining the level and quality of assessment applied to the desired learning outcomes and their impact, which was also a purpose of our review. Kirkpatrick’s hierarchy provides a framework to stratify assessment methods and impact, from surveys of student satisfaction at the lowest level (1), to student patient care interventions affecting patient care at the highest level (4b). Organizing the CAPE outcome subdomains by various KH levels provides perspective on the range of assessment methods used within and across subdomains, but the level and rigor of assessment methods used varies within each KH level. For example, within KH level 2 (learning), assessment methods include the use of nonvalidated and validated surveys. These assessment methods are useful for their intended purposes, but are not necessarily meant to be comparable, as they would not be presumed to have the same validity. And while it is generally desirable to have the highest possible impact with our interventions (a benefit to patients), even studies coded at this level could be viewed as having a “surrogate marker” of impact such as clinical interventions accepted vs demonstration of improved patient outcomes. So KH has utility in the organization of publications according to an increasing level of learning “impact,” but heterogeneity exists within each level. The validity and reliability of the assessments and results reported are outside the scope of our primary objective; however, Hoover and colleagues’ review described areas for general improvement in pharmacy education literature.⁹⁹ Continuously improving scholarly design and quality assessment within pharmacy education is highlighted, and resources are available for consideration when planning and executing experiential education.^100-104

The authors acknowledge limitations inherent to the Review. While we deployed a broad literature retrieval strategy across several search engines, supplemented the research through table of contents for relevant nonindexed journals, and conducted abstract reviews to identify articles meeting our prespecified criteria, relevant publications may not have been captured. Our objective was to characterize the extent to which published APPE literature addressed current CAPE outcomes, and our results provide a representative aggregate of work meeting our criteria. It is also feasible that our decisions regarding the coding of publications according to the CAPE 2013 outcomes and KH included degrees of subjectivity. We attempted to minimize arbitrary decisions through two independent reviews of each publication according to prespecified criteria, discussion of discrepancies, and obtaining consensus when necessary for resolution.

Our results provide a basis to reflect on published work of APPE education during a time of significant change in the pharmacy academy. The CAPE outcomes and ACPE standards have been revised three times since 1996, which was the starting point of our literature review. Each iteration has guided schools and faculty members in the contemporary delivery of professional programs and presented opportunities to experiment with curricula, subsequently sharing results through publication. The CAPE 2013 Outcomes are in keeping with a forward-thinking curriculum and have been embraced by the ACPE 2016 Standards, which will guide the academy’s focus for the next several years. This review is intended to provide a historic window of how published APPE education is reflected in the CAPE 2013 Outcomes, stimulate inquiry into the improvement of assessment quality, and determine new methods to address outcomes of emerging importance to the profession and society.

CONCLUSION

Faculty members and preceptors report pharmacy student contributions and learning for the majority of the current CAPE 2013 outcomes. Outcomes related to caregiver (2.1), learner (1.1), self-aware (4.1), communicator (3.6), and educator (3.2) are frequently addressed in APPE reports, while achievement of newer outcomes such as leadership (4.2) and innovation and entrepreneurship (4.3) are notably absent. The level of assessment according to Kirkpatrick’s Hierarchy shows a wide distribution, from qualitative survey methods to change in behavior and improvement in the health/well-being of patients. Scholarly design and quality of assessment methods used in APPEs are areas for continued improvement.

Appendix 1. Kirkpatrick’s Hierarchy (KH) Levels and Selected Examples of Representative Assessment Evidence