Correlation of the Pharmacy Curriculum Outcomes Assessment and Selected Pre-pharmacy and Pharmacy Performance Variables

Abstract

Objective. To identify the correlation between the Pharmacy Curriculum Outcomes Assessment (PCOA) and pre-pharmacy and pharmacy performance variables.

Methods. Four years of PCOA data (2012-2015) were analyzed for students taking the assessment in the third professional year (P3). Using the Pearson correlation coefficient, data was correlated to a series of performance variables: pre-pharmacy grade point average (GPA), pre-pharmacy science GPA (pre-pharmacy science courses only), Pharmacy College Admission Test (PCAT) composite score, P3 pre-advanced pharmacy practice experience (APPE) GPA and the North American Pharmacist Licensure Examination (NAPLEX). Scores that were correlated at r ≥ 0.30 were added to a multivariate linear regression model to compare their unique contributions.

Results. There was a moderate correlation between PCOA and PCAT (r=0.60), P3 pre-APPE GPA (r=0.60) and the NAPLEX (r=0.64). The multivariate regression analysis explained 60% of the variance of the total PCOA score, with PCAT making the largest unique contribution.

Conclusion. The PCOA was moderately to strongly correlated to the pre-APPE GPA, thus providing an acceptable assessment of student learning. The PCOA was also moderately to strongly correlated to the NAPLEX, making it a useful tool in predicting performance and identifying students in need of further remediation before the NAPLEX.

INTRODUCTION

The Pharmacy Curriculum Outcomes Assessment (PCOA) is a standardized, validated examination created by the National Association of Boards of Pharmacy (NABP) to assess individual student performance in the curricula.¹ The assessment can also be used for pharmacy programs to provide formative analyses, as well as monitor within-program results longitudinally and benchmark results with other programs nationally. The questions are grouped into four domains: basic biomedical sciences; pharmaceutical sciences; social, behavioral and administrative sciences; and clinical sciences. These four domains are further divided into 28 subtopics. In 2015, the PCOA was used by 55 (increased from 38 programs in 2014) pharmacy schools and colleges. For 2015, the Cronbach’s alpha was 0.92, supporting a high degree of reliability.¹ In 2016, the PCOA examination became a required assessment for pharmacy colleges and schools in the U.S.

Currently, the American Association of Colleges of Pharmacy (AACP) and the Accreditation Council for Pharmacy Education (ACPE) consider the PCOA examination to be a validated method to assess the foundational science knowledge of student pharmacists. Standard 24 of the 2016 ACPE Standards require the use of the PCOA as an assessment of knowledge of the essential content areas identified in Appendix 1 and recommend administering the assessment to students nearing the completion of the didactic curriculum.²

Previous studies have explored the relationship between PCOA total score and pre-pharmacy characteristics, PharmD academic performance, and NAPLEX score.^3-6 Scott and colleagues evaluated the correlation of PCOA total scale scores to grade point average (GPA) in P1, P2, and P3 students over 3 years.³ For the P3 class, there was a strong correlation between current GPA and PCOA total scaled score initially (R=0.71), followed by weaker correlations over the next 2 years (R=0.46 and 0.26, respectively). The institution also used the assessment to target perceived weaker areas of the curriculum by identifying subtopics that students scored 10% or below the national average percent correct. Two studies focused on predictors of student performance on the PCOA. Giuliano and colleagues evaluated two years of data for P2 students taking the PCOA by examining predictors of performance, including learning styles and pre-pharmacy characteristics.⁴ GPA after the P1 year and PCAT-reading were associated with higher PCOA scores. Gillette and colleagues studied specific admissions criteria as predictors of PCOA performance across the first three years of the professional program.⁵ PCAT, the Health Sciences Reasoning Test and cumulative pharmacy GPA were significant predictors of higher PCOA total scores. Naughton and Friesner⁶ examined the correlation between P3 PCOA scores and NAPLEX performance. Students with higher PCOA scores were more likely to have higher NAPLEX scores than students with lower PCOA scores. None of these studies evaluated pre-pharmacy characteristics, PharmD academic performance, and NAPLEX score with PCOA score concurrently in the same student cohort.

We sought to examine the relationship between PCOA and a pre-pharmacy admission criteria (PCAT), academic performance in the curriculum as measured by cumulative P3 GPA, and a major postgraduate assessment (NAPLEX). The primary objective of this evaluation was to identify relationships between PCOA and pre-pharmacy variables and markers of student performance.

METHODS

During 2012-2015, the PCOA was administered annually to P3 students in the winter prior to starting advanced pharmacy practice experiences (APPE). The examination was used as part of a comprehensive assessment plan for student achievement. Prior to the assessment, a faculty member explained the rationale for and importance of the assessment to students via a formal presentation. The presentation included a detailed description of the examination format, the different domains being assessed and sample questions from each domain. The faculty member explained how the data would be used for the student and the college to identify curricular “gaps” and areas where curricular enhancements could be made. The students were encouraged to “do their best”; however, no incentives were provided for high achievers. Likewise, no remediation or other penalty was used for students not performing well. Students were advised they each would have an individual meeting with a faculty member to review results. In addition to student advice and guidance, a goal of the faculty meeting was to increase student motivation and accountability.

Performance of the P3 cohort on the PCOA was presented to the students during class time. Students then met individually with a faculty member for 15 minutes; generally, this was a pharmacy practice faculty member. Other interested faculty were typically paired with a practice faculty member for the initial year of consultation. All faculty members also met with the associate dean for professional education and assessment prior to meeting with students to review the goals of the meeting. The meeting consisted of the faculty member reviewing the results with the student, how the student compared to his/her peers within the college and nationally, and how to leverage his/her advanced practice rotations to improve areas of weakness. When the college first used the faculty-student meetings, faculty members provided the students with resources specific to areas of weakness. However, as this process was reassessed in subsequent years, it became clear that the resources were not useful. Useful resources were difficult to find due to the lack of detailed information provided around a student’s specific weakness in a topic area.

Students’ total PCOA content area scores (Basic Biomedical Sciences, Pharmaceutical Sciences, Social/Behavioral/Administrative Sciences, and Clinical Services) were assessed for their correlation with the students’ pre-pharmacy GPA, pre-pharmacy science GPA (pre-pharmacy science courses only), PCAT composite score, P3 pre-APPE GPA (cumulative GPA prior to APPE), and NAPLEX score using Pearson correlation coefficient. Multivariate linear regression analysis with backward elimination was used to examine the correlation between each score and compare their unique contributions to the total PCOA score. Backward elimination was chosen to ensure variables involved in suppressor effects were considered in the model.⁷ The in p value was set at .1 and the out p value (p criteria for removing a variable) was set at .2. Variables identified as significant from backward elimination were then used to conduct additional multivariate linear regression to identify their contribution toward each of the PCOA content area scores. All statistical analyses were computed using SPSS Statistics (version 22, IBM Corp, Armonk, NY) and p values <.05 were considered significant.

RESULTS

The PCOA examination was administered to 384 P3 students from 2012-2015. Mean scaled scores by domain and overall are presented in Figure 1. Preliminary analyses showed no violation of normality, linearity and homoscedasticity for the data.

Figure 1.

Mean Total PCOA Scale Scores and by Domain. Hashed line = Clinical Sciences, X = Social/Behavioral/Admin Pharm Sciences, Circle = Total Score, Triangle = Pharmaceutical Sciences, Square = Basic Biomedical Sciences

Using PCOA data from 2012-2015, there was a significantly positive correlation with P3 pre-APPE GPA, NAPLEX score and PCAT score and students’ total and content area PCOA scores (Table 1). There was a small positive correlation found between PP-GPA and students’ total and content area PCOA score. There was a small positive correlation between PP-Science GPA and students PCOA total score and some of its content area scores (Table 1).

Table 1.

Correlation of Students’ PCOA Total and Domain Scores with their Grades/Assessments (2012-2015)^a

The final multivariate regression model included P3 pre-APPE GPA, NAPLEX score and PCAT scores from 2012 to 2015. The regression model explained 60% of the variance of students’ total PCOA score, with their PCAT score making the largest unique contribution (β=0.38, Table 2). The regression models of each of PCOA’s content area scores showed that students’ PCAT score also made the largest unique contribution toward the Basic Biomedical Sciences (β=0.32) and Social/Behavioral/Administrative Sciences (β=0.33) scores. In these models, the students’ NAPLEX score did not significantly contribute. However, the students’ NAPLEX score made the largest unique contribution toward the Pharmaceutical Sciences (β=0.36) and Clinical Services (β=0.33) scores (Table 2).

Table 2.

Multivariate Linear Regression Models of Students’ Total PCOA Scores and Their Individual Components with their Grades/Assessments (2012-2015)^a

DISCUSSION

PCOA scaled scores (total and each of the four domains) were relatively stable across the four years of administration. Clinical sciences scaled score decreased somewhat over this period. Consistent with findings from Scott and colleagues³ and Gillette and colleagues,⁵ PCOA scores were moderately to strongly correlated with P3 pre-APPE GPA (r = 0.60). In addition, our data showed correlation between PCOA scores and PCAT and NAPLEX performance. Similar correlation between PCOA and NAPLEX was found by Naughton.⁶ The correlation between PCOA scores and P3 pre-APPE GPA and NAPLEX scores is easily explained by the curricular content overlap between the college curriculum, PCOA content blueprint and NAPLEX content blueprint. More puzzling is the correlation between PCOA scores and PCAT scores; although, a similar finding was reported by Gillette and colleagues.⁵ This may be due to a strength in taking standardized tests or a strong foundation in pre-pharmacy courses that prepares students to be successful in the PharmD curriculum. Also, it could be that students who learn and retain knowledge well are able to demonstrate this across a variety of assessment types. Although the PCOA was not developed as a predictor of NAPLEX performance, we found NAPLEX had the largest contribution in correlation with the PCOA Pharmaceutical Sciences domain and the PCOA Clinical Sciences domain. Considering the NAPLEX focuses on medications and pharmacotherapy, the correlation found with the PCOA Pharmaceutical Sciences and Clinical Sciences domain is not surprising.

Notably, there are some limitations. Although this study was conducted over four years, it was at a single institution, which could limit the generalizability of the results. The blueprint for the PCOA was developed based on curricula at pharmacy colleges and schools in the US, and not our specific institution. As with any retrospective study based on student record review, only a relationship can be identified and cause/effect cannot be made.

Several benefits were identified with administering the PCOA examinations. It was useful for curricular assessment to help identify areas that needed strengthening if students across multiple years consistently performed poorly in certain areas. We were hopeful the assessment could be used as a type of formative assessment, allowing students to self-evaluate areas of strength and areas to improve upon prior to and during APPE. While the PCOA was viewed to be useful, it was realistically regarded as just one component of an overall evaluation plan. It is limited to evaluating cognitive attributes while skills and attitudes/values are assessed via other means (eg, Skills Lab, OSCE).

In addition to other measures of curricular effectiveness, information gained from student performance on the PCOA has helped inform the curricular quality improvement process. Student performance over the four years consistently demonstrated a weakness in the biomedical science domain. Historically, the pre-pharmacy curriculum required for our college was two years, so the relative weakness in this area could be related to less foundational science coursework than other programs using PCOA. Less consistently, the subtopics of immunology and pharmacogenomics have been identified as areas of weakness and this is being addressed in an upcoming curricular revision.

After four years of administering the PCOA, the faculty decided to administer the PCOA as a “high stakes” assessment in 2016. Faculty members felt confident the PCOA blueprint overlapped with our curriculum; and given the PCOA performance was moderately correlated with GPA, faculty members decided to transition to high stakes. Faculty members believe this will motivate students to perform their best and provide a better benchmark of true performance. Threshold for remediation was determined based on prior years’ performance. Any of the following would require remediation prior to APPE: GPA <2.75 and overall PCOA scale score 1 standard deviation (SD) below University of Cincinnati (UC) mean, overall PCOA scale score 2 SD below UC mean, or PCOA Clinical domain 2 SD below UC mean.

CONCLUSION

Student performance on the PCOA moderately correlated with P3 pre-APPE GPA and NAPLEX. Correlation with P3 pre-APPE GPA provides relative confidence that the PCOA is an acceptable assessment of student learning at our college. Correlation between PCOA and NAPLEX, particularly the Pharmaceutical Sciences and Clinical Sciences, is useful in advising students on leveraging their APPEs to fill in any gaps in content and/or application of knowledge. The PCOA may be a useful tool in predicting performance and identifying students who may benefit from additional studying or completing review sessions prior to NAPLEX. After several years of administration, this college felt the PCOA was a useful assessment of student learning and faculty were comfortable in transitioning to a high stakes approach. Future areas of study include the utility of PCOA as a high stakes pre-APPE readiness assessment, the effectiveness of remediation processes and the effectiveness of incentives for optimal performance. Finally, institutions that make curricular changes based partly on PCOA performance may be able to evaluate the effectiveness of those changes using future PCOA scores.