Determination of predictors impacting performance on the third-year Pharmacy Curriculum Outcomes Assessment at a historically Black college of pharmacy

Abstract

Introduction:

The Pharmacy Curriculum Outcomes Assessment (PCOA) is a standardized exam developed by the National Association of Boards of Pharmacy (NABP) in 2008 to measure the curriculum in relation to student progress. The purpose of the study was to determine the impact of pre-admissions and pharmacy school variables on third-year student PCOA performance at a Historically Black College or University (HBCU) College of Pharmacy.

Methods:

A retrospective analysis was conducted using data from three cohorts of students who took the PCOA in their third professional year from 2015 to 2017. An independent samples t-test, correlation analysis, and multivariate linear regression were conducted to determine the relationship between student characteristics and the PCOA score.

Results:

The mean PCOA scaled score for the third-year pharmacy students was 349.6 ± 46.20 while the mean Pharmacy College Admission Test (PCAT) percentile was 62.7 ± 14.5. Most students (67%) self-identified as Black and the majority (54.9%) were female. The PCOA scores were correlated with the PCAT percentile (P < .001) and the cumulative grade point average (GPA) through the fall semester of the third professional year (P < .001). After adjusting for other factors, the cumulative GPA through the fall semester of the third professional year (P < .001) and PCAT percentiles (P < .001) remained predictive of students PCOA scores.

Conclusions:

The cumulative GPA through the third-year fall semester and PCAT percentiles are important factors in helping to predict PCOA scores among third year pharmacy students at a HBCU.

Introduction

The Pharmacy Curriculum Outcomes Assessment (PCOA) is a standardized exam developed by the National Association of Boards of Pharmacy (NABP) in 2008. As designed, the PCOA was developed to measure the curriculum in relation to student progess.¹ Prior to creation of an exam outline and formal implementation, a NABP survey was given periodically to all pharmacy colleges and schools to assess coverage in the content areas.² Currently, the PCOA is comprised of 225 questions and scaled scores can range from 0 to 700.³ The content areas include basic biomedical sciences (10%), pharmaceutical sciences (33%), social/behavioral/administrative pharmacy sciences (22%), and clinical sciences (35%).³ In 2016, the Accreditation Council for Pharmacy Education (ACPE) made the PCOA a requirement for all schools and colleges of pharmacy.⁴ From 2016 onward, colleges of pharmacy have had a validated and objective way of assessing their curricula alongside peer institutions in accordance with ACPE Standard 24.2 and as categorized as foundational knowledge under Standard 1 in Appendix 3.^3.4

At Howard University College of Pharmacy, the PCOA is used as the summative evaluation preceding progression to advanced pharmacy practice experiences (APPEs). Specifically, it is used as a high-stakes examination for third-year pharmacy students with the intention of assessing an individual student–s content knowledge prior to APPE matriculation and identifying gaps in the didactic curriculum. The PCOA is administered in January of the spring semester, and a failing score at our college is defined as a score that is lower than one standard deviation below the national mean scaled score set by NABP. A major point of contention with regards to the scoring of the PCOA is that there is no set minimum competency level or threshold for passing.³ However, as recently as August 2020, NABP convened to recommend a passing scaled score of 338 on PCOA to correlate with the Foreign Pharmacy Graduate Equivalency Examination passing score of 284.³ At the time of this publication, this recommendation has not yet been adopted. Furthermore, NABP has not set guidance on PCOA adoption from the perspective of mandating it as a low-stakes or high-stakes offering.³ As a consequence, schools have been given the autonomy to set minimum passing scores and offering types (high-stakes vs. low-stakes) based upon their curricular needs.¹ This position has created a dilemma for colleges because student motivation for high performance on the PCOA may vary based upon the minimum passing scores as well as low-stakes or high-stakes status assigned to exam by an individual college of pharmacy. Given this potential for variation, it may be important for individual pharmacy programs to identify predictors of PCOA performance as these factors may vary from institution to institution.

Several studies have evaluated predictors of academic performance in pharmacy schools.^5–16 Factors such as pre-admission factors as well as sociodemographic variables have been identified. However, some studies have evaluated predictors of performance on the PCOA exam.^17–24 Of them, none have assessed predictors among a largely underrepresented minorities (URMs) population of students. Across nearly all the studies, grade point average (GPA) has been found to be a significant predictor of PCOA performance.^17–21 Other factors such as critical thinking and Pharmacy College Admission Test (PCAT) scores have also been reported as predictive factors of PCOA performance.^{17,19,22–25} While actual data on PCOA performance data for URM pharmacy students are lacking, there is some evidence of low performance of URMs on other standardized tests.^26–32 As such, more data are needed to evaluate PCOA performance among URMs. In order to fill all the prior mentioned knowledge gaps, the purpose of this study was to conduct a retrospective cohort analysis of third-year pharmacy students to determine predictors of PCOA performance at a historically Black college of pharmacy.

Methods

Study design

We employed a retrospective cohort study design to review academic records of third-year pharmacy students who took the PCOA from 2015 to 2017 to determine predictors of PCOA performance. Students were excluded from the study if they withdrew or were dismissed from the doctor of pharmacy (PharmD) program prior to taking the PCOA. The study was reviewed and approved by the Howard University Institutional Review Board.

Study variables

The dependent variable in the study was the scaled PCOA examination score during the third professional year. Data for the PCOA scaled scores were abstracted from PCOA scores reported to the college by NABP. The included admissions characteristics were age, gender, race/ethnicity, having a degree prior to matriculating into the college of pharmacy program, and the cumulative GPA prior to entering into the college of pharmacy. The pharmacy school characteristics included whether a student was placed on probation due to their academic performance during the first semester in pharmacy school, and the cumulative GPA through the fall semester of the student’s third year in the college of pharmacy.

Statistical analyses

Means and standard deviations were used in describing the baseline characteristics for continuous variables while frequencies were employed in describing categorical variables. Race/ethnicity was dichotomized into Black/non-Black for the purpose of conducting inferential statistics. Non-Blacks included Asian, White, Hispanic, and others. The independent samples t-test was utilized to assess if there were significant differences in the mean scaled PCOA scores by gender, race/ethnicity, probation after the first semester, and degree status prior to entering into the college of pharmacy. Pearson’s correlation was used to assess if age, PCAT percentile, cumulative GPA at admission, and cumulative GPA through the third year fall semester had a significant association with the third year PCOA score. Multivariate linear regression was utilized to determine which characteristics were independent predictors of the third-year pharmacy student PCOA score after adjusting for other factors. Students with missing data were excluded from bivariate and multivariate analyses. SPSS, version 23 (IBM Corp.) was used to analyze the data. All analyses were conducted at an alpha of .05.

Results

Admissions characteristics

One hundred eighty-two students took the PCOA during the study period. The mean age of the students was 26.0 ± 5.7 years as summarized in Table 1. Most students (67%) self-identified as Black, and the majority (54.9%) were female. In addition, most students (74.7%) had obtained some type of degree prior to admission. The mean PCAT percentile was 62.7 ± 14.5, and the cumulative GPA at admission was 3.18 ± 0.39.

Table 1.

Demographic and academic characteristics of pharmacy students.

Characteristic	Value
Gender, n (%)
Male	82 (45.1)
Female	100 (54.9)
Race/ethnicity, n (%)
Black	122 (67)
Asian	42 (23.1)
White	11 (6)
Hispanic	5 (2.7)
Other	2 (1.1)
Type of degree, n (%)
No prior degree	46 (25.3)
Associates	10 (5.5)
Bachelors	121 (66.5)
Masters or higher	5 (2.7)
Year of entry, n (%)
2012	70 (38.5)
2013	56 (30.8)
2014	56 (30.8)
Probation during first semester, n (%)
Yes	17 (9.3)
No	165 (90.7)
Age (years), mean ± SD	26.0 ± 5.7
PCAT percentile, mean ± SD	62.7 ± 14.5
Undergraduate GPA, mean ± SD	3.18 ± 0.39
Cumulative GPA through five semesters,a mean ± SD	3.20 ± 0.35
P2 PCOA scaled score, mean ± SD	307.94 ± 45.41
P3 PCOA scaled score, mean ± SD	349.64 ± 46.20

GPA = grade point average; PCAT = Pharmacy College Admissions Test; PCOA = Pharmacy Curriculum Outcomes Assessment; P2 = second year; P3 = third year.

^aIncludes data from the first and second year of pharmacy school and the fall semester of third year.

PCOA scores and bivariate analysis

The mean scaled PCOA score in the third year of pharmacy school was 349.64 ± 46.20. The scaled PCOA score for students not placed on probation after the first semester (mean 352.6 ± 45.3) was significantly higher (P < .001) in comparison to the scaled score for students placed on probation (mean 320.9 ± 46.1). There was no significant difference in the mean scaled PCOA scores by gender, race/ethnicity, or degree status prior to entering into the college of pharmacy. The scaled PCOA score during the third year was significantly correlated with the PCAT percentile (r = 0.432; P < .001) and the cumulative GPA through the third-year fall semester (r = 0.534; P < .001). As seen in Figure 1, the scaled PCOA scores ranged from 320.59 among students in the bottom quartile for cumulative GPA to 385.44 for students in the top quartile for cumulative GPA. Figure 2 shows the PCOA scaled score ranged from 332.56 among students in the bottom quartile for PCAT percentile to 378.52 for students in the top quartile for PCAT percentile.

Fig.1.

Mean third-year scaled PCOA score according to cumulative third year GPA quartile.

GPA = grade point average; PCOA = Pharmacy Curriculum Outcomes Assessment.

Fig.2.

Mean third-year scaled PCOA score among according to PCAT percentile quartile.

PCAT = Pharmacy College Admissions Test; PCOA = Pharmacy Curriculum Outcomes Assessment.

Predictors of third-year PCOA scores

After adjusting for other variables, the cumulative GPA through the third-year fall semester was a significant predictor of third-year scaled PCOA scores with a beta coefficient of 70.80 (P < .001) as summarized in Table 2. In addition, the PCAT percentiles (P < .001) were predictive of students– scaled PCOA scores with a beta coefficient of 1.20. Accordingly, each one unit increase in the cumulative GPA during the third-year fall semester was predictive of an increase of 71 points in the third-year PCOA scaled score, and each one unit increase in the PCAT percentile was predictive of an increase of 1.20 points in the third-year PCOA scaled score.

Table 2.

Multivariate linear regression model of predictors of PCOA scaled scores at an HBCU.

Variable	Beta coefficient	Standard error	Standardized coefficient	P value
Constant	81.19	35.18		.02
Probation during first semester	6.16	10.23	0.04	.55
Age	0.390	0.50	0.05	.44
PCAT percentile	1.20	0.20	0.38	< .001
Cumulative undergraduate GPA	−15.77	8.24	−0.13	.06
Cumulative GPA through five semestersa	70.80	9.26	0.53	< .001
At least a bachelor’s degree	−2.69	6.34	−0.03	.67
Black race	8.19	5.96	0.08	.17
Male gender	4.37	5.55	0.05	.43

R² = 0.433

GPA = grade point average; HBCU = Historically Black College University; PCAT = Pharmacy College Admissions Test; PCOA = Pharmacy Curriculum Outcomes Assessment. Gael Summary F 30.92, p <0.001

^aIncludes data from the first and second year of pharmacy school and the fall semester of third year.

Discussion

This study sought to evaluate the predictors of third-year pharmacy student PCOA scores. To our knowledge, this is the first study to examine this issue in a largely URM student population. As observed in other studies,^{17,19,22–24} our study found similar associations and predictive effects of PCAT and GPA as academic factors on PCOA score. In our study, no sociodemographic factors at pre-admission were found to be significant predictors of PCOA scores. This is notable as other studies have not shown similar findings.²³

Third-year cumulative GPA was the strongest predictor of PCOA score. These findings are in agreement with the majority of other studies.^{17,19,22–24} A meta-analysis by Daugherty and Malcolm²⁴ utilized data from eight studies to examine the relationship between cumulative GPA and PCOA score during the third year and demonstrated that there was a statistically significant association between PCOA scores and GPA during the third year of pharmacy school. Overall, the third year GPA explained 14% to 48% of the variability in PCOA scores.²⁴ Gillete et al¹⁷ found that a one unit increase in cumulative GPA led to a 26.597 increase in PCOA score (P < .01). In their adjusted analysis, Giuliano et al¹⁹ also found a higher effect with a one unit increase in GPA leading to a 49.950 increase in PCOA score on average. Our study found a much higher impact of the cumulative fall semester third-year GPA with a one unit increase in GPA contributing to a 70.80 increase in PCOA score. Collectively, the evidence points to GPA as an important predictor, further reinforcing the fact that high performance on pharmacy curricula will be reflected in the PCOA.

PCAT scores were also found predictive of the PCOA scaled scores with an increase of 1.20 points in PCOA score observed for one unit increase in PCAT scores. We compared our findings with Giuliano et al¹⁹ who studied PCOA performance in the second year and Gillette et al¹⁷ who studied PCOA performance in the first year through third year. Gillette et al¹⁷ and Giuliano et al¹⁹ also found that PCAT scores were predictive of a relatively small change in PCOA scaled scores with each one unit increase in PCAT score corresponding to increases of 0.389 and 0.634 points, respectively on the PCOA scaled score. Garavalia et al²² provided insights on the observed relationship between PCAT and PCOA. In their analysis, they hypothesized that higher PCOA scores could generally be associated with pre-requisite learning.²² Further explained, students with higher incoming PCAT scores would have more comprehensive knowledge that would be critical for good performance on the PCOA.²² Alternatively, it could also be argued that high performance on the PCAT could be a proxy for better performance on standardized tests like the PCOA. Nevertheless, across three studies the effect of PCAT on PCOA was statistically significant; however, its impact was not observed to be as impactful as GPA based upon the small effect sizes. This finding could also support the argument on the reduced importance of the PCAT at admission with 35 out of 142 (25%) accredited PharmD programs no longer requiring PCAT as a preadmission factor.³³

Our study did not find any association between PCOA and socio-demographic factors. Specifically, there was no predictive effect of gender or race. The findings on gender were contrary to those found by McDonough et al.²³ In their study, they found that male students had higher PCOA scores in all the areas including the total scaled scores except Area 3 (social, behavioral and administrative sciences section) compared to female students.²³ While general gender differences in standardized test performance have been reported by Reardon et al,³⁴ their findings have not been replicated in other studies.^17,19 To our knowledge our study was the first to examine the predictors of PCOA within a largely URM pharmacy students. As a HBCU, Black students comprise nearly 70% of our student population. Our findings showed no statistically significant relationship between being Black and PCOA score. These findings were highly encouraging given our institution’s commitment to sustaining URM students on the path to pharmacy careers. These findings also agree with those observed by McDonough et al²³ who found no relationship between race and PCOA scores in their bivariable analysis. Specifically, in the sample of 159 examinees comprising non-Hispanic Whites (71.7%), African Americans/Blacks (15.1%), Asians (10.7%), Alaskans (1.3%), and unknown (1.3%), there was no statistically significant racial group differences across all the PCOA content areas.²³

While not directly comparable, our study and McDonough et al²³ found no difference within the race/ethnicities comprising URM unlike previous studies showing lower performance on standardized tests in URM vs. non-URM counterparts.^26–32 Beyond the fact that race may not itself be a predictor, it could also be reasoned that the provision of more academically supportive programming at our college may have led to these findings. Over the past few years, our college through grant funds from Health Resources and Services Administration has been able to provide quality and consistent academic enrichment throughout our students’ matriculation. These programs are equally beneficial to URM and non-URM students. On this basis our findings are unique to our college. Additional planned analysis stratified by race is scheduled for our new curriculum to examine how PCOA performance is impacted by the interaction effect of race and participation in PCOA support programming which now can be quantified as a predictive factor. This support includes a PCOA practice exam from Exam Master (Exam Master Corporation) and incentivized points for completion of more practice exams. Similar proposed studies are also encouraged to continue to examine the effect of race on PCOA performance.

Of note, other trends unique to our study were observed with regards to inconsistencies in performance areas comparing the PCOA scores for the third-year pharmacy students to the national average for all three class cohorts. We hypothesize that the consistency of lower performance was not indicative of student related factors but a reflection of a needed upgrade in our curriculum delivery to accommodate the new generation of learners and the need for earlier introduction of clinical exposure. As a result of these findings, the college of pharmacy changed to a new curriculum in 2018. While not reported in this study, early analysis has noted improved PCOA performance for the second-year pharmacy students in the curriculum; we await more confirmatory data when they take the exam as a high-stakes offering in their third year in January 2021.

Speculation on the potential for biased third-year PCOA results due to prior PCOA exposure of our students in the second year has also been considered; however, out of caution in the differences in the low-stake vs. high stake status in the two offerings, we chose not to include adjustments for second-year PCOA in our model. A study by Waskiewicz³⁵ evaluated whether students– test-taking motivation efforts on the PCOA was a true reflection of their knowledge if the test was low-stakes. In their commentary, they suggested that high-stakes exam should correlate with improved test scores and therefore are a more reliable way of assessing test-taker ability.³⁵ In the future, studies could also further examine the impact of offering type (low-stakes vs. high-stakes) on PCOA performance.

Our study also had a few limitations. Foremost is that the data only analyzed 2015–2017 cohorts and did not include the more recent cohorts from 2018–2020. In some cases, demographic information on admission was limited and possibly missing, therefore there may be residual confounding effects. Additional sources of bias due to confounding is the lack of consideration of other factors such as level of preparation or critical thinking addressed in other studies. Lastly, the external validity of our study may only be limited to other HBCUs. In spite of these limitations this is the first study to provide evidence on the predictive factors of PCOA score and will add to the existing body of evidence.

Conclusions

Our study found that PCAT and GPA were academic predictive factors of PCOA score similar to studies performed at non-HBCU colleges of pharmacy. Race and other socio-demographic factors were not significant predictors of performance on the PCOA. This study showed that there are multiple predictors that can affect performance on the PCOA that may vary among different colleges of pharmacy. Once NABP finalizes adoption of the minimum PCOA passing score, further studies are recommended on predictors as they may change.