Examination of Factors Relating to Student Performance on the Pharmacy Curriculum Outcomes Assessment

Abstract

Objective. To examine relationships between students’ demographic and academic performance factors and their scores on the Pharmacy Curriculum Outcomes Assessment (PCOA).

Methods. Students’ PCOA scores and demographics (eg, age, race/ethnicity, sex), preadmission data [eg, cumulative and science grade point average (GPA), Pharmacy College Admissions Test (PCAT)], and academic performance variables (eg, pharmacy GPA, academic standing) were analyzed for one class of third-year pharmacy students (N=159). Independent t-tests and Analysis of Variance (ANOVA) were used to compare scores by demographic variables. Pearson’s r correlations were used to assess relationships between PCOA scores and age, PCAT scores, and GPA. Stepwise linear regression was conducted to determine the predictive ability of variables with significant correlations to PCOA performance.

Results. Significant correlations were found between sex and PCOA scores with males scoring higher than females. Significant correlations with PCOA scores were also found for PCAT scores, pre-pharmacy science GPA, and pharmacy didactic GPA. Significant differences were found by academic standing, where students in academic difficulty who were allowed to proceed without repeating curricular content scored significantly lower on the PCOA than those who did not experience academic difficulty. Conversely, there were no statistical differences between those who repeated courses and those who never experienced academic difficulty. PCOA performance predictors in the final regression model included PCAT composite score, pharmacy GPA and sex.

Conclusion. New findings included differences in PCOA scores by sex and by academic standing, a variable not previously explored in published studies. Findings have implications for remediation decisions in pharmacy curricula.

INTRODUCTION

The Pharmacy Curricular Outcomes Assessment (PCOA) is an examination developed and administered by the National Association of Boards of Pharmacy (NABP). It was designed to provide an external measure of student performance in United States pharmacy curricula and was operationally launched in 2009. This 3-hour examination includes 225 multiple-choice items, 200 of which are operational and thus calculated into the scores. The content spans four domains: basic biomedical sciences (16%), pharmaceutical sciences (30%), social/behavioral/administrative sciences (22%); and clinical sciences (32%). Although administration of the PCOA was optional prior to 2015, it had been administered in 65 different pharmacy programs,¹ which constitutes roughly half of the schools and colleges in the nation.

The Accreditation Council for Pharmacy Education (ACPE) mandated the use of the PCOA in their most recent accreditation standards revision, effective in 2016, requiring all pharmacy programs to administer the examination to students nearing the end of the didactic portion of the curriculum.² A 2016 review of PCOA studies focusing on implementation strategies, utilities, score interpretation, and limitations of the examination, noted that evidence of predictive value is limited and called for more studies to be conducted to determine the utility of PCOA scores.³ Given that the PCOA requirement is quite recent, and that only about half the programs in the country used the test before the ACPE mandate, exploration into the best use of this examination, including other factors that may be associated with student performance on the PCOA, is in the interest of all programs. It is important, at minimum, to understand how the results of the PCOA may best be used to improve the program and support student success. Further, it is important for pharmacy programs to contribute, and thereby expand, validity evidence relating to the use of this assessment. A literature review revealed that academic standing in the pharmacy program and undergraduate performance in science courses are two variables that have not been previously assessed in PCOA studies. Examining how these factors, in addition to others previously studied, may relate to PCOA performance contributes uniquely to the literature. To that end, the purpose of this study is to examine relationships between students’ demographic, pre-pharmacy and pharmacy academic data, academic standing in the program, and performance on the PCOA.

METHODS

Third-year student pharmacists in the Class of 2018 (N=159) sat for the PCOA on the University of Tennessee College of Pharmacy’s three campuses (Memphis, Nashville, Knoxville) in November 2016. This Doctor of Pharmacy program is completed over 4 years, which provides five semesters (2.5 years) of didactic instruction and 1.5 years of experiential instruction. Thus, at the time these students sat for the PCOA, they had already completed all the required didactic coursework and would begin experiential rotations the following semester. Students were encouraged to take the practice examination beforehand, come well rested, and take the examination seriously, but there were no stakes or consequences associated with level of performance. They were not provided any formalized preparation beyond the opportunity to take the practice examination. The PCOA was administered on the same day and proctored at the college’s three campuses by test administrators contracted by the NABP. Scores were reported to the college in early January 2017. The Institutional Review Board at the University of Tennessee Health Science Center reviewed the study application and determined that it qualified for not human subjects research (NHSR) status.

Data Collection and Analyses

Descriptive data for PCOA scores, student demographics, and academic performance measures, both in pre-pharmacy and in the pharmacy program, were collected and summarized. Demographic variables included age, sex, race/ethnicity, and campus assignment (either Memphis, Knoxville or Nashville). Pre-pharmacy academic performance measures included Pharmacy College Admissions Test (PCAT) Composite score as well as subscores (ie, Biology, Chemistry, Quantitative, Reading, Verbal), undergraduate cumulative GPA upon application to pharmacy school, and undergraduate overall science GPA upon application to pharmacy school. Academic performance in the pharmacy program included cumulative GPA at the end of the didactic portion of the curriculum (P3 fall semester). Academic standing was also noted using the college’s criteria for progression, which requires a student who makes two grades of a “D” or a “D” and “F” at any time to repeat a course, semester or year. All grades of an “F” must be repeated. Students were categorized into three groups. Group 1 included students with no academic issues that affected progression on time. This group included students with a single “D” who were placed on academic probation. Group 2 included students with poor academic performance who were allowed to progress on time without repeating any academic content. This group consisted of students who made two grades of “D” in non-consecutive semesters. Group 3 included students required to retake curricular content due to poor performance. Depending on the specific circumstances, these students repeated all courses in either a semester or year.

Categorical data were summarized by frequencies and percentages, and continuous data were summarized using means and standard deviations. In addition to the summary statistics, four types of analyses were conducted. Independent t-tests were used to compare PCOA scores by sex. Analysis of Variance (ANOVA) tests were performed to compare PCOA scores by race/ethnicity, by campus assignment, and by academic status. Pearson’s r correlation statistics were calculated to examine relationships between students’ PCOA scores and age, PCAT scores (composite and all subscores), undergraduate cumulative and science GPAs upon application, and pharmacy school cumulative GPA at the end of five semesters. Lastly, forward stepwise linear regression was used to further examine the significance of these variables relating to PCOA performance. Preliminary analyses were conducted to ensure that the data met all assumptions of multiple linear regression. The a priori level of statistical significance was established as ≤.05. All analyses were conducted using IBM SPSS Statistics, Version 24 (SPSS Inc., Chicago, IL).

RESULTS

A total of 159 P3 students in the Class of 2018 took the PCOA. Table 1 presents the summary of PCOA scores. Student performance across the four domains was similar with the lowest scores obtained in Area 1: Biomedical Sciences, and the highest scores in Area 4: Clinical Sciences. Overall, 35 students (22%) scored at or above the 75^th percentile, and 27 students (17%) scored below the 25^th percentile in reference to a national sample of P3s who tested from 2012 to 2016.

Table 1.

PCOA Scaled and Percentile Scores Overall and by Domain

The mean and standard deviation (SD) age of students was 26 years (4.0) with a range of 22-52 years. No significant correlations were found between age of examinees and their total PCOA scores or any of the four PCOA subscores.

There were 96 females (60%) and 63 males (40%). Independent samples t-tests were calculated to examine differences in PCOA scores by sex. Statistically significant differences were found for the total scaled scores, and Areas 1, 2 and 4 of the PCOA, with males scoring higher than females as reflected by mean score comparisons (Table 2). Effect sizes (Cohen’s d) were between small and medium, per Cohen’s general guidelines (ie, 0.2=small, 0.5=medium, 0.8=large). To examine additional academic performance measures by sex, independent samples t-tests were conducted but found no significant differences by sex for PCAT composite mean and SD scores [females=64.52 (21.89), males=70.76 (19.89), p=.072] or cumulative PharmD didactic GPA means and SD [females=3.21 (0.45), males=3.11 (0.40), p=.167].

Table 2.

Results of Independent Samples t-tests for PCOA Scores by Sex

The group of 159 examinees was composed of non-Hispanic whites (71.7%), African Americans/blacks (15.1%), Asians (10.7%), Alaskans (1.3%) and unknown (1.3%). ANOVA revealed no statistically significant differences for total PCOA score (p=.194), Area 1 (p=.188), Area 2 (p=.118), Area 3 (p=.505), or Area 4 (p=.242).

The majority of students were assigned to the Memphis campus (54.7%) with 27.1% and 18.2% assigned to the Knoxville and Nashville campuses, respectively. There was no statistically significant difference found by campus assignment for total PCOA score (p=.832), Area 1 (p=.823), Area 2 (p=.576), Area 3 (p=.756), or Area 4 (p=.259).

Academic Correlates

The first correlation analyses conducted were to investigate the relationships between PCOA total and subscores and PCAT composite and subscores. Statistically significant positive correlations were found for all variables of these examinations except between the PCAT quantitative subscore and PCOA Area 1 (p=.125). All significant positive correlations were weak to moderate, ranging from .17 (PCAT Chemistry and PCOA Area 3: Social/Behavioral/Administrative Sciences) to .54 (PCAT Verbal and PCOA Area 3). Table 3 displays the results.

Table 3.

Results of Pearson’s r Correlations between PCOA Scores and PCAT Scores

The second correlation analyses examined relationships between PCOA scores and GPA in the pre-pharmacy and pharmacy programs (Table 4). There was no correlation found between cumulative undergraduate GPA and total PCOA or area scores. However, statistically significant positive correlations in the weak to moderate range were found between undergraduate science GPA and PCOA total score, Area 2: Pharmaceutical Sciences, Area 3 and Area 4. Statistically significant positive correlations in the weak to moderate range were found between cumulative pharmacy GPA at the end of the didactic portion of the program and all areas of the PCOA.

Table 4.

Results of Pearsons’ r Correlations Between PCOA Scores and Grade Point Average

Of the 159 students who completed the PCOA examination, 20 students (12.6%) had academic issues that required them to appear before the college’s progression committee. Eleven of the 20 (55%) were allowed to progress for on-time graduation and were not required to repeat any course content (Group 2). The other nine students were required to successfully repeat a semester or year of curricular content before proceeding (Group 3).

The ANOVA showed statistically significant differences among the three groups for scores on PCOA total score (F=7.26, p=.001), Area 2 (F=5.87, p=.003), and Area 4 (F=10.4, p<.001). Bonferroni post hoc tests showed significant differences in mean scores between two groups only: those without any academic issues (Group 1) and those with academic issues who were allowed to continue without repeating curricular content (Group 2) (Table 5). These differences occurred for the total PCOA score (p=.003), Area 2 (p=.024), and Area 4 (p<.001), with those in Group 1 scoring higher. Conversely, there were no significant differences on total score or any Area score between those students without any academic issues (Group 1) and those with academic issues who were required to repeat curricular content (Group 3).

Table 5.

Post hoc Comparisons for Academic Standing

The final model in the stepwise linear regression analysis accounted for approximately 44% of the variance in total PCOA score (adjusted R²=0.44; p<.001) and included the following variables: PCAT composite percentile, pharmacy didactic GPA, and sex. PCAT composite percentile accounted for the largest percentage of variance in the regression model (24.0%), while pharmacy didactic GPA accounted for 15.5%, and sex accounted for 4.3% of the variance. The regression equation for the final model was as follows: PCOA Total Score = 168.9 + .78 (PCAT composite percentile) + 42.9 (pharmacy GPA) + 18.9 (sex; 0=female and 1=male).

DISCUSSION

The goal of this study was to examine relationships between students’ demographic and academic performance factors and their performance on the PCOA, in an effort to understand how PCOA results may best be used to improve the program and support student success. To date, few studies have been published in the pharmacy literature that examine relationships between PCOA scores and academic performance measures.^4-10 This study included two variables not previously examined (academic standing and undergraduate science GPA) in relation to the PCOA and reveals findings not yet reported in the literature.

The analysis of demographic variables showed no differences in PCOA performance by age, race/ethnicity, or campus assignment. However, a statistically significant difference was found by sex, with males scoring higher than females on the total PCOA score and all subscores except for Area 3. Although effect sizes were small, the sex variable fit into the regression model as a predictor for PCOA performance, accounting for approximately 4% of the variance in PCOA scores. Even though there have been reports of sex differences on standardized tests and different testing formats,¹¹ similar findings have not yet been reported in the literature on PCOA studies. Only two published studies have examined PCOA score differences by sex, and neither found significant differences.^4,5 Future analyses of the PCOA should be monitored for any trend on sex differences. There would most likely not be any direct implication for improving student success within a program, were such a bias to become evident. However, it would be important to bring to light any demographic bias on the PCOA if there were evidence to support it, as such information would have implications for test developers.

PCOA scores correlated positively with PCAT scores on all but one comparison, which was between PCOA Area 1 and PCAT Quantitative subscore. It is perhaps not surprising to see correlations between performance on the PCOA and PCAT, given that both are standardized tests. The PCAT composite score was also the strongest predictor of the PCOA in the final regression model, accounting for 24% of the score variance. This finding is consistent with other results reported in the literature finding PCAT scores to be a predictor of PCOA scores.^4,5

Cumulative undergraduate GPA did not correlate with PCOA or area scores. Consistent with findings of this study, other studies also found no relationship between PCOA scores and pre-pharmacy GPAs in pre-requisite courses and in cumulative GPA for undergraduate courses upon admission.^4,5 Conversely, the authors of this study found that undergraduate science GPA, a variable previously unexplored in PCOA studies, showed positive correlations with the total PCOA score and with Areas 2, 3, and 4 but did not show a correlation with Area 1. This finding is interesting because one might have expected Area 1 to be more likely to correlate with undergraduate science GPA. No comparison results are available in the literature to support or refute this finding. The only other undergraduate GPA variables included in PCOA correlation studies thus far are limited to an overall undergraduate “baseline” GPA and a GPA in pre-requisite course work, which included other courses outside of science (eg, English, statistics, calculus).^4,5 This is the first published study to examine the relationship between PCOA scores and undergraduate science GPA.

Pharmacy GPA correlated positively with the total PCOA score and with all four PCOA domains, which is consistent with results from other studies where relationships between PCOA scores and cumulative didactic pharmacy GPA were examined.^5-7 Additionally, positive correlations with PCOA scores have been found for GPA in other parts of the curriculum, specifically for APPE grades and for cumulative pharmacy GPA at the end of the P1 year.^4,8 Finally, another group of investigators used the PCOA to establish criterion-related validity for their own internally developed cumulative assessment and progressions examination, and found a strong correlation between performance on their internal examination and the PCOA.⁹ Findings from all of these studies provide evidence of the correlation between PCOA scores and pharmacy curricular content.

Perhaps the most interesting and potentially useful finding of this study surfaced in the analysis involving student academic status, a type of investigation that has not been previously reported in the literature on PCOA studies. It stands to reason that students who have struggled academically may have lower scores on the total PCOA or on select areas than students who have not. The fact that there was no effect of academic status on performance in Area 1 is not surprising given that all students took this content as part of the pre-pharmacy curriculum. It is also not surprising that there was no difference in social/behavioral/administrative sciences subscores as this content has less emphasis in the program than the areas of pharmaceutical and clinical sciences. It is noteworthy that post hoc tests showed the significant differences in mean scores between two groups only: those who had not experienced academic difficulty (Group 1) and those who did experience academic difficulty but were allowed to continue without repeating any curricular content (Group 2). Group 2 students scored significantly lower on total PCOA score, as well as Areas 2 and 4, than the group who never experienced academic difficulty. However, PCOA scores for the group with academic issues who repeated courses were not statistically different from the group who had no academic difficulty. This finding could certainly be interpreted as support for having struggling students to repeat courses.

Previous studies on the PCOA, although few, encompass a variety of interests. One interest is in the ability of the PCOA to predict NAPLEX scores. Two separate groups of investigators have assessed this and found some evidence of positive correlation;^6,10 however, one of these studies included pharmacy didactic GPA in their analysis and, interestingly, found GPA to be stronger than PCOA in predicting NAPLEX scores.⁶ The first cohort to take the PCOA as P3s has not yet taken the NAPLEX, hence analyses are planned to examine the relationship between scores on these two tests once the class graduates and their NAPLEX scores are obtained.

In addition to conventional demographics and academic performance variables, other factors that may correlate with PCOA performance have been recently explored. For example, one group examined the relationship between students’ learning style preferences and performance on the PCOA, finding that accommodators performed significantly worse on the examination than assimilators.⁴ Another group of investigators explored the effect of student motivation on PCOA performance.¹² They used a statistical motivation filtering technique to measure the level of student motivation on PCOA and filter out scores from students with low motivation to obtain data that would more accurately reflect student knowledge.¹² Additionally, they found that incentivizing students improved the motivation level on the PCOA and reduced the need to filter out scores for low effort.¹²

No doubt, an important point to consider when approaching the findings of this study is the issue of student motivation, which limits interpretation of the results. There were no consequences, positive or negative associated with student performance on the PCOA in this study. As long as it is uncertain whether students put forth their best effort, it is also uncertain whether the results are a valid indication of what students have learned. Pharmacy programs across the country have chosen to use the PCOA in various ways throughout their curricula with varying stakes attached.¹³ For example, some programs use the PCOA as a high-stakes examination, where scores affect progression in the program or need for remediation, while others attach medium stakes (eg, part of a course grade), or low stakes (eg, development plan required).¹³ Others use performance-based rewards (eg, bonus points, scholarship money) to motivate students to perform at the highest level on the test.¹³ Some administer the PCOA in multiple years, allowing students to have some earlier exposure to the test before they sit for the ACPE-required administration in the P3 year. As long as pharmacy programs are using the PCOA in different ways rather than in a standardized process with the same stakes for all, student motivation across programs is bound to be inconsistent, and any comparisons on a national scale are problematic.

This study included one cohort of students in one pharmacy program, so results may not generalize to the national pharmacy student population. Nevertheless, demographics of the class studied are similar to national demographics,¹⁴ and average PCOA scores for the class fell into the 50^th percentile nationally, hence the sample reflects the national population to a large extent. A variable of academic progress status was included in this study as well as a variable of undergraduate science GPA, neither of which have been examined in other published studies. The data showed significant correlations, some of which have direct implications of how these data can be used to help improve student success.

CONCLUSION

With the recent accreditation requirement for all pharmacy programs in the US to administer the PCOA, it is in the best interest of all colleges and schools of pharmacy to increase understanding of how best to use the results to improve their programs. This study found correlations between PCOA and other academic measures, thereby expanding support for the examination as a measure of content taught in pharmacy curricula. Findings within this study regarding differences in performance by sex and correlations with certain academic difficulty indicators have not previously been reported in the literature on PCOA. Results have implications for decisions about how to support students in academic difficulty. Future research should continue to build evidence to help programs make good decisions regarding the use of the PCOA, including how it correlates with students’ ability to practice pharmacy.