Predicting Pharmacy Curriculum Outcomes Assessment Performance Using Admissions, Curricular, Demographics, and Preparation Data

Abstract

Objective. To determine the factors, including a preparation test, that best predict pharmacy students’ performance on the Pharmacy Curriculum Outcomes Assessment (PCOA).

Methods. Two cohorts of third-year pharmacy students completed a 100-item locally created PCOA pre-test, the PCOA Prep. This PCOA Prep was a cumulative knowledge test that was administered in the fall semester. In the spring semester, the students completed the 200-item PCOA and a separate survey on study habits and confidence. A retrospective review of students’ demographics data, pre-pharmacy admission variables, and pharmacy school factors were collected. Correlation and regression analyses were conducted to evaluate which factors predicted students’ PCOA total scaled score as well as scores in areas 1-4.

Results. One hundred seventy-nine students were included in the study. The majority were female (55%), white (54%), and 28 (SD=5.4) years old on average. Students’ average score on the PCOA Prep test was 80.7% (SD=7.8). The stepwise multiple linear regression model for the PCOA total scaled score included the PCOA Prep test, cumulative GPA at the end of the didactic curriculum, race/ethnicity, Pharmacy College Admission Test (PCAT) Verbal, PCAT Biology, and a class identifier. Including the PCOA Prep test explained more variance than the model without the test.

Conclusion. This study revealed that student performance on a locally created cumulative knowledge test best predicted the PCOA Total Scaled Score. These results offer insights into additional contributing factors that influence students’ PCOA performance and how colleges and schools of pharmacy could identify at-risk students who may need knowledge remediation prior to beginning advanced pharmacy practice experiences.

INTRODUCTION

The 2016 Doctor of Pharmacy (PharmD) accreditation standards established by the Accreditation Council for Pharmacy Education (ACPE) require colleges and schools of pharmacy to administer the Pharmacy Curriculum Outcomes Assessment (PCOA) to all students prior to starting advanced pharmacy practice experiences (APPEs).^1,2 The PCOA, which was developed by the National Association of Boards of Pharmacy (NABP), was designed to measure students’ progress in the didactic curriculum and student readiness for APPEs.^1,2 All pharmacy schools are required to offer the PCOA test, a program’s results can, with limitations, be systematically compared across schools for benchmarking.¹ Benchmarking allows schools to monitor areas where curricular improvement is needed .¹ The PCOA also offers students an opportunity to review content, and programs can use the results to identify students who may need remediation.¹ Although the ACPE is not currently requiring the PCOA as a high stakes test, benchmarking and use in accreditation decisions may impact stakeholders’ perceptions of its purpose. As a result, there is some uncertainty in the Academy about how the results should be used and the utility of engaging in PCOA preparation.

Prior to the mandatory PCOA requirement, the 2007 ACPE standards encouraged the periodic use of comprehensive knowledge and performance-based summative assessments,³ which resulted in the University of Oklahoma College of Pharmacy’s creation and seven-year use of six locally developed summative knowledge examinations. These “integrated examinations” were embedded into the final examinations of the pharmacy practice courses offered in each semester of the first three didactic years (six tests total) and were worth 10% of the overall course grade.⁴ The college’s course coordinators, along with the assessment and curriculum committees, identified the most important content in the required didactic courses and created the objectives and tests questions for the integrated examinations, which were different from courses’ lecture objectives and test questions.⁴ After the release of the 2016 ACPE Standards,² the college decided that, because of the time, effort, and resources that were being dedicated to creating the six integrated examinations, and because the PCOA lacked student preparation and study resources, and finally, because students would benefit from focused content review and structured feedback prior to APPEs and the PCOA test, the assessments were repurposed for use as PCOA and APPE preparation tools. Five of the six integrated examinations were combined into one 100-item multiple-choice cumulative knowledge test called the “PCOA Prep” test, which was to be administered during the fall semester finals week in the third professional (P3) year and would account for 10% of the overall grade for the Pharmacy Practice VI course delivered in that semester. Students were informed that the goal of the PCOA Prep test was to help them refresh their memory of the cumulative knowledge gained from course content covered over the previous three years in preparation for starting their APPEs. A second goal was to get students accustomed to preparing for and completing a large cumulative test (eg, PCOA) that covered what they had learned in their didactic coursework.

Previous research has revealed associations between students’ PCOA total scaled score and third-year GPA (grade point average);⁵ PCAT (Pharmacy College Admission Test) reading score and first-year pharmacy GPA;⁶ cumulative pharmacy GPA and score on the Health Sciences Reasoning Test;⁷ gender, PCAT score, pre-pharmacy science GPA, pharmacy didactic courses GPA;⁸ and PCAT and cumulative GPA.⁹ However, no studies have examined the impact of a PCOA preparatory tool/practice test on PCOA scores. Although one study examined the factors related to pre-pharmacy, pharmacy, and demographic factors related to Pre-NAPLEX test scores, the study did not evaluate the relationship between the Pre-NAPLEX and NAPLEX.¹⁰ Therefore, the purpose of the current study was to examine the relationship among PCOA Prep scores, the didactic curriculum (GPA and course remediation), demographic variables, admissions criteria (undergraduate GPA, PCAT scores), amount of study preparation for the PCOA, and student PCOA knowledge confidence (self-awareness) to P3 students’ performance on the PCOA examination.

METHODS

Third-year pharmacy students in the class of 2017 (N=99) and class of 2018 (N=80) at the University of Oklahoma College of Pharmacy completed the 100-item PCOA Prep cumulative knowledge test developed inhouse, the week before fall final examinations. The test was worth 50 points and accounted for 10% of their overall P3 Pharmacy Practice course grade. The test did not include content from the spring P3 semester as the PCOA Prep was a P3 fall test and students had not been exposed to the content covered in the spring coursework yet. The PCOA Prep creation and rationale was explained to the P3 students at the beginning of the P3 fall semester as well they were given PCOA Prep objectives four weeks prior to the test. Students were encouraged to study and were told that the goal of test was to help them prepare for APPEs and the PCOA. To help students prepare for the PCOA Prep test objectives and for the examination, all previous courses’ (P1-P2) lecture handouts, slides, and videos were made available to students on Desire2Learn (Desire2Learn, Inc, Ontario, Canada). All of the PCOA Prep test questions were loaded into Desire2Learn (D2L), and each of the 100 PCOA Prep Examination questions in D2L were tagged to four areas: the PCOA Prep test objective, Bloom’s Taxonomy (high or low level), one of the 15 CAPE Outcomes abbreviated terms,¹¹ and one of the ACPE Appendix 1 terms.² After students completed the fall PCOA Prep examination, they received their test grade before the end of the semester. In the spring semester, they received a locally developed report card called the Test Tracker, which was created by exporting test data from D2L and outlined their performance on the four tagged areas. Students were educated on what the four tags represented and encouraged to review the PCOA Prep test report card results. They were encouraged to study areas of weakness using course videos and/or notes prior to taking the PCOA, which was scheduled for the week before spring final examinations. Students were also encouraged to complete the formative quizzes available on the National Association of Boards of Pharmacy’s (NABP) website and review the four PCOA content areas prior to taking the PCOA examination. The PCOA was administered to P3 students during the first week of May in 2016 and 2017. Students were informed that the college had incentivized the PCOA in that they would receive bonus points in the P4 fall seminar course based on their PCOA score. The bonus points were based on a scale that coincided with a percent score (eg, 50-59=1 point; 60-69=2.5 points; 70-79=5 points; 80-89=7.5 points; and 90-99=10 points).

After completing the PCOA examination, all P3 students were invited to complete a 12-item survey regarding their demographics, the amount of time they spent studying for the PCOA Prep and PCOA tests (0, 1-5, or ≥6 hours, which were arbitrary cut points selected to reflect no studying, a single study session, or longer/multiple study sessions), study methods used for both tests (assessed because although the PCOA Prep was a study method, the college wanted to know what other ways students prepared for the PCOA), the perceived utility of the fall PCOA Prep (scale 1=strongly disagree to 5=strongly agree) and study objectives (scale 1=not at all useful to 5=very useful), and confidence in each of the four PCOA areas (scale 1=not at all confident to 5=very confident). The last question was an open-ended question that solicited overall comments. The questionnaire was administered after completion of the spring PCOA test. Student participation in the survey was not blinded and was voluntary. All students were invited to complete the survey as part of a quality improvement project and were told that the college was evaluating whether it should continue to offer the PCOA Prep. The survey was administered prior to students receiving their PCOA scores. The University of Oklahoma Institutional Review Board approved this project.

Data Collection and Analyses

To gather the factors related to performance on the PCOA examination, institutional databases and the Pharmacy College Application Service (PharmCAS, Liaison International, LLC, Stone Mountain, GA) were used to perform a retrospective review of student records. Three categories of data were collected: demographics, including age at PCOA test date, gender/sex, and race/ethnicity; pre-pharmacy and admissions GPAs, including undergraduate science GPA, undergraduate cumulative GPA, graduate coursework, more than one Pharmacy College Admission Test (PCAT) attempt, PCAT Composite score and section scores (Biology, Chemistry, Quantitative, Reading, and Verbal); and didactic curriculum including history of remediation or delay in graduation, and cumulative GPA at end of the didactic program.

To statistically analyze the factors related to PCOA performance, continuous variables were summarized using means, standard deviations, minimum and maximum values. Categorical variables were summarized using frequency number and percent. Dummy coding of categorical variables was performed as follows: sex/gender (0=male, 1= female); race/ethnicity (0=white; 1=All others); graduate coursework (0=none, 1=at least one credit hour); PCAT attempts (0=only one attempt; 1=more than one attempt); remediation (0=no courses remediated, 1=one or more courses remediated); delay in graduation (0=graduated on-time, 1=delayed due to any reason).

Bivariate correlations were examined using zero-order Pearson correlation coefficient (r) for continuous variables and point biserial for dichotomous variables. Confidence intervals (95% CI) were added to show the precision of the correlation coefficient. Stepwise regression was used to model the linear relationships between PCOA scores and multiple variables.¹² Stepwise selection adds and removes candidate variables until a best fit model is achieved. Standard regression diagnostics were performed, including residual analysis and checks for multicollinearity. Parameter estimates from the final model and adjusted R² (aR²) were reported. Additionally, both type II semi-partial correlations (representing the unique correlation between predictor and PCOA score after controlling for other variables in the model) and type II squared semi-partial correlations (representing the proportion of variance that is uniquely explained by the predictor after accounting for other model predictors) were reported.

One-way analyses of variance (ANOVA) were performed with Tukey adjustments for multiple comparisons to examine mean PCOA scaled score differences between groups on variables of interest. All analyses were performed using SAS software, version 9.4 (SAS Institute Inc, Cary, NC). The a priori significance level was set at p<.05.

RESULTS

The demographic, pre-pharmacy, and pharmacy academic characteristics of the 179 students in the study are reported in Table 1. All P3 students from the classes of 2017 and 2018 were included in this analysis. The mean age of the students was 28 years at the time of the PCOA administration and participants were mostly female (55%) and white (54%). The PCOA study habits and perception survey was administered to all 179 students. One hundred sixty-eight (94%) students completed the entire survey, with the remaining 11 completing some but not all items.

Table 1.

Demographic, Pre-Pharmacy, and Pharmacy Academic Characteristics (n=179)

The amount of time students reported studying for the PCOA Prep and PCOA examinations differed. Eighty-two percent of students reported at least some amount of studying for the PCOA Prep compared to 73% for the PCOA, p<.001. Rates of studying less than 5 hours were similar for both the PCOA Prep and PCOA, with 49% and 51%, respectively. However, the percentage of students studying more than 5 hours was significantly higher for the PCOA Prep (33%) than the PCOA (21%). Students used a variety of study methods to prepare for the PCOA: 53% used the lecture objectives provided for the PCOA Prep examination; 20% used content and topic areas for the PCOA that were listed on the NABP website; 3% watched course lecture videos; and 22% studied course handouts, slides, and/or notes.

Four percent of students strongly agreed and 27% agreed that the PCOA Prep examination increased their comfort level with taking the PCOA examination; however, 8% strongly disagreed and 17% disagreed with this statement. Forty-five percent of students gave a neutral response, indicating that completing the PCOA Prep made no change in their comfort level.

When asked how useful the PCOA Prep examination was in helping them prepare, a majority (62%) of students reported that it was of average or above average usefulness. However, 29% found the PCOA Prep of very little use and 10% reported it being not at all useful.

Students self-reported their level of confidence relative to each PCOA content areas. Twenty-two percent of students reported having little to no confidence in area 1, basic biomedical sciences; 11% had little to know confidence in area 2, pharmaceutical sciences; 11% had little to know confidence in area 3, social, behavioral, and administrative sciences; 8% had little to know confidence in area 4, clinical sciences. Most students reported having an average level of confidence in each content area: area 1, 56%; area 2, 62%; area 3, 55%; and area 4, 49%. The areas in which students reported having the most confidence (rated confident or very confident) was area 4 with 42%, followed by area 3 with 35%, area 2 with 27%, and area 1 with 22%.

Students’ confidence level and average scaled score for PCOA content areas are reported in Table 2. The 22% of students who reported having no to little confidence in their knowledge of basic biomedical sciences scored a mean of 345.2 (SD=60.4). Compared to the scores of these students, the mean scores of students reporting average confidence were 31.5 points higher, and mean scores for students who reported feeling confident or very confident were 56.6 points higher. All three pairwise comparisons were significant (p<.05) using the Tukey procedure following a significant result using a one-way ANOVA, p<.05.

Table 2.

Pharmacy Students’ Self-Reported Confidence in Their Knowledge and Study Habits, and Scaled Scores for Pharmacy Curriculum Outcomes Assessment Content Areas (N=168)

Students average scores for the pharmaceutical sciences content area did not have as large of increases across confidence levels. The scaled score average of students reporting little to no confidence was 367.7, with an increase of 11.2 point for those with average confidence and an increase of 41.5 points for those who were confident or very confident. Significant differences (p<.05) were found between students with high confidence and those with low or average confidence, but not between students with low or average confidence (p=.70).

No significant differences were found in average scores for social, behavioral, and administrative sciences between confidence levels. The range of average scores between students with low and high confidence was 31.9 points. Finally, students reporting little to no confidence in their knowledge of clinical sciences had an average scaled score of 351.6, which was significantly different (p<.05) from that of those reporting being confident to very confident in their knowledge whose average was 44.1 points higher. No other differences were significant for area 4.

The relationships between pre-pharmacy, demographic, didactic program, and select survey variables and the PCOA total scaled score are reported in Table 3. Out of the 19 variables examined individually, 13 were found to be significantly correlated with PCOA total scaled score. The largest was the PCOA Prep examination (r=0.66), followed by cumulative didactic GPA (r=0.57) and PCAT composite score (r=0.51).

Table 3.

Stepwise Multiple Linear Regression Modeling of the Pharmacy Curriculum Outcomes Assessment Total Scaled Score

The first linear regression analysis (reported as Model 1) included all variables for consideration except for the PCOA Prep examination results. This analysis was performed to show the maximum amount of variance that could be explained without the PCOA Prep examination results. The final model for analysis 1 included cumulative didactic GPA (13% of unique variance explained), course remediation (1%), race/ethnicity (4%), and PCAT Verbal (7%), Biology (1%) and Chemistry (1%). The selection process allowed for either the PCAT Composite or the five PCAT area tests to be included for consideration. Despite the PCAT Composite having a higher correlation with PCOA total scaled score, the regression model included three area tests, which collectively explained more variance than the Composite score alone. Class identifier (2017 vs 2018) had a significant bivariate relationship with PCOA total scaled score and was found significant in some adjusted models. Here it was included as a nuisance variable to maintain parity with other models. This model accounted for 53% of the variance in the PCOA total scaled score (p<.05).

Model 2 added PCOA Prep as a variable to consider in the stepwise model selection. The final model accounted for 61% of the variance, an increase of 8 percentage points from model 1. The PCOA Prep explained the largest amount (9%) of unique variance in the model, with cumulative GPA decreasing to account for only 1%. The second largest predictor was PCAT Verbal (5%), followed by PCAT Biology (4%). In the final model, race/ethnicity accounted for only 2% of uniquely explained variance. Class identifier was kept as a control variable, but in this model, it was significant (p=.04).

The PCOA Prep and cumulative didactic GPA were correlated, r= 0.68 (95% 0.60-0.75). However, a three variable partial correlation analysis between the PCOA Prep examination and PCOA total scaled score while controlling for cumulative GPA resulted in a Pearson partial correlation of 0.46. By reversing it and controlling for PCOA Prep, the partial correlation between cumulative GPA and PCOA total scaled score was only 0.22. Additionally, multicollinearity was examined in both model 1 and 2. Neither model had a tolerance less than 0.1, nor a variance inflation greater than 2.2, indicating all variables included in the final models explained a unique proportion of variance.

Table 4 reports the multiple variable linear regression results from modeling each PCOA Area Score. Only variables included in any of the final models were reported. The aR² for these models were collectively lower than the PCOA Total Score’s aR² of 0.6125; the lowest was area 1 (aR² = 0.3792) and the highest was area 2 (0.5495). Significant relationships were found between the in-house PCOA Prep examination and each PCOA area score, though its proportion of unique variance explained varied depending on the area, ie, lowest for area 3 (3%) and highest for area 2 (17%). It remained the strongest predictor in each model, except for area 3, where PCAT Verbal explained 6% of unique variance compared to PCOA Prep’s 3%.

Table 4.

Stepwise Multiple Linear Regression Modeling Pharmacy Curriculum Outcomes Assessment Area Scaled Scores

Collectively across the models, six variables were found to have a negative relationship, with at least one PCOA area score. Demographically, being female was associated with a lower score in area 1, and being non-white was associated with lower Area 2-4 scores. Higher undergraduate science GPAs was associated with a lower area 3 score, students who took the PCAT more than once was associated with a lower score in area 1, and PCAT Quantitative was associated with decreased area 4 scores. Study preparation had a role in two area models. On average, students who studied less than five hours had lower scores in area 1 compared to students who did not study. Additionally, on average, students who studied more than five hours were associated with higher area 3 scores compared to those who did not study. The remaining variables reported in Table 4, most of which are academically related, all had some degree of a positive relationship with at least one PCOA area score.

A post-hoc analysis was performed on gender/sex and race/ethnicity to determine if differences exist on observed PCOA SS, pre-pharmacy and didactic-related variables. The t test was used to examine differences between male and female students, and one-way ANOVAs for race/ethnicity with Dunnett tests for pairwise differences between each minority and the control group (set to white).¹³ With six race/ethnicity groups, a total of 15 pairwise comparisons could be made, thus resulting in very conservative tests if using Bonferroni or Tukey multiple comparison procedures. Thus, a post-hoc Dunnett test was chosen as it adjusts for five simultaneous comparisons to a control group.

No differences in observed scores were found between sex/gender groups for PCOA overall scaled score, area 2, 3, and 4 scaled scores, PCOA Prep examination score, PCAT Composite score, and PCAT Chemistry score, quantitative score, and reading subtest scores. Overall, female students scored lower on the PCAT Biology section (p=0.01) and PCAT Verbal section (p=.03), and slightly lower on PCOA area 1 (p=.07). However, female students had a higher cumulative didactic GPA than male students, p=.01.

Fifty-four percent of students reported their race/ethnicity was white and 29% reported their race/ethnicity was Asian. The remaining 17% of students reported their race/ethnicity as American Indian (2%), black (6%), Hispanic (2%), or as being two or more races (7%). These small, disproportionate numbers often resulted in underpowered statistics to determine if differences in mean scores on selected variables existed. Where differences were found, minority averages were higher than majority on some variables and lower on others. Differences were found on mean scores for the PCAT Chemistry and Quantitative sections, with Asian students’ average score being higher than white students’ average; for the PCAT Verbal (underpowered to detect which differed), for the PCOA Overall SS and Area 4 with average scores for Whites being higher than averages for both Asian and Black students; and for PCOA Areas 2 and 3 average scores for Asian students were lower than average scores for White students. No differences in mean scores were found in the remaining variables: PCAT Biology, PCAT Reading, PCAT Composite, PCOA area 1, cumulative didactic GPA, and PCOA Prep.

DISCUSSION

Since taking the PCOA became a requirement in 2016, uncertainty has existed regarding how PCOA scores for students and programs would be used in the future, which can be stressful for both students and programs. In addition to this uncertainty, questions also exist about whether pharmacy schools should help prepare students for the PCOA. If NABP’s intent for the PCOA was to measure APPE readiness, then pharmacy schools helping students prepare for the PCOA would ultimately help them reinforce their knowledge prior to APPEs, which would provide a benefit to students beyond PCOA performance. Students may forget some content over the span of the program because of lack of repetition or perceived lack of relevance. Therefore, encouraging them to study the most important content determined by the school’s faculty members the semester before the PCOA test is administered should give students an advantage with remediating any deficient knowledge and increase the likelihood of retrieving that content for future use in APPEs as well as on the PCOA. As a result, the rationale for studying for the PCOA is not grounded only in performing better on the test, but also in reviewing essential content needed for APPEs.

Another area of uncertainty is whether schools of pharmacy should use the PCOA as a measure of their program’s curricular effectiveness. One disadvantage of using the PCOA as a metric for curricular effectiveness is that there are a limited number of test questions per content area. Therefore, students’ poor performance in a given area may not be related to a curricular weakness, but instead may be related to sampling, (ie, students may not remember specific aspects of course content rather than being weak overall in a course). A more complete measure of curricular effectiveness could include the tagging of assessment questions as to whether they are high or low level on Bloom’s Taxonomy and to the CAPE outcomes and ACPE standards the questions address. This would allow schools to map their curriculum and evaluate how often the content is assessed in a course, at what level it is assessed, if it is assessed across the curriculum, and how well students perform on the content areas in aggregate. Schools may discover that the content is not assessed in a course, is not longitudinally assessed, or is not assessed at a level that advances across the curriculum. These data could then be compared to the PCOA Prep and PCOA results to provide a more comprehensive picture of students’ knowledge.

The goals of this study were to evaluate factors related to student performance on the PCOA in order to better understand how to help students reactivate or retrieve prior knowledge or refresh content knowledge and prepare for the PCOA test and APPEs. While previous literature has found that pharmacy GPA, undergraduate GPA, PCAT scores, and demographic variables influenced PCOA performance, our current study evaluated the impact of three new variables on PCOA scores, PCOA pre-test scores (PCOA Prep), number of PCAT attempts, and student self-reported confidence in PCOA knowledge (self-awareness, confidence, and metacognition). Our study results revealed that the PCOA Prep could predict success on the PCOA, which may be most beneficial for predicting at-risk students who may have knowledge deficits. Being able to predict these knowledge deficits would allow for earlier identification and remediation of content knowledge to ultimately improve APPE preparedness.

The overall regression model that accounted for 61% of variance in the PCOA total scaled score included one of the variables not previously reported in the literature, the PCOA Prep test. The variable that accounted for the greatest proportion of unique variance in the stepwise multiple linear regression model for the PCOA total scaled score was students having taken the college’s locally created PCOA pretest, the PCOA Prep, which explained nine percent of the variance. Another finding related to the PCOA Prep study variable was that, although students reported on the survey that they did not find the PCOA Prep test useful when taking the PCOA test, it was the most correlated with PCOA scores. Students’ scores on a test similar to the PCOA Prep may help programs identify at-risk students with knowledge deficits that may limit their performance on the PCOA and APPEs. The earlier that programs can identify at-risk students, the earlier they can help those students remediate knowledge so they can successfully complete the program requirements.

Number of PCAT attempts was a variable that was associated with area 1 scores. On average, students who took the PCAT more than once did worse on area 1, the basic biomedical sciences, by 20 points. Number of PCAT attempts was a unique variable in this study that had not been evaluated in previous PCOA studies. These results may suggest that students who need to take the PCAT multiple times struggle with standardized tests or have knowledge deficits. The PCAT Verbal and PCAT Biology scores were two additional admissions variables that were positively associated with the PCOA Total Scaled Score, explaining five and four percent of the unique variance, respectively. In addition to the overall PCOA score, the PCAT Verbal score was also positively associated with area 2, 3 and 4 scores. The PCAT Biology score was positively associated with the area 1, 2 and 4 scores, the PCAT Chemistry was associated with the area 2 score, and the PCAT Quantitative score was positively associated with the PCOA area 4 score. Overall, schools of pharmacy that want to screen for at-risk students for the PCOA and APPEs could use the PCAT results to identify students, with the exception of the PCAT Verbal section as the section was retired by the National Association of Boards of Pharmacy. In the future this regression model will need to be reevaluated with students who complete the new PCAT.

Students’ self-reported level of confidence in knowledge of each of the four PCOA areas was another outcome assessed in this study. The results revealed that students’ metacognition or self-awareness of their level of confidence in the four PCOA test areas was an accurate self-assessment, as students who had little to no confidence in areas 1, 2, and 4 had lower scores than students who had high confidence in those three areas. Previous reports in the literature indicate that students lack self-assessment skills, as seen in low correlations between students’ self-assessment and actual performance or achievement.¹⁵ In contrast, the third-year students in this study demonstrated accurate self-awareness, ie, students with low confidence in a PCOA area (1, 2, and/or 4) earned low scores. This finding offers programs an additional strategy for assisting students with knowledge remediation prior to APPEs. When determining how to prepare for the PCOA and for APPEs, students may feel overwhelmed with the amount of prior content they need to study. Encouraging or requiring students to self-assess their confidence can offer them insight into whether they need to study a topic area more as metacognition is related to planning, monitoring, and evaluating. This coupled with a knowledge pre-test may help students focus on how much to study and what to focus on specifically. Programs may also want to consider strategies for supporting knowledge remediation as academic at-risk students who accurately self-assess that they have low confidence may not respond to this by increasing their study time in a given area.¹⁵

Cumulative GPA at the end of the didactic curriculum was significantly associated with the PCOA Total Scaled Score. While this variable was positively associated with the overall score, as seen in previous PCOA studies, the PCOA Prep test explained a greater portion of the variance. Therefore, although GPA at the end of the didactic curriculum may help programs identify at-risk students, a better screening tool for at-risk students may be to offer them a cumulative knowledge test. One study found that students who read a passage and then took a test retained more than students who repeatedly studied material or drew detailed diagrams that documented what they learned.¹⁴ Ultimately, students should take tests if they want to know how well they know given content. Tests such as the PCOA Prep help students identify what they know and what they do not know, which can lead to more strategic studying approaches.

The demographic results revealed that students’ gender/sex accounted for two percent of the unique variance in scores for the Area 1 Basic Biomedical Sciences model where being female was associated with a decreased score. A difference in gender/sex was also noted in another PCOA study that found a significant difference in scores based on the gender/sex variable, which is why we collected this variable for analysis.⁸ The demographic results also revealed that race/ethnicity accounted for two percent of the unique variance for the overall PCOA Total Scaled Score with the PCOA Prep model. Although previous PCOA studies did not find race/ethnicity differences on the PCOA test, one study found a difference on Pre-NAPLEX test scores,¹⁰ which is why we collected this variable for analysis. In addition to the PCOA Total scaled score, being a minority was associated with a decrease in predicted scores in area 2 (pharmaceutical sciences), 3 (social, behavioral, and administrative sciences), and 4 (clinical sciences) scores. However, race/ethnicity accounted for no more than 2% of the unique variance in these models. Additional research is needed to further evaluate the impact of race/ethnicity and gender/sex on the PCOA test.

The study had some limitations which should be considered. First, the research was conducted at a single institution, which may limit the generalizability of the results. Second, the PCOA Prep was a local examination created at the University of Oklahoma and is not available to other colleges and schools of pharmacy. However, the test could be created by asking content experts within a pharmacy school to identify the five most important content areas in a given course and then write an objective and related test question for each important area. If every course offered approximately five test questions, a 100-item multiple-choice test could be created. Programs could also recycle a limited number of existing test questions from courses to create a cumulative knowledge examination. Creating this preparatory examination should not be resource or time intensive, especially if programs are currently using electronic testing.

CONCLUSION

This study revealed that student performance on a locally created cumulative knowledge test best predicted the PCOA Total Scaled Score. These results offer insights into additional contributing factors that influence students’ PCOA performance and how schools of pharmacy could identify at-risk students who may need knowledge remediation prior to APPEs. While early preparation may help increase PCOA scores, it is important to emphasize that engaging in early PCOA preparation and assessment allows students more time to remediate knowledge deficiencies so that they can retrieve and use that knowledge while on APPEs. Programs may also want to evaluate additional predictors of PCOA performance including the number of PCAT attempts and student PCOA content area confidence, as these were also correlated with PCOA Total Scaled Scores.