A 10-year Study of the Academic Progress of Students Identified as Low Performers after Their First Semester of Pharmacy School

Abstract

Objective. To examine whether pharmacy students characterized as low performers at the conclusion of their first semester remained low performers throughout their academic career.

Methods. Bottom quartile performance on first semester grade point average (GPA) was compared to licensing examination success, cumulative grade point average at the end of the didactic education and whether the student graduated on time, using cross tabulation analysis. Relative risk ratios and confidence intervals were calculated.

Results. Students in the bottom quartile for GPA at the end of their first semester in pharmacy school were approximately six times more likely not to graduate on time, not to pass the North American Pharmacist Licensure Exam on their first attempt and to remain in the bottom quartile for GPA at the end of their didactic education.

Conclusion. This study suggests that pharmacy students who score in the bottom quartile for GPA at the end of their first semester are more likely to underperform academically unless they take corrective action.

INTRODUCTION

Students, faculty members, and administrators anticipate that each student admitted to a school or college of pharmacy will successfully meet program requirements and graduate on time. Standard 17 (Admissions) of the 2016 Accreditation Council for Pharmacy Education (ACPE) Standards directs schools to make public their on-time graduation rates as an indicator of program quality.¹ As the number and size of pharmacy training programs continues to increase while the number of applicants decreases, concerns about the decreasing number of qualified applicants arise.² Programs struggling to meet enrollment targets with a shrinking applicant pool may be compelled to consider prospects who do not meet historical standards. In addition, Rupp argues that when academic standards are relaxed, faculty members are often asked to exert more effort to support struggling students.³ This may lead to tensions between those focused on Standard 17, which requires schools to select only qualified students, and Standard 18, which requires faculty members to identify students predicted to fail early enough to provide effective assistance.

Several articles highlight attempts by programs to identify predictive and causative factors associated with poor student performance. Culbertson determined that examination scores in pharmacotherapy courses did not predict performance in advanced pharmacy practice experiences (APPEs).⁴ Kinder and Knecht found that students who did not earn passing grades in introductory modules outperformed students in subsequent therapeutic modules.⁵ Schlesselmann and Coleman suggested that prepharmacy grade point average and verbal Scholastic Aptitude Test scores are useful in predicting poor academic performance defined as failure to graduate on time, cumulative grade point average (GPA) less than 2.7 for the first three years of pharmacy school, or a grade of C+ or below on any APPE.⁶ Allen and Diaz determined that prepharmacy predictors of success on the North American Pharmacist Licensure Exam (NAPLEX) included no unsatisfactory grades, cumulative GPA, and science and math GPA. On-time graduation rates, cumulative GPA, and no unsatisfactory grades in the prepharmacy or doctor of pharmacy (PharmD) program also correlated with NAPLEX performance.⁷ Chisholm-Burns et al found significant correlations between pre-NAPLEX performance and Pharmacy College Admission Test (PCAT) scores, race/ethnicity, and pharmacy school GPA.⁸ Alston and colleagues studied prepharmacy and first semester performance to identify variables that could be used to identify students most at risk to fail to graduate on time. Students with a first semester GPA below 3.0 were 17.6 times more likely to fail, and students with a second semester GPA below 3.0 were 35.8 times more likely to fail. Prepharmacy variables paled in comparison.⁹ Rupp highlighted the differences between administrative and faculty incentives with respect to student retention and challenged members of the academy to reconcile them.³

To effectively evaluate the impact of student remediation efforts at a school of pharmacy, it is essential to understand the historical academic trajectory of low performing students at the institution. Without the baseline data, future studies on the effect of a specific remediation would be difficult to analyze accurately. Thus, this study aimed to examine whether pharmacy students at Wingate University School of Pharmacy (WUSOP) characterized as low performers at the conclusion of their first semester remained low performers throughout their academic career. This study can then be used to inform future study on the effect of a remediation program to determine whether student performance improved from the historical baseline as a result of an intervention. While results may not be generalizable outside WUSOP, the methodology may be useful for schools seeking to create a baseline to analyze performance of their own remediation efforts.

METHODS

This study was approved by the WUSOP Research Review board as part of an ongoing institutional review of student progression. The analysis of low performing students required identification of proxies for academic performance that could be tested for evidence of a relationship. The indicators chosen to represent terminal academic performance were failure to graduate from the program on time with the student’s entering class, inability to pass the NAPLEX on the first attempt, inability to pass the Multistate Jurisprudence Exam (MPJE) on the first attempt, and inability to earn a GPA in the top 75% of all students. While the twenty-fifth percentile student has historically earned an acceptable didactic GPA of 3.13 at WUSOP, this marker was selected as the end point to begin the analysis to maximize the study’s ability to identify all potential academic failures. These markers were chosen because they were universal data points available to every pharmacy program and possessed a reasonable level of face validity. The cumulative GPA at semesters 6 and 8 were analyzed because of accuracy concerns for semester 8 grades. The preponderance of As earned during APPEs during semesters 7 and 8, and the fact that most students who fail out of the program do not complete these semesters, caused researchers to question which semester would be more predictive of failure. Therefore, the goal of phase 1 of this study sought to determine which terminal GPA was most appropriate for this dataset.

Student performance data were included for all students who had enrolled in WUSOP between fall 2003 and fall 2013. The number of students enrolled who remained in the program long enough to obtain a semester one GPA was 842. With a targeted first year of enrollment of 108 since 2011, 611 students had a semester 6 GPA available by the time of this analysis, and 545 had the opportunity to graduate on time. Graduates from the classes of 2011-2014 signed releases to allow the school to receive their identified scaled score, thus 266 student specific scores were obtained for the NAPLEX and 246 for the MPJE. Students who transferred or dropped out of the program were excluded from any analysis that required performance scores as such scores would not have been available because of these students’ lack of progression.

Additional variables were crafted from each of the original data points to facilitate analysis. The cumulative GPA for semesters 1, 6, and 8 were converted to percentile rank scores. For each semester, a dummy variable was created to identify whether the student’s GPA was in the bottom 10%, 15%, 20%, or 25% of the dataset. For NAPLEX and MPJE scores, the new study variables identified whether or not the student passed the examinations on his or her first attempt and whether his or her scores were in the bottom decile rank of all scores in the dataset. Each study variable was assigned a value of (0) if the score fell below the targeted performance and a value of (1) if the score rose above the targeted performance. To represent on-time graduation, a single field was created to identify whether students failed to graduate on time. Students who failed to graduate or failed to graduate on time were assigned a value of (0). Students who graduated with their entering class on time were assigned a value of (1).

Statistical comparisons for nominal data comparisons were completed using Fisher exact tests, whereas relative risk ratios were computed for ratio scale data. Systat13, v13.00.05 (Systat Software, San Jose, CA) software was used to complete all statistical analysis. Data were analyzed in four sequential phases. While some study variables may have shown a significance, the authors applied a standard of usefulness to determine whether the variables proved helpful in guiding school policy. Usefulness was determined during phases 2 and 3 and was defined as: those for which p values were significant (<0.05), those that identified more than 50% of the cohort that eventually failed, and those that resulted in substantial relative risk ratios (>3). A minimum threshold of 50% identification of students who failed was selected based on an attempt to have no more than 10-15 students from each class under active watch by student services. Relative risk ratios below 3 are viewed with suspicion.⁸

For phase 1, selection of the best proxy for end-of-program (terminal) GPA performance, student scores for first semester GPA (GPA1), cumulative GPA at the end of the didactic curriculum (GPA6), cumulative GPA at the end of the curriculum (GPA8), NAPLEX first attempt score, and MPJE first attempt score were analyzed to determine the correlation between the variables. The bottom decile percentile rank scores for GPA6 and GPA8 were compared to on-time graduation, NAPLEX failure, and MPJE failure dummy variables using a cross tabulation analysis. The GPA that best predicted failure was then selected to serve as the proxy for terminal GPA performance in pharmacy school. Many models assess the strength of a correlative relationship, and they all vary slightly in the breakpoints used. Guidance provided in the British Medical Journal was used to determine the strength of the Pearson correlation for this project. The journal contends that, “If we wish to label the strength of the association, for absolute values of r, 0-0.19 is regarded as very weak, 0.2-0.39 as weak, 0.40-0.59 as moderate, 0.6-0.79 as strong and 0.8-1 as very strong correlation…”⁹

Based on the proxy terminal GPA identified in phase 1, GPA6 was analyzed to determine its ability to predict graduation on time, NAPLEX failure, and MPJE Failure in phase 2. A cross tabulation analysis was completed to determine whether a student scoring in the bottom 10%, 15%, 20%, or 25% of GPA6 was more likely to fail to graduate on time or fail the NAPLEX or MPJE on his or her first attempt. Results were reviewed to select which percentile score was most useful to predict failure.

For phase 3, GPA1 was analyzed to identify the correct early indicator of low performance. A cross tabulation analysis was completed to determine whether a student in the bottom 10%, 15%, 20%, or 25% of GPA1 was more likely to fail to graduate on time, fall in the bottom quartile for cumulative GPA6, or fail the NAPLEX or MPJE on his or her first attempt. The p values were calculated and results were reviewed to select which percentile score was most useful to predict failure.

To determine if students who scored in the bottom quartile for GPA1 were more likely to remain low performers at the end of their academic career (phase 4), the GPA1 performance in the bottom quartile was compared to all four GPA6 rank brackets, GPA8 bottom decile, NAPLEX bottom decile and first attempt failure, and MPJE bottom decile and first attempt failure. A cross tabulation analysis was completed. A summary of the methodology used to create baseline study of low performers is available in Table 1.

Table 1.

Methodology Used to Create a Low Performers Baseline Study

RESULTS

Grade point average 1 was a strong predictor of GPA6 and GPA8 as evidenced by Pearson correlations above (0.7), a moderate indicator of NAPLEX score as evidenced by a correlation above (0.4), and a weak predictor of MPJE score as evidenced by a correlation below (0.2). Grade point average 6 was a strong predictor of GPA8 (0.96), a moderate predictor of NAPLEX score (0.58), and a weak predictor of MPJE score (0.3). Grade point average 8 was a strong predictor of NAPLEX score (0.61) and a weak predictor of MPJE score (0.34). No significant differences appeared when correlating GPA6 and GPA8 to GPA1, NAPLEX, or MPJE (Table 2).

Table 2.

Pearson Correlation of Academic Performance Variables in Pharmacy School

Table 3 summarizes comparisons of GPA6 and GPA8 bottom decile performance. The bottom decile of GPA6 significantly predicted failure to graduate on time and failure to pass the NAPLEX on the first attempt. Results for MPJE were not significant. The bottom decile of GPA8 predicted failure to graduate on time but did not predict either NAPLEX or MPJE failure. Thus, GPA6 was selected as the proxy for terminal GPA performance in pharmacy school based on its ability to predict graduation on time and NAPLEX score. Grade point average 8 only predicted graduation on time and identified a smaller cohort of the 526 students who had the opportunity to graduate on time.

Table 3.

Comparison of Sixth Semester and Eighth Semester Cumulative Grade Point Average Low Performance as the Best Indicator of Academic Performance

Regarding phase 2, Table 4 summarizes comparisons between GPA6 percentile brackets compared to the markers of academic failure. All four percentile brackets revealed significant relationships to failure to graduate on time, first attempt NAPLEX failure, and first attempt MPJE failure with the exception of the bottom decile’s ability to predict MPJE failure. Each bracket also met the standard of usefulness described earlier in this article with the exception of the bottom decile prediction of MPJE failure.

Table 4.

Cumulative GPA at Semester 6 (GPA6) Percentile Brackets Compared to Graduation on Time, NAPLEX Failure, and MPJE Failure

The bottom 25% of GPA6 captured the highest percentage of at-risk students and had the strongest relative risk profile for NAPLEX failure so it was selected as the optimal GPA6 rank to use in the remainder of the analysis. The bottom 10% of GPA6 was not significant for first attempt MPJE failure and was rejected. The bottom 20% of GPA6 had the weakest relative risk numbers for each comparison and was also rejected.

Given the selection of GPA6 bottom quartile, graduation on time, NAPLEX failure, and MPJE failure as the proxies for academic failure, GPA1 decile brackets were compared to these proxies (phase 3). Results are summarized in Table 5. All four GPA1 percentile brackets showed significant relationships with performance in the bottom quartile of GPA6, failure to graduate on time, and first attempt NAPLEX failure. Only the bottom 25% bracket significantly predicted first attempt MPJE failure. The bottom quartile bracket for GPA1 met the standard of usefulness previously described for GPA6 bottom quartile, graduation on time, and NAPLEX failures, but not for MPJE failures. While the results were significant for the MPJE failure, the variable only identified three of eight students who actually failed. The results of the analysis were reviewed, and the GPA 1 bottom quartile was selected as the predictor of poor academic performance because it identified the largest percentage of low performers, demonstrated the highest relative risk ratios, and was the only bracket that resulted in significance for all markers analyzed. Therefore, the GPA1 bottom quartile was the best early indicator to categorize students as low performers at the end of the first semester.

Table 5.

Cumulative GPA at Semester 1 (GPA1) Percentile Bracket Comparison to Low Performers Represented by Semester 6 GPA Bottom Quartile, Failure to Graduate on Time, NAPLEX Failure, and MPJE Failure

Table 6 shows the propensity for students who were low performers at the end of semester 1 to remain low performers at the end of their formal education as calculated by cross tabulation analysis (phase 4). Students performing in the bottom quartile of GPA1 comprised 80% of the students who ended up in the bottom decile of GPA6, 50% of the students in the bottom decile for NAPLEX score, 58% of the students who failed on the NAPLEX first attempt, and 65% of the students who failed to graduate on time.

Table 6.

Low Performance on Program Educational Outcomes Predicted by Being in the Bottom Quartile of First Semester Grade Point Average in Pharmacy School

Based on relative risk calculations, students who performed in the bottom quartile of GPA1 were roughly five times more likely to perform in the bottom decile of GPA6. In addition, they were roughly six times more likely to experience first time NAPLEX failure and on time graduation failure and seven times more likely to score in the bottom decile of NAPLEX. Comparisons to MPJE measures and GPA at semester 8 were not significant. Overall, early low performers (defined as those in the bottom quartile at GPA1) were likely to remain low performers for important educational outcomes of pharmacy education.

DISCUSSION

As schools seek to maintain or improve on-time graduation rates, it is important to determine if those initially labeled “poor performers” early in their didactic careers are likely to remain low performers who do not graduate on time or fail to pass the NAPLEX or MPJE on the first attempt.

These data are useful across the academy because they can help faculty members and administrators identify and better support at-risk students. While the data are limited to the results at one institution, data points were collected for 10 years and the use of GPA, on time graduation, and NAPLEX failure provide a measure of generalizability if other institutions use their own data and complete similar analyses. Additionally, the 10-year time frame and relatively large sample sizes of the subanalyses (between 611 and 250 cases) are fairly robust. However, this study has several limitations including the inability to control for changes in the delivery of the curriculum, remediation efforts, faculty quality, grading policies, or class size over the 10-year period for which data was collected. In addition, because WUSOP is an integrated curriculum, accepts students with a minimum of 60 units of prerequisites, is a 4-year program with three years of didactic training, and is a private university not integrated with a medical center, the specific results may not be generalizable.

When interpreting these data, one needs to understand that relative risk determines increased probability of the same outcome occurring in the defined group.¹⁰ Taubes and colleagues suggested that significant relative risks must indicate 3- to 4-fold risk increases with a 95% confidence interval that does not cross one.¹⁰ Based on these criteria, GPA1 had an insignificant impact on MPJE passage and GPA8; therefore, these markers were not reported. Moreover, relative risk calculations are measures of association not causation. Academicians must be careful not to misinterpret these data as causative. Otherwise, these results may be inappropriately used to eliminate students from programs who fail to reach a certain performance threshold. These results identify a tendency in the larger student population but should not be applied directly to the performance of any one student. It would not be appropriate to suggest that because a student scored in the bottom quartile at semester 1 that he or she should be prevented from continuing in the program. It would be more valid for a school to suggest that a student who scores in the bottom quartile is more likely to underperform unless the student alters behavior to improve performance.

Regardless of admission criteria, schools should identify available resources to assist poor performers while considering financial and time limitations. Although the bottom quartile of performers at GPA1 was the best predictor of continued poor performance, this pool of students identified was likely too large (25% of the student body) for institutions to support in a one-on-one fashion. Some students will self-select out of assistance programs inherently decreasing the pool, but schools should consider approaches to solving this problem. For example, schools with limited resources may wish to focus energy and resources on smaller groups (eg, students in the bottom GPA decile). The tradeoff for this approach is that a smaller number of potential failures will be prevented by this method. Alternatively, institutions may wish to perform a thorough review of their data to best determine where to focus limited resources. This research provides a step-by-step process to identify the best predictors of failure.

The institution studied currently requires students to repeat a course if they fail with a final grade <69.5. Because courses are offered only once per year, this frequently results in delayed progression. Multiple course failures may lead to probation, suspension, and, ultimately, dismissal. While remediation is focused on mastery of course content, personal issues students share with faculty members as potential causes for failure are also addressed when appropriate. However, this data shows the current remediation strategy does not consistently help students move out of the low performance area, despite some attempts for a holistic approach. Additionally, students who are low performing but passing courses often remain unidentified and unassisted. Alternatives to course remediation need to be developed and evaluated and, while important, are beyond the scope of this paper. However, the first step in any student assistance strategy requires identification of at-risk students, ideally early in the curriculum before they are eligible for suspension.

It is notable that GPA1 is strongly predictive of GPA6 but is less likely to identify students in the bottom quartile at GPA8. While these data cannot provide definitive explanations, the authors hypothesize this could be a result of lack of retained knowledge, limited ability to apply “textbook” knowledge in a clinical setting, or struggles with patient and health care provider communication.

Overall, this analysis does not suggest causation but it does identify a persistent pattern of low performance that should be addressed. On-time graduation and NAPLEX passage are two markers of failure that represent the greatest concern for most pharmacy programs since potential candidates may use these values to determine their preferred schools. Additionally, ACPE requires that schools report on-time graduation rates of the most recent graduating class which highlights the importance of this measure.¹¹ In time, as schools improve their processes to address low performing students, the current GPA markers could become less relevant. Ideally, all students committed to the curriculum will be supported so that, regardless of placement in GPA quartiles, they will graduate on time and pass the NAPLEX on the first attempt. Until that is achieved, the GPA is a valid performance measure.

CONCLUSION

A 10-year study of student pharmacists at Wingate University School of Pharmacy demonstrated that students who performed in the bottom quartile for GPA at the end of their first semester in pharmacy school were approximately six times more likely to not graduate on time, six times more likely to not pass the NAPLEX on their first attempt, and five times more likely to remain in the bottom decile for GPA at the end of their didactic education (semester 6 of 8). This study suggests that pharmacy students who score in the bottom quartile for GPA at the end of semester 1 are more likely to underperform academically unless they take corrective action.