Predictors of Student Failure or Poor Performance on Advanced Pharmacy Practice Experiences

Abstract

Objective. To determine factors predictive of student failure or poor performance on advanced pharmacy practice experiences (APPEs) at a single pharmacy program.

Methods. This retrospective cohort evaluated students entering the Doctor of Pharmacy (PharmD) program from 2012-2014 at St. Louis College of Pharmacy. Students who received a grade of F for one or more APPEs (failure group) were compared to all other students (non-failure group). A secondary evaluation compared students with a C or F on one or more APPEs (poor performers) to all other students (non-poor performers). Data were collected on didactic and experiential performance, identifiable professionalism issues from introductory pharmacy practice experiences (IPPEs), and academic honor code violations. Univariable and multivariable logistic regressions were performed to determine factors associated with APPE failure and poor performance.

Results. A total of 669 students were analyzed. Twenty-eight students (4.2%) failed one or more APPEs and 81 students (12.1%) were identified as poor performers (grade of C or F). For the primary outcome, professional grade point average (GPA) of less than 2.7, practicum failure, IPPE professionalism issue(s), and pharmacotherapy course failure were identified for inclusion in the multivariable analysis. The IPPE professionalism issue(s) (HR 4.8 [95% CI 1.9-12.4]) and pharmacotherapy course failure (HR 4.2 [95% CI, 1.6-11.1]) were associated with APPE failure on multivariable regression. On the secondary analysis, the same variables were identified for multivariable regression, with professional GPA of less than 2.7 (HR 2.7 [95% CI 1.5-5]), IPPE professionalism issue(s) (HR 3.9 [95% CI 2.2-6.9]), and pharmacotherapy course failure (HR 2.0 [95% CI 1.1-3.7]) associated with poor performance.

Conclusion. Poor academic performance and/or identified unprofessional behavior while completing IPPEs are associated with APPE failure and poor performance. Interventions should be aimed at identifying at-risk students and addressing risk factors prior to APPEs.

INTRODUCTION

Doctor of pharmacy (PharmD) students are expected to be prepared to begin advanced pharmacy practice experiences (APPEs) in their final year of the program because there is a stark increase in the level of responsibilities that they must assume between the classroom, where patient interactions are hypothetical and simulated, and the healthcare environment, where patients are real. Despite being supervised during APPEs, students can either elevate or worsen the level of patient care they provide based on the knowledge, skills, and attitudes they master prior to entering their final year of training. Fortunately, the Accreditation Council for Pharmacy Education (ACPE) provides curricular standards to support students’ success on APPEs.¹ Based on the educational outcomes described in Standards 1-4, Appendix A of “Guidance for Standards 2016” outlines the core domains students must achieve prior to beginning APPEs. However, institutions must determine how they will identify students’ readiness for APPEs and are encouraged to look at students’ success on introductory pharmacy practice experiences (IPPEs) and to conduct other assessments.² Thus, pharmacy schools have used simulations, standardized patients, objective structured clinical examinations (OSCE), and progress examinations to assess students as described in the literature.^3-7 While schools have placed emphasis on determining student readiness, they have not consistently identified predictive factors for poor student performance on APPEs. Therefore, identifying at-risk students remains difficult.

Several variables related to poor student performance in pharmacy education have been studied, but with contradictory findings. Common variables studied can be categorized into pre-admission academic factors, non-cognitive traits, and demographics (gender, race, and national origin).^5,8-25 Examples of pre-admission factors studied include Pharmacy College Admission Test (PCAT) scores, pre-pharmacy grade point average (GPA), grades in science/math courses, and prior completion of a pharmacy degree.^8-25 Examples of noncognitive traits range from interviews to essays that assess motivation, professionalism, responsibility, communication, self- and social awareness, self-control, and moral reasoning.^10-16 In one study examining academic factors, the authors found that the highest predictors of failure were: scores on the first examination (or overall grade) in an integrated science course, a GPA less than 3.0 in the first semester, and a cumulative GPA of less than 3.0 for the first year of pharmacy school.²⁶ Students who were in the bottom 25% of their class for GPA at the end of their first semester of pharmacy school were six times more likely not to pass the North American Pharmacist Licensure Examination (NAPLEX) on the first try, six times more likely not to graduate on time, and five times more likely to remain in the bottom percentile for GPA at the end of their didactic education.²⁶ More recently, a model to predict NAPLEX outcomes and identify students needing more preparation associated the following factors with poor outcomes: age greater than 28 years at graduation, PCAT scaled score less than 74, grades of less than 74% in more than three courses, and a Pharmacy Curriculum Outcomes Assessment (PCOA) scaled score less than 349.²⁷

Most variables to date have been largely examined to determine whether they were predictive of overall academic performance and not necessarily APPE performance. However, in 2019, Nyman and colleagues conducted a four-year review of their students’ (n=226) demographics, pre-pharmacy performance, didactic performance, and external assessment tests compared against APPE performance to create a model to predict APPE readiness.²⁸ The primary outcome was the average of a core APPE midpoint score, and multiple models were examined including all core APPEs, just acute care and ambulatory care APPEs, and only students who were failing at midpoint. The study concluded that aggregate pharmacy educational knowledge-based variables such as the PCOA or performance in therapeutics, as well as the entering age of students, are more predictive of a student’s performance at midpoint on core APPEs than skills-based variables (OSCEs), admission measures, or other student demographics. However, the findings only demonstrated modest predictability and the authors describe that additional student variables likely to contribute to APPE success are communication, professionalism, and personal factors (such as physical and mental health).

Knowledge of specific factors can better inform institutions on how to determine pre-APPE readiness and help at-risk students prior to entering patient care environments. The objective of this study was to determine which additional factors are predictive of student failure and poor performance on APPEs.

METHODS

This single-institution, retrospective cohort study evaluated students entering the professional PharmD program at St. Louis College of Pharmacy from 2012 to 2014. This corresponded to students entering APPEs during the 2015-2016, 2016-2017 and 2017-2018 academic years. At the time of this study, the majority of students entered a 0-6 program, first completing pre-professional (two years) coursework and then professional (four years) coursework, with a relatively small number of students transferring into the program and completing only the four years of the professional curriculum. The professional curriculum included three years of pre-APPE curriculum followed by eight APPEs, each lasting five weeks. All students were followed from the time of enrollment through completion of the professional curriculum. Data collected on all evaluated students included: undergraduate GPA, pre-APPE professional GPA, individual professional didactic and experiential course grades, practicum scores, calculation proficiency examination scores, professionalism issues on IPPEs, and academic honor code violations.

For analysis, professional GPA (based on a 4.0 scale) was a priori categorized as a nominal variable with individual cut points of 2.5 and 2.7, based on previous institutional admissions and progression policies. Pharmacotherapy courses were evaluated individually and as an aggregate single variable that included all pathophysiology, medicinal chemistry (seven credit hours across two courses in the second professional year), pharmacology (eight credit hours across two courses in the second professional year), and therapeutics courses (four 4-hour courses across the second and third professional years). These didactic course grades were based on a traditional A-F grading scale, with course grade totals below 60% constituting failure (F) for the course. Practicums, which evaluated individual student performance of skills essential in pharmacy practice and on APPEs (eg, conducting patient interviews, educating patients) were conducted as part of the pharmacy practice skills laboratory course. Student performance on these practicums were collected and included in the analysis as pass or fail. Academic honor code violations included any reported acts of academic dishonesty (eg, cheating, plagiarism) that were reviewed and investigated by a board of faculty members and peers who ultimately decided on the responsibility of the student(s) for the reported act. Only students found responsible through this process would have a record of a violation of the academic honor code entered in their student record.

Three hundred hours of IPPEs were completed by each student, including three longitudinal experiences (one during each pre-APPE professional year; 20 hours/year) and two concentrated experiences in the summer (community experience=120 hours after the first professional year; health system experience=120 hours after the second professional year). Professionalism issues on IPPEs were defined as one or more incidents when expectations in any of the following five categories were not met (examples of unprofessional behavior given in parentheses after each category): exhibiting professional demeanor at the site or the classroom (eg, tardiness or inappropriate dress), recognizing opportunities to fulfill social and professional responsibilities (eg, disinterest or non-substantive reflections), treating patients, peers, faculty members and other health care professionals in a respectful and caring manner (eg, lack of timely response to emails or disrespectful communication), demonstrating a professional work ethic (eg, scheduling site time during classes or submitting assignments late), and complying with all laws and regulations governing the practice of pharmacy (eg, letting intern licensure lapse). Site preceptors communicated any professionalism incidents to the faculty member serving as course coordinator who then documented the incident and notified the student.

The APPEs assigned to each student included: ambulatory care, community care (community pharmacy practice), health system management (health system/institutional pharmacy practice), acute care/general medicine, patient care selective (clinical, patient care rotation in a medicine subspecialty), and three electives (combination of patient care and non-patient care settings). The final grade for each APPE was comprised of clinical performance and assignments. The clinical performance included the outcomes of thinking and decision-making, communication, and professionalism. If any clinical performance outcome or an assignment was evaluated as not passing, the student would fail the APPE. These final APPE grades were based on an altered A-C or F grading scale, with a C remaining a final grade of 70%-79% and F being any final grade below 70%.

The primary evaluation was conducted to determine predictors of student failure on APPEs. All students earning an F on one or more APPEs were defined as the “failure cohort.” All other students were defined as the “non-failure cohort.” A pre-specified secondary evaluation was conducted to determine predictors of student poor performance on one or more APPEs. Students earning a grade of C or F on one or more APPEs were classified as the “poor-performer” cohort.

Statistical Analysis

Statistical analysis was performed using SAS, version 9.4 (SAS Institute Inc., Cary, NC). Baseline characteristics were analyzed using a Student t test for continuous variables and a chi-square or Fisher exact test for nominal data. For the primary evaluation, the failure group was compared to the non-failure group using univariable and multivariable logistic regression to determine predictors of APPE failure. All variables were included in univariable analysis and were eligible for inclusion in multivariable analysis based on a predefined stepwise process first involving achieving a p value of <.2 in univariable analysis, achieving a p value of <.05 in univariable analysis, and consideration of hazard ratio size. One variable was eligible for inclusion in the multivariable analysis for every seven students included in the failure group of the analysis. A predictor was deemed statistically significant based on a p value of <.05 on multivariable analysis. This study was approved by the St. Louis College of Pharmacy Institutional Review Board.

RESULTS

A total of 669 students entered the PharmD program during the specified academic years and were included in the final cohort. Twenty-eight students (4.2%) failed one or more APPEs and 641 students (95.8%) did not fail an APPE. For the secondary evaluation, an additional 53 students received one or more Cs on an APPE, yielding 81 students (12.1%) who were considered poor performers.

Twenty-eight students failed a combined 32 APPEs during the evaluation, with four students failing more than one experience. The failures occurred primarily on APPEs in acute care/general medicine (31.3%), patient care selective (21.9%), and ambulatory care (18.8%). The remaining failures occurred on health system management and community care, which resulted in approximately 12.5% and 9.4% of total failures, respectively. Only 6.3% of all APPE failures occurred on elective APPEs. The failure cohort had a mean undergraduate GPA of 2.93 (SD=.3) and mean professional GPA prior to APPEs of 2.67 (SD=0.4), while the non-failure cohort had a mean undergraduate GPA of 3.18 (SD=.4) and mean professional GPA prior to APPEs of 3.10 (SD=.4). All baseline characteristics were found to be significantly different with the exception of calculation proficiency examination failures, academic honor code violations, and didactic course failures in Therapeutics 1, Pharmacology 2, and Medicinal Chemistry 1 & 2 (Table 1). The most common characteristics of students in the failure group included professionalism issues on IPPEs (78.6%), one or more pharmacotherapy course failures (72.4%), and a professional GPA <2.7 (60.7%).

Table 1.

Analysis of Predictors of Pharmacy Students’ Failure on Advanced Pharmacy Practice Experiences

Similar baseline characteristic differences were noted between the poor performer and non-poor performer cohorts. The poor performer cohort had a lower mean undergraduate GPA (2.95, SD=.4) and lower mean professional GPA prior to APPEs (2.75, SD=.4) than the non-poor performer cohort (3.20, SD=.4 and 3.13, SD=.4, respectively). The most common characteristics present in the poor performer cohort again included professionalism issues on IPPEs (67.9% vs 32%; p<.001), a pharmacotherapy course failure (52.5% vs. 21.9%; p<.001), and professional GPA less than 2.7 (49.4% vs. 16.3%; p<.001), and these occurred with increased frequency compared to the non-poor performer cohort. Individual didactic course failures in Therapeutics 1, Pharmacology 1, Pharmacology 2, and both medicinal chemistry courses were without significant differences between the cohorts. Though not significant, the number of honor code violations was higher in the poor performer cohort (20.1% vs 13.6%; p=.08).

Univariable analysis was performed for predictors of APPE failure on all pre-specified exploratory predictors (Table 1). Calculation proficiency examination failures, academic honor code violations, Therapeutics 1, Pharmacology 2, and Medicinal Chemistry 1 and 2 course failures were each not significantly associated with failure on APPEs. The variables with the strongest association with APPE failure included professional GPA <2.5 (hazard ratio (HR)=5.3 [95% CI 2.3-12.3]), professional GPA <2.7 (HR 6.6 [95% CI 3.1-14.2]), practicum failure (HR 4.4 [95% CI 1.7-11.4]), professionalism issues on IPPEs (HR 7.3 [95% CI 2.9-18.2]), Therapeutics 2 course failure (HR 6.5 [95% CI 2.4-15.2]), Therapeutics 3 course failure (HR 8.5 [95% CI 3.6-20.2]), Therapeutics 4 course failure (HR 7.3 [95% CI 3.3-16.4]), and composite of pharmacotherapy course failure (HR 9.0 [95% CI 3.9-20.6]).

Based on an APPE failure cohort of 28 students, using the a priori determined criteria and in an effort to avoid duplication for related variables, the variables included in the multivariable analysis were professional GPA <2.7, practicum failure, professionalism issues on IPPEs, and composite of pharmacotherapy course failure (Table 2). After multivariable regression, professionalism issues on IPPEs (HR 4.8 [95% CI 1.9-12.4]) and pharmacotherapy course failure (HR 4.2 [95% CI 1.6-11.1]) were associated with APPE failure.

Table 2.

Multivariable Regression Analysis of Pharmacy Students’ Failure on Advanced Pharmacy Practice Experiences

On the prespecified secondary analysis of poor performance, univariable analysis found similar associations with the exploratory variables. The same variables of professional GPA less than 2.7, practicum failure, professionalism issues on IPPEs, and composite of pharmacotherapy course failure were included in the multivariable analysis for poor performance (Table 3). After multivariable regression, professional GPA less than 2.7 (HR 2.7 [95% CI 1.5-5.0]), professionalism issues on IPPEs (HR 3.9 [95% CI 2.2-6.9]) and pharmacotherapy course failure (HR 2.0 [95% CI 1.1-3.7]) were associated with APPE poor performance.

Table 3.

Multivariable Regression Analysis of Pharmacy Students’ Poor Performance on Advanced Pharmacy Practice Experiences

DISCUSSION

In the present study, poor academic performance and having professionalism issues while completing IPPEs were associated with failure and poor performance on APPEs, with professionalism issues while completing an IPPE having the highest association with both failure and poor performance. Notably, students who did not perform well in the didactic curriculum were also identified as at risk of not succeeding on APPEs. Failing a pharmacotherapy course was found to be a significant academic performance predictor for both outcomes, particularly failure of an APPE, while a professional GPA <2.7 was identified to be a significant predictor for poor performance on an APPE.

Maintaining a cumulative GPA of ≥2.0 is the most common criteria used to determine progression at 80% of pharmacy schools that had progression policies viewable online.²⁹ At the time of the present study, St. Louis College of Pharmacy also required a student to have a GPA of 2.0 to progress to APPEs. Although many pharmacy schools have set a GPA of ≥2.0 as a requirement, inconsistent data exist to support this particular cut point. A study by Heldenbrand and colleagues that was conducted in a large four-year pharmacy program found that pre-pharmacy GPA (r=0.20, p<.001) and pharmacy school GPA (r=0.17; p<.001) were positively correlated with APPE performance.³⁰ Additionally, when compared to students with a pharmacy GPA >3.5, a negative correlation was observed with APPE scores and a pharmacy GPA of 3.0-3.5 or a pharmacy GPA of 2.6-3.0. A pharmacy GPA less than 2.6 did not reach statistical significance (β=-0.06; p=.09); however, this could be related to including only a few students in this cohort. The study was also unable to assess for factors associated specifically with APPE failure because of the low number of practice experience failures during the study period. The present study validated professional GPA as a potential predictor of performance on APPEs. This three-year cohort included 28 students identified as having received a failing grade on an APPE and 81 students deemed poor performers on APPE, providing sufficient sample size to assess the association of specific GPA cut points with these outcomes. These particular findings provide greater justification for pharmacy programs to use GPA as a criterion to determine “APPE readiness.”

There have been conflicting results regarding other variables (eg, PCAT scores and undergraduate GPA).^30,31 McLaughlin and colleagues found no correlation between undergraduate GPA or admission interview scores and APPE performance, though a significant relationship was noted between PCAT scores and APPEs, specifically ambulatory care (r=0.14 p<.05) and clinical specialty APPEs (r=0.15, p<.01).³¹ Additionally, OSCE performance prior to APPEs appeared to be associated with poor performance on certain APPE types. The mean APPE scores for low OSCE performers (<1 standard deviation below the mean on the final OSCE) were lower than those for the rest of the students, and significant differences were seen on acute care (p<.01), ambulatory care (p<.01), and community (p=.046) APPEs. However, only performance on APPEs was assessed, and data regarding students who failed was not provided. The present study, which assessed practicum failures, did not observe an association with poor performance on an APPE or with APPE failure when taking into account other markers of academic performance and professionalism. While poor performance on OSCEs or practicums may be associated with poor performance on APPEs, the present study argues against the use of this variable and favors alternative criteria as a determinant for progression.

Professionalism is a relatively undocumented variable for predicting APPE performance in the pharmacy literature. One study observed a positive correlation between overall APPE scores and the multiple-mini interview (MMI) scores from the admission process for pharmacy school (r=0.14; p<.01). The MMI scores had a significant positive correlation with several individual APPE domains (eg, patient care, documentation, drug information/evidence-based medicine, public health, and communication), but was not correlated with the APPE professionalism domain.³⁰ In the present study, behavior deemed not professional during IPPEs had the highest association with both failure and poor performance on an APPE. Interestingly, academic honor code violations were not found to be associated with poor performance or failure on an APPE. Academic honor code violations rely on voluntary submission of acts identified as breaching this policy. It is plausible that incidents of minor professionalism violations may not consistently be submitted, resulting in only “severe” unprofessionalism (eg, plagiarism) being captured, which may have limited those particular results. Overall, the data support tracking professionalism issues in colleges of pharmacy, particularly as part of IPPEs. These measures may help to identify students exhibiting unprofessional behavior and allow for colleges of pharmacy to help improve necessary professional abilities prior to APPEs.

Some results of this study may prove challenging to apply to other colleges of pharmacy as academic curricular structures may vary. Problem-based learning (PBL) is a teaching method associated with student perception of preparedness for APPEs and can be a method to identify students at risk for poor performance on APPEs.^32,33 The skills developed through PBL are generally assessed on clinical patient care rotations (eg, acute care/general medicine, patient care selective, ambulatory care, etc.), and these rotation types had the highest rates of failure in the present study. At the time of this study, this learning method was not incorporated as part of the curriculum at St. Louis College of Pharmacy, which may make extrapolation of results more difficult for programs using PBL throughout their curriculum. The curriculum shell is another factor that may limit external validity. During the study period, pathophysiology, pharmacology, medicinal chemistry, and therapeutics were all standalone courses. Many schools of pharmacy, including St. Louis College of Pharmacy, have an integrated curriculum organized by organ system rather than discipline. For example, Integrated Pharmacotherapy-Cardiology encompasses the pathophysiology, pharmacology, medicinal chemistry, and therapeutics of cardiovascular disease. While the results of this study may suggest that a student’s failure to successfully complete one disease-state module might predict APPE failure later in their academic career, this relationship has not been specifically evaluated. Additionally, the majority of professionalism issues in IPPEs were identified during longitudinal IPPE courses that spanned across an academic year and consisting of only 20 contact hours per course. The structure and oversight of these longitudinal IPPEs may create unique opportunities for St. Louis College of Pharmacy to systematically and consistently identify unprofessional behavior in our students compared to that ability in other schools. Finally, the investigators were unable to derive what component of the APPE grading criteria the student failed (eg, thinking and decision making, communication, professionalism, or a combination). Preceptor demographics such as years of professional experience, preceptor training and experience, and number of students precepted were also unknown.³⁴ While performance criteria were used to evaluate student performance and calculate grades, some subjectivity and inter-rater variability may have occurred.

CONCLUSION

This is the largest study conducted to date that focused solely on APPE failures. While other studies have suggested correlation between APPE performance with prior academic performance measures (eg, PCAT, undergraduate or pharmacy GPAs), this study uniquely identified professionalism issues occurring during IPPEs as a predictor of both a student’s poor performance in and failure of APPEs. Additionally, this study identified pharmacotherapy course failure and a professional GPA <2.7 as being associated with APPE performance. While failing a pharmacotherapy course was found to be a better academic performance marker for failure of an APPE, a professional GPA <2.7 may be a better academic marker for poor performance on an APPE. To ensure student success, interventions should be aimed at those identified as exhibiting behavior deemed unprofessional or poor academic performance prior to APPEs. Given that academic curricula differ among schools and colleges of pharmacy, each program should conduct their own internal evaluation to determine which specific factors have the strongest correlation with their students’ APPE performance and use this data to determine students’ APPE readiness.