Association Between NAPLEX Preparation Program Characteristics and First-Time Pass Rates

Abstract

Objective. The objectives of this study were to (1) describe characteristics of preparation programs for the North American Pharmacist Licensure Examination (NAPLEX) that are currently used by colleges and schools of pharmacy and (2) evaluate these program characteristics in relation to first-attempt NAPLEX pass rates.

Methods. This cross-sectional study was based on an online survey administered between February and March 2020. Assessment leads from 143 PharmD programs were invited to answer questions on their schools’ PharmD program characteristics and various aspects of NAPLEX preparation programs. The study included regression analyses to investigate associations between the NAPLEX first-attempt pass rates and PharmD demographic characteristics as well as between first-attempt pass rates and various aspects of the NAPLEX preparation programs. Finally, common themes from open-ended questions were identified.

Results. Fifty-eight participants completed the survey out of 132 successfully delivered email invitations (response rate = 44%). Fifty participants (86%) indicated that their PharmD program offers a NAPLEX preparation program. Our data indicate that offering a NAPLEX preparation program was not significantly associated with higher NAPLEX first-attempt pass rates. The analysis identified possible explanations for this lack of association, including student concerns with balancing a preparation program with advanced pharmacy practice experiences (APPEs) and the faculty workload associated with delivering such programs.

Conclusion. The current findings show no association between offering a NAPLEX preparation program and NAPLEX first-attempt pass rates. Future research should continue to examine the impact of these programs on individual school pass rates and factors that may enhance student motivation to engage in these programs.

INTRODUCTION

The North American Pharmacist Licensure Examination (NAPLEX), developed and administered by the National Association of Boards of Pharmacy (NABP), is used to assess a graduate’s competence to practice as a licensed pharmacist. ¹ Performance on the NAPLEX is evaluated as part of the Accreditation Council for Pharmacy Education (ACPE) accreditation standards; NAPLEX performance is mentioned in guidance documents for both Standard 15 (Academic Environment) and Standard 24 (Assessment Elements for Section I: Educational Outcomes) as required information for colleges and schools of pharmacy (C/SOP) to assess and provide to students. ² In order to fulfill Department of Education requirements, ACPE ensures that each Doctor of Pharmacy program annually posts its most recent first-attempt pass rate for the NAPLEX to its website. As the number of schools of pharmacy has increased since 2000, there is increasing competition for students. A program’s NAPLEX pass rates may be an important indicator for prospective students seeking to compare programs.

First-attempt NAPLEX pass rates have recently become a serious concern for colleges and schools of pharmacy, as national pass rates began a downward trend in 2014, with the most significant drop occurring between 2015 and 2016. ^{3, 4} The NAPLEX competency statements were revised in 2015; in 2016, the examination length was increased from 4.25 hours to 6 hours with an increase from 185 to 250 questions. ³ At the same time, the model changed from computer adaptive to preassembled, with set percentages of questions varying in difficulty. The greatest change in the first-attempt pass rate occurred between 2015 and 2016, perhaps corresponding with the changes to the examination, hitting a nadir at 85.86%. ³ Since that time, the mean national first-attempt NAPLEX passing rates have improved (89.46%, 88.34%, and 88.43% in 2018, 2019, and 2020, respectively). ⁵ NAPLEX competency statements are reviewed every five years to reflect licensure best practices and changes in pharmacy practice. As of January 2021, the blueprint for the NAPLEX has expanded from two competency areas to six, with the largest portion of the examination (35%) devoted to developing or managing treatment plans. ⁶ Short-term fluctuations in pass rates may be anticipated in response to these changes.

Following the decline in nationwide pass rates, considerable interest has arisen in identifying characteristics that are associated with students’ success on the examination. While early studies focused on individual schools, a study published in 2019 by Williams and colleagues sought to identify correlations between school characteristics and NAPLEX pass rates, using data from all US pharmacy schools accredited as of July 2019. ³ That study concluded that higher first-attempt pass rates were significantly correlated with pharmacy schools that resided within an academic health center, that were established prior to 2000, that had a public (rather than private) structure, and that had a four-year (rather than an accelerated three-year) curriculum. Additionally, a positive correlation emerged between the percentage of the total graduating class that matched for postgraduate year 1 (PGY-1) residencies and NAPLEX passing rates.

A 2017 study by Lebovitz and colleagues reported that approximately 80% of pharmacy schools offer some type of NAPLEX review program. ⁷ Common descriptions of the review programs included question banks developed by a vendor (n=45, 73.9%), live reviews provided by a vendor (n=34, 55.7%), and live reviews provided by local faculty (n = 28, 46%). The most commonly identified motivator for providing a review was to offer a service to students, though private pharmacy schools were more likely than public pharmacy schools to identify as a motivator poor historical performance on the NAPLEX (χ ² = 8.15, p=.004). For pharmacy schools considering whether to use NAPLEX preparation programs, they must evaluate the effectiveness of the program as well as the associated costs, including but not limited to the costs of the program, faculty salary, and space allocation.

The effectiveness of commercial test preparation has been evaluated in both the medical and pharmacy literature. ^8-12 As with pharmacy schools, some medical schools offer, require, and/or recommend commercial and school-sponsored preparation programs. Several reviews of the impact of preparation programs in medicine indicate no significant differences in United States Medical Licensing Examination (USMLE) Step 1 scores between students who had commercial coaching versus those who prepared on their own. ^{8, 9, 12} Similar results have been seen in pharmacy schools, where the use of no single examination preparation tool has been associated with higher examination pass rates, but some products are favored over others by students. ¹¹

In a study by Zhang and colleagues that evaluated student perceptions of the preparation methods, students reported that the two most significant factors in successful Step examination preparation were personal learning habits and advice from other medical students. ⁹ All students who had participated in a commercial test preparation program indicated that having an organized study schedule was important, while only 59% of those who did not participate in a formal test preparation program identified the importance of an organized study plan. More surprisingly, participants rated commercial test preparation programs highly but did not demonstrate higher performance levels. Most students reported that their curriculum prepared them for the examination. Based on an inverse relationship in perceived course quality and perceived quality of the corresponding segment of the preparation course, the authors postulated that participants joined preparation courses to complement content areas within their curriculum that they felt were weaker.

Although a systematic review of previous studies evaluating test preparation was attempted, it proved difficult because of limitations in these studies; for example, most did not have control groups. ¹² Further, selection bias was common since enrolled students were self-selected. Also, most studies did not use prospective randomized controlled trials, provided few details about the commercial educational interventions, and lacked cost-benefit analyses. ¹² Some studies only evaluated student perceptions of perceived utility. ¹² The review also noted a trend in marketing strategies, which focused on student perceptions rather than the impact of verifiable interventions.

The purpose of this study was to evaluate, in greater detail, the NAPLEX preparation programs currently used by pharmacy schools and to investigate associations between the programs’ characteristics and first-attempt NAPLEX pass rates. Though test preparation programs have been evaluated in relation to the traits of the pharmacy schools offering them, the specific placement, timing, and structure of these programs have not been fully evaluated in the pharmacy literature to date. The current study sought to provide details related to program administration and resources so that stakeholders might be better informed about the NAPLEX preparation structure currently used by pharmacy schools.

METHODS

In this cross-sectional study, the primary tool for data collection was a self-administered survey. The survey included highly structured quantitative as well as qualitative items. Participants were recruited via email invitations that included a link to access the online survey. Qualtrics software (Qualtrics International Inc) was the platform used for data collection.

The survey consisted of 36 items that were developed based on previous research. ^{3, 7-12} The survey items measured a series of program characteristics, including the number of graduated classes (0-5, 6-10, 11-25, 26-50, and more than 50), the typical number of students in each class (0-50, 51-75, 76-100, 101-150, 151-200, and more than 200), and the accreditation status (precandidate, candidate, and fully accredited). Additionally, the survey assessed multiple aspects related to the NAPLEX preparation programs, including development, coordination, implementation timeline, updating, grading, use of practice examinations, and use of third-party resources such as Exam Master (Exam Master Corp), Pass NAPLEX Now (Pass NAPLEX Now Inc), or RxPrep (RxPrep Inc).

The survey was sent to one individual from each of the 143 accredited or candidate PharmD programs in the United States with an invitation to take part in the survey. The contacted individuals were identified using a list of program assessment leads, mostly directors or assistant/associate deans for assessment, maintained by the Assessment Special Interest Group (SIG) of the American Association of Colleges of Pharmacy (AACP). Recipients were asked to suggest an alternative responder from their school if they believed someone else in their program had more extensive knowledge of NAPLEX preparation.

Data collection began in February 2020, and the survey was open for four weeks. Two reminders, approximately one week apart, were sent to individuals who did not complete the survey. Additional reminders were not sent due to the COVID-19 pandemic. After data were collected from the survey, we manually inputted the 2019 NAPLEX first-attempt pass rates published by NABP ⁵ for each participating school to evaluate associations between NAPLEX preparation program characteristics and pass rates.

After the basic characteristics of the participating schools were collected, the associations between specific school characteristics and NAPLEX first-attempt pass rates were examined either with point biserial or Spearman rank correlation analyses, as applicable. Then, two linear regression analyses were conducted to investigate the adjusted coefficients and confidence intervals related to the associations between the NAPLEX first-attempt pass rates and selected variables. The first regression analysis examined the association of NAPLEX first-attempt pass rates with offering a NAPLEX preparation program while accounting for program demographic characteristics (school type [public vs private]; the number of classes the program graduated; the average number of students in each class). The second regression analysis investigated the association of NAPLEX first-attempt pass rates with various aspects of the NAPLEX preparation programs offered by the pharmacy schools. Variables included in this analysis were whether the school required the preparation program for every student, use of the pre-NAPLEX examination, use of a third-party resource, use of a “home-grown” preparation program, and demographic characteristics of the program as listed above. Variables with substantial missing values or those measured qualitatively were excluded from the regression analyses. Excluding these variables is not expected to impact the results due to the great variability in how the preparation programs are assessed and implemented. Significance was determined at α = 0.05.

We also conducted a qualitative analysis of the challenges faced by pharmacy schools that attempted to implement a NAPLEX preparation program. The responses were reviewed by two investigators and coded based on themes identified de novo. The codes were compared to ensure consistency, and discrepancies were discussed. Frequencies of each code were recorded. The study was deemed exempt by the institutional review board (IRB) at Nova Southeastern University.

RESULTS

A total of 143 emails were sent, and 11 were returned as undeliverable. Fifty-eight of 132 participants completed the survey (response rate=44%). This response rate represented 41% of accredited and candidate PharmD programs. The participating programs covered all regions in the United States, with the largest representation from the Midwest (Table 1). There was similar representation of private and public institutions. Fifty-seven programs (98%) were fully accredited, with the majority having graduated more than 50 classes. There was good distribution of programs by class size. The mean for the 2019 NABP-published NAPLEX first-attempt pass rates for the participating schools was 88%, whereas the median was 91%.

Table 1.

Participating Programs’ Institutional Characteristics

Fifty participants (86%) indicated that their pharmacy school offers a NAPLEX preparation program. Of these, 71% were required for all students, and 56% updated their program in response to the changes in the NAPLEX that took effect in 2016. Programs were most often coordinated by academic affairs or assessment-related offices (74%), experiential education–related offices (39%), and/or faculty (30%). Programs took place during the advanced pharmacy practice experience (APPE) year (80%), after the APPE concluded but before graduation (52%), and/or after graduation (15%). Forty-eight percent of preparation programs resided within the PharmD curriculum. Student performance was commonly graded (objectively in 50% of programs, subjectively in 20%), but in 45% of the programs, performance was ungraded or checked only to ascertain completion (totals may not add to 100% as respondents could select multiple grading options).

About 33% of programs reported using the pre-NAPLEX examination, with 60% of programs administering the examination in a proctored setting. About 78% of the programs used a third-party resource to provide their NAPLEX preparation program; of these, the most common products used were RxPrep (n=30) and Exam Master (n=7). Of those pharmacy schools administering a third-party resource’s practice examination, 67% did so in a proctored setting. At the time of the survey, 46% of the review courses were delivered in a live format, 26% blended, 14% asynchronous, and 14% other or a combination.

The bivariate association analyses results revealed that the 2019 NAPLEX first-attempt pass rates were significantly lower for pharmacy schools that required the NAPLEX preparation program for every student, for programs that used a third-party resource (eg, Exam Master, Pass NAPLEX Now, RxPrep), for private pharmacy schools, and for pharmacy schools that graduated fewer classes. More details on the bivariate analyses results are presented in Table 2.

Table 2.

Bivariate Association Analyses Between the 2019 NAPLEX First-Attempt Pass Rates and PharmD Program Characteristics

The first linear regression analysis assessed the associations of program demographic characteristics with NAPLEX first-attempt pass rates. Results of this analysis demonstrated that the primary independent variable (offering a NAPLEX preparation program) was not significantly associated with the NAPLEX first-attempt pass rate (Table 3). However, the number of classes that the school graduated was positively associated with it, such that each one-level higher score in the number of graduated classes was associated with a 2% increase in the NAPLEX first-attempt pass rate.

Table 3.

Linear Regression of 2019 NAPLEX First-Attempt Pass Rates on PharmD Demographic Characteristics

In terms of the measured aspects of the NAPLEX preparation programs, the second linear regression analysis revealed that the number of classes that a PharmD program had graduated continued to be the only significant variable associated with NAPLEX first-attempt pass rates. Each one-level higher score in the number of graduated classes was again associated with a 2% increase in the NAPLEX first-attempt pass rate. More information about the outcomes of this linear regression is presented in Table 4.

Table 4.

Linear Regression of 2019 NAPLEX First-Attempt Pass Rates on Various Aspects of the NAPLEX Preparation Programs and PharmD Program Characteristics

Thirty-eight participants commented on the challenges they experienced with their NAPLEX preparation programs. The most frequent challenge identified was a lack of student motivation in participating and/or completing a preparedness program during their APPE (22 responses), as students’ priorities were either on their APPEs or they had difficulties devoting dedicated time for a preparation program. The next most frequent challenge (seven responses) was scheduling, regarding whether the program took place during the APPE or after completion of the APPE (but before graduation). The final two common themes (each receiving four responses) involved perceptions about whether the preparation program makes a difference and concerns about the time commitments associated with a preparedness program for faculty, staff, and the administration.

DISCUSSION

Overall, this study found that the number of graduated classes by a pharmacy school was the only significant predictor of NAPLEX first-attempt pass rates, while controlling for other program characteristics and characteristics of the NAPLEX preparation program itself. This is the first study to our knowledge that characterizes in detail the NAPLEX preparation programs offered by pharmacy schools. Responses to the survey indicate that the structure, timing, and resources used to deliver such programs vary across the Academy. Comments by survey respondents indicated that engaging students in these preparation programs tends to be challenging while they are busy participating in APPEs.

The number of graduated classes, or, in other words, how established a program is, has been correlated with higher NAPLEX pass rates in prior studies. ^{3, 7} Lebovitz and colleagues found that legacy programs (defined as schools founded before 1995) had significantly higher pass rates than more recently established schools (93% vs. 91%, p = .031). ⁷ Williams and colleagues similarly found that NAPLEX pass rates between 2014 and 2016 were significantly higher in schools established before rather than after 2000. ³ In our study, the number of graduated classes was a significant predictor for the NAPLEX pass rate, whether controlling for other characteristics of the NAPLEX preparation program or not. A linear increase was found in the number of graduated classes and NAPLEX pass rates, with an approximate 10% improvement in first-attempt pass rates between schools graduating 0 to 5 classes (mean pass rate = 82%) and those graduating more than 50 classes (mean pass rate = 91%). It is unknown why more established programs may have higher NAPLEX pass rates. In this study, we did not have access to student-level admission data, such as GPAs or PCAT scores, so it is difficult to determine whether certain programs are recruiting academically stronger students. There also may be some influence of a program’s ability to consistently prepare students for success on the NAPLEX, as evidenced by dependably high pass rates across varying years and cohorts. Previous research has shown that a program’s prior years’ NAPLEX pass rates can be a predictor of performance in the following year. ³

According to our survey, the most significant challenge identified by respondents was students’ lack of motivation to participate in or complete prep programs. Previous research on standardized examinations (eg, PCOA or NAPEX) has shown that student motivation has been a consistent and major factor in determining performance. ¹³ Depending on how a school implements these types of preparation programs, performance may be low stakes, which may lead to diminished intrinsic motivation. The next most frequent challenge was uncertainty about when to schedule a program during the curriculum and/or dedicating time for NAPLEX preparation during APPEs. As seen with PCOA implementation and use, ¹⁴ a variety of techniques and considerations exist for offering these programs. These programs could be curricular or cocurricular, for course credit or not, or they could be required for all students or only poor-performing students. To determine the best administration of these offerings, faculty should dedicate time, energy, and effort into developing a program that fits with their school. Given the increased demand on faculty time and the potential for burnout, this may be problematic or not feasible. Finally, limited research has been published on these programs and their impact on NAPLEX pass rates.

This study has several limitations. First, not all schools responded to the survey, so the findings may not be generalizable across all schools. However, respondents were equally distributed between private (52%) and public (48%) schools, which is similar to the national landscape (51% private, 49% public). ¹⁵ The majority of respondents to the survey had a class size between 76 and 100 (19%) and 101 to 150 (19%), which is similar to the mean class size of 103 in the fall 2020 Profile of Pharmacy Students. ¹⁶ The mean NAPLEX first-attempt pass rate for our sample was 88%, which is similar to the 2019 (88.3%) and 2020 (88.4%) national pass rates published by NABP. ⁵ While the survey response rate was less than desirable, the results are likely representative of the Academy.

There was a wide variety in the type and structure of NAPLEX preparation programs used by each school or college of pharmacy, so the intervention was not the same among programs that responded to the survey. However, when performing regression analyses, we attempted to control for demographic differences among programs that have been shown to impact NAPLEX pass rates (ie, number of graduated classes). Further, this study involved a retrospective review of NAPLEX pass rates, so temporal relationships between the use or change of NAPLEX preparation programs and the impact on school pass rates are difficult to determine. Also, the challenges expressed by schools in administering NAPLEX preparation programs were only representative of faculty or administrator opinions and, thus, did not capture students’ perceptions of these programs. Finally, as this study did not collect individual student-level study habits or adherence to the study programs offered by each school, we are unable to determine whether these factors may have impacted a program’s pass rates above and beyond the offer of the program itself.

Schools and colleges of pharmacy may want to consider the results of this study in planning for NAPLEX preparation programs offered at their institutions. Schools should investigate the impact of NAPLEX preparation programs on first-attempt pass rates at their institution with student-level data regarding the amount and effectiveness of study time associated with them. Programs may use these data in conjunction with other potential predictors of NAPLEX performance (eg, PCOA scores, program GPAs) to determine whether student performance within NAPLEX preparation programs may predict NAPLEX pass rates above and beyond existing academic measures. Programs should also evaluate whether all students, or only those at high risk of NAPLEX failure, should receive these resources. Programs should be structured to minimize impact on students during APPEs, while they should still foster students’ intrinsic motivation to study. Involving recent alumni or students in the planning process for these programs may help overcome challenges with deploying such programs during the APPE year and may help students balance examination preparation with other responsibilities.

CONCLUSION

Overall, we found no association between offering a NAPLEX preparation program and first-attempt NAPLEX pass rates. This study identified a wide variety of types and methods of implementation of NAPLEX preparation programs. Established schools and colleges of pharmacy that had graduated more than 50 classes showed a significantly higher NAPLEX pass rate. A common concern in delivering such programs was the lack of student motivation to actively participate in the program. Future research should explore strategies to enhance student motivation to engage in NAPLEX preparation programs and evaluate the benefits such programs provide to students and colleges/schools of pharmacy.