Effect of a Commercial NCLEX-RN Preparation Product on First Time Pass Rates

Abstract

Introduction

Passing the National Council Licensure Exam-Registered Nurse (NCLEX-RN) is the entry into the profession for nurses, and rates of passing the exam on the first attempt are used as an important outcome measure for schools, students, and accrediting agencies.

Objective

This study was conducted in order to investigate if a commercial NCLEX-RN test preparation product impacted first time pass rates.

Methods

Data were collected from 688 students from six institutions who were taking the NCLEX-RN for the first time. The intervention group (n = 396) received a 6-month subscription to a commercial product with which to prepare for the NCLEX-RN, and was compared to the control group (n = 292) to test for differences in demographics and test pass rates.

Results

The intervention group was more likely to be younger than 25 years of age (n = 160, 55.8% vs. n = 76, 34.6%; p < .001) when compared with the control group. No other significant differences in demographics or NCLEX-RN pass rates were observed between the groups.

Conclusion and Recommendations

This study failed to demonstrate a change in the likelihood of passing NCLEX-RN on the first attempt in a group of students who used a commercial test preparation product. We recommend that instructor-guided use of a test-preparation product, along with other interventions to support test success, be considered in creating an individualized plan for nursing students preparing for NCLEX-RN.

Introduction/Background

It is important for nursing school graduates to pass the licensure exam, ideally on the first try. First time passing rates are scrutinized by accrediting agencies and programs can lose accreditation from their state board of nursing and others if their pass rates go below a benchmark. First time pass rates are also a mark of quality that potential students consider when choosing a program.

When an individual fails the licensure exam on the first try, there are personal and professional consequences. The individual experiences the expense of paying for the exam again, as well as a delay in starting their first professional nursing job and earning the associated salary (Glasgow et al., 2019). Additionally, the odds of passing on second or subsequent attempts are significantly lower than the odds of passing on the first try (National Council of State Boards of Nursing [NCSBN], n.d.). The impact on the nursing profession is felt when there is a delay or even loss of a nurse for the workforce, which is currently experiencing significant shortages (American Association of Colleges of Nursing [AACN], 2023).

The National Council of State Boards of Nursing (NCSBN) develops the National Council Licensure Exam for Registered Nurse (NCLEX-RN), revising it regularly based on analyses of current professional nursing practice (NCSBN, n.d.). In 2023, the NCLEX-RN has undergone numerous changes, which the NCSBN has called “NextGen NCLEX.” The NextGen NCLEX represents significant changes from the prior exam, including a variety of new and more challenging item types, and a general focus on clinical judgment which is in alignment with current nursing practice and is a more rigorous standard than prior versions of the exam (NCSBN, n.d.). It is typical for first time pass rates to drop immediately after the NCLEX-RN changes or increases in rigor (Serembus, 2016). This NextGen NCLEX has created a heightened concern for nursing faculty and nursing students, with specific concern about how to prepare for the exam.

There are many components that contribute to nursing school graduates being prepared and able to pass NCLEX-RN, including program policies and practices, and studying that students do outside of school. Nursing program admission policies may require evidence of academic achievement in the form of grade point average and scores on entrance exams. Progression policies typically require students to maintain high grades or risk repeating a course or being dismissed from the program (Lewis et al., 2021). Program policies are often developed while considering the evidence that associates academic achievement before and during nursing school with first time success on the NCLEX-RN.

Nursing program practices can support student preparation for NCLEX-RN. For example, testing with questions that mimic NCLEX-RN in terms of content, rigor, and level on Bloom's taxonomy give students practice for the licensure exam. Testing throughout a program, with a commercial product that offers standardized tests that are nationally benchmarked, can also be helpful (Collaborative Momentum, 2018). Another practice that can help students prepare for NCLEX-RN is assigning and giving academic credit for study activities, such as remediation and review of practice exams (Conklin & Cutright, 2019).

After graduation, most students engage in focused preparation for NCLEX-RN. This preparation ranges from live review classes and structured, supervised study plans, to independent student review of class notes and books of practice questions (Conklin & Cutright, 2019). There are numerous commercial products for NCLEX-RN preparation, with new products coming on the market each year. Some products are entirely self-paced, while others provide guidance to the student in the form of a virtual coach or class. Additionally, some products include elements that may be integrated into the nursing school curriculum, with content area exams and study materials (Stuckey & Wright, 2021).

Most commercial products have only internal market research to support their claims of guiding students to NCLEX-RN success. A few products have peer-reviewed evidence to correlate scores on their assessments with NCLEX-RN success. Schools and students combine teacher-made or student-made resources with commercial products, further confusing the issue (Thompson et al., 2023).

Review of Literature

There is minimal evidence in the literature of best practices or interventions to prepare students to pass NCLEX-RN. Researchers who have written about successful programs have typically used a collection of interventions, often including a commercial test preparation product (Quinn et al., 2018; Thompson et al., 2023). There is no consensus or even pattern of success with any individual product.

There are several examples in the literature of nursing schools that implemented a variety of actions to support increased first time pass rates on NCLEX-RN. Appreciative advising and emotional intelligence training (Opsahl et al., 2018), continuous quality improvement processes (Serembus, 2016), coaching for graduates (Schlairet & Rubenstein, 2019), and even curricular revision (Czekanski et al., 2018) are some of the interventions that schools have used with resultant improved NCLEX-RN pass rates. The majority of the literature about strategies to increase NCLEX-RN success also include the use of commercial test preparation products such as content exams for use during the nursing program, predictor exams for use at the end of the nursing program, and question banks and content review for studying (Conklin & Cutright, 2019; Thompson et al., 2023).

The market for NCLEX-RN test preparation products is large, including review books and flashcards, online programs that may be self-paced or guided by a coach, mobile applications, and in-person classes or tutors. New products become available each year. Only a few of these products have been studied and results published in peer-reviewed literature, such as products from Assessment Technologies Institute, Health Education Systems Incorporated, Kaplan, and Hurst. Many schools use more than one commercial test preparation product (Czekanski et al., 2018; Serembus, 2016).

Some schools of nursing purchase NCLEX-RN preparation products for their students from funds obtained from tuition or fees. Many students are obliged to purchase such products in addition to tuition and fees, and some students who are provided products by their school still choose to purchase different or additional products. There is a wide range in price for NCLEX-RN preparation products. An internet search in 2022 demonstrated that popular products with a high market share cost from $100 to $500, depending on the duration of the subscription and the variety of products included.

In a study focused on institutional characteristics that correlate with higher first time pass rates, Odom-Maryon et al. (2018) found that the use of commercially prepared standardized exams for admission or progression was not associated with increased NCLEX-RN success. Rather, these authors found increased NCLEX-RN success in public schools, schools with more full-time faculty, and schools that did not require a standardized admission test. While the postgraduation use of a commercial test preparation product may be valuable, products that are integrated with the curriculum or are used as an entrance exam may be less valuable.

There is no evidence to support one particular path to success on NCLEX-RN in terms of preparation. In one study of faculty perceptions, participants agreed that best practices included the use of commercial products for remediation and assessment of test readiness (Davis & Morrow, 2021). Nursing students and new graduates may have barriers to using these best practices due to the expense of NCLEX-RN preparation products.

Methods

A study was conducted to assess the impact of an NCLEX-RN test preparation product on first time pass rates on the NCLEX-RN-RN.

Research Design

The study was quasi-experimental, with the cohort of students graduating in December 2020 serving as a control group; which were subsequently compared to the cohort of students graduating in May 2021 who were provided a 6-month subscription to the NCLEX-RN preparation product for free.

Setting and Sample

Six-hundred and eighty-eight students participated in the study, representing a convenience sample of six prelicensure nursing programs: three associate degree programs at community colleges, and three generic baccalaureate programs at universities (Table 1). Power calculations were based on an anticipated sample size of 700 students. This sample size was selected to result in a 90% power to detect a small change in pass rate, a 4% change. The sample size of this study was 688 students, just under the anticipated sample size. As recommended by biostatisticians, a post hoc power analysis was not done (Dziadkowiec, 2021).

Table 1.

Institutions Included in the Study (N = 6).

	Type of institution	Type of degree offered	Setting and geographic region of U.S.
School A	Public community college	Associate degree	Small city, Southeast
School B	Public community college	Associate degree	Suburban, Midwest
School C	Private not-for-profit	Bachelor of science	Large city, Pacific
School D	Private not-for-profit	Bachelor of science	Large city, Mid-Atlantic
School E	Public community college	Associate degree	Large city, Southwestern
School F	Private not-for-profit	Bachelor of science	Rural town, Midwest

Information from National Center for Education Statistics College Navigator (https://nces.ed.gov/collegenavigator/).

Tools

The only instrument used in this study was the survey administered to the control group students. This survey consisted of three questions: “What school do you attend?” “What is your school identification number?” [used for school to remove the NCLEX-RN preparation product users from the control group], and “What NCLEX-RN test preparation product(s) have you used?”

Method of Data Collection

Data were collected from the institutions about demographics and NCLEX-RN pass rates for each cohort. Institutional level data was provided by the participating schools as files that were uploaded into a secure shared drive. Individual student level data was collected from the control group, via web-based electronic survey software, in the form of a survey asking if they had used the commercial NCLEX-RN preparation product. This allowed the researchers to determine if the control group had exposure to this product, which was the intervention for the experimental group.

The intervention was a commercially available NCLEX-RN preparation product. The intervention was selected based on researcher familiarity with the product, and the product's large market share in the United States. The product consists of a large question bank of practice test items, as well as practice exams designed to give the test-taker a probability of passing the NCLEX-RN. The product is entirely web-based.

Ethical Considerations

The institutional review board (IRB) of the author's institution declared the study exempt. Approval to conduct the study, and declaration that the study was exempt, was also received from the IRBs of the participating schools. Informed consent was obtained from all participants who participated in this survey. Participation in the survey was voluntary.

Statistical Analysis

Using aggregate data provided by the institutions, we tested for differences in participant demographics between groups, using chi-squared tests. Aggregate data was also used to test for differences in pass rates between groups. This analysis was done overall as well as separately for each school. Using participant-level data, we examined mean usage of the NCLEX-RN preparation product, in total time used over the semester, by institution, using an analysis of variance test.

Results

A total of 688 students, representing six institutions, participated in the study, with 292 in the control group and 396 in the intervention group. Unfortunately, NCLEX-RN passing rates for the entire intervention group was implausible because, almost 6 months after graduation (at the end of the study period), participating schools were unable to determine test results for 105 graduates (approximately 25%) due to delays in graduate testing and in reporting. Four of the six participating institutions provided demographic data (Table 2).

Table 2.

Demographics by Cohort (N = 2).

	Control group: December 2020 graduates	Intervention group: May 2021 graduates	χ2	p-Value
Gender: Female	214 (73.29%)	292 (74.30%)	0.09	.77
Race/ethnicity			4.17	.76
American Indian	4 (1.82%)	3 (1.03%)
Asian	10 (4.55%)	17 (5.86%)
Black	24 (10.91%)	34 (11.72%)
White	120 (54.55%)	167 (57.59%)
Hispanic	40 (18.18%)	42 (14.48%)
Non-United States citizen	10 (4.55%)	12 (4.14%)
Two or more	3 (1.36%)	1 (0.34%)
Unknown	9 (4.09%)	14 (4.83%)
Age group			22.50	<.001
Younger than 25	76 (34.55%)	160 (55.75%)
25 or older	144 (65.45%)	127 (44.25%)

No difference existed between the control and intervention groups by gender and race. Those in the intervention group, however, were more likely to be younger than 25 years of age (n = 160, 55.8% vs. n = 76, 34.6%; p < .001) when compared with the control group.

While, the rate of NCLEX-RN passing on the first attempt decreased in the intervention group compared to the control group for five out of the six institutions, the observed decrease was only statistically significant for one institution (see Table 3). One institution saw their NCLEX-RN pass rate increase from 84.8% to 91.3%, but the observed increase did not reach significance.

Table 3.

NCLEX-RN-RN Pass Rates by Cohort (N = 2), at Each School and Total Across Schools.

	Control group: December 2020 graduates			Intervention group: May 2021 graduates
	n	n passing	Pass rate	n	n passing	Pass rate	χ2	p-Value
School A	43	40	93.0%	58	44	75.9%	5.19	.02
School B	46	39	84.8%	46	42	91.3%	0.93	.34
School C	35	28	80.0%	39	31	79.5%	0.003	.96
School D	55	47	85.5%	63	53	84.1%	0.04	.84
School E	63	54	85.7%	55	43	78.2%	1.14	.29
School F	20	17	85.0%	30	24	80.0%	0.20	.65
Total	262	225	85.9%	291	237	81.4%	1.97	.16

Note. NCLEX-RN=National Council Licensure Exam-Registered Nurse.

Exposure to the intervention, the NCLEX-RN preparation product, for those in the control group was measured via a survey administered to the control group students. Participation in the survey was voluntary with a 51% response rate. More than half of those in the control (57%), who responded, used the product in their preparation for the NCLEX-RN, which is similar to the product's stated market share (Table 4).

Table 4.

Control Group Exposure to the Intervention (N = 262).

	Number of survey respondents/total number of students in cohort (percent responding)	Number of students using UWorld/number responding (percent using UWorld)
School A	24/43 (56%)	8/24 (33%)
School B	37/46 (80%)	21/37 (57%)
School C	21/35 (60%)	15/21 (71%)
School D	22/55 (40%)	19/22 (86%)
School E	17/63 (27%)	11/17 (65%)
School F	12/20 (60%)	2/12 (17%)
Total	133/262 (51%)	76/133 (57%)

The intervention group student usage of the NCLEX-RN preparation product was examined (Table 5). Overall, students in the study used the product for 44 h, 35 min on average. There was a significant difference in usage by school (F = 7.14, p < .001). Usage ranged greatly from little use (3 h, 56 min) to extended use (57 h and 11 min).

Table 5.

Intervention Group Usage by Site (N = 6).

Site	Usage	p-Value
		<.001
School A	21:28a(5:44)
School B	38:54b(5:02)
School C	57:11c(5:14)
School D	53:37bc(5:48)
School E	54:21c(5:14)
School F	3:56a(15:57)

Note. Usage reported in hours and minutes. Mean and standard error reported. Means having subscripts of different letters differed significantly in post hoc mean comparisons.

Discussion

It is necessary to first address the nonsignificance of the study results. According to Visentin et al. (2020), it is important to report nonsignificant findings to decrease the potential of publication bias and meta-analyses bias, and to prevent future researchers from wasting resources repeating the study. While the lack of significance in the NCLEX-RN pass rates of the experimental group could mean that the intervention has little effect, it is also possible that the variation within the groups was too large, or that there were too many confounding variables.

Throughout the duration of this study, the world, including the United States, was living through the COVID-19 pandemic. This health crisis triggered many changes that affected nursing education and practice. Some of these changes were:

• Abbreviated NCLEX-RN test, consisting of 85 to 145 questions, when prior to the pandemic it was 75 to 265 questions. The NCSBN has stated that the statistical and psychometric evaluations of the NCLEX-RN with fewer questions supports its reliability, however, the impact of this change on test takers and pass rates is still not fully understood. Nationally, NCLEX-RN pass rates have declined for first time test takers who were educated in the United States during the pandemic. During the first and second quarters of 2021, when the control group was taking NCLEX-RN, pass rates were 84.83% and 85.69%, respectively (compared to 85.9% observed in our sample). The third-quarter of 2021, when the intervention group was taking NCLEX-RN, the national pass rate declined to 81.10% (NCSBN, n.d.) (compared to 81.4% in our sample).

• Altered classroom experiences, with schools of nursing moving to remote/virtual delivery. At the onset of the pandemic, colleges and universities moved to online learning or, in some cases, canceled learning entirely. By fall of 2020, only about a quarter of schools had returned to fully face-to-face learning, with the remainder continuing with virtual learning (44%) or a hybrid model (21%) (Smalley, 2021).

• Altered clinical experiences, with limited opportunities for face-to-face clinical placements and laboratory experiences. At the start of the pandemic, most clinical agencies closed their doors to learners in attempt to conserve limited personal protective equipment and to mitigate risk of transmission of infection. Cancelations and delays in clinical experiences pushed nursing schools to provide increased virtual and simulated experiences for students (International Council of Nurses, n.d.).

• Introduction of a graduate registered nurse temporary permit in many states that did not have such a role prior to the pandemic. Many states, including several of the states where this study took place, responded to the pandemic by allowing new graduate nurses to begin practice under an emergency permit, prior to passing NCLEX-RN. Beginning the transition to nursing practice and working full time while preparing for NCLEX-RN posed a challenge for graduates who chose to begin working as a graduate nurse. It may have also resulted in more graduates delaying their NCLEX-RN-RN test date.

The NCLEX-RN pass rates for the intervention group were varied, with overall no statistically significant change in comparison to the control group with the exception of School A. School A saw a statistically significant decrease in the pass rate for the intervention group. Possible explanations for this outcome include pandemic changes, institutional attributes, student attributes, and student usage of the product.

Although small changes in pass rates may not be statistically significant, it is important to note that they may have practical significance. Schools of nursing jeopardize their accreditation if their NCLEX-RN pass rate falls below the benchmark set by the accrediting body; for example, a 79% pass rate results in probationary status for the program while an 80% pass rate does not. Many programs additionally benchmark to the national average passing rate, thus defining success as anything above that national average and failure as anything below. An example of practical significance in this study is School B. Their NCLEX-RN passing rate increased from approximately 85% to 91% with the use of the product. That change was not statistically significant, but was valuable to the school such that they decided to purchase an institutional membership so that all of their students have access to the product going forward.

Although both the control group and intervention group were affected by the pandemic changes, they may have been impacted differently. The control group started their nursing education in the spring or fall of 2019, while the intervention group started in either fall of 2019 or spring of 2020, thus experiencing more of their education during pandemic restrictions. The impact of virtual education has not been entirely positive, with a majority of students finding that it hindered their academic and emotional experience (McCoy, 2021). It is possible that the longer duration of virtual learning experienced by the intervention group impacted their ability to pass NCLEX-RN. The data from NCSBN showing decreased pass rates for this group supports this idea.

The selection of the sample for this study was a convenience sample of institutions that agreed to an invitation to participate. Comparing two graduating cohorts from the same school, from the same academic year, was intended to decrease the variability between cohorts. However, it is not known if the schools had other changes during the study that may have impacted the students. For example, one participating institution (School C) had numerous faculty resignations throughout the study, requiring the primary investigator to file addenda with that school's IRB twice to update the contact person for the study. It is possible that those personnel changes affected the intervention group in some way.

While students in sequential cohorts at the same institution should be from a similar population, there are known and possibly unknown differences between the groups. Analysis of the demographic data showed that the intervention group was statistically significantly younger (<25 years old) than the control group. There may have been other differences that were not captured in the data we collected.

Finally, this study was not designed with a prescribed type or amount of usage of the product for the intervention group. Usage varied greatly from one institution to another. Students varied in the number of hours they spent using the product, and varied in the proportion of time that they took practice questions versus reviewing answers and explanations. It is possible that the intervention group students used the product ineffectively, and thus failed to benefit. It is also worth noting that the control group at School A had the least exposure to the product. It is possible that faculty and peer lack of experience with the product contributed to the intervention group at School A using it less effectively.

The evidence suggests that a variety of interventions, including commercial test preparation products, are correlated with increased likelihood of passing NCLEX-RN on the first attempt. While some students may be successful without using a test preparation product, the observation of nursing faculty is that most students will need to use such a product (Davis & Morrow, 2021; Stuckey & Wright, 2021; Thompson et al., 2023).

Although cost can be an obstacle to nursing students using NCLEX-RN preparation products, the results of the study described in this paper demonstrate that access to a product is not independently correlated with improved outcomes. Students may need individualized guidance, based on their academic strengths and weaknesses, to most effectively use a test preparation product. Students may also need additional supports that work in parallel with the test preparation product, such as coaching or mentoring (Mailow et al., 2019; Schlairet & Rubenstein, 2019; Stuckey & Wright, 2021).

Some of the commercial NCLEX-RN products offer predictor assessments, to enable the student to determine their readiness to test. It is valuable for faculty and students to understand the meaning of the predictor exam the student uses. There is evidence that predictor exams are most accurate in identifying students with a high probability of passing NCLEX-RN on the first attempt, but are less accurate in identifying students who are likely to fail (Phelan, 2014).

Interventions that provide individualized guidance for students preparing for NCLEX-RN has been demonstrated to increase first time NCLEX-RN pass rates. Czekanski et al. (2018) had increased NCLEX-RN success with the use of a postgraduation coach, who used a variety of commercial test preparation products as well as cognitive-behavioral interventions customized to the student. Conklin and Cutright (2019) emphasized the importance of individualized guidance for linguistically diverse students. Stuckey and Wright (2021) had success with postgraduation faculty mentoring, individualized to the student. The involvement of a trusted coach or mentor may be an important element, along with a commercial test preparation product, for NCLEX-RN success.

Strengths and Limitations

The strengths of this study included the study design and sample size. The power analysis suggested that the sample of the study would be sufficient to show significant effect of the intervention. The results of the study showing no difference between the intervention and control groups were not likely due to the sample size.

The study was significantly limited by the changes in nursing education and the nursing licensure exam that occurred with the global pandemic.

Implications for Nursing Education

This study suggests that providing an NCLEX-RN preparation product, as a single intervention, is not sufficient to increase first time pass rates. Students may need individualized guidance and recommendations in order to effectively prepare for NCLEX-RN. Future studies should examine other interventions that may be combined with the test preparation product to result in an effective bundle or collection of interventions.

Conclusion

This study set out to assess the impact of a test preparation product on first time pass rates on the NCLEX-RN. The study failed to demonstrate change in NCLEX-RN pass rates for the intervention group overall, and for one institution in the study, NCLEX-RN pass rates decreased for the intervention group; therefore, the impact of the test preparation product was not significant.

While the effect of the pandemic is an important consideration that may have affected the results of this study, varied usage of the product may also have affected the results. It is also likely that using an NCLEX-RN preparation product is insufficient as a single intervention to increase pass rates. The best results may be achieved with an individualized plan of study and other interventions for students preparing for NCLEX-RN using a commercial product.