Reproductive endocrinology and infertility nurse online orientation program evaluation: a quality improvement initiative

Abstract

Objective

To evaluate the effectiveness of a novel online orientation program (Nurses in REI Communication, Knowledge, and Skills [NRCKS]) among new and early-career reproductive endocrinology nurses to make recommendations for program quality improvement.

Design

Quality improvement.

Subject(s)

Reproductive endocrinology nurses.

Exposure

Online orientation program featuring 8 micromodules.

Main Outcome Measure(s)

Knowledge acquisition, confidence, user experience, skills, and abilities

Result(s)

Nurses demonstrated a statistically significant increase in reproductive knowledge after completion of NRCKS. Most nurse users felt confident applying knowledge in the clinical settings after program completion. Nurse users had an overall positive experience with interactive components and quality of visuals. Most nurse users gained skills and abilities that they will implement into professional practice. Nurses suggested improving navigation, adding more visuals in content expert presentations, and teaching soft skills necessary for the role of reproductive endocrinology nurses.

Conclusion(s)

Nurses in REI Communication, Knowledge, and Skills was well received by nurse participants and emphasized the receptivity and desire for reproductive health and fertility training among novice reproductive endocrinology nurses. Nurses in REI Communication, Knowledge, and Skills increased nursing knowledge, and nurses felt confident applying newly gained knowledge in the clinical setting on completion. Nurse participants provided invaluable feedback for quality improvement to inform the hard launch. The results emphasize the online orientation’s strong potential to close the theory-practice gap.

Introduction

Problem Description

Reproductive endocrinology and infertility (REI) nursing is a highly specialized field requiring comprehensive knowledge in reproductive health, infertility, endocrinology, pregnancy loss, and assisted reproductive technology (ART). As frontline caregivers serving as coordinators between the interdisciplinary team and patients, REI nurses should translate theoretical knowledge into direct patient care tasks using hard and soft skills. Nurses and nurse practitioners receive minimal to no REI didactic or clinical training in school, and transferability of skills is limited in experienced nurses transitioning from other specialties.

Available Knowledge

The inability to properly train and retain nurses in professional practice has significant consequences (1). The novice nurse’s immediate transition from academia to practice, theorized by Duchsher as “Transition Shock,” encompasses anxiety, stress, insecurity, and exhaustion, all risk factors for patient safety and practice errors (1, 2, 3, 4, 5). Fifty-five percent of nurses with <5 years of experience admitted to a medication error (5). Furthermore, Ortiz (6) noted lack of confidence expressed by all new graduate nurse study participants during their first year, and a leading factor was navigating communication with patients and interdisciplinary team. Unsuccessful transition risks nurses leaving their place of employment or nursing profession altogether (2). Approximately 25% of nurses leave a position in their first year of practice because of detrimental transition experiences (1). Consequently, increased turnover rates yield staffing gaps, adverse patient outcomes, and increased costs to hospitals (1, 7).

In 2017, the Nurses’ Professional Group (NPG) of the American Society for Reproductive Medicine (ASRM) conducted a scope of practice survey among members (8). Ninety percent of respondents selected “observational on-the-job” as type of training received as an REI care coordinator, and only 40.5% of nurses reported having a formal orientation. Similar findings are reflected in a study by Applegarth et al. (9) with most nurses reporting training via unstructured on-the-job mentorship. When asked to free text how training could improve, most respondents in the NPG survey suggested themes of a standardized competency-based training program (8). Furthermore, in the ASRM professional group meetings, REI physicians noted high nursing turnover shortly after hiring. Concurrently, established REI nurses expressed concerns regarding inconsistent training for nurses entering the field.

Rationale

To improve practice transition among novice REI nurses nationwide, a standardized approach to training was implemented. The National Council of State Boards of Nursing emphasizes the need for evidence-based training of new nurses that can be replicated across the country to improve quality patient care and decrease attrition rates (1). An online competency-based curriculum is an effective strategy to minimize discrepancies in training and instill knowledge, confidence, and feasibility.

As care in REI settings becomes more complex, REI nurses require advanced knowledge of reproductive health and ART to effectively counsel patients (9, 10, 11, 12). The most beneficial approaches for REI nurses to gain knowledge are short courses specific to ART nursing (9). Online competency-based curriculums for medical education have demonstrated statistically significant increases in both knowledge acquisition and retention (13, 14, 15, 16, 17). Gillan et al. (14) explored user experience of high-fidelity vs. low-fidelity e-learning and found that high-fidelity modules have a greater impact on knowledge acquisition than low-fidelity modules. When compared with traditional classroom models, e-learning yields statistically similar or increased levels of knowledge acquisition (13, 18, 19, 20).

Knowledge acquisition improves self-efficacy and confidence in REI nurses (9). Confidence in ability to effectively communicate with patients is essential because REI nurses spend at least half their time counseling patients on medications and treatment (10, 12). Nurse respondents from a study by Catherino et al. (12) felt more confident counseling patients and discussing results after receiving an educational intervention. Online learning modules have demonstrated statistically significant increases in self-rated confidence and self-efficacy (15, 17, 21, 22).

Online learning opportunities are beneficial to REI nurses for professional development because in-person trainings are not always feasible (9). Research supports integrating e-learning into medical training so learners can master basic knowledge and skills offline in a self-directed format (14, 18). Furthermore, e-learning can be centralized and benefit a larger volume of learners than traditional classroom settings (14).

Program Description: Nurses in REI Communication, Knowledge, and Skills

In response to the lack of formalized training among novice REI nurses nationwide, ASRM, in collaboration with NPG, developed an evidence-based, standardized orientation program entitled the Nurses in REI Communication, Knowledge, and Skills (NRCKS) (23). Nurses in REI Communication, Knowledge, and Skills is a novel online program aimed to orient early-career REI nurses and nurse practitioners to reproductive health by improving communication with interdisciplinary team, didactic knowledge, and technical skills. The comprehensive competency-based orientation is an engaging platform featuring 8 micromodules (Appendix A, available online) with short didactics by content experts, immersive 3-dimensional models, videos, case scenarios, and optional interactive “Ask the Expert” sessions. Nurses in REI Communication, Knowledge, and Skills was designed to be self-paced and estimated to take 4.5 hours.

Specific Aims

This quality improvement project aimed to evaluate the effectiveness of a novel online orientation program (NRCKS) among new and early-career REI nurses to make recommendations for program improvement. The first phase comprised 4 aims to guide program evaluation after user completion of NRCKS soft launch: assess changes in reproductive health and fertility nursing knowledge; assess perceived nursing confidence; assess user experience of interactive modules; and assess perceived skills and abilities informing REI nursing practice. Using findings from phase 1, the second phase aimed to develop robust formal recommendations for ASRM stakeholders to inform the hard launch in 2023.

Materials and methods

Study of the Intervention

The quality improvement project was designed in 2 phases. The first phase intended to assess novice REI nurses’ experience participating in NRCKS soft launch. Before beginning 8 modules, nurse participants completed a 15-question multiple choice pretest developed by REI physician content experts for NRCKS. After course completion, nurses completed a posttest designed with the same 15 questions. To receive certificate of completion, a minimum score of 70% within 3 attempts was required. Nurses could receive continuing education credits after filling out a continuing medical education postsurvey competence evaluation with the Likert scale and free-text style responses. After completion of the soft launch, the ASRM education department director emailed participants a Survey Monkey link to gather additional feedback. The survey included 8 questions, designed by the ASRM education department, with the Likert scale and free-text style responses. Then, in phase 2, the primary investigator transformed results from phase 1 into concrete recommendations for the ASRM stakeholders to enrich the hard launch.

In July 2022, the NRCKS soft launch began by granting access to new and early-career REI nurses on a first-come-first-serve basis. The soft launch closing date was November 30, 2022. The hard launch released January 2023 using nurse feedback from the soft launch. In collaboration with ASRM, the evaluation of the NRCKS soft launch occurred by collecting and analyzing quantitative and qualitative data from nurse participants.

Phase One

To evaluate project aims, existing data were used from preassessments/postassessments and 2 surveys integrated into the NRCKS soft launch experience. Four outcome measures guided program evaluation. Knowledge acquisition (aim 1) was defined as the percent increase in correctly answered questions on the 15-question multiple choice before and after assessment before and after course completion. Data were collected by the ASRM Learning Management System (LMS); SPSS software was used to conduct a paired t-test to determine whether there was a difference in the mean test scores before and after NRCKS completion.

Confidence (aim 2), defined as perceived nursing confidence in applying new knowledge after course completion, was measured by asking a Likert scale question about participants’ perceived confidence in applying new knowledge. Quantitative data were collected from Qualtrics and analyzed using descriptive statistical analysis.

User experience (aim 3) was defined as nurses’ experience with interactive platform format. User experience was assessed with 2 quantitative line items ascertained on a Likert scale and 4 free-text components. Quantitative items assessed the quality of visuals as well as structure and user navigation. Additionally, open-ended questions asked participants to describe experience with interactive components, express any moments of frustration, make suggestions to improve user experience, and offer any additional feedback. Data were collected via Qualtrics, and quantitative data were analyzed using descriptive statistical analysis. For open-ended questions, the primary investigator completed thematic content analysis, with another investigator (E.S.) providing intercoder reliability of identified themes.

Skills and abilities (aim 4) were defined as perceived skills and abilities gained informing future nursing practice after course completion. Skills and abilities were assessed by 1 quantitative line item and 2 free-text components. The quantitative item assessed whether users gained skills and abilities that will be implemented into practice using a Likert scale response. Qualitative questions were open-ended and asked participants to describe skills and abilities gained and how professional practice may change as a result. Data were collected by the ASRM LMS, and quantitative data were analyzed using descriptive statistical analysis. The primary investigator completed thematic content analysis on free-text components, with another investigator (E.S.) providing intercoder reliability of identified themes.

Of note, for consistency across all line items in this analysis, data from questions formatted with a 100-point Likert scale were converted to a 5-point Likert scale under the assumption that the user would perceive a 1–5 scale similar to a 0–100 scale. For this conversion, it was transformed as follows: 1, 0–20; 2, 21–40, 3, 41–60; 4, 61–80; and 5, 1–100.

Phase Two

The objective of phase 2 was to develop formal recommendations for the ASRM stakeholders on the basis of the outcomes of 4 project aims outlined in phase 1. The recommendations would reflect program strengths and weaknesses with the intention that the ASRM development team consider quality improvement before the hard launch.

Ethical Considerations

Soft launch users voluntarily participated for a discounted fee in exchange for program feedback. Data were originally collected by ASRM and did not contain Protected Health Information. Before granting investigators access to results, the ASRM LMS administrator deidentified data by assigning numbers to users. This project was accepted as a quality improvement project and deemed exempt from Institutional Review Board approval at Duke University School of Nursing and ASRM.

Results

The NRCKS soft launch started with 56 nurse participants who completed the pretest. Of 56 participants, 20 (35.71%) completed the NRCKS program concluding with the posttest. Of 20 participants who completed the program, 12 (60%) completed the Survey Monkey user experience survey, and 19 (95%) completed the continuing medical education survey for continuing education credit. Figure 1 shows the evolution of user participation.

Figure 1.

Evolution of user participation. CME = continuing medical education; NRCKS = Nurses in REI Communication, Knowledge, and Skills.

Knowledge Acquisition

The results of the paired t-test indicated a statistically significant improvement in knowledge from the pretest (mean, 82.33; standard deviation [SD], 7.26%) to posttest (mean, 97.33%; SD, 4.3%) (t (19) = −8.68, P < .001). The mean improvement was 15% points (95% confidence interval, 11.4%–18.6%). Figure 2 illustrates the pre- and postintervention mean scores. All nurse users received the minimum required score of 70% within the first attempt of completing the posttest.

Figure 2.

Nurses in REI Communication, Knowledge, and Skills knowledge scores at the preintervention and postintervention (0–100%).

Confidence

Confidence applying new knowledge had a mean score of 4.29 (SD, 0.75), with a minimum of 3 and maximum of 5. Seventy-five percent of 12 respondents rated their confidence as ≥4 on the scale.

User Experience

Quality of visuals and navigation

For quality of visuals, the mean score was 4.33 (SD, 0.78), with a range from 3 to 5. Of the participants, 83.3% rated their experience as 4 or 5 stars (≥4). Regarding structure and user navigation, the mean was 3.47 (SD, 0.69), with a range from 2 to 5. Twenty-five percent of the users rated structure and navigation as ≥4 on the scale.

Interactive components

Three themes emerged about user experience with interactive components: positive user feedback (n = 8); mixed user feedback (n = 3); and negative user feedback (n = 1) (Appendix B, available online). The positive responses highlighted how interactive components were an asset to learning modules, applicable to real-life scenarios, and kept the user engaged. Negative feedback detailed how some interactive content was confusing or outside the scope of novice fertility nurses.

Moments of frustration

When asked to describe any moments of frustration, analyses revealed themes navigation (n = 11), completion time (n = 1), content (n = 1), presenters (n = 1), and display (n = 2) (Appendix B). Most responses stressed navigation as a main point of frustration with a majority experiencing challenges navigating between modules and being allowed to complete the course to take the postassessment.

Suggestions to improve user experience

When asked for suggestions to improve user experience, analyses revealed themes navigation (n = 5), content (n = 3), presenters (n = 2), and display (n = 4) (Appendix B). First, several users suggested improving navigation of course modules and interactive components. Second, users recommended clarifying conflicting content among presenters and adding more nursing-specific content. Third, respondents proposed enhancing user experience by adding nurses as expert presenters. Fourth, to improve user friendliness of platform display, respondents proposed incorporating more visuals, offering a mobile application, and simplifying the design.

Additional feedback

When asked for any additional feedback, analyses revealed themes navigation (n = 3), content (n = 7), presenters (n = 3), sound quality (n = 2), and overall positive remarks (n = 6) (Appendix B). The most prevalent theme noted among general feedback was the need to revise course content such as simplifying the male anatomy section, explaining an example in vitro fertilization protocol, and adding a section focused on soft skills rather than just theory. The second most frequent theme to emerge was overall positive feedback about NRCKS; respondents stated they gained a foundation of knowledge essential to their practice.

Skills and Abilities

Skills and abilities to be implemented into professional practice were found to have a mean score of 4.26 (SD, 1.05), with a range from 1 to 5. Eighty-four percent of the users moderately agreed or strongly agreed with the statement. When asked to describe skills and abilities gained, themes emerged included increased knowledge (n = 16), patient communication (n = 6), and content (n = 1) (Appendix C, available online). Several nurses highlighted increased confidence communicating with and educating patients. However, a user noted a gap in content expectation vs. reality expressing disappointment that there was not more training on soft skills.

Themes about how professional practice may change because of gained skills and abilities included patient education (n = 8), nursing care (n = 6), and rationale (n = 3) (Appendix C). Several nurses stated that NRCKS would enhance the quality of their patient education. Second, nurses felt that they would provide more thorough nursing care. Finally, a few nurses expressed how NRCKS increased their understanding of the rationale behind patient care tasks.

Discussion

Knowledge Acquisition

Nurses in REI Communication, Knowledge, and Skills demonstrated the ability to increase REI nursing knowledge. The results indicated a statistically significant difference in the pre- and postassessment knowledge scores. These findings are congruent with previously published studies evaluating knowledge acquisition in medical professionals after implementing online training modules (13, 14, 15, 16, 17). Numerous qualitative accounts emphasized multiple types of foundational knowledge gained such as infertility causes, evaluation, and treatment. Furthermore, several users highlighted how interactive components and real-life scenarios helped apply new knowledge. Nurses in REI Communication, Knowledge, and Skills established the groundwork of knowledge, missing from nursing school curricula, necessary to thrive as a novice REI nurse.

Confidence

The mean and top box findings suggest that most nurse users felt confident after completion of NRCKS. In addition, qualitative accounts emphasized nurses’ increased confidence in counseling patients because of newly gained foundational knowledge. The results demonstrate the importance of an REI-specific training program to improve confidence, especially with communication. Confidence is an essential trait to successfully assimilate into the REI nurse role and navigate delicate patient encounters rooted in a web of grief, loss, and hope. As care coordinators, the multidisciplinary team relies on nurses to translate complex diagnostics into understandable care plans for the patient. During treatment cycles, REI nurses partner with their patients to provide step-by-step medical guidance and emotional support. Patients rely on nursing teams to answer questions and provide clarity, and nurses should be confident in education they provide and obtaining informed consent (12).

User Experience

The NRCKS interactive platform was a novel format for ASRM’s curriculum development team because previously offered trainings were strictly lecture based. The primarily positive feedback regarding user experience was reassuring as it kept the user engaged and will help set a foundation for future ASRM courses. Striving for a positive user experience is meaningful given the important role of perceived usefulness and perceived ease of use on adoption of new technologies, theorized by Davis (24) as the Technology Acceptance Model (25). The benefit of healthcare technology can only be achieved if nurses accept and intend to fully use it (25). Because REI nurses expend time and energy on complex patient care tasks and demanding schedules, optimizing user friendliness is essential to uptake and sustainability of the NRCKS platform.

One of the most well-received interactive components was visual displays. The quantitative analysis suggested positive user experience with quality of visuals. Qualitative accounts emphasized how visuals coinciding with content expert presentations were a helpful addition to clarify challenging concepts. Because of positive experience with existing visual components, a few nurse users expressed desire for more visuals to be included in every presentation. When designing nursing professional development opportunities, learning style preferences should be considered to enhance learning. Using the Felder-Silverman Model, a study by Mangold et al. (26) revealed that nursing staff preferred visual over verbal learning. To facilitate meaningful education and engage nurses, it is recommended to use visual learning tactics when possible. On the basis of feedback, including graphics and bullet point notes alongside all presentations should be considered for the NRCKS hard launch.

Navigation was a barrier to successfully using NRCKS and requires improvement to elevate user experience for the hard launch. The quantitative results revealed that the average user felt that the interactive platform was logical but not easy to follow. The free-text responses highlighted the need for smoother transitions between interactive components using a consistent clicking feature to signal module completion and allow access to the postassessment. Adjustments to navigation should be considered because perceived ease of use is related to technology acceptance and perceived usefulness. If nurses find technology easy to use, they are often more likely to accept it, intend to use it, and believe that it is useful for patient care (25).

Several qualitative accounts offered feedback about course content and advocated for consistent statistics, revision of male anatomy pop-up quiz, and a module dedicated to the REI nurse role. Nurse users requested that nursing-specific content be delivered by nurse experts covering hard and soft skills such as therapeutic communication, patient education, patient advocacy, psychosocial care, and the in vitro fertilization coordinator role.

Skills and Abilities

The quantitative analysis revealed that most nurse users gained skills and abilities they will implement into professional practice. However, when asked to expand on skills and abilities gained in the free text, a majority responded with the types of foundational knowledge gained as opposed to actual nursing skills. In a study by Mitchell et al. (10), participants reported that the primary role of REI nurses is direct patient care; skills essential for practice include teaching, counseling, and physical assessment. Furthermore, in the American Nurses Association Social Policy Statement, a key component characterizing advanced practice nursing is expansion. Expansion refers to the ability of nurses to integrate technical skills with theoretical knowledge to enhance autonomy (27). Although NRCKS provides a solid foundation of theory, additional education on tangible skills and abilities should be considered for a more robust orientation. Because the course title suggests that nurses will be trained in communication and skills in addition to knowledge, it was recommended that monthly live workshops be mandatory or an additional module be created for the hard launch.

Recommendations to the ASRM

The primary investigator presented the NRCKS soft launch data and analysis to the ASRM key stakeholders in January 2023. Detailed recommendations were provided in the form of a PowerPoint presentation via live video call. Feedback was well received and under consideration at the time of this publication.

Limitations

Data collection tools were created before formulating the quality improvement project. On program evaluation, it was noted the pretest displayed correct answers once users submitted. Furthermore, the posttest allowed users to check answers before submitting and then return to the test to fix incorrectly answered questions. This format jeopardized the reliability of the instrument to determine that knowledge was gained from completing NRCKS vs. memorizing answers. Regarding the Survey Monkey instrument, participants were provided a series of “star” visuals with which to rate the quality of visuals. However, no anchors were explicitly stated defining 1 star as poor quality and 5 stars as best quality. Therefore, Survey Monkey automatically averaged responses assuming that the highest number of “stars” meant the most positive rating. Finally, the small sample size limits the generalizability of findings.

Conclusion

The quality improvement project aimed to evaluate the effectiveness of the NRCKS soft launch among new and early-career REI nurses to make recommendations for program improvement. Nurses in REI Communication, Knowledge, and Skills was well received by nurse participants and emphasized receptivity and desire for REI-specific training among nurses entering the field. Nurses in REI Communication, Knowledge, and Skills increased REI nursing knowledge, and nurses felt confident applying knowledge in their clinical setting on completion. The results emphasize the online orientation’s strong potential to close the theory-practice gap in novice REI nurses and promote successful transition to clinical practice. The program aims were met and helped guide quality improvement to enrich the hard launch.