Evaluation of an evidence-based practice education workshop focused on critical appraisal for advanced practice nurses: A before-after intervention study

Abstract

Background

Regarding the evidence-based practice (EBP) process, nurses tend to perceive that critical appraisal of research articles is challenging. Though critical appraisal in EBP has been a frequent topic in medical education, implementing and disseminating existing EBP education for nurses are challenging in countries with limited EBP educational resources. This study aimed to evaluate changes in practice, attitude, knowledge and skills of EBP before and after an EBP workshop focused on critical appraisal among advanced practice nurses.

Methods

This study was a before and after educational intervention study. The workshops were provided in small groups in 2017 and 2018. The targeted nurses were certified nurse specialists (CNSs) who provide patient care and are certified as advanced practice nurses. The workshop was designed to include at least one CNS in each group. The Japanese version of the Evidence-based Practice Questionnaire, with four subscales—practice, attitude, and knowledge and skills of research and practice—was evaluated four times: baseline, immediately after the workshop, and 3 and 6 months after the workshop. At the end of the workshop, the participants wrote their learning objectives and plans for improving their critical appraisal of EBP. A self-reflection questionnaire was distributed 3 months after the workshop, and a qualitative descriptive analysis was used.

Results

Eleven CNSs participated in this study. Only the score of the knowledge and skills of research showed a statistically significant change. Many participants did not perform critical appraisal of research articles after the workshop. The facilitators were to have a role in integrating research evidence into practice and the barriers to being too busy, personnel transfers, and insufficient conditions.

Conclusions

Self-efficacy for critical appraisal in EBP might increase through the EBP education workshop. Even nurses interested in EBP require ongoing learning opportunities and organizational support for EBP activities.

Trial Registration

This study was retrospectively registered with University hospital Medical Information Network Clinical Trial Registry (UMINCTR) on 09/14/2018 (Registration Number. UMIN000034146).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12909-024-06315-z.

Introduction

Evidence-based practice (EBP) is the integration of the best available evidence with clinical expertise and patient values [1]. Clinical decision-making should be supported by current clinical information that reflects the best available research evidence [2]. Among the five steps of EBP (Ask, Acquire, Appraise, Apply, and Assess) [1], “Appraise” involves the critical evaluation of the integrity, reliability, and applicability of health-related research [3] to identify and select the most relevant evidence. Nurses are expected to contribute to and enhance EBP [4], and they generally have positive beliefs and attitudes toward it [5]. However, they practice critical appraisal infrequently and often lack the knowledge or skills required, with “Appraise” being the least performed step [6–10]. Advanced practice nurses (APNs) with high EBP competency levels [11, 12] are expected to play a leading role in EBP [2, 13].

EBP education is a fundamental approach to the development of EBP competencies. Numerous educational intervention studies have already been conducted [14–18]. “Appraisal” is the most frequently addressed topic in EBP medical education [19]. However, many nurses have perceived infrequent action of critical appraisal and inadequate knowledge and skills [5–10]. In Japan, many nurses also perceive low frequency of participation and low ability for critical appraisal [6, 13]. Furthermore, EBP education studies from Japan are limited, suggesting challenges in implementing and disseminating current EBP education for Japanese nurses and nursing students. For instance, nursing baccalaureate and master’s programs for certified nurse specialists (CNSs) as APNs are not regulated, although some official statements refer to the need for EBP education [20]. Although there are nursing research courses, there is often a lack of distinction between creating new evidence and integrating evidence into clinical practice (i.e., EBP). Additionally, there is a limited learning environment and practice of critical appraisal in clinical settings after graduating from a university or graduate school [13]. Even CNSs have few opportunities to participate in journal clubs and human resources are insufficient to teach critical appraisal in the context of EBP in their organizations [13].

Considering Japan’s unique cultural and societal context, directly implementing educational interventions that have been successful elsewhere can be challenging. EBP educational interventions, often based on problem-based learning, are typically conducted at single institutions or specific multisite settings, such as residency or organizational programs [13, 21, 22]. In these contexts, participants are often colleagues who collaborate on EBP within the same facility. However, in countries like Japan, implementing such programs can be difficult due to factors such as shortage of EBP mentors and educators, limited organizational readiness for EBP, and challenges in recruiting participants from a single institution. In such cases, participants may come from various institutions, potentially creating a disconnect between the learning environment and the actual clinical setting. Therefore, developing innovative educational approaches is essential in resource-constrained settings.

A new educational program must be designed with continuity between the place of learning in educational interventions and the place of daily practice, considering inadequate EBP resources, as well as evidence-based educational strategies, adult learning theory, and self-directed learning for participants [23–30]. Additionally, to evaluate its impact, tracking learners’ subsequent behaviors and outcomes is essential. Although the duration of educational interventions has been reported to vary among studies [15, 31], the long-term impact of the program may depend on the environment in which the participant is placed. Therefore, this study aimed to evaluate the impact of a new EBP workshop that focused on critical appraisal in a setting with limited EBP resources for nurses’ practice, attitude, and knowledge and skills of research and practice at baseline, immediately, 3 months, and 6 months after the workshop and the learning process after the workshop.

Methods

Study design

This educational interventional study with a before-after single-arm trial was conducted from October 2017 to March 2018 in Japan. The education program was developed based on the Critical Appraisal Skills Program (CASP) [32]. The CASP has been adopted for EBP education of clinical nurses [33, 34]. This study was designed based on the learning package of evidence-based medicine developed by CASP Japan. The details of our educational program are shown in Supplemental file 1, which refers to the guideline for reporting EBP educational interventions and teaching [23].

Participants and settings

The target population was CNSs with a master’s degree who were certified by the Japan Nursing Association as advanced practice nurses and working as a nurse in patient care. The participants were recruited through convenience sampling and a mailing list distributed to CNSs.

Participants applied to join the study as groups, forming their own groups that met the following criteria: (a) a group consisted of no more than four members, and (b) at least one member of the group was a CNS. If these criteria were met, other nurses without a CNS license or those working as faculty with a CNS license, were allowed to participate. This setting was used because this study aimed to provide a setting as similar as possible to an actual and daily clinical setting in Japan.

Educational intervention

Learning objectives

The overall objectives of this educational program were that learners would be able to formulate clinical questions using the PICO (patients/population, intervention, comparison, and outcome) format, appraise research articles while critically acquiring knowledge and skills, identify their challenges in formulating clinical questions and appraising research articles critically, and recognize their opinions on EBP and its process.

Theoretical frameworks

The educational program adopted the following theoretical frameworks: the experiential learning model for designing this program was based on an experimental learning cycle, instructional design of this program to enhance learning, Knowles’ adult learning theory for clinical nurses as adult learners, and problem-based learning for creating a setting that is closely associated with the EBP process in clinical practice.

The three modules of the workshop and the follow-up period after the workshop were structured such that the participants experienced the learning cycle of “concrete experience,” “reflective observation,” “abstract conceptualization,” and “active experimentation.”

Planned program

The educational program was designed based on the EBM learning package developed by CASP Japan. It was based on a clinical scenario based on the participants’ clinical questions in each group and comprised three modules: Module 1 focused on critical appraisal in EBP, Module 2 was a self-learning period, and Module 3 included reformulating the clinical question and searching the literature. This learning strategy, which combines lectures and small group discussions, has been used in several EBP studies [33, 34]. This workshop was held per group, which comprised 2–4 participants, including at least one CNS and one facilitator.

Educational materials

The materials included the handouts provided for lectures, worksheets, a scenario, and research articles. The EBM learning package was modified to suit this study, and CASP Japan approved the revisions. Scenarios were prepared for each group tailored to the interests of the participants, and the facilitator participating in the workshop selected a research article that matched the scenario.

Data collection

Data were collected from October 2017 to August 2018, four times per participant: pre (baseline and time 1) and post (time 2 and time 3) during the educational program and 3 and 6 months after the educational program as follow-ups. The participants’ demographics were collected at baseline, and a self-reported questionnaire regarding evidence-based practice was distributed at all four times. At time 3, a reflection sheet that describes the participants’ learning process was distributed by mail(Supplemental file 2). The timeline of the educational program and data collection are shown in Table 1.

Table 1.

Timeline of educational program and data collection

Educational program		Data collection
Module	How to provide educational programs	Time	Collecting data
		Baseline data before Module1 (Time 1)	Questionnaire
Module 1	Workshop (face-to-face, 5 h in one day)
Module 2	Self-learning (1 month after Module1)
Module 3	Workshop (face-to-face, 3 h in one day)	After Module 3 (Time 2)	Questionnaire
		3 month after Module 3 (Time 3)	Questionnaire and focus-group interviews
		6 month after Module 3 (Time 4)	Questionnaire

Measurements

The primary outcome was the total score of the evidence-based practice questionnaire of the Japanese version (EBPQ-J) [35]. This is the translated version in Japanese from the original EBPQ in English [36]. The EBPQ-J comprises four subscales: practice, attitude, knowledge and skills of research, and practice. This self-report questionnaire comprises 18 items ranked on a 7-point Likert scale (1 = never and 7 = frequent), and the range of the total score is 7–126. The higher the score, the better the participant’s outcomes for EBP. The reliability and validity of the EBPQ-J have already been tested [35]. The secondary outcomes were the four subscales of the EBPQ-J. In addition, quantitative changes were evaluated using reflection sheets.

Data analysis

The participants’ demographics were descriptively analyzed. The EBPQ-J scores from the four subscales were analyzed to determine the difference between the four subscales using repeated two-way analysis of variance (significance level of α = 0.05). The learning process was qualitatively analyzed using text data from the self-report questionnaires, which focused on whether the participants had achieved their behavior goals and reasons why they had or had not achieved them. The target population was the CNSs eligible in this study; therefore, the data for other nurses are described in Supplemental files 3 and 4. Statistical analysis was performed using SAS 9.4 (SAS Institute Inc., Cary, NC, USA.) and Microsoft Excel 2010.

Sample size

The sample size was 20 CNSs, calculated to provide the mean difference (standard deviation) of the total scores of the EBPQ-J as 20 (20) before and after the educational program, with α error 0.05 and power 0.80.

Ethical considerations

This study was approved by the Institutional Review Board. After the approval, the researchers informed all the participants about the study in writing and orally, and informed consent was obtained from all participants in this study. As an incentive for participating in this study, each participant received a gift card of 5,000 yen. The study was retrospectively registered with University hospital Medical Information Network Clinical Trial Registry (UMIN-CTR) on 09/14/2018 (Registration Number. UMIN000034146).

Results

Participants

Nine groups applied to participate in this study, and eleven CNSs were enrolled. Along with these participants, fifteen nurses without a CNS license or with a CNS license working as faculty members participated in this study with the eleven CNSs. Finally, seven CNSs and eleven other nurses completed the program and all questionnaires (Fig. 1). Each group contained two to four nurses and included 1–2 CNSs. The most common unit of application was “community colleagues” (5 groups), followed by study groups with hospital staff (including different hospital affiliations in the same family) (2 groups) and study groups with staff in hospital wards (2 groups). The characteristics of the CNS participants are detailed in Table 2. The mean age of the CNS participants was 38.4 years, and the mean clinical experience was 15.5 years. They had held their CNS certification for an average of 3.5 years. Most CNSs were employed in hospitals (81.8%) and worked as staff nurses (72.7%). Additionally, 72.7% of the CNSs received education on EBP and had experience with EBP in their practice, although few participated in journal clubs in the past year. Other nurses had similar characteristics in terms of age and years of clinical experience; however, they had less experience learning and implementing EBP compared to the CNSs (Supplemental File 3).

Fig. 1.

Participants’ flow

Table 2.

Participants’ characteristics

		CNS eligible in this study
		(n = 11)
Age	Mean (standard deviation)	38.4(3.3)
Years of clinical experience	Mean (standard deviation)	15.5 (3.5)
Academic background	Master’s degree	11 (100%)
Certification of certified nurse specialists*1)	Yes	11 (100%)
Year after being certified as a certified nurse specialist	Mean (SD)	3.5 (2.9)
Organization	Hospitals	9 (81.8%)
	Visiting nursing station	0 ( 0.0%)
	University	0 ( 0.0%)
	Research Institute	0 ( 0.0%)
	The others	2 (18.2%)
Job position*2	Staff nurses	8 (72.7%)
	Chief nurses	3 (27.2%)
	Specialized nurses	2 (18.2%)
	Others	0 (0.0%)
Learning experience of EBP	Yes	8 (72.7%)
Experience doing EBP	Yes	8 (72.7%)
Participation in journal club per year	None or missing	10 (91.1%)
	One time	0 (0.0%)
	30 times	1 (8.9%)
	More than 60 times	0 (0.0%)

EBP: evidence-based practice, CNS: certified nurse specialist, SD: standard deviation

*1) The CNSs’ specialties were cancer nursing, pediatric nursing, maternal nursing, chronic disease nursing, acute and critical care nursing, and infectious disease nursing

*2) Job positions were selected via multiple choice

Changes in EBPQ-J scores before and after the education program

At baseline of the CNSs shown in Table 3, the EBPQ-J’s total score and the subscales’ score, including the practice and knowledge and skills of research and practice, were moderate, with means and standard deviations of 70.4 (19.3), 20.0 (9.3), 21.7 (8.3), and 9.3 (2.9), respectively. The attitude score was particularly high, with a mean of 19.4 (1.4). Changes in scores over the four time points through 6 months (Table 3) showed a slight upward trend, but no statistically significant changes were observed in the total score or the three subscale scores—practice, attitude, and knowledge and skills of practice. Only the score for knowledge and skills of research increased by an average of approximately 10 points, which was statistically significant. The other nurses’ scores trended to be lower than those of the CNSs, with some significant changes noted in the total scores and the subscales of practice and knowledge/skills of research (Supplemental File 4).

Table 3.

Changes in the evidence-based practice questionnaire of the Japanese version (EBPQ-J) scores for six months

EBPQ-J	CNSs eligible for this study (n = 7)				p value
	Time 1	Time 2	Time 3	Time 4
	Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)
Total scores (range: 18–126)	70.4 (19.3)	81.9 (12.0)	78.0 (16.5)	85.6 (16.1)	0.21
Practice (range: 6–42)	20.0 ( 9.3)	23.0 ( 8.3)	21.3 ( 8.2)	24.3 ( 7.7)	0.61
Attitude (range: 3–21)	19.4 ( 1.4)	17.3 ( 6.0)	18.9 ( 2.0)	17.0 ( 5.4)	0.64
Knowledge/skills of research (range: 7–49)	21.7 ( 8.3)	30.4 ( 4.6)	27.3 ( 6.7)	31.1 ( 6.0)	< 0.05
Knowledge/skills of practice (range: 2–14)	9.3 ( 2.9)	11.1 ( 1.5)	10.6 ( 2.3)	11.1 ( 1.7)	0.23

SD: standard deviation

*1) Time 1: baseline before the workshop, time 2: immediately after the workshop, and times 3 and 4: three and six months after the workshop

Learning processes

Almost all the participants were satisfied with the workshop. However, many participants did not follow the plan of performing critical appraisal of research articles after the workshop. The CNSs who practiced critical appraisal told the facilitators that they found instances in their clinical practice that provoked them to read research articles. The CNSs who did not practice critical appraisal had barriers such as a job transfer, a job change, or changes in their physical condition or workload. Other reasons for not doing critical appraisal were a mismatch of the interests of EBP between the CNSs and the organization and their unspecific action plan, which they devised at the end of Module 3.

Discussion

The educational program in this study showed an increasing trend in the knowledge and skills subscale regarding research during the 6 months, but the total score and the other subscales did not change significantly. Furthermore, even for the CNSs with a strong interest in EBP and critical appraisal, continuous learning and engaging in critical appraisal of research articles in clinical settings was difficult. Previous studies reported that health care professionals with a master’s degree required ongoing support in learning EBP [37] and that learners in a monthly 6-month educational program using CASP required subsequent learning of EBP [34]. Our findings indicate that this program needs an additional follow-up program for enhancing CNSs’ action of critical appraisal in clinical settings or the CNSs require support to increase opportunities to read research articles for EBP after completing the program.

This program was evaluated in terms of three domains: knowledge and skills, attitude, and practices for EBP. First, this study indicated that providing learning opportunities focused on critical appraisal in EBP for CNSs may likely contribute to positive self-efficacy for knowledge and skills; this result is similar to previous EBP educational studies [15–17, 34]. Second, the insignificant changes in practice scores on the EBPQ-J suggest that this intervention does not encourage a behavior change in CNSs. It implies that only action plan development and evaluation after completing the program are insufficient. Third, a small change in the attitude score of the EBPQ-J after the educational intervention is probably due to the higher score at baseline. In general, many studies have high scores for attitudes toward EBP [33, 38]. This study included CNSs who are interested in EBP, and their attitude toward EBP was less affected by the educational program. To not only acquire knowledge and skills in critical appraisal, but also to increase its frequency, there is a need to further design an approach to provide a clinically integrated EBP education program [39] as a part of continuous education and the lifelong learning process.

The findings that this educational program did not notably improve the total score of the EBPQ-J over the six months are interpreted that the program did not enhance self-directed learning. To increase the total score of the EBPQ-J, participants need to continue reinforcing their EBP efforts in the workplace. Even including action plan development for conducting critical appraisal in EBP for APNs interested in EBP, the program did not enable them to overcome the barriers to engaging in critical appraisal. For example, the absence of the ability to proactively manage their own time for EBP and the authorities to change usual care into alternative care is often considered a barrier to EBP [40, 41]. The lack of such organizational readiness may have hindered the implementation of EBP by the participant APNs [42].

Strengths and limitations

In this study, university faculty members facilitated an educational program as an external educational resource for learning critical appraisal for EBP. This study setting could serve as a model for collaboration between academia and clinical practice. Another study reported that clinical nurses’ journal clubs were held with researchers or faculty members who supported selecting articles and managing the journal clubs [43]. Such activities might provide an opportunity to encourage the interpretation of research evidence in clinical practice and its incorporation into practice.

This study has some limitations. First, this study did not have a control group; thus, whether the change was affected by only the educational intervention remains unknown. For example, if CNSs conduct or participate in nursing research and projects, the scores related to EBP might change even if the educational program has no effect. Second, there was a strong selection bias for participants in terms of generalizability; thus, the findings would not be applicable to CNSs uninterested in EBP and critical appraisal. Third, the number of the participants for data analysis was smaller than the designed sample size; thus, the power was low.

Implications for education

Learning opportunities for the critical appraisal of EBP should be disseminated to countries where EBP is deficient. In-house educational human resources for EBP and critical appraisal are limited in such countries. The educational program developed in this study has the potential to enhance the participants’ self-efficacy in research skills and knowledge of EBP, despite some limitations. In Japan, there is a shortage of EBP instructors who can educate critical appraisal in the context of EBP. CNSs are expected to become instructors; however, EBP is not a mandatory role for CNSs [44]. Education for the critical appraisal of research questions is partially provided in graduate school, but there are few opportunities to apply it to solve clinical questions [13]. In nursing education, especially for APNs including CNSs, an education of critical appraisal for EBP should be provided apart from that in traditional research design education.

Conclusions

This study developed and evaluated an educational program based on CASP, developed by CASP Japan. The EBP workshop, which focused on critical appraisal of EBP, improved the score of knowledge and skills of research on the EBPQ-J six months after the workshop, but the total score and those of the other subscales did not change significantly. Additionally, even if CNSs had a strong interest in EBP and critical appraisal, ongoing learning of critical appraisal of research articles was difficult for them in practice. Continuous learning and practicing EBP and critical appraisal require organizational support and readiness for EBP.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

Ai Tomotaki contributed to the conceptualization, methodology, investigation, data collection, formal analysis, writing – original draft, writing – review & editing, project administration, and funding acquisitionIkuko Sakai, Hiroki Fukahori, and Yasunobu Tsuda contributed to the conceptualization, writing – review & editing, Surpervision. All authors read and approved the final manuscript for publication.

Funding

This work was supported by Japan Society for the Promotion of Science (JSPS) KAKENHI (grant numbers: JSPS-JP-16H07464 and 18K17452).

Data availability

Data sharing for this study has been restricted by the Institutional Review Board because the consent for sharing raw data for publications was not obtained from all participants.

The Educational materials are available upon request to the corresponding author.

Declarations

Ethics approval and consent to participate

This study was approved by the Certified Review Board of National Center for Global Health and Medicine (No. NCGM-G-002272-00). After the approval, the researchers informed all participants about the study in writing and orally, and informed consent was obtained from all participants in this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.