Effectiveness of nursing virtual training courses based on the Kirkpatrick Model

Abstract

BACKGROUND:

Educational evaluation is one of the most significant programs of any organization. In fact, evaluating training effectiveness allows managers and staff of the organization to get a clear picture of the qualities of training activities. The purpose of this study was to evaluate virtual nursing intensive care training courses at Shariati Hospital based on the Kirkpatrick Model in 2020.

METHODS AND MATERIAL:

The current cross-sectional study is an evaluation of the outcome-oriented program to evaluate the results of the e-learning course on nursing intensive care at Shariati Teaching Hospital from March 20, 2020 to September 20, 2020. The total number of nurses working in intensive wards was 168, of whom 150 participated in this research study with convenience sampling. The data collection tool consisted of four-part questionnaires including demographic information and three levels of the Kirkpatrick Model.

RESULTS:

The mean age of participants was 34.41 ± 6.63 years. The mean score of participant satisfaction was 76.77, indicating that they showed good reaction to virtual training courses. Virtual training course for intensive care nurses was statistically significant at the levels of learning (P < 0.0001) and behavior (P < 0.0001).

CONCLUSIONS:

Virtual training courses are effective educational method based on three levels of Kirkpatrick's Model in intensive care nurses and it can be used along with traditional training approaches.

Introduction

Human resources are one of the most important factors for organizations to reach its goals.[1] Nursing staff are a critical part of healthcare and make up the largest section of the health profession.[2,3] At present, nursing education programs involve the training of independent, responsible, and patient care nurses who think critically and are familiar with the practical skills that are continually renewed and refreshed.[4,5]

In 1990, the law on continuing education of the medical community was approved by the Islamic Consultative Assembly (Iranian parliament) to promote knowledge, skills, and improve quality of the provision of health services. At present, only graduates of various medical disciplines are required to pursue continuing education across the country.[6] In Iran, associations, scientific institutes, and public sectors, which are the same universities of medical sciences, provide continuing education. In 2000, medical universities (79.1%), scientific and specialized associations (11.1%), and other organizations (98.9%) participated in continuing education programs.[7]

The growing number of advancements in technology and information have brought changes in the field of education and learning and is widely implemented in most e-learning institutes as a method of in-service staff training.[8,9] E-learning and virtual learning in the educational systems of developed and developing countries has a growing trend and is one of the new and active methods to improve the quality of education. Virtual education, a medium for distance education, is not defined just as courses offered online. This type of training consists of integrating educational resources, interactions, performance enhancement, and structured training programs. Virtual learning is essentially the art of using network technology to design, select, develop, and manage the learning process.

As staff have to leave their workplace to attend various training programs which causes serious problems in performing organizational tasks and duties, virtual in-service staff training provides an opportunity to save money and time of learners. Moreover, many advantages have been enumerated for virtual tutorials. Despite the above-mentioned advantages for e- learning, it is necessary to determine the effectiveness of these courses.[10,11] Numerous studies around the world have shown the positive outcomes of e-learning with the effect of continuing education on improving the quality of care.[8,12]

Educational program evaluation, one of the most significant programs of any organization, provides good feedback in designing and reviewing any system. Indeed, evaluating the effectiveness of training courses helps the managers and staff of the organization to obtain a clearer picture about the quality of training activities and equips planners and training staff to be aware of the positive and negative aspects of the program.[13]

To determine the value of training courses, there are several evaluation models, in which the Kirkpatrick Model—a four-level model—of evaluation is one of the important, simple, and practical models in evaluating educational programs.[10,11] This model has been widely used in the evaluation of continuing nursing education programs in organizations.[14,15] In this simple and practical model, the effectiveness of learning programs can be evaluated using four levels of criteria: reaction, learning, behavior, and results. Reaction measures the trainee's impressions of the training program and the trainee's level of satisfaction with a particular training experience. Learning is concerned with measuring the extent to which learning skills, techniques, and facts have been acquired during training, pre- and post-tests. Behavior is the change that occurs as a result of attending the training program. Finally, results refer to the achievement of goals that are directly connected to the organization.[16,17,18]

Virtual trainings are mostly used in training of the nursing staff in many hospitals in Iran. Unfortunately, educational programs are rarely evaluated. One of the reasons is that employees and managers are not serious about the proper evaluation of courses because it is usually associated with incorrect feedback..[15,19] This study attempts to evaluate the virtual nursing intensive care training courses at Shariati Teaching Hospital based on three levels of the Kirkpatrick Model in 2020.

Materials and Methods

Study design and setting

The current study employed a cross-sectional design to evaluate the outcome-oriented program focusing on assessing the results of the e-learning course of nursing intensive care at Shariati Teaching Hospital from March 20, 2020 to September 20, 2020. The survey was conducted in five intensive care units of Shariati Hospital, including the general intensive care unit (GICU), the medical intensive care unit (MICU), the neurological intensive care unit (NICU), the cardiac surgery intensive care unit (ICUOH), and the cardiac care unit (CCU).

Study participants and sampling

Study participants included all nurses employed in intensive care units and nurses who had completed an e-learning course. The sample size was equal to the study population representing the target population. A total of 168 nurses who had completed the electronic intensive care courses were invited to participate in this study, and then 150 staff working in these departments entered the study based on convenience sampling.

Data collection tool and technique

The data collection tool consisted of four-part questionnaires. The first part included demographic information, age, gender, work experience, place of work, level of education, position, and the number of the relevant courses in which the participants had enrolled. The second to fourth parts of the questionnaires included the evaluation of continuing nursing education in three levels of the Kirkpatrick Model (reaction, learning, and behavior). A 20-item survey questionnaire was used to evaluate the first level. Responses were collected on a five-point Likert scale ranging from 1 to 5, where the total score ranged from 20 to 100. Participants took a 20-item, pre and post, multiple choice test to assess the second level of the Kirkpatrick Model (learning). The questions were created based on course topics and learning objectives. The minimum and maximum scores were zero and 20, respectively. A checklist questionnaire with a five-point Likert scale ranging from 1 (very low) to 5 (very high) approved by the Tehran University of Medical Sciences was applied to assess the third level of the model. The total scores ranged from a minimum score of 5 to a maximum score of 25.

To determine the validity of the data collection tool (data entry sheet), content validity was used. The instrument was then presented to a group of experts in the field (n = 10) for evaluation of the content validity. After applying corrective feedback, the relevant tools were used. The validity of each instrument were determined separately.

Cronbach's α was used to assess the scientific reliability of the instrument. The questionnaire was provided to 20 nursing staff individuals, and then the collected data were entered into the Statistical Package for the Social Sciences (SPSS) software for analysis. The reliability of each instrument was evaluated separately. The reliability of the instrument checking the level of Kirkpatrick's pyramid was 0.86, 0.94, and 77.3 at levels 1, 2, and 3, respectively. Inclusion criteria included having at least a bachelor's degree in nursing, being employed in the intensive care unit, and full participation in virtual courses. The exclusion criteria were the reluctance of participants to participate in the project and non-participation in the test.

To conduct research work, all nurses who had passed the intensive care courses at Shariati Hospital along with clinical and educational supervisors were included in the study. All participants provided informed consent before participation in the study.

The aims considered for learning were as follows:

1. Bedsores and related care in the intensive care unit

2. Basic cardiopulmonary resuscitation

3. Advanced cardiopulmonary resuscitation

4. Fall prevention strategies

5. Patient monitoring and hemodynamic evaluation

The course contents were prepared based on the aims in the Microsoft PowerPoint and educational videos that were uploaded on the website of Shariati Hospital designed by the IT department in cooperation with the nursing office and electronically visible in the personal profiles of all participants after login. They could download and read it anytime and anywhere asynchronously. Before conducting the research, learners were provided with essential information and instruction about the course structure.

At first, the participants answered demographic questions and then they logged in with the username and password previously issued by the administrator to access necessary information uploaded on their account. The virtual training course included training materials and relevant multiple-choice questions. All participants were asked to study materials developed for the online course and respond to multiple choice questions. After a month, a two-week opportunity with a specific schedule to attend the electronic tests was determined for the learners.. Two months after the end of the course, the effectiveness of the training at the behavioral level was evaluated based on the standard checklist by the head nurses of the relevant intensive care units.

Ethical consideration

The current study was approved by the Ethics Committee of Shahid Beheshti University of Medical Sciences (IR.SBMU.SME.REC.1398.097). A written, informed consent to participate in this research study was obtained from all nurses. All data collected during the survey were kept confidential and all participants were informed that participation in the study was voluntary.

Data were analyzed using the SPSS version 19 software. Paired t-test was used to analyze data, and various discussions such as frequency, mean, and standard deviation were addressed by descriptive statistics.

Results

The mean age of participants was 34.41 ± 6.63 years. A majority of participants were in the age range of 30–35 years. The vast majority of participants (29.3%) had five to ten years of work experience. The average work experience was 10.03 years with a standard deviation of 5.62 years. Study participants were 78% female and 22% male. The ICUOH had the greatest number of participants in this study [Table 1].

Table 1.

Demographic characteristics of study participants

Variable	n (%)
Age (Years)
Under 30 years	37 (24.7%)
30-35 years	56 (37.3%)
35-40 years	24 (16%)
Above 40 years	33 (22%)
Wards
Under 5 years	39 (26%)
5-10 years	44 (29.3%)
10-15 years	36 (24%)
15-20 years	21 (14%)
Above 20 years	10 (6.7%)
Wards
Female	33 (22%)
Male	117 (78%)
Wards
ICUM	23 (15.3%)
ICUG	36 (24%)
ICUNS	29 (19.3%)
ICUOH	39 (26%)
CCU	23 (15.3%)
Education Level
Bachelor’s degree	126 (84%)
Master’s degree	24 (16%)
Types of Employment
Official employment	53 (35.3%)
Contract employment	66 (44%)
Corporation employment	31 (20.7%)

The mean score of participant satisfaction was 76.77, which indicates that they had a good degree of satisfaction. Moreover, the majority of participants (66.7%) stated that e-learning courses were well structured [Table 2].

Table 2.

Mean reaction level score of studied units and absolute and relative frequency distribution

Education Reaction Level	Virtual number (%)
Very good	46 (30.6%)
Good	100 (66.7%)
Medium	4 (2.7%)
Mean±standard deviation	76.77±8.22

The mean score of participants before and after e-learning courses was 15.34 and 16.88, respectively. Paired t-test showed a statistically significant relationship between the scores before and after e-learning courses (P < 0.0001) [Table 3].

Table 3.

A comparison between mean and standard deviation of scores before and after level 2 (learning) of the Kirkpatrick Model

Variable	Mean	Standard deviation	Mean difference	Upper bound	Lower bound	Statistical Test
Before	15.34	4.89	−1.54	−0.83	−2.24	t=−4.30, df=149, P<0.0001
After	16.88	5.20

According to the questionnaire developed to measure behavior change, 50.7% of participants displayed a moderate level of behavior and the overall mean score for all participants was 14.8. On the other hand, 63.3% of participants showed significant changes in behavior after learning. Moreover, there was an increase (18.01) in the average of scores. A paired t-test showed a statistically significant difference between pre- and post-behavior scores [Table 4].

Table 4.

Frequency distribution of behavior level among participants in e-learning courses and comparison of mean and standard deviation of scores

Time of Evaluation Level of Behavior	n (%)
	Before	After
Very high degree	5 (3.3%)	25 (16.7%)
High degree	49 (32.7%)	95 (63.3%)
Moderate degree	76 (50.3%)	29 (19.3%)
Low degree	19 (12.7%)	1 (0.7%)
Mean±standard deviation	14.80±3.22	18.01±2.67

t=−4.30, df=149, P<0.0001

Discussion

The results of the present study showed that most of the participants had a good and very good reaction to holding these courses electronically. The outcomes of the present study were evaluated in terms of nurses' satisfaction with the length of the courses, educational materials, educational facilities and equipment, manner of material presentation, and attractiveness of learning materials presented in their continuing nursing education courses, according to the first level of the Kirkpatrick Model. Meanwhile, more than half of the nurses were satisfied with the timing and presentation of the courses because they had chosen very good and good options, and the rest of the participants had chosen moderate and poor options. At the first level of this model, the reaction of participants was evaluated. Although there are several instruments to measure the reaction of people participating in the courses, questionnaire-based surveys are the most commonly used method in research.[20] The first reason for evaluating the reaction level in the Kirkpatrick Model is to be aware of the trainee's feelings about participating in training programs and the changes needed to improve the program. The second reason is that trainees realize that their reaction to and satisfaction with the program is important to the course organizers. In a study conducted by Ghanbari et al.[21] Donald et al.,[22] and Baghaei et al.,[23] students were not satisfied with the online training courses due to low speed of the network, but fortunately this problem was solved in the present study. Nurses were interested in e-learning courses due to the following reasons: updating information, being attracted and familiar with online courses, and virtual tests. The study conducted by Ghanbarnia and Khorasani[24] also showed that participants were often interested in e-learning, which is consistent with the present study. A number of nurses were not interested in these courses, and perhaps the main reason was the unwillingness to use or unfamiliarity with electronic equipment.[24] In the study conducted by Fathi,[20] the following results were obtained: the effectiveness of the content of the training program, teaching-learning activities, and the design of the pages were at the desired level; the organization of educational materials, the feedback provided, and flexibility were at the average level; helping students, the effectiveness of e-learning courses, and the assessment methods were undesirable. It seems that the effectiveness of online courses at the level of reaction and public satisfaction was acceptable. Perhaps one of the most important reasons for staff satisfaction is that most nurses do not have enough time to attend continuing nursing education courses due to their busy schedule and compulsory overtime work, and e-learning courses allow nurses to attend training classes without the stress of and it interfering with work shifts.

The results of the present study showed that the mean test scores of the participants increased significantly after the virtual training. The findings of the study, which are consistent with those in the studies by Ghanbarnia, show that e-learning has been effective at the level of learning. In the present study, there was a significant difference between delivering courses electronically and evaluating the knowledge of nurses before and after courses. Various studies show that learning through online courses plays an important role in promoting knowledge, and distance-learning courses produce the same level of learning as traditional face-to-face courses. However, successful distance learning requires proper management and adequate expertise.[24] As shown in the study by Salari and Karami,[25] the learning of the participants through electronic courses has no priority over the face-to-face courses. The findings of their study also showed the effectiveness of learning by combining learning environments in on-job-training courses for employees of the industry sector. Moreover, delivering education through online courses sees no priority over traditional face-to-face courses. It seems that the effectiveness of the online course at the level of learning depends on various factors, including the management and proper presentation of the course. Delivering education through online courses was performed properly because it was previously performed in Shariati Hospital with the cooperation of the nursing office. This is why it can be considered as a reason for the course effectiveness and learning enhancement of staff. Furthermore, the requirement to attend on-the-job training courses and the effect of the online course scores on the evaluation of nurses at the end of the year can be a double motivation to obtain a passing score and attend courses. Therefore, it can be said that according to the obtained data, the virtual training courses of Shariati Hospital intensive care nursing have been effective at the level of learning.

The results of the present study showed that the participants had a significant change in behavior after virtual learning, indicating that training changed the behavior of staff in the workplace. Behavior refers to changes that occur in the behavior of participants as a result of participating in the training course, and it can be determined by continuing to evaluate in the real work setting. As this level is more challenging and sensitive compared to the previous one, Kirkpatrick mentioned the following three reasons: 1) participants should be given an opportunity for behavior change; 2) the time of behavior change does not appear to be predictable and 3) organizational context can influence participant behavior at the job. According to findings of a study conducted by Kennedy et al.,[26] it seems that the main reasons for the lack of studies on the educational effectiveness at levels 3 and 4 are related to lack of support by the organization and lack of awareness of education experts about the purpose and potential benefits of evaluation at levels 3 and 4 for organizations.[26]

According to the the Association for Talent Development, two-thirds of education managers believed that they were under more pressure to show the effectiveness of the training courses to senior officials in the organization, but only about 10% of their educational interventions were examined at level 3 and about 25% at level 4. Over the past two decades, the frequency of assessments at level 3 has increased significantly, but only about half of all educational interventions are being performed.[27] Most previous studies have evaluated the effectiveness of courses only on the first two levels of the Kirkpatrick Model, and a few studies in Iran evaluating the effectiveness of electronic courses have focused on the four levels of the model. It seems logical for an organization that spends time on the evaluation of its training program to figure out whether the staff have actually learned or acquired the desired knowledge, and then use this information to make any necessary improvements or other changes to the training programs. It can be concluded from the findings of the study that virtual training can make changes in the behavior of the staff and improve the quality of nursing care services to patients. Therefore, based on the research data, virtual nursing intensive training courses in Shariati Hospital has caused a change in the behavior of the staff members who have passed the courses.

Limitation and recommendation

The major limitations of the study were the non-participation of a number of participants in the tests, inability to provide feedback to the participant because of asynchronous nature and disconnection of internet on some exam days.

Conclusion

Regarding the findings of the study and the effectiveness of virtual courses at three levels of reaction, learning, and behavior, the benefits of virtual courses such as ease of implementation, reduction of organizational costs, and providing more opportunities for the organization to evaluate staff training, these courses can be widely used in educational centers of training. Finally, the implementation of blended learning involving traditional face-to- face learning format and delivering learning content virtually can have an impact on the learning of trainees. The researchers suggest that other program evaluation models be used and the nursing staff of other wards be also examined.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.