Abstract
Background:
There are several evaluation models that can be used to evaluate the effect of in-service training; one of them is the Kirkpatrick model. The aim of the present study is to assess the in-service training of cardiopulmonary resuscitation (CPR) for nurses based on the Kirkpatrick's model.
Materials and Methods:
This study is a cross-sectional study based on the Kirkpatrick's model in which the efficacy of in-service training of CPR to nurses was assessed in the Shahadaye Lenjan Hospital in Isfahan province in 2014. 80 nurses and Nurse's aides participated in the study after providing informed consent. The in-service training course was evaluated in reaction, learning, behavior, and results level of the Kirkpatrick model. Data were collected through a researcher-made questionnaire.
Results:
The mean age of the participants was 35 ± 8.5 years. The effectiveness score obtained in the reaction level (first level in the Kirkpatrick model) was 4.2 ± 0.32. The effectiveness score in the second level of model or the learning level was 4.70 ± 0.09, which is statistically significant (P < 0.001). The effectiveness score at the third and fourth level were 4.1 ± 0.34 and 4.3 ± 0.12, respectively. Total effectiveness score was 4.35.
Conclusions:
The results of this study showed that CPR in-service training has a favorable effect on all four levels of the Kirkpatrick model for nurses and nurse's aides.
Key words: Evaluation, in-service training, Kirkpatrick model, nursing
INTRODUCTION
Nowadays, economic growth and development of societies depends on their human resources; and the efforts for increasing efficiency and effectiveness, are efforts to improve life situation for all people in the society. Thus, effective insight, knowledge, and behavior should be developed in staff.[1,2] Today, accelerated growth of information and complexity of careers has increased the importance of staff training.[3] Training has a positive role on the employees of an organization. Organizations that pay attention to training and human resources are more successful.[4] Hence, it can be claimed that in-service staff training is an important step for improving the capabilities of every organization.
In health care professions, in—service training has a special role. In nursing—as one of the most important health care profession-cardiopulmonary resuscitation training is a vital component, and it is necessary that we become confident that nurses act efficiently during care of patients who need CPR. To act efficient, it is necessary that nurses have acquired essential knowledge and skill.[5] One of the way for improving knowledge and skill is in-service training.
In today's organizations, in-service training for improving scientific level of staff is essential, however, the efficacy evaluation of these trainings is more important.[6] Evaluation is a means for determining the extent to which we have been successful in achieving educational goals. Evaluation should determine whether we have achieved goals or not.[6,7]
Nowadays because of socioeconomic changes and training technologies related to them, evaluating training programs has also changed.[8] Because staff knowledge and skill in health care professions are important, choosing a proper method for evaluation is also important. There are different evaluation models that are used for evaluating effectiveness around the world such as the Context-Input-Process-Product (CIPP) evaluation model, Kirkpatrick evaluation model, and Outcome-based Evaluation model (OBE). Considering the present study and evaluation of effectiveness of in-service training of nurses, the Kirkpatrick evaluation model was used, which is one of the best evaluation methods and includes 4 levels, namely, reaction, learning, behavior, and results.[9,10,11,12] Kirkpatrick evaluation model was proposed for the first time by Kirkpatrick in 1960s, and has been accepted and used extensively since then.[12] Kirkpatrick model has been used for more than thirty years as the primary design for organizing and evaluating training in non-profit organizations.[13] The popularity of this model for evaluation of training programs can be because of the fact that this model provides a system for demonstration of training results and different data that can be used for evaluation. Hence, this model simplifies the complex process of evaluation of training programs.[10]
Reaction is the act that learners demonstrate to all effective factors during a training period. In fact, reaction evaluates participants’ feelings about a program. By learning, we refer to the way and extent of changes in participants by attending training courses. Behavior implies that whether training course has caused a desired change in learners’ behaviors. And finally, results imply that whether training course has resolved the existed problem and helped in achieving organizational goals.
Unfortunately in Iran, training programs are rarely evaluated. One of its causes is that staff and managers are not serious about proper evaluation of courses, which is also associated with improper feedback.[14,15] In Iran, few studies have investigated in-service training programs based on the Kirkpatrick model, especially in health care domain, and even in these few studies based on Kirkpatrick model, evaluation is about first and second level and the next two levels are not considered significantly. Therefore, this study was done to evaluate if this model can be a proper model for evaluation of in-service training courses in health care and especially nursing field.
MATERIALS AND METHODS
This study is an evaluation, cross-sectional study. Statistical sample comprised all nurses and nurses’ aides in the Shohadaye Lenjan Hospital. These nurses and nurses’ aides worked in the women's and men's internal and surgery wards as well as the emergency room and they participated voluntarily during 20–30 December CPR in-service training course (N = 80) conducted in 2014. Participants were divided in three groups based on working shift and their requests. For every group, 2 day and 5 h CPR in-service training sessions using slides, models, and resuscitation instruments such as laryngoscope and DC shock were held. Training content and teachers in all three groups were the same.
For evaluation of reaction level (determining relish extent and participants’ feelings regarding the training course is done after the course and is based on the participants’ immediate reaction), feedback questionnaires concerning training course and teachers were evaluated. The questionnaire about training course had 13 questions that evaluated content, equipment and managing. Evaluation of teachers contained rhetorical and presentation capabilities, knowledge and skill related to training course topics, making learners participate in discussions, proper use of training aids and teacher's general evaluation (discipline, behavior, Etc.). 5-point Likert's scale (absolutely agree, agree, no comments, disagree, absolutely disagree) was used for the evaluation of participants’ comments. Points were calculated based on 5. Because obtaining correct and meaningful answers from participants was very important at this stage, a questionnaire with closed questions was designed so that respondents give accurate answers. Respondents were assured that their information will remain anonymous and secret. The effect of training course and the learning was evaluated by a researcher-made test in the form of pretest and posttest 1 month after training. The effect on learning was calculated by the difference between scores of pre and posttests. Face and content validity of questionnaires were investigated by professionals in two stages of pre-test. Reliability of questionnaire was evaluated and its Cronbach's alpha was 0.84, which was acceptable. For analyzing data, Descriptive and inferential statistical methods were used with SPSS (statistical package for social science) version 18 (SPSS Inc, Chicago, IL, USA). For evaluation of learning, attempts were made to all three domains, knowledge, understanding, and application, were evaluated [Table 1].
Table 1.
Distribution of test questions (blue print of questions)
For evaluating behavior level, a 360° evaluation checklist for CPR was used. Behavior was evaluated by supervisors, head nurses, CPR physician, and one colleague after 3 months.
For evaluation of the fourth level or results, it should be noted that the results of the education can be studied in one of the four domains, i.e., avoiding costs, saving, profits, and strategic results. In this study, achieving strategic results was considered. Acceptable strategic results were considered to be the extent of achieving special and general goals of training course that had been agreed earlier by head nurses and supervisors and CPR physicians. The overall utility was calculated based on the average weight of scores in four levels, and because Kirkpatrick has evaluated the importance and sensitivity of the third level to be high, third level or behavior was given an average weight of 2 and other levels were given a weight of 1.
Ethical considerations
The study was explained verbally to participants and they also receive a written explanation. They were informed that participation was voluntary. It was emphasized that none of the information would be identifiable. Informed consent was also obtained from the participants.
RESULTS
Using description statistics and central and dispersion indices, obtained average utility in reaction level in the Kirkpatrick model was 4.2 ± 0.32, that is average of participants’ answers to every item in the feedback questionnaires of course and teacher. Score of content was 4.5 ± 0.26, equipment 3.92 ± 0.4, and teacher 4.2 ± 0.18. In learning level of the Kirkpatrick model, average score of utility, obtained from the test made by the researcher, was 4.7 ± 0.09 [Table 2]. It is worth noting that comparing results from pretest to posttest of participants by t-test and P < 0.05 were statistically significant, which implies that level of knowledge and skill of nurses in CPR was improved after one month from training course. This table showed also that this improvment was significant for both nurses and Nurses’ aides.
Table 2.
Results of second level of the Kirkpatrick model (pretest and posttest)
In the third level, the average utility score was 4.1 ± 0.34. This score was obtained by 360° evaluation by the head nurse, supervisor and CPR physician, and one colleague after 3 months. Criteria for being ineffective was scores less than 50 from 100 or 2.5 from 5.
In the fourth level, which was achieving strategic goals, utility score was 4.3 ± 0.12. Table 3 lists the strategic goals. Finally, the total utility score, based on a weight of 2 for behavior level and 1 for other levels, was 4.35 [Table 4].
Table 3.
Strategic goals of the course and its utility score from 5
Table 4.
The utility of four levels using descriptive statistics
DISCUSSION
Human resource is considered to be one of the most important resources of an organization. An organization, even with the best technologies and equipment, without specialist and trained staff cannot achieve success.[16] Hence, training should be one of the organization's duties and should be considered an investment. Today, organizations for in-service training consider budget. Organizations in the USA have reported an increase in the budget of training.[9] Moreover, managers want to see feedback and scientific reports regarding the effectiveness of these trainings. Receiving feedback needs evaluation. To obtain confident results, this evaluation should be based on scientific principles and objectives as well as the conditions of the organization.
Results of this study show that staff that participated in CPR training course evaluated teacher and the course as useful. In the second level, changes made in knowledge and learning of staff were desired. In the third level, it was determined that the course could make desired changes in learners up to 3 months after the training. In the fourth level, it was determined that CPR training course could achieve strategic goals. In general, despite the fact that training course was not 100% effective, but based on statistical calculations, it can be claimed that the effectiveness of this course was desired.
In Iran, no study has been conducted on the effectiveness of the Kirkpatrick model in in-service training of CPR for nurses and nurses’ aides. A similar study to ours was reported by Pour-Jahromi et al., which evaluated the effectiveness of in-service training concerning working with electroshock in nurses based on the same model in every four levels. Results of their study showed the utility of the course in four levels.
Iranian articles based on the Kirkpatrick model are few and are usually limited to level 1 and 2 of this model. Studies by Bakhshandeh et al.,[17] Amiri-Mehr[18] and Tavakoli[19] showed evaluation utility in the reaction level. The work by Omar et al.[15] showed desired results based on the Kirkpatrick model. Work of Omar et al. was conducted on 35 health care managers in different provinces, and it was determined that the most effective training was in the 1st and 2nd levels and it was reduced in the 3rd and 4th levels. In a work by Abbasian et al.,[20] in IranKhodro company, it was determined that the general utility was satisfactory, but it was decreased in level 3 and especially 2, such that effectiveness in 4 levels were 76, 64, 72, and 13%, respectively.
Considering the importance of CPR in health care profession, proper training and its evaluation is emphasized. In explaining the importance of training, especially for nurses and the necessity of proper evaluation of CPR, it should be noted that heart diseases caused the early worldwide deaths. Madden in her article mentioned the rate of early deaths to be more than 40% annually in Ireland, and reported that 6000 of these deaths were because of sudden cardiac arrests. She pointed out that more than half of these deaths occurred in hospitals, and hence considered the role of nurses in this regard vital and emphasized on the competence of nurses in administering CPR.[21] Training nurses for CPR is one of the principles that has been mentioned in guidelines such American Heart Association guidelines.[22] It becomes clear that training nurses about CPR and receiving feedback in order to increase patients’ survival and preventing early deaths is very important. It appears essential to improve the quality of these programs and receive feedback from them.
To increase quality and utility of training courses in nursing regarding CPR, studies show that it is better that training programs for CPR are held every 3 to 6 months for improving knowledge and skill. In addition, it has been recommended that computer simulation applications are used in order to provide the possibility of self-study and practice. These applications should be associated with providing feedback to user.[5] Hence, it is possible to improve the quality of in-service training using simple and inexpensive strategies and receive feedback and correct weaknesses.
In foreign studies, we could not find an article assessing the effectiveness of in-service training of CPR for nurses based on all 4 levels of the Kirkpatrick model. The most similar article that we found was the work by Aoki et al.[23] They observed the desired results of their training program concerning CPR in the 1st, 2nd, and 3rd levels of the Kirkpatrick model. They studied nursing, physiotherapy, speech therapy, and work therapy students and their objective was to investigate the effectiveness of American CPR in Japanese students. Regardless of its objective and the fact that this was not in-service training, it can be said that the results of Aoki et al. demonstrate the effectiveness of CPR training programs based on the Kirkpatrick model, which is also in agreement with the present study.
CONCLUSION
The results of this study showed that in-service training of CPR is effective in all the levels of the Kirkpatrick model for nurses and nurses’ aides. CPR training for nurses is a complex activity. Hence, it is recommended that we improve nurses’ learning and functional skills. Nurses who work in intensive care units encounter more cardiac arrests and dangerous situations and require training for advanced CPR. Thus, training courses for these nurses should be based on valid guidelines and should provide the ability to diagnose at-risk patients in hospitals.
Therefore, the importance of CPR in nursing and the results of the present study show that this training can be effective in the reaction, learning, behavior and results levels; it is recommended that organizations and managers try to conduct more such in-service trainings and evaluate their effectiveness based on useful models because their results have a direct impact on improving services and patients’ survival.
Considering that the behavior level compared to the reaction and learning levels is challenging and sensitive and Kirkpatrick mentioned three causes for this (first, participants should find an opportunity to change their behaviors; second, the time for change in behavior cannot be really predicted; third, organizational atmosphere can have an impact on changing behavior during work), it is recommended that evaluation be repeated enough times to be confident about the permanent changes in the behavior. In the present study, attempts were made to evaluate behavioral indices at different times from different people (head nurse, physician and colleague), however, survey of behavioral index in the Kirkpatrick model is difficult and time consuming. The limitation of the present study is a lack of control group. It is recommended that future studies employ a control group in order to omit factors effective on behavior. Furthermore, it is better that changes occur in a more natural environment and be surveyed for a longer duration, a task that was not feasible in the present study.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
Acknowledgement
At the end of the research we thank all the staff of the Shohadaye Lenganan Hospital.
REFERENCES
- 1.Chiang FF, Birtch TA, Kwan HK. The moderating roles of job control and work-life balance practices on employee stress in the hotel and catering industry. Int J Hospitality Manage. 2010;29:25–32. [Google Scholar]
- 2.Aghaee M, Aghaee R, Aghaee A. The role of behavioral dimensions job on efficiency and effectiveness of human resources. J Manage Org Culture. 2015;12:809–30. [In Persian] [Google Scholar]
- 3.Alhoseini H. The Theory of Continuous Engineering C.E.O. Identification of Permanent Authenticity in the Organizations as the Field of Development of the Individual and Society. 1st ed. Bandar Abbas: University of Hormozgan Publication; 2000. p. 103. [Google Scholar]
- 4.Ologunowa CS, Akintunde BA, Adu BO. Manpower training and development: Pathway to efficient organizational performance. Am Int J Res Hum Arts Sci. 2015;9:44–52. [Google Scholar]
- 5.Hamilton R. Nurses’ knowledge and skill retention following cardiopulmonary resuscitation training: A review of the literature. J Adv Nursing. 2005;51:288–97. doi: 10.1111/j.1365-2648.2005.03491.x. [DOI] [PubMed] [Google Scholar]
- 6.Shojae K, Karami M, Ahanchian MR. Evaluation the effectiveness of training programs for staff of Mashhad Ferdowsi University. Master Thesis in field of Education Management. 2011, thesis nomber: ED EDC 114. Available from: http://thesis.um.ac.ir/moreinfo-45241-pg-1.html .
- 7.Esteves J, Pastor J, Casanovas J. A framework proposal for monitoring and evaluating training in ERP implementation projects. Technical Research. 2002 [Google Scholar]
- 8.Rajeev P, Madan MS, Jayarajan K. Revisiting Kirkpatrick's model-An evaluation of an academic training course. Curr Sci. 2009;96:272–6. [Google Scholar]
- 9.Kirkpatrick DL. Evaluating Training Program: The Four Level. 2nd ed. San Francisco, CA: Berrett-Koehler Publisher; 1998. [Google Scholar]
- 10.Bates R. A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Eval Program Plan. 2004;27:341–7. [Google Scholar]
- 11.Kirkpatrick DL. Training and Development Handbook: A Guide to Human Resource Development. New York: McGraw-Hill; 1976. Evaluation of Training. [Google Scholar]
- 12.Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ. 2005;39:561–7. doi: 10.1111/j.1365-2929.2005.02173.x. [DOI] [PubMed] [Google Scholar]
- 13.Shelton S, Alliger G. Who's afraid of level 4 evaluation? A practical approach. Training Dev. 1993;47:43–7. [Google Scholar]
- 14.Pourjahromi Z, Ghafarian HR, Ghaedi H, Momeninejad M, Mohamadi M, Abasi A, et al. The effectiveness of training Course on “how to work with DC Shock Devices” for nurses, based on Kirkpatrick model. Iranian J Med Educ. 2011;11:896–901. [Google Scholar]
- 15.Omar M, Gerein N, Tarin E, Butcher C, Pearson S, Heidari G. Training evaluation: A case study of training Iranian health managers. Hum Resour Health. 2009;7:20. doi: 10.1186/1478-4491-7-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Abbasian A. Effectiveness of training course. Tadbir J. 2006;170:52–5. [Google Scholar]
- 17.Bakhshandeh H, Ahmadi H, Behnam M, Hamidi M. Evaluating the effectiveness of arbitration and coaching of the students based on Kirkpatrick's model. J Sport Manage. 2014;5:161–78. [In Persian] [Google Scholar]
- 18.Amiri-Mehr M. Evaluate the effectiveness of training courses Saipa Diesel Company according to Kirkpatrick model in summer 2006. Master's thesis, Department of Education and Psychology, University of Shahid Beheshti. 2007:11–25. [Google Scholar]
- 19.Tavakoli S. Assessment of learners reaction from short-term training courses for librarians of public library of Astan Quds Razavi on the first level of Kirkpatrick's model. Electronic Journal of Libraries, Museums and Documents Center of Astan Quds Razavi. 2010;2:1–16. [In Persian] [Google Scholar]
- 20.Abbasian A, Salimi G, Azin R. Evaluation of engineering training: Survey the effectiveness of resistant welding training course based on Kirkpatrick model, Irankhodro Co. as a case study. J Iran Eng Educ. 2008;39:37–62. [In Persian] [Google Scholar]
- 21.Madden C. Undergraduate nursing students’ acquisition and retention of CPR knowledge and skills. Nurse Educ Today. 2006;26:218–27. doi: 10.1016/j.nedt.2005.10.003. Epub 2005/11/30. [DOI] [PubMed] [Google Scholar]
- 22.Association AH. BLS for Healthcare Providers. Texas: American Heart Association; 2001. [Google Scholar]
- 23.Aoki T, Asada Y, Hiroe T, Tomizawa Y. Board 105-Program innovations abstract effectiveness of American style CPR training for Japanese medical students (submission #1494) Simul Healthcare. 2013;8:403. [Google Scholar]