Abstract
Introduction
Nowadays, the employees` in-service training has become one of the core components in survival and success of any organization. Unfortunately, despite the importance of training evaluation, a small portion of resources are allocated to this matter. Among many evaluation models, the CIPP model or Context, Input, Process, Product model is a very useful approach to educational evaluation. So far, the evaluation of the training courses mostly provided information for learners but this investigation aims at evaluating the effectiveness of the experts’ training programs in SUMS and identifying its pros and cons based on the 4 stages of the CIPP model.
Method
In this descriptive analytical study, done in 2013, 250 employees of SUMS participated in in-service training courses were randomly selected. The evaluated variables were designed using CIPP model and a researcher-made questionnaire was used for data collection; the questionnaire was validated using expert opinion and its reliability was confirmed by Cronbach’s alpha (0.89). Quantitative data were analyzed using SPSS 14 and statistical tests was done as needed.
Results
In the context phase, the mean score was highest in solving work problems (4.07±0.88) and lowest in focusing on learners’ learning style training courses (2.68±0.91). There is a statistically significant difference between the employees` education level and the product phase evaluation (p<0.001). The necessary effectiveness was not statistically significant in context and input level (p>0.001), in contrast with the process and product phase which showed a significant deference (p<0.001).
Conclusion
Considering our results, although the in-service trainings given to sums employees has been effective in many ways, it has some weaknesses as well. Therefore improving these weaknesses and reinforcing strong points within the identified fields in this study should be taken into account by decision makers and administrators.
Key words: Inservice training, Training, Effectiveness
Introduction
Nowadays, the employees ` in-service training has become one of the core component in survival and success of any organization .In the past decade, the rapid evolution of science and information technology, not only has heighten the need for but also mandate such trainings. Living in the age of information and technology advancement, the employees can no longer fulfill their diverse duties by relying on their old and out dated knowledge and skills. Moreover, the organizations also cannot remain indifferent to these advances, since they must compete with parallel organizations which utilize these new technologies. These facts has pointed out a very important need for organizational survival and improvement which is essential designing, implementing and constant evaluating of in-service educations. In addition, by extending occupational knowledge, attitude, and teaching group work through cooperative learning, the in-service training can enhance human resources capability and help them to do their job more efficiently (1). It also will result in lifelong learning since philosophically speaking, in - service education has origins in it (2).
Unfortunately, despite the importance of training evaluation, a small portion of considerable resources are actually allocated to this matter. What’s more, many expensive in- service training programs which are even designed thoughtfully lack evaluation (3).
Homen believed that whatever is done to gather information, improve training problems and provide feedbacks for managers can be considered as training evaluation. Therefore, in order to make intelligent decisions and as a result improve the quality of training programs an evaluation based on these factors must be done. Such a feedback, if designed properly, will guarantee the success of training programs (4).
As Stuffelbeam has proven in 1983, among many evaluation models, the CIPP model or Context, Input, Process, Product model is a very useful approach to educational evaluation which as a standard, provides a systematic and constructional way of evaluation in many training programs. Based on CIPP model the most important goal of evaluation is the training courses improvement rather than its confirmation. .In addition, instead of focusing only on an individual improvement, this model offers information which can be used by decision makers in a training institute for ongoing evaluation of the programs. By providing an organized feedback of the current affairs, this model helps the managers to prioritize the essential needs and allocate available resources to more effectual activities (5, 6).
In context evaluation, primary programming and the program environment are examined. Since in this phase, the program may have not been started yet, the need to implement the program and required facilities may also be examined (7).
The input evaluation phase which starts after the execution approval includes: program time table, facilities and human resources management (7). The goal of input evaluation is facilitating the implementation of the program designed in the context phase. In addition this phase also focuses on human and financial facilities, politics, training changes, instructional strategies, obstacles and limitations of the training system (8).
In the process evaluation phase, the program itself goes under close observation and the changes of input during the program and the later on output is investigated. In fact, it is a quality control phase during its execution (7). The performance of process evaluation is largely depended on the implementation of the two previously mentioned phases; in other words, if the context and input evaluations are done properly, the implementation of the program is defiantly easier and more successful; as a result the process phase face less obstacles (8).
The final phase or product evaluation is considered as the goal of evaluation. By demonstrating all the desired and undesired outcomes, this phase shows what has or has not been done (7). As Stuffelbeam put it, product evaluation is measuring and judging the achievement of predefined goals. In product evaluation this measurement and interpretation is done not only at the end of the program but also during the program execution and completing each task. This is due to the periodic feature of the evaluation process, in other words evaluation is a process that should be done constantly and the results of each phase should be utilized to improve the previous phase and program the next phase. In some cases product evaluation is expanded until it covers long term effects of a program and its positive and negative products (8).
So far, the evaluation of the training courses mostly provided information for learners but this investigation aims at evaluating the effectiveness of the experts’ training programs in Shiraz University of Medical Sciences based on the CIPP model, and identifying weak and strong points of the program, and providing results for administrators and decision makers, and can obtain guidelines for designing better quality programs.
Methods
In 2013, this analytical and illustrative study was done on all the employees of Shiraz University of Medical Sciences (2219) who took part in different in-service trainings. We used Kerjesy and Morgan formula for calculating the sample size.
In an attempt to increase the accuracy of the study and compensate for possible low response rate, 10 percent was added to the actual sample size. After gaining informed consent , 250 employees which were selected randomly entered the survey .A questionnaire based on Stuffelbeam’ s CIPP evaluation model was designed (9). This questionnaire consists of two parts: The first part collect the demographic data such as :gender, education level, age, employment type, work experience, field of study, type of job and place of work ;whereas the second part which contained 62 ( 58 quantitative and 4 qualitative ) questions gathered the effectiveness data . The employees ranked each quantitative item based on a five rating scale which was :( 1= very low, 2= low, 3= average, 4= high, 5= very high). Achieving 3 points was considered the cut point of effectiveness for each phase which meant that the effectiveness is undesirable if the average of points given to the items in each phase is less than 3 (1-2.99) and desirable if it is 3 or more (3-5) ,providing that the (p<0.001) in each case.
The qualitative questions were open –ended and used for gathering personal opinions and suggestions. The questionnaire`s questions were based on four phases of CIPP model and previous studies,. These phases are:
Context Phase: Training infrastructure, opportunities, satisfaction, training needs assessment, training resources, learning styles, location status, and manager support.
Input Phase: Staff, training content, program designing, budget and financial resources, training environment, supports staff, and instructors.
Process: Training process, learning, number of instructors, instructors’ performance, support and encouragement, performance of the library and announcement.
Product or outcomes Phase: The quality of educational content, the rate of reaching goals, effectiveness of trainings, enhancing the knowledge and skills, the motivation for getting educational certificate, evaluation of trainees' information and empowering the staff.
Questions 1 to 10 evaluate the context component, Questions 11 to 25 evaluate the input component, Questions 26 to 48 evaluates the process component, Questions 49 to 58 evaluates the product component, and Questions 59,60,61,62 measures the results, qualitatively. The reliability of the questionnaire was determined by experts through Cronbach’s alpha test. The cronbach’s alpha coefficient was 0.89 which suggests that the questionnaire has the required reliability. The validity was also established using experts opinion. The questionnaires were distributed and collected via face to face or email approach. The data were then analyzed using descriptive and inferential statistics including distribution of population, mean, standard deviation, correlation coefficient, and t-test.
Results
In the present study, 68% of the participants were women and 32% were men. 163 participants (67.2%) had Bachelor’s, 57 (23.4%) had associate and 23(9.4%) had Master’s and PhD degrees. The mean and standard deviation of the study participants’ age and work experience were (36.3±6.14) and (11.25±6.79), respectively. The employees enrolled in the study were from deferent departments such as healthcare, administrative - financial, educational- cultural and technical-engineering departments. The gender affected the process phase `s results significantly (p=0.045); in contrast with the other phases in which the gender didn’t play any role (p>0.001). In the product phase , the educational degree of the employees which included Associate , Bachelor’s , Master’s and PhD degrees significantly affected the results (p<0.001).
Quantitative data showed that in the context phase, the mean score was highest in solving work problems (4.07±0.88) and lowest in focusing on learners’ learning style training courses (2.6 ± 0.94). Although the total score achieved in the context phase was 3.17 overall, this phase didn’t have the necessary effectiveness (p>0.001) (Table 1).
Table 1.
The context phase
| Evaluation component | Research question: How are the staff training programs with respect to evaluation of context? | ||
|---|---|---|---|
| Context | Variable | Num | Mean±SD |
| Training as solving work problems | 246 | 4.07±0.88 | |
| Interested in participating in training courses | 250 | 4.02±0.95 | |
| The status of training environment | 249 | 3.39±0.87 | |
| Appropriate opportunity for using skills | 248 | 3.12±0.86 | |
| Facilities and substructures of electronic education | 248 | 2.69±0.94 | |
| Focusing on learners’ learning style | 246 | 2.68±0.91 | |
| Support and encouragement from managers | 247 | 2.89±0.95 | |
| Availability of educational resources | 247 | 2.87±0.99 | |
| Appropriate environment from managers and executive administrators in order to use educational content | 249 | 2.87±0.95 | |
Mean from 5
In the input phase, the maximum mean score was related to holding courses in office hours and the instructors’ interest in teaching (3.6±0.731) and the minimum mean score was related to needs assessment of training courses (2.4±1.63). The total score was reported to be 3.03 Overall, but like the context phase, this phase also lacked the necessary effectiveness (p>0.001) (Table 2).
Table 2.
Frequency distribution, mean score and response rates of questions about input
| Evaluation component | Research question: How are the staff training programs with respect to evaluation of input? | ||
|---|---|---|---|
| Input | Variable | Num | Mean±SD |
| Holding courses in office hours | 249 | 3.63±1.14 | |
| Instructors’ interest in teaching | 247 | 3.62±0.731 | |
| Using competent instructors | 249 | 3.26±0.871 | |
| Training environment( hotness, coldness, …) | 250 | 3.26±0.929 | |
| Using competent training administrators and liaisons | 249 | 3.46±0.884 | |
| Conformity of facilities with educational needs | 250 | 3.10±0.789 | |
| Panning and programming training programs before holding courses | 243 | 3.19±0.819 | |
| Scientific capability of Instructors in teaching | 247 | 3.25±0.812 | |
| Scientific capability of library administrators | 228 | 3.01±0.890 | |
| Compiling training contents based on the needs of the organization and learners | 250 | 2.69±0.924 | |
| Allocation of budget and financial resources | 226 | 2.65±0.835 | |
| Allocation of training resources ( books, …) | 244 | 2.69±0.897 | |
| Needs assessment of training courses | 246 | 2.47±1.063 | |
| Setting programs according to work activities of learners | 246 | 2.84±0.988 | |
| Sufficient resources and facilities in libraries | 237 | 2.38±0.929 | |
Mean from 5
In the process phase of the evaluation, using new educational methods and technologies had the maximum (3.50±.862) and sufficient budget and facilities for educational needs had the minimum mean score (2.6±0.855). The total score was reported to be 3.15. Overall and it was statistically significant (p<0.001), therefore it can be concluded that this phase had the necessary effectiveness (Table 3).
Table 3.
The process phase: Frequency distribution, mean score and the questions ` response rate
| Evaluation component | Research question: How are the staff training programs with respect to evaluation of process? | ||
|---|---|---|---|
| process | Variable | Num | Mean±SD |
| Applying new educational technologies( slides, overheads, …) | 248 | 3.52±0.862 | |
| Announcing the dates of holding training courses | 245 | 3.43±0.962 | |
| Conformity of number of instructors with teaching the educational content | 243 | 3.40±0.799 | |
| Introducing the goals of training programs | 249 | 3.07±0.855 | |
| Focusing on effective teaching methods | 246 | 3.01±0.815 | |
| Quality of educational tools | 248 | 3.15±0.854 | |
| Motivating learners to learn | 247 | 3.03±0.896 | |
| Using practical examples during teaching | 246 | 3.09±0.898 | |
| Up to date content of courses | 245 | 3.31±0.860 | |
| Conformity of workshop duration with training subjects | 242 | 3.04±0.910 | |
| Observing a logical continuation between subjects | 244 | 3.17±0.844 | |
| Allowing learners to express opinions | 247 | 3.36±0.978 | |
| Encouraging learners to learn | 245 | 3.33±0.900 | |
| Providing support services | 245 | 3.24±0.865 | |
| Meeting the needs and solving educational problems of learners | 249 | 3.22±0.875 | |
| Work and technical conformity of learners with each other | 248 | 3.14±0.925 | |
| Conformity of educational environment with the number of learners | 247 | 3.29±0.904 | |
| Using problem solving teaching method | 243 | 3.00±0.876 | |
| Focusing on learner based teaching method | 243 | 3.25±0.876 | |
| Focusing on skills and practical components of training | 244 | 3.06±0.906 | |
| Conformity of training duration with subjects | 244 | 2.93±0.889 | |
| Conformity of education content with the allocated time | 244 | 2.87±0.882 | |
| Sufficient budget and facilities for educational needs | 239 | 2.67±0.855 | |
Mean from 5
In the product phase, “the quality of materials offered” had the highest mean score (3.2±0.749) while the lowest mean score belonged to“the offering feedback item” (2.8±0.966). The total score was reported to be 3.13, Overall, and since it was statistically significant, the product phase has the necessary effectiveness (P<0.001) (Table 4).
Table 4.
The product phase: Frequency distribution, mean score and the questions` response rates
| Evaluation component | Research question: How are staff training programs with respect to evaluation of product? | ||
|---|---|---|---|
| Product | Variable | Num | Mean±SD |
| Receiving certificate of education | 246 | 2.55±1.011 | |
| Quality of materials offered | 248 | 3.27±0.749 | |
| Achieving goals of the program | 244 | 3.05±0.796 | |
| Effectiveness of education in action | 246 | 3.06±0.904 | |
| Desirable increase of knowledge | 246 | 3.26±0.907 | |
| Meeting scientific needs | 243 | 3.04±0.933 | |
| Evaluating information by performing pre test and post test | 243 | 3.13±1.027 | |
| Work and occupational empowerment | 247 | 3.18±0.994 | |
| Offering feedback | 243 | 2.83±0.966 | |
| Using the benefits of training programs by managers and executive administrators | 241 | 2.97±1.014 | |
Mean from 5
The CIPP Model full reports based on the participants answer are as follows:
Regarding the question “how to hold a better training course”, the most effective factors included:
Appropriate place selection with respect to proximity to workplace, holding virtual and distance courses, motivating learners and providing appropriate situation for attending training courses, in the context phase. Moreover, utilizing instructors from inside and outside of the organization, needs assessment, holding courses based on the employees work requirements and degrees, programming before implementation, holding the courses in office hours, Allocating enough budget, educational resources and facilities and general environment of the courses were the most important factors mentioned in the input phase, and last but not least regarding the process phase, the main contributing factors were : holding practical courses, applying new educational methods such as learner- based teaching, reducing the number of learners, allocating enough time to each course and announcing the dates before head .
In response to the question” whether evaluation can result in the improvement of programs and why” in the product phase:
The most influential factors in in-service training evaluation included:
Methodical evaluation
Weaknesses and strong points identification
Attending these training programs should have a better motivation rather than just receiving a certificate in education
Improving the quality of education and level of knowledge and science
Improving programs
Offering feedbacks
Discussion
The results of the current study have shown that the effectiveness of in-service training depends on various factors. Therefore, in order to be effective, these factors must improve. Regarding the training courses presented to SUMS` employees, the CIPP evaluation done in this survey indicated that, there are some strong points and some weaknesses.
According to our study, high interest in taking these courses and the training which result in solving work problems are the program strong points while the weaknesses included : the lack of educational facilities for holding virtual training courses, inaccessibility of educational resources, inappropriate environment for applying educational materials, and lack of support and encouragement by administration .These results confirm the results of other studies which aimed at investigating the weaknesses of in-service training courses such as Zandvarian et al.’s (2009), Naeeni et al.’s (2006) and Kazemi et al.’s (2009) studies (10-12) .
Like Hosseini et al in 2008 ,our study have found that factors such as allocating budget, financial and educational resources, needs assessment of training courses and take the staffs` problems in to consideration while choosing the subject and content of these courses, to be the most important issues in the component of input (1).
The results of the present study showed that the most influential factors which decrease the effectiveness of the programs are ignoring needs assessment procedures, educational-welfare conditions and facilities (12). Also, our results are in agreement with Mahmoodi et al (2012) and Yarmohammadian et al.’s. (2011) studies in which needs assessment are mentioned as the greatest change of innovation in sustaining the in-service training programs which is lower than expected (2, 13). Decision makers and education programmers should pay more attention to the necessity of needs assessment that is the first and most important stage in each organization (13).
One of other effective factors is to utilize instructors from inside and outside of the organization. In this way, the program can benefit from the insiders awareness due to their familiarity with the organization problems and also the specialty and experience of outsiders in special fields. Other factors include: programming before implementation and holding courses compatible with the employees’ work and degree during their work hours which are in agreement with Mahmoodi et al.’s study in 2010 (13).
Other training effectiveness obstacles included: lack of budget and facilities for educational needs, insufficient educational contents and time allocated to them. Items such as goals introduction methods, effective teaching style, quality of educational tools, learners’ motivation enhancement and the use of practical examples during teaching and relatively up-to-date contents must also be taken in to consideration and revise. Khotami’s study (2012) who also conducted an evaluation based on CIPP model in Saudi Arabia also emphasized on the importance of facilities and equipment, training environment and capability of instructors as a useful controlling model (14).
Other effective factors of this component may include holding practical courses, applying new teaching methods, reducing the number of learners in training sessions, learner-based training, allocating enough time for each course and announcing the time of holding courses that are in agreement with a study done by Tazakori et al.’s (2010) which reported that new methods of teaching were not used (15).
In the product phase, the employees believed that one of the main weaknesses of the evaluation is the fact that many attend these courses with sole purpose of receiving educational certificate. Other weak points in this phase included the lack of utilization of program results by administrators and their ignorance of such programs’ benefits, as well as lack of feedbacks to staffs. Likewise, in Hosseinpour et al’s study (2012) which also found motivations such as a guaranteed certificate for work promotion and raise upon fulfillment of the course as one of the trainings weak points, since it could consequently lead to lack of active participation in training and less effective program (16).
Also, results from Akhlaghi and Yarmahmmodian (2011) and Heidari et al.’s. (2005) studies showed that evaluation of the product phase was not desirable and even lower than expected indicating some problems; therefore a sustainable evaluation and identification of weak and strong points of programs should be used to improve the quality level of training courses (17, 18).
Conclusion
In conclusion, although the in-service trainings given to sums employees have been effective in many ways, they also have some weaknesses especially in the input and context phases, in which more effectiveness was expected. The product and process phases of the training, however, were desirably effective.Therefore, Improving these weaknesses and reinforcing strong points within the identified fields in this study should be taken into account by decision makers and administrators. In this regard it seems that to enhance the positive points of these training courses, an environment of creativity and productivity in all phases must be created and actively supported by the administration and those in charge of in-service training program .If done correctly and continuously, this may also lead to weak points` improvement. Moreover, needs assessment, electronic learning facilitation and the employees points of view regarding the time and place and content of the trainings must be taken under consideration in
Acknowledgment
This article resulted from the research project No.6469 funded by the Deputy of research, Shiraz University of Medical Sciences thereby their cooperation and assistance is appreciated.
Conflict of Interest: None declared.
References:
- 1.Amini M, Doostkam A, Kojuri J, Abdollahifard GhR, Iravani K, Nabeiei P, et al. Evaluation of virtual training for senior management of public health (MPH) Family Practice Shiraz University of Medical Sciences. Strides in Development of Medical Education. 2013; 10(3): 204–9. Persian. [Google Scholar]
- 2.Yarmohammadian MH. Evaluating in-service training programs of the employees of Islamic Azad University based on international standard ISO 10015, Khoorasgan Azad University. Journal of Systems Manangement Marvedasht Unit. 2011; 5(2):90. Persian. [Google Scholar]
- 3.Darabi M, MehrAlizadeh Y. Assessing the effectiveness of action research courses of the organization of Education in Khozestan province using Kirkpatrick model. Journal of nursing management. 2008; 2(2): 35–42. Persian. [Google Scholar]
- 4.Ebrahimi A. Investigating the programs of technical and vocational boys’ schools of art in Shiraz based on CIPP model [dissertation] Shiraz: Shiraz University of Medical Sciences; 2008. Persian. [Google Scholar]
- 5.Kazemi M, Sohrabinezhad N, Hamrahi M. Evaluating in-service trainings in Fars tourism industry based on CIPP model. The Journal of Tourism Studies . 2009;(10):35–52. Persian. [Google Scholar]
- 6.Frye AW, Hemmer PA. Program evaluation models and related theories. Medical Teacher: AMEE Guide. 2012;67(34):e288–99. doi: 10.3109/0142159X.2012.668637. [DOI] [PubMed] [Google Scholar]
- 7.Kojuri J, Dehghani MR, Amini M. Essential note in medical education. Shiraz: Irsa; 2009. Persian. [Google Scholar]
- 8.Bazargan A. Educational evaluation: concepts of action models and process. Tehran: Semat; 2011. Persian. [Google Scholar]
- 9.Stufflebeam D. CIPP evaluation model checklist [Internet] Netherlands: Ministry of foreign affairs of the Netherlands; 2002. [cited 2002 Apr]. Available from: betterevaluation.org/resources/overview/Stufflebeam_CIPP. [Google Scholar]
- 10.Zandvarian A, Mashayekhi N. Evaluating Maktab Alzahra Seminary of Yazd based on CIPP model. Journal of Shia women. 2009;6(22):1–4. Persian. [Google Scholar]
- 11.Zandvarian Naeini A. Evaluating Khozestan Teacher Training Centers using CIPP model. Journal of Educational Psychology. 2006;3(13):137–62. Persian. [Google Scholar]
- 12.Kazemi M, Hamrahi M. Pathology of in-service training based on CIPP model (Fars case study) Journal of management research. 2009;2(4): 113–30. Persian. [Google Scholar]
- 13.Mahmoodi MT, Hosseini Chaleshtari SGhR. A survey of the effectiveness of physicians training courses from participants’ point of view in Chaharmahal and Bakhtiari. Journal of Shahrekord University of Medical Sciences. 2012;14(4):79–87. Persian. [Google Scholar]
- 14.Alkhathami AD. Evaluation of Saudi family medicine training program: The application of CIPP evaluation format. Med Teach. 2012;34:S81–9. doi: 10.3109/0142159X.2012.656752. [DOI] [PubMed] [Google Scholar]
- 15.Tazakori Z, Mazaheri A. Evaluating PhD course in nursing in Iran based on CIPP model. Journal of school of nursing and midwifery. 2010; 6(9):12. Persian. [Google Scholar]
- 16.Hosseinpour Z, Heshmati Nabavi F. Observing adult learning principles in the implementation of sustainable training programs and the effectiveness of such programs from nurses’ point of view in Mashhad University of Medical Sciences. Iranian journal of education in medical sciences. 2012; 12(11):836–41. Persian. [Google Scholar]
- 17.Akhlaghi F, Yarmahmoodian MH. Evaluating the quality of educationalprograms in higher education using CIPP model. Journal of Health Information Management. 2011;8(5): 621–9. Persian. [Google Scholar]
- 18.Heidari Kh., editor. Assessing the database of national organization of management andprogramming based on CIPP model. The Third international conference of management; 2005 Dec; 2005. Tehran: Shahid behshti University; 2005. Persian. [Google Scholar]