Employing Kirkpatrick's Evaluation Framework to Determine the Effectiveness of Health Information Management Courses and Programs

Abstract

Evaluation of the impact and effectiveness of courses is necessary so that strengths and weaknesses can be identified and improvements made. This article uses Kirkpatrick's evaluation framework to present a model that health information management (HIM) instructors can use to improve upon the standard course evaluation form. Kirkpatrick's model stresses evaluation on the levels of reaction, learning, behavior, and results. The proposed course evaluation model addresses the first three of these levels and focuses on the conditions necessary for transfer of learned knowledge and skills into on-the-job application. The article provides concrete tips that HIM instructors can apply in the process of evaluating the effectiveness of their courses and programs.

Evaluation of the impact and effectiveness of courses is necessary so that strengths and weaknesses can be identified and improvements made. As in most academic fields, health information management (HIM) instructors typically use the course evaluation tool their organization suggests or requires they use. It is commonly referred to as a smile sheet. Smile sheets are not necessarily of no value. They can identify whether the course was a negative experience likely to inhibit learning and application back on the job.¹ However, they do not measure learning or on-the-job application.

Now is the time to employ a more comprehensive model that incorporates the desires of all relevant stakeholders: the student, the instructor, and the employer. The following model is based on the seminal work of Kirkpatrick.

Developed more than 50 years ago as his dissertation, Kirkpatrick's framework² for evaluation has been used as a basic model for the identification and targeting of training-specific interventions in business, government, the military, and industry alike.³ It has been successfully used by Duke Energy, Arthur Andersen and Company, Intel (University), and St. Luke's Hospital, among thousands of others.^4,5 An example of its use in education includes Baskin's exploration of student-learning outcomes related to online group work.⁶

Kirkpatrick stressed that an evaluation should go beyond immediate reactions of the attendees. The study could be carried out on four different levels:

1. reaction,

2. learning,

3. behavior, and

4. results.⁷

For HIM instructors' purposes, the first three levels will suffice, as the fourth level looks at systemwide or organizational impact. This level seeks to determine if an increase in company profits, customer satisfaction, or similar measures occurred as a result of the training. A longer-term evaluation to measure holistic improvement in the workplace after the learning experience would be highly beneficial but is beyond the scope of most HIM programs.

The Model

Level one evaluation assists an organization in assessing a participant's reactions to a course's instructor, setting, materials, and learning activities. This level of training evaluation is essential. It involves gaining direct feedback. Many organizations use this level as their sole means of evaluation.⁸ The strength of this level of evaluation is the ease of obtaining the information. However, positive satisfaction numbers do not ensure learning and subsequent application of program content.^9,10

From an instructor's point of view, it is important to get good satisfaction ratings. An instructor must get favorable ratings in order to attract new participants and get current participants to return for other courses. Also, if participants are not satisfied, they probably will not be motivated to learn. So while good satisfaction ratings do not guarantee learning, bad ratings most likely decrease the probability of it occurring.^11,12 To keep things simple, I would suggest using the organization's course evaluation form. Evaluate it to determine if all the relevant information you wish to obtain will be ferreted out. You may need to add questions. It should be given immediately after the learning event.

The second level of evaluation involves determining the extent to which learning has occurred.¹³ A variety of techniques can be used for determining if the learning objectives have been met. Types of level two assessments include performance testing, simulations, case studies, plays, and exercises.^14,15 My suggestion would be to develop a pretest and a posttest. Have the students display actual knowledge of the subject before and after instruction. Quantify the results most likely using t -tests. A t -test is a statistical test used to determine if a set of results are statistically significant.¹⁶ It should be noted that although a participant may possess the knowledge, skills, and attitudes taught in the course, there is still no guarantee of application on the job.^17–19 Of critical concern to any HIM department and employing organization is the actual on-the-job application of acquired knowledge and skills.

The third level of evaluation attempts to determine the extent to which new skills and knowledge have been applied on the job.^20–22 Level three evaluation should not be conducted before completing level one and level two evaluations.^23,24 Even when satisfaction ratings are good and the learning objectives are met, transfer of knowledge into behavior may not occur.^25,26

Kirkpatrick lists four conditions that must be met for change to occur:

1. the person must have a desire to change,

2. the person must know what to do and how to do it,²⁷

3. the person must work in the right climate, and

4. the person must be rewarded for changing.²⁸

The first two conditions can be met by fathering a positive disposition toward the sought-after change and by instilling the requisite knowledge and skills to be successful.²⁹

The third condition, the right climate, refers to the participant's work environment. For knowledge and skills to be transferred, the environment must be receptive to the transfer.^30,31 Swanson and Holton maintain that the environment in which a learner earns a livelihood is more influential than the learning itself when it comes to execution.³²

Barriers to using knowledge and skills on the job include:

1. the lack of the opportunity to use one's learning,^33,34

2. the lack of the personal capacity to try out the learning,

3. a belief that the effort exerted will not change performance,³⁵

4. a belief that the desirable performance will lead to outcomes the learner values,^36,37

5. the extent to which the supervisor or manager actively inhibits the use of the new knowledge and skills,^38–40 and

6. the support or resistance that peers provide when using new approaches.^41,42

The fourth condition, rewards, can be either intrinsic or extrinsic.⁴³ Intrinsic rewards are the psychological compensation an individual gets from work. When an individual is intrinsically rewarded, he or she is energized and fulfilled by doing a job well. Extrinsic rewards are the economic rewards received from others; they include pay increases, bonuses, and benefits.⁴⁴

Many organizations avoid level three evaluations because they take time, add cost to the training and development process, and are often disruptive. For these reasons, only training programs that are critical to organizational success, that represent significant investments, or for which skill application is critical to the goals of the organization should be evaluated at this level.⁴⁵ HIM skills meet all three criteria.

Level three evaluations can be obtained in numerous ways. I would suggest a survey of the direct supervisor of the former student. It could be conducted by phone, e-mail, letter, or other means.

The fourth level of evaluation involves measuring systemwide or organizational impact of training and is generally beyond the scope of HIM course evaluations.⁴⁶ Most organizational leaders are interested in knowing how education actually improved the business in terms they understand. This is often quite difficult because some needs assessment processes do not link skills and knowledge deficiencies to business performance problems or opportunities. Only a small percentage of programs should be evaluated at level four because of the increased time requirements, additional cost, and the complexity of measuring business impact. ⁴⁷

Conclusion

Although Kirkpatrick's model was never intended to describe exactly what to evaluate and how to do it, it does provide an overview of how to proceed.⁴⁸ The model is still in widespread use.⁴⁹ Also, it is the standard to which other techniques are compared.⁵⁰ Finally, adult education practitioners generally hold this approach to be efficacious.^51–54 Given the proliferation of HIM courses and programs, Kirkpatrick's evaluation framework provides an excellent framework to determine strengths and weaknesses of HIM instruction.