Review evaluation indicators of health information technology course of master's degree in medical sciences universities’ based on CIPP Model

Abstract

Background:

Sensitivity of teaching and learning processes in universities emphasizes the necessity of assessment of the quality of education which improves the efficiency and effectiveness of the country. This study was conducted with an aim to review and develop the evaluation criteria of health information technology course at Master of Science level in Tehran, Shahid Beheshti, Isfahan, Shiraz, and Kashan medical universities in 2012 by using CIPP model.

Materials and Methods:

This was an applied and descriptive research with statistical population of faculty members (23), students (97), directorates (5), and library staff (5), with a total of 130 people, and sampling was done as a census. In order to collect data, four questionnaires were used based on Likert scale with scores ranging from 1 to 5. Questionnaires’ validity was confirmed by consulting with health information technology and educational evaluation experts, and questionnaires’ reliability of directorates, faculty, students, and library staff was tested using the Cronbach's alpha coefficient formula, which gave r = 0.74, r = 0.93, r = 0.98, and r = 0.80, respectively. SPSS software for data analysis and both descriptive and inferential statistics containing mean, frequency percentage, standard deviation, Pearson correlation, and Spearman correlation were used.

Results:

With studies from various sources, commentary of experts, and based on the CIPP evaluation model, 139 indicators were determined and then evaluated, which were associated with this course based on the three factors of context, input, and process in the areas of human resources professional, academic services, students, directors, faculty, curriculum, budget, facilities, teaching–learning activities, and scientific research activities of students and faculty, and the activities of the library staff.

Conclusion:

This study showed that in total, the health information technology course at the Master of Science level is relatively good, but trying to improve and correct it in some areas and continuing the evaluation process seems necessary.

INTRODUCTION

In the changing world, which increasingly is adding to uncertainty, all higher education institutions should provide favorable responses to social needs. Experience has proven that universities can provide best services to the community if they have concerns of continuous improvement in the quality of their services.[1] Higher education over the past two decades has been faced with many challenges and issues; some among them are the following: Failure to produce theoretical knowledge and use of the substantive and theoretical knowledge in other countries, lack of applied academic training, lack of proper relationship between universities and other sectors of society, neglecting research functions and providing service in the universities, the problem of multiple decision-making centers and multiple custodians,[2] growth of students and applicants in universities and higher education institutions, quantitative expansion of higher education, regardless of the existing capacity and economic, social, and cultural potential of the society for acceptance of the graduates, reduced funding, and pressure from the community to take responsibility and accountability.[3] Higher education system, to deal with these challenges, must maintain, improve, and promote quality in higher education. To succeed in this case, all the functions of higher education should be considered equally and by using valid and appropriate criteria and measures to evaluate their operations and processes continuously.[4]

Entezari, in his article entitled “A model for harmonization of higher education system with the knowledge-based evolution the employment system; the fifth economic, social and cultural development plan of Islamic Republic of Iran,” states that interpretation of human source as a key factor of production has led economists and policy makers in developing countries to believe that by investing in education and learning can achieve long-term goals of economic development.[5]

A researcher proposed an alternative approach with the ability to change perspectives of quality from focusing to control activities to focusing on improvement of activities.[6] Another researcher in the article entitled “The phases and paradoxes of educational quality assurance” refers to quality assurance applied strategies in Singapore education system. This study describes the quality assurance of fuzzy model of Singapore education system.[7] Another research concluded that there are seven dimensions which are used to evaluate faculty and educational administrators in higher education institutions. It consists of leadership for teaching, leadership for research, fair and efficient management, vision and strategy, participative leadership, developmental, recognition, and interpersonal skills.[8]

Unfortunately training programs of advanced graduate study that are delivered by the ministry do not have the evaluation part, and this issue is highly important as the educational departments need to design the evaluation indicators in order to judge the programs’ characteristics and quality continuously without bias, and determine the dark spots of the program and achieve the evidence to establish proper development in training.[9]

Over the last 10 years, the most important issues of higher education in our country were declining qualitative indicators. Quantitative growth of higher education, regardless of the lack of resources and the high sensitivity of the society to this system, make necessity the attention to the effectiveness and efficiency of the higher education system. Awareness of efficiency and effectiveness of the higher education system requires an accurate assessment.[10] Evaluation helps higher education to improve quality, and also makes it accountable. Also, using a systematic approach can facilitate the usage of evaluation because it can be assessed on each element of the education system (input, process, and output), and judge them.[2]

Studies show that the quality of M.Sc. course of health information technology programs to realize the goals and objectives of the Higher Education Council planning is weak and needs to be upgraded, and most of the interviewees are of the opinion that the adoption goals of this course are appropriate and have high compliance levels, but in fact, these goals will not be pursued and materialize. So, we must focus on the quality of this educational course with consideration on the objectives intended.[11]

Success of universities and colleges of education in achieving the goals and expectations of education requires attention, review, and improving the quality of education. Quality in higher education is defined as compliance to the current status of each of the elements and principles of education system including input, process, and output with standards, goals, and overall and detailed expectations.[12] The main advantage of a course evaluation is that with a systematic approach to the evaluation and its performance, strengths and weaknesses of the course will be cleared. This can help decision makers and policy makers at the macro level to take correct and master actions.[13]

Internal evaluations in the Department of Management and Medical Information Sciences School have shown that health information technology department in this faculty, in the component of the mission and goals of teaching–learning process and the organizational structure and management is good, in the research process is bad, and in the other criteria, the faculty and students, courses and curricula, and educational facilities, is relatively good.[14] Other researchers in their study showed that the average quality improvement of education in Isfahan University faculty is 2.72 on the Likert scale.[15] A research in Washington State University has made some changes to the process to evaluation of students’ academic finding in scale of related to helping students learn experiences. Results show that taking full revision of the assessment processes, particularly for public education, is essential to develop a systematic process to describe students’ findings and priorities to support student learning.[16]

Application of analytical methods (evaluations) for appropriating activities of higher education system is important because higher education covers a significant percentage of the population and imposes huge cost to the economy. So, if it fails to achieve its objectives, in addition to wasting money, it would have adverse consequences of training unskilled and inefficient manpower. During evaluating the quality of higher education inputs (faculty and students), processes (methods of teaching–learning) and outputs (graduates and generating new knowledge) should be tested.[17] Practical and comprehensive model of CIPP (context, input, process, product) for improving the quality can help managers and planners prioritize needs based on a systematic approach, gives attention to context, input, resources, processes, and results, and also monitors stage by stage and pays the available resources to serve the best type of activity with continuous monitoring program.[18]

Features of medical universities are an important mission of higher education, and they are responsible for an important part of health care and also minor and major research done in this series.[19] The level of professional competence and the efficiency of medical graduates for health services in order to maintain and improve the level of personal and social care depend on the realization of training objectives.[20] The health care field is currently experiencing a change in the pattern and these changes will affect all health workers. Concurrent with these changes, health information technology also, both in terms of composition and reliance on information technology, is growing every day.[21] In the United States, health information technology specialists are known as health information management.[22] Dramatic changes in health information management require new thinking in relation to training tomorrow's professionals. Due to changes and increasing dependence on information and communication technologies, students’ training should enable them to learn critical thinking, creative problem solving, data recovery management, effective communication, and continuous learning.[23]

This study was conducted to gain knowledge about the quantum of achievement of educational objectives of health information technology master's degree. It is clear that identifying the strengths and weaknesses of this course will help the authorities to correct the weaknesses of the course and try to stabilize and reinforce the strengths, which eventually will promote the quality of this educational course.

MATERIALS AND METHODS

This was an applied, descriptive, and evaluation research conducted at five medical sciences universities of Iran which were executives of advanced graduate study in the health information technology field in 2012. All the people who were involved in the master's course of health information technology, and Based on educational evaluations and evaluation methods in the evaluation of different educational courses, all participants which were involved in the master's course of health information technology, were interviewed and studied. The statistical population consisted of four sub-groups of directorates (5 people), faculty members (23 people), students (97 people), and library staff (5 people) of health information technology course in five Iranian medical sciences universities, and contained 130 people in total, and sampling was done as a census. In order to collect data, four questionnaires were used based on Likert scale ranging from 1 to 5; the scoring was based on the answers, very low, low, average, high, and very high, respectively. Questionnaires’ validity (content validity) was confirmed by consulting with health information technology and educational evaluation experts, and the Cronbach's alpha coefficient formula was used to determine questionnaires’ reliability of the directorates, faculty members, students, and library staff, which showed r = 0.74, r = 0.93, r = 0.98, and r = 0.80, respectively. In order to collect data, we used four separate questionnaires,[12,24,25,26,27,28,29,30] based on the factors in the three areas of CIPP evaluation model, including two parts of demographic information and multiple choice questions based on Likert scale rating that were divided into five grades. (Directorates’ questionnaire consisted of 11 open demographic questions and 49 closed Likert questions; faculty members’ questionnaire consisted of 11 open demographic questions and 64 closed Likert questions; students’ questionnaire consisted of 9 open demographic questions and 37 closed Likert questions; and library staff's questionnaire consisted of 6 open demographic questions and 13 closed Likert questions.) SPSS software was used for data analysis, and both descriptive and inferential statistics containing mean, frequency percentage, standard deviation, Pearson correlation, and Spearman correlation were used. In this research, first the field and goals of the training course were studied, and then, by using facilitate decision-making model of CIPP, indicators of educational system that are factors of evaluation were determined. In determining indicators and making questionnaires, the following were observed: Getting and presenting license and referral to the educational department of universities, getting permission and approval from the authorities, and keeping the information confidential. These factors included: Context (human specialists and scientific services for needs of the local community), input (head of department, faculty, students, curriculum, funding, training facilities), and process (activities of group manager, students, administrators of library; scientific research and teaching–learning activities of faculty). By using a focus group discussion (FGD) and interviews with professors and professionals, educational courses, indexes through review of various sources, and scholars’ interpretations were determined (the mean score of their opinion), and then according to these specified criteria, questionnaires were developed. Through them, the current situation of selected universities was found. Some consider the FGD and interviews as one way to gather information and others consider it as one of the qualitative approaches; but in any case, it is an interview which is done in a semi-structured group led by a certain person in an informal condition. So, group's dynamics increases the quality and quantity of information and is not based on only a single researcher's or a participant's opinion. Because of the characteristics of the group, new ideas will also be established. This method is used when sharing different views and opinions (mind share) of people is important. Therefore, this method does not work in isolation, but rather tries to understand the mentality of the entire population. The number of samples depends on the subject, but those who are selected usually have common role and experiences. Usually 5–7 is the best number of members to be had in the group.

RESULTS

Demographic data show that among the five directors, four were females and one was male. Four of them were PhDs and associate professors, and one had an MS and was a lecturer. The mean age of them was 44.8 years and the mean of management experience was 6.6 years. Among the 23 faculty members, 86.95% were females and 13.04% were males; 39.13% were MS holders and 60.86% were PhDs; 56.52% were lecturers, 34.78% were assistant professors, and 8.69% were associate professors. The mean age of them was 40.63 years and the mean of teaching experience was 12.42 years. They had a mean of 2.66 research projects, 4.68 Persian articles, 1.40 Latin articles during three recent years, and 1.18 translated and pirated books during their teaching period. Among the five library staff in five universities, four were females and one was male, with an average of 9.6 years of management experience. Among the 97 students, 29.89% were males and 70.10% were females, and 32.98% were indigenous and 67.02% were non-indigenous. They had a mean age of 27.52 years, mean undergraduate average of 17.17, and an average time of 3.43 h a day for studying.

The objectives of this study were setting the evaluation criteria for master's degree in health information technology course in context, input, and process areas and determining the rating of them in Tehran, Shahid Beheshti, Isfahan, Shiraz, and Kashan Medical Sciences Universities. To achieve these goals and evaluate this training course, 13 factors associated with 139 indicators were tested. In the context of the training course area, two factors of human specialists and scientific services for needs of the local community with 20 indicators; in the input of the training course area, six factor of directors, faculty members, students, curriculum, funding, and training facilities with 54 indicators; and in the process of the training course area, five factors of directors, students, faculties’ academic research and teaching–learning, and library information administrators’ activities with 65 indicators were tested. Highest level of acceptability was related to the faculty's factor (4.47) in the input field (the human resources area) and the lowest level of acceptability was related to the budget's factor (2.93) in the input field (the fund resources area). In total, in the three areas that were studied, Isfahan University of Medical Sciences obtained the highest level (3.75 score) and Kashan University of Medical Sciences obtained the lowest level (3.38 score).

On determining the desirability of these factors, the results shown in Table 1 were obtained.

Table 1.

Areas of course's evaluation, scores, and their acceptability level based on CIPP model

As shown in Table 1, the average scores were obtained from Likert scale, ranging from 1 to 5. If the calculated score for answers of a question is between 1 and 2.33, it is an undesirable situation. If the score is between 2.33 and 3.66, it is a relatively acceptable situation, and if the score is between 3.66 and 5, it is an acceptable situation.

Spearman correlation test showed that there was a significant inverse relationship between the academic rank of faculty members and the score of the context in human resources specialist required for the local community (r = −0.415, P = 0.049). This means that the faculty members who had higher rank, had less belief in the necessity of these goals [Table 2].

Table 2.

Mean and standard deviation of context area's item separation of the academic rank of faculty members in all the universities under study

Pearson correlation test showed that there was a significant inverse relationship between the teaching background of the faculty members and their academic research and teaching–learning activities (r = −0.457, P = 0.049) and (r = −0.540, P = 0.028). This means that whatever experience the teachers had, they had less belief in evaluation indicators in these fields.

Pearson correlation test showed that there was a direct correlation between the final grades and scores of students’ activities (P = 0.005, r = 0.339). This means that academic and non-academic activities of students with higher grades are more likely to have more points in this field.

Spearman correlation test showed that there was a significant direct relationship between the rate of interest of students in their field and student activities (r = 0.25, P = 0.003). This means that the students who had more interest in their field had more academic and non-academic activities and more points in this area [Table 3].

Table 3.

Mean and standard deviation of academic and non-academic activities of students in selected medical sciences universities

DISCUSSION

Regarding the course's objectives, the results of field and context show that the goals set for this course are good and acceptable, but revising them according to the updated knowledge is necessary.

Dixon in his study concluded that due to changes and increasing dependence on information and communication technologies, students’ training should enable students to learn critical thinking, creative problem solving, data recovery management, effective communication, and continuous learning.[23]

Results of Zandvanian's study entitled “Comprehensive evaluation of teaching training institutions in Khuzestan based on CIPP model” show that current targets of this institution are quite favorable. Also, the educational context of the center was evaluated by six criteria which indicated that it was desirable.[31]

Skurka in her article entitled “Health information management education in USA: Curriculum and competencies in conjunction with the 21^st century” concluded that in all revised curricula, emphasis is on information technology, changes in the health care labor market, and the increasing need for health care professionals with more qualifications and competence in information management and computer applications. All health information management programs must have the mission and goals which coordinate and are consistent with the new curriculum.[32]

The results of these studies are consistent with the present research results wherein the majority of the respondents believed that the approval goals for this educational course are appropriate and have high level of compliance, but in action, these goals are not pursued and will not materialize. So, we must focus on the quality of this course with these intended objectives and training students with the focus on information and communication technologies and abilities.

In the input field, results of this study show relatively acceptable level and weaknesses in the curriculum, budget, and facilities. Weaknesses factors are in the curriculum Appropriateness with the approval outline, previous lesson requirement, career interests and abilities of students, expectations and needs of students, progress of science and technology, specified size and time, the rate of providing course's goals by the curriculum, and the rate of relationship and sequence between curriculum. Weaknesses of the budget factors are in the Appropriateness of budget with the educative, research, and library needs. Weaknesses of the facilities factors are in the Appropriateness of facilities with educative and research needs, library recourses, and number of computers.

Findings of Bazrafshan's study entitled “Evaluation of educational inputs of education programs of medical library and information MSc in medical sciences universities in the country based on CIPP model” indicate that curricula from the educational inputs view have many limitations. Not updating the curriculum, lack of proper research and educational facilities, and shortage of faculty expertise were the most encountered problems of curricula in the country.[33]

Akhlaghi et al., in their study entitled “Evaluating the quality of educational programs in higher education using the CIPP model,” concluded that the continuous evaluation of medical records groups and medical records education courses would determine the strengths and weaknesses of the programs and improve their quality. Also, they stated that there are many limitations in budget and facilities resources in these groups.[34]

The results of Ghadipasha's research which evaluated the mathematics bachelor's degree in input area at the state universities of Tehran with five factors and 46 indicators included the following: Students, faculty, and directorates were good, curriculum was relatively good, and the facilities were unfavorable.[26]

Westbrook et al., in their study entitled “A glimpse into the future of health information management: A survey of the expectations and ambitions of Australian health information management students,” concluded that health information management career is strongly dependent on the potential of curricula and future of health care professionals largely depends on the ability of training programs to meet the specific needs and to create training students with skills that allow them to respond to the demands of the health system appropriately. Drafters of health information management training programs should take advantages of this strategy for improving the quality of education to enhance the skills and competencies of graduates in the workplace.[35]

These studies confirmed the results of the present study and emphasize on bringing in changes in teaching students in the course because training is not appropriate and sufficient for graduates to enter the job market and this issue can cause unemployment, dissatisfaction with their performance in education, and dissatisfaction with the functional groups and the educational curriculum that was taught. Also, all the above researches have shown weaknesses in curriculum and course outlines, which should be revised and reformed with the advancements in science, keeping in mind the future needs of students in the field of new technologies and having updated curriculum. To enhance the quality of teaching, curriculum review and revision seems necessary, and educational evaluation of the course should be done at least every 5 years. Weaknesses of the budget factors that are in the Appropriateness of budget with the educative, research, and library needs, and also weaknesses of the facilities factors in the Appropriateness of facilities with educative and research needs, library recourses, and number of computers are the common results in all the above-mentioned studies.

In the process field, results of this study show relatively acceptable level and weaknesses are in activities of students, faculty, and library staff. Weaknesses of the students’ activities are in the amount of extracurricular and research activities and participation in solving problems of their educational group; weaknesses of faculty activities are in educational–training processes; and weaknesses of library staff's activities are in updating the resources in the library.

Pakdaman in her research entitled “Evaluation of the achievement of educational objectives of the Community Oral Health and Periodontics Departments using the CIPP model of evaluation – Students’ perspective” in both context and process areas shows that there are significant differences between the two departments. Reload subheads in the areas where the achievement was poor for upgrading program is recommended.[36]

The results of Ghadipasha's research which evaluated the process of mathematics bachelor's degree at the state universities of Tehran with four factors and 40 indicators included the following: Student activities (relatively desirable), faculty training activities (relatively desirable), faculty research activities (unfavorable), director of group (desirable), and the activities of the library staff (relatively desirable).[26]

Results of Zandvanyan's thesis titled “Comprehensive evaluation of teaching training institutions in Khuzestan based on CIPP model,” which examines the process area by using eight criteria, show the relatively desirable level of this area.[31]

The results of these studies are consistent with the present research results, as all have shown relatively acceptable level of the process area of course evaluation. All of them prove weaknesses in students, faculty, and library staff activities, which must be improved by suitable strategies. It is necessary that curriculum planning, course content, methods of teaching and learning, research, etc., are up-to-date and flexible and are always disposable into the new experiences.

In total, all three areas of courses that were evaluated had acceptable and relatively acceptable level in the five medical sciences universities which were reviewed.

Yarmohammadian's study goal was to determine the educational and research condition of the teaching department of the management and medical information faculty in the Isfahan Medical Sciences University in the period 2003–2004 through internal evaluation as the first step of the accreditation process. Results show that medical records department with the mission, goals, and teaching–learning process, the optimal organizational structure, and administration was good, in research was unacceptable, and the rest was relatively good.[37]

The results of his study on the mission and goals, structure and administration areas in that faculty are the same as obtained in this research, but in teaching–learning process, library activities, and research areas, they are different because our results show relatively acceptable level of these factors and weaknesses in them.

Yarmohammadian et al., in an article entitled “Evaluation of quality of education in higher education based on Academic Quality Improvement Program (AQIP) model,” assessed the quality of medical records of group of four universities from the viewpoints of students and faculty of the universities in terms of standard models of AQIP and its nine criteria. All these universities were found to have relatively favorable status in these nine dimensions and there was not any difference among them, but there were differences between the attitudes of students and faculty.[38]

Lagrosen conducted a research entitled “A review of the educational quality aspects in higher education.” Results show that 11 quality criteria were identified: Group collaboration, information and accountability, offered subject courses, facilities of university, teaching activities, internal assessments, external assessments, computer facilities, collaboration and comparing the factors after research, and library resources. From the 11 quality criteria, 7 factors had an average higher than 5 in the seven-degree scale.[39]

At the University of Wisconsin, CIPP model had been used for evaluating clinical researchers’ training programs. Four factors of CIPP model (context, input, process, and output) and the participants (trainees) were evaluated to analyze the effectiveness of programs to ensure the success of students.[40]

The results of these studies are consistent with the present research, and show that context, input, and process areas and their subgroups and indicators are in good position. Also, the MSc course of the health information technology field in the country is in a relatively acceptable condition.

CONCLUSION

The most important and desirable objective of evaluation in education and educational system is the awareness of the current state and to find the distance with the desired state in the phenomenon or object of evaluation, which can be achieved by relying on data from the evaluation process with a comprehensive and strategic planning to improve the condition and maximize the use of available resources to achieve the desired goals and objectives. The results of this research will form the context for similar studies in other groups, and also will help the group members to identify the strengths and weaknesses in teaching–learning programs and try to overcome the disadvantages in order to improve the quality of education.

Results of the research show that all areas of context, input, process, and output are placed in relatively acceptable category. The results of the study indicate that continuous planning evaluation of the health information technology groups results in a better analysis of strong and weak points of the plans and improves their quality levels. So, the ongoing and continuous evaluation of educational system's quality and improvement of educational programs and courses in order to maintain the dynamics of educational system for coordination with the advancement of science and knowledge are essential, and developing an acceptable standard for the evaluation of different degrees in different fields is necessary.

RECOMMENDATIONS

Due to the level of acceptability obtained that shows weaknesses in some areas and factors which were evaluated, attention to the following factors is essential: Balance between the job's interests and capabilities of the students and curriculum program, and also change in training courses according to the progress of science and technology; increase the funding for research needs, extracurricular activities of educational courses, and as per students’ welfare; apply the necessary and suitable reforms for increasing students’ motivation for conducting educational and research activities; make efforts to improve the teaching–learning process and extracurricular activities; increase students’ participation in decision making to improve the condition of library; provide the context of student participation in decision making and problem solving of groups by directors; strengthen the databases used by students and organize the state budget and the budgets of the department in education and research dimensions; and increase the activities of the library responsible for providing library books and magazines based on the need and number of students.