Meta-evaluation of published studies on evaluation of health disaster preparedness exercises through a systematic review

Abstract

OBJECTIVE:

Exercise evaluation is one of the most important steps and sometimes neglected in designing and taking exercises, in this stage of exercise, it systematically identifying, gathering, and interpreting related information to indicate how an exercise has fulfilled its objectives. The present study aimed to assess the most important evaluation techniques applied in evaluating health exercises for emergencies and disasters.

METHODS:

This was meta-evaluation study through a systematic review. In this research, we searched papers based on specific and relevant keywords in research databases including ISI web of science, PubMed, Scopus, Science Direct, Ovid, ProQuest, Wiley, Google Scholar, and Persian database such as ISC and SID. The search keywords and strategies are followed; “simulation,” “practice,” “drill,” “exercise,” “instrument,” “tool,” “questionnaire,” “ measurement,” “checklist,” “scale,” “test,” “inventory,” “battery,” “evaluation,” “assessment,” “appraisal,” “emergency,” “disaster,” “cricise,” “hazard,” “catastrophe,: “hospital”, “prehospital,” “health centers,” “treatment centers,” were used in combination with Boolean operators OR and AND.

RESULTS:

The research findings indicate that there are different techniques and methods for data collection to evaluate performance exercises of health centers and affiliated organizations in disasters and emergencies including debriefing inventories, self-report, questionnaire, interview, observation, shooting video, and photographing, electronic equipment which can be individually or collectively used depending on exercise objectives or purposes.

CONCLUSION:

Taking exercise in the health sector is one of the important steps in preparation and implementation of disaster risk management programs. This study can be thus utilized to improve preparedness of different sectors of health system according to the latest available evaluation techniques and methods for better implementation of disaster exercise evaluation stages.

Introduction

Among several components of disaster management, the health centers and affiliated units can reduce physical, financial, and social damage due to disasters by providing the preparedness plans and appropriate strategies.[1,2,3] In developed countries, most health centers are requested to have codified programs to prepare and improve their abilities to response unexpected events,[4] and these programs should be designed in a way that; the operators, time and methods, and activate or deactivate programs are known, and the ways of hospital transferring and discharge are identified, and information communication and management are taken into account. These programs should be also up-to-dated, and all staff should be adequately familiar with program for disaster.[5] Revision and improvement of health centers’ preparedness plans for the proper and timely reaction is major role for reducing damages caused by disasters. Otherwise, carrying out disaster exercises is the most important way to create, maintain, and improve preparedness plans.[6,7]

Running exercise courses in different sectors of health system are one of the important steps to prepare and deploy disaster risk management programs, especially response phase. Exercises simulate the realistic conditions so that people improve their mental and physical skills in situations similar to real conditions and provide an appropriate response based on existing programs to emergencies and disasters.[1,8] Disaster exercises can be used for testing and validating policies, programs, procedures, teaching personnel, their roles and responsibilities, as well as improving the individual performance, and improving interorganizational communication and coordination.[9] There are two types of disaster exercises: (a) Discussion-based exercise and (b) Operation-based exercise. Discussion-based exercise includes four types of exercises: Seminars, workshops, games, and tabletop; and operation-based exercise, which is, in fact, the real exercise implementation, consists of 3 types of exercises including drills, functional-scale, and full-scale exercises.[6] Types, features, and objectives of exercises are presented in Table 1.[10]

Table 1.

Specifications and objective of disasters preparedness exercise.

Different steps should be taken to carry out an exercise evaluation. Exercise evaluation is important step in designing and implementing exercise. It systematically investigates and indicates how much the exercise has fulfilled its objectives. It also determines strengths and weaknesses of disaster exercise program.[11] After exercise, evaluators should investigate exercises during a session with all key participants and delegates of involved units. This session aims to collect information on individual performance and application of information to revise and improve preparedness program and response process. It should be noted that identification of unskilled and unprofessional staff and finding their fault, error, and poor performance is not among the objectives of the session; and programs will be reviewed and modified based on the available information and results of evaluation at the end of session.[6,11]

Evaluation is defined as the review and investigation of value and utilization of phenomena program, plan, policy, or procedure and finding a way for improving the quality of program through utilizing available proper, moral, and accurate methods. Common questions of all evaluations are as follows: Do the components of program have appropriate and effective performance? How is the good performance or best practice? Why do the program and its components do not well? How durable are the program and its consequences? Is this program more effective than other programs? What do we learn about this program? What do customers think about this program?[12] Despite numerous studies on the need for evaluation, unfortunately, evaluation is not effectively an integral part of most programs, and we usually think of a program when we are faced with problems or questions about itself.[13] Evaluation experts believe that all evaluations may be encountered with bias because evaluators’ decisions on what they investigate, what methods and tools they use, to whom they talk, and even, their professional and personal experiences, affect the implementation and result of evaluation. Therefore, we should take measures to assess credibility, validity, and accuracy of evaluation program. Therefore, the evaluation of evaluation (meta-evaluation) program should be first included in evaluation program.[14]

Given the importance of health preparedness and important strategy for doing disaster exercise to maintain and promote the preparedness for appropriate and timely response to disasters and reduction of physical, psychological, social, and economic damages of society, the present study is a meta-evaluation taking advantages of a systematic review with the aim of evaluation of related and involved units and organizations at health sector during exercises for emergencies and disasters.

Methods

The present study was a meta-evaluation through systematic review of published studies relating to evaluation of health preparedness exercises for emergencies and disasters. This study performed based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.[15]

Search strategy

This study was conducted during July 2017 to review all published English and Persian articles in the field of evaluation of health disaster preparedness exercises. For this purpose, it has been studied databases including ISI web of science, PubMed, Scopus, Science direct, Ovid, ProQuest, Wiley, Google Scholar, and Persian database such as from January 1, 2000 to June 24, 2017. The search keywords and strategies are followed; “simulation,” “practice,” “drill,” “exercise,” “instrument,” “tool,” “questionnaire,” “ measurement,” “checklist,” “scale,” “test,” “inventory,” “battery,” “evaluation,” “assessment,” “appraisal,” “emergency,” “disaster,” “cricise,” “hazard,” “catastrophe,” “hospital,” “prehospital,” “health centers,” “treatment centers,” were used in combination with Boolean operators OR and AND. Key words were combined and written in search box of databases included ([simulation OR practice OR drill OR exercise] AND [instrument OR tool OR questionnaire OR measurement OR checklist OR scale OR test OR inventory OR battery] AND [evaluation OR assessment OR appraisal] AND [emergency OR disaster OR crisis OR hazard OR catastrophe OR tragedy OR mass casualty incident] AND [hospital OR prehospital OR treatment center OR health center]). All synonyms of the key words were search with using MESH strategies.

Selection of articles and document

Independent reviewers (HS and MN) screened abstracts and titles for eligibility. When the reviewers felt that the abstract or title was potentially useful, full-text copies of the article were retrieved and considered for eligibility by both reviewers. If discrepancies occurred between reviewers, the reasons were identified and a final decision was made based on third reviewer (MY).

Inclusion and exclusion criteria

The inclusion criteria were included: Published papers during 2000 till 2017, Published in English and Persian language, Published in indexed and peer-reviewed research journal and only allocated to disaster exercise evaluation. The exclusion criteria were included: the study were reports or papers which aim to provide experiences in designing and developing exercise without testing performance, evaluation of equipment and emergency essentials, assessment of health preparedness in disasters, evaluation of clinical exercise response including clinical diseases such as respiratory, cardiovascular diseases.

Database search

The initial electronic database search of the literature resulted in a total of 5578 documents. At the next step, duplicated, books, dissertations, and presentations were filtered and the number of documents decreased to 2789 articles. Based on systematic screening, at the first stage, it reviewed the titles and abstracts to find those related to evaluation of health exercises for emergencies and disasters and extracted 123 eligible articles. In the next step, all 123 selected full-text papers were considered, and finally, 10 papers which reported evaluation of health preparedness exercises for emergencies and disasters. Figure 1 shows the search strategy and the selected articles in accordance with the PRISMA guidelines.[15]

Figure 1.

Flow diagram showing selection of articles reviewed

Study quality assessment

Quality assessment of the included studies were done using the CASP tools.[16] The score of quantitative studies ranged from 2 to 7 and majority of those did not provide any ethical statement, study design, sampling, and reflexivity related to research process. In the cross-sectional studies, only three out of ten articles[17,18,19] were used appropriate methods, also majority of them did not consider important confounding factors accounted.

Results

Demographic of studied

Number of health and treatment centers of these ten studies were consist 34 hospital, 4840 personnel (disaster management expert, staff, evaluator, volunteer, firefighters, and police). Details of each study and their special features were reported regarding exercise type and level, duration, location, year, participants, instrument type, evaluation dimensions, evaluation methods and technique, scoring, validity and reliability, and rescores. The studies were mainly conducted in United States,[18,19,20,21,22,23,24] the Netherlands,[25] Australia,[26] and Italy.[17] According to conducted studies, eight exercises were often operation-based and full-scale[17,18,23,26] and drill[19,22,24,25] and two studies on their exercises were discussion-based and tabletop.[19,21] Furthermore, the majority of disaster exercises varied from 2 h to 3 days at hospitals and during the exercises. Research results also indicated that most organizations which participated in full-scale exercises of hospitals including firefighting, police officer and infectious disease control center, and voluntary organizations.[26] The results of studies indicated that evaluation of 3 exercises[19,25,26] was carried out by external evaluator and 6 exercises by internal evaluators[18,20,21,22,24] and one exercise[23] by both internal and external evaluators.

Main results and meta-evaluation

Evaluation is among the most important stages of disaster exercises which should have prepared tools before exercise.[11] According to results of study, there are different tools and techniques to collect data for evaluating performance of health and medical systems through disaster exercise and they include self-report (completion of questionnaires by participants), questionnaire, interview, observation, shooting video, photographing, electronic equipment, as well as use of qualitative research techniques such as Delphi to prepare tool items which can be utilized alone or together depending on objective of exercise. Research results indicate that different exercises mostly aimed to enhance self-confidence in employees,[18,23,24] improve perception of preparedness, increase awareness of roles and responsibilities of our organizations or other organizations which participated in response operations,[26] implement the incident command system,[24] identify gaps and limitations of plans, programs and protocols of health disasters,[17] share information between participating organizations in disaster response program, and provide an opportunity for exercising emergency programs of organizations,[17,21,24] review and improve interorganizational communications, provide an opportunity for interorganizational collaboration, and development of knowledge, attitude, perception, skill, and behavior in participants.[18,25]

The studies indicated that the most important functions of hospitals were evaluated by exercise. Those functions included early warning system; leadership; control; coordination; inter- and intra-organizational communications; risk perception; hospital discharge process; triage; contingent planning; documentation; incident command system; and decontamination and surveillance of communicable diseases. Selection of people for the evaluation of different exercises was the most important finding which was emphasized by studies.[9,25] The results of studies indicated that selection of evaluators for implementation of standard evaluation processes was one of the most important steps in evaluation process because an evaluators decisions on what they would investigate, what methods and tools they would use, to whom they talk, and even their personal and professional experiences affect implementation and outcome of evaluation; and thus the evaluators’ training before evaluation could play significant roles in identification of strengths, weakness, and improvement ability of programs.[14,21] According to research results, the most important features of a disaster exercise evaluation tool should include experience in design, implementation, and evaluation of disaster exercises; disaster-related academic knowledge and attitude; experience in disaster response and preparedness programs as well as participation in disaster training courses.[17,19,21,23,24] Majority of studies utilized standard program of Homeland Security Exercise and Evaluation Program Guidelines to design tools for the disaster exercise evaluation.[20,22,23] Some of them also reviewed past studies and used panel of experts as the basis for the preparation of evaluation tools.[21] Lack of transparency in examination time of exercise results based on the evaluation method was one of the important points in the assessment of exercise evaluation techniques, and only the research by[24] reported the organizational performance on the basis of evaluation results immediately after holding a joint session consisting of participating officials and staff in exercise. According to the assessment of evaluation techniques, except for 4 evaluation techniques.[17,19,22,23] Majority of tools and scales were designed without any validity and reliability.[18,20,21,24,25,26] Assessment of evaluation techniques indicated that each of existing tools only evaluated a part of disaster management activities in a health sector including hospitals and health centers; and an evaluation tool could not alone cover health system disaster response programs due to diverse activities. Moreover, the summaries of each paper related to evaluation of health field preparedness exercises in emergencies and disasters are shown in Table 2.

Table 2.

Papers summary of health preparedness exercises evaluation in emergencies and disasters

Discussion

This systematic paper reviewed the latest evaluation methods and techniques of health exercises to prepare for responding to incidents and disasters. This research determined that among different health centers, most hospitals carried out different operation and discussion-based exercises to be prepared for response to incidents and disasters.[17,22,23,24,25,26] The results of conducted studies indicated that there were various techniques and methods for the evaluation of health exercises including observation,[18,22,23,26] interview,[21] photography,[24,25] shooting video,[25] and use of electronic equipment[17] by Hot wash and Debriefing.[17,26] Use of any of the abovementioned techniques depended on type and objectives of exercise, various applications, and strengths and weaknesses which should be taken into account by maneuver officials.[27] The studies emphasized that an evaluation technique cannot solely evaluate an exercise in a standard manner and it is better to investigate various dimensions of an index or performance by a combination of different evaluation methods.[26,27] Disaster experts also believe that the evaluation outcomes are valuable when disaster exercise evaluations are based on the quantitative as well as qualitative data.[6] Studies also indicate that a disaster exercise evaluation method will be superior in the case of considering different items including its ease of use, function-based nature, accuracy, transparency, reliability and validity, and compliance with cultural indices.[18,21,23]

There is not any study on success and superiority of an exercise evaluation method, but there is also a research on evaluation and effectiveness of one of the evaluation methods including shooting video and photography.[17] Benefits of shooting video as one of evaluation methods include secondary evaluation of individual performance; investigation of participants’ performance in exercise based on the time and place; possibility of playing the exercise video for participants to show their performance; recording the exercise document; better evaluation of teamwork; and management, improvement, and help in participants’ fast and stable learning; and also behavior improvement compared to the mere verbal feedback.[25]

Studies also indicate that various tools designed for evaluating different disaster exercise; and implementing these tools basically is responsibility at hospitals and are often provided as function-based in checklists, but there is not any comprehensive tool which is applicable to all health sectors including health, treatment and support sector, for designing and implementing a variety of exercises.[17,19,20,22,23,24,26] According to the study on available tools, a majority of them did not have any validation process (validity and reliability) and only 4 studies were reported validation including the validity (face, content, and construct) and reliability.[17,19,22,23] The studies also reported that a function-based tool as well as a set of valid and reliable tools (toolkit) should be designed for exercise evaluation according to a variety of exercises and their objectives.[28,29,30]

In addition to existence of a valid tool, evaluator's role is one of the most important results of studies on evaluation of health exercises. Some studies utilized the evaluators outside the exercise organization,[19,25,26] and several studies used the internal evaluators[17,18,20,21,22,24] and other exercises used both types of evaluators.[23] Studies reported that selection of evaluators was one of the most important stages of design and implementation of exercise evaluation and they emphasized that a person, who was selected as evaluator, should have academic knowledge about disaster management, emergency medicine, disaster preparedness, experience in exercise design, acceptable knowledge on preparedness of fatal incidents, and organizational plans and structure for a preparedness system during an exercise because results of an exercise evaluation could be strongly influenced by evaluator's beliefs, decisions, physical, and mental health; and they emphasized that an evaluator should play an impartial and inactive role and carry out exercise evaluation only on the basis of exercise tool.[17,19,21,23,24] The studies also report that it is essential to hold sessions for justification and familiarization of evaluators by application of evaluation tools to evaluate disaster exercise.[14,21] The most important limitation of this research was the lack of study on effective evaluation tools due to the lack of reported results of evaluation and change and improvement of disaster management programs and structures.

Conclusion

The results of literature review indicate that there are different techniques and tools which can be used based on the types and objectives of exercises to evaluate health sector exercises and improve preparedness for appropriate response to disasters and incidents. Furthermore, many existing tools are not validated (validity and reliability), and thus the localization and validation stages need to be performed for the scientific use of these tools according to culture of any community. According to an important point of studies, there is not still any scientific document for superiority of existing exercise evaluation techniques, and thus it is suggested that disaster management researchers should conduct interactive studies to assess effectiveness of various types of exercise evaluation techniques in the future. Since every tool and evaluation method can be used to evaluate one or more dimensions of performance in a health sector and given the variety of types and objectives of exercise, this paper recommended developing a valid and reliable tools (toolkit) for exercises evaluation of different dimensions of health feild including hygiene, treatment, education, and logestic. This study can be thus utilized to improve preparedness of different sectors of health system according to the latest available evaluation techniques and methods for better implementation of disaster exercise evaluation stages.

Financial support and sponsorship

This study was financially supported by Isfahan University of Medical Sciences.

Conflicts of interest

There are no conflicts of interest.