A Cognitive Task Analysis for an Emergency Management Serious Game

Abstract

The Bethesda Hospitals' Emergency Preparedness Partnership identified a need to design training systems for hospital emergency management scenarios that included incident command situations. As part of this partnership, the National Library of Medicine (NLM) was challenged to develop an engaging, learner-centered simulation to specifically address hospital procedures for highly infectious diseases (HIDs) for multiple hospital roles. A serious game approach was selected for the simulation because collaborative (multiplayer) immersive, game-based simulations have been proven to generate realistic and engaging learning experiences and, when properly designed, can enhance training while minimizing cost compared to full-scale disaster exercises (Spain et al., 2013). Although substantial research effort has been put into design and evaluation of serious games, less time has been spent on developing sound instructional design methodologies to support serious game development. So how does one collect the appropriate, relevant, contextualized content and then align with serious game design elements? This paper describes how a cognitive task approach supported by a live demonstration with a think-aloud protocol was used to collect the rich psychomotor, procedural, and cognitive data necessary for the design of a serious game for handling HIDs. Furthermore, the paper presents a process to translate the collected data into meaningful content to support rapid prototyping. Recommendations for data collection and translation for a serious game close the paper.

Introduction

Infectious diseases are the leading cause of death worldwide (“Infectious diseases,” 2014). Given this fact, and given the largest outbreak of Ebola occurred in 2014 (“Genetic clues to the 2014 Ebola outbreak,” 2014), the Bethesda Hospitals' Emergency Preparedness Partnership (BHEPP) identified a need to improve hospital staff readiness to manage emergency events that involve patients with highly-infectious diseases (HIDs). BHEPP reached out to their partner, the National Library of Medicine (NLM), for their expertise and support. As an organization, NLM provides information services that promote health literacy, improve health outcomes, and reduce health disparities by strengthening workforce skills and enabling service delivery mechanisms. NLM works collaboratively with local communities to explore innovative, plausible, scalable solutions. Within NLM, the Disaster Information Management Research Center (DIMRC) provides the resources for disaster remediation and conducts research on information solutions to support disaster preparedness. DIMRC took the lead within NLM to explore disaster prepared solutions for HIDs. Their focus was on staff preparedness for suspected HIDs patients from point of arrival at the hospital to patient care outfitted in personal protective equipment (PPE). DIMRC would develop a training product to meet this need which would also support their organizational goal to research new innovative technologies for information solutions.

DIMRC decided to pursue virtual worlds (VWs) as the information technology with the intention to push to a virtual reality (VR) delivery mechanism if the preliminary desktop prototype proved plausible and advantageous. Virtual worlds are becoming an accepted technology for clinical and healthcare professional development with an attraction to its technology affordances of immersion, team-based activities, realistic environment, safe practice, and relative low cost compared to customized game development (Heinrichs, Fellander-Tsai, & Davies, 2013; Primack et al., 2012). As part of their research effort, DIMRC was interested in:

• Exploring the applicability of VWs for training hospital staff

• Developing in-house knowledge, experience, skills, and tools on VW and VR applications

• Applying instructional design to a VW application

• Researching the usability of a VR-based training product

This paper describes how a cognitive task analysis (CTA) was applied to collect and convey instructional design requirements for a prototype VW-based training product to maintain hospital staff readiness for handling potential HIDs patients. The training product would be designed as a serious game to intentionally include fun, immersive gameplay elements. In the Background section below, additional context about the training need is provided. The subsequent section explains the purpose of a CTA and describes the particular type of analysis method used in this project. For purposes of this paper, a literature review was conducted after the preliminary prototype was developed to review other documented occurrences of using a CTA for serious game design. The Method section describes how the CTA was conducted and the ensuing analysis procedures. The paper concludes with a discussion of the results and recommendations for using a CTA for serious game design.

Background

Initial DIMRC discussions with the local BHEPP community hospital educators led to the identification of initial training goals. Given hospital staff work 24/7, 365 days a year, there was a need for consistent, reliable training on the skills associated with caring for possible HIDs patients. While a serious game was considered an appropriate solution to enhance the existing training and maintain readiness, hands-on training would continue to provide first-hand psychomotor learning that affords staff the opportunity to recognize personal limitations of time spent in full personal protective equipment (PPE) when caring for a potential HIDs patient. The training needed to be simple, focused, quick, engaging, and easy to use that addressed procedures while fine motor skills would be addressed in the hands-on training. Multiple training topic areas were identified as indicated in the prototype's teleport navigation menu shown in Figure 1. Training completion needed to be tracked whereby completion would be measured by accurate accomplishment of the procedures. The target audience was reflected in the care path of a potential HIDs patient: registrars, nurses, technicians, and doctors. The target audience was approximated as 125 staff members who would likely complete the training annually.

Figure 1. Prototype's Teleport Navigation Menu.

In addition to these training goals, DIMRC and the hospital educators were also interested in determining if the modified CDC guidelines would work with the hospital staff and within their physical environment, hence the desire for a three dimensional learning space. Additionally, the hospital educators wanted the simulated environment to replicate their existing facility to increase learning effectiveness through their site-specific visual cues but to also provide healthcare staff a tool to evaluate and potentially enhance their healthcare procedures according to their physical environment conditions. DIMRC and the hospital educators were also interested in staff perceptions of the training as well as exploring the ability to monitor learner progress. The serious game design began with these basic training goals and explorations.

Cognitive Task Analysis

Experts are unaware of how much unconscious knowledge they use when performing a task. For some, their actions and decisions rely on as much as 70% automatic, unconscious knowledge to complete the task (Clark & Elon, 2006). The purpose of a cognitive task analysis (CTA) is to capture this rich unconscious knowledge necessary for proficient task performance. CTAs have been used in the medical profession to extricate the thought processes in complex decision-making and deconstruct experts' unconscious automated skills (Riggle, Wadman, McCrory, Lowndes, Heald, Carstens, & Hallbeck, 2014). From an instructional design perspective, the CTA output needs to describe the implicit and explicit knowledge to support the actions and decisions made by the expert as well as the context in which the task is performed (Jonassen, Tessmer, & Hannum, 1999). Furthermore, the output should include learning objectives, conceptual knowledge, and procedural knowledge, including cues and decision points, plus expected performance standards (Clark, Feldon, van Merriënboer, Yates, & Early, 2008).

A CTA approach was selected for this project because we were particularly interested in understanding why certain tasks were performed as prescribed; striving to capture the thinking behind the tasks and not just the how. This became readily apparent during an initial interview with the hospital trainers. They indicated that nursing staff commonly look for efficiencies to increase productivity. If an efficiency is perceived available, they will apply it. Hence, the hospital trainers noted that staff work best when they feel they understand the why, so that whatever efficiency they identify, they're making a good choice. Additionally, the procedural documentation was declarative in nature, task-oriented knowledge without explanation, reasoning, or the challenges to successful performance.

Many types of CTA methods exist depending on the type of task being explored, the knowledge supporting that task, and the problem that needs to be addressed. Cooke (1994) identified three broad families for knowledge elicitation to support a CTA based on the mechanics of the elicitation method as shown in Figure 2. First, observations and interviews relies on watching and talking to experts and supports task analysis methods. Second, process tracing is suited for tracking specific tasks via pre-determined data sources such as verbal reporting from think-aloud protocols, eye movements, and actions taken (nonverbal) from which cognitive inferences are made. Lastly, conceptual techniques are focused on developing domain concepts, structures, and interrelationships. We chose to use observations and interviews since the task procedures and challenges were not initially well-defined (Clark et al., 2008; Wei & Salvendy, 2004).

Figure 2. Cognitive Task Analysis Methods.

CTA For Serious Game Design

A literature review was conducted after the preliminary prototype was developed to explore how other researchers have used a CTA in the development of a serious game. Serious game design is a complex process, requiring the combination of ensuring learning objectives are met while simultaneously applying meaningful gameplay elements (Kelle, Klemke, & Specht, 2013). Furthermore, it is a multidisciplinary process relying on expertise in pedagogy, subject matter domain, programming, and gameplay (Reese, 2013). Little research was found at the intersection of CTA and serious game design even though the CTA output must support the serious game design.

Serious game design research commonly focused on design frameworks and applicable learning theories. de Freitas and Oliver (2006) developed a framework to support the design and evaluation of a serious game or educational simulation. The 4D Framework addressed pedagogic considerations, learner specifications including learning paths, learning context, and representation, i.e., fidelity, interactivity, and immersion. Dalgarno & Lee (2010) developed a framework that draws from the affordances of a virtual world, more specifically, representational fidelity, learner interactions, construction of identity, sense of presence, and co-presence which is perceived to yield the learning benefits of spatial knowledge representation, experiential learning, engagement, contextual learning, and collaborative learning. Fowler (2015) expands this framework to include more pedagogical descriptors, i.e., learning stages, objectives, and activities. Carvalho, (2016) developed an Activity Theory-Based Model of Serious Games (ATMSG) that aligns gaming components (e.g., game interactions and game elements) with instructional components (e.g., learner goals and learner actions). Additional theories have also been applied in the design of virtual worlds and serious games such as constructivist, experiential learning, and flow (Loke, 2015; Tsai et al., 2016). Many more design frameworks can be found in the literature (Kapp & O'Driscoll, 2010; Raybourn, 2014; van Staalduinen & de Freitas, 2011). Yet these new design frameworks and theoretical foundations do not address how to collect and align content with them. How does one collect the appropriate, relevant, contextualized content and then align with serious game design elements?

A CTA can provide an abundance of information useful in online learning environments (van Gog, Sluijsmans, Joosten-ten Brinke, & Prins, 2010). Reese, Tabachnick, and Kosko (2015) describe a serious game approach that relies on analogical reasoning theory, cognitive task analysis, and knowledge representation to develop cyberlearning through game-based, metaphor-enhanced learning objects (CyGaMEs). This approach required the use of cognitive task analysis to develop a 101-node concept map on lunar science (Reese, 2014). Using an iterative process, 18 subject matter interviews were conducted, notes taken, diagrams drawn, and concepts related via indexed note cards. Seventy-six concept nodes were numbered for inclusion in the serious game. Each numbered concept node was cross-walked against the game elements to ensure knowledge requirements were addressed in the game elements. Concepts could be addressed in multiple game elements. For example, the concept of collision was addressed in the game elements Accretion animation, projectile collisions in protoMoon, and projectile collisions with the Moon surface in Surface Features. This cross-walk specification led to an iterative development whereby game designers and developers would at times make minor modifications to the concept map to accommodate gameplay.

Similar to Reese (2013), Borro-Escribano, del Blanco, Torrente, Martinez Alpuente, and Fernandez-Manjon (2014) developed a serious game development approach using an iterative process that required interviewing subject matter experts but they did not use a CTA process. They applied this development approach to transplant management procedures whereby their goal was to provide standard case histories that could be taught independent of instructors as initial training. A four step, iterative development process was created that explicitly brought game designers and subject matter experts together for “tight collaboration.” The first step, Specification, required game developers to interview the subject matter experts which they deemed difficult because “experts tended to overlook information that is not obvious for non-domain experts.” The next step, Game Design, transformed the Specification output into game elements including a game progression flowchart. During this step, the game designers also addressed instructional design elements such as devising alternate incorrect options and developing feedback mechanisms. In the next step, Simulation Development, a game-authoring tool was used to develop the game elements for review and revision. In the last step, Quality Assurance, the product is reviewed for content accuracy, software reliability, ease of use, educational value (learning goals addressed), and engagement. These four steps were repeated with each iteration adding refinement until the product is ready for release, basically, an agile software development process was applied.

Borro-Escribano et al. (2014) relied on game developers to address instructional design. They noted that capturing, formalizing, and adapting domain knowledge into a game design was a difficult task. Perhaps instructional designers applying a CTA may help bridge this perceived difficult task. This paper adds to the literature on applying a CTA to serious game design. The paper describes the CTA process used and how the collected data was translated to a design document with storyboards for prototyping a simulation-based serious game.

Method

The serious game team included DIMRC staff and their summer college student interns for game architecture, game programming, and graphic artist requirements; ICF International staff for instructional design; and two lead hospital clinical educators for subject matter expertise (SMEs) who were registered nurses and had worked with HID patient policy and procedures. An agile software approach was applied to provide multiple cycles, i.e., sprints, to iteratively support the CTA and subsequent design and development of a prototype. Each sprint would last three weeks with two weeks to conduct the CTA and analyze the data, followed by one week for client review and discussion. Three sprints were conducted; one for initial training requirements followed by two CTA-based sprints. The next section discusses the CTA process used in this project.

Conducting the CTA

To inform the serious game design, the team had to inquire about the hospital's emergency procedures at a very detailed level from an expert's perspective. Since the game would both instruct and assess the player, the team had to collect information to serve four purposes as indicated in Table 1. More importantly, the team required an expert to walk through situations, actions, consequences, and outcomes while describing his or her thought process. In life or death situations, information cannot be omitted, and equally, all discretionary decisions need to be explained, i.e., an instructional designer cannot work with an answer such as, “It's just a gut feeling; something I can't explain.”

Table 1. CTA Purpose.

Purpose		Description
1	Didactic Instruction	Instruction the player receives before or during gameplay
2	Virtual Environment	Scenery, equipment, characters, visual and audible cues
3	Storylines	Situations that can occur and how they typically unravel in a particular sequence (common mistakes, critical incidents)
4	Actions & Outcomes	Correct actions, outcomes of correct actions, incorrect actions, consequences to incorrect actions

The training systems development interviews (called interviews from this point forward), lasting between 1.5 and 2 hours each, were conducted in the hospital environment with the serious game team in attendance. The first CTA-based interview addressed donning and doffing PPE and the second addressed x-ray procedures for a HIDs patient. The instructional designer led the interview while the development team would interject questions about details that would help inform game design and taking pictures of scenery, equipment, and accomplished tasks. When certain tasks involved psychomotor skills, the SMEs would demonstrate and execute those tasks while talking aloud through their thought process and explaining relevant knowledge, and at times letting the team experience the task as indicated in Figure 3.

Figure 3. SME Explains Glove Length Requirements.

The interviews were semi-structured relying on detailed protocols designed to ask initial questions that would generate the data required to address the four purposes in Table 1. Table 2 below provides sample questions from the protocol used during hospital staff interviews that were related to x-ray policies and procedures in a potential HID situation. Each time a question was answered, the instructional designer would ask the SME to elaborate, depending on the answer to the question. Often times, answers to questions would spark more questions in order to gather data needed for the four purposes listed in Table 1 above. Since a large majority of the data collection would stem from the elaborated statements, the interview was digitally recorded with audio. The first interview was transcribed by a third party vendor while the x-ray interview was scripted by the instructional designer. The transcription process also afforded the team an opportunity to capture exact expert statements that could be used within the game for instructional guidance, e.g., as a tip or hint.

Table 2. Sample Questions from X-Ray Interview Protocol – Role X-Ray Technician.

Example Semi-Structured CTA Interview Questions		Supported Game Design Elements
1	How do you know that x-ray procedures must begin? (i.e., how is that communicated to you?)	Storyline, Injects
2	What is the first concern you would have once you know x-ray procedures must begin? (i.e., I know there is a checklist, so I retrieve that, etc.)	Procedures, Decisions, Actions
3	How long should it take optimally to get the x-ray machine and what could slow it down?	Time factor, Decisions, Actions
4	What are the main concerns during this process in terms of patient contact and how are these concerns handled?	Decisions, Actions
5	What are the main concerns during this process in terms of the x-ray equipment and how are these concerns handled?	Decisions, Actions
6	Describe what the x-ray technician does next.	Procedures, Decisions, Actions

As seen in Table 2, the answers provided during the interview informed all aspects of the game design. In addition to collecting information about the scenery and details about all equipment, materials, characters, and other props, the team was able to gather data about specific procedures and the thought processes behind those procedures in order to establish all possible actions that one could take, and specifically, feasible actions. When developing a serious game, decision points with advertently incorrect actions serve very little purpose, not unlike distractor items in a multiple choice question that are obviously wrong. The person taking the assessment will most certainly select the correct answer, and this does not truly measure knowledge transfer. For example with question (6) in Table 2 above, the answer from the interviewee was, “The x-ray machine is placed outside of the patient room door.” The instructional designer then asked, “What is something that the x-ray technician might do at this moment that puts him or her at risk of contamination?” The key word is “might” in this case, since the action must be feasible. The response that the SME provided was, “opening the door, it's supposed to stay closed.” Obtaining a set of feasible actions at each decision point allows the game designer to provide the player a distinct set of options, all of which are feasible yet one is considered correct. This set of options is then linked to a distinct set of consequences or outcomes for the player, allowing for the development of performance feedback and scoring within the game.

CTA Data Analysis

After each interview was completed, the audio files from the interview were scripted, providing the ISD team a detailed account. After a complete review of the scripts and any images taken during the interview visit, the ISD team created a Training Requirements Report which included:

• Identified Learning Objectives (outcomes for all involved procedures, skills, and decisions)

• Instructional Strategies (types of activities that support the learning, e.g., view a video, access a checklist, and perform a task)

• Storylines (the sequence of events leading up to this particular set of tasks, including assumptions, i.e., an x-ray technician has been assigned, a patient is on a stretcher in the patient room, and a portable x-ray machine is available)

• Task-by-Task description following the normal sequence of events for that particular focus area.

  • Visuals (props needed)

  • Auditory Cues

  • Other characters needed during this task (Non-Player Characters (NPCs))

  • Evaluation Criteria (what the player needs to do correctly)

  • Actual quotes from the script for that task that will assist with verbiage for hints and also assist with developing feasible incorrect actions at each task decision point.

The Training Requirements Report for each focus area was then used as a foundation to create storyboards that would blueprint the serious game design. It is during the development of the storyboards that the specific game elements, content, and instructional strategies are developed in sufficient detail such that the development team can create the prototype as instructionally envisioned. Without a CTA approach, it would have been difficult to adequately and effectively define the content to meet the four data purposes as listed in Table 1.

Example Process: CTA to Game Developers

To illustrate how the CTA process supported prototype development, we'll use the CTA data and its transition to the developers for a relatively simple task in the donning PPE procedure, Task 2 Secure Hair. Local hospital policy indicates donning and doffing PPE is a two person process. Both staff members don PPE together and watch each other for problems. The Buddy role directs the donning process by following the donning checklist. Figure 4 provides the checklist as implemented in the prototype. The Caregiver role follows the Buddy's direction and subsequently provides the patient care while the Buddy watches for problems. Table 3 lists the steps and their outputs used to conduct the CTA and transition the content to a meaningful document for the team developers to create a prototype. Each step is then further described below.

Figure 4. Donning PPE Checklist as Implemented in the Prototype.

Table 3. CTA Process Steps with Outputs.

Process Steps from CTA to Developers	Process Step Output
1. Collect existing documentation	Personal notes on gaps and unknown items
2. Develop interview protocol questions	Interview protocol
3. Conduct CTA	Personal notes, digital recording, photographs
4. Analyze	Transcription file, corpus document
5. Identify training requirements	Training requirements for PPE process
6. Create storyboards	Blueprint for developers to create prototype

1. Collect existing documentation. For Task 2, Secure Hair, the existing guidelines stated: not touching collar, not in eyes. You may use a surgical bonnet if necessary.

2. Develop interview protocol questions. The protocol called for determining any concerns or barriers to task completion; tips and solutions; special tasking by the Buddy or Caregiver to perform this task; and determining if a hair bonnet and elastics are included in the PPE cart. The latter question was to determine if long-haired staff needed to come prepared with their own materials to secure their hair.

3. Conduct CTA interview. During the donning/doffing interview, the team included the instructional designer who led the discussion and the graphic artist, programmer, and system architecture asking questions as they arose to meet their development needs. The two hospital clinical educators served as subject matter experts (SMEs). The instructional designer digitally recorded the interview and took photographs that were later shared with the team. Questioning for Task 2 began with repeating the documented Task 2 procedure and then asking about the hair bonnet. The SMEs stated the PPE cart contained items to secure hair. To elicit more detailed information about concerns or problems, the instructional designer asked about handling hair bangs. Again, the SME's provided a simplistic answer of “use bobby pins.” Rich data was not forth coming, so the instructional designer asked about the hair bonnet. Here we learn that the bonnet is good to use because it helps to have the hair covered when removing the face mask as well as helps to keep the hair off the collar and face. To elicit more conversation about hair, the instructional designer then asks how staff would handle her type of hair (worn up with a plastic clip). This provided even more details such as my hair is not a problem because it is not voluminous; staff with braids or voluminous hair have more difficulty. Then a spontaneous discussion ensues about eyeglasses since the instructional designer is wearing glasses. The SMEs provide information on what is suggested for them. Eyeglasses are not mentioned in the documented guidelines. At this juncture the instructional designer believed the conversation on Task 2 was finished and the interview continued to the next donning task. No pictures were obtained for securing the hair; both SMEs had short, non-voluminous hair and did not warrant the bobby pins. The hair bonnet is optional and they choose not to use one. However, when doffing the facemask, photographs of the hair tangled in the elastic bands was obtained to illustrate why a hair bonnet could be useful.

4. Analyze. The recorded interview was professionally transcribed and proofed by the instructional designer which also served as a refresher on the donning procedures. During this proofing, any content related to securing hair was collocated to a single table. The discussion on Task 2 was first parsed out but then a keyword search for hair was also performed on the 65-page transcription file to help ensure content was not overlooked. By so doing, other conversational areas about the hair could also be added to the Task 2 table. This keyword search proved valuable in that at the start of the interview hair was mentioned as an important step while the caregiver is dealing with any initial anxiety in the upcoming interaction with a potential HIDs patient. Additionally, personal notes from the discussion were also added to this table to include perceived attitudes and sentiments. Eyeglasses were also included because it seemed naturalistic to include that content at the same time the SMEs became cognizant of that guidance. The Task 2 table became the corpus for the training content and training requirements for the instructional design.

5. Identify training requirements. An overarching design document had already been developed as a result of the initial SME interview on training goals and requirements. The overarching design included a description of the target audience, the learning objectives, the learning strategies, the storyline regarding the HIDs patient, and an overview of the topic areas and general procedures. Based on SME training experience, they made recommendations that included: let the staff fail at times, evaluate training per task completion, provide real-life video as an example, and support two-person interactive training. With completion of the analysis for donning and doffing procedures, training requirements now focused on donning and doffing procedures. A description of the donning PPE area was provided to include the purpose; learner interactions such as view a checklist and a video; visual cues such as signage and equipment; audible cues (none beyond ambient); non-player character interactions for single-player practice; evaluation tracking requirements such as did the player view the video and did the player successfully complete the task; and other notes to the development team such as checklist items should be visible without zoom. Additionally, the training requirements document included selected interview excerpts by task as example of content sourcing. Task 2 is only addressed in this example of content sourcing. It is the storyboards that outline how Task 2 will be executed in the training program but in support these training requirements.

6. Create storyboards. During the storyboard process, the content, learning objectives, and learning strategies are aligned with the gameplay. Similar to the affordances framework by Dalgarno and Lee (2010), possible gameplay elements were then identified to be described in the storyboards if applied. For Second Life (SL), potential gameplay elements included SL email, notecards, heads-up display, narration, video, chat, text messaging, imagery, inventory repository, avatar attachments, non-player characters, teleportation, and scoring. Additionally, the overarching gameplay is represented in a flowchart to help guide the developers as shown in Figure 5. Gameplay was initially designed for single-player with the anticipation of expanding to multiplayer at a later date. When the player is the Caregiver, scoring (evaluation) is based on selecting the correct PPE item from the table. So when the NPC Buddy directs the Caregiver/learner to secure hair, the Caregiver must select the hair bonnet with bobby pins or lose points. When the player is the Buddy, scoring is based on assessing if the system properly donned the PPE or accomplished the task as the case may be. The Buddy reads the checklist and selects Task 2 in proper sequence which signals the NPC Caregiver to secure their hair with or without the use of the hair bonnet with bobby pins. The Buddy must assess if the system properly secured the NPC Caregiver hair. Three incorrect hair options were provided to the developers that could be randomized with the correctly secured hair. It was envisioned that a player would initially play the Caregiver to first learn the procedure through correct selection of a PPE item and then play the Buddy to assess if the NPC properly accomplished the task. In this manner, the learner is introduced to the correct procedure and then can assess if accomplished correctly. Notecards were developed so players could collect a summary of tips and hints for each task for later review. Furthermore, it was recommended to insert a photo of real items and with an image of the gameplay graphic items (bonnet, bobby pins, and hair elastic) to add to the notecard. Lastly, messaging was developed to convey the content. The term messaging was chosen to represent the notion that the content was to be presented to the learner. At this juncture, it was not known if narration would be used or some type of text messaging. Therefore, the messaging was written as if narration with the recognition that the script could be redesigned if to be presented in text form. Narration script is commonly more verbose than would be written text. The amount of text and narration would likely need to be balanced at some point. The messaging for Task 2 was: “Great! It looks like you decided to use the surgical hair bonnet. I know how difficult it can be remove the facemask with the elastics pulling on your hair if it's not protected. Since I don't see any hair bangs or any hair touching your collar, I know you must have pinned your hair back. Good job! If you don't need your glasses, you can remove them as well. As for contacts, some staff keep a second pair and simply throw away the used ones even though there's no science supporting this action.”

Figure 5. Portion of the Gameplay Flowchart.

This CTA-to-Developer process was applied to each task in the donning and doffing of PPE.

Discussion

Using an agile development approach proved advantageous to accommodate findings that impacted design and development of the prototype. Ultimately, the initial interview and only two of four topic areas were explored, i.e., donning and doffing PPE and x-ray procedures. Additionally, storyboards were only produced for donning and doffing PPE. As the prototype development evolved, several changes were made. Changes included adding an omniscient character to direct and track gameplay who could also provide content, and determining messaging would be via onscreen text. It was good to capture the richness in the message in the storyboard so it could be accurately trimmed back or parsed for the actual text messaging.

The CTA approach is independent of domain while the specific methodology depends on the desired content (Riggle et al., 2014). However, 3D experiential learning via a role-play simulation, the team needs to pay particular attention to capture the fidelity of the cues (visual, spatial, audio, tactile, olfactory) and how those cues interact in the cognitive processes. Note-taking, audio recording, imagery (photos or video), and documentation are not uncommon to a CTA and should be applied regularly, but in the case of creating a 3D environment, imagery and audio recording is highly recommended to ensure production accuracy with rich fidelity.

The CTA method offered an interactive approach to elicit rich, relevant information from the SMEs who otherwise may not readily self-identify details, reasoning, or subconscious decision-making. Presenting different cases to the SMEs helped elicit that unconscious knowledge about hair concerns as opposed to simply stating in the training to use bobby pins and the hair bonnet if so desired. Additionally, asking questions of what might occur also elicited details for plausible but incorrect actions which could be injected into the gameplay.

Live demonstration by a SME with at least the instructional designer, if not the team, observing prompted additional unforeseen questions; additionally the SME could perform a more accurate think-aloud process and reasoning. While the pictures provided the visual, it was the interactive questioning and think-aloud demonstration that provided subconscious and experience-based details. The interactive questioning brought out the personal challenges (cognitive, attitudinal, psychomotor) and environmental challenges (e.g., sweating in PPE) as well as mechanisms to overcome those challenges.

Reviewing the CTA-derived data relied on a constant comparative analysis to combine comments of similar nature, ensure consistency across occurrences, and denote nuance differences in the collected data. Keyword searches also proved beneficial as a means to validate that no data went unlooked by task.

Creating the hospital environment three dimensionally did in fact lead to the discovery that the proposed hospital room for the donning and doffing of PPE would not be practical; the room would be too small for the equipment and personnel.

Conclusions and Recommendations

Based on our experience, we recommend using a CTA approach, and if appropriate, incorporate a live demonstration with a think-aloud protocol to elicit the rich details required for a simulation-based role-play serious game. Be sure to take photographs, and preferably video for complex or intricate procedures. Using an agile approach for rapid prototype development may prove advantageous to constantly keep a multidisciplinary team abreast of evolving design and development requirements and to align CTA outputs with serious game design. Eliciting details of automatic actions and unconscious knowledge from subject matter experts can be challenging but the use of a CTA approach may more readily elicit this type of information for subsequent alignment with gameplay elements. Furthermore, using instructional designers or cognitive scientists may be advantageous in that they are commonly more versed in conducting different types of interviews to elicit the required knowledge, the context, and the skills (cognitive, attitudinal, and psychomotor) to successfully perform tasks as well as capture the challenges, barriers, and plausible alternative but incorrect options or actions. The CTA proved a valuable method to prototype a serious game for donning and doffing PPE. However, given that a CTA is domain independent, these findings are equally extensible to other domain applications. Since this was not a research study, we look forward to the opportunity to explore training effectiveness for a CTA-based training design.

Biographies

Susan Dass, Ph.D., is a Senior Instructional Designer, Technical Specialist at ICF International. Dr. Dass received her Ph.D. in Education and her M.Ed. in Instructional Design. She has over 15 years of experience in instructional design supporting both private and government entities. Dr. Dass has many publications including co-authoring three book chapters that explore the design of help-seeking and self-regulated learning skills in 3D learning environments as well as faculty adoption of virtual worlds as a learning environment. Current projects include designing 3D serious games for the medical industry and exploring data analytics for learner performance and course improvements.

Victor Cid is a Senior Computer Scientist at the Disaster Information Management Research Center, Specialized Information Systems Division of the National Library of Medicine. His R&D work involves hardware and software solutions to improve the resiliency of health facilities and systems in the U.S. and abroad. Over the last 10 years, he has been researching and applying computer simulation and gaming techniques to training the health workforce for crisis events.

Joanne Barnieu, Senior Technical Specialist at ICF International, has over 24 years of experience in instructional design with focus on game-based training for Defense-related projects.

Paul Cummings, Senior Fellow at ICF International, has over 20 years of research and technical and management leadership experience in the education, simulation, and training community.