Applying cognitive task analysis to health services research

Abstract

Objective

Designing practical decision support tools and other health care technology in health services research relies on a clear understanding of the cognitive processes that underlie the use of these tools. Unfortunately, methods to explore cognitive processes are rarely used in health services research. Thus, the objective of this manuscript is to introduce cognitive task analysis (CTA), a family of methods to study cognitive processes involved in completing a task, to a health services research audience. This methods article describes CTA procedures, proposes a framework for their use in health services research studies, and provides an example of its application in a pilot study.

Data Sources and Study Setting

Observations and interviews of health care providers involved in discharge planning at six hospitals in the Veterans Health Administration.

Study Design

Qualitative study of discharge planning using CTA.

Data Collection/Extraction Methods

Data were collected from structured observations and semi‐structured interviews using the Critical Decision Method and analyzed using thematic analysis.

Principal Findings

We developed an adaptation of CTA that could be used in a clinical environment to describe clinical decision‐making and other cognitive processes. The adapted CTA framework guides the user through four steps: (1) Planning, (2) Environmental Analysis, (3) Knowledge Elicitation, and (4) Analyses and Results. This adapted CTA framework provides an iterative and systematic approach to identifying and describing the knowledge, expertise, thought processes, procedures, actors, goals, and mental strategies that underlie completing a clinical task.

Conclusions

A better understanding of the cognitive processes that underly clinical tasks is key to developing health care technology and decision‐support tools that will have a meaningful impact on processes of care and patient outcomes. Our adapted framework offers a more rigorous and detailed method for identifying task‐related cognitive processes in implementation studies and quality improvement. Our adaptation of this underutilized qualitative research method may be helpful to other researchers and inform future research in health services research.

What is known on this topic

• Cognitive task analysis (CTA) is a robust methodology to systematically characterize cognitive processes involved in decision‐making and task performance.

• There is little guidance on how CTA can be applied in health services research studies and clinical environments.

What this study adds

• We propose a 4‐step CTA framework that includes multiple sources of information to understand the knowledge, expertise, thought processes, procedures, actors, goals, and mental strategies of task‐related cognition.

• A better understanding of the cognitive processes that underly clinical decisions is the next step to developing more effective health care technology tools such as patient care dashboards, decision support applications, and patient portals.

• Our CTA framework enables explicit exploration of the cognitive processes underlying routine health care processes and decision‐making in a clinical environment such as an operating room, clinic, or patient ward.

1. INTRODUCTION

Designing practical decision support tools and other types of health care technology for implementation science and quality improvement relies on a clear understanding of the cognitive processes (i.e., the way people think) that underlie the use of these tools. ¹ Examples of factors that influence cognitive processes related to health care technology include the ordering of subtasks, influence of prior experiences on thinking, tools available to support thinking, people involved in a cognitive process, and even the organizational constraints to cognitive processes. ² Understanding these elements can help us build or improve more efficient and useful clinical decision support tools such as dashboards, applications, and patient portals that will meet health care providers' needs better. Unfortunately, there is very little guidance in the health services literature on methods to explore cognitive processes. Thus, in this era of rapidly evolving health care technology, most decision support tools are not designed with cognitive processes in mind. ³ , ⁴

Commonly used methods for exploring cognitive processes include expert or subject matter interviews, ⁵ surveys, ⁶ experiments on human behavior, ⁷ case studies, and observations. Each of these has limitations that may lead to biased results or make them challenging to implement in a clinical environment. ⁶ As an example, expert interviews are often incomplete, resulting in biased or incomplete information about the knowledge required to complete the task. ⁵ Many miss up to 70% of the subtasks (or “steps”) needed to describe a specific cognitive task for teaching purposes. ⁵ In addition, experimental designs may not be possible in some health care settings due to patient safety or time and space constraints, especially in clinical settings. Cognitive task analysis (CTA) may provide a less intrusive, less biased, and more complete exploration of the cognitive processes that underlie certain clinical decisions or processes. Still, it is rarely used in health services research studies, quality improvement, or the design of health care tools. ⁸ , ⁹

Recently, our multi‐disciplinary study team developed a study protocol to design a clinical decision support tool to be deployed while a patient was being discharged to improve post‐discharge outcomes. The goal of this study was to understand and describe the task of discharging from the hospital after surgery. ¹⁰ Essentially, we sought to describe the cognitive processes involved in the task of discharging a patient. To address our study goal, we planned to use CTA. While we found guidance and several examples for using CTA in numerous other fields, we found very few manuscripts describing the method in a health services research study. ² , ¹¹ , ¹² A recent review by Swaby et al. supports our finding, identifying 81 published studies over the past three decades that use CTA in a clinical setting. ¹² The systematic literature review was extensive, representing 13 countries and a wide range of specialties including surgery (n = 30), intensive care (n = 11), primary care (n = 6), and even obstetrics and gynecology (n = 3). However, only 35 studies used CTA to explore a particular clinical scenario. And, no studies provided clear guidance on how to design health services research studies that use CTA. Thus, the impetus of this methods manuscript is to illustrate the value of CTA and provide concrete guidance based on our thorough research on its use so that other health services researchers can apply it to their research questions. ¹³ , ¹⁴ , ¹⁵ , ¹⁶

Drawing on prior studies and work from other fields that have used CTA, such as aviation, ¹⁷ psychology, ¹⁸ and ergonomics, ¹⁹ we developed a health care setting‐oriented CTA framework that can be used in health services research, quality improvement, and implementation science. Below we attempt to address the gaps we found in the current literature by describing our adapted CTA framework with the hopes that it will be helpful to other researchers to inform future research in surgery and other areas of health services research.

2. WHAT IS COGNITIVE TASK ANALYSIS?

Cognitive task analysis is a suite of methods that provides an iterative and systematic approach to studying cognitive processes such as reasoning, problem‐solving, or decision‐making. Key elements studied with CTA include knowledge, expertise, thought processes, procedures, actors, goals, and mental strategies that underlie the completion of a task (Table 1). ²⁰ At its core, CTA aims to define the cognitive, or the conscious intellectual, activities required to perform a specific task. ²¹ A task is defined as something that must be done within a specified period of time or, more broadly, the goals of a specific activity or series of activities. ¹⁹ Examples of a task may be as simple as completing a form or as complicated as responding to a natural disaster.

TABLE 1.

Key components of cognitive processes in a health care setting

Knowledge/expertise

Thought processes

Procedures

Actors

Goals

Mental strategies

CTA is often described as a family or suite of methods ²² because multiple sources of information are used to moderate the biases and limitations of single sources of information. Including multiple sources of information is also conducive to an iterative process, allowing the researcher to return to and refine prior steps to better inform subsequent steps in understanding task‐related cognition. More details on these steps are provided below.

3. STEPS OF THE ADAPTED COGNITIVE TASK ANALYSIS FRAMEWORK

Through a thorough review and synthesis of the current literature, we distilled four (4) main steps in our health care setting‐adapted CTA framework (Figure 1). These steps include (1) Planning, (2) Environmental Analysis, (3) Knowledge Elicitation, and (4) Presenting Results. The goal of the first step (Step 1) is to focus on the task to be examined and prepare for data collection and analysis. Steps 2 and 3 are data collection steps, one focused on the broader environment where the cognitive process or task occurs (Step 2) and another more narrowly focused on key elements of the task (Step 3). The final step (Step 4) distills the information collected into meaningful deliverables.

FIGURE 1.

Adapted cognitive task analysis (CTA) framework [Color figure can be viewed at wileyonlinelibrary.com]

We used an iterative process not only to develop but also to pilot the framework. Reiterating through the CTA steps described below allowed us to continue to improve the final products of our research and also helped us to develop a more complete and inclusive framework that could be used in future studies. Our adaptation is mainly based on the work of Crandall et al.'s book titled Working Minds: A Practitioner's Guide to Cognitive Task Analysis (2006). ² We first describe the major elements and considerations in each of the 4 steps, then illustrate the use of the framework in the context of our pilot study of hospital discharge of surgery patients.

3.1. Step 1: Planning

Our adapted CTA framework begins with in‐depth framing and preparation of the research question. ² Step 1 is the foundational planning step for the study. During this step, the researcher should clearly identify the task to be examined, define key questions, and define the target population/location (framing). In addition, we propose to identify data collection methods and materials for Steps 2 and 3 a‐priori and begin to think about the study's deliverables and results for Step 4 (preparation).

Our framework's first and most central piece is framing a clear key study question or study goal related to the cognitive process(s) or task being studied. All other steps in the adapted CTA framework will evolve from this study question. Our study team found it helpful to frame our key question in light of a particular theory or conceptual framework. Given our goals of exploring decision making processes in our pilot study, we chose the Ottawa Decision Support Framework. ²³ , ²⁴ We found this framework useful for our study because of its focus on decisional needs. Other theories that could guide key questions in CTA include, but are not limited to cognitive load theory, multiple resource theory, or perceptual load theory, all of which conceptualize how an individual processes information. ²⁵ , ²⁶ , ²⁷ , ²⁸ As with all other steps in our framework, the key question may also be refined slightly as the study progresses.

The keys to writing a strong key question for CTA are similar to those for writing a strong research question. The key question should be simple, focused, and limited to a particular time and place. The key question should identify the task to be explored and the population involved in the task. As an example, “How do you transfer a patient?” would be a poor question. While it is simple and identifies a clinical task (patient transfers), it is not focused and does not clearly identify the place and population to be studied. An example of a better question might be, “What is the process for transferring a patient from the Surgery service to the Medical service”? The question is still simple but much more focused, identifying a specific element of the task (the process or ordering of sub‐tasks) and a specific population and location (health care providers in the Medical and Surgical unit).

A solid key question will lead to a more precise definition of the population and locations to be studied. This will help identify the qualitative methods that could be applied to these populations and locations. Examples of these methods include literature reviews, observations, interviews, or experimental techniques. Crandall et al. recommends considering the following four key questions to help guide the selection of methods for a CTA ² :

1. When does the task take place? Past, present, or future?

2. What type of data would you like to collect? Real‐world, simulation, or artificial environment data?

3. How frequently does this task occur? Is the task an everyday occurrence, or does it happen more infrequently?

4. Is the goal of the analysis to gather more abstract data and general knowledge, or is it to collect more detailed information on specific tasks or events?

Working through these questions can help to identify or eliminate qualitative methods that could be used for data collection. Options and examples of potential methods for each step are discussed in more detail below (Steps 2 and 3).

Once data collection methods are determined, drafts of materials should be developed for the environmental analyses (Step 2) and perhaps for the knowledge elicitation step (Step 3). Including at least two data collection methods is unique to our health care‐setting adapted CTA framework. We developed this progression to minimize intrusion into the clinical environment. The goal of the environmental analysis or environmental scan (Step 2) is to identify internal and external factors related to conducting the task and to refine materials and methods for the subsequent knowledge elicitation. Step 3, knowledge elicitation, expands on the data collected during the environmental analysis to dive into the detailed nuances of performing a task or series of tasks. Materials for either step may include observation guides, interview guides, or draft data collection forms for literature reviews if proposed.

Finally, for Step 1, we recommend developing a draft of the final deliverables and a data analysis plan. Proactively identifying study outputs like project deliverables or data structures can help improve the study design and quality of the study's findings. The draft of project deliverables will of course evolve throughout the study as the data are actually collected but imagining what the results might look like can guide study design and data collection methods. Project deliverables may include tables describing key elements of the cognitive process, process flow diagrams, fishbone diagrams, or even figures mapping relationships between key actors in a task or subtasks related to the task. More information on specific types of deliverables is included below (Step 4). In addition, we have provided some examples of the initially proposed deliverables from our Pilot study (Appendix S1).

3.2. Step 2: Environmental analysis

Step 2, the Environmental Analysis, is a data collection step focused on understanding the health care environment where the task occurs. We propose using observational methods that involve fewer interactions and less interference with the environment that is being studied. We included this step in our health care setting‐adapted framework in order to minimize intrusion in an already busy health care environment. The secondary goal of the environmental analysis in our framework is to further frame questions and data collection methods for the potentially more intrusive knowledge elicitation in Step 3 (Figure 1).

An environmental analysis, or environmental scan, is essentially a systematic analysis of the setting or circumstances in which a task is conducted. ²⁹ During an environmental analysis for CTA, the researcher should identify and describe the physical location where the task is performed and the population involved in conducting the task. Sources of information may include in‐depth literature reviews, structured and unstructured observations, or informal discussions within the health care setting where the task is taking place. Again, the goal is to minimize interruptions while conducting data collection. Examples of data that may be collected include key actors involved in completing a task, environmental barriers or facilitators to performing a task, or even organizational policies related to the task. In our pilot study, data gathered in our environmental scan included providers involved in the discharge process (actors), cues that a patient is ready to be discharged, the procedure to discharge a patient, and problems that may delay a discharge.

Data collected during an environmental analysis also helps refine the data collection processes and forms for Step 3, knowledge elicitation. For example, a researcher may propose a web‐based application to expedite clinic check‐ins. The environmental analysis may include unstructured observations during the clinic to collect information on where a patient checks in, who is involved in checking the patient into the clinic, what information is necessary to check the patient in, and where this information goes once it is completed. Assuming in‐person interviews are planned for Step 3, collecting this data a priori with an environmental scan will help identify whom to interview and may also identify barriers or facilitators to completing the task that could be better defined during the interviews.

3.3. Step 3: Knowledge elicitation

Knowledge elicitation, Step 3, is the core component of CTA (Figure 1) in our adapted framework. There are a variety of methods that can be used in the knowledge elicitation step. ³⁰ These include interview methods, observational methods, surveys, focus groups, and even experimental methods. ²¹ , ³¹ Interview and observational methods tend to be the most commonly used. ¹² In a clinical setting, observational methods or surveys will likely be the least intrusive. If the topic involves sensitive or private information, one‐on‐one interviews of clinicians and other health care providers will likely provide the most information.

The chosen strategy for knowledge elicitation may consist of any of these methods and should also be guided by the proposed project deliverables and types of information needed: actors, processes, goals, relationships, barriers, etc. Some interview techniques for knowledge elicitation like concept mapping ³² or interruption analysis ³³ are best used to describe or map the procedures and actions involved in a specific task. Other techniques such as user needs analysis, ³⁴ task action mapping, ³⁵ or other decomposition methods like hierarchical task analysis, ¹² can help to further explore the knowledge and decisions required to complete a task. Lastly, structured and unstructured interviews can be used to identify and explore barriers or facilitators to task completion.

In addition to interview techniques, knowledge elicitation methods may include more involved observational methods such as card sorting, ³⁶ process tracking, ³⁷ , ³⁸ or expert/novice comparisons ³⁹ depending on the question, population, and location studied. These can be used concurrently with the more frequently used interview methods to strengthen study findings. Experimental methods may also be used for knowledge elicitation, although these are often hard to implement in a clinical environment without interrupting patient care. ²¹ Experimental methods that have less impact on patient care include role playing ⁴⁰ or simulation methods. ⁴¹ These can be particularly useful in eliciting knowledge and actions that interview techniques may miss because they have essentially been “automated” by the interviewee. ⁴²

3.4. Step 4: Presentation of results

Our adapted CTA framework's fourth and final step involves analyzing and presenting the study's results (Figure 1). This step aims to structure the data, identify findings, and discover meaning. ² There are a variety of analytic techniques that can be used to code and structure the data collected in Steps 2 and 3. These data may include both qualitative and quantitative data. While qualitative data will likely be more common, using literature/chart reviews, surveys, or experimental methods may result in quantitative data. For qualitative data, systematic coding should be used to increase study validity and enable transparency and reproducibility. Qualitative coding aims to discover patterns and themes occurring in the qualitative data. While quantitative analyses are typically done after data collection, coding of qualitative data can begin as early as the first environmental scan.

Common types of deliverables that should be considered include tables, diagrams, or lists of cognitive functions, cues, and sub‐processes needed to complete a task. ²¹ One classical figure to consider is a process flow diagram. CTA uses process flow diagrams to document cognitive processes or steps to complete a task. Tables might also be used to break out required cognitive tasks by event type or describe the actions necessary to complete the task being studied. These are frequently referred to as “Cognitive Demands Tables” when used in the context of CTA. ²¹ , ³¹ Additional information in these tables could include descriptions of task types or cognitive actions, actors involved in the cognitive task or process, cues to the cognitive actions, or potential barriers and facilitators to completing the task or cognitive activities related to the task. Fishbone diagrams, flowcharts (i.e., task diagrams), or network diagrams may also be proposed to detail a series of tasks or processes involved in completing an overarching goal. ³¹ These can detail the sequence of events, key actors involved in completing the task, or places where potential errors may occur. Other types of information that might be desired include descriptions or classifications of procedures and processes, strategies for problem‐solving, goals of involved parties or stakeholders, or even relationships between individuals or tasks and sub‐tasks. Decisions about which deliverables are most appropriate should be made based on the research audience.

4. APPLYING COGNITIVE TASK ANALYSIS IN POSTOPERATIVE CARE COORDINATION

As discussed above, this adapted framework was developed while we designed and performed a research study to develop a postoperative decision support tool for surgical health care providers. Thus, we have been able to pilot the framework in this ongoing study. For our study, the key cognitive task we focused on was discharging a patient after surgery. Our key question was, “Who is involved and what information is needed to discharge a patient after surgery?”

Working through Step 1, we broadly identified our study population and location as health care providers involved in postoperative care in a hospital setting. We identified structured observations of the surgery team's morning teaching rounds as the starting point for our environmental analysis. We planned to use a semi‐structured interview for the knowledge elicitation phase. We also developed an observation guide and a preliminary list of questions to be used during the knowledge elicitation step. And lastly, we began to draft the final project deliverables. Our initial list of project deliverables included a task flow diagram for the discharge process, a table of key actors in discharging a patient, and a table of the most frequently occurring “cues” that a patient is ready for discharge (Appendix S1).

As part of our environmental analysis (Step 2), we began observing morning teaching rounds for two surgical specialties at two different hospitals using a structured observation guide. Thus far, we have conducted 16 observations at two hospitals. During morning teaching rounds, a resident on the postoperative care team summarizes the current patient status, discusses new developments, and reviews the treatment plan for patients under the care of the surgical service. The team discusses the patient's hospital course and any updates to the care goals while also preparing and planning for discharge. Readiness for discharge and discharge planning is frequently discussed during these rounds, making them an excellent opportunity to observe the sub‐tasks and key stakeholders involved in discharging a postoperative patient. Although not originally known, we also found that staff nurses and other specialties such as physical therapy or social services periodically drop in for these morning rounds. This allowed us to observe interactions across the many disciplines and specialties involved in discharge care coordination and identify other stakeholders for our knowledge elicitation step.

For Step 2, one study staff member conducted all observations to ensure consistent data collection methods, and the study team met bi‐weekly to discuss findings. The study staff conducting the observations collected the data using a single‐page observation guide and short‐form notes during each observation. After each observation, the staff entered the data into an electronic file along with long‐form notes and also reviewed the electronic health care record to supplement information gathered on decision making processes during the observations.

The limits of our environmental analysis were open so as not to miss additional sources of information that were not previously identified during planning. Throughout the observations, we were mindful of any new information that should be included in the observation guide and other locations that should be included in our environmental analysis. This was part of our iterative approach to the study design and the adapted framework. Although not identified during our initial planning (Step 1), we also found that resident morning rounds, resident shift changes, daily interdisciplinary (IDT) team meetings, and weekly/monthly M&Ms (Morbidity and Mortality conferences) provided additional information on key roles and sub‐tasks in discharging a surgical patient. We did not use a structured guide to collect data from these other observations. In addition, we continued to review and update our knowledge elicitation interview guide throughout the environmental analysis.

We began conducting interviews (Step 3) after we completed 12 site observations. Our updates and revisions to the interview guide were minimal after the first 10 site observations. For Step 3, knowledge elicitation, we employed the critical decision method (CDM) to structure our semi‐structured interviews. A traditional CDM interview involves an iterative process of four phases. ² , ¹⁴ , ¹⁹ First, the interviewee is asked to recall and reflect upon the real‐life examples of a specific task (Identification, Phase 1). The interviewer can also prompt the interviewee at this point with examples. After reflecting upon their real‐life experiences, the interviewee and interviewer work to reconstruct a timeline of events around the specific task (Timeline Verification, Phase 2). Finally, the interviewee is asked to identify strategies for problem‐solving along the steps of the constructed timeline (Knowledge Deepening, Phase 3), and the interviewer digs deeper into the task by asking “What if” questions (“What if” Queries, Phase 4). ¹⁴

We use purposive sampling of disciplines identified during our environmental analysis to recruit interview participants. Thus far, we have conducted eight interviews with various health care providers including nurses, residents, case managers, medical students, and dieticians. The semi‐structured interviews have expanded upon the information collected during our observations. Interviews are audio‐recorded to allow us to transcribe, code, and re‐review interviews as needed. Interviews begin by identifying the steps, key actors, and roles involved in some of the participant's most recent patient discharges (Identification, CDM Phase 1). This information is used to construct a timeline of patient discharge from the interviewee's perspective (Timeline, CDM Phase 2). We then focus on the judgments and decision making processes around discharge by asking about factors that make some discharges more complex, including disagreement among team members or lacking information (Deepening, CDM Phase 3). In the last portion of the interview, the interviewer digs deeper into the interviewee's responses with “What if” questions and scenarios to further understand the interviewee's role in discharging a patient (“What if”, CDM Phase 4). We have found that our interviews take between 20 and 30 min to complete.

While analyses are ongoing (Step 4), we plan to conduct a thematic analysis, identifying and interpreting the patterns that arise from our observations and interviews. Throughout the study, we continue to revisit and refine the task flow diagram of the discharge process as represented by our ongoing data collection. We are also creating a table of key actors and responsibilities for discharging a patient to highlight the multidisciplinary nature of the process. Finally, we plan to develop a table of the most frequently occurring “cues” that a patient is ready for discharge.

Once completed, we plan to use this information to design and pilot a user‐centered decision support tool for intervening in post‐discharge infections. While still ongoing, CTA has helped us uncover key stakeholders in the discharge process of which we were initially unaware. We found that different actors in the environment may offer different perspectives. These perspectives provide insights into where people have consistent representations of the environment and where these perspectives differ. We have also uncovered numerous barriers to discharging a patient after surgery which has sparked future research ideas. Finally, we plan on developing a conceptual model of the discharge process that can help other researchers continue to build user‐centered decision support tools deployed around the time of patient discharge.

5. CONCLUSIONS/MOVING FORWARD

The impetus of this methods manuscript is to address the lack of guidance we found for designing health services research studies that focus on cognitive processes and apply the CTA methodology. ¹² We are currently conducting a research study to develop a postoperative decision support tool for surgical health care providers. Our study's goals were to assess the feasibility of this tool in a busy and fast‐moving clinical environment and to collect information that would improve the outcomes of a future implementation study (i.e., acceptability, appropriateness, adoption, sustainability). We realized that we needed a better understanding of the discharge process or the tasks necessary for discharging a patient to improve our future implementation study outcomes. Our research led us to CTA, which, while used in many other fields such as aviation and ergonomics, was rarely used in health services research studies. Therefore, we developed a framework to guide our study by referencing research conducted across various fields. This article presents that framework with detailed information on how CTA can be applied in a clinical environment for health services research.

To conclude, our adapted CTA framework provides a robust methodology to systematically characterize cognitive processes involved in completing a task in a clinical environment. Deeply understanding processes of care is the underpinning of success in quality improvement and implementation sciences research. We believe that our framework will allow future studies to employ more rigorous qualitative and mixed methods research exploring and describing cognitive processes in health services research to further improve outcomes.

FUNDING INFORMATION

This material is based upon work supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development grants (I21HX003217‐01A1; RCS 14‐232; RCS 17‐154).

Supporting information

Appendix S1. Examples of proposed deliverables for pilot project.

Graham LA, Gray C, Wagner TH, et al. Applying cognitive task analysis to health services research. Health Serv Res. 2023;58(2):415‐422. doi: 10.1111/1475-6773.14106