Workflow Driven Cognitive Decision Support Systems for Clinicians:

Abstract

This paper investigates how the knowledge of clinical workflow or clinical problem solving process can support the development of decision support system and improve the performance of the clinicians. A prototype of a decision support system called Intra-operative Visualization System (IVS) was developed based on a workflow centered design framework. IVS was designed for an upcoming minimally invasive surgery called Radio Frequency Ablation or RFA (for treatment of liver cancer). IVS was evaluated in an experimental study by comparing the performance of eight expert intervention radiologists and eight medical students while performing selected tasks from RFA procedure on the patient phantom by using two systems: IVS and the conventional Ultrasound (US). The results reveal significant evidence for improved decision-making when using the IVS by both the clinical experts and students. This study demonstrates the benefits of workflow centered design in medical visualization to support clinical decision-making and hence improvement of task performance and prevention of errors.

Introduction

Significant research in the area of medical informatics points to the importance of understanding cognitive processes to support human centered development of clinical decision-support systems for complex workspaces^1,2. Empirical studies illustrate the benefits of including cognitive theories into system design to develop information systems, which lead to safer working environments and prevention of errors. Recent examples of such web based systems in the clinical workspace are: computer based patient record systems ³. knowledge management systems for biomedical engineering ⁴, and computer based training systems in pathology ⁵. The introduction of new clinical techniques such as Minimally Invasive Surgeries (MIS) has led to several technological innovations in the operation theatre⁶. However, inadequate information transparency, limited access, and poor visualization, compel clinicians to rely on advancements in medical imaging technology. These limitations in MIS are constantly giving rise to new research and development in the area of decision-support systems. Such systems are providing real-time image guidance and task automation support while the clinician is performing the intra-operative tasks ⁶

Recent literature on technological developments indicates that the design of expert systems involves major deficiencies with respect to the following issues: First, the development of visualization support tools such as augmented reality ⁷ and pre-operative planning ⁸ are often centered around introducing new technologies in the clinical workspace. Second, there is rare evidence that sufficient understanding of clinical requirements is integrated in the early technology development phase ⁹. As a consequence solutions are often more influenced by the latest technological trends rather than required by the clinicians ². The introduction of such technology may even lead to increase in information load, resulting in low performance and clinical errors. To avoid information overload in the clinical workspace it is crucial that the decision support system presents only necessary information at the right time and in the right form to the clinician. One way to achieve this goal is to correspond the information visualization to the clinicians’ problem solving process during the ‘clinical workflow’. The paper addresses these requirements by focusing on the following two research questions:

• – How can the knowledge of the clinical workflow be included into the system development cycle to provide a foundation for designing expert decision-making systems?

• – To what extent do expert systems, developed on the basis of the knowledge of the clinical workflow, aid in decision-making and improving the performance of the clinicians by preventing errors?

Workflow centered framework

Workflow centered design framework (Fig.1) integrates the cognitive processes in different phases of the human centered development cycle, which are: 1) specify context of use, 2) analyze requirements, 3) design prototype and 4) evaluate the prototype of the system. Workflow Integration Matrix or WIM ¹⁰ has been incorporated for clinical workflow analysis.

Figure 1.

Workflow centered development framework including the 5 stages the WIM is incorporated in the requirement analysis.

In the design prototype phase theory of situation awareness (SA) has been incorporated. SA is regarded as the theoretical backbone for improving information visualization in system design. WIM is built upon the theory of problem solving in complex workspaces and cognitive task analysis. WIM consists of two main components, the existing workflow and the future workflow. The existing workflow allows the task decomposition of the three clinical phases (pre-operative, intra-operative and post-operative). The new workflow creates a bridge between the current clinical workflow and the future technological solutions. WIM includes a task-based analysis of the clinical and technological requirements to create concept storyboards.

Information Visualization in IVS

As an application case, a decision support system called Intra-operative Visualization (IVS) was developed. IVS aims to provide decision support to perform an upcoming MIS called percutaneous Radio Frequency Ablation or RFA. RFA is commonly performed by intervention radiologists through Ultrasound guided procedure to treat malignant tumor in liver. Inadequate visualization limits the global acceptance of performing RFA. A workflow analysis based on WIM revealed several clinical milestones, where IVS could support the performance of RFA ⁹. IVS prototype was developed for two Clinical Milestones (CM) that were selected by clinicians from the workflow analysis: CM 3- refers to the task of identification of the target tumor in the liver and CM 4- requires the placement of the needle in the center of the tumor. Fig. 2 illustrates the combination of the three screens which assisted the intervention radiologists in identifying the target tumor and in needle navigation. Real-time information visualization in IVS was achieved by fusion of pre-operative data obtained by CT scan with intra-operative data gained through real-time US.

Figure 2.

Physical set up and components of Intra-operative Visualization System (IVS).

Situation awareness in information visualization

In a complex and dynamic environment such as the surgical workspace, decision-making is highly dependent on situation awareness [12].Together, the three screens of IVS are aimed at enhancing perception of the relevant clinical cues, comprehension of critical structures and projection of action plans.

Screen 1:

Screen 1 provides the clinicians with a real-time view of the patients’ anatomy. This augmented information is visualized through image fusion between real-time US and pre-operative CT scan. The information is augmented in 2D on the US video feed. This means that only the key abstracted information related to critical structures is extracted from the pre-operative CT data set and superimposed on the original US image. Thus, the system supports the decision-making in two levels of task complexities:

• ▪ Routine scenario: The image fusion provides an augmented image of the target tumor, which is represented by a red arrow. As the clinicians swipe the US on the phantom, the system recognizes the target tumor as a result of CT and US fusion. The arrow indicates the target tumor, thus minimizing the error of selecting the wrong tumor (the tumor not intended to be ablated). The system recognizes and tracks the location of the needle and generates a needle trajectory, which is augmented in the US image in real-time. This, in turn, supports the needle navigation and the clinician’s ability to create an accurate projection plan.

Complex scenario:

In a complex scenario the visualization system aids the clinician to locate the newly detected tumor, and allows the clinician to plan dynamically. The image fusion between pre-operative data and real-time US provides information about the newly detected tumor, represented by a blue arrow.

Screen 2:

This screen provides context information about the location of the tumor, the vessels and the position of the US probe. A 3D model of the liver is generated based on image fusion between CT scan and US image. Decision-making related to positioning and guiding the needle to the center of the tumor depends on the anatomical and clinical constraints. The standard ablation zone is considered to be 5 cm. The screen provides the following critical cues in both routine and complex scenarios:

• ▪ Context view: When the clinician places the needle on the phantom, screen 2 displays the needle in a 3D model of the liver along with an augmented needle trajectory. This offers the clinician a context view of the patient anatomy to spatially orient the US probe and the needle towards the target tumor. The target tumor is marked in red in the 3D model for the routine scenario. In the complex scenario, the model updates itself and the new tumor found is marked in blue. As a consequence, the clinician is able to comprehend the task related information appropriately, thus better action planning is supported.

• ▪ Navigation and verification: The liver parenchyma, tumor and main vessels are visualized in 3D. When the clinician positions the US probe on the patient phantom, she/he knows its location corresponding to the tumor and the vessels. This assists in placing the US probe optimally and navigating the RFA needle by avoiding the critical anatomical structures. By visualizing critical cues related to patient anatomy, IVS can aid in a better perception of patient data and thus supporting the situation awareness.

Screen 3:

This screen displays the original pre-operative CT scan. This is included because during the procedure the clinicians often prefer to recapitulate the overview of the patient anatomy. No additional information is augmented on the CT scan.

Evaluation Study

An experimental study was conducted to compare whether the IVS is better at supporting the decision-making and performance of expert intervention radiologists and medical students in comparison to the conventional US guided intervention. The purpose of study was clearly explained to all participants. All the participants gave verbal agreement to participate.

Participants:

It is important to note that to perform RFA percutaneously with US guidance is a spatially challenging and complex procedure. Since it requires special expertise in understanding of US for interventional procedures, the proportion of experts practicing this procedure is limited worldwide, with each hospital having limited number of experts.

• ▪ Experts: Eight expert intervention radiologists who were practicing RFA or biopsy procedures were selected as participants. These experts were associated with the Rikshospitalet and Radium Hospitalet Oslo, Norway and had eight to twenty years of experience in intervention radiology. Eight experts was the maximum number of experts available for the study in Oslo.

• ▪ Students: Final year medical students (n=8) of the Rikshospitalet Oslo participated in the study. All the selected student participants were required to have primary knowledge and understanding of CT scans and of working with Ultrasound system.

Experiment design:

Specifically designed clinical scenarios were developed in the IVS. Each participant was given an hour of training time on the IVS and the US. Although one hour is limited time to get acquainted with IVS, it was the maximum time available with the experts and the students. In the case of students, half an hour more was allowed for the training as most of them were new to performing interventions by using the US. In the training period, the participants were required to hit the center of a tumor by using both systems. For the experimental task, each participant was given two tasks (routine and complex) of hitting the center of the tumor using both systems. The two systems were alternated in a way that first four experts and four students were asked to perform the tasks using the US and then the IVS. For the next group of participants this situation was reversed. Two levels of task complexity: routine and complex, were selected.

• ▪ Routine scenario: This task required the participants to ablate the tumor selected for ablation during the pre-operative planning stage. To simulate this clinical scenario in the experiment, one of the tumors was highlighted on the CT of the abdominal phantom. The participants were required to ablate the selected tumor using the two different systems.

• ▪ Complex scenario: This task required the participants to ablate the tumor which was newly detected while conducting intra-operative US. This newly detected tumor was not visible in the preoperative CT, thus causing uncertainty in the previously planned clinical action. To simulate this complex scenario in the experiment, a tumor existing in the abdominal phantom were made invisible in the CT scan. The hidden tumor was visible to the participants only while conducting the intra-operative US. The participants were expected to dynamically adjust the RFA procedure.

Measurement Criteria:

e following three measure criteria were selected: Intra-operative task planning time: The planning time was measured as the time taken by the participant to plan the procedure intra-operatively. This was measured as the total time taken by the participants for problem analysis time after explanation of the tasks until the start of task execution.

• ▪ Hitting the tumor in the center: Clinical findings state that the major cause of clinical errors performing the RFA procedure is either caused by the wrong tumor hit, or by not hitting in the center of the tumor-causing unablated cancer cells. The accuracy of hitting the center of the tumor was measured as the distance between the point of needle insertion by the participant and the mathematical center of the tumor. This was measured by specially programmed software which tracked the position of the needle tip and calculated its distance from the center of the virtual tumor. This was possible because the position of the virtual tumors had already been obtained from the US data of the phantom.

• ▪ Hitting the target tumor: The participant’s accuracy of hitting the correct target tumor during the task was measured. A wrong tumor hit was measured by the distance between the needle hit and the center of the target tumor.

Results

Reduced intra-operative planning time:

Both of the participant groups, showed a significant reduction in intra-operative planning time for both levels of task complexity (routine and complex) using IVS, compared to only US intervention (Table 1).The planning time of the experts was significantly reduced when performing routine tasks (p=0.012, Med_IVS=2.85 min and Med_US=6.80 min) and complex tasks (p=0.036, Med_IVS=4.37 min and Med_US=5. 62 min). n addition, the planning time of the student participants was significantly reduced for routine tasks (p=0.012, Med_IVS=5.39 min and Med_US=8.55 min) and complex tasks (p=0.012, Med_IVS=5.78 min and Med_US=9.67 min).

Table 1:

Intra-operative planning time

Expert Intervention Radiologists(n =8)
Routine tasks			Complex tasks
Systems	Median	Range (min)	Systems	Median	Range (min)
IVS	2.85	2.11–4.41	IVS	4 37	2.80–5.54
US	6.80	5.54–10.27	US	5.62	4.33–7.40
Significance p=0.012			Significance p=0.036
Students(n=8)
Routine tasks			Complex tasks
Systems	Median	Range (min)	Systems	Median	Range (min)
IVS	5.39	3.54–6.34	IVS	5.78	4.21–6.89
US	8.55	5.67–13.65	US	9.67	5.15–14.70
Significance p=0.012			Significance p=0 012

Results in Table 1 show that there is a significant difference in reduced planning time between the experts and the students, the experts were quicker in conducting the intra-operative planning. Intra-operative planning involves routine but also the perception of critical situations and coping with high uncertainty. These scenarios required seeking alternative courses of actions, which the experts were able to retrieve from their action repertoire accumulated during their past experiences. This can be explained by that the IVS supported the experts’ experiential knowledge by providing the necessary critical cues through integrated information.

Task accuracy of hitting the tumor in the center:

The results show that using the IVS, both experts and students showed an increase in accuracy in hitting the center of the tumor as compared to the US (Table 2). The task accuracy of hitting the center of the tumor by the expert participants did not increase significantly for the routine tasks, (p<0.05, p=0.69, Med_IVS=2.10 mm and Med_US=2.65 mm.) but increased significantly in the complex tasks, (p=0.017, Med_IVS =1.80 mm and Med_US=3.20 mm.). The task accuracy of the student participants increased significantly while performing routine tasks, (p=0.025, Med_IVS =1.25 mm and Med_US=5.76 mm.) and complex tasks, (p=0.012, Med_IVS =2.65 mm and Med_US=6.36 mm).

Table 2:

Task Accuracy

Expert Intervention Radiologists(n = 8)
Routine tasks			Complex tasks
Systems	Median	Range (mm)	Systems	Median	Range (mm)
IVS	2.10	0.70–2.90	IVS	1.80	1.10–2.80
US	2.65	2.10–12.80	US	3.20	1.90–6.80
Significance p=0.069			Significance p=0.017
Students(n =8)
Routine tasks			Complex tasks
Systems	Median	Range (mm)	Systems	Median	Range (mm)
IVS	1.25	0.90–2.90	IVS	2.65	0.80–4.80
US	5.76	1,40–49 00	US	6.35	3.80–22.78
Significance p=0.025			Significance p=0.012

Note: Wilcoxon signed ranked tests p<0.05)

Errors in hitting the target tumor:

No wrong tumor was hit either by the experts or by the students while performing the task with the IVS. However, experts hit three wrong tumors while performing with US. Student participants hit four wrong tumors while using the US, with two hits for each task complexity.

The results in Table 2 indicate improved accuracy and reduced errors on two counts. First, the experts hit three wrong tumors and students hit four wrong tumors using the US, but there were no errors of hitting the wrong tumor using the IVS. The reduced errors can be attributed to the visualization of critical cues in IVS which guided the participants in selecting the target tumor. Second, although the experts showed an overall improved accuracy in hitting the center of the tumor by using the IVS, significant differences between both systems were only found in the complex task scenario. Studies investigating problem solving in complex workspaces show that due to prior experience, experts can perceive underlying causes quicker and this high performance can hardly be improved. Therefore, no significant difference in performing routine tasks was found for experts. The students showed significant differences in achieving task accuracy of hitting the center of the tumor for both of the task scenarios by using the IVS. The results indicate that IVS supports the learning curve as even with almost no experience of conducting the RFA procedure, the students performed better using the IVS.

Conclusion

The paper presented the application of the workflow centered framework and its theoretical underpinning as a structured approach for designing a cognitive decision support system for clinicians. As an example application, the framework was applied to develop and evaluate an intra-operative visualization system (IVS) for an upcoming minimally invasive surgery-radiofrequency ablation (RFA). The results reveal significant evidence for improved decision-making when using the IVS by both the clinical experts and students. In terms of three performance measures both the groups: (1) needed less intra-operative planning time; (2) illustrated increase in accuracy of hitting the tumor in the centre and (3) fewer errors in hitting the wrong tumor. In order to support the surgical decision making, the information provided through decision support systems such as IVS, the visualization should correspond with the ‘clinical workflow’. The term workflow emphasizes the importance of information flow between the patient, the system, and the surgeon during the surgical procedure and forms the basis of surgical problem solving.