Measuring Workload Demand of Informatics Systems with the Clinical Case Demand Index

Abstract

Introduction: The increasing use of Health Information Technology (HIT) can add substantially to workload on clinical providers. Current methods for assessing workload do not take into account the nature of clinical cases and the use of HIT tools while solving them. Methods: The Clinical Case Demand Index (CCDI), consisting of a summary score and visual representation, was developed to meet this need. Consistency with current perceived workload measures was evaluated in a Randomized Control Trial of a mobile health system. Results: CCDI is significantly correlated with existing workload measures and inversely related to provider performance. Discussion: CCDI combines subjective and objective characteristics of clinical cases along with cognitive and clinical dimensions. Applications include evaluation of HIT tools, clinician scheduling, medical education. Conclusion: CCDI supports comparative effectiveness research of HIT tools. In addition, CCDI could have numerous applications including training, clinical trials, design of clinical workflows, and others.

Introduction

In recent years, there has been increasing interest in assessing and quantifying healthcare provider workload^1–3. A major motivation is that excessive workload has a substantial negative effect on quality of care provided and patient safety^4–6. The introduction of Informatics tools such as Electronic Health Records EHRs, Computerized Prescription Order Entry, CPOE systems, mobile health decision support tools, is known to contribute to workload^7,8 for example by increasing cognitive and physical demands on the provider^9–11. Therefore, there is a need for methods to quantify the impact of Health Information Technology (HIT) on provider workload and, especially, to perform comparative effectiveness research of informatics tools with respect to workload.

While objective measures, such as time performances, and subjective measures like the National Aeronautics and Space Administration-Task Load Index (NASA-TLX)¹², and the Subjective Workload Assessment Technique (SWAT)¹³ have productive uses in healthcare, they do not explicitly consider an important factor: the nature of the clinical case¹⁴. For example, it seems intuitively clear that 8 hours spent in handling routine and simple outpatient cases impose less clinical demands and workload on healthcare providers than 8 hours in an intense trauma and critical care setting. This aspect of clinical tasks is not captured explicitly or adequately in current workload analysis. This means that evaluation of HIT tools is limited to task efficiency or subjective factors while ignoring the influence of the nature of the clinical problem on effectiveness of HIT tools. Quantifying workload characteristics at the clinical case level enables detailed analysis, modeling, and simulation of clinical cases. For example, when performing comparative effectiveness research of informatics tools using clinical trials it can become necessary to provide patient cases that make comparable clinical demands across study design groups.

In this paper, we present the Clinical Case Demand Index (CCDI), a tool that succinctly combines objective and subjective characteristics of clinical cases with respect to the multifactorial demands they place on clinicians. The CCDI focuses on evaluating clinical cases while the NASA-TLX and SWAT are primarily focused on evaluating subjects^15,16. A major application of CCDI is to compare two or more cases and determine how similar or different they are in terms of the demands these cases make on providers.

Broadly speaking, the CCDI represents application of ideas from Industrial engineering to informatics tools, analysis, and modeling of healthcare practice with implications for patient safety. The CCDI provides a quantitative, objective metric to identify such comparable clinical cases. In the following we describe such a comparative effectiveness study in some detail.

Related Work: Workload factors in Clinical Care

Provider workload can be measured objectively by metrics such as physical hours spent in patient consulting, number of patients seen, type of patients / diseases seen¹⁷. For example, in a study¹⁸, primary care physicians worked about 50-60 hours per week, with about 18 patient visits per day per physician. These sorts of measures are aggregates focused on quantity of provider services¹⁹ and do not account for the multifactorial demands made on clinicians by specific clinical cases and types of cases¹⁴.

In addition to the purely objective measures, subjective measures of workload, developed initially for engineering and manufacturing contexts have also been applied in the clinical domain. Notable among these are The National Aeronautics and Space Administration-Task Load Index (NASA-TLX) ¹², and the Subjective Workload Assessment Technique (SWAT)¹³

The NASA-TLX incorporates mental, physical, and temporal demands as well as the operator’s assessment of his/her performance into its model of workload. The SWAT considers the operator’s time pressures, mental effort, and psychological stress. When assessing the workload, the scenario is distilled into its tasks. In this context, we note that physical workload can be defined as physical strength and resources required for completing a task. Mental (Cognitive) workload is defined as effort required for thought processes, reasoning, calculating, decision-making, memory, learning, and other cognitive activities.

NASA-TLX and SWAT are aggregates of purely subjective perceptions that do not consider the multiple characteristics of the clinical case. Some of these case characteristics are objective, such as the number of clinical tasks needed, whether the case is routine, is an emergency, and so on. Others are subjective such as perceived cognitive and physical demands.

Workload and performance in clinical cases have also been quantified using case complexity measures^20,21. Such case complexity measures consider intrinsic factors including clinical, social, and epidemiological factors. These complexity measures are used in training and research^22,23. However, these measures do not consider extrinsic factors such as provider skill and available resources, including HIT tools, for solving clinical cases. So far, workload assessment of HIT tools supporting clinical practice has been done using total time spent and conventional human factors assessment of task performance^24,25, ignoring the nature and complexity level of clinical cases. It is useful to combine both intrinsic and extrinsic factors since irrespective of its intrinsic complexity, provider factors such as decision-making skills and executing needed tasks are crucial to outcomes^26,27.

In addition, the comprehensive relationship between sources of workload and human performance evaluation does not seem to have been investigated in detail for this purpose^28-30. Florez-Arango, Iyengar, and Smith (2016) developed an ontological approach to model properties of actions performed by clinicians and related human factors³¹. This ontological approach combines previous work describing task as goals or processes, and human factors derived from NASA TLX.

Methods:

Clinical Case Demand Index

Clinical cases are a fundamental unit in healthcare, comprising of a complex micro-environment of physical and mental processes. While solving a clinical case a healthcare provider typically performs several actions such as taking blood pressure, performing auscultation, arriving at a diagnosis, providing a treatment plan, and so on. We denote these actions as Tasks. Each task contributes to the overall demand and workload on the provider. With this nomenclature, solving a clinical case is equivalent to performing a sequence of one or more tasks by the provider or team. For example, most clinical practice guidelines (CPGs) are structured as a series of tasks to be performed considering clinical signs and symptoms. In this paper, we are concerned with the use of HIT tools while performing one or more of these tasks.

The CCDI is based on five dimensions derived from a formalism to represent tasks that are widely regarded among researchers and practitioners as contributing significantly to case demand^28-30. It includes clinical and cognitive dimensions that are related to use of HIT tools. Physical Demand (PD) and Cognitive Load(CL) are ubiquitous when assessing provide workload. The Time Pressure (TP) dimension is related to urgency as a factor for decision making ^32,33. The Expertise (EX) dimension is a combination of importance / criticality^34,35 and familiarity / uniqueness ³⁶ as a factor for decision making. The Task Complexity(TC) dimension counts the number of tasks required to solve the clinical case, noting that the ability to make clinically meaningful clinical decisions decreases as the number of tasks in a case increases³⁷. An issue here is that even for the same clinical case, the sequence of tasks that is performed can vary widely depending on the institution, the qualifications of the healthcare provider, national and local standards. To account for these variations in clinical practice the Task complexity is normalized over a given cohort of clinical cases. This means that CCDI is most meaningful over that cohort, such as within a specific care context or research setting.

CCDI consists of both a numerical summary, CCDI-S and a visual depiction, CCDI-V, of these 5 dimensions. CCDI-S provides an overall idea of the case demand and lends itself to statistical analysis, while CCDI-V enables rapid comprehension of the differential roles of the five dimensions listed above. We suggest that maximum benefit of the CCDI will be obtained by utilization of both the summary value and the visual component.

Development of the Clinical Case Demand Index

The CCDI is computed as follows based on five dimensions.

1. Time Pressure(TP): This dimension of CCDI describes the temporal constraints on tasks in case. If a single task in a case must be executed immediately, for example, because the patient is in imminent risk of death, it is an emergent case. When a single task is required to be executed to avoid death or permanent sequels it is an urgent case. Cases with lack of emergent or urgent task are non-urgent.

2. Expertise(EX): This dimension relates to the level of knowledge or skills required to complete the case. It is rated Low if the provider can complete the task by himself or herself without needing additional resources. If any task in the case requires the provider to call on another individual of same knowledge and skills, the task is rated Medium. Finally, if specialized expertise, knowledge, or skills is needed, the expertise level is rated High.

3. Cognitive Load(CL): In the context of HIT tool use, this dimension is rated High when majority of tasks in the case require recall from long-term memory. A medium cognitive load occurs when tasks only require use of working memory such as calculations. Any other kind of cognitive processing, such as mechanical click-through and simple note-taking, is scored as low.

4. Physical Demand(PD): When a task in a case requires heavy manual labor, muscular strength, the whole body, or constantly switching positions PD is scored as high. If a task. requires use of both hands to manipulate and solve the problem it is graded as medium. When there is little or no physical effort is low.

5. Task Complexity(TC): This component is a function of the cases in the cohort under consideration, as explained above. It is a continuous variable in range 0-3.

   $$TC=\frac{n_i*3}{max(n)}$$

   n_i = number of tasks in the case i, obtained by heuristic task analysis ³⁸ max(n) is the number of task in the largest case.

Each factor is scored between 0 and 3 as shown in Table 1.

Table 1–

Case Demand Index Assessment Scale

Axis/values	0	1	2	3
Time Pressure (TP)	No factor	Non-Urgent	Urgent	Emergent
Expertise (EX)	No previous knowledge	Low	Medium	High
Cognitive Load (CL)	Absent	Low	Medium	High
Physical Demand (PD)	Absent	Low	Medium	Heavy
Task Complexity (TC)		$\frac{n_i\ast 3}{\max (n)}$

Clinical Case Demand Index – Summary (CCDI-S):

Summarizing the 5 axes, a composite numerical measure is calculated as shown below:

$$CDI-S=\frac{TP+EX+CL+PD+TC}{15}$$

Interpretation: The values range from 0-1. Lower numbers represent lower case demand.

Clinical Case Demand Index – Visual (CCDI-V):

The graphical component of the CCDI is a 5-axis radar plot corresponding to each of the five dimensions, TP, EX, TC, CL, and PD. In the rest of this paper we shall refer to each of this interchangeably as a dimension or an axis. Radar Plots (Figure 1) are a graphical method of displaying multivariate data with axes radiating from a central point. The plot consists of spokes radiating from a central point separated by equal angles. The spokes correspond to the axes. The axis values determine the length of a spoke. The number of axes determines the shape of the plot - from a triangle, pentagon, to full circle. Radial lines connecting the major and minor ticks in the axis can be drawn to help in both plotting and reading the graph. The values of the variables are encoded into the lengths of the axes and normalized to fit in the overall area.

Figure 1.

CCDI-Visual (CCDI-V) component of CCDI Time Pressure

In each of the five axes the value of that axis increases from the origin to the perimeter. A case in which all 5 dimensions have a value of 1 (typically interested as defining low demand in that dimension) would appear as a filled in symmetric pentagon with internal radii equal to 1. Asymmetric radar plots indicate that at least one dimension makes greater demands than others. For example, in the 6 radar plots shown in Figure 2 below, the top left plot shows high demand (need) for expertise and low demand for physical effort.

Figure 2.

CCDI-V plots of 6 cases.

Evaluating CCDI for grouping cases in a clinical trial for a mobile health tool

An example application of CCDI is a mobile Health study at the simulation lab of Universidad de Antioquia (UdeA), Medellin, Colombia³⁹. The purpose of this study was to determine the benefits of presenting clinical practice guidelines in a media-rich step-by-step algorithm presented on smart phones vs the same content on paper. The study design was a randomized cross-over in which each subject managed 15 cases using the mobile health tool and another 15 using paper-based guidelines. To make the comparison meaningful the two sets of cases needed to be “equivalent” and yet different, to prevent learning effects. The CCDI was used to quantify clinical demands on 30 clinical cases and determine 15 comparison of cases per subject between treatments. The cases were selected from an existing case bank at Simulation Center and Clinical Skills lab (CS) in UdeA. Physician experts from UdeA reviewed the cases and trained Community Health Worker (CHW) performed the clinical tasks. The CHWs who performed the task were predominately male (58%), and were qualified as CHW by training or experience, comparable by their training. Mean years of training was 3.24 years.

Three physicians, as a group, conducted ontological informed task analysis over the 30 cases (10 pediatric, 10 trauma, and 10 non-trauma) and came to a consensus on the values of the five factors (axes) for all 30 cases. The CCDI was computed for all 30 cases based on the consensus values developed by the 3-physician group for each of TP, EX, TC, CL, and PD. CCDI-S and CCDI-V were calculated for the 30 cases using a Microsoft Excel spreadsheet. Based on the CCDI, cases were indexed from 1-5 and grouped under 6 categories (A & D – Pediatrics, B& E- Trauma, C&F – non-trauma). They are grouped in such a way that cases A1-D1 are equivalent, A2-D2 are equivalent and so on.

Figure 2 shows radar plots of six cases from the sample. The cases illustrated in Figure 2 are A1-Mastoiditis, B2 Burns, C4-Hypertension, D1-Severe Malnutrition, E2-Below-Knee Amputation, F4-Headache. Visual analysis, based on area and length of axes in the CCDI-V enabled grouping of the pairs of cases with respect to case demand. It is seen that A1 and D1, B2 and E2, C4 and F4 are equivalent. Using a combination of CCDI-V and CCDI-S allowed to avoid pairing cases like B2 and A1 with similar CCDI-S values but clearly different CCDI-V.

A total of 50 CHWs were recruited. We selected 5 of these CHWs to pilot the CCDI on 30 cases. Three independent physician observers noted that no additional tasks needed to be added to any cases, and that the evaluation of other axes were appropriate. Based on this review no changes to the scale or study design were made, and the data from the 5 pilot study CHWs were included in the final analysis of 50 CHWs. They were divided into two groups and each group performed 15 cases out of total 30. In Group 1, Cases A-C were included and in Group 2, Cases D-F were included. In Group 1, 21 CHWs, and in Group 2, 23 CHWs completed all the 15 cases allotted. Only these completed cases were included in the final analysis. The groups were exclusive and independent samples with no overlap of cases.

To understand whether CCDI provides meaningful values consistent with workload analyses we compared the case groupings described above with the NASA-TLX values assigned by care providers to the 30 cases as follows. One Laerdal Simman® and one Laerdal Simbaby® human patient simulators at the CS were programmed with the selected cases. The simulated cases were presented to CHW subjects. Cases were completed by CHWs in each treatment (mobile vs paper guidelines) group. At the end of each case, the subjects completed NASA-TLX questionnaire in Spanish. For each case the NASA-TLX was computed as average of all subjects’ score in the group. Consistency of grouping by CCDI with NASA TLX grouping is important to validate CCDI as a predictive measure of case demand. NASA TLX has been validated in health-care contexts as a measure of perceived workload ^15,16.

We compared the CCDI to NASA-TLX using Pearson correlation, regression analysis and paired t-test. Our null hypothesis is that CCDI and NASA-TLX are not comparable with a 0.05 level of significance. We used R software package for statistical analysis.

Results:

CCDI is a valid, strong predictor of NASA-TLX with R-squared of 0.33, and Pearson correlation is 0.58. The difference between the means of CCDI and NASA-TLX would be between 0.11 and 0.01 with 95% confidence interval. The accuracy of CCDI is 0.06 - i.e. 68% of values predicated from CCDI would lie within 0.06 of the NASA- TLX value in the population. Table 2 shows that the grouping of cases by CCDI, which is a prospective, pre-task scale is consistent with NASA-TLX which is a subjective post-task measure off perceived workload.

Table 2–

Statistical Analysis of CCDI vs NASA-TLX

A) Pearson Correlation:
Pearson Correlation coefficient ρ = 0.5815037
t-statistic = 3.7823, df = 28, p-value = 0.000751
B) Regression
Coefficients:
	Estimate	Std. Error	t-value	p
(Intercept)	0.34277	0.04760	7.200	0.0000000776
CCDI	0.29925	0.07912	3.782	0.000751
Residual standard error: 0.06482 on 28 degrees of freedom
Multiple R-squared: 0.3381, Adjusted R-squared: 0.3145
F-statistic: 14.31 on 1 and 28 DF, p-value: 0.000751
C) paired t-test:
t = -2.8946, df = 29, p-value = 0.00714
95% CI =-0.11200744, -0.01925922
Mean of differences = -0.06563333

Discussion:

The Clinical Case Demand Index is a measure of the multifactorial subjective and objective demands required to handle clinical cases in healthcare. In Human Factor research, workload is defined as “that portion of the operator’s limited capacity actually required to perform a particular task” ⁴⁰. Measuring workload helps in assessing the capacity required for optimal performance. Workload assessment is typically performed to improve productivity, reduce adverse events, plan resource allocation, training, and for quality management. CCDI serves as a tool to assess workload in healthcare. The CCDI includes both a summary numerical value (CCDI-Summary) and a succinct visual in the form of a pentagonal radar plot (CCDI-Visual). The radar plot generated helps to visualize the workload, while maintaining the weights of the dimensions. Radar plots have been used in healthcare to display multivariate data ^41,42. They show the areas of strength and weakness, as well as a general overall summary. The CCDI-V, among other benefits, provides a rapid way to differentiate cases with respect to the specific factors, TP, EX, TC, CL, and PD. CCDI can be applied in selection of clinical cases for training, evaluation, resource management and research purposes.

The CCDI-V is designed to support visual analytics, a technique that’s growing rapidly in popularity in biomedical informatics. They can be inspected rapidly both in regards to the relative sizes of the axes and the size and shape of the shaded area. For example, the relative roles of each of the five axes, representing the impact of each of the five factors making up the overall CCDI, can be assessed at a glance. If multiple cases are being considered, a side-by-side visual inspection can simply assess their similarity. For example, cases that have high cognitive load (CL) can begrouped together if the corresponding axes on their CCDI-V plots are similar in length. The size and shape of the shaded area also rapidly yields useful case demand information. If the shape of the shaded areas of the CCDI-V plots for several cases are similar, but the sizes are different it is immediately apparent that these cases make similar demands in terms of the 5 factors, TP, EX, TC, CL, and PD, but the overall demand varies.

One of the applications of CCDI is in comparative effective research of HIT. As described above, we used CCDI to allocate cases with equivalent workload in two treatment groups. CCDI enabled presentation of cases matched with respect to difficulty level from simple to more difficult in each group. This enabled meaningful comparisons of clinician performance, learning and burnout. Thus, CCDI can be applied to create standardized cases, normalized with respect to clinical demand, for clinical student evaluation and training purposes.

Applications of the CCDI include training, evaluation, planning healthcare provider schedules, and quality management. The combined CCDI can be used in training contexts such as evaluation of resident’s skills, allotment of cases for examinations. Comparing cases based on CCDI can be useful in quality assessments, planning clinical workflow, and for insurance purposes.

Limitations:

Formal reliability and validity analyses of CCDI need to be performed. The scale was developed as part of study to train community health workers; thus, it may not represent the workload of another clinical environment. The cases used for testing the scale were subjectively chosen and may be subject to selection bias. The task complexity component involves summary of count of number of tasks, irrespective of inherent complexity of the task itself. The CCDI is designed to be applied only when at least two fully specified cases (using an ontology for example) are being considered.

Future Work:

Immediate future tasks include reproducing the evaluation in a variety of clinical settings with various types of providers including nurses, physicians, and specialists with larger sample sizes. These studies will facilitate factor analysis studies. Also, we are currently designing software tools to generate CCDI from case repositories.

Conclusion

The Clinical Case Demand Index measures the combined effects of five dimension typically contributing to provider workload in solving clinical cases especially when using HIT tools. Unlike currently used measures of workload CCDI takes into account intrinsic and extrinsic factors related to solving clinical cases. In this paper, its utility in comparative effectiveness research of a mobile health tool was demonstrated. In addition, CCDI could have numerous applications including training, clinical trials, design of clinical workflows, and others.