SYFSA: A Framework for Systematic Yet Flexible Systems Analysis

Abstract

Although technological or organizational systems that enforce systematic procedures and best practices can lead to improvements in quality, these systems must also be designed to allow users to adapt to the inherent uncertainty, complexity, and variations in healthcare. We present a framework, called Systematic Yet Flexible Systems Analysis (SYFSA) that supports the design and analysis of Systematic Yet Flexible (SYF) systems (whether organizational or technical) by formally considering the tradeoffs between systematicity and flexibility. SYFSA is based on analyzing a task using three related problem spaces: the idealized space, the natural space, and the system space. The idealized space represents the best practice—how the task is to be accomplished under ideal conditions. The natural space captures the task actions and constraints on how the task is currently done. The system space specifies how the task is done in a redesigned system, including how it may deviate from the idealized space, and how the system supports or enforces task constraints. The goal of the framework is to support the design of systems that allow graceful degradation from the idealized space to the natural space. We demonstrate the application of SYFSA for the analysis of a simplified central line insertion task. We also describe several information-theoretic measures of flexibility that can be used to compare alternative designs, and to measure how efficiently a system supports a given task, the relative cognitive workload, and learnability.

1 Introduction

Efforts to improve health care quality have led to an increased push to develop and adopt systems that enforce or encourage consistent processes based on best practices and evidence-based medicine. These efforts follow similar successful practices in other safety-critical industries, such as aviation and nuclear power. Within healthcare, these efforts include clinical guidelines, structured documentation, standardized terminologies, decision support systems, checklists, as well as policies.

Although systems that enforce or encourage consistency can improve safety and efficiency, health care is filled with complexity, variations, and exceptions that are not easily captured by idealized processes. Systems that are too rigid to support such deviations can lead to decreases in quality, along with caregiver resistance and creative workarounds that decrease the positive effects of best practices (see, for example Koppel, et al.[1]). Hollnagel's ETTO (efficiency-thoroughness tradeoff) principle is an informal way to express the tradeoff between systematicity and flexibility [2]. Recognition of similar tradeoffs in other industries have led to the design of Systematic Yet Flexible (SYF) systems [3], in which the system supports and sometimes enforces a systematic approach, while allowing flexibility. Thimbleby [4] has argued that user interfaces are easier to use when they are “permissive” (that is, giving the user flexibility, and hence lowering learning costs) but again this is an informal treatment. Norman [5] emphasizes the role of design constraints and forcing functions in user interfaces, but not how to design the appropriate blend.

Although there are general design goals for SYF systems [3], there are no analytic frameworks that allow one to analyze the tradeoffs to determine the appropriate blend of systematicity and flexibility. Without analytic frameworks, organizations (or system developers) will inevitably make arbitrary, often sub-optimal, design choices. The usual response is to require iterative design, which is a period of repeated implementation and evaluation to guide improved re-implementation of the procedures; essentially a “trial and error” design process.

In this paper, we present an analytic framework for designing SYF systems (whether organizational or technical) by formally considering the tradeoffs between systematicity and flexibility. In particular, we propose that the ideal SYF system is one that supports graceful degradation from idealized practices to those that better fit the situation at hand. The framework, which we call Systematic Yet Flexible Systems Analysis (SYFSA), is based on analyzing a task using three related problem spaces: the idealized space, the natural space, and the system space. The idealized space represents the best and most efficient practice—how the task should best be accomplished, assuming that only actions that ultimately lead to a goal state are taken and all logical task constraints are met; in general, the least number of actions will be taken to achieve the goal. For example, the idealized space for choosing a medication includes a number of constraints, such as that the medication is therapeutically appropriate, has the correct dose and route, is safe, is available for purchase in the form and dose prescribed and within the required time frame, and is as economically efficient as possible. The natural space captures the task actions and constraints on those actions imposed by the physical world. For example, if the natural space is a paper-based, handwritten prescription we see that it enforces almost none of the idealized constraint—it is too flexible. However, this flexibility allows a physician to use non-standard formulations and dosing regimens to better personalize care and to easily prescribe new medications that may not yet be in more systematic IT-based ePrescribing systems. Lastly, the system space specifies how the task is done in a redesigned, or newly designed, system, including how it may deviate from the idealized space, and how the system supports or enforces constraints in the idealized space. A system space for ePrescribing explicitly considers the constraints of the idealized prescribing space, supports known constraints, while recognizing the need to cope with the inevitable exceptions and variations that are common in healthcare.

SYFSA is a design and analysis framework, not a set of prescriptive guidelines or principles for producing SYF systems. Prescriptive guidelines give explicit design advice, but usually at a very high level of abstraction that leaves considerable details underspecified. For instance, one of Perer and Shneiderman's guidelines for SYF systems that we discuss below is to allow the user to “See an overview of the sequential process of actions,” [3] but this guideline does not help the designer decide which of many possible sequences to highlight. In contrast, SYFSA's primary value as a design and analysis framework is to allow stakeholders to explore trade-offs in systematicity and flexibility by making constraints (and lack of constraints) on actions and sequences of actions an explicit part of the design and evaluation process. Specifically, SYFSA forces designers (and others involved in the design or evaluation process) to think about the constraints or lack of constraints in each of the spaces and whether a specific system design supports those constraints. It is then up to the designer to use the results of the analysis to inform system design. Returning to Perer and Shneiderman's example, SYFSA can help designers decide which sequence of actions to highlight.

We also propose three quantitative, information-theoretic measures of task flexibility that allow designers to compare the flexibility of alternative system designs and how closely these designs match the idealized flexibility required to complete a task. These measures are motivated by an intuitive notion of flexibility, whereby a task that can be done by carrying out actions in any order has maximum flexibility, and a task that can only be done with a specific sequence of actions has the least flexibility.

2 Background

2.1 Flexibility Characteristics

The concept of system or process flexibility has been explored for at least 30 years in a number of fields, including chemical process engineering [6], manufacturing design [7,8] and more recently business process design, and workflow automation systems [9-12]. One general consensus is that flexibility is a multidimensional concept, where the relevant dimensions depend on the kind of process or system being analyzed and the analyst's goals. For example, Sethi and Sethi [8] identified eleven different, but complementary definitions of manufacturing flexibility, including production flexibility, which is the range of products a system can produce without need for major changes, and operation flexibility, which is the ability for a system to produce a product in different ways.

Despite the lack of a single, precise definition of flexibility, or even a fixed set of dimensions, there is a general consensus that flexibility is the ability of a system to tolerate and adjust to variations in operating conditions. One common distinction is between short-term and long-term flexibility, where short-term flexibility is the ability to tolerate variations without changing the goal, whereas long-term flexibility is the ease with which a system can be changed to meet new goals. An example of short-term flexibility is the ability of an automotive manufacturing process to adjust to a part substitution. In contrast, long-term flexibility refers to the ease of changing the assembly line to manufacture a different vehicle.

There are often trade-offs between different dimensions. For example, a multipurpose woodworking machine that acts as a router, planer, jointer, and table saw has a lot of functional flexibility, but because it takes time to convert from one function to another and it can only perform one function at a time, a shop with a multipurpose machine loses scheduling flexibility over the same shop with a dedicated machine for each function. In addition, dedicated machines often perform better (e.g., with more precision or speed) than multipurpose ones.

Some researchers have argued that the general definition of flexibility, with its emphasis on adapting to and tolerating variation, also implies that there are some invariants that are meant to be maintained by flexible systems [13]. This implies that a flexible system must also be somewhat resistant to changes, in the same way that the wings of plane must flex, but still return to their original positions. Some of the more formal definitions and approaches to measuring flexibility operationalize this concept by defining a range of operation that a system must maintain in the face of variation. Flexibility is then the amount of variation that can be tolerated while maintaining operation in the desired range [6]. For example, a chemical process that works only when ambient temperature varies by no more than 5 degrees is less flexible than one that works within a wider temperature range.

The multidimensional nature of flexibility means that there are also many different measures of flexibility. For instance, in a review of flexibility concepts and measures Gupta and Goyal [14] identified six different classes of measures and then further subdivided these into qualitative and quantitative measures. In chemical process design, researchers have developed a flexibility index, which is a single number that defines the maximum variation in a set of normalized variables that the system can tolerate while still producing the desired output

2.2 Flexibility in Healthcare

Healthcare system flexibility, including organizational and health information technology, is perhaps most similar to business process flexibility. Researchers exploring business process flexibility have discussed measures such as the number of possible initial states of a system, the number of reachable goal states, and the number of paths from some initial state to the goal states. Bider has also applied mathematical systems theory to business processes [10]. However, research on business process flexibility is less mature than the other domains, so the conceptual and analytical frameworks are not as well developed.

Like many of the aforementioned industries, healthcare has also experienced a push to adopt and enforce consistent procedures based on both best practices and evidence-based medicine. While such systems can improve efficiency and safety, healthcare is complex and is not always amenable to idealized processes. Some health information systems are too rigid, leading to negative consequences such as decreased quality, user resistance, and workarounds [1,15–19]. As an example, one study concluded that many unintended consequences of clinical decision support systems (CDS) are attributable to insufficient flexibility [16]. Indeed, an overly rigid system can cause medication errors by not allowing clinicians to enter atypical prescriptions [17].

On the other hand, there are also instances when errors can occur due to excessive flexibility. Consider the nurse who intended to program a pump to infuse 5 mcg/minute, but accidently selected a rate of 5 mcg/kg/minute (equivalent to 350 mcg/minute for a 70 kg patient). While an alert appeared, the flexible system allowed the nurse to simply bypass the warning [20].

2.3 Systematic Yet Flexible Design

Perer and Shneiderman, working in the context of exploratory data analysis systems, proposed seven Systematic Yet Flexible (SYF) design goals for systems that support exploratory data analysis [3]. The design goals are to enable users to: (1) See an overview of the sequential process of actions. (2) Step through actions. (3) Select actions in any order. (4) See completed and remaining actions. (5) Annotate their actions. (6) Share progress with other users. (7) Reapply past paths of exploration on new data. These design goals provide useful advice for tasks that generally require a single sequence of actions, but they do not provide guidance on assessing task flexibility or the trade-offs among user interfaces that support different amounts of flexibility for the same task. In summary, previous work on flexibility provides considerable insight on the nature of flexible systems, how to measure flexibility, and how to design user interfaces to support some kinds of flexible systems. However, there are no clear operational definitions or measures for the kinds of flexibility that interests us in the context of healthcare systems. Further, there is no specific design process to help us produce flexible systems and understand the trade-offs among alternative designs.

2.4 Cognitive Work Analysis

Cognitive Work Analysis (CWA) is a design and analysis framework that was specifically created to develop systems that allow workers to flexibly adapt to unanticipated situations [21,22]. It does this by using a number of methods to uncover the intrinsic constraints of a work domain.at multiple hierarchical levels. Once these constraints are visible, a designer can look for places where flexibility may be unnecessarily restricted. This gives workers more flexibility to adapt to unanticipated situations. In addition, CWA emphasizes the development of information displays and controls that maximize a worker's situation awareness so that he or she can more readily understand an unexpected situation and respond appropriately.

Although CWA is specifically designed to support flexible systems, it does not explicitly provide tools for analyzing trade-offs in systematicity and flexibility. CWA emphasizes increasing flexibility to allow workers to adapt. We found only one paper that explicitly addressed flexibility in the context of CWA, but it too focused on increasing flexibility [23]. It did, however, contain a brief comment that sometimes limiting flexibility can be beneficial because fewer choices can speed decision making. This is followed by a recommendation to develop interfaces that present the most common strategy while still allowing alternative strategies. This is the essence of an SYF system. Unlike CWA, SYFSA provides an explicit mechanism for understanding trade-offs in flexibility and systematicity. However, CWA is highly complementary to SYFSA, because it provides a number of methods and tools for uncovering, relating, and visualizing intrinsic constraints in a work domain. A designer can use these constraints to develop the idealized and natural spaces.

2.5 Types of Flexibility

Based on this review, we differentiate among three different types of flexibility: procedural, functional, and operational. Procedural flexibility is the number of ways to successfully complete a task and achieve a given goal. Procedural flexibility can result from multiple paths to a single goal state or multiple goal states each with one or more paths. Functional flexibility is the number of functions a system is designed to support. For example, an epinephrine auto-injector that delivers a single measured dose of epinephrine, has less functional flexibility than a programmable infusion pump that can deliver a variety of meds at different rates and volumes. Operational flexibility is the amount of variation a system can tolerate while still allowing task completion. Variation is measured with respect to one or more variables and one or more tasks. For example, if the only task of interest is delivering a dose of epinephrine, and available time to deliver the dose is the only variable used to measure variation, then the epinephrine autoinjector has greater operational flexibility than a programmable infusion pump, because the autoinjector can deliver its dose under a wider range of available times. In contrast, if the variation is measured by the range of patient-types (e.g., adult, pediatric, neonate, etc.) and the conditions to be treated, then a programmable infusion pump has higher operational flexibility.

At this time, SYFSA addresses only procedural flexibility. Although this is a limitation of the current framework, , we feel that the focus on procedural flexibility is warranted for several reasons. First, procedural flexibility is an important component of system design that can affect both functional and operational flexibility. For instance, the high procedural flexibility of a programmable infusion pump allows it to do more functions (increased functional flexibility) under more conditions (increased operational flexibility) and do each function several different ways (procedural flexibility) than an epinephrine autoinjector. An analysis of procedural flexibility is therefore necessary for analyzing operational and functional flexibility. Second, many best practices in healthcare are highly procedural and attempts to improve practice or to enforce best practices often take procedural forms. This is especially true of regulations, standard operating procedures, structured data entry, as well as Health IT forcing functions and interaction design. The motivation for this approach comes from decades of experience that shows that the healthcare work domain is underconstrained and that even experienced workers often don't know or don't follow best practices. This has resulted in a well intentioned, but often ineffective, reaction to erect barriers to force workers to do the “right” thing. As noted in our review above, this can result in a system that is so inflexible that it prevents or hinders workers from delivering appropriate care, or it leads workers to create workarounds that can jeopardize themselves or the institution legally, and even bring harm to patients. For example, estimating a required patient weight when there is no way to weigh the patient can lead to future dosing errors. In future work we plan to extend SYFSA to incorporate the other two types of flexibility.

3 A Framework for Systematic Yet Flexible Systems Analysis (SYFSA)

To illustrate our framework and how it can be used to design SYF systems, we consider a simplified procedure: central venous line insertion [24]. Central lines are used to establish reliable access to large (central) veins to deliver medications and fluids, to draw blood for testing, and to obtain measurements, such as central venous pressure. Once inserted, a central line remains in place for days or weeks. As a result, patients may develop central line infections that substantially increase morbidity and mortality. However, the chance of infection is reduced by following infection control guidelines during insertion and by minimizing the number of days that the central line stays in the body.

Our example is a simplified version of the insertion procedure and sacrifices realism for clarity. For example, hands are usually washed (scrubbed) before putting on a sterile gown to avoid contaminating the gown. We consider only the following actions, listed in the approximate order required to comply with best practices for infection control:

• Sterilize Site

• Drape patient

• Put Hat On

• Put Mask On

• Put Gown On

• Wash hands

• Glove Up (put gloves on)

• Insert Central Line

• Apply Sterile Dressing

Under ideal circumstances, a caregiver first prepares the patient by sterilizing the insertion site and then fully draping the patient. The caregiver inserting the central line, must then put on a mask, hat, and gown. The gown prevents one from donning a mask and hat, so while the order of mask and hat does not matter, they must both come before donning a gown. Once the gown is on, the caregiver washes his or her hands, and then puts on sterile gloves. Following this, he or she inserts the central line, and places a sterile dressing over the insertion site.

Following Newell and Simon [25], a problem space consists of a symbolic representation capable of capturing each problem state, a set of operators which are information or physical processes that transform one state into another, an initial state, and one or more goal states. Just prior to setting up a new programmable infusion pump for a patient, the initial state is one in which the pump is turned off, whereas the goal state is one in which the pump is infusing the prescribed drug at the prescribed rate and volume. Infusion pump operators consist of the actions (such as the buttons on the front panel) available to install the drug administration set and then program the pump.

In general, a problem space of a real world task may consist of hundreds, thousands or even millions of states and transitions between states (operator applications); manual analysis is difficult or impossible. Thus, we implemented each space as a model in Mathematica [26] that generates a finite state machine (FSM) that contains every possible state and operator application. We then used the FSM to visualize the space, and to calculate measures for each space, such as all possible paths between a pair of states, the number of states, different goal states, and so on. A Mathematica notebook containing the code for the examples presented here is available from the first author and may be used to develop new models. We do not describe the details of this approach in the paper, because it is just one of many possible ways to automatically calculate the equations described below. The basic approach to generating and using FSMs for the analysis of user interaction is fully described by Thimbleby in a book [27] and several articles (e.g., [28–30]).

In the remainder of this section we walk through the specification and implications for each of the three spaces, beginning with the idealized space. Although we present the spaces sequentially, we expect the framework to be used in an iterative fashion. Part of the value of the framework is that it provides insight needed to better understand a task and how to design an SYF system to support the task.

3.1 The Idealized Space

The idealized space is best specified as a work domain ontology (WDO) for the task [31]. A WDO defines an explicit, abstract, implementation-independent description of a task by separating the task from work context, and the technology used to accomplish the task. In other words, the WDO separates the inherent constraints of the task from constraints that are due to system design. Rather than focusing on details of the current system, the WDO highlights the fundamental nature of the work, thereby providing guidance for designing an appropriate system to support the work. WDO does not provide explicit methods for discovering and visualizing constraints; however, CWA (see Section 2.4) provides a range of such methods and visualization tools.

A WDO is easy to express as a problem space. The WDO goal is specified as one or more goal state(s). Operations in the WDO are specified as problem space operators. Constraints are specified as sets of preconditions on the operators.

3.1.1 Assumptions

As with all models, a WDO is based on a variety of assumptions that set the scope of the model; i.e., which elements of the real world are considered relevant and which are not. When we specify the idealized space we must always clearly specify our assumptions.

For the idealized central line insertion space, we assume that a single caregiver will accomplish the entire task, that all required supplies are available, and that there is sufficient time to do the entire procedure according to best practices. We also assume that the objects needed to follow the best practice and the caregiver are specified in the WDO. That is, they are inherent components of the abstract task.

Explicitly listing these assumptions allows us to better assess the validity and scope of the idealized space and, subsequently, the results of the entire analysis. For example, Berenholtz, et al. [24] found that lack of ready access to supplies was a major barrier to following the best practice for central line insertion. Part of their intervention for lowering central line infections was to develop a central line insertion cart, restocked on a regular basis. We assumed that all supplies were on hand to simplify our example, but in an actual design setting, making this assumption explicit would allow one or more of the stakeholders in the design process to question its validity, with the possibility of modifying the analysis.

3.1.2 State Representations

To specify a problem space we must decide how to represent system state. Abstractly, we think of the state representation in terms of a set of state variables, and a specific state as a specific assignment of values to each variable. In this example we use a simple Boolean representation of the state components to record whether an action was done or not. For instance if nothing has been done the components would all be false, thus:

• centralLineInserted = False

• drapePatient = False

• glovesOn = False

• gownOn = False

• hatOn = False

• maskOn = False

• sterileDressing = False

• sterilizedSite = False

• washedHands = False

Here, you can read “=” to mean “is.” This representation captures the state of the system regardless of whether an element of the system state is visible or hidden. For instance, putting on gloves is a readily visible change to the system state. In contrast, washing hands is not. There are many different ways to represent system state. We suggest including the minimum properties of the state needed to support the idealized problem space. One should model the “relevant” features. As the model is analyzed, other significant features may be recognized and added to the model. We will discuss the importance of this advice below when we describe the natural and system spaces.

3.1.3 Operators

We define operators using a set of logical preconditions on the state and how the operators change the state (see Table 1).

Table 1.

Operator and conditions for the idealized central line insertion space.

Operator	Precondition	Postcondition
Sterilize Site	¬sterilizedSite	sterilizedSite'
Drape patient	¬drapePatient ∧ sterilizedSite	drapePatient'
Put Hat On	¬hatOn ∧ drapePatient	hatOn'
Put Mask On	¬maskOn ∧ drapePatient	maskOn'
Put Gown On	¬gownOn ∧ hatOn ∧ maskOn	gownOn'
Wash hands	¬washedHands ∧ gownOn	washed Hands'
Glove up	¬glovesOn ∧ washedHands	glovesOn'
Apply Sterile Dressing	¬sterileDressing ∧ centralLineInserted	sterileDressing'
Insert Central Line	¬centralLineInserted ∧ glovesOn	centralLineInserted'

Here, we use the conventional symbols ¬ for logical NOT and ∧ for AND. The prime (′) notation means the value after the operator has been applied to a state. For instance, the preconditions for the operator “Drape patient” state that the operator can only be applied to states in which drapePatient is false and sterilizedSite is true, and that after the operator is applied the state component drapePatient will be true. The prime notation allows this to be stated mathematically: operator(“drape patient”) = ¬drapePatient ∧ sterilizedSite ∧ drapePatient'. As usual, any state component not mentioned is unchanged; if we wished we could have written operator(“drape patient”) = ¬drapePatient ∧ sterilizedSite ∧ drapePatient' ∧ maskOn=maskOn', which means the same thing, except it redundantly says that the state of the mask is unchanged.

Coincidentally in this example all operators only achieve setting the corresponding state component; thus “wash hands” implies washedHands', but in general many components might be affected. For example, if we tracked left and right hands separately, then the single “wash hands” would achieve two outcomes: washedLeftHand' ∧ washedRightHand'. Note also, that operators are formal problem space constructs that specify one or more task actions. In the central line insertion example each operator corresponds to a single task action, but, in general, an operator can take parameters that define a set of task actions. For example, in an interface for selecting from among several patients we could define a Select(<patient>) operator, where <patient> is any patient shown on the screen. If 20 patients are shown on the screen, this single operator could be instantiated 20 times resulting in 20 different possible task actions.

Finally, we note that automated model checking can (and should) be used on specifications such as this. For example, it is easy to check automatically that centralLineInserted always implies maskOn, even though this is never stated explicitly (and it would be tedious and error-prone to try to say so for all relevant states).

3.1.4 Initial State

The initial state is one in which nothing has yet been done: all components are False.

3.1.5 Goal State

The goal state for this example is one in which all of the operators have been applied (equivalently, all of the actions have been done), and thus all the components are true:

• centralLineInserted = True

• drapePatient = True

• glovesOn = True

• gownOn = True

• hatOn = True

• maskOn = True

• sterileDressing = True

• sterilizedSite = True

• washedHands = True

This is equivalent to the more concise logical statement centralLineInserted ∧ drapePatient ∧ glovesOn …

The goal state specifies only that all operators have been taken, not that they have been done in the correct order. There is no way to specify sequences of operators in terms of state properties alone. Instead, we constrain the sequence through the operator preconditions. Taken together, the initial state, goal state, operators, and operator preconditions, restrict the problem space to paths that reach the goal using an appropriate sequence of operators. However, we are not restricted to using this representation. Other representations may help us understand and explore the space from different perspectives. For example, we might choose to track whether the field is sterile or not and how actions affect whether or not a sterile field is created or maintained. We could then specify that some actions should only be done in a sterile field. Taking this further, we could choose to represent the urgency of the procedure and then modify the goal and operators to explicitly consider numerical time factors. Exploring alternative problem space formulations may inform system design.

3.1.6 Goal State

When a space is small, visualizing it can aid in understanding and pinpointing sources of flexibility and systematicity. From the idealized space shown in Figure 1, we can see that there is only one goal state with two different paths to it. The shortest path from the initial to the goal state is nine steps. There is clearly, very little flexibility — one choice — in the idealized space.

Figure 1.

The idealized problem space. The initial state is square and the goal state is a diamond. The black circle is a state in which the central line is in place but the sterile dressing is not yet applied.

3.2 The Natural Space

The natural space captures the task actions and constraints on those actions imposed by the physical world. For example, one natural constraint is that you cannot remove a surgical glove that you have not put on. In contrast, you can wash your hands with surgical gloves on. In the natural space we also separate the primary goal from secondary goals. For instance, inserting the central line is the primary goal, while putting a sterile dressing on the insertion site is secondary.

Unlike the idealized space, the natural space need not be a WDO (work domain ontology; see above). Since the natural space is intended to reflect the real world, we can capture aspects that may affect task performance, such as non-task critical artifacts or cognitive limitations and assumptions. For instance, we might assume that no clinicians will apply the sterile dressing prior to inserting a central line, even though there is nothing to physically prevent this.

When representing the state in the natural space, we must consider that some state variables may be measurable and some may be hidden (or latent). Distinguishing between the two is a matter of perspective. For example, in a typical automatic teller machine (ATM) the user has no visible indication of whether his or her ATM card is in the machine. However, this state variable is readily available to the ATM. When considering which variables are hidden vs. visible, we recommend taking the perspective of the human or humans who are part of the system. If the human cannot readily detect the value of a state variable, consider it hidden. In addition, assume that cognitive state variables are hidden. The former recognizes that the human in a system is likely to forget or distort the values of state variables that are not readily observable in the environment. The latter recognizes that cognitive states are also likely to be forgotten or distorted. Both are likely to occur given the stress and interruptions present in many real world settings.

3.2.1 Assumptions

For the natural central line insertion space, our assumptions are very similar to those of the idealized space. We assume that a single care-giver will accomplish the entire task, that all necessary supplies are available, and that there is sufficient time to do the entire procedure. We also assume that the artifacts needed to follow the best practice and the caregiver are part of the task model. In contrast to the idealized space, we define central line insertion as the primary goal. Creating and maintaining a sterile field are possible, but not required, because there are no natural constraints that enforce these requirements.

3.2.2 State Representations

We use the same representation as the idealized space.

3.2.3 Operators

The operators for the natural space are identical to those of the idealized space, but the preconditions reflect hard constraints found in the task environment (see Table 2). These are that the hat and mask cannot be put on after the gown is on and that the sterile dressing will not be put over the insertion site prior to inserting the central line. The preconditions also reflect our assumption that all other operators, except applying the sterile dressing, will not be done once the central line is in place.

Table 2.

Operators and conditions for the natural central line insertion space.

Operator	Precondition	Postcondition
Sterilize Site	¬sterilizedSite ∧ ¬centralLineInserted	sterilizedSite'
Drape patient	¬drapePatient ∧ ¬centralLineInserted	drapePatient'
Put Hat On	¬hatOn ∧ ¬gownOn ∧ centralLineInserted	hatOn'
Put Mask On	¬maskOn ∧ gownOn ∧ ¬centralLineInserted	maskOn'
Put Gown On	¬gownOn ∧ ¬centralLineInserted	gownOn'
Wash hands	¬washedHands ∧ ¬centralLineInserted	washedHands'
Glove up	¬glovesOn ∧ ¬centralLineInserted	glovesOn'
Apply Sterile Dressing	¬sterileDressing ∧ ¬centralLineInserted	sterileDressing'
Insert Central Line	¬centralLineInserted	centralLineInserted'

3.2.4 Initial State

The initial state is the same as the idealized space.

3.2.5 Goal State

The goal states are any states in which the central line is in place. The goal is therefore a set of states.

3.2.6 Analysis of the Natural Space

The network diagram in Figure 2 shows that the natural space is more complex and has considerably more flexibility than the idealized space. As with the idealized space, the initial state is shown as a square, goal states are black, and the goal state with all operators applied, though not necessarily in the right order, is shown as a black diamond. There are many more goal states in the natural space, because it recognizes that a person may stop once he or she has accomplished the primary goal (central line placement).

Figure 2.

The natural central line insertion space. The initial state is the square in the lower right quadrant of the central image. The goal state in which all operators have been applied is the black diamond in the upper left corner. Black circles are states in which the central line has been placed. White circles are states where the central line has not been placed.

The natural space has 384 states of which 256 are goal states. There are 13,004 paths that lead to a state in which the central line is inserted with the shortest being 1 step and the longest 9. Although there are 1,680 possible paths to the “ideal” goal state, only two of these paths contain the appropriate sequence of 9 steps that reflect best practice.

Comparing the natural space to the idealized space, we can see that the ideal sequence of actions is not enforced or encouraged by physical constraints. In addition, some actions, such as washing hands or sterilizing the site, may leave no visible record, meaning that the current system state is not visible. A lack of visibility of system state is a major usability problem that can lead to errors of omission (omitting a necessary step; e.g., not washing hands) and commission (including an unnecessary step; e.g., washing hands twice). Further, the system state contains insufficient information to allow an observer to detect the ideal goal state. The state variables in our problem space indicate only which actions were done, not the sequence of actions. However, the ideal goal depends, in part, on the action order. Finally, because the sterile dressing is placed after the primary goal of central line insertion is achieved, there is a strong chance of post-completion errors [32], which are errors that occur when a person forgets to do an important task action that must be taken after they have accomplished the primary goal. Typical post-completion errors are forgetting to retrieve your ATM card after you have received cash from the machine, or leaving an original document on a copier after you have made copies.

Taken together, the characteristics of the natural space allow flexibility that makes idealized task performance less likely to be achieved (i.e., intuitively the task might be considered “error prone.”). Below we use the comparison between these two spaces to consider a SYF (Systematic Yet Flexible) system that encourages ideal performance, while supporting graceful degradation under unexpected or unusual conditions.

3.3 The System Space

As noted above, stakeholders can use SYFSA to design a new system or to evaluate (and possibly refine) an existing system. For this demonstration of SYFSA, we base the system space on the existing intervention proposed and implemented by Berenholz, et al. which has nearly eliminated central line-related bloodstream infections in multiple institutions [24,33]. Although the intervention was widely reported to consist of a simple checklist, it actually has five components: (1) educating the staff on the best practices and the intervention; (2) creating a central line insertion cart to ensure easy access to all supplies needed to comply with the best practice; (3) asking daily whether the central line could be removed; (4) a checklist to ensure adherence to best practices; and (5) empowering nurses to stop the procedure if the guidelines were not followed during non-emergency situations. Here, we are concerned only with the elements of the intervention that directly affect central line placement.

These interventions lead to a system that addresses several of the characteristics, assumptions, and problems noted in our idealized and natural spaces. The supply cart supports our idealized space assumption that all supplies will be available at the start of the procedure. The checklist, external monitoring by a nurse, and the nurse's power to stop the procedure, encourages and enforces the ideal practice. The checklist itself increases visibility of system state and externalizes knowledge of the ideal action sequence. Taken together, these factors provide and encourage systematicity. At the same time, the system provides flexibility by allowing the provider to deviate from the best practice in situations where the central line must be inserted emergently.

The resulting system space is a combination of the graphs from the natural (Figure 2) and idealized spaces (Figure 1) with a new root state that switches between the two original root states depending on whether there is an emergency. Switching to the natural space relaxes the action constraints imposed by the idealized space and allows the provider to accept a goal that trades off the chance of an infection with the need to quickly insert the line.

As presented above, SYFSA provides a means of qualitatively analyzing trade-offs in systematicity and flexibility during organizational or information system design. The explicit descriptions of each of the three spaces, in terms of initial state, goal state(s), and operators and their preconditions (constraints on actions) force stakeholders to explicitly describe their assumptions and understanding of each of the spaces. By making these descriptions explicit, stakeholders can share, debate, and refine each space. This allows stakeholders to determine whether each space adequately models the best practice (idealized space), the current system (natural space) and the new or redesigned system (the system space). Comparing the descriptions of these spaces can then reveal trade-offs or potential opportunities to iteratively refine each space to better address stakeholder needs.

In the next section we consider information theoretic measures for qualitatively comparing the flexibility of different system designs and how closely they match the flexibility required to complete a task.

4 Information-Theoretic Measures of Procedural Flexibility

As we noted earlier, there are many different measures of flexibility. Here, we propose several flexibility measures that capture our intuitive notion of procedural flexibility and allow us to compare the flexibility of different SYF system designs with respect to one or more tasks. We distinguish between inherent task flexibility and system flexibility. The former is the amount of flexibility required to do a task, whereas the latter is the amount of flexibility in a system that is designed to support the task. For instance, if the task is to deliver a single dose of epinephrine, the inherent task flexibility is low and is best met by designing a device, such as an epininephrine autoinjector, that has similarly low system flexibility. System flexibility often differs from task flexibility, because a particular system may admit actions that are incorrect or irrelevant to completing a task, or may not allow actions that are actually needed to complete a task. Thus, a system may support more or less flexibility than is inherent in the task. When a system design allows more flexibility than is inherent in the task, it allows actions that may lead to errors or inefficiencies. In contrast, when a system design supports less flexibility, it may be impossible to complete the task.

To derive appropriate measures of flexibility, we start by considering the extreme end points of task flexibility: no flexibility and complete flexibility. We propose that if there is only a single correct way to complete a task, then that task has 0% flexibility; whereas if any possible sequence of task actions completes a task, then that task has 100% flexibility. Between these limits, flexibility should increase monotonically (that is, if there are more ways of accomplishing the task, then flexibility should not decrease).

To explore this concept, consider the following three simple tasks:

• Any-object: Table A has ten objects and Table B is empty. The goal is to place any one object from Table A onto Table B.

• All-objects: Table A has ten objects and Table B is empty. The goal is to place all ten on Table B.

• Sort-objects: Table A has ten numbered objects. The goal is to move all ten objects in increasing order to Table B (i.e., object 1, 2, 3 …, object 10).

In our running example, Table A might be the central line supply cart and Table B might be the sterile field. By our intuitive definition of flexibility above, Sort-objects is the least flexible of the three tasks, but which of the other two is the most flexible? If we define flexibility as the number of paths to the goal, then All-objects with 10!=3,628,800 paths is clearly more flexible than Any-object with only 10 paths. But intuitively, it seems that Any-object is equally, if not more flexible than All-objects, because Any-object allows any choice of action, and just one choice is needed. In contrast, although All-objects allows any sequence of actions to lead to the goal, each choice constrains the actions that follow, which intuitively would seem to decrease flexibility. In fact, a system space that allowed a person to move an object from Table B back to Table A would be overly flexible for the All-objects task. Thus, the number of paths in a space can have more to do with the size of the space, rather than constraints on actions.

Instead of using the number of paths to the goal to define flexibility, we can use the average amount of information needed to choose an action per non-terminal state (whether those states lead to a goal or non-goal terminal state). In information theory [34], the amount of information (measured in bits) in a choice between n equally likely actions is log₂(n), so the total information required to perform a sequence of actions is the sum of the information for each decision along the path. Suppose that there are n non-terminal states S_i, and these states have a corresponding number of equally probable actions a_i (in terminal states there are no actions). Then the average bits per non-terminal state F is given by

$$F=\frac{{\sum }_{i=1}^n{log}_2(a_i)}{n}$$ (Equation 1)

We can convert F to an indicative flexibility score. Of the many possibilities, here we define a percentage so it is conveniently measured as a number increasing to 100:

$$\frac{100F}{F+1}$$ (Equation 2)

Equation 2 approaches 100% as F increases. In addition, because of the definition of F in Equation 1 together with Equation 2, a space where every state has a single action has 0% flexibility, whereas a binary tree (in which all non-terminal states have two actions) has 50%.

Table 3 shows the flexibility of the three simple tasks described above. Consistent with our intuitive notion of flexibility, Sort-objects has zero flexibility, Any-objects has the most flexibility, whereas All-objects has less flexibility because each action further constrains the remaining available actions. Although any possible action leads to completion of Any-object, it falls short of our intuitive 100% flexibility measure because the information theoretic measure considers the number of choices at each step. As a result, the flexibility of Any-object will approach 100% as the number of objects increases.

Table 3. Flexibility of three simple tasks using bits per state (Equations 1 and 2).

Task	F	% Flexibility
Any-object	3.32	76.86%
All-objects	0.51	33.64%
Sort-objects	0	0%

Table 4 shows the flexibility of the three types of spaces for central line insertion. As expected, the idealized space has the least flexibility, whereas the natural and system spaces have considerably more, with the system space being nearly as flexible as the natural space. The small difference in flexibility between the natural and system spaces is misleading, because the more flexible path through the system space can only be taken in emergency situations—situations that are less likely to occur than non-emergent situations. The general problem is that Equation 1 assumes that all states have an equal chance of being visited (which is false because of structural properties of the space, e.g., the top state is always visited) and because the actions from any single state may be chosen with differing probabilities. The problem is easily corrected by computing the average amount of information based on the probability of each action in each state. If a non-terminal state Si has ai actions and those actions have probabilities pⁱ₁, …, pⁱ_ai, then a choice of action at S_i conveys an average number of bits given by:

Table 4. Flexibility of three central line insertion spaces using bits per state (Equations 1 and 2).

Space	F	% Flexibility
Idealized	0.1	9.1%
Natural	0.94	48.5%
System	0.91	47.6%

$$B_i=\sum _{j=1}^{a_i}{\mathit{\text{p}}}_{\mathit{\text{j}}}^{\mathit{\text{i}}}{log}_2\left(\frac{1}{{\mathit{\text{p}}}_{\mathit{\text{j}}}^{\mathit{\text{i}}}}\right)$$ (Equation 3)

This results in a version of Equation 1 that considers the probability of actions:

$$F=\frac{{\sum }_{i=1}^n{B}_i}{n}$$ (Equation 4)

However, this equation alone does not consider how action probabilities affect the likelihood of reaching future states. In the central line insertion space, Equation 1 assumes that emergency and non-emergency situations are equally likely, resulting in 1 bit for the initial state. If we instead assume that an emergency occurs, say, 10% of the time, Equation 3 reduces the required bits for the initial state from 1 to 0.469. Given the number of states in the space, however, and assuming that actions for all subsequent states are equally likely, this decrease for the initial state has very little effect on overall space flexibility (47.63% to 47.58%).

In general, it is important to consider the probabilities of actions in a SYFSA analysis, because SYF systems support graceful degradation by making common actions and action sequences easy and uncommon ones possible. For example, in a user interface, common actions may be made more salient and/or faster to select than less common actions. This provides for graceful degradation in the face of unanticipated events. To account for the probabilistic effects of actions on future states, we need to weight the average bits per state, B_i, by the probability of reaching each state. If there are n non-terminal states and these states have probabilities s₁, …, s_n, then the weighted average bits per non-terminal state is given by:

$$F=\frac{{\sum }_{i=1}^n{s}_i{B}_i}{{\sum }_{i=1}^n{s}_i}$$ (Equation 5)

Because the probabilities of the non-terminal states need not sum to one the weights are normalized by dividing by their sum. Table 5 compares the percent flexibility of the three central line insertion spaces using the non-weighted, non-probabilistic F from Equation 1 to that of Equation 5. The weighted measure for the idealized space shows very little difference. However, there are larger differences in the measures for the natural and system spaces. The natural space nearly doubles the required number of bits per state, reflecting that earlier states have both higher probabilities of being reached and a larger number of possible actions. The system space mean bits per state decrease from 0.91 to 0.78, reflecting the lack of flexibility in the idealized path. More importantly, under Equation 5, the system space is now less flexible than the natural space (43.8% vs. 65%), as compared to their difference under Equation 1 (47.6% vs. 48.5%).

Table 5.

Comparison of the flexibility of three central line insertion spaces using non-probabilistic (Equation 1) vs. probabilistic (Equation 5) flexibility measures.

	Non-Probabilistic (Equation 1)		Weighted Probabilistic (Equation 5)
Space	F	%F	F	%F
Idealized	0.1	9.1%	0.11	10%
Natural	0.94	48.5%	1.86	65.0%
System	0.91	47.6%	0.78	43.8%

Another useful information-theoretic measure for comparing spaces is the average information per path. This measure tells us, on average, how much information a person must convey in a particular space.

The total information conveyed by a single path is equivalent to the information content as measured by the probability of following the path (i.e., choosing a sequence of actions that result in taking the path to the goal). For instance, the probability of a path that has 6 states and 5 edges, where each edge has a probability of 0.5, is 0.5⁵. The sum of the information conveyed by each of the 5 decisions is 5log₂(1/0.5) = 5, which is equal to the log of the probability of the path: log₂(0.5⁵). Thus, the average information over all paths P₁, …, P_n with probabilities p₁, …, p_n is given by:

$$P^{\text{avg}}=\sum _{i=1}^n{p}_i{log}_2\left(\frac{1}{p_i}\right)$$ (Equation 6)

This measure is sensitive to the size and complexity of a space, in that spaces that are deeper and have more choices per decision will naturally have greater average information per path. As noted in the previous section, it is often useful to compare the average information of specific paths, such as the correct paths in both the idealized space and the natural space. Table 6 shows the average bits per path for the three central line insertion spaces. The difference between the natural and system spaces results from the fact that the first state of the system space is an equally likely choice between an emergency situation, which leads to the natural space (requiring 9.62 bits), and a non-emergency, which leads to the idealized space (requiring only 1 bit).

Table 6.

Average bits per path for the three central line insertion spaces using Equation 6.

Space	Average bits per path (Eq. 6)
Idealized	1.00
Natural	9.62
System	6.31

We can also use a similar measure to quantify how efficiently the natural space supports the best-practice by comparing the amount of information a clinician requires to do the best practice in the natural space versus the idealized space. In the idealized space there are two equivalent paths of nine non-terminal states. Eight of the nine states permit a single action, whereas one state has two possible actions. This means that a person need only convey one bit of information to correctly perform the task in the idealized state. In contrast, the natural space has the same two paths, but because of the lack of natural constraints on possible actions, seven of the non-terminal states allow more than one action. The initial state has eight possible actions, the second state seven, and so on, with each correct action eliminating one possible action until the final two non-terminal states admit a single action each (with zero bits of information). Assuming actions are equally probable, this makes the total bits in either correct path:

$$\sum _{i=2}^8{log}_2(i)=15.2992$$ (Equation 7)

Since the idealized space requires only 1 bit, the efficiency of the natural space for supporting the best practice is only 100 * 1/15.2992 = 6.5%.

According to the Hick-Hyman law, the time to make a decision is proportional to the amount of information in the available choices [35,36]. As a result, the information theoretic analysis of a system provides a prediction of cognitive load and relative task times (e.g., a task that requires more information is likely to take longer than a task that requires less information). In addition, through practice a person can automate a consistent sequence of task actions, resulting in fast, nearly subconscious behavior. This means that a person must acquire, through practice, over 15 bits of information to fully automate the idealized task in the natural space, but only 1 bit in the idealized space. We can use this kind of analysis to compare the learnability of different spaces for different system designs.

5 Discussion and Future work

While the current approach is clear and rigorous, there are a number of limitations to SYFSA that should be noted, and could provide inspiration for further work. Primarily, SYFSA as described in this paper is designed to analyze systems that support a single task. However, many systems (such as an infusion pump) must support more than one task. In SYFSA such systems are modeled by expanding the spaces so that they admit all tasks and then separately analyzing each task. For example, a programmable infusion pump supports many different volumes and rates of delivery, so the idealized space must include operators that can be applied to achieve each possible (and allowable) combination of volume and rate. Each task, such as the task of starting at a state where the rate is 0 and then moving to a state where the rate is 125, can then be analyzed using the equations described above. To analyze the entire system, a designer must analyze each task separately. It is up to the designer to decide how to aggregate the results of each task analysis. For instance, the designer could produce a single flexibility measure using a weighted average of each task's flexibility, where the weights are the expected frequency of each task.

SYFSA does not provide designers guidance on how to determine which tasks should be included in an analysis, so it is important for the designer to use other work-centered or user-centered methodologies to determine which tasks a system should support. In addition, any system designed to support multiple tasks necessarily requires additional procedural flexibility because the user has more possible actions to take at each step. This flexibility can lead to errors and inefficiencies for any one of those tasks. For example, since a programmable infusion pump must provide actions that allow a user to enter different volumes and rates of delivery, but since the device does not know what the user wants to enter it cannot completely constrain the users' behavior for the specific task at hand. Designers of infusion pumps have dealt with this problem by including dose error reduction systems, wherein the user must first specify a drug and concentration prior to programming the pump. Once the pump knows the drug, the pump can enforce additional, drug-specific constraints on rate and volume. Developing a work domain ontology to inform the idealized space (as we suggest in Section 3.1) can help designers better explore intrinsic task constraints. In any design for supporting multiple tasks, common user-centered design principles recommend providing error reversal, or undo, functionality to traverse back through prior choices, to change them or to review them. For example, if a clinician accidentally sets an infusion pump to 100 mcg/min instead of 10 mcg/min, it should be possible to clear or re-enter the infusion rate. This reflects an increase in flexibility over the idealized space (which assumes a perfect user), but is an appropriate trade-off given the realities of the natural space in which even highly trained users can make mistakes.

Design frameworks, such as SYFSA, are often difficult to validate. They tend to be used (or abandoned) based on whether designers find them useful and easy to use. Any evaluations are often qualitative in nature, consisting of case studies and arguments that outline strengths and weaknesses. However, some aspects of SYFSA may be empirically testable. SYFSA assumes that systems that are too flexible relative to the task (the idealized space) will be harder to learn and use, as will systems that support too little flexibility. Building on several existing laws and cognitive results, we also believe that SYFSA can predict relative efficiency, cognitive load, and learnability. However, we have not yet empirically evaluated these claims.

Another challenge, as we noted above, is that many real world tasks and systems can have dozens or hundreds of possible actions leading to thousands or even hundreds of thousands of states in each problem space. There are at least three solutions to this problem. The first is to generate and analyze the spaces computationally as we have done for the examples in this paper. Thimbleby describes these techniques in detail and they are also demonstrated in the Mathematica code available from the first author. [27] The second is to reduce the complexity of the spaces by selecting an appropriate level of abstraction. For example, in the central line examples we did not model the detailed cognitive steps required to determine the best location to insert the central line, nor all of the physical steps involved in the process, such as opening equipment packages. As with any modeling approach, selecting the right level of abstraction is challenging and remains part art and part science. The third solution is to separately analyze subparts of a complex system. For instance, we analyzed an infusion pump by analyzing the number entry tasks (for specifying rate and volume) separately from the other tasks involved with the pump (e.g., entering various data entry modes, pausing the infusion, responding to an alarm, etc.). In practice, it is often necessary to use a combination of these approaches to tame the complexity of real world tasks.

Finally, as noted above, the measures described in this paper characterize procedural flexibility only, not functional or operational flexibility. These other forms of flexibility are also important for health information and organizational systems, and will require extensions to SYFSA.

6 Conclusions

SYFSA is a systematic approach to analyzing and designing Systematic Yet Flexible (SYF) systems. By explicitly representing three spaces, the idealized space, the natural space, and the system space, designers and domain experts can examine the assumptions behind task analysis and system design, and the possible tradeoffs between systematicity and flexibility. By making assumptions and constraints on actions explicit, the framework provides a means for designing novel systems that better support the constraints inherent in a task, but not in the natural environment. In addition, the quantitative information theoretic flexibility measures allow analysts to compare different spaces and system designs in terms of relative efficiency for supporting a task, cognitive workload, and learnability.

• Healthcare quality can suffer when systems are too rigid or too flexible

• Systematic Yet Flexible systems support a mix of systematicity and flexibility

• SYFSA is a design framework for selecting the right mix

• SYFSA provides qualitative and quantitative methods for comparing designs