Impact of Human Factors on Clinical Protocol Performance: A Proposed Assessment Framework and Case Examples

Abstract

Background

Hyperglycemia is prevalent in critical care and tight control can save lives. Current ad-hoc clinical protocols require significant clinical effort and can often produce highly variable results. Thus, tight control remains elusive as there is not enough understanding of the relationship between control performance and protocol design, particularly with regard to how a given protocol is implemented.

Methods

This article examines the role of human factors and how individuals relate to technological protocols in clinical settings. The study consists of an overall brief review that is used to create a first graphical representation of the impact of human factors in clinical medical protocol implementations. This initial framework is examined in the context of two similar, but different, case studies—the specialized relative insulin and nutrition tables glycemic control protocol and the TREAT system for antibiotic selection.

Results

A graphical framework relating the human factors impact on medical protocol implementation is created. This framework describes the primary impacts on performance as resulting from clinical burden and protocol transparency. Their primary effect is on compliance with the protocol, which directly affects performance and outcome, particularly in long-term studies versus short pilot studies.

Summary

Compliance is a key element in obtaining the best clinical outcome that a given protocol can provide. The issues that most affect compliance are quite often unrelated to the patient or treatment, but are a function of the protocol design and its ability to integrate (by its design) into a given clinical setting. A framework for examining these issues in design and in post-hoc assessment is therefore proposed and examined in two brief case studies.

Introduction

Intensive insulin therapy (IIT) in the intensive care unit (ICU) has been of great interest since the landmark studies of van den Berghe et al.^1,2 and Krinsley.³ The potential economic benefits of tight control have only served to increase this interest.^4,5 Therefore, the idea that tight glycemic control in critical care saves lives is increasingly less questioned. In contrast, the “how” and “for whom” remain elusive.

To find these answers, increasingly more complex, often computerized and/or model-based protocols have been designed. Results are still often very variable due to differences in cohort, protocol, and, potentially, compliance. In particular, studies^6,7 have noted difficulties with compliance to timing or dosing, while others report excessive clinical burden associated with IIT.^8–13

The performance or success of any clinical protocol will always depend primarily on its clinical efficacy. However, human factors must also be taken into consideration. A protocol may perform perfectly to clinical expectations and results with 100% compliance, but suffer much degraded performance if compliance is reduced, significantly influencing the clinical outcome and effectiveness. Thus, when a practitioner is faced with a clinical decision, the degree of decision support and how it is presented is an important factor for the decisions and actions they choose.

In a rigorous review of clinical decision support systems, Kawamoto and colleagues¹⁴ found the following four independent predictors of effective clinical decision support.

1. Decision support provided automatically as part of clinician workflow.

2. Decision support provided at the time and place of decision making.

3. Actionable recommendations provided.

4. Computer based.

In this study, 94% of systems possessing all four features significantly improved clinical practice versus 46% for systems lacking one or more features. The results of Garg et al.¹⁵ and Haynes et al.¹⁶ found that automatically generated versus user-initiated decision support resulted in better delivery. These findings suggest that effective clinical decision support systems must minimize the effort required by patients or caregivers to receive and act on recommendations, as well as pointing to compliance as a determining factor for clinical outcome.

Overall, these results show that

1. Clinical burden is a major issue and can affect compliance and thus outcome.

2. Systems that reduce clinical effort can produce improved results.

3. Despite the improvements of computerization, compliance and outcome are still variable in IIT and, in general, are less than ideal.

Hence, successful uptake of a protocol may well depend on nonphysiological factors, relying instead on the ability to implement it effectively—the human factors. The goal of this article is to introduce an initial, more formal problem definition to the field rather than a specific solution.

Problem Definition

An overall assumption is that the full success of an effective IIT protocol is a function of compliance to the requirements of that protocol in timing and treatment. Thus, an effective IIT protocol that delivers tight control will provide strong objective feedback that bolsters compliance, while a poor one may not. However, several nonphysiological factors may also affect compliance or make it difficult to comply.

1. Management and collaboration. Relationships among nursing staff, doctors, pharmacists, and hospital management affect collaboration, especially if there is not a multidisciplinary approach or lack of a clinical leader.^17–20

2. Clinical burden. The time and effort needed to administer a protocol. Complex protocols or those requiring many interruptions, trips from the bedside, extensive data management or input, or outside input directly affect compliance.^{17–19,21–23} These issues may all be enhanced by geographically distributed ICUs or blood glucose sensing, the time and effort required to maintain patient records in a given unit, and the level of critical illness of the patients.^18,19,21,22

3. Transparency. Complex protocols, difficult interfaces, and frequent deviations or changes serve to reduce consistency, remove insight into how the protocol works, and decrease trust in the protocol, all of which are important for good compliance to the guideline and reducing workarounds outside the protocol.^{17,18,21–24}

4. Training and education. The level of training and education affects both the transparency and the clinical effort required. Regular in-service training can also improve the efficiency of care and compliance to a protocol in longer term use.^20,25–29

These four factors represent the specific human factors or issues that can affect compliance and/or performance. They may all map to one or more of the specific outcomes noted in the prior section resulting from the work of Kawamoto and colleagues¹⁴ or the final three points made with respect to IIT protocols. This article proposes a framework in which addressing these issues will optimize the more general conclusions used to define the problem in the prior section. In particular, while clinical burden may be cited in many cases, these four factors more specifically target other aspects outside of that one issue or that may be conflated with it.

Figure 1 shows a schematic of how these factors might interact to affect compliance and clinical outcome. The human factors box shows where human factors design could directly affect clinical outcome of a protocol. Finally, note that many of these same issues are also relevant to the compliance of individuals to prescribed or recommended therapies.^26,30–32

Figure 1.

Graphical framework definition for the human factors aspects of tight protocol design as related to compliance and clinical outcome. Double-lined boxes show main issues, and single-lined boxes show selected relevant issues.

Figure 1 shows that a human factors-oriented design will not necessarily solve all problems. Certainly, training and education would be required in the implementation of any new protocol. Similarly, an atmosphere or organizational structure that makes collaboration among doctors, nurses, and other prescribing or purchasing authorities difficult will affect the ability to comply with a protocol, regardless of whether the protocol is otherwise well designed and effective.

Thus, the assumption is that where human factors design can play a role is in most effectively designing a protocol to match the clinical situation at the bedside. More specifically, it can help design the protocol to meet the workload, workflow, patient level, and structural layout requirements of the specific ICU. In addition, its impact on interface design and ease of use will increase transparency of the protocol to the clinical end users.

Related Case Studies and Experiences

This section briefly examines the framework of Figure 1 in terms of a paper-based glycemic control protocol developed from computerized versions, specialized relative insulin and nutrition tables (SPRINT),^33–35 and a computerized decision support tool for antibiotic selection, TREAT.^36,37 These examples are presented to show how any ICU decision support tool (computerized or not) might be analyzed within this type of framework rather than as representing any specific solution. Hence, these two systems represent, to an extent, the extremes of media (paper for SPRINT and computer for TREAT) and complexity (a few calculations for SPRINT versus more extensive data entry and a Bayesian network for TREAT).

TREAT

Appropriate antibiotic treatment decreases mortality, and superfluous treatment is associated with antibiotic resistance. TREAT is a computerized decision support system for antibiotic treatment targeting these outcomes.^36,37 In multicenter prospective and intervention cohort studies, TREAT improved empirical antibiotic treatment selection prior to microbiological diagnosis and reduced both cost and the superfluous use of broad-spectrum antibiotics. In particular, length of hospital stay, costs associated with future resistance, and total antibiotic costs were reduced significantly in the intervention groups.

However, these positive results were not equivalent compared across the three different centers utilized in three different countries (Italy, Germany, and Israel). In particular, the first treatment recommended by TREAT in the intervention group was appropriate significantly more often (76%) than what was prescribed by physicians in the control group (65%). The actual rate of appropriate treatment achieved in the intervention group was 73%,^36,37 whereas the rate of appropriate treatment when the advice of TREAT was followed was 85% (per protocol analysis). Results were thus affected by incomplete acceptance and compliance with the systems recommendations.

Factors that differed among clinical sites and presumably contributed to differences in compliance included a reported poor integration of the tool into the clinical workflow, as it was not made a regular part of their process. Thus, insufficient training of staff played a role directly and in terms of reported difficulty using a relatively complex, detailed, and multilevel interface. It was a particular problem in some institutions during periods with a high exchange rate of staff. Finally, because this system could require as many as 20 different pieces of data entry to refine a solution (symptoms, test results, etc.), a too high clinical burden was associated with data entry and thus its use. Finally, there were also obvious differences among the three clinical sites in terms of the support of the clinical trial from management, which then were reflected directly into the results.

All of these factors can be found in the framework of Figure 1. Additionally, the long time lag between antibiotic selection and clinical outcomes reduces the positive effect from objective feedback, as these are only seen days or weeks later.

The overall result was that in the intervention study the potential improvement of the clinical outcomes was not fully realized because of incomplete compliance.

SPRINT

SPRINT is a paper-based tight glycemic control system derived directly from a computerized protocol.^{33–35,38,39} To date, SPRINT has been utilized for 1.5 years and in over 400 patients, delivering tight control and reduced mortality in the Christchurch Hospital ICU. While not a computerized protocol per se, it offers added insight into the same human factors and design issues.

Compliance with the protocol in dosing of insulin and nutritional inputs is over 93–95% for the first 384 patients and 42,000 interventions. Many of the existing errors are due primarily to small misinterpretations or use of the covered paper wheels that the system utilizes, with the clinical staff choosing a value from an incorrect exposed column. Details on the SPRINT wheels and their ergonomics, design from computerized control methods, virtual trials, pilot testing, and ongoing results are presented in several related works.^34,35,40

A primary reason for this level of compliance has been objective feedback with staff being able to see positive results relatively immediately in a clinical sense. Additionally, the protocol has remained unchanged and requires no outside clinical intervention or deviation, reducing both a measure of clinical burden and adding transparency. Transparency is also increased by the simplicity of the paper interface—nothing is hidden and what-if scenarios or other questions can be seen or answered easily. Another major factor may well be education and its continuing input. When clinical staff changes, they are trained by experts, while every few months informal surveys collect feedback and provide answers to questions.

Finally, the interface designed was created in consultation with ergonomics experts. The goal was to ensure that it was as easy to use as possible and that mistakes would thus be minimized. This design was then taken to clinical nursing staff whose input on shape (wheels versus slides), coloring, size, and other metrics was also taken onboard in the design process. Note that this consultation also provided them a link and ownership of the protocol that might have ameliorated issues about management and ownership of the protocol and its use, although this specific aspect has not been surveyed.

With regard to clinical burden, SPRINT requires no outside collaboration from bedside staff. It is well integrated into the clinical workflow because all measurements are taken at bedside where the wheels are located and used rather than at a centralized blood gas analyzer. The Christchurch Hospital also has an approximately 1:1 to 1:2 nurse-to-patient ratio so the 1- or 2-hour intervention frequency (overall average 1.5 hourly) is not outside of normal duties. Finally, in an early survey of clinical staff users on the safety, efficacy, and ease of use after the first ∼30 total patients (1–2 per staff surveyed at ∼1-month use), 95% of responses rated SPRINT as satisfactory or better, with 74% rating it good or very good.³⁵ These results show how designing the protocol in consultation with staff and in recognition of the local work practice can result in high compliance and end-user satisfaction, thus minimizing any impact on the clinical outcome.

Discussion

A common theme emerging from many studies is one of clinical burden or effort required and its impact on the clinical results. The clinical burden of IIT has not gone unnoticed.^6–13,41 In particular, van den Berghe and et al.^2,42 utilized extra nursing staff, whereas Krinsley³ adjusted the protocol design to accommodate what might be accomplished with the staff available; more recently, Rea et al.²⁵ utilized a dedicated pharmacist. However, the idea that clinical effort can affect compliance to guidelines has been discussed only recently in the context of tight glycemic control,^6–8,27,41 although it is well reported in more generic ICU research.^17–19

Shulman and colleagues⁶ addressed the issue of burden and compliance directly by computerizing their protocol to reduce administration effort and then tracking compliance in the timing of glucose measurements and thus perhaps compliance and performance in control. Only 53% of glucose measurements were performed in the specified 1- to 2-hour time frame, including a 50% (30–60 minutes) buffer. This result clearly shows the potential difficulty of integrating any protocol into the typically hectic ICU environment.

Similarly, Rood et al.⁷ investigated the compliance differences between paper-based and computerized versions of the same protocol. In this study, timing compliance to a 2-minute plus 5% buffer on 1- to 2-hourly measurements was 35% for paper-based and 40% for computerized versions, indicating that, as reported elsewhere, computerized order systems can offer increased efficiency. Interestingly, Rood and colleagues⁷ also investigated insulin dose compliance and found that there was also a difference (64% for paper-based vs 77% for computerized version) between protocols, which would have had a flow-on effect in terms of performance.

The various degrees of compliance with TREAT recommendations compared to the SPRINT protocol may also be accounted for in at least two related ways. The two systems target different groups of health care personnel. TREAT gives recommendations to medical doctors, whereas SPRINT is executed by nurses. A willingness to comply with rigid clinical recommendations may be inversely correlated with the level of clinical training and skill, as well as the different demands for time and workflow on these two groups. In particular, more rigidly scheduled nursing staff monitoring a patient amidst other duties and interruptions¹⁹ may be more willing to comply with recommendations and have less time to second guess them.

TREAT and SPRINT also utilize different modes of knowledge representation. TREAT is computer based, whereas SPRINT was derived from a computerized protocol, but is implemented using cognitive artifacts—the paper-based look-up tables are designed as covered wheels to minimize user error and to increase ease of use of the interface. Computer designs, by virtue, often abstract away implementation details, thereby removing cognitive load from practitioners. In contrast, cognitive artifacts stimulate and support cognition in clinical work activities.^43,44 For example, the SPRINT wheels are used in an algorithm that involves cognitive processing of nursing staff at a number of decision points in execution of the SPRINT protocol. A fully computerized protocol may remove this cognitive load by taking control over each decision point and simply presenting an end action, but without explanation or intermediate steps that provide transparency.

While removal of cognitive load from practitioners to computers may be experienced as a relief by some, it may also be counterproductive to compliance with system recommendations if these are perceived as the outcome of a magical black box, working in mysterious ways that are beyond the comprehension of even highly skilled clinicians. Thus, the designers of computer-based medical decision support systems face the challenge of developing computer representations that balance the need for transparency, recognition, and making sense of processes underlying system recommendations, with the contrasting need to minimize clinical load. Specifically, balancing transparency and clinical burden in Figure 1 represents a potentially nontrivial task for computerized decision support.

With respect to the TREAT case study it is possible to hypothesize that the introduction of more effective objective feedback metrics would improve the compliance. Similarly, TREAT shows the difficulty that can be encountered if objective feedback is not available or difficult to supply with confidence, an issue that is less directly relevant to the glycemic control case. More specifically, the response to glycemic input is relatively rapid (30–90 minutes) and measured easily (blood glucose measurement). In contrast, the response to antibiotic therapy is not measured easily or directly and occurs potentially over several days. Hence, objective feedback is lost. In addition, there is less “immediacy,” which may also affect compliance and outcome, illustrating a fundamental difference potentially in designing protocols for insulin versus those for longer acting therapeutics such as antibiotics. However, it is important to recollect that, per some of the results cited here, even immediate therapeutics such as insulin can suffer from lack of compliance, so while a fast objective feedback may make the design easier, it does not necessarily free the protocol designer from having to consider these issues.

It should not be taken that paper-based protocols are better than computerized protocols given this difference in abstraction. Both may suffer from compliance or related issues in this framework. A more relevant conclusion would be that the design of a computer interface should better reflect the users it is intended for and their needs in terms of transparency and effort. It may therefore require more thought and effort in design, to which the field of human computer interaction is devoted.

Given the time periods for response, computers may offer some advantages in certain instances. In particular, computerized glycemic control methods, such as those of Wong et al.,³⁸ from which SPRINT was created, would not be feasible without a computational platform. However, glycemic control still typically provides 3- to 10-minute windows for action, at a minimum, which are suitable for either type of protocol. Hence, there is no preference for any protocol in media used, but rather that how it is designed to be used best reflects the users needs and those issues outlined in the proposed framework.

Finally, both TREAT and SPRINT were not integrated with computerized medical records. For TREAT, this integration would save time on data entry for users³⁶ and thus reduce burden, while also providing, and gaining access to, more data to create objective feedback metrics. For a computerized form of SPRINT it can be hypothesized that this type of integration would potentially reduce clinical burden and provide greater transparency in the context of the patient's ongoing care by providing immediate access to a wider range of data on patient condition and evolution.

In terms of using the framework of Figure 1 or the elements within it, it is not possible to draw specific or quantitative conclusions. A great deal of work in the arena of interface design, in general, has not yet led to such a result. In this sense it is better suited to guiding design and post-hoc assessment. A similar post-hoc approach followed by educational follow-up was used by Berenholtz et al.²⁹ to improve ventilator therapy. In particular, the proposed framework would imply that not meeting any single criteria (e.g., education) would likely lead to a loss of performance. However, how much a loss would depend very much on several other factors, including the clinical staff capability of those using it. Thus, one cannot use it to design for a certain level of acceptability or performance, where most likely better performance is related to meeting all of the criteria as much as possible.

Speculatively, that last point may indicate that any protocol will require some measure of continuous update in education or assessment to ensure ongoing compliance and relevance. A specific limitation of this study is that it cannot state conclusively or concretely what does or does not make an interface more or less transparent, particularly for a computerized protocol. What is obvious to an experienced user may be confusing to a less experienced person. Thus, each would require a different interface with different assumptions, similar to the prior discussion on the type of educational background of the user and their expectations. As a result, it might at least be concluded that such interfaces may well need to be designed to evolve with user preferences over time.

Summary

Comparing SPRINT and TREAT, as well as other results in the area of computerized decision support in medicine, the same common themes emerge that are being seen by others in the field of glycemic control in critical care.^6,7,27 Specifically, that compliance is a key element to obtaining the best clinical outcome that a given protocol can provide. The issues that affect compliance the most are quite often unrelated to the patient or treatment, but are a function of the protocol design and its ability to integrate into a given clinical setting.

A framework for examining these issues in design is therefore proposed in Figure 1. It is designed as (at least) an opening statement in a discussion of how to better design protocols that are clinically and economically effective beyond well-controlled pilot studies. As such its use would be intended for two phases: (1) design and guiding the development and testing of interfaces in particular and (2) post-hoc assessment by guiding the surveying and objective data collected.

Finally, human factors and interface design are extremely broad research areas with a great deal of literature that cannot be summarized here. In addition, given the many variables of user, user education, geographical structure of the ICU, type of therapy being automated, and other factors described, any specific answers may well reside at the level of an individual protocol. However, the framework provided gives a foundation from which important questions can be asked in designing or implementing any protocol. Hence, the main result of this work may well be questions for analysis rather than a conclusive recipe that overarches many or all glycemic control (or other) protocol implementations.

Abbreviations

ICU

intensive care unit

IIT

intensive insulin therapy

SPRINT

specialized relative insulin and nutrition tables