Reasons for declining computerized insulin protocol recommendations: application of a framework

Abstract

Clinical decision support systems (CDS) can interpret detailed treatment protocols for ICU care providers. In open loop systems, clinicians can decline protocol recommendations. We capture their reasons for declining as part of ongoing, iterative protocol validation and refinement processes. Even though our protocol was well-accepted by clinicians overall, noncompliance patterns revealed potential protocol improvement targets, and suggested ways to reduce barriers impeding software use. We applied Rita Kukafka and colleagues' (2003) IT implementation framework to identify and categorize reasons documented by ICU nurses when declining recommendations from an insulin-titration protocol. Two methods were used to operationalize the framework: reasons for declining recommendations from actual software use, and a nurse questionnaire. Applying the framework exposed limitations of our data sources, and suggested ways to address those limitations; and facilitated our analyses and interpretations.

INTRODUCTION

Guidelines and protocols can increase the consistency of clinical decision making and link decisions with evidence-based care [1] and expert opinion about best practices [2]. However, clinicians exhibit variable practice patterns [3, 4] and ICU clinicians will inevitably disagree with some protocol recommendations. Clinician compliance varies widely and depends on the specific protocol, clinician, and the implementation location [2, 5]. Increasing the detail in a protocol, to increase the consistency with which it is applied across multiple clinicians and patients, increases protocol complexity and may make implementation more difficult [6]. Computerization within a clinical decision support system (CDS) can promote protocol compliance [7-11]. However, computer systems that fail to adequately support the decision-makers may degrade rather than enhance decision making [12].

In clinical research studies, the intervention of interest often involves a set of clinician behaviors. Such behaviors must be studied under clinical research and care conditions [13, 14]. Computerized decision support systems will not be optimally effective unless they are accepted and monitored at the time and point of care [15]. Examining a computer decision support system in a clinical setting can reveal issues that may not have been apparent during development [16, 17].

We describe an approach for decision support system validation and refinement, based on a formal, framework-guided technique. We used this approach to evaluate a computerized protocol for managing blood sugar in ICU patients (eProtocol-insulin). The patient's blood glucose is evaluated on average every 2 hours and ICU nurses accept or decline each recommended titration of IV insulin dosage. Although compliance with recommendations exceeded 90%, we recognized that high overall compliance could mask noncompliance with specific portions of the protocol [18].

Frameworks and models provide structure, terminology, and a perspective for evaluating and integrating research results. We wanted to apply an existing, published framework to our analysis rather than creating a framework de novo. Because clinician noncompliance appears to be multifactorial [19], we needed an adaptable framework that accommodated multiple simultaneous factors at different levels. After reviewing relevant published frameworks, we chose the Kukafka et al. information technology (IT) implementation framework [20], which met our criteria for adaptability and multifactorial approach, and included clinician behavior and other concepts we believed would be important for our analysis.

Our initial literature review revealed two major categories of factors that influence overall clinician compliance: those that influence clinician use of computers, and those that affect clinician compliance with guidelines (regardless of format). These factors are encompassed within Kukafka's framework [20], as is the protocol (computer system) and implementation location. Kukafka and colleagues described their framework at the level of overall IT system implementation, but the extent to which the framework would apply to the detailed level of clinician noncompliance with individual computerized protocol recommendations was not known.

The perspective of nurses regarding barriers to compliance, especially the perspective of ICU nurses, was under-represented in the literature. We describe the application of the Kukafka et al. IT implementation framework to our analysis of ICU nurses' reasons for declining recommendations from a computerized insulin protocol (eProtocol-insulin). Two data sources were used to operationalize the framework: reasons for declining recommendations from actual software use, and a nurse questionnaire. The questionnaire evaluated individual nurse's attitudes and perceptions about their likely (intended) behavior. The software use data reflect ICU nurse perspectives captured at the point of care and at the time of decision making, and their actual behavior while using the software in a clinical setting.

Framework

Kukafka's information technology (IT) implementation framework [20] integrates multiple behavioral and system theories, and includes technical and human factors that influence IT use. Kukafka et al. recognized their framework might need to be modified because determinants of clinician behavior are context specific. We modified their IT implementation framework by adding detailed constructs applicable to our analysis.

The IT implementation framework includes five assessment phases: (1) organizational needs, (2) IT system assessment, (3) behavior and environment assessment, (4) an educational and organizational assessment of behavioral factors and (5) IT use-inducing strategies. The IT implementation framework organizational needs (phase 1) assessment is an evaluation of the users perceptions about the organization's needs and goals [20]. We operationalized the organizational needs assessment as perceived organizational support for use of clinical decision support within the ICU setting. Our study was conducted in an organization that is known for clinical decision support (LDS Hospital in Salt Lake City). Perceived organizational support was not anticipated to be a barrier and was evaluated with a single question in the nurse questionnaire.

The IT system (phase 2) assessment is specific to the system and context of use. Needed but absent functions, software errors, and user interface issues are examples of factors explored in the IT system assessment [20]. A computerized protocol is a type of clinical decision support system (CDS) [10]. A CDS typically contains a user interface, a set of input patient data, and a knowledge base [21-23].The knowledge base contains the rules and logic statements that encapsulate the knowledge required for clinician decisions—in our case, rules and mathematical algorithms for titrating intravenous insulin infusion rate to maintain a desired blood glucose concentration in intensive care unit patients. The knowledge base rules generate a patient-specific recommendation based on individual patient data [24]. Like most other CDS, eProtocol-insulin is an open-loop system. In open-loop systems the clinician assesses, and accepts or declines, each computerized protocol recommendation before initiating an action [21].

eProtocol-insulin is a stand-alone computerized protocol, with a single computer screen interface, for titrating intravenous insulin in ICU patients. The bedside nurse inputs only the patient's blood glucose value, the intravenous insulin infusion rate, and indicates if the patient is being fed. The software then computes the recommended insulin infusion rate based on the protocol rules and mathematical algorithms. The nurse accepts or declines the recommendation. If the nurse declines, they enter their reason for declining the recommendation. The nurse can choose one or more reasons for decline from a drop-down list on the eProtocol-insulin interface, or can type a free text response. We operationalized the IT system assessment as computer hardware difficulties, software programming errors (including errors in coding the knowledge base rules), missing functions or functions the nurse does not know how to use, interface difficulties, and data inputs including patient data and the nurse's reasons for decline. Questionnaire items that supplemented the IT system assessment asked about the importance of proposed additional software functions

In the IT implementation framework, the behavioral and environmental assessment (phase 3) includes evaluating individual and collective behaviors related to use of the IT system, and environmental factors such as social norms and communication patterns [20]. The behavior of interest in this study was bedside clinician noncompliance with recommendations from eProtocol-insulin, operationalized as accepting or declining software recommendations. The environmental assessment was operationalized as perceived peer attitudes toward using a computerized decision support system. Because our study site had a long history of using computerized decision support, environmental factors were not anticipated to be a barrier to compliance; perceived peer attitudes were assessed via a single question on the questionnaire.

The educational and organizational assessment of behavioral factors (phase 4) evaluates factors that can influence behavior. Predisposing factors are pre-existing psychosocial barriers [20]. Based on our literature review, predisposing factors we felt might be important for this study included perceived usefulness of the software, perceived ease of use, nurse attitudes, and nursing expertise. These factors were assessed via the questionnaire, and were potential categories when we classified declined recommendations from actual software use. Based on our experience with decision support systems, we also felt that perceived knowledge base (KB) accuracy could be a predisposing factor. This reflects clinicians' perceptions about the appropriateness of rules, rather than the accuracy with which a rule was programmed. Actual inaccuracies in programming are system errors. However, a rule that was programmed correctly could still be perceived as inaccurate if the nurse disagrees with a particular rule or does not understand how the rules led to a recommendation. The questionnaire asked about perceived KB accuracy in general terms (e.g., would the nurse decline an instruction if they did not understand the logic/rule), whereas the content analysis looked at perceived KB accuracy in the context of specific recommendations.

Enabling factors are conditions external to the user, such as resources needed to use the system [20]. Lack of those resources can be barriers to using the system. For this study, we operationalized access to the computer system, access to the patient, and educational materials (help files and tutorials) as enabling factors, and were assessed in the data from software use and in the nurse survey. Reinforcing factors are subsequent to the behavior and provide incentives to repeat behaviors [20]. Reinforcing factors we thought could be pertinent to our analysis were feedback to the users, and potential increases in nurse knowledge. These factors were assessed in questionnaire items that asked about proposed additional software functions.

Assessment of IT use-inducing strategies is the fifth evaluation within the IT implementation framework. Education and organizational policy are among strategic areas to consider when planning interventions to influence behavior [20]. Because this was primarily a retrospective analysis, and compliance with the protocol was high, we did not formally assess use-inducing strategies in our study.

METHODS

Study Design and Sample

The study reported here was a portion of a dissertation study [25]. This was a descriptive-exploratory study in a tertiary-care hospital with extensive computerized clinical decision-support experience. Data were acquired during usual care of adult ICU patients' blood sugar with continuous intravenous insulin infusions. Bedside nurses used eProtocol-insulin for this purpose. After obtaining IRB approval, data were extracted from three databases: the eProtocol-insulin database, a quality improvement database, and the LDS hospital electronic medical record. Data were de-identified and evaluated for consistency, completeness, and duplication. We conducted a content analysis of the reasons that the nurses chose and recorded in eProtocol-insulin at the time they declined eProtocol-insulin recommendations. As an exploratory study, breadth of data was essential to provide a comprehensive picture [13]. This study used the data from all patients who were managed with the eProtocol-insulin software from January 2004 through December 2005.

Nurses were not individually identified in the data from use of the software. In addition, the IT implementation framework suggested factors that might be important, but that were not likely to be found in the software use data. Therefore, we administered a questionnaire to nurses working in this ICU. The questionnaire was conducted as a human factors usability study, because it provided descriptive data about the nurses' interaction with the computer interface. Our goal was to obtain questionnaires from at least 10 nurses. The study was described and questionnaires distributed at ICU staff meetings; questionnaires were available for 2 weeks. We asked that nurses who had used the eProtocol-insulin software complete the questionnaire. Questionnaires were returned from 14 nurses, which is approximately 28% of the total nursing staff on this unit. Individual nurses are not identified during software use, so we could not verify how many of the total nursing staff actually used the software. Because there has been little or no nursing staff turnover in this ICU during and after completion of this study, we felt that the nurses who responded to the questionnaire adequately represented the nurses who listed reasons to decline recommendations in the data set.

Questionnaire

The questionnaire was developed for a dissertation study [25] and was specific to eProtocol-insulin. We focus our report here on the findings pertinent to our application of the IT implementation framework. The background information section contained 9 items addressing respondent demographic characteristics, nursing experience, and computer experience. Nursing experience included years of ICU experience, educational preparation, and a question corresponding to Benner's levels of nursing expertise [26]. Level of computer experience included length of time that the nurse had used electronic charting, experience with other electronic decision support systems, and a self evaluation of the level of computer experience that rated experience on a Likert scale in which 0 represented no computer use and 8 represented an expert user [22]. The literature review noted the importance of organizational issues and the environment [27, 28] leading to 2 questions about organizational support and peer attitudes.

Subsequent sections contained 9 items that evaluated perceptions about computerized decision support in general, and 11 items that were specific to perceptions about eProtocol-insulin. For analysis, negatively worded items were recoded so that all items were oriented with higher scores reflecting positive perceptions. Individual nurse attitudes, perceived usefulness, and perceived ease of use questions were derived from the literature [19, 27, 29-32]. A 5 point Likert scale in which 1 = strongly disagree/not useful and 5 = strongly agree/useful was used to rate usefulness. The questionnaire asked about the nurses' perceptions of the accuracy of the eProtocol-insulin knowledge base, ranking 5 statements on a scale where 1 = strongly disagree and 5 = strongly agree. Nurses were asked about the influence of eProtocol-insulin on their workload, using a scale where 1 indicated negative impact on workload and 5 indicated not an influence on workload.

A list of 18 potential reasons to decline was derived from the eProtocol-insulin software pick-list, and from questions that nurses asked while using the software. The first 11 items ranked potential reasons to decline on a Likert scale in which 1 = strongly disagree with the item as a decline reason and 5 = strongly agree. For the other 7 items, nurses were asked to indicate which of listed clinical situations would increase the likelihood that recommendations would be declined. An additional open-ended question asked about other reasons the nurse might decline protocol recommendations.

The final section of the questionnaire evaluated potential software enhancements. Three scenarios based on actual protocol recommendations explored mechanisms of explaining recommendations (no explanation, display of the rules, and a text explanation). A list of possible new features was based on suggestions made by ICU nurses. Nurses were asked to indicate the extent to which they felt a feature might be important in promoting protocol compliance, ranking items from 1 = not at all important to 5 = critically important. The questionnaire was reviewed by the dissertation committee [25] and by the ICU nurse manager, the ICU nurse educator, and an ICU physician, prior to use.

Content Analysis

Our initial analysis compared the free-text reasons to decline recommendations to the items on the eProtocol-insulin reasons-to-decline pick-list. However, we found that nurses seldom selected items from the pick-list (only 3 out of 2077 declines). They typed free-text responses instead, even when their response was conceptually similar to a pick-list choice.

Therefore, we conducted a formal content analysis of the text entered by ICU nurses as reasons to decline recommendations. Content analysis is a common exploratory process that can be used to ascertain meaning from textual data [33, 34]. Content analysis reduces the total content of unstructured data to a set of categories or themes that represent similar content [33, 35]. We transcribed the free text responses to a consistent set of phrases. For example, transport out of the ICU, patient in surgery, and similar phrases were transcribed as lack of access to patient. A team consisting of a nurse informaticist, an ICU physician, an ICU nurse study coordinator, and a physician researcher grouped phrases that were conceptually similar, iteratively refining and consolidating similar phrases, resulting in a set of 12 categories. Definitions and examples were refined to explain the categories, and then the declined recommendations were coded using the 12 categories. A second rater assigned categories to 208 (10%) of the declined recommendations and inter-rater reliability was calculated [36].

We calculated frequencies for each category. We mapped the final list of “reasons to decline” categories to concepts in the IT implementation framework, and calculated cumulative frequencies for the framework concepts.

RESULTS

The framework was operationalized through two methods: a nurse questionnaire and data from software use. Fourteen nurses responded to the questionnaire (Table 1). The nurses were predominantly female (78.6%) with bachelor's degrees (78.6%), and clinically experienced, averaging 12.7 years of ICU experience. None of the nurses were beginners. The nurses appeared comfortable with computers, with an average of 5.2 out of 8 for self-ranked computer experience. Most of the nurses (85.7%) had used electronic charting for longer than 2 years, and more than one third (35.7%) of the nurses had used other computerized protocols in addition to eProtocol-insulin.

Table 1.

Characteristics of 14 nurses who responded to the questionnaire.

Characteristic	Number (%)
Gender: Male	3 (21.4%)
Female	11 (78.6%)
Education: Diploma/Associate degree	2 (14.3%)
Bachelor's degree	11 (78.6%)
Graduate degree	1 (7.1%)
Time using electronic charting
Less than 1 year	0
1 to 2 years	2 (14.3%)
More than 2 years	12 (85.7%)
Nursing expertise
Novice/Advanced beginner	0
Competent/increased skills	6 (42.9%)
Proficient	3 (21.4%)
Expert	5 (35.7%)
Characteristic	Mean (SD)
Age	38.3 (10.2)
ICU experience (Years)	12.7 (7.4)
Level of computer experience (0 = none to 8 = expert)	5.2

After cleaning, the retrospective data set contained 39,640 recommendations for blood glucose management in 830 adult ICU patients. Bedside nurses declined 2077 recommendations (5.2% of 39,640). The nurses declined at least 1 recommendation in 548 patients (66% of 830). Demographic characteristics of the patients in our sample were similar to the general population of the study ICU (Table 2).

Table 2.

Patient characteristics

Characteristic	No (%)
Sample size	830
Gender
Male	487 (59.5)
Female	331 (40.5)
Ethnicity/Race
Asian	5 (0.7)
Black	13 (1.9)
Hispanic	40 (5.8)
Pacific Islander	2 (0.3)
Native American/ American Indian	8 (1.2)
Other	2 (0.3)
White	625 (89.9)
Diabetes diagnosis present	230 (27.7)
Characteristic	Mean (SD)
Age (years)	60 (17.8)
Number of recommendations per patient	47.8 (56.9)
Total protocol use (hours per patient)	109.2 (128.1)

Table 3 lists the results of the content analysis. Only 11 of 2077 decline reasons contained even a partial textual match to a software pick-list choice. Inter-rater reliability indicated our content categories were sufficiently well-described for our study purpose. For the 208 (10%) of the declined recommendations compared, simple percent inter-rater reliability agreement (N agree/total N) was 99.5%, and Cohen's kappa as computed by SPSS version 15 software was 99.4%. Sometimes the bedside nurse listed more than one reason to decline a recommendation (2186 reasons were found for 2077 declined recommendations), so percents reported here may sum to more than 100%. We successfully mapped all of the categorized reasons to decline back to the IT implementation framework.

Table 3.

The content analysis results

Framework Concept / Content Analysis Category	Framework Concept		Content Analysis Category
	n	%*	n	%*
Patient Data	1022	49.2%
Patient history or trends			401	19.3%
Non-steady state perturbations			373	18.0%
Incorrect data			248	11.9%
Predisposing Factors	567	27.3%
Perceived KB accuracy			566	27.3%
Attitudes			1	0.05%
Enabling Factors	255	12.3%
Unable to comply			193	9.3%
MD orders			51	2.5%
Nurse workload			5	0.2%
Medication errors			4	0.2%
Patient or family request			2	0.1%
IT System (software) issues	97	4.7%	97	4.7%
Reason for decline
No reason given	245	11.8%	245	11.8%

^*Percentages based on number of recommendations (n = 2077)

Framework assessment phase 1: Organizational support

The framework concept Organizational support was not found in the text of the reasons for declining instructions. In the questionnaire all nurses indicated that the organization supported clinical computerized decision support use “to some extent” (30%) or “a great deal” (70%).

Framework assessment phase 2: IT system assessment

The IT system assessment was subdivided into: IT system (software), IT system (patient data), and IT system (reasons to decline recommendations). The IT System (software) concept included interface issues, software errors, and functions that the nurse had difficulty using[20]. Issues with the software accounted for 99 declined recommendations (4.7% of 2077). Nurses reasons for declining recommendations included problems with the software or computer (e.g., “had a computer glitz”), process errors (e.g., “I forgot to accept the previous recommendation”), and incorrect timing (e.g., “glucose checked early”).

The most common reasons for declining recommendations were related to patient data input by nurses into the system to get a recommendation. The concept IT System (patient data) accounted for almost half (49.2%) of the 2077 declined recommendations. The IT System (patient data) concept included concerns over the possibility of increased hypoglycemia risk and encompassed the content analysis categories: patient history or trends (19.3%), non-steady state perturbations (18%), and incorrect data (11.9%). Reasons such as “glucose has been falling” were categorized as patient history or trends. Non-steady state perturbations are temporary or short-term influences on glucose equilibrium such as having recently administered an antibiotic mixed in a dextrose containing solution (e.g., D5W) or changes in the rate of enteral feeding. Co-interventions such as administration of epinephrine, and co-morbidities such as hypothermia or agitation, were also categorized as non-steady state perturbations. Incorrect data included both errors in the current data (e.g., “oops, wrong glucose”), and the nurses' assessment that previous data were incorrect (e.g., “enteral feeding has never been given”).

No reason was entered by the nurse for 245 declined recommendations (11.8%). We mapped the “No reason” category to the IT System (reason to decline) concept.

Framework assessment phase 3: behavioral and environmental assessment

The behavioral assessment included the behavior of interest for this study (noncompliance with recommendations) and the environmental assessment, which focused on perceived peer attitudes toward CDS usage. Like organizational factors, peer attitudes were not chosen by nurses as reasons for declining eProtocol-insulin recommendations. In the questionnaire, the nurses indicated that their colleagues were “accepting of” (79%) or “enthusiastic about” (21%) the use of computerized decision support.

Framework assessment phase 4: Educational and organizational assessment of behavioral factors

The educational and organizational assessment of behavioral factors included predisposing factors, enabling factors, and reinforcing factors. Predisposing factors were the second most common reason for declined recommendations (27.3% of 2077 declines). Predisposing factors encompass nurse characteristics and internal factors such as attitudes and perceptions that can influence the nurse's behavior [20]. All but one of the declined recommendations mapped to predisposing factors were issues with perceived knowledge base accuracy (clinician opinion or disagreement with the recommendation). Perceived knowledge base accuracy issues included unspecified opinion (e.g., “will wait and recheck”), disagreement with the dose change increments (e.g., “piddly change”), disagreement with minimum/maximum thresholds (e.g., turning off drip instead of running at the minimum threshold rate), disagreements with the wording of the recommendation (e.g., “I can't ‘decrease’ to the same rate”), and lack of understanding (the nurse indicated they did not understand why the recommendation was given).

The questionnaire revealed generally positive attitudes toward use of computers and decision support tools. The overall mean score for attitudes toward computers and computer-based decision support was 3.85 out of 5. The nurses agreed with 3 statements evaluating the software's ease of use (the software was seen as easy to use). The nurses felt that decision support tools in general were useful (mean = 4.1 out of 5) and somewhat improve their ability to make decisions (mean = 3.5 out of 5). The eProtocol-insulin software was also ranked as fairly useful in two questions (3.7 and 4.1 out of 5). A question about trust (“I will accept a recommendation from the software if the instruction appears safe, even if I do not understand the rule”) had a mean score of 3.4 out of 5 suggesting that the nurses trust decision support software but remain vigilant regarding potential problems.

The nurses agreed that the recommendations from the software are appropriate (mean ranking = 3.7) but that recommendations should be declined if the nurse does not agree with the logic for insulin dosing (mean ranking = 3.4), or if the nurse agrees to give dextrose but disagrees with the dextrose dose (mean ranking = 3.6). They mildly disagreed with statements that recommendations should be declined for problems with the wording (text) of the recommendation (mean ranking = 2.6) or when the nurse does not understand the logic (mean ranking = 2.6).

Enabling factors accounted for 255 declined recommendations (12.3% of 2077 declines). Enabling factors influence behavior by facilitating or inhibiting the nurse's ability to carry out a behavioral intent. The most common barrier (9.3% of 2077 declines) we identified was when the nurse was unable to comply with the recommendation, such as because of lack of access to the patient or lack of IV access. Physician orders, nurse workload, and medication errors, which might have been expected to be important reasons for noncompliance, collectively accounted for less than 3% of the 2077 declined recommendations. In the questionnaire, nurses were neutral regarding agreement with two statements that recommendations should be declined because of access to the patient (mean aggregated score was 2.8, SD = 0.7). Slightly more than half of the nurses (8 of 14, or 57%) felt that recommendations would be declined if there were equipment problems such as the IV line not working. Overall, use of the software did not appear to be overly burdensome (mean score = 3.4, SD = .9); the nurses indicated that use of eProtocol-insulin is a routine part of their patient care (mean = 4.0, SD = 03). None of the nurses indicated that recommendations would be declined because the nurse is too busy.

Nurses did not identify Reinforcing factors as reasons for declining eProtocol-insulin recommendations during software use. The nurses provided mixed questionnaire responses about whether the importance of providing background information (potential increase in knowledge) was important as a potential method for improving compliance. Some nurses ranked each of these items as not important at all and others ranked them as very important. The nurses indicated that an explanation of the protocol goals was somewhat important (mean rank 2.9), a general explanation of the protocol logic and algorithms was somewhat important (mean rank 2.9), and a summary of literature regarding the need for glucose control was “a little” important (mean rank 2.3).

User feedback in the form of explanations was explored in three scenarios. The scenarios indicated that explanation of rules might influence the nurse's decision to accept or decline a recommendation. The first scenario presented patient data resulting in a potentially confusing, but correct, instruction with no explanation. Half (54%) of the nurses indicated they would decline the instruction. The second scenario presented the same recommendation plus displayed the knowledge base logic frame. The nurses were evenly divided in their agreement whether viewing the logic frame would increase the likelihood of accepting the recommendation (average rank 2.9). The third scenario provided the same recommendation plus a simple text explanation of the logic. The nurses more strongly agreed that a text explanation would increase their likelihood of accepting the recommendation (average rank 3.8).

Framework assessment phase 5: Use-inducing strategies

Use-inducing strategies were not formally assessed in this study.

DISCUSSION

Clinical decision support systems do not simply capture knowledge and represent it in a static system; the software and knowledge base must be continuously evaluated and refined [4]. Factors that the ICU nurses appeared to perceive as important for decision making with eProtocol-insulin emerged from formal evaluation guided by Kukafka's IT implementation framework [20]. We acquired a better understanding of noncompliance behavior. Because the data were collected at the point of care, at the time the bedside nurse declined the eProtocol-insulin recommendation, we were able to gather ICU nurses' perspectives about clinical use of the software in their own words. While this is a strength, our study has several limitations. When the nurse gave no reason for declining, we lost an opportunity to evaluate the software from the nurse's perspective, and lost an opportunity to improve the computerized protocol. We think the retrospective nature of our analysis of this large cohort dataset is only a minor limitation to our study. We did not identify individual nurses and this limited our retrospective data analysis. The questionnaire, although based on concepts from the literature review and the theoretical framework, was specific to the software used in this study and had no established reliability or validity.

An important study limitation is our use of a single study site that is familiar with the software development process. This site has a long history of using computerized decision support tools. The characteristics of our study site explain, in part, why IT implementation framework concepts such as organizational factors, user attitudes, software usability, and reinforcing factors were not identified by any of our nurses as reasons for declining eProtocol-insulin recommendations. This ICU is accustomed to delivering care via nurse-driven protocols, and was the site of initial eProtocol-insulin development and refinement. Other sites may require time for clinicians to adjust to making standardized decisions [42]. Although we believe them to be internally valid, our findings may not be generalizable to other sites of eProtocol-insulin use, or to other computerized protocols. The analytical method we used, however, appears to be generalizable.

The study reported here illustrates how the IT implementation framework was applied to identify and categorize reasons documented by ICU nurses when declining individual computerized protocol recommendations. The framework was described as a guide for planning IT system implementations, and for planning multi-level interventions to enhance IT system use. The framework was readily applied as a post-implementation, evaluation framework at the micro-level of our analysis (individual instructions within a computerized protocol).

The IT implementation framework [20] was useful for analyzing the questionnaire data. The questionnaire allowed individual nurses to articulate their attitudes and perspectives. By using the framework to organize structure our evaluation, questionnaire responses could be compared to responses from the content analysis and interpreted.

We found good conceptual matches between framework concepts and content analysis categories. However, content analysis is context sensitive. Like other means of analyzing qualitative data the analysis is inevitably a process of selection and reduction. Text can assume different meanings once relationships between concepts are taken into account [33, 34]. Theoretical frameworks interrelate concepts, provide symbols and labels, and describe the circumstances under which a process will occur [13, 14, 37, 38]. By grounding our content analysis in the IT implementation framework, we came to our interpretation about ICU nurses' reasons for declining instructions. Had we chosen a different framework, we might have a different interpretation.

The IT implementation framework [20] also provided a means for evaluating our study methods. By using two methods for operationalizing the framework (questionnaire and content analysis), we hoped to complement each approach's strengths and weaknesses. This was only partially successful. Without addressing the full range of factors, our conclusions are limited and strategies we develop to address barriers to compliance risk being ineffective.

Not surprisingly, organizational support was not listed by the nurses in the text of any reasons for declining recommendations. Our phase 1 assessment was only weakly evaluated in the questionnaire. We evaluated organizational support with a single question, because we were conducting our evaluation in an environment known to be supportive of CDS. Our phase 2 assessment was primarily focused on the IT system. Most of our findings about the CDS came from the software usage data, with some supplemental data in the questionnaire. The phase 3 assessment focused on a single behavior of interest (noncompliance with protocol recommendations). Like organizational support, peer attitudes (our operationalization of the environmental assessment) was felt to be already known, and was assessed with only a single question on the survey.

Our phase 4 assessment best demonstrates the complementary nature of using two approaches, and was well supported by the content analysis and several items in the questionnaire. The software usage data demonstrated ICU nurses' actual behavior, at the point of care and time of decision making, but lacked the ability to group findings by nurse. The questionnaire allowed individual nurse perspectives to be articulated, but reflected intent and attitudes rather than actual behaviors. We did not formally assess IT use-inducing strategies (phase 5), because this was a retrospective analysis, overall compliance was high, and we found no subsets where high overall compliance masked noncompliance with a subset of the protocol.

We cannot infer, however, that compliance behavior will be the same at other sites or with other protocols. IT acceptance and use involves multiple factors at the organizational, group, and individual level [15, 20, 39]. Approaches for improving acceptance and use must be strategically designed to affect the determinants of behavior in each situation [20]. The IT implementation framework organizes disparate behavioral theories, which in turn can allow us identify approaches applicable to each identified barrier to compliance.

Although we conducted a limited evaluation a single site, our evaluation suggested potential areas where we could improve the eProtocol-insulin computerized protocol:

1. Nurses noted patient data trends extending over longer times than accommodated by the protocol. The protocol uses the current and previous patient data, typically spanning a 2 hour interval, whereas some nurses described trends extending across 12 hour intervals. We are conducting statistical modeling evaluations to assess trends over time.

2. Co-interventions such as glucose-containing IV fluid boluses with antibiotic administration, and co-morbidities such as agitation, restlessness, or hypothermia, can temporarily disturb glucose metabolism. We are collecting additional data to evaluate the potential impact of rules addressing these short-term influences.

3. More explicit rules regarding feeding may be needed. More than one third of the “incorrect data” reasons to decline involved incorrectly categorized feeding data.

Usability was not cited by ICU nurses as a barrier in our study. Nevertheless, potential user interface improvements were identified. Clinician opinion (disagreement with recommendations) was the second most common reason for declining recommendations. We realized that in some cases, enhanced explanatory functions were needed to clarify rules that the nurses did not understand. In addition, several nurses indicated to us on follow up that they didn't realize the software had a pick-list for reasons to decline, so the pick list was changed from a drop-down to a list box that displays the choices without requiring extra mouse clicks. The ability to also enter free-text was retained.

Although they were not found as barriers to compliance in our current study, the IT implementation framework [20]indicates that organizational needs and intrinsic factors within the environment are likely to be key factors that influence behavior. Acceptance of an IT system can vary depending on the magnitude of organizational culture and environmental influence [15, 20]. This is a nurse-driven protocol, so organizational culture factors such as the extent to which a unit allows ICU nurse autonomy could influence acceptance [39].

Preliminary data from protocol use at other sites suggests that compliance will vary by site. We plan to compare patterns of noncompliance between sites and site characteristics (e.g., adult versus pediatric sites and experienced versus naïve sites) in our currently ongoing multicenter study. Although we plan to use similar methods, it will be vital for future evaluations to include a more thorough evaluation of the organization and environment. The questionnaire used in this survey should be revised for future studies. A broader questionnaire should more fully assess IT implementation framework concepts such as organizational culture and environmental variables. Future questionnaires would be strengthened by using validated scales. For example, the questionnaire might be redesigned like the instrument described by Ash [40], who made use of validated scales to assess organizational influences. An instrument measuring Predictors of Use of Computerized Protocols by Clinicians [41, 42] is specific to computerized protocols, and is compatible with Kukafka's IT implementation framework [43], and so could be a valuable addition to future studies.

CONCLUSIONS

Nurses are expected to follow CDS recommendations, but will occasionally decline those recommendations. Some rejection of individual recommendations is expected because patients are complex and can experience events not accommodated by the protocol [44]. However, to reduce unnecessary variability the protocol should be followed unless there is a clear risk to the patient. The nurses' reasons for noncompliance with protocol recommendations, including the interconnectedness of factors at multiple levels, needs to be better understood and managed if such systems are to be optimally used. We plan to analyze noncompliance with eProtocol-insulin recommendations in other clinical sites as part of ongoing multicenter evaluation. The method described in this paper appears applicable, with modification of the questionnaire, for other analyses of CDS recommendations.

Kukafka and colleagues encouraged other investigators to use their framework to guide implementation plans [20]. This analysis demonstrated that the IT implementation framework can also be utilized after implementation, and at a detailed level, to understand ICU nurses' noncompliance with individual recommendations from a computerized insulin protocol. The framework identified factors that could influence compliance behavior, but which were not likely to be found simply by evaluating software data, leading us to complement our content analysis data via a nurse questionnaire. Applying the framework facilitated organizing and interpreting our data. In addition, the framework was useful for evaluating our methods, suggesting methodological aspects that could be improved for future studies. The results guided us to identify areas for potential software refinements, which should in turn improve the eProtocol-insulin protocol and promote ICU nurses' compliance with the computerized protocol recommendations.

Based on our results and analyses, we believe that Kukafka's IT implementation framework has high utility. The framework can be readily applied to evaluate computerized protocol implementations in the ICU setting.

Appendix I: Reengineering Critical Care Clinical Research Investigators

The following persons and institutions are participants in the BAA Roadmap Initiative Reengineering Research in Critical Care: Principal Investigator and Steering Committee Chair–A. Morris; Clinical Coordinating Center – B.T. Thompson, D. Schoenfeld, C. Oldmixon, H. Zheng, C. Bliss; Data and Safety Monitoring Board – H. Wiedemann (Chair), G. Rubenfeld, M. Meade, S. Anand; Clinical Centers - University of Utah – D. Sorenson, H.R. Warner, K. Sward, P. Haug; Primary Childrens Medical Center - E. Hirshberg, G. Larsen; Children's Hospital of Philadelphia - V. Nadkarni, V. Srinivasan, C. Bayer Roth, L. Hutchins; Vanderbilt University - G. B. Bernard, S. Bozeman; Wake Forest University - R.D. Hite, A. Howard; Massachusetts General Hospital - B.T. Thompson, C. Oldmixon; Johns Hopkins University - R. Brower, K. Boucher; LDS Hospital - J. Orme, L. Baumann; University of Virginia - J. Truwit, M. Marshall; University of Virginia Children's Hospital - D. Willson, M. Ball; Yale University - C. Bogue, V. Faustino, I. Lazar; Penn State Children's Hospital, Hershey - N. Thomas, J. Hess; Baystate Medical Center - J. Steingrub, M. Tidswell, L. Kozikowski; Vanderbilt Childrens – N. Patel, T. Shalaby; Childrens Hospital Central California – A. L. Graciano; Hospital for Sick Children – P. Cox, A. Guerguerian; St. Justine Hospital – J. Lacroix, G. Cannizzaro; Dartmouth Hitchcock Medical Center – D. Levin, D. Jarvis; Childrens Hospital Minnesota/St. Paul – Kurachek, L Blumberg; Childrens Hospital Michigan – S. Heidemann; Childrens Hospital Los Angeles – C. Newth, F. Fajardo; Children's Medical Center Dallas – P. Luckett; Baylor Childrens – L. Jefferson; Childrens Hospital Boston – A. Randolph.