Abstract
Computerized provider order entry(CPOE) and other clinical information systems can help reduce medical errors, promote practice standardization, and improve the quality of patient care. However, implementing these systems can result in unintended adverse consequences. Our multidisciplinary team used qualitative methods to gather and analyze data describing unintended adverse consequences related to CPOE adoption and use. Overdependence on technology emerged as one of nine major types we identified. Careful analysis of these data revealed three themes: 1) system downtime can create chaos when there are insufficient backup systems in place, 2) users have false expectations regarding data accuracy and processing, and 3) some clinicians cannot work efficiently without computerized systems. We provide recommendations for mitigating these important issues.
Introduction
Healthcare delivery has become increasingly dependent on information technology to computerize almost all aspects of patient care, as evidenced by the proliferation of systems ranging from billing and accounts management to computerized provider order entry (CPOE) to sophisticated image-guided surgery systems. 1 It is not surprising that introducing these systems into the healthcare environment causes shifts in the way health care providers perform their work. As both personnel and organizations adapt to these new technologies, unintended adverse consequences may emerge. To identify and evaluate the full array of the types of consequences related to CPOE, we held a conference of experts and also conducted fieldwork at five hospitals where CPOE systems had been successfully implemented. Qualitative analysis of our data identified over 300 unintended adverse consequences, which we categorized into nine major types: more/new work for clinicians, unfavorable workflow issues, never-ending system demands, changes in communication patterns and practices, paper persistence, negative emotions, new kinds of errors, changes in the institutional power structure, and overdependence on technology. 2 The last of these types, overdependence on technology, is not often discussed in the healthcare literature. However, this topic raises some significant issues, which are explored here.
Theoretical Background
The theory of technological determinism holds that technology is the prime force in initiating social change, and that the introduction of new technology fundamentally shifts work activities, resulting in transformations of individuals and their social interactions as well as the organizations in which they work. 3 In contrast to this deterministic approach, the theory of social construction of technology posits that technology does not directly shape society; instead, the social context in which the technology is used determines how it is created, diffuses, and becomes part of the organization. 4 Both theories imply that the introduction of technology is associated with significant change; they differ in whether the change is initiated by the technology or the social context in which it is used. Regardless of the theoretical basis for understanding the change, it is reasonable to assume that some degree of dependence on any technological innovation will inevitably occur if the technology provides users with some perceived, relative advantage over whatever system it supersedes. 5 This reliance is expected and necessary if the technology is to realize the potential for which it is designed. This fundamentally differs from overdependence on technology, in which those using technological innovations no longer treat them as flexible tools to support work activities, but instead make incorrect assumptions about how these systems work, and begin to rely on them, without question or skepticism, to manage critical work activities. To answer the question “How does the introduction of CPOE create the potential for overdependence on technology in healthcare organizations?” we conducted a detailed analysis of all references to overdependence on technology in our data. The results are presented here.
Methods
Site and Subject Selection
We selected five hospitals based on their reputations for excellence in their implementations of CPOE. In addition, we selected hospitals representing different organizational types (e.g., tertiary teaching vs. private community) using CPOE systems developed either commercially or “in house,” to assure we did not limit our investigation to a single organizational structure or CPOE system. Selected sites included Wishard Memorial Hospital in Indianapolis, IN, using the locally developed Regenstrief system, Brigham and Women’s and Massachusetts General Hospital in Boston, both using in-house developed systems, The Faulkner Hospital in Boston, MA, using MediTech (Westwood, MA), and Alamance Regional Medical System in Burlington, NC, using Eclipsys (Boca Raton, FL). Our fieldwork consisted of unobtrusive observations of clinicians and other personnel interacting with CPOE systems during their regular work, as well as hour-long, semistructured oral history interviews with hospital administrators, physicians, nurses, pharmacists, lab workers, medical records specialists, information technology leaders, and others suggested to us by local principal investigators. The Institutional Review Boards of Oregon Health & Science University and each study site granted approval for this study.
Data Gathering and Analysis
Our multidisciplinary research team consisted of two physicians, two Ph.D. researchers, a pharmacist, and a nurse. The team visited each of the study sites for 3–4 days, completing a total of approximately 390 hours of observation of 95 clinicians, in addition to 32 interviews. More detailed descriptions of our methods are published elsewhere. 6 We used QSR N6 software to categorize over 300 unintended adverse consequences identified in 1,849 pages of transcribed data. To gain further insight and understanding of this category, we reviewed the 20 instances we categorized as best representing overdependence on technology, then investigated them using axial coding. 7
Results
We identified three themes among the unintended adverse consequences related to overdependence on technology. Because some examples were richly descriptive, a single observation or quote could sometimes be coded in two ways. The first, and largest theme clustered around the problem of practice disruption and loss of patient safety during system unavailability. The second theme highlighted false expectations related to data accuracy and processing, spanning a range from strong skepticism to potentially inappropriate complete trust in computerized information. The third theme involved the perception that clinicians cannot work without CPOE technology, because they cannot keep current on the wealth of clinical knowledge (e.g., drug-drug interactions, clinical guideline recommendations, hospital formulary contents, etc.) required to perform their work effectively. Each of these themes is described below. Where direct quotes from our data were particularly descriptive, they are included in italics.
System Downtime
System unavailability, regardless of primary cause, can “create chaos” for users and organizations. A system is unavailable if users cannot access it, even though it appears to be operational, as when the system interface is working but the back end database is down, when there is insufficient hardware to support workers’ needs, or when the system is so slow that work activities cannot be efficiently completed. Such situations can “…create a real fight at times to get work done, because [people] are always in need of a computer,” and frustrate busy end-users when they incorrectly assume the system is entirely functional, must find workarounds (e.g., leave the unit to find an available terminal) perform redundant work (e.g., document on paper until a computer becomes available), or, in a worst case scenario, elect to skip documenting important clinical information.
Because hospital systems are so complex, and require the careful integration of disparate, specialized software and hardware systems, single component downtime can greatly interrupt workflow. For example, if the laboratory result reporting system becomes unavailable, clinicians must rely on phone calls and pagers to get results, may lose the benefits of display of historical data for trending, and thus may miss important results necessary for optimal clinical care. In addition, a single system component failure in one area can dramatically impact other areas, leading to cascading effects that may not be anticipated prior to their occurrence: “They use a white board screen saver in the ER that keeps track of people in the ER. When the hospital registry system goes down, the registry can’t provide the [patient’s] ID number, it wreaks havoc in the ER.” In this case, failure of the hospital registry system disrupts the functioning of the entire emergency department because no backup system exists to provide temporary registration numbers until the main system can be brought back on line.
Complete system downtime, though generally rare, can have disastrous repercussions for clinicians and institutions. One faculty physician summed it up this way: “It’s funny now. When the computer goes down, we don’t remember how to document on paper.” Even if supplemental paper forms for clinical documentation are available during a downtime event, clinicians can be severely hampered by their inability to access medical data that exist only in electronic from. This loss of important historical data can result in potential for medical error because the clinician must work with incomplete information. In addition, poor preparation for downtime events (e.g., no preparatory training exercises, poorly defined downtime procedures, lack of sufficient paper forms to support documentation until the systems are back up, etc.) can magnify the negative repercussions of downtime, as clinical staff are left without necessary electronic resources and have few practical alternatives for efficiently completing their work.
Data Accuracy
Clinicians report a variety of attitudes regarding data accuracy with the computer. When assessing the accuracy of “read” data (e.g., data stored in and provided by the computer for viewing), opinions range from strong skepticism to complete trust of what is presented. In one system we observed, the original source of data was listed beside any displayed information imported from an outside hospital. One clinician reported that she did not trust any of the data from the outside source, and would not use it until she could verify it. In a lifethreatening situation, especially if the patient, his family or friends were not available to provide medical history, such lack of trust in the data could be a concern. In marked contrast, others reported “… if it’s in a computer it must be accurate and complete...’.” Such an assumption is also concerning; if clinicians do not pause to question electronic data, they may fail to recognize errors in the record.
When discussing “write” data (e.g. information input into the system via keyboard, mouse, etc.), clinicians appear to make assumptions about what the system can and cannot do with data as it is entered, or how it is ultimately processed. For example, “ [some doctors] don’t understand that the free text allergy information cannot be used by the decision support system…[they] have a false sense of security as a result.” This misunderstanding is common; there is a perception that data, once entered into the computer, is fully accessible and useful, regardless of how or where they were entered. Because of this, clinicians often enter data into miscellaneous fields when the correct location is not readily found. Other providers accessing the record at a later date may not think to look for information in these non-standard fields, so the data may be inadvertently lost even though they have been entered.
Finally, there is a strong tendency to assume that processes complete once they have been electronically initiated. For example, it is common to assume that because a medical order has been entered into and processed by the system, the requested action has actually occurred. This is especially true in systems where the medication administration record is not available on-line, so clinicians do not have a single electronic resource to review both the medications that have been ordered and which of them have been administered. “The difference between the medication order list and the medication administration list causes the physicians to have a false sense of security…many physicians assume that all of the medications that have been ordered have been administered.”
Clinicians Cannot Work Without Automated Systems
CPOE systems with embedded clinical decision support (CDS) provide clinicians with a variety of knowledge support tools, such as notification of drug-drug interactions, warnings about allergies, recommendations for clinical guideline compliance, and more. For CDS to be effective, it must be current, context-sensitive, and well integrated into the CPOE system, so that clinicians can rely upon its suggested clinical guidance to supplement knowledge gaps. When these criteria are met, CDS can be enormously effective for supporting clinical practice. However, when clinicians rely on CDS to the exclusion of sound clinical judgment, the potential for errors can increase.
CDS alerts often “fire” when they have not been properly programmed to leverage important and available information. For example, anticoagulants such as heparin are not commonly administered with aspirin. However, this drug combination is often ordered intentionally in the coronary care unit for its heart-protective benefits. In this setting, firing the clinical alert to warn the prescribing clinician is likely unnecessary. But if the alert does fire, and if the clinician relies on the alerting information exclusively, the possibility exists that the clinician will delete one of the two necessary medications for therapy, thus increasing the potential risk to the patient.
Because many CPOE systems we studied have been in place for decades, there are medical students and residents trained at these institutions who have never had to practice medicine without the support of computerized systems. These clinicians may be over dependent on technology—they may not be able to efficiently work without it. Studies have shown that the presence of CDS does not appear to negatively impact learning, and may actually improve it, 8 however, moving from the highly integrated electronic medical record or CPOE system back to a non-automated system can be very difficult for the clinician not familiar with a paper-based clinical record: “We had a resident who was voted the best resident…two years in a row…a wonderful guy. He took a new position at a new hospital…and the head of the medical staff called the residency program director about a month after he got there and said, ‘I just don’t understand, this guy is non-functional.’ He didn’t know how to work in a place that didn’t have order entry or results retrieval. He took almost six months to re-acclimate, [and to] figure out how to order in a different environment.”
Discussion
Reasonable dependence on technology is a desirable outcome of the automation of patient care systems. Clearly, CPOE systems with integrated CDS provide many distinct advantages to the busy clinician who must synthesize and remember an ever-increasing body of clinical knowledge. However, overdependence on technology can arise when computerized clinical systems are not robust (e.g., are slow, partially or completely unavailable), when clinicians begin to trust these systems without question, and when healthcare workers have no exposure or training in non-automated clinical environments. For these reasons, it is imperative that healthcare organizations consider and prepare for potential problems related to technology adoption.
System Downtime
One hundred percent reliable systems do not exist; downtimes, whether planned or unanticipated, are inevitable and costly. Even a system with 99% uptime potentially might be unavailable for 14.6 hours a month, with the actual number of personhours of work lost depending on the number of staff impacted by the downtime. One study suggests that for each minute of unavailable system time, staff must spend 4.5 minutes to complete the work they would have done had the system been available, and to reenter the data once the system comes back on line. 9 The actual costs of this downtime range from about $264 per minute of downtime for a 500-bed hospital, to as much as $1000/minute for a threehospital integrated delivery network (IDN} with 1,400 beds. Over a single year, each 1 percent of downtime could result in an additional $1.4 million in operating costs for the 500 bed hospital and as much as $10 million for the IDN. 9
Given these high costs, organizations should formally measure the performance of their systems, and develop metrics to assess the financial and workflow impacts of system unavailability. At a minimum, organizations should be able to report their uptime percentages, time to recovery for different types of system failures, and overall system usage statistics (e.g., percentage of providers entering orders on-line, average user load, average network speed. etc.). In addition to developing and monitoring these system performance measures, healthcare organizations must also develop and test contingency plans for continued operations during system downtimes, so that system unavailability is minimally disruptive to health care work. Plans should include scheduled downtimes for system maintenance, preparations for short-term and long-term system outage, and rigorous protection of data against loss. In addition, downtime preparations should detail provisions for workable paper backup systems, procedures for operating in the absence of electronic resources, and training for employees. To protect against data loss, robust and reliable backup systems must be in place, and they should be rigorously tested with periodic “downtime drills” to assure that they function as expected.
Data Accuracy
Clinicians cannot reliably trust information accuracy if the organization does not take steps to assure it. In the optimal practice of clinical care, information accuracy should be as ubiquitous and necessary as hand washing. Data imported into systems from any source should be rigorously validated for accuracy. This requires periodic manual checking of results, development of quantifiable benchmarks for data quality, careful attention to variation from established benchmarks, development of procedures to identify and report data inconsistencies, and dissemination of this information to system users and administrators. Data integrity and reliability should be a key organizational goal.
System end-users should be trained in the proper use of clinical applications, including correct data placement on electronic forms, why standardized data entry can improve data reliability and interoperability (and how to use system tools provided for entering these data), and methods for assuring clinical orders are actually completed in a timely fashion. Many of these training areas imply the need for integrated informatics education in medical curricula. In addition, clinicians should be reminded to carefully evaluate and leverage the information provided to them by CDS, so as to make informed and clinically relevant decisions regarding patient care, using this tool as one of many potential sources of information. Finally, clinicians should be given periodic feedback regarding the data they enter in such areas as use of CDS (e.g., how often alerts are ignored), comprehensiveness (e.g., the presence and/or absence of clinically vital data), and quality (e.g., use of nonstandard abbreviations or over reliance on free text entry when standardized entry is available), so that they might improve their documentation.
Clinicians Cannot Work Without Automated Systems
As clinical care becomes more and more dependent on automation, we expect clinicians to rely heavily on technology. However, great care should be taken to educate clinicians that over reliance on technology can be dangerous when it is used to the exclusion of sound clinical judgment. No automated system can yet discern and evaluate all of the subtle physical cues displayed by a patient in a clinical encounter. Despite continued improvements in functionality, proven gains in practice efficiency, and improved access to knowledge sources, these systems are not foolproof. The clinician should therefore utilize his or her education and experience in combination with these tools to provide optimal care.
Regardless of the system in use, healthcare organizations should find ways to measure how well these systems are supporting or enhancing clinical work and quality of care. Organizations should create specific, robust, repeatable, and scalable measures of system performance above and beyond basic return on investment calculations. It is vital to know how often systems are unavailable and to understand and prepare for the impact on staff who must rely on these systems to do their work. Measures must be developed to evaluate data quality, accuracy, and comprehensiveness, particularly with regard to CDS effectiveness and the potential for over reliance on these tools. Finally, organizations should share benchmarking strategies with other institutions so that organizations can develop common, useful, and repeatable methods for assessing system performance
Limitations
The results of this study are limited by the small set of observations we collected reflecting this theme, possibly because this theme may be less of a pressing (and subsequently less often discussed) concern than other unintended adverse consequences, such as workflow disruption and the subsequent potential for new kinds of errors with CPOE adoption.
Conclusion
Overdependence on technology can be an important unintended adverse consequence of the automation of patient care with CPOE systems. Awareness of this issue is vital if organizations are to prepare for and effectively deal with system downtime, assure data accuracy, and help clinicians understand that these tools are designed to support clinical judgment rather than replace it. Finally, organizations should develop methods for measuring the overall efficiency of these systems and quantifiable strategies for system improvement.
Acknowledgments
This research was funded by research grant LM06942 and training grant LM07088 from the U.S. National Library of Medicine, National Institute of Health.
References
- 1. Shortliffe EH , Perreault LE , editors. Medical informatics: Computer Applications in Health Care And Biomedicine . 2nd ed . New York : Springer-Verlag ; 2001 . [Google Scholar]
- 2. Campbell EM , Sittig DF , Ash J , Guappone K , Dykstra R . Types of unintended consequences related to computerized order entry . Journal of the American Medical Informatics Association . 2006 ; 13 : 547 – 556 . doi: 10.1197/jamia.M2042. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Heap N , Thomas R , Einon G , Mason R , Mackay H , editors. Information Technology and Society . Thousand Oaks : Sage Publications ; 1995 . [Google Scholar]
- 4. Pinch TJ , Bijker WE . The social construction of facts and artefacts: Or how the sociology of science and the sociology of technology might benefit each other . Social Studies of Science . 1984 Aug ; 14 ( 3 ): 399 – 441 . [Google Scholar]
- 5. Rogers EM . Diffusion of Innovations . 5th ed . New York : Free Press ; 1998 . [Google Scholar]
- 6. Ash JS , Smith A , Stavri P . Interpretive or qualitative methods: Subjectivist traditions responsive to users . In: Friedman CP , Wyatt JC , editors. Medical Informatics . 2nd ed . Springer-Verlag ; 2005 . [Google Scholar]
- 7. Strauss A , Corbin J . Basics of qualitative research: Techniques and procedures for developing grounded theory . 2nd ed . Thousand Oaks, CA : Sage Publications ; 1998 . [Google Scholar]
- 8. Kho A , Henderson L , Dressler D , Kripalani S . Use of handheld computers in medical education. A systematic review . J Gen Intern Med . 2006 May ; 21 ( 5 ): 531 – 537 . doi: 10.1111/j.1525-1497.2006.00444.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Anderson M . The toll of downtime. Healthcare Informatics Online [WWW document] http://www.healthcareinformatics.com/issues/2002/04/leading.htm Accessed February 13, 2007.