Unintended Consequences: New Problems, New Solutions Contributions From 2015

Summary

Objective

To select the best of the 2015 published papers on unintended consequences of healthcare information technology (HIT).

Method

Literature searches in several areas of scholarship, including IT, human factors, evaluation studies, medical errors, medical informatics, and implementation science. Also, because the specific terms “unintended consequences” were not often included in abstracts and titles, a more nuanced search algorithm was developed.

Results

We identified 754 papers that had some empirical research on unintended consequences of HIT. An initial screen of titles and abstracts reduced this to 171 papers of potential interest. We then further filtered out papers that did not meet the following criteria: 1) the paper had to report an original empirical investigation, and 2) the impact reported had to be not negligible, i.e., in quantitative studies, the results related to unintended consequences were statistically significant; and in qualitative studies the relevant themes emerged were prominent. This resulted in 33 papers of which 15 were selected as best paper candidates. Each of these 15 papers was then separately evaluated by four reviewers. The final selection of four papers was made jointly by the external reviewers and the two section editors.

Conclusions

There is a growing awareness of the importance of HIT’s unintended consequences—be they generated by the HIT vendors, the implementation process, the consultants, the users, or most probably, some combination of the above. There has also been greater creativity in use of data sources, including secondary data (e.g., medical malpractice cases and surveys) and a wider acceptance of mixed methods to identify unintended consequences. Unfortunately, the complexity of causes mitigates the value of recommendations to avoid unwanted outcomes. Suggestions are often contentious rather than obvious, setting-specific, and not universally applicable. “Lessons learned” often take on generalized—and perhaps platitudinous—forms, such as: “plan extra time,” “involve all of the stakeholders,” “recognize the different needs of different units or disciplines.” The greater awareness of these problems, and the increased desire to identify and eliminate them is clearly reflected in the area’s growing literature. We are hopeful the topic will receive additional attention and the discipline will improve its ability to identify and address these unexpected and usually adverse outcomes.

Introduction

Several landmark papers were published over a decade ago reporting the adverse impact of computerized prescriber order entry (CPOE) systems on patient safety [1, 2, 3, 4, 5, 6]. Since then, there has been a growing consensus in the medical informatics community that implementation of health information technology (HIT) is often associated with unexpected, unintended, and usually unwanted, outcomes [7]. In recent years, researchers and practitioners continue to report such unintended consequences of HIT. Over time, this body of literature has been greatly enriched by including more types of HIT applications and more rigorous research designs [8].

Given the prevalence of unintended adverse consequences of HIT, and their significant impact on efficiency, clinician satisfaction, patient evaluations, quality of care, and patient safety, the 2016 IMIA Yearbook introduces a Special Topic section that focuses on this area. We surveyed the 2015 literature in an attempt to identify the “best” research on this topic. In reviewing the literature and selecting best papers, we adopted the working definition of unintended consequences proposed by the Steering Committee of the American Medical Informatics Association (AMIA) 2009 Annual Health Policy Meeting: “Unintended consequences are outcomes of actions that are not originally intended in a particular situation (e.g., HIT implementation).” Unintended consequences thus reflect the effects of technological interventions that deviate from the functionality expected from the originally system design. These might include lost or delayed data, reduced access to information, extra time required to complete tasks, communication failures, errors caused by dual systems (using both paper and digital records), decreased clinician satisfaction, need for extra staff, and/or unexpected financial losses after the introduction of a new IT system [9]. Of course, some unintended consequences are beneficial, providing efficiencies and cost savings [10].

In addition to the difficulty of defining “unintended consequences,” there is the perennial debate about causality and responsibility. This becomes a wicked problem because of the complexity of the systems, the many interactions, and the vast number of unknowns. For example, even before the system “goes live,” there are thousands of customizations required for each installation, some of which are part of the vendor contract, and some of which are built by the local team. Often, these interact in ways that are not only unknown but often unknowable—and certainly unknowable before the system is in operation. Unintended consequences usually result from a complex interplay among: 1) what the HIT vendor provides as a product; 2) the customizations offered by the vendor; 3) the role of consultants; 4) the local clinical team; and 5) the local IT team involved in the implementation process. These actions, motivations, limitations, and expectations are at best hard to disentangle. Therefore, recommendations on best practices gleaned from each implementation are often contentious rather than obvious; and often specific to each setting and not universally applicable. “Lessons learned” often take on generalized—and perhaps platitudinous--forms, such as: “plan extra time,” “involve all of the stakeholders,” “recognize the different needs of different units or disciplines.”

Also, adverse events—especially those resulting from unintended consequences--are frequently unknown and undetected. Hospitalized patients are usually old and sick, have several comorbidities, and are taking many medications. Key organs, like the liver, kidney, and heart, are compromised. Bad things can happen to these patients even when we do everything right; conversely, good things can happen even when we do much wrong. We usually miss the results of, say, a wrongly prescribed medication. We must also acknowledge that cognitive dissonance makes recognition of unintended consequences hard to see. Hospital personnel spend up to five years and sometimes billions of dollars with the implementation process [11]. Careers and budget justifications are on the line for such massive investments. It’s very easy to blame users rather than the system or its developers [12].

Our search of the recent literature identified four trends in unintended consequences scholarship:

1. Continuing research areas: Studies on unintended consequences that have been previously discussed in the literature – largely focusing on electronic health records (EHRs) and their influences on workflow, staffing, and time allocation in a particular medical setting or specialty area.

2. New research areas: Papers that address new problems associated with HIT, such as economics, privacy and safety concerns. While these studies are relatively rare, they have gained more attention in recent years.

3. Methodological concerns: The methods used in studying unintended consequences can seldom if ever rely on random clinical trials. Therefore, many of the studies combine interrupted time series statistics, field observations, interviews, surveys, time-motion research, log analyses, etc. Mixed method approaches or qualitative studies are crucial for understanding the complex interplay of factors involved in any HIT implementation.

4. Secondary data use: Employing national surveys conducted about medical practices in general, or database that are used to submit malpractice claims, was used to discover the trends or issues of unintended consequences that occur frequently across difference practices. Such methods offer new ways of identifying common and frequent consequences on a large scale. These data can be used to address subsequent HIT designs.

The next section presents the best paper selection process and quantitative features about the review. The last section briefly summarizes the selected papers, emphasizing their contribution to patient safety, privacy, decision support, efficiency, error detection, patient trust, etc. Following that, we discuss the future research and challenges.

About the Paper Selection

This is the first time “unintended consequences” are included in the IMIA Yearbook as a Special Topic. As such, there are no established protocols to guide our literature search process. Further, only a handful of relevant systematic review studies are available, and each study employed a distinct literature research strategy tailored to its purpose [13, 14, 15, 16, 17]. Therefore, we employed an iterative process to develop our own literature search strategy.

First, we searched in PubMed for synonyms of “unintended consequences” and “unintended outcomes.” These were then combined with keywords that restricted the search to be HIT-specific, such as “health information technology,” “health IT,” “electronic health records,” “EHR,” “electronic medical records,” “EMR,” “order entry,” “CPOE,” and “e-prescribing.” However, this search strategy only resulted in 14 papers retrieved from PubMed that were published between January 1, 2015 and December 31, 2015.

After reviewing these papers, in addition to several very relevant papers known to us but missed by the search, we realized that many of the papers we needed did not explicitly mention the keywords “unintended consequence” and “unintended outcomes” in their titles or abstracts. We also found that many IT system evaluations or implementation science studies did not originally target unintended consequences, but rather iterated into these areas as their research work evolved. Thus, in the final literature search, we expanded the scope of the search to include keywords such as “evaluation” and “implementation.” The final search returned a total of 754 papers published in 2015¹.

These 754 papers were reviewed to exclude those that did not meet our inclusion criteria. Specifically, the two editors of this Special Topic section first screened the titles and abstracts of the papers to exclude studies that had no relevance (e.g., not regarding HIT). This round of screening resulted in including 171 papers of potential interest. We then again reviewed the abstracts and key sections of those papers, occasionally reading the entire documents, to ensure we did not exclude relevant research. In this round of our reviews, we further filtered out papers that did not meet the following criteria: 1) the paper had to report an original empirical investigation, and 2) the impact reported had to be not negligible, i.e., in quantitative studies, the results related to unintended consequences were statistically significant; and in qualitative studies the relevant themes emerged were prominent. Whenever the two editors had different opinions on the eligibility of a paper, the disagreement was resolved through consensus meetings. This round of screening resulted in the selection of 33 papers.

To select the final pool of 15 papers for the best paper selection, the full texts of these 33 papers were reviewed independently by each of the two editors, and disagreements were resolved in further discussion. Following the IMIA Yearbook best paper selection protocol, these 15 papers were then peer-reviewed by the Yearbook editors and external reviewers. At least four reviewers were assigned for each paper. This process led to four papers selected as the best papers for the IMIA Yearbook 2016’s Special Topic section on Unintended Consequences. They are listed in Table 1.

Table 1.

Best paper selection of articles for the IMIA Yearbook of Medical Informatics 2016 in the section ‘Unintended Consequences: New Problems, New Solutions’. The articles are listed in alphabetical order of the first author’s surname.

Section Unintended Consequences: New Problems, New Solutions

Campos-Castillo C, Anthony DL. The double-edged sword of electronic health records: implications for patient disclosure. J Am Med Inform Assoc 2015;22(e1):e130–e140. Cifuentes M, Davis M, Fernald D, Gunn R, Dickinson P, Cohen DJ. Electronic Health Record Challenges, Workarounds, and Solutions Observed in Practices Integrating Behavioral Health and Primary Care. J Am Board Fam Med 2015;28 Suppl 1:S63–S72. Georgiou A, Prgomet M, Lymer S, Hordern A, Ridley L, Westbrook J. The Impact of a Health IT Changeover on Medical Imaging Department Work Processes and Turnaround Times: A mixed method study. Appl Clin Inform 2015;6(3):443–53. Graber ML, Siegal D, Riah H, Johnston D, Kenyon K. Electronic Health Record-Related Events in Medical Malpractice Claims. J Patient Saf 2015 Nov 6.

Discussion and Outlook

One primary intended outcome of HIT is to advance the efficiency of clinical work via enhanced information access and process improvements. To that end, much of the prior literature sought to examine the impact of these technologies on the efficiency of medical work. The first selected paper, by Georgiou et al. [18] continues this line of research but focuses on the impact of a new Picture Archiving and Communication System (PACS) and Radiology Information System (RIS) in radiology practices. The study’s objectives were to assess how the new PACS and RIS systems affect the medical imaging work processes and the turnaround time (TAT). They measured the time from when the orders were initiated from the Emergency Department until they were completed. Their mixed method study analyzed both qualitative semi-structured interviews with the imaging department staff, and quantitative, retrospective data captured in the systems for calculation TAT.

They found that the introduction of the PACS/RIS systems dramatically improved the efficiency of the work practices both for patients, who were discharged from the emergency room, and those who were admitted to the hospital -- with median TATs shortened by a third to half. For patients who were discharged from the hospital, the median time dropped from 76 hours to 38 hours; and for patients who were admitted to the hospital, the median TAT reduction was even greater, from 84 hours to 35 hours. The easier access of images as well as patient-related information in the digital systems was seen as the main reason behind the efficiency gain – an intended benefit that most health IT systems were designed to deliver. However, the authors’ interviews with clinicians revealed several significant deficiencies of the RIS system, i.e., critical functions of patient notification, in the previous RIS system, no longer existed in the new system. This loss resulted in communication breakdowns and prolonged patient wait time. In response, staff developed workarounds using “manual” spreadsheets to track patients. Also, the new dictation via voice recognition was called slow and inconvenient. This study raised essential questions about how efficiency should be defined and whether a quantitative indicator alone is sufficient to assess system performances. How do researchers balance the value of shorter task turnaround times versus longer documentation time by radiologists and delayed patient/clinician notification times?

The best paper authored by Cifuentes et al. [19] addresses another widely discussed issue in health informatics. While the EHR may (or may not) deliver its designed benefits to general practices, it often fails to meet the needs of various medical specialties. The study examines cases where behavioral health information and primary care data had to be connected to deliver “integrated care” to patients. Through an observational cross-case study of eight primary care clinics and three mental health community centers, the researchers found that all of the practices’ EHRs generally lack the ability to document both the behavioral health data and general care information in the same chart. Thus, EHRs did not support the medical home concept of shared planning needed by primary care doctors and behavioral health practitioners. In addition, even with two EHR systems being used at the same time, the lack of interoperability made it impossible to document the behavioral health surveys in the general EHR. To overcome these problems, various workarounds were developed, which often generated increased workload, fragmented data, inefficient data management, and even additional standalone systems. The authors believe that the findings described in their paper are limited to “practices that were specific to behavioral health and primary care integration.” This research, never the less, again offers warnings about the need to design centralized IT systems that support the needs of heterogeneous practices that are often (and increasingly) sharing data or are in need of sharing data. The suggested “solutions,” such as customizing EHR templates to accommodating the additional behavioral measurements, however, have been repeatedly shown to be counterproductive. That is, prior research illustrates that many workarounds prevent seeking longer-term and more satisfactory solutions. Worse, they also frequently require special data entry systems or processes and increased opportunities for error [20, 21].

Departing from the usual methods of unintended consequences papers—which are generally empirical studies examining HIT used in clinical practices—our search also identified a few papers that used secondarily data analysis from national surveys or databases. While such approaches are not uncommon in studying HIT, they are seldom used to examine unintended consequences of HIT. The reason for this rarity is that these unintended consequences are always deeply embedded in the specific sociotechnical environments in which HIT systems are situated. The third best paper, a study by Graber et al. [22], investigated safety issues related to HIT from a national database of medical malpractice claims. Two years of malpractice claims were coded using a mixed methods approach. Among all malpractice claims, less than 1% of them were found to be related to HIT (248 cases). Malpractice cases occur in a wide variety of service areas, with most reported in medicine, surgery, nursing, and obstetrics and gynecology. Further analysis suggests that most of these HIT-related malpractice cases involved medication errors (31%), followed by diagnosis errors (28%), and treatment complication (31%). More than 80% of these cases involved moderate or severe harm leading to serious injuries or even death. The overall severity of these cases is understandable because only the most grievous and “provable” are accepted by attorneys and become “legal cases,” even if settled without trials.

The qualitative coding showed a wide range of system- and user-related socio-technical factors involved in adverse patient safety issues. IT system-related factors were technology or software design flaws, such as not being able to access critical clinical notes on time, or information errors due to the poorly designed interfaces. In contrast, user-related factors were sometimes due to the lack of proper training or education on how users should access and respond to the system requests.

This study revealed that the majority of HIT-related malpractice cases were centered on the EHR, usually the centerpiece of all digital medical systems. Most important, the study revealed that many patterns of errors were re-occurring across different medical malpractice claims, indicating that many of the patient safety hazards are foreseeable and potentially avoidable. Of course, IT system designers can only learn the lessons from the practices and communities if they can access the actual case documents. The fact that such information is usually sealed as part of the settlement with the vendors and providers makes these lessons hard to find. The Graber et al. paper is to be especially commended for presenting this information, even though it does not reveal the specific issues of each HIT system.

The authors highlighted four main problems related to EHRs. These were: “the danger inherent to hybrid systems and EHR conversion,” “the dangers of delayed, missing, or incorrect data, services, or actions,” “the danger of over-reliance on the EHR,” and “the inherent risks using copy\paste functionality, overriding alerts, and employing ‘workarounds.” It is essential to point out that even the “user-driven” errors, such as copy/paste and overriding alerts, were mostly the result of the poorly designed or implemented HIT systems. In other words, the user behaviors are seldom if ever the only cause (and indeed, most scholars refer to these as “use errors,” rather than “user errors”). The task of EHR vendors is therefore to design EHRs that minimize the probability of errors and to eliminate unnecessary burdens borne by users.

The last of the four best papers [23] extended the focus of unintended consequences of HIT from practitioners to patients. Patients, of course, are not direct EHR users, but may perceive risks associated with their providers’ use of HIT. Campos-Castillo and Anthony specifically examined whether patients withheld personal information from their providers who used an EHR. Data were from a national survey by the National Cancer Institute study about peoples’ access to health information. The survey items used to measure patients’ non-disclosure behaviors were: 1) “Have you ever kept information from your health care provider because you were concerned about the privacy or security of your medical record? (Yes/No);” and 2) about the providers’ uses of an EHR -- “As far as you know, do any of your doctors or other health care providers maintain your medical information in a computerized system? (Yes/No).” In addition, authors applied the global rating for care based on the five-point Likert scale, “Overall, how would you rate the quality of health care you received in the last 12 months?’ (Poor–Excellent).” The results showed that 13% of responding patients have withheld personal information from a provider due to privacy and security concerns. After statistical adjustments, the authors also found that patients, who believe their clinicians used EHRs, are more likely to have ever withheld information from a provider. That is, the authors conclude that there is a greater likelihood of patient non-disclosure of relevant medical information if the provider uses an EHR. Campos-Castillo and Anthony offer solutions: 1) that clinicians should discuss privacy concerns with patients during their medical visits; and 2) that policies should be set to address the real and perceived privacy and security risks inherent to the use of EHRs.

The obvious limitations of this type of data analysis are that the measurements used in the original study may not be sufficient to infer the causal relationships, and that the wording of the questions may not be the best to solicit patients’ concerns regarding the goals of the analysis. Future research in this area should supplement the survey analysis with additional qualitative data, such as determining these EHR-related perceived privacy and security risks. Also, one asks why some patients perceived such risks when their providers’ offices were not even equipped with EHRs? What do patients desire to (need to?) know to eliminate their privacy concerns? What might be the best strategies to educate patients about these concerns?

Conclusion

In surveying the literature for the 2016 IMIA Yearbook on “Unintended Consequences: New Problems, New Solutions,” we found a large number of papers in a wide range of fields, that did not specifically use the term “unintended consequences.” This obliged us to develop more nuanced search algorithms, which proved useful. Most of the papers we identified are focused on IT systems, in particular EHRs, and their effects on clinicians. They generally used empirical methods. The use of secondary data—previously collected data for other purposes—added considerably to the field, and indeed two of the best papers reflected these sources.

One of the larger lessons for unintended consequences research is the need for all errors and near errors to be collected, aggregated, and analyzed. Only then can researchers offer lessons that transcend local practices and better inform HIT design. Thus, the need for a protected reporting institution or space – the need to build a community of practice to report the issues – is essential to patient safety and to improving HIT design.

Looking forward, it may be valuable to move beyond clinical personnel and errors to include other stakeholders affected by HIT use, such as supporting staff, IT staff, and non-clinically trained caregivers at home, or visiting patients in hospitals. HIT studies could also benefit from the work of human-computer interaction scholars who frequently incorporate non-clinical personnel in technology design [24, 25]. Also, with the rise of patient-centered technologies such as online patient portals and mobile health systems, researchers will increasingly need to address the needs of patients—including misunderstandings and access limitations.

Appendix: Content Summaries of selected best papers for the 2016 IMIA Yearbook, special section “Unintended Consequences: New Problems, New Solutions”

Georgiou A, Prgomet M, Lymer S, Hordern A, Ridley L, Westbrook J

The Impact of a Health IT Changeover on Medical Imaging Department Work Processes and Turnaround Times: A mixed method study

Appl Clin Inform 2015;6(3):443–53

Objective: The study examines the impact of a new Picture Archiving and Communication System (PACS) and Radiology Information System (RIS) on medical imaging work processes and turnaround times (TATs).

Methods: A mixed method approach that includes qualitative interviews of radiology staff and logged TAT times for images ordered from the Emergency Department (ED).

Findings: The introduction of the new PACS/RIS system significantly reduced the TAT times and improved accessibility of images and patient-related information by medical imaging staff. On the other hand, qualitative interviews revealed some negative effects of the new system, such as prolonged documentation times, a missing patient tracking function, and the mismatch of system requirement with the physical layout of the workplace.

Conclusion: While offering efficiency improvement as designed, the introduction of the new PACS/RIS system was coupled with unwanted, unintended consequences on medical imaging work.

Cifuentes M, Davis M, Fernald D, Gunn R, Dickinson P, Cohen DJ

Electronic Health Record Challenges, Workarounds, and Solutions Observed in Practices Integrating Behavioral Health and Primary Care

J Am Board Fam Med 2015;28 Suppl 1:S63–S72

Objective: To examine the use of electronic health records (EHRs) in practices that integrate behavioral health and primary care.

Methods: An observational study of 11 diverse practices, including eight primary care clinics and three community mental health centers. Field study data were analyzed using grounded theory to identify challenges of EHR use and workarounds developed by healthcare providers.

Findings: Practices found it difficult or impossible to include behavioral health data and documentation and physical health information in one EHR system. Challenges included lack of support for communication and coordination in the team-based practices, and non-interoperability of the systems that provided both behavioral health and physical health information. Healthcare providers adopted various workarounds to accommodate these challenges, such as double documentations, duplicated data entry, scanning and carrying copies of documents, reliance on patient or clinician recall for inaccessible EHR information, and use of freestanding tracking systems. Some of these temporally workarounds were then turned into permanent EHR features.

Conclusion: EHR system need to be designed to support both: a) documentation of integrated care that connects behavioral health and primary care, and b) care coordination in team-based practices.

Graber ML, Siegal D, Riah H, Johnston D, Kenyon K

Electronic Health Record-Related Events in Medical Malpractice Claims

J Patient Saf 2015 Nov 6

Objective: To identify HIT-related medical errors by analyzing malpractices cases in a large database.

Methods: Qualitative and quantitative methods were employed to analyze the cases related to HIT use, including responsible services, error types, recurring patterns of malpractice cases, and the sociotechnical factors associated with the malpractice cases.

Findings: Among all cases submitted to the malpractice database within 2012--2013, less than 1% (248) were found to be HIT-related. 59% of the cases were at ambulatory care facilities, followed by inpatient (31%), and emergency settings (10%). The most frequent HIT-related errors are medications (31%), a complication of treatment (31%), then diagnosis (28%). More than 80% of cases reported moderate or severe harm to patients.

Conclusion: The study shows the value of using national malpractice database to identify recurring patterns of errors related to health IT use, and to provide lessons for avoiding many errors.

Campos-Castillo C, Anthony DL

The double-edged sword of electronic health records: implications for patient disclosure

J Am Med Inform Assoc 2015;22(e1):e130–e140

Objective: To investigate whether patients choose to not disclose their personal information when they perceive privacy and security risks resulted from providers using of an electronic health record (EHR).

Methods: The 2012 Health Information National Trends Survey data were analyzed to identify relationships between EHR provider use and patient non-disclosure.

Findings: Thirteen percent of responding patients have withheld personal information from a provider due to privacy and security concerns. However, a simple correlation did not find that withholding information was not associated with providers’ use of EHR. After statistical adjustments, it was found that patients, who believe their clinicians used EHRs, are more likely to have ever withheld information from a provider.

Conclusion: There is a greater likelihood of patient non-disclosure of personal health information if the provider is believed to use an EHR.