Clinician Perspectives on the Quality of Patient Data Used for Clinical Decision Support: A Qualitative Study

Abstract

Objective:

Clinical decision support (CDS), defined broadly as patient-specific information and knowledge provided at the point of care, depends on a foundation of high quality electronic patient data. Little is known about how clinicians perceive the quality and value of data used to support CDS within an electronic health record (EHR) environment.

Methods:

During a three-year research study, we collected ethnographic data from ten diverse organizations, including community hospitals, academic medical centers and ambulatory clinics.

Results:

An in-depth analysis of the theme “data as a foundation for CDS” yielded a descriptive framework incorporating five subthemes related to data quality: completeness, accessibility, context specificity, accuracy, and reliability.

Conclusion:

We identified several multi-dimensional models that might be used to conceptualize data quality characteristics for future research. These results could provide new insights to system designers and implementers on the importance clinicians place on specific data quality characteristics regarding electronic patient data for CDS.

Introduction

Clinical decision support (CDS), combined with electronic health records (EHRs) and computerized provider order entry (CPOE), has shown mixed potential to increase patient safety, improve the efficiency and effectiveness of care, and to help slow the growing costs of healthcare delivery.[1–4]

While previous studies of clinical decision support have largely focused on computer generated alerts and reminders such as drug interaction alerts, duplicate order checks, and preventative care reminders[2–4], the meaning of CDS has continued to evolve.[5] For example, a recent definition states: “CDS is a process for enhancing health-related decisions and actions with pertinent, organized clinical knowledge and patient information to improve health and healthcare delivery...information delivery formats can be drawn from a rich palette of options that includes data and order entry facilitators, filtered data displays, reference information, alerts and others.”[6]

Although the number of studies evaluating the impact and use of CDS continues to grow, very few have looked at the effects of electronic patient data quality on the functioning of CDS.[7–9] We were unable to find peer-reviewed studies that examined the experiences and attitudes of clinicians with respect to electronic patient data quality and its effect on CDS.

Objective

In 2007 our team, based at Oregon Health & Science University (OHSU), began an ongoing program of research to provide insight into the perceptions and attitudes of clinicians who use CDS.[10] As part of this earlier study, we identified “data as a foundation for CDS” as an important theme leading to a new research question: What are clinicians’ perceptions about the quality and value of electronic patient data used for CDS in their organizations?

Methods

We purposively selected ten organizations known for their use of EHRs, CPOE, and CDS. Study sites included: two academic medical centers, six community hospitals, an outpatient Veterans Administration clinic, and a group of independent ambulatory practices (Table 1).[10]

Table 1.

Research sites for CPOE and CDS studies, 2007 to 2010.

Visit Date	Site	Type	Location	Setting	EHR
Dec, 2007	Providence Portland Medical Center	Community	Portland, OR	Inpatient	Commercial
Feb, 2008	El Camino Hospital	Community	Mountain View, CA	Inpatient	Commercial
June, 2008	Partners HealthCare	Academic and Community	Boston, MA	Both	Both
Sept, 2008	Wishard Memorial Hospital Clinics	Academic and County	Indianapolis, IN	Outpatient	Homegrown
Sept, 2008	Roudebush VA	Veterans Administration	Indianapolis, IN	Outpatient	National
Dec, 2008	Mid-Valley IPA	Community	Salem, OR	Outpatient	Commercial
Feb, 2009	RWJ Medical Group	Academic	New Brunswick, NJ	Outpatient	Commercial
Jan, 2010	LDS Hospital (IHC)	Community	Salt Lake City, UT	Both	Commercial
March, 2010	Caritas	Community	Boston, MA	Both	Commercial
July, 2010	Health East	Community	St. Paul, MN	Both	Commercial

Data were gathered from on-site ethnographic interviews and observations focused on the participants’ own conception of CDS, which turned out to be surprisingly inclusive.[5,10] Semi-structured interviews were conducted with physicians, nurses, pharmacists, other types of clinicians, and CDS technical experts. During the same visits, the team conducted multiple participant observations to study clinicians interacting with CDS in naturalistic environments. Field data were collected using a novel application of a team-based ethnography called the Rapid Assessment Process (RAP). An especially intriguing result of these studies, arising spontaneously as a concern among many of our participants, was about clinical data used for CDS. For detailed reports about these previous field studies and methods see Ash, et al. and McMullen, et al. [10,11]

The present study extends our team’s earlier work by applying axial coding methods (informed by a grounded theory perspective) to a subset of 59 participant interviews (26 physicians, 21 nurses, 11 pharmacists, and 1 imaging technician) previously coded with the theme “patient data as a foundation for CDS.”[10] Through a process of continuous comparison, we generated codes for actions, outcomes, perceptions and relationships from the interview text. These codes were compared and contrasted over several iterations to form themes and relationships.[12] Emerging themes were discussed with members of the research team over the course of our analysis to produce the results and theoretical model reported below.

Our research was approved by the Institutional Review Board at OHSU and at each participating organization, and funded through grants from the National Library of Medicine and the Agency for Healthcare Research and Quality.

Results

Our analysis of 59 clinician interviews revealed strong opinions about the quality of patient data used for CDS in their organizations. Five characteristics of patient data quality relevant to CDS were broadly represented across all study sites and clinician roles: 1) completeness, 2) accessibility, 3) context specificity, 4) accuracy, and 5) reliability. No clear trends between settings (inpatient/outpatient), organizational characteristics, or the type of electronic health record emerged.

Characteristics of Patient Data Quality

Clinicians said that the quality of electronic patient data in their organizations was sometimes poor, but this was variable. For example, one nurse in a leadership role said flatly: “No one knows how bad data is in hospitals--on a good day it’s bad, on a bad day it’s terrible.” It is natural for interviewees and those observed to focus on problems, so we heard many general statements expressing negativity. For example, one nurse noted: “A lot of people get data and they’re very disappointed because [CDS] isn’t what they expected. It’s because they don’t have good data.” One physician said: “The mammogram is a simple example about how just a little bit of the wrong or missing data can make a rule 1/5 of the time not fire correctly.” Another physician made the link between data and CDS explicit by saying: “To do good decision support, you need good data.”

Completeness of data

Missing or incomplete patient information was the most frequent example clinicians gave of poor data impacting CDS. System interoperability (or the lack thereof) and problems with human data entry were both described as causing gaps in the electronic record. They said, for example, with the role of the speaker in brackets: [Nurse] “We don’t have that connectivity”; [Physician] “They’re still handwriting progress notes”; [Imaging Tech] “Most of our patients don’t get their blood work done in the hospital lab, so we can’t check it by the hospital system”; [Physician] “The question came up, ‘would nurses fill it out; would doctors fill it out?”; and [Physician] “Even my own partner doesn’t really, you know, capture or ‘do’ the data.”

Accessibility of the data

Our participants described examples of both inaccessible data and missing data. Data were considered inaccessible when the information existed, but could not be easily retrieved or used; data were considered missing when they did not exist in an electronic form.

For example, one pharmacist noted frequent difficulties in navigating the EHR: “I think [the EHR] is fairly comprehensive with the data that’s available; getting to it is the problem.” Interviewees also described some patient information as being “hidden”. For example, a nurse said: “[The medication] doesn’t have to be in the order; it can be in the progress note.” One physician noted: “As you see, in spite of everything that we do, occasionally the only time that CHF is mentioned is in the discharge summary. By then, the patient’s gone”, and another pointed out during observation: “When I look at this flowsheet, there’s no lab results from that visit ‘cause it’s not signed.”

Clinicians often considered patient data in scanned documents or external systems to be hidden, frequently making it difficult to locate and use, and all but invisible to CDS rules and alerts. One nurse called the scanned documents in her EHR: “…a big bucket of images.” Data were also considered hidden within portals, outside systems, or multiple EHRs, frustrating efforts to assemble a complete clinical picture of the patient. One physician described the proliferation of clinical data sources as: “[Yet] another icon on the desktop.”

Data relevance to context

Many clinicians found that CDS was less effective because it failed to adapt to individual and clinical contexts. For physicians and pharmacists, this often took the form of CDS alerts or reminders that were irrelevant or inappropriate for a given patient or situation. For example, a physician said: “[If] my patient is going to die in three months, most people would forgive a computer saying ‘you should do a mammogram.’”

A pharmacist described her challenges reviewing medication alerts by saying: “You get these potential flags and go from there. You investigate, see, because most of ‘em are false positives”; she went on to say: “You get better at investigating; being a detective.”

To the participants, the relevance of patient information was not only situational, but the ideal representation and presentation of the data could also be dependent on context. For example, many physicians lamented the loss of rich narrative under pressure to use structured documentation and templates. A physician gave a colorful example of the value of his preference for narrative: “Instead of reading ‘hematoma under the big toenail’, [I’d prefer] ‘patient was lifting the frozen turkey out of the freezer and dropped it on his toe’…[this] gives so much more flavor; so much more richness.” Another physician, referring to the wide use of templates and copied text, said: “Every note sounds the same and you’re not really getting information. You’re just getting [expletive] back!”

Nurses also shared concerns about moving away from narrative in favor of structured documentation. For example, one said: “Everybody documents on [templates] so documentation should be somewhat consistent, but, [sometimes it is] too uniform ‘cuz people just copy what the other person entered.”

Data accuracy and reliability

The clinicians also complained of inaccurate and unreliable patient data. Although these two themes appeared less frequently than the three described above, they have obvious importance to clinicians.

An example of miss-entered data was provided by a nurse, who said: “There are check boxes that [nurses] are supposed to fill in, and you’re thinking, ‘what were they thinking when they checked those?’ …I can’t imagine how you can check this and also that.” One pharmacist pointed out: “[The physicians are] just about as sloppy in here as they are with handwriting; it doesn’t change.” We also were told of miss-identified records because of wrongly entered patient names, noted one physician: “I happened to notice [while searching for a progress note on Francis] that there was somebody in there [named] ‘Frances’,‘ces’; it turns out it was him.” Finally, outdated information was a problem. Said one physician: “This medication might still be wrong, but at least as far as the computer knows, it’s the medication this patient should be getting.”

Descriptive model of data quality and CDS

We adapted an eight stage socio-technical model by Sittig and Singh[13] and a generic data lifecycle[14] to develop a simple theoretical model (Figure 1) relating the factors thought, by our participants, to affect data quality with consequences for CDS.

Figure 1.

A descriptive model of factors impacting data quality of CDS from 59 clinician interviews.

Our participants described a progression of patient data from acquisition through delivery and use, while interacting with four components with direct impacts on data quality for CDS: Humans, Computers, Human-Computer Interfaces (HCI), and Organizations (represented in the figure as dashed lines).

Examples of each of the twelve possible nodes were represented in our interviews and observations. For example, many user interface issues were observed during data entry (represented in the model as the intersection of Acquisition-HCI) with the potential to impact the perceived quality of the data for CDS. These factors were often amplified when data crossed organizational boundaries.

Discussion

We conducted this study to more deeply analyze one of ten themes found in our multi-year, multi-site study of CDS[10]: What are clinicians’ perceptions about the quality and value of electronic patient data used for CDS in their organizations? Our use of ethnographic fieldwork to answer this question is unique.

Previous non-ethnographic studies have investigated the quality of computerized patient data including: completeness[15], accuracy[15,16], timeliness[17], and the impact of data quality on CDS.[7–9] There is a need, however, for a more comprehensive multi-dimensional framework to simultaneously address the many characteristics of patient data quality important to clinicians.

One such model, by Wang, Lee and Strong[18,19], comprehensively addresses multiple characteristics (or “dimensions”) of data quality based on empirical research of (non-healthcare) data consumers. Similar models (Table 2) have been adapted for health care by several authors[20] and organizations[21,22]. No studies were found, however, that used more than two dimensions to describe or evaluate patient information in EHRs. Based on our study, we recommend the use of a multi-dimensional framework to incorporate a rich understanding of clinicians as “data consumers” and to ensure that electronic clinical data are, to quote Joseph Juran, “fit for intended use.”[23]

Table 2.

Data quality dimensions from the literature of computer science and healthcare.

Strong, Lee, and Wang (1997)	Kerr (2006)	AHIMA (2007)	CIHI (2009)
Intrinsic Data Quality - Believability - Accuracy - Objectivity - Reputation - Security and privacy Contextual Data Quality - Value-added- Relevancy - Timeliness - Completeness - Appropriate amount Representational Data Quality - Interpretability - Ease of understanding - Representational consistency - Concise representation Accessibility Data Quality - Accessibility - Access security	Accuracy Timeliness Comparability Usability Relevance Privacy and Security	Accessibility Accuracy Consistency Comprehensiveness Currency Definition Granularity Relevancy Precision Timeliness	Accuracy Timeliness Comparability Usability Relevance

Abbreviations: AHIMA: American Health Information Management Society

CIHI: Canadian Institute for Health Information

Our proposed framework is supported by a recent study by Magrabi et al. Analyzing a government database of voluntary reports of adverse events related to health information technology, the authors describe a multi-factor model built around the flow of patient data that is similar in many respects to our own.[24]

The strengths of this study include the diversity of our sites and the real-world perspectives of clinicians using state-of-the-art CDS while delivering patient care. Our study is limited by the small number of organizations we studied and our choice to focus on advanced adopters of health information technology.

Conclusion

This study identified five consistent and recurring characteristics of patient data that impact the perceived effectiveness of CDS among physicians, nurses, and pharmacists: data must be complete, accessible, context specific, accurate, and reliable. These characteristics were found to be important to clinicians across a diverse set of organizations, settings, electronic health record environments. In addition to proposing a new descriptive framework, we identified several multi-dimensional models from the literature and communities of practice that might be used to conceptualize subjective and objective data quality characteristics as a useful framework for future research. These results could provide new insights to system designers and implementers on the relative value and contextual importance clinicians place on specific data quality characteristics regarding electronic patient data for CDS.