Workflow Differences Affect Data Accuracy in Oncologic EHRs: A First Step Toward Detangling the Diagnosis Data Babel

Abstract

PURPOSE

Diagnosis (DX) information is key to clinical data reuse, yet accessible structured DX data often lack accuracy. Previous research hints at workflow differences in cancer DX entry, but their link to clinical data quality is unclear. We hypothesized that there is a statistically significant relationship between workflow-describing variables and DX data quality.

METHODS

We extracted DX data from encounter and order tables within our electronic health records (EHRs) for a cohort of patients with confirmed brain neoplasms. We built and optimized logistic regressions to predict the odds of fully accurate (ie, correct neoplasm type and anatomic site), inaccurate, and suboptimal (ie, vague) DX entry across clinical workflows. We selected our variables based on correlation strength of each outcome variable.

RESULTS

Both workflow and personnel variables were predictive of DX data quality. For example, a DX entered in departments other than oncology had up to 2.89 times higher odds of being accurate (P < .0001) compared with an oncology department; an outpatient care location had up to 98% fewer odds of being inaccurate (P < .0001), but had 458 times higher odds of being suboptimal (P < .0001) compared with main campus, including the cancer center; and a DX recoded by a physician assistant had 85% fewer odds of being suboptimal (P = .005) compared with those entered by physicians.

CONCLUSION

These results suggest that differences across clinical workflows and the clinical personnel producing EHR data affect clinical data quality. They also suggest that the need for specific structured DX data recording varies across clinical workflows and may be dependent on clinical information needs. Clinicians and researchers reusing oncologic data should consider such heterogeneity when conducting secondary analyses of EHR data.

INTRODUCTION

Secondary analysis of electronic health record (EHR) data is essential to the development of learning health care systems,^1-3 oncologic comparative effectiveness research,^4-7 and precision oncology decision support.^8-11 This often relies on patient diagnosis (DX) data for cohort selection^12,13 in spite of its quality limitations.¹⁴ Decades of research have shown alarmingly high DX inaccuracy rates,^15-17 which can greatly affect secondary analysis results.¹⁵ Despite error rate improvements from 20%-70% in the 1970s to 20% in the 1980s, their reliability remains questioned.^5,18 The complex nature of clinical knowledge, variable clinical workflows, and billing-oriented data recording are partially to blame^5,19-22 but the challenges are exacerbated by EHR systems that provide multiple descriptions for individual DX codes.²³

CONTEXT

• Key Objective

• To uncover the relationship between oncologic electronic health record (EHR) diagnosis (DX) data quality and both workflow and user profiles to inform reliable reuse of oncologic EHR data in clinical care for precision oncology and clinical decision support.

• Knowledge Generated

• Both clinical department and care location were predictive of accurate or suboptimal DX entry (eg, DX entered in nononcology departments had higher odds of being accurate; outpatient care locations had lower odds of recording inaccurate DX). A user’s clinical role was predictive of accurate, inaccurate, or suboptimal DX data entry (eg, DX recoded by a physician assistant had fewer odds of being suboptimal compared with those entered by physicians).

• Relevance

• Clinical oncologists using EHR data to make patient care decisions or reusing clinical data to develop clinical decision support tools should account for these significant accuracy differences across clinical workflows and clinical personnel. There may be different levels of ideal DX specificity across clinical workflows.

Inaccurate DX data entry is particularly complex in oncology because of both cancer DX coding structures and oncology workflow constraints. Standard DX code descriptions are not designed to support oncology data reuse,^5,24,25 leaving information locked in progress notes.^5,26 For example, International Classification of Diseases (10th revision) codes C71.XX correspond to “malignant neoplasm of the brain” DX codes and allow encoding anatomic site (eg, C71.1 represents a malignant neoplasm of the frontal lobe). However, unlike International Classification of Diseases for Oncology (3rd edition) codes,²⁷ which are not as broadly adopted, they do not encode neoplasm type (eg, IDH wild-type glioma, glioblastoma, and so on), which is crucial to treatment selection and patient classification. Customized DX descriptions provided by EHR vendors include some neoplasm type information but present wide-varying levels of specificity, complicating structured DX entry.²⁸ Because cancer care is team based and requires patients to interact with multiple specialties and units (eg, scheduling, imaging, surgery, and so on); EHR systems rarely support consistent data recording across clinical workflows, making DX logging burdensome to oncologists.^29-31 Although both aspects cause structured DX data unreliability, much more has been published on resulting data issues^14,21,22 and limitations of coding terminology^25,21 than on the impact of clinical workflows on EHR data.^32-35

Oncologic records suffer from high DX data variability,²⁸ EHR section-dependent accuracy levels,³⁶ and statistically significant differences in DX entry accuracy across subspecialties.³⁷ This hints at a link between clinical workflows and resulting data quality, in concordance with prior work.^5,38 However, the literature is still unclear on which factors affect data quality. To address this gap, we conducted a statistical analysis. We assessed which clinical workflow factors correlate with accurate DX entry. We hypothesized that there is a statistically significant relationship between workflow-describing variables (eg, care location, department, and users) and accurate, inaccurate, and suboptimal (ie, correct but imprecise) data entry. We tested this hypothesis on EHR data from patients diagnosed with brain neoplasms. We selected this disease for its large number of textual diagnosis descriptions of varying levels of precision for a limited list of specific diagnosis codes and the availability of a definitive histopathology report stating the most precise and accurate DX description possible. This analysis improves our understanding of oncologic data entry within clinical workflows and its impact on clinical data quality. Our findings identify a new avenue for clinical data quality improvement, thereby facilitating reliable secondary uses of oncology data within learning health care systems and future clinical oncology applications, such as clinical data–driven clinical decision support for cancer treatment selection.

METHODS

We extracted structured DX data and relevant covariates across multiple clinical workflows from the Wake Forest Baptist Medical Center’s EHR database. Our study was approved by Wake Forest University School of Medicine’s Institutional Review Board (IRB; No. 00044728). Our initial extract contained oncologic DX entries for a set of 36 patients treated for brain neoplasms. The data set contained DX descriptions entered during care and covariates describing clinical workflows and involved personnel. The covariates selection was driven by relevance to clinical workflow and personnel-EHR interaction descriptors, but also data availability within the EHR’s data and metadata tables. Our initial list of covariates included days between biopsy and DX entry, DX chronologic rank in entry sequence, clinical department and location where the DX was entered, visit providers and their specialties, users entering the data and their clinical role (eg, physician, nurse, assistant, and so on), the authorizing provider, and the order type for order DX.

To identify accurate, inaccurate, and suboptimal DX description, we used a clinician-generated gold standard containing each patient’s accurate diagnosis. Patient charts were preselected from an existing chart review–based glioma registry (IRB No.: 00038719). Patient inclusion was based on completeness of information within the registry. Patients having received cancer care outside our institution were excluded to ensure that all relevant care information was available within our EHR. Comprehensive medical record review was performed by 2 independent reviewers. The primary postoperative diagnosis was determined based on a review of pathology reports and clinician notes. All treating clinicians were available for consultation when needed. Discrepancies between the 2 reviewers were resolved by an independent neuro-oncologist. Two features were used to determine DX accuracy: neoplasm type (eg, astrocytoma, glioblastoma, and so on) and anatomic site (eg, frontal lobe, temporal lobe, and so on). We compared the standard’s DX description with each DX entry. Accurate DX entries matched the standard’s neoplasm type and anatomic site; inaccurate DX entries contradicted neoplasm type or anatomic site. For example, a frontal lesion DX for a patient with a temporal lesion would be inaccurate; an astrocytoma DX for a patient treated for a glioblastoma would also be inaccurate. Partially accurate (ie, suboptimal) DX descriptions were categorized separately because they did not contradict the standard DX because they failed to provide a specific DX description.

Our final data set contained 10,052 DX attached to procedure orders and 3,718 encounter DX observations of 31 patients, recorded from January 1, 2016, to June 1, 2018. This time frame was defined to ensure ICD coding version consistency (ie, to include DX after October 2015; ICD10 implementation date). We only analyzed data after a biopsy (BX) to allow for accurate recording on the clinical side. However, 4 patients did not have BX data within the selected time window and were excluded. One patient was excluded because of a confirmed neurofibromatosis DX, which made the patient not clinically comparable to other patients.

We selected our predictors based on the strength of correlation (Nagelkerke R²)³⁹ with each outcome variable, because it solves Cox-Snell correlation’s⁴⁰ upper bound issues and provides a generalized correlation metric similar to our modeling approach.⁴¹ We used a stepwise selection approach for covariate inclusion. To test our hypothesis, we built logistic regressions⁴¹ to predict the odds of accurate, inaccurate, and suboptimal DX across patient charts using R’s generalized linear model package.⁴² Some of our variables had too many categories to be useful in our model. Thus, we reclassified clinical departments into oncology and nononcology departments and represented user factors via provider type rather than using user identification numbers. We maximized goodness of fit using Akaike’s information criterion.⁴³ We tested for variable interactions and collinearity effects in all models with more than one predictor. Adjustments for multiple comparison were made using R’s p.adjust function⁴⁴ selecting Holm’s correction method.⁴⁵ We reran each regression using a time window of 90 days before and after the BX to confirm the effect’s robustness.⁴⁶ Data extraction was performed with DataGrip software (version 2017.2.2; JetBrains, Prague, Czech Republic), exploratory analysis relied on Tableau (version 10.2.4; Tableau Software, Seattle, WA), and graphics were generated using Prism (version 8; GraphPad Software, San Diego, CA). Data cleaning and statistical analyses relied on R version 3.6.1³⁰ and RStudio (version 1.2.1335; RStudio, Boston, MA).

RESULTS

The final analytical data set contained 10,052 order DX entries and 3,718 primary encounter DX entries for 31 patients (Table 1). Each patient had at least one encounter per visit and at least one order per encounter that would each have DX entries attached for clinical care and billing purposes. Minimum and maximum follow-up times were 119 and 1,185 days, respectively, with an average follow-up time of 654 ± 308 (mean ± standard deviation). Order DX entries contained 1,899 (18.9%) accurate records and 1,536 (15.3%) inaccurate records; the remaining were suboptimal. There were 180 visit providers, 108 order-authorizing providers for 27 different kinds of orders, and 431 distinct users over 63 departments, across 8 care locations, covering 23 clinical specialties recorded in this data set. Encounter DX entries contained 712 (19.1%) accurate DX and 522 (14.0%) inaccurate DX, recorded in 66 departments across 8 care locations, covering 28 clinical specialties by 162 visit providers.

TABLE 1.

Data Set Size and Features

Correlation analysis revealed relatively strong relationships between workflow variables and accuracy (Fig 1). Department and care location had high correlation for most outcomes on both data sets (ie, order and encounter DX). Department presented 0.31, 0.26, and 0.30 correlation values for order accurate, inaccurate, and suboptimal DX, respectively. Care location presented 0.31, 0.21, 0.32 correlation values, respectively. Clinical personnel variables also returned higher correlation values. User entering the data (R² = 0.3, 0.34, 0.36), visit provider (R² = 0.29, 0.33, 0.39), and authorizing provider (R² = 0.16, 0.11, 0.16), respectively, had high correlation values for all outcomes for order DX data. For encounter DX data, department (R² =0.31, 0.21, 0.32), care location (R² = 0.30, 0.20, 0.27), and visit provider (R² = 0.15, 0.20, 0.19), respectively, had relatively high correlation values. All high correlation values were significant (P < .0001).

FIG 1.

Correlation analysis results. DX, diagnosis.

We found significant statistical relationships between clinical workflow and user variables in the order DX data set (Table 2). For clinical workflow variables, DX entered in a nononcology department (eg, surgery, magnetic resonance imaging, outpatient laboratories, and so on) had 2.89 times higher odds of being fully accurate (ie, correct and specific histology and anatomic site; adjusted [adj]-P < .0001) and 49% lower odds of being suboptimal (adj-P < .0001) compared with oncology departments (eg, oncologic hematology, radiation oncology, and so on). We also found that DX data recorded at care locations where patients were seen regularly presented increased accuracy and reduced inaccuracy odds. For example, a DX entered at a care location including a cancer survivorship center had 68 times higher odds of being fully accurate (adj-P < .0001), was 95% less likely to be inaccurate (adj-P < .0001), and had 62% lower odds of being suboptimal (adj-P < .0001) compared with the main campus care location, which included the comprehensive cancer center. Similar results were found for a care location including a geriatric outpatient clinic and a chronic disease management facility. DX entered at these facilities had 19 and 8 times higher odds of being accurate (adj-P < .0001), respectively, 88% and 62% lower odds of being suboptimal (adj-P < .0001), respectively, and 98% lower odds of being inaccurate in the chronic disease management center (adj-P < .0001). Imaging facilities were the only care locations with significantly less accurate logging. A DX entered at an imaging facility location had 99% lower odds of being accurate (adj-P < .0001) and over 400 times higher odds of being suboptimal (adj-P < .0001) compared with DX data recorded at the main campus. For our user variable, we found that a DX entered by a physician assistant had 16% lower odds of being fully accurate (adj-P < .0001) but only 9% higher odds of being inaccurate (P = .0015) compared with DX entered by physicians. We also found differences in the logging habits of pharmacists. DX entered by users with pharmacist roles in the system tended to be much more accurate (odds ratio [OR], 2.99; P = .012; adj-P = .059) and had 85% lower odds of being suboptimal (adj-P = .005).

TABLE 2.

Order DX Regression Results

We further tested this relationship between clinical workflow variables and data quality by rebuilding this regression using the encounter DX data (Table 3) and by rerunning our initial regression on a data set containing only data within a 90-day range of the BX. We found that a DX entered in a nononcology department had 66% higher odds of being fully accurate (adj-P = 0.008) and 28% lower odds of being suboptimal (adj-P = .025) compared with oncology departments. A DX entered at a care location including a cancer survivorship center had over 36 times higher odds of being fully accurate (adj-P < .0001), 89% lower odds of being inaccurate (adj-P = .017), and 50% lower odds of being suboptimal (adj-P = .003) compared with the main campus care location. DX entered at a geriatric outpatient clinic and a chronic disease management facility had over 7 and 3 times the odds of being accurate (adj-P < .0001), respectively, 96% lower odds of being inaccurate (adj-P < .0001) for the chronic disease management center, and 50% lower odds of being suboptimal in the outpatient geriatric care facility (adj-P = .003). A DX entered at an imaging facility had 95% lower odds of being accurate (adj-P < .0001) and 66 times higher odds of being suboptimal (adj-P = .025) compared with DX data recorded at the main campus. We also rebuilt our models for data sets containing data for the first 90 days after each patient’s BX to further confirm our findings. The resulting model for order DX data confirmed our finding on user differences revealing differences between physicians and physician assistants (OR, 6.3, adj-P < .0001 for accurate DX; OR, 0.38, adj-P < .0001 for inaccurate DX; OR, 0.33, adj-P < .0001 for suboptimal DX). This model also revealed the same differences for clinical departments (OR, 3.5_; adj-P < .0001 for accurate DX; OR, 0.44, adj-P < .0001 for suboptimal DX) and for the outpatient geriatric care facility (OR, 8.0, adj-P < .0001 for accurate DX; OR, 0.44, adj-P < .0001 for suboptimal DX). Interestingly, we found the opposite effect for the chronic disease management outpatient facility (OR, 0.47, adj-P < .0001 for accurate DX; OR, 6.8, adj-P < .0001 for suboptimal DX) hinting at a transient effect where such clinics might be more likely to record less accurate and more suboptimal DX data early in cancer treatments. Rebuilding the model for encounter DX confirmed that geriatric care facilities were more likely to record accurate DX (OR, 10.8; adj-P < .0001) and less likely to record suboptimal DX (OR, 0.29; adj-P = .035).

TABLE 3.

Encounter DX Regression Results

DISCUSSION

We used statistical regressions to uncover potential relationships between accurate, inaccurate, and suboptimal DX data entry in oncologic EHRs and clinical workflow-describing variables, such as care location, clinical department, and EHR user types. We found that both clinical department and care location were predictive of accurate or suboptimal DX entry. Care location was also predictive of inaccurate recording. We also found that a user’s clinical role predicted the odds of accurate, inaccurate, or suboptimal entry. Our findings support the hypothesis that clinical workflow factors affect the accuracy of clinical data recording; they also suggest that there may be significant differences across clinical workflows and clinical personnel’s logging habits. Clinical oncologists using EHR data to make patient care decisions or reusing clinical data to develop clinical decision support tools should be aware of this variability.

Our study expands beyond existing literature by exploring DX code assignment beyond accuracy^1,22,47 and shifts the view of static clinical data as raw analysis material to data as the product of clinical workflows. Our findings are congruent with the existing literature^15-17,36 but also unlock new dimensions of oncologic data quality assurance. We provide quantitative evidence of the heterogeneous nature of clinical workflows and, most importantly, their impact on clinical data quality. This work also provides preliminary evidence to suggest that different clinical departments, care locations, and clinical roles may have different data logging, which we were not able to find reported quantitatively in prior publications. This is the core contribution of our analysis.

Our findings hint at differences in the ideal level of DX specificity across clinical workflows, care locations, and clinical departments. One interesting finding is the higher degree of accuracy and lower suboptimal logging odds in nononcology departments, potentially explained by oncologic progress note accessibility at oncologic departments. There may be a reliance on unstructured clinical data for information foraging⁴⁸ during clinical practice at these sites. This is congruent with the idea that the most accurate and precise DX is contained in the clinical progress note.^49-51 However, this raises questions about the accessibility of oncology notes to users outside oncology departments and in the postclinical data lifecycle.⁵² Current clinical data reuse research seeks to develop data quality assessment methods separately from clinical practice.^6,53-64 However, our work shows that clinical data and clinical workflows are closely related and should be viewed as a product and production process. There may also be a link between interface design and entry accuracy, given that order and encounter entry interfaces are different in our EHR system. This may counter current data aggregation and warehousing trends^65,66 but would allow for a data quality assessment and reuse approach more in tune with the paradigm of learning health care.⁶⁷

Our analysis has 5 core limitations that will be addressed in future work. First, we analyzed data for 36 patients with cancer. Because we relied on a clinician-defined gold standard, developing a larger cohort would have been labor intensive and cost prohibitive at this stage. Still, our final data set provided adequate statistical power to test our hypothesis and previous work.³⁷ Second, we relied on data from a single institution, which may limit the external validity of our findings. We will address this limitation in future work by replicating our analysis at multiple sites. Third, we had a simple definition of DX accuracy. The definition was based on information available in our EHR and information needed to identify patient charts for secondary analysis. This is in concordance with the clinical data quality literature,^68,69 which recommends accuracy definitions to fit the intended data use. Fourth, limited clinical workflow and user factors were explored because of availability and analytic method limitations; other variables will be explored in future work. Finally, we only studied one type of cancer. Future work will include reproducing this analysis for other patient cohorts to assess generalizability. We will use a larger cohort of patients and cohorts diagnosed with other oncologic conditions.³⁷ We will also explore the availability of user interaction data within our EHR database to better understand whether data entry modes (eg, drop-down menu selection and search interfaces) have an impact on the accuracy of DX data entry in oncologic EHRs. Additional secondary analyses of clinical data using analytic methods, such as machine learning⁷⁰ and simulation modeling techniques,⁷¹ will also be conducted.

In conclusion, clinical departments and care locations were predictive of DX data quality. A user’s clinical role (eg, physician, assistant) was also predictive of accurate, inaccurate, and suboptimal DX data recording. Clinical oncologists using EHR data to make care decisions or reusing data to develop clinical decision support tools should take such differences into account. Additional analytic work is needed to tease out this heterogeneity in clinical data recording.^1,2,5

AUTHOR CONTRIBUTIONS

Conception and design: Franck Diaz-Garelli, Roy Strowd, Brian J. Wells, Umit Topaloglu

Financial support: Franck Diaz-Garelli, Umit Topaloglu

Administrative support: Franck Diaz-Garelli

Provision of study materials or patients: Franck Diaz-Garelli, Roy Strowd, Umit Topaloglu

Collection and assembly of data: Franck Diaz-Garelli, Roy Strowd, Virginia L. Lawson, Umit Topaloglu

Data analysis and interpretation: Franck Diaz-Garelli, Roy Strowd, Maria E. Mayorga, Brian J. Wells, Thomas W. Lycan, Umit Topaloglu

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).

Roy Strowd

Consulting or Advisory Role: Monteris Medical, Novocure

Research Funding: Southeastern Brain Tumor Foundation, Jazz Pharmaceuticals, National Institutes of Health

Other Relationship: American Academy of Neurology

Virginia L. Lawson

Employment: Walgreens

Stock and Other Ownership Interests: Walgreens

Brian J. Wells

Research Funding: Boehringer Ingelheim (Inst)

Thomas W. Lycan

Travel, Accommodations, Expenses: Incyte Corporation

Open Payments Link: https://openpaymentsdata.cms.gov/physician/4354581

No other potential conflicts of interest were reported.