Research-grade data in the real world: challenges and opportunities in data quality from a pragmatic trial in community-based practices

Abstract

Randomized controlled trials face cost, logistic, and generalizability limitations, including difficulty engaging racial/ethnic minorities. Real-world data (RWD) from pragmatic trials, including electronic health record (EHR) data, may produce intervention evaluation findings generalizable to diverse populations. This case study of Project IMPACT describes unique barriers and facilitators of optimizing RWD to improve health outcomes and advance health equity in small immigrant-serving community-based practices. Project IMPACT tested the effect of an EHR-based health information technology intervention on hypertension control among small urban practices serving South Asian patients. Challenges in acquiring accurate RWD included EHR field availability and registry capabilities, cross-sector communication, and financial, personnel, and space resources. Although using RWD from community-based practices can inform health equity initiatives, it requires multidisciplinary collaborations, clinic support, procedures for data input (including social determinants), and standardized field logic/rules across EHR platforms.

IINTRODUCTION

The HITECH Act has spurred electronic health record (EHR) adoption and innovation over the past decade,^1–3 accelerating the use of real-world data (RWD) gathered as part of clinical care to support meaningful use and to inform health policy and regulatory decision-making within the United States.^4–6 RWD broadly include EHR, medical claims, and patient-generated data in contrast to data obtained from clinical trials designed to answer specific research questions.⁶ Population health and medical research has relied on randomized controlled trials (RCTs), the gold standard for inference on intervention effectiveness.^6,7 Greater investment in pragmatic trials⁸ have enabled health care settings to serve as learning laboratories to assess intervention efficacy and compare treatments in real-world settings.^9–12 Pragmatic trials improve trial engagement and generalizability to broader settings and more diverse populations by minimizing burden of data collection while harnessing existing clinical populations using EHR data.¹³

For example, much of the literature using EHR-based RWD has been generated from large health care systems.^14–16 However, community-based practices serve a large proportion of immigrants and minorities, particularly in urban settings.¹⁷ These practices serve patients with high social and medical complexity; a substantial proportion of patients pay with public insurance which generates lower care reimbursement, resulting in fewer resources at their disposal.¹⁸ The inability to account for patient diversity (eg, language, culture)^19–21 may affect generalizability to diverse populations and, therefore, limit external validity.²² Few studies have explored the unique aspects of pragmatic trials in community-based practices, including the acquisition and quality of EHR data for meaningful use.

This report describes the challenges and lessons learned through Project IMPACT (Implementing Million Hearts for Provider and Community Transformation)²³ in acquiring EHR-based RWD from small community-based practices serving largely South Asian populations and implications for analysis and interpretation. These practices face challenges inherent to delivering care to a medically complex population with limited health care and financial resources.^24,25 South Asian communities experience a disproportionate burden of cardiovascular disease risk factors, including comorbid conditions, compared with other racial/ethnic groups.^26–31 This report highlights the unique barriers to research in geographically disparate community-based practices serving immigrant populations and the impact of these barriers on using RWD to evaluate EHR interventions.

THE CASE

Project IMPACT is a 5-year stepped-wedge quasi-experimental study testing the feasibility, adoption, and impact of an EHR-based health information technology (HIT) intervention to improve hypertension control among patients in 16 small New York City practices serving South Asians.²³ These underresourced practices employ 2 full-time physicians seeing 178 patients per week on average, with nearly 75% Medicaid-based revenue, and 88% of patients speaking a primary language other than English.³² The initiative involves a multi-stakeholder collaboration between community-based clinics, a Medicaid payer, and academic institutions (Figure 1). With stakeholder input, the project team trained providers to generate hypertension-based patient registry reports and medical alerts and order sets tailored to South Asian patients. They also provided ongoing technical assistance. The main outcome was clinic-level hypertension control, defined in Table 1. The academic team collaborated with 2 EHR vendors to acquire data; they also collected data from clinic EHRs directly. Additional intervention details are published elsewhere.²³

Figure 1.

Collaborative partners and associated roles. Abbreviations: CDC, Centers for Disease Control and Prevention; EHR, electronic health record.

Table 1.

Challenges faced by small community-based practices and IMPACT strategies to address them

Challenges	Implications	Strategies
Challenge: Data accuracy—Capabilities of built-in registries/queries
Capabilities of EHR Systems
Built-in EHR registry functions could not pull accurate patient lists matching our outcome definition. The main outcome was clinic-level hypertension control (proportion of patients in the previous 6 months with an ICD diagnosis of hypertension [HTN] and elevated blood pressure [BP] [≥ 140/90 mm Hg] at the last visit). For instance, when defining HTN control as BP reading at last visit in a given time period, the registry within 1 EHR platform only evaluated the most recent BP reading—not the last BP reading in a previous time period (see Figure 2). The registry function in the other EHR platform could only evaluate against any BP value in a time range, not the last value in a range. One of the 2 EHR platforms allowed providers to create their own ICD codes, resulting in multiple versions of same ICD code (Table 2).	If unresolved, this deficiency would produce an artificially high proportion of patients with controlled hypertension for previous time periods, because returning patients were more likely to have controlled hypertension. Registries and reports to obtain number of patients with a particular ICD code needed to account for additional nonstandard ICD codes to avoid underestimating the number of patients with hypertension. For example, at 1 clinic, when using 1 standard ICD-9 code for hypertensive diagnosis (401.9), our report yielded many fewer patients than when we included all nonstandard codes (Table 2) for the same diagnosis.	Research team needed to be nimble in practice and adjust data query specifications as needed. Acquired data directly from the EHR vendor, requiring additional grant resources. Our custom reports using add-on business-based software allowed us to account for these additional codes.
Acquiring Data from EHR Vendors
When necessary, the research team commissioned data reports from the EHR vendors. However, due to communication barriers with report developers and the lack of focus/resources by the EHR vendors on research, it was a challenge to communicate data specifications, and we experienced prolonged time to acquire valid data reports.	Because the research team collected and analyzed data from 2 EHR systems, it was imperative that the report specifications were comparable between the EHR vendors in order to integrate data from both systems. For example, ensuring that the report identified the BP value at last office visit in a given time period versus the BP value ever measured.	Scheduled times when the EHR developer could modify the report in real time while research staff validated the report, reducing time to final report. Validated subsets of reports against manually collected data to identify discrepancies
Challenge: Data accuracy—Data entry practices and EHR field availability, logic and rules
EHR System Design
The clinic EHR systems did not contain fields and/or did not mandate field entry for social and cultural factors known to influence health. For example, EHRs in this study provided limited options for racial/ethnic categories. Most patients were captured under the “Asian American” or “Other” categories. Asian Americans are a diverse population, including immigrants from South Asia, China, Philippines, etc., with unique social/cultural influences on health.	IMPACT was not able to account for most social factors in the evaluation of practice level outcomes, including ethnic subgroup when assessing the effect of the EHR intervention on clinic-level hypertension prevalence. IMPACT was not able to determine if the practice-level intervention had a differential effect on outcomes by racial/ethnic subgroup.	Used proxies (eg, surnames and language) to identify South Asian patients for a separate study component which assessed the impact of community health worker-led coaching on improving patient blood pressure control.
Provider/Staff Documentation Practices
Language preference, among other social and cultural determinants, was not systematically documented across practices.	IMPACT was unable to account for language preference (and many other social factors) when assessing the effect of the EHR intervention on hypertension prevalence at the clinic level; thus, the role of language preference in EHR intervention efficacy is unclear.	Encouraged providers to document language preference for all patients upon check-in; however, fidelity varied. Conducted a study among a subsample of the clinic population, which collected data on many important social determinants.
Challenge:Access to data—Resource intensive
Geographic Distance between Participating Clinics
Clinics were located throughout NYC in geographically disparate neighborhoods where there was a greater proportion of South Asians. Manual data collection from clinics throughout a geographically wide area was time-intensive for research staff.	Data that were not collected, or were not collected on time in some cases due to staff resource limitations, could have biased the estimate of intervention efficacy.	Trained CHWs to collect data using structured tools and remote guidance from research staff. Built trust with providers and considered gaining permission for EHR remote access.
Research Staff Time and Expertise
To develop systems to acquire accurate data, research staff needed to have ample knowledge of the unique EHR systems, including field availability, navigation, and reporting functions. Research staff needed epidemiological expertise to match the technical capabilities of the EHR system with the operational outcome definitions. Initially, the collection of counts of hypertensive patients with controlled BP for a single time frame took 2+ days. After a months-long process of creating and deploying costly business-based software, this time was reduced to 30 minutes/clinic per time period.	Without careful attention to designing a data collection system that produces data that match the operational definition of the outcome and are equivalent across EHR systems, the estimate of the effect of the intervention on the outcome may be inaccurate. This process evoked the challenges in communication with EHR vendor–developers described above and required some clinics to upgrade their servers and EHR platforms.	Employed research staff with graduate level epidemiology training and experience with the clinic EHR systems to design the collection systems and develop custom reports from EHR vendors. Consulted with EHR experts to identify EHR reporting capabilities and opportunities. Considered “train the trainer” models to enhance CHW skills in using the EHR to run registries and carefully considered the implications of acquiring data remotely.
Physical Space
Data collection needed to occur in person at clinics, often on a specific date; however, the clinics were very busy and operated within small physical spaces. Finding space and scheduling an appropriate time for data collection was a challenge at the busiest and (physically) smallest clinics.	If data were not collected on time, we risked over- or underestimation of the outcome. In this case, the outcome was a proportion of patients with controlled hypertension in the past 6 months. This would threaten validity of intervention efficacy results. Physical space demands from the research team could strain relationships with the clinic and lead to reduced access to follow-up data.	Developed data collection protocols that took the least amount of in-person time. This included creating parsimonious collection templates and identifying alternatives for data acquisition that involved partnerships with EHR vendors. Established some flexibility around time-sensitive data collections (eg, +/− 3 days)
Challenge: Access to data—Relationships and collaboration with Clinics
Clinic Concerns about the Privacy of Patient and Clinic Data
At times, clinics expressed concern over the privacy of their patient and clinic data. For example, while partnering with an insurance agency lent credibility to the project, the research team needed to address concerns over whether the researchers and insurance company were sharing information.	These concerns affected the intervention progress and effectiveness and limited access to data collection. Timely data collection was requisite to obtaining accurate data.	Drew from principles of CBPR to build trust. Employed research personnel from the target population facilitating trust building. Created 1-page BP control progress reports for the clinics. Offered opportunities for publication authorship for practice staff.
Clinic Staff Training and Turnover
IMPACT provided training on running registry reports, however frequent clinic staff turnover made it difficult to maintain learned knowledge and rapport. Clinics using 1 EHR were instructed to manually save the reports to document use, but this was uncommon.	Inability to account for the frequency of use of registry reports Saving registry lists to the computer raised concerns over the security/privacy of patient data	Provided ongoing technical support visits over the first-year, post-intervention. Offered training on running and saving registry reports by IMPACT staff at each data collection visit (biannually).

Abbreviations: BP, blood pressure; CBPR, community-based participatory research; CHW, community health worker; EHR, electronic health record; HTN, hypertension; ICD, International Classification of Diseases; IMPACT, Implementing Million Hearts for Provider and Community Transformation; NYC, New York City.

CHALLENGES OF RWD IN A COMMUNITY-BASED SETTING

Most studies using EHR data for research and quality are from larger health care systems.^14–16 Using EHR data to evaluate and inform quality improvement initiatives in small, underresourced community-based clinics evokes unique challenges, opportunities, and implications for health equity.

Data accuracy: capabilities of built-in registries/queries

In pragmatic trials, data accuracy is paramount for valid findings that are applicable to the settings of interest.²² During validation procedures for IMPACT, we discovered that EHR registry functions could not pull accurate patient lists that matched our outcome definition, thereby compromising the validity of findings (Tables 1 and 2, Figure 2) and limiting the utility of reports for clinic staff. Lack of registry functionality and commonalities across EHR platforms has been documented as a challenge in quality initiatives.^33,34 Small practices often lack resources to customize registries and reports, and vendors are often resistant to making changes.³⁵ Additionally, these challenges in data accuracy also trace back to documented limitations of using clinical classification systems (eg, ICD-10)—which were originally created to code death certificates—to assess quality initiatives.³⁶

Table 2.

Examples of standard ICD-9 codes and associated nonstandard ICD codes

Standard ICD-9 Code	Examples of Additional Nonstandard ICD Codes*
401.9 Unspecified essential hypertension	401.9 Benign hypertension 401.9 Continued arterial hypertension 401.9 HTN [Hypertension] 401.9 Hyperpiesia 401.9 Hyperpiesis 401.9 Hypertension 401.9 Hypertensive disease NOS 401.9 Hypertension NOS 401.9 Huchard’s disease 401.9 HTN

^* Note: Same ICD code, but different labels and therefore different underlying codes.

Abbreviation: ICD, International Classification of Diseases.

Figure 2.

BP value assessed on 12/31/2018 using 2 methods (custom data query vs built-in registry report) to identify patients with uncontrolled HTN between 1/1/18 and 6/30/18. Abbreviations: BP, blood pressure; HTN, hypertension.

To obtain accurate data, we worked with the EHR vendors to acquire data and create custom reports using add-on software. We encountered challenges of creating a shared language, communicating data specifications, and operationalizing it in the registry due to differences in technical expertise. Specifically, we were less versed in the technical language of creating custom data queries and EHR vendors were less versed in scientific terminology and its application. Further, validation of the reports required lengthy on-site testing at the clinics, where space and computer resources were limited (Table 1).

Data accuracy: data entry practices and EHR field availability, logic, and rules

EHR systems are designed for patient management and not for research. Studies using EHR data are limited by provider entry practices and the EHR data entry form, content, and field logic/rules, often yielding incomplete data that vary by practice and EHR vendor. Researchers using observational designs are interested in adjusting for potential confounding effects. However, providers may not systematically enter data on demographic, behavioral, and social factors in the EHR. For instance, language preference was not systematically documented (despite training) and neither EHR platform provided options for racial/ethnic subgroups (Table 1). Further, differences in structured fields across EHR platforms often limit the ability to aggregate data and evaluate outcomes across practices.³³ This challenge has been documented in regard to pragmatic trials in clinics more generally,²² but it is exacerbated in practices serving immigrant communities because these confounders have a greater influence on outcomes. As social determinants account for health inequalities, poor documentation of these factors threatens the validity of findings on intervention effectiveness; this ultimately limits our ability to target and tailor interventions to high-risk patients.

Access to data: resource intensive

A primary goal was to collect accurate data with minimal practice workflow interference. The barriers to obtaining RWD from the EHR vendors necessitated considerable resources across several dimensions: 1) human resources, including research team time and expertise, 2) clinic space and location, and 3) access to computers/EHR systems. For the research team, gathering data manually from clinics throughout a geographically wide area was time-intensive. Intermittently slow internet/computer speeds and lack of space and computers also increased data collection time. Further, research staff designing and validating the data collection protocols needed to be well-versed in navigating multiple EHR platforms and outcome definitions to identify data discrepancies. For example, the research team undertook a months-long process of creating and deploying costly software reports to reduce the time needed to collect 1 data point from 2 days to 30 minutes, and training community health workers (CHWs) to collect data (Table 1).

The clinics themselves were similarly over-extended. Few staff resulted in a high patient load per provider, and practices had a high proportion of walk-in patients. Data access was limited by few computers, constrained space, occasional slow/non-responsive internet, and other technical issues slowing workflow. Thus, scheduling data collection that minimized workflow disruptions was challenging, though data collection was facilitated if clinic personnel had a strong interest in HIT solutions or had participated in past quality improvement initiatives.

Access to data: relationships and collaboration with clinics

Developing good rapport and a trusting relationship between the researchers and providers were paramount to addressing challenges of collecting RWD and ensuring meaningful use in research and clinical practice. Concerns about the privacy of patient data are a challenge to establishing this relationship in any trial.²² Providers needed to trust research staff to maintain data privacy and confidentiality and to ensure that their involvement would not affect reimbursements or licensing. Our project’s partnership with a large health maintenance organization lent credibility to the project but also generated concerns about whether the researchers and the payer were sharing information. Like other pragmatic trials,¹⁸ we also saw substantial clinic staff turnover, which made it difficult to establish trusting relationships and sustain learned knowledge over the multi-year project (Table 1).

To surmount these challenges, IMPACT drew from principles of community-based participatory research, including creating opportunities for communication, bidirectional learning/input, data-sharing data, and adapting protocols to accommodate organizational context and workflow. IMPACT also employed CHWs from the target population to facilitate trust-building between researchers, practices, and patients (Table 1).

OPPORTUNITIES AND FUTURE DIRECTIONS

While there is growing interest in how RWD can improve and accelerate the development of medical products and therapeutics,^4–6 there is limited focus on using RWD to evaluate community-based interventions addressing health disparities. We describe opportunities to improve the acquisition of high-quality RWD from resource-constrained practices in the following sections.

Informatics solutions: EHR integration and compatibility

Researchers and policy makers have sought to ensure that EHR features, fields, and functions align with quality targets and are similar across vendors to ensure connectivity and seamless integration of data.^33,35 Policies promoting collection of standardized data measures across EHR platforms will facilitate data harmonization across diverse practices and systems,³³ and thereby provide the capability for real-time assessment of care quality and data to inform health policy.

Various incentives are increasingly focused on inclusion of social determinants of health^37–39 in EHR platforms,⁴⁰ which is particularly important in supporting the health of vulnerable populations. The power to adapt EHR entry templates and structured fields to specific patient populations through HIT would allow for the ability to better characterize the population and examine intervention efficacy. Engaging EHR vendors at the outset of research and in policy discussions around health systems and population health will help gain their support for these changes to EHR platforms.

Payers and larger health system providers are increasingly willing to share data and innovations in extracting, transforming, and integrating data across EHR platforms and other sources into researchable databases (eg, “data lakes”)⁴¹; however, community-based practices serving minority and immigrant communities continue to be left out of these initiatives. Our study shows that data can be extracted to inform public health research and practice generalizable to populations often excluded from traditional clinical studies.

Funding and/or programs for assistance in selecting the most appropriate EHR

EHR selection is a complex, costly, and time-intensive process; and it is particularly challenging for small, underresourced clinics that may only be able to afford a substandard platform or 1 that lacks customization features.^34,35 Even though IMPACT practices adopted EHRs certified for meaningful use, these systems were not able to produce valid clinical quality reports (as other research has found),³⁵ an element crucial to successful patient care management and participating in quality improvement initiatives.³³ Funding and/or programs to assist in EHR selection, customization, and integration into practice would help ensure the production of evidence that is accurate and relevant to the populations served.

Initiatives to engage clinics serving vulnerable populations in pragmatic trials

Municipal and state quality improvement efforts are needed to engage community-based practices in pragmatic trials that also consider contextual factors (eg, staff turnover, cultural factors) to improve engagement in, and generalizability of, findings to diverse populations.¹³ Using initiatives engaged with community-based practices in the same way that large health systems initiatives have been leveraged^42–44 would help identify the barriers and facilitators of implementing and evaluating pragmatic trials of HIT-based quality initiatives. The use of implementation frameworks, such as the Consolidated Framework for Implementation Research (CFIR), can contribute to rapid uptake of innovative solutions in small practices, offering key constructs (eg, outer setting, inner setting, and characteristics of individuals) with which to examine implementation success. In IMPACT, we use a mixed-methods approach and CFIR to evaluate program implementation and engage payers and municipal partners in considerations of program replication and sustainabilty.²³

Engaging stakeholders across the health care spectrum (including academic institutions, providers, EHR vendors, and payers) in designing pragmatic trials and potentially drawing on human-centered design approaches,⁴⁵ is necessary to improve health outcomes and equity. These collaborations are mutually beneficial and allow for access to innovative data sources and expertise to strengthen the credibility of these novel data sets.

CONCLUSION

While larger health care settings have vastly greater IT and analytic resources to support quality initiatives, Project IMPACT demonstrates that it is possible to harness RWD from small community-based practices to support pragmatic and adaptive study designs aimed at improving health equity. We demonstrate the potential to capitalize on RWD for outcome-driven research in such settings as increasing resources are allocated toward pragmatic trials.⁸ IMPACT demonstrates that community-based providers and underserved populations are willing to engage in pragmatic research, and that, given sufficient resources, these trials can help address health inequities. The experience of IMPACT illustrates key lessons learned to optimize RWD use for an array of intervention research to promote health equity.

FUNDING

This publication was supported in part by cooperative agreement U48DP005008 from the Centers for Disease Control and Prevention (CDC), Prevention Research Centers (PRC) Program. NI and CTS contributions were also partially supported by the National Institutes of Health (NIH) National Institute on Minority Health and Health Disparities (NIMHD) grants P60MD000538 and U54MD000538; NIH National Center for the Advancement of Translational Science (NCATS) grant UL1TR001445; and NIH National Institute of Diabetes and Digestive Kidney Diseases (NIDDK) grants R01DK110048. The findings and conclusions in this journal article are solely the responsibility of the authors and may not represent the official view of the CDC, NIH NIMHD, NIH NIDDK, or NIH NCATS.

CONTRIBUTORS

AD developed the concept for this article, developed the results/challenges and examples outlined in the article, led the writing of the manuscript and the acquisition and analysis of practice-level data.

PL provided input on results/challenges and led the acquisition and analysis of claims data from the health insurer. TH contributed to the concept/perspective of this article and integrated a systems design perspective. LT and CTS contributed to the concept/perspective of this article and integrated knowledge on health systems and health equity. NI designed and led the IMPACT study, oversaw data acquisition, contributed to the concept/perspective of this article, and provided input on results/challenges. All authors contributed to interpreting the results/challenges and the broader implications, critically revised the article for intellectual content, and approved the version to be published.

CONFLICT OF INTEREST STATEMENT

None declared.