Strategies for working with pragmatic clinical trial observational data—lessons learned from the Pain Management Collaboratory

In 2017, an inter-governmental agency partnership between the National Institutes of Health (NIH), the Department of Defense (DOD), and the Department of Veteran Affairs (VA) was announced to support multimodal pragmatic clinical trials (PCTs) targeting nonpharmacologic management of chronic non-cancer pain among active service members and Veterans, entitled the NIH–DOD–VA Pain Management Collaboratory (PMC).^1,2 PCTs are usually embedded in routine healthcare workflows and leverage routinely collected data from electronic health records (EHRs) to reduce data collection costs and facilitate replication.^3,4

While the DOD and VA have long-established EHR systems, there were significant limitations in the collection and use of EHR data within the PMC PCTs. These data are largely limited to routinely documented ratings of pain severity and pain-relevant diagnoses, tests and procedures, clinical encounters, and medications, with a large portion of the information in narrative text.

For the PMC Coordinating Center (PMC³), the EHR work group (WG) was 1 of 7 WGs established to support the program. The WG’s responsibilities included supporting the PCTs understanding, optimizing, and documenting the use of EHR data and supporting prospective data collection and EHR-aligned interaction with clinical care delivery. Here, we describe key activities and lessons learned within the EHR WG of the PMC³ as they evolved based on client research project use, interest, and engagement.

EHR WG—structure and governance

The WG membership was represented by personnel funded by the coordinating center and each of the participating projects (supported through the projects). The WG leadership team included 2 co-chairs who were VA clinicians with expertise in VA EHR infrastructure and systems, a DOD representative with expertise in the DOD EHR and Military Health System Data Repository (MDR), and a PMC³ project manager to provide organization. The remaining WG membership included PMC project principal investigators and technical faculty and staff involved with EHR data collection, integration, and synthesis. All projects used EHR data in some way, such as demographics, medication, or laboratory data.¹

EHR WG support was the most frequent during the initial 2 years, and meetings were initially monthly, then bimonthly, and lastly as needed. WG activities were communicated to PMC-wide audiences through meetings with leadership and PMC sponsors, monthly reports to the PMC³ Steering Committee, and cross-WG coordination meetings.

The EHR WG leadership team coordinated with projects to compile key observational data elements each intended to use. During initial meetings, the WG updated the scope and goals to reflect client project needs, which were: (1) information and knowledge for retrospective and prospective EHR and patient-reported outcomes data collection,⁵ (2) guidance on harmonization of definitions for data elements to align between VA and DOD projects where possible, and (3) the need to support cross-project harmonization of data element definitions and transformation processes.

We found this was a critical step to establish the working goals and processes to support projects. This allowed us to understand the general data requirements for each project, when EHR data support was needed, and identified special needs for information gathering or data collection support.

EHR situational awareness

During this project, the VA and DOD were modernizing their EHRs and migrating to commercial applications. Both chose Oracle Health (previously Cerner) as the vendor, but implementation strategies differed in timelines and how each organization modeled and harmonized data. Because of concerns for unexpected project impacts, the WG provided regular updates on changes to legacy and Cerner systems in both environments. WG co-chairs held organizational leadership roles that facilitated inviting national EHR technical leaders to provide updates directly to researchers in the WG meetings and facilitated individual project meetings with those leaders to help resolve questions around data requirements. This highlighted the utility of identifying and recruiting organizational leaders to provide championship for project execution, particularly when unusual data access or active prospective data collection is required.

Clinical integration and prospective data collection

The EHR WG provided updates and guidance on interacting with live clinical environments, which included getting approvals for tools, forms, and prospective data collection or data/information delivery. They also facilitated project members quickly gaining data access and identifying and implementing software tools or solutions that satisfied user needs and complied with privacy and security requirements.

One example of this was a VA trial that sought to improve Veteran access to multi-model low back pain management strategies. This trial used operational data to provide reports to enrolled sites about their clinical progress, and the investigators were interested in establishing data storage and visualization tools for research team review. The WG leadership team made an inquiry on the study’s behalf to operational IT partners, who then supported with project team with guidance on how this environment would be set up as well as data ingestion and transformation for the desired use.

As these activities required more stringent approvals and understanding of the processes of healthcare IT, the organizational leadership roles in health IT operations, privacy, and security that WG leadership team held were strong facilitating factors in providing this critical support. We recommend ensuring that WG members have organizational roles that facilitate project support.

Retrospective EHR data collection support

While prospective integration with EHR resources was more challenging than retrospective data use, this category of support comprised a majority of EHR WG consultations. Projects requested identification of data storage locations and clinical variable definitions. The WG reviewed best practices within the VA and DOD data warehouses and provided guidance for selection and transformation of data sources.

An example of this type of support was a DOD trial that included assessment of how behavioral health consultants delivered care and impacted patient outcomes following a chronic pain training program. The researchers requested support for data variable assessment, collection, and transformation, and the WG provided a detailed review of definitions and access policies for DOD data resources. The MDR had limited reporting of ambulatory diagnostic/procedural codes and pain measures needed for the study, so WG leaders facilitated the project working with DOD Health Information Technology experts to obtain additional data. This request also highlighted collaboration with the Data Sharing WG, and the 2 WGs facilitated recommendations and support for a data sharing agreement.⁶

Another category of request was to receive standardized code libraries that could be used across research projects. Each project had its own data analysis personnel, and the group meetings functioned to engage members in group problem solving to establish code repositories and variable harmonization. While useful, the major challenges in supporting this activity were (1) that projects had different needs regarding when they needed EHR data, and (2) some projects had well-established data transformation methods, and changing these processes presented programming costs and risks.

An example was the implementation of a standardized morphine equivalent daily dose (MEDD) calculation, which was a common requirement across projects. This resulted in an important EHR WG effort, which was based on established clinical guidelines and adapted to VA and DOD source data. This highlighted the need to partner with other WG’s for requests with shared scope, and together with the Phenotypes and Outcomes (P&O) WG, we wrote and evaluated the data analytic code for MEDD, provided it to pilot project users for validating and updating, and then circulated across the consortium. The challenge was engaging clinical subject matter experts and translating documentation variation in dose and frequency of medication fill records. The lesson learned was to identify common variable and phenotype definitions early to facilitate harmonization as quickly as possible to minimize potential inertia challenges in using shared code repositories.

In addition, some projects wished to use VA and DOD together, but aligning these data in a usable form was challenging. The WG consultants developed data harmonization crosswalk recommendations between VA and DOD EHR data elements for inpatient and outpatient care received both from direct care and as a third-party payer. This crosswalk included harmonized metadata, procedures, specialty care, and telehealth codes. These products were provided to project members in the EHR WG for their review and questions and then made available across the PMC.

Lastly, the WG was originally scoped to provide guidance for leveraging natural language processing tools within projects that automated text extraction, but these services were not requested, primarily because projects using NLP already had well-established capacities in this area as a condition of successful funding.

We recommend that WG leadership and members have experience in collecting and transforming EHR data. Most importantly, we recommend ensuring that WG meetings are structured to communication learnings and relevant information among projects.

Conclusions

We have provided a summary of the organizational structure, challenges, and lessons learned with examples of services delivered as part of the PMC³ EHR WG. Our findings are consistent with prior experience, in which multiple challenges were noted for using EHR data in PCTs.^7,8 We addressed research project challenges such as identification and implementation of phenotype definitions, data cleaning and transformation, and data quality assessments. An effective strategy was to engage in a mix of group and individual project support activities and oscillating the intensity of support as projects moved through their lifecycles.

Funding

Research reported in this publication was made possible by Grant Number U24 AT009769 from the National Center for Complementary and Integrative Health (NCCIH) and the Office of Behavioral and Social Sciences Research (OBSSR). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NCCIH, OBSSR, and the National Institutes of Health.

Conflicts of interest: The authors have no conflicts of interest to disclose.

Supplement statement

This article appears as part of the supplement entitled “Pain Management Collaboratory: Updates, Lessons Learned, and Future Directions.”

This manuscript is a product of the NIH–DOD–VA Pain Management Collaboratory. For more information about the Collaboratory, visit https://painmanagementcollaboratory.org/.