Invited Commentary: Go BIG and Go Global—Executing Large-Scale, Multisite Pharmacoepidemiologic Studies Using Real-World Data

Abstract

At the time medical products are approved, we rarely know enough about their comparative safety and effectiveness vis-à-vis alternative therapies to advise patients and providers. Postmarket generation of evidence on rare adverse events following medical product exposure increasingly requires analysis of millions of longitudinal patient records that can provide complete capture of data on patient experiences. In the accompanying article by Pradhan et al. (Am J Epidemiology. 2022;191(8):1352–1367), the authors demonstrate how observational database studies are often the most practical approach, provided these databases are carefully chosen to be “fit for purpose.” Distributed data networks with common data models have proliferated in the last 2 decades in pharmacoepidemiology, allowing efficient capture of patient data in a standardized and structured format across disparate real-world data sources. Use of common data models facilitates transparency by allowing standardized programming approaches that can be easily reproduced. The distributed data network architecture, combined with a common data approach, supports not only multisite observational studies but also pragmatic clinical trials. It also helps bridge international boundaries and further increases the sample size and diversity of study populations.

Abbreviation

COVID-19

coronavirus disease 2019

EHR

electronic health record

NIH

National Institutes of Health

PCORnet

National Patient-Centered Clinical Research Network

Editor’s note: The opinions expressed in this article are those of the authors and do not necessarily reflect the views of the American Journal of Epidemiology.

Clinicians and patients often consider many aspects—overall therapeutic performance, patient preferences, predicted adherence, and cost—when choosing among various treatment options. Most therapeutic performance data come from prelicensure randomized controlled trials (RCTs), but these data are often limited to restricted patient populations—for example, those without polypharmacy or multiple comorbid conditions. Many RCTs also focus narrowly on efficacy endpoints demonstrating some benefit over placebo. Developing actionable evidence on infrequent but potentially serious safety risks (e.g., the 3–5 anaphylaxis cases per 10,000 years studied by Pradhan et al. (1)) is challenging, since it usually requires patient enrollment, which is prohibitively expensive. Quantifying these risks requires larger populations when subgroups of patients are expected to respond differently to treatments. All told, once a drug comes to market, there are still considerable knowledge gaps about its benefits or risks in comparison with alternative therapies.

Executing efficient and robust postmarket studies to fill those gaps in a timely and cost-efficient manner is an important undertaking. Ideally, we would like to study multiple effectiveness and safety outcomes simultaneously with head-to-head comparisons among therapeutics for which there is clinical equipoise. Because some outcomes are rare, data from 1 study site alone are rarely sufficient to meet sample-size requirements. Pooling of observational data from multiple sites has emerged as a viable solution to the sample-size issue.

The study by Pradhan et al. (1) appearing in this issue of the Journal is part of a larger effort to better understand the risks and benefits of several second- or third-line antihyperglycemic medications. Clinicians and patients want to know whether a glucagon-like peptide 1 receptor agonist, a dipeptidyl-peptidase 4 inhibitor, or a sodium glucose cotransporter 2 inhibitor is the best choice for them. In their study, Pradhan et al. focus on anaphylactic reactions, a relatively narrow question but an important piece of the overall safety profile of antihyperglycemic treatments.

A concern in this multisite database study, as with most observational studies, is confounding and other biases. For pharmacoepidemiologic studies analyzing routinely collected electronic health data (i.e., “real-world data” (2)), a first, and central, step is choosing databases that are “fit for purpose.” Common choices include electronic health record (EHR) databases, administrative claims databases, or databases that link EHRs to administrative claims. A fit-for-purpose data source includes accurate and complete information on the study treatment, the comparator, the outcome, confounders, and additional covariates (e.g., effect modifiers) that allows researchers to answer the study question. In practice, requiring longitudinal capture of data on health-care utilization is often the minimum requirement to guard against missing data in these important variables. In the paper by Pradhan et al., the authors chose 1 EHR database from the United Kingdom (the Clinical Practice Research Datalink) and 3 administrative claims databases from the United States (1). These data sources have been used in numerous pharmacoepidemiologic studies and are generally considered appropriate to answer Pradhan et al.’s study question, with some important caveats.

In the United States, data gaps in patient records in EHR databases are a predictable result of a fragmented health-care system. It is possible for a patient to visit 2 separate health-care organizations (i.e., EHR data systems) in the same metropolitan area without either of them ever knowing about the other. Hence, the absence of evidence of a particular outcome (especially an urgent one like anaphylaxis) in one EHR system is not a guarantee that it did not occur. Administrative claims databases capture encounter data from multiple health-care organizations but suffer from a different kind of missingness: the in-depth granular information within each health-care visit that might be present in an EHR. While administrative claims databases have longitudinal (i.e., breadth) advantages, EHRs have depth advantages. There are a few exceptions, however. Integrated health-care delivery systems, such as the Veterans Health Administration and the Kaiser Permanente health-care system, are often able to combine their clinical and administrative data. The Clinical Practice Research Datalink, an ambulatory EHR with enrollment in mutually exclusive practice areas, is a close analog of a complete-capture system when linked to hospital records. Unfortunately, the total number of patients covered in these linked EHR-claims database systems is generally quite limited relative to stand-alone EHR databases or administrative claims databases.

In pharmacoepidemiologic studies where prescription medications are often exposures of interest, administrative claims databases are fit for purpose because of the high accuracy and completeness of outpatient pharmacy dispensing data. EHR databases often capture prescribing—that is, an intended assignment of the study drug—but not whether the prescription was filled. On the other hand, administrative claims databases may fall short in the capture of critical confounders and effect modifiers, such as “lifestyle” variables, that are more routinely captured in EHR data. Pradhan et al. demonstrate these differences: Hemoglobin A1c measurements are not available in the administrative databases and may be an important variable that helps measure diabetes severity.

Outcome measurement is an additional challenge, and anaphylaxis illustrates the concern. Historically, anaphylaxis has mediocre performance as a computable phenotype. That is, its approximately 60%–70% positive predictive value did not improve with the introduction of International Classification of Diseases, Tenth Revision, Clinical Modification, diagnosis codes (3). It is difficult to isolate drug-induced anaphylaxis from other common causes, such as food allergies or insect stings, without the unstructured text found in the EHR. In general, unstructured clinical text remains a largely untapped field of useful data. A common remedy for mitigating subpar algorithm performance is to use quantitative bias analysis to check for the robustness of effect estimates in the presence of outcome misclassification. The International Society for Pharmacoepidemiology has created a user-friendly website to enable simple calculations (4).

Ensuring that data are fit for purpose is a prerequisite when using real-world data, but it is not sufficient to generate valid postmarket evidence. In addition to data not being collected exclusively for an RCT, observational database studies also largely lack the advantages that randomization brings in creating exchangeable treatment groups. Outlining the hypothetical “target trial” (5) and using other tools like directed acyclic graphs to reveal potential biases are very beneficial in study design. Prespecifying analyses in a transparent and reproducible way is also of great importance (6).

For many years, multisite studies were performed using a “common protocol approach,” where investigators at individual study sites each created analytical data sets using site-specific programming based on a shared protocol. Pradhan et al. take this approach. While this may be reasonable when conducting a handful of studies, it is not a scalable approach for evidence generation. There is a more cost-efficient way.

In the last 10–15 years, reusable distributed-data infrastructures that leverage real-world data have emerged to generate postmarket evidence on the benefits and risks of medical products (7). These infrastructures, embedded in research-oriented health-care organizations or health plans, support large-scale observational studies and pragmatic or practical clinical trials (8). One of their key advantages is the ability to repurpose existing health data architectures rather than create entirely new study databases repeatedly (9). In pragmatic trials, much of this work is done with the aforementioned common protocol approach, which leverages existing structured administrative claims or EHR data elements to form an analytical data set as described in a shared protocol. Within the National Institutes of Health’s (NIH) Pragmatic Trials Collaboratory, a major initiative to create “learning health-care systems” using real-world data, there have been substantial challenges in standardizing the native clinical data (10). While the use of existing data structures inherent to the health-care organizations was intended to save time and resources, the required amount of data-wrangling for interoperable and research-ready data sets was still a bottleneck in evidence generation.

Rather than rely on still elusive EHR interoperability, it is possible to create a dedicated, standing common data model to facilitate multisite studies. In a common data model, all data are structured (or structurized) and quality-checked in a common format so that a common programming code can be used across all data sites to develop an analytical data set. This code can be posted for transparency and replicability against databases formatted into the common data model structure. While these site-specific patient-level analytical data sets could be pooled together for analysis, they are typically maintained at each site using a distributed network (7). Summary-level information is typically shared to complete the final analysis, or meta-analytical techniques are employed to summarize the site-specific results (11).

A primary driver of the common data model approach has been the substantial investment of US federal agencies to facilitate large-scale multisite pharmacoepidemiologic studies (12). Early federal adopters of the common data model approach were the Centers for Disease Control and Prevention’s Vaccine Safety Datalink (13) and the Food and Drug Administration’s Sentinel Initiative (14). Both of these common data models are closely related to the Health Care Systems Research Network’s Virtual Data Warehouse common data model, which was the pioneering example of a multipurpose, sustainable distributed data network (15). Since then, the NIH Pragmatic Trials Collaboratory has adopted the Sentinel Common Data Model (16), and the National Patient-Centered Clinical Research Network (PCORnet), funded by the Patient-Centered Outcomes Research Institute, has developed the PCORnet Common Data Model (17), leveraging much of the Sentinel Common Data Model as its starting point. Other important common data models include the NIH-funded Informatics for Integrating Biology and the Bedside (i2b2) model (18) and the Observational Medical Outcomes Partnership (OMOP) model (19), which have further accelerated the adoption of the common data model approach.

International adoption by government agencies of common data models in pharmacoepidemiology has also been strong, including the Canadian Network for Observational Drug Effect Studies (CNODES) (20) and the European Medicines Agency’s Data Analysis and Real World Interrogation Network (DARWIN). The migration to a common data model approach from a common protocol has been primarily about gaining study efficiency and statistical power across many sites, minimizing preventable heterogeneity (e.g., different interpretations of the common protocol), while still allowing generation of site-specific estimates (21).

In addition to observational studies, these distributed data network infrastructures also support multisite pragmatic trials. Examples include Sentinel’s Implementation of an RCT to Improve Treatment With Oral Anticoagulants in Patients With Atrial Fibrillation (IMPACT-AFib), which was an individually randomized education intervention geared to patients and providers (22), and PCORnet’s Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-Term Effectiveness (ADAPTABLE), a randomized trial wherein patients were given one of 2 aspirin dosing schedules (23).

It is important to resist the temptation to accept nothing less than perfection in clinical studies. Minimum requirements are that data be fit for purpose and that shortcomings of the data be well-delineated whenever real-world data are employed. Study design tools should be employed to mitigate biases, and transparency is critically important. The good news is that these distributed data networks are growing larger, and cross-country collaborations that increase size and diversify study populations are more easily within reach now than even 5 years ago. The European Medicines Agency, the Food and Drug Administration, and Health Canada have collaborated on multiple coronavirus disease 2019 (COVID-19)-related projects during these last 2 years, including use of novel therapeutics in COVID-positive pregnant patients, use of outpatient glucocorticoids among COVID-19 patients, and research on thromboembolic risk among COVID-19 patients. These large, multicountry database studies that use multiple common data models can improve what we know about the risk-benefit profiles of medications and can ask the salient clinical questions about comparative risk-benefit for patients in clinical equipoise.