Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Mar 1.
Published in final edited form as: Circulation. 2016 Mar 1;133(9):872–880. doi: 10.1161/CIRCULATIONAHA.115.019902

Good Clinical Practice Guidance and Pragmatic Clinical Trials: Balancing the Best of Both Worlds

Robert J Mentz 1,2, Adrian F Hernandez 1,2, Lisa G Berdan 1, Tyrus Rorick 1, Emily C O’Brien 1,2, Jenny C Ibarra 1, Lesley H Curtis 1, Eric D Peterson 1,2
PMCID: PMC4777975  NIHMSID: NIHMS753594  PMID: 26927005

Abstract

Randomized clinical trials are commonly regarded as the highest level of evidence to support clinical decisions. Good Clinical Practice (GCP) guidelines have been constructed to provide an ethical and scientific quality standard for trials that involve human subjects in a manner aligned with the Declaration of Helsinki. Originally designed to provide a unified standard of trial data to support submission to regulatory authorities, the principles may also be applied to other studies of human subjects. While the application of GCP principles generally led to improvements in the quality and consistency of trial operations, these principles have also contributed to increasing trial complexity and costs. Alternatively, growing availability of electronic health record data has facilitated the possibility for streamlined pragmatic clinical trials (PCTs). The central tenets of GCP and PCTs represent potential tensions in trial design (stringent quality and highly efficient operations). In the present manuscript, we highlight potential areas of discordance between GCP guidelines and the principles of PCTs and suggest strategies to streamline study conduct in an ethical manner to optimally carry out clinical trials in the electronic age.

Keywords: pragmatic clinical trial, good clinical practice, clinical trial, ethics

INTRODUCTION

Health care decision makers need evidence-based medicine to support clinical and health policy choices,1 and randomized clinical trials (RCTs) are the highest level of evidence to support these decisions2, 3. Good Clinical Practice (GCP) guidelines were developed to provide an ethical and scientific quality standard for investigators, sponsors, monitors, and institutional review boards (IRBs) throughout each stage of clinical trials4. These guidelines were initially designed to harmonize conduct for clinical trials intending to submit data to regulatory authorities. GCP principles are commonly applied to contemporary clinical investigations of human subjects with the intent of supporting the safety and well-being of study participants. GCP serves “as a roadmap of responsibilities” for those involved in research and can improve the quality and consistency of trial operations5.

In contradistinction, it has been suggested that some of these guidelines if inflexibly applied may result in challenges6. For instance, some GCP processes can lead to markedly increased trial complexity, duration and costs without substantially improving the quality of these trials, their ability to correctly answer clinical questions or support the safety of human subjects7, 8. Furthermore, sponsor interpretation of GCP may complicate trial conduct via implementation of regulatory and monitoring approaches that increase the workload and dissatisfaction of site staff and research monitors.

More recently, the growing widespread availability of electronic health record (EHR) data in community practice has led to the potential to use such data to streamline trials and conduct pragmatic clinical trials (PCTs)9, 10. EHR-based PCTs represent one contemporary strategy to improve the efficiency of clinical trials, reduce costs and support more “real-world” study conduct. While recent EHR empowered trial designs offer remarkable opportunities, there is a potential tension between certain central tenets outlined in GCP guidelines and the core principles of PCTs. In the present manuscript, we highlight potential areas of discordance between GCP guidelines and the principles of PCTs and suggest strategies to balance these perspectives to optimally carry out clinical trials in the electronic age.

The Increasing Cost and Complexity of Contemporary Trials

Many clinical trials in the 1980s and 1990s were characterized by relatively streamlined protocols with assessment of a few hard clinical outcomes (e.g., mortality), modest financial support for sites and few regulatory hurdles. For instance, the International Studies of Infarct Survival (ISIS) studies assessing acute myocardial infarction (MI) therapies used a 1-page case report form without site monitoring, endpoint adjudication or site payments11, 12. In this era, integrity of the trial results were supported by randomization, large sample sizes, and unbiased outcome assessment. However, a number of competing forces including financial incentives and perceived conflicts of interest led to increased trial bureaucracy12. In response, GCP guidelines were developed to ensure patient safety, prevent and/or detect fraud and assure the validity of trial findings. However, these guidelines also led to increased trial complexity and cost. Challenges with participant recruitment and retention, declining funding, and poor engagement of recruiting clinicians placed further strain on the research infrastructure13. With increased documentation and regulatory requirements as well as a strained research structure, clinical trial enrollment shifted from the US and Western Europe to other world regions. Factors promoting the globalization of clinical research and the potential adverse impact of these trends were recently outlined14. The current paradigm shift in clinical trials requires a re-appraisal of GCP guidelines and their applicability to contemporary research.

Several studies from the Tufts Center for the Study of Drug Development have evaluated the cost and complexity associated with study conduct in the contemporary GCP environment. In 2011, Getz et al reported temporal changes in protocol design complexity and study staff burden in more than 8300 clinical trial protocols from 2000 to 2007 (Table 1)15. The typical phase III protocol in the 2000–2003 period had an average of 20 unique procedures compared to 28 unique procedures conducted 5 times in 2004–2007. In cardiovascular trials, there was a nearly 50% increase in total procedures and 30% increase in total work burden comparing these periods. A separate analysis found that for phase III studies approximately 25% of procedures supported regulatory requirements and non-core data with an average direct cost of nearly $2 million (~20% of the total)16. The authors conservatively estimated that the total direct costs for these procedures in active phase II and III studies is approximately $4 billion annually. Intensive study-specific testing not only influences the burden on site staff but also reduces patient participation in clinical trials17.

Table 1.

Temporal changes for phase III protocols across disease states.

2000–2003 2004–2007 Change
Unique procedures 20.0 28.4 42.0%
Total procedures 93.6 147.5 57.6%
Total work burden (units) 27.0 43.1 59.7%
Median CRF pages 55 180 227.3%

Presented as mean or % unless noted.

A recent analysis also assessed temporal changes in the size, duration and enrollment rates for cardiovascular trials published between 2001 and 2012 in select high-impact journals18. Comparing the 2001–2003 period with 2009–2012, trials involved more patients (median from 400 to 500) and sites (from 20 to 22) and enrollment rate decreased from 1.2 to 0.9 patients/site/month. Importantly, low enrollment rates may influence event rates and the validity of trial results19, 20. Median trial duration (2.1 years) did not significantly change over time. Taken together, these temporal changes are due to a number of factors including advances in medical therapy which require larger numbers to demonstrate a net clinical benefit as well as an overall globalization of trials and regulatory challenges14, 21. Regardless, these overall trends in trial complexity, work burden, cost and enrollment challenges suggest the need for a reappraisal of the current trial environment.

Central Tenets of GCP: Advantages and Criticisms

The initial intention of GCP criteria was to ensure the safety and rights of participants in trials and the reliability of trial data to support the safety of future patients4. In brief, the guidelines detail the responsibilities, procedures and recording that are necessary for appropriate trial conduct by investigators, study staff, sponsors, and IRBs. In general, these guidelines have led to improved quality of clinical trial conduct and reporting. For instance, GCP stipulates that trials are conducted in compliance with the IRB-approved protocol with appropriate adverse event monitoring and reporting. These consistent expectations supported a reduction in site/investigator misconduct, enhanced protection of patients’ and improved data quality and standardization across trials for reporting to regulatory bodies.

Importantly, the development and implementation of GCP guidance were related to and complementary to other documents focusing on the ethics of human subject research; these concepts are covered in documents including the Declaration of Helsinki22, the Belmont Report23 and the Code of Federal Regulations, Protection of Human Subjects24. We refer the reader to previous summary statements on these documents25, 26. In brief, the Declaration of Helsinki was developed by the World Medical Association’s Committee on Medical Ethics in 1964 to serve as a guide for physicians engaged in clinical research involving human subjects25. Basic principles include that studies be conducted by qualified persons with the health, interests, privacy and integrity of the patient as the first consideration with careful assessment of “predictable risk and foreseeable benefit.” The Belmont Report was subsequently developed in 1978 by the US National Commission for the Protection of Human Services of Biomedical and Behavioral Research Biomedical to articulate the fundamental ethical principles underlying human subjects research: beneficence, justice and respect for persons26. The US Department of Health and Human Services has also developed a specific Code of Federal Regulations detailing basic policies for protection of human subjects (Title 45, Part 46)24. This document outlines policies related to informed consent and protection of special populations and applies to all research involving human participants that is subject to regulation by any US federal department or agency. Thus, GCP guidance includes details on scientific quality standards but also stems from the principles outlined in documents involving the ethics of human subject research.

However, not all of the downstream effects of GCP implementation have been evidence-based or positive. In some circumstances, there is an inappropriate emphasis on reporting (e.g., progress reports, safety reporting, final study reports), monitoring, auditing, and “essential documents” as a result of GCP guidelines. Other reviews6, 27 have previously critiqued the challenges of GCP guidance. In brief, the guidance is derived from informal consensus rather than evidence-based data. The document’s writing process did not include academic researchers well versed in trial methodology. In fact, the documents lack details on authorship and references are not included. As a result, key details are missing from the documents. One example that is commonly highlighted is that there is no discussion on adequate allocation concealment in randomized trials. Allocation concealment refers to the technique used to implement the sequence of randomization in a manner that keeps clinicians and participants blinded to treatment assignment28. The lack of appropriate concealment has been shown to introduce bias and significantly influence trial quality29. Moreover, despite the significant changes in the clinical research environment over the past decade, the documents have not been updated since 1996.

Implementation of GCP guidelines has also resulted in intensive site monitoring, which is costly, time consuming and of unproven benefit6. Key GCP considerations include clinical monitoring and data audits to confirm that trial data are “verifiable from source documents”. As discussed by Reith C et al, “they focus to an inappropriate extent on ensuring the completeness and accuracy of each piece of data that is recorded, even though minor errors occurring with similar frequency in the treatment groups should not materially affect the findings.6” While in many circumstances site monitoring has become less intensive over time30, some ongoing trials continue to require substantial on-site verification of documents detailed in GCP guidelines. Major goals include the detection of fraud and inaccurate transcription of data. While these goals are important, the tendency to emphasize minutia may direct focus away from critical aspects of trial conduct such as appropriate patient enrollment and retention, study intervention and outcome assessment. There is an emphasis on “essential documents” unrelated to research validity (e.g., updating curriculum vitae, signature sheets, drug-storage records, temperature logs) rather than activities designed to directly improve research quality and enhance scientific validity (e.g., minimizing lost-to-follow-up). While the intention of GCP is to protect and promote patients’ rights and safety while increasing the overall quality of clinical trials, the interpretation and implementation of GCP guidelines has been far from ideal.

PCT Background, Study Characteristics and Limitations

While contemporary trials seem to be becoming more complex, and costly, the emergence of EHR and registry data offer the possibility of novel PCT designs whose form and function are much more similar to the streamlined clinical trials of the 1970s and 80s31, 32. The key characteristics of PCTs have been previously reviewed33, 34. In brief, PCTs focus on whether the intervention is effective under usual conditions or in a “real-world setting” rather than under controlled “ideal” circumstances. In general, PCTs have broad entry criteria to enroll a diverse study population across heterogeneous practice settings to enhance generalizability of study results, a shortcoming of traditional, highly-selected clinical trials35. A major aim of PCTs is to simplify eligibility criteria, screening and overall study conduct to improve trial efficiency. Clinically relevant alternative interventions are compared such that results are readily applicable to usual care. Certain types of interventions may be more appropriate in a PCT as compared with a conventional trial. For instance, blinding and placebo-controlled comparators may be cost prohibitive in the setting of a large scale PCT aligned with usual care. In order to balance concerns related to the scientific rigor or validity of a trial conducted without blinding, additional design considerations may be necessary such as the use of objective endpoints (e.g., all-cause mortality) and utilization of an endpoint-capture strategy that is systematic for all patients. Similarly, the strategy to address monitoring procedures is also not a simple one-size-fits-all approach and may require a tailored approach to match the research goals in a manner that optimizes ethical and scientific quality.

Study conduct is incorporated into routine clinical practice rather than through study-specific visits, reducing the burden and inconvenience to participants. The adherence of practitioners and patients is assessed in an unobtrusive manner (if at all). Endpoints are clinically meaningful with assessment in a manner that is consistent with usual care (i.e., site-reported) with less reliance on central adjudication. Thus, while the development of so-called pragmatic trials appears to be a relatively recent paradigm, in many respects the design of PCTs has its roots in earlier clinical trials as exemplified by the ISIS and GUSTO36 studies. More recently, in order to help trialists design studies and characterize the study’s position on the spectrum from explanatory to pragmatic, the PRECIS tool was developed34 and updated37. This tool provides a framework to characterize the different components of a clinical trial (e.g., entry criteria, follow-up schedule) in order to promote study designs that are consistent with the intended degree of pragmatism.

The design considerations of PCTs represent two sides of a coin with respect to the strengths and limitations. Trials that enroll a broad patient population have a more heterogeneous study population compared to those with strict entry criteria. On one hand, this may improve generalizability of study results. However, when neutral results are observed in a heterogeneous population, it may be uncertain whether the intervention does not work or whether positive results would have been observed under optimal conditions (i.e., effectiveness vs. efficacy design). Effectiveness studies may leverage a pragmatic methodology to assess interventions under usual circumstances within a broader population targeting formulary approval and/or real-world comparative-effectiveness. In contrast, efficacy studies are routinely designed with an explanatory methodology that seeks to answer whether an intervention works under optimal circumstances. There is usually a strictly defined patient population and the results may be targeting regulatory approval. The FDA requires placebo-controlled studies for pharmaceutical approval (unless superiority over another marketed product is targeted) and has historically required efficacy rather than effectiveness data33. As a result, most prior and ongoing PCTs compared clinically available therapeutics (i.e., phase IV studies) rather than novel therapeutics requiring regulatory review. Other potential concerns with PCTs include limitations of real-world safety reporting, reduced patient retention and the potential for reduced adherence and data acquisition outside the context of conventional monitoring, and study-specific visits/procedures.

Progress in Pragmatism: Registry-based Studies and Cluster Randomized Trials

Several strategies that have been used in contemporary clinical trials in order to improve pragmatism include leveraging additional data sources such as registries and incorporating cluster designs to streamline trial conduct. The TASTE trial assessed whether thrombus aspiration during ST-elevation MI reduced mortality in a multi-center trial with enrollment of patients and endpoint acquisition from national registries in Sweden38. SAFE-PCI was a randomized trial comparing radial versus femoral arterial access in women undergoing percutaneous coronary intervention (PCI)39. The study embedded the randomized trial into the existing infrastructure of the National Cardiovascular Data Registry CathPCI Registry through the NIH’s National Cardiovascular Research Infrastructure. Similarly, the TRANSLATE-ACS trial was a longitudinal observational study of myocardial infarction patients managed with PCI40. The trial design built upon the PCI registry platform and incorporated a systematic telephone interview follow-up process, a cluster-randomized substudy to investigate the utility of platelet inhibition testing and an assessment of the dissemination of site-specific, quality-of-care data benchmarked to peer performance. Additional clinical trials have incorporated streamlined cluster designs as in the Post-MI FREEE trial which assessed the clinical impact of reducing cost-sharing for cardiac medications following myocardial infarction41. These clinical trial designs represent incremental progress toward improved pragmatism compared with so-called explanatory or conventional trials that do not incorporate the streamlined processes proposed in PCTs.

EHR-Facilitated PCTs

The growth of EHR data across health systems has generated enthusiasm for EHR-facilitated PCTs. Importantly, while the EHR is a rich source of clinical data, it is specifically designed to support clinical care and reimbursement. The assumption that EHR data are fit for use in high-quality clinical research has not been rigorously evaluated to date. Several ongoing investigations will assess the fitness of EHR data to facilitate efficient, reliable and cost-effective clinical research. For instance, the Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-term Effectiveness (ADAPTABLE trial) is one of the first chronic intervention trials that leverages a PCT design as compared to more conventional RCTs. ADAPTABLE is assessing lower-dose or higher-dose aspirin in patients with heart disease through the PCORnet (Patient Centered Outcomes Research Network) infrastructure42. PCORnet is a coordinated network of Clinical Data Research Networks (CDRNs) representing health system collaborations and Patient-Powered Research Networks (PPRNs) of patients/stakeholders with representation in all 50 states and coverage of >25 million Americans. The trial is embedded within usual care with minimal entry criteria, electronic patient-directed consent and data collection that incorporates data that have been standardized to a common data model, Medicare claims and patient-reported outcomes. The trial will recruit 20,000 patients and by leveraging EHR data with reduced burden on patients, clinicians and practices it is expected to cost less than conventional trials.

An example of a prior interventional PCT that leveraged EHR data for outcome acquisition was a real-world, randomized, open-label trial of anti-depressants43. In addition, the Time to Reduce Mortality in End-Stage Renal Disease (TiME) Trial is an ongoing cluster-randomized NIH Collaboratory trial that is assessing the implications of hemodialysis duration on clinical outcomes and quality of life through a pragmatic design that leverages EHR data and collaboration between academic investigators and industry (clinicaltrials.gov identifier: NCT02019225). Moreover, the BPMedTime Trial is an NHLBI-funded randomized pragmatic trial designed to evaluate the safety and efficacy of nighttime dosing of anti-hypertensives that will leverage the EHR for recruitment and data acquisition at two collaborating health systems with follow-up for 36 to 42 months (NIH Project Reporter: 1UH2AT007784-01).

Despite these recent initial steps towards streamlining clinical trials, EHR-facilitated PCTs face significant challenges. Our understanding of how EHR-based platforms might increase the efficiency of data collection, outcome surveillance, and confirmation in an integrated manner is in a nascent state. For instance, evidence from the West of Scotland Coronary Prevention Study suggests that cardiovascular endpoints can be ascertained from routinely recorded EHR-type data, but classification was imperfect44. Specifically, fatal endpoints ascertained via routine mechanisms (i.e., source documentation, case report form completion, adjudication) showed excellent matching with register data (97%). However, non-fatal events such as myocardial infarction and stroke matched approximately 80% of the time. These discrepancies were due to factors including duplicate events, disagreement between site-reported and adjudicated events, events outside the catchment area, miscoding and linkage errors. Importantly, the authors demonstrated that the observed risk reduction for the primary endpoint differed using the record-linked data compared with the original trial results (39% risk reduction [24,51) vs. 29% [15–44]), yet the qualitative conclusions were similar regardless of the methodology.

Similarly, a population-based study in Sweden comparing heart failure diagnoses in the hospital register with adjudication via chart review found that roughly 80% of register cases were classified as definite HF with echocardiographic data increasing this to nearly 90%45. Lower validity was observed when comparing general medicine clinics vs. cardiology clinics (86 vs 91%) but assessment of HF as the primary diagnosis increased validity to 95%. Taken together, these examples suggest that event classification varies depending on the specific endpoint but is around 80% or greater for commonly assessed cardiovascular endpoints. Importantly, the methodology for classification using both structure and unstructured EHR data is a rapidly growing area of research and will likely continue to improve over time.

EHRs are inherently heterogeneous, and “common” data elements are defined in different ways46, 47. The complexity of the health care system impacts the EHR as well. Patients receive care from multiple providers48 and in multiple health care systems49, so complete information does not exist in a single location. Thus, while significant advances have been made in recent years with respect to EHR-facilitated trials, many challenges remain.

Comparison of Conventional Trials and EHR-Facilitated Trials

EHR-facilitated trials may offer financial advantages over more conventional explanatory trials in the contemporary GCP environment. A recent conventional trial of more than 14,000 diabetic patients enrolled at 660 sites from 2008–2012 with follow-up through 2015 cost nearly $250 million with monitoring constituting more than $56 million (23%). In comparison, the ADAPTABLE trial that leverages EHR data to target enrollment of 20,000 patients over a shorter enrollment period is estimated to cost approximately $14–18 million with reduced costs for trial management and monitoring and increased costs for informatics42.

Tension between GCP and PCTs

Areas of potential tension exist between the guidance provided by GCP and the key characteristics of PCTs. From a patient enrollment perspective, GCP guidelines indicate the need for screening and written informed consent procedures by a qualified study team member, whereas PCTs emphasize streamlined identification of study participants via electronic mechanisms and consideration of mass enrolling. In addition, there is growing interest in pragmatic trials using patient-directed consent and electronic consent50 rather than more traditional coordinator/investigator-facilitated consent. Additional details related to study personnel responsibilities are detailed throughout the GCP guideline document. For instance, GCP indicates that qualified physician investigators should be responsible for all trial-related medical decisions and patient follow-up related to use of the investigational product as well as medical care for comorbid conditions. The responsibilities outlined in the GCP guidelines may appear at odds with PCTs that emphasize inclusion of practitioners with a range of research experience, flexibility of study intervention in routine care and minimal trial-specific training. Documentation, monitoring and reporting guidelines are extensively detailed in the GCP document, whereas PCTs leverage routinely collected data with monitoring that is risk-based (i.e., scalable depending on need). A formal process for clinical event classification is invoked via the language in GCP whereas PCTs support streamlined event capture and minimal (if any) formal adjudication. Streamlined safety reporting32 including large-scale surveillance of safety data51 is central to PCTs whereas GCP details on-site monitoring with auditing and source document verification.

Harmonization of GCP and PCTs

Despite the potential for tension between GCP guidelines and PCT characteristics, innovative approaches to clinical trials can be harmonized with these historic trial guidelines. While prior trials designed to support FDA approval were routinely conventional trials with explanatory methodology, we propose that future trials across the spectrum of product development should consider incorporating elements of pragmatism as able. Ultimately, each trial design should be constructed in an individualized manner that is fit for purpose. Rather than a one-size-fits-all approach to trial design, different trials may incorporate various degrees of operational simplicity while leveraging available data, PCT concepts and logical implementation of GCP. Table 2 provides an overview of potential solutions to address GCP guidance in the context of PCTs. For example, a strategy of simplifying the informed consent process can be conducted in a manner that reduces the burden on patients and investigators, while supporting the rights and safety of patients in a manner consistent with GCP recommendations. Previous studies have demonstrated that when patients do not read informed consent documents, they are nearly three times more likely to decline trial participation compared to those patients who read the document52. Therefore, a simplified consent form in language accessible to the layperson may be designed in collaboration with patient partners to balance the need to adequately inform the patient of critical study elements and convey any risks to the patient. These consent forms should focus on presentation at the eighth-grade level with typical oral reading rates of no more than 150 words per minute to ensure comprehension53 and should also implement comprehension questions. Similarly, the historic perspective that medical contact for those involved in all trials needs to be orchestrated by qualified study investigators through study-specific visits may represent an overinterpretation of the GCP guidelines that can be adapted to the PCT model. A potential advantage of clinical trials that are aligned with routine care is reduction in the Hawthorne effect that may be seen with additional study visits and patient monitoring. The Hawthorne effect involves an alteration in the behavior of research participants as a consequence of the awareness of study participation, which may bias trial results54.

Table 2.

Harmonization of GCP and PCTs.

Domain Potential Solutions
Patient Enrollment and Consent
  • Patient identification via electronic health record search with real-time point-of-care triggers to alert clinician of potential eligibility. Such strategies may support patient autonomy and justice with improved access to available trials while also reducing the burden of trial participation on clinicians.

  • Simplified informed consent text and procedures to reduce burden on patients/families and investigators while supporting the rights, safety and well-being of patients.

    • Enhanced efforts to better inform patients regarding expectations, responsibilities and rights for trial participants in contemporary clinical practice that capitalize on multi-media approaches that are individualized to patients (e.g., video platforms for trial overview and consent)

Study Intervention and Medical Care
  • Incorporation of intervention as part of routine care by usual care provider with input/mentorship from a qualified trial physician

    • Protocol orders (e.g., tests and procedures) and visit schedules are incorporated into EHR orders and documentation

High quality data
  • Implementation of risk-based site monitoring strategies

  • Central statistical monitoring of sites

  • Streamlined event adjudication as the preferred option, if appropriate, with formal clinical events committee as needed

Study Personnel
  • A real-world clinical trial team that includes members with varied training and experience with appropriate support from a qualified physician and trial champion

Study Visits, Follow-up, and Documentation
  • Incorporate existing electronic data from routine follow-up into the study documentation as well as registry data and/or national registry data (as able)

    • Follow-up via ongoing clinical registries, Centers for Medicare and Medicaid Services claims and direct patient contact (e.g., email and/or telephone contact)

Monitoring and Adherence Measures
  • Focus on consent procedures, randomization integrity, safety, and complete follow-up

Reporting and Safety
  • Streamline reporting with an emphasis on generating safety data that are similar to that reviewed by a Data Safety Monitoring Board and leveraging safety procedures used in routine clinical care (e.g., MedWatch) rather than the historic perspectives of industry safety groups

Focused and abbreviated study-specific and general research training of usual care providers may be appropriate to support a PCT model that leverages streamlined participation of a real-world clinical trial team with support and mentorship from a trial physician. Furthermore, for some PCTs that are aligned with standard of care procedures, minimal (if any) additional research training or study-specific training may be needed for usual care providers. With ascertainment of baseline and outcomes data from electronic sources including the EHR, registries and national registers, the responsibility and burden of research participation for enrolling sites may be reduced and the requirements for provider training may not apply or may be reduced. A balanced approach for monitoring and safety reporting would incorporate large-scale surveillance of safety data51 as able, with more conventional reporting of specific adverse events of interest. Importantly, some clinical trials (e.g., a cardiovascular outcomes study for a novel agent) may be best conducted with a more traditional trial design that incorporates streamlined operations, endpoints and data acquisition, as appropriate, in order to reduce cost and improve efficiency. At present time, there are several ongoing efforts that may improve harmonization of GCP perspectives with contemporary clinical trials. The National Heart Lung and Blood Institute also has keen interest in reforming trial design and conduct55, 56.

TransCelerate BioPharma is a non-profit entity that includes biopharmaceutical companies, regulatory bodies, and academicians with a mission of collaborating across the research community “to identify, prioritize, design and facilitate the implementation of solutions to drive efficient, effective and high-quality delivery of new medicines”57. The consortium has focused on improving the quality and efficiency of clinical trials via incremental advancements in the following areas: risk-based monitoring58, site-qualification and training that meets benchmarked minimum GCP criteria, industry-wide clinical data standards to support research data exchange and patient safety, and development of a shared investigator platform to exchange data and protocols to facilitate trial development. Additional initiatives of TransCelerate include creation of common clinical trial protocol templates and a global investigator registry to streamline trial conduct and optimize trial efficiency with supporting appropriate trial conduct and patient safety.

The Clinical Trials Transformation Initiative (CTTI) was co-founded in 2007 by Duke University and the FDA to identify and promote clinical trial practices that prioritize quality and efficiency59. CTTI’s membership includes academic research organizations and representatives from industry and government as well as patients and investigators. The group has generated data on clinical trial conduct in order to provide recommendations for improvement on topics such as informed consent, patient recruitment, and IRB conduct. Several of the specific areas of advancement that have been the focus of CTTI to date include the development of a Quality by Design (QBD) document that includes evidence-based recommendation for improving trial quality60 and collaboration with the FDA-established Mini-Sentinel program to facilitate future randomized trials the leverage the distributed database model61.

Additional think-tanks including representation from academia, industry and regulatory bodies have extended these discussions on improving clinical trial conduct to topics including data safety monitoring board processes62, post-marketing evaluations63, and reducing racial and sex disparities in clinical trials64.

Future Directions

Despite the potential tension between GCP guidance and PCT methodology, we have highlighted strategies to help harmonize and individualize the guidance as applied to PCTs. These considerations may inform future trial design and conduct. In addition, these areas of tension suggest the need to revise and update the historic GCP guidelines to improve relevance to the contemporary research environment. GCP reform is necessary not only for the implementation of PCTs but also to improve the efficiency of conventional trials. The inclusion of academic trialists, patient partners and evidence-based data in these revisions will be necessary. We suggest the possibility of a reduced emphasis on monitoring, auditing, and “essential documents”. Rather, we favor shifting the focus to stream-lined and “real-world” enrollment, study conduct, and reporting to ensure internal and external validity of trial results. The guidance could benefit from changing the guidelines to more specifically cover “Good Clinical Trial Practice” in the contemporary research environment6. In brief, the emphasis should be on making sure that the “right” patient (i.e., satisfies entry criteria with adequate consent) receives the appropriate intervention (i.e., correct randomization, blinding and treatment assignment) with adequate assessment of outcomes (i.e., complete, correct, and timely event ascertainment). With appropriate engagement of patients, clinicians, researchers, policymakers, and regulators, these issues can be clarified in order to improve the clinical research enterprise while maintaining high standards of protection of the rights, safety, and well-being of study participants. TransCelerate, CTTI and Mini-Sentinel represent ongoing collaborative efforts that may help to harmonize GCP principles with the design of PCTs. GCP does not necessarily preclude the conduct of PCTs, but rather, requires alignment between the different trial stakeholders particularly sponsors. A central theme is the alignment of industry subgroups including those from compliance, regulatory and safety groups with those of investigators designing PCTs. Implementation of GCP is dependent in large part on differential interpretation by these parties. Improved partnerships between regulators, industry representatives, trialists, and patients on the interpretation of GCP is long overdue and the era of EHR-facilitated PCTs may represent the ideal time for a reappraisal and redesign of study conduct.

Acknowledgments

Funding Sources: RJM receives research support from the National Institutes of Health (U10HL110312).

Footnotes

Disclosures: RJM receives research support from the National Institutes of Health (U10HL110312), Amgen, AstraZeneca, Bristol-Myers Squibb, GlaxoSmithKline, Gilead, Novartis, Otsuka, and ResMed; honoraria from HeartWare, Janssen, Luitpold Pharmaceuticals, Novartis, ResMed, and Thoratec; and has served on an advisory board for Luitpold Pharmaceuticals, Inc. AFH reported consulting for Amgen, AstraZeneca, Bayer, BMS, Corthera, Cytokinetics, Gilead, GSK, Janssen, Johnson & Johnson, Merck, Pluristem Therapeutics and Sensible; research support from Amylin, Bristol-Myers Squibb, Johnson & Johnson, GSK, Merck, Portola. LHC receives funding from Boston Scientific, GlaxoSmithKline, Gilead, NCQA, and Novartis. EDP has received institutional grant support from the American College of Cardiology, the American Heart Association, Eli Lilly, and Janssen; and consulting fees (including Continuing Medical Education) from Merck & Co., Boehringer Ingelheim, Bristol-Myers Squibb, Pfizer, Genentech, Janssen, and Sanofi. ECO receives research support from the NIH, PCORI, Bristol-Myers Squibb, GlaxoSmithKline, Janssen Scientific, and Boehringer-Ingelheim and has served on an advisory board for Portola Pharmaceuticals. The remaining authors report no relevant conflicts of interest.

References

  • 1.Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–2. doi: 10.1136/bmj.312.7023.71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.ACCF/AHA Task Force on Practice Guidelines. [Accessed May 12, 2015];Methodology Manual and Policies From the ACCF/AHA Task Force on Practice Guidelines. 2010 Available at: http://assets.cardiosource.com/Methodology_Manual_for_ACC_AHA_Writing_Committees.pdf.
  • 3.Collins R, MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. Lancet. 2001;357:373–80. doi: 10.1016/S0140-6736(00)03651-5. [DOI] [PubMed] [Google Scholar]
  • 4.International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. ICH Harmonized Tripartite Guideline. [Accessed May 12, 2015];Guideline for Good Clinical Practice E6(R1) 1996 Available at http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R1_Guideline.pdf. [PubMed]
  • 5.Devine S, Dagher RN, Weiss KD, Santana VM. Good Clinical Practice and the Conduct of Clinical Studies in Pediatric Oncology. Pediatr Clin North Am. 2008;55:187–xii. doi: 10.1016/j.pcl.2007.10.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Reith C, Landray M, Devereaux PJ, Bosch J, Granger CB, Baigent C, Califf RM, Collins R, Yusuf S. Randomized clinical trials--removing unnecessary obstacles. N Engl J Med. 2013;369:1061–5. doi: 10.1056/NEJMsb1300760. [DOI] [PubMed] [Google Scholar]
  • 7.Califf RM. Clinical trials bureaucracy: unintended consequences of well-intentioned policy. Clin Trials. 2006;3:496–502. doi: 10.1177/1740774506073173. [DOI] [PubMed] [Google Scholar]
  • 8.McMahon AD, Conway DI, Macdonald TM, McInnes GT. The unintended consequences of clinical trials regulations. PLoS Med. 2009;3:e1000131. doi: 10.1371/journal.pmed.1000131. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med. 2011;365:1758–9. doi: 10.1056/NEJMp1108726. [DOI] [PubMed] [Google Scholar]
  • 10.Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med Inform Assoc. 2014;21:576–7. doi: 10.1136/amiajnl-2014-002864. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Randomised trial of intravenous atenolol among 16 027 cases of suspected acute myocardial infarction: ISIS-1. First International Study of Infarct Survival Collaborative Group. Lancet. 1986;2:57–66. [PubMed] [Google Scholar]
  • 12.Yusuf S. Randomized clinical trials: slow death by a thousand unnecessary policies? CMAJ. 2004;171:889–92. doi: 10.1503/cmaj.1040884. discussion 892–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Butler J, Fonarow GC, O’Connor C, Adams K, Bonow RO, Cody RJ, Collins SP, Dunnmon P, Dinh W, Fiuzat M, Georgiopoulou VV, Grant S, Kim SY, Kupfer S, Lefkowitz M, Mentz RJ, Misselwitz F, Pitt B, Roessig L, Schelbert E, Shah M, Solomon S, Stockbridge N, Yancy C, Gheorghiade M. Improving cardiovascular clinical trials conduct in the United States: recommendation from clinicians, researchers, sponsors, and regulators. Am Heart J. 2015;169:305–14. doi: 10.1016/j.ahj.2014.12.001. [DOI] [PubMed] [Google Scholar]
  • 14.Glickman SW, McHutchison JG, Peterson ED, Cairns CB, Harrington RA, Califf RM, Schulman KA. Ethical and scientific implications of the globalization of clinical research. N Engl J Med. 2009;360:816–23. doi: 10.1056/NEJMsb0803929. [DOI] [PubMed] [Google Scholar]
  • 15.Getz KA, Campo RA, Kaitin KI. Variability in Protocol Design Complexity by Phase and Therapeutic Area. Drug Inform J. 2011;45:413–420. [Google Scholar]
  • 16.Getz KA, Stergiopoulos S, Marlborough M, Whitehill J, Curran M, Kaitin KI. Quantifying the magnitude and cost of collecting extraneous protocol data. Am J Ther. 2015;22:117–24. doi: 10.1097/MJT.0b013e31826fc4aa. [DOI] [PubMed] [Google Scholar]
  • 17.Martin SS, Ou FS, Newby LK, Sutton V, Adams P, Felker GM, Wang TY. Patient- and trial-specific barriers to participation in cardiovascular randomized clinical trials. J Am Coll Cardiol. 2013;61:762–9. doi: 10.1016/j.jacc.2012.10.046. [DOI] [PubMed] [Google Scholar]
  • 18.Butler J, Tahhan AS, Georgiopoulou VV, Kelkar A, Lee M, Khan B, Peterson E, Fonarow GC, Kalogeropoulos AP, Gheorghiade M. Trends in characteristics of cardiovascular clinical trials 2001–2012. Am Heart J. 2015;170:263–272.e2. doi: 10.1016/j.ahj.2015.05.006. [DOI] [PubMed] [Google Scholar]
  • 19.Butler J, Subacius H, Vaduganathan M, Fonarow GC, Ambrosy AP, Konstam MA, Maggioni A, Mentz RJ, Swedberg K, Zannad F, Gheorghiade M. Relationship between clinical trial site enrollment with participant characteristics, protocol completion, and outcomes: insights from the EVEREST (Efficacy of Vasopressin Antagonism in Heart Failure: Outcome Study with Tolvaptan) trial. J Am Coll Cardiol. 2013;61:571–9. doi: 10.1016/j.jacc.2012.10.025. [DOI] [PubMed] [Google Scholar]
  • 20.Lovato LC, Hill K, Hertert S, Hunninghake DB, Probstfield JL. Recruitment for controlled clinical trials: literature summary and annotated bibliography. Control Clin Trials. 1997;18:328–52. doi: 10.1016/s0197-2456(96)00236-x. [DOI] [PubMed] [Google Scholar]
  • 21.Bailey W, Cruickshank C, Sharma N. [Accessed October 7, 2015];Make your move: taking clinical trials to the best location. Available at https://www.atkearney.com/health/ideas-insights/article/-/asset_publisher/LCcgOeS4t85g/content/make-your-move/10192.
  • 22.World Medical Association. Declaration of Helsinki serves as guide to physicians. JAMA. 1964;189:33–34. [Google Scholar]
  • 23.U.S. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical principles and guidelines for the conduct of research involving human subjects. Washington, D.C: DHEW; 1978. [PubMed] [Google Scholar]
  • 24.US Department of Health and Human Services. [Accessed on October 7, 2015];Code of Federal Regulations. Title 5, Part B. Public Welfare: Protection of Human Subjects. Available at http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html.
  • 25.World Medical Association Declaration of Helsinki. Recommendations guiding physicians in biomedical research involving human subjects. JAMA. 1997;277:925–6. [PubMed] [Google Scholar]
  • 26.Sims JM. A brief review of the Belmont report. Dimens Crit Care Nurs. 2010;29:173–4. doi: 10.1097/DCC.0b013e3181de9ec5. [DOI] [PubMed] [Google Scholar]
  • 27.Grimes DA, Hubacher D, Nanda K, Schulz KF, Moher D, Altman DG. The Good Clinical Practice guideline: a bronze standard for clinical research. Lancet. 2005;366:172–4. doi: 10.1016/S0140-6736(05)66875-4. [DOI] [PubMed] [Google Scholar]
  • 28.Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359:614–618. doi: 10.1016/S0140-6736(02)07750-4. [DOI] [PubMed] [Google Scholar]
  • 29.Mills EJ, Ayers D, Chou R, Thorlund K. Are current standards of reporting quality for clinical trials sufficient in addressing important sources of bias? Contemp Clin Trials. 2015;45(Pt A):2–7. doi: 10.1016/j.cct.2015.07.019. Epub 2015 Jul 30. [DOI] [PubMed] [Google Scholar]
  • 30.US Department of Health and Human Services. Food and Drug Administration. [on August 17, 2015];Guidance for Industry: Oversight of Clinical Investigations - A Risk-Based Approach to Monitoring. Accessed at www.fda.gov/downloads/Drugs/Guidances/UCM269919.pdf.
  • 31.Fordyce CB, Roe MT, Ahmad T, Libby P, Borer JS, Hiatt WR, Bristow MR, Packer M, Wasserman SM, Braunstein N, Pitt B, DeMets DL, Cooper-Arnold K, Armstrong PW, Berkowitz SD, Scott R, Prats J, Galis ZS, Stockbridge N, Peterson ED, Califf RM. Cardiovascular drug development: is it dead or just hibernating? J Am Coll Cardiol. 2015;65:1567–82. doi: 10.1016/j.jacc.2015.03.016. [DOI] [PubMed] [Google Scholar]
  • 32.Roe MT, Mahaffey KW, Ezekowitz JA, Alexander JH, Goodman SG, Hernandez A, Temple T, Berdan L, Califf RM, Harrington RA, Peterson ED, Armstrong PW. The future of cardiovascular clinical research in North America and beyond-addressing challenges and leveraging opportunities through unique academic and grassroots collaborations. Am Heart J. 2015;169:743–750. doi: 10.1016/j.ahj.2015.03.002. [DOI] [PubMed] [Google Scholar]
  • 33.Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290:1624–32. doi: 10.1001/jama.290.12.1624. [DOI] [PubMed] [Google Scholar]
  • 34.Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–75. doi: 10.1016/j.jclinepi.2008.12.011. [DOI] [PubMed] [Google Scholar]
  • 35.Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365:82–93. doi: 10.1016/S0140-6736(04)17670-8. [DOI] [PubMed] [Google Scholar]
  • 36.An international randomized trial comparing four thrombolytic strategies for acute myocardial infarction. The GUSTO investigators. N Engl J Med. 1993;329:673–82. doi: 10.1056/NEJM199309023291001. [DOI] [PubMed] [Google Scholar]
  • 37.Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. doi: 10.1136/bmj.h2147. [DOI] [PubMed] [Google Scholar]
  • 38.Frobert O, Lagerqvist B, Olivecrona GK, Omerovic E, Gudnason T, Maeng M, Aasa M, Angeras O, Calais F, Danielewicz M, Erlinge D, Hellsten L, Jensen U, Johansson AC, Karegren A, Nilsson J, Robertson L, Sandhall L, Sjogren I, Ostlund O, Harnek J, James SK. Thrombus aspiration during ST-segment elevation myocardial infarction. N Engl J Med. 2013;369:1587–97. doi: 10.1056/NEJMoa1308789. [DOI] [PubMed] [Google Scholar]
  • 39.Hess CN, Rao SV, Kong DF, Aberle LH, Anstrom KJ, Gibson CM, Gilchrist IC, Jacobs AK, Jolly SS, Mehran R, Messenger JC, Newby LK, Waksman R, Krucoff MW. Embedding a randomized clinical trial into an ongoing registry infrastructure: unique opportunities for efficiency in design of the Study of Access site For Enhancement of Percutaneous Coronary Intervention for Women (SAFE-PCI for Women) Am Heart J. 2013;166:421–8. doi: 10.1016/j.ahj.2013.06.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Chin CT, Wang TY, Anstrom KJ, Zhu B, Maa JF, Messenger JC, Ryan KA, Davidson-Ray L, Zettler M, Effron MB, Mark DB, Peterson ED. Treatment with adenosine diphosphate receptor inhibitors-longitudinal assessment of treatment patterns and events after acute coronary syndrome (TRANSLATE-ACS) study design: expanding the paradigm of longitudinal observational research. Am Heart J. 2011;162:844–51. doi: 10.1016/j.ahj.2011.08.021. [DOI] [PubMed] [Google Scholar]
  • 41.Choudhry NK, Brennan T, Toscano M, Spettell C, Glynn RJ, Rubino M, Schneeweiss S, Brookhart AM, Fernandes J, Mathew S, Christiansen B, Antman EM, Avorn J, Shrank WH. Rationale and design of the Post-MI FREEE trial: a randomized evaluation of first-dollar drug coverage for post-myocardial infarction secondary preventive therapies. Am Heart J. 2008;156:31–6. doi: 10.1016/j.ahj.2008.03.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Hernandez AF, Fleurence RL, Rothman RL. The ADAPTABLE Trial and PCORnet: Shining Light on a New Research Paradigm. Ann Intern Med. 2015;163:635–6. doi: 10.7326/M15-1460. [DOI] [PubMed] [Google Scholar]
  • 43.Simon GE, VonKorff M, Heiligenstein JH, Revicki DA, Grothaus L, Katon W, Wagner EH. Initial antidepressant choice in primary care. Effectiveness and cost of fluoxetine vs tricyclic antidepressants. JAMA. 1996;275:1897–902. [PubMed] [Google Scholar]
  • 44.Barry SJ, Dinnett E, Kean S, Gaw A, Ford I. Are routinely collected NHS administrative records suitable for endpoint identification in clinical trials? Evidence from the West of Scotland Coronary Prevention Study. PloS One. 2013;8:e75379. doi: 10.1371/journal.pone.0075379. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Ingelsson E, Arnlov J, Sundstrom J, Lind L. The validity of a diagnosis of heart failure in a hospital discharge register. Eur J Heart Fail. 2005;7:787–91. doi: 10.1016/j.ejheart.2004.12.007. [DOI] [PubMed] [Google Scholar]
  • 46.Bayley KB, Belnap T, Savitz L, Masica AL, Shah N, Fleming NS. Challenges in using electronic health record data for CER: experience of 4 learning organizations and solutions applied. Med Care. 2013;51:S80–6. doi: 10.1097/MLR.0b013e31829b1d48. [DOI] [PubMed] [Google Scholar]
  • 47.Botsis T, Hartvigsen G, Chen F, Weng C. Secondary Use of EHR: Data Quality Issues and Informatics Opportunities. AMIA Jt Summits Transl Sci Proc. 2010;2010:1–5. [PMC free article] [PubMed] [Google Scholar]
  • 48.Pham HH, Schrag D, O’Malley AS, Wu B, Bach PB. Care patterns in Medicare and their implications for pay for performance. N Engl J Med. 2007;356:1130–9. doi: 10.1056/NEJMsa063979. [DOI] [PubMed] [Google Scholar]
  • 49.Bourgeois FC, Olson KL, Mandl KD. Patients treated at multiple acute health care facilities: quantifying information fragmentation. Arch Intern Med. 2010;170:1989–95. doi: 10.1001/archinternmed.2010.439. [DOI] [PubMed] [Google Scholar]
  • 50.Geller NL, Kim DY, Tian X. Smart Technology in Lung Disease Clinical Trials. Chest. 2015 doi: 10.1378/chest.15-1314. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Psaty BM, Breckenridge AM. Mini-Sentinel and regulatory science--big data rendered fit and functional. N Engl J Med. 2014;370:2165–7. doi: 10.1056/NEJMp1401664. [DOI] [PubMed] [Google Scholar]
  • 52.Sen Biswas M, Newby LK, Bastian LA, Peterson ED, Sugarman J. Who refuses enrollment in cardiac clinical trials? Clin Trials. 2007;4:258–63. doi: 10.1177/1740774507079434. [DOI] [PubMed] [Google Scholar]
  • 53.Hasbrouck J, Tindal GA. Oral Reading Fluency Norms: A Valuable Assessment Tool for Reading Teachers. The Reading Teacher. 2006;59:636–644. [Google Scholar]
  • 54.McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol. 2014;67:267–77. doi: 10.1016/j.jclinepi.2013.08.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Lauer MS, Kiley JP, Mockrin SC, Mensah GA, Hoots WK, Patel Y, Cook NL, Patterson AP, Gibbons GH. National heart, lung, and blood institute (NHLBI) strategic visioning: setting an agenda together for the NHLBI of 2025. Circulation. 2015;131:1106–9. doi: 10.1161/CIRCULATIONAHA.115.015712. [DOI] [PubMed] [Google Scholar]
  • 56.Lauer MS, D’Agostino RB., Sr The randomized registry trial--the next disruptive technology in clinical research? N Engl J Med. 2013;369:1579–81. doi: 10.1056/NEJMp1310102. [DOI] [PubMed] [Google Scholar]
  • 57.Gill D. Re-inventing clinical trials through TransCelerate. Nat Rev Drug Discov. 2014;13:787–8. doi: 10.1038/nrd4437. [DOI] [PubMed] [Google Scholar]
  • 58.TransCelerate. [on August 12, 2015];Risk-Based Monitoring. Accessed at http://www.transceleratebiopharmainc.com/assets/risk-based-monitoring/
  • 59.Tenaerts P, Madre L, Archdeacon P, Califf RM. The Clinical Trials Transformation Initiative: innovation through collaboration. Nat Rev Drug Discov. 2014;13:797–8. doi: 10.1038/nrd4442. [DOI] [PubMed] [Google Scholar]
  • 60.CTTI. [on August 12, 2015];Quality By Design. Accessed at http://www.ctti-clinicaltrials.org/files/QbD_QRM/QbDQRM-PrinciplesDocument-DeviceWorkshop-2014-01-16.pdf.
  • 61.Mini-Sentinel and Clinical Trials Transformation Initiative. [on August 12, 2015];Developing Approaches to Conducting Randomized Trials using the Mini-Sentinel Distributed Database. Accessed at http://www.mini-sentinel.org/work_products/Statistical_Methods/Mini-Sentinel_Methods_CTTI_Developing-Approaches-to-Conducting-Randomized-Trials-Using-MSDD.pdf.
  • 62.Hess CN, Roe MT, Gibson CM, Temple RJ, Pencina MJ, Zarin DA, Anstrom KJ, Alexander JH, Sherman RE, Fiedorek FT, Mahaffey KW, Lee KL, Chow SC, Armstrong PW, Califf RM. Independent data monitoring committees: preparing a path for the future. Am Heart J. 2014;168:135–41.e1. doi: 10.1016/j.ahj.2014.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Rao SV, Califf RM, Kramer JM, Peterson ED, Gross TP, Pepine CJ, Williams DO, Donohoe D, Waksman R, Mehran R, Krucoff MW. Postmarket evaluation of breakthrough technologies. Am Heart J. 2008;156:201–8. doi: 10.1016/j.ahj.2008.01.036. [DOI] [PubMed] [Google Scholar]
  • 64.Berger JS, Bairey-Merz CN, Redberg RF, Douglas PS. Improving the quality of care for women with cardiovascular disease: report of a DCRI Think Tank, March 8 to 9, 2007. Am Heart J. 2008;156:816–25. 825.e1. doi: 10.1016/j.ahj.2008.06.039. [DOI] [PubMed] [Google Scholar]

RESOURCES