Abstract
The widespread adoption and use of electronic health records and their use to enable learning health systems (LHS) holds great promise to accelerate both evidence-generating medicine (EGM) and evidence-based medicine (EBM), thereby enabling a LHS. In 2016, AMIA convened its 10th annual Policy Invitational to discuss issues key to facilitating the EGM-EBM paradigm at points-of-care (nodes), across organizations (networks), and to ensure viability of this model at scale (sustainability). In this article, we synthesize discussions from the conference and supplements those deliberations with relevant context to inform ongoing policy development. Specifically, we explore and suggest public policies needed to facilitate EGM-EBM activities on a national scale, particularly those policies that can enable and improve clinical and health services research at the point-of-care, accelerate biomedical discovery, and facilitate translation of findings to improve the health of individuals and populations.
Keywords: research informatics, clinical informatics, learning health systems, evidence-generating medicine, evidence-based medicine, policy
INTRODUCTION AND BACKGROUND
As a consequence of the almost universal adoption of electronic health records (EHRs) in the United States (US), “data-driven” healthcare—long considered merely theoretical—is now within reach and holds great promise.1 However, despite the unprecedented volumes of clinical data generated each day across hospitals, physician offices, urgent care facilities, and neighborhood walk-in clinics, the ability to leverage these data to increase our knowledge of health and disease and drive improvements in care remain overwhelmingly unrealized.2
The evidence-based medicine (EBM) paradigm has become the bedrock paradigm of clinical decision-making over the last two decades. Today, EBM is being facilitated via the use of EHRs, and its corollary—“evidence-generating medicine” (EGM)—can be operationalized in order to enable the generation of evidence from real-world practice. This approach has the potential to transform a severely limited clinical research system, one that is expensive, inefficient, and incapable of scaling to address the myriad unanswered clinical questions faced by clinicians and patients.3,4 EGM is the “systematic incorporation of research and quality improvement considerations into the organization and practice of healthcare in order to advance biomedical science and thereby improve the health of individuals and populations.”5 The EGM paradigm recognizes that clinical care activities are not entirely distinct from research activities and that the generation of evidence is critical to the EBM lifecycle.6 EGM is therefore an important and necessary element of enabling a learning health system (LHS), one that seeks to systematically learn from each healthcare encounter, facilitate discovery, and advance our collective understanding of effective approaches to healthcare.7
Increasingly, all health stakeholders have the opportunity—and obligation—to leverage health information technology (IT) systems, and the valuable data they contain, for the development and delivery of new interventions. The success of several current high-impact, high-visibility research initiatives depend on actualizing this evidence cycle. Examples include the All of Us Research Program8 and Cancer Moonshot initiative,9 the Health Care Systems Research Collaboratory,10 and the development of a national system of real-world evidence generation system as pursued by such groups as the US Food & Drug Administration (FDA), Patient-Centered Outcomes Research Institute (PCORI), National Institutes of Health (NIH), and other federal agencies.11
It was with this environment and motivation in mind that for 2 days in September 2016, AMIA convened its 10th annual AMIA Policy Invitational (API2016) meeting that brought together key thought leaders to discuss informatics issues related to realizing the emerging paradigm of an EBM-EGM cycle to enable the LHS. As further described below, through a series of facilitated group discussions before and during the API, a strategy for realizing the vision of nationally scaled EGM was developed. The emergent framework focused on facilitating: (1) research opportunities at the “point-care” (nodes), (2) research conducted across organizations (networks), and (3) ensuring the ongoing viability of research at regional/national levels (sustainability). This article explores those discussions and findings in more detail and suggests a range of public policies needed to facilitate EGM activities on a national scale. Special attention in paid to those policies that can enable and improve clinical research at the point-of-care, accelerate biomedical discovery, and enable translation of findings to improve the health of individuals and populations. Below, we synthesize discussion from the meeting and supplement those conversations with relevant context to inform ongoing policy development.
MEETING STRUCTURE
The 2016 AMIA Policy Invitational (API2016) included over 70 participants that were invited based upon experience and research contributions focused at the intersection of care delivery and research. Approximately 80% were AMIA members, with the remaining 20% composed of policy professionals and non-AMIA members from industry, government, professional associations. Attendees were identified by the conference chair and were not paid for their participation. Based upon premeeting deliberations led by a core group of experts that constituted the API2016 planning committee, the 2-day meeting was segmented into three broad levels of abstraction: (1) nodes, (2) networks, and (3) sustainability. Each level was covered in a portion of the meeting that included expert presentations on key topics followed by breakout working group discussions to further explore the topic and its policy implications. The first session focused on how to integrate research at the point-of-care, symbolized as a node. Next, meeting attendees considered issues and policy implications related to conducting research that spanned multiple healthcare and research organizations, or networks. Finally, attendees considered public policy issues that would contribute to the long-term sustainability of such a national research-practice ecosystem. (See Table 1 for questions presented to API2016 participants.) In preparation for the API2016, AMIA staff deployed a website12 that supported preconference collaboration and discussion. Breakout session questions were posted 2 weeks in advance of the meeting, and participants were encouraged to engage with fellow attendees by considering the questions via discussion threads.
Table 1.
Breakout A: Nodes: Evidence Generation at the Local Level |
|
Breakout B: Networks: Clinical Research Across Organizations |
|
Breakout C: Sustainability: Maintaining a National Research Ecosystem |
|
To supplement these discussions, API2016 attendees also heard keynotes given by National Library of Medicine Director Dr. Patricia Brennan, FDA Commissioner Dr. Robert Califf, and Agency for Healthcare Research and Quality Director Dr. Andrew Bindman. A summary of their remarks and links to their presentations can be found in Table 2.
Table 2.
Title/Speaker | Key findings | |
---|---|---|
Keynotes | The role of the NLM in Reimagining the Research-Practice Relationship in the Post-Meaningful Use Era |
|
Patricia Flatley Brennan, RN, PhD, Director, United States National Library of Medicine | ||
Generating Evidence to Inform Decisions in the Era of Precision Medicine |
|
|
Robert Califf, MD, Commissioner, Food and Drug Administration | ||
Building a Research Strategy to Support Learning Health Systems |
|
|
Andy Bindman, MD, Director, Agency for Healthcare Research and Quality |
FINDINGS AND RECOMMENDATIONS
Key findings and recommendations from the meeting participants were captured, reviewed, categorized, and cataloged by the authors using an iterative, qualitative analytic approach that led to group consensus. The findings are summarized below. Recommendations for each set of findings are further detailed in Table 3.
Table 3.
Topic | Findings | Recommendations |
---|---|---|
Nodes: Evidence Generation at the Local Level | Clinicians, patients and health systems are not routinely engaged in research and often treat it as a separate component from care delivery. As it represents a single node in what should be a complex and integrated network, focusing on a single point of care as the most basic locus of research is vital. |
|
Current regulations present real and perceived barriers to evidence generation at the local level. |
|
|
Technical work on data standards and certified health IT functionality is needed to enable EGM and local learning health systems. |
|
|
Networks: Clinical Research Across Organizations | The clinical and research workforce, including Institutional Review Board (IRB) staff, data stewards and data curators often lack a fundamental understanding of informatics-driven research methodologies. |
|
Incompatible data standards—including metadata and patient identifiers—and data use agreements create technical challenges and tremendous administrative burden on multisite research. |
|
|
Sustainability: Maintaining a National Research Ecosystem | The effectiveness of our national research networks is dependent on shared infrastructure that require appropriate levels of government support to complement initiatives from the private sector. |
|
The benefits of federally sponsored research must extend as much as possible beyond the awardees who conduct the research. |
|
Nodes: policies needed to facilitate evidence generation at the local level
Finding: clinicians, patients, and health systems are not routinely engaged in research and often treat it as a separate component from care delivery. Focusing on enabling evidence-generation activities at the point-of-care as the most basic locus of research is vital because it represents a single, yet connected, node in a complex and integrated network
Policy implication: incentives are needed for key stakeholders to prioritize research at the point-of-care—the nodes of an EGM system
Participating in certain research-related activities at the point-of-care can involve an opportunity cost to clinicians and healthcare systems that must be recognized and addressed to align incentives for enabling research and care nodes. Actions by Congress and the Executive Branch over the past decade have helped improve the engagement of patients and clinicians in research. Through research programs led by PCORI and federal research funding agencies, patient participation in research design and policy development has been explicitly required.13–16 Further, these agencies have made strides in having patients and clinicians work together to fund, design, and implement research studies that address real-world questions that are important to patients.17 However, to be effective, this trend must continue with the current Administration and Congress; steps should be taken to replicate this kind of engagement with both clinicians and health systems. Specifically, CMS should leverage its Quality Payment Program to reward clinical practice Improvement Activities (IAs) that involve research components. This would encourage office-based physicians to invest time and resources needed to realize EGM. Further, federal agencies should develop funding priorities and special emphasis notices that encourage clinicians and health systems to collaborate in research studies and commit to share study results, in both raw data and curated formats, back to clinicians and patients. Practice-Based Research Networks (PBRNs) also remain a promising model for primary care clinicians and practices working together to answer community-based health questions and to translate those findings into practice.18
Finding: current regulations present real and perceived barriers to evidence generation at the local level
Policy implication: regulatory modifications are needed to facilitate research at the point-of-care
There is wide disparity in how current regulations governing informed consent and privacy protections are applied across healthcare delivery settings, often resulting in an environment where research is stymied. As part of the January 19, 2017 final rule revising the Common Rule,19 the Office of Human Research Protections (OHRP) created a new exemption meant to relieve researchers from Common Rule compliance if they are already regulated by HIPAA. This is a positive development for EGM, and we encourage OHRP to produce educational materials and guidance to help ensure that such research-facilitating provisions are uniformly understood and implemented across care delivery settings when the compliance date arrives in January 2019. In addition, HHS should promote recent guidance from the Office of Civil Rights (OCR) regarding individual authorization for uses and disclosures of protected health information for research.20 Last, the administration’s Office of Management and Budget (OMB) was tasked with establishing a Research Policy Board within 1 year of enactment of 21st Century Cures to make recommendations regarding the modification and harmonization of research regulations and policies.21 More than a year and half since 21st Century Cures became law, the Research Policy Board has not been established. The administration must implement this section of legislation to better calibrate and harmonize our sprawling and incoherent federal research regulations.
Finding: technical work on data standards and certified health IT functionality is needed to enable EGM and local learning health systems
Policy implication: investment in the “basic science” of health IT, including supporting standards and processes for implementing and enhancing functional requirements, is necessary
Data standards are necessary to the exchange of clinical data and interoperability between EHR systems will be critical for data aggregation and sharing. The development, harmonization, and use of standards for vocabulary, format, and transport will be essential for the continued improvement of health IT tools necessary to support EGM and LHS. While interoperability is an on-going challenge, agreement on standards is foundational to realizing the potential benefits of IT. As tools for care delivery, EHRs are not readily configured to facilitate EGM or supplemental uses of patient data for research, even when a patient explicitly wants to enable that use. IT tools that support the screening and recruitment of patients into research in the context of a healthcare encounter are needed, as are tools that enable patients to easily access their health data and transfer it to (share it with) research activities and investigations as they wish.
Networks: clinical research across organizations
The development of processes and systems to generate evidence at points-of-care (nodes) is an essential component of a LHS.22–24 The value of research conducted at organizational nodes can be magnified when the nodes are integrated into networks. Networks of organizations, working in tandem, can leverage efficiencies of scale to include more clinicians and more patients in the generation of important knowledge, thereby mitigating common research challenges related to sufficient sample size, “representativeness” of sample populations, and the “generalizability” of study results.
Research networks are of two types—regional (typically less than 5 organizations, often privately funded and geographically organized) and national-scale networks, which include a larger number of organizations, wider geographical distribution, and often rely on federal or foundation funding for large centralized infrastructure. (These national networks are described in Sustainability: maintaining a national research ecosystem section.) To identify the factors that enable regional and national research networks, API2016 participants organized into smaller breakout groups with targeted discussion on policy issues surrounding multisite research. Specifically, these groups discussed information flows to improve reproducibility, quality, veracity and completeness of data, and barriers to data sharing. A number of common themes emerged, falling into the familiar categories of people, process, and technology. Later, the breakout groups reconvened into the larger API2016 group to develop consensus findings on the major challenges for networked research, which are presented below.
Finding: the clinical and research workforce, including institutional review board (IRB) staff, data stewards, and data curators often lack a fundamental understanding of informatics-driven research methodologies
Policy implication: informatics-driven research requires improved workforce competencies and federal support for informatics training
The healthcare and research enterprise envisioned for the 21st century will require a workforce competence beyond the mechanics of health IT and health information management.7,24 The API2016 participants identified three kinds of education and training that are necessary to EGM and the LHS: (1) basic “informatics literacy” for all health professionals incorporated with health practitioner education,25–27 biomedical research training,28 and public health education29; (2) intensive applied informatics training for an additional cohort of practitioners to improve leadership and expertise in applying informatics principles to healthcare problems30,31; and (3) Support for informatics education professionals who will advance the science and train the next generation of informatics professionals.32–34 “Informatics literacy” includes awareness of specific data standards and their importance to multisite research, the mechanics of data sharing, and the fulfillment of user needs that are specific to EBM, EGM, and learning cycles. Informatics professionals should be represented across all IRBs and grant selection committees where data sharing and informatics tools are part of the research.
Finally, a competent workforce for the future will depend upon federally funded recruitment and educational programs—targeted to high school, undergraduate, and professional students—that will encourage the next generation of health practitioners and researchers to engage in the science and practice of informatics.35 This includes continuation of federal support for programs at the National Library of Medicine,36 the Agency for Healthcare Research and Quality,37 and Clinical Informatics Fellows funded through Graduate Medical Education.38
Finding: incompatible data standards—including metadata and patient identifiers—and data use agreements create technical challenges and tremendous administrative burden on multisite research
Policy implication: convergence of technical standards and governance can facilitate multi-site research and reduce legal burden
Organizations should be able to “consume” research findings from other organizations and networks (as part of their LHS mission) and share results with the broader community as EGM. This requires technical standards for data representation and exchange.39–41 Regardless of whether they are in regional ad hoc research networks or part of nationally recognized research networks, health delivery organizations should be able to participate in multisite research, collaboration, and knowledge sharing. The development of a trusted exchange framework and common agreement, as required by the 21st Century Cures Act,42 is meant to ease legal burdens to exchange clinical data for providers in treating patients. A similar mechanism should be spearheaded to help private-sector stakeholders share clinical data for research purposes, especially for small institutions that wish to contribute to research and participate in EGM, but are not part of PCORnet,43 CTSA44 or other national research networks. Such trust and collaboration frameworks can be informed by the experience of the NIH Health Care Systems Research Collaboratory, one part of the overall effort to improve the national capacity to generate evidence that informs healthcare decisions by patients, providers, and payers.45,46 Indeed, via a series of demonstration projects using a pragmatic research approach, Collaboratory investigators have shared lessons learned regarding the design, execution, and dissemination research conducted in partnership with multiple healthcare systems.47 Among other recommendations, the Collaboratory EHR Core advocates for more federally driven (data) standards (to reduce the complexity and transformations required for multisite research) and policies to ensure that future electronic health record systems have the flexibility to support research, including implementation of new standardized data collection that fits into clinical workflows.45,46
Sustainability: maintaining a national research ecosystem
Finding: the effectiveness of our national research networks will be dependent upon shared infrastructure requiring adequate support from the private sector and federal agencies
Policy implication: federal funders should see an increasing proportion of their research portfolios as strategic investments rather than time-limited projects and coordinate them as such
Large scale infrastructure, supported by federal funding, is essential to the continued dominance of US biomedical research. This infrastructure includes both technology and the policies and procedures that enable best use of the technology. Federal investments in this infrastructure should take a long-term view, analogous to the concept of the National Science Foundation’s Supercomputing Program,48 rather than a project-based view, which places predetermined end points on federal support. Use of more agile funding mechanisms such Other Transaction Authority, and Cooperative Agreements, can help these programs work at the cutting edge of technology and be responsive to future needs in a rapidly changing environment.
Finding: the benefits of federally sponsored research must extend as much as possible beyond the awardees who conduct the research
Policy implication: federal policy must improve the use of research data and the application of new knowledge for multiple stakeholders to derive value
Policies should target the reuse of data through improved data sharing plans and harmonized data management/sharing policies across NIH institutions and centers. Similar policies could target reusability of biomedical knowledge that results from research studies, so this knowledge can be more readily applied to the improvement of health and care. Trans-NIH collaboration, as well as collaboration among other funding agencies, should compose a minimal percentage of the federal biomedical research portfolio so that research results can be more readily translated into practice. This concept should extend to the CMS, Veterans Health Administration and Defense Health Services, so that national-level research findings can be implemented at scale, as appropriate.
Finally, patients and families should be recognized as key stakeholders in our national research ecosystem, and they should be leveraged to enhance health IT systems for research, promote data standards, and help develop organizational research participation policies.49 Patients and the public will ultimately be the beneficiaries of research, and they can promote the value of research through participation and driving research questions. Federal research funders should promote projects that leverage health informatics tools that enable patients and their families to participate in research as part of their healthcare experience whatever their geographical location, actively share their own health data with researchers of their choosing, and obtain the return of results of research in which they participate.
CONCLUSION
The AMIA 2016 Policy Invitational brought together informatics experts from across the spectrum of care and research with national policy leaders for a structured dialog designed to address barriers to EGM within the context of regional and national-level research and participants identified a number of possible solutions that could be supported by policy and advocacy. The findings of API2016 confirm that strategic co-ordination and co-operation on a national scale will be required in order to achieve safe and effective patient care amid continuous advances in medical knowledge and proliferation of HIT tools. As a multidisciplinary, multistakeholder organization representing more than 5400 members, AMIA appreciates the need for synergy between people, technology, and systems. Innovation and impact in care delivery and clinical research will depend on public policies that encourage collaboration, co-operation, and shared digital infrastructure.
In order to achieve the vision of nationally scaled evidence-generating medicine where individual nodes contribute to larger networks and—ultimately—a sustainable LHS, national leadership, and investment is needed. Continued and increased funding for advanced health informatics training and cyberinfrastructure will be essential to ensure that biomedical research and EGM can be used to improve population health and healthcare delivery in the US. Without federal support to train clinicians and researchers on the fundamentals of health informatics literacy, we will fail to overcome current challenges and discover answers to important questions that face clinicians and patients. Further, without the cyberinfrastructure necessary to ask and answer clinical questions across disparate research networks, based on the best evidence-based knowledge and growing real-world data, we will not realize our vision of a LHS. The National Science Foundation has initiated important research in this area and NCATS has continued to fund large-scale projects. Moving forward, other agencies (eg AHRQ, ONC, CMS, CDC, and the VA) that would benefit from the promulgation of EGM should consider collaborative approaches to support the infrastructure required to generate evidence and implement findings more routinely and quickly.
Second, national policies should enable organizations to consume research findings from other organizations and networks easily, and share results with the broader community of healthcare and research practitioners. The development of trust frameworks, as required by the 21st Century Cures Act, is meant to ease legal burdens to exchange clinical data for providers in treating patients, so similar mechanisms should be spearheaded to help private-sector stakeholders share clinical data for research purposes, especially for small institutions who wish to contribute to research, but are not part of PCORnet, the CTSA program, or other national research networks. The NIH Collaboratory is vital resource for development of such processes and policies, and it should garner continued support from Congress and the White House. Additionally, revisions made to the Common Rule must be implemented in a co-ordinated fashion with the private sector, so that the benefits of these new changes can be realized across the country in consistent ways.
Finally, federal support is needed to ensure that better standards are developed and adopted in the private sector. Data standards are necessary to the exchange of clinical data and interoperability between health IT systems, are critical for data aggregation and sharing in research. The development, harmonization, and use of standards for vocabulary, format, and transport will be essential for the continued improvement of health IT tools necessary to support EGM and LHS. While interoperability is an on-going challenge, agreement on standards is foundational to realizing the potential of IT, and the federal government has important levers in certification and payment policies to help align private-sector adoption and use of health IT. The above recommendations will require the cooperation and engagement of many disciplines and stakeholders, but the return will have an impact on the costs of healthcare and biomedical research. Optimizing the use of health IT to support health care and research will create the infrastructure for LHS that ultimately can improve the lives of all Americans, now and into the future.
CONTRIBUTORS
All authors made substantial contributions to the conception and drafting of the manuscript. PJE, RR, and JS led the majority of the writing, with all authors reviewing and revising critically for important content and providing final approval of the version to be published.
FUNDING
The conference that informed this manuscript was supported by a grant (R13HS023969-01) from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
Conflict of interest statement. None declared.
REFERENCES
- 1. Friedman CP, Wong AK, Blumenthal D.. Achieving a nationwide learning health system. Sci Transl Med 2010; 257: 57cm29.. [DOI] [PubMed] [Google Scholar]
- 2. Smoyer WE, Embi PJ, Moffatt-Bruce S.. Creating local learning health systems: think globally, act locally. JAMA 2016; 31623: 2481–2. [DOI] [PubMed] [Google Scholar]
- 3. Malakoff D. Clinical trials and tribulations. Spiraling costs threaten gridlock. Science 2008; 3225899: 210. [DOI] [PubMed] [Google Scholar]
- 4. Vickers AJ. Clinical trials in crisis: four simple methodologic fixes. Clin Trials 2014; 116: 615–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Embi PJ, Payne PR.. Evidence generating medicine: redefining the research-practice relationship to complete the evidence cycle. Med Care 2013; 51(8 Suppl 3): S87–91. [DOI] [PubMed] [Google Scholar]
- 6. Califf RM. The Patient-Centered Outcomes Research Network: a national infrastructure for comparative effectiveness research. N C Med J 2014; 753: 204–10. [DOI] [PubMed] [Google Scholar]
- 7. IOM. The Learning Healthcare System: Workshop Summary. Washington, DC: The National Academies Press; 2007. [PubMed] [Google Scholar]
- 8. NIH. All of Us Research Program. Secondary All of Us Research Program; 2017. https://allofus.nih.gov/. Accessed September 1, 2018.
- 9. NCI. Cancer Moonshot Initiative. Secondary Cancer Moonshot Initiative. https://www.cancer.gov/research/key-initiatives/moonshot-cancer-initiative. Accessed September 1, 2018.
- 10. NIH Health Care Systems Research Collaboratory. Secondary NIH Health Care Systems Research Collaboratory. http://www.rethinkingclinicaltrials.org/about-nih-collaboratory/. Accessed September 1, 2018.
- 11. Sherman MD, Califf RM. What We Mean When We Talk About EvGen Part I: Laying the Foundation for a National System for Evidence Generation. FDA Voice Blog. Secondary What We Mean When We Talk About EvGen Part I: Laying the Foundation for a National System for Evidence Generation. FDA Voice Blog 2016. https://blogs.fda.gov/fdavoice/index.php/2016/04/what-we-mean-when-we-talk-about-evgen-part-i-laying-the-foundation-for-a-national-system-for-evidence-generation/.Accessed May, 1, 2016.
- 12.API2016 discussion website. Secondary API2016 discussion website. Available at: https://connect.amia.org/communities/community-home? CommunityKey=8fe48158-0b9f-4cd5-bbc3-9976c778d07d). Accessed November 1, 2016.
- 13. Smith MY, Hammad TA, Metcalf M, et al. Patient engagement at a tipping point—the need for cultural change across patient, sponsor, and regulator stakeholders: insights from the DIA Conference, “Patient engagement in benefit risk assessment throughout the life cycle of medical products”. Drug Inf J 2016; 505: 546–53. [DOI] [PubMed] [Google Scholar]
- 14. Terry SF, Patrick-Lake B.. Hearing voices: FDA seeks advice from patients. Sci Transl Med 2015; 7313: 313ed12.. [DOI] [PubMed] [Google Scholar]
- 15. Smith SK, Selig W, Harker M, et al. Patient engagement practices in clinical research among patient groups, industry, and academia in the United States: a survey. PLoS One 2015; 1010: e0140232.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Finney Rutten LJ, Alexander A, Embi PJ, et al. Patient-centered network of learning health systems: developing a resource for clinical translational research. J Clin Trans Sci 2017; 1(1): 40–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Fleurence R, Selby JV, Odom-Walker K, et al. How the Patient-Centered Outcomes Research Institute is engaging patients and others in shaping its research agenda. Health Aff (Millwood) 2013; 322: 393–400. [DOI] [PubMed] [Google Scholar]
- 18.Practice-Based Research Networks. Agency for Healthcare Research and Quality. https://pbrn.ahrq.gov. Accessed September 1, 2018.
- 19.Federal Policy for the Protection of Human Subjects, 82 Fed. Reg 7149. Federal Register. 19 Jan 2017. [PubMed]
- 20.Guidance on HIPAA and Individual Authorization of Uses and Disclosures of Protected Health Information for Research. Office of Civil Rights.
- 21.Section 2034 of the 21st Century Cures Act of 2016. PL 114–255 130 STAT 1060.
- 22. Sterman JD. Learning from evidence in a complex world. Am J Public Health 2006; 963: 505–14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Richesson RL, Hammond WE, Nahm M, et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory. J Am Med Inform Assoc 2013; 20 (e2): e226–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.In: Smith M, Saunders R, Stuckhardt L, McGinnis JM, eds. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington (DC: ): National Academies Press; 2013. [PubMed] [Google Scholar]
- 25. Shortliffe EH. Biomedical informatics in the education of physicians. JAMA 2010; 30411: 1227–8. [DOI] [PubMed] [Google Scholar]
- 26. Stead WW, Searle JR, Fessler HE, Smith JW, Shortliffe EH.. Biomedical informatics: changing what physicians need to know and how they learn. Acad Med 2011; 864: 429–34. [DOI] [PubMed] [Google Scholar]
- 27. Risling T. Educating the nurses of 2025: technology trends of the next decade. Nurse Educ Pract 2017; 22: 89–92. [DOI] [PubMed] [Google Scholar]
- 28. Valenta AL, Meagher EA, Tachinardi U, Starren J.. Core informatics competencies for clinical and translational scientists: what do our customers and collaborators need to know? J Am Med Inform Assoc 2016; 234: 835–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Brownson RC, Samet JM, Bensyl DM.. Applied epidemiology and public health: are we training the future generations appropriately? Ann Epidemiol 2017; 272: 77–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Kennedy MA, Moen A.. Nurse leadership and informatics competencies: shaping transformation of professional practice. Stud Health Technol Inform 2017; 232: 197–206. [PubMed] [Google Scholar]
- 31. Ingebrigtsen T, Georgiou A, Clay-Williams R, et al. The impact of clinical leadership on health information technology adoption: systematic review. Int J Med Inform 2014; 836: 393–405. [DOI] [PubMed] [Google Scholar]
- 32. Perlin JB. Health information technology interoperability and use for better care and evidence. JAMA 2016; 31616: 1667–8. [DOI] [PubMed] [Google Scholar]
- 33. Kannry J, Sengstack P, Thyvalikakath TP, et al. The Chief Clinical Informatics Officer (CCIO): AMIA Task Force Report on CCIO knowledge, education, and skillset requirements. Appl Clin Inform 2016; 71: 143–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Kannry J, Fridsma D.. The Chief Clinical Informatics Officer (CCIO). J Am Med Inform Assoc 2016; 232: 435.. [DOI] [PubMed] [Google Scholar]
- 35. Unertl KM, Finnell JT, Sarkar IN.. Developing new pathways into the biomedical informatics field: the AMIA High School Scholars Program. J Am Med Inform Assoc 2016; 234: 819–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. NLM. National Library of Medicine. NLM's University-based Biomedical Informatics and Data Science Research Training Programs. Secondary National Library of Medicine. NLM's University-based Biomedical Informatics and Data Science Research Training Programs. https://www.nlm.nih.gov/ep/GrantTrainInstitute.html. Accessed September 1, 2018.
- 37. AHRQ. Agency for Healthcare Research and Quality. AHRQ Health IT Funding Opportunities. Secondary Agency for Healthcare Research and Quality. AHRQ Health IT Funding Opportunities. https://healthit.ahrq.gov/ahrq-health-it-funding-opportunities. Accessed September 1, 2018.
- 38. AMIA. Clinical Informatics Fellowship Programs. Secondary Clinical Informatics Fellowship Programs. https://www.amia.org/programs/academic-forum/clinical-informatics-fellowships. Accessed September 1, 2018.
- 39. Haynes K, Selvam N, Cziraky MJ.. Bidirectional Data Collaborations in Distributed Research. Egems 2016; 42: 1.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. AMIA. AMIA Public Policy Principles and Positions. https://www.amia.org/public-policy/policy-priorities.
- 41.ONC. Office of the National Coordinator for Health IT. Interoperability Standards Advisory. https://www.amia.org/public-policy/policy-priorities. Accessed September 1, 2018.
- 42.Section 4003 of the 21st Century Cures Act of 2016. PL 114-255 130 STAT 1165.
- 43.The National Patient-Centered Clinical Research Network. PCORnet. Secondary The National Patient-Centered Clinical Research Network. PCORnet. http://www.pcornet.org/. Accessed September 1, 2018.
- 44. National Institutes of Health. National Center for Advancing Translational Sciences. Secondary National Institutes of Health. National Center for Advancing Translational Sciences. https://ncats.nih.gov/ctsa. Accessed September 1, 2018.
- 45. Weinfurt KP, Hernandez AF, Coronado GD, et al. Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol 2017; 171: 144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Richesson RL, Green BB, Laws R, et al. Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory. J Am Med Inform Assoc 2017; 245: 996–1001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials.
- 48. NSF. National Science Foundation. Envisioning supercomputers of the future. Secondary National Science Foundation. Envisioning supercomputers of the future. https://www.nsf.gov/discoveries/disc_summ.jsp?cntn_id=137961&org=NSF. Accessed December 1, 2018.
- 49. Terry SF. The study is open: participants are now recruiting investigators. Sci Transl Med 2017; 9371: eaaf1001. [DOI] [PubMed] [Google Scholar]
- 50. Lehmann CU, Longhurst CA, Hersh W, et al. clinical informatics fellowship programs: in search of a viable financial model: an open letter to the Centers for Medicare and Medicaid Services. Appl Clin Inform 2015; 62: 267–70. [DOI] [PMC free article] [PubMed] [Google Scholar]