Abstract
Introduction
To describe the development and implementation of learning health system (LHS) infrastructure for a pediatric specialty care health system to support LHS research in pediatric rehabilitation settings.
Methods
An existing pediatric common data model (eg, PEDSnet) of standardized medical terminologies for research was expanded and leveraged for this stud, and applied to SHOnet, a clinical research data resource consisting of deidentified data extracted from the electronic health record (EHR) from the Shriners Hospitals for Children speacialty pediatric health care system. We mapped EHR data for laboratory, procedures, drugs, and conditions to standardized vocabularies including ICD‐10, CPT, RxNorm, and LOINC to the common data model using an established extraction‐transformation‐loading process. Rigorous quality checks were conducted to ensure a high degree of data conformance, completeness, and plausibility. SHOnet data elements from all sources are de‐identified and the server is managed by the SHC Information Systems Department. SHOnet data are refreshed monthly and data elements are continually expanded based on new research endeavors.
Interventions
Not applicable.
Results
The Shriners Health Outcomes Network (SHOnet) includes data for over 10 000 distinct observational data elements based on over two million patient encounters between 2011 and present.
Conclusion
The systematic process to develop SHOnet is replicable and flexible for other pediatric rehabilitation research settings interested in building out their LHS capabilities. Challenges and facilitators may arise for building such LHS infrastructure for rehabilitation in areas of (a) data capture, curation, query, and governance, (b) generating knowledge from data, and (c) dissemination and implementation of new institutional knowledge. Further research studies are needed to evaluate these data resources for scalable system‐learning endeavors.
SHOnet is an exemplar of an LHS for rehabilitation and specialty care settings. The success of an LHS is dependent on engagement of multiple stakeholders, shared governance, effective knowledge translation, and deep commitment to long‐term strategies for engaging clinicians, administration, and families in leveraging knowledge to improve clinical outcomes.
Keywords: health informatics, infrastructure, learning health systems, quality improvement
1. INTRODUCTION
Large‐scale healthcare systems recognize the need to build infrastructures capable of continuous learning and improvement in care delivery to address critical population health issues. 1 , 2 In pediatric rehabilitation, existing research infrastructures require sustainable solutions to support the continuous learning and improvement necessary to provide real‐time evidence to patients and providers and realize improved patient rehabilitation outcomes. This concept, referred to as a learning health system (LHS), is a system‐level approach to improve care delivery, patient outcomes, and reduce costs by systematically and rapidly developing new knowledge as a by‐product of real‐world clinical data and experiences and implementing this knowledge into clinical practice. 3 , 4 , 5 , 6 , 7 Patients and families are integral members of the LHS infrastructure and become active participants in collaborative learning, clinical decision‐making, and knowledge translation.
Pediatric healthcare organizations and research networks use electronic health record (EHR) structures and data to develop system‐wide LHS activities for learning and improvement in best practices and processes of care delivery. 8 , 9 , 10 , 11 Many healthcare systems participate in learning networks as a part of an LHS or the infrastructure for an LHS. 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 Although real‐world examples of mature pediatric LHSs are sparse in the literature, the efforts by publicly and privately funded learning networks continue to support an extensive number of outcomes and comparative effectiveness research studies that improve the literature base for building LHS capacity. 9 , 23 , 25 , 26 Along with PEDSnet, other examples of pediatric‐specific research networks include ImproveCareNow and Learn From Every Patient. 9 , 19 , 23 , 27 However, the extent of LHS infrastructure, initiatives, and guidance specific to rehabilitation research is limited.
Therefore, in this paper, we aim to advance the science of LHS for rehabilitation by describing the process of developing the technical infrastructure of an LHS for a pediatric specialty care rehabilitation network, the Shriners Hospitals for Children (SHC) Health Outcomes Network (SHOnet). In addition, we discuss three major areas where challenges and facilitators (eg, existing structured programs that facilitate patient‐centered quality improvement activities; informaticians with expertise in interfacing with clinical researchers) may arise for building and expanding such LHS infrastructure and research enterprise to support large‐scale rehabilitation research across multiple levels and stakeholders within the SHC system based on our SHOnet experiences: (a) data capture, curation, query, and governance, (b) generating knowledge from data, and (c) dissemination and implementation of new institutional knowledge.
2. METHODS
2.1. Development of SHOnet
In 2015, SHC funded the SHOnet clinical research project to facilitate system‐wide multisite, efficient research across SHC care sites (Figure 1). The SHC system provides specialty services for children with orthopedic conditions, burns, spinal cord injuries, cleft lip, and other complex surgical needs. The long‐term objective of the SHOnet project is to establish a sustainable LHS infrastructure to support rapid, continuous learning about patient care and patient outcomes in SHC and serve as an incubator for research on dissemination and implementation of new practices. SHOnet adds to the landscape of existing LHSs given the cohort of child‐onset conditions seen in a pediatric specialty care network. Given this cohort and setting, the inclusion of specific rehabilitation and surgical data elements (eg, physical and occupational therapy forms, intra‐operative data) was a focus to build in the data model.
FIGURE 1.
Schematic of SHOnet learning model
SHOnet data derive from data routinely captured in an EHR by a variety of clinical providers. SHOnet was built based on the PEDSnet technological infrastructure, which is a pediatric‐specific learning network, or clinical research network (CRN), funded by the Patient‐Centered Outcomes Research Institute (PCORI). PCORI funds PCORnet, the National Patient‐Centered Clinical Research Network, and its federated network of CRNs in the United States to improve the digital infrastructure for outcomes research and comparative clinical effectiveness research, and engage all stakeholders (patients, families, clinicians, researchers, health system leaders, health systems) from over 70 care sites to improve health. 12 , 15 , 25 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 These CRNs provide foundational data infrastructure, such as the PCORnet common data model (CDM), and data quality processes for PCOR and learning activities. Through collaboration with a community of SHC clinical and system leaders, patients, and their families, SHOnet supports large‐scale system research studies to impact clinical practice and improve the health of children cared for by SHC sites.
Early efforts in SHOnet focused on establishing governance and strategic planning on data access, quality, and the build of the data resources to accomplish SHOnet Phase 1 (Years 1‐3) aims (Figure 2). The initial development of SHOnet in 2015 required institutional support across three operationally distinct systems: research, medical affairs, and informatics. One of the first challenges for this cross‐cutting system project was defining the resources and key project proponents from each subdivision. In this case, the knowledge translation research needs were supported by the research division, while medical affairs supported the technical, informatics, and performance improvement expertise. In addition, legal, compliance, healthcare quality, and clinical informatics divisions supported those critical areas of the network. SHOnet Phase 2 activities continue to refine the processes developed in Phase 1 while expanding, utilizing, and sustaining the rapid generation of new, highly impactful knowledge about best practices for patient care and translate that new knowledge at the point of care. The primary aims of SHOnet Phase 2 are described in Figure 2. The final deliverable of SHOnet is to have a fully functioning and sustainable data and scientific infrastructure that is interoperable with other national CRNs and LHS infrastructures, and that will serve as a mechanism to develop active research programs with a clear and measurable impact on patient care and outcomes across SHC.
FIGURE 2.
Aims for each phase of SHOnet
SHC, although multisite, is more of a homogenous organizational care system than care systems in other CRNs, and this facilitates leveraging EHR data elements for system‐wide measurement and learning purposes. Despite its potential practice variations by geographic locations, SHC has uniformity in many areas that are excellent criteria for building a mature CRN for LHS 35 : (a) SHC has the same primary EHR vendor across all SHC care sites; (b) It maintains a similar EHR structure and interface across sites referred to as the SHC Information System; (c) SHC has long‐term experience using an EHR with full, system‐wide EHR implementation since 2005; (d) SHC provides specialty care to a relatively narrow set of health conditions; (e) SHC employs health professionals that share the organization's mission, vision, and values; and (f) SHC requires all clinicians to complete the same system‐wide standardized trainings and clinical competencies. All these factors minimize, but do not eliminate, the variation in practices across the care system. However, given the SHC model of care includes more charity care than many other systems, some aspects of EHR documentation are less mature than others, for instance, Current Procedural Terminology (CPT) coding and other billing technologies and requirements.
2.2. Informatics: Building the SHOnet data resource
SHOnet is based on the pediatric‐specific CDM used by PEDSnet. PEDSnet, which consists of eight large pediatric healthcare systems in the United States, modified the Observational Medical Outcomes Partnership (OMOP) CDM V5 19 , 20 , 36 , 37 with a defined set of pediatric common data elements. OMOP is designed to enable research on associations between healthcare interventions and outcomes for specific patient cohorts (eg, patients with a certain condition or those who have undergone a specific procedure). PCORnet and other large national data research consortia use this CDM approach to standardize data structures for heterogeneous partners. 38 Therefore, SHOnet data are interoperable with data maintained by other CRNs that use OMOP and existing terminology standards (eg, PEDSnet).
In the initial build of SHOnet, our team queried available EHR discrete data elements and clinical events in the EHR structure as well as from related embedded applications for surgical and radiological data such as Surginet, PHARMnet, and RADnet. Additional data, such as information on the care providers (eg, profession, specialty practice areas, gender, age, etc), or specific orthotics/prosthetics devices rely on extraction from non‐EHR (external) databases. A team of experienced clinicians and bioinformaticians developed crosswalk tables that mapped the EHR “event codes” for all identified data elements to OMOP codes. Laboratory, procedures, drugs, and conditions represented as standardized vocabularies in the EHR included International Classification of Disease‐10 (ICD‐10) for conditions, CPT codes for procedures, RxNorm for drug exposures, and Logical Observation Identifiers Names and Codes (LOINC) for many laboratory and culture data elements. We focused on mapping the clinical/EHR source data to existing OMOP codes, but if no existing standard OMOP codes were available for an EHR data element, custom OMOP codes were assigned. In SHOnet, this was necessary for many of the physical and occupational therapy, patient‐reported outcomes and pediatric clinical performance tests, and measures present in the OMOP measurement and observation standard tables. This process did not change the OMOP data model itself; rather, it allowed us to assign customized OMOP concept codes for many of the measurement and observation table elements that did not have existing standard terminology. Over time, OMOP expanded its standard data elements (eg, PROMIS items are now mapped to SNOMED and OMOP); however, the need to assign custom OMOP codes early on for many observation and measurement table data elements was driven by the fact that the PEDSnet modified OMOP CDM did not include codes for these data elements when we built out this infrastructure. This resulted in the addition of approximately 1500 custom‐coded data elements, which were shared with the OMOP community. For example, OMOP custom codes were created for the physical therapy clinical examination elements (eg, muscle tone, right gastrocnemius OR ankle plantarflexion with knee extended, passive, right) and several of the developmental assessment item‐level elements (eg, Gross Motor Function Classification System [GMFCS] level, rolling back to front, etc) or parent‐reported development questions.
Once these crosswalk tables linking EHR codes to the OMOP CDM were completed and verified, the extraction‐transformation‐loading (ETL) process was implemented in Oracle Data Integrator tools by a single ETL analyst. The “data build” process required an iterative approach, whereby as additional data elements were identified, they were crosswalked to the OMOP model, the ETL was performed, data underwent extensive quality reviews, and the ETL process was subsequently adjusted. Data quality reviews included monthly ETL and informatic level review (eg, significant changes in levels of missingness and counts from month to month) and periodic confirmation of source value (raw) data from EHR reviews as well as review by clinician experts‐informatician teams for specific use cases to confirm values and correctness of the transformation process. During this time, monthly data refreshes were completed to pull in current clinical care data, as well as any expanded elements across all time. The SHOnet data resource undergoes rigorous quality checks to ensure a high degree of data conformance, completeness, and plausibility that aligns with existing practices. 39 The current SHOnet data resource now contains over a decade worth of data elements, across multiple OMOP categories or tables (eg, procedure, diagnosis, conditions, drugs, measurements, clinical and psychosocial outcome tools, observations, providers).
3. RESULTS
SHOnet includes discrete data elements that were generated as part of routine clinical practice during over two million patient encounters that took place from 2011 to 2021. Given the nature of the cohort and setting, the SHOnet observation and measurement tables remain consistent with OMOP standard CDM but have significant expansion in the number and breadth of data elements critical for pediatric rehabilitation research. For the user‐facing open‐access SHOnet data portal, de‐identification consistent with safe harbor criteria is performed to allow for broad engagement in “prep‐for‐research” activities such as cohort identification within the SHC system. SHOnet data elements from all sources are de‐identified (stripped of information that would allow end‐users to identify individual patients), transformed into the CDM, and loaded onto a secure, dedicated server. The server is managed by the SHC Information Systems Department and monthly audits are conducted. Additional data span as far back as 2007, but the data are considered high quality after 2011 because of stabilization of the billing and coding system at SHC. A unique code to the set of de‐identified health information is assigned to each patient to permit re‐identification for IRB‐approved studies that is managed by an honest data broker in the SHOnet data coordinating center.
SHOnet can be operationalized for system research and learning purposes and presents an opportunity to move the fields of learning health sciences and rehabilitation sciences forward to improve and innovate measurement of rehabilitation practice. The SHOnet data resource is a massive repository of de‐identified patient‐level data that are extracted from the SHC Information System, and other clinical and administrative databases. Currently, SHOnet is expanding to include data from a wide range of patient/family‐generated sources, including caregiver and patient‐reported outcomes, wearable sensor data (eg, activity trackers, geo‐trackers), and data generated as by‐products of healthcare delivered outside the SHC system (eg, external pharmacies, non‐SHC healthcare providers).
4. DISCUSSION
This study describes our approach to build SHOnet and rigor necessary for other institutions to support a pediatric rehabilitation LHS research enterprise. SHOnet is an exemplar of an LHS for pediatric rehabilitation settings. SHOnet is currently used to design and study multiple practices and processes across SHC, and we continue to identify additional data elements to expand scope and scale of system research endeavors. SHOnet also provides a reusable CDM and ETL for other pediatric rehabilitation research settings to extract EHR data and study and inform point‐of‐care rehabilitation interventions and generate real‐world knowledge as a by‐product of healthcare delivery.
There is growing interest by healthcare systems specializing in pediatric rehabilitation, such as SHC, to design mechanisms for continuous learning and improvement of rehabilitation treatments and to address critical rehabilitation issues, reduce healthcare costs, and compete in a complex healthcare environment. 1 , 2 One of the areas where learning is needed is in understanding clinician workflow, value, and documentation of discrete data elements in the EHR. 40 , 41 SHOnet is focused on addressing this gap. Clearly, rich clinical information resides in “free‐text” electronic or transcribed clinical notes; however, the process of extracting this information is difficult due to the complexity of achieving complete patient de‐identification. In rehabilitation settings, documentation of discrete data is more often used to characterize the nuances of patients. These discrete data elements also help characterize system‐level issues, and thus, are integral to measuring clinical performance, quality, and patient health outcomes. A challenge, however, is that generalizability of large‐scale clinical findings for research purposes is often difficult due to variability in clinical vocabularies and terminologies of institution‐specific data models. While EHR data can support the development of real‐world knowledge, not all discrete data elements are mapped to common terminologies in other LHS data infrastructures. SHOnet elevates the value of discrete data for research and learning in pediatric rehabilitation. Many of these mapped discrete data elements in SHOnet may support classifiers of functional performance and rehabilitation outcomes integral to improving care delivery and learning. 42
The EHR is an essential infrastructure for a functioning LHS. However, in our experience extracting data from EHRs, we noticed that EHRs are currently not conducive to efficient assessment of system‐wide clinical performance, especially in rehabilitation settings. Extracted data require systematic, extensive quality control procedures and data are often difficult to interpret without understanding nuances of clinical practices within individual care systems. The structure of EHRs was developed with a primary goal of replacing traditional paper‐based documentation to streamline care and care coordination. The structure, approach, and practice to documentation have changed little aside from accommodating the transition from paper to electronic modes. 43 Documentation practices primarily focused on information completeness to meet clinical decision‐making for individual patients and for reimbursement requirements 44 and include a mix of unstructured narrative formats and structured fields for “free‐text” and discrete data entry.
As the concept and field of LHS have evolved, key elements of an efficient and effective LHS include the composition of a core LHS team, identification of external supports, barriers and facilitators to data curation, queries and governance, and knowledge generation and dissemination with stakeholders. Additionally, competencies of LHS researchers have been specified 45 and several training programs have emerged. For rehabilitation researchers, the recently funded P2C Medical Rehabilitation Research Resource Centers provide a specialized opportunity for training in LHS science: notably the Learning Health Systems Rehabilitation Research Network (LeaRRn). While PCORI and privately funded CRNs maintain critical infrastructure for pediatric‐centered LHS activities, there is minimal inclusion of the rehabilitation and perioperative detail that is embedded in SHOnet, and minimal efforts exist to expand these infrastructures for pediatric rehabilitation. PCORI and privately funded LHS CRNs also do not incorporate and evaluate rehabilitation‐related data elements in their CDMs, 46 despite the potential benefit to measure clinical effectiveness of surgical and rehabilitation interventions during routine care. These LHS data infrastructures are built across disparate healthcare systems employing numerous institutional review boards and discrete types of patient data routinely collected, especially by rehabilitation professionals, which influences coordinated dissemination and implementation science research across institutions. Based on our experiences with building SHOnet, we identify three key challenges and facilitators for building institutional LHS infrastructure in pediatric rehabilitation settings.
4.1. Data capture, Curation, query, and governance of data resources
The SHOnet data infrastructure supports SHC‐wide research studies that are clinically impactful, address practice variation and quality of care, and improve the health of its patients. Asher et al describe this concept as a science of practice that is “not simply a response to abstract or irrelevant external requirements, but an opportunity to survive and indeed thrive amid the increasing competition and demands of the informatics age.” 47 While SHC is a highly specialized pediatric care system, building a culture of learning and a science of practice is essential, specifically, for measuring value of care delivery and patient outcomes informed by system‐derived knowledge and advance practice innovation. The processes and infrastructure necessary to stabilize an organizational LHS surrounding pediatric rehabilitation are emerging.
One of the areas we focused our efforts on related to this is understanding and improving mechanisms for capturing data on functional performance for patients with cerebral palsy (CP). Our early SHOnet work incorporated the formation of a collaborative group of clinicians with deep expertise in the care of children with CP. This group informed the development and prioritization of data elements for extraction, development of use cases that would inform practice, and are involved in the ongoing dissemination of the findings from several of the studies. The use cases included phenotyping of CP, hip surveillance protocols for children with CP, effect of anesthesia type on re‐catherization post‐operatively, and predictors of blood loss during orthopedic surgery in children with CP. These use cases help inform both care practices, as well as the building of the LHS. Several gaps in the data for children with CP that were necessary for SHOnet included outcomes and measures documented in the rehabilitation forms (eg, range of motion, functional status, etc). For example, the workgroup noted that the Gross Motor Function Classification System (GMFCS) level as a clinical decision‐making tool was viewed as increasingly important, particularly with respect to new treatment guidance such as hip surveillance protocols for children with CP. In the currently configured EHR, GMFCS documentation as a discrete data element (numeric data 1‐5) was present on a form typically filled out only by occupational and physical therapists, with physicians including GMFCS levels within their visit notes as narrative free‐text. To address this issue, physicians at one site began documenting GMFCS in the diagnostic section as a discrete data element. Initially, the EHR coders later mapped these to a standard diagnostic code (required) to “Cerebral Palsy‐unspecified.” This caused a loss of information on GMFCS as well as an increase in the frequency of “Cerebral Palsy‐unspecified” when ICD codes for CP were queried. This was addressed across the system by informing the physicians and others documenting in the EHR, as well as the system‐level informatics teams. This again highlights the need for the entire system: from clinician to EHR and data coding teams to work together to simultaneously solve issues and provide transparency across levels to address issues efficiently and comprehensively. This demonstrates a need to understand the routine processes to collect and document clinical data and to inform both the coders and clinicians of how clinical data may be produced and utilized for generating real‐world clinical knowledge.
4.2. Generating knowledge from data
A core LHS team of researchers supports the use of SHOnet data resources to optimize an understanding of practice patterns and patient outcomes for SHC. The SHOnet core team of researchers designated cerebral palsy (CP) as an initial use case for operationalizing the LHS structure and processes in SHOnet. CP is one of the largest patient populations managed by SHC, with many system‐wide opportunities to generate meaningful knowledge and inform clinical practice for managing CP. A CP collaborative research group of SHC investigators was formed to develop LHS‐focused research priorities for children with CP. As noted earlier, the group drafted use cases based on consensus criteria to test the value of SHOnet for measuring functional performance, perioperative outcomes, and long‐term surveillance programs of CP cohorts, which are instrumental to measure quality of care, patient health outcomes, and response to treatment interventions. Subsequently, the research group developed demonstration projects in parallel to the development of SHOnet to help operationalize processes within a realistic use case. The findings were discussed by the collaborative research group internally and are being disseminated through scientific publications. Even with its organizational homogeneity, there remains a long‐term need to identify and remediate gaps and inconsistencies of key data elements as additional research needs and projects emerge.
As SHOnet matured in terms of the data resource, the core team focused on provision of data visualization tools and dashboards for end‐users, and support for application of machine learning and similar advanced analytic tools. Support for the use of SQL analytics as well as PowerBI (Microsoft) platforms was developed with a train‐the‐trainer program initiative. Given the incremental nature of the underlying EHR over time (eg, addition of documentation modules, change in use of standardize vocabularies etc), the importance of clear documentation of when and how underlying documentation changed became critical (eg, searching based on ICD9 codes after ICD10 was introduced) in preparation for the widespread use of more advanced analytics. Future work involves inclusion of natural language processing to extract information from free‐text data (eg, detailed clinical and surgical notes) and mechanisms to extract radiographic images. The difficulty in ensuring consistent de‐identification of such data remains an issue not only for SHOnet but for the field of clinical research informatics.
4.3. Dissemination and implementation of new institutional knowledge
In addition to the data resources, SHOnet is an organizational initiative to support a climate of LHS research and practice across the 20 geographically diverse specialized pediatric healthcare centers in the SHC system. Based on our experiences and consistent with LHS principles, we recommend early and sustained incorporation of hospital administration, clinical and research stakeholders within a care system contemplating establishing an LHS. The initiation and sustained successes of SHOnet build on a system of support from SHC administration, medical affairs, information services, compliance and research department stakeholders. Involvement of individuals both “top‐down” and “bottom‐up” are instrumental to SHOnet's success and require consistent communication and synergy among all. The optimization of coordination across sites, as well as effective dissemination and implementation activities, is a critical priority and one of the most challenging for the LHS core team. The challenge arises not from any stakeholder group's lack of enthusiasm or support but the competing priorities at a day‐to‐day level for individuals whose primary responsibilities lie in administering and measuring success of the hospital network.
Through blended support and champions from both research and medical affairs at SHC Headquarters, priorities and timelines for LHS activities were identified and the SHOnet team focused on building capacity in four key elements of LHS: data infrastructure, stakeholder engagement, data quality monitoring and control, and knowledge translation. Our experiences support the earlier identification of specific research affinity groups that provide relevant, impactful use cases for the system and also serve as content experts for building the data warehouse as well as engagement of other clinicians in changing practice. SHOnet's early and continued success relies on the open communication, information‐sharing, consistency in strategic activities, respect for often competing priorities, and a constant high value of the project by all key stakeholders. The successful synergy across stakeholders and competing priorities support the successful dissemination of knowledge and uptake of LHS approaches to research.
5. CONCLUSION
SHOnet is a pediatric rehabilitation CRN that was established in 2015 to support an LHS research enterprise across SHC. Early years focused on building the data infrastructure and was faced with challenges of extraction of data from the EHR and related databases, gaining familiarity with EHRs, understanding that the EHR was built out of many modules and evolved over time, and required investment in developing an LHS team that works across multiple areas. Ongoing challenges remain related to data governance, data access, management, and sustainability; however, we have built foundational LHS infrastructure for SHC and infrastructure that is reusable by other healthcare systems and CRNs to expand LHS capacity for rehabilitation research.
CONFLICT OF INTEREST
The authors declare no conflicts of interest.
Abbreviations
- CDM
common data model
- CDRN
clinical data research network
- CP
cerebral palsy
- CPT
current procedural terminology
- EHR
electronic health record
- ETL
extraction‐transformation‐loading
- ICD
International Classification of Disease
- LHS
learning health systems
- LOINC
Logical Observation Identifiers Names and Codes
- OMOP
Observational Medical Outcomes Partnership
- PCORI
Patient‐Centered Outcomes Research Institute
- SHC
Shriner's Hospitals for Children
- SHOnet
Shriners Health Outcomes Network
ACKNOWLEDGMENTS
The Shriners Hospitals for Children Learning Health System (SHOnet) was funded by Shriners Hospitals for Children clinical research grant #79157 (PI: Carole A. Tucker, PhD). The content presented here is solely the responsibility of the authors and does not necessarily represent the official views of the Shriners Hospital for Children.
Koscielniak N, Jenkins D, Hassani S, et al. The SHOnet learning health system: Infrastructure for continuous learning in pediatric rehabilitation. Learn Health Sys. 2022;6(3):e10305. doi: 10.1002/lrh2.10305
Funding information Shriners Hospitals for Children, Grant/Award Number: Clinical Research Grant/79157
REFERENCES
- 1. Etheredge LM. A rapid‐learning health system. Health Aff. 2007;26(2):w107‐w118. doi: 10.1377/hlthaff.26.2.w107 [DOI] [PubMed] [Google Scholar]
- 2. Budrionis A, Bellika JG. The learning healthcare system: where are we now? A systematic review. J Biomed Inform. 2016;64:87‐92. doi: 10.1016/j.jbi.2016.09.018 [DOI] [PubMed] [Google Scholar]
- 3. Friedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high‐functioning learning health system. J Am Med Inform Assoc. 2015;22(1):43‐50. doi: 10.1136/amiajnl-2014-002977 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Institute of M . The National Academies Collection: reports funded by National Institutes of Health. In: Grossmann C, Powers B, JM MG, eds. Digital Infrastructure for the Learning Health System: the Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Washington (DC): National Academies Press (US), National Academy of Sciences; 2011. [PubMed] [Google Scholar]
- 5. Grossmann C. Institute of Medicine (US). Roundtable on Value & Science‐Driven Health Care: Clinical Data as the Basic Staple of Health Learning: Creating and Protecting a Public Good: Workshop Summary. Washington, DC: National Academies Press; 2010. [PubMed] [Google Scholar]
- 6. Institute of Medicine Roundtable on Evidence‐Based M . The National Academies Collection: reports funded by National Institutes of Health. In: Olsen LA, Aisner D, JM MG, eds. The Learning Healthcare System: Workshop Summary. Washington (DC): National Academies Press (US), National Academy of Sciences; 2007. [PubMed] [Google Scholar]
- 7. Friedman CP, Rubin JC, Sullivan KJ. Toward an information infrastructure for Global Health improvement. Yearb Med Inform. 2017;26(1):16‐23. doi: 10.15265/iy-2017-004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Cameron CB. Users Guide to Computable Phenotypes. Durham, NC: Duke University; 2016. [Google Scholar]
- 9. Crandall W, Kappelman MD, Colletti RB, et al. ImproveCareNow: the development of a pediatric inflammatory bowel disease improvement network. Inflamm Bowel Dis. 2011;17(1):450‐457. doi: 10.1002/ibd.21394 [DOI] [PubMed] [Google Scholar]
- 10. Forrest CB, Crandall WV, Bailey LC, et al. Effectiveness of anti‐TNFalpha for Crohn disease: research in a pediatric learning health system. Pediatrics. 2014;134(1):37‐44. doi: 10.1542/peds.2013-4103 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Walsh KE, Margolis P, Marsolo KA, et al. Accuracy of the medication list in the electronic health record—implications for care, research, and improvement. J Am Med Inform Assoc. 2018;25(7):909‐912. doi: 10.1093/jamia/ocy027 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Amin W, Tsui FR, Borromeo C, et al. PaTH: towards a learning health system in the mid‐Atlantic region. J Am Med Inform Assoc. 2014;21(4):633‐636. doi: 10.1136/amiajnl-2014-002759 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Califf RM. The patient‐centered outcomes research network: a national infrastructure for comparative effectiveness research. N C Med J. 2014;75(3):204‐210. [DOI] [PubMed] [Google Scholar]
- 14. Hartzler AL, Chaudhuri S, Fey BC, Flum DR, Lavallee D. Integrating patient‐reported outcomes into spine surgical care through visual dashboards: lessons learned from human‐centered design. EGEMS. 2015;3(2):1133. doi: 10.13063/2327-9214.1133 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Mandl KD, Kohane IS, McFadden D, et al. Scalable collaborative infrastructure for a learning healthcare system (SCILHS): architecture. J Am Med Inform Assoc. 2014;21(4):615‐620. doi: 10.1136/amiajnl-2014-002727 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. McGlynn EA, Lieu TA, Durham ML, et al. Developing a data infrastructure for a learning health system: the PORTAL network. J Am Med Inform Assoc. 2014;21(4):596‐601. doi: 10.1136/amiajnl-2014-002746 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Randhawa GS. Building electronic data infrastructure for comparative effectiveness research: accomplishments, lessons learned and future steps. J Comp Eff Res. 2014;3(6):567‐572. doi: 10.2217/cer.14.73 [DOI] [PubMed] [Google Scholar]
- 18. Bhandari RP, Feinstein AB, Huestis SE, et al. Pediatric‐collaborative health outcomes information registry (Peds‐CHOIR): a learning health system to guide pediatric pain research and treatment. Pain. 2016;157(9):2033‐2044. doi: 10.1097/j.pain.0000000000000609 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Forrest CB, Margolis P, Seid M, Colletti RB. PEDSnet: how a prototype pediatric learning health system is being expanded into a national network. Health Aff. 2014;33(7):1171‐1177. doi: 10.1377/hlthaff.2014.0127 [DOI] [PubMed] [Google Scholar]
- 20. Forrest CB, Margolis PA, Bailey LC, et al. PEDSnet: a national pediatric learning health system. J Am Med Inform Assoc. 2014;21(4):602‐606. doi: 10.1136/amiajnl-2014-002743 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Lannon CM, Peterson LE. Pediatric collaborative improvement networks: background and overview. Pediatrics. 2013;131(Suppl 4):S189‐S195. doi: 10.1542/peds.2012-3786E [DOI] [PubMed] [Google Scholar]
- 22. Margolis PA, Peterson LE, Seid M. Collaborative chronic care networks (C3Ns) to transform chronic illness care. Pediatrics. 2013;131(Suppl 4):S219‐S223. doi: 10.1542/peds.2012-3786thinspaceJ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Marsolo K, Margolis PA, Forrest CB, Colletti RB, Hutton JJ. A digital architecture for a network‐based learning health system: integrating chronic care management, quality improvement, and research. EGEMS. 2015;3(1):1168. doi: 10.13063/2327-9214.1168 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207‐210. doi: 10.7326/0003-4819-157-3-201208thinspace070-00012 [DOI] [PubMed] [Google Scholar]
- 25. Britto MT, Fuller SC, Kaplan HC, et al. Using a network organisational architecture to support the development of learning healthcare systems. BMJ Qual Saf. 2018;27:937‐946. doi: 10.1136/bmjqs-2017-007219 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Adler J, Dong S, Eder SJ, Dombkowski KJ. Perianal Crohn disease in a large multicenter pediatric collaborative. J Pediatr Gastroenterol Nutr. 2017;64(5):e117‐e124. doi: 10.1097/mpg.0000000000001447 [DOI] [PubMed] [Google Scholar]
- 27. Evans RS, Lloyd JF, Pierce LA. Clinical use of an enterprise data warehouse. AMIA Annu Symp Proc. 2012;2012:189‐198. [PMC free article] [PubMed] [Google Scholar]
- 28. Ali J, Califf R, Sugarman J. Anticipated ethics and regulatory challenges in PCORnet: the national patient‐centered clinical research network. Account Res. 2016;23(2):79‐96. doi: 10.1080/08989621.2015.1023951 [DOI] [PubMed] [Google Scholar]
- 29. Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med Inform Assoc. 2014;21(4):576‐577. doi: 10.1136/amiajnl-2014-002864 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Corley DA, Feigelson HS, Lieu TA, McGlynn EA. Building data infrastructure to evaluate and improve quality: PCORnet. J Oncol Pract. 2015;11(3):204‐206. doi: 10.1200/jop.2014.003194 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient‐centered clinical research network. J Am Med Inform Assoc. 2014;21(4):578‐582. doi: 10.1136/amiajnl-2014-002747 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Kaushal R, Hripcsak G, Ascheim DD, et al. Changing the research landscape: the new York City clinical data research network. J Am Med Inform Assoc. 2014;21(4):587‐590. doi: 10.1136/amiajnl-2014-002764 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Ohno‐Machado L, Agha Z, Bell DS, et al. pSCANNER: patient‐centered scalable National Network for effectiveness research. J Am Med Inform Assoc. 2014;21(4):621‐626. doi: 10.1136/amiajnl-2014-002751 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Waitman LR, Aaronson LS, Nadkarni PM, Connolly DW, Campbell JR. The Greater Plains collaborative: a PCORnet clinical research data network. J Am Med Inform Assoc. 2014;21(4):637‐641. doi: 10.1136/amiajnl-2014-002756 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Lannon C, Schuler CL, Seid M, et al. A maturity grid assessment tool for learning networks. Learn Health Syst. 2020;5:e10232. doi: 10.1002/lrh2.10232 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Observational Medical Outcomes Partnership . OMOP Common Data Model – OHDSI. 2020. https://www.ohdsi.org/data-standardization/the-common-data-model/
- 37. Stang PE, Ryan PB, Racoosin JA, et al. Advancing the science for active surveillance: rationale and design for the observational medical outcomes partnership. Ann Intern Med. 2010;153(9):600‐606. doi: 10.7326/0003-4819-153-9-201011thinspace020-00010 [DOI] [PubMed] [Google Scholar]
- 38. Institute PCOR . PCORnet Common Data Model (CDM). 2018.
- 39. Khare R, Ruth BJ, Miller M, et al. Predicting causes of data quality issues in a clinical data research network. AMIA Jt Summits Transl Sci Proc. 2018;2017:113‐121. [PMC free article] [PubMed] [Google Scholar]
- 40. Koscielniak N, Tucker C. Thinking beyond SOAP: clarifying clinical documentation for daily‐use in pediatric rehabilitation learning health systems. Arch Phys Med Rehabil. 2020;101(11):e79–e80. doi: 10.1016/j.apmr.2020.09.241 [DOI] [Google Scholar]
- 41. Koscielniak NJ, Tucker CA, Grogan‐Kaylor A, et al. Evaluating completeness of discrete data on physical functioning for children with cerebral palsy in a pediatric rehabilitation learning health system. Phys Ther. 2021;102(1):1‐10. doi: 10.1093/ptj/pzab234 [DOI] [PubMed] [Google Scholar]
- 42. Koscielniak N, Piatt G, Friedman C, Vinson A, Richesson R, Tucker C. Development of a standards‐based phenotype model for gross motor function to support learning health systems in pediatric rehabilitation. Learn Health Syst. 2021;6:1‐10. doi: 10.1002/lrh2.10266 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Rosenbloom ST, Denny JC, Xu H, Lorenzi N, Stead WW, Johnson KB. Data from clinical notes: a perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc. 2011;18(2):181‐186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Embi PJ, Weir C, Efthimiadis EN, Thielke SM, Hedeen AN, Hammond KW. Computerized provider documentation: findings and implications of a multisite study of clinicians and administrators. J Am Med Inform Assoc. 2013;20(4):718‐726. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Forrest CB, Chesley FD Jr, Tregear ML, Mistry KB. Development of the learning health system researcher core competencies. Health Serv Res. 2017;53:2615‐2632. doi: 10.1111/1475-6773.12751 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Pletcher MJ, Forrest CB, Carton TW. PCORnet's collaborative research groups. Patient Relat Outcome Meas. 2018;9:91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47. Asher AL, McCormick PC, Selden NR, Ghogawala Z, McGirt MJ. The national neurosurgery quality and outcomes database and neuropoint alliance: rationale, development, and implementation. Neurosurg Focus. 2013;34(1):E2. [DOI] [PubMed] [Google Scholar]