The DARTNet Institute: Seeking a Sustainable Support Mechanism for Electronic Data Enabled Research Networks

Abstract

Context:

Clinical data research networks require large investments in infrastructure support to maintain their abilities to extract, transform, and load data from varied data sources, expand electronic data sources and develop learning communities.

Case Description:

This paper outlines a sustainable business model of ongoing infrastructure support for clinical data research activities. The DARTNet Institute is a not-for-profit 501(c)(3) organization that serves as a support entity for multiple practice-based research networks. Several clinical data research networks working closely with a professional society began collaborating to support shared goals in 2008. This loose affiliation called itself the “DARTNet Collaborative.” In 2011, the DARTNet Institute incorporated as an independent, not-for-profit entity. The business structure allows DARTNet to advocate for all partners without operating its own practice-based research network, serve as a legal voice for activities that overlap multiple partners, share personnel resources through service contracts between partners, and purchase low-cost (nonprofit rate) software.

Major Themes:

DARTNet’s business model relies upon four diverse sources of revenue: (1) DARTNet licenses and provides access to a propriety software system that extracts, transforms, and loads data from all major electronic health records (EHRs) utilized in the United States, and which also provides clinical decision support for research studies; (2) DARTNet operates a recognized, national professional-society-quality improvement registry that enables organizations to fulfill Meaningful Use 2 criteria; (3) DARTNet provides access to data for research activities that are funded by direct research dollars, provided at prices that generate excess revenue; and (4) DARTNet provides access to large primary care datasets for observational studies and pregrant analyses such as for sample size development. The ability of the system to support pragmatic trials will be described.

Conclusion:

The DARTNet model facilitates the use of direct grant dollars to generate revenue to support the overall enterprise through a purchased services arrangement. Other services provided through subcontracting provide facilities and administration fees as well as direct dollars to support the system. The flexibility of the business model overcomes the complicated financial arrangements and governance requirements of many professional associations and academic medical centers.

Introduction

Practice-Based Research, Pragmatic Trials and Access to Electronic Data

Practice-based research networks (PBRNs) have been in existence for over 30 years and include a wide variety of organizing disciplines ranging from primary care to oncology to neonatal intensive care.1–4 Several features distinguish PBRNs from academic research collaboratives.5–7 Three of the characteristics considered essential to PBRNs by the Agency for Healthcare Research and Quality (AHRQ) include participation of nonacademic clinical locations “devoted principally to the care of patients,” a “shared mission to investigate questions” important to improving the quality of care delivered in their practice settings, and direct data collection from clinicians primarily engaged in patient care.3 Maintaining an engaged and productive network that is able to provide high reliability research activities and attract potential investigators requires a skilled and committed central organizing body to facilitate collaboration, to lessen the burden of research participation for practices, and to govern the network such that that all members receive benefits from participation. However, the infrastructure necessary to accomplish these goals is not easily funded through direct research dollars. The benefits of central support, although deemed essential by network participants and funders, are difficult to quantify, and assigning costs to individual projects in an environment of uncertain and fluctuating funding results in under resourced central support and an inability to maintain the skilled and experienced persons necessary for network engagement. Most primary care PBRNs have struggled with infrastructure funding.8

Recently, the scientific community has realized the importance of pragmatic research and the translational benefits of implementation science.9–13 Pragmatic clinical trials conducted in real-world settings as well as the use of real-world clinical data for comparative outcomes research have been proposed as methods to provide evidence to overcome this translational gap.14–16 Over the past decade the AHRQ, the National Institutes of Health (NIH) and now the Patient Centered Outcomes Research Institute (PCORI) have provided various infrastructure funding mechanisms to develop clinical data research networks.

This support for infrastructure has led to a growth in the number of clinical data research networks, but the business model for long-term sustainability, which includes the resources needed to adapt to a quickly evolving environment, has been inadequately addressed. In theory, a successful network, or network of networks, would derive the needed capital for infrastructure as a by-product of research grants, i.e., the facilities and administration fees (F&A) would cover most of the ongoing infrastructure. In reality this is rarely the case. Academic centers tend to provide small levels of funding back to investigators, and independent organizations may not have high enough F&A rates to generate substantial infrastructure funding. Therefore other approaches to stable, long-term funding are required.

Some research programs receive infrastructure support from funders, such as NIH Cancer Centers or Clinical Translational Sciences Awards (CTSA). Some electronic PBRNs have been supported by CTSA funds, but with decreasing funding to CTSA sites over the next several years, the long-term stability of this source is far from guaranteed. Researchers typically think in terms of annual budgets, not long-term business models that provide stable income for ongoing programmatic support. This paper describes the path the DARTNet Collaboration has taken in seeking to develop a business model to support this network of networks.

Formation of DARTNet Institute

The Distributed Ambulatory Research in Therapeutics Network (DARTNet) began with the award of two successive AHRQ Task Orders to the University of Colorado School of Medicine, Department of Family Medicine over the three-year period 2007–2010, focused on improving data available for comparative effectiveness research. The initial network relied upon proprietary third party software to extract, transform, and load (ETL) data from multiple different electronic health records into a common data model (CDM) for research and quality improvement. During this three-year period the American Academy of Family Physicians National Research Network (AAFP NRN) assumed day-to-day operations of the original network and began collaborating with several other practice-based research networks in their attempts to use existing electronic data for research, quality, safety, and learning. To advance all members of the group, the decision was made to utilize the DARTNet name as the umbrella name for the burgeoning collaboration, and the original network was renamed. The full original name was dropped and was replaced by simply the “DARTNet Collaborative.”

As the collaborative became more active in developing grants, alternative ETL models and developing data translation software, the DARTNet Advisory Board felt that a legal voice and business entity for the group was essential. This was critical to house the business functions of contracting, providing shared support of software to help advance a learning community, and to develop business activities that could provide ongoing infrastructure support to maintain the collaborative activities. Multiple business structures were explored; among them was the development of a new, not-for-profit entity organized under the Internal Revenue Service Code Section 501(c)(3). While initially more complex than other business structures, the 501(c)(3) structure provided the greatest flexibility and potential for DARTNet’s growth as a federation of research networks regardless of geography. Thus, the DARTNet Institute (DARTNet) was incorporated as an independent, not-for-profit entity in December, 2011, with support and assistance from the American Academy of Family Physicians, and the Advisory Board was transitioned to a Board of Directors.

DARTNet’s partner PBRNs (see Table 1) are constructed around a very different model than are networks of academic institutions or large integrated delivery systems. The data sharing clinical partners range from solo clinicians to large, academic integrated-delivery systems. The local sites may not have technology staff that can access and extract electronic health record (EHR) data nor do they typically have any dedicated research staff. Thus, the networks must provide the support required to maintain productive and reliable research activities within these primarily clinically focused organizations. DARTNet provides a shared resource to support clinical organizations willing to participate in PBRN projects but lacking the necessary technical expertise. DARTNet also links networks with in-house technology capabilities to other partner networks for shared development and joint projects. Most clinical organizations within DARTNet partner networks actively participate if the research activities do not have an adverse impact on work flow and research time is at least modestly reimbursed. This is a very different environment than one where infrastructure support is expected from network members.

Table 1.

DARTNet Partner Networks and Academic Partners

Network Partners

Academic Partners

AppNet – Appalachian Research Network eNQUIRENet – Electronic National Quality Improvement and Research Network CCP – onnecticut Center for Primary Care FREENet – Free Clinic Research & Educational Engagement Network LA Net – Los Angeles Network of Community Health Centers MAFPRN – Minnesota Academy of Family Physicians Research Network MAPPR – Mecklenburg Area Partnership for Primary Care Research NCNC – North Carolina Network Consortium SAFTINet – Scalable Architecture for Federated Therapeutic Inquiries Network STARNet – South Texas Ambulatory Research UNYNet – Upstate New York Practice Based Research Network WPRN – WWAMI region Practice and Research Network

East Tennessee State University Ohio State University University of Alabama, Birmingham University of Buffalo University of California, San Diego University of Colorado University of Massachusetts University of Minnesota University of North Carolina University of Texas Health Sciences Center San Antonio University of Vermont University of Washington

The DARTNet Institute is governed by a self-perpetuating board of directors. The board includes clinician members from partner networks, researchers from partner networks, and individuals from across the country with various skill sets from business, informatics, ethics, and finance backgrounds. The DARTNet governance structure is depicted in Figure 1. DARTNet has clarified its role as a support entity for research, quality improvement, and safety activities with its partner PBRNs serving as the research home for all clinical organizations engaged in network research; i.e., DARTNet will not directly operate a PBRN. This model allows DARTNet to advocate for all partner PBRNs without being conflicted by operating a PBRN itself. Through sharing of projects, expertise, staff, and data through a single coordinating organization, all partners benefit. Furthermore, DARTNet has several lines of business, shown in Figure 2, that generate excess marginal revenue that is used to maintain the organization and add new services, including a soon-to-be-formed Patient Safety Organization (PSO).

Figure 1.

Governance Structure of the DARTNet Institute

Figure 2.

DARTNet Institute Strategic Scope

Notes:

1. Analytic Datasets: Created through the Ql registry activities as well as various specific projects. Further de-identified and data limitations applied and then reused for grant development and secondary analysis by partners as well as outside entities. Price depends on role.
2. Software licensing: ETL, decision support, data transfers to third parties, panel management software all available at discounted prices generally over standard commercial rates for research and quality improvement partners. Pricing varies based on services requested and size of clinical organization.
3. Quality Improvement Registry: Free to DARTNet clinical sites not seeking Meaningful Use-2 or Meaningful Use-3 credit. Small fees attached to data providers seeking Meaningful Use certification based on number of physicians being certified
4. PBRN Research: Support data extraction, participant eligibility, point of care study reminders: inked patient reported outcomes data collection through multiple systems. Prices depend or the services provided and systems utilized

Partner networks each operate independently and agree to participate in research projects on a case by case basis. Networks are principally primary-care focused and share similar values and expectations of their clinical partners.3 Many of the partners have worked together off and on for decades across multiple activities including research projects, network development activities, and various convening activities.17–19 Each network may have projects entirely conceived of and conducted within a single network as well as joint projects only possible through shared activities.

DARTNet serves as a convening body for joint projects, which are generally primed through one of the partner networks. DARTNet may provide data management services for projects that are entirely within a single network as well as serving to standardize data across networks. DARTNet provides data stewardship and re-use support for large projects that require ongoing access to project data by the funding agency. Current requirements to join the collaboration are minimal, including a willingness to participate in future projects, work toward data standardization, and share ideas and best practices. Some subgroups within the collaboration have substantial, highly detailed consortium agreements. Partner networks include over 4 thousand clinicians and over 5 million patients and include sites in over 30 states.

Data Standardization Contributes to Sustainability

DARTNet has gone through two rounds of data standardization, starting with a system that utilized SNOMED, RxNORM, and ICD-9 as the underlying codification schema along with local site as well as several proprietary software systems to perform the ETL functions. In 2011, through the careful work of the Scalable Architecture for Federated Therapeutic Inquiries Network (SAFTINet) and vetting with other DARTNet partners, the entire system converted to the Observational Medical Outcomes Partnership (OMOP) data standard.20,21 A number of data models were considered, including Integrating Biology to the Bedside (i2b2),22 the Health Maintenance Organization Research Network Virtual Data Warehouse,23 open access EHR systems, as well as continuing to use the data standards promoted by Clinical Data Interchange Standards Consortium24 (the original DARTNet approach).

For a number of reasons, the OMOP model was selected. These included: (1) a comprehensive standardized vocabulary, (2) an efficient hybrid data model that combines both traditional column-based and entity-attribute-value schemas, and (3) an active user’s community of investigators working to share development and analytical activities. The SAFTINet project brought resources to allow collaboration with the OMOP25 and members of the SCANNER project26 to modify the OMOP CDM) to better support practice-based comparative effectiveness research.20 The Version 4 data model went through a period of open comment by the OMOP research community and was officially accepted by the partnership in the first half of 2012. The data model forms the basis for the federated data system that DARTNet partners are utilizing.

Selecting a nationally recognized CDM and working towards adoption of the model across all collaboration networks has further cemented the common mission and shared resource development. As improvements in transforming EHR, claims or patient reported data into the OMOP CDM are realized all members of the collaborative more clearly recognize the value this work adds to the system. Sustainability of the collaborative is enhanced through ease of data sharing for research and quality improvement.

To further advance data standardization within DARTNet, the SAFTINet project supported development of the Reusable OMOP and SAFTINet Interface Adaptor (ROSITA), an ETL and data harmonization middleware tool20 designed to lessen partners’ data sharing burden and improve adherence to the networks data harmonization and interoperability standards. ROSITA allows clinical data to be transformed to the OMOP V.4 CDM, allows claims data to be linked to clinical data at the time of this transformation, tracks translational errors, tracks the integrity of the process, and creates both a limited data set version for research and a fully identified version for local use. The ROSITA software allows data partners to create data files from their EHR or clinical data repository (CDR), which can be rapidly converted to the OMOP data standard. The OMOP data dictionary maps27 to all major coding schema in the country, and if the data are already coded in any of these underlying coding systems ROSITA automatically maps the data to the OMOP terminology. This work requires substantial infrastructure support and will be a major focus of DARTNet’s use of revenue streams and shared informatics support in the future.

DARTNet Partner’s Research History

After the initial AHRQ task orders, which studied oral hypoglycemic medications and antidepressants,28,29 the DARTNet partner networks have been tested through a number of other projects. Three pragmatic trials demonstrate both the power of access to practice-level data through the EHR and the research approach of PBRNs.

Electronic National Quality Improvement and Research Network (eNQUIRENet)

eNQUIRENet (the new name for the original DARTNet electronic network) conducted a large-scale demonstration project to assess the effectiveness of implementing the Centers for Disease Control and Prevention (CDC) issued guidelines to improve the management of skin and soft-tissue infections (SSTIs) in primary care with a focus on community-acquired, methicillin-resistant Staphylococcus aureus (CA-MRSA). The DARTNet technology made it possible to assess the prevalence of CA-MRSA using an electronic chart audit and then evaluate SSTI management strategies consistent with CDC guidelines. A total of 3,112 SSTI cases (cellulitis or purulent) were observed during the control period and 1,406 cases were studied during the intervention. The study included collection of information on clinician decision-making near the point of care,30 as well as patient reported outcomes two weeks after the initial diagnosis. In the treatment of SSTIs, this intervention resulted in increased use of antibiotics that are believed to cover MRSA strains. The intervention provided actionable findings regarding the use of these guidelines, and disseminated a replicable and portable intervention with applicability for improved antibiotic selection in other settings.31 The project demonstrated techniques for using EHR data to guide near point-of-care data collection from clinicians: a “card study”32 to supplement the interpretation of the clinical data.

SNOCAP-USA

Using DARTNet technology and with support from AHRQ, the PBRN consortium State Networks of Colorado Ambulatory Practices and Partners (SNOCAP-USA) (which included eNQUIRENet) conducted a large-scale cluster randomized, pragmatic trial to improve antibiotic stewardship in primary care practices. Clinical pathways for eight common adult and pediatric outpatient infections were developed. Each pathway was a one-page decision support algorithm designed to assist providers in determining whether an antibiotic should be prescribed, the optimal antibiotic choice when indicated, and the shortest appropriate duration of therapy. In addition to the clinical pathways, the intervention consisted of patient education materials developed as part of a prior community antibiotic stewardship campaign. The study demonstrated significant improvements in antibiotic use in intervention compared to control practices, including decreases in antibiotics for likely viral infections and decreased use of broad spectrum antibiotics in general.33 The clinical pathways and patient educational material became part of the DARTNet Learning community and have been widely used by other DARTNet associated clinical organizations.

Improving Evidence-Based Primary Care for Chronic for Chronic Kidney Disease (CKD)

A study currently underway, “Improving Evidence-Based Primary Care for Chronic Kidney Disease (CKD)”, funded by the National Institute of Diabetes, Digestive and Kidney Diseases, conducted by the University at Buffalo, The State University of New York, and the AAFP National Research Network, has worked with five DARTNet Institute partners to enroll 44 primary care practices to improve the recognition and care of patients with CKD. The five-year project will evaluate both the impact of different approaches to improving care as well as the impact of improved guideline concordance with CKD treatment guidelines. The project is projected to track over 30,000 patients with CKD as it works with practices to improve care.34

The DARTNet Institute data standardization processes have made it possible for five networks to include practices in the study; DARTNet software provides access to much of the patient data as well as support of the clinical decision support systems at many locations. The DARTNet relationship with study practices provides a potential avenue for obtaining claims data for a planned economic analysis more efficiently than through the practices themselves or the individual research networks. Because this is a comparative effectiveness trial, DARTNet is able to create a true concurrent control third arm of the study using data from similar DARTNet practices that were not part of either intervention arms. This has the strength of being able to test for secular trends (for example, the National Kidney Foundation’s release of new guidelines35) that might have improved care irrespective of the study.

Other Observational Studies

DARTNet Institute partners have also worked together to conduct a number of observational studies including examining the variance in primary care testing approaches for patients with possible cardiac disease, evaluation of treatment approaches for type 2 diabetes mellitus related to underlying patient characteristics including insulin initiation, and the impact of Maintenance of Certification activities on quality of care.36–38 Pediatric studies have linked the American Academy of Pediatrics electronic network to DARTNet partners and utilized DARTNet technology to study the impact of Attention Deficit Hyperactivity Disorder medications on growth, the use of atypical antipsychotics in pediatric patients, and the recognition and treatment of pediatric hyper-tension. DARTNet Institute partners have proven their ability to work together to ask and answer questions of interest to primary care practices, policy makers and funders utilizing existing electronic health data, incorporating patient reported outcomes and clinician input and interactions. The DARTNet Institute has allowed diverse medical societies to share resources in ways that would be difficult without a trusted third party.

Business Structure Advantages

DARTNet’s Structure and Business Features

The data management required to conduct large scale projects as described above require ongoing maintenance, support, innovation, and coordination. The DARTNet Institute fulfills this role as an independently structured entity, yet is accountable to its partner networks and funders. Some of the key characteristics of this model include the following DARTNet features:

• Serving as a legal voice for activities that overlap multiple partners.

• Providing a mechanism to share personnel resources through service contracts between DARTNet and its partners that utilize personnel with technical expertise that are physically employed by DARTNet partners but required across the collaborative.

• Providing ongoing access to unique propriety software as well as common third party software, such as operating systems, databases, and connectivity software at low cost, in addition to central maintenance for the ETL software for all interested partners.

• Providing access to a quality reporting system utilizing business intelligence software, and a data cube fed from the OMOP data warehouse. The quality reporting system is a recognized, national professional-society-quality improvement registry;39 thus, reporting to this system through approved software fulfills Meaningful Use 2 criteria,40 and, as currently specified, Meaningful Use 3 criteria.

• Serving as the prime organization for selected grants that include infrastructure funding and multiple network partners.

This combination of infrastructure and features has resulted in an active organization that is now providing billable services to multiple networks across approximately 80 clinical organizations as well as quality metrics for a growing number of organizations utilizing multiple data-input methods. The work is conducted by members of the partners’ technical teams as well as employees directly hired by DARTNet. The use of shared resources allows DARTNet to work with a diverse and robust group of information technology experts while only paying for what is actually needed, an important consideration in maintaining low costs with robust support.

Sources of Revenue

DARTNet’s business model relies upon four primary sources of revenue at this time.

Software Licensing

As mentioned previously, DARTNet licenses and provides access to a propriety software system that has proven ETL abilities against all major EHRs utilized in the country. The pricing structure through this agreement for “research only” purposes is highly competitive, and has so far been sustained through the flow of direct research dollars from sites that actively participate in both observational and interventional studies. DARTNet also licenses clinical decision support software to clinical organizations that participate in research studies.

Quality Improvement Registry

DARTNet has worked with members of partner PBRNs as well as using the work of several research grants to develop a quality improvement registry that has been endorsed by a national professional society.39 Eligible providers that submit data to the quality improvement registry through an approved EHR or component system receive credit for Meaningful Use 2 and 3. Through an agreement with one EHR vendor DARTNet receives funding to manage quality and safety reporting. DARTNet expects to add other data providers once this initial arrangement matures. The registry is a growing portal for peer-to-peer interaction among providers and practice participants within DARTNet. Performance reports based on clinical metrics of interest to DARTNet clinicians and clinical organizations provide a venue for data-driven discussion and learning.

Research and Data Support

DARTNet provides access to data, or can serve as a support entity for practices for research activities through the Business Associates Agreement (BAA) maintained with multiple clinical sites. For instance, through the BAA DARTNet can mail recruitment letters on behalf of a practice for a research project with little to no work on the part of the practice.

Analytic Data Sets

DARTNet provides access to large primary care data sets for observational studies or for background data during grant preparation. Projects are often conducted as secondary analyses of fully de-identified existing data sets or may involve new data mappings and new data extractions. In either case, DARTNet again utilizes direct grant dollars to generate revenue to support the overall enterprise.

DARTNet’s current revenue stream is highly diversified. Software licensing, Quality Improvement Registry activities, and research support activities each provide approximately 30% of total revenue. Access to analytic data sets provides the other revenue and comes from internal partners, external academic organizations, and research activities funded by industry but conducted by partner organizations. Data have not been provided to commercial entities for their own analyses to date, though this model holds the potential to become an important and significant source of revenue. The ability of commercial entities to directly access fully de-identified versions of DARTNet data sets is currently under consideration by the DARTNet partners and invokes considerable concerns and discussion.

Business Structure Constraints

The not-for-profit corporate model does have some limitations. For instance, access to start-up and operating capital is more difficult for a small not-for-profit organization. Potential for-profit partners, seeing value in DARTNet, could provide initial support through a combination of cash payments and future stock options if DARTNet were a for-profit company. This avenue of support for specialized support or marketing activities is not available to a not-for-profit start-up corporation.

In addition, DARTNet’s mission and not-for-profit status are focused on research and quality- and safety improvements. This mission restricts DARTNet’s ability to license software purely for clinical purposes. Thus, if a clinical organization that is using DARTNet supported software wishes to continue to use a DARTNet product but not participate in any of the data sharing activities, then that clinical organization must find another vendor for the software. In addition, the mission restricts DARTNet from making databases available to for-profit entities for nonresearch purposes. Overall, these limitations are considered relatively minor concerns compared to the benefits for all partners in the not-for-profit model.

Direct Value to Smaller Clinical Environments

DARTNet has been able to help stabilize data availability for a number of partner networks that work in smaller clinical environments where internal information technology capacity is limited. The system allows direct research grant dollars to be used for a portion of the infrastructure costs of maintaining electronic data networks through the provision of direct grant services. If the primary revenue streams can continue to grow then the organization envisions support of other common infrastructure activities across network partners such as patient engagement activities. The ability of DARTNet to provide grants back to network partners is a long-term goal. Currently the DARTNet approach has stabilized access to EHR and Patient-Reported Outcomes data across multiple networks at a cost that appears sustainable for the foreseeable future. Whether the business model will be supported long-term by partner networks and secure sufficient revenue to develop a sustainable infrastructure is not known at this time.

The Future

Moving forward, DARTNet expects to establish a formally negotiated federal facilities and administration fee structure that will improve recovery of administrative costs for grant activities (currently set by default at a low level) while maintaining a competitive rate compared to large academic centers. The organization will continue to provide administrative support for work between partners on items such as semantic interoperability, improved methods of collecting Patient-Reported Outcomes data, delivery of complex clinical decision support to clinical organizations, and ongoing development of quality and safety reports and cross partner learning. This business model provides real revenue to support partner clinical data research networks and a plan to contribute to overhead and access philanthropic support funds. The flexibility of using the not-for-profit business model overcomes the complicated financial arrangements and governance requirements of many professional associations and academic health sciences centers. We have learned from the nature of our work that pragmatic and translational research often leads to technologies and processes that practices wish to keep. Key next steps for sustaining DARTNet should include learning how to capitalize upon the proven value of such efforts, and to translate them into sustainable tools for business over time.