Sustaining the Effective Use of Health Care Data: A Message from the Editors

Abstract

Introduction:

Over the past decade, several initiatives have funded large projects to develop clinical research data infrastructures totaling several hundred million dollars. While most of this funding has ended or is expected to end soon, the projects themselves must struggle to continue operations beyond the initial funding. Examples of sustained research-data infrastructures are lacking, and recommended approaches to improve sustainability of developing infrastructures are even rarer. Early on, the Electronic Data Methods (EDM) Forum—and the Agency for Healthcare Research and Quality (AHRQ) as its sponsor—recognized the need to study strategies for sustainability.

Themes:

Three prominent themes relating to sustainability arise among the articles in this special issue: the importance of project maturity, commercialization activities, and stakeholder support. Maturity was relevant to all the papers since a project’s maturity directly influences the opportunities that are available, while commercialization and stakeholder support emerged from comparisons among subsets of articles.

Next Steps:

The papers in this issue create a useful initial set of case studies to help in understanding sustainability issues for data infrastructures needed for research and QI. Each paper includes important lessons learned from the authors’ experience with the different projects that should resonate with the broader fields of clinical research and clinical research informatics. There is an ongoing need for greater understanding of sustainability beyond what this issue provides. As more case studies of sustainability are accumulated, it is expected even more important themes will emerge from qualitative reviews that can eventually be demonstrated quantitatively.

Introduction

Over the past decade, several initiatives have funded large projects to develop clinical research data infrastructures totaling several hundred million dollars.1-6 Other substantial investments have occurred with health information technology (HIT) and clinical data exchange projects, with characteristics similar to infrastructure for clinical research data.7 Initial funding was critical for each project to build collaborations among institutions contributing data, create agreements and methods to actually accumulate the data, and develop and implement software to manage and use the data. While most of this funding has ended or is expected to end soon, the projects themselves must struggle to continue operations beyond the initial funding.

These recent time-limited investments have emphasized the importance of considering sustainability early in the process of developing data infrastructure, not only to continue to support operations but also to create an appropriate return on the public funding investments that have been made. In some cases, sustainability plans were not implemented successfully, and those projects failed to survive beyond the initial investment.8 In other instances, some additional investment was awarded to ensure progress according to a defined sustainability plan. Projects lacking sustainability awareness were less likely to be selected for additional funding from other sources, as subsequent awards required sustainability plans.9

Taken together, these examples highlight the need to better understand approaches to achieve sustainability for large clinical data projects. Examples of sustained research-data infrastructures are lacking, and recommended approaches to improve sustainability of developing infrastructures are even rarer. One reason for this gap in knowledge is that, early on, projects often focus substantial effort on navigating data governance and use among changing regulations and incentives during the infrastructure development. As a result, considerations of long-term sustainability were comparatively less urgent.10 Toward the end of projects, teams may have little time and few resources to discover and develop effective sustainability strategies. Consequently, successful case studies and lessons learned in order to achieve sustainability are often not published or shared—but are desperately needed.

Early on, the Electronic Data Methods (EDM) Forum—and the Agency for Healthcare Research and Quality (AHRQ) as its sponsor—recognized the need to “look under the hood” of sustainability. The first publication by the EDM Forum in a special issue of Medical Care discussed the need to build sustainable, electronic clinical-data systems.6 The first issue of eGEMs included a framework for sustainability, and a January 2013 meeting led by the EDM Forum and AHRQ suggested the value of such case studies to existing data infrastructure projects.11

In response to recommendations raised at the January 2013 meeting, in August 2013 the EDM Forum convened a meeting on sustainability that included stakeholders building various research data infrastructures and other clinical data resources. In this meeting participants discussed a broad range of sustainability activities, but most significantly identified the importance of collecting case studies on sustainability. Participants were interested in generating examples of successful as well as less successful models of sustainability, and in efforts to identify conditions under which specific models are most likely to be successful. In late 2013 the EDM Forum announced a call for papers for a special issue focusing on sustainability to build on these discussions. Case examples, research studies, and position papers on ways to sustain electronic-research data infrastructures were solicited.

The results of the solicitation for the special issue have been impressive. Submissions covered a broad spectrum of data infrastructures at various levels of sustainability, from successful to struggling. The special issue includes projects with decades of development, and other projects just recently started with American Recovery and Reinvestment Act of 2009 (ARRA) funding. We received submissions describing sustainability strategies of distributable tools to support a research data infrastructure within an institution, and papers focused on health data exchange across multiple institutions. Descriptions included purely research-driven (and funded) models and business development approaches. In sum, we received a wide range of submissions documenting successes and failures, ongoing and new infrastructures, and papers that illustrate a variety of funding models for sustaining data networks for research and quality improvement (QI). This was primarily due to two things: the increasing recognition of the importance of understanding sustainability; and planned and successful targeted recruitment of articles from specific contributors who represented the breadth we wanted.

Because of this range of experience in examples and the need for understanding a range of approaches, in introducing this issue it is important to include a commentary summarizing the overall lessons learned in the examples. Here we introduce and summarize the papers, describing results of a compare-contrast exercise with the different articles to identify core themes that emerged. In this way, the commentary is a qualitative meta-synthesis of the articles, following the methods described by Walsh and Downe.12 The goal is not just to introduce the articles, but also to interpret the overall process of creating this issue.

Description of Papers in This Issue and Sustainability Themes

Three prominent themes relating to sustainability arise among the articles: the importance of project maturity, commercialization activities, and stakeholder support. Maturity was relevant to all the papers since a project’s maturity directly influences the opportunities that are available, while commercialization and stakeholder support emerged from comparisons among subsets of articles. We describe each article in the context of maturity, then describe the other themes with the relevant subsets of papers.

Project Maturity

Differences in the lengths of time that projects have existed influenced the stages of sustainability discussed. Three of the papers describe sustained projects that started primarily as research activities with federal research dollars, all of which existed in some form well before ARRA funding. Steiner et al. describe the 20-year history of the HMO Research Network (HMORN), a data collaboration among clinical research centers in geographically spread, integrated health care delivery networks. The HMORN has been sustained in large part through funded projects that have leveraged the efficiency, unique capabilities, and prior track record of the network. At the same time, governance of the network has had to remain nimble to steer the network toward areas of value. Pace et al. explain the history of the Distributed Ambulatory Research in Therapeutics Network (DARTNet) institute, which began in 2007 as a data network centered at the University of Colorado School of Medicine, but has since expanded to include multiple data networks across the country. Like HMORN, DARTNet is a large research data network supporting research in multiple clinical areas, and is supported by research grant funding. Both networks have had to invest heavily in defining data models to make data sharable across the networks.

Unlike HMORN, DARTNet has used software licensing of its data extract-transform-load (ETL) capabilities as a substantial source of revenue. The Informatics for Integrating Biology and the Bedside (i2b2) project began in 2004, though it was derived from the existing Research Patient Data Repository (RPDR) at Partners Healthcare that was created in 1999. Like HMORN and DART-Net, i2b2 has been funded largely by federal grant dollars, but it continues to share functionality and development with the internally funded RPDR. The expansion of i2b2 has been broad—as with HMORN and DARTNet—but it has disseminated software rather than data.

Other projects were developed and implemented more recently, but have reached a stage where their sustainability is more dependent on supporting participation and maintenance than expansion and development. Dutton describes the creation of the National Anesthesia Clinical Outcomes Registry (NACOR), which began in 2009. Its development depended on sponsorship from a national medical society, which established and funded the separate quality institute to develop the registry. Even though most of the sustained funding for the registry comes from the medical society, the development and continued value of NACOR is dependent on broad participation from practice groups across the country, in order to collect data from enough sites that comparisons with the data are representative. In order to reduce participation costs, NACOR discounted participation fees, defined minimum data requirements, accepted a broad set of available data if submitted, and created programs to extract data from the most common information system vendors. At the same time, NACOR provided value back to contributing practices in the form of access to their data compared with national benchmarks to measure performance.

Reams et al. describe the Kentucky Cancer Registry (KCR), which was initially formed in 1986, then expanded to be a standardized data exchange with funding from a federal research program in 2011. Like NACOR, KCR was dependent on obtaining a critical mass of participation from stakeholders, but focused on local stakeholders in Kentucky. Because of the more limited geographic scope, KCR was more directly influenced by local regulations—mandatory reporting to KCR was required by state law beginning in 1991. KCR was funded externally in its initial creation, but expansion efforts were dependent on leveraging existing social networks across participating organizations. Initially a small group of connected collaborators were able to find value in the KCR expansion, which reinforced their collaborative intents. The alignment of the collaborators on goals of improving service became critical in order for the project to overcome technical challenges.

A third set of papers in this issue describe projects and ideas that have not yet reached a sustained point, yet are reflective of important lessons in pursuing a sustainability strategy. Payne describes the Translational Research Informatics and Data Management (TRIAD) project, which is pursuing sustainability through licensing and commercialization. TRIAD was derived from an architecture funded as part of the caBIG initiative. Additional funding evolved the architecture into a scalable and deployable software solution that could be applied to different domains and in different institutions. With time the technology has been adopted by multiple stakeholder groups and organizations, which led to a focus on maintenance and enhancements rather than development. This success in dissemination was accompanied by increased difficulty obtaining federal funding to sustain TRIAD, since such funding is generally more focused on supporting innovation. The project leaders thus pursued commercialization to support the software platform as it evolved out of the research and development stages. Even though such evolution is appropriate, the process of commercialization is generally unfamiliar to researchers and developers who create research-based resources, leading to difficulties in navigating the commercialization process. Payne focuses on lessons learned from navigating this process.

Van Eaton et al. describe the Comparative Effectiveness Research and Translation Network (CERTAIN) project, which began in 2010 with ARRA funding. CERTAIN expanded the capabilities of the existing Surgical Care and Outcomes Assessment Program (SCOAP) registry, which had nearly complete participation by hospitals in Washington State. CERTAIN was intended to automate data extraction and usability, thus reducing the volume of manual chart abstraction needed for QI initiatives. The project collaborated with a software vendor to deliver efficiently the data automation platform, with the understanding that participating hospitals would later take over software license and maintenance costs. The project is a good example of how unanticipated technical barriers and external factors can limit project sustainability even with a defined sustainability strategy. Competing priorities for hospital IT teams (e.g., Meaningful Use (MU)) reduced the priority of the CERTAIN participation. With a lower priority, organizations limited their exposure in the CERTAIN project, which sought to reduce the data participation to a subset of patients. This decision in turn complicated the data automation implementation, and implementation efforts became much higher than forecast, undermining the sustainability strategy.

In the last paper, Wilcox et al. document important considerations for sustainability based on observations of multiple projects rather than reporting sustainability of a specific project. This makes it difficult to compare it directly with the other articles on project maturity. The paper was spurred directly by the early AHRQ and EDM Forum activities to understand sustainability, and the authors were primarily convened by these activities. Many illustrative examples are taken from the WICER project, an ARRA-funded research-data infrastructure in New York City, but most of the authors were not participants in that project specifically. The sustainability factors and lessons learned were derived from broad experience, and focused heavily on stakeholders and how they receive sufficient value from a resource in order to effectively sustain it. In many respects the Wilcox et al. paper is a microcosm of the overall issue, with a qualitative approach to developing the factors and lessons much like this meta-synthesis. The sustainability factors are consistent with the projects described in the other articles, but the overall focus and described lessons learned are markedly different from the themes that emerged.

Commercialization Activities

Commercialization strategies emerged as a theme in six of the articles. HMORN and DARTNet are both sustained research-data networks that include institutions across the United States; however, the two networks took markedly different approaches with respect to commercialization. In the 2000s, following the initial development of the HMORN, the network was largely supported by federal funding for multi-institutional research projects that leveraged the network. DARTNet, on the other hand, completed initial development nearly a decade later when federal funding for research projects was no longer expanding. As a consequence, the need to diversify sources of support, including potential fees from software licensing, was more urgent for DARTNet. Limited federal support for research was also part of the difficulty faced by TRIAD, leading to more substantial commercialization efforts. From these three projects a conclusion emerges that grant funding is pursued to the extent it is available and that commercialization as a strategy has been influenced most by the scarcity of grant funding. However, readers should be cautioned that the success of the commercialization efforts described may be due to the degree to which these teams are familiar with the process of commercializing systems and software. Additional case studies of successful efforts to translate intellectual property from research to commercial domains are needed in order to increase the likelihood researchers will view commercialization as an early sustainability strategy rather than provisional approach.

Wilcox et al. identify for-profit companies as stakeholders that may find value in the infrastructure. Effective technology transfer education, incentives, or brokering of partnerships between academic and commercial institutions may also increase the consideration of commercial approaches. On the other hand, risks and benefits of commercialization must be weighed carefully. Murphy et al. considered commercializing i2b2 but determined that maintaining the developmental direction and original governance structure was more important to i2b2’s long-term goal and mission than was a potentially lucrative partnership. Similarly the CERTAIN project partnered with a commercial vendor in the sustainability strategy; yet even with no issues in software functionality, sustainability of that commercial partnership could not be guaranteed.

Stakeholder Support

While all of the articles were dependent on stakeholders, comparisons among three papers describing similar resources underscored how this dependence can vary. NACOR, KCR, and CERTAIN are all data registries among multiple organizations, though their scopes and goals differ. NACOR is a national registry designed to support QI through benchmark reporting for its participants. CERTAIN was also designed to support QI through benchmarking, but its scope was for a specific geographic area. KCR was also defined for a geographic area, but its intent was more for disease surveillance than benchmark reporting.

Both KCR and CERTAIN were based on previous successful affiliations among the participants and had leadership support at the participating sites. Both faced significant technical and governance obstacles, but with opposite resolutions. It is difficult to determine if the difference in outcomes was due to a difference in actual commitment of the organizations, or whether the competing priority of MU was itself too disruptive to overcome for a project. Another likely important difference is that KCR was able to effectively leverage the support of the Regional Extension Center, while CERTAIN did not have such support.

Given that research has shown that MU requirements have been disruptive to many organizations, in the end, the difference between the two projects may have been the perception regarding whether or not the project was aligned with MU. NACOR completed much of its initial recruitment prior to the availability of MU incentives, and therefore was less disrupted by the competing priorities within the IT departments of contributing organizations. CERTAIN, on the other hand, was directly hampered by MU as a competing priority for IT resources. In addition, NACOR required only a representative subset of the stakeholders’ participation in order to develop a useful critical mass to successfully achieve the project’s objectives. KCR required mandatory reporting, and therefore achieved participation of all independent critical access and rural hospitals in the state. By comparison, CERTAIN had the support of nearly 80 percent of hospitals approached for participation, but only about 25 percent of hospitals were able to complete all process steps of the project. Interestingly, the participation rate of CERTAIN among targeted hospitals was higher than that for NACOR, yet CERTAIN stakeholder participation was viewed as too low while NACOR participation was sufficient for sustainability. Clearly the dependence on stakeholder support can vary according to the resource goals.

Next Steps

The papers in this issue create a useful initial set of case studies to help in understanding sustainability issues for data infrastructures needed for research and QI. Each paper includes important lessons learned from the authors’ experience with the different projects that should resonate with the broader fields of clinical research and clinical research informatics. To our knowledge this is the first collection of case examples and elucidated lessons about sustaining projects to support the use of clinical data for research and QI. While others may have introduced or discussed sustainability as an important issue, the collection of papers together gives even deeper insight into sustainability—revealing themes and issues not otherwise identified by the individual projects. For example, questions about how to approach commercialization are newer and surface indirectly in the papers. Yet, across the whole this emerged as a prominent theme in this collection. Wilcox et al. describe lessons learned from a group of projects, but the focus of the lessons on stakeholders led to different themes than what emerged here. For example, their lessons learned give possible approaches to engaging stakeholders, while the themes of this issue are broader than just that of involving stakeholders.

There is an ongoing need for greater understanding of sustainability beyond what this issue provides. As more case studies of sustainability are accumulated, we expect even more important themes will emerge from qualitative reviews that can eventually be demonstrated quantitatively. eGEMs remains interested in publishing to expand our knowledge of sustainability—including other case studies, comparative case studies, frameworks, and policy reviews to develop a broader understanding of important sustainability practices. We welcome future sustainability-related submissions to eGEMs’ general issue. Please see the eGEMs Instructions for Authors for more information.

We encourage you to read the papers in this collection—and to think about other relevant examples that could add to the general understanding, and to identify additional themes across the papers that resonate based on your own perspectives and work. Further case examples from diverse settings are important to add to the literature and continued conversations if we are to understand how to design sustainable infrastructure. And for those who need to develop their own sustainability strategies to support research and QI data, we recommend applying the lessons learned from these articles in your strategy. Gaps in our understanding of how to consistently and successfully sustain infrastructure for electronic data persist, yet this collection is a critical step toward defining appropriate methods to ensure the future of nascent research efforts using electronic health data.

Thank you for your interest in this topic and this issue—and our thanks in advance for the contributions you and your peers will make to develop innovative and useful pathways to sustainability.