A system dynamics evaluation model: implementation of health information exchange for public health reporting

Abstract

Objective

To evaluate the complex dynamics involved in implementing electronic health information exchange (HIE) for public health reporting at a state health department, and to identify policy implications to inform similar implementations.

Materials and methods

Qualitative data were collected over 8 months from seven experts at New York State Department of Health who implemented web services and protocols for querying, receipt, and validation of electronic data supplied by regional health information organizations. Extensive project documentation was also collected. During group meetings experts described the implementation process and created reference modes and causal diagrams that the evaluation team used to build a preliminary model. System dynamics modeling techniques were applied iteratively to build causal loop diagrams representing the implementation. The diagrams were validated iteratively by individual experts followed by group review online, and through confirmatory review of documents and artifacts.

Results

Three casual loop diagrams captured well-recognized system dynamics: Sliding Goals, Project Rework, and Maturity of Resources. The findings were associated with specific policies that address funding, leadership, ensuring expertise, planning for rework, communication, and timeline management.

Discussion

This evaluation illustrates the value of a qualitative approach to system dynamics modeling. As a tool for strategic thinking on complicated and intense processes, qualitative models can be produced with fewer resources than a full simulation, yet still provide insights that are timely and relevant.

Conclusions

System dynamics techniques clarified endogenous and exogenous factors at play in a highly complex technology implementation, which may inform other states engaged in implementing HIE supported by federal Health Information Technology for Economic and Clinical Health (HITECH) legislation.

Background and significance

In 2009 the Health Information Technology for Economic and Clinical Health Act (HITECH) created state-level grant programs to support health information exchange (HIE) capacity among healthcare providers and hospitals, a critical step toward realizing the full potential of electronic health records (EHRs) to improve the coordination, efficiency, and quality of care.¹ Electronic HIE for public health reporting (HIE for PH) is a powerful strategy for shaping both short and long term policies to promote the health of populations through: rapid and efficient identification, monitoring, investigation, and treatment of communicable and emerging diseases; early identification of food borne outbreaks and environmental exposures; identification of health risk factors; and planning and evaluation of public health services. HIE for PH initiatives are underway in all 50 states, but there have been few evaluation studies that can inform these complex technical implementations.²

Objective

The objective of the evaluation presented here is twofold. First, we evaluated the process of implementing the HIE for PH use case at New York State Department of Health (NYS DOH), using system dynamics modeling techniques. Second, we used the model to identify leverage points that may have policy implications for NYS and for others engaged in similar HIE initiatives, such as states funded through federal HITECH programs.³

In 2004, well ahead of federal legislation, the NYS legislature passed the Health Care Efficiency and Affordability Law for New Yorkers (HEAL-NY). The law created a visionary capital grants program that has invested about $840 million in state, federal, and local matching funds in a multi-year initiative to transform healthcare service delivery through health information technology.⁴ In 2007, as part of HEAL-NY Phase 5, NYS DOH awarded three regional health information organizations (RHIOs) 3-year contracts to implement a technology use case demonstrating HIE for PH. A RHIO facilitates access to and exchange of health information within a geographic area, for the benefit of the community in that area. A significant dependency of the HIE for PH use case was the implementation of web services and operational policies at NYS DOH to enable bi-directional exchange. Services include the capability to electronically query RHIOs for standardized electronic health information to fulfill public health reporting and population health monitoring objectives (ie, patient demographics and identification/re-identification, line list queries for symptoms/diagnoses, and analytic queries for specific populations). These services are executed on the web through a universal public health node (UPHN), an integral feature of the State Health Information Network in New York. Standards and policies are developed through a Statewide Collaboration Process (SCP). This unique private–public partnership is supported by a combination of federal, state, and private funds. It consists of health information entities (RHIOs and others), technology vendors, and NYS DOH.⁵

A high level depiction of HIE in NYS during this study is illustrated in figure 1, in the form of a stock-and-flow structure, a notation used for modeling dynamic systems. The diagram depicts the flow of electronic health information from a local hospital, laboratory, or provider, through aggregation at the RHIO level, and transmission to NYS DOH for public health reporting. Points where loss or delay can occur are shown as the ‘valves’ on the pipes that connect entities. The evaluation examined the dynamics of implementing HIE for PH between the RHIO and NYS DOH. It was conducted as part of a larger evaluation led by the Health Information Technology Evaluation Collaborative (HITEC), a consortium of four academic institutions, led by Weill Cornell Medical College. HITEC is designated by NYS as the evaluation entity for health information technology projects funded under the HEAL-NY initiative.⁶ The Columbia University Medical Center Institutional Review Board approved this study.

Figure 1.

A high-level stock-and-flow diagram showing the flow of electronic health information in the health information exchange for public health reporting use case implementation. The evaluation considered the dynamics of implementing a flow of electronic health information between regional health information organizations (RHIOs) and New York State Department of Health (NYS DOH). UPHN, universal public health node. This figure is only reproduced in colour in the online version.

Materials and methods

Data sources

Primary qualitative data were collected in 2010 and 2011 from seven public health experts identified based on their familiarity with the project by the director of operations at the Office of Health Information Technology Transformation, which coordinates HEAL-NY activities. The experts represented three NYS DOH offices involved in implementing the project and included project managers and technology implementation specialists. All held master's degrees in science or public health. Two were physicians with expertise in surveillance and clinical information systems. The evaluators invited the experts to participate via email. All seven accepted. One expert retired after two data collection sessions and did not participate in validation of the model. Data from the experts were supplemented with secondary analysis of protocols, implementation guides, and other documentation produced by the SCP or by staff at NYS DOH.⁷ In separate, concurrent work, the evaluators conducted traditional case studies of the three RHIOs implementing HIE for PH,⁸ which afforded additional perspective on the efforts at NYS DOH.

Study design

System dynamics modeling techniques were applied within a case study evaluation framework.^9–11 The HIE for PH use case evolved over 4 years, between August 2007 and August 2011. It involved multiple stakeholders and many unanticipated events. In the formative stage of the evaluation, it became clear to the team that a traditional case study would not adequately capture aspects of this use case. For example, participants in the process at the RHIOs and at NYS DOH had differing perspectives, incorporating completely plausible yet totally conflicting interpretations of the same events.¹² This suggested complex dynamics were involved.¹³

System dynamics modeling is a computer aided approach that can be applied to evaluation and policy analysis to augment our understanding of complex social, managerial, economic, or ecological systems.^14–19 These models are less useful when system behavior is explained only by exogenous factors or a single event, or when time produces little change in system relationships.

The method is best applied when one or more of the following are present: non-linear relationships; quantities that accumulate and deplete over time; delays from information or material processes; and/or active feedback processes. Modeling helps explain how compensating interactions reduce, change, or even cancel out desired outcomes. Less rigorous evaluation methods can fail to identify the non-linear impact of decisions over time, or clarify endogenous balancing and reinforcing processes, where decision makers have the most control over outcomes.

Models are built from causal loop diagrams that map the relationships between the relevant variables in the system.²⁰ Each causal loop diagram consists of individual feedback loops, which are the foundational structures of any system.¹⁵ Feedback loops take two forms: a balancing loop represents movement from a current state (the way things are) to an objective state through action (whatever is done to reach the objective), while a reinforcing loop represents an action that results in more of the same action, either producing growth (ie, a ‘virtuous’ cycle) or decline (ie, a ‘vicious’ cycle). The behavior of a dynamic system tends toward equilibrium, where the variables driving feedback loops exhibit less change over time.¹⁶

System dynamics models rely on three sources of information: numerical data (eg, time series data), the written database (eg, operational protocols), and the expert knowledge of individuals engaged with the system. Numerical data are used to produce ‘running’ models with testable quantitative outcomes, which are the gold standard in operations research and engineering.²¹ Such data have the disadvantage of being time consuming and costly to collect. The difficulty and uncertainty involved in quantifying parameters for this multi-stakeholder technology implementation suggested that a running simulation might be of questionable value.^22–24 More readily available was extensive expert knowledge, documents, and other artifacts, which were key to understanding the implementation process.^25–27 The model described here is based on these qualitative data.

Data collection

The evaluation model was iteratively developed by the evaluation team which consisted of two evaluators and two system dynamic modelers. Data were collected from seven public health experts during two face-to-face group sessions, each lasting 4 hour.²⁸ A preliminary model was produced, which the experts individually reviewed. A 2-hour webinar followed, to collect feedback from the experts. All sessions were audio taped and transcribed. After a final round of individual review, emails were used to collect any additional feedback.

In the initial face-to-face session, the evaluation team elicited the history of the process, the specific issues that arose over time, and descriptions of unintended consequences.²⁹ At the next face-to-face session, 1 month later, the experts engaged in two model-building exercises. In the first exercise, the evaluation team facilitated a discussion of the sequence of the project by asking the experts to draw reference modes. A reference mode is used in system dynamics to describe key indicators in the system. It is drawn as a two axis graph where time is always represented on the horizontal axis. Examples of reference modes for this project included data requirements that were discovered over time, the availability of funds over time, and growth in expertise over time.

In the second exercise, the experts drew influence diagrams to illustrate how multiple factors interacted to contribute to the behaviors described in the reference modes. An influence diagram is an intuitive way to identify essential elements in a system (such as decisions, uncertainties, or goals) and how they influence each other.^{30 31} The experts worked in small teams to draw several diagrams of interactions they saw as important in the implementation. For example, one drawing showed how their capability to implement HIE for PH was affected by a shortage of HIE expertise in the national market, by changes in leadership, and by uncertainty about federal information technology standards.

Model development

The evaluation team used the reference modes and influence diagrams to produce a series of causal loop diagrams representing events over time. The reference modes pointed to the dynamic behavior of variables in the model. The inference diagrams pointed to the balancing or reinforcing relationships between variables. Transcripts of the group sessions and review of extensive project documentation helped to verify and further inform the model.

Confidence was built in the model and its policy implications through iterative review with the experts. In the webinar conducted to verify the preliminary causal loop diagrams, each feedback loop (accompanied by a text description) was reviewed and discussed. Before moving on, the experts were polled on the accuracy of changes made. If they did not concur, discussion continued until all concerns were addressed. The verified diagrams and text explanations were circulated to the experts a final time for individual review.

Findings

NYS DOH successfully implemented UPHN protocols for requesting and receiving standardized electronic health information and certified the capability of three RHIOs to process electronic queries and to deliver an accurate payload of electronic health information. Two important adjustments were made to bring the system to equilibrium. NYS DOH extended the HEAL-NY Phase 5 contracts through a 1-year no-cost extension for the RHIOs to implement a connection with the UPHN. Second, NYS DOH allowed HIE for PH to be demonstrated in a ‘test’ environment (using a prepared database of standardized electronic health information) rather than a ‘live’ production environment.

The model is bounded by endogenous causal relationships. Exogenous factors (shown in gray in the figures) are events that were not under the control of the NYS DOH but which had important effects on the dynamics of the HIE for PH use case implementation. They are addressed implicitly via their effect on endogenous feedback processes. Funds to develop the UPHN came from the Centers for Disease Control and Prevention (CDC), not from HEAL-NY. The federal process for specifying HIE standards changed after the 2008 elections, as new legislation (American Recovery and Reinvestment Act (ARRA) and HITECH) redefined the national agenda for achieving HIE. Finally, there were three important changes in leadership: at CDC, at the SCP, and at NYS DOH.

The model

The final evaluation model consisted of three casual loop diagrams that identified important dynamics in the process: Sliding Goals, Project Rework, and Maturity of Resources. Each diagram consists of several feedback loops that are explained. A description of potential leverage points in the diagrams follows. Color figures are available in the online version.

The Sliding Goals Causal Loop Diagram displayed in figure 2 shows the observable goals and resources to meet those goals. It consists of four balancing feedback loops. Loop B1 (blue) represents the gap between the supply of electronic health information and NYS DOH's goal to build capability to receive standardized electronic health information through the UPHN. RHIOs had the goal (ie, contract deliverables) to supply electronic health information in a standardized format. HEAL-NY funds for RHIOs to implement HIE for PH were released in 2007 ahead of the UPHN implementation process at NYS DOH. Funds to develop the UPHN came to NYS from the CDC in 2008. A goal–gap structure is a configuration frequently found in dynamic systems. The behavior of this structure typically exhibits an s-shaped decline over time. The gap between the desired level of health information exchange and the standardized electronic health information collected would be expected to approach zero over time. In this case, CDC funding to NYS DOH was disrupted when a new leader was appointed to CDC following the 2008 elections. Progress on the UPHN stalled, and the gap increased.

Figure 2.

The Sliding Goals Causal Loop Diagram shows the dynamics involved in resolving the gap between the desired level of health information exchange and the electronic health information available. CDC, Centers for Disease Control and Prevention; HEAL 5, Health Care Efficiency and Affordability Law for New Yorkers Phase 5; HIE, health information exchange; NYS DOH, New York State Department of Health; RHIO, regional health information organization; UPHN, universal public health node. This figure is only reproduced in colour in the online version.

Loop B2 (red) represents activities to close the standardized electronic health information gap to achieve implementation goals. Resources allocated to collect standardized health information had associated costs. Time and expertise to resolve the gap depleted resources available for other steps in the implementation. The behavior of this loop drained resources from the system. To maintain equilibrium, funding had to keep pace with activities to close the gap. Disruption in CDC funding diminished the flow of resources. This made it difficult for NYS DOH to close the standardized data gap, for example, by producing a technical interface and protocols for RHIOs to connect to the UPHN. The federal process for specifying HIE standards also stalled after the 2008 elections, which diminished another resource that NYS DOH had tapped for developing protocols to close the gap.

Loop B3 (green) represents pressure to change the UPHN goals, a ‘sliding goal’ structure that is also frequently found in dynamic systems. Projects with multiple goals are prone to delays especially, as in this case, where objectives depended on other sectors of the system.³² This loop describes how delays created pressure to modify goals in the face of changing conditions. Over time as NYS DOH capability to receive electronic health information faltered, the RHIOs’ perceptions about progress on the UPHN flagged. This led to risk management decisions by the RHIOs—to delay work until NYS DOH provided the connection to the UPHN, or to develop their own solutions. The delays placed pressure on NYS DOH to adjust goals until the system found an equilibrium level. As described above, uncertainty about federal standards specification was an external factor that made NYS DOH objectives more difficult to achieve.

The B4 (brown) loop addresses the same issue as above, but with respect to the RHIOs’ HIE for PH use case goals, which were not independent of UPHN goals. As perceptions of progress changed over time, this loop interacted with the ‘goal–gap’ loop (B1) and the ‘activities to close the gap’ loop (B2). These dynamics led to a shift in the desired level of electronic health information exchange until equilibrium was reached. The time to reach goal was modified with a no-cost extension of HEAL-NY Phase 5 contracts and the RHIOs’ deliverable to demonstrate HIE for PH was modified to a test environment. The no-cost extension allowed time for RHIOs to connect to the UPHN and time for NYS DOH to create testing protocols and certify the RHIOs’ capability for HIE for PH . These feedback loops were sensitive to the dynamics of the SCP, which are described in the Maturity of Resources diagram (figure 4).

Figure 4.

The Maturity of Resources Causal Loop Diagram shows the dynamics of developing expertise and collaboration to achieve the HIE for PH use case goals. ARRA, American Recovery and Reinvestment Act; CDC, Centers for Disease Control and Prevention; HIE, health information exchange; HIE for PH, health information exchange for public health reporting; HITECH, Health Information Technology for Economic and Clinical Health Act; NYS DOH, New York State Department of Health; RHIO, regional health information organization; SCP, Statewide Collaboration Process; UPHN, universal public health node. This figure is only reproduced in colour in the online version.

The Project Rework Causal Loop Diagram is displayed in figure 3. It consists of two balancing loops and one reinforcing loop. Technology implementations typically need to revise and rework processes. This project had a degree of known rework, but it also had significant undiscovered work. Feedback loop B1 (blue) represents a balancing process where gaps in standardized electronic health information are discovered and reworked—that is, standardized and validated for public health reporting. A portion of the rework was undiscovered for some time, while staff at NYS DOH developed the expertise to recognize all the levels of validation needed for information to be reusable by multiple public health programs. The balancing behavior of this feedback loop reduced the gap only when rework was discovered. Dormant undiscovered work created delays implementing the UPHN. Over time, had adequate resources been available to discover rework, the accumulation of (non-standardized) electronic health information would have approached zero, as NYS DOH staff specified standards to meet public health reporting objectives.

Figure 3.

The Project Rework Causal Loop Diagram shows the dynamics involved in the undiscovered rework that was needed to resolve the gap in standardized electronic health information. CDC, Centers for Disease Control and Prevention; HEAL 5, Health Care Efficiency and Affordability Law for New Yorkers Phase 5; HIE, health information exchange; NYS DOH, New York State Department of Health. This figure is only reproduced in colour in the online version.

Balancing loop B2 (brown) shows how resources drained from the system to discover rework. As NYS DOH staff expertise increased, accumulated rework was discovered. But rework competed for resources with other tasks, such as developing a technical interface for RHIOs to connect to the UPHN, and for developing testing and certification protocols. This loop shows a ‘tradeoff’ dynamic between preparing the infrastructure to receive standardized electronic health information and assuring that incoming data would meet public health reporting goals. With disrupted CDC funding and insufficient expertise, these processes stalled. External, or exogenous, influences included the fact that HEAL-NY had no funds allocated for UPHN implementation at NYS DOH. The SCP boosted the capability to discover rework by tapping the in-kind expertise of the RHIOs and their technology vendors, although this ‘in-kind’ resource had obvious limits to growth.

Reinforcing loop R1 (green) shows how availability of resources influenced progress. When the R1 loop behaved as a ‘virtuous cycle,’ resources were sustained. Progress towards closing the standardized electronic health information gap freed up resources to receive standardized electronic health information. With fewer rework issues, fewer resources should have been allocated to identifying gaps. Over time, there should have been more resources for NYS DOH to meet project goals. Instead this loop behaved as a ‘vicious cycle’ when resources declined. The behavior of the balancing loops (B1 and B2) diminished the resources to discover rework, and fewer resources remained for technical development. The potential supply of standardized electronic health information leveled off when the staff expertise was not sufficient to discover gaps and rework. Exogenous factors contributed to a vicious cycle: funding disruption, and limits to in-kind work by staff from NYS DOH programs (who helped with data validation) and at the SCP.

The Maturity of Resources Causal Loop Diagram is displayed in figure 4. It consists of four reinforcing loops. Feedback loop R1 (blue) shows how capacity, expertise, and resources matured to achieve deliverables. The maturity of resources facilitated development at NYS DOH to test and certify the three RHIOs. This allowed the RHIOs to achieve their contract deliverables, and increased their willingness to collaborate. However, the R1 loop had a ‘limit to growth’ behavior constrained by the availability of HIE experts in the labor market around 2008–2010. This influenced how the final deliverable was achieved (see the variable: willingness to reduce scope of HEAL 5 objectives in figure 2).

Feedback loop R2 (red) involves the SCP. This feedback loop has virtuous and vicious cycle implications. Early on, strong collaboration improved the technical specification process for HIE for PH and reduced uncertainty from the perspective of the RHIOs and their technology vendors. When SCP leadership changed, this diminished the specification process and uncertainty increased. Deliverables were not being achieved and willingness to collaborate stalled. In the early stages of the project, the maturity level of resources increased with collaboration. Later declining collaboration and limits in the maturity level of resources fed off each other for a rapid decline. Collaboration behaved like an engine that needed fuel (resources and participation) to achieve project goals.

External factors contributed to uncertainty. HITECH legislation led to reorganization of the federally managed process to identify HIE standards. Simultaneously, there was loss of leadership at NYS DOH, the leader of the SCP moved to NYS DOH, and the SCP activities shifted to NYS DOH. The communication previously fostered by the SCP lapsed.

Feedback loop R3 (green) assists the ‘capacity, expertise, resource’ loop (R1) with a surge from champions. The 2008 elections and HITECH legislation, although disruptive in the short term, eventually increased support for HIE and boosted the maturity of resources for building the UPHN. These changes were beyond the control of NYS DOH, which made the R3 ‘champions’ loop subject to oscillations. When champions increased, more deliverables were reached; a loss of champions had the opposite impact, which happened when leadership changed at CDC and funding was delayed.

Feedback loop R4 (brown) shows the influence of the time delays for NYS DOH to build out the technical interface that allowed RHIOs to connect to the UPHN. The specifications became available just 1 month before the HEAL-NY Phase 5 contracts were scheduled to end. The delay created uncertainty from the perspective of grantees and their vendors, delayed deliverables, lowered willingness to collaborate in the future, and had a negative impact on the maturity level of resources. The technical specifications feedback loop dampened other loops in the system and created problems for collaborative efforts. Finally, communication suffered during this period and grantees were unaware of progress at NYS DOH. The adjustments made to project goals helped to bring the system to equilibrium, by reducing the scope of data exchange, and increasing the time available to build out the UPHN and for RHIOs to meet their HEAL-NY contract obligations.

Points of leverage in the causal loop diagrams

The evaluation model clarified endogenous and exogenous factors at play in a highly complex technology implementation. The three causal loop diagrams describe dynamic phenomena which conform to well-recognized system dynamics.^{33 34} The presence of well-recognized dynamics suggests that by analyzing leverage points in the model we may infer policy implications that have potential for general application in similar implementation processes.³⁵

In the Sliding Goals structure (figure 2), there were points of weakness in contingency planning and risk mitigation that threatened system resilience.³⁶ Two dependent projects (the NYS DOH part of the project and the RHIO part of the project) had independent sources of funding. Timelines were based on contracts between NYS and the three RHIOs. State contract requirements do not easily accommodate the flexibility usually needed in complex implementations.

In the Project Rework structure (figure 3), a budget and specific planning for rework could have prevented delays in the system.^{37 38} Resources earmarked for identifying gaps in standardized electronic health information at each stage of the project would have reduced dormant data issues and moved the project forward. Further, the leadership change at NYS DOH left uncertain policies and procedures for completing work. At times implementation tasks became an add-on for staff with other assignments and were not always a priority.

The Maturity of Resources (figure 4) was critical in this pioneering project. Initial assessment of NYS DOH capacity for HIE may not have been adequate, amplifying the delay time created by the standardized electronic health information gap. Further, the expertise required was not clear at the outset. The collaborative process in NYS pooled the expertise of many stakeholders and helped to moderate the effects of change at the state and federal levels. The SCP helped to sustain the commitment of all stakeholders, although it was stressed by dynamics outside the control of NYS DOH. Supplemental communication could have moderated uncertainty and its dampening effect on collaboration.^{39 40}

Discussion

Policy implications

States that are implementing HIE may benefit from understanding the dynamics involved in this project, regardless of the placement or timing of events in NYS. The model illustrates how exogenous shocks common to any HIE implementation (eg, leadership change, funding disruptions, political environment) interact with endogenous processes. Analyzes of leverage points in this model suggest several issues that may have generalizable implications for similar HIE projects.

Expertise to implement HIE should not be assumed

Electronic health information is not necessarily formatted to meet public health reporting objectives. A supply of electronic health information is just one step in the process. Transmission of aggregated electronic health information and utilization of that data (by a state health department, local health departments, or other state agencies) requires subsequent levels of standardization. Gaps in expertise emerged among all parties (RHIOs, vendors, and NYS DOH). There was a lack of common understanding regarding the technical requirements necessary for data standardization, and the scope of validation required for data reuse by public health programs. This was not only a problem of short supply, but also of understanding the type of expertise needed. While industry-wide expertise to implement HIE has improved in the last 4 years, it is not yet clear such expertise resides in health departments, which during the same period have experienced budget cuts, layoffs, and hiring freezes.⁴¹

Rework is a hallmark of HIE

Projects that involve exchange of health information standardized to be reusable in many settings, will contain significant undiscovered work, often stemming from assumptions made during the design phase.⁴² Rework can become a significant problem if there are no resources earmarked for this purpose. While some rework can be anticipated a priori, additional undiscovered work will emerge in the process of reaching agreement on HIE requirements. Planners must account for both the staff expertise and the budget to address rework.

Contingency plans to accomplish project goals

The HIE for PH use case was dependent on an external source of funds. HEAL-NY legislation resulted in an implicit mandate to implement an infrastructure at NYS DOH, although no funds were allocated to DOH for this purpose. This put the UPHN implementation on a different schedule than use case implementation by the RHIOs. Delays undermined confidence and trust. When an HIE project involves governmental contributors, at the state or the local level, resources need to be allocated. It is unrealistic to expect that government agency staff will have time for ‘in-kind’ development of HIE protocols and/or data validation. Contingency plans for all stages of statewide HIE projects are needed to ensure steady progress on multi-stakeholder projects that are interdependent. NYS now funds ongoing development through two parallel and coordinated tracks. One, focused on modernizing DOH system capacity for reporting and bi-directional exchange, is funded through federal grants and by funds earmarked from internal program budgets (immunization, newborn screening, cancer registry, etc). The other, focused on deployment of external web service interfaces, continues to be funded through HEAL-NY capital grants to RHIOs.

Leadership, consistent champions and communication

Mid-point in the implementation, leadership changed at both SCP and NYS DOH. Leadership change disrupted the public/private collaboration. Communication lapsed. RHIOs were uninformed about technical work being done at NYS DOH. There was uncertainty among all stakeholders regarding contractual obligations. A project of this magnitude needs consistent champions to propel it forward, to put policies and procedures in place, to negotiate consistent funding, adequate staff and resources, and a reasonable timeframe. Attention to communication could have addressed some of the uncertainty that undermined collaboration.

Managing timelines in publicly funded HIE implementations

Time delays are expected in large scale technology implementations.⁴³ When dealing with known and/or undiscovered rework, the scope of work originally planned may be changed or deliverables may be delayed. In this case, rework took resources away from technical UPHN development at NYS DOH. Flexibility needs to be built into the project along with adequate resources and expertise. These needs may be difficult to reconcile within government funded contracts that require firm deliverables often based on legislative mandates. Champions need to negotiate approaches for flexibility into otherwise strict contracts when large scale IT implementations are involved.

Limitations

The model was specified and validated through iterative development with involved experts, supported by review of extensive project documentation. Typically the next stage in the development of a system dynamics model is specification of algorithms using time series or other quantifiable data to animate and test the accuracy of the model in relation to real events. Such quantitative data, even were it possible to collect them, would place a prohibitive burden on an evaluation such as this. Indeed, the HIE landscape is progressing so quickly that quantification for any process could be irrelevant before model completion.

Conclusion

The healthcare system is on the brink of wide-scale implementation of HIE, supported by significant legislation that has created incentives aimed at broad health IT adoption.^{1 44 45} The use case evaluated here is an early example of HIE implementation at the level of a state health department. The evaluation found well-recognized system dynamics that shed light on what are likely to be common problems associated with HIE projects. The discussion of policy implications may help government leaders and staff plan and implement similar complex projects in the future.

Collaboration between public and private sector organizations, be they RHIOs or other qualified entities, has an important role to play in shaping policy to assure the benefits to population health that EHRs promise. The work of the SCP was a critical factor in the hard won successes realized by NYS DOH in this implementation. The innovative work done in NYS to implement HIE for PH continues to evolve and has already influenced national standards.⁴⁶

This evaluation illustrates the value of a qualitative approach to system dynamics modeling.²¹ Such models are valuable tools for strategic thinking on complicated and intense processes. They can be produced with fewer resources than a full simulation, yet still provide benefits and insights to policymakers that are timely and relevant. Through analysis of leverage points in the system, we have identified general areas where knowledge and resources can be applied to mitigate untoward outcomes. System dynamics modeling provided a comprehensive evaluation of a project with high significance for public health systems, which the authors believe could not have been achieved with other methods.