Hospital readiness for health information exchange: development of metrics associated with successful collaboration for quality improvement

Abstract

Objective

The development of readiness metrics for organizational participation in health information exchange is critical for monitoring progress toward, and achievement of, successful inter-organizational collaboration. In preparation for the development of a tool to measure readiness for data-sharing, we tested whether organizational capacities known to be related to readiness were associated with successful participation in an American data-sharing collaborative for quality improvement.

Design

Cross-sectional design, using an on-line survey of hospitals in a large, mature data-sharing collaborative organized for benchmarking and improvement in nursing care quality.

Measurements

Factor analysis was used to identify salient constructs, and identified factors were analyzed with respect to “successful” participation. “Success” was defined as the incorporation of comparative performance data into the hospital dashboard.

Results

The most important factor in predicting success included survey items measuring the strength of organizational leadership in fostering a culture of quality improvement (QI Leadership): 1) presence of a supportive hospital executive; 2) the extent to which a hospital values data; 3) the presence of leaders’ vision for how the collaborative advances the hospital’s strategic goals; 4) hospital use of the collaborative data to track quality outcomes; and 5) staff recognition of a strong mandate for collaborative participation (α = 0.84, correlation with Success 0.68 [P < 0.0001]).

Conclusion

The data emphasize the importance of hospital QI Leadership in collaboratives that aim to share data for QI or safety purposes. Such metrics should prove useful in the planning and development of this complex form of inter-organizational collaboration.

Introduction

The promise of health information exchange (HIE) to bring about dramatic improvements in healthcare quality has fueled scores of attempts to share clinical data at regional and national levels [1–3]. Access to information from multiple inpatient and outpatient settings should not only increase the efficiency and effectiveness of doctors’ decisions regarding their patients, but should also boost hospitals’ ability to discover, implement and encourage the best healthcare practices [4]. Equally important, development of an HIE infrastructure is an integral component of national public health efforts to use clinical data to strengthen our emergency preparedness and resilience in the face of pandemics, natural disasters, and bioterrorism [5–7].

Certain provisions of the American Recovery and Reinvestment Act (ARRA) passed in February 2009 granted substantial financial incentives to healthcare providers for using health information technology (IT). These provisions are known as the Health Information Technology for Economic and Clinical Health (HITECH) Act. Under the HITECH Act, providers eligible for these incentives must demonstrate "meaningful use" of electronic health records, which includes structured documentation and sharing of clinical information within and across organizations for patient care, for public health purposes, and for quality of care reporting [8,9].

These incentives are now shaping current efforts by hospitals and physician practices to participate not only in EHR implementation, but also in HIE endeavors, and increasing numbers of HIEs are now appearing [10]. In the U.S., the number of HIEs operational in 2010 was 73, up from 57 in 2009 according to the eHealth Initiative [10], although the data exchanged still appear quite limited in scope [10,11], and many more efforts remain relegated to the planning stages. Progress in building HIEs has been slow, and these attempts commonly stall or fail [1,12–15]. The most significant challenge continues to be the development of a sustainable business model [10,12]. The difficulty in holding diverse stakeholders together, forming a collaborative governance system and gaining stability until achieving membership financing for the new organization, is formidable.

Indeed, HIEs may be developed for several different purposes, each of which makes its own distinct demands on design and implementation. The goal of sharing data for quality improvement (QI) has shaped the foundation for collaborative data-sharing in the United States over the last 10–15 years, and experiences of QI collaboratives have taught us much of what we know about successful HIE endeavors. The more recent campaign to share clinical data for direct patient care through the promotion of HIEs must build on the same foundation and face the same basic technological, human, usability, managerial and political risks as outlined by Sicotte and Pare [19]. In fact, the complexity of clinical HIE increases exponentially with the technical requirements of real-time round-the-clock data-sharing capabilities, as well as the market-related challenges of sharing data among competing organizations.

Although the HIEs may vary widely, all must accomplish specific complex inter- and intra-organizational tasks to ultimately achieve functionality [16–19]. These include: 1) the development and implementation of a data collection system that is common to all participants; 2) the development and implementation of professional and technical infrastructures for sharing the collected information; 3) the development and implementation of toolkits and protocols for integrating this information into administrative and patient care practices; and 4) the development of measures and documentation of improvements in actual patient outcomes.

Previous Work

In an earlier case study of a regional perinatal information system that was designed to link clinical data of mothers and newborns across four local hospitals, we identified four elements that appear necessary for the achievement of successful data-sharing among hospitals: 1) an assessment of the readiness of each hospital to participate; 2) a widespread recognition of a “perceived mandate” for cooperation and contribution by all participants; 3) the creation of a formal governance structure; and 4) the utilization of a third-party information technology (IT) component with fiduciary responsibility to all participants individually, as well as to the governing body [20]. Utilizing Snyder-Halpern’s framework for organizational readiness for IT innovation [21], and analyses of “critical incidents” [22] to uncover the details of organizational capacities and incentives that appeared necessary to achieve an inter-hospital system for sharing clinical data, we then formed hypotheses regarding the organizational incentives and capacities that would be required to achieve the project milestones. Hospital capacities required for HIE development were categorized as leadership, organizational policies, organizational systems, IT resources, legal resources, and the existence of cross-organizational collaborative relationships [20].

Study objective

The purpose of the current study was to determine which of these capacities and incentives appeared associated with successful HIE participation by member hospitals. The rationale guiding this study was to create reliable measurements of these organizational attributes to allow collaborative organizations to "test" their potential participants in advance for their capacity to share data with the group. If collaborative organizations can estimate these participants' readiness to join, they could then guide them in closing the gap between their current operational state and the anticipated operational functioning as part of the collaborative. An understanding of this gap should assist potential participants and funding organizations in determining the likelihood of successful investment in data-sharing efforts, and enable the formation of strategies to achieve a functional, and successful collaborative. Readiness assessment has been shown to be a critical component of both EHR and HIE implementation and development [12,19,21–24], and practical tools to measure readiness, i.e. readiness metrics, should greatly assist in the development of collaborative organizations.

Methods

Setting

Our original survey concepts had been derived from experience based on a network of hospitals (as distinct from outpatient practices) located within one U.S. city. That HIE was being developed for research purposes, with a view to eventually upgrading the database for use in real-time clinical patient care. Here, in an effort to build on the previous case and work with a "simpler" infrastructure that did not require information directly used for patient care, we examined an American hospital-based data-sharing collaborative for QI.

In this collaborative, data from participating hospitals are sent into a central repository via customized Excel files provided by the collaborative, and the data shared are for quality tracking purposes, and do not involve patient identifiers or highly sensitive information. Thus, the threshold for participation in the collaborative was lower in terms of technical and legal resources, staff and infrastructure for maintaining critical real-time availability of data, human subjects institutional review board requirements, privacy and federal regulatory issues (especially as governed by the Health Insurance Portability and Accountability Act or HIPAA). Hypothetically, testing for critical organizational capacities in such a "simple" HIE should provide a foundation for such testing in the more diverse and resource-intensive environments required for sharing data for direct patient care.

The name of the collaborative tested was the Collaborative Alliance for Nursing Outcomes (CALNOC--formerly known as the California Nursing Outcomes Coalition) [25–29], which shares clinical information regarding the quality of nursing care among California hospitals for the purposes of benchmarking and quality improvement. CALNOC has been in existence since 1996, growing steadily with an increase in membership since 2006 prompted by the increasing demand for comparison data by insurance companies and consumer groups as a condition for receiving contracts [30,31]. CALNOC measures are included in the Implementation Guide for the National Quality Forum (NQF) Endorsed Nursing-Sensitive Care Performance Measures, which is made available to hospitals nationwide by The Joint Commission [32]. CALNOC member hospitals meet the Centers for Medicare & Medicaid Services’ (CMS) Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) program’s new requirement (July 2010) that hospitals participate in a registry for nursing-sensitive quality metrics. CALNOC data have been used to examine the 2004 California mandated nurse staffing ratios, and aggregated trends have been published to help nurses use benchmarks and clinical dashboards for expediting improvement in patient outcomes[29].

At the time of the survey, in January 2008, CALNOC had 192 member hospitals, with 79.5% being nonprofit, 89.5% urban, and 88.9% members of a multi-hospital system. Although some CALNOC members were new and had only recently begun reporting, participating hospitals report quarterly regarding nurse staffing and specific nursing outcomes such as the occurrence of pressure ulcers and patient falls. Each hospital has a primary site coordinator who is responsible for data collection and submission to the CALNOC data center. CALNOC’s website provides member hospitals with a variety of benchmarking reports tailored to the specific needs of users, including the hospital’s board of directions, nurse executives, unit managers, and frontline staff.

Study Design

This study used a cross-sectional design with data collected via an electronic survey instrument that was administered to the CALNOC primary site coordinator [33,34]. The items in the survey were directly derived from literature review and the direct experience in the previous case study. Subject areas covered in the survey were: incentives for joining the collaborative, leadership support, hospital policies (including legal and institutional review board), and operational experience and resources (including IT); full details regarding the derivation of these subject areas have been previously published [20]. Nurse coordinators for a similar but distinct perinatal data-sharing project assisted in the piloting and testing of the survey questions for face and construct validity.

Based on the pilot testing, we revised the survey and in January 2008, sent it via an e-mail link from the CALNOC Administrative Director to the primary site coordinators at each member hospital. Two follow-up requests were made over the subsequent month. Respondents were encouraged to participate in a re-test of the same survey given two to four weeks later for a small financial incentive. Respondents were not required to identify themselves unless they agreed to be re-tested. This anonymity was viewed as important to maximize the response rate and to allow candid assessment of the CALNOC organization and its leadership as well as their own hospital’s level of participation and support.

“Success” in using the CALNOC system was determined by the hospital’s agreement with the following question: “CALNOC data are incorporated into the hospital dashboard with other metrics.” Dashboarding is a good indicator of whether the data are being used to achieve organizational goals [27,28]. It is indicative that the data are being collected, aggregated into a reporting format, contextualized with respect to normative measures, and tracked at a high level within the organization.

Each survey item was examined individually for face and content validity, and comment fields attached to each item were explored. Answer choices ranged from 1 to 9 (strongly disagree to strongly agree). Univariate statistics were calculated using non-parametric methods, and individual survey items were tested independently against this operational definition of Success, either as a continuous variable, or categorized into three groups [High (7–9), Medium (4–6), and Low (1–3)]. These independent items were also entered in a stepwise fashion into a multivariate logistic regression model with Success defined categorically with values from 1–6 as “No,” and values from 7–9 as “Yes.”

Principal component and factor analysis were used to analyze the data. Principal component analysis is useful when investigators have obtained measures on a number of observed variables and wish to develop a smaller number of artificial variables (called principal components) that will account for most of the variance in the observed variables [35]. Here, principal components analysis was used to explore the data and identify those constructs that appeared to be represented by the survey, and that had acceptable internal reliability [36]. This yielded a reduced number of variables, which was then submitted to factor analysis. Factor analysis allows identification of the underlying factor structure, assuming that the observed variables are linear combinations of the underlying hypothesized factors [37]. Exploratory factor analysis using squared multiple correlations as prior communality estimates was performed. The principal factor method was used to extract the factors, and this was followed by a promax (oblique) rotation, which allows the factors to be correlated. Appropriate methods were used to interpret the rotated factor pattern and the loading of individual items. A multi-factor/multi-item correlation matrix was constructed.

Univariate analysis of the identified factors (using factor-based scores) was then performed with respect to the operational outcome of interest, Success. Multi-factor/multi-item correlations were explored.

The reliability of the survey instrument was examined by Spearman (non-parametric) correlations of the test and retest of independent items and the factor scores. Where possible, item results were categorized as Yes (score 7–9) and No (score 1–6), and kappa statistics calculated.

All calculations were performed in SAS, v. 9.1 (Cary, NC). Chi-square testing for proportions, and parametric and non-parametric testing for continuous variables were used as appropriate. Statistical significance was defined as P <0.05. The study was approved by the appropriate institutional review boards (IRB) for human subjects.

Results

We received responses from a total of 68 hospitals (35.9%). Although the survey was anonymous, over half of the respondents did identify themselves. A qualitative review of respondent identifiers indicated that the sample included hospitals from all health systems participating in CALNOC, including those with a long history of membership and those new to CALNOC at the time of the survey. The majority of responding hospitals (75.0%) indicated that they submitted data at least quarterly, and the majority of the staff responding to the survey were involved in data preparation and submission, and downloading benchmarking reports for facility use. Sixty-nine percent of the respondents chose the following reasons why their hospital implemented the CALNOC system (not mutually exclusive): for benchmarking purposes (46%); for internal quality improvement purposes (26%); for research purposes (10%); as a hospital system requirement (4%); as a magnet hospital expectation (4%); for educational purposes (4%); or for regulatory purposes (4%). None of these reasons was associated with the operational definition of a successful implementation (data not shown).

The mean value of Success was 7.62 ± 2.18, with a median of 8.00 (range 1.00–9.00). Table 1 describes the distribution of the independent survey items by level of Success (High/Medium/Low). Most of the leadership measures, and the compatible hospital policy measure, were associated with the level of Success. The number of years of hospital experience with CALNOC was not associated with the level of Success (data not shown).

Table 1. Distribution of independent survey items by levels of success in CALNOC system usage.

Survey item scores range from 0 (strongly disagree) to 9 (strongly agree) and are expressed as mean ± standard deviation. High Success is defined as a score of 7–9, Medium Success as 4–6, and Low Success as 1–3. The number of hospitals answering each set of questions varied slightly because of one or more missing responses in the items making up the predictor or response variables.

Survey Item	High Success N = 48	Medium Success N = 7	Low Success N = 6	P value
Leadership
Your participation included a senior champion (executive level) who both supports your participation in the collaborative and is influential within your hospital organization	8.6 ± 0.8	6.1 ± 2.9	6.7 ± 3.4	0.0030
Your senior champion has had prior experience in guiding your hospital in similar strategic data sharing and/or benchmarking projects	8.6 ± 0.9	7.0 ± 1.4	7.0 ± 2.8	0.0050
Your hospital highly values the availability of electronic data, e.g. using dashboarding, routine electronic reports, quality assessments, etc.	8.7 ± 0.7	7.3 ± 2.1	7.3 ± 2.3	0.0049
Your hospital’s leaders have a well-defined vision of how participating in the collaborative will advance the strategic goals of the organization	7.7 ± 1.6	5.1 ± 2.3	4.0 ± 3.4	0.0023
Your hospital’s senior leadership uses collaborative data to track quality outcomes	8.2 ± 1.3	6.6 ± 1.0	3.3 ± 3.3	<0.0001
Your hospital’s senior administrative-level personnel view themselves as personally accountable for you’re your hospital’s participation	7.5 ± 1.9	6.6 ± 1.1	3.3 ± 1.9	0.0017
The hospital staff recognizes that there is a strong mandate to participate in the collaborative	7.0 ± 1.9	5.4 ± 0.8	3.2 ± 2.2	0.0016
Personnel working with the collaborative data view this work as important in helping the hospital achieve its strategic objectives	8.2 ± 1.1	8.1 ± 1.1	6.5 ± 1.2	0.0085
The hospital holds you accountable for your own activities related to the collaborative (e.g., activities such as project leadership, data collection, submission, report generation, or using data for quality improvement)	8.8 ± 0.4	8.0 ± 1.4	8.7 ± 0.5	0.0628
The senior champion (executive level) and primary site coordinator communicate effectively with each other	8.2 ± 1.3	6.2 ± 3.3	5.4 ± 2.9	0.0134
Hospital Policies
The collaborative data submission specifications are compatible with standing hospital policies, e.g. policies regarding privacy, sharing data with other organizations, HIPAA, and data security processes	8.8 ± 0.5	8.1 ± 0.4	8.7 ± 0.5	0.0006
At the time you joined the collaborative, your hospital had prior experience with projects involving data-sharing across organizations, e.g. Joint Commission Core Measures	8.4 ± 1.31	8.1 ± 0.9	7.7 ± 2.8	0.3045
Your hospital has confidence in the collaborative’s systems to ensure data security and confidentiality	8.8 ± 0.5	8.4 ± 0.8	8.0 ± 1.7	0.2675
Approval from the Institutional Review Board for Human	4.1 ± 3.8	4.0 ± 4.1	6.0 ± 3.6	0.5445
Subjects (IRB) was required by your hospital to join the collaborative
Your hospital had prior experience in approving legal agreements for data-sharing projects with other hospitals or organizations	8.6 ± 0.7	8.2 ± 0.8	8.8 ± 0.5	0.3537
Your hospital’s legal counsel was needed to assist during the contracting process with the collaborative	4.7 ± 3.7	1.0 ± 0.0	6.0 ± 4.2	0.1979
If needed, your hospital’s legal counsel was available to assist during the contracting process	7.6 ± 2.7	7.3 ± 2.1	8.5 ± 0.7	0.7079
Hospital Operations
The activities required for participation in the collaborative are compatible with standing hospital work procedures, e.g. availability of procedures and personnel for data collection, data entry, and data reporting	7.6 ± 2.13	6.6 ± 1.7	6.3 ± 2.9	0.0940
Your hospital was already collecting patient data electronically prior to joining the collaborative	6.6 ± 3.2	6.7 ± 2.6	7.5 ± 3.2	0.6363
Your hospital uses operational data from electronic clinical information systems to plan its services and staffing	7.6 ± 2.2	6.9 ± 1.7	6.2 ± 3.4	0.3123
Your hospital’s personnel resources are adequate to ensure the accuracy of your data collection, submission, and report generation	7.6 ± 1.9	6.5 ± 2.1	7.5 ± 2.3	0.3948
Your hospital provides adequate physical and technical resources to assist with data collection, submission, and data use, e.g. hardware, software, and physical location	7.9 ± 1.5	7.2 ± 1.9	6.8 ± 2.3	0.3630
Your hospital provides support, e.g. finances and protected time, to integrate collaborative activities related to data collection, submission, sand reporting into routine work procedures	7.9 ± 1.6	7.3 ± 1.8	5.7 ± 3.3	0.2348
Your hospital personnel working with the collaborative data generally have had prior experience in working with similar data	7.3 ± 2.1	7.3 ± 1.1	7.5 ± 1.8	0.7120
Your hospital’s work practices allow regular communication among your hospital’s team members if needed	8.0 ± 1.5	7.6 ± 1.9	6.8 ± 2.3	0.4129
Incentives
External pressures demanding data encouraged your hospital to join the collaborative	5.9 ± 2.3	6.9 ± 1.4	7.6 ± 1.5	0.2376
The encouragement of nearby peer hospitals and/or the opportunity to benchmark among marketplace peers encouraged your hospital to join the collaborative	6.8 ± 2.3	7.0 ± 1.3	6.4 ± 1.8	0.6944
Before joining the collaborative, your hospital had well- established collaborative relationships with other member hospitals	4.5 ± 2.7	5.6 ± 2.2	4.8 ± 1.8	0.5312
Your hospital is part of a network or system that includes other hospitals participating in the collaborative	6.7 ± 3.4	6.3 ± 3.7	7.0 ± 3.1	0.8432
Your hospital system encouraged or required your hospital to join the collaborative	5.4 ± 3.7	7.9 ± 1.7	5.5 ± 4.1	0.3974
Your participation in the state benchmarking project required your hospital to submit data	5.2 ± 3.8	6.3 ± 3.1	4.8 ± 4.4	0.8241
The usefulness of the collaborative’s benchmarks to your hospital’s quality improvement program encouraged your hospital to join	7.7 ± 2.2	7.3 ± 1.5	7.2 ± 1.9	0.2785

Principal components analysis was used to determine those items that appeared to represent the original constructs proposed by the survey. Several areas, such as legal resources, IRB requirements, and incentives, had substantially fewer respondents, and items associated with these areas were excluded from further analysis.

The first and most important factor was comprised of items that together measured the strength of organizational leadership in fostering a culture of quality improvement (QI). The following five items were included in the QI Leadership factor (α=0.83):

1. Presence of a supportive hospital executive;

2. The extent to which the hospital values data;

3. Presence of hospital leaders’ vision for how CALNOC advances the hospital’s strategic goals;

4. Hospital use of CALNOC data to track quality outcomes; and

5. Staff recognition of a strong mandate to participate in CALNOC.

The following three items were included in the second factor, here termed Hospital Resources: 1) personnel resources are adequate; 2) technical resources are adequate; and 3) finances and time are adequate (α = 0.84).

Factor analysis confirmed the above two factors, which were then examined for independence (Table 2). The item “Compatible Hospital Policies” was also considered independently because of its strong association with Success. Univariate analysis of these final three identified factors (using factor-based scores) was then performed with respect to the operational outcome of interest, Success (Table 3). The correlation between Success (as a continuous variable) and the QI Leadership factor was 0.68 (P < 0.0001), and between Success and the Hospital Resources factor was 0.28 (P = 0.0383). The item Compatible Hospital Policies item yielded a correlation of 0.44 (P = 0.0004).

Table 2.

Multi-factor/Multi-item correlation matrix for the Quality Improvement Leadership Factor and the Hospital Resources Factor

Item	Quality Improvement Leadership Factor	Hospital Resources Factor
Senior champion	0.70	0.34
Value data	0.55	0.35
Vision	0.88	0.42
Track outcomes	0.86	0.37
Mandate	0.54	0.19
Enough personnel	0.35	0.80
Adequate technical resources	0.32	0.89
Hospital support	0.62	0.78

Table 3.

Association of factor-based scores with Success (High/Med/Low). High Success is defined as a score of 7–9, Medium Success as 4–6, and Low Success as 1–3. QI = Quality Improvement

Factor	N	Mean ± standard deviation., median, range	P Value
QI Leadership
High Success	43	8.0 ± 1.0, 8.2, 5.8–9.0	<0.0001
Med Success	7	6.1 ± 0.9, 6.2, 4.4–7.4
Low Success	3	3.6 ± 1.4, 4.4, 2.0–4.4
Hospital Resources
High Success	43	7.7 ± 1.6, 8.0, 2.3–9.0	0.2990
Med Success	6	6.8 ± 1.6, 7.0, 4.7–8.7
Low Success	6	6.9 ± 2.2, 7.2, 3.3–9.0
Policies Compatible
High Success	48	8.8 ± 0.5, 9.0, 7.0–9.0	0.0006
Med Success	7	8.1 ± 0.4, 8.0, 8.0–9.0
Low Success	6	8.7 ± 0.5, 9.0, 8.0–9.0

Twenty respondents were re-tested. Factor-based scores on the original and repeat tests are described in Table 4.

Table 4.

The correlation of factor-based scores from original testing and repeat testing (N=20). Scores are expressed as mean ± standard deviation, median, range

Factor	Original score	Repeat Score	Correlation	P value
Quality	7.2 ± 1.7	7.5 ± 1.55	0.75	0.0009
Improvement	8.0, 2.8–9.0	8.0, 3.8–9.0
Leadership
Hospital	7.7 ± 1.4	7.4 ± 1.6	0.67	0.0035
Resources	8.2, 5.0–9.00	8.0, 3.0–9.0
Policies	8.8 ± 0.4	8.7 ± 0.6	0.43	0.0633
Compatible	9.0, 8.0–9.0	9.0, 7.0–9.0

Discussion

These data confirm the importance of organizational leadership for successful participation in a data-sharing collaborative, and demonstrate that the concept of QI leadership can be measured reliably. The items in our resulting leadership factor appeared to mirror the leadership characteristics of successful hospital quality improvement (QI) and safety “cultures” [38,39]. These cultures encompass values, beliefs and routine practices that enable organizations to accomplish ongoing improvements in quality and safety. Successful QI and safety programs are built on trust and transparency established between staff and leadership [40,41], and rely on the creation of a “learning culture,” in which “leadership’s key role becomes providing an atmosphere that promotes adaptive work and mobilizes people to tackle tough problems” [42]. These cultures also rely heavily on data-sharing for result tracking and benchmarking, both within and across organizations.

The QI Leadership factor tells us several important things about leadership for successful data-sharing among hospitals. First, it tells us who must be involved: namely, a supportive hospital executive. The QI literature states that top management (i.e. the administrators and the executive team) must be responsible for QI program implementation and project oversight. Indeed, several studies of QI initiatives have found that top-down projects have advantages over bottom-up initiatives, as they are generally better supported by organizational resources [43], and have greater legitimacy [44]. Our original case study of the perinatal data exchange, which was not a QI project, also found that its development suffered due to lack of upper-level support [20].

According to Shortell et al [45], organizations must develop their QI capabilities along four key dimensions in order to successfully achieve continuously improving performance: cultural, structural, technical and strategic. Only executive leaders are in a position to manage the development of their organizations along all four dimensions, and leaders’ communication practices are fundamental to this development [46].

Communication was a second theme within the QI Leadership factor, which included the hospital leaders’ communication of a vision regarding the relevance of CALNOC to their own hospital, and their communication of a strong mandate to participate. There is a substantial literature that sees leaders as “managers of meaning.” Leaders’ communication practices affect their followers’ perceptions, and thereby influence organizational behavior [47–49]; the resulting behaviors may reinforce cultures or result in organizational change. Singla et al.’s review of patient safety culture survey instruments found that two core dimensions of these cultures were communication and leadership support for patient safety [50]. When a hospital decides to join a data exchange, the scope of organizational change can be expected to vary depending on the maturity of the collaborative and the readiness of the hospital that wants to join. If the data-sharing effort is nascent, or if the hospital needs to do a lot of work in order to get ready to join (e.g., collecting new indicators, hiring new staff, implementing new IT systems), then an elaborate communication effort may be required. In such complex cases, which may involve many constituencies at multiple levels, a change vision should be implemented via an internal campaign [47], which is a “planned, organized effort to mold corporate images, manage issues, and articulate values”. Such an implementation effort has also been compared to “delivering a new product to market,” requiring that leaders act like marketers, which calls for both interpersonal and organizing skills [51].

The QI Leadership factor also suggests that the use of CALNOC data to track quality outcomes and the extent to which a hospital values data are related. When hospital leaders use CALNOC data to track quality outcomes, it reinforces employees’ perception of the hospital as a place where data are valued. It is one way of signaling the hospital’s continuing commitment to QI, and it also communicates the relevance of CALNOC to the hospital’s QI efforts.

As an independent item, the presence of “Compatible Hospital Policies” did appear to be related to Success, although it will require further exploration to tease out the critical policy issues that arise with such an endeavor. Such issues may be related to conflicts between hospital policies and the collaborative’s requirements that may exist within multiple hospital areas, such as nursing, IT, legal, and human subjects requirements. The perceived importance of responding to the survey may itself be shaped by hospital policy that determines which HIE-related work is given priority.

Apart from the QI Leadership factor and the Compatible Hospital Policies item, no other factors or items demonstrated an important and interpretable association with Success. The Hospital Resources factor resulted from the factor analysis, although there was not sufficient statistical power to support the conclusion that this factor was associated with CALNOC Success. Other capacity items tested, such as those involving legal resources and IRB use, were not answered consistently by the respondents, and were therefore difficult to interpret. It is likely that such capacities are not relevant to this kind of data exchange because it is based on QI, and has never required patient identifiers, unlike the perinatal database in our previous study. Incentives for participation were also difficult to assess because the survey was administered to those who managed the CALNOC data-sharing process, and these staff may have been unaware of the rationale for their hospital’s participation, resulting in an incomplete and potentially inaccurate participant response.

Limitations

Apart from the QI leadership concept, multiple other concepts were hypothesized based on our previous experience and on the literature, but either were not developed sufficiently or were underpowered to produce interpretable results. Our 36% response rate is similar to the 32% median response rate reported by Cycyota and Harrison in a meta-analysis of surveys of executives during 1992–2003 [52], and in keeping with a 2006 review that found 4 of the 6 major information systems journals published articles with average response rates of less than 40% [53]. Multiple authors note that expected response rates are continuing to decrease annually as organizations are increasing limits on employees’ participation in surveys both because of workload and concerns over sensitive information [52,54]. These issues are particularly prevalent in health care organizations today.

Irrespective of the response rate, non-response bias, in which the study sample is not representative of the collaborative as a whole, is always a potential concern. In this study, such bias may have manifested in either the prevalence of the estimates of the survey items themselves, or in the relationship between the estimated factor-based scores and the multiple levels of Success.

With respect to estimates of the specific survey items, our sample is likely to have over-represented the more “successful” hospitals, since the more active participants in CALNOC were more naturally motivated to complete the survey for the same cultural reasons that they were more likely to incorporate the CALNOC metrics in their hospitals’ dashboards. Because the survey was administered anonymously, likely reasons for non-response include: a heavy workload, apathy, and a perception that nothing would change (for better or worse) as a result of their response [55]. Such reasons may have discouraged staff from less “successful” hospitals from participating. Thus, the estimated proportion of hospitals that were successful may be exaggerated.

However, the distinction between the factor-based scores for QI Leadership for the different levels of Success should not have been affected. Our limited number of hospitals with low and medium Success scores still allowed enough power to differentiate between QI Leadership levels in those with high, medium, and low Success scores. If non-responders had both low levels of Success and QI Leadership (a scenario that seems likely), inclusion in the study would have made the differences among the Success groups even more extreme. For non-response bias to have caused us to over-estimate the relationship between leadership scores and Success, non-responders would have had to have a low level of Success and strong QI Leadership scores, or a high level of Success with weak QI Leadership scores, both situations which, although they have not been tested, seem unlikely. Thus, if a non-response bias existed, it would likely have biased our results toward the null hypothesis, rather than toward the positive relationship found between QI Leadership and Success.

Although the concepts of "success" and operability may be different for HIEs sharing data for patient care (e.g. many of the measures used may not be relevant to a hospital dashboard [56]), in the end, the overarching vision behind HIEs for QI and for direct patient care is to seek improvement in both organizational operations, and patient outcomes. At the present time, the horizon for demonstrating improved patient outcomes secondary to both QI and health IT implementation seems distant [57–59]. Future research will be needed to address more technical operational measures of successful data-sharing, such as the amount of data shared, and the accuracy of data shared, as well as to demonstrate the critical link between data-sharing and improved outcomes.

The difficulties inherent in using the hospital as the unit of analysis were apparent in this study. It became clear that certain survey items could not be answered by the designated respondents, particularly the questions regarding organizational incentives for participation in the collaborative. Several types of respondents (e.g., hospital executives, collaborative staff) may be necessary for learning more about these relevant constructs. For a survey of this type, further exploration of a variety of methodologies for achieving maximum response rates for a number of individuals in different positions from each of the participating organizations will likely be required.

Conclusion

Hospitals that are considering embarking on the difficult road of collaborating with the goal of HIE would be well advised to assess their readiness for data-sharing. Our survey has derived two important domains (QI Leadership and Hospital Resources), and points to a third (Compatible Hospital Policies), that were testable in this mature QI collaborative environment. To our knowledge, this paper is among the first to empirically test these domains as potential constituents of a readiness metrics tool.

As we respond to the current mandate to move toward deriving further measures of readiness, we must begin to explore these domains in different types of collaboratives (research, QI, clinical) at varying levels of maturity. In business environments, stages of organizational development have been described as “maturity models” [60]. There are factors that characterize each stage, and tools for developing the organization to move to the next level. In healthcare, we are just beginning to apply such thinking to our HIE collaboratives. The eHealth Initiative has recognized that this readiness journey can be long and involved, and developed a framework with seven stages for assessing and tracking HIE development [10]. Research is needed to flesh out the model of the levels of maturity of HIE collaboratives, develop the characteristic capacities that independent organizations require to reach the next level, and generate toolkits that apply to each level to enable the collaborative to mature.

Performance tracking of these capacity domains would allow both individual organizations and the collaborative to assess their readiness status. The performance tracking process has become highly familiar to hospitals and other healthcare organizations, although it is usually applied to processes and outcomes for patient care [61,62]. Just as sets of metrics are now being developed for quality and safety of patient care in hospitals, “readiness metrics” for participation in data-sharing can also be tracked and used to encourage collaboratives to identify and employ best practices to achieve the desired outcome of successful data exchange. This same strategy is being used by global efforts to develop metrics for community resilience to natural disasters [5]. The benchmarking process (by which individual organizations can view their status with respect to the domains of readiness compared to their peers) should enable these organizations to recognize and consider the gaps that they must close prior to participating in an operational collaborative.

As in benchmarking for clinical care, identification of the organizational gaps that must be closed to become a successful data-sharing partner should then encourage efforts both internally and on the part of the collaborative, to anticipate and resolve problems, to share information and experience, and to develop workarounds to move the organization forward. The relatively new fields of knowledge management (KM) and collaborative technologies (CT) can offer assistance in these efforts.

Knowledge management in healthcare has been defined as “aligning people, processes, data and technologies to optimize information, collaboration, expertise, and experience in order to drive organizational performance and growth” [63]. CT provides tools, including groupware and document management, for KM endeavors [64]. The fragmentation and complexity of the healthcare environment is in dire need of tools that increase the efficiency and effectiveness of information, and under many circumstances, tools that ease the transition between tacit and explicit knowledge [65–67]. Although the tools associated with CT are most often utilized to bring together people within an organization, or within collaborative structures that have a formal management structure, such as hospital networks [68–70], they may indeed play a more important role in groups with highly disassociated members.

Examples of CT that would apply to improving readiness of healthcare organizations to participate in data-sharing collaboratives would be:

• Leadership deficiency: use of learning communities to educate and develop “champions” within the organizational environment [71–73]

• Operational limitations: use of workflow management systems [74] and decision-support [68] to assist in understanding options for adapting to data exchange

• Policy barriers: document libraries [71] including policies that can be re-used or adapted, document management systems [71] for developing and annotating new policies

• Legal challenges: expert locators [71,75], research databases [71] for determining relevant legislation and its interpretation

• Health IT deficiencies: project snapshots [72] to educate participants regarding potential workarounds for technical problems

• Lack of experience with collaborative relationships: use of groupware [71] for sharing experiences and developing project goals through relationships across organizations

In summary, these data support the importance of QI leadership in a mature clinical benchmarking collaborative. Such results should now be explored within younger and contextually-varied data-sharing environments to assess hospital readiness for participation prospectively, and to test whether the strength of QI leadership is associated with their success.