Developing and Testing a Model to Predict Outcomes of Organizational Change

Abstract

Objective

To test the effectiveness of a Bayesian model employing subjective probability estimates for predicting success and failure of health care improvement projects.

Data Sources

Experts' subjective assessment data for model development and independent retrospective data on 221 healthcare improvement projects in the United States, Canada, and the Netherlands collected between 1996 and 2000 for validation.

Methods

A panel of theoretical and practical experts and literature in organizational change were used to identify factors predicting the outcome of improvement efforts. A Bayesian model was developed to estimate probability of successful change using subjective estimates of likelihood ratios and prior odds elicited from the panel of experts. A subsequent retrospective empirical analysis of change efforts in 198 health care organizations was performed to validate the model. Logistic regression and ROC analysis were used to evaluate the model's performance using three alternative definitions of success.

Data Collection

For the model development, experts' subjective assessments were elicited using an integrative group process. For the validation study, a staff person intimately involved in each improvement project responded to a written survey asking questions about model factors and project outcomes.

Results

Logistic regression chi-square statistics and areas under the ROC curve demonstrated a high level of model performance in predicting success. Chi-square statistics were significant at the 0.001 level and areas under the ROC curve were greater than 0.84.

Conclusions

A subjective Bayesian model was effective in predicting the outcome of actual improvement projects. Additional prospective evaluations as well as testing the impact of this model as an intervention are warranted.

This paper describes the development of a short survey instrument and the creation of a companion Bayesian model that uses the survey data to predict the potential for successful implementation of a health system change and to explain (identify) the factors that mitigate for and against success in that particular change. The goal of the instrument and model is to help change agents (persons responsible for bringing about a process improvement) detect potential obstacles to and improve chances for successful implementation. The development of the survey instrument and the Bayesian model were conducted with an expert panel. In a subsequent independent empirical study, the performance of the Bayesian model was tested using data from actual hospital and clinic improvement projects. In this paper, we (a) briefly review literature on organizational change, (b) provide details of the expert panel process used to develop the survey instrument and assemble the Bayesian model, (c) provide details of the methods used to collect and analyze data to ascertain the predictive ability of the Bayesian model, and (d) present and discuss the results. When the survey instrument and accompanying model are discussed in combination we refer to them as the “Organizational Change Manager” (OCM).

Review of Organizational Change Research

The literature relating to organizational change is extensive. In 1951 Lewin suggested three stages of change that influenced much future research. Since then many conceptual models have been developed to help understand organizational change (Finstand 1998; Weick 1995; 1976; Starbuck 1976; Orton and Weick 1990; DiMaggio 1988; Zucker 1987). Related research developed theories concerning strategic planning, adaptive learning, decision making, management applications, diffusion of innovations, and social–psychological processes of organizations (Van de Ven and Poole 1995; Chakravarthy and Lorange 1991; Burgelman 1991; March and Olsen 1976; March and Simon 1958; Carroll and Hannan 1989; Weick 1979; Rogers 1995; Gersick 1991). Still others examined how differences in organizational structure, environment, and relationships affect how members manage and achieve change (Kabanoff, Waldersee, and Cohen 1995; Coghlan 1997; Beckhard and Harris 1987; Rashford and Coghlan 1994; McCaughan and Palmer 1994; Senge 1990).

Research and development project selection models have been developed that weight factor importance (Roberto and Pinson 1972). For instance Cooper (1980) used regression analysis on 195 research and development projects to show how his eight-factor model correctly predicted the success or failure in 84 percent of the cases.

However, there is a scarcity of such models for organizational change. Lee and Steinberg (1980) suggested areas to consider in assessing the probability of successful implementation including: characteristics of company management, system users, collaboration between users, the project, the project team and their approach, and the project solution. A model based on Sink (1998) computed an organization's readiness for change using four variables: level of dissatisfaction with the status quo, a desired future state, first steps for achieving that future state, and perceived costs of changing. Models intended to facilitate organizational change have been proposed but have not been thoroughly tested (Hall, Wallace, and Dossett 1973; Snyder 1985; Pearce and Osmond 1996).

The intent of this paper is to describe the development and testing of the OCM, not to provide a comprehensive review of the organizational change field. Hence, the remaining part of this review will concentrate on the 18 factors chosen for inclusion into the OCM. Those factors are summarized in Table 1.

Table 1.

Factors and Levels Used in the Survey and Bayesian Model

		Factor Levels
Factor	Highest Rating	Middle Rating	Lowest Rating
Exploration of problem and customer needs	• Team talked to many customers to understand problem.	• Team experienced problem firsthand and knows it well.	• Team has neither experienced problem firsthand nor talked with customers. They have no data proving problem severity.
	• Personally experienced customer need.	• Have no data to prove severity.
	• Have data on severity.	• Did not involve customers.
Change agent prestige, commitment, and customer focus	• Change agent is committed to making project a success.	• Sees the project as a job and is being a good soldier.	• Does not support this project and is just going through the motions.
	• Has power and prestige.	• Has little power; has potential.	• Has no power or prestige.
	• Respects values of staff.	• Respects values of staff.
Source of ideas	• Key solution ideas came from outside the organization.	• Solution was based on successful models in other organizations.	• No attempt was made to learn from experience of other organizations.
	• Solution was then tailored to this organization.	• However, there was little tailoring to this organization' situation.
Funding	• Leaders committed money to support both problem solving and implementation.	• Either no money was needed or external source of funds was found.	• No money was committed and no external source is available.
Advantages to staff and customers	• Parties involved clearly understand the solution, feel it has many more advantages than disadvantages, and meets their needs well.	• The parties don't understand the solution and don't see its advantages or disadvantages or how it meets their needs.	• Clearly understand the solution, but believe it doesn't meet their needs and has fewer advantages than disadvantages.
Radicalness of design	• New process fits in with current philosophy and operation.	• New process seems very unconventional.	• New process seems unconventional to this organization and the industry as a whole.
	• Not a radical departure.	• Nothing like it in this organization.
Flexibility of design	• Proposed solution can (without hurting effectiveness) be easily modified to make it more appropriate for the setting.	• Design can be modified and still be effective, but it will be difficult (either for political or technical reasons).	• Design is very difficult to modify without hurting its effectiveness.
Mandate	• Leaders assigned a change agent, thought about the project carefully, clearly described the need and task, and set high performance expectations.	• Leaders initiated the project, assigned the change agent, but didn't clearly define need, task, or expectations.	• Leaders opposed the project from the start.
Leader goals, involvement, and support	• Regularly involve/inform leaders.	• Regularly involve/inform leaders.	• Solution conflicts with leadership goals.
	• Solution helps meet their goals.	• Leader goals not met by solution.	• Some leaders state opposition and are working to defeat.
	• Managers spend time and resources to support.	• Leaders endorse project but are not spending time or resources.
Supporters and opponents	• Supporters will gain much more than opponents will lose if the project succeeds.	• Opponents and supporters are about equal in what they stand to lose or gain.	• Opponents stand to lose much more than supporters stand to gain.
Middle manager goals, involvement, and support	• Regularly involve/inform managers.	• Regularly involve/inform managers.	• Solution conflicts with middle manager goals.
	• Solution helps meet their goals.	• Their goals not met by solution.	• Some are working to defeat the project.
	• Managers spend time and resources to support.	• While they don't oppose it, they don't actively support it either.
Tension for change	• Staff hate current situation and believe change is essential.	• Most feel no need to change, but a few very influential members feel change is essential.	• Staff feel no need to change.
			• They feel they have more to lose than to gain with new process.
Staff needs assessment, involvement, and support	• Team knows staff needs	• Staff leaders believe project doesn't meet their needs.	• Staff leaders believe project conflicts with its needs.
	• Solution meets some of the needs.	• They won't actively support or oppose it.	• They will fight the new process.
	• Staff wants solution.
Evidence of effectiveness	• Concrete evidence exists that the new process worked well in an organization like this one.	• No strong evidence that the new process will work, but experts believe it will work.	• Strong evidence that the new process failed when tried in organizations like this one.
Complexity of implementation plan	• Implementation plan is very simple; all understand it.	• Plan is complex but everyone understands it.	• Plan is vague and complex.
	• Implementation schedule and task assignments are detailed and clear.	• Schedule and tasks are carefully designed and clear.	• Schedule and task assignments are not clear.
		• A pilot test was conducted.	• No pilot test was conducted.
Work environment	• Leader roles, organization structure, incentives, and staffing already support the change well.	• Roles, organization structure, incentives, and staffing were modified to support the change.	• Leader roles, organization structure, incentive systems, and staffing are not set up to support the change.
		• Changes have not been tested.
Monitoring and feedback	• A specific method exists to get honest staff and customer feedback and use it to improve the process.	• No system to obtain and use staff and customer feedback.	• No feedback system exists.
	• Data on performance of new process will be collected.	• However, organization has culture of open communication with staff.	• Communications with staff are strained.
		• Will collect performance data.	• No process performance data collection is planned.
Staff changes required	• Job changes are few and clear.	• Some job retraining will be needed.	• Many job changes are needed.
	• Staff has needed skills.	• Protocols training program and materials are difficult to understand.	• There will be no protocols, training, materials, or coaching.
	• Excellent protocols and training materials were developed.	• Coaching is available.
	• Coaching is available.

Mandate/Project Launch

Creating and communicating a mandate for change enhances chances of successful implementation. Leonard-Barton and Kraus (1985) found that change was more successful when the need for innovation was defined at a high level within the organization. Lee and Steinberg (1980) found that successful implementation was positively correlated with clear objectives and explicitly identified, tangible, and measurable tasks. Delbecq and Mills (1985) suggest that leaders must energize the organization's members and enable employees to face challenging goals.

Leader Goals, Involvement, and Support

Leadership commitment, involvement, and accountability are key features for successful implementation. Among the most effective ways to engage that support is to have a change effort that is consistent with and contributes to achieving the organizational goals of the leadership (Goodman 1982; Quinn and Cameron 1989; Hurley 1992; Wynne 1979; Lee and Steinberg 1980). Support from leaders is more likely if the leaders are kept involved in and informed about the project (Kanter 1983).

Supporters and Opponents

Support of a proposed change by informal opinion leaders is essential for success. However, leaders are not always innovators. Opinion leaders are more likely to be innovative and to support a change if the organization's norm is to adopt changes (Rogers 1995). The issue then is the relative strength of the supporters and opponents.

Middle Manager Goals, Involvement, and Support

Several researchers address the important role of middle managers in the change process, emphasizing for instance their involvement in networking and negotiating (Beatty and Lee 1992; Rousseau and Tijoriwala 1999; Hosking 1988). Like senior leaders, middle managers are more likely to support a change if they believe that doing so will promote their own organizational goals (Schneider and Goldwasser 1998; Duck 1993), and if they feel involved in discussions about the change (Rousseau and Tijoriwala 1999).

Tension for Change

One of the key predictors of a successful change is whether the affected staff are dissatisfied with the current process. It is difficult to create this tension for change when none exists, suggesting that such tension should be an important consideration in deciding what projects to select. However, effective communication can spread dissatisfaction with status quo as well as announce a change, cultivate commitment, and reduce resistance (Jick 1993; Beckhard and Pritchard 1992; Hunsucker and Loos 1989; Morgan 1988; Kotter and Schlesinger 1979; Ford and Ford 1995; Donnellon 1986).

Staff Needs Assessment, Involvement, and Support

Like senior leaders and middle managers, employees are more likely to support a project that they believe is in their own best interests. The project team should understand employee needs and ensure that employees understand how the project will meet those needs (Hage and Aiken 1970; Ginzberg 1981; Delbecq and Mills 1985; Smith and Carayon 1995; Hurley 1992).

Exploration of Problem and Understanding Customer Needs

Prior to change, the organization's circumstances, problems, and needs must be analyzed. One particularly important key to success is the ability of the change to meet customer needs (Rogers 1983; 1995; Maidique and Zirger 1984). Guimaraes (1981) found incomplete and inaccurate evaluation of user needs results in implementation failure.

Change Agent Prestige and Commitment

A change agent is critical to establishing a climate for creating, implementing, and sustaining change (Meyer and Goes 1988; Stjernberg and Philips 1993). In health care organizations, a physician change agent is often most effective because of stature. Other characteristics improving a change agent's effectiveness include prestige (Freeman 1982), persistence (Schon 1963), political influence, and access to resources (Kanter 1983; Maidique 1980).

Source of Ideas

When studying research and development projects in manufacturing companies, Utterback (1971) found successful projects were more likely to use information from consultants outside the organization. Rogers (1995) also found that innovators look outside their organization for ideas. Amabile (1990) found competition from outside organizations drives creativity.

Funding

Several researchers have found change implementation more likely when sufficient money, time, and personnel are allocated (Delbecq and Mills 1985; Damanpour 1991; Hannan and Freeman 1989; Rogers 1995; Amabile 1990).

Relative Advantages

Romano (1995) found that staff will be less likely to resist if they see powerful advantages of the proposed change (Lorenzi and Riley 2000). Demonstrable benefits and valued consequences had a positive impact on implementation success in a study that examined employee opinions regarding reasons for change (Rousseau and Tijoriwala 1999).

Radicalness of Design

Change is more likely to be adopted if perceived as reasonable (Stjernberg and Philips 1993; Delbecq 1975; Kotter and Schlesinger 1979; Rogers 1995; Alavi and Henderson 1981; Guimaraes 1981; Utterback et al. 1976). Changes seen as threatening or incompatible with current conditions will be resisted (Van de Ven, Angle, and Poole 1989; Kotter and Schlesinger 1979; Schneider and Goldwasser 1998).

Flexibility of Design

Successful implementation is more likely if the new design can be easily adapted to fit into existing culture and practices (Rogers 1983; 1995; Schultz and Slevin 1975; Vertinsky, Barth, and Mitchell 1975; Kotter and Schlesinger 1979; Cooper 1980; Dickson 1976; Maidique and Zirger 1984).

Evidence of Effectiveness

Those affected by a change evaluate the change's expected consequences (Rogers 1995). This information is usually obtained from peers whose subjective opinion is often more convincing than empirical evidence, although empirical evidence is not without value. A demonstration of the change on a trial basis will also influence adoption, especially if the demonstrator is an opinion leader (Rogers 1995).

Complexity of Implementation Plan

Many changes are unnecessarily complex. A simple implementation complete with well-defined tasks and a schedule increases likelihood of successful implementation (Dane, Gray, and Woodworth 1979; Rousseau and Tijoriwala 1999; Lonnstedt 1985; Lee and Steinberg 1980).

Work Environment

Too often change is simply dropped into an organization without modifying surrounding organizational systems that influence success. Paolillo and Brown (1978) found rewards and incentives for staff to be positively related to innovativeness of organizations. Others found that clearly defined reporting relationships are needed and that organizational turbulence interferes with chances of success (Coch and French 1948; Hunsucker and Loos 1989; Schneider and Goldwasser 1998; Iskat and Liebowitz 1996; Lawler and Rhode 1976).

Staff Changes Required

Resistance results when those affected fear they cannot develop the new skills required. Hence, the fewer the changes required the greater staff self-efficacy (Bandura 1977; Compeau and Higgins 1995; Lorenzi and Riley 2000). Training that employs peer role modeling minimizes resistance and increases likelihood of successful implementation (Kotter and Schlesinger 1979; Coch and French 1948).

Monitoring and Feedback

Change that can be measured and is tracked tends to be implemented more successfully (Rogers 1983; 1995; Maidique and Zirger 1984; Nadler and Tushman 1990). Furthermore, soliciting employee feedback in a risk-free manner during implementation can shape attitudes favorably toward the change (Hunsucker and Loos 1989; Kanter, Stein, and Jick 1992; Beckhard and Pritchard 1992; Beer, Eisenstat, and Spector 1990).

Survey and Bayesian Model Development

This section will describe the process by which those 18 factors were identified, measured, and agreed upon for the survey. It will also describe how the Bayesian model was developed.

The survey instrument and the accompanying Bayesian model were developed using the Integrative Group Process (Gustafson, Cats-Baril, and Alemi 1992). A panel of experts identified and quantified the factors used in the instrument and the Bayesian model. To identify and select the experts, we employed a snowball nomination process in which six leaders in organizational change were asked to nominate experts who (1) understood the theoretical and practical aspects of organizational change in health care, (2) were respected by their peers, and (3) would function effectively in a group (thoughtful, not domineering). When an expert was nominated twice they were listed as a candidate for our panel. Those receiving the most nominations were sent a letter describing the project and indicating that they would receive a phone call to discuss it further. Ultimately the panel was composed of four theoreticians (e.g., a professor of organizational change specializing in health care) and three healthcare practitioners (e.g., an administrator of a state department of health with a reputation as a “mover and shaker”).

The senior author interviewed each panelist for approximately one hour by phone to determine what “implementation success” meant to the expert, what questions they would want answered before they predicted whether an organizational change would be successful (our strategy to identify factors), and what answers to each question would make them optimistic or pessimistic about success. This provided clues on how to define levels for each factor.

Because our intent was to develop a model that not only predicted chance of success but also provided a tool for improving chances of success, experts were encouraged to choose only factors that were modifiable and causally related to successful implementation.

During interviews, the experts would identify factors by responding to the following question: “Suppose you were asked to predict whether a project would be successfully implemented. You can ask me any question you want about the project and I will find the answer for you. What questions would you ask of me? Also please give me examples of answers that would make you optimistic and pessimistic about the chances of success.” The optimistic and pessimistic responses gave examples upon which the levels of the factors would be determined.

For example, two experts said they would want to know what type of problem exploration occurred. One expert said they would be optimistic if the response was that the team talked with several customers to identify problems. Another said they would be optimistic if the team personally experienced the problem and also had data to demonstrate the severity of the problem. In preparation for the meeting these example answers were combined and the panel, after deciding that problem exploration would be a factor, created the levels. As it turned out, the levels for “Exploration of Problem and Customer Needs” were:

Highest (strong positive influence) rating:	The team talked to many customers to understand the problem, personally experienced the problem, and had data proving severity of the problem.
Middle (minor influence) rating:	The team experienced the problem firsthand and knows it well. They had no data to prove severity of the problem and did not involve customers.
Lowest (strong negative influence) rating:	The team had neither experienced the problem firsthand nor talked with customers. They had no data proving problem severity.

We also reviewed the literature to identify definitions of success, factors, and levels of performance. We considered the literature review to be a secondary source of this information. Our prime source was the interviews with the experts because their suggestions would be based on a specific understanding of how the factors would be used in this particular application. The literature review provided a way to ensure completeness. Prior to a face-to-face meeting of the panelists, the senior author combined the interview and literature data into a nonduplicative taxonomy of more than one hundred factors and possible measures—the “Straw Model.”

Two weeks after the telephone interviews were completed, the panel convened for a face-to-face meeting that lasted from noon one day to noon on the next. During the first day the panelists reviewed the straw model's definitions of success, factors, and associated measures. They were told that before adjourning that day they had to agree on: (1) a definition of success, (2) a small number of factors (<20) that were conditionally independent and would not only predict but also explain whether an improvement project would be successfully implemented, and (3) ways of measuring each factor.

The panel decided that success meant a process improvement that persisted six months after implementation and still had the support of both management and staff.

As a test of conditional independence, the factors that resulted from the discussion were each placed on a 3“×5” card. Panelists were first asked to assume a successful project. They were then asked to sort the cards so that cards would be in the same pile only if knowing the answer to one of them would tell a lot about the answer to the other(s). The task was repeated assuming a failed implementation. Cards sorted into the same pile were then discussed by the panel, and if they agreed that they belonged together then the factors were either rewritten to distinguish the real differences, or all but one factor in a pile were eliminated. For example, “problem exploration” and “understanding of customer needs” were grouped into one common factor.

Because most factors could not be measured quantitatively, the panel created three or four descriptions (which we call “factor levels”) of their potential influence on implementation outcome (strong positive influence, minor influence, strong negative influence). A strong positive influence would increase the chance of implementation; a strong negative influence would decrease it.

By 7:00 PM the panel had completed its first day's work. Table 1 shows the version of the instrument used to predict the outcome of 221 change efforts in health care organizations that formed the basis for the evaluation reported later.

That evening the project staff created forms to allow the panel to estimate the parameters of a Bayesian model. We also created a set of 60 hypothetical profiles of implementation projects described in terms of one level for each factor. We did so by first creating profiles at the “corner points” (e.g., one profile with all factors at the highest levels and one with all factors at the lowest level). Then a computer program generated the remaining ones using a random number generator. We reviewed the profiles to check whether each factor level occurred with about the same frequency. We also examined whether any profiles did not make sense; one factor level could not be present if another factor was at a certain level. If we found unrealistic combinations it would have suggested that the factors were not independent. None were found.

As discussions were winding down, the panel was asked whether there were any types of changes that would be much more difficult to make than others. It was agreed that changing culture was much more difficult than changing process. The purpose of this question was to decide what issues should be considered in defining Prior Odds, a key element in the Bayesian model and a concept we will discuss next.

Bayesian Model Development

Bayesian modeling refers to a wide range of models applied in many areas. In this particular modeling effort, a simple, yet extremely valuable, application of Bayes' rule permitted us to quantify—through expert subjective estimation—the influence of the 18 factors identified and defined by the expert panel to construct a predictive model of implementation success. Such subjective Bayesian models (see for example Gustafson, Cats-Baril, and Alemi 1992, chapter 8; Gustafson et al. 1993) estimate the probability of an event, given certain information about factors that influence it by applying the odds version of Bayes' theorem:

Posterior Odds = Product of Likelihood Ratios × Prior Odds

• Prior Odds are the initial estimates of how much more likely it is that a change will be a success than a failure. Mathematically, prior odds are the ratio of prior probability of success to prior probability of failure, that is, [P (S) /P (F)], where “S” is a successful change and “F” is a failed change.

• Likelihood Ratios indicate the diagnostic power of particular information, that is, the extent to which a datum revises the prior odds. Likelihood ratios are the ratio of the conditional probability of observing the level of a particular factor given a successful implementation, to the conditional probability of that same datum given a failed implementation: [P (D|S)/P (D|F)].

• Posterior Odds are the ratio of the probability of success given the datum to the probability of failure given the same datum, that is, [P (S|D)/P (F|D)]. They represent the revised odds given the datum of information learned.

If there are n data points (D₁, D₂,…,D_n), and they are conditionally independent, the odds version of Bayes' theorem can be written as:

$$\begin{array}{l}[P(S|D_1,\dots ,Dn)/P(F|D_1,\dots ,D_n)]\\ =[P(D_1|S)/P(D_1|F)]\times \dots \times [P(D_n|S)/P(D_n|F)]\times [P(S)/P(F)]\end{array}$$

In many situations, likelihood ratios and prior odds can be estimated empirically. However, if necessary data do not exist or are insufficient, behavioral decision theorists assert that likelihood ratios can be estimated subjectively by trained experts (Edwards, Lindman, and Savage 1963; Slovic and Lichtenstein 1971; Hogarth 1975; von Winterfeldt and Edwards 1986).

In the research reported here, no data were available to empirically estimate the prior odds and likelihood ratios for the 18 factors, therefore the expert panel was assembled and guided through a structured process to estimate them. Such process has been used successfully in developing a similar Bayesian model in the context of evaluating the probability of high quality care being delivered in psychiatric emergencies (Gustafson et al. 1993).

While it may appear that experts could directly estimate the probability of success, the cognitive burden in forming an evaluative response to the simultaneous influence of n factors (D₁, D₂,…, D_n) is much greater than the cognitive burden in forming an evaluative response to one factor at a time (Gustafson et al. 1993; Luke, Stauss, and Gustafson 1977). Thus, experts are better equipped in subjectively estimating n individual likelihood ratios than in subjectively estimating posterior odds given n pieces of data presented simultaneously. To ensure that likelihood ratios can be estimated one by one, it is necessary to define the pieces of data (D₁, D₂,…, D_n) so that conditional independence can be assumed. As mentioned above, the panel was provided specific tasks to assess conditional independence, and if necessary to redefine factors to achieve it.

Hence, the next morning the panel estimated the prior probability of implementation success. These estimates, like all estimates that morning, involved an estimate-talk-estimate approach (Gustafson et al. 1973) and followed a strategy we have used in other health care applications (Gustafson et al. 1977; 1990). Panelists individually provided numerical estimates, which were posted on a flip chart, and if estimates were widely different (a judgment made by the senior author), the panelists were asked to discuss the rationale behind their scores. After discussion panelists were allowed (but not required) to individually modify their estimates. The revised estimates were then averaged.

The panel estimated each likelihood ratio for the Bayesian model in two different ways.

1. Likelihood ratios were estimated directly by responding to the following two questions:

   • Think about two healthcare improvement projects. One was successfully implemented and the other was not. Which one is more likely to have the following characteristics. (Then a factor level was specified, such as “a staff that hate the current situation and believe that change is essential.”)

   • How much more likely? (a lot, somewhat, barely)

2. Setting these estimates aside, the panel then gave numerical estimates of the individual likelihood ratios for each factor. They were asked to assume that we had dossiers on a random selection of one hundred successful implementations and one hundred failures. They were shown the levels of that factor (from Table 1) and asked to distribute the one hundred successful projects and the one hundred failed projects among the different levels on the factor being considered. For instance they each completed a table similar to Table 2 for the “problem exploration and customer needs understanding” factor. Once these estimates were completed, again using the estimate-talk-estimate approach, the scores in the success row were divided by scores in the failure row to yield likelihood ratios as shown in third column of Table 2.

Table 2.

Likelihood Ratios for the Factor “Exploration of Problem and Customer Needs”

Factor Level	Successful Projects P(D1∥S)	Failed Projects P(D1∥F)	Likelihood Ratio $\frac{P(D_1|S)}{P(D_1|F)}$
The team talked to many customers to understand problem, personally experienced the customer need, and has data proving severity.	77	13	∼6/1
The team experienced problem firsthand and knows it well, but has no data to prove severity and did not involve customers.	22	78	∼1/3.5
The team has not experienced problem firsthand and has no data proving problem existence or severity.	1	9	∼1/9
Total	100	100

As a consistency check, these ratios were compared to the judgments obtained in response to question A. If there were differences (e.g., a response of “much more likely” in task A and the numerical estimate was small in task B) then further discussion ensued and, if needed, task B was redone.

A similar process was used to estimate prior odds. The expert panel decided that the probability of successful implementation was 5 percent in projects involving cultural change and 16 percent for changes involving only process changes.

The final task was to conduct a preliminary first test of model performance. The panel (again using the estimate-talk-estimate approach) directly estimated the chance of the implementation success for each of the 60 profiles created the previous evening. The 60 profiles were also each assigned a score calculated using the Bayesian model. The scores assigned by the panelists were averaged for each profile. The correlation between those average scores and the Bayesian model scores was .77. This performance test, however, does not ascertain the validity of the model in predicting success probability because (1) it compares two types of subjective assessments, (2) it involves a small number of hypothetical profiles, (3) it is performed by the same panel that developed the model, and (4) as explained earlier, such holistic judgments are difficult to make. Thus, to investigate the validity and performance of the Bayesian model developed by the expert panel, an independent evaluation study involving actual change projects was performed and is described in the next section.

Retrospective Evaluation of The Bayesian Model

This section describes how the survey and Bayesian model were evaluated using actual change results. Senior leaders from 198 health care organizations attending seminars on organizational improvement offered by the Institute for Healthcare Improvement (http://www.ihi.org) identified an improvement project with which they were intimately familiar and for which outcome was known. These leaders, who were not part of the expert panel that developed the model, completed a survey instrument, which asked them to indicate:

• whether the project involved cultural change (to determine which prior odds to use, 5 percent or 16 percent).

• the project's status in terms of success (don't know, big success, small success, small failure, big failure), time since implementation (less than six months, six months or more, don't know) and current status (ongoing, no longer in place, don't know). Table 3 displays the joint distribution of success, time since implementations, and current status for all 221 projects (big success, modest success, modest failure, big failure).

• which factor levels (as shown in Table 1) best described their project. They could choose level 1, 2, or 3, or they could say their project was somewhere between 1 and 2 or between 2 and 3. Our research team used the appropriate likelihood ratios and prior odds to calculate the posterior odds for the project.

Table 3.

Project Outcome, Duration, and Current Status (N=221)

		Ongoing?
Outcome	Duration of Implementation	DK*	No	Yes	Total
Don't know	Don't know	11	0	7	18
	Less than six months	1	0	6	7
	Six months or more	2	0	2	4
	Total	14	0	15	29
Small success	Don't know	14	0	0	14
	Less than six months	0	0	3	3
	Six months or more	2	1	13	16
	Total	16	1	16	33
Big success	Don't know	31	0	25	56
	Less than six months	3	1	14	18
	Six months or more	5	1	23	29
	Total	39	2	62	103
Small failure	Don't know	14	8	0	22
	Less than six months	2	2	1	5
	Six months or more	1	6	11	18
	Total	17	16	12	45
Big failure	Don't know	4	2	1	7
	Less than six months	1	0	0	1
	Six months or more	0	2	1	3
	Total	5	4	2	11

^*DK=Don't know

The model has been designed to predict probability of success or failure of a project prior to its implementation. The data at hand ascertains success or failure of 221 actual projects. Hence, using actual success/failure as a dependent variable and Bayesian model scores as a predictor, we used both logistic regression and ROC analysis (Metz 1978; Swets 1988; Sainfort 1991) to evaluate the performance of the model. Specifically, the chi-square statistics from the logistic regression, the overall percentage of correct classification, and the area under the ROC curve are reported as measures of goodness of fit.

To analyze how well the model predicted the outcome, the outcome (success or failure) has to be known, leading to a smaller set (n=192). Three sets of analyses were performed using three definitions of success. In the first set, a very stringent selection criterion was applied to define success. In the second set, a slightly less stringent selection criterion was applied. The third set included all successful projects, independent of their duration.

Selected cases for the first analysis:

• For successful (small or big success) projects, implementation must be ongoing for six months or more (n=45).

• All failed projects (n=56).

Selected cases for the second analysis:

• For successful (small or big success) projects:

  Either project has been implemented for six months or more (n=45),

  Or project is considered successful but has been implemented for less than six months and is still ongoing (n=42).

• All failed projects (n=56).

 Selection criteria for the third analysis:

• All successful (small or big success) projects whether ongoing or not (n=136). The rationale here is that a project could have been intended as a temporary, rather than permanent, intervention.

• All failed projects (n=56)

Results

Analysis 1

With a very stringent definition of success, 45 successful and 56 failed projects were retained. The logistic regression chi-square statistic is 29.62 and significant at the 0.001 level. The area under the ROC curve is 0.849 (standard error=.039) and is significantly greater than 0.50 at the 0.001 significance level.

Analysis 2

With a less stringent definition of success, 87 successful and 56 failed projects were retained. The logistic regression chi-square statistic is 42.24 and significant at the 0.001 level. The area under the ROC curve is 0.842 (standard error=.035) and is significantly greater than 0.50 at the 0.001 significance level.

Analysis 3

Including all successful projects, 136 successful and 56 failed projects were retained. The logistic regression chi-square statistic is 53.98 and significant at the 0.001 level. The area under the ROC curve is 0.846 (standard error=.032) and is significantly greater than 0.50 at the 0.001 significance level.

Table 4 shows summary results (chi-square statistics, area under the curve, standard error, significance level, and 95 percent confidence interval) for the three different analyses performed above. Thus, independent of the success definition used, the model did well in predicting success or failure.

Table 4.

Logistic Regression and ROC Results for the Three Outcome Definitions

Analysis	Total N	Chi-Square (P-value)	Area under Curve	Standard Errora	95% Confidence Interval
1	101	29.62	.849	.039	.773 to .925
		(0.001)
2	143	42.24	.842	.035	.774 to .911
		(0.001)
3	192	53.98	.846	.032	.783 to .908
		(0.001)

^aUnder the nonparametric assumption.

Discussion and Conclusions

This was a study to determine the extent to which a subjective Bayesian model could predict the outcome of organizational change attempts. The model performed well on all three definitions of success using logistic regression and ROC analysis. While results are encouraging, further research is needed.

First, more tests of reliability and validity of the survey and model are needed. For instance, only one person completed the survey for each of the 221 projects. In addition, the assessment of predisposing factors and the judgment of success were conducted by the same raters. Thus, this could have led to a slight overestimation of the model performance. What interrater reliability would be achieved had several people familiar with the project completed the survey? We subsequently asked the members of one improvement team (ranging in size from three to six people) in each of six hospitals, not part of the research reported above, to independently complete the survey. Average team member agreement on item scoring was 84 percent. On a related issue, our application of Bayes' Theorem assumes conditional independence. While we did not conduct a statistical analysis, the expert panel did engage in a formal process (described earlier) to identify and remove conditional dependencies. Further testing could be needed to ensure the validity of this assumption.

Second, there is no objective measure of success. Success was measured by opinions of the people who completed the survey. The survey was completed by one person who, although being very familiar with the project, may not have had an objective perspective. Ideally, a team of people from each organization would independently complete the survey and then after discussion reach a consensus on the survey as well as the level of success.

The process we employed is intended to produce a survey/model that both predicts and explains reasons for success and failure. During the factor elicitation process experts were encouraged to select factors that were not only correlated with success but were causally related to it and were actionable. While the survey/model was thus intended to predict and explain, the evaluation reported here did not assess the validity of the causal links.

Some factors that are arguably important in predicting and causally linked to organizational change were not included in the OCM; for example the characteristics of the project team and issues related to organizational culture. Both were included in the straw model, but not in the final model. The expert panel felt that the change agent was more important than the team and that culture was not easily modifiable (a key consideration in selecting factors for this model). Still, strong arguments could be made for their inclusion in such models.

Despite the study's limitations, the results are encouraging and the survey is easy to complete (taking approximately 15 minutes). Hence the OCM may be useful in applications such as those described below.

The OCM may help educate those seeking to bring about change. Many in these roles lack formal education in research and theory of successful change. Others may be blind to potential barriers, even though they understand the theory. The model may introduce factors critical to successful change and to current realities. It may also assist in establishing a common language and in preventing future conflict arising from lack of consensus. In theory these people could also learn about change by reading a literature review. However, a simple survey that provides feedback forces the user to think about issues in ways that a journal article might not.

The model may assist in determining whether change is likely to be worth the effort required. Failed change can be costly in resources consumed, opportunity costs, and an organization's loss of confidence in its ability to successfully change. Repeated failures can breed cynicism and increase resistance when the experience has been, “This too shall pass.” Thus, the model's predictive ability may help evaluate the wisdom of pursuing a particular change.

In planning change, the survey and model may lead to preventive action directed toward low scoring factors with high likelihood ratios. Barriers can be removed and critical aspects strengthened before an initiative is even begun. For instance it might be determined that leaders were going to make change desirable but not essential. Discussions prior to project launch might convince the leadership to take a more aggressive position on the importance of change.

In implementing change, the model may assist groups in evaluating an ongoing change process over time. Scoring the change at key checkpoints, and having group members complete the instrument and discuss resulting scores, can strengthen the likelihood of success.

The model may also help quantitatively track the direction of change. Completing the instrument several times during a major initiative may help keep the effort on track and measure the effect of corrective actions taken between evaluations. Using an instrument can help depersonalize issues and legitimatize change agents' requests to strengthen critical factors.

The model may help organizations learn from past efforts. Many organizations experience repeated failure of change efforts due to their inability to learn from experience. An organization's agility may be improved through repeated use of the model. This knowledge may be more systematically applied to future efforts, leading to more successful change efforts, and greater collective confidence in the change process.

However, the model's value in the applications described above requires empirical validation. While the model has been used for such purposes, the only evidence of effectiveness as an intervention (rather than a predictor) is anecdotal. Randomized trials, where improvement teams in one arm receive feedback from analyses of the survey and those in the other arm do not, would provide important insights into the OCM's potential as an agent of effective change. Other useful insight would come from prospective studies in which several people rate the chances of success and score the predisposing factors, and also from studies assessing the reliability of those assessments. These studies would help us determine the number of raters needed to ensure adequate reliability.