Organizational Measurement and the Implementation of Innovations in Mental Health Services

Abstract

This paper presents a brief review of organizational measures related to implementation of new practices and technologies in sectors other than mental health, and discusses potential application of these measures to mental health implementation research. A few standardized organizational measures are presented along with considerations regarding the appropriateness of adapting existing measures rather than creating novel ones or using additional methodologies. Challenges and opportunities for researchers in measuring key organizational constructs related to implementation in mental health settings are discussed.

Introduction

The recent awakening of interest in the dissemination, implementation, and sustained use of evidence-based mental health and substance abuse practices has generated widespread discussion of the impact of the organizational settings into which the practices are introduced on their adoption and sustained use and a search for the kinds of organizational attributes whose impact might be significant. Research on the influence of organizational attributes on adoption and sustained use of innovation, while plentiful, suffers from conceptual ambiguity and a variety of measurement issues.

It is important to understand the types of factors that influence the adoption and sustained use of innovations in organizations on both theoretical and practical levels. From a theoretical perspective, this understanding is achieved by examining the extent to which there is a common set of attributes that influence implementation across both organizational settings and types of technologies and practices. The goal is to determine whether there is a macro-level general model of implementation that encompasses a multitude of sectors (e.g., business, medical and mental health, and education) or whether there are instead a number of mid-range theories that are context specific in their application [e.g., implementation of new treatments for Post-traumatic Stress Disorder (PTSD) in Department of Veteran Affairs]. From a practical perspective, whether the model is general or context specific, it is vital that an agent of change understand how to utilize what is known in order to achieve successful implementation of a new practice in any given organization.

Two common basic concerns in implementation research and practice are the what and the how. In the process of instituting change the what refers to the nature of the change itself (target; e.g., a new treatment protocol, computer system, set of work practices, or a reconfigured design for the organization as a whole) and the how refers to implementation (process; see Klein and Sorra 1996) or the steps required in order to insure that the new practice or technology is both adopted and used by members of the organization. The what and the how are inextricably linked both by needs within the organization and by those who will utilize the new practice. The most elegant and sophisticated of new practices will not be implemented if they are not embraced by potential users. The study and design of implementation are concerned precisely with the issue of compatibility or “fit” between the new technology or practice and the organizational system into which it is being introduced. Research on implementation, therefore, needs to be concerned with the constellation of factors that influence adoption and subsequent use.

As might be expected given the significance of “fit,” a great deal of research has focused on what at the most general level might be called the management of change. This paper focuses on one piece of the much larger puzzle of organizational change: The impact of organizational level attributes on implementation. A review of organizational attributes that have traditionally been used by organizational researchers is provided, followed by a presentation of a number of “off-the-shelf” or standardized organizational measures. A discussion of the advantages and disadvantages of using standardized measures in research on implementation is presented. This includes an overview of a number of salient issues that investigators must inevitably confront as they design research on the role of organizational attributes in implementation of new technologies or practices in mental health organizations. Additionally, we suggest that investigators give serious consideration to several limitations or problems in existing research in this area.

Before identifying and evaluating measures of various organizational attributes that are related to implementation, we would like to call the reader’s attention to two features of the literature we have reviewed. First, the literature includes both qualitative and quantitative research. For researchers in mental health, where studies tend to be quantitative rather than qualitative, this may be somewhat frustrating. However, for reasons we will suggest later, a mix of qualitative and quantitative approaches is appropriate for studying implementation, and we would refer the interested reader to an excellent review of work in this area by Greenhalgh and colleagues (Greenhalgh et al. 2004) for a more detailed explanation of this point. Another feature is that some studies aggregate the perceptions of individuals to measure organizations. The problem with this is that the perceptions of individuals are necessarily limited. Individuals may know their part of the organization very well, but the researcher needs to be careful to avoid the implicit assumption that all parts of the organization are similar, as they would if they simply averaged perceptions to measure organization-level variables. If in our presentation here, we were to limit ourselves to the kinds of measures that traditional mental health researchers typically utilize, we would poorly inform readers of the activity in this area. The issue is one of methodologies versus measures. For example, the widely used measures of organizational culture are based mainly on self-report surveys. While the underlying constructs of these measures provide important information, this knowledge is limited or potentially biased by the fact that the methodology used is self-report. Self-reports have the disadvantage of only capturing a restricted range of content, and they can be influenced by intentional false reporting, inattentive responding, cognitive or memory limits, and acquiescence.

There is some programmatic research (e.g., Moos and Moos 1998) where the investigators developed a set of constructs, measured them across diverse environments and marketed these tools for others to use. However, there are far more instances where a number of different authors have started with the same construct (e.g., organizational culture) and measured it in more idiosyncratic fashion. This obviously makes comparisons difficult because the same label is being applied to different measures, none of which have been widely used.

Our goal in this paper is to raise these issues and implications and introduce a select number of organizational variables, methodologies and measures that might be particularly attractive to researchers interested in implementation rather than covering a more extensive, less focused, list of organizational variables and their measures.

Organizational Measures: Their Role in Implementation

Researchers have traditionally used four categories of variables when examining organizational influences on implementation: (1) Structure, (2) culture or climate, (3) internal processes, and (4) leadership. Because of space limitations, we chose in this paper to focus on the first two categories, organization structure and culture/climate with limited reference to organizational processes and leadership.

Organization Structure

In a meta-analysis of determinants and moderators of organizational innovation in the change management literature, Damanpour (1991) found that 13 organization level variables were significant. The names and definitions of these variables and their relationship to implementation of innovations are presented in Table 1. There appears to be no specific standardized measures that assess these organizational variables. Ideally, one then would want to start from scratch by developing such measures specifically tailored for mental health organizations. One-way to do this is to identify a sample of organizations that vary with respect to their success in implementation. Identification of parameters, or organizational variables distinguishing between differently effective implementers, would then serve as a starting point for developing standardized measures. Some of these organizational variables such as specialization, administrative intensity as well as functional and vertical differentiation could be measured with simple counts. Others require somewhat greater investment by the researcher. To measure formalization, for example, one might code the extensiveness and complexity of procedure manuals in the sample of organizations being studied and develop an index based on the variability that was found. Still others would be more difficult to quantify and standardize across organizational settings. For example, professionalism could be assessed by obtaining an employee’s resume and noting degree(s), years of experience, post-graduate school, and number of specialized trainings/certificates. To link the construct to actual behavior, one would have to include ratings received from supervisees, peers, and supervisors, thereby introducing an important social component that would be impossible to capture otherwise.

Table 1.

Organizational structure measures and their relationship to innovation

Variable	Definition	Relation to innovation
Structure
Specialization	Different specialties or role complexities in an organization (e.g., number of different occupational types or jobs)	+
Functional differentiation	Number of different units in terms of structure, department or hierarchy	+
Professionalism	Education, training and experience of staff	+
Formalization	Degree to which organization is run by rules and procedures	−
Centralization	Extent to which locus of authority and decision-making are dispersed or concentrated	−
Managerial attitude toward change	Extent to which managers are in favor of change	+
Managerial tenure	Manager’s level of service and experience	+
Technical knowledge resources	Staff technical current knowledge and potential	+
Administrative intensity	Ratio of administration/managers to total staff	+
Slack resources	Unencumbered budget, finance and expenditures as well as human resource slack	+
External communication	Degree of staff involvement and participation in extraorganizational professional activities	+
Internal communication	Extent of communication among organization units or groups	+
Vertical differentiation	Number of levels in an organization’s hierarchy	−

This table is based on tables and appendix from Damanpour (1991)

In order to assess centralization, one can examine the budget approval and decision-making process (e.g., hiring and firing). Specifically, this may involve the identification of how many individuals are involved in this process, at what level of the organization these decisions are made and how many iterations are required before a decision is made.

In order to assess managerial tenure, one can view human resource records to ascertain how many years an employee has been with the company, in how many different positions have they served and with what level of performance. A numerical index could be created to rank the managerial complexity of these jobs and this could be multiplied by the number of years the person served in that particular job. Managerial attitude toward change could be assessed via face valid surveys or on the job demonstrations of willingness.

Technical knowledge could be assessed via a knowledge, attitude and practice survey in a particular content area or as an on the job demonstration/role-play. Slack resources might involve the examination of the actual versus spent budget and/or the total potential hours each employee is available versus actual time spent working.

External communication could be assessed via employee attendance and participation (such as presentations or committee roles) in local, state, and national associations and their conferences. Internal communication in an organization might be best measured via social network analysis (Wasserman and Faust 2005). The number of direct and indirect ties between employees could be calculated based on various communications technologies such as email or documents sent through interoffice mail.

Each of these variables presents its own special measurement challenges, but each has a history in the organizational literature and should therefore be accessible to the researcher interested in implementation in mental health.

Organization Culture/Climate

Theoretically, one might expect there to be a relationship between organizational culture and implementation through its effect on levels of collective engagement with, or collective resistance to, efforts to introduce new technologies or practices to an organization. Culture typically refers to basic assumptions, values, and behavioral norms and expectations found in an organization or its subunits (Rousseau 1990). In contrast to measures of organization structure described above, measures of culture or climate are typically generated from the perceptions of organizational members and are based on some average value or other central tendency of those perceptions. Because they are based on member perceptions and are hence subjective by definition, measures of culture or climate tend to be viewed as “soft” in comparison to measures of structure, and there are extensive debates regarding definition, measurement, and appropriate level of analysis of study.

The methods that have been used to measure organizational culture/climate in non-mental health settings typically rely upon self-report instrumentation, but also include a broader range of approaches such as ethnography, participant observation, and qualitative analysis of archival materials. The investigator interested in an organizational measure that can simply be administered by implementation researchers to stakeholders can choose among several standardized self-report measures of organizational culture/climate, all of which are psychometrically sound and have acceptable levels of construct and predictive validity. Some of the most widely used are described in detail below.

Zammuto and Krakower (1991) developed an organizational culture measure, variants of which are commonly used in the literature (see, Zammuto et al. 2000; Zammuto and O’Conner 1992). This instrument assesses organizational character, cohesion, emphases, rewards, and managerial attributes. Organizational character assesses the degree to which the environment is: (1) Personal, (2) dynamic and entrepreneurial, (3) formalized and structured, and (4) production-oriented. Cohesion assesses the degree to which the organization is based on: (1) Loyalty and tradition, (2) commitment to innovation and development, (3) formal rules and policies, and (4) tasks and goals. Emphases assess the degree to which an organization stresses: (1) Human resources, (2) growth and acquiring new resources, (3) permanence and stability, and (4) competitive actions and achievement. Rewards are measured based on whether or not they are: (1) Fairly distributed amongst members, (2) based on individual initiative, (3) based on rank, and (4) based on achievement of objectives. The last subscale on this culture measure reflects whether the managers are: (1) Warm and caring, (2) risk-takers, (3) rule-enforcers, and (4) coordinators and coaches (Table 2).

Table 2.

Organizational culture measures: Source and concepts covered

Name	Source/how to obtain	Concepts covered
Organizational culture/climate
Organizational Culture	Zammuto and Krakower (1991)	Organizational character, cohesion, emphases, rewards and managers
Organizational Culture Assessment Instrument	Cameron and Quinn (1999)	Organizational characteristics, leadership, management of employees, strategic emphases, and criteria of success
Work Environment Scale	Consulting Psychologists Press Phone 1-800-624-1765	Organizational structure and functioning, physical resources, staff characteristics, and work relationships
Organizational Culture Profile	O’Reilly et al. (1991)	Person-organization fit
Individual and Organizational Performance	Burke and Litwin (1992)	Organizational mission and strategy, management systems and practices, leadership, employee motivation, needs and values
Organizational Readiness for Change
Readiness for Organizational Change	Davis and Salasin (1977)	Organizational resources, change values, resistance to change, and motivation
Organizational Readiness for Change	Lehman et al. (2002)	Organizational functioning and readiness to change

The Organizational Culture Assessment Instrument (OCAI: Cameron and Quinn 1999) was constructed based on an empirically derived theoretical model called the “Competing Values Framework.” It has questions assessing six dimensions of culture: Dominant characteristics, organizational leadership, management of employees, organization glue, strategic emphases, and criteria of success. Each question poses four alternative responses based on culture types. Total scores fall along two dimensions. One dimension represents a continuum ranging from organizational adaptability and flexibility to organizational stability and permanence. The other dimension represents a continuum ranging from organizational cohesion and harmony to organizational division and independence. These two dimensions form four quadrants that are contradictory or competing on the diagonal and illustrate the four major culture types (i.e., clan, adhocracy, market, and hierarchy). According to Cameron and Quinn (1999), the OCAI has been used in more than a thousand organizations and has excellent face and empirical validity.

The Work Environment Scale (WES; www.cpp.com) is a measure of the social climate of a work unit, such as organizational functioning, physical resources, staff characteristics, and relationships among employees and between employees and supervisors. It has been utilized for numerous purposes, including evaluation of productivity, measurement of employee/employer satisfaction, assessment of needed improvements in workplace and as a tool to monitor change over time. The WES scoring key and manual are published by and available from Consulting Psychologists Press at www.cpp.com.

Interestingly, the WES has been utilized extensively in substance abuse treatment settings. Moos et al. (1990) reviewed the use of the WES in long-term studies of treatment outcome among alcoholic patients. Moos and Moos (1998) described the use of the WES to assess and identify the connections between substance abuse treatment programs’ staff work environment and treatment environment. This measure stands out as one with good face validity, good psychometric properties, and a history of use in mental health care and substance abuse treatment settings.

The Organizational Culture Profile (O’Reilly et al. 1991) is another quantitative assessment of organizational culture, developed and validated as an instrument to measure person-organization fit. It contains a set of 54-value statements that can be used to assess both the extent to which certain values characterize an organization and individual’s preference for those values (person-culture fit). The statements were developed on the basis of an extensive review of academic and business practitioner-oriented literatures. A larger pool of items was then reviewed for content and redundancy by a range of individuals, including academic faculty members in a school of business. The measure can be found in the appendix of O’Reilly et al. (1991).

Another comprehensive organizational functioning measure is a 90-item questionnaire (Burke and Litwin 1992) based on the Burke–Litwin Model of Individual and Organizational Performance (Burke and Litwin 1989). This model includes external environment, mission and strategy, leadership, culture, structure, management practices, systems, work group climate, task requirements and individual skills, motivation, individual needs, and values and organizational performance.

Organizational Readiness for Change (Openness) Measures

The concept of readiness for change has a long history of application to the uptake of and adherence to mental health and substance abuse treatments by patients. When applied to implementation in non-mental health settings, measures of readiness for change examine components of both structure and climate which contribute to an organization’s ability to make change in procedures, objectives, goals or human resources. Readiness for change encompasses attitudes of employees, training and approach of leaders, level of motivation for all organization members as well as the actual physical resources of the organization. These resources may simply include available office space or may encompass existing utilization of information technology and means of communication both within the organization and between organizations or individuals.

Prochaska et al. (2001) applied their Transtheoretical Model for individual behavior change to organizational change theory, research, and practice. Below are descriptions of measures that capture this construct.

Texas Christian University Organizational Readiness for Change (ORC: Lehman et al. 2002) is a 115-item Likert-type measure of organizational functioning and readiness to change. It represents four major areas: Motivational readiness, personality attributes of program leaders and staff, institutional resources and organizational climate. Motivational readiness includes perception of program strengths and weaknesses, training needs and internal/external pressure for change. Personality attributes of program leaders and staff includes growth, efficacy, influence, adaptability, and clinical orientation. Institutional resources include adequacy of office and physical space, number and quality of available staff, management and financial support for staff training and development, adequacy and use of computers, and use of e-mail and Internet for professional communications and information access. Organizational climate includes mission, cohesion, autonomy, communication, stress, and openness to change. Lehman et al. (2002) described the ORC’s development, including rationale, and structure, as well as its preliminary psychometric properties. It was developed in part by adapting existing organizational climate scales to human services organizations and can be downloaded free of charge at www.ibr.tcu.edu.

An older instrument, Readiness for Organizational Change (Davis and Salasin 1977) measures obligation, ideas, ability, values, circumstances and timing, resistance, and yields. Specifically, obligation refers to a participants’ awareness of need, drive, motivation, and desire to bring about change. The subscale on ideas includes information or new knowledge about the innovation, and a specific action plan to bring about change. Ability attempts to capture the capacity and resources to implement the innovation such as personnel, training, budget, space, and time. Values refer to the perceived organizational purpose, accustomed operational conduct and self-concept. Circumstances and timing was designed to capture current environmental activities and conditions that might facilitate or impede implementation. The subscale of resistance was designed to describe inhibitions or anxieties related to change, including personal and product risks and losses if action is taken or innovation is adopted (e.g., loss in job status, power, and worth). Lastly, yield measures perceived benefits and rewards of change. Lehman et al. (2002) point out that the primary aim of the Readiness for Organizational Change measure is evaluation, while the ORC focuses on technology transfer particularly in substance abuse settings, and places greater emphasis on organizational climate and staff attributes.

Additionally, an important resource for names of additional measures of organizational structure/effectiveness can be found on the Internet site sponsored by the Center for Mental Health Services Research at Washington University in St. Louis sponsored (www.gwb.wustl.edu/cmhsr/measure/categoryu.html).

Advantages and Disadvantages of Standardized Measures

Standardized measures have a number of distinct advantages, most of which are obvious to the researcher. First, they are easily accessible and ready to use, often with little or no modification. Second, they typically have been empirically validated and thus have a certain amount of face validity and psychometric legitimacy. Third, they allow for ease of replicability. These attributes result in what is considered a scientifically sound or rigorous means of measurement. Consistent use of identical measures allows exploration of areas of convergence and divergence without endless and seemingly counterproductive discussions of measurement error, construct validity, etc. This is particularly important in scientific discourse where the dominant logic is incremental improvement, and one study is valued to the extent it builds on a previous one.

These standardized off-the-shelf measures also have some disadvantages that are paradoxically a direct result of their positive attributes. The very availability of the measures often means that they may be utilized without regard to relevance to either the setting or group which they are being applied. The incentives for rapid publication that permeate many fields can lead the researcher to suspend judgment about the substantive appropriateness of the measure used and focus more on its statistical robustness, thus potentially giving priority to form over content. None of this may be intended by the researcher, but may happen as a consequence of easy availability and pressure to obtain funding and deliver scientific results. Furthermore, utilization of standardized measures may actually diminish the chances of new substantive insights. By encouraging replicable research, exclusive use of standardized measures may inadvertently push the investigator away from the possibility of framing the research in fresh, innovative directions. Thus, we believe that although off-the-shelf standardized organizational measures can be a real asset for researchers interested in improvement in the delivery of mental health services, they must be used with care to insure that they are substantively appropriate and contextually meaningful.

Using qualitative investigative or process-type methods may appear to be taking a wrecking ball to the edifice of scientifically rigorous academic intervention research. In some academic circles, the premium placed on being “right” appears so high that there is little room for speculation and imagination. It would be a mistake for academic mental health researchers to be so focused on technique that they miss key variables and relationships related to effective implementation of new treatments in mental health. The process of implementation is rarely linear, and understanding it involves more than administering a survey pre- and post-implementation interventions. Organizational measures that are included in research designed to examine implementation should not just be taken off-the-shelf and plugged into a study, but carefully screened for their appropriateness and relevance. The easy availability of a “validated” instrument may lead to inappropriate use without full consideration of its suitability for mental health settings. Assessment of the appropriateness of instruments in the context in which they will be used should be a necessary precursor to use. This involves more for the researcher than “plug and play.”

Organizational Attributes and their Proposed Role in Mental Health Implementation

There are many influences on the implementation of new technology and practices in provider organizations in health care (for review, see Rye and Kimberly 2007). Theoretically and substantively, what makes implementation of new technologies and practices sustainable includes distribution of control and authority within an organization, behavior of leaders as well as engagement of top-level management, incentives for change, interplay of interests within an organization, and the role of champions (Rye and Kimberly 2007). So far, the relevance of these factors for sustained use of new practices and technologies in mental health services organizations is not well understood. As Rosenheck (2001) noted, the role of organizational variables in facilitating and inhibiting the effective delivery of mental health services has not been examined extensively, and considerably more research in this area is needed. However, it is exciting to see the recent role of organizational variables examined in the dissemination and implementation of evidenced-based practices in children’s mental health services (Glisson 2002; Glisson and James 2002; Hemmelgarn et al. 2006).

Organizational attributes that may influence implementation of innovations in mental health settings are likely to be similar to those that have been found to be important in other settings; e.g., formalization, centralization, managerial tenure and attitudes toward change, professionalism, and internal communication. Our experience suggests that when mental health organizations are overly layered or fractured in terms of structure, department or hierarchy, that is, when they are overly “bureaucratic,” implementation will at best be extremely challenging and at worst impossible. Fragmented groups may be competing for resources and patients and therefore may be less amenable to adopting an innovation such as integrated forms of treatment [e.g., despite high rates of comorbidty between PTSD and Substance Use Disorders (SUD) in veteran populations, VA PTSD and SUD clinicians are rarely trained in treatments for disorders]. Reflecting the previous discussion of readiness for change, we might hypothesize that if there were greater integration or alignment among practitioners, then implementation of consistent and comprehensive treatment would be more readily achieved.

Some of the most highly visible and innovative work on the effect of organizational variables on change in health care organizations has come from a group of researchers in UK. In the British health services literature, there are numerous qualitative interview-based assessments. Although Pettigrew and colleagues have not constructed a measure per se, they are building an empirical knowledge for the issue by exploring readiness and capacity for action and delivery in a number of ways.

In their study of Britain’s National Health Service, Pettigrew and his colleagues utilized a sample of health authorities in matched pairs and controlled for the content of change they were attempting and then explored what features of context and action helped to deliver the different rates of change (Pettigrew et al. 1992). The argument for conceptualizing readiness for action and actual action was eventually put together in the final chapter of the book called Receptive Contexts for Change. In summary, the authors conclude that readiness for change is necessary, although in and of itself, insufficient to initiate change and delivery is far more important in the change process.

Pettigrew and colleagues’ latest work also explores readiness for change, but from a much different theoretical angle in an international study of the links between innovative forms of organizing and company performance (Pettigrew et al. 2003). Using the Milgrom and Roberts work on the new economics of complementarities as a guide, the investigators concluded that the higher performing organizational forms made innovations in mutually reinforcing sets, while the lesser performers made singular innovations or subsets of some theoretically possible set. This work is confirmed by econometric analysis. Additionally, they also explored in comparative longitudinal cases why and how the higher performers put together these mutually reinforcing sets of innovations and connected them to improved performance.

Pettigrew and other business researchers and practitioners use “readiness for change” differently than psychologists. The concepts are only distant relatives. Both groups begin with an intuitively appealing language about people and organizations being willing to consider and implement change in their lives. Both approaches assume that one corresponds one’s own interest to the state of the individual/organization while ranging from pre-contemplative to fully committed. One difference is that psychological investigators may be overly impressed with rather straight-forward pencil and paper assessments of readiness for change. The actual literature concerning applications to health behavior suggests weak effects. As critics have noted, there are some bases for assuming that one cannot appeal to people to make greater effort or more of a commitment than they are ready to provide. However, as the organizational investigators suggest, shifts in readiness for change can sometimes be non-linear. Also, in addition to stages of change and readiness unexpected events and variables influence the process of change. Learning through observation, motivation through social influence, occurrence of positive reinforcement that increases confidence, and individuals reaching the limits of their frustration tolerance with their current circumstances, for example, may influence the process of change, and all are poorly modeled in a rigid readiness to change stage theory. Psychological investigator proponents of stage theory are beginning to see their limits, but it is difficult to accommodate the full criticism and maintain their unique perspectives. The main limits appear to be that the theory of readiness for change is typically one of orderly progression, is highly individualistic and does not accommodate social process (like contagion) or context.

Conclusion

Designing Process Research for Organizational Systems in Mental Health

As new practices and technologies are introduced into any system, there is a normal and natural interest in assessing their impact and the extent to which the impact was consistent with the expectations of the advocates of change. The ultimate success of a new practice or technology hinges as much on how well it is implemented as on its intrinsic performance characteristics. However, examining the impact of new practices and technologies in any organization illuminates the economic, political and psychological agendas their introduction triggers. The potential gains or losses for various stakeholders can be substantial; therefore research on the development and application of these new practices and technologies is not neutral. It will be used by detractors and advocates alike, and for this reason it must be as relevant as it is rigorous. Particularly in the case of mental health organizations, where the potential loss associated with ineffective or inefficacious innovation includes detrimental effects to the emotional and cognitive well-being of the consumer, relevance should not be sacrificed on the altar of rigor.

In an effort to determine the criteria that should be used to judge whether a given piece of research on implementation is well-constructed, further tension between rigor and relevance is encountered. This tension is present to a greater or lesser extent in every field of inquiry, but is particularly salient in fields that border on the “hard” sciences. It is precisely those fields that are coupled to and are judged by those squarely in the hard sciences that feel the tension most acutely.

To balance rigor with relevance, to insure that the research carried out is substantively meaningful, and to avoid becoming overly influenced by “hard science” thinking, we would argue that researchers interested in understanding the factors that influence the implementation of new practices and technologies in mental health should use process research designs. As we have argued elsewhere (Kimberly et al. 1972), process research can be defined as a systematic attempt to gather data in a particular organizational setting that, on the one hand, can be used both to inform existing theory and to track the impact of change while being used, on the other, to provide feedback to participants regarding these changes so that they can make real-time adjustments should they feel them to be necessary.

As such, process research falls somewhere between “pure” or “basic” research and “action” research in terms of the role of the researcher. In the case of the former, the researcher attempts to remain uninvolved with the phenomena being studied in order to maximize “objectivity” and remove possible sources of bias; in the case of the latter, the researcher deliberately intervenes in the setting being studied as a change agent. The role of the researcher in “process” research is intended to bridge these two extremes and to take advantage of the strengths of each.

In brief, designs for process research should be built around the following five criteria: (1) Flexibility of procedures, (2) longitudinal perspective, (3) focus on behavior, (4) focus on the systemic nature of change, and (5) provision of feedback to key stakeholders (Kimberly et al. 1972).

Flexibility of Procedures

Techniques used to monitor the implementation of new practices and technologies in mental health service organizations should be context-sensitive, that is, they should reflect the unique characteristics of work and professional organization in mental health. As we argued above, although techniques developed in other contexts may be useful and may yield data that are relevant, there is often a problem of “fit,” particularly when survey instruments are being used. A second dimension of flexibility is the use of multiple methods. Where resources permit, use of multiple techniques is highly desirable both for purposes of triangulation and for enhancing the richness of the data. Finally, procedures need to be situationally adaptable. Very rarely is it possible to develop a design that perfectly anticipates the uncertainties that inevitably accompany the introduction of new practices and technologies into an organizational setting. In process research, a premium is placed on adapting techniques to meet situational demands. This approach may seem to violate certain cherished canons of “scientific” research to some investigators. However, consider the alternative, the case of a study of leadership and job satisfaction that was halted because a strike took place during one phase of the research, “contaminating” the design. Should the strike be regarded as a source of unwanted, contaminating error variance or as a source of data?

Longitudinal Perspective

Process research, by definition, involves diachronic as opposed to synchronic design. Methods used should be based on the premise that it is primarily through a careful monitoring of processes over time that questions of either a theoretical or a more policy-oriented nature associated with the implementation of new practices and technologies can be answered. Researchers need to have a theory-driven view of the importance of time for the phenomena they are studying (Kimberly 1976) and to construct research designs based on these theoretical considerations.

Focus on Behavior

Data collected by process research should be anchored, insofar as possible, in behavior. In particular, techniques should be used that reflect behavioral adaptation to formal constraints, or what sociologists call the “unintended consequences” of formal structure. This focus is particularly important in situations where new practices or technologies are being introduced because of the tensions that inevitably arise between and among stakeholders regarding the preservation of old certainties and the need for new patterns if implementation is to be successful.

Focus on the Systemic Nature of Change

Process research should be based on recognition of the nature of the interdependencies among the various stakeholders who are a part of the setting. Techniques used, therefore, should be designed to monitor these interdependencies as carefully as possible and to reflect changes in the relationships among the stakeholders over time as a consequence of the introduction of the new practices or technologies.

Provision of Feedback to Stakeholders

One of the most useful (and potentially controversial) characteristics of process research is the provision of data about their own behavior and the behavior of others in the setting to the stakeholders. Advocates of the “pure” research model might argue that such an activity may alter the very nature of the phenomenon being studied. However, we contend that this model is inappropriate when considering the introduction of new practices or technologies in mental health service organizations. Key stakeholders in the organizational systems in which these innovations are being introduced need to debate and reflect on the consequences of their introduction in an informed fashion. Well-designed process research can provide a foundation for these debates and reflections.

The general process research approach has been used, with local variation and adaptation, since the mid-1970s. Two of the best known champions of process research in the management area are Andrew Van de Ven at the University of Minnesota and Andrew Pettigrew at the University of Bath. Some of Pettigrew’s work has been highlighted earlier in this paper. Since 1983, Van de Ven and colleagues have been engaged in longitudinal field studies that monitor the development of a wide array of innovations from concept to implementation. For more information on this large body of programmatic research, see Van de Ven et al. (1989), Van de Ven et al. (1999), and Poole et al. (2000).

Process studies of change and implementation over-whelmingly show that, across a variety of settings and contexts, the implementation trajectory is non-linear, complicated, and unpredictable. It is likely similar in mental health settings. Measures of organizational influences in mental health settings need to reflect this reality. To be effective and to speak to real issues, approaches to research in this context should not be shackled by the imagery of the randomized controlled trial (RCT). Appropriate as the RCT is for some problems, the RCT mind-set is less appropriate for others, and investigators as well as clinicians should be working together to distinguish between appropriate and inappropriate circumstances. If we are able to do this effectively, research on the implementation of new practices and technologies in mental health will help uncover reliable paths for service improvement for users of the system.