Multimethod Evaluation of Information and Communication Technologies in Health in the Context of Wicked Problems and Sociotechnical Theory

Abstract

Objective

Few research designs look at the deep structure of complex social systems. We report the design and implementation of a multimethod evaluation model to assess the impact of computerized order entry systems on both the technical and social systems within a health care organization.

Design

We designed a multimethod evaluation model informed by sociotechnical theory and an appreciation of the nature of wicked problems. We mobilized this model to assess the impact of an electronic medication management system via a three-year program of research at a major academic hospital.

Measurements

Model components include measurements relating to three dimensions of system impact: safety and quality, organizational culture, and work and communication patterns.

Results

Application of the evaluation model required the development and testing of purpose-built measurement tools such as software to collect multidimensional work measurement data. The model applied established research methods including medication error audits and social network analysis. Design features of these tools and techniques are described, along with the practical challenges of their implementation. The distinctiveness of doing research within a unique paradigm of complex systems, explicating the wickedness and the dimensionality of sociotechnical theory, is articulated.

Conclusion

Designing an effective evaluation model requires a deep understanding of the nature and complexity of the problems that information technology interventions in health care are trying to address. Adopting a sociotechnical perspective for model generation improves our ability to develop evaluation models that are adaptive and sensitive to the characteristics of wicked problems and provides a strong theoretical basis from which to analyze and interpret findings.

Introduction

In 2004 we reported on a program of research that involved implementation of a comprehensive evaluation framework to measure the impact of computerized order entry systems on major academic medical institutions. ¹ This evaluation framework was informed by sociotechnical theory. We added a complexity dimension, thereby extending the conceptualization of sociotechnical theory in its application to health care organizations, where a large component of work is nonlinear ² and where organizational and professional cultures are characterized by factors inconsistent with the central goals of traditional sociotechnical system design. ³ A multimethod approach was adopted whereby a range of data collection methods was used to measure three dimensions of the study organization: safety and quality, organizational culture, and work and communication patterns. These dimensions reflect both the areas on which clinical information systems are expected to impact positively, and those that impact on the successful use and integration of systems into clinical practice. This article reports substantial progress with this program of research, delving into the deep structure of one of the study sites. We focus on the measurement tools that have been developed as part of this work, and some of the practical challenges in undertaking multimethod evaluations of health informatics interventions.

Wicked Problems and Information and Communication Technologies

Information and communication technologies are being targeted as a possible way of addressing what is in essence a “wicked problem”, ⁴ namely, how do we deliver safe and sustainable health care systems in the future through information technology (IT)–enabling responses? Wicked problems are those that are “ill-defined, ambiguous and associated with strong moral, political and professional issues.” ⁵ According to Ritchey, ⁵ there is little consensus about the definition of wicked problems or how to resolve them. Wicked problems are dynamic with multiple sets of complex, interacting issues that evolve in an emergent social context. A significant challenge in addressing such dilemmas is that it is difficult to know when an end point has been reached. Solutions will be socially constructed depending on the observer’s standpoint and predilections. Fixes to wicked problems will often contain within them other wicked problems, and attempted solutions invariably generate new wicked problems. Wicked problems are characteristic of complex adaptive systems. ⁶ For those dealing with the implementation and evaluation of the impact of information and communication technology (ICT) in health, such descriptions help explain the magnitude and complexity of their task.

How Does Sociotechnical Theory Help in Understanding How to Investigate the Impact of Information and Communication Technologies in Health Care Organizations?

The earliest studies that resulted in the formulation of the sociotechnical perspective are still among the most illuminating for those seeking to understand sociotechnical theory. Trist and Bamforth ⁷ studied coal miners in Britain in the 1940s and 1950s. They investigated the impact of a new form of mining “shortwall” versus the traditional “longwall” mining. In the shortwall mine, miners worked closely together in teams and became multiskilled. In the longwall mines, workers performed single, limited tasks. Workers were physically isolated from others in their shifts and also from those on the coal face. Problems created on other shifts could make work extremely dangerous and difficult, but the longwall miners had no way of influencing the behavior of others. As a result, miners felt vulnerable and developed elaborate strategies, including bribery and absenteeism, to cope with this work environment. ⁸ In comparison, the shortwall miners who had formed relatively autonomous work groups, in which workers interchanged roles and shifts and experienced minimal supervision, were highly successful in terms of improved productivity and reduced absenteeism. ⁹

This research led to the formulation of a radical theory at the time that espoused the need to consider and optimize both the technical work processes and the social systems operating within the work environment to improve organizational performance. For these early theorists, the social system comprised elements such as the workers, e.g., employees, managers, and contractors; their practices, e.g., their behaviors, activities, and skills; their mental constructs, e.g., their attitudes and beliefs; and their interactions, e.g., their relationships, coalitions, and political milieu. The technical system of an organization consisted of the artefacts, e.g., the tools, devices, materials, and techniques by which work is performed. According to sociotechnical theorists, by achieving the right balance and designing processes and policies that recognize and address the interdependency between the social and technical subsystems of an organization, the performance of an organization can be optimized.

Fast forward to today’s health system, and sociotechnical organizational design theory is particularly relevant as we grapple with the wicked problem of how to deliver safe and sustainable health systems using IT. Sociotechnical theory would predict that introduction of a new tool (i.e., a sophisticated clinical information system with its accompanying new work processes) into a complex organization such as a hospital is unlikely to bring immediate benefits without careful investigation of, and responses to, the impact on other technical and social subsystem elements.

Many studies of health IT system failures confirm such predictions. ^10,11 As Fox ⁹ describes it, the goal from a sociotechnical perspective is to integrate “… the social requirements of people doing the work with the technical requirements needed to keep the work systems viable with regard to their environments” (p 92). Each subsystem needs to be considered independently and interdependently because optimization of one may have a negative impact on the other. Hence proponents argue the need for dual focus and joint optimization. ⁹

Added to this is the challenge that the problems to be addressed by the introduction of IT into health care organizations are in themselves wicked problems. They include improving the quality and safety of health care and delivering more efficient care, and while often presented as straightforward and simple, are in fact resistant to solutions, intractable, political, and multidimensional. They are prone to be at best indefinite and contradictory, and at worst impossible to resolve satisfactorily. The decades of strategies for improving the safety and quality of health care are a testament to the wickedness of this problem.

We argue that although recognizing that the problems being targeted for solution by IT in health are wicked, sociotechnical theory provides a way forward. The wicked problem paradigm informs us that in addressing one aspect of a problem new ones will be created that also need detection and attention. Sociotechnical theory in particular provides a basis from which to identify methods for studying and measuring aspects of an organization’s social and technical systems evident in these problems. Thus the aim of evaluation is both to deliver robust evidence of the impact of IT on organizations, and also to develop an evaluation model that is adaptive and sensitive to wicked characteristics.

A central tenet of sociotechnical theory is that context will influence individuals’ behaviors within an organization, and that technology is one of the strongest elements of context that affects behavior. ⁸ Organizations today face increasingly unpredictable, continuous, and often chaotic change. ^12,13 To survive and prosper, organizations must be flexible and responsive to their contexts, requiring high levels of system redesign. ⁸ In many newer models of work organization, such as those found in modern hospitals, the systems are complex and nonlinear, problems are often poorly structured, and “… the inputs and outputs, or the nature of the problem itself and its solutions, are erratic” (p 203). ⁸ Examples of nonlinear work in the health system include the process of diagnosing a patient, securing a bed for an emergency room patient, and determining the allocation of budget across multiple departments or institutions. Pava ² argues that in these characteristically complex, nonlinear organizations, deliberations (sequences of exchange and communication) about specific problems are an important element of the technical subsystem. The aim of these deliberations is to deal with “… complex or uncertain issues that cannot be solved with a specified rule or algorithm.” ² The social system includes the coalitions or groups of people engaged in these deliberations. Such groups may be defined by shifting professional boundaries or informal organizational networks. ¹⁴ Recognition of hospitals as examples of these complex organizations highlights the importance of studying how such deliberations occur within hospitals and how information and communication technologies may interfere or inhibit or effectively support such deliberations. ¹⁵

Implementing an Evaluation Model for an Electronic Medication Management System

Our aim in this program of research was to conceptualize an evaluation model reflecting the need for health care organizations to understand (and thus have the capacity to work on) the impact of the introduction of computerized physician order entry (CPOE) systems on both the technical and the social systems within organizations. To achieve this, we adopted a multimethod approach to data collection and analysis. Where tools did not exist, we set out to devise and test new ones. Before the development of our model, we undertook an extensive review ^1,16 that identified key limitations of previous attempts to evaluate CPOE systems. Evidence gained from this review led to the formulation of a set of evaluation principles and subsequently to the design of the multimethod evaluation model. This model incorporates a prospective study design with measurements collected before and after the implementation of the CPOE systems and the application of multimethods that allow the triangulation of these indicator data, thereby respecting the characteristics of wickedness and incorporating concepts from sociotechnical theory.

Setting

The research reported here was undertaken at a major academic medical center located in Sydney. This public hospital has 400 beds and provides general medical and surgical services. The hospital has used mandatory computerized pathology ordering systems for a decade and more recently added diet and consultation orders via the Cerner Powerchart system. In 2007 the hospital will implement the Cerner medication management system, which incorporates both electronic prescribing functions and an electronic medication administration record. The hospital intends to initiate a basic level of decision support in the first instance, and it is anticipated that this will be phased in slowly. The research team members were all independent from the hospital staff. Ethics approvals were obtained from the study site and the researchers’ academic institutions.

▶ illustrates the evaluation model dimensions and the core substudies undertaken. Each major substudy within these dimensions is briefly described, with a focus on measurement tools and datasets produced. The triangulated, multifactorial nature of the design is a core feature. A requirement of all substudies was the need to be equally effective in measuring and collecting data before and after system implementation. The overall design feature of the evaluation model was a controlled before-and-after study whereby measures were taken before and after implementation. Not all wards will receive the intervention initially, thus allowing them to act as controls. By including control wards in our study we can make a more reliable judgment about the role of the intervention on our measurement indicators.

Figure 1.

Conceptual evaluation model illustrating the interconnected dimensions and substudies undertaken to measure these dimensions of system impact on a health care organization.

Safety and Quality Indicators

A central argument for the implementation of CPOE systems is their significant positive impact on health care safety, principally through the reduction in adverse events. Because the hospital was to introduce an electronic medication management system, a key element of the framework was to obtain accurate measurement of medication errors before and after system introduction. Although great weight is often given to the potential of electronic prescribing systems to reduce adverse drug events, there are few empirical data showing the effectiveness of commercial electronic prescribing systems to produce these outcomes. Existing evidence relates almost entirely to home-grown systems. ¹⁷

Prescribing and medication administration errors are the largest categories of medication errors. ^18–20 The voluntary medication error reporting systems used by most hospitals were deemed insufficient as a way of measuring error rates because they have been shown to underestimate error rates significantly. ²¹ Thus two independent studies were initiated.

Controlled Time Series Study of Prescribing Errors

We have commenced a prescribing error study using a controlled time series design (▶). Using prospective error surveillance measures, all patients’ medication charts are reviewed daily by a hospital pharmacist across four general medical/surgical wards to record errors. Data were collected for eight months before system implementation to detect prescribing errors. These data are supplemented with data from the voluntary incident database, pharmacy intervention logs, and verbal reports by nursing and medical staff. Similar approaches have been used in the evaluation of home-grown electronic prescribing systems in the United States ^22–24 but have not been applied to the evaluation of commercial systems, which by their very nature are more generic. Commercial systems are thus potentially more difficult to integrate into health care organizations, and hence may not be as effective in reducing error rates.

Figure 2.

Schematic diagram of the design and outcomes of the controlled time series study aimed at evaluating the impact of electronic prescribing systems (e-PS) on prescribing error rates.

In the post–system implementation period, the electronic medication charts will be reviewed using the same approach as in the preintervention stage. A focus also will be the detection of new types of errors related to both system design features and automation bias. ^25,26 This will include undertaking a computer log analysis. Further postintervention error analysis will be conducted drawing on techniques used elsewhere to examine the underlying causes of clinical incidents. ^27,28 Where it is unclear whether a prescribing error is related to use of the electronic prescribing system, a researcher will interview the prescriber within 48 hours of detecting the error. A brief semistructured interview to elicit system factors contributing to the error and the role of the electronic prescribing system will be conducted.

Observational Study of Medication Administration and Preparation

One third of medication errors that cause patient harm arise from medication administration errors (MAEs). ²⁹ Research regarding rates of MAEs is limited and has predominantly relied on voluntary incident reports or observational studies. ^30,31 The quality of evidence from these studies is poor due to the study designs and MAE detection techniques used. Traditionally, observational studies of medication administration processes have used paper-based data collection. This is time intensive and creates distraction from actual observation, and the amount of information that can be recorded is limited. Technological advances allow us to develop software able to capture multiple layers of information in an efficient manner. In designing this substudy, our objective was to develop and test a personal digital assistant (PDA, a handheld computer) data collection device for use in observational studies to monitor MAEs within hospitals.

The PDA data collection tool (▶) was designed following identification of data elements needed to calculate MAE rates. The program was developed using Microsoft Visual Studio C# and was implemented on the PDA running the Microsoft CE platform. Using ActiveSync enabled the use of SQL and the synchronization of multiple PDA databases with one central database on a PC. Through an iterative process the PDA device was tested in the field to examine usability issues and study participant acceptance and to calculate observers’ inter-rater reliability scores. The final version was tested during a two-month prospective observational study of nurses as they administered medications. The study had two stages. In stage one, trained researchers followed and observed nurses on the wards as they prepared and administered medications. Information was recorded using the PDA about procedures undertaken by the nurse (for example, whether the patient’s identification was checked or medication label read) and medications administered (for example, the medication type, dose, and route). In stage two, the observational data were downloaded from the database and details of each administration were printed. A different researcher then compared the observational data against each patient’s medication chart to identify MAEs. Data collection occurred across four hospital wards for a period of eight months before system implementation.

Figure 3.

Personal digital assistant (PDA) medication administration data collection tool.

The PDA tool proved to have high levels of usability by the researchers. Training in the use of the PDA and data definitions was important to gain acceptable levels of data collection accuracy and speed of recording. On average this process took 15 to 20 hours. Inter-rater reliability scores exceeded 85% after several training sessions in the field. Observers are required to be physically close to study participants to record the necessary data, thus establishing good rapport and trust with participants is essential. ▶ shows the close proximity of the observer to the clinician needed to record drug administration details. The PDA reliably captured a greater number of variables than is possible with paper-based methods; for example, information about interruptions to nurses’ work along with detailed medication and procedural information. Much secondary data entry was avoided by direct PDA data entry. Data on over 3,000 medication administrations have been collected in the preimplementation phase. The observational techniques and tools were designed to ensure that these would be equally applicable to both study stages to allow valid comparisons to be made. This has been confirmed by undertaking a trial of the observational methods and PDA within a second hospital that has already implemented an electronic medication administration record. This trial showed that the research methods used in the preimplemention stage were equally effective in the postimplementation stage.

Figure 4.

Researcher with a personal digital assistant (PDA) collecting data during the observational medication study.

Safety Climate Survey

It is well accepted that errors predominantly occur as a result of system failures and that organizational, managerial, human, and technical factors all may play a role. ³² Thus there has been growing interest in measuring the culture of safety within health care organizations. The Safety Attitudes Questionnaire (SAQ) ³³ is a reliable and validated tool for measuring health providers’ beliefs and attitudes about the safety culture within their organization. The tool measures participants’ attitudes about six patient safety-related domains and permits multilevel comparisons to be made, for example, within and across clinical units and professional groups. Constructs (factors) that are measured in the survey include: team work climate, safety climate, perceptions of management, job satisfaction, working conditions, and stress recognition. The factor scores generated by the survey have been associated with important clinical outcomes, including medication errors. ³⁴ The Safety Climate survey was administered to all clinical, scientific, and management staff in the study site. To assist completion, medical staff were given the option of completing an online version of the survey, and all nursing and allied health staff (who have less ready access to e-mail at work) were provided with paper-based surveys to complete. Over 450 surveys were completed before system implementation. We will use the same survey protocol in the postimplementation stage to examine changes in the safety culture of the organization. The purpose of applying this tool is to obtain a measure of the safety culture and the extent to which this changes after system implementation.

Organizational Culture

To identify and explore the central issues of concern to hospital staff within the organization before the implementation of the electronic medication management system, a series of interviews and focus groups was held. Participants included: (1) senior staff predominantly in management (n = 10), (2) senior clinical management staff (n = 21), and (3) predominantly clinical staff (n = 19). Purposive sampling techniques were used to obtain a broad representative cross section. Systematic concurrent analysis of the data was undertaken by the team of researchers with the aid of Nvivo software to explore major concerns. This analysis showed several key areas that were of concern to all three groups. This work was supplemented by video observational studies of clinical staff as they undertook hospital ward rounds to gain an understanding of the ways in which paper records, of which medication charts are an integral part, are used in everyday practice. This involved a researcher with a handheld video recorder following a clinical team on a standard ward round while the team members were reviewing patient care. The video data showed that staff access different parts of the record at the same time, as well as performing different tasks with the record such as reviewing patient notes, documenting care decisions, ordering medications, reviewing results, and reading aloud information to share with colleagues or the patient. Ways in which these behaviors will, by necessity, change will be a focus in the postimplementation stage. Questions of interest include how ward rounds and the educational component of these change when a central part of the paper record is computerized. We are developing approaches ³⁵ to analyzing these video data to identify themes around the communication and exchange of information during ward rounds that will allow us to make comparisons in these processes in the postimplementation stage.

Social Network Analysis

Much attention has been placed on the role of clinical teams within health care, and several studies have now shown the association between well-functioning teams and positive patient outcomes. ^36,37 Good relationships between clinicians also have been found to be important in the diffusion of innovation among clinicians of different professions ³⁸ and within the same profession. ³⁹ Communication between clinicians is complex and has not been studied to any great extent.

Social network analysis provides a useful approach by which to examine and measure the communication patterns and interactions of professional groups in hospital wards. It has shown the central role that specific individuals can play in the exchange of information within their networks. ^40–43

A small number of previous studies have revealed inconsistent and sometimes unexpected ways in which electronic medication management systems have impacted the flow of information and the communication patterns within hospitals. ^44,45 We undertook a social network analysis across three hospital units to gain an understanding of current information flows regarding medication-related communications. All clinical and administrative staff working on each of the study wards were given a social network survey to complete that included questions such as, “How often do you seek advice from (name of staff member) about medication decisions/tasks?” and “How often do you socialize with this person?” ▶ provides an illustration of one of the resulting network diagrams.

Figure 5.

Network diagram of a hospital ward. This diagram represents the extent to which individuals reported seeking help from a colleague to solve a work-related problem at least several times per month. Each staff member in the ward is represented by a node in the diagram. The lines between the nodes are the interactions between the staff members. The arrows show the direction of the asking for help. The shape of each node indicates the profession of each staff member. Doctors are circles, nurses are squares, allied health professionals are triangles, and the ward clerk is shown as an upside-down triangle. RN = registered nurse; EN = enrolled nurse, which is equivalent to a nurse’s aide; TEN = trainee enrolled nurse.

A whole-network approach was used to capture the interactions that occur between staff members of the ward when carrying out their work. This approach allowed the elicitation of patterns of connections that occur between staff. ⁴⁶ The results show great similarities in the communication patterns of staff on different hospital wards in terms of the frequency and direction of communication between ward staff. Results show that on average there is a low frequency of interaction between staff members in terms of seeking advice regarding medication decisions and tasks ⁴⁷ and asking for help to solve work-related problems, and in terms of socializing. The interaction that does occur is mainly between members of the same professional group: nurses interacting with nurses, and doctors interacting with doctors. This is illustrated in the network diagram (▶), where nurses (squares) are grouped together on the right-hand side of the network and doctors (circles) are located away from most nurses, in the bottom left of the network diagram. However, there are key individuals from various professions who play vital roles in the ward. For example, the pharmacist currently plays a key role in providing medication advice to many members of the ward. Junior doctors are important in bridging or brokering between the senior doctors and the nurses, ⁴⁶ as shown by ▶, where junior doctors are located between the nurses and the senior doctors. The ward clerk takes an important liaison role in providing help to others in the ward to solve work-related problems, as do many of the senior nurses.

Social network analysis provides a systematic approach for measuring complex interactions that occur between health care professional groups and individuals. In the postimplementation period, this study will be repeated using the same methods to identify how patterns of communication and information seeking relating to medication advice change.

Work Practice and Communication Patterns

Studies of communication practices among doctors have been the focus of a limited number of observational studies (e.g., Coiera et al. ⁴⁸ and Chisholm et al. ⁴⁹ ) that have applied observational techniques matched with paper recording and audiotaping. These studies have shed light on the complexity of clinician information exchange and the predominance of face-to-face communication in health care. Evidence regarding how electronic medication management systems influence clinical work patterns and support efficient practices is limited. Studies that have focused on measuring the changes in work and communication after the introduction of ICT have in general adopted either work sampling, or time and motion methods. ⁵⁰ Traditional paper-based data collection methods used in such studies have been limited in their ability to capture clinical work complexity and communication patterns. Results from a small number of previous studies ^51,52 suggested that the application of PDAs may be useful for this task. Our objective was to build on previous work in this area and to design and test an electronic data collection tool for work measurement studies that would allow efficient, accurate, and reliable data collection and capture work complexity within our hospital site before and after system introduction. We developed software on a PDA that applied a multidimensional work task classification that captures details of nurses’ and doctors’ work (▶); what task, with whom, and with what; multitasking; interruptions; and event duration. A particular advantage of the data items collected is the ability to identify not only what the clinician was doing, but also who else was involved in the task, and if and when the task was interrupted or undertaken in parallel (multitasking) with another task. Although this level of detail was possible to derive from observational studies using audiorecordings of clinicians, ⁴⁸ the data transcription and coding procedures were laborious and the inter-rater reliability of data collection was more difficult to measure. The PDA tool provides the opportunity to collect larger volumes of data with greater accuracy, but without the detailed context provided by audiorecordings. This tool has provided us with rich data about clinical work patterns on the study wards. A similar approach to that adopted in the technical development of the medication administration error tool described above was used. The design of the tool followed extensive pilot work and the conduct of comparison studies of paper-based work sampling studies. ⁵³

Figure 6.

Doctors’ work measurement personal digital assistant (PDA) data collection tool.

Nurses on four wards have participated in over 250 hours of direct observation, ⁵⁴ and in a separate substudy doctors were observed for a total of 151 hours before system implementation. In each observational study, trained observers followed individual clinicians for one-hour blocks as they performed their normal work tasks, and recorded all work tasks performed using the PDA, which automatically time-stamped activities. The sampling frames used were designed to gain a representative view of work across the day by different categories of clinical staff. Inter-rater reliability of observers was tested by two researchers independently yet concurrently observing the same clinician until inter-rater reliability reached 85% or above, before formal data collection could commence. An additional study observing pharmacy staff is planned.

Results from the work measurement studies have been valuable in three ways. Firstly they have provided a profile of how doctors and nurses distribute their time on the wards. This has revealed, for example, that the greatest proportion of doctors’ time is spent in professional communication (33%) whereas for nurses it is spent in direct care (24%), followed by professional communication (22%). Secondly, comparisons between wards have shown, for example, that nurses’ patterns of work task time did not vary substantially across hospital wards. ⁵⁵ Thirdly, interestingly, we found that time spent on some work tasks, predicted to decrease after the introduction of the electronic medication management system, such as time nurses spent clarifying a drug order or time doctors spent searching for a missing medical record, actually consumed very small (<1%) proportions of time. ⁵⁴ Thus even if there were substantial savings in time for these tasks after system implementation, it would be unlikely to make a significant impact on overall patterns of work–time distribution. What we have learned from this experience is the value of the pre–system implementation data in terms of allowing an organization to develop a more evidence-based approach to identifying the areas of health care delivery more likely to reap rewards and thus those areas where post–system implementation evaluation should focus. These data are also of value in creating realistic expectations of staff in terms of the likely benefits of CPOE for their patterns of work.

Evaluation Model Implementation Challenges

The implementation of this research program has presented several major challenges. The central practical one is the slippage of system implementation timelines. The research program commenced in 2003 after receipt of a large government-funded research grant. At that time, system implementation for this study site was planned to occur in 2004-2005. For numerous reasons, which include changes in the administrative structure of the health regions, difficulties the hospital encountered in negotiating government funding to support system implementation, and delays in achieving system upgrades from the commercial vendor, the planned system implementation has now been scheduled to occur in mid-2007. Such a situation will be familiar to those involved in large complex system implementations. It highlights a central reason why there are so few long-term studies of health IT undertaken, why most evaluations occur once the system is in place, and why evaluations are most frequently undertaken in-house. ¹⁶ Few independent evaluators have the resources or capacity to allow their research timelines to extend in excess of three years, and few funding bodies will support this. Although government health departments or private health care providers may value the availability of high-quality evidence to inform decision making processes, policy timelines rarely extend to allow such research to be undertaken or supported. This places our ongoing program of research in a unique position in relation to previous CPOE evaluations.

A strength of our model is the system perspective it takes, reflected in the measurement of multiple dimensions of the health care process and outcomes, as opposed to focusing solely on specific processes such as ordering. The model has benefited from the several years that were spent designing and testing the data collection tools and techniques and measuring their reliability and validity. In addition, the tools and methods have been designed to provide flexibility in terms of being applicable to both preimplementation and postimplementation stages, as well as differing clinical specialty areas, and with regard to the wicked nature of the problems under investigation and the sociotechnical environment.

Designing an effective evaluation model requires a deep understanding of the nature and complexity of the problems that IT interventions in health care are trying to address. Unlike most previous evaluations of CPOE, our starting point has been to gain a deep understanding of how health care organizations operate (encompassing both the social and technical systems) before system implementation. Further, we have benefited from the experiences of the early large-scale evaluations of CPOE implementations 22,23,56–58 and have, for instance, incorporated measures for detecting both expected and unexpected outcomes and errors produced as a result of system use that do not feature in many early investigations of CPOE.

A major design feature of the model is the concurrent measurements that are then able to be triangulated. One outcome of this is an ability to validate data obtained from one measure with that from another. For example, the work measurement data confirmed conclusions drawn from the social network analyses. We have identified few previous studies that have attempted to compare multiple, concurrent, qualitative, and quantitative indicators of safety and quality, organizational culture, and work and communication patterns before and after CPOE introduction. Although the multimethod approach to the model is a strength, it also is a weakness. Each of the techniques applied are associated with their own specific potential limitations. For example, direct observation of clinical work to detect medication administration errors may result in an underrepresentation of the true error rate because observed nurses may be more careful in their practice as they are aware they are being watched. Although strategies, such as those we used, to reduce such biases have been shown to be effective in reducing possible influences on data collection, ^53,59,60 researchers need to continue to be vigilant in considering such effects. We have applied our model to one large teaching hospital, and thus the generalizability of the model to other hospitals will need to be tested. We have confirmed the use of specific tools, such as the medication administration observational techniques, in other hospitals with an existing electronic medication administration record system that provides some evidence of the transferability of the model.

In summary, it is the broad range of data sources incorporated in the evaluation model that will allow us to examine the relationships between the social and technical subsystems operating in this hospital and to quantify how, in what ways, and why CPOE systems bring about change. The extent to which the social and technical subsystems of the organization are enhanced and the tradeoffs that are made will become explicit. Importantly, if performance is not enhanced to the level expected, we will have good data to identify strategies by which to design a better fit between the organization’s social and technical systems. The importance of moving away from uni-dimensional evaluation studies and the limitations of traditional positivist scientific methods such as randomized controlled trials have been well argued. ^61,62 Several previous health informatics researchers, drawing on social interactionist, ^61,63,64 actor-network, and sociotechnical theory, ^65,66 have made significant contributions to focusing attention on the impact of organizational issues on ICT integration and use. Our multimethod model builds on this important previous work, adding further theoretical considerations as well as demonstrating in practice large-scale, longitudinal, qualitative and quantitative data collection methods and tools and the ways in which data generated potentially may be triangulated.

Health informatics evaluators need to appreciate that they are researching a wicked sociotechnical problem with no mechanistically deterministic resolution. We are predicating our research strategy on this understanding, and are focused on the dimensions of this problem. It may be, consistent with an acceptance of both the wickedness of the problem being addressed by IT in health and IT implementation itself, along with an understanding of sociotechnical theory from the early days, that what we can provide is new perspectives from which to apprehend the intricacies and complexities of the topic. That would be no small contribution to the hermeneutics of the field.