LOGIC ANALYSIS: TESTING PROGRAM THEORY TO BETTER EVALUATE COMPLEX INTERVENTIONS

Abstract

Evaluating complex interventions requires an understanding of the program’s logic of action. Logic analysis, a specific type of program theory evaluation based on scientific knowledge, can help identify either the critical conditions for achieving desired outcomes or alternative interventions for that purpose. In this article, we outline the principles of logic analysis and its roots. We then illustrate its use with an actual evaluation case. Finally, we discuss the advantages of conducting logic analysis prior to other types of evaluations. This article will provide evaluators with both theoretical and practical information to help them in conceptualizing their evaluations.

INTRODUCTION

Evaluating complex interventions presents a paradox. On one hand, complex interventions are considered to be systems that evolve over time and are subject to perturbations that render prediction impossible (Callaghan, 2008). Yet as evaluators we want evaluation to be useful. We want to improve interventions so their impact can be maximized to improve people’s general well-being. For an evaluation to be useful and have a positive impact on practice, it needs to transmit simple messages (Contandriopoulos, Brousselle, & Kêdoté, 2008). An important lesson from Walker and Kubisch’s evaluation of a complex system of initiatives is that we should avoid conducting complex evaluations of complex interventions:

Do not be lured by the assumption that a comprehensive and complex systems-building initiative should be matched by an equally comprehensive and complex evaluation design.

(Walker & Kubisch, 2008, p. 497)

But complexity is not chaos. Interventions can also evolve along certain trends or patterns. If prediction is not possible, explanation is (Callaghan, 2008).

This article proposes that one path to follow is to design the evaluation to properly integrate the characteristics of these complex interventions in a way that is enlightening for stakeholders. As P. J. Rogers pointed out,

the art of dealing with the complicated and complex real world lies in knowing when to simplify and when, and how, to complicate.

(P. J. Rogers, 2008, p. 30)

Logic analysis is a type of program theory evaluation that uses scientific knowledge to evaluate the validity of the intervention’s theory and identify promising alternatives to achieve the desired effects. It can be used as part of a larger evaluation, but we consider it to be a distinctive type of evaluation because it raises a particular evaluation question about the validity and appropriateness of the intervention. In this article, we first briefly present the principles and roots of logic analysis, then illustrate its application in the specific case of an innovative Health Promoting Hospital project. We conclude with a discussion of the advantages of conducting logic analysis prior to other types of evaluations in complex interventions.

LOGIC ANALYSIS

Definition and Types

Logic analysis is a type of evaluation that allows us to look critically at the program’s theory using available scientific knowledge— either existing scientific evidence or expert knowledge (Brousselle, Contandriopoulos, & Lemire, 2009; Brousselle, Lamothe, Mercier, & Perreault, 2007; Champagne, Brousselle, Contandriopoulos, & Hartz, 2009; Contandriopoulos et al., 2008; Contandriopoulos, Champagne, Denis, & Pineault, 2000; Rossi, Lipsey, & Freeman, 2004). The aim of logic analysis is to identify the best ways to get where we want to go, that is, to achieve the desired effects. Logic analysis will identify (a) the important characteristics the interventions must have to achieve the effects and (b) the critical conditions required to facilitate the implementation and produce the effects.

Logic analysis belongs to the large family of theory-driven evaluations, program theory evaluations, and so on, that have been at the centre of methodological debates over the past 30 years (Bickman, 1987, 1990; Rogers, Petrosino, Huebner, & Hacsi, 2000). Recently several authors have proposed innovative methodologies using intervention theory as a main component of their theory-driven evaluation. First, the realist review approach (Davis, 2005; Greenhalgh, Kristjansson, & Robinson, 2007; Pawson & Tilley, 2005) uncovers context—mechanisms—outcome configurations that shed light on why an intervention should work, for whom, and under what conditions, based on existing writings (Pawson & Tilley, 2005). In his approach to program theory evaluation called “the plausibility check,” Donaldson (2007) identifies the program’s active ingredients and how they lead to desired effects. Intervention theory has recently played a central role in impact evaluations. A case in point is contribution analysis (Mayne, 2001, 2008), which explores “attribution through assessing the contribution a programme is making to observed results. It sets out to verify the theory of change behind a programme and, at the same time, takes into consideration other influencing factors” (Mayne, 2008, p. 1). Intervention theory is also central in prospective impact studies (Kautto & Similä, 2005).

There are two types of logic analysis. The first, direct logic analysis, allows us to see whether the intervention is designed in such a way as to achieve the desired effects (Figure 1). This analysis judges the design of the intervention and its appropriateness, but also provides important insights into how it could be improved. It identifies the crucial characteristics of the intervention and the critical conditions for achieving the effects. In that sense, it simplifies the complexity of the intervention, with the criteria for simplification being the importance of the causal contribution of the characteristics and the contextual conditions in the path toward the effects. Not all elements will have the same importance in achieving the effects, and the analysis of scientific knowledge should help in identifying those that may have a greater importance.

Figure 1.

Direct Logic Analysis

In the second type, reverse logic analysis, the evaluator looks for the best ways to achieve the desired effects (Figure 2). This can be helpful for identifying alternatives that will also produce the intended effects, broadening the array of possible interventions. It will also help identify the critical conditions to successfully implement the alternatives and produce the effects.

Figure 2.

Reverse Logic Analysis

Three Steps

A logic analysis can be conducted in three steps: building the logic model of the intervention, developing the conceptual framework, and evaluating the theory of the intervention (see Brousselle & Champagne, 2011, for a complete description).

Building the logic model

First, the evaluator uses logic modelling to represent the intervention’s program theory and the basis upon which it is supposed to lead to the desired effects. Once the model is built, the evaluator will select a few issues to be explored in detail in the logic analysis. Issues are selected based on stakeholders’ interests, objectives, or implementation difficulties encountered.

Developing the conceptual framework

Next, the evaluator gathers information on the best ways of doing things, either by looking at the intervention’s main components to see if the optimal characteristics and conditions have been assembled to achieve the desired effects (direct logic analysis), or by uncovering alternative ways of achieving them (reverse logic analysis). This step should be based upon existing scientific knowledge, that is, expert advice or research evidence identified through a literature review (Rossi et al., 2004). There may be different fields of knowledge leading to various conceptual frameworks. The evaluator is not constrained to choose one domain of knowledge, but can instead propose a knowledge synthesis, highlighting divergences between domains. Further, the objective here is not to develop a systematic synthesis, which would require a considerably more intensive literature analysis, but rather to clarify stakeholders’ representations using scientific knowledge. Citing foundational and recent scientific work (without necessarily consulting all the literature) or using evidence-based data such as systematic syntheses (or practice guidelines in the clinical field), if available, allows the evaluator to develop a sound representation of the intervention’s theory. Logic analysis can be applied to innovative interventions as well. There may not exist documented writings on the interventions as such, but it is very likely that these innovations’ principles of action have already been studied and documented.

Evaluating program theory

Finally, the evaluator compares the real intervention with the conceptual model that emerged from consulting experts or from the literature analysis. The evaluator will work out this step differently depending on whether a direct or a reverse logic analysis is being conducted.

In direct logic analysis, by comparing the model of the intervention to the conceptual model, the evaluator can identify what characteristics are essential to ensure the intervention successfully follows the causal path toward the intended effects.

If a reverse analysis is done, the evaluator will be able to confirm whether the intervention being evaluated is appropriate (Brousselle & Champagne, 2011; Brousselle et al., 2007). Alternatives would have been identified in the conceptual modelling, allowing the evaluator to identify the best ways to achieve the intended effects. Generally, the appropriateness of using one particular intervention rather than others will depend on contextual characteristics. This step could be conducted deliberatively with stakeholders. However, it is important to stress that reverse logic analysis is mostly a summative exercise, and stakeholders directly involved in the intervention could be reluctant to participate in such an activity. Direct logic analysis is much more formative or developmental.

In both cases, often by relying on experts or the literature, the evaluator will also identify important conditions that will either impede or foster the process through which effects will be produced. Not all of these elements can be manipulated; many belong to settings in which no leverage is available. However, some of these conditions can be targeted as levers to foster implementation and maximize the chance of reaching the objectives.

AN EXAMPLE OF LOGIC ANALYSIS: IMPLEMENTING THE HEALTH PROMOTING HOSPITAL CONCEPT IN A PERINATAL CONTEXT

Background

In 2007, a Montreal hospital embarked on the process of becoming a Health Promoting Hospital by joining the Réseau montréalais des CSSS et des hôpitaux promoteurs de santé, member of the World Health Organization (WHO) International Health Promoting Hospitals (HPH) Network that was initiated in 1988 subsequent to the Ottawa Charter for Health Promotion in 1986 (WHO, 1986). The hospital was accepted as a member of the HPH network based on its having created a Health Promotion Department, also in 2007. That department began implementing the HPH concept through several strategies that included creating both a health-promoting hospital setting and a health-promoting workplace, and providing health-related services, training, education, and research. The objective in assuming the HPH designation was to achieve WHO standards related to (a) management policy, (b) patient needs assessment, (c) patient information and interventions, (d) a healthy workplace, and (e) continuity of care and collaboration. The hospital decided to begin its pilot intervention by implementing the HPH concept in a perinatal context, which was perceived as a healthcare innovation.

To identify the characteristics and conditions required to implement the HPH concept comprehensively and effectively, the Health Promotion Department sought an evaluation. The intervention evaluated was the implementation of HPH standards within the hospital’s birthing centre. As such, this constituted a direct logic analysis.

Evaluating complex interventions is a huge challenge, especially when dealing with programs anchored in an ecological approach to health (Deschesnes, Martin, & Hill, 2003; Dooris, 2006; Poland, Green, & Rootman, 2000; St. Leger, 1997). The HPH concept includes health promotion

which is defined as “the process of enabling people to increase control over, and to improve, their health” (Ottawa Charter for Health Promotion [WHO, 1986]) and is here understood to embrace health education, disease prevention, and rehabilitation services. It is also understood to include health enhancement by empowering patients, relatives, and employees in the improvement of their health-related physical, mental and social well-being.

(WHO, 2004, p. 8)

In fact, very little is known about how best to implement a comprehensive and integrated approach like the HPH concept. Therefore, based on available scientific knowledge, we decided to use direct logic analysis to gain a better understanding of the factors that could influence the HPH standards implementation process.

Building the Logic Model

The hospital’s birthing centre records around 2500 deliveries per year. Among the pregnant women followed by health professionals there, 21% are at risk for psychosocial problems including drug addiction, domestic violence, and other harmful lifestyles. To address these problems, the birthing centre, as a centre of excellence in pregnancy and birthing, carries out diverse health promotion activities in its various units (i.e., general obstetrics clinic, first prenatal visit clinic, high-risk pregnancy clinic, perinatal clinic, delivery room, neonatology unit, post-partum unit). With the aim of achieving HPH standards, the centre’s activities are related to doing needs assessments for pregnant women and their families, systematically helping them to engage in healthy lifestyles, and screening pregnant women for psychosocial problems. The project leaders believe these activities will increase knowledge among pregnant women and their families, and empower them through counselling, health information, and education. The aim is to encourage them to engage in behavioural change processes that will enable them to better control and improve their health.

In addition, health promotion and capacity building activities are organized for the benefit of health professionals. The intended effect of such interventions is to promote a workplace in which professionals feel healthy and empowered.

The implementation of the HPH concept in the birthing centre is based upon the hospital’s health promotion policy, developed by the Health Promotion Department. Based on our informal interviews with people involved in the intervention’s design and implementation, we constructed the logic model collaboratively with the Health Promotion Department. We also analyzed documents related to the intervention (policy documents, reports, activity plans). Moreover, project leaders were involved in developing the model by providing relevant information on the intervention’s overall resources, activities, and outcomes, both intermediate and long-term. The logic model was first validated in discussions with some members of the Health Promotion Department (Figure 3).

Figure 3.

The Logic Model of the Health Promoting Hospital Concept

Developing the Conceptual Framework

In line with its mandate to be a “healthy hospital organisation” (Pelikan, Krajic, & Dietscher, 2001), the hospital is considered to be an “advocate” and a “change agent” for health promotion, in both its external community and its internal environment.

As organized systems of action, hospitals host different groups of actors, and thereby different sources of legitimacy and power that coexist and evolve through competition and cooperation (Contandriopoulos, 2003). They represent complex entities characterized by task diversity, multiplicity of actors, and contradictory expectations from the public and governments (Kervasdoué, 2007). Moreover, hospitals are characterized by the “silo effect” generated by hierarchical relations, divergent philosophies, and competing domains (Degeling, Kennedy, Carnegie, & Holt, 1998; Lane, 2005). Indeed, each healthcare profession has its own culture, including values, beliefs, attitudes, customs, and behaviours that often impede collaboration and interprofessional teamwork (Hall, 2005). In a context of change, hospitals, like any other organization, are dynamic and adaptive settings where actors play by divergent strategic rules, which are blurred and difficult to identify. Because of uncertainty, change can create conditions that lead either to interventions’ failures or to unexpected results. At the same time, change makes possible innovations whose impacts can vary significantly. Change processes are continuous and not time-limited (Orlikowski, 1996; Weick & Quinn, 1999). Programs such as the HPH, implemented to drive change, can be effective only if they are adaptive and adjusted to actors within their context (Boden, 1994). In fact, change consists in transforming actors’ routine practices in the organization (Becker, Lazaric, Nelson, & Winter, 2005; Levitt & March, 1988).

To develop the conceptual framework, we first tried to gain a solid understanding of the hospital context and of the intervention described in the logic model. Having grasped the intervention’s context, we oriented our review of existing knowledge toward two types of literature: first, the literature on change and innovation in organizations and, second, writings that document key factors influencing the implementation of HPH standards in health settings.

Change and innovation as organizational processes

According to Steckler, Goodman, and Kegler (2002), the most effective strategies used to achieve innovations (i.e., any policy, program, or technology that is new to the potential users) are based on principles of innovation diffusion and organizational change. They suggest that organizational change proceeds through four stages: (a) the organization becomes aware of the innovation (awareness); (b) it decides to adopt it (adoption); (c) it implements it (implementation); and (d) if the implementation is successful, it becomes institutionalized (institutionalization) (Steckler et al., 2002). As a planned social change process, innovation is an idea, practice, or object perceived as new by an individual or any other unit of adoption (Earl, 2002). The degree to which an individual perceives an idea to be new determines his or her reaction to it (E. M. Rogers, 1995). In Rogers’ diffusion of innovations model, the innovation diffusion process passes from (a) first knowledge of an innovation, to (b) forming an attitude toward the innovation, to (c) taking a decision to adopt or reject, then to (d) implementation of the new idea, and finally to (e) confirmation of this decision (E. M. Rogers, 1995, 2010). As an organized action, innovation is aimed at producing change (Desbiens & Dagenais, 2002). Therefore, an innovation’s success relies on the presence of liaison-agents (Orlandi, Landers, Weston, & Haley, 1990), or agents of change in the organization (E. M. Rogers, 1995) who devote their personal influence to encouraging the innovation’s adoption. Moreover, the process of innovation diffusion hinges upon organizational, sociocultural, and political factors. Indeed, change is a complex and unpredictable phenomenon that necessarily involves multiple actors and champions whose roles in the organization can evolve over time. It is also dependent on local opportunities, available resources, individual or collective initiatives, and professional differentiation (Champagne, 2002). Introducing the HPH concept into the hospital’s birthing centre was therefore a complex intervention, influenced by several factors that have been identified in the literature.

Implementing the Health Promoting Hospital Concept in a Hospital Context

Within a hospital are many—sometimes competing—interests. The HPH concept, described as an umbrella concept (Pelikan et al., 2001), incorporates the expectations of different groups of actors such as managers, professionals, nurses, and patients. In their everyday practices and actions, these actors are encouraged to collaborate to achieve an optimal quality of health-related services. Moreover, it is widely recognized that hospitals’ involvement in promoting health and preventing disease not only has positive impacts on the organization’s management and quality of services and on people’s health and quality of life, but also induces change in the local environment and community (Groene & Garcia-Barbero, 2005; Pelikan et al., 2001; Zhao, Carretta, & Hurley, 2003).

In the following paragraphs, we highlight some crucial characteristics and critical conditions that could influence the implementation process. We found that existing scientific knowledge is focused mostly on factors that could potentially influence the implementation of hospital-based health promotion activities (1). We also identified articles specifically on HPH standards implementation (2). And finally we integrated key elements found in the literature (3).

(1) Factors influencing the implementation of health promotion activities in hospitals

For McBride (2004), the key elements required for successful integration of health promotion activities are organizational leadership; financial support; health promotion that is anchored in the hospital’s philosophy and integrated into the hospital’s overall care and prevention strategies, policies, and plans; health promotion training and staff involvement; and collaborative strategies between internal and external actors. The literature also highlights that hospital-based health promotion activities depend on strong partnerships, good communication, and effective networking between healthcare settings (Johnson & Baum, 2001; Johnson & Nolan, 2004). The underlying condition for all of these elements is the hospital administrators’ strong will to integrate health promotion into the hospital’s mission and structure as a new organizational strategy (McBride, 2004).

In hospitals, different groups of actors have their own perceptions related to factors involved in implementing health promotion activities. For instance, while nurses may be willing to integrate health promotion into their everyday practices, they cite lack of time and of training in health promotion as obstacles to this integration (Mc-Bride, 1994). To alleviate the burden on nurses’ time, McBride and Moorwood (1994) suggest hiring a health-promotion facilitator to support them in integrating health promotion activities into their practices and routines.

From the managers’ standpoint, financial barriers, difficulties in meeting even the basic needs of patients, and employees’ lack of interest in health promotion are impediments to implementing health-promoting activities, especially in public hospitals (Stanton, Balanda, Gillespie, & Lowe, 1996). As frequently noted in the literature, the most important factors influencing all stages of effective health promotion practice are committed and skilled people, funds and resources, and interest for health promotion activities. Health promotion capacity building and activities implementation are also influenced by the quality of leaders and champions, of partnerships, of support structures and coordination mechanisms, of communications, and of the credibility of messages conveyed in health promotion programs, as well as by available evidence on the effectiveness of such programs (Robinson, Driedger, Elliott, & Eyles, 2006).

Professionals practicing in hospitals consider that implementing health-promoting services is facilitated by their organization being involved, by employees being recognized as “health-promotion instruments,” and also by ensuring a proper balance between resources and tasks (Johansson, Weinehall, & Emmelin, 2010). Hospital-based health professionals view health promotion as a long-term process and feel the organization should allocate much more time to prevention and health promotion. They also would like the objectives of services reorientation to be realistic and in line with resource availability. Moreover, health professionals want freedom of action in their practice to be able to implement health promotion activities with flexibility. Indeed, when top-down directives are imposed, professionals feel frustrated, whereas health promotion processes established through a bottom-up approach are more readily adopted. Finally, it has been observed that physicians and nurses practicing in hospitals that develop health promotion activities tend to pay more attention to patients’ needs for information and health education (McBride, 2004; Robinson et al., 2006).

In a study in British hospitals, 70% of patients interviewed considered hospitals to be appropriate settings for health promotion activities, especially those targeting risk factors related to nutrition and physical activity. Patients strongly believed that communication between healthcare services and general practitioners in the community and access by health professionals to electronic patient records were essential to promote their health (Haynes, 2008). The increased prevalence of chronic diseases calls for the expansion of collaborative relations between patients, hospital-based professionals, and community-based services to strengthen continuity of care (Robinson et al., 2006).

Even though each group has its own perceptions about the factors that influence the implementation of health promotion activities, the common objective pursued by integrating health promotion into the organization and care practices is to provide care and services that correspond to HPH quality standards.

(2) Factors influencing the implementation of HPH standards in healthcare systems

Beyond implementing health promotion activities for the patients’ benefit, the WHO Health Promoting Hospital project seeks to incorporate, in a normative way, health promotion concepts, values and standards (i.e., management policy, patient needs assessment, patient information and intervention, healthy workplace, continuity, and collaboration) into hospitals’ organizational structure and culture in order to improve the health of patients and staff, support healthy environments, and cooperate actively with the community. The project gives hospitals an opportunity to contribute to the public health agenda (Groene, Jorgensen, & Garcia-Barbero, 2004) and to achieve optimal quality management (Groene & Garcia-Barbero, 2005).

Even though standards seem to be essential for implementing the HPH concept and “were judged as being relevant and applicable to hospitals,” it appears that “most hospitals had only moderate compliance” (McHugh, Robinson, & Chesters, 2010, p. 5). Although those standards are perceived as necessary for assessment and continuous monitoring for quality improvement (Groene & Garcia-Barbero, 2005), there is a lack of evidence about their effectiveness in creating a health-promoting setting (WHO, 2004). Moreover, key factors influencing the implementation of those standards in hospital settings are rarely documented.

A recent literature review on health-promoting health services summarized the findings of articles that identified factors influencing the implementation of HPH standards (McHugh et al., 2010) and ultimately concluded that there was “a dearth of high-level research on HPH … and there is limited evidence therefore of the efficacy of HPH” (McHugh et al., 2010, p. 1). Meanwhile WHO, which initiated the Health Promoting Hospital project, continues to encourage hospitals to develop participative and collaborative strategies, organize training, and strengthen employees’ and patients’ involvement in order to make implementation effective, ultimately providing opportunities for research in this area (Groene, 2006; WHO, 2004).

The literature reports diverse methods used to implement the HPH standards. In Scotland, for example, when a strategic plan called the Health Promoting Health Service Framework was implemented in various health services, actors involved in the implementation gained a better understanding of health promotion concepts and methods, developed skills, and worked to improve patients’ health. However, because the framework rarely served to improve managers’ and professionals’ practices, its standards were considered insufficient to produce change in health services (Whitelaw, Martin, Kerr, & Wimbush, 2006).

The study by Whitelaw and colleagues (2006) showed that underlying the standards integration issue are necessary conditions for implementation. Other studies looking at the relation between health promotion policy—the first HPH standard— and hospitals’ quality management have demonstrated that hospitals in an HPH network apparently “have better preconditions for providing high-quality health services” (Polluste et al., 2007, p. 333) than do stand-alone hospitals.

Most studies that analyzed determining factors in HPH standards achievement confirmed the findings of other studies that looked at health promotion activities in general. Among the determining factors, McHugh et al. (2010) identified organizational support; Whitelaw et al. (2006) highlighted having health improvement on the political agenda, building health promotion capacity, leadership, and advocacy, having access to and support from health promotion experts, proximity to the “practical context,” and the presence of “multipliers”; and Tountas, Pavi, Tsamandouraki, Arkadopoulos, and Triantafyllou (2004) cited the need for financial and human resources, incentives, and experience in health promotion, training, collaboration, teamwork, supportive work conditions, and a good hospital reputation. Some studies observed an absence of coordination and lack of continuity (Aujoulat, Le Faou, Sandrin-Berthon, Martin, & Deccache, 2001; McHugh et al., 2010; Polluste et al., 2007), while yet others concluded that a bottom-up rather than top-down approach was more effective in implementing the standards (Guo et al., 2007). Above all, it appeared there was a need to clarify the standards’ contents in order to promote better understanding of the Health Promoting Hospitals concept (Lin, Huang, & Tung, 2009). Moreover, it is important to note that even though there are writings focused on health promotion activities in a perinatal context, we found no evidence related particularly to HPH standards implementation in birthing centres.

(3) Integrating key elements

As shown above, the literature has identified many factors influencing the implementation of health promotion activities and HPH standards. We first classified these factors according to whether they were external or internal. We then categorized internal factors into organizational factors and factors related to the intervention itself. Organizational factors are related to resources (i.e., human, financial, and infrastructure), organizational structure (i.e., organizational characteristics), organizational support, and working conditions. Factors related to the intervention are categorized into the organization level (i.e., strategy, vision, mission, and goals), the activities level (i.e., processes and project components) and the actors level (perceptions, attitudes, and agency). Table 1 summarizes our findings from the literature review.

Table 1.

Factors Influencing the Implementation of Health Promoting Hospitals Standards

EXTERNAL FACTOR Health improvement political agenda (Whitelaw et al., 2006)
INTERNAL FACTORS
ORGANIZATIONAL FACTORS	FACTORS RELATED TO THE INTERVENTION
RESOURCES Availability of resources (Johansson et al., 2010; Champagne, 2002) Funds and resources (Robinson, 1996). Physical infrastructures dedicated to HP (Johnson & Baum, 2001; Johnson & Nolan, 2004) Financial support and human resources available for HPH standards implementation through HP activities (Stanton et al., 1996; Tountas et al., 2004; Guo et al., 2007)	ORGANIZATION LEVEL Integration of health promotion (HP) into the hospital’s mission and structures (McBride, 2004) Integration of HPH standards into hospital quality management (Polluste et al., 2007) Collaboration between services within the hospital to enhance the HPH implementation process (Johnson & Baum 2001; Johnson & Nolan, 2004; Tountas et al., 2004; Robinson et al., 2006, WHO, 2004; Groene, 2006) Communication / information / diffusion between hospital care services and the community (Haynes, 2008; Robinson et al., 2006) Strong partnerships between the hospital and the community to develop HP programs (McBride, 2004; Robinson et al., 2006) Networking between the hospital and community organizations (McBride, 2004; Tountas et al., 2004) HP policy and program planning (Johnson & Baum, 2001; Johnson & Nolan, 2004)
ORGANIZATIONAL STRUCTURE Professional differentiation (Champagne, 2002) Health promotion anchored in hospital philosophy (Johnson & Baum, 2001; Johnson & Nolan, 2004) Hospital’s attitude and image with respect to HP (internally perceived) (McBride, 2004) Hospital’s attitude and image with respect to HP (externally perceived) (McBride, 2004) Continuity of care in the healthcare network (Aujoulat et al., 2001; Tountas et al., 2004; Polluste et al., 2007)	ACTIVITIES LEVEL Training in HPH standards for professionals (Johnson & Baum, 2001; Johnson & Nolan, 2004; Tountas et al., 2004; WHO, 2004; Groene, 2006) Recruitment of a health-promotion facilitator (McBride, 1994) HPH standards integrated into hospital program planning (Tountas et al., 2004) Integration of HP principles into professional practice (Whitelaw et al., 2006; McBride, 2004) Patient-centred approach to meeting patients’ needs (Stanton et al., 1996; McBride, 2004; Robinson et al., 2006) Credibility of messages transmitted and programs implemented (Robinson et al., 2006) Coordination in HPH standards implementation (McHugh et al., 2010) Access to and support from HP experts (Whitelaw et al., 2006) Collaboration between patients and physicians (Robinson and al, 2006) Coordination of HP activities (Robinson et al., 2006; Aujoulat et al., 2004; Polluste et al., 2007) Implementation of evidence-based HP programmes (Robinson et al., 2006) Considering HP as a long-term process (Johansson et al., 2010) Participative approach to HP activities implementation (Johansson et al., 2010) and HPH standards implementation (Lin et al., 2009) Bottom-up approach (Lin et al., 2009; Guo et al., 2007; WHO, 2004; Groene, 2006) Teamwork (Tountas et al., 2004)
ORGANIZATIONAL SUPPORT Organization’s interest in HP (Robinson et al., 2006) Organization’s support for HPH standards (Whitelaw et al. 2006; McHugh et al., 2010) Organization’s commitment to facilitating implementation of HP services (Johansson et al., 2010) Presence of organizational leadership (Johnson & Baum 2001; Johnson & Nolan, 2004) HP activities integrated into the hospital’s care and prevention strategy (McBride, 2004) Supportive structure (Robinson, 1996) Organization’s involvement (Johansson et al., 2010) Recognition of employees as “health-promotion instruments” (Johansson et al., 2010) Training in HP for professionals (McBride, 1994; Guo et al., 2007) HP capacity building (Whitelaw et al., 2006) Leadership and advocacy development for employees (Whitelaw et al., 2006)
WORKING CONDITIONS Improvement of working conditions necessary for HPH standards implementation (Tountas et al., 2004) Balance between resources and tasks (Johansson et al., 2010) Professionals’ perception of time load involved in implementing HPH standards (Tountas et al., 2004; Guo et al., 2007) Additional time allocation for HP (McBride, 1994, Johansson et al., 2010)	ACTORS LEVEL Employees’ perceptions of HP in hospitals (McBride, 2004; Stanton et al., 1996) Patients’ perceptions of HP in hospitals (McBride, 2004; Robinson et al., 2006) Better understanding of HPH concept (Lin et al., 2009; Guo et al., 2007) Development of individual or collective initiatives (Champagne, 2002) Managers’ proximity to field realities (practical context) (Whitelaw et al., 2006) Presence of “multipliers” (Whitelaw et al., 2006) Presence of champions serving as relays in new HP practices (Robinson et al., 2006) Presence of liaison-agents (Orlandi et al., 1990) Presence of change-agents (Rogers 1995) Lack of time and training in health promotion (McBride, 1994) Difficulties in meeting patients’ needs (Stanton et al., 1996) Lack of interest from employees (Stanton et al., 1996) Lack of incentives (Tountas et al., 2004) Presence of committed and skilled people (Robinson et al., 2006) Liberty of action in professional practice (Johansson et al., 2010) Staff mobilization for HP development in hospital setting (Whitelaw et al., 2006) Staff involvement in HPH standards implementation (Johnson & Baum, 2001; Johnson & Nolan 2004; WHO, 2004; Groene, 2006)

Evaluating the Program Theory

With a good understanding of the research field, combined with what we learned from the literature, we evaluated the program theory by comparing the real intervention as designed in the logic model with the findings from the literature analysis. We then identified the intervention’s strengths that appeared essential to reach its intended goals and the potential weaknesses that could impede its success.

The intervention’s strengths according to the logic model

Comparing the logic model with the factors identified in the literature that are said to influence HPH interventions, we found that the intervention implemented within the birthing centre had some real strengths. As conceived, the implementation of HPH standards in the birthing centre received financial support from the hospital foundation and from a private foundation. Moreover, the plan included human resources dedicated to implementing activities at the organizational (HP policy communication, health professionals needs assessment, etc.) and clinical (patient needs assessment, patient counselling, etc.) levels. Another lever presented by the intervention as planned was the integration of some activities into the healthcare network and community organizations. The organization has a Health Promotion Department in charge of communicating the health promotion policy, attesting to the hospital’s interest and leadership in implementing the HPH standards. As represented in the logic model, the standards constitute the long-term outcomes targeted by the organization. We can therefore conclude that at the activities level, HPH standards are integrated into hospital policy and program planning. Furthermore, all the activities presented in the birthing centre’s logic model are definitely patient-centred and are planned to be implemented through teamwork and collaboration among all the organizational and clinical units. Finally, the Health Promotion Department functions as a liaison-agent between the hospital as a whole and the Birthing Centre in the implementation of the intervention.

The intervention’s potential weaknesses

Some other factors that would be expected to have an influence over the intervention, based on the literature, are not clearly represented in the logic model. For example, there was no explicit plan for training the professionals in order to integrate the health promotion concept and values with the HPH standards. Moreover, even though the health promotion policy was planned, there was no indication of how the hospital intended to promote itself externally as a health-promoting hospital. Establishing the hospital’s image as a health-promoting setting could therefore constitute a real communication challenge. This is an aspect that could be improved when developing the intervention plan.

To implement HPH in the perinatal context, some external conditions (political context, health system reform, etc.) that could affect the implementation process had to be considered. In our literature review, the only external factor that emerged was the health improvement political agenda. Such an agenda would explicitly promote a new perspective in hospital-based healthcare focused on health improvement and especially health promotion. However, other external factors, such as healthcare system pressures, have an influence. Severe spending cutbacks in the system today impede the development of activities such as training and access to health promotion experts. To date, the birthing centre intervention has been financially supported by both the hospital foundation and a private one, but there are concerns about its sustainability in the long term when those funds come to an end.

Implementing the HPH concept in the hospital’s context can constitute a real challenge. Hospitals are characterized by fragmentation of services, professional differentiation, and considerable time overload that could impede optimal development of the intervention. Given the constraints of the overall hospital setting, it might be difficult for agents of change such as the Health Promotion Department to spread health promotion principles and philosophy into a highly specialized and curative organization whose primary goal is to treat diseases, allocating budgets to tertiary and quaternary care. To provide effective leverage to implementation, Johansson et al. (2010) cautioned that there must be a balance between resources injected and a realistic level of tasks related to health promotion activities.

Given the deeply anchored curative culture of the hospital, it is not easy to integrate into hospital-based care—that is, into all the healthcare departments, services, and professional practices—the health promotion activities and prevention strategy that are essential to support continuity of care as well as care coordination, prevention, and promotion activities. Managers responsible for planning and implementing health promotion activities will be challenged on a daily basis, especially considering the hospital’s work conditions. The literature on human resources in hospitals reveals that most employees, especially nurses, work considerable overtime with very few breaks, resulting in a stressful working environment. Consequently, it becomes difficult to mobilize and motivate them for other types of activities (e.g., health promotion, health education) that are not, but should be, included in their professional practices. To exercise the levers of leadership and advocacy within their teams, medical unit managers must be aware of their employees’ needs and concerns. Their proximity to the “practical context” (Whitelaw et al., 2006) and to field realities could allow them to identify multipliers and leaders who could relay to their team the importance of health promotion principles, values, and activities.

DISCUSSION

Conducting logic analysis from an organizational perspective allowed us to identify strengths and weaknesses of the HPH implementation in a perinatal setting embedded in a hospital. Our analysis offered a better understanding of the factors that might significantly influence the intervention’s implementation.

The preliminary logic model constructed for this project served as a basis for discussions with the project’s key users. It was purposefully presented in a simplified form so that they could identify clearly, sequentially, and logically the objectives of each of the intervention’s components. It should be stressed that this first step was not intended to minimize the complexity of the intervention. Indeed, according to Patton, “in face of complexity, the first task is to identify clear, specific and measurable goals … Everything seems complex until you do a logic model” (Patton, 2011, p. 6).

In a next step, the evaluator will go back to the actors to discuss in greater depth the complex relationships between the different elements and add to the original logic model the factors thought to influence the intervention. This will provide a more accurate representation of the intervention’s complexity.

Some factors and conditions identified in the literature are represented in the intervention’s logic model. The hospital-specific context, which is complex and more focused on curative care, could encumber the intervention, whose perspective is preventative rather than curative. It would therefore not be surprising to encounter resistance throughout the implementation process.

Achieving a condition such as anchoring health promotion in the hospital’s philosophy requires a cultural and a paradigmatic shift. It takes time for actors in a hospital to incorporate a broader, more holistic perception of health. The definition, principles, and values of health promotion have to be mainstreamed throughout the hospital’s departments and services, and integrated into professional practices, while respecting the organization’s structural and cultural realities. That being said, the birthing centre’s project leaders and managers can influence their HPH project more effectively by remaining close to those working in the units. In developing local solutions to strategic challenges such as training professionals in health promotion and allocating time for health promotion activities, this project will ultimately have a greater and longer-term impact over the hospital as a whole.

By making more explicit the factors influencing good implementation of the HPH concept in a perinatal context, the logic analysis calls for an implementation analysis to investigate in depth the influence of these factors, notably by collecting the actors’ perceptions and experiences. We are firmly convinced that, if this pilot program were given more visibility, the innovative experience of the birthing centre could ultimately be extended to other areas within and outside the hospital’s boundaries.

CONCLUSION

Our previous experience of logic analysis (Brousselle & Champagne, 2011; Brousselle et al., 2009; Brousselle et al., 2007) has shown that it is important to conduct this type of evaluation before launching other evaluation activities. Doing so has important consequences for the various stakeholders, as well as for the evaluation itself. It can provide, in a relatively short time (from several weeks to six months, depending on the complexity of the intervention and the time devoted to the project), a sound interpretation of the intervention’s strengths and alternatives for action. Logic analysis highlights which factors are more important for obtaining the effects and can help identify levers to foster the implementation of the intervention.

First, it is directly useful to stakeholders. It rapidly gives them a sound conceptualization and understanding of the intervention they are trying to implement and of the critical conditions involved. The knowledge derived from the conceptual framework can be used to rapidly reorient actors’ actions to maximize their impact.

Second, logic analysis is an important tool for evaluators because it enhances their knowledge of the strengths and weaknesses of the intervention’s theory and helps them to identify factors over which stakeholders have little or no control. With logic analysis, evaluators are then able to choose the appropriate type of evaluation to conduct.

Finally, logic analysis produced, within a short time, an original and illuminating interpretation of the intervention, providing information that was directly useful for improving actors’ practices while setting the foundations for a valid and relevant subsequent evaluation project.

Evaluators may sometimes find the complexity of certain interventions daunting and feel discouraged about designing their evaluation. We have found that logic analysis offers a well-structured approach to analyzing interventions by stressing the most important factors, characteristics, and conditions involved in the successful realization of desired effects.

Biographies

Lynda Rey is a Ph.D. student in Public Health at the University of Montreal, and holds a degree in political science from the Institut d’Études Politiques of Aix-en-Provence, and a Master’s in cooperation and international development policies from the Sorbonne University in Paris. In recent years she has worked in international cooperation as a project officer for health (HIV/AIDS, maternal mortality) and human rights. Her research interests are participatory approaches in evaluation, health systems organizations, and health promotion. Her doctoral dissertation focuses on evaluating the implementation of the WHO’s Health Promoting Hospitals concept in a birthing centre.

Astrid Brousselle is an Associate Professor in the Department of Community Health Sciences at the Université de Sherbrooke and researcher at the Charles LeMoyne Hospital Research Center. She holds a Canada Research Chair in Evaluation and Health System Improvement–EASY (CIHR-FRQS). Dr. Brousselle contributes to the development of innovative evaluation approaches, where her objective is to use evaluation as a lever for health system improvement. She applies innovative methods to various areas of research in public health and health system organizations.

Nicole Dedobbeleer, Sc.D., is a full professor in the Department of Health Administration at the Université de Montréal, where she has been teaching new practices and health since 2002. She is also a researcher in the Research Institute in Public Health of the Université de Montréal (IRSPUM). Her research interests focus now on the adoption and implementation of the WHO’s Health Promoting Hospitals concept and its adaptation to health and social centres.