Evaluating Service Organization Models

Abstract

Based on the example of the evaluation of service organization models, this article shows how a configurational approach overcomes the limits of traditional methods which for the most part have studied the individual components of various models considered independently of one another. These traditional methods have led to results (observed effects) that are difficult to interpret. The configurational approach, in contrast, is based on the hypothesis that effects are associated with a set of internally coherent model features that form various configurations. These configurations, like their effects, are context-dependent. We explore the theoretical basis of the configuration approach in order to emphasize its relevance, and discuss the methodological challenges inherent in the application of this approach through an in-depth analysis of the scientific literature. We also propose methodological solutions to these challenges. We illustrate from an example how a configurational approach has been used to evaluate primary care models. Finally, we begin a discussion on the implications of this new evaluation approach for the scientific and decision-making communities.

Relevance

In recent years, the evaluation of service organization models, particularly in primary health care, has been the subject of much debate. Over the past decade (WHO, 2008) and in several countries, various primary care reforms (Marriott and Mable, 2000) have attempted to address problems related to the accessibility and continuity of primary care services (Commission d’Étude sur les Services de Santé et les Services Sociaux, 2000; Commission on Medicare, 2001; Government of Alberta, 2001; Health Services Restructuring Commission, 1999; Ministère de l’Emploi et de la Solidarité Datar-Credes, 2000; Russel and Mitchelle, 2002). Despite the number of reform initiatives, however, there exists relatively little information about their potential and actual impact. Of the few studies that have evaluated primary care policies, most are ecological in design and suggest that primary health care does have a significant impact on population health (Macinko et al., 2003; Shi, 1994; Starfield, 1994; Starfield et al., 2005); at the same time, it also seems to reduce costs (Starfield, 1994). Other results, also constrained by the limits inherent in ecological analysis, suggest that access to primary health care is a promising avenue for reducing inequities in health (Blumenthal et al., 1995; Shi and Starfield, 2000; Shi et al., 1999; Starfield et al., 2005). Nonetheless, the many discrepancies between findings make it difficult to draw clear conclusions about linkages between service organization models and the observed impacts (Hutchison et al., 2001; Levesque, 2003). The following statement by the Canadian Health Service Research Foundation summarizes the situation: ‘although everyone embraces primary healthcare as the cornerstone for healthcare restructuring, there is no consensus on what it is’ (CHSRF, 2004). It is our belief that this situation results from the evaluation approaches followed to date, which for the most part have examined the individual attributes of organizations independently of one another. We believe that, in order to address this gap in our understanding, it is imperative to consider an alternative approach, based on the concept of organizational configuration. The merits of this approach were initially stressed a decade ago but have been underexploited so far. It is thus pertinent and useful to consolidate, through an in-depth theoretical analysis and analysis of concrete applications, the position of this approach and to launch a theoretical and empirical discussion on methodological issues that it raises.

Our article is organized in the following manner. We begin by showing that empirical studies have revealed limits to our knowledge about the impacts of primary health care models. We also suggest hypotheses explaining the linkage or the lack of linkage between service organization models and their impact, which leads us to propose the configurational approach as an alternative approach. Next, we emphasize the relevance of the concept of configuration for organizational analysis by underscoring its theoretical basis. This section concludes with a discussion of the usefulness of the concept of configuration for the evaluation of health service organization models. We then discuss the methodological challenges inherent in the application of a configurational approach through in-depth analysis of research that has used the concept of configuration in organizational analyses. We next suggest methodological solutions to these challenges. Through an example, we illustrate the issues and added value of a ‘configurational’ approach. Finally, we launch a discussion on the implications this approach has for the scientific and decision-making communities.

The Current State of Knowledge: The Limits of ‘Classical’ Evaluation Approaches

Most studies of primary health care models have examined the relationship between selected attributes of a given model and a particular aspect of the model’s performance (quality, effectiveness, accessibility, etc.). The attributes most studied are gatekeeping (Blumenthal et al., 1995), physician remuneration methods (Gosden et al., 2001; Hutchison et al., 1996; Starfield et al., 1998), multi disciplinary clinical teams (Cohen et al., 1986; Yano et al., 1995), patient identification with a regular source of care (Baum et al., 1998; Fint, 1987), case management (Druss et al., 2001), medical practice support measures (such as electronic reminders) (Yano et al., 1995), feedback on preventive measures (Yano et al., 1995), patient file review committees (Yano et al., 1995), computerization of patient records (Starfield et al., 1977), the autonomy of referral of physicians (Starfield et al., 1998), greater reliance on nurse practitioners (Yano et al., 1995), and clinic operating hours (Yano et al., 1995).

This literature has confirmed that some organizational attributes are associated with specific effects. For example, capitation payment systems are shown to be associated with the reduced use of specialized services (Gosden et al., 2001). Similarly, access to primary health care is enhanced in facilities staffed by multidisciplinary teams, 24-hour telephone medical access and extended clinic operating hours (Yano et al., 1995). In their systematic review of primary care (2005), Starfield and colleagues also show that a regular source of primary health care increases both the effectiveness and efficiency of care.

However, a number of notable inconsistencies between study results are striking. Studies of the impact of gatekeeping on the use of specialized services, for example, have yielded varying results (Blumenthal et al., 1995). It is also difficult to determine whether financial incentives (such as capitation or bonuses for achieving immunization targets) motivate physicians to practise a greater number of preventive measures (Ritchie et al., 1992; Shimmura, 1988, cited in Geneau, 2004). Hulscher et al.’s (2002) meta-analysis on interventions aimed to improve prevention also shows varying results. Similarly, Yano and colleagues (1995) found that no single measure has a clear positive impact on overall continuity of care. Finally, concerning relational continuity, Forrest et al. (2002) have shown that having both a designated physician as well as a longer relationship with one’s physician are factors associated with greater patient satisfaction with the physician–patient relationship. According to others, however (Freeman et al., 2000, cited in Reid et al. 2002), these factors do not necessary result in relationships of loyalty and trust.

In summary, when analysed separately, some attributes of organization models tend to have effects but these often lack consistency and are difficult to interpret because of the lack of theoretical basis. This conclusion suggests that we must consider other hypotheses about the linkages between service organization attributes and observed effects. Our proposition is that observed effects are associated with a set of internally coherent attributes that form various configurations. These configurations, like their effects, are influenced by the context in which they operate. Accordingly, we suggest that the evaluation of service organization models adopt a configurational approach based on the premise that the features of a given intervention cannot be disassociated but are in fact interdependent.

Analysing Organizations Using the Concept of Configuration: The Idea of External and Internal Fit

The purpose of the concept of configuration (Meyer et al., 1993), like that of related concepts such as archetypes (Greenwood and Hinings, 1993) and gestalt (Miller, 1981), has been to analyse organizations holistically by assessing the coherence between the various attributes of the organizations in question. Links between these attributes are reciprocal and non-linear. Accordingly, the concept of configuration is based on two premises: that of ‘internal fit’, meaning the internal coherence of the attributes of an organization, and that of ‘external fit’, the configuration’s coherence vis-a-vis the external environment and the fact that organizational performance depends on an organization’s ability to adjust to its environment or context.

Internal Fit: Theoretical Interpretations

Different theories incorporate this idea, each advancing a different interpretation of the internal coherence of the features of an organization.

• Systems theory (Angyal, 1969), for example, emphasizes the primacy of the whole over the parts and the importance of non-linear interrelationships between elements. A more recent addition to systems theory is complexity theory (Gleick, 1988; Waldrop, 1992), which studies the behaviour of complex systems (systems characterized by a high number of interacting elements). For systems theorists, internal fit includes the idea that systems are self-organizing and naturally tend towards order.

• Institutional theory, which refers to archetypes far more frequently than other theories, argues that the coherence of the elements of an organization (its structure and systems) is upheld by the organization’s underlying interpretive schemes (Greenwood and Hinings, 1988, 1993, 1996). These interpretive schemes embody the purpose of the organization, its optimal structure, and the criteria it uses to evaluate its performance. Because organizational structures and systems affect interpretive schemes in turn, the relationships between the three variables are reciprocal and non-linear. Institutional theorists emphasize the significance of the values expressed in a given institutional field, that is, a domain of institutional life made up of a set of organizations or ‘a community of organizations that shares a common meaning system and whose participants interact more frequently and inevitably with one another than with actors outside the field’ (Dimaggio and Powell, 1983; Scott, 1995). Institutional theory places greater weight on the similarities between organizations (isomorphism) than it does on their differences; furthermore, it considers isomorphism to be necessary for organizational legitimacy. We will see, further, that this principle of isomorphism has been questioned.

External Fit: Theoretical Interpretations

Two types of theories have sought to relate a given configuration’s performance to its external environment.

• Contingency theories (Child, 1972; Lawrence and Lorsch, 1967), as their name suggests, postulate that contextual elements impact the structures and processes of an organization and that organizational performance depends on the organization’s ability to adapt to its environment.

• Population ecology-type theories (Aldrich, 1979; Hannan and Freeman, 1977, 1984) maintain that the role of the environment is determinant. They question the principle of adaptation and encourage greater attention to the constraints that face organizations. Ecology theorists believe that in any given niche, superior performance results in certain organizational forms being selected.

In practice, there is general agreement that these two perspectives are not exclusive but in fact complementary: that the fit between an organization and its environment is a function of the effects both of the environment and of management decisions (Ketchen et al., 1993). These considerations support the view that the concept of configuration, which rests on the notions of external and internal fit, is a widely accepted one.

Theories about the Organization of Health Services: How Suitable is the Configurational Approach?

The configurational approach has proved a popular tool for the description of the evolution of service organizations, particularly in the United States (see Dubbs et al., 2004). Several studies on this topic have used a taxonomic approach to classify organizational models along three dimensions: integration, differentiation and centralization. Others, working from institutional theory, have moved beyond description to analyse the transformation of health care systems in various contexts (the USA, Great Britain, Germany and Canada). This research tradition (Denis et al., 1999; Dent et al., 2004; Kitchener, 1999; Scott, 2003) suggests a general movement towards a new archetype or variations thereof. In summary, the research findings show transformations of health care systems to be dependent on their institutional context (as would be expected) and to be multi-faceted in nature.

Despite the interest and significance of these findings, it is evident, however, that the evaluation of the impacts of health care system transformations is not yet fully developed. Concerning specifically the evaluation of the primary care reforms, most studies, as mentioned before, have failed to consider the relationships between the various model features or the relationships between those features and the environment, as factors that determine overall performance. Some exceptions exist. We can for example cite the numerous works of B. Crabtree (Crabtree et al., 2001a, 2000b; Miller et al., 2001), based on the complexity theory and on qualitative research. These studies have shown that successful practices are those that ‘minimize errors, make good sense of what is happening and effectively improvise to make good practice jazz’. Consequently, efforts to change practice must recognize the potential benefits of some kind of variation. For example, the incorporation of preventive services into primary care practice requires a more comprehensive approach that takes into account factors relating to the physician (physician style, doctor – patient relational continuity, etc.) and to the practice (practice organization, office staff, patient population). Crabtree’s studies provide a deep understanding of primary care practices but need to be further developed to draw meaningful typologies of primary care practices, necessary for decision-making. We argue that a more extensive utilization of the configurational approach and the exploration of related methods can help overcome these shortcomings.

While this evaluation approach may seem attractive, its application poses a number of methodological difficulties which we will now examine.

Using the Configurational Approach in Evaluation Research: A Theoretical Discussion of the Methodological Challenges

Because the performance of service organization models is dynamic by nature, the methodological challenges related to the evaluation of this performance are best identified by drawing on analyses of organizational dynamics. Studies relying on the concepts of archetypes, related to institutional theories, will help us to identify these challenges.

The need to conceptualize theoretically based models

By definition, a configurational approach calls for understanding the linkages between the features of an organization and their effects on performance. To reach this understanding, it is necessary to relate configurations to explanatory theories (Miller, 1996).

In addition, studies have shown that theoretically based deductive approaches to the development of typologies explain variability in performance better than do purely inductive approaches (Ketchen et al., 1993). Authors including Doty and Glick (1994) insist on the need for theories to explain both the coherence between the features of a configuration and the relationships between configurations and performance. Furthermore, they state, it must be possible to validate the theories in question (Doty et al., 1993): in other words, it must be possible to predict the dependent variable – in this case performance – according to the configuration’s deviation from the ideal type. The development of theoretically based typologies must also conform to certain guidelines, such as clear theoretical assertions, the identification of all possible ideal types and the description of those types using the same dimensions.

But whereas the proponents of this approach insist that it is useful, they also admit that it is complex: ‘it becomes more difficult to ensure that only those dimensions that are related to the dependent variable are included in the typology’. Of course, testing the plausibility of the configuration and its effect, through a logic analysis, can help fill this gap (Brousselle et al., 2007) but we think also that the solution probably is to combine a deductive approach with an inductive one.

The need to integrate different levels of analysis

Strongly inspired by resource dependency theory (Pfeffer and Salancik, 1978), Hinings et al. (1999; Reay and Hinings, 2005) have suggested using different levels of analysis in order to understand the process of organizational change. These levels include environmental change factors, characteristics of the institutional field (permeability and the extent of coupling), the role of actor-entrepreneurs, and specific characteristics of a given organization (power relations, commitment to values and capacity for action). In summary, as Powell et al. (1999) have pointed out, the transformation of an institutional field can only be understood when the reflexive relationship between the institutional field and the organizations that compose it is taken into account. Similarly, Scott et al. (2000) identify two factors that act as intermediaries in processes of inter-level transformation: the degree of congruence of a given actor (the extent to which a social actor embodies or reflects the rules, norms prevailing in its context) and the degree of structuration of the field. Overall, it appears that the study of organizational change must integrate several levels of analysis: the general environment, the institutional field and the organization’s internal environment.

The need to account for the time factor

Close analysis of organizations’ trajectories shows that organizations never lie in a state of static equilibrium: rather, they most often move between two archetypes. Greenwood and Hinings (1988) used the concept of ‘tracks’ to analyse the dynamics of organizational change. They studied the non-linear movements between two archetypes: delays, oscillations, reorientations and aborted changes are some examples. Their results showed that an absence of fit can last a long time, and organizations can find themselves in situations of embryonic coherence or schizoid incoherence. Cooper et al. (1996) refer to sedimentation to suggest that archetypes should also be seen as structures in progress. According to this view, an organization could very well change its structures and processes without changing its interpretive schemes. In his analysis of the transformation of Great Britain’s hospital system, Kitchener (1999) confirmed that the process of constructing archetype is primarily a process of identity building.

Inspired by the results of a longitudinal study, Dyck (1997) suggests that analysis would gain by recognizing that organizations are constantly taking measures to change. These measures may or may not take root, depending on the type of rationality they reflect. This dynamic leads to tension between the transformation and the continuation of a configuration. Such a view of organizational dynamics attenuates criticisms of the concept of configuration (Kirkpatrick and Ackroyd, 2003) that consider configuration reflects a functionalist view of organizations that does not reflect reality – reality being that organizations are constantly changing without necessarily heading towards a configuration. They advocate the concept of configuration be totally dismissed. This position neglects the fact that it is not necessary for organizations to be seen as static in order for the concept of archetypes to be relevant and analytically useful.

As a result, the evaluation of the performance of organizations, seen as archetypes, requires a detailed understanding of the sequence of interactions that produce the impacts observed at any given point in time. In other words, analysis must include the time factor by using a longitudinal research design, among other measures. Inclusion of the time factor requires combining two views of time (Poole, 2004): the Newtonian view (time as a linear continuum) and the transactional view (the temporal occurrence of significant events).

The need to identify archetypes by referring to interpretive schemes

Hinings et al. (1996) demonstrate that the values of elite management had a critical role both as initiators of archetype change and as interpretive schemes of organizational structures. Because the coherence of an archetype is based on these schemes, the evaluation of the performance of a service organization model, seen as a configuration, must identify underlying values.

The fact that the systems and structures that correspond to the state of an organization may or may not correspond to the values of the actors (see the principle of sedimentation) makes this particularly true. The phenomenon of ‘desirability’, however, complicates the identification of values in the data collection procedure.

The need to delineate the context and understand its effect

As previously mentioned, the environment of an institutional field and of the organizations that make it up has an important influence on both the structure and the performance of the organizations. According to precursors of the institutional theories, organizations within the same field tend to evolve towards a single archetype (isomorphism). Other authors have questioned the validity of this principle (Brock, 2006; Dent et al., 2004; Pinnington and Morris, 2002). In accordance with these authors’ views, we feel that the analysis of institutional change would benefit from studying the variability among archetypes in a given institutional field: a variability that is partly attributable to the unique history of the various organizations and professions and their specific institutional contexts. Other authors, such as Dyck (1997), go as far as taking the extreme position of maintaining that an archetype can be specific to a single organization.

For that reason, it is important to clearly delimit the particular context that surrounds service organization models as well as to evaluate with precision how that context affects performance. Scholars have long emphasized the importance for all evaluation research projects to take context into account. Indeed, this is a well-defined field of research in itself (Type 3 implementation evaluations; Brousselle et al., 2009). Pawson and Tilley (1997) even advocate using what they call ‘realistic evaluation’. As Scott et al. (2000) contend, however, delimiting the context is far from simple: ‘environmental factors vary in terms of their spatial and temporal scope as well as in their salience to particular field participants’. It is therefore important to question how relevant environmental factors are to different types of organization. According to Scott et al. (2000), two types of environmental variables must be integrated into an analysis of the context: material resource environments and institutional environments.

In addition to delimiting the environmental factors that seem to influence models and their performance, it is also necessary to understand the impact of these factors. Some studies (see especially Alter and Hage, 1993) have attempted to theorize the impact of environmental variables on the extent of collaboration between organizations – a collaboration that conditions performance in turn. The authors identified two categories of analytic variables: variables that correspond to external controls (the degree of resource dependency, the amount of network regulation, etc.) and variables that correspond to technology (task dimensions).

The need to adopt a synthetic view of the ‘phenomenon’ under study

The concept of configuration leads us beyond purely analytical approaches that are founded on the principle of separating a whole into its parts in order to reveal an underlying order. In this perspective, evaluation of service organization models requires a synthetic approach more centred on comprehension (as in ‘com’, completely, and ‘prehendere, to grasp or seize). This perspective allows us to compensate for shortcomings in the analytic approach that include the following (Miller and Mintzberg, 1983: 60): ‘1- relationships are generally assumed to be linear and causation unidirectional; 2- the focus is on bivariate relationships’. A synthetic approach, in contrast, ‘is really the search for networks of causation. There are no purely dependent or independent variables in a system: over time, everything can depend on everything else.’

From this discussion, it is evident that a number of methodological challenges are associated with the use of a configurational approach. Meeting these challenges requires applying appropriate analytical methods.

Overcoming these Challenges: What Methods Work Best When Using a Configurational Approach?

In our opinion, evaluating service organization models using a configurational approach requires combining two types of methods:

• Factor analysis (more precisely, multiple correspondence analysis – Greenacre, 1992) type and classification type (Bailey, 1994) quantitative methods that recognize and formalize the links between variables (the intervention, the effects and the contextual variables). Factor analysis seeks to group as much initial information as possible into a limited number of factors, according to the associations between variables. As their name suggests, classification techniques assign individuals to groups according to their similarities. Statistically speaking, these techniques minimize intragroup variance and maximize inter-group variance. Classifying individuals according to the dimensions identified in factor analyses therefore leads to a more accurate understanding of the links between an organization model with which an individual is associated and the individual’s experience of care. These techniques can ‘detect’ non-linear and reciprocal relationships between variables. In contrast to the classical regression-type quantitative approach, the techniques mentioned strive to comprehend the causal complexity of phenomena without resorting to simplistic hypotheses (how the independent variable x influences the effect variable throughout all the cases). Instead, the question they ask is the following: ‘in what context and in what kinds of cases is a given variable related to a given effect?’ Use of these techniques does not mean that the analysis of the impact of each variable becomes obsolete. On the contrary, we believe that, in an initial phase, it will remain important to study the impact of each variable in order to make sense of the data. Nevertheless, it is essential to go beyond these analyses in order to develop methods that will enable researchers to grasp the impact of the entire set of variables, by considering their interactions overall.

• Qualitative, ‘case study’ type methods, both in order to complement the findings of the quantitative methods and more specifically to better understanding the whys and hows of the observed outcomes (see ‘the need to delimit and understand the effect of context’ and ‘the need to develop theoretically based models’). The explanatory theory that emerges from these qualitative case studies will help orient the quantitative methods. As Bailey (1994) has written, the choice of classification methods (the choice of variables, the decision whether or not to weight variables, the presence or absence of ‘outlier individuals’) does not result from a magic formula but should be guided by the underlying theory that explains the phenomenon under study. Qualitative methods are not merely ‘fodder’ for quantitative methods, only furnishing preliminary data for more sophisticated quantitative methods. Rather, qualitative methods provide a complementary perspective on the problem of study, and thus have merit in their own right.

Combining these two types of methods (quantitative and qualitative) therefore permits researchers to compensate for the shortfalls of each. Numerous programme evaluation theorists have long endorsed a combined approach (Patton, 1990, cited in Dubois and Marceau, 2005) in order to reinforce the internal and external validity of evaluation research (Greene et al., 2001). As Ragin (2004) has pointed out, the methodological challenges of the configurational approach consists of combining the merits of both types of methods (establishing regularity across cases and recognizing case specificity). It is necessary to understand the parts in order to understand the whole.

Illustrative Example of Methodological Challenges and Added Value of the Configurational Approach: The Evaluation of Primary Healthcare (PHC) Organization Models

Context and Problem Definition

Comparative analysis of the performance of health care systems among Canadian provinces reveals that Quebec lies slightly below the average, particularly in terms of accessibility and continuity of PHC services (Castonguay et al., 2008). To improve this situation, Quebec has engaged in recent years in important reforms. The most recent one follows the recommendations of an inquiry commission (Commission d’Étude, 2000) and has led to the creation of 95 health and social services centres (Centres de services sociaux et de santé, in French – CSSS), resulting from the merging of public establishments located in their territory (Levesque et al., 2007). These CSSSs are responsible for their population’s health and for building a local services network with their partners, namely the private medical clinics. Their major task of strengthening PHC services is to improve accessibility, comprehensiveness and continuity of services. General practitioners in private clinics were encouraged, through financial and professional incentives, to join Family Medicine Groups (FMGs) that receive public financing for developing collaboration with other professionals, namely nurses, and for providing multi-disciplinary services.

Several studies, using different methodologies and addressing the question of the impact of different PHC organizational models on the practice of physicians and the care experience of users of services, have been initiated to support decision-making. We review two of these studies in this section. These two studies are considered, for the purpose of this article, as a research programme comprising two components, one qualitative and the other quantitative. These studies, headed by two autonomous researchers, are both inspired by the configurational approach and, as we will see, are strongly academically interrelated.

The Theoretical Framework

The conceptual framework of this research programme posited that care experience results from the interface of organization and population characteristics in different contextual environments (Figure 1).

Figure 1.

A Theoretical Perspective

Methods

A mixed methods approach, combining qualitative and quantitative methods, has been adopted by the researchers. More precisely, they adopted an integrated research design (Greene et al., 2001) where ‘there are scheduled points of interaction between different methods through duration of the study and mixing methods is iterative and ongoing’. This contrasts with a coordinated design where ‘mixing of methods happens at the end of the study at the stage of drawing conclusions’. As we will see later, the qualitative study of the programme served to suggest hypotheses to explore in the quantitative study and to better understand and interpret its findings.

The qualitative approach

This study consists in the analysis of the medical practice of general practitioners (GPs) from a sociological perspective that views it as a social practice. The objective was to look inside the black box of the routine work of GPs (Geneau, 2004). Geneau based his analysis on the structuration theory developed by Giddens (1984) that holds that structures and thus modes of organization act simultaneously as facilitators and barriers for action. In other words, structures do not exist independently of actors (they are in fact ‘mnesic’ traces), but only in the course of their practice; this implies that modes of organization do not have a determining effect on medical practice. Given that actors are socially competent, able to understand and explain their own actions, as posited by Giddens, Geneau opted for a research strategy based on ethno-methodology, completed by qualitative case studies in order to grasp the dynamics of physician practice in PHC organizations. Cases were selected according to a strategy akin to purposeful/theoretical sampling. The selected cases were distributed as follows: four in private clinics where physicians were paid on a fee-for-services basis, of which two were in rural and two in urban areas; the other four were in community health centres (CLSCs: publicly owned organizations providing medical and social services and in which physicians are paid on a salary basis); two of these were in rural and two in urban areas. Data were gathered from responses to semi-structured interviews and were then analysed according to a theory-building strategy (Strauss and Corbin, 1998).

The quantitative component

This study involved three levels of analysis: population (a telephone survey of 9206 respondents in two regions); organization (a mail survey of 473 clinics); and context (information gathered from various sources in 23 CSSSs) (Pineault et al., 2008) – a nominal link was established between respondents to the population survey and his/her regular source of care (PHC organization) (Figure 2).

Figure 2.

Research Design

One of the two regions under study was urban (Montreal) and the other contained both urban and rural areas (Monteregie). The population survey (response rate: 64%) provided information about the usual source of care, unmet needs, and care experience including perceived outcomes of users of services. Indices of care experience were constructed as follows: first, 28 items were selected based on conceptual judgment according to their link with the various dimensions of care experience (accessibility continuity, etc.). Factor analysis led to the grouping of items around eight indices (Table 1). Cronbach’s α showed a satisfactory level of consistency. In order to compare indices, they were dichotomized, the cutoff point being determined by a conceptual judgement on performance and a score was calculated in percentage for each index.

Table 1.

Eight Care Experience Indices

Global index	29 items
Accessibility (geographical) Ex: distance and opening hours	5 items
Accessibility (economic) Ex: fees for non-insured services	3 items
Affiliation continuity Ex: regular source of care	4 items
Informational continuity Ex: follow-up after a test or visit to specialist	3 items
Responsiveness Ex: non-clinical aspect of care; respect and dignity of the individual	5 items
Comprehensiveness Ex: global/biopsychosocial approach of care	4 items
Care outcomes Ex: control of the condition; acquiring good lifestyle habits	5 items

The organizational part of the study involved a mailed questionnaire completed by 473 organizations (response rate: 75%). Information was gathered on various aspects of organizations, namely their vision, their resources, their structure and the mechanisms of coordination and collaboration that enhance continuity, accessibility and the type and scope of services offered. To construct a taxonomy of organizations, a method combining factor and cluster analysis was used (Lebart et al., 1995). First, 43 variables retained were transformed into categorical variables. This qualitative coding was based on two principles: the existence of a natural threshold and the grouping of similar individuals in modalities. Multiple factor analysis was then conducted to analyse the relationships between the variables and their modalities and to reduce the information under factorial spaces or axes. The choice of the number of factors was based on classical criteria (e.g. explained variance) and on the interpretation of factors, as recommended by Lebart et al. (1995) who attach great importance to the object of study. A principal component analysis enabled the research team to elaborate the complete model that integrates the factorial axes of each dimension. Using a hierarchical ascendant classification technique, the researchers could then regroup the organizations into homogeneous and differentiated categories (Figure 3). Two criteria were used to achieve the optimal number of partitions and categories: the first is the variance threshold criterion that is reached when there is no further significant increase in the ratio of variance between classes to the variance within classes; the second, the differentiation criterion, is met when no meaningful differences among categories are generated by further partition.

Figure 3.

Taxonomy of Organizations

The contextual component of this study was also examined using a configurational approach. A taxonomy of contexts was constructed, using the statistical techniques to group the organizations. Data were gathered from different sources: publications, data banks, administrative reports, census data. Variables obtained were demographic (age, mortality, etc.), level of resources and their utilization and collaboration among organizations.

Finally, the researchers evaluated the effects of the organizational models on care experience. Based on their mean score on each dimension, organizations were ranked. The analysis consisted in comparing the organizational models globally and specifically for each care experience index.

A complementary configurational analysis was done by constructing an index of conformity to an ideal type of PHC organization derived from the literature and various policy documents (Brunelle, 2006; Ministère de la Santé et des Services Sociaux, 2007). The index was based on 15 attributes distributed in the four organizational dimensions: vision, resources, structure and practices and expressed in percentage, 100 percent corresponding to the presence of all attributes in an organization. Data were presented for the total score and for specific scores by dimension. The index correlates with the taxonomy, the highest score of the index corresponding to the more complex models of the taxonomy.

Facing and Solving Methodological Challenges

Let’s now review the different methodological challenges encountered in the course of this research and see how concrete situations were handled, paying more attention to ‘less classical’ solutions to these challenges. In this regard, we will not explain how qualitative case studies rest upon theory building since this is true for all qualitative studies.

Theory building

As mentioned earlier, the organization models were mainly constructed according to a taxonomic approach (inductive). But the process actually involved both a typologic and a taxonomic approach (thus deductive and inductive). In addition, as a complementary approach, an ideal type was derived from the literature thus following a deductive approach. As mentioned earlier, conformity to that ideal type was expressed by an index measuring the percentage of attributes possessed by an organization.

The conceptual view of organizations is that of an organized system of action (Friedberg, 1993) consisting of individuals interacting with each other in social fields to mobilize and utilize resources in order to generate activities to pursue their collective project and attain their objectives. In operational terms, organizational models are configurations of attributes grouped around four dimensions: vision (values and representation), resources (quantity and variety of available resources), structure (rules and regulation governing actor’s behaviour) and practices (processes underlying the production of services). This conceptual formulation thus becomes the basis for constructing both the taxonomy and the ideal type, and thus has been used both in an inductive and in a deductive way.

The time horizon consideration

In the quantitative study, because of the cross-sectional nature of the design, consideration given to time horizon is less obvious, but still present. For example, context analysis extends back to the time period preceding the organization and population surveys. Similarly, interpretation of the results took into account events related to the changes brought about in PHC services before the surveys.

Integration of the different levels of analysis

Both studies have addressed this challenge. The quantitative one clearly establishes links between the organization and the population levels, namely through the nominal link between the two surveys. The context level also connects with the two other levels by looking at the performance of different models in the various contexts to see whether the relationships between organizational models and performance hold or change depending on the context. The qualitative component of the programme also integrates the different levels of analysis through Giddens’s structuration theory. This theory enables the researcher to examine several dynamics of action linked with individual actors, mainly through its core concept of ontologic security; with organizations throughout the concept of social integration that reflects the reciprocal relationships among actors in co-presence situations; and with the ecological level through the concept of systemic integration that refers in the study to the extent to which ‘physicians orient their practice in relation to population needs and have established values and functions of family medicine in a system of universal access to care’.

Delineating contexts

As mentioned earlier, delineating contexts is essentially interpretive. Since the research took place at the time of a reform that created CSSS, the research team opted for a concordance between their own delineation of territories and that dictated by the reform, that is the CSSS territories. Near 20 variables were retained to construct the taxonomy of contexts along the following dimensions: (1) demographic and socioeconomic variables, including health needs; (2) available resources on the territory; and (3) collaboration, clinical (internal) and interorganizational (external). Several of these variables assess the uncertainty of the activity in relation to users’ needs and the interdependence among organizations (Alter and Hage, 1993).

Synthetic view of the phenomenon

Relying on Giddens’ theory, and more particularly on his work on non-recognized conditions for actions and their unintended consequences, Geneau attempted to understand the complex non-linear web of factors that can influence medical practice. Likewise, in the quantitative study, correspondence analyses are aimed to reveal non-linear relationships among variables. Of course, as mentioned by Fiss (2007), taxonomic approaches, based on classification techniques, have certain drawbacks, mainly related to the fact that empirical clustering does not necessarily reflect causal relationship; the results largely depend on the nature of the sample, the choice of variables, etc. Giving consideration to these, we think that the juxtaposition of quantitative and qualitative approaches mitigates these drawbacks, since the results of quantitative analyses are equally interpreted in the light of the results of the qualitative analyses, thus strengthening the internal validity of the studies.

Some Results of the Research

Here are some results that emerge from the research (for more details, see Geneau, 2004; Pineault et al., 2008).

Medical practice: a complex phenomenon

The qualitative study has shed some light on the influence of certain factors on medical practice, more precisely on managing work time and schedules in medical service components, on served clientele and on acquiring and generating new knowledge (Geneau, 2004). Aside from usual factors, such as mode of remuneration, the study has revealed the importance of poorly documented factors, such as modalities of co-presence (patients’ attitude, contacts with other professionals). This research has also shown the complex and non-linear webs of causation and retroaction among factors. For instance, modes of remuneration may influence the type (e.g. with or without appointment) and the duration of a visit; but in the process of providing care, the degree of uncertainty in the flow of medical activities and in decisions concerning diagnostic tests and consultations to specialists becomes of paramount importance.

PHC organizational models: different forms of configurations of attributes

Pineault et al. (2008) have proposed taxonomy of five organizational models: four are professional and one is a community model. The four professional models (single provider, contact, coordination and integrated coordination) share the same objective to respond to the needs of patients who present or consider these clinics as their regular source of care. The community model has a broader, more inclusive objective, to meet the needs of the whole population for which the organization is responsible. Its governance is public in contrast with the professional model. The attributes of these models are summarized in Table 2.

Table 2.

Main Discriminant Characteristics of the Five PHC Models (Synthesis)

		Professional models
		Single provider (5)	Contact (1)	Coordination (2)	Coordination integrated (3)	Community model (4)
Vision	Responsibility	Clientele	Individuals who present	Clientele	Clientele/Population	Population/Clientele
Structure	Governance	Private - Professional				Public
	Integration	Low	Medium/Low	Medium/Low	High	High/Medium
Resources	Quantity and variety	Low	Medium	Medium	High	High
	Appointment/Walk-in	Mostly appointment	Mostly walk-in	Mostly appointment	Mixed	Mixed
Practice	Scope of services	Narrow	Narrow	Average	Wide	Wide

The contexts: different configuration of attributes

In the Pineault et al. study (2008), the 23 geographic (CSSS) territories were grouped according to the gap between the population needs and the available resources of the territory, the utilization/production of services (degree of use of outside services by the population and the degree of use of services produced by the resources of the territory and used by outside patients).

These characteristics of the contexts were validated with actors of the field who confirmed that this taxonomy made sense to them. Table 3 briefly presents these results.

Table 3.

Taxonomy of CSSS Territories

Rank or characteristic of context taxonomy models
Dimensions	Independent well off 1	Adequate coordinated 2	Low resources dependent 3	Affluent open market 4
•Adequacy needs-resources	1	0	−3	2
•Market openness	no	no	yes (import)	yes (import/export)
•Clinical collaborations	weak	high	weak	weak

The PHC organization model: prevalence and effects that vary by context

Organization models are distributed differently in the contexts. Professional, single provider and contact models are more prevalent in urban contexts. In rural contexts, professional coordination integrated models are more prevalent. Overall, the professional coordination integrated model performs best on most dimensions, whereas the contact model performs least well, even on accessibility. The single provider professional model is associated with the best care experience reported by the patients, presumably because of the quality of the patient–physician relationship it provides. This is particularly true in rural contexts, where the physician is closer to the patient and more integrated in the community. In this context, medical practice is also more diversified and better integrated. As shown by the Geneau study, physicians in these contexts combine different practice sites (office, hospital, emergency) and this fosters continuity of care. The performance of the community model is positive, as far as care experience is concerned, but relatively deficient with respect to productivity. Again, the qualitative study helps understand such results. According to Geneau (2008), these factors can explain this lack of productivity: mode of physician remuneration (salary), more severe cases and work rhythm of personnel over whom physician has no authority. The weak population coverage is probably due to the fact that, in urban areas, community organizations project a symbolic image associated with more severe cases and disadvantaged clienteles, thus reducing its attractiveness for other types of clienteles.

In sum, these examples illustrate the interest and the usefulness of a configurational approach to evaluation based on mixed methods, qualitative and quantitative. This approach yields rich information on the links between organizational models and performance.

Instead of stressing contradictions that seem to emerge from analysis, as in most studies, the configurational approach proposes solutions to solve problems of continuity of care and by the same token problems of accessibility, particularly for patients with chronic conditions. These objectives are better achieved by the coherent alignment of vision (priority given to continuity and responsibility), structure (coordination mechanisms between professionals), resources (namely presence of information technologies) and practices (namely formal links with a reference network). Given its added value, the configurational approach deserves to be more fully exploited.

Conclusion: Implications for Scientific and Decision-Making Communities

In this article, we have argued the merits of rethinking current approaches to the evaluation of service organization models. As we have been reiterating, we believe that configurational approaches allow the research community to better grasp complex phenomena. Classical methods for evaluating service organization models have paid insufficient attention to either sets of organizational attributes or sets of effects; in addition, they have tended to neglect the context in which an intervention takes place (Lamarche et al., 2006; Lomas et al., 2005). The configuration approach, however, is perfectly in line with current attempts to reinforce the theory and practice of programme evaluation, with the specific aim to bring greater rigour to the study of the complex linkages between interventions, their contexts and their outcomes. These attempts have resulted in an increased theorization of interventions in order to validate hypotheses on the links between the effects and the intervention (Brousselle et al., 2006, 2007, 2009); for example, through an analysis of the scientific literature. Similarly, ‘good practice’ in the field of evaluation requires analysing how interactions between interventions and their contexts impact observed outcomes (type 3 implementation analysis: Brousselle et al., 2009) through correlation analyses, case studies and other means. We believe that juxtaposing classification-type quantitative analysis methods and qualitative methods would advance these efforts to theorize interventions and their impacts.

In addition to its scientific merit, we are convinced that this approach also has practical value because of the compatibility of its global conception of service organization with the actual practices of managers and decision-makers. Having said this, we acknowledge that working towards this kind of approach is far from simple, given the major changes in perspective it would require on the part of both the scientific and the decision-makers’ community.

On the Need to Rethink the Scientific Evidence on the Organization of Services

The reasoning elaborated here has insisted at length on the linkages between the performance of service organization models and their surrounding context. As a result, it is clear that scientific evidence in this field cannot be other than contextual (Lomas et al., 2005). Because of this, practices based on these findings will diverge from the evidence-based medicine paradigm long rooted in a perspective that regards evidence as universal and absolute (Cookson, 2005). In the field of service organization, ‘contextual’ facts will help shed light on the following enquiries. What works? Under what conditions? And how can this solution be implemented in the present case and in other contexts? We have argued for public health research practices that go beyond randomized trials (Victora et al., 2004) that fail to establish the plausibility of inferences (linkages between outcomes and interventions; Habicht et al., 1999).

On the Need to Reconcile Research Traditions

Combining quantitative and qualitative methods is the best means of resolving the methodological problems inherent in the implementation of a configurational approach. To do so, however, is far from simple, given that numerous researchers see the two methods as paradigms founded on very different ontologies (definitions of what exists and constitutes reality), epistemologies (accounts of what we know and how we know it) and methodologies (the conduct and rules to follow in order to understand or discover an object). The multiplicity of approaches within each of these two paradigms further hampers open discussion. As might be expected, the ‘discipline’ of programme evaluation does not escape this cleavage (Dixon-Woods et al., 2005; Dubois and Marceau, 2005; Mays et al., 2005). Having said this, a growing number of authors have criticized this dualistic perception of research and instead advocate methodological pluralism (Bryman, 2006). In addition to philosophical barriers, this approach must also overcome practical and political obstacles (the need to meet the acceptance standards of scientific journals with strict research traditions; the influence of reviewers).

On the Need to Rethink Knowledge Transfer

There are two principal reasons for which a configurational approach to evaluating service organization models does not provide simplistic solutions to the problems facing policy-makers. First, to attain a given performance goal requires juxtaposing several features, an exercise considerably more complicated than ‘throwing a lever’. Second, performance also depends on the context within which an organization model evolves. Given these restrictions, it might seem that this kind of research will not easily develop. It is true, in fact, that the results of evaluation research do not lend themselves to an instrumental use of research findings (meaning that they will not constitute the input used to make a decision). They are instead better suited to either a ‘conceptual’ use, corresponding to an enlightenment function (Weiss, 1977), meaning that they will deepen understanding of the subject matter, or a ‘strategic’ use, meaning that they will help persuade various actors to collaborate in the effort to achieve a certain objective. The strategic use of knowledge pertains particularly to the field of service organization because organization models, seen as configurations, are never neutral: rather, as we have seen, they reflect a specific vision and therefore have a political dimension, bringing us back to the fact that in the field of management, ‘what counts as evidence and how it should be understood are never merely technical questions’ (Learmonth and Harding, 2006). Using findings to make complex decisions, then, takes place through a deliberative process (Lomas et al., 2005) in which the various stakeholders, including policy-makers and researchers, are confronted with different points of view. The hypothesis is that this process will result in more ‘rational’ and ‘effective’ decisions, assuming that there is some degree of symmetry between the viewpoints of the researchers and the policy-makers. This being said, it is important to bear in mind that because social systems are by nature open systems, the potential for transferring knowledge into practical use and therefore the cumulability of evaluation research are limited: ‘We can never know that what works today will always work since the conditions for things to work cannot ultimately be guaranteed, and the reflexive penetration of conditions for action and for choice by actors mean that their responses cannot mechanically be predicted.’ (Pawson and Tilley, 1997: 150–1)

Biographies

NASSERA TOUATI, PhD, is professor at École Nationale d’Administration Publique (ENAP) and researcher at the Groupe de Recherche Interdisciplinaire en Santé of the Université de Montréal (GRIS).

RAYNALD PINEAULT, PhD, was nominated Emeritus Professor at the Université de Montréal in September 2003 and is currently pursuing his professional activities at the Direction de santé publique in Montreal.

FRANÇOIS CHAMPAGNE, PhD, is professor at the Université de Montréal and researcher at the Groupe de Recherche Interdisciplinaire en Santé (GRIS).

JEAN-LOUIS DENIS, PhD, is professor at the Université de Montréal and researcher at the Groupe de Recherche Interdisciplinaire en Santé (GRIS).

ASTRID BROUSSELLE, PhD, is researcher at the Department of Health Administration of Université de Montréal and at the Groupe de Recherche Interdisciplinaire en Santé (GRIS).

ANDRÉ-PIERRE CONTANDRIOPOULOS, PhD, is professor at the Université de Montréal and researcher at the Groupe de Recherche Interdisciplinaire en Santé (GRIS).

ROBERT GENEAU, PhD, is professor at the University of Ottawa and researcher at the Public Health Agency of Canada.