Tackling the wicked problem of health networks: the design of an evaluation framework

Abstract

Networks are everywhere. Health systems and public health settings are experimenting with multifarious forms. Governments and providers are heavily investing in networks with an expectation that they will facilitate the delivery of better services and improve health outcomes. Yet, we lack a suitable conceptual framework to evaluate the effectiveness and sustainability of clinical and health networks. This paper aims to present such a framework to assist with rigorous research and policy analysis. The framework was designed as part of a project to evaluate the effectiveness and sustainability of health networks. We drew on systematic reviews of the literature on networks and communities of practice in health care, and on theoretical and evidence-based studies of the evaluation of health and non-health networks. Using brainstorming and mind-mapping techniques in expert advisory group sessions, we assessed existing network evaluation frameworks and considered their application to extant health networks. Feedback from stakeholders in network studies that we conducted was incorporated. The framework encompasses network goals, characteristics and relationships at member, network and community levels, and then looks at network outcomes, taking into account intervening variables. Finally, the short-term, medium-term and long-term effectiveness of the network needs to be assessed. The framework provides an overarching contribution to network evaluation. It is sufficiently comprehensive to account for many theoretical and evidence-based contributions to the literature on how networks operate and is sufficiently flexible to assess different kinds of health networks across their life-cycle at community, network and member levels. We outline the merits and limitations of the framework and discuss how it might be further tested.

Strengths and limitations of this study.

• With the prevalence of network forms and the high level of society’s reliance on and investment in them, evaluation models to assess their effectiveness and sustainability are needed.

• This paper reports on progress in evaluating networks and presents an evaluation framework designed for health settings.

• The framework is informed by systematic reviews of health networks and literature on existing frameworks that have been applied to the evaluation of health and non-health sector networks.

• Brainstorming and mind-mapping techniques were used in expert advisory group sessions to assess existing frameworks, to consider how they might be applied to extant health and clinical networks, and to identify any gaps.

• The new evaluation framework may assist health researchers, social scientists, policy-makers and managers with the evaluation of clinical and health networks.

• A limitation is that, although this framework has been applied in our health and clinical network evaluations, there is a need for wider testing of the applicability and validity of the framework.

Background

Since the emergence of network science last century as a putative solution to the way people are organised, many countries have introduced network forms of governance in their health systems.¹ Networks are said to be compatible with principles of devolved governance and decentralisation of services.^2–4 It is claimed that they can be conceptualised as ‘postbureaucratic’ entities where the standard bureaucratic forms are hollowed out.³ These forms were implemented to solve ‘wicked’ public problems not amenable to simple solutions,^{5 6} addressing such issues as how health, social care and education can be more effectively configured and provide improved services to consumers and customers.^{7 8}

In healthcare, many kinds of networks have emerged from loose forums to share information and experiences to middle-range collaborations and partnerships, to tightly defined and integrated organisational forms. No level of healthcare—from macrosystems to microsystems—remains unaffected from deploying, funding or mandating network structures.

Health systems have supported networks for improved relationships, interprofessional service delivery and to give effect to complex policy requirements,^9–11 as structures to support quality improvement¹² and to support implementation activities.¹³ Networks also manifest naturally in health systems through workplace, professional and social associations. The term ‘network’ is used extensively in healthcare research and in health services delivery. It is often used as a synonym for ‘partnership’, ‘collaboration’, ‘alliance’ and ‘group’. Or, it may be used more specifically to describe the structure of relationships between people, groups or organisations, joined together through nodes and ties.

Various scholars from differing traditions have researched or discussed how networks operate.^14–18 Provan and Kenis list the advantages of network coordination as: enhanced learning, more efficient use of resources, increased capacity to plan for and address complex problems, greater competitiveness and better services for clients and customers.¹⁹ Nevertheless, they identify a discrepancy between the acclamation and attention networks receive and the knowledge we have about their overall functioning. The process by which certain network conditions lead to various network-level outcomes is unclear. In spite of the burgeoning literature on networks, most have been descriptive (eg, work by Agranoff and McGuire,²⁰ Huxham and Vangan²¹ and van Bueren).²² Braithwaite et al contend that this also applies in the health sector, with claims for the effectiveness of networks and communities of practice tending to be theoretical or conceptual rather than empirically well grounded.²³ Also, relevant to the measurement of effectiveness of networks is the argument posited by Herman and Renz in the literature on organisational effectiveness, that the measurement of effectiveness of organisations should not be seen as an objective reality.²⁴ Rather, effectiveness is a social construction, an achievement of organisational agents and other stakeholders in convincing each other that an organisation (or network) is pursuing the right objectives in the right way. We also note the wide theory of change literature elaborating effectiveness from the stakeholder perspective, linking actions to outcomes and what an organisation or network can expect.²⁵

The issue of how to evaluate networks has received the intermittent attention of social and behavioural scientists. With the prevalence of network forms, and society’s reliance on and investment in them, we need evaluation models to assess their effectiveness and sustainability. Securing an evaluation framework which could be applied to assessing how well health professional networks perform and how they could be organised to work more effectively is a signature problem.

Evaluating networks to date

There have been multiple reviews of social and organisational networks outside of the health literature.^26–30 Borgatti and Foster developed a set of dimensions along which network studies vary, including direction of causality, levels of analysis, explanatory goals and explanatory mechanisms.²⁶ The two latter dimensions were used to construct a 2-by-2 table cross-classifying studies of network consequences into four types: structural social capital, social access to resources, contagion and environmental shaping. The leadership network classification framework of Hoppe and Reinelt categorises four leadership network types, providing a tool for network analysis to assist in strengthening, using and evaluating leadership networks effectively.³¹ Other authors such as Owen-Smith and Powell argue for going beyond a narrow application of social network analysis (SNA), highlighting the importance of the context and settings of networks.³²

According to Provan and Milward, empirical researchers have employed network analysis techniques to understand how agencies coordinate and integrate their activities, but they have tended to emphasise differences in network structures and governance.¹⁸ Examinations of the relationship across network structures and activities and measures of effectiveness have been lacking. Turrini et al remark on the fragmentation of the literature on determinants of network effectiveness, with a plurality of definitions, multiple theories, multiple methods and multiple explanations.⁷ Thus, despite the growth in research on network effectiveness since the early 1990s, many authors note the lack of widely accepted theories about network effectiveness and its determinants.

There have been several systematic reviews of health professional networks and collaboratives.^33–35 Cunningham et al’s review found that creating cohesive, collaborative networks (of professionals or agencies) could pay dividends in coordinating care and attending to quality and safety issues and agendas, and positively functioning networks were likely to contribute more broadly to an effective organisational climate.³³ However, network vulnerabilities can include cliques, professional and gender homophily and over-reliance on central agencies or individuals. Only one-third of the studies reviewed by Cunningham et al linked network structure with evidence of outcomes. There was a need for further well-designed research examining the relationships between professionals’ network structures and health outcomes in a range of different care settings.

In related work, the systematic review of the public affairs literature by Varda et al found that network structure affects governance, management strategies of administrators and effective collaboration.³⁴ In a systematic review assessing the effectiveness of one type of health network, quality improvement collaboratives (QICs), in improving the quality of care, Schouten et al found that the evidence underlying quality improvements was positive but limited.³⁵ The authors concluded that further knowledge of effectiveness, cost-effectiveness and success factors was crucial to determine the value of QICs. In their review of the literature on QICs, Dückers et al state, ‘the problem is that despite its popularity, the evidence for QIC effectiveness is positive but limited’.³⁶ They call for additional research to test the effectiveness of QICs as a spread agency and to examine their sustainability. Identifying the need to develop methodologies to analyse the impact of integrated healthcare networks in Latin America, Vázquez et al developed an evaluation framework, taking into account context, process and intermediate and final outcomes.³⁷ However, they note that the benefits and risks of such networks have scarcely been evaluated.

All in all, the literature evaluating networks indicates that more work is needed to demonstrate effectiveness, and to identify factors related to improved outcomes. It suggests that no widely accepted evaluation framework has yet emerged. The aim of this paper is to propose a framework to evaluate the effectiveness and sustainability of clinical and health networks for the purpose of rigorous research and policy analysis. Our work is part of a larger study to progress the development of measures and tools for a framework to evaluate health networks and communities of practice.²³

Methods

Procedure

Prior to this developmental study, we conducted and published several systematic reviews on networks of health professionals and communities of practice, with a particular focus on quality improvement.^{33 38–40} We also reviewed the wider literature to locate existing frameworks that have been applied in the evaluation of health and non-health sector networks. Informed by the literature, we used brainstorming and mind-mapping techniques in expert advisory group sessions to assess existing health and non-health sector evaluation frameworks (particularly in relation to the organisational studies literature on public sector administration and evaluation of organisational performance), and to consider how they might be applied to extant health and clinical networks. Three expert advisory group sessions were convened to review extant literature and frameworks and to develop the new framework. All members of our expert advisory group were at that time with the Australian Institute of Health Innovation which is now located at Macquarie University, Sydney. The group included extensive expertise in: health services and health systems research in Australia and internationally; in health sector quality improvement; in organisational studies and clinical and health professional network studies in the health sector; and, in multimethod health sector evaluation studies. Members had professional backgrounds covering research and evaluation, education, medical practice, executive health management in public and private health sectors, and health policy work. All had both Australian and international experience in the health sector. Our approach also took into account feedback from stakeholders in network studies that we have conducted.^{41 42}

Drawing from work to date in evaluating health networks

Recognising that evaluating organisational network effectiveness is complex and has generally been neglected, Provan and Milward developed a benchmark framework, from a study of mental health provider networks, for evaluating public sector organisational networks.¹⁸ In presenting this framework, the authors argue that networks must be evaluated at three levels of analysis: community, network and organisation/participant. According to Turrini et al,⁷ this framework, subsequently expanded by drawing from work by Provan and Sebastian,⁴³ remains a milestone in the research about the determinants of a network’s overall effectiveness, paving the way to subsequent insights and further conceptualisation of network effectiveness.

As networks have pressures to perform effectively from a broad range of stakeholders, these authors argued that networks must be evaluated at three levels of analysis: community, network and organisational/participant levels. Each of these levels needs to have its own set of effectiveness criteria that must be considered. Milward and Provan looked at network structural characteristics, finding that the relationship between network structure and network effectiveness was mediated by network context (eg, network resources and network stability).^{3 44} Adding to this, Provan and Kenis explored the impact of governance and the role of management on network effectiveness.¹⁹ Building on the work of Kenis and Provan,⁴⁵ Lucidarme et al examined the combined influence of different determinants, and found that environmental, structural and managerial determinants were associated with network effectiveness in a comparative study of health promotion networks.⁴⁶

Turrini et al refined and extended Provan et al’s benchmark framework.⁷ They reviewed and classified theoretical and evidence-based studies on network effectiveness and its determinants to develop their evaluation framework. Their model proposed five dimensions of network effectiveness: (1) client-level effectiveness; (2) overall community-level effectiveness; (3) ability to reach stated goals (the network is more likely to survive if it can achieve the key tasks or goals it has been set and has set itself); (4) capacity for innovation and change and (5) sustainability and viability. The first two dimensions are indicators of external impact and the last three are indicators of internal capacity building performance. The Turrini et al approach was applied by Ferlie et al in their evaluation of National Health Service (NHS) clinical networks.⁶

Subsequently, Bergenholtz and Waldstrøm added that, at a minimum to explain performance, the analytical approach must involve ‘longitudinal, multiplex data and whole network conceptualisation that goes beyond ego-networks and includes overall network structures’.²⁹ The issue of partnership synergy—the mechanism that enables partnerships to achieve more than individuals and organisations on their own—is identified by Lasker et al and they provide an assessment framework.⁴⁷ The practical tools and strategies developed by Woodland and Hutton to assist evaluators with operationalising and assessing the related construct of collaboration may contribute to network evaluation.⁸ Various authors have contributed to our understanding of how network analysis tools can be applied to explore network topology, connectivity and diffusion of innovations.^48–51

Specific guidance addressing the unique challenges with designing, implementing and evaluating large-scale QICs is proffered by Goeschel et al with their logical framework approach and associated tools.⁵² Britto et al have described how the network model has supported their common framework and methods for their Learning Networks in healthcare systems, and they identify the need for attention to the underlying organisational architecture of such networks.⁵³ However, challenges in evaluating large-scale collaboratives, as evidenced, for example, by the US Veterans Health Administration’s ‘FIX’ QIC, implemented in their 130 hospitals, include: (1) the issue of fidelity to the collaborative and its interventions, (2) understanding barriers and facilitators and their impact on short-term and long-term success in achieving goals and (3) understanding the impact of other simultaneous initiatives.⁵⁴ Nadeem et al in their systematic review of QICs found that it was impossible to identify active QIC ingredients linked to improved care, as reporting on specific components of the collaborative was imprecise across studies.⁵⁵ They identified the need for further controlled research examining the core components of these collaboratives related to patient-level and provider-level outcomes.

Patient and public involvement

This paper is a conceptual development of a framework to evaluate health networks. There was no patient or public involvement in setting the research question, in designing the study, the conduct of the study or the interpretation of the results.

Results: an evaluation framework for health networks

Adding relevant information from systematic reviews33 34 38–40 56 and related network and collaborative research29 52 55 57–60 to Provan and Milward,¹⁸ and Turrini et al,⁷ we embraced a three-level approach in our model: at the community level, the network level and the member level. We accepted the three additional dimensions of internal capacity building performance proposed by Turrini et al,⁷ and endorsed by Ferlie et al ⁶ (ability to reach stated goals, capacity for innovation and change, and sustainability and viability). Stakeholders in our studies recognised the temporal aspect of measuring network effectiveness and the need for appropriate measurement for the network’s development stage.⁴¹ Therefore, our framework addresses how evaluation can be applied at the three levels proposed by Provan and Milward, while recognising the three additional dimensions of Turrini et al across the network life-cycle, as identified by Mandell and Keast⁶¹ and Kenis and Provan.⁴⁵ We also included network governance as a determinant of network effectiveness as this dimension was recognised in the literature but underplayed in prior evaluation frameworks. In addition, we identified through the literature a range of ‘intervening variables’ to include in our model: leadership, network management, communication strategies and interprofessionalism. Figure 1 presents the evaluation framework.

Figure 1.

Evaluation framework for clinical and health networks.

The framework begins by asking the evaluator to assess the network goals by gathering information about its characteristics (type of network), to take into account the role of network synergy or antagony, and to provide a summary of network relationships at three levels. At the community level, the effectiveness of the network can have an impact on services to community members. This is important for policy-makers, and for governments who provide and fund networks, in terms of value for money. In addition, those who represent these clients, such as consumer lobby organisations, must be satisfied by the network’s activities, in addition to the broader public. Networks can also be evaluated at the community level in terms of their contribution to building social capital.⁶² The cooperation and collaboration among members engendered by the network may benefit the community in ways that would not have been possible if no social capital had been created and maintained.¹⁸ Some scholars distinguish between the client and the community level to elaborate the difference between more short-term (better individual health) and long-term results (community resilience or social capital).

At the network level, if a network is to be effective in achieving its objectives, it must become a viable entity and sustainable. For Provan and Milward, to operate effectively individual members must act as a network, and this means incurring organising and transaction costs.¹⁸ Some of these costs will be borne by individual members. However, for networks established by government bodies such as in the UK’s NHSs, network establishment and maintenance are led, coordinated and governed by a central, local administrative entity. This entity is referred to as a network broker by Lawless and Moore⁶³ and Mandell.⁶⁴ As the disseminator of funds and resources, administrator, and coordinator of the network, this type of broker is both the agent of the community and the principal of the network participants: this needs to be factored into the framework. Another measure of network effectiveness is the attraction and retention of members. Network outputs provide a further measure, along with the efficiency of information flows in the network, and the strength of relationships among members across the network.

At the participant or member level, network affiliation may or may not accrue benefits for the member. These may include gaining new knowledge, facilitating collaboration, professional acknowledgement and collegiate support. Through network involvement, members may also change their own practice and effect changes in their organisations.

How these relationships feed into network outputs and outcomes, taking into account the specified intervening variables, must also be considered. Researchers will need to address questions about how to distinguish between outputs (what the network does), outcomes (what immediate effects are visible) and impact (what sustainable changes occur). They may decide to involve those participating in the network in defining effectiveness. From our review of the literature, and from our evaluations of health networks, we suggest that a number of factors should be considered. For networks in the quality improvement area, network outputs and outcomes could include: (1) development of models of care; (2) implementation of models of care; (3) establishment of trials, new services, changed practice, redesign projects; (4) education and training of health professionals, consumers, carers; (5) development of consumer and carer materials; (6) raising the profile of the disease/clinical condition; (7) information dissemination (eg, newsletters, website); (8) journal publications, conference presentations; (9) increased resources for the disease/clinical condition and (10) measurable improvement in health outcomes. Intervening variables could include factors relating to (1) leadership; (2) network management; (3) communication strategies and (4) interprofessionalism. Finally, temporality must be taken into account: an assessment of the network’s short-term, medium-term and long-term effectiveness should be made.

Discussion: how might the framework be applied

In this paper, we present a new framework that can be applied to evaluating how well health networks are performing and to assessing how they can be organised to work more effectively. By reconciling key findings across disciplines, especially from the organisational studies literature on public sector administration and health sector literature, with an expert-informed framework, we have taken a step forward to aid evaluators of health networks. Although researchers in various fields have studied the problem of how to evaluate the effectiveness of networks, this knowledge has not been harnessed across research disciplines to provide a framework that can be applied to evaluate health networks.

This framework can help with providing a shared understanding of key domains and inputs for network evaluation for administrators, funders and researchers working with a range of different types of networks in the health sector. The framework responds to calls to address the complex challenge of evaluating the effectiveness of health networks. With increased international public and private funding of health network structures and reliance on such structures in the 21st century, it is important to have a robust framework to assist in their evaluation. The framework presented here explicitly focuses on key domains associated with network effectiveness, as it identifies organisational and contextual characteristics that we hypothesise are necessary aspects to be taken into account in assessing network effectiveness.

We have attempted to be both comprehensive and flexible in designing our framework. In applying the framework to the evaluation of health networks, consideration needs to be given to the underlying research methods, tools and measures needed to produce an assessment. By way of example in the measurement of performance of one type of health network, integrated healthcare networks, Vázquez et al identify a number of methods and tools, and sources of material, that researchers could apply.³⁷

Approaches to evaluating health networks need to select appropriate methodologies and need to be aware of the types of data that are required for the evaluation. For example, in their review of eight English managed clinical networks, Ferlie et al employed a qualitative approach to assessment.^{6 65} However, they noted methodological difficulties in evaluating network performance, identifying the need to complement qualitative data with more quantitative data—perhaps clinical outcome data. In another study of English clinical networks, Currie et al used mixed methods (SNA and qualitative fieldwork) to examine the current status and potential for leadership agency and knowledge management to transcend institutional hurdles in network functioning.⁶⁶

In previous evaluations of clinical and health networks, researchers have employed a range of mixed methods. For example, to evaluate clinical networks, we have used qualitative fieldwork, including stakeholder and member interviews to assess perceptions of the key factors relating to network effectiveness.⁴¹ We have also employed an online survey to collect network member demographic, performance assessment and network data (for the purpose of SNA). Similarly, mixed methods have been applied to the evaluation of the role of brokers in a translational cancer network.^{42 67}

The approach we have taken in this study has a number of strengths. We drew from a foundation of several literature reviews that were wide in scope, from an examination of existing network evaluation applications, from our own experience in evaluating a number of different health and clinical networks, and we also reviewed a range of different health sector framework constructions. Experts working in, and with background in, diverse health settings informed the mind-mapping process. Finally, our framework was informed from feedback received from participants in several health network studies that we have conducted.

While recognising these strengths, there are also a number of important limitations to this study some of which suggest the need for future research. Although mind mapping is a useful and flexible research tool, it is reliant on expert opinion. In addition, we acknowledge that all models and frameworks are simplifications of the real world, and thus can fail to capture real life messiness and complexity. Although this framework takes into account recent literature reviews and currently available assessment models there may be other literature and models we have missed. Although we have applied the framework in our health and clinical network evaluations, there is a need for wider testing of the applicability and validity of the framework to show that the various domains (and their interactions) are all important in the evaluation of the effectiveness of health networks.

Conclusion

This paper reports on progress in evaluating networks and presents an evaluation framework designed for health settings. With the high level of societal investment in health networks, it is timely to provide a framework to address the need for evaluation models to assess the effectiveness and sustainability of network forms in the health sector.

We now release this framework as an overarching contribution to network evaluation. We encourage further research to test the utility of the framework, and its reliability and validity, using comparative research designs to assess network effectiveness of different types of networks in different contexts and settings.

The conceptual framework presented here may assist health researchers, social scientists, policy-makers and managers with the evaluation of clinical and health networks in different settings. It is designed to inform our understanding of factors associated with their effectiveness and sustainability.