Conceptualization and measurement of integrated human service networks for evaluation

Abstract

Introduction

Integration has been advanced as a strategy for the delivery of a number of human services that have traditionally been delivered by autonomous agencies with independent processes and funding sources. However, measurement of the dimensions of integration has been hampered by numerous factors, including a lack of definitional and conceptual clarity of integration, and the use of measurement tools with atheoretical foundations and limited psychometric testing.

Theory/methods

Based on a review of integration measurement approaches, a comprehensive approach to the measure of multiple dimensions of integrated human service networks was conceptualized. The combination of concepts was derived from existing theoretical, policy, and measurement approaches in order to establish the content validity and comprehensiveness of the proposed measure.

Results

The dimensions of human service integration measures are:

(1) Observed (current) and expected structural inputs, or the mix of agencies that comprise the network (e.g. extent, scope, depth, congruence within an agency, and reciprocity between agencies).

(2) Functioning of the network both in terms of the quality of the network or partnership functioning and ingredients of the integration of the networks' working arrangements and range of human services provided.

(3) Network outputs in terms of network capacity (e.g. what is accomplished, for how many and how quickly given the local demand) measured from dual perspectives of the agency and the family.

Conclusion

This newly developed measure unites multiple perspectives in a comprehensive approach to the measurement of integration of human service networks. Content validity has been established. Future work should focus on further refinement of this instrument through psychometric evaluation (e.g. construct validity) in diverse networks and relating these measures of network integration to client and system outcomes.

Introduction

Networks of integrated human services have emerged as one way of coordinating currently autonomous services for populations with complex needs. The autonomous service agencies have independent governing processes, mandates and funding sources. The human service network is a strategic alliance between human service agencies with its own governance and service agreements for a target population with complex needs. The integration of human service networks has been viewed as a key strategy for overcoming resource deficiencies and gaining credibility for those who need comprehensive services [1].

There are a number of deterrents that hinder the measurement of the degree of human service integration [2] including: (i) a lack of definitional and conceptual clarity of integration [3] with (ii) the use of interchangeable terms such as coalitions, partnership, collaboration and integration (iii) atheoretical approaches found in the majority of existing tools [2, 4] (iv) with limited psychometric evaluation, and finally (v) a focus on process issues with inadequate attention to structural components of the network.

As Provan and Milward [5] note, “the study of health service delivery networks needs to move from a general discussion of what they are and why they are useful to empirical measures of the components of integrated networks and their interrelationships”.

There is an early consensus in the integration literature that a comprehensive measurement approach that considers multiple dimensions, components and perspectives on integrated networks is needed [2, 4].

All of this measurement work must precede any attempt to measure network effectiveness in terms of client outcomes.

Purpose

The purpose of this paper is to conceptualize four distinct dimensions of integrated human service networks and to suggest that various reliable and valid measures of coalitions, partnerships, and collaborations are distinct dimensions of a more comprehensive view of integrated human service networks. Collectively, these dimensions are supported by theoretical and human service policy criteria.

For the purpose of this paper an integrated human service network is a coalition or strategic alliance between appropriate agencies from multiple sectors (social, health, education) or funding sources (public, not for profit, private) that together collaborate and function to provide a continuum and spectrum of comprehensive services and opportunities for people of various ages with complex needs.

Background: model development

Theoretical approach

The theoretical framework for the proposed model is based on a compilation of work accomplished during the past decade by Provan et al. [1, 5, 6]. Based on organization and agency-theory, Provan and Milward [6] were the first theoreticians to consider criteria that evaluated the network at multiple levels, including the network, its member agencies and clients, and the community, and to integrate this concept with stages of growth of the network (Table 1). As such, these criteria provide theoretical guidance for the evaluation of network levels and growth of dimensions of the network. Furthermore, Provan and Milward's criteria support the importance of the existence and evaluation of an overall network administration organization, without which equitable distribution of resources may not occur and, in fact, whose absence may propagate a system of resource competition and service duplication, rather than cooperation.

Table 1.

Questions related to the measurement of human service network integration: an alignment of theoretical, political, and empirical perspectives

What should we measure? A theoretical approach	Why should we measure this? A policy approach	How should we measure this? A measurement approach
Criteria Levels of Network Analysis [1, 6]	Human Service Policy Criteria [8]	Comprehensive Measure of Integration [12]
Membership growth: • Effective mix of core (critical) and peripheral services Relationship strength (multiplexity)	Appropriateness: (To evaluate appropriateness) Are the appropriate number and type of services involved given the NEED and CONTEXT (geographic, legislative)	Structure of Network: Current and Expected – • Extent • Scope • Depth • Congruence • Reciprocity [12]
Integration/coordination: Absence of duplication Range of services (rather than only number of services) Minimum conflict Member commitment to network goals	Acceptability: To providers To recipients' culture (processes by which services are provided)	Function of network: • Quality of • Value of [13]
Administrative structure: Cost of network maintenance Creation and maintenance of network administrative structure Resource acquisition and distribution Enhanced legitimacy	Accountability: • Governance • Agreements • Monitoring	Ingredients of integration: Governance, common record, case coordination, etc. [14] service agreements
Service access: Network capacity (“ability to develop and sustain action to improve health”) [1]	Accessibility: Location Intake assessment and eligibility Adequacy: Wait list/times Equity: Horizontal  – Similar needs  – Similar resource Vertical  – Different needs  – Different resources	Network system capacity: • Wait times/lists; client volume • Joint programs [1]

In 2004, Provan et al. [1] expanded their existing criteria by the addition of the concept of network capacity, the “ability to develop and sustain action to improve health”. The importance of this concept lies in the notion that capacity is viewed as a transferable entity [1, 7] in that the skills and relationships that develop in solving one health issue can be applied to a different one. Provan et al.'s work contributes to the proposed model by providing a framework for understanding “what” to measure in assessing the dimensions of human service network integration.

Policy approach

Based on an expanded version of Maxwell's [8] framework designed to evaluate the quality of health services, Thomas and Palfrey [9] developed criteria for evaluating the performance of public sector organizations from various perspectives, including clients, providers, and decision-makers (Table 1). Suggesting that each group of stakeholders is likely to emphasize and value certain criteria, Thomas and Palfrey acknowledge that singular perspectives and criterion are unable to capture all the dimensions of a human service network. In terms of relevance for the proposed model, Thomas and Palfrey's criteria may be viewed as supplying a framework for the goals of the service network, including services that are (i) appropriate given the needs of the target population, (ii) acceptable, (iii) accessible, (iv) accountable, and (v) adequate in terms of wait times.

Measurement approach

Quantitative measures for the evaluation of these multiple dimensions of network integration have been developed and serve as general measures for the assessment of integration across diverse forms of networks [10, 11]. The proposed model represents an elaboration of the conceptualization of human services integration that led to the development of the Human Services Integration Measure [10]. The Human Services Integration Measure facilitates measurement of integration across a continuum of care within a service sector (assessment, early intervention and treatment), across service sectors (health, social, education), and between public, not-for-profit and private sectors of financing services [10]. It quantitatively measures the scope and depth of integration for each sector and service in a network, and its recent expansion includes the measurement of the actual or observed versus the expected extent, scope and depth of integration of services within a network [12].

Description of proposed model

Assumptions of proposed model

A recent review of 146 integration tools [4] concluded that existing measurement tools represent a fragmented collection of instruments that lack theoretical foundations, rigorous psychometric evaluation, and comprehensive approaches. This plethora of existing tools and the lack of consensus regarding the conceptualization and measurement of network integration suggest that a comprehensive approach which acknowledges that “everyone has captured some of the truth” should be the foundational assumption of the proposed model. A second, key assumption of this model is that integration measurement models continue to proliferate because foundationally each individual model is incomplete, and lacks comprehensiveness and content validity.

An alignment of approaches

A systems approach to measuring network integration suggests the use of a variety of measures in order to address all of the health services policy criteria and network organizational theoretical criteria for assessing network function [12]. These system or clinical dimensions of Human Network Service Integration found in Table 1 and Figure 1 are not captured by any one published measure alone. An alignment of theoretical, policy, and measurement criteria (Table 1) reveals that concepts and criteria arising from these different perspectives are largely consistent and can guide the development of a more comprehensive approach to measuring human service network integration.

Figure 1.

Dimensions of human service network integration to be measured [12].

Specifically, the levels of analysis provided by Provan and Milward [6] can be aligned with Thomas and Palfrey's [8] health service policy effectiveness criteria (Table 1). Thomas and Palfrey's criterion of appropriateness, acceptability, accessibility, accountability, are consistent with Provan and Milward's concepts of network membership, coordination/absence of duplication, accessibility, and network maintenance/administrative structure, respectively. In addition, the dimensions of the Comprehensive Measure of Integration (e.g. structure, function, and ingredients of integration) align with both of these approaches.

Description of proposed model

Based on the alignment of theoretical, policy, and measurement criteria for evaluating network function (Table 1), dimensions of human service network integration that require measurement are conceptualized as: (1) observed (current) and expected structural inputs, or the mix of agencies that comprise the network (e.g. extent, scope, depth, congruence within an agency, and reciprocity between agencies); (2) functioning of the network both in terms of the quality of the network or partnership functioning and ingredients of the integration of the networks' working arrangements and range of human services provided; and (3) network outputs in terms of network capacity (e.g. what is accomplished, for how many and how quickly given the local demand) measured from dual perspectives of the agency and the family (Figure 1). Each of these structure, function and capacity dimensions of service network integration are distinct aspects of integrated service systems and are similar to Donabedian's [15] recent reference to his classical concepts of structure, process and outcome when evaluating the quality of health care. The proposed model (Figure 1) guides the rationale for their collective use. Specific measurement tools to capture these dimensions are also suggested in Figure 1.

Each dimension represented makes an independent contribution to the description and measurement of integrated human service networks. For example, an integrated service network could have a high quality of functioning in terms of leadership, efficiency, and synergy, but could at the same time be limited in extent and scope as in the number of types of services from different sectors and/or funding sources. Neither a measure of human service network structure [10] nor a measure of quality of human service network function [13] captures the assessment provided by the other measure. Similarly, any network structure and the quality of its function could be deficient in having the necessary funding and accountability arrangements, or ingredients or processes of providing/recording services (e.g. common intake or electronic record) necessary to minimize duplication [14]. Finally, a network may have a high level of inter-sectoral structure of member agencies, a high level of functioning, and a number of ingredients including funding for the work of integration, but still be deficient in network capacity in terms of the volume of clients served, size, and length (in time) of the waiting list. While Figure 1 appears to be linear, feedback loops are added to illustrate how each dimension of the network can affect the other dimensions.

Discussion

The proposed model provides an approach to measuring human service network integration based on existing theoretical and policy criteria for the evaluation of the network structure, functioning and capacity resulting from integration. The model structure depicts Donabedian's [15] concepts of structure, process and outcome when evaluating the quality of medical care. Aligning integration evaluation criteria from theoretical and policy arenas, and building on the author's previous measure of integration, this model is unique in that it represents an amalgamation of existing criteria in a conceptual framework that has direct utility for measurement of human service network integration. Content validity for the proposed measure of Human Service Integration is established in that the incorporation of existing criteria suggests that the measure “adequately covers the domain under investigation” [16]. It should be noted that although examples of instruments measuring integration structure, function, and output have been given (Figure 1), others could be suggested as alternatives depending on the context.

Based on Provan and Milward's [6] concept of analysis of the network at various levels, the proposed measurement model is designed to be administered at any level of a network, including the client, front-line staff, and management. Different results can be expected from different respondents, and are valued as multiple perspectives of the network from diverse stakeholders. This is a critical aspect of addressing network issues. However, if a discrepancy exists in observed or expected depth of integration between levels of respondents, the view of the frontline worker is viewed as representative of what is happening at the service delivery level.

Although this model is a measure of present integration structure, function, and output, and therefore present capacity, it can be used to guide evaluations of future network capacity and sustainability. In order to ensure sustainability of the network, structures and processes related to funding and accountability must be in place, including budgets and designated ‘integrators’. Future needs of the network could be assessed by ongoing measurement of trends in waiting list length and times, growth in target populations, the growing needs that accompany survival of those with complex conditions (e.g. AIDS), and the comparative cost of not addressing such needs, and by establishing strategic alliances with other community groups (e.g. not-for-profit or private groups) that offer the opportunity to pool resources.

Given that the content validity of the proposed measure has been documented, future efforts should focus on the assessment of construct validity of this measure in diverse networks. Based on a priori hypotheses of network structure, function, and output, psychometric evaluation of the construct validity of the tool will contribute to our understanding of how well the dimensions of integration combine to predict the hypothesized system and client level outcomes, and guide further development of the model and its analysis.

In addition, this tool should be used in research studies to evaluate fidelity to and the full impact of integrated services. For example, an emphasis on integrated services may unwittingly erode the value of client (family) autonomy and empowerment that can result from choosing between alternative sources of care. Thus, it is necessary to evaluate the full impact of the integrated approach to service versus a client (family's) self directed approach to obtaining service. At present, this is being evaluated in a Randomized Trial of the use of these dimensions of integration in a Children's Disability Treatment Network compared to usual self directed care with the objectives of determining (i) what are the characteristics and circumstances of individuals who benefit from the integrated approach, and at what cost, compared to (ii) who does well with self-directed care? Finally, this approach to measuring network integration can be related to measures of effectiveness from client, family, service agency and system perspectives and can be analyzed longitudinally to document change over time. Such future work would help establish the relative importance of each of these dimensions to client and system outcomes.

Reviewers

Helen Dickinson, Lecturer in Healthcare policy and management, Health Services Management Centre, School of Public Policy, University of Birmingham, Birmingham, UK.

Jean-Claude Henrard, MD, Emeritus Professor of Public Health and former Director Health and Ageing research unit Versailles-Saint-Quentin-en-Yvelines University and Centre of Gerontology, Sainte Périne Hospital, Paris, France.

John Ovetvreit, Professor and Director of Research, The Karolinska Institute Medical Management Centre, Stockholm, Sweden and Professor of Health Policy and Management, Bergen University Faculty of Medicine, Norway.