Monitoring the degree of implementation of an integrated delivery system

Abstract

Introduction

The aim of the study was to develop a method to measure the implementation of specific components of an Integrated Service Delivery system for the frail elderly. The system includes six mechanisms and tools: (1) coordination of all organizations involved in delivering health and social services, (2) a single entry point, (3) case management, (4) a single assessment tool with a case-mix classification system, (5) an individualized service plan, and (6) a computerized clinical chart.

Method

Focus groups of researchers, clinicians, managers and policy-makers identified quantitative indicators for each component. The six components were weighted according to their relative importance in order to generate a total score. Data were collected every six months over 30 months to establish the implementation degree in the three experimental areas: Sherbrooke, Granit and Coaticook in the Province of Quebec, Canada.

Results

After 30 months, coordination is the most developed component in the three experimental areas. Overall, in July 2003, the Integrated Service Delivery system was implemented at the rate of 73%, 71% and 70% in Sherbrooke, Granit and Coaticook, respectively.

Discussion

This type of quantitative assessment provides data for managers and researchers to monitor the implementation. Moreover, when there is an outcome study, the results of the outcome study can be correlated with the degree of implementation, thus allowing for dose-response analyzes and helping to decrease the “black box” effect.

Introduction

Integrated Service Delivery (ISD) systems have been proposed for improving efficacy and efficiency of health care system, particularly for patients with multiple needs and complex interactions of many professionals and organizations. It is hypothesized that ISD systems would improve continuity of care and the health and satisfaction of clients, together with decreasing the use of costly resources, like hospitals and institutions. While there is some indications of the efficacy of ISD systems for some clientele like frail older people [1], their real effectiveness at the population level remains to be demonstrated. Although many ISD systems have been well described and compared [2–4], those experimented so far have not reported much on the implementation of the components of the system, nor on its process and real functioning. The lack of implementation data limits the replication of studies and could explain some negative results about the impact of ISD systems. It also deprives policy makers of critical information for applying ISD in the health care system.

The implementation of ISD systems should be assessed and monitored in order to inform managers and policy-makers of how the project is evolving, to account for the use of resources, identify obstacles quickly and find strategies to foster full implementation. Such a process analysis is also helpful in evaluative research to describe facilitating factors and obstacles that future implementation will have to consider. When an outcome evaluation is performed, a process analysis is used to ensure that the intervention really has been implemented as planned. This assessment is usually qualitative, but it is also informative to generate some quantitative ratings, especially when an economic analysis is done. Ratings could then help policy-makers to evaluate the degree of implementation and extrapolate the effort and resources needed for full implementation. In an outcome evaluation study, such ratings can be used to measure the “dosage” of the intervention and carry out dose-response analyses. The objective of this study was to develop a methodology to rate the degree of implementation and to monitor the implementation of an ISD system, the PRISMA model in three different areas of the Eastern Township in the Province of Quebec, Canada (the PRISMA-Estrie project).

The PRISMA model

According to Leutz, there are three levels of integration in health care: (1) linkage; (2) coordination; and (3) full integration [5]. ISD refers to systems targeting either coordination or full integration. In full integration ISD systems, the integrated organization is responsible for all services, either under one structure or by contracting some services with other organizations. Many examples of this level of ISD programmes have been developed. In the United States, the California On Lok project [6] gave rise to the PACE (Program of All inclusive Care for the Elderly) projects [7]. In Canada, the CHOICE (Comprehensive Home Option of Integrated Care for the Elderly) project in Edmonton is an adaptation of the PACE projects [4]. These programmes are built around Day Centres where the members of the multidisciplinary team who evaluate and treat the clients are based. Clients are selected according to relatively strict inclusion (degree of disability compatible with admission to a nursing home) and exclusion (e.g. behavioural problems) criteria. These systems usually function in parallel with the socio-health structures in place. Services are delivered by structures operated by the system or by external structures linked through contracts (hospitals, specialized medical care, long-term care institutions). The Social HMO in the United States [8] and the SIPA (“Systéme de services intégrés pour personnes âgées en perte d'autonomie”) project in Montreal are also integrated services but do not include a Day Centre [3]. However, home care services are provided by personnel hired by or under contract with the organization. All these fully integrated models are nested in the usual health and social services in a particular area but are run in parallel to them. They do not involve significant changes to the structure or processes of existing services, except for the negotiation of protocols for referring clients to ISD and the provision of some services not covered by ISD. Capitation budgeting is usually a key component of these programmes.

Although there were many attempts to design and implement full integration models, very few experiments of coordination models were implemented or tested. The PRISMA (Program of Research to Integrate Services for the Maintenance of Autonomy) model is a new model of integrated care based on coordination [9]. As opposed to full integration systems, this model includes all the public, private or voluntary health and social service organizations involved in caring for older people in a given area. Every organization keeps its own structure but agrees to participate within an “umbrella” system and to adapt its operations and resources to the agreed requirements and processes. At this level, the ISD system is not only nested in the health care and social services system (like the full integration models) but is also embedded within it.

The PRISMA model includes six components: (1) coordination between decision-makers and managers at the regional and local level, (2) a single entry point, (3) a case management process, (4) an individualized service plan, (5) a single assessment instrument coupled with a management system based, and (6) a computerized clinical chart. Coordination between institutions is at the core of the PRISMA model. Coordination must be established at every level of the organizations. First, at the strategic level (governance), by creating a Joint Governing Board (“Table de concertation”) of all health care and social services organizations and community agencies where the decision-makers agree on the policies and orientations and what resources to allocate to the integrated system. Second, at the tactical level (management), a Service Coordination Committee, mandated by the Board and comprising public and community service representatives together with older people, monitors the service coordination mechanism and facilitates adaptation of the service continuum. Finally, at the operational level (clinical), a multidisciplinary team of practitioners surrounding the case manager evaluates clients' needs and delivers the required care.

The single entry point is the mechanism for accessing the services of all the health care institutions and community organizations in the area for the frail senior with complex needs. It is a unique gate which older people, family caregivers and professionals can access by telephone or written referral. A link is established with the Health Info Line available to the general population in Quebec seven days a week, 24 hours a day. Clients are referred to the ISD system after a brief needs assessment (triage) to ensure they meet the eligibility criteria for the integrated system. Otherwise, they are referred to the relevant service. ISD eligible clients are then referred to a case manager. The case manager is responsible for doing a thorough evaluation of the client's needs, planning the required services, arranging to admit the client to these services, organizing and coordinating support, directing the multidisciplinary team of practitioners involved in the case, and monitoring and re-evaluating the client. The case manager is legitimized to work in all institutions or services. The individualized service plan results from the overall assessment of the client and summarizes the prescribed services and target objectives. It is led by the case manager and established at a meeting of the multidisciplinary team including all the main practitioners involved in caring for the older person.

The single assessment instrument allows for evaluating the needs of clients in all organizations and by all the professionals working either in home care organizations or in hospitals and institutions. The instrument implemented in the PRISMA model is the SMAF (Système de mesure de l'autonomie fonctionnelle—Functional Autonomy Measurement System), a 29-item scale developed according to the WHO classification of disabilities [10]. It measures functional ability in five areas: activities of daily living (ADL) [7 items], mobility [6 items], communication [3 items], mental functions [5 items] and instrumental activities of daily living (IADL) [8 items]. For each item, the disability is scored on a 5-point scale: 0 (independent), −0.5 (with difficulty), –1 (needs supervision), –2 (needs help), –3 (dependent). The resources available to compensate for the disability are also evaluated and a handicap score is deducted. A case-mix classification system based on the SMAF has also been developed [11]. Fourteen Iso-SMAF profiles were generated using cluster analysis techniques in order to define groups that are homogeneous in regard to their profiles, but heterogeneous in other respects. These profiles are used to establish the admission criteria to the different services and to calculate the required budget of the organizations, given the autonomy of the clientele served [12]. Finally, the PRISMA model includes a computerized clinical chart for facilitating communications between organizations and professionals. The SIGG (“Système d'information géronto-gériatrique”) has been developed and implemented in a pilot project in Victoriaville (Quebec, Canada). This shareable clinical chart uses the Quebec Ministry of Health and Social Services Internet network and was developed from a Lotus Notes platform.

The PRISMA-Estrie project

The study reported in this paper is part of a larger project assessing the implementation and impact of the PRISMA model. After piloting the model in a different area [13], we extended the PRISMA model to three areas in another region (Eastern Townships of Quebec) that present different environments:

• Sherbrooke: an urban area (population 145,000; 13%>65) with 3 university establishments (a tertiary care hospital, a geriatric institute and a primary care agency), two large public nursing homes (745 beds), several private and public residential facilities, a public home care agency and some voluntary agencies providing complementary services in home care;

• Granit: a rural area (population 22,000; 15%>65) with one merged public establishment that includes a local hospital, a primary care program (including home care), a nursing home (106 beds), public and private residential facilities, and some voluntary agencies providing complementary services in home care;

• Coaticook: a rural area (population 16,000; 14% >65) with one merged establishment that includes a primary care program, a nursing home (88 beds), an emergency service but no hospital beds, mostly private residential facilities and some voluntary agencies providing complementary services in home care.

The implementation evaluation focuses on the process of implementing the mechanisms and tools and how they function. One of the objectives is to explain the variations observed between the different implementation settings using a case study approach (multiple case study design) developed by Yin [14]. The questions that are documented try to define the extent to which the clientele using the services corresponds to the clientele initially targeted; if the services delivered correspond to those planned; if the resources planned were effectively made available; and if the delivery procedure corresponds to the one initially defined. Other questions focus on evaluating the process itself and identifying its strengths and weaknesses in order to reinforce or correct some of the elements comprising the new mechanisms and tools. There are three cases (each of the selected areas) analysed using different perspectives (multiple units of analyses). Data are collected from policy-makers, managers and clinicians, as well as clients and informal caregivers using different methods (interviews, focus groups, surveys). Other data are obtained from documentation analysis (minutes from the meetings), participating observation, management data monitoring or clinical files analysis.

Effectiveness is being evaluated using a quasi-experimental design (pre-test, multiple post-tests with control group). A sample of frail older people in the three study areas is followed for 5 years, as is a comparative sample in three comparable areas elsewhere in the Province of Quebec. The variables measured are: functional autonomy, satisfaction in regard to the services received, client empowerment, caregivers' burden, utilization of health and social services, and drug use. An economic analysis is also being performed.

More information on the PRISMA model and the PRISMA-Estrie Project can be found in a previous paper published in the Journal [15]. The present paper reports on one of the study included in the implementation analysis targeting the degree of implementation of the six components of the PRISMA model.

Methods

Based on the PRISMA model, a set of objective and measurable indicators of implementation were generated for each component. Those indicators were fully discussed and approved by two committees acting as focus groups meeting researchers, policy-makers, managers and clinicians involved in the PRISMA project. The first group is provincial and includes two teams of researchers coming from Laval and Sherbrooke universities interested in ISD systems, policy-makers from the Quebec Ministry of Health and Social Services, and managers from five Regional Health and Social Services Authorities. The second is a local group involved in the implementation of the Estrie Project. It includes researchers from the Sherbrooke University, a policy-maker from the Quebec Ministry of Health, managers from the Estrie Regional Health Authority and the health organizations of the three experimental areas, and clinicians. The provincial group met once and the local group twice for discussing this issue. The groups were also asked to weight the relative importance of each indicator and the relative importance of each component. For the latter, it was agreed to assign 20% each to coordination, the single entry point and case management, 15% each to the single assessment tool and computerized clinical chart, and 10% to the individualized service plan. A description of the indicators and their relative weights can be found in Table 1.

Table 1.

List of indicators used to rate the implementation of the ISD system

Indicators	Rating
	(pts)
Coordination	20
1. Presence of a structure designed to enhance cooperation between partners	3
2. All partners concerned represented	3
3. Representatives stability over time	3
4. Representatives participate regularly	3
5. Partners informed of how services are changing (or not changing)	4
6. Partners criticize the organization of the services change process	4
Single entry point	20
1. Presence of a single entry point in each local area	5
2. Clearing functions done by dedicated professionals	5
3. Professionals use a validated screening instrument to identify eligible frail elderly	5
4. Follow-up with older people in the group at high risk of functional decline	5
Case management	20
1. Variation between actual number of case managers and number needed according to proportion of senior citizens in the area	10
2. Variation (above or below) between actual average caseload and recommended caseload (45)	10
Single assessment tool and case-mix classification	15
1. Percent of clients under case management evaluated with SMAF tool	5
2. Percent of partners systematically using SMAF tool with their elderly patients	5
3a. Use of the case-mix classification system (ISO-SMAF profiles) for efficient utilization of resources	5
3b. Use of the ISO-SMAF profiles system as a new standard for financing services
Computerized clinical chart	15
1. Availability of a computer program for sharing clinical information in real time	5
2. Sufficient number of computers for all partners	5
3. Utilization of the computerized computer chart by partners	5
Individualized service plan	10
1. Percent utilization of the individualized service plan by case managers (as indicated in the clinical files of patients under case management)	10
TOTAL	100

From the beginning of the implementation evaluation (July 2001), data were collected every six months to assess all the indicators. Coordination was rated according to two sources of data about the different meetings held by the Joint Governing Boards and the Service Coordination Committees. Formal minutes of those meetings were systematically reviewed and were complemented by the notes of one of the research assistants who attended all the meetings using participating observation strategy. The single entry point indicators were completed using statistics from the local agencies where they are based. The number and caseload of case-managers were collected from the coordinators responsible for the implementation of the ISD system in each area. These data were validated by financial information from the Regional Health and Social Services Authority. The charts of all clients involved in the ISD system and referred to case-managers were systematically reviewed to quantify the utilization of the SMAF tool and the individualized service plans. For the computerized clinical chart, we did a monitoring of the equipments in each area and we used data on the real utilization of the system.

Results

Table 2 shows the implementation degree for each of the six components and each area. At the beginning of the implementation evaluation study, the coordination process was already well underway in all three areas, especially Sherbrooke where the implementation degree for this component was over 90%. In Sherbrooke, however, there was a broad consultation process and a reform of the ISD governance model during the spring of 2003, which explains why the rate declined to 76% in July 2003. The implementation of the single entry point started in January 2002 in the two rural areas and later (July 2002) in Sherbrooke. The first case managers were hired in January 2002 in Sherbrooke, but there was a slight delay in the rural areas. Consequently, the individualized service plan use started at these times. Implementation of the SMAF tool began before the official start of the study, and the Iso-SMAF case-mix classification system was being used by the local agencies for admitting clients to institutions and by the regional board for financing by July 2002. Finally, the computerized clinical chart was piloted in the Granit Region in July 2002 and extended to the other areas in November 2002. Overall, in July 2003, the ISD system was implemented at the rate of 73%, 71% and 70% in Sherbrooke, Granit and Coaticook, respectively.

Table 2.

Evolution of the implementation degree (in percent) for each component in each area under study from July 2001 to July 2003

	07/2001	01/2002	07/2002	01/2003	07/2003
Sherbrooke
Coordination	93	93	91	91	76
Single Entry Point	0	0	50	80	80
Case Management	0	30.5	53	55	53
Individualized Service Plan	0	81	47	69	62
Single Assessment Tool	40.7	80	88.7	88.7	88.7
Computerized Clinical Chart	0	0	0	20	76.7
Total	24.7	46.1	55.5	68.4	72.8
Granit
Coordination	79	75	82.5	82.5	82.5
Single Entry Point	0	50	50	70	70
Case Management	0	0	60.5	67.5	63
Individualized Service Plan	0	0	65	50	50
Single Assessment Tool	36	52.7	80	75.3	74.7
Computerized Clinical Chart	0	0	33.3	80	80
Total	21.2	33	62.2	72.3	71.3
Coaticook
Coordination	79	72.5	72.5	86.5	86.5
Single Entry Point	0	50	50	70	70
Case Management	0	0	17.5	33.5	50
Individualized Service Plan	0	0	62	59	53
Single Assessment Tool	35.3	52	77.3	80.7	76
Computerized Clinical Chart	0	0	0	80	80
Total	21.1	32.3	46.7	68	70

Discussion

In the literature on ISD systems, there are few data on the implementation process [4, 16, 17]. Even when it is reported, implementation is mostly summarized very briefly within a paper focusing mainly on the outcome of the intervention. In this study, we attempted to quantify the implementation in order to monitor the introduction of the components of the program under study.

The weighting of the components was arbitrary and based on the opinions of the researchers, clinicians, managers and policy-makers involved in the implementation. There is a hierarchy within these components: coordination is the base without which the other components cannot be implemented. Case management is also essential for implementation of the individualized service plan, the single assessment tool and computerized clinical chart. Therefore, coordination and case management are enabling factors that need to be weighted more than the others. The single entry point was also heavily weighted, because of its importance and the complexity of implementing it, especially in the urban area where multiple public, private and voluntary agencies are geographically scattered in town and must converge on a unique point.

The coordination element was already very highly rated at the beginning of the study and showed a ceiling effect. This suggests that more indicators should have been developed to illustrate higher level of coordination, particularly at the governance level. However, it must be said that discussions around coordination were already well engaged in the region for several years and that the suggested indicators could be appropriate and more modulated in other areas where the coordination process has to be initiated.

The PRISMA research group does not control the actual implementation. This is managed by the Regional Health and Social Services Authority and the local Governing Boards and agencies. Although the implementation of the PRISMA model was supported by a government policy working towards the integration of services for frail older people, it was implemented during a period of health and social services budget constraints. The implementation was delayed and has not yet been completed, mostly because of the slowness of the regional authority to provide funding to create new case manager positions. As noted by Leutz, “integration costs before it benefits” [4]. The investment in ISD system should be significant enough to generate some impact and benefit by improving the efficacy and efficiency of the system. We originally expected that the implementation would be completed over an 18-month period. This period has now been extended to 30 months, which is more than double the time anticipated.

The major and most costly component of the PRISMA model is case management, the implementation of which was only half that expected in July 2003. The weighting of this element should probably have been even higher given its importance in the integration of services. Recently, new investments have been announced to complete the staffing of case managers and we anticipate that the implementation percentage of this component should be over 90% by July 2004. Such an improvement will push the overall implementation degree to over 80% in the three areas. It is probable, although there are no data to support this hypothesis so far, that there is a threshold over which ISD begins to have a significant impact on the health and social services delivery system.

In general, the individualized service plan implementation degree decreased as the caseload of the case managers increased and approached the planned objective of 45 cases. This probably means that this case number is too high and does not allow case managers to complete the plan properly for their clients within an appropriate timeframe. Because of this, and other indicators mentioned in the interviews with case managers, the coordinators are considering decreasing the caseload to 40.

This method of rating the implementation cannot be fully validated, since there is no concurrent method of assessing quantitatively the implementation of a program. The use of focus groups and a well-defined theoretical model confers it a content validity. The relation of the implementation rating with the impact of the system on health services utilization and cost could be used in the future as construct validity. The data were collected and the rating was applied by only one person in the present study, so the inter-rater reliability is not an issue here. Since there were only three areas and the memory bias would have been very important, it was not possible to check the test-retest reliability of the method.

The data generated by this method have been very useful to managers and policy-makers. It gives them continuous information on the implementation process. It has been also a very powerful tool when arguing for more investments to complete the implementation. In our project, the implementation degree was a very strong incentive for investing more in the ISD system, for creating more case-manager positions, for financing the computerized clinical chart, and for implementing the case-mix classification system. It provided also good evidence for adjusting the case load of case-managers.

This method could also be useful for quantifying the “dosage” of the intervention for the outcome study. We may then correlate the implementation degree with the outcome measures relating to the individual, the family or health care utilization. It will help to decrease the “black box” effect of health services intervention and to avoid a “type 3 error” in measuring the effect of an intervention that has not really been or was not sufficiently implemented. It will be interesting to check if there is a threshold in this percentage over which significant impact may be demonstrated.