Evaluation of DementiaNet, a network-based primary care innovation for community-dwelling patients with dementia: protocol for a longitudinal mixed methods multiple case study

Abstract

Introduction

Primary healthcare professionals will increasingly be required to manage and optimise their treatment for patients with dementia. With DementiaNet, we aim to reduce the burden of dementia on healthcare services and society through implementation and facilitation of integrated network-based care with increased dementia expertise. DementiaNet is designed as a stepwise approach including clinical leadership, quality improvement cycles and interprofessional training, which are tailor-made to the local context. For example, the composition of the network and improvement goals are tailored to the local context and availability. Here, we describe the linked evaluation study which aims to provide insight in effectiveness, process and mechanism of the DementiaNet approach through an innovative evaluation design.

Methods and analysis

We designed a longitudinal, mixed methods, multiple case study. Study population consists of two levels: (i) local DementiaNet networks of primary care professionals and (ii) patients and informal caregivers who receive care from these networks. At the start and after 12 and 24 months, quantitative data are collected for each network on: level of network maturity, quality of care indicators and outcomes reported by informal caregivers of dementia patients. We assess changes in networks over time and the association with quality of care and informal caregiver-reported outcomes. Throughout the study, logs about each network are registered. Additionally, semi-structured interviews with network members and informal caregivers will provide insight in experiences and opinions regarding effects and mechanisms through which changes in quantitative outcomes are effectuated. Rich narratives will be constructed about the development of the local networks using collected data.

Ethics and dissemination

The study protocol was reviewed by the local medical ethics committee; formal judgement was not required (protocol number: 2015–2053). The findings of this study will be disseminated through peer-reviewed publications, conference presentations and presentations for healthcare professionals where appropriate.

Strengths and limitations of this study.

• Primary care innovations are not always subjected to the right rigorous evaluation, especially if their complexity is at odds with the conceptual assumptions of the randomised controlled experiment. This evaluation study adds to evidence-based healthcare, by employing research methods that help to understand whether DementiaNet is effective or not and focuses on why, how and in which context certain outcomes can be expected. Therefore, comprehensive data collection is designed with quantitative and qualitative methods.

• The knowledge resulting from this longitudinal multiple case study emanates from theoretical generalisability rather than statistical generalisability, and may have great importance in allocating healthcare resources in such a way that patients benefit most.

• Quality indicators of care were derived based on widely supported primary care guidelines and were developed specifically for the current study to fit the innovation. Hence, these have not been employed in research before. Indicators’ face validity has been established and will be reviewed for feasibility and reliability before final data analyses.

• The time span of the current evaluation study is likely too short to result in impacts on informal caregiver-reported outcomes; however, it may provide important data for further evaluation of DementiaNet with extended follow-up.

Introduction

Healthcare needs of elderly are characterised by high prevalence of chronic conditions, multimorbidity and strong heterogeneity between individuals and over time.¹ As a result, numerous health and social caregivers are involved in care for this population. Additionally, over the last years, care systems and services have changed with a shift from long-term residential care facilities towards increased community-based care for elderly, resulting in increased requirements for primary care. Despite many initiatives, care arrangements are still suboptimally designed to deal with the complexity of care, that is, the large number of different available services, the involvement of many different professionals and the accompanying lack of certainty and agreement about the best treatment plan. This has led to a lack of integration, coordination and continuity.^2–5 Possible explanations might be the facts that, in general, new guidelines are not fully taken up in clinical daily practice and are not adapted to each other, and improvement strategies merely target only parts of the system or aim at regional instead of local systems.

Community-dwelling patients with dementia present an illustrative example of the challenges that are posed on complex chronic primary care. First, much diversity exists in care needs since both the manifestation of dementia and the patients’ social contexts are multiform. Second, many different primary care professionals are involved from different health and social disciplines to provide care for patients with dementia. This urges the need for a high level of collaboration, as clinical practice is still mainly characterised by ad hoc collaboration. Hence, reorganisation of primary care is needed, in a way that is innovative, effective, scalable and also cost-effective.⁶ An overview of usual care is provided in box 1.

Box 1. Usual primary dementia care and DementiaNet care.

Usual care for patients with dementia in the Netherlands:

Dementia care in the Netherlands is characterised by practice variation among regions. The most important characteristics and common shortcomings are:

• key players in primary dementia care are general practitioners, practice nurses, case managers, district nurses;

• originally focused on acute episodes of single diseases instead of chronic multimorbidity patients;

• care is fragmented with professionals working in their own domain, with limited interprofessional communication and ad hoc collaboration;

• many professionals do not know each other, are unfamiliar with each others’ disciplines, responsibilities and competencies;

• there is little adherence to guidelines;

• knowledge about dementia diagnosis and management is often insufficient.

Care with the DementiaNet innovation:

The DementiaNet innovation aims to promote a shift, addressing these limitations, towards integrated dementia care through:

• network-based care with high levels of collaboration;

• a network leader to stimulate and coordinate the network;

• care improvement through quality improvement cycles with tailor-made goals and improvement plan to fit the situation of each individual network;

• high dementia-specific expertise through interprofessional training and practice-based learning.

Education alone is insufficient to improve primary dementia care.⁷ Also, interventions targeted at improving case management, a crucial factor in primary dementia care, show limited improvements on outcomes such as caregiver burden⁸ or care needs and quality of life.⁵ Another UK-based analysis showed disappointing results from efforts on dementia recognition, diagnosis and management.^{9 10} In contrast, innovations aimed at a more comprehensive system, such as the PRISMA model for integrated service delivery system for frail older people in Canada, were positively evaluated on several relevant outcomes such as functional decline rate and unmet care needs.¹¹ Another intervention study that targets dementia management in primary care as a whole, the Delphi study in Germany, shows promising preliminary results (on general practitioner attitude and caregiver burden),¹² but is yet to publish the overall results.

Both the necessity and possibility for improvement in primary care for patients with dementia are evident, which led to the development of DementiaNet. This innovation aims at network-based care for community-dwelling patients with dementia, following a stepwise, tailor-made approach. The innovation is integrated with a parallel running evaluation study which aims to assess implementation of DementiaNet in primary care, and to assess the merits and harms of this approach.

DementiaNet is complex in nature, as it alters a services delivery system with many different players involved and many external factors potentially influencing the pathways through which effects can be accomplished. The evaluation study, thus, has to fit the complexity of the healthcare innovation. In contrast to most medical and healthcare research where the influence of context is minimised, this is of particular interest in the evaluation of complex innovations. Therefore, research should not solely be aimed to answer the question of ‘does it work?', but should prioritise on how and why does it work.¹³ Therefore, the current evaluation study aims to answer the following questions: what are the merits and drawbacks of the DementiaNet approach; how are these achieved and which factors influence these processes? This paper describes the innovative methods used for the evaluation of DementiaNet along with background on these methods.

Methods

DementiaNet innovation

With DementiaNet we work towards high-quality, network-based care, which is organised on a local level with professionals from medical, care and social disciplines. DementiaNet aims to optimise care processes and outcomes, both from a perspective of community-dwelling patients with dementia and their informal caregivers, as well as from care professionals’ perspective. This is pursued through multidisciplinary network-based care with a high level of collaboration. A tailor-made approach is employed to ensure fit to the large practice variation as seen in daily clinical practice.

DementiaNet encompasses the transition towards network-based care through practice facilitation.¹⁴ These clinical networks are designed in primary care, and include professionals from multiple disciplines and from varying organisations. Hence, these networks include collaborations between individuals and organisations across institutional and professional boundaries. These clinical networks thereby ensure quality of and access to care for patients, including those who require coordination of care across a range of settings.¹⁵ This is pursued through formation of networks of primary care professionals who jointly and locally provide care to a number of patients with dementia, desirably including at least one professional of the medical (eg, general practitioner), care (eg, community nurses or case managers) and welfare (eg, social workers) discipline. Inclusion of healthcare professionals is adapted to local sources and needs. As a consequence, each network in the programme is different from another in terms of size, represented disciplines and starting level of collaboration and care. A baseline data collection assessment takes place to map the starting position of the network. This includes measurements regarding network members and their backgrounds as well as the quality of care in their network. Feedback of the findings in the baseline data collection is then provided to the networks. Local network meetings are scheduled which start by making several actions to improve dementia care. These goals and actions are part of the quality improvement cycle, which are tailor-made to each networks’ specific situation. Tailoring the approach to fit their local diversity is key in this innovation.¹⁶

Each network will employ four key components that are central to the approach of DementiaNet. Primarily, it relies on network-based care. The professionals in the network generally share a caseload of patients, the majority of whom have multiple professionals involved, requiring structured and organised collaboration to ensure continuity in care.

Second, the network leaders take up a central role in the process. Their task is to connect all professionals in the network and to stimulate and facilitate collaboration and improvement actions. Specifically, there is a leadership support programme for network leaders to help them take up this role.

Third, networks work through quality improvement cycles (Plan-Do-Check-Act (PDCA)). This means that at the beginning of each PDCA cycle, a comprehensive assessment is performed to get an overview of the quality of care and their network characteristics. The network jointly identifies improvement goals based on this measurement and their own experiences. A plan is drawn up with specific actions, tasks and a timeframe to achieve their goals. At the end of the yearly cycle, another assessment is performed to evaluate improvement and to identify new goals.

The last key element has a facilitating function. Interprofessional training and practice-based learning are used to increase knowledge and competencies. The contents of these training and coaching sessions are tailored to each network's own goals, as they have different starting levels and different improvement goals. Preferably, the training topics are linked to the quality improvement cycles. Also team training sessions are applied to increase team coherence, with sufficient team working skills, attitudes and competencies in the individuals involved in the team. Furthermore, professionals from different networks can take part in other sessions that were planned for these groups together, to be able to learn from each other and from best practices. More detailed information on the development of the DementiaNet innovation are described elsewhere.¹⁷

Study design and population

The evaluation study is designed as a longitudinal, mixed methods, multiple case study.¹⁸ Each participating network serves as a case in this study. Networks will be followed over time. Quantitative data will be collected at baseline and after every 12 months, with a maximum of three measurements within the current study period (January 2015–July 2017). Qualitative data will be collected throughout the course of the innovation programme to gain in-depth knowledge on processes and experiences of involved persons (ie, care professionals, patients and informal caregivers). Triangulation of quantitative and qualitative data will be used to strengthen insight in patterns.

The study population consists of two levels. The first level includes the local DementiaNet networks participating in the DementiaNet programme. The second level includes patients and informal caregivers who receive care from care professionals in these local networks.

Data collection

We will collect data from multiple sources to describe the networks and to measure outcomes. First, for each network, data will be documented by the research team regarding the number and discipline of professionals involved. Log documents will be kept for each network with information on the process of network formation and actions taken before enrolment of networks into the programme, as well as specifics that may influence the way their network develops and is able to execute the quality improvement cycles. Of this log, a narrative is to be constructed about each network. Additionally, a yearly online questionnaire will be distributed among network members, including instruments including their attitude towards healthcare teams¹⁹ and dementia,²⁰ their perceived team skills²¹ and enabling factors for collaboration.²²

The following data will be collected to assess the effects of the DementiaNet innovation on care processes and outcomes (figure 1):

Figure 1.

Overview of data collection for the evaluation of the DementiaNet innovation. §Continuous collection of data; †Data collected at start and after 12 and 24 months; ‡Data collected at one time point in a selected number of networks. References for the informal caregiver outcome instruments^37–46.

Network maturity

Network maturity is defined as the level at which the care professionals operate as a network. To assess the starting level and changes over time, we will use a model for integrated primary care called the ‘Primary Care Maturity Model’,²³ which includes eight items in three domains: (1) person-focused care, population-focused care; (2) clinical integration, professional integration, organisational integration, system integration and (3) functional integration, and normative integration. Each item is rated on four defined levels, ranging from (1) ad hoc, through (2) defined, and (3) controlled, to (4) synchronised. By summing the scores on the eight domains, a global maturity score will be derived for each network for each measurement point, reflecting their network maturity at each time.

The rating of network maturity will be based on information obtained directly from the networks by means of interviews. Structured interviews with the network's leader(s) will be held at each measurement point (baseline and after every 12 months) by an independent researcher. An interview guide is developed based on the content of the Primary Care Maturity Model in such a way that sufficient information is obtained on each of the eight items to be scored. This approach is chosen in order to allow a certain degree of flexibility to each networks composition and context, while still targeting the specific topics to be scored. Interviews are recorded on audio tape and stored until the end of the evaluation study. At that point, another independent and blinded researcher, who is unfamiliar with the study design and networks in the programme, will be instructed to rate the eight aspects of the Primary Care Maturity Model based on the information in the interview, to obtain the global network maturity score.

Quality of care indicators

Quality of dementia care will be assessed by means of quality indicators. The quality indicators will be reported on by the local networks through a registration file. A composite score will be constructed of the indicator scores of the final quality indicators to obtain a single overall score reflecting the network's quality of care.

This set of quality indicators was developed by an expert panel consisting of a geriatrician, general practitioner, community nurse/researcher, primary care researcher, epidemiologist and geriatric researcher prior to the current study to fit this particular evaluation. As it regards an innovative services delivery approach, it therefore requires different indicators then have already been developed for primary care settings. First, a framework was drafted with the basic concepts of the DementiaNet innovation, which were translated into criteria and subsequently operationalised into indicators that care should meet. These were checked to comply with relevant current guidelines and agreements regarding primary dementia care. Consensus on 13 final indicators was reached after multiple meetings in which relevance and feasibility of indicators were reviewed, as well as the comprehensiveness of the total set. This set was tested for face validity, acceptability and perceived feasibility in a pilot survey among 18 primary care professionals and showed good results on every aspect.

As these indicators have not been used before, they will be subjected to additional assessment based on the baseline data before the actual analysis of the data. Assessment will ensure the use of only reliable indicators, for instance, taking into account missingness, floor and ceiling effects and coherence with definitions. Therefore, the final set of indicators used in actual data analysis is expected to be more concise.

Informal caregiver-reported outcomes

Data on informal caregiver-reported outcomes will be gathered through paper-and-pencil questionnaires. Patients will be informed about the project and associated evaluation study through a letter from their general practitioner. This letter includes an answering card in which they can indicate whether they are interested in participation in an informal caregiver questionnaire. If so, the research team will contact them to obtain consent from informal caregiver and the patient where possible, and the postal address to send them the questionnaire. The informal caregiver questionnaire consists of demographic questions about the patient and informal caregiver, as well as validated instruments on several outcomes (figure 1).

Experiences and perspectives

In conjunction with the quantitative evaluation, a qualitative approach will be employed. For this part of the evaluation study, we will use semi-structured interviews with both care professionals in the networks as well as patients and informal caregivers. These data will provide insight in experiences and complex processes influencing potential results to be examined in the quantitative part. Interviews will be held by a trained researcher, starting after the first year of the project. By purposive sampling of participants, we aim to include relevant perspectives from different disciplines of care professionals, as well as patients and their informal caregivers originating from different networks.

Analysis

It is expected that this innovation has effects on multiple levels which may vary. Also, it is expected that the networks have different starting levels and divergent progression rates. Hence, the study considers both within and between network analyses, as follows.

Within each network, all data sources will be conjoined in order to identify any changes resulting from the DementiaNet innovation. We will look for patterns in trends over time in quantitative measures and we look for possible explanations for trends in activities carried out by the networks and their improvement goals. More specifically, we will look into associations that follow from a hypothesised pathway of effects, where we expect that network maturity will increase over time, and will be associated with quality of care as measured by the quality indicators. Potentially, an increase in patient-reported outcomes will eventually follow the increases in quality of care. This will be analysed by using mixed effects growth models to account for repeated measures and clustering of data within networks.

As the course of this innovation will proceed differently in each network, there will be a natural contrast between different networks with regard to the maturation into networks and the subsequent approach to care. Given the fact that these aspects will be also monitored over time within each network, this will allow for cross-case comparisons. This approach has been used previously, for example, on an integrated services delivery system in primary care for elderly, in which they monitored the degree of implementation of integrated services in a quantitative manner.²⁴ In outcome evaluation studies, such a quantification of implementation can be used as a measure of ‘dosage’ of the intervention to be able to look for dose-response patterns to strengthen plausibility of found patterns.

By comparing cases (ie, cross-case comparison) on the extent these have matured into a coordinated network and how much improvement efforts have been made and output (ie, trends in quality of care and informal caregiver-reported outcomes), it will be possible to increase plausibility of causality to attribute changes to the DementiaNet innovation similar to a dose-response manner.

Furthermore, the qualitative data from the semi-structured interviews will be used to explore experiences of professionals and patients and informal caregivers with the DementiaNet innovation. A thematic analysis will be used to analyse the verbatim transcripts of the semi-structured interviews. The analysis will be partly guided by a predetermined framework of potential experiences and perceived benefits based on the development of the innovation. We will remain open to discovering unanticipated nuances and topics in the data. First, transcripts will be independently coded by two trained researchers. Subsequently, both coding schemes will be jointly reviewed to reach consensus about most appropriate coding. After that, codes will be categorised and major themes will be identified by the same two researchers. Lastly, both researchers will independently draw overall findings from the codes in each category, after which a consensus round will be applied to these findings. Qualitative data analysis will be performed in Atlas.ti software.

The findings from these qualitative data will be conjointly used with the quantitative findings in the interpretation phase of the study in multiple manners: a) through triangulation, to corroborate findings and provide a stronger basis for conclusions, b) the qualitative findings will be used to augment quantitative findings, c) the qualitative findings will be used to identify unexpected and/or unintended effects that are not covered by the quantitative data.

Discussion

DementiaNet is an innovation that aims to tackle the current shortcomings in primary care for patients with dementia by effectuating a transition from ad hoc collaboration towards more integrated network-based care with increased dementia expertise. With the current evaluation study, we aim to provide insight in implementation of the DementiaNet innovation and its merits and harms by means of a longitudinal, mixed methods, multiple case study. Here, we will also take DementiaNet as an example of a complex intervention to elaborate further on the viewpoint that rigorous evaluation of these types of innovations in health services systems is essential and which considerations should be taken into account when designing such an evaluation study, to ensure adequate capturing of the complexity while achieving high external validity.

Rationale of the study

Unlike clinical treatments, innovations in health services and primary care are not always subjected to rigorous evaluation.^25–27 Such evaluation studies add to evidence-based healthcare, which is essential in order to distinguish innovations that change healthcare organisations for the better, from those that lack beneficial effects. Such knowledge has great importance in allocating healthcare resources to spread innovations and ensure actual implementation.

Innovations in health services systems are often complex in terms of multiple components that interact, the number of involved professionals, the extent to which they have to alter their behaviours and the flexibility and tailoring necessary to fit the situation in which it is implemented,²⁸ which is particularly the case in the DementiaNet innovation. In such complex innovations, it is often difficult to accurately predict to what extent and through which pathways the intervention may affect outcomes, and how the context in which it is implemented influences these pathways. In other words, it is hardly possible to predict if and how healthcare innovations will lead to the intended outcomes.^{29 30} Many examples exist of previous efforts in healthcare innovations that seemed promising but did not induce the desired changes, or even worsened outcomes or expenses.^{31 32} For instance, interventions aimed at reduction of emergency admissions have failed to produce the desired outcomes or even produced counterproductive outcomes because several aspects had been ignored, such as alternative explanations, regression to the mean and supply-induced demand.³³ The degree of uncertainty in effective pathways through which interventions work and therefore the results they lead to, increases with a higher degree of complexity of healthcare change. In general, but especially in times of limited resources, it is of invaluable importance to evaluate innovations in healthcare services to know which ones are worth adopting and investing in.

Study design

From the viewpoint that evaluation is indispensable, one inevitable choice is the optimal study design. From the perspective of traditional scientific (statistical) generalisation, the highest form of evidence for efficacy of interventions comes from randomised clinical trials (RCTs). The key methodological components of an RCT are the use of a control group and random assignment to groups to balance distribution of potential confounders, to allow for causal inferences. These components ensure high internal validity, but often limit external validity. However, several differences are encountered between the evaluation of relatively simple (medical) interventions and of complex healthcare innovations. For instance, the nature and complexity of health services innovations often cause assumptions underlying the RCT design to not be upheld, therefore compromising internal validity of RCTs and thus advocating the use of alternative study designs.³⁴ The most often violated assumption is the assumption of context independence, but the assumption of equipoise may not apply if preference for the intervention over usual care exists.

The DementiaNet innovation is complex on multiple aspects, according to definitions from the Medical Research Council (MRC)^{28 35}: it consists of multiple interacting components; healthcare professionals have to alter their behaviours considerably and multiple organisational levels are targeted. Additionally, the innovation is tailored to the specific situation of each local network, which has been recognised as a logical fit for complex interventions to be adapted to local contexts rather than completely standardised.²⁸ Logically, the context in which the intervention is implemented is of great influence and therefore of interest to the evaluation. This will be taken into account by constructing narratives of each network with specific attention to their context and by looking for patterns in different contextual factors that may account for different trends in outcomes.

For these reasons, we designed the evaluation study as a longitudinal multiple case study. The unit of analysis is the individual network participating in the DementiaNet project. This makes it impossible to set up a comparable control unit, as these networks do not exist yet without the innovation. Additionally, necessary investment in data collection was not endorsed by professionals if participation in the project was not ensured. In case studies, the context is explicitly taken into account as part of the evaluation, in contrast to experimental designs which employ the opposite approach by controlling the context as much as possible.¹⁸ Therefore, a multiple case study is found very suitable for this type of evaluation. In a multiple case study, each case can be seen as a single experiment. Hence, a multiple case study may then be considered the equivalent to multiple experiments. Under this assumption, generalising from case studies can be equivalent to generalising from experiments.¹⁸ Inferences are drawn both from within-case changes over time and cross-case comparison. The longitudinal multiple case study design allows for the addition of this latter approach, thereby providing the potential to replicate findings and identify patterns, which increases explanatory power and generalisability of findings.³⁶

Although there is a certain selection underlying the participating networks in the evaluation study, we believe that the results will extrapolate to other locations as well. This is assumed because the innovation is not specific to this innovators and early adopters group, but is applicable to every network as it is strongly tailor-made to the specific needs of every network. We will evaluate the suitability for networks that start at higher and lower levels of collaboration and quality, leading to higher external validity.

Each network is enrolled into the evaluation study as a case on starting in the project. Hence, the evaluation study commences at the same time as the implementation of the innovation. This timing allows for the most optimal within-case comparison between the situation right before implementation started and during increasing levels of implementation within the network. This outweighs the fact that effects take time to develop and thus may not come to full fruition within the timeframe of the study in our opinion as it strongly increases the validity of inferences to be drawn from this evaluation.

Expectations

Successful transition towards network care will be evidenced by an increase in the rating of network maturity. It is expected that this is not the case in all networks, as some probably fail to succeed in transitioning after the starting initiative to take part in the innovation, for instance, because of organisational problems or because network leaders are unable to fulfil their role. Moreover, it is expected that rating of network maturity is associated with the score on quality of care as measured by indicators. Hence, we expect that quality of care scores will increase along with network maturity, although possibly with a considerable delay. It is not hypothesised that informal caregiver-reported outcomes will already be affected by the DementiaNet innovation in a way that is timely and strong enough to be picked up by this evaluation study. However, as it is an extension of the hypothesised pathway and the ultimate goal of many health services innovations, we do consider the inclusion of these outcomes relevant to incorporate the patient and informal caregiver's perspective to expand on in further studies.

We expect that the mixed methods design provide us with insight in how the innovation actually was implemented in each network, how it worked and which contextual aspects influenced this. Furthermore, we expect information on which aspects of the innovation are most effective in which circumstances. Possibly, the innovation and future implementation can be improved with this information. Next to highly valuable data for effective and efficient network-based care for chronic conditions in older populations, starting with dementia care, this study may yield important methodological data on the value of a multiple case study analysis for other complex interventions as well.