The PRIME-NL study: evaluating a complex healthcare intervention for people with Parkinson’s disease in a dynamic environment

Abstract

Background

An innovative, integrative care model for people with Parkinson (PRIME Parkinson) has gradually been implemented in a selected region of the Netherlands since 2021. A prospective evaluation of this model (PRIME-NL study) was initiated in parallel, spanning the year prior to implementation (baseline) and the implementation period. Following publication of the original study protocol, the COVID-19 crisis delayed implementation of the full PRIME Parkinson care model by two years and hampered the recruitment of study participants.

Objective

To describe which methodological adjustments were made to the study protocol because of these developments.

Methods

We compare various outcomes between a region where PRIME Parkinson care was implemented (innovation region) versus the rest of the Netherlands (usual care region). We use healthcare claims data of virtually all people with Parkinson in the Netherlands and annual questionnaires in a representative subsample of 984 people with Parkinson, 566 caregivers and 192 healthcare professionals. Four major methodological adjustments had to be made since publication of the original protocol. First, we extended the evaluation period by two years. Second, we incorporated annual process measures of the stage of implementation of the new care model. Third, we introduced a real-time iterative feedback loop of interim results to relevant stakeholders. Fourth, we updated the statistical analysis plan.

Discussion

This manuscript provides transparency in how the design and analyses of the evaluation study had to be adapted to control for external influences in a dynamic environment, including eruption of the COVID-19 crisis. Our solutions could serve as a template for evaluating other complex healthcare interventions in a dynamic environment.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12883-024-03778-x.

Introduction

Parkinson’s disease (PD) is the second most common neurodegenerative condition, and its prevalence is growing rapidly [1]. The needs of people with Parkinson (PwP), including PD and atypical parkinsonism, are not met optimally by current care models. Based on the needs of PwP, their relatives and healthcare professionals, we proposed a new care model termed PRIME (Proactive and Integrated Management and Empowerment in Parkinson’s Disease) [2]. Our four hypotheses are that the model will improve (I) the health of PwP, (II) the experienced quality of care by PwP and their informal caregivers and (III) the work-life balance for healthcare professionals, (IV) without raising the total costs of care. These hypotheses reflect the four-fold aim of this evaluation, which collectively referred to as the ‘quadruple aim’ [3].

The new care model introduced various healthcare innovations in a stepwise fashion, which are being implemented in the South-East region of the Netherlands from 2021 onwards. The PRIME-NL study compares various outcomes between the region where this care model was implemented (innovation region) and the rest of the Netherlands (usual care region). The data sources include healthcare claims data of virtually all PwP in the Netherlands plus annual questionnaires obtained in a representative subsample of 982 PwP, 566 caregivers and 192 healthcare professionals. The rationale and design of that prospective observational evaluation was published earlier [4].

However, following the publication of the original study protocol, we faced unforeseen challenges. In the first two years since this project started, the coronavirus disease 2019 (COVID-19) outbreak emerged, which had an enormous impact on global healthcare. For example, waiting times for diagnostics and elective care increased. The COVID-19 pandemic impacted our project in various ways. First, it delayed the implementation of the new care model, because of attrition of personnel and because meetings as well as in-person pilot assessments had to be cancelled. Second, it delayed the recruitment for the questionnaire-based study, which resulted in a smaller than planned sample size for the questionnaire study. In addition, we have gained a lot of experience during implementation and evaluation period, which also resulted in additions to the study protocol.

The purpose of this paper is to outline the methodological adjustments that had to be made since the publication of the original protocol. We aim to be transparent about such modifications to enhance scientific integrity and validity. We hope that this will inspire the design of other healthcare evaluations by providing insights into real-life challenges in an observational healthcare evaluation that is being implemented and evaluated in a highly dynamic environment.

Methods

We will illustrate how the PRIME Parkinson care model is operationalized, explain the changes to the original study protocol and describe the prespecified statistical analyses in detail.

PRIME Parkinson care

PRIME Parkinson care is operationalized as a home, spoke and hub model (Supplementary Material A). This model comprises the following core elements: (a) specialized Parkinson’s nurses who operate across the patients’ total care network and play an important role in coordination and integration of care; (b) regional teams which include specialist neurologists and a network of specially trained allied health professionals (ParkinsonNet); (c) an expertise center that supports Parkinson’s nurses and regional teams; and (d) self-management and active involvement in decision-making (empowerment) by well-informed patients [2]. This complete model is supported by various technology products and a range of centralized services, such as reliable information services, to support PwP and professionals.

The elements and services of this home, spoke and hub model are implemented in the South-East region of the Netherlands. In previous publications, we termed this region the PRIME region [4, 5]. However, from now on we will call this the innovation region, because of possible confusion with the name of the evaluation study. The new care model was implemented in a stepwise fashion, starting from 2021. In Fig. 1, we visually illustrate the timeline of implementation of all innovations of the new care model. We did not yet reach full implementation of all innovations, and expect to reach this in June 2025. Full implementation indicates the integration of innovations in routine regional clinical practice. Moment in time of full implementation is different for each hospital, depending on the possibilities per hospital. We developed the healthcare innovations from 2019 onwards in an iterative process, adhering to principles of design thinking [6]. Adopting a design thinking approach, enables PwP, caregivers and healthcare professionals to give input throughout the entire development process, which has led to changes in the intervention during the project phase. A detailed description on the operationalization of the care model will be published elsewhere.

Fig. 1.

Timeline of development and implementation of the PRIME Parkinson care model. Dotted lines indicate development period, light-colored lines indicate pilot implementation in a subgroup and dark-colored lines indicate full implementation of the innovation in the innovation region. Full implementation indicates integration of innovations in routine regional clinical practice in the intervention region. Moment in time of full implementation is different for each hospital, depending on the possibilities per hospital. Innovations are linked to the solutions of the logic model [3]

Changes to the initial protocol

We made several changes to the initial protocol. Four major methodological adjustments were made since publication of the original protocol. First, the evaluation period has been extended by two years. Second, we incorporated annual process measures of the stage of implementation of the new care model. Third, a real-time iterative feedback loop of interim results to stakeholders has been effectuated. Fourth, we updated both the statistical analysis plan and the methodology of the qualitative work regarding the evaluation of the PRIME Parkinson care model.

Extension of the evaluation period

First, the evaluation period had to be prolonged by two years because the COVID-19 crisis delayed the implementation of the new care model. Specific examples include the absence of personnel due to illness, restrictions in physical meetings during the lockdown, and slowing of the development of innovations with PwP and their caregivers. We therefore assume that PRIME Parkinson care was ineffective in the first year of evaluation (2021).

The COVID-19 crisis also caused a delay in the recruitment process of the PRIME-NL questionnaire study. In total, 984 PwP (414 in innovation region; 570 in usual care region), 566 caregivers (240 in innovation region; 326 in usual care region) and 192 healthcare professionals (67 in innovation region; 125 in usual care region) were recruited. These numbers were somewhat smaller than the original target sample size of 1200 PwP, 600 caregivers and 250 healthcare professionals. The recruitment process was completed in December 2020 and we do not plan to continue recruitment to reach the original target sample size.

To account for the delay and the smaller recruitment numbers, we have extended the study duration by two years (2024 and 2025). For the questionnaire-based study, we added two further follow-up assessments, resulting in a total of five follow-up measures between 2021 and 2025, i.e., one at baseline and five follow-up assessments at 12, 24, 36, 48 and 60 months. We complemented this with medical claims data collected during the same time frame. The updated power calculations given this extended study duration are presented in Supplementary Material B.

Incorporation of process measures

Since 2022, we have incorporated process measures of the implementation of the new care model in the innovation region. The implementation process was evaluated by adding specific questionnaire items which assess familiarity, usage and experiences concerning each healthcare innovation. The latter is based on the Net Promotor Score (NPS), an easy way to assess user satisfaction with one question and is therefore widely used in business and healthcare worldwide [7].

Furthermore, the roles of neurologists and Parkinson’s nurses has been shifted and extended within the new care model. This process is evaluated by extracting the number of consultations and the duration thereof from hospital systems.

In addition, we are preparing qualitative studies to obtain insight in various components of the new are model, including —but not limited to—shifts in the roles of PD nurses and neurologists and the work-life balance of healthcare professionals involved in the model. The qualitative studies will entail semi-structured interviews in a subgroup of healthcare professionals in the innovation region. We will publish the methodology of the qualitative studies separately in due time.

Annual feedback of interim results to stakeholders

Throughout the innovation and implementation process, we have experienced the value of incorporating key stakeholders in the development process. We therefore incorporated a feedback loop of process measures and interim results of the main analyses to key stakeholders in the care model, such as healthcare innovators and healthcare professionals. The main reason is to engage those stakeholders in the care model as much as possible throughout the implementation period and beyond. This dynamic strategy of soliciting ongoing feedback is quite unlike in clinical studies (which are typically characterized by a fixed protocol that is maintained throughout the trial), as such intercurrent feedback itself can affect the outcomes of the evaluation. By contrast, however, this strategy has previously been deployed to enhance the uptake of other healthcare innovations that relied in part on behavioral change, including the PD specialized allied health network named ParkinsonNet that is operational in the Netherlands [8, 9]. The rationale for this strategy is that this information feedback loop is part of the model itself [10], and will thus also be incorporated in potential future scaling efforts following the current evaluation period. Process measures will also be fed in real-time to healthcare innovators, to finetune innovative solutions (Fig. 2), for example by increasing awareness of certain products through additional implementation efforts. Feedback is provided during regular meetings between the innovation and evaluation team and an annual project meeting with other important stakeholders.

Fig. 2.

Evaluation of process indicators. From 2022, we annually evaluate process indicators, indicated by the blue-colored cycles. Based on these evaluations, we will finetune implementation strategies where necessary

Importantly, the results of the process measure analyses and interim analyses have no bearing on the publication of the final results of the evaluation study. We will submit the results of the prespecified statistical analysis plan, which is presented in Sect. 2.3, for publication no later than 1 year after completion of the final assessment year, irrespective of whether the results are in line with the hypotheses of the study.

Further operationalization of outcomes

We have previously described the quantitative outcome measures both within the healthcare claims-based data and questionnaire-data of PRIME-NL study [4]. The quantitative outcome measures are summarized in Table 1. In Supplementary Material C and D, we provide additional details on how the outcomes are operationalized.

Table 1.

Measurable outcome measures

		Dimension		Outcome	Instrument/Source
Primary analysis
		Population health		Primary	Healthcare claims
				Number of parkinsonism-related complications	Vektis
Secondary analysis
		Population health		Primary	Questionnaire
				Quality of life	Parkinson’s Disease Questionnaire-39 [14]
				Secondary	Questionnaire
				Depressive symptoms	Beck Depression Inventory
				Anxiety	State Trait Anxiety Inventory for Adults
				Autonomic symptoms	SCales for Outcomes in PArkinson’s disease - autonomic dysfunction
				Cardinal motor features	Numeric rating scale for bradykinesia, tremor, rigidity, postural imbalance
				Freezing	New Freezing of Gait Questionnaire
				Acceptance of Illness	Acceptance of Illness Scale
				Coping strategy	Ways of Coping Questionnaire for Parkinson’s Disease
				Activities of Daily Living	Academic Medical Center Linear Disability Score
				Cognitive performance	Telephone Montreal Cognitive Assessment (MoCA)
		Patient experience		Primary	Questionnaire
				Experience of Integrated Care	Patient Assessment of Chronic Illness Care+(PACIC+)
				Secondary	Questionnaire
				Experience of Continuity of care	Nijmegen Continuity of care questionnaire
				Self-management	Patient Activation Measurement (PAM)
				Patient empowerment	Self-designed questionnaire
				Being well-informed	Single items
		Caregiver experience		Primary	Questionnaire
				Caregiver burden	Zarit Caregiver Burden Inventory
				Secondary	Questionnaire
				Quality of life of caregiver	Parkinson’s Disease Questionnaire-Carers
				Coping strategy of caregiver	BriefCOPE
				Empowerment/self-management	Caregiver Patient Activation Measure
				Perceived social support	Multidimensional Scale of Perceived Social Support
		Work-life balance for healthcare professional		Primary	Questionnaire
				Professional wellbeing	Professional Fulfillment Index
				Secondary	Questionnaire
				Shared decision making	Shared decision making
				Experience of integrated care	Assessment of chronic illness care
		Costs of care		Primary	Healthcare claims
				Total healthcare expenditure	Vektis
				Secondary	Healthcare claims
				Distribution of costs over community or hospital care	Vektis
				Costs of parkinsonism-related complications	Vektis
		Process indicators			Questionnaire
				Awareness of innovations	Single item: Are you familiar with the innovation?
				Usage of innovations	Single item: Did you make use of the innovation in the past year?
				Experience with innovations	Net Promotor Score (How likely would you be to recommend the innovation?)

Population Health refers to the population of people with parkinsonism

Statistical analyses

Primary- and secondary analyses

We previously described the statistical analyses for the primary outcome of this study, but then also indicated that we would later publish a detailed statistical analysis plan. Here, we provide this update, including our analytical approach and sample size calculations for secondary outcomes [4]. Previously, we assumed the standardized difference between groups to be linear over time. However, given practice effects and delays in implementing, this assumption may not be true and either a threshold or curvilinear effect may be present. Therefore, we will explore including exponential terms in the regression models. We previously proposed to use negative binomial regression models in the primary analysis, on the assumption that the outcome data will be over-dispersed. We will now estimate the outcome variance and if this is equal or smaller than the mean, we will use Poisson regression. Otherwise we will revert to our initial intention of negative binomial regression models. For the secondary analyses, we will use generalized mixed-effects models for all outcome measures. In the models, the region of care (innovation region or usual care region), time, and their interaction (region of care * time) are the independent variables and the outcome measure is the dependent variable. We will adjust for potential covariates (described in [4] and visualized in a Directed acyclic graph in [5]) as fixed effects. Random intercepts and random slopes over time for participant ID will be included. The regression coefficient and corresponding p-value for the interaction term describes the difference in annual change in the outcome measure between regions. We will consider p < .05 for the interaction term between time and region as statistically significant.

Sensitivity analyses

We have added several sensitivity analyses. Because of the observational nature of our study, we were not able to randomize participants between groups and therefore certain types of bias may occur. Earlier work [5] concluded that our groups differ slightly in terms of age and disease duration. Therefore, we will use a propensity score matching procedure in sensitivity analyses.

Also, we will repeat the main analysis after incorporating inverse probability weighting to account for these regional differences in age and disease duration, to quantify the influence of potential selective inclusion of participants by region as much as possible.

We are aware of changing healthcare strategies and innovations in both the innovation and control region, with possible “contamination” due to our pragmatic real world evaluation. For example, in other areas of the Netherlands, a different innovative healthcare model has been introduced which focuses on intensive inpatient rehabilitation of patients who are on the brink of permanent nursing home admission [11]. Therefore, we will repeat the main analysis on the population health dimension, in which we will exclude people potentially receiving other complex healthcare interventions that we are aware of in the Netherlands as a sensitivity check. We will do this based on their four digit postal code.

Separately, we will explore the intervention effect as a time-varying covariate.

Furthermore, we will examine region-specific mortality rates. If these are different between the PRIME Parkinson region and usual care region, we will conduct a sensitivity analysis to account for competing risk bias [12].

Recruitment for our study occurred during the COVID-19 pandemic. Different phases of this pandemic in the Netherlands have been described in a previous publication [13]. To rule out that the moment of recruitment during the pandemic had an impact on selective dropout or main outcomes, we will conduct a sensitivity analysis. In this sensitivity analysis, we will divide participants into 3 strata based on their recruitment date. These 3 strata will reflect participants recruited during (1) the so called “first-wave” of the pandemic, (2) the period between the “first-wave” and the “second-wave” and (3) the “second-wave” of the pandemic. We will determine whether the main outcomes and selective dropout differ between those strata.

Deep brain stimulation may be a confounding factor in our analyses on parkinsonism-related complications and costs of care. Therefore, we will repeat these analyses in which we will exclude people who have received any form of device-aided therapy, including a deep brain stimulation surgery, as a sensitivity check. Another sensitivity analysis will use not the actual costs of care (as primary outcome in the costs of care dimension), but compute total costs as claims volume times average national claims prices to correct for any price differences between hospitals.

Lastly, we will perform an exploratory analysis in which we repeat the main analysis in the patient- and caregiver experience dimension, but excluding people in the innovation region who reported not using any healthcare innovations. We are aware that this is a biased approach to evaluate the utility of PRIME Parkinson care. However, this analysis serves to gain insight in the effect of the information resources on patient- and caregiver experience dimension rather than evaluating the utility of PRIME Parkinson care as a whole.

Discussion

This paper provides a transparent overview of how the design and analyses of the PRIME-NL study [4] have been adapted to control for external influences in a dynamic real world environment. For example, since the introduction of our healthcare initiative, a devastating pandemic occurred which necessitated a development of new health care strategies in both the innovation- and control regions. Another example was the introduction of complimentary healthcare innovations in other parts of the Netherlands, which serve as a control region for our innovation region – these may dilute the contrast between the innovation and control arms in our study. Typically, researchers do not make major changes to a study design after the start of the study as this is typically regarded as scientifically unsound. Making late adjustments to a study protocol while the study is already ongoing could be interpreted as data dredging [14]. Being an observational study, PRIME-NL is situated within an ever-changing environment that dynamically responds to such external influences. Instead of regarding it as a weakness, we leveraged the observational nature of this study to adapt and improve the study’s design to suit its dynamic environment. With these changes, however, also comes the responsibility to make any decisions and planned analyses transparent to ensure scientific integrity and to enhance the validity of our conclusions. We have committed to reporting future updates regarding the study’s design and analyses on the Open Science Framework. This approach might serve as a template for -and could inspire- other studies on healthcare innovations.

Although we aimed to optimize the study design, there are still some remaining limitations in our updated design. The observational nature introduces for example selection bias by the inability to randomize participants. However, our team recently published a manuscript which showed that the generalizability of the PRIME-NL study seems to be good. For example, there is no difference in the prevalence or proportion of atypical parkinsonism between the intervention and usual care region [5]. Furthermore, PRIME Parkinson care has also been operationalized in the Bath area in the United Kingdom (PRIME-UK) as a single-centre open label randomized controlled trial [15].This enables us to investigate the utility of PRIME Parkinson care both in a real-life situation (in the Netherlands), as well as in a controlled environment (in the United Kingdom) and triangulate our results. Another limitation might be any underestimation in the inflation of the cost of care dimension. At the moment, we assume a discount rate of 3% as advised by Dutch economic evaluation guidelines [16], in sensitivity analyses we explore discounting costs according to the Dutch medical consumer price index [17]. Another limitation is that a ceiling effect might occur on outcome measures, for example on the Patient Activation Measure [18].

Taken together, the evaluation of a complex healthcare model that is implemented in the real world with external “perturbations” introduces challenges but also allowed us to adapt and improve our study design. This approach may be used in the design of a similar model for other chronic health conditions. Our solutions and transparent descriptions could both serve as a template for other complex healthcare evaluations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

B.R.M., R.v.d.B. and S.K.L.D. wrote the main manuscript text. B.R.B., M.M., Y.B.S. and S.K.L.D. conceptualized the study. All authors gave input on the methodology and all authors reviewed the manuscript.

Funding

This research is part of the PRIME project, which was funded by the Gatsby Foundation [GAT3676] as well as by the Ministry of Economic Affairs by means of the PPP Allowance made available by the Top Sector Life Sciences & Health to stimulate public-private partnerships. The Center of Expertise for Parkinson & Movement Disorders was supported by a center of excellence grant by the Parkinson Foundation.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

The PRIME-NL study has been approved by the Ethical Board of the Radboud University Medical Center, reference number 2019–5618. The study has therefore been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments. Participants signed a digital or written informed consent before inclusion in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.