Detecting placebo and drug effects on Parkinson's disease symptoms by longitudinal item‐score models

Abstract

This study tested the hypothesis that analyzing longitudinal item scores of the Unified Parkinson's Disease Rating Scale could allow a smaller trial size and describe a drug's effect on symptom progression. Two historical studies of the dopaminergic drug ropinirole were analyzed: a cross‐over formulation comparison trial in 161 patients with early‐stage Parkinson's disease, and a 24‐week, parallel‐group, placebo‐controlled efficacy trial in 393 patients with advanced‐stage Parkinson's disease. We applied item response theory to estimate the patients’ symptom severity and developed a longitudinal model using the symptom severity to describe the time course of the placebo response and the drug effect on the time course. Similarly, we developed a longitudinal model using the total score. We then compared sample size needs for drug effect detection using these two different models. Total score modeling estimated median changes from baseline at 24 weeks (90% confidence interval) of −3.7 (−5.4 to −2.0) and −9.3 (−11 to −7.3) points by placebo and ropinirole. Comparable changes were estimated (with slightly higher precision) by item‐score modeling as −2.0 (−4.0 to −1.0) and −9.0 (−11 to −8.0) points. The treatment duration was insufficient to estimate the symptom progression rate; hence the drug effect on the progression could not be assessed. The trial sizes to detect a drug effect with 80% power on total score and on symptom severity were estimated (at the type I error level of 0.05) as 88 and 58, respectively. Longitudinal item response analysis could markedly reduce sample size; it also has the potential for assessing drug effects on disease progression in longer trials.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Large trials are required for detecting drug effects on the symptoms of Parkinson's disease, as measured by the total score of the Unified Parkinson's Disease Rating Scale instrument. Differentiating disease‐modifying drugs from symptom‐relief drugs is a major challenge.

• WHAT QUESTION DID THIS STUDY ADDRESS?

To what extent can analyzing item scores, instead of the total score, reduce the sample size of placebo‐controlled trials?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

Longitudinal modeling showed an approximately 30% sample‐size reduction for detecting the treatment effect observed in a placebo‐controlled trial of ropinirole.

• HOW MIGHT THIS CHANGE DRUG DISCOVERY, DEVELOPMENT, AND/OR THERAPEUTICS?

Item‐score analysis, where applicable, can reduce the time and cost of developing drugs for Parkinson's disease. Used in the longitudinal modeling of trials of adequate duration, it also has the potential to more efficiently estimate a drug's effect on the symptom progression rate.

INTRODUCTION

Parkinson's disease (PD) affects approximately 1% of the population aged older than 65 years. ¹ It is a slowly progressive neurodegenerative disease manifested by a broad range of symptoms. ² Diagnosis occurs with the onset of motor symptoms, for example, bradykinesia, muscular rigidity, rest tremor, and postural and gait impairment, but the disease begins years before diagnosis, with nonmotor symptoms such as constipation and rapid eye movement sleep behavior disorder. Following diagnosis, additional nonmotor features such as pain, fatigue, and mild cognitive impairment develop. In advanced PD, further motor and nonmotor symptoms develop, for example, postural instability with frequent falls, freezing of gait, urinary symptoms, orthostatic hypotension, and dementia. Accordingly, patients experience increasing clinically significant disability over time as a result of symptoms and long‐term complications of dopaminergic therapy, including fluctuations, dyskinesia, and psychosis. ² Available treatments relieve symptoms without clear evidence of reducing the rate of disease progression. ³

The most widely used clinical measurements to quantify the disability and impairment of patients with PD over time are the Unified Parkinson's Disease Rating Scale (UPDRS) and, lately, the newer version, the Movement Disorder Society–sponsored revision of the UPDRS (MDS‐UPDRS). ⁴ , ⁵ The scales assess the consequences of the disease in areas such as mental status, daily living, motor functionality, treatment complications, and adverse effects by scoring 44 questions (items) mostly from 0 to 4, with higher scores reflecting more severe disease. The items are composed of the following components: (I) mentation, behavior, and mood; (II) activities in daily living; and (III) motor examination (including left/right assessment).

The common approach for evaluating treatment effects is to analyze the sum of all item scores, that is, the total score. A key assumption for this approach is that all items have equal importance in reflecting the overall disability. ⁶ In contrast, item response theory (IRT) offers an alternative. It estimates a patient's disease severity (or disability level) as a latent variable directly from the item level scores and with the acknowledgment that each item bears varying importance for informing the disease severity. Several pharmacometric models applying IRT for disease areas using composite scales have been developed, ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² and a comprehensive tutorial presenting the framework of the IRT in the pharmacometric context is available. ¹³ Simulations using the UPDRS and Alzheimer's Disease Assessment Scale–Cognitive Subscale (a standard efficacy end point for clinical trials in Alzheimer's disease) have shown promising evidence that longitudinal modeling of a composite end point analyzed using IRT could enhance the study power. ⁷ , ¹²

The efficacy of an investigative drug is typically assessed using the end‐of‐treatment observations in a placebo‐controlled study. However, multiple assessments made during the treatment present an opportunity for longitudinal analysis to describe the time course of treatment response, which can be important for understanding whether a drug slows down the disease progression. Longitudinal modeling is also advocated by the pharmacometric community and supported by regulatory bodies for its potential to enhance trial efficiency. ¹⁴ , ¹⁵ , ¹⁶

In a previous report of simulation‐based longitudinal modeling of PD, where IRT analysis demonstrated the potential for higher study power than total score analysis, a hypothetical drug effect was assumed to reduce the linear disease progression with an immediate onset, and the placebo effect was not accounted for. ⁷ Because a drug effect typically takes time to be observed and a placebo effect is inevitable in clinical trials, it is useful to capture these aspects in a trial simulation. Furthermore, in the previous report, the total score was derived from item scores that were simulated from the item response model (IRM); as such, the comparison could potentially favor the IRT analysis.

Given this background, we explored a framework of longitudinal modeling for understanding the time course of the drug effect (symptom and slowdown of disease progression) and placebo effect on PD and compared the study power of two longitudinal modeling approaches—analyzing total score versus disease severity estimated by IRT—using historical clinical trial data.

METHODS

Studies analyzed

We used the UPDRS data from two published studies of ropinirole, a nonergot dopamine agonist. One was a cross‐over comparison of immediate‐release and extended‐release formulations in patients with early‐stage PD, ¹⁷ and the other was a parallel comparison of ropinirole to placebo as an adjunct therapy to levodopa (l‐dopa) in patients with advanced‐stage PD at individually titrated doses between 6 and 24 mg/day, where symptoms were measured at 11 visits over 24 weeks starting from baseline. ¹⁸ The analysis included the baseline data from the first trial and all data from the second trial. The postbaseline data from the first trial were not included; the lack of placebo arm made those data unsuitable for comparing the time course between the drug and placebo treatments. Both trials were approved by the relevant institutional review boards. Details on study designs, inclusion/exclusion criteria, and assessment schedules were reported in the original publications. ¹⁷ , ¹⁸ For summaries of patient characteristics and the data included in the current analysis, see Table 1.

TABLE 1.

Summary of patient characteristics and data used

Patient baseline characteristics
	Age, years, mean (min‐max)	Weight, kg, mean (min‐max)	Sex, count	Prior concomitant drugs, count
				Selegiline	Amantadine	Anticholinergics	Levodopa
Early PD, N = 161	61 (37–84)	77 (50–137)	Female 70 Male 91	No 114 Yes 47	No 132 Yes 29	No 143 Yes 18	No 161 Yes 0
Advanced PD, N = 391	66 (34–87)	74 (42–144)	Female 145 Male 246	No 302 Yes 89	No 300 Yes 91	No 381 Yes 10	No 0 Yes 391

UPDRS item score dataa
Early‐PD study, N = 161		Advanced‐PD study, N = 391
Baseline		Baseline placebo		Baseline ropinirole		Longitudinal placebo		Longitudinal ropinirole
N	n	N	n	N	n	N	n	N	n
161	7004	190	8330	201	8765	189	22,882	200	25,186

UPDRS total score dataa
Early‐PD study, N = 144		Advanced‐PD study, N = 389
Baseline		Baseline placebo		Baseline ropinirole		Longitudinal placebo		Longitudinal ropinirole
N	n	N	n	N	n	N	n	N	n
144	144	179	179	185	185	187	484	199	542

The analysis included baseline data from the early‐PD study, and both baseline and longitudinal data from the advanced‐PD study. The total score is the sum of the scores of 44 items.

Abbreviations: max, maximum; min, minimum; PD, Parkinson’s disease; UPDRS, Unified Parkinson's Disease Rating Scale.

^a N = number of patients; n = number of item scores.

Pharmacometric analyses

Initially, we developed two longitudinal models to describe the placebo/drug treatment effects in the advanced‐PD study on UPDRS total scores (total score model [TSM]) and on the disease severity estimated using the IRT (IRM). Based on these models, we performed simulations investigating the power for detecting the drug–placebo treatment difference. Model estimation and simulations were performed by means of nonlinear mixed effect modeling using NONMEM version 7.3. ¹⁹ , ²⁰

Item response model

The methodology for applying the IRT to pharmacometric modeling has been described previously in detail. ¹³ The UPDRS is designed to measure the PD disease severity through the individual item scores, and the IRT is well suited for the analysis of such composite scale data. Essentially, the more severe a patient's condition is, the more likely the patient would have a higher score for an item in the instrument (e.g., 0, 1, 2, etc. for normal, mild, moderate, etc., respectively). In other words, the probability for a patient to have a certain score on a given item is a function of that patient's disease severity. The IRT thus uses the probability functions for all scores of all items collectively to estimate the disease severity for each patient at a given time. We applied a previously reported IRT model for UPDRS, which was developed using the baseline and placebo data from the two studies included here. The model included three latent severity variables reflecting certain aspects of the symptom severity and estimated through items that were (i) patient reported (items 1–17), (ii) clinician assessed and nonsided (items 18, 19, 20, 22, 27–31), and (iii) clinician assessed and sided (items 20, 21, 22, 23–26, evaluated for the right and left sides) for which a mixture component estimated the proportion of two subpopulations depending on their more disabled (dominant) side at baseline. Correlations between the latent variables were estimated within each study. Further information can be found in the original publication. ⁶

Longitudinal models

The conceptual framework guiding this analysis is illustrated in Figure 1, which shows the longitudinal models of disease progression, the effect of a placebo treatment, the effect of drugs that cause symptom relief without affecting the progression, and the effect of drugs that slow down the progression.

FIGURE 1.

Longitudinal model with arbitrary scales to illustrate the importance of trial duration for correctly describing drug effects on the worsening of a symptom over time. The black dotted line indicates an underlying natural progression, which is not observable in a trial. The red line indicates observations for placebo treatment. The purple line indicates observations for symptom‐relief agents. The pink line indicates observations for the progression slow‐down agents. The left panel shows a trial of adequate duration. The right panel shows a shorter trial is inadequate for correctly describing these effects

It is well established that the symptom progression of PD, as measured by the UPDRS, is approximately linear over time over a relatively short period such as the duration of a clinical trial. ²¹ , ²² , ²³ Many reports have shown that during a double‐blind trial, the placebo treatment typically improves the symptoms to a small extent without altering the progression rate—effectively causing a slight downward shift of the underlying disease progression—and such effect takes a short time to stabilize. ¹⁸ , ²⁴ , ²⁵ , ²⁶ The onset of this effect was modeled as an exponential function of time as supported by the data pattern; a greater onset rate meant an earlier achievement or stabilization of the placebo effect. Equation (1) describes Symptom in terms of either the UPDRS total score or each of the latent symptom severity variables estimated by the IRT model, for placebo‐treated patient i at time t. In this equation, Baseline, Progression rate, Placebo, and Onset rate are the baseline, the progression rate over time, the placebo effect, and the onset rate of the placebo effect, respectively, for patient i.

$${\text{Symptom}}_i\left(t\right)=\text{Baseline}+\left(\text{Progression rate}\right)·t+\left(\text{Placebo}\right)·\left(1‐e^{‐\text{Onset rate}·t}\right)$$ (1)

Conceivably, a drug may relieve the symptoms further, also without altering the progression rate. In a placebo‐controlled trial, this drug effect would be reflected by a further shift of the time course, compared to the placebo effect. Equation (2) describes Symptom, also in terms of either the UPDRS total score or each of the latent symptom severity variables estimated by the IRT model, for drug‐treated patient i at time t. In this equation, Baseline, Progression rate, Placebo, Drug, and Onset rate are the baseline, the progression rate over time, the placebo effect, the drug effect, and the onset rate of the drug effect, respectively, for patient i.

$${\text{Symptom}}_i\left(t\right)=\text{Baseline}+\left(\text{Progression rate}\right)·t+\left(\text{Placebo + Drug}\right)·\left(1‐e^{‐\text{Onset rate}·t}\right)$$ (2)

Alternatively, the drug may slow down the progression, so patients treated by the drug would show a different underlying progression rate while maintaining the placebo effect on the symptoms. ²⁷ This is described in Equation (3), where Drug is the drug effect on the progression rate, and all other parameters are the same as in Equation 2.

$${\text{Symptom}}_i\left(t\right)=\text{Baseline}+\left(\text{Progression rate + Drug}\right)·t+\left(\text{Placebo}\right)·\left(1‐e^{‐\text{Onset rate}·t}\right)$$ (3)

Therefore, the effect of the symptom‐relief drugs is reflected as a downward parallel shift from the time course of placebo, whereas the effect of the progression‐altering drugs is reflected as an eventual divergence from the placebo time course (Figure 1).

We fitted the longitudinal models described in Equations ((1), (2), (3)) using both the IRM (where Symptom_i represents each of the symptom severity variables estimated by IRT) and the TSM (where Symptom_i represents UPDRS total score) to explore whether ropinirole altered the symptom progression rate or simply showed greater symptom relief than placebo. Description on the individual level was ensured through a random effect associated with each of the symptom severity variables in the IRM and a random effect associated with some of the fixed effect parameters in the TSM. The IRM included multiple latent variables estimated from different UPDRS items, and we estimated the placebo and drug effects on each latent variable separately. The fit was performed on all data, the Drug parameter was assumed null for the placebo data and estimated for the drug data.

Evaluation of study power

For a comparison of power between the IRM and the TSM, we generated the power curves using both longitudinal models by employing a modified Monte Carlo mapped power (MCMP) approach. ²⁰ , ²⁸ Compared with alternative methods, ²⁹ , ³⁰ the MCMP approach offers greater computational efficiency by drawing samples from the results of model fitting to a single (large) dataset, instead of requiring model fitting to a large number of sample data sets.

In the MCMP, a stochastic simulation is preceded by the estimation of full and reduced models, followed by hypothesis testing using the likelihood ratio test (LRT), where the overall likelihood is substituted by the sum of individual contributions to the likelihood. In NONMEM, this corresponds to the sum of each individual's contribution to the objective function value (iOFV), the OFV being approximately proportional to minus twice the natural logarithm of the likelihood of the data.¹⁹ In our case, the MCMP was modified such that the “observed” iOFVs were used (no simulation) based on the real data.

Thus, we fitted two models to the data from both ropinirole‐treated and placebo‐treated patients in the advanced‐PD trial—a full model that included a drug effect for the ropinirole arm and a reduced model not including any drug effect but otherwise identical—to generate the iOFV for each patient. The difference in iOFV between the full and reduced models (ΔiOFVs) was calculated for each patient. We then drew 10,000 treatment‐stratified virtual trial data sets of ΔiOFV values for a wide range of samples sizes. The LRT was then applied to the sum of the ΔiOFVs from all patients in that trial (ΣΔiOFVs), which is approximately χ² distributed. For each sample size, the proportion of the virtual trials that showed a difference in the ΣΔiOFVs corresponding to p = 0.05 or 0.1 for the χ² distribution would represent the study power at that p value. The actual critical ΣΔiOFVs values used in the MCMP were derived through the randomization test in Perl speaks NONMEM applied to the observed data of the placebo patients letting them be assigned randomly to active and placebo treatments. ²⁰ , ³¹ The critical values for the IRM were 11.62 (p = 0.05) and 8.10 (p = 0.1), and the corresponding values for the TSM were 4.00 and 2.76.

RESULTS

Understanding the time course of placebo and drug effects

When the longitudinal IRM was fitted to the estimated disease severity, the disease Progression rate could not be estimated with adequate precision. This was somewhat expected because of the short treatment duration of 24 weeks and the small data set, considering the slow progression and high variability in the data (Figure 2). Effectively, without the Progression rate, the model describing the placebo effect and drug effect time courses took the form of Equation (4). In this model, the placebo produced a gradual effect of symptom relief, and the drug produced a greater relief.

$${\text{Symptom}}_i\left(t\right)=\text{Baseline}+\left(\text{Placebo + Drug}\right)·\left(1‐e^{‐\text{Onset rate}·t}\right)$$ (4)

FIGURE 2.

Visual predictive checks for total scores simulated from the item response model (upper) and total score model (lower). The lines describe the 5th, 50th, and 95th percentiles of the observations; the bands are the model‐simulated 90% confidence intervals of the corresponding percentiles

The data supported the estimation of separate parameters for the Placebo effect and the Onset rate on the four latent variables (Figure 3). The Placebo effects were −0.167, −0.172, −0.300, and −0.059 for the latent variables reflecting patient‐reported items, clinician‐assessed nonsided items, clinician‐assessed items on the initially more severe side, and clinician‐assessed items on the initially less severe side, respectively. The Onset rate of the Placebo and Drug effects for these latent variables varied slightly at 0.15 – 0.3 week⁻¹ (0.15, 0.30, 0.27, and 0.22 week⁻¹ for patient‐reported, nonsided, initially less‐severe side, and initially more‐severe side items, respectively). The data were insufficient to estimate separate Drug effects on the different latent variables; a common symptomatic effect of −0.539 was estimated for all latent variables

FIGURE 3.

The model‐estimated speed and magnitude of change from baseline for the latent variables. See the “Item Response Model” section in “Methods” for descriptions of the multiple latent variables

Similarly, when the longitudinal TSM was fitted to the observed total score data, the Placebo effect was successfully estimated, but the disease progression rate could not be described (p > 0.05). The model parameters of Placebo and symptom‐relief Drug effects were estimated to be −3.7 and −6.5 respectively, with an Onset rate of 0.13 week⁻¹.

Both longitudinal TSM and IRM exhibited adequate predictive performance when compared with the total UPDRS observations (Figure 2). The TSM‐estimated median (90% confidence interval) change in total UPDRS at week 24 for placebo and ropinirole groups were −3.7 (−5.4 to −2.0) and −9.3 (−11 to −7.3), respectively. Similarly, the IRM‐simulated corresponding values were −2.0 (−4.0 to −1.0) and −9.0 (−11 to −8.0).

The estimated model parameters with their precision and corresponding NONMEM control streams are presented in the Supplementary Material.

Evaluation of study power

The sample size comparison for detecting a drug effect by the IRM versus the TSM analysis generated by the modified MCMP, in a parallel study of equal randomization, is shown in Figure 4. For the same study power, applying the IRM needed fewer patients than applying the TSM. At the significance level of 0.05, the sample sizes required for 80% power to detect a drug effect by the TSM and IRM were found to be 88 and 58 patients (34% smaller for the IRM), and the corresponding sample sizes for 90% power would be 126 and 90 patients (29% smaller for the IRM). At the less stringent significance level of 0.1, the sample sizes required for 80% power to detect a drug effect by the TSM and IRM would be 70 and 48 patients (31% smaller for the IRM).

FIGURE 4.

Power for detecting the drug effect (significance levels 0.1 and 0.05, one degree of freedom) based on analyses using a longitudinal item response model or total score model

DISCUSSION

There is a pressing need for drugs that can slow down functional deterioration in PD. Differentiating drugs that change the disease progression trajectory from drugs that relieve the symptoms without altering the progression trajectory is a challenge for drug developers and approvers alike. For example, there is inconsistent evidence whether l‐dopa, the standard‐of‐care pharmacotherapy for PD, slows down the disease progression. ²⁴ Based on the paradigm shown in Figure 1, we conducted longitudinal modeling to test whether ropinirole, an approved dopaminergic agent, showed evidence of altering the disease progression rate. Both the longitudinal TSM and IRM adequately estimated the placebo effect and its onset rate. The models also successfully quantified the magnitude of the drug effect; however, it is important to point out that the parameters estimated from the short trial in patients with advanced PD may not be applicable to other patient populations or trial settings.

Neuroimaging data from patients with PD have suggested possible neuroprotection by ropinirole. ³² However, neither model could estimate the underlying progression rate and hence were unable to test whether the drug altered this progression rate. This was not unexpected: the treatment duration was 24 weeks—most likely too short to observe any notable progression, as noted by the original trial publication. ¹⁸

These findings highlighted the importance of the treatment duration: a short trial does not provide enough information to describe the underlying progression rate; it could even provide misleading information on a placebo or drug effect. For example, observations in the early phase of the trial for a symptom‐relief drug described in Figure 1 might erroneously show a divergence between the placebo and drug responses, suggesting a change in the progression rate. Similarly, observation times are also important for differentiating the nature of the drug effect: infrequent observations in the early phase of a trial could also lead to unreliable data for this differentiation. ³³ Crucially, the claim of whether a drug changes the disease progression should not rely solely on the slope of the symptom progression; it must also be supported by biological evidence. ³³

The differential progression among different aspects of the symptoms in PD is well recognized. ²¹ , ²² , ³⁴ It is conceivable that a drug or placebo treatment can relieve the symptoms or slow down the progression of these different aspects to varying extent. The IRM used in this investigation included multiple latent variables, representing patient‐reported, nonsided, and sided items. ⁶ , ⁸ Data from the advanced‐PD study suggested that the effect of placebo treatment varied for the different latent variables (Figure 3). The data could not differentiate the drug effect on the respective latent variables. These observations speak to the utility of item response analysis in understanding the differentiated symptom profile in an integrated fashion and assessing the placebo or drug effect on this profile; they also prompt the need for further investigations in larger and longer trials in more diverse patients.

Sample size requirement for detecting a drug effect depends on the size of the effect as well as the assessment schedule and duration of the trial. Given the nature of the data set that we used—the observed magnitude of symptom relief and the observation frequency and schedule—our analysis suggested reduced sample size with the IRM compared with the TSM. For example, the sample sizes saving for detecting the drug effect with 80% power at p = 0.05 would be 34%.

The potential sample size savings by IRM analysis of PD trials has been previously reported. ⁷ Buatois et al. ⁷ built a linear longitudinal IRM using 48‐month MDS‐UPDRS observational (instead of clinical trial) data. They then used the model to simulate longitudinal item–level data for a hypothetical drug that would reduce the slope of the progression rate, derived the total scores from the item scores (in absence of real data), and demonstrated notable potential savings in sample size by item response analysis compared with total score analysis for detecting the effect of a hypothetical drug on the progression slope. The exercise did not consider the placebo effect in a trial.

In comparison, the placebo time course captured by our model would be essential for simulation‐based design of future trials: it informs the assessment time requirement for understanding the nature of the drug effect and for optimizing trial efficiency. Furthermore, deriving total scores from an IRM and subsequently comparing the efficiency between the TSM and the IRM could conceivably bias in favor of the IRM approach. Our work using real total score data during both placebo and drug treatments in a historical trial conceivably allowed a more reliable and fairer sample size comparison for the two approaches, complementing and confirming their findings.

In summary, this work generated strong evidence that item response analysis of the UPDRS could significantly reduce the sample size for drug effect detection compared with the standard total score approach. We also propose that longitudinal modeling could be used to explore whether a drug slows down symptom progression. Trials for drugs aimed at slowing down neurodegeneration are large and lengthy because of slow disease progression, a large placebo effect, and high end‐point variability. Better understanding of these emerging methodologies through broad application will be highly valuable in the pursuit of disease‐modifying treatments, especially benefiting trials for small patient populations.

CONFLICT OF INTEREST

C.C. and S.Y. conducted this research as salaried employees of GlaxoSmithKline. S.J., E.L.P., and M.O.K. conducted this research as salaried employees of Uppsala University. The authors’ employment did not directly influence this work.

AUTHOR CONTRIBUTIONS

C.C. and S.J. wrote the manuscript. C.C., M.O.K., and E.L.P. designed the research. C.C., S.J., S.Y., E.L.P., and M.O.K. performed the research. S.J. analyzed the data.

Supporting information

Supplementary Material