Pseudoreplication Can Lead to an Apparent Association of Increasing Proton Linear Energy Transfer With Radiation-Associated Image Changes in Cohorts Where No Such Relationship Exists

Introduction

Spatially varying proton relative biological effectiveness (RBE) in vitro, as measured by cellular clonogenic survival assays, is well established.1, 2, 3, 4 Prior to 2016, concerns about proton range uncertainties and proton-RBE were raised by some retrospective studies of proton therapy for ocular, prostate, esophageal, and brain malignancies, which reported toxicity rates similar to or exceeding those observed following treatment with megavoltage photon therapy.5, 6, 7, 8 It was not until a 2016 study by Peeler et al⁹ that quantitative clinical evidence linked proton-RBE to the incidence of posttreatment toxicities.⁹ Peeler et al⁹ found an association between track-averaged linear energy transfer (LET_t) and radiation-associated image changes (RAIC) in posttreatment T2-weighted magnetic resonance imaging (MRI) series of the brain.

At present, contradictory evidence exists in the literature for whether a relationship between linear energy transfer (LET) and RAIC can be observed following proton therapy. In the decade following the Peeler et al⁹ study, several related retrospective studies have been conducted, yielding inconsistent findings.10, 11, 12, 13, 14, 15 As LET optimization in proton therapy is increasingly considered, reconciling sources of disagreement between clinical studies of proton LET and tissue toxicity is important to inform whether and to what extent it is appropriate to perturb the dose distribution to optimize proton LET.

In this work, several analytical approaches established in prior studies investigating the connection between RAIC and proton LET were each applied to a single data set (the cohort from Peeler et al⁹). The goal of this work was to demonstrate whether methodological differences can, in part, explain the contradictory results observed in the literature regarding the association between proton LET and RAIC. Additionally, this work provides guidance on the appropriateness of published analysis techniques and best practices for future studies in this field.

Methods and Materials

Cohort description

A cohort of 34 pediatric patients with ependymoma who were treated with passively scattered proton therapy were included in the study cohort; this cohort was identical to that analyzed by Peeler et al⁹. Of the 34 patients, 14 presented with hyperintense RAIC regions on follow-up T2-weighted fluid-attenuated inversion recovery MRI. The hyperintense regions were contoured and transferred to the planning computed tomography scan via rigid registration. Demographic information about the cohort is described in Table 1 of Peeler et al.⁹

Table 1.

Model parameters, 95% CIs, and associated P values for both probit regression models developed

Without ventricular proximity
Parameter	Estimate	95% CI	P value
${\kappa }_1$ (intercept)	−5.40	−5.44, −5.36	0*
${\kappa }_2$ (dose)	2.08	2.06, 2.10	0*
${\kappa }_3$ (LET)	0.80	0.78, 0.81	0*

With ventricular proximity
Parameter	Estimate	95% CI	P value
${\kappa }_1$ (intercept)	−5.38	−5.42, −5.34	0*
${\kappa }_2$ (dose)	2.08	2.06, 2.11	0*
${\kappa }_3$ (LET)	0.80	0.79, 0.82	0*
${\kappa }_4$(ventricles)	−0.22	−0.23, −0.20	.0051

^⁎The P value is so small that it could not be represented as a floating-point number. κ_i refers to the i-th fitting parameter.

Naïve voxel-wise analysis

The simplest of the 4 analysis methods we conducted for extracting relationships between RAIC and LET_t involved pooling voxels from all patients and determining the fraction of voxels within an image change contour, referred to as the image change proportion (ICP). This analysis is inspired by a recent study of lung fibrosis induction.¹⁶ By taking the ratio of doses required to achieve the same ICP at different LET_t, a clinical RBE is determined.

Probit regression

A probit regression model, equivalent to that developed by Peeler et al,⁹ was applied to the pooled voxel data of all patients. A probit link function connects the probability of RAIC to a risk model, which is functionally equivalent to using the Lyman normal tissue complication probability model.¹⁷ Two risk models were developed: one containing dose and LET_t as predictors, and another additionally containing a binary predictor term for regions within 4 mm of the ventricles of the brain.

All fitting was conducted using MATLAB R2023b (MathWorks Inc) and the “fitglm” function. Model parameters were estimated by averaging the results of leave-one-out cross-validation across the 14 patients with image changes. Consistent with the assumptions underlying the model, the standard errors and residual degrees of freedom were determined by treating each voxel as an independent observation.

Comparison of LET differences between dose-matched voxels

The work of Niemierko et al¹³ introduced a dose-matching approach that compares a patient’s voxels within an image change region, to that patient’s voxels receiving a similar dose outside an image change region. By using one’s own dose-matched voxels as a reference, one can determine whether heightened LET exists within image change regions.

We have reproduced the greedy dose-matching algorithm employed by Niemierko et al¹³ as faithfully as possible. Our algorithm for dose matching involved taking each voxel within an RAIC region and randomly grabbing a voxel with a dose within 0.1 Gy of it outside the image change region. This matching was performed without replacement. The difference in LET_t was calculated for each voxel pair.

Mixed-effects probit regression

Additionally, we generalized the probit regression approach employed by Peeler et al⁹ to a mixed-effects probit regression model. Mixed-effects models characterize population-level averages (fixed effects) and patient-by-patient radiosensitivity (random effects). Two mixed-effects probit models were developed: one with fixed effects for dose and LET_t, and random effects for LET_t; and another with fixed effects for dose, LET_t, and ventricular proximity, and random effect terms for LET_t and ventricular proximity. All fitting was conducted using MATLAB R2023b using the “fitglme” function.

Results

Naïve voxel-wise analysis

For ranges of LET_t, the total proportion of RAIC voxels as a function of dose is shown in Fig. 1A. As LET_t increased, a lower dose was required to achieve a given necrotic proportion. Clinical RBEs for the same ICP approaching 2 were observed for proton LET_t of 4 to 5 keV/$\mu$m compared with the reference curve of 2 to 3 keV/$\mu$m (Fig. 1B).

Figure 1.

(A) Image change proportion as a function of dose in a series of track-averaged linear energy transfer ranges. (B) Measured relative biological effectiveness image change (RBE_IC) as a function of dose.

Probit regression

The fitting parameters and associated P values for 2 probit regression models for probability of RAIC development are shown in Table 1. As in the original Peeler et al⁹ analysis, a positive correlation between increasing dose, increasing LET_t, and increasing RAIC risk was observed. Unlike prior studies of adult glioma patients, a lower risk of image change was observed in regions within 4 mm of ventricles. Wald tests for each parameter indicate that all predictors were statistically significant and the null hypothesis could be rejected.

Comparison of LET differences between dose-matched voxels

The average difference in LET_t between RAIC regions and non-image change regions, as determined using the dose-matching methodology, is depicted in Fig. 2. Only 5 of 14 patients showed enhanced LET_t in their RAIC contours compared with other regions of the brain. When the differences in LET_t in dose-matched voxels for all 14 patients were combined using the weighted meta-analysis approach described by Niemierko et al,¹³ an average difference of −0.18 keV/$\mu$m (95% CI, −0.34 to −0.02 keV/$\mu$m; P = .03) was observed between RAIC regions and non-image change regions.^13,18,19 The I² statistic was 99.97% (P = 0), indicating that variation between patients was not explained by random chance alone and it would not be appropriate to pool data from patients together.²⁰

Figure 2.

Average difference in track-averaged linear energy transfer (LET_t) between the dose-matched radiation-associated image change region and healthy regions for each of the 14 patients with image changes. Error bars for individual patients represent the 95% confidence interval determined from the standard error of the mean. Confidence intervals for the combined data were calculated using the meta-analysis approach outlined by Dersimonian and Laird.¹⁸

Mixed-effects probit regression

Fit parameters, 95% confidence intervals (CIs), and associated P values for the developed mixed-effects probit regression models developed are provided in Table 2. Fixed effects for LET_t were statistically insignificant for predicting RAIC in a mixed-effects model. In the mixed-effects model with random and fixed effects for LET_t and ventricular proximity, the model estimate for the LET_t coefficient was −0.30 (95% CI, −0.96 to 0.36; P = .4). The standard deviation (SD) of the random-effects parameter for LET was 1.27. Together, these values indicate that the contribution of LET_t to image change risk is negligible in this patient population; however, at the patient-by-patient level, there is substantial variation in whether LET_t is associated with increased or reduced RAIC risk.

Table 2.

Fixed-effects model fit parameters, 95% CIs, and associated P values for both the mixed-effects probit models developed

Without ventricular proximity
Parameter	Estimate	95% CI	P value
${\kappa }_1$ (intercept)	−8.28	−10.1, −6.42	3 × 10-18
${\kappa }_2$ (dose)	1.74	1.72, 1.76	0*
${\kappa }_3$ (LET)	−0.27	−0.93, 0.38	.4
σ(${\eta }_1$) (intercept)	3.78	-	-
σ(${\eta }_3$) (LET)	1.24	-	-

With ventricular proximity
Parameter	Estimate	95% CI	P value
${\kappa }_1$ (intercept)	−8.32	−10.2, −6.48	9 × 10-19
${\kappa }_2$ (dose)	1.80	1.77, 1.82	0*
${\kappa }_3$ (LET)	−0.30	−0.96, 0.36	.4
${\kappa }_4$ (ventricles)	−0.65	−1.22, −0.077	.03
σ(${\eta }_1$) (intercept)	3.73	-	-
σ(${\eta }_3$) (LET)	1.27	-	-
σ(${\eta }_4$) (ventricles)	1.05	-	-

^⁎The P value is so small that it could not be represented as a floating-point number. κ_i refers to the i-th fitting parameter. SDs of random effects parameters are indicated by σ.

The ventricular proximity coefficient was −0.65 (95% CI, −1.22 to −0.077; P = .03), with a random-effects SD of 1.05. In this patient population, ventricular proximity had a statistically significant association with reduced image change risk.

The sum of fixed and random effects for LET_t (i.e., each patient’s individual correlation between LET_t and RAIC) is shown for the mixed-effects model not containing ventricular proximity (Fig. 3A). Of 14 patients who developed image changes, 8 showed a positive correlation between LET_t and RAIC within their image change contours. Patient 9 was an extreme outlier. Refitting the model, excluding patient 9, still yielded a statistically insignificant fixed-effects parameter estimate for LET_t of −0.038 (95% CI, −0.42 to 0.34; P = .8).

Figure 3.

(A) Fixed and random effect coefficients for each of the 14 patients with image changes. (B) The number of radiation-associated image change voxels for each patient. The color of the bar indicates whether the random- and fixed-effects term for the track-averaged linear energy transfer (LET_t) for that patient was positive (blue) or negative (red).

The number of RAIC voxels for each of the 14 patients who developed image changes is plotted, along with whether the sum of their fixed- and random-effects parameters for LET_t is positive or negative. The 2 patients with the largest numbers of RAIC voxels in the study, who contributed 67% of all image change voxels, both demonstrated an association between LET_t and increased image change risk (Fig. 3B).

Discussion

Does evidence of spatially variable proton-RBE exist in this cohort?

In this work, four established analysis techniques were applied to the patient population in which it was claimed that quantitative clinical evidence of spatially variable proton-RBE could be first observed.^9,13,15,16 The two analysis approaches, which pooled voxel data from all patients, yielded a positive correlation between RAIC and increased proton LET_t. The other approaches considered associations between RAIC and proton LET_t at the patient-by-patient level and did not find a statistically significant relationship between the two.

The reason for the discrepancy in findings across the different analysis approaches becomes clear once one realizes that the naïve voxel-wise and probit regression analyses replicated in this study have made a common data mishandling practice known as pseudoreplication.²¹ Pseudoreplication occurs when highly correlated data points are erroneously treated as statistically independent observations. Pseudoreplication can lead to enormously underestimated type I uncertainties (ie, the unrealistically small 95% CIs reported in Table 1). In this patient cohort, a failure to control for pseudoreplication allowed for a minority of patients whose image changes contained up to 200 times as many voxels as other patients being overrepresented in the model, drowning out the signal (Fig. 3, right). This is a relevant finding because even very recent studies on RAIC following proton therapy continue to develop predictive models trained on voxel data pooled across all patients (albeit with an upper threshold on image change size to somewhat prevent large image changes from overwhelming the model).²²

Factors that may obscure clinical evidence of spatially varying proton-RBE

The results of this study do not in any way contradict the claim that spatially varying proton-RBE effects exist in vivo. That spatially varying proton-RBE exists is well established. However, in heterogeneous patient populations, the impact of proton-RBE can be obscured by confounding variables and uncertainties. This study demonstrates the difficulty of measuring those effects and how improper data handling can lead to incorrect conclusions being drawn.

We believe that in order to maximize the probability of observing a measurable association between proton LET and tissue toxicities, not only will larger study sizes be needed, but also extreme care to minimize confounding factors will be required. Whenever possible, we recommend that studies of the relationship between voxel-based imaging outcomes following proton therapy and voxel-level predictors: (1) avoid unjustified pooling (pseudoreplication) of voxel-level data across patients; (2) attempt to isolate comorbidities and confounding of clinical/demographic factors (prior surgery, chemotherapy, age, baseline health status, etc); (3) make use of deformable image registration techniques to link post-treatment MRI sets to the image set on which dose and LET are calculated; (4) accommodate for the growth and temporal development of RAIC (e.g., see Cogno et al²³); and (5) avoid averaging of LET across multiple beams or make use of more suitable microdosimetric descriptors of proton energy deposition density.24, 25, 26, 27 Under these conditions, we believe it is more likely that an accurate determination of the magnitude of spatially variable proton-RBE effects and their association with post-treatment toxicities will be observed.

Conclusions

Reanalysis of the original cohort, which revealed quantitative clinical evidence of spatially variable proton-RBE, indicates that a statistically significant correlation between image change risk and increasing LET_t does not exist in this patient population. Careful choice of modeling approach is needed to avoid pseudoreplication and prevent patients with more voxel data from introducing bias. The choice of analysis method used to derive the relationship between LET and image change risk can have a profound impact on whether evidence of spatially variable proton-RBE is observed.

Disclosures

Joseph M. DeCunha: receives funding from the Cancer Prevention Research Institute of Texas and the Natural Sciences and Engineering Research Council of Canada. Christopher Peeler: none to declare. Dragan Mirkovic: none to declare. Uwe Titt: none to declare. Pablo Yepes: receives funding from the National Institutes of Health and the University of Texas MD Anderson Cancer Center. Mark Newpower: receives funding from the Oklahoma Clinical and Translational Science Institute; has a research agreement with RaySearch Laboratories, which is unrelated to this work. David Grosshans: none to declare. Radhe Mohan: receives funding from the National Institutes of Health and holds financial assets in GE and Berkshire Hathaway.