Validation of a Prognostic Index for Huntington's Disease

Abstract

Background

The aim was to develop a prognostic index for motor diagnosis in Huntington's disease and examine its predictive performance in external observational studies.

Methods

The pre-diagnosis Neuro-biological Predictors of Huntington's Disease study (N=945 gene-positive) was used to select a Cox regression model for computing a prognostic index. Cross-validation was used for selecting a model with good internal validity performance using the research sites as natural splits of the data set. Then the external predictive performance was assessed using pre-diagnosis data from three additional observational studies, The Cooperative Huntington Observational Research Trial (N=358), TRACK-HD (N=118), and REGISTRY (N=480).

Results

Model selection yielded a prognostic index computed as the weighted combination of the UHDRS total motor score, symbol digit modalities test, baseline age, and cytosine-adenine-guanine expansion. External predictive performance was very good for the first two of the three studies, with the third being a much more progressed cohort than the other studies. The databases were pooled and a final Cox regression model was estimated. The regression coefficients were scaled to produce the prognostic index for Huntington's disease, and a normed version, which is scaled relative to a 10-year 50% probability of motor diagnosis.

Conclusion

The positive results of this comprehensive validity analysis provide evidence that the prognostic index is generally useful for predicting Huntington's disease progression in terms of risk of future motor diagnosis. The variables for the index are routinely collected in on-going observational studies and the index can be used to identify cohorts for clinical trial recruitment.

Introduction

Huntington's disease (HD) is an inherited progressive neuro-degenerative disorder caused by a cytosine-adenine-guanine (CAG) repeat expansion in the HTT gene. HD signs and symptoms include motor, cognitive, and psychiatric features. There are no current treatments to slow the progression of the disease, but early-phase clinical trials have commenced with the ultimate goal of changing the disease course¹.

Resources for trial recruitment include clinical research platforms, such as Enroll-HD². Enroll-HD has an embedded observational study with regular visits at which clinical data are collected. Characterizing the progression level of candidate participants is essential for helping to recruit individuals for whom a treatment has measurable efficacy.

Pre-diagnosis progression has traditionally been characterized by a combination of age and CAG expansion because of the well-known association with the timing of motor onset^3–5. Additional work has shown that the prediction of motor diagnosis, defined as the highest examiner rating on the UHDRS diagnostic confidence level (DCL)⁶, is enhanced by considering clinical variables along with the influence of CAG length⁷.

Recently, data from several HD observational studies has become available: PREDICT-HD⁸, COHORT⁹, TRACK-HD¹⁰, and REGISTRY¹¹. The databases provide a unique opportunity to develop a prognostic index for HD (PI_HD) and externally validate its predictive performance. Developing a PI_HD that can be used across studies would be useful for estimating progression levels in pre-diagnosis HD and recruiting appropriate candidates for clinical trials. For example, the PI_HD can be useful when researchers want to recruit individuals from disparate sources who are just a few years away from estimated diagnosis.

In this study, the pre-diagnosis PREDICT-HD database will be used to develop a PI_HD using CAG, baseline age (age at visit entry), and UHDRS motor and cognitive variables that are common among the studies. Imaging and other technology intensive variables will be omitted for the pragmatic reasons that they are not measured in all studies and existing registries like Enroll-HD do not have such variables. Once the PI_HD is developed with the PREDICT-HD database, the PI_HD will be externally validated in the remaining studies by predicting the timing of motor diagnosis in the remaining pre-diagnosis gene-positive HD individuals.

Methods

Study population

Neuro-biological Predictors of Huntington's Disease (PREDICT-HD) is a longitudinal observational study of pre-diagnosis HD with 32 sites in six countries (AUS, CAN, DEU, ESP, GBR, USA)^8,12–15 with data collected 2002-2014. TRACK-HD is a longitudinal prospective observational study of pre-diagnosis and early HD with four sites in four countries (CAN, FRA, GBR, NE)^10,16,17 with data collected 2008-2011. The Cooperative Huntington Observational Research Trial (COHORT) is a longitudinal observational study of HD or at-risk individuals with 38 sites in three countries (AUS, CAN, USA)⁹ with data collected 2006-2011. REGISTRY is a longitudinal observational study that includes pre-diagnosis HD, manifest (diagnosed) HD, and at-risk individuals, with a total of 150 mostly European sites^11,18 with data collected 2004-2012.

HD gene carriers who did not have a motor diagnosis at baseline (study entry) were used for the analysis. Motor diagnosis was defined as DCL = 4, which is the highest rating and indicates that the examiner has ≥ 99% confidence that the patient exhibits unequivocal HD motor signs⁶. Additional inclusion criteria were ≥ 18 years of age (REGISTRY did not exclude Juvenile HD), a lab-confirmed CAG ≥ 36 (CAG range among studies was 36-66), and complete data on the variables for the analysis. Sample sizes and descriptive statistics for key variables at baseline after applying the inclusion criteria are shown in Table 1.

Table 1.

Descriptive statistics for variables measured at study entry for the pre-diagnosis participants meeting the analysis inclusion criteria. Mean (SD) for quantitative variables and count (percentage) for categorical variables.

	PREDICT	COHORT	TRACK	REGISTRY
N	945	358	118	480
Sites	32	38	4	89
Site Sample Size	28.64 (16.42)	9.42 (8.89)	29.50 (0.58)	5.39 (6.02)
Female	605 (64)	225 (63)	64 (54)	257 (54)
Follow-up(Years)	4.83 (2.95)	2.06 (1.02)	2.71 (0.64)	2.34 (2.13)
Motor DX	225 (24)	83 (23)	21 (18)	319 (66)
Age	40.40 (10.36)	42.06 (12.51)	40.80 (8.86)	45.19 (12.18)
CAG	42.42 (2.68)	42.41 (2.81)	43.14 (2.41)	43.91 (3.73)
DCL=3	44 (5)	46 (13)	2 (2)	136 (28)
TMS	4.93 (5.29)	6.20 (8.39)	2.53 (1.68)	12.07 (15.61)
STROOP	98.88 (17.37)	90.62 (19.87)	99.67 (16.10)	79.41 (24.86)
SDMT	50.64 (11.57)	44.93 (12.11)	51.41 (10.24)	38.48 (17.01)

Note. Motor DX = prospective motor diagnosis (DCL = 4); CAG = cytosine-adenine-guanine expansion; TMS = total motor score; STROOP = Stroop word, SDMT = symbol digit modalities test.

Study activities were reviewed and approved by institutional review boards (PREDICT-HD, COHORT) or local ethics committees (TRACK-HD, REGISTRY). Informed consent procedures were carried out for each participant, and signed consents for participation and the distribution of de-identified data for collaborative research were obtained.

Statistical analysis

The first analysis used the PREDICT-HD database for variable selection. Ten pre-planned Cox regression models were specified based on previous work with PREDICT-HD⁷. The models were specified to represent different levels of parsimony, with the full model (Model 1) having the most predictors (see Figure 1). All models used the predictor scores at baseline to predict the hazard of motor diagnosis (the first occurrence of DCL = 4). A method to reduce bias in model selection is cross-validation (CV), which is often used with several splits of data in order to reduce variance. The participating research sites in PREDICT-HD provided natural and convenient splits (subgroups of participants). Thus, leave-one-site-out CV (LOSO-CV) was used for model selection¹⁹. For LOSO-CV, each Cox model was estimated using all the sites but one, and then the performance of the model for predicting the timing of motor diagnosis was evaluated in the omitted site. The process was repeated leaving each site out in turn, resulting in a predictive performance index for all 32 sites. For each estimated Cox model, the PI_HD was computed as the weighted combination of the regression weights and the predictor scores. Focus was on prediction, so the proportional hazards assumption was not a primary concern. However, Schoenfeld residuals showed general consistency with the proportional hazards assumption for the main model discussed below.

Fig. 1.

Boxplots and individual values (circles) of Harrell's C for 10 models among the PREDICT sites. TMS, total motor score; STROOP, Stroopword; SDMT, symbol digit modalities test; AGE, baseline age; CAP5Age × (CAG - 34).

Predictive performance was assessed using the integrated Brier score²⁰, Harrell's C (concordance) statistic²¹, and a C statistic weighted for censoring²². The indexes provided similar model evaluations and we focus on Harrell's C because it is the most commonly reported (extended results are presented in the appendix). C indexes the ability of the PI_HD to discriminate between individuals with longer time to motor diagnosis and those with shorter time. C is similar to the area under the ROC curve in sensitivity/specificity analysis, but it accounts for the right-censoring due to dropout and study termination. C = 0.5 indicates no better than chance prediction of the timing of motor diagnosis, whereas C = 1 is perfect prediction (each pair of individuals is correctly ordered in terms of timing of diagnosis). A guideline for the evaluation of the magnitude of C is provided by a survey of values found in external validation studies in oncology and cardiovascular disease²³. The survey showed mean C = 0.78, and we consider this value to be the benchmark for typical and acceptable predictive performance.

The second analysis was an external validation that included all the databases and used the model selected in the first analysis. To provide a benchmark for comparison, internal validation for each study was performed using LOSO-CV, and the mean C was computed among sites. For the external validation, the weights estimated from PREDICT-HD were used to compute the PI_HD in each other study, then C was computed. The Cox regression weights and standard errors (SEs) of the PI_HD predictors were estimated independently in each study (values were multiplied by 1000 to avoid very small numbers). A significance test of the difference between a study's regression weight and the corresponding PREDICT-HD regression weight was computed using a multiple-group analysis based on dummy coding with combined databases²⁴.

Risk groups for motor diagnosis were formed by using the quartiles of the PI_HD as computed in PREDICT-HD applied to all the studies. In PREDICT-HD the quartiles divided the PI_HD distribution into four equal groups (25% each). These same quartiles were used in the other studies, but they did not necessarily divide the other distributions into quarters. Survival was defined as the probability of not having a motor diagnosis, and survival curves were estimated using cubic splines²⁵ to provide smooth curves over the follow-up period. Similar cubic spline survival curves and bootstrapped 95% CIs were constructed for the pooled data as described below.

Results

Model selection results using only the PREDICT-HD database are shown in Figure 1. The figure consists of boxplots of distributions of Harrell's C among the sites for the 10 models. Predictive performance varied by site, with some sites showing worse than random performance for some models (C < 0.5), and other sites showing perfect performance (C = 1.0). Several models had similar good performance (Model 1 through Model 6), but the summary statistics slightly favored Model 4. Model 4 had the largest median (C = 0.87) and the largest lower and upper quartiles (25% = 0.82, 75% = 0.93) indicating its C distribution had the closest proximity to perfect prediction (shifted closest to the top of Figure 1). Similar results were shown for the Brier score and the weighted C, and Model 4 had the best aggregate rank among all the measures (see appendix). Therefore, we considered Model 4 to be an adequate model to carry forward for further consideration (other models could have been carried forward as well). In all subsequent analysis, the PI_HD was computed as the weighted composite of the UHDRS total motor score (TMS), symbol digit modalities test (SDMT), and the CAG-Age Product (CAP)⁵, with the latter being CAP = Age at baseline × (CAG – 34).

External validation results using C are shown in Table 2. For COHORT and TRACK-HD, the PI_HD external predictive performance (C_EXT) was very similar to the internal performance (C_INT). For example, COHORT had internal performance of C_INT = 0.85 and external performance of C_EXT = 0.84. The REGISTRY study showed low internal predictive performance (C_INT = 0.64) and even lower external performance (C_EXT = 0.56). The Cox regression weights for TRACK were larger in absolute value than their PREDICT-HD counterparts, and much more so for TMS. However, the estimated SEs were also large and there were no statistically significant differences. (Entry criteria for the TRACK-HD study included a maximum screening TMS of 5, and the restricted range may be related to these estimates.) The COHORT SDMT weight was significantly smaller than the PREDICT-HD weight, but the differences for TMS (larger) and CAP (smaller) were not significant. Each REGISTRY regression weight was significantly smaller in absolute value than the corresponding PREDICT-HD weight, which was indicative of the general weakness of the predictors in the REGISTRY database.

Table 2.

External validation results: C statistic (internal and external validation), Cox regression weights (SEs), and tests of difference with the PREDICT weights.

Study	CINT	CEXT	TMSa	SDMTa	CAPa
PREDICT	0.85		56.93 (9.44)	-35.01 (7.02)	7.63 (0.79)
COHORT	0.85	0.84	58.25 (12.71)	-22.45*** (13.09)	4.46 (1.41)
TRACK	0.79	0.79	219.99 (121.86)	-61.45 (27.21)	9.23 (5.39)
REGISTRY	0.64	0.56	0.43*** (5.27)	-5.56*** (5.05)	2.33** (0.45)

Note.

^aRegression weights and SEs are multiplied by 1000 to avoid small numbers;

C_INT = internal validation Harrell's C; C_EXT = external validation;

^***p<.001,

^**p<.01,

^*p<.05;

TMS = total motor score; SDMT = symbol digit modalities test; CAP = Age × (CAG - 34).

Figure 2 shows the smooth survival curves for the motor diagnosis risk groups along with the counts and percentages of group membership (survival is the probability of not being diagnosed). The risk gradient of the survival curves displayed by PREDICT-HD (upper left panel) was similar for COHORT (upper right) and TRACK-HD (lower left) for the early years (up to year 3) in the sense that the risk curves were in the same order from top to bottom (Low ≥ Mid-Low > Mid-High > High). In contrast, the curve ordering for REGISTRY (lower right) was inconsistent, as there were several order changes (crossing lines) in the early years among the three lower risk groups. However, similar to the other studies, the REGISTRY High risk group curve was consistently the lowest over time (there was some early overlapping with Mid-High). By design, the PI_HD quartiles define four equal groups for PREDICT-HD, but the same PREDICT-HD PI_HD cutoffs did not result in equal groups in the other studies. However, COHORT and TRACK-HD had a similar 50% split as PREDICT-HD into lower risk of motor diagnosis (Low + Mid-Low) and higher risk of motor diagnosis (Mid-High + High). On the other hand, REGISTRY had more skewed classification percentages, with 82% of the sample classified as having a higher risk of motor diagnosis.

Fig. 2.

Cubic spline survival curves for risk groups of motor diagnosis defined by the PREDICT-HD study cutoffs. Counts (percentages) are shown for each group.

Pooling among databases is justified when baseline hazards (and survival curves) are similar²⁶. Cubic spline estimates of the baseline hazards are shown in the appendix. Similar to Figure 2, there was reasonable similarity among PREDICT-HD, COHORT, and TRACK-HD. Thus, the three databases were combined, and a final Cox regression model was estimated. Based on the estimated Cox regression weights (multiplied by 1000 and rounded for simplicity), the formula is computed as

$${\mathit{\text{PI}}}_{\mathit{\text{HD}}}=51\times \mathit{\text{TMS}}+(-34)\times \mathit{\text{SDMT}}+7\times \mathit{\text{Age}}\times (\mathit{\text{CAG}}-34).$$

Larger values index greater risk of motor diagnosis, and hence, greater predicted progression. For example, if a person has TMS = 10, SDMT = 90, age = 40, and CAG = 42, then PI_HD = -310, whereas a person with TMS = 15, SDMT = 70, age = 50, and CAG = 42 has PI_HD = 1185. The PI_HD was “normalized” (scaled) to enhance interpretation. The survival curve was estimated for each PI_HD based on the fitted model. For the 10-year time point, the PI_HD value associated with a 0.5 survival probability was selected as the centering constant (= 883), and the SD of the PI_HD distribution was selected as the scaling constant (= 1044). The prognostic index normed for HD (PIN_HD) is

$${\mathit{\text{PIN}}}_{\mathit{\text{HD}}}=\frac{{\mathit{\text{PI}}}_{\mathit{\text{HD}}}-883}{1044}.$$

PIN_HD is in SD units and is interpreted relative to 50% 10-year survival. PIN_HD < 0 indicates greater than 50% 10-year survival, and PIN_HD > 0 indicates less than 50% 10-year survival. From the example above, the person with PI_HD = -310 has PIN_HD = -1.14 that is 1.14 SDs below the 50% 10-year survival mark, and the person with PI_HD = 1185 has PIN_HD = 0.29 that is 0.29 SDs above the mark. Figure 3 shows the survival curves and bootstrapped 95% CIs for different ranges of PIN_HD defined by a lower cutoff value. For example, a lower cutoff of 1 defines the right-tail range of PIN_HD ≥ 1 for the pooled data. For this cohort, the figure indicates that 40% are expected to have a motor diagnosis a little after 2 years (60% remain pre-diagnosis), and 70% are expected to have a motor diagnosis by 4 years (30% remain pre-diagnosis).

Fig. 3.

Cubic spline survival curves (95% CIs) for ranges of the prognositc index normed for Huntington's disease (PINHD). Curves are based on pooling PREDICT-HD, TRACK-HD, and COHORT.

Discussion

To our knowledge, this is the first external validation of a prognostic index for motor diagnosis (conversion) in pre-diagnosis HD. External validation is the gold standard for reproducibility^27,28, and the very good predictive performance in two out of the three external studies underscores the general usefulness of the PI_HD. The external predictive performance was slightly higher than that found in similar validation studies²³.

The poor performance in the REGISTRY study can be explained by a lack of variability in progression level. The REGISTRY cohort was much more progressed at baseline, with many individuals having clinical scores in the ranges typically associated with diagnosis. The lack of progression variability resulted in the PI_HD variables (TMS, SDMT and CAP) having limited predictive power. Like the other studies, REGISTRY had a standard protocol, but it was the only study to enroll individuals less than 18 years of age, it had the largest variety of languages (13), the largest number of sites (see Table 1), and all the sites were European for our analysis.

Though the predictive performance of the PI_HD was poor for the REGISTRY study, this does not necessary threaten the validity of the PI_HD. Consistent with conventional benchmarks of motor and cognitive performance, the PI_HD was successful in correctly classifying most of the REGISTRY individuals as having a high risk of motor diagnosis.

The PI_HD or its normed version (PIN_HD) can be used to predict progression, with higher scores indicating greater risk of motor diagnosis. For clinical purposes, predicted progression provides general information about the current status and course of the disease. The survival curves indicate what happens to an individual's cohort with a common PI_HD. Such information can be useful for treatment strategies and life planning.

For research purposes, PI_HD provides a more accurate index. Progression is commonly indexed by the CAG-Age Product (CAP), which is a type of “burden score” of age and CAG expansion that has several variants^3–5,29. Our results show that when CAP is supplemented with the TMS and SDMT, predictive performance increases. The added variables include one of the core components of the UHDRS, the summation of motor signs (TMS), and the other is one of the three UHDRS cognitive tests (SDMT) that also has a motor component.

Because PIN_HD is easy to compute with commonly measured variables, it is useful for recruiting cohorts of individuals for clinical trials from among those who are registered in on-going observational studies, such as Enroll-HD. The anonymized Enroll-HD database is available to researchers online (https://www.enroll-hd.org/for-researchers/access-data/), and the PIN_HD can be computed using an individual's last wave of data. There is an optional consent for participation in future research studies, which has a very high rate of endorsement (> 90%)². One use of PIN_HD is to identify consenting individuals who have progression levels appropriate for a clinical trial. If a preventative trial requires participants to be many years from motor diagnosis, then a certain bottom percent (e.g., 20%) of individuals with the smallest PIN_HD values might be selected from Enroll-HD. More finely tuned selection is possible if the researcher can identify the desired proportion of converters at the end of the proposed study. Each PIN_HD value (or range) is associated with a cohort that has a specific expected rate of conversion over a given time interval. After deciding upon the non-treated survival curve that is appropriate for the clinical trial, the potential pool of candidates can be identified by the associated PIN_HD value(s). For example, an intervention that targets relatively advanced progression might require the untreated group to have 40% survival (60% conversion) at a study terminus of 3.5 years. Figure 2 indicates that the appropriate survival curve is associated with the range of PIN_HD ≥ 1. Consenting individuals from Enroll-HD whose PIN_HD falls within this range might be candidates for recruitment.

In addition to identifying whom to recruit, PIN_HD is useful for determining the sample size when the endpoint of the clinical trial is conversion itself (time to DCL = 4). In such trials, the goal is to determine if motor diagnosis is delayed by treatment relative to placebo. The expected number of events and the hypothesized treatment effect on the hazard ratio are the key determinants of sample size for survival-based clinical trials³⁰. The survival curve for a PIN_HD can be used to estimate the number of expected conversions in the (untreated) placebo group at study end.

PIN_HD can also be informative for recruitment when the trial is not directly concerned with survival, as when change in TMS is the outcome. Based on the analysis of large observational studies, it is well established that the rate of deterioration for imaging and clinical variables accelerates as they increasingly depart from control-like initial levels^10,15,17,31. The rate of deterioration also increases with CAP³². Because PIN_HD is based on clinical variables and CAP, deterioration of clinically relevant outcome measures is expected to accelerate as PIN_HD increases. Consider the example of change in ICV-corrected putamen volume × 1000 (the scaling is used to avoid small numbers) using the PREDICT-HD database. We found that PIN_HD = -1 was associated with an annual rate of decrease of β = -0.0627 (95% CI = [-0.0714, -0.0539]), and PIN_HD = 1 was associated with the greater rate of β = -0.0881 (95% CI = [-0.0987, -0.0774]).

An ancillary analysis with results presented in the appendix showed that the weights of the PI_HD and its predictive performance were similar when estimated separately at seven annual follow-up visits. Thus, there is evidence that the PI_HD will not change much when computed over a several-year span. The implication for recruitment is that if individuals have repeated observations, then only the last wave of data might be sufficient for predicting progression level.

The emphasis of this study was on developing a prognostic index with variables that are routinely collected in observational studies. Potentially important predictors, such as putamen volume, were intentionally excluded because COHORT and REGISTRY did not have imaging, and neither does Enroll-HD. Recent results using PREDICT-HD⁷ show that imaging variables were predictive of motor diagnosis when considered in isolation. However, when important clinical variables (e.g., TMS, SDMT) and genetic variables (i.e., CAP) were already in the model, the boost in predictive power was modest when imaging variables were added. Therefore, the exclusion of imaging variables does not substantially lower the predictive power of the PI_HD. A potentially more powerful prognostic index could be developed using a combination of newly developed or newly discovered imaging, clinical, genetic, or wet biomarker variables.

A caveat regarding the analysis is that only individuals without a motor diagnosis at baseline were analyzed. This resulted in the exclusion of diagnosed participants. There is a potential for selection bias if the included and excluded individuals were representative of different populations.

The databases used in this study may not represent the more general clinic population. PREDICT-HD and TRACK included participants who had pre-manifest genetic testing. Individuals who have such testing may be less than 10% of the HD population in North America³¹ (though the percentage is probably higher in Europe and elsewhere).

Another qualification is the overlap of participants for COHORT and PREDICT-HD. The data from these studies was concurrently collected and they had many sites in common. Because of patient confidentiality and the anonymized data, it is not possible to identify unique participants. Nonetheless, we did use birth year, CAG length, and gender to examine commonalities among the studies. Approximately a third of the COHORT subjects could not be mapped to the PREDICT-HD subjects (overlap is probably 75% at most).

We caution that the PIN_HD computed for one person does not necessarily predict progression for that individual. The survival curves of the models and the figures indicate the risk of motor diagnosis for a cohort with a common PIN_HD. Due to individual variability, prediction for a single person is difficult, and the survival curve for a person's cohort may have little bearing on their actual risk for motor diagnosis³².

Finally, our findings demonstrate that a relatively simple risk score computed on readily available variables measured at a single visit has reasonable generalizability and usefulness for predicting progression towards HD motor diagnosis, and thus, may be valuable for targeted recruitment for clinical trials. Further analysis will be needed to assess if usefulness is improved adding newly developed variables.

Supplementary Material

Supp Appendix2

Figure A1. Harrell's C and Cox regression weights for PREDICT-HD follow-up visits.

Figure A2. Estimated baseline hazards based on the pooled data.

Table A1. Extended model selection results. Harrell's C, C with inverse censoring weights, integrated Brier score, mean standardized rank, and p-value of aggregated rank. Trimmed means among sites are reported (20% trimming).