Abstract
Background:
The neuronal ceroid lipofuscinoses (NCLs) are a group of disorders characterized by neurodegeneration and intracellular accumulation of an auto-fluorescent lipopigment. Together, NCLs represent the most common cause of cerebral neurodegenerative disease in children. CLN3 disease, the classic juvenile-onset form (JNCL) due to mutations in CLN3, is characterized by progressive vision loss, epilepsy, dementia, behavioral difficulties, and motor impairment. The Unified Batten Disease Rating Scale (UBDRS) is a disease-specific rating scale that was developed to assess disease severity in 4 domains: physical, behavior, seizures, and functional capability. Validity and reliability of the UBDRS has been established in a large North American cohort of over 130 individuals. The purpose of this study was to determine whether the UBDRS is valid and reliable when tested in an independent sample.
Methods:
Over the course of one week, 13 individuals with genetically confirmed CLN3 disease were evaluated with the UBDRS by 5 examiners at the University Medical Center Hamburg Eppendorf (UKE). One rater (JWM), one of the developers of the UBDRS, served as the reference standard. The other 4 raters were physicians with expertise in various forms of Batten Disease. After a formal training session, 13 individuals (age 16.5 ± 5.6 yrs) were evaluated simultaneous in parallel by the 5 raters. Inter-rater reliability of the Physical subscale was assessed with Intra-class Correlation (ICC) analysis. The relationship between age and severity was assessed and compared to previously published data from the North American cohort.
Findings:
The ICC among the 5 independent raters was 0.92, demonstrating excellent inter-rater reliability. The individual correlations of each UKE rater compared to the reference standard rater were all >0.95. The average UBDRS Physical Subscale score in this sample was 28 ± 21 (mean ± SD) with a range from 1 to 61. When evaluated as a function of participant age, the slope was 3.06 points/year (R2 = 0.66).
Interpretation:
We have shown excellent interrater reliability for the UBDRS as a clinical rating scale for CLN3 disease in a sample independent from previous work. The results of this study are comparable to those published by Kwon et al., 2011 in a North American cohort showing a slope of 2.86 points per year with a 95% CI of 2.27–3.45 (N = 82). Our results demonstrate excellence inter-rater reliability after training a new group of raters and provide additional evidence for construct validity of the UBDRS. The UBDRS is a valid and reliable rating scale that can used by trained raters to assess the severity and rate of progression of CLN3 disease.
Keywords: Neuronal ceroid lipofuscinosis type 3, Juvenile neuronal ceroid lipofuscinosis, Unified Batten Disease Rating Scale (UBDRS), Natural history
1. Introduction
The neuronal ceroid lipofuscinoses (NCLs) are a heterogenous group of lysosomal storage diseases characterized by neurodegenerative brain disease and accumulation of auto-fluorescent lipopigment. Individuals with CLN3 disease, due to mutations in the CLN3 gene (also known as Juvenile NCL, or Batten disease) typically suffer from rapid-onset of vision impairment as first symptom around the ages of 4–7 years, followed by cognitive impairment (progressive dementia), behavior problems, epilepsy, and a movement disorder. In the later course of the disease (around 10–20 years of age) a loss of mobility, speech, vision and feeding is common with progression to death typically by late adolescent [1-4]. Many affected individuals also develop cardiac problems in the later stages of CLN3 disease [11-13]. The exact function of the CLN3 encoded protein is still unknown. It is localized in the lysosomal membrane and it is proposed in modeling studies to have a role in substrate trafficking [5]. Several disease-causing mutations have been described, but the great majority of affects individuals have a 966 bp (~1-kb) deletion in exons 7 and 8) [6].
The Unified Batten Disease Rating Scale (UBDRS) is a global disease-specific rating instrument that was developed to quantify motor, behavioral, seizure, and functional capability in CLN3 disease in order to assess the severity of the disease. The UBDRS was developed at the University of Rochester Batten Center (URBC) by a group of NCL experts based, in part, on the successful use of the Unified Parkinson Disease Rating Scale and the Unified Huntington Disease rating scale considering the multimodal scales for evaluation of Parkinson disease and Huntington disease [7]. Scoring systems like this are necessary to quantify disease progression in complex disorders that evolved slowly over time [9]. Reliability and validity to measure clinical progression in patients with confirmed CLN3 disease has been established in two different studies and in a large North American cohort that now includes over 130 affected individuals [7,8,14]. The physical subscale of the UBDRS has been shown to change linearly with age in CLN3 disease, and has good cross-validity with functional impairment [8]. Recently the first gene therapy trial for CLN3 disease started in 2018 in Columbus, Ohio, USA, they use the UBDRS as one of their in- and exclusion criteria. Additionally, the primary outcome measures of efficacy in this study are declared as change in rating of the physical assessment of the UBDRS. Consequently, this scale is already used as an evaluation tool for clinical trials [10].
As previous studies involved assessment by experts at the single center that created the UBDRS (UBRC), we performed this study to determine whether the UBDRS is valid and reliable when tested in an independent sample with newly trained raters and patients from Europe. This multi-site validation is important as this scale will be used for future clinical trials.
2. Methods
This study was initiated to determine whether the UBDRS is valid and reliable when tested in an independent sample. It was an observational study based on pediatric and neurologic exams performed during of standard of care routine consultation visits at the University Medical Center Hamburg-Eppendorf, NCL specialty clinic at the Department of Pediatrics. These data were collected on patients who had a genetically confirmed diagnosis of CLN3 disease.
The study protocol of this observational study was approved by the local ethical committee of the Ärztekammer Hamburg (PV7215) and informed consent was obtained for all patients prior to enrollment. The principles of the Declaration of Helsinki were followed.
To determine whether the UBDRS is valid and reliable when tested in an independent sample five examiners evaluated 13 patients with genetically confirmed CLN3 disease. All 13 individuals were invited previously for the voluntary evaluation in the NCL clinic at University Medical Center Hamburg-Eppendorf (UKE) in Hamburg, Germany. All parents gave the permission for this assessment. Assessments were performed in March 2017.
The 2007 version of the UBDRS was used, which is essentially unchanged in current use. It contains four domains: physical, behavior, seizure, and functional capability. In the physical subscale there are 28 different items which are each rated on a scale 0 (normal) to 4 (severe impaired). The total range of the physical subscale includes from 0 to 112 points. The higher the points the more severe the status of the disease. The behavior, seizure and functional capability assessments are based on interviews with the parents or primary caregivers. The 9 behavioral items are rated by severity (0 = normal, 3 = severe) and frequency (0 = never, 3 = almost always). In the 12 seizure assessment questions to form, duration, frequency, complications and medication changes are asked. The patients’ capability items include 5 items: school, chores, play, activities of daily living and overall care levels, additionally these items are asked with the hypothesis how this would be without the influence of blindness [7].
One rater (JWM) came from the URBC as one of the developers of the UBDRS to serve as the reference standard [7]. The other four raters were physicians from the University Medical Center Hamburg-Eppendorf in Hamburg, Germany with expertise in various forms of NCL including CLN3 disease. The 13 individuals were evaluated using the UBDRS over the course of one week. First all four raters received a formal training session by the trainer JWM. Training was done in English. Three participants were evaluated by the trainer with the independent raters watching and independently scoring. Afterwards ten individuals were evaluated by one of the independent evaluators and the scoring was done by all five raters simultaneous in parallel. These subsequent evaluations were performed in German, English, or a combination depending on the primary language of the participant and parents. The results were compared using an intraclass correlation (ICC) analysis of the physical subscale to measure the inter-rater reliability [7]. Our results (the relationship between age and severity) were compared with previously published results from the North American cohort [8].
3. Results
Thirteen individuals (6 males, 7 females) were evaluated. All 13 patients had genetically confirmed mutations in the CLN3 gene and were diagnosed with classic juvenile CLN3 disease. 8 patients carried the classic homozygous 1 kb deletion (c.461-280_677 + 382del) in the CLN3 gene, 5 patients showed compound heterozygous mutations: 4/5 were compound heterozygous for classic 1 kb deletion and another mutation in the CLN3 gene and one patient was compound heterozygous for two other mutations in the CLN3 gene. Detailed patient genotypes are listed in Table 1.
Table 1.
Characteristics of the patients with CLN3 evaluated by UBDRS.
| Subject # |
Gender | Genotype | Age at Examination y = years m = months |
|---|---|---|---|
| 1 | Male | homozygous c.461-280_677 + 382del | 10y 3 m |
| 2 | Male | homozygous c.461-280_677 + 382del | 22y 2 m |
| 3 | Male | homozygous c.461-280_677 + 382del | 20y 3 m |
| 4 | Female | compound heterozygous c.461-280_677 + 382del / c.424delG | 8y 7 m |
| 5 | Male | homozygous c.461-280_677 + 382del | 22y 2 m |
| 6 | Female | compound heterozygous c.105G>A / c.222+5G>C | 18y 4 m |
| 7 | Female | homozygous c.461-280_677 + 382del | 23y 2 m |
| 8 | Male | homozygous c.461-280_677 + 382del | 17y 5 m |
| 9 | Female | homozygous c.461-280_677 + 382del | 12y 7 m |
| 10 | Male | homozygous c.461-280_677 + 382del | 22y 11 m |
| 11 | Female | compound heterozygous c.461-280_677 + 382del / c.1054C>T | 10y 5 m |
| 12 | Female | compound heterozygous c.461-280_677 + 382del / c.1054C>T | 16y 5 m |
| 13 | Female | compound heterozygous c.461-280_677 + 382del / deletion in exon 10-13 | 9y 6 m |
The age at time of evaluation ranged from 9.5 to 23.2 years (Mean 16.5 years, SD 5.6 years). The cohort showed a severity on the physical subscale ranging from 1 to 61 points (possible range 0–112 points) (Mean 28, SD 21) (Table 2). Severity on the physical subscale correlated with participant age with slope of 3.06 point/year (R2 = 0.66) (Fig. 1). These data were then compared with previously reported results (Kwon et al., 2011).
Table 2.
UBDRS Physical Subscale Scores by subject and rater of the Hamburg cohort.
| Subject# |
Evaluator |
||||
|---|---|---|---|---|---|
| Trainer | A | B | C | D | |
| 1 | 6 | 8 | 5 | 5 | 8 |
| 2 | 45 | 48 | 46 | 47 | 42 |
| 3 | 53 | 54 | 54 | 53 | 55 |
| 4 | 1 | 1 | 1 | 1 | 1 |
| 5 | 53 | 52 | 54 | 53 | 56 |
| 6 | 31 | 33 | 34 | 32 | 30 |
| 7 | 61 | 61 | 61 | 61 | 58 |
| 8 | 42 | 44 | 44 | 44 | 38 |
| 9 | 20 | 19 | 19 | 17 | 21 |
| 10 | 13 | 13 | 11 | 9 | 16 |
| 11 | 5 | 6 | 3 | 3 | 8 |
| 12 | 26 | 28 | 26 | 26 | 30 |
| 13 | 8 | 8 | 8 | 8 | 10 |
Fig. 1.
Graph of UBDRS Physical Subscale Scores of the Hamburg cohort. Individual data points shown with dashed line represented linear fit to the data with slope of 3.06 points/yr. The solid line represents the slope (2.86 points/yr) from Kwon et al., 2011 [8], and the dotted lines represent the 95% confidence intervals the Kwon et al., 2011 study.
The intraclass correlation coefficient (ICC) for all five raters was 0.92, with individual agreements between each rater and the trainer >0.98 (Table 3).
Table 3.
Agreement between each rater and the trainer.
| A | B | C | D |
|---|---|---|---|
| 0,99 | 0,99 | 0,99 | 0,98 |
4. Discussion
The UBDRS was developed as a global disease assessment tool for all forms of NCL at the URBC, a center in the USA with extensive experience with NCL [7]. Although it is now an established evaluation procedure and commonly used for evaluation of individuals with CLN3 disease in North America, published data on validity and reliability of the UBDRS have been generated at a single expert center, the URBC [8]. Validation studies assessing data generated at multiple centers are still missing. With the expanding therapeutic pipeline in the NCLs [15], there is increasing need for valid, reliable, and utile evaluation tools that can be used at multiple centers to provide comparable results. Therefore, our study was initiated to determine whether the UBDRS is valid and reliable when tested in an independent sample by newly trained raters. Because CLN3 disease is rare, the need for multi-site and international collaborations is essential to maximize the impact of clinical research on this and other NCLs.
The NCL program at UKE includes the largest cohort of CLN3 patients in Germany and includes patients from all over Europe. To foster the ability to collaborate, an in-person collaboration in Hamburg was initiated in which one of us (JWM) from the URBC spent time in Hamburg to train the other authors of this manuscript in the use of the UBDRS and to subsequently assess interrater reliability and construct validity.
The results of the UBDRS physical scale score from this assessment compared well to previously reported data from the URBC on a sample of 82 affected individuals [8].
The slope of the physical subscale score in the present study (3.06 points per year) is within the 95% confidence interval of the slope (2.27–3.45) reported by Kwon et al. (2011) (Fig. 1). Thus, the relationship between UBDRS physical subscale score and age is comparable between the URBC and UKE samples, providing validation of the UBDRS in an independent sample.
There is one outlier in Fig. 1, this is subject #10, interestingly this patient is homozygous for the classic 1 kb deletion. This underlies once again that in patients with homozygous 1 kb deletion the clinical course can be variable as previously published [16].
The interrater reliability across the 5 raters and between each rater and the trainer was excellent. The ICC of all five raters with 0.92 was excellent compared to Marshall et al., 2005 who found an ICC of 0.83 for the physical subscale in an untrained, but experienced, group of three rates. The individual interrater reliability in the present study (>0.98) was outstanding.
The primary limitation of this study was the small sample size, which was due to the rarity of CLN3 disease and the practical limitations of simultaneous in-person evaluations of multiple affected individuals. Nonetheless, in this sample of 13 participants that included one clear outlier similar to the outlier reported by Kwon et al. (2011), the data clearly support our conclusions about validity and reliability. A second limitation is the potential confound of UBDRS administration in German for most participants and English for some. However, all scoring was done on forms in English, all of the UKE authors are fluent in English, and the USA author understands spoken German moderately well. Consistent administration in a single language might be expected to improve interrater reliability, however the results of this assessment leave little room for improvement.
Based on the confirmation of validity in this independent sample and the excellent interrater reliability, even among raters and participants with different primary languages, shows that the UBDRS can be used by trained raters rigorously to assess the severity and rate of progression of CLN3 disease. Based on these results, the UBDRS is now employed at UKE for assessment and to collect data that can be used for international collaborative studies.
Acknowledgements
We thank the patients and their families for contributing important data supporting this study.
We thank Our Promise to Nicholas., the Batten Disease Support and Research Association, the German Federal Ministry of Education and Research (Grant NCL2Treat to AS, Grant No. 01GM1516A), the European Union’s Horizon 2020 research and innovation program (under grant agreement No. 66691 (BATCure), “Freundeskreis UKE für Kinder mit Demenz e.V.” and “Ein Herz für Kinder – Bild hilft e.V.” for funding this work.
Abbreviations:
- NCL
Neuronal ceroid lipofuscinosis
- UBDRS
Unified Batten Disease Rating Scale
- ICC
Intra-class Correlation
References
- [1].Williams RE, Mole SE, New nomenclature and classification scheme for the neuronal ceroidlipofuscinoses, Neurology 79 (2012) 183–191. [DOI] [PubMed] [Google Scholar]
- [2].Mole SE, Williams R, Goebel HH, The Neuronal Ceroid Lipofuscinoses (Batten Disease), second ed., Oxford University Press, Oxford, 2011, pp. 24–34. [Google Scholar]
- [3].Ostergaard JR, Juvenile neuronal ceroid lipofuscinosis (Batten disease): current insights, Degener. Neurol. Neuromuscul. Dis 6 (2016) 73–83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [4].Schulz A, Kohlschütter A, Mink JW, Simonati A, Williams R, NCL diseases-clinical perspectives, Biochem. Biophys Acta 1832 (2013) 1801–1806. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [5].Phillips SN, Benedict JW, Weimer JM, Pearce DA, CLN3, the protein associated with batten disease: structure, function and localization, J. Neurosci. Res 79 (2015) 573–583. [DOI] [PubMed] [Google Scholar]
- [6].Kousi M, Lehesjoki AE, Mole SE, Update of the mutation spectrum and clinical correlations of over 360 mutations in eight genes that underlie the neuronal ceroid lipofuscinoses, Hum. Mutat 33 (2012) 42–63. [DOI] [PubMed] [Google Scholar]
- [7].Marshall FJ, de Blieck EA, Mink JW, Dure L, Adams H, Messing S, Rothberg PG, Levy E, McDonough T, DeYoung J, Wang M, Ramirez-Montealegre R, Kwon JM, Pearce DA, A clinical rating scale for Batten disease, Neurology 65 (2) (2005) 275–279. [DOI] [PubMed] [Google Scholar]
- [8].Kwon JM, Adams H, Rothberg PG, Augustine EF, Marshall FJ, Deblieck EA, Vierhile A, Beck CA, Newhouse NJ, Cialone J, Levy E, Ramirez-Montealegre D, Dure LS, Rose KR, Mink JW, Quantifying physical decline in juvenile neuronal ceroid lipofuscinosis (Batten disease), Neurology 77 (20) (2011) 1801–1807. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].de Blieck EA, Augustine EF, Marshall FJ, et al. , Methodology of clinical research in rare diseases: development of a research program in juvenile neuronal ceroid lipofuscinosis (JNCL) via creation of a patient registry and collaboration with patient advocates, Contemp. Clin. Trials 35 (2) (2013; Jul) 48–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Gene Therapy for Children with CLN3 Batten Disease. 2018, December 10. Retrieved from https://clinicaltrials.gov/ct2/show/NCT03770572. [Google Scholar]
- [11].Ostergaard JR, Rasmussen TB, Mølgaard M, Cardiac involvement in juvenile neuronal ceroid lipofuscinosis (Batten disease), Neurology 76 (2011) 1245–1251. [DOI] [PubMed] [Google Scholar]
- [12].Ostergaard JR, Paroxysmal sympathetic hyperactivity in Juvenile neuronal ceroid lipofuscinosis (Batten disease), Auton. Neurosci 214 (2018) 15–18. [DOI] [PubMed] [Google Scholar]
- [13].Rietdorf K, Coode EE, Schulz A, Wibbeler E, Bootman MD, Ostergaard JR, Cardiac pathology in neuronal ceroid lipofuscinoses (NCL): more than a mere comorbidity, Biochim. Biophys. Acta (BBA) - Mol. Basis Dis 1866 (9) (2020) 165643. [DOI] [PubMed] [Google Scholar]
- [14].Masten MC, Williams JD, Vermilion J, Adams HR, Vierhile A, Collins A, Marshall FJ, Augustine EF, Mink JW, The CLN3 Disease Staging System: a new tool for clinical research in Batten disease, Neurology 94 (23) (2020) e2436–e2440. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [15].Masten MC, Mink JW, Augustine EF, Batten disease: an expert update on agents in preclinical and clinical trials, Expet Opin. Invest. Drugs (12) (2020) 1317–1322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [16].Lebrun AH, Moll-Khosrawi P, Pohl S, Makrypidi G, Storch S, Kilian D, Streichert T, Otto B, Mole SE, Ullrich K, Cotman S, Kohlschütter A, Braulke T, Schulz A, Analysis of potential biomarkers and modifier genes affecting the clinical course of CLN3 disease, Mol. Med 17 (11–12) (2011) 1253–1261. [DOI] [PMC free article] [PubMed] [Google Scholar]