Caregiver Meaningful Score Differences and Meaningful Score Regions for the Observer-Reported Communication Ability (ORCA) Measure for Individuals with Angelman Syndrome

Abstract

Objective

This study presents a novel method for estimating meaningful score differences (MSDs) and meaningful score regions (MSRs) for the Observer-Reported Communication Ability (ORCA) Measure for individuals with Angelman syndrome (AS).

Methods

The AS Natural History Study collected longitudinal data on an individual’s communication abilities on the ORCA Measure, Vineland Adaptive Behavior Scales-3, and the Bayley Scales of Infant and Toddler Development-4. In a separate Delphi-based study, caregivers provided MSDs and MSRs for the Vineland-3 and Bayley-4 scores. Leveraging the strong correlations between ORCA and these measures, mixed models with random intercepts were used to estimate ORCA MSDs and MSRs based on the previously derived Vineland-3/Bayley-4 MSDs and MSRs.

Results

The ORCA Measure was completed by 192 caregivers, with 113 also completing the Bayley-4 and 159 completing the Vineland-3. ORCA T-score MSRs for low, intermediate, and high communication ability were estimated for expressive, receptive, and pragmatic communication domains using the Vineland-3 and Bayley-4 previously-derived MSDs and MSRs. ORCA MSDs varied depending on the MSR and ranged from 0.38–4.55 (based on Vineland-3) and 1.79–4.62 (based on Bayley-4).

Conclusion

We present a novel anchor-based methodological approach for deriving MSD and MSR thresholds for clinical outcome assessments. While these MSD and MSR estimates for the ORCA measure are preliminary, our methods offer a strategy for establishing meaningful change thresholds in rare disease populations where traditional longitudinal methods may be challenging to implement.

Précis

This study uses anchor-based methods with Vineland-3 and Bayley-4 and a mixed model to estimate meaningful score differences and regions for the ORCA Measure.

Introduction

Angelman syndrome (AS) is a rare neurodevelopmental disorder estimated to affect 1 in 20,000 to 30,000 people.¹ It is associated with significant motor and neurological deficits, including seizures, ataxia, and impairments in communication, cognition, mobility, fine motor skills and sleep.^2–6 Diagnosis usually occurs by age 1–2 years,⁷ though improvements in genetic testing are allowing for earlier detection. Novel genetic therapies enable a potential treatment for AS by activating the silenced paternal UBE3A allele or UBE3A gene replacement therapy.^8–11 While preliminary mouse studies have successfully restored UBE3A expression,^9,12,13 translating these findings to human trials requires measuring biological and phenotypic improvements, including patient-centered outcomes such as the individual’s communication ability.

Caregivers consistently identify communication as a meaningful clinical trial endpoint in AS.¹⁴ However, measuring communication ability is challenging, as most existing tools were developed for verbal individuals and often exhibit floor effects, limiting their ability to detect meaningful variation in this population.¹⁵ Most individuals with AS have little to no verbal speech and follow an atypical trajectory.¹⁶ They primarily communicate through gestures, sign language, or augmentative and alternative communication (AAC) devices, which generate speech from symbols or pictures.¹⁷ AAC devices vary in input (e.g., touchscreens, eye tracking) and output (e.g., synthesized speech, visual displays) to accommodate individual needs.¹⁸ Yet even tools designed for nonverbal populations often fail to account for AAC use. To accurately assess communication ability in AS, measures must be specifically tailored to the abilities and modalities of individuals with AS.

The Observer-Reported Communication Ability (ORCA) measure was specifically created to evaluate and describe communication in individuals with AS. Created in collaboration with caregivers, advocacy groups, and communication experts,¹⁹ the ORCA Measure was designed following best practice guidelines from the U.S. Food and Drug Administration (FDA),^20,21 the International Society for Quality-of-Life Research (ISOQOL),²² and the Professional Society for Health Economics and Outcomes Research (ISPOR).²³ Evidence for the validity of its scores comes from qualitative and quantitative studies which included caregivers of individuals with AS.²⁴ However, to maximize its utility in clinical trials and other research settings, the ORCA Measure scores must be accompanied by meaningful score differences (MSDs) and meaningful score regions (MSRs) to support interpretation of an individual’s communication abilities and how they change over time in response to therapy and other interventions.

MSDs are defined as the smallest level of score change on a measure that holds practical or clinical importance for patients or caregivers.^20,21 Traditional MSD methods rely on longitudinal data and an anchor—an external variable with an established MSD that is assessed alongside the measure of interest in the target population over time.²⁵ MSRs are defined as specific score ranges associated with distinct levels of communication ability.^20,21 Interpreting an individual score (e.g., 50, 51, 52) on the ORCA Measure is complex and unintuitive. The FDA recommends specifying distinct categories (for example, low, medium, and high) to descriptively and easily interpret a level of functioning.^21,26 Determination of categories differs from normative score ranges found in standardized assessments like IQ tests. For condition-specific measures like the ORCA Measure, MSRs must reflect meaningful distinctions within the target population, in this case, individuals with AS. The determination must ensure they represent true differences in communication as perceived by caregivers and other stakeholders, rather than arbitrary cut-points in the score range (e.g., quartiles). The FDA recommends various approaches, including bookmarking, use of Delphi panels, or using illustrative items from the measure itself to help describe what different score ranges represent in terms of actual abilities.²¹ However, to date, there are no established MSDs or MSRs for the ORCA Measure, given its recent development.

A valuable approach to determining MSDs and MSRs in rare diseases such as AS is using caregiver-informed qualitative methods. These methods can capture real-world perspectives on functional and quality of life improvements, ensuring that MSDs and MSRs reflect meaningful score changes or regions, respectively, rather than purely statistical cut-offs.^27,28 Additionally, qualitative approaches are well-suited for rare diseases as they allow for collection of rich data, even with small sample sizes, a critical advantage given that many rare disease trials struggle with participant accrual, leading to high rates of discontinuation and nonpublication.²⁹ By bridging the gap between clinical measures and lived experience, qualitative data provides essential context for interpreting changes.

This study presents a novel anchor-based approach to estimating MSDs and MSRs by using qualitatively derived thresholds for meaningful score changes on established communication measures, the Vineland Adaptive Behavior Scales, Third Edition³⁰ (Vineland-3) and Bayley Scales of Infant and Toddler Development, Fourth Edition³¹ (Bayley-4), to estimate corresponding ORCA Measure thresholds in AS. With the addition of these preliminary AS-specific MSDs and MSRs, interpretation of within-person score changes over time will be enhanced, facilitating the use of ORCA Measure in clinical trials and for regulatory decisions.

Methods

Participants and Study Design

Figure 1 shows the study timeline and methodological workflow, including the integration of MSD and MSR anchors from an separate caregiver Delphi panel with longitudinal Bayley-4, Vineland-3, and ORCA Measure data from the AS Natural History Study (AS-NHS). The AS-NHS is a prospective observational study starting in 2006 that includes ten sites in the United States and Canada. Eligibility required a molecular diagnosis of AS, confirmed through genetic testing. Participants were excluded if the individual with AS had an unrelated comorbid condition that affected neurodevelopment. The Bayley-4³¹ was administered during in-person clinic visits. The ORCA Measure was completed independently by caregivers at least once within three months of the clinic visit. The Vineland-3³⁰ was administered by a trained administrator virtually with the caregivers within three months of the clinic visit. The AS-NHS is registered with ClinicalTrials.gov (NCT04507997) and received Institutional Review Board (IRB) approval at each participating site.

Figure 1.

Study Design and Timeline

Figure 1 shows an overview of the study timeline and design used to derive meaningful score differences (MSDs) and meaningful score regions (MSRs) for the ORCA measure. The natural history study was conducted 2021 through 2025 and included repeated administrations of the ORCA measure and anchor measures (Bayley-4 and Vineland-3). ORCA and Bayley/Vineland assessments could occur in either order, with minimum time between measure administration requirements based on age (for children <2 years old, ≥1 month, for ≥2 years old, ≥6 months). A separate Delphi panel conducted over approximately six months (July-December 2022, shown in gray) provided caregiver-derived MSD and MSR values for Bayley-4 and Vineland-3 communication domains from a different sample. These Delphi-derived values were used to derive MSD and MSR estimates on the ORCA T-score scale using mixed-effects models.

Measures

Observer-Reported Communication Ability Measure

The ORCA Measure assesses overall communication ability, which include three subdomains of communication: expressive, receptive, and pragmatic.²⁴ Expressive communication reflects how individuals convey thoughts, needs, and emotions, including gestures, vocalizations, words, signs, or AAC device use. Receptive communication reflects how well individuals understand language and respond to directions. Pragmatic communication reflects social aspects including greetings, turn-taking, and engaging with others.

Within these domains, the ORCA Measure includes 24 communication concepts. Concept-level responses were converted into scores using the Emerging & Mastery algorithm. The “Emerging & Mastery” scoring algorithm was developed by our study team and is currently recommended for the ORCA Measure. “Mastery” refers to consistent and/or frequent performance of communication behaviors (i.e., “Yes, almost all the time”), while “emerging” reflects occasional performance (i.e., “Sometimes”) over the past 30 days. ORCA T-scores are scaled based on the original calibration sample of individuals with AS and have a mean T-score of 50 and a standard deviation of 10.²⁴ For ORCA T-scores, and every other measure used in this study, higher scores reflect greater communication ability.

Vineland Adaptive Behavior Scales (Third edition)

The Vineland-3 is a caregiver-reported measure that evaluates functional skills needed in daily life and is often used in conjunction with tests of cognitive functioning to diagnose intellectual disability.³² While it has not been psychometrically tested specifically for individuals with AS, it has been used previously to assess their communication ability.^33–35 The interview form of the Vineland was used in the AS-NHS where primary caregivers were interviewed with the Vineland-3 by skilled clinicians. Growth scale value (GSV) scores for the expressive, receptive, and interpersonal subscales were used for this study.³⁶ The expressive domain captures all forms of communication, including gestures, vocalizations, and use of AAC devices. GSV scores, generated through item response theory, offer an interval-scaled measure, allowing for precise and sensitive tracking of developmental progress.^37,38 GSV scores are recommended for use in neurodevelopmental disability and older age (i.e., not infants and toddlers) to help avoid floor effects.^39,40

Bayley Scales of Infant and Toddler Development (Fourth edition)

The Bayley-4 is a norm-referenced assessment designed to evaluate early childhood development and identify developmental delays.⁴¹ For administration, a trained examiner directly evaluates a child’s cognitive, language, and motor skills through structured tasks and observations.⁴² The Bayley has been psychometrically validated in both typically developing children and those with developmental disabilities up to 42 months old,^43–48 and has been used in older children and adults with the AS.^34,49–51 The receptive and expressive communication GSV scores were used for this study. While the Bayley includes some items assessing early gestures, it primarily measures vocalizations, sound, and word production.

These anchor measures (i.e., the Vineland-3 and Bayley-4) were selected as they have been used in AS populations.^{33–35,47,49,52} The Bayley-4 combines performance-based assessment with clinician observation and some caregiver report elements,^26,53,54 while the Vineland-3 interview version involves structured caregiver report with clinician interpretation and scoring.³² Both measures are administered by trained clinicians and draw on multiple sources of information but differ in their primary data-collection approach. Specifically, the Bayley-4 is administered in a controlled setting, focusing mainly on direct assessment of the participant with caregiver report used as necessary. The Vineland-3 mostly relies on caregiver report in a structured interview interpreted by a clinician. Additionally, Bayley-4 administration can be impacted by attention, motivation, and environmental distractions.³⁵ The conceptual alignment between the ORCA Measure and Vineland-3 led us to prioritize the Vineland-derived MSD and MSR estimates for preliminary interpretation of ORCA T-scores in this study.

MSD estimates for the Vineland-3 and Bayley-4

In a separate study, MSDs for the Bayley-4 and Vineland-3 subdomains were determined using a Delphi panel approach with caregivers of individuals with AS.^55,56 A group of 19 caregivers, including 3 fathers and 16 mothers, participated in structured discussions meeting every two weeks from July to December 2022. Caregivers were presented with several scenarios for each domain and rated the smallest change in GSV scores they considered meaningful. Each MSD was linked to a specific baseline GSV score. An expanded range (used as the MSR) was initially defined using midpoints between adjacent baseline scores and was limited to regions where the conditional standard error of measurement did not exceed 0.5 points above that of the baseline score. The resulting MSD and MSR estimates, which serve as anchors for the current analyses, are presented in Table 1. More details on this study are provided elsewhere.⁵⁵

Table 1.

Bayley-4 and Vineland-3 GSV Score Meaningful Score Regions (MSRs) and Meaningful Score Differences (MSDs) in Angelman Syndrome from Delphi Panel⁵⁶

	Bayley-4 GSV		Vineland-3 GSV
	MSR	MSD	MSR	MSD
Expressive
Low	477 – 491	5	61–82	3.8
Intermediate	492 – 505	9	83–105	2.1
High	506 – 545	4	106–167	2.1
Receptive
Low	451 – 501	8	52–66	4.6
Intermediate	502 – 512	5	67–91	2.5
High	513 – 556	4	92–127	1.7
Pragmatic/Interpersonal
Low	---	---	42–71	2.7
Intermediate	---	---	72–92	1.5
High	---	---	93–118	0.4

Abbreviations: GSV: Growth Scale Value; MSD: Meaningful Score Difference; MSRs: Meaningful Score Region

Analysis

A previous study found that children with AS less than two years old had greater change in communication than children older than two years.¹⁹ To minimize confounding from developmental changes, the time between assessments of the ORCA Measure with Vineland-3 and Bayley-4 was restricted to less than or equal to one month for children less than two years and less than or equal to six months for children two years or older. Multiple ORCA Measure assessments were included if they met this criterion.

Because the Vineland-3 and Bayley-4 provide GSV scores only at the subdomain level, we estimated MSDs and MSRs for the ORCA Measure separately by domain—receptive, expressive, and pragmatic communication—rather than as an overall score. Although neither the Vineland-3 nor the Bayley-4 includes a pragmatic communication domain, the Vineland-3 includes the Interpersonal Relationships subdomain within the broader socialization domain, which captures some aspects of pragmatic functioning (e.g., initiating and maintaining social interactions). We used this subdomain as a proxy for pragmatic communication.

To ensure that the Vineland-3 and Bayley-4 could serve as appropriate external anchors for estimating MSDs and MSRs, we evaluated the strength of their associations with ORCA T-scores using the Pearson correlation coefficient between the measurement scores at the first assessment. A commonly accepted threshold for using an external anchor in anchor-based methods is a correlation coefficient of at least 0.30.⁵⁷

Mixed models with random intercepts were used to account for both single and multiple observations per individual, allowing for the capture of within-person variability over time. The estimated coefficients from the mixed models were used to determine the change in the ORCA T-score per unit change in the Vineland-3 and Bayley-4 scores. These unit changes were directly applied to pre-determined MSDs and MSRs for the Vineland-3 and Bayley-4. The ORCA MSDs and MSRs were derived solely through this anchoring approach.

Since MSDs varied across the Vineland-3 and Bayley-4 depending on the MSR, a sensitivity analysis was conducted to assess whether these variations impacted study findings. Specifically, regression models were fitted using piecewise functions based on predefined MSD thresholds, ensuring that slope estimates reflected differences in developmental trajectories across MSRs. All analyses were conducted in RStudio version 4.4.2.

Results

Demographic characteristics are shown in Table 2. Most caregivers were mothers, and most children were white and male, with an average age of 11.7 years. Characteristics were largely similar between the overall sample and those with Bayley-4 or Vineland-3 data, though the Bayley-4 group had a slightly higher proportion with a deletion diagnosis (60.2% vs 53.6%). Participants without Vineland-3 (n=33) or Bayley-4 (n=79) scores were largely similar to those with these assessments in demographic and clinical characteristics.

Table 2.

Demographic Characteristics for Total ORCA Sample and those who have Matched ORCA with Bayley-4 Data, and ORCA with Vineland-3 Data

Variable	Total (n=192)	Bayley-4 (n=113)	Vineland-3 (n=159)
Child’s Age, mean, SD	11.7 (10.0)	12.0 (10.9)	12.7 (10.3)
Child’s Sex, n (%)
Female	87 (45.3)	50 (44.2)	73 (45.9)
Male	105 (54.7)	63 (55.8)	86 (54.1)
Child’s Race, n (%) a
American Indian or Alaska Native	4 (2.1)	0 (0)	4 (2.5)
Asian	17 (8.9)	10 (8.8)	14 (8.8)
Black or African American	16 (8.3)	12 (10.6)	13 (8.2)
White	167 (87.0)	97 (85.8)	138 (86.8)
Other	8 (4.2)	4 (3.5)	7 (4.4)
Hispanic/Latino, n (%)	32 (16.7)	17 (15.0)	25 (15.7)
Child’s Molecular Diagnosis, n (%)
Deletion	103 (53.6)	68 (60.2)	82 (51.6)
UBE3A Mutation	49 (25.5)	25 (22.1)	40 (25.2)
UPD or Imprinting Defect	31 (16.1)	17 (15.0)	30 (18.8)
Abnormal DNA Methylation, FISH or CGH Negative	7 (3.6)	0 (0)	5 (3.1)
Abnormal DNA Methylation only	2 (1.0)	3 (2.7)	2 (1.3)
Caregiver’s Relationship to Child, n (%)
Father	12 (9.2)	8 (10.0)	10 (9.0)
Mother	119 (90.2)	71 (88.8)	100 (90.1)
Other	1 (0.8)	1 (1.2)	1 (0.9)
Number of observations b
1	97 (50.5)	83 (73.5)	101 (63.5)
2	59 (31.7)	24 (21.2)	46 (28.9)
3+	36 (18.8)	6 (5.3)	12 (7.5)

^aParticipants were able to select multiple races.

^bAll participants who completed the ORCA and either the Bayley-4 or Vineland-3 only had up to 3 observations. Those who completed only the ORCA had up to 5 observations.

Abbreviations: CGH: Comparative Genomic Hybridization; FISH: Fluorescence In Situ Hybridization; UBE3A: Ubiquitin Protein Ligase E3A; UPD: Uniparental Disomy

ORCA, Bayley-4, Vineland-3 Measures’ Scores and their Association

Bayley-4 and Vineland-3 GSV scores for expressive, receptive, and interpersonal/social communication abilities at the first assessment are shown in Table 3. For the total sample (n=192), the mean ORCA Overall Communication T-score was 49.4 (SD = 11.4), Expressive T-score was 49.7 (SD = 10.7), Receptive T-score was 48.8 (SD = 11.1), and Pragmatic T-score was 49.9 (SD = 9.5). The average number of days between measure administration was 20.0 days in the ORCA-Bayley group and 21.6 days in the ORCA-Vineland group. Vineland norm-referenced scores showed impairment in this sample (Table 3). A large proportion of participants scored at the floor of norm-referenced V-scaled scores, including 141 (89%) for Expressive Communication, 96 (60%) for Receptive Communication, and 37 (23.3%) for Socialization-Interpersonal. No participants scored at the floor for any GSV or ORCA T-score domain.

Table 3.

Communication Scores and Sample sizes for those with matched Bayley-4 and ORCA data or Vineland-3 and ORCA data at first assessment

Variable	Bayley-4 (n=113)	Vineland-3 (n=159)
Expressive, Mean (SD)
Raw Scores	17.3 (8.2)	21.0 (15.4)
Norm-Referenced Scores	--	1.6 (1.9)
GSV Scores	488.2 (9.8)	93.0 (25.1)
Expressive MSRs, n (%)
Low	70 (65.4)	57 (35.8)
Intermediate	35 (32.7)	45 (28.3)
High	2 (1.9)	49 (30.8)
Receptive
Raw Scores	34.5 (11.9)	33.6 (17.5)
Norm-Referenced Scores	--	3.1 (3.3)
GSV Scores	499.8 (14.3)	87.6 (20.8)
Receptive MSRs, n (%)
Low	70 (62.5)	20 (12.6)
Intermediate	22 (19.6)	66 (41.5)
High	20 (17.9)	66 (41.5)
Interpersonal/Social
Raw Scores	--	33.1 (12.0)
Norm-Referenced Scores	--	5.1 (3.7)
GSV Scores	--	81.5 (13.1)
Interpersonal/Social MSRs, n (%) a
Low	--	52 (32.7)
Intermediate	--	73 (45.9)
High	--	34 (21.4)

^aPragmatic score on the Vineland-3 was represented by Socialization Interpersonal

MSR percentages were calculated out of the total n. Not all participants had scores for all communication domains

Abbreviations: GSV: Growth Scale Value; MSRs: Meaningful Score Region

Based on pre-established thresholds, defined as small (0.10–0.29), moderate (0.30–0.49), strong (0.50–0.69), and very strong (≥0.70),^58,59 the ORCA Expressive T-scores had strong to very strong correlations with the Bayley-4 Expressive GSV scores (r = 0.55) and the Vineland-3 Expressive GSV scores (r = 0.72). Similarly, ORCA Receptive T-scores also correlated strongly to very strongly with the Bayley-4 Receptive GSV scores (r = 0.68) and the Vineland-3 Receptive GSV scores (r = 0.84). ORCA Pragmatic T-scores showed a strong correlation with the Vineland-3 Interpersonal Social GSV scores (r = 0.55). All correlations exceeded the recommended threshold of r ≥ 0.30,⁵⁷ supporting the use of the established Vineland-3 and Bayley-4 MSDs to estimate preliminary MSDs and MSRs for the ORCA Measure.

Regression Results

For the Bayley-4 and ORCA data, the regression model for expressive communication yielded an average intercept of −203.08 (SE = 33.3, p < 0.01) and a slope coefficient of 0.51 (SE = 0.07, p < 0.01). For receptive communication, the average intercept was −176.35 (SE = 21.81, p < 0.01), with a slope coefficient of 0.45 (SE = 0.04, p < 0.01).

For the Vineland-3 and ORCA data, the regression model for expressive communication yielded an average intercept of 24.93 (SE = 1.88, p < 0.01) and a slope coefficient of 0.27 (SE = 0.02, p < 0.01). For receptive communication, the average intercept was 12.94 (SE = 1.78, p < 0.01), with a slope coefficient of 0.41 (SE = 0.02, p < 0.01). For pragmatic communication, the average intercept was 19.29 (SE = 3.61, p < 0.01), with a slope coefficient of 0.38 (SE = 0.04, p < 0.01).

Meaningful Score Differences and Meaningful Score Regions

Figure 2 shows ORCA T-score MSDs across MSRs, derived from Bayley-4 and Vineland-3. Based on Bayley-4 estimates, predicted ORCA Expressive T-scores MSR were 42.0–49.2 (Low), 49.7–56.4 (Intermediate), and 56.9–76.9 (High), with MSDs of 2.57, 4.62, and 2.05, respectively. For ORCA Receptive T-scores, predicted MSR were 25.2–47.5 (Low), 47.9–52.4 (Intermediate), and 52.9–72.1 (High), with MSDs of 3.57, 2.23, and 1.79, respectively (Table 4).

Figure 2.

Preliminary ORCA Meaningful T-Score Differences by Meaningful Score Region from Bayley-4 and Vineland-3 Derived Thresholds in Angelman Syndrome

Figure 2 shows meaningful score regions (MSRs) and meaningful score differences (MSDs) on the ORCA T-score scale in AS derived from Bayley-4 and Vineland-3 Delphi panel results in communication domains: Expressive, Receptive, and Pragmatic (corresponding to the Interpersonal subdomain of the Vineland-3 Socialization scale). Each MSR is shown as a colored horizontal bar with labeled endpoints and associated MSD values. Blue indicates the lowest MSR, purple the intermediate, and red the highest. Lighter extensions of the blue and red bars reflect scores that fall below or above the Vineland-3-derived range but are within the broader ORCA Measure’s T-score range.

Table 4.

Preliminary ORCA Measure Meaningful Score Regions (MSRs) and Meaningful Score Differences (MSDs) for Expressive, Receptive, and Pragmatic Communication in Angelman Syndrome.

	Bayley-4 Derived ORCA T-Scores		Vineland-3 Derived ORCA T-Scores
	MSR	MSD	MSR	MSD
Expressive
Low	42.0 – 49.2	2.5	41.3 – 46.9	3.8
Intermediate	49.7 – 56.4	4.6	47.2 – 52.8	2.1
High	56.9 – 76.9	2.1	53.0 – 69.7	2.1
Receptive
Low	25.2 – 47.5	3.6	34.7 – 40.5	4.6
Intermediate	47.9 – 52.4	2.2	40.9 – 50.2	2.5
High	52.9 – 72.1	1.8	50.7 – 65.1	1.7
Pragmatic/Interpersonal
Low	---	---	35.3 – 46.4	2.7
Intermediate	---	---	46.8 – 54.4	1.5
High	---	---	54.8 – 64.3	0.4

Abbreviations: GSV: Growth Scale Value; MSD: Meaningful Score Difference; MSRs: Meaningful Score Region

Based on Vineland-3 estimates, estimated ORCA Expressive T-score MSRs were 41.3–46.9 (Low), 47.2–52.8 (Intermediate), and 53.0–69.7 (High), with MSDs of 3.75, 2.14, and 2.14, respectively. For ORCA Receptive T-scores, estimated values were 34.7–40.5 (Low), 40.9–50.2 (Intermediate), and 50.7–65.1 (High), with MSDs of 4.55, 2.48, and 1.65, respectively. For ORCA Pragmatic T-scores, estimated values were 35.3–46.4 (Low), 46.8–54.4 (Intermediate), and 54.8–64.3 (High), with MSDs of 2.67, 1.53, and 0.38, respectively (Table 4). These estimates, anchored to the Vineland-3 and Bayley-4, provide initial thresholds for interpreting ORCA T-scores. They should be viewed as preliminary benchmarks rather than definitive gold-standard values for assessing communication change in AS.

Sensitivity analyses using piecewise regression models (Appendix Tables 1–4) showed similar overall trends to the linear models. However, these analyses produced overlapping score ranges when translated to MSRs, and did not substantially improve model fit.

Discussion

We used an anchor-based method to estimate MSDs and MSRs for ORCA T-scores to reflect expressive, receptive, and pragmatic communication ability for individuals impacted by AS. We believe this approach is appropriate given the strong association of the ORCA Measure with Bayley-4 and the Vineland-3 subdomains. While this anchoring approach builds upon the qualitative foundations of the Delphi-derived MSDs, it represents a novel method for translating established thresholds to new instruments without requiring comprehensive qualitative studies for each measure. In our piecewise model, overlapping MSRs introduced ambiguity in score classification (Appendix), so we opted for a simpler linear model to improve interpretability. Derived MSD and MSR estimates for the ORCA Measure followed similar trends across both the Bayley-4 and Vineland-3 measures. However, some variations emerged. For example, in the intermediate expressive region, ORCA T-score MSDs derived from the Bayley-4 were larger than MSDs derived from the Vineland, potentially indicating that larger changes are required to be perceived as meaningful when using the Bayley-4.

Some variation in MSD estimates across anchor measures is expected, as different instruments measure and conceptualize communication ability slightly differently. Rather than reflecting inconsistency, these differences highlight variation in measurement approach. The ORCA Measure and Vineland-3 have much higher correlations than the ORCA Measure and Bayley-4, potentially reflecting their similar administration and focus on non-verbal communication. Thus, the Vineland-derived MSD and MSR estimates may serve as a preliminary reference for ORCA T-scores, reflecting closer conceptual alignment and fewer administration constraints in AS relative to task-based Bayley-4.³⁵

Generally, MSDs were highest in the low communication region, indicating that larger score changes are needed to be perceived as meaningful for children with relatively lower communication skills. Several studies in other populations have also reported smaller MSDs at higher score regions;⁶⁰ however, this pattern contrasts with findings from another Vineland-2 study, which reported smaller MSDs for participants with Autism Spectrum Disorder with lower scores.⁶¹ Thus, our results should be interpreted cautiously and without strong causal inference. One possible explanation, that would need to be confirmed in future qualitative or quantitative studies, is that individuals with more limited abilities may show progress in subtle or inconsistent ways that blend into natural behavioral fluctuations, making small meaningful changes harder to detect. For example, a child may progress from producing a single vowel sound to a consonant-vowel combination, an early change that may lack functional value and be difficult to distinguish from natural variability. In contrast, those at higher communication levels may use clearer detectable forms of communication that are more functional, such as word approximations or clearer gestures, in which changes may be easier to recognize as meaningful progress. However, our explanation of lower MSDs in higher regions remains speculative and should be viewed as a possible explanation rather than a definitive conclusion.

MSD estimates across all regions were generally small in magnitude, especially within high communication regions. Small MSD estimates do not imply that changes are unimportant. For example, a child’s emerging ability to make a sound to gain attention, rather than relying on pulling or hitting, may register as only a small change in T-score, yet can make daily life safer, allow the child to communicate needs more directly, and reduce reliance on physically disruptive behaviors.

An important note is that one of our MSD estimates was less than one point change (Vineland-derived pragmatic domain in the highest MSR). We used the Vineland-3 social-interpersonal subdomain as a proxy for pragmatic communication in the ORCA Measure, recognizing that it captures only a subset of pragmatic skills and may differ conceptually from the broader construct. Nevertheless, it was the most related domain in either measure, and had a strong correlation with the ORCA pragmatic T-scores. Further research is needed to confirm this estimate and ensure that MSD thresholds reflect changes that are both relevant in real-world contexts.

There are several limitations of this study. First, data collection was limited to English-speaking caregivers in the U.S. and Canada. As such, the findings may not be generalizable to other languages or cultural contexts, where caregiver interpretation and reporting of communication may differ. Secondly, although strong correlations between the ORCA Measure and the Bayley-4 and Vineland-3 measures justify their use, they are not perfectly correlated, meaning that MSD and MSR translations should not be taken as definitive. Differences in scaling and measurement could introduce discrepancies in how change is quantified across assessments, especially considering that the Bayley-4 is primarily task-based rather than caregiver-reported. Third, measurement error was modeled only in the predictor variable, not in both the predictor and outcome. This introduces asymmetry, as results may vary depending on which measure is treated as the predictor. While a fully symmetric latent variable model could address this, a mixed-effects approach was prioritized in this study to account for repeated observations and estimate MSDs and MSRs anchored to established benchmarks. This method aligned with our primary goal—interpreting ORCA scores in relation to known communication measures—and allowed us to generate predicted ORCA Measure values across relevant score regions. Future work should explore latent models that incorporate measurement error in both variables.

Additionally, the Delphi panel study was primarily designed to estimate MSDs for the Bayley-4 and Vineland-3, and the expanded baseline ranges we used as MSRs were derived heuristically based on those MSDs, using conditional standard error thresholds to guide range boundaries.⁵⁵ These MSRs were therefore not empirically validated in the original study, and should be considered preliminary. Even with this limitation, we believe our use of mixed-effects models to translate anchor-based MSDs and MSRs from highly correlated communication measures onto the ORCA scale represents a novel and practical approach for enhancing score interpretability in rare disease contexts. A next step is to evaluate the consistency of these MSR categories to determine whether individuals are consistently classified in the same range. Such evidence would strengthen our confidence that the MSRs reflect stable levels of communication ability.

Lastly, although anchoring to the Bayley-4 and Vineland-3 allowed us to estimate MSDs and MSRs using familiar reference points, these tools were not originally designed to capture the full range or subtlety of communication changes in AS. The Bayley-4 and Vineland-3 were chosen because they have been used in the AS population and have strong correlations with the ORCA Measure.³⁵ However, one of the core motivations for developing the ORCA Measure was to provide a caregiver-centered measure specific to AS. This represents a paradox wherein new tools are often validated against the older measures they were designed to improve upon. While the ORCA Measure and Vineland-3 share construct overlap, the ORCA Measure provides enhanced sensitivity to nonverbal communication modalities, including gestures, signs, and AAC use, that are central to AS communication but inadequately captured by measures designed for typically developing children. Thus, anchoring too rigidly to Bayley or Vineland may unintentionally reintroduce the bias we aimed to address. These anchor-derived estimates should therefore provide preliminary statistical support toward developing more precise, AS-centered definitions of meaningful change. Future work should consider complementary approaches that are specific to the language and content of the ORCA Measure.

Despite these limitations, our study has several strengths that lend credibility to these initial estimates of MSDs and MSRs for the ORCA Measure in AS. They are grounded in well-established anchor-based methods that have been widely applied in the development of clinical outcome assessments.⁵⁷ While anchor-based methods are commonly used, a novel contribution of this work is the use of mixed-effects models to translate MSDs and MSRs from two strongly correlated external anchor measures, the Vineland-3 and Bayley-4, onto ORCA T-scores. While future work may continue to refine these thresholds, our methods are scientifically sound and represent the best available guidance for interpreting ORCA scores at this time.

Consistent with best practices in the field, multiple alternative approaches to estimating MSDs and MSRs are recommended, not because one method is inherently superior, but because each offers complementary insight into score interpretation. A future scale judgment study⁶² in AS will further refine the estimates for MSDs. In this approach, caregivers will review structured summaries of hypothetical communication profiles representing distinct ORCA T-score and indicate the smallest change they would consider meaningful. This work can examine the validity of our current anchor-based estimates and allow for comparison between anchoring and alternative methods to estimate MSDs and MSR. Additionally, future work is needed to map concrete examples of communication abilities in AS within low, intermediate, and high MSRs to make these categories more interpretable. The same is true for MSDs, which at this point lack clearly defined communication behaviors that correspond to specific score changes. We are currently working on developing vignettes that illustrate the communication profiles associated with each MSR and can be used to examine the types of observable changes that correspond to MSDs.

In line with FDA guidance,²¹ these MSDs and MSRs are not intended to replace traditional outcome comparisons (e.g., between trial arms), but rather to aid interpretation. In AS clinical trials, investigators may compare ORCA T-scores over time or between groups, with or without baseline adjustment, and use MSD and MSR estimates to contextualize those findings. For example, researchers may use these estimates to describe how many participants showed a meaningful change, or how group-level shifts align with meaningful thresholds.

This study is the first to estimate the MSDs and MSRs for the ORCA Measure in AS, and while these estimates are preliminary, they help to address a key gap for the interpretation of ORCA T-scores. The use of mixed models strengthens the estimation process by accounting for within-subject variability and maximizing available data. This approach can serve as a model for similar estimations in other rare diseases where traditional MSD methods are impractical. The ORCA Measure is currently being adapted for use in 12 additional neurodevelopmental disorders,⁶³ and establishing consistent MSD and MSR thresholds across these conditions will support its use as a caregiver-centered outcome measure for both clinical trials and real-world applications. Continued research is needed to refine and contextualize these estimates based on real-world impact.

Conclusion

This study presents a novel method for determining MSDs and MSRs, and provides the first anchor-based preliminary estimates for ORCA T-scores in AS, supporting its use as a fit-for-purpose outcome tool in clinical trials for AS. Using mixed models and two well-established communication measures (Bayley-4 and Vineland-3), we translated MSDs and MSRs onto the ORCA Measure across expressive, receptive, and pragmatic domains. MSDs were larger at lower ability levels. While minor differences between anchors emerged, estimates were largely consistent, reinforcing their robustness. Future research will use alternative methods to provide additional estimates for MSDs and MSRs in AS and improve interpretability of ORCA T-scores.

Highlights.

• The Observer-Reported Communication Ability (ORCA) Measure was developed to assess communication in individuals with Angelman syndrome (AS), a rare neurodevelopmental disorder characterized by atypical communication and very limited verbal speech. To improve its fit-for-purpose for use in clinical trials, meaningful score differences (MSDs) and meaningful score regions (MSRs) must be established to support score interpretation grounded in real-world experience.

• We present a novel anchor-based approach that leverages the ORCA Measure’s strong to very strong correlations with the Vineland-3 and Bayley-4. Using a mixed model with random intercepts, we translate previously established MSDs and MSRs from a Delphi panel comprised of caregivers onto ORCA T-scores. This approach provides a methodological framework that could be adapted for other rare disease measures.

• Further research is needed to replicate these findings, validate thresholds, and describe communication ability represented within each MSR and observable changes that correspond to MSDs in AS. This includes methods that directly incorporate caregiver feedback into meaningful change thresholds on the ORCA Measure. Given the stronger associations and methodological alignment between the ORCA Measure and Vineland-3, Vineland-derived estimates can serve as a preliminary reference for interpreting ORCA scores.