Functional Rating Scale 2.0 (FuRST 2.0): A patient-reported outcome measure of function for Huntington’s disease

Abstract

Background

Current functional rating scales are not sensitive to the earliest functional changes in Huntington's disease (HD).

Objective

The Functional Rating Scale 2.0 (FuRST 2.0) is a patient-reported outcome (PRO) measure designed to be sensitive to the initial functional changes in HD, specifically stage 2 and mild stage 3, as defined by the Huntington's Disease Integrated Staging System (HD-ISS).

Methods

We followed standard assessment development methodology to create a PRO. Study 1 consisted of a Delphi panel which analyzed data from focus groups comprised of people with HD and companions from 6 countries. This was followed by four rounds of cognitive interviews through which we evaluated respondents’ comprehension of the instructions, understanding of question and response options, and comfort with the material. Informal advice from a regulatory agency was garnered throughout the process.

Results

Concerns from the target population and regulators regarding instructions, questions, response options, and comfort with the material were addressed through modifications to the scale's wording and format.

Conclusions

We developed the FuRST 2.0 official working document (OWD), the penultimate version of the scale, using focus groups, a Delphi panel, iterative rounds of cognitive interviewing, and informal regulatory advice. Its psychometric properties are being evaluated in the FOCUS-HD validation studies from which the final version of the scale will be derived. The FuRST 2.0 OWD is available for use.

Introduction

Huntington's disease (HD), a progressive neurological disorder caused by expansion of the polymorphic CAG repeat in the huntingtin gene, ¹ affects every domain of life, eventually resulting in death. Functional ability is the capacity to perform everyday activities. ² These activities include an individual's ability to meet basic needs, maintain the ability to fulfill roles in family and society, perform in an occupational capacity, manage financial affairs, and maintain overall health and well-being. ³ In people with HD (PwHD), decline in functional ability begins subtly in the period prior to clinical motor diagnosis (CMD) and progresses throughout the disease. As such, functional ability is a meaningful patient-oriented health outcome, ⁴ and when combined with other endpoints in clinical trials, it can provide data that connects neuropathological change with impact on everyday life. Currently, the Total Functional Capacity (TFC) scale—a clinician-reported outcome (ClinRO) measure that is part of the Unified Huntington's Disease Rating Scale (UHDRS)—is the most frequently used assessment of function in HD research. ⁵ Because this scale was developed for PwHD after CMD, the scale shows a ceiling effect in PwHD prior to CMD.^6–8

Recognizing the limitations of the UHDRS TFC in PwHD prior to CMD, the Functional Rating Scale Taskforce for Pre-Huntington's disease (FuRST pHD)—comprised of a multidisciplinary team of clinical experts, drug developers, and psychometricians who worked in close association with advocacy groups and PwHD—was convened to develop a functional rating scale to assess changes in symptom severity in PwHD, both prior to and after CMD. This taskforce developed the FuRST 1.0, a ClinRO assessment for HD ⁹ with GRID-type ¹⁰ scaling in which both severity and frequency were considered when scoring each item. A separate independent advisory board reviewed this scale and recommended modifications to its structure, scoring, and items. Because the reporting requirement for functional ability is internally determined, the advisory board recommended a patient-reported outcome (PRO) measure in line with regulatory guidance. ¹¹ Pursuant to this guidance, the Functional Rating Scale 2.0 (FuRST 2.0) was developed to provide a fit-for-purpose, ¹² psychometrically-sound, PRO measure sensitive to the earliest functional decline in HD.

In line with recommended best practice,^12,13 we established a data-driven, iterative, and collaborative process for scale development. Here, we present findings from a series of five studies, including focus group input to a Delphi panel followed by four rounds of cognitive interviewing, culminating in the FuRST 2.0 official working document (OWD), the penultimate version of the scale. This OWD is being tested in a series of large sample field studies, called FOCUS-HD, to assess the psychometric properties of reliability and validity, in which changes are usually limited to item reductions, resulting in the final version of the scale. ¹²

Methods

Three of the five studies, including the focus group and cognitive interviewing rounds 1 and 2, were approved by ethics review boards, with all participants providing informed consent. Cognitive interviewing rounds 3 and 4 received exempt status from Western Institutional Review Board (WIRB Copernicus Group) and assent to these studies were provided by each respondent after reading an information sheet about the study. Focus group data were collected prior to the creation of FuRST 1.0 but were analyzed later during the development of FuRST 2.0. Figure 1 presents each step in the scale development process and the number of participants in each study.

Figure 1.

The scale development process undertaken and number of participants in each study. Qualitative data was collected in the focus groups and analyzed by the Delphi panel to create the initial draft of the FuRST 2.0. That draft underwent cognitive interviewing. The data were analyzed by the steering committee who received regulatory input. Based on all information the draft FuRST 2.0 was updated and went through the next round of cognitive interviewing. Each round followed this same procedure until the official working document was reached. PwHD-CMD: people with Huntington's disease after clinical motor diagnosis, PwHD-pCMD: people with Huntington's disease prior to clinical motor diagnosis, Com: Companion/Caregiver, UK: United Kingdom, US: United States, FuRST 2.0: Functional Rating Scale 2.0, CI: Cognitive Interview, R: Round, NA: North America, SC: Steering Committee, RI: Regulatory Input, HD-COPE: HD Coalition for Patient Engagement, OWD: Official Working Document.

Focus groups and Delphi panel

A total of 103 respondents, ages 19–65 years, participated in one of eighteen different focus groups. A semi-structured interview was used to elicit conversation about feelings, behaviors, and symptoms experienced or beginning to emerge. All meetings were audio-recorded and transcribed.

In the focus groups, respondents were sampled from 6 countries (France, Netherlands, Portugal, Spain, United Kingdom, and United States), with three focus group in each country. Respondents were recruited by clinical staff either in person, by phone, or email. Saturation of concepts was obtained for all subgroups in all languages. In qualitative research, saturation refers to the point at which no new concerns or issues are raised by respondents. ¹⁴ Each focus group consisted of one of 3 types of respondents. ¹

1. PwHD after CMD (n = 27), had a CAG repeat > 39 and had a Diagnosis Confidence Level [DCL] ⁵ of 4 indicating unequivocal motor signs of HD within the past 3 years.

2. PwHD prior to CMD (n = 35), had a CAG repeat > 39 and had DCL < 4.

3. Companions and/or family members (n = 41), had a family member with a positive HD gene test or shared a home with a family member with symptoms of HD.

We evaluated the focus group responses with a Framework analysis ¹⁵ approach using 5 stages (familiarization, thematic framework development, indexing, charting, and interpretation) to identify patterns, themes, and generalizable features of data across the different subgroups and reduce the content to relevant domains.

The results of the framework analysis were presented to a Delphi panel of 12 international experts (Supplemental Table 1) for development of the concept of interest and the designation and ranking of domains. The Lawshe Content Validity Ratio (CVR) ¹⁶ was used to identify the proportion of Delphi panel members rating the domains as essential to measuring functional ability in PwHD. Those domains with a CVR > 0.8 were retained. The Delphi panel used these data, along with the results from the psychometric examination of the FuRST 1.0 and a review of the literature to develop an item bank for possible inclusion in the FuRST 2.0 and a scaling metric. The original GRID-type ¹⁰ scaling requires each item to be evaluated for frequency and severity, a complicated and time-consuming system that assumes equivalence between some pairings of frequency and severity. For example, a “very severe” symptom which occurs “rarely” earns the same score (3) as a “moderate” symptom which occurs “almost all the time”. In addition, some items had a three-point maximum score, while others had a four-point maximum score. Therefore the FuRST 1.0 scoring system was replaced by a 5-point Likert-type scaling of 0 = “none”, 1 = “slight”, 2 = “mild”, 3 = “moderate”, 4 = “severe” in FuRST 2.0. To ensure clarity of items and response options, a primary 7th grade (12–13 years) reading level was confirmed using Flesch-Kincaid Grade Level Readability Statistic. ¹⁷

Cognitive interviewing

Cognitive interviewing is used to elicit respondents’ interpretations of the items in an instrument being tested. The aim of the four rounds of cognitive interviewing was to test both the usability and clarity of the draft FuRST 2.0 questionnaire to enable revisions to enhance the tool before subjecting it to psychometric evaluation. Cognitive interviewing for the FuRST 2.0 involved in-depth interviews with respondents with HD and their companions using a semi-structed interview guide designed to elicit understanding and interpretations of the wording of the instructions, items, and response options. All cognitive interviews were conducted with English speaking participants as the initial draft of the scale was in English. Cognitive interviews in other languages will occur after the final version of the scale is translated. All interviewers were trained in cognitive interviewing techniques by an expert in cognitive interviewing (NL).

Cognitive interviewing rounds 1 and 2

Round 1 was conducted with respondents recruited from four North American sites participating in the Enroll-HD longitudinal study. ¹³ Participant demographic, clinical characteristics, and HD Integrated Staging System (HD-ISS) stage ¹⁸ are presented for each group (Table 1). ¹⁸ Because the HD-ISS was created after recruitment for this study closed, the participants were retrospectively classified according to HD-ISS using clinical data collected at the time of the visit.

Table 1.

Demographic characteristics of the HD participants in Cognitive Interviewing Rounds 1 and 2.

Characteristic	CI Round 1 n = 40	CI Round 2 n = 39
Age	49.65 (14.13)	52.28 (10.35)
Sex
Female	20 (50.0%)	17 (43.6%)
Male	20 (50.0%)	22 (56.4%)
Education
ISCED level 0–3 (≤HS)	10 (25.0%)	8 (20.5%)
ISCED level 4–6 (>HS)	30 (75.0%)	31 (79.5%)
CAG	43 (38, 53)	42 (39, 48)
TMS	17.05 (16.54)	10.67 (9.68)
TFC	11.22 (2.03)	12.05 (1.30)
SDMT	41.30 (16.07)	44.74 (15.55)
DCL
<4	20 (50.0%)	19 (48.7%)
=4	20 (50.0%)	20 (51.3%)
HD-ISS Stage*
0/1	14 (35.0%)	11 (28.2%)
2	3 (7.5%)	9 (23.1%)
3	23 (57.5%)	19 (48.7%)

ISCED: International Standard Classification of Education; HS: high school; CAG: cytosine-adenine-guanine repeat length; TMS: Total Motor Score; TFC: Total Functional Capacity; SDMT: Symbol Digit Modalities Test; DCL: diagnostic confidence level. Mean (SD) for continuous variables; N (%) for categorical variables; median (range) for CAG. *HD-ISS stage assigned retrospectively. No demographic data were collected for CI Rounds 3 and 4 (see text for more details).

In Round 1, the FuRST 2.0 was individually evaluated with a sample of 75 study respondents (n = 35 companions, n = 20 PwHD with DCL < 4 and n = 20 PwHD with DCL = 4 and TFC score ≥ 7). Special emphasis was given to the comprehension of the instructions and scale items, the ease of use of the scale overall, and respondents’ comfort level with the items. Interviewers were encouraged to observe the respondents for signs of difficulty as well as record verbatim responses provided to questions about scale items and wording. The data were analyzed by a qualitative analyst (NL) and the results were reviewed by a steering committee (Supplemental Table 1) who modified the assessment.

Through the Critical Path HD Regulatory Science Consortium, CHDI met with the Food and Drug Administration (FDA) Center for Drug Evaluation and Research (CDER). After reviewing the assessment, the FDA made the following recommendations. Modify the response options to avoid quantification of events, such as “few” or “many” difficulties, and make them more explicitly tied to function, for example “moderate = …more than a few difficulties” was changed to “moderate = difficulty doing this but could still do it well”. The steering committee implemented these changes and made modifications to the assessment based on the results of the first round of cognitive interviewing. The modifications were evaluated in Round 2, which included 50 respondents recruited from the same four North American Enroll-HD sites (n = 11 companions, n = 19 PwHD with DCL < 4 and n = 20 PwHD with DCL = 4 and TFC score ≥ 11). Five respondents participated in Round 1. In this round, special emphasis was placed on comprehension of the new items introduced following Round 1, comfort with the scale, and wording and comfort with the new disclaimer. The disclaimer was added to reassure respondents that not all symptoms covered in the assessment occur in all PwHD as this concern arose during the Round 1 cognitive interviews.

The FDA CDER had several suggestions and questions following Round 2. These included avoiding the use of “average” in the instructions as this implies mathematical mean, eliminating questions with overlapping themes, determining if respondents understood what was meant by difficulty, considering if the 2-week timeframe was appropriate for PwHD in the FuRST 2.0's context of use, and whether the examples provided facilitated the respondent's understanding of what was being asked of them. Based on these recommendations, changes were made to the draft FuRST 2.0 for testing in Round 3.

Cognitive interviewing round 3

Round 3 of cognitive interviewing involved a single interview of a group of PwHD conducted over a video web application with one trained interviewer using a semi-structured interview guide. Because the changes made after Round 2 were minimal, a smaller sample size of five PwHD were considered sufficient and were recruited from the HD Coalition for Patient Engagement (HD-COPE), a patient engagement group of the Huntington's Disease Society of America (HDSA). No demographic data were collected.

The aim of the group interview was to encourage consensus on the wording of the response options. Respondents could choose to have their cameras on or off, but the interviewer remained on camera. The session was audio-recorded, and a transcript was provided. Questions included asking respondents for their preference between 2 sets of response options, (Option 1) “No: I had no difficulty doing this” and “Moderate: I had difficulty doing this well” and (Option 2) “No: I had no difficulty doing this well” and “Moderate: I could do this but not well”. Additional questions targeted the specific wording of the response options, the respondents’ understanding of differences among the response options of no, mild, moderate, and severe, and the instructions that had been modified in the last round.

Cognitive interviewing round 4

In Round 4, individual interviews were conducted with six HD-COPE members over a video web application with one trained interviewer using a semi-structured interview guide. No demographic data were collected. This round tested the response options derived from prior rounds with additional testing on tense preference (past or present) for the current response options in the fourth round of cognitive interviewing, for example, “I have no difficulty doing this” compared to, “I had no difficulty doing this”. Questions included asking for respondent's preference between 2 sets of response options targeting tense, the time frame of the past two weeks, and the instructions that had been modified in the previous round (as listed in Table 2). The session was audio-recorded, and a transcript was provided.

Table 2.

Findings from the cognitive interviewing rounds and impact on scale.

Round	Findings	Changes made
Round 1	Respondents found some instructions and responses too vague or too wordy The tense used was confusing Items did not differentiate between physical, cognitive, and emotional functions Differences between slight, mild, and moderate were not clear Formatting difficulties	Instructions were simplified and edited to emphasize that the respondent should rate the function regardless of impairment reason (i.e., physical, cognitive, or emotional symptoms) Questions and response options were reworded to improve clarity, and to make the response options explicitly tied to function The number of response options was reduced from 5 (no/slight/mild/moderate/severe) to 4 (no/mild/moderate/severe) Compound questions were split into multiple questions Disclaimer added to address respondent concerns about the likelihood and severity of potential future symptoms
Round 2	Unclear time frame for questions Respondents found some instructions too complex o Unclear if the ‘difficulties’ are physical, motivational, or cognitive o Some questions were missing examples o Use of the word ‘OR’ is confusing o Perceived redundancy across similar questions	Time frame of past two weeks was emphasized Instructions were simplified, and language was revised to focus on the functional ability without specifying the reason for this impairment Examples were revised and/or added to cover function only, not the underlying cause Double-barreled response options were removed (e.g., I could barely do this or I could not do it al all) Items were re-ordered to group together items of a particular theme (e.g., communication so that the items do not seem to overlap and the granularity of differences between such items would be more obvious “item 1: communicating with other people” and “item 11: how well are you talking” were put next to each other as items 1 and 2 and reworded to make the difference between the two items more clear. Two distinct responses were added, “I never did this activity”, and “I used to do this activity but can no longer do it”. The term ‘average’ was removed
Round 3	Four of the five respondents preferred Option 1 pwHD preferred circles to squares for response options Respondents did not like the separation of 4 response options followed by 2 more options Respondents found it difficult to distinguish between “Severe: I could not do this” and “I used to do this activity but can no longer do it”.	Option 1 was selected The number of response options was reduced to 5, by removing “I used to do this activity but can no longer do it” All responses were combined into one set Squares were replaced with circles
Round 4	Regarding tense of the response options, respondents preferred the response, “I have no difficulty doing this” Respondents stated they would select “I never did this activity” if they had not done the function in the last two weeks For the instructions, the respondents understood “usual everyday level of difficulty” and “past two weeks” timeframe but indicated that the instructions were too long.	Instructions were simplified as follows “This questionnaire asks you about how well you did your daily activities over the past two weeks. We are not asking about the cause of the difficulty, only if you had any difficulty doing these activities. For each question, consider the usual everyday level of difficulty you have had over the past two weeks . Consider how well you did these activities by yourself.”

Results

The distribution of the respondent types (companions, PwHD DCL < 4, PwHD DCL =4) did not differ significantly across the 6 focus groups, X² (10, N = 103) = 2.47, p > 0.1, see Supplemental Table 2.

Overall, 22 functional activities were identified by the Delphi panel to be used in the FuRST 2.0 item bank (Table 3). These were refined through the cognitive interviewing process to produce a 24-item PRO.

Table 3.

Functional activities identified by the Delphi panel for use in the FuRST 2.0 item bank.

Communicating

Working in home or job

Managing finances

Handling cash or credit

Getting started doing activities

Planning day-to-day activities

Getting around

Walking

Doing hobbies

Using hands

Talking

Staying clean and neat

Changing clothes or getting dressed

Keeping to daily routine

Getting to appointments on time

Keeping surroundings clean

Exercising

Keeping interest in what is going on

Controlling temper

Having sexual interest or function

Driving a car

Sleeping

Respondent characteristics for those participating in the first two rounds of cognitive interviewing are summarized in Table 1. For the PwHD respondents, the median CAG length was 43 [range 38–53] in Round 1 and 42 [39-48=] in Round 2. The mean age of PwHD respondents was 49 years in Round 1 and 52 in Round 2; the majority of respondents had an education level beyond high school (75% in Round 1 and 80% in Round 2). PwHD respondents had a mean ± SD UHDRS Total Motor Score of 17.05 ± 16.54 in Round 1 and 10.67 ± 9.68 in Round 2 (score range of 0–124 with higher scores indicative of more severe impairment) and a mean ± SD TFC score of 11.22 ± 2.03 in Round 1 and 12.05 ± 1.30 in Round 2 (score range of 0–13 with lower scores indicating more severe impairment). Mean ± SD Symbol Digit Modalities Test (SDMT) scores were 41.30 ± 16.07 in Round 1 and 44.75 ± 15.55 in Round 2 (maximum 110 with more correct pairings indicating better performance). These characteristics are consistent with the respondents in the first two rounds of cognitive interviewing being in HD-ISS stages 0/1–3, suggesting that all HD-ISS stages are represented in this sample. ¹⁹

Refinements made to the FuRST 2.0 during each round of cognitive interviewing are summarized in Table 2. The iterative refinement process following the Delphi panel through cognitive interviewing rounds is exemplified for the financial domain in Figure 2. The final response options following cognitive interviewing Rounds 3 and 4 are shown in Figure 3.

Figure 2.

Iterative development of FuRST 2.0 questions and responses (exemplified by the ‘managing finances’ domain).

Figure 3.

Response options from the FuRST 2.0 following round 3.

Discussion

We present the development process behind the FuRST 2.0, a 24-item PRO designed to assess functional ability in PwHD prior to CMD, and shortly after CMD, specifically Stage 2 and mild Stage 3 as measured by the HD-ISS. ¹⁸ We developed the FuRST 2.0 using a rigorous scale development approach that included focus groups, a Delphi Panel, multiple rounds of cognitive interviewing, steering committee input, and informal regulatory feedback. Following the fourth round of cognitive interviewing, when no new problems were identified, the Official Working Document (OWD) was obtained.

One limitation is the sample was reasonably well educated and the HD-COPE participants (Rounds 3 & 4) are trained for participating in protocol reviews, thus may have a better understanding of clinical research jargon than the PwHD who will ultimately use the scale. However, through our structured interviews all participants were able to provide feedback regarding their understanding of the items, instructions, and disclaimer, including those with lower education attainment levels. Furthermore, the assessment reading level is below the high school level at a primary 7th grade level (12–13 years).

Developing fit-for-purpose ¹² scales involves several steps that can be summarized into item and scale development and qualitative and quantitative scale evaluation. While the literature often focuses on quantitative scale evaluation, rigorous qualitative evaluation of item and scale development are critical to success. The development of the FuRST 2.0 exemplifies the value of repeated interaction with the assessment's target population and regulatory agencies during the development process of a PRO. By conducting several iterative rounds of cognitive interviewing, we ensured that items are interpretable to PwHD in order to minimize misunderstanding and prevent future measurement error. The remediation of poorly worded questions and responses will also act to reduce the cognitive burden on research participants when the scale is used in future clinical trials. Furthermore, having input from regulatory agencies early in scale development allows for the wealth of knowledge and experience with endpoint development for clinical trials across indications to be incorporated in the structure and wording of the PRO.

To date, PRO's have not yet been prominently used in registration trials, anecdotally because many PROs do not meet the methodological standards required by the regulatory agencies.^20,21 However, these agencies, including the FDA, actively encourage research based on the patient voice, and their landmark guidance ²² has led to major advancements in establishing methodological standards for developing and using PRO measures. The inclusion of informal regulatory feedback in the scale development process is intended to ensure the scale meets the required standards for use clinical trials later.

Next steps

The OWD is undergoing psychometric testing in a series of three studies called FOCUS-HD. In study one, FOCUS-Online, people who self-identify as having HD complete the FuRST 2.0 online. In study two, FOCUS-In Person, participants in the Enroll-HD study are invited to complete the FuRST 2.0 at their Enroll-HD site. Finally, in study three, FOCUS-Longitudinal, Enroll-HD participants in HD-ISS Stages 2 and mild Stage 3 will complete the FuRST 2.0 at nine study visits over three years. The FuRST 2.0 OWD is available for use, please contact info@CHDIfoundation.org for more information or to obtain a copy of the scale.