DEFINING DISABILITY: DEVELOPMENT AND VALIDATION OF A DISABILITY SEVERITY INDEX IN INHERITED NEUROPATHY

Abstract

Objective

To develop and validate a reliable patient-reported scale that grades the severity of disability in inherited neuropathy, from an in-depth analysis of patient and healthcare provider perspectives on what constitutes mild, moderate and severe disability.

Design

In this prospective, cross-sectional study, a 19-item Disability questionnaire was developed following literature and expert review. Between 2011–2012, the Disability Questionnaire was provided to Health Care Providers experienced in inherited neuropathy attending national scientific meetings, and to patients self-registered with the Inherited Neuropathy Consortium – Rare Diseases Clinical Research Consortium on-line contact registry. Provider and patient responses were compared utilizing a 2-sided unpaired t-test with Bonferroni correction. The questionnaire was then assessed for validity, reliability, and unidimensionality.

Results

We analyzed 259 Disability Questionnaires (167 patients, 92 providers); these showed perfect agreement between patient and provider responses on qualitative descriptions of disability, but significant differences in quantitative responses on items corresponding to minimal or severe disability (p < 0.001). Validity and Test-retest reliability of the Questionnaire was excellent (Cronbach’s alpha =0.96; ICC= 0.977 [0.951–0.993]. Exploratory factor analysis and the Mokken Scaling Procedure supported the unidimensionality of the Disability Severity Index.

Conclusion

The Disability Severity Index is a unique instrument, categorizing disability from the patient’s perspective, and will undergo further cross-validation studies in inherited neuropathy. This Index may have applications to other peripheral neuropathies, and would thus benefit from future validation studies in the appropriate cohorts.

INTRODUCTION

There are currently no approved curative therapies for the 31 million people in the United States alone who suffer from chronic peripheral neuropathy.[1] Specifically, there is a critical need for therapies to reduce the disability associated with neuropathy; however, definitions of disability can differ between patients and providers. Inherited neuropathies (Charcot-Marie-Tooth disease or CMT), with their known genetic causes, offer the opportunity to develop targeted disability treatments with potential for broader clinical applications. Our NINDS-funded Inherited Neuropathy Consortium (1U54-NS065712) is leading efforts to assess disability progression in inherited neuropathy through the development of validated outcome measures. Clinician-developed measures ascribing to the WHO ICF framework can quantify the degree of disability depending on participation in life activities[2] and resulting functional limitations;[3–6] however, there is no measure that provides the patient’s direct input on the impact of their disability: their increasing dependence on aids to ambulate, and how they perceive this dependence. The objective of this study was to to develop and validate a practical index to grade the severity of disability in inherited neuropathy, by critically analysing patient and health care provider perspectives on what constitutes mild, moderate and severe disability.

DESIGN

Development of Disability Questionnaire

This was a prospective, cross-sectional instrument development and validation study. A systematic literature review helped identify items pertinent for disability assessment in neuropathy.[2–6] The identified items were reviewed by genetic counselors and neurologists with experience in CMT to develop a standardized 19-item Disability Questionnaire (see eTable 1 in the supplement). Each item pertained to how the responder would classify the disability severity in a generic patient, based on pre-specified functional limitations and utilization of mobility aids. Questionnaire responders were asked to answer each question both qualitatively (choosing from ‘none,’ ‘mild,’ ‘moderate,’ or ‘severe’ disability), and quantitatively (0–10 scale, with 10 being the most disabled). Standardized demographic data (age, gender, race, and ethnicity) and self-reported clinical data (CMT genotype) were also collected.

Participants

The eligibility criterion specified was that the responder either be a patient with CMT, or a health care provider who takes care of patients with CMT. The Disability Questionnaire was distributed by paper to health care professionals (attending the Peripheral Nerve Society annual meeting in Potomac, MD in June, 2011, 4^th International CMT Consortium meeting in Potomac, MD in July, 2011, or the Muscular Dystrophy Association national Clinical Conference in March, 2012; providers were requested to only participate once). The Disability Questionnaire was also sent via an online link to the 450 patients worldwide, who self-registered with the Inherited Neuropathy Consortium (INC) – Rare Diseases Clinical Research Consortium (RDCRC) contact registry: (https://rarediseasesnetwork.epi.usf.edu/registry/index.htm); the online link was open from August 2011 through September 2011. Additionally, 25 participants from the patients that completed the Disability Questionnaire were randomly selected to receive the questionnaire again in an 8 to 12 week period to assess test-retest validity. Both the pen-and-paper questionnaire and the online questionnaire were self-completed by the participant, to minimize mode effect bias.

Standard Protocol Approvals, Registrations, and Patient Consents

The protocol was approved and monitored by the institutional ethics review board at Stanford Hospital and Clinics. All participants received a summary sheet (paper or online) explaining the study; this sheet notified them that completion of the questionnaire constituted informed consent to participate in the study.

Statistical Analysis

Descriptive statistics to characterize the study sample were calculated using Stata-IC 12.1 (College Station, TX) and SPSS (version 20.0). An a priori decision was made to exclude samples with less than 50% data imputed, as per standard scale development analyses. An unpaired t-test with Bonferroni correction for multiple testing was used to compare mean responses between patients and health care providers. The internal consistency of the patient responses to the Disability Questionnaire, a measure of validity, was calculated using the Cronbach’s alpha coefficient. Test-retest reproducibility of the Disability Questionnaire, a measure of reliability, was calculated through intra class correlation coefficients (ICC) using a two-factor mixed effects model and type consistency.[7, 8] The structural validity of the Disability Questionnaire was assessed using Stata-IC 12.1 through exploratory factor analysis using Principal-Component Analysis and oblique rotation for correlated factors,[9] followed by the Mokken Scaling Procedure,[10, 11] a non-parametric item response theory model analogous to Rasch analysis, to test unidimensionality of the final scale.

RESULTS

Two hundred and ninety questionnaires, plus 16 retest patient questionnaires, were collected during the study period. Of the 290 questionnaires, 184 were from online participants and 97 from health care providers at scientific meetings Figure 1 (participant flow chart) shows the breakdown of questionnaires received and ultimately analyzed. 31 questionnaires were ruled ineligible for the following reasons: 12 questionnaires did not meet the eligibility criterion (these 12 responders were neither a patient with CMT, nor a health-care provider taking care of patients with CMT), and 19 questionnaires had less than 50% of items completed in the questionnaire and were excluded from the final analysis, as specified by our a priori methodology. Item 6 also had a large number of missing responses (only 67 responses collected), and was excluded from further analyses. The demographic and clinical characteristics of the 167 patients and 92 health care providers who completed the 19-item disability questionnaire are summarized in Table 1.

Figure 1.

Participant Flow Chart

Table 1.

Demographics

Characteristic	Patient mean (SD) n = 167	Health care provider mean (SD) n = 92
Age	50.6 (14.7)	46.5 (11.7)
Gender	69% F	57% F
Race	96% Caucasian	71% Caucasian
Ethnicity	90% non-Hispanic	76% non-Hispanic

The distribution of CMT types of the patients responding through the INC RDCRC is presented in Table 2. Historical distributions from large academic centers are presented alongside for comparison.[12, 13] The genotype distribution of our patient population was similar to what has been reported in larger academic centers in the US and UK, validating that we reached our intended population.

Table 2.

CMT Genotype Distributions

CMT Type	Online Questionnaire Self-Report	Historical Rates12,13
CMT1A	51.2%	37–40.9%
CMT1B	4.2%	3.1–5.7%
CMT1X	3.6%	10.2–10.8%
CMT2A	12.1%	2.7–4.4%
HNPP	0.6%	0.05–6.1%
CMT4	3.6%	n/a
Unknowns	24.7%	37.4–42%

Patient and health care provider responses to the Disability Questionnaire are provided in Table 3, with summary statistics for quantitative and qualitative responses. A two-sided unpaired t-test with Bonferroni correction for multiple testing was used to determine significant differences in the mean quantitative scores per question between the patient group and the health care providers’ group. In addition, mean scores from data points excluded from the final analysis are provided alongside to allow the reader to draw their own conclusions. On average, there was perfect agreement on qualitative disability classifications between health care providers and patients. Quantitative responses differed significantly between the groups on five items: 3, 4, 5, 7, and 9, which corresponded to either extreme of the disability severity spectrum.

Table 3.

Disability Questionnaire Responses

Disability Questionnaire Item	Mean Patient response scores (obs, SD); and most frequent qualitative response n = 167	Mean Health care provider response scores (obs, SD); and most frequent qualitative response n = 92	Unpaired t-test (Bonferroni correction for significance: p < 0.002)*	Mean scores of excluded data (obs, SD) n = 31
1. Uses a scooter all the time	7 (164,2.9); Severe disability	7.5 (92, 1.8); Severe disability	0.14	5.8 (24, 3.3)
2. Uses a walker some times	5 (166,2.5); Moderate disability	5 (91, 1.7); Moderate disability	0.99	4.7 (21, 2.4)
3. Uses Ankle Foot Orthosis (AFO)/ leg brace, on one leg	4.6 (165,2.5); Mild disability	3.3 (89, 1.5); Mild disability	<.0001*	4.5 (20, 2.2)
4. Uses a shoe insert or similar foot orthosis on one foot	3.1 (167, 2.5); mild disability	1.9 (92, 1.3); mild disability	<.0001*	2.1 (20, 1.3)
5. Bed bound	8.1 (164, 3.6); Severe disability	9.5 (91, 1.4); Severe disability	0.0004*	7.9 (21, 3.9)
7. Uses a shoe insert or similar foot orthosis on both feet	3.4 (164, 2.4); Mild disability	2.3 (91, 1.5); Mild disability	0.0001*	2.6 (20, 1.5)
8. Uses a wheelchair some times	5.5 (164, 2.8); Moderate disability	6.4 (91, 1.6); Moderate disability	0.005	5.9 (22, 2.4)
9. Uses Ankle Foot Orthoses/ leg braces, on both legs	5.5 (166, 2.7); Moderate disability	4.5 (91, 1.7); Moderate disability	0.0015*	5.8 (24, 1.9)
10. Uses one cane	4.2 (164, 2.5); Mild disability	3.6 (91, 1.6); Mild disability	0.04	3.3 (23, 1.8)
11. Uses Ankle Foot Orthoses/leg braces, plus one cane	5.4 (163, 2.8); Moderate disability	5.4 (92, 1.7); Moderate disability	0.99	5.1 (22, 2.2)
12. Uses Ankle Foot Orthoses/leg braces, plus 2 canes	6.3 (165, 3); Moderate disability	6.3 (90, 1.6); Moderate disability	0.99	6.5 (16, 2.2)
13. Uses a 4-point cane	4.7 (164, 2.8); Moderate disability	4.7 (91, 1.9); Moderate disability	0.99	4.1 (18, 2.5)
14. Uses two canes	5.4 (164, 2.9); Moderate disability	5.2 (90, 1.7); Moderate disability	0.55	4.4 (18, 2.6)
15. Uses a walker all the time	5.9 (162, 3); Moderate disability	6.5 (89, 1.6); Moderate disability	0.08	5.9 (19, 3)
16. Uses a walker plus Ankle Foot Orthoses/ leg braces	6.4 (162, 3.1); Moderate disability	6.9 (90, 1.5); Moderate disability	0.15	6.5 (19, 2.9)
17. Uses a wheelchair all the time	7.4 (161, 3.5); Severe disability	8.4 (89, 1.3); Severe disability	0.01	7.3 (19, 3.3)
18. Uses a scooter some times	5.9 (163, 2.9); Moderate disability	6.4 (89, 1.9); Moderate disability	0.145	5.4 (20, 2.6)

The internal consistency of our 19-item Disability Questionnaire, a measure of validity, was excellent (Cronbach’s alpha coefficient of 0.96). The questionnaire also exhibited excellent test-retest reliability (ICC = 0.977) with narrow 95% confidence intervals (0.951–0.993). The 167 patient responses to the 19-item Disability Questionnaire underwent exploratory factor analysis, using Principal-Components factor Analysis. The Kaiser-Meyer-Olkin value[14] was excellent at 0.91, supporting factorability of the correlation matrix. The factor analysis revealed two components with eigenvalues > 1, explaining 75% of the total variance. Oblique rotation to account for correlated factors revealed a simple structure with all but two items (item 4 and item 7) loading substantially on one factor. Next, the Mokken Scaling Procedure assessed the 19-item Disability Questionnaire’s structural validity by calculating Loevinger’s scalability coefficient H_i.[15] The results stratified to a 17-item scale where none of the H_i coefficients fell below the threshold of 0.4. Items 4 and 7 (“Uses a shoe insert in one foot” and “Uses a shoe insert on both feet”) formed the 2^nd scale, with H_i coefficients greater than 0.8 (see eTable 2 in the supplement).

We noted that the items on Scale 2 corresponded to minimal disability. Based on these results, and the patients’ qualitative responses, we re-categorized the 19 items of the Disability Questionnaire into a four-scaled Disability Severity Index (DSI): (1) Minimal disability (items 4 and 7), (2) Mild disability (items 3, 10), (3) Moderate disability (items 2, 8, 9, 11, 12, 13, 14, 15, 16, 17, 19), and (4) Severe disability (items 1, 18, and 5). The internal consistency of the DSI was good (Cronbach’s alpha = 0.72). Assessment of the structural validity of the DSI using the Mokken Scaling Procedure yielded H_i coefficients between 0.4 and 0.8 for items within the minimal, mild, moderate and severe scales (Table 4).

Table 4.

The Disability Severity Index (DSI)

	1 “Minimal Disability”	2 “Mild Disability”	3 “Moderate Disability”	4 “Severe Disability”
Severity Scales	Uses a shoe insert or similar foot orthosis on one or both feet	Uses Ankle Foot Orthosis (AFO)/leg brace, on one leg Uses one cane	Uses a walker sometimes or all the time Uses a wheelchair or scooter sometimes Uses Ankle Foot Orthoses/leg braces on both legs, or Ankle Foot orthoses/leg braces plus cane or walker Uses 2 canes or a 4-point cane	Uses a scooter or wheelchair all the time Bed bound
Hi coefficients (item scalability)	0.83	0.46	Individual 0.61–0.89; overall 0.73	Individual 0.83–0.92; Overall 0.86

DISCUSSION

Recent studies have shown that chronic disability accounts for approximately 50% of the US health burden;[16] this parallels the global shift in disease burden from deaths to disability.[17] In order to develop meaningful interventions to reduce the prevalence and impact of disability, it is necessary to understand how disability is defined by not only the physician, but also the patient. In this study, we characterized the responses from patients with inherited neuropathy to a 19-item Disability Questionnaire to develop a valid and reliable Disability Severity Index (DSI). The DSI is a patient-reported instrument: based on what patients state that they use to ambulate, we can classify them as minimally, mildly, moderately or severely disabled.

The NINDS-funded Inherited Neuropathy Consortium has validated two measures to quantify disability in CMT: The Charcot Marie Tooth Neuropathy Score (CMTNS)[4,5] is a composite score derived from historical, physical examination and nerve conduction data, that can distinguish between mild, moderate, and severe disability in CMT. The CMT Pediatric Examination Score (CMTPedS),[6] is an age-adjusted, Rasch-built functional score, and can assess disability in children with CMT. While both measures allow the clinician to determine the degree of disability in their patients with CMT, neither measure offers the patients’ perspective on their disability status. For example, in the CMTNS, it has been inferred that the wearing of ankle-foot-orthotics (AFOs), the use of additional walking aids such as canes or “sticks,” or the use of wheelchairs, are appropriate markers for ‘mild,’ ‘moderate,’ or ‘severe’ disability, respectively. Similarly, the CMTPedS relies on stringent physician-determined functional assessments for its scoring algorithms. The DSI was developed in order to include the patients’ voices among our clinical outcome measures.

One of the more interesting findings in this study was the concordance and discrepancy between provider and patient responses. We found that on average, patients and health care providers show perfect agreement on qualitative descriptions of disability; however, there is significant disagreement on quantifying the extent of disability, especially on the extreme ends of ability. For example, using a shoe insert on one foot (item 4) or both feet (item 7) were scored at 1.9 and 2.3 respectively by health care providers; patients, however, scored these respectively at 3.1 and 3.4 (p < 0.0001). It emphasizes that utilization of any ambulatory aid, no matter how minimal they may seem to a physician, has a significant impact on the patient, and represents a significant change from the norm, from the patient’s perspective. Conversely, “bedbound” status was scored at 9.5 by providers and 8.1 by patients (p = 0.0004), perhaps indicating the disability paradox:[18] despite seemingly severe physical limitations imposed by the descriptor “bedbound,” patients are unwilling to characterize that numerically as the worst disability. This is important to bear in mind while scoring this instrument in future studies: as clinicians, we may state that a patient would strongly benefit from an orthotic insert, an AFO or cane, but the patient may refuse to use these aids, since they perceive the use as a visible marker of significant disability. To maintain the “patient-reported” aspect of this instrument, we recommend that scoring of the DSI reflect what the patients use, rather than what the clinician thinks they should be using.

Comparing table 4 to the CMTNS,[5] we note that there is excellent concordance between that clinician-determined neuropathy severity scale (scores 1, 2, 3, 4) and the patient-determined DSI (minimal, mild, moderate and severe). One notable exception is that in the CMTNS, using a cane is assigned severity grade 3; based on our results, using a single 1-point cane is classified by most patients (and physicians) as mild disability; using a 4-point cane or two canes changes the perception to moderate disability. We plan to cross-validate the DSI against our validated clinical measures of disease severity in CMT, the CMTNS, and the CMTPedS. If there are consistent differences between the DSI and other outcome measures, we will alter the severity classifications of the other instruments based on the patient’s interpretations from the DSI.

Our study has several limitations. The participant’s self-completion of the paper questionnaire or online questionnaire should minimize mode effect bias, but these may still exist and complicate the duplication of our results. The lack of responses to item 6 (“walks noticeably differently than other people, but does not require any aid to walk”), may indicate that patients internalized the questions despite the instructions, and only responded to questions perceived as pertinent to them. The online contact registry we used to target patients is open to patients world-wide, and the genotype distribution of our patient population was similar to what has been reported in larger academic centers in the USA and in the UK[12,13], thus validating that we reached our intended population, and supporting the generalizability of our findings to all CMT patients. One striking difference was the large number of self-identified “CMT type 2”. Since type 2 presents phenotypically as an axonal neuropathy, there is a possibility that some of those patients screened in the study were axonal neuropathies of another etiology, and not CMT. On the other hand, it also helps demonstrate the universality of the findings to disability from peripheral neuropathy of any etiology.

CONCLUSION

The Disability Severity Index, developed from the inherited neuropathy patient’s perspective, is a reliable and valid patient-reported measure of disability. Results of on-going cross-validation studies will determine the utility of the DSI as an anchor to characterize disease severity in patients with CMT. Longitudinal correlations between DSI and other outcome instruments will enable us to determine whether particular components of some clinical outcome measures correlate better than others with patient-reported data, and whether these should be emphasized in future trials. The DSI may have broad applications to other neuromuscular diseases as well, and would thus benefit from future validation studies in the appropriate cohorts. The Disability Severity Index will allow us to incorporate the neuromuscular patients’ voices into our outcome measures, to fully capture both benefit and harm to patients in future clinical trials.