Study to Assess Content Validity and Interrater and Intrarater Reliability of the Inclusion Body Myositis Functional Rating Scale

Abstract

Background and Objectives

Sporadic inclusion body myositis (IBM) is a rare, muscle-wasting disease that negatively affects health-related quality of life. Although a measure that has been developed to assess the impact of IBM, the IBM Functional Rating Scale (IBMFRS) has limited evidence of content validity or reliability, and what constitutes a meaningful change threshold; this study was conducted to address these gaps.

Methods

Adult patients with a clinical diagnosis of IBM from the United Kingdom and disease area expert health care professionals from the United States and United Kingdom took part in this study. This study consisted of 5 stages including phone interviews (physicians), face-to-face interviews (patients), face-to-face ratings, phone ratings, and ratings of videos using the IBMFRS.

Results

The IBMFRS adequately captures all core functional impacts of IBM, which was corroborated by both patient participants and physicians when debriefing the measure. Physicians and patient participants all thought any change on the measure would be meaningful change for a patient, either improvement or worsening. The quantitative analysis demonstrated good interrater reliability for face-to-face ratings (intraclass correlation coefficient [ICC] >0.7) and for video ratings (ICC >0.9). Intrarater reliability was excellent for face-to-face and video ratings (ICC >0.9). Equivalence between the modes of administration, face-to-face vs phone, was also excellent (ICC >0.9).

Discussion

The IBMFRS is content valid in assessing the key functional impacts of IBM, and any change would be meaningful. It is reliable both within and across raters, and there is equivalence between different modes of administration (face-to-face vs phone).

Sporadic inclusion body myositis (IBM) is the most common idiopathic inflammatory myopathy older than 50 years and more commonly in men than women.¹ It is a debilitating autoimmune and degenerative condition manifesting as progressive muscle weakness, typically initially affecting the finger flexors and/or quadriceps, leading to loss of dexterity and falls. Although there is a suggestion of shortened life expectancy in patients with IBM,² several causes of death have been ascribed to dysphagia or weakness of the respiratory muscle, including malnutrition, cachexia, aspiration, or respiratory failure.³

The prevalence of IBM is unknown in the United States but is conservatively estimated at 5–10 cases per million, while the upper prevalence estimate is 71 per million⁴ and up to 182 per million in people 50 years or older.⁵ A 2017 meta-analysis estimated a worldwide prevalence of between 25 and 46 cases per million.⁶ There is a significant unmet need for treatment options because currently there is no approved treatment for IBM. Therefore, patients typically undergo several treatments with unapproved medications without sustained effect. Despite a clear inflammatory component in IBM, no significant effect has been found with immunosuppressive therapy, such as corticosteroids, IV immunoglobulin, methotrexate, or azathioprine.^7,8 The mainstays of supportive treatment are physical therapy, exercise, nutritional treatment, falls prevention, treatment of dysphagia, and treatment of disease-related psychological symptoms.⁹

The IBMFRS is a disease-specific Clinician Reported Outcome measure that assesses a patient's ability and independence in completing 10 functional activities such as swallowing, handwriting, dressing, hygiene, walking, and climbing stairs¹⁰ and is administered during a patient assessment. Responses are selected using a 5-point scale, ranging from 0 (unable to perform) to 4 (normal). The measure is scored using a total score of all 10 items. The IBMFRS was developed by Jackson et al.¹⁰ and the Muscle Study Group and was derived from the Amyotrophic Lateral Sclerosis (ALS) Functional Rating Scale (FRS),¹¹ a widely accepted and used measure in ALS clinical trials.

Clinical outcome assessments (COAs) positioned as primary or secondary end points within clinical trials require evidence for reliability and validity before product labeling approval.¹² In rare indications, novel methods for the psychometric evaluation of COA measures must be considered because common methods may lack the statistical power to make psychometric inferences, given the small population and increased heterogeneity of the disease experience.¹³ Although there is adequate evidence of the psychometric properties of the IBMFRS,^10,12,14 there is a lack of robust evidence for content validity for its use in IBM, interrater and intrarater reliability, phone vs in-person administration, and for what constitutes a meaningful change. Therefore, the objective of this study was to address the identified gaps in the IBMFRS (i.e., content validity, reliability, and meaningful change threshold).

Methods

Study Design and Participants

This study included adult patient participants with IBM from the United Kingdom (n = 9) and expert health care professionals (HCPs) in IBM from the United States and U United Kingdom K (n = 18). Expert HCPs in IBM were initially identified by 2 of the clinicians involved in this study (P.M.M. and M.M.D.). These HCPs were approached based on their clinical experience with IBM; none of the invited clinicians declined participation in this study. To be included in the panel, HCPs had to be qualified as a clinician, nurse, or physiotherapist with experience and knowledge of the history of IBM, had to have at least 2 years of experience working with IBM patients, and see at least 5–10 patients a year with this condition. Inclusion criteria for patient participants required the participant to sign an informed consent form before any study-related procedures. The specific inclusion criteria were male or female, of any race 45 years or older; participant had a clinical diagnosis of IBM as determined by a specialist doctor in neuromuscular diseases and met the European Neuromuscular Centre IBM research diagnostic criteria 2011 categories for IBM¹⁵; could complete all study requirements; and was sufficiently fluent in speaking English to participate in the interview. Participants were excluded if they had previously been or were currently taking part in an ongoing clinical trial to treat their IBM. HCPs had to be qualified as a physician, nurse, physiotherapist, or health care assistant with experience of the IBMFRS and IBM; have at least 2 years of experience in IBM; and see at least 5–10 patients with IBM per year. Severity of IBM was collected using a physician-reported severity rating scale. The severity of a patient participant's IBM was defined as the perceived degree of impact of the disease on overall function (i.e., the overall level of disability).

There were 5 stages to this study (Figure). The first 2 stages were used to address the content validity of the IBMFRS with physicians and patients. The next 3 stages explored the interrater and intrarater reliability of the IBMFRS. Details of each stage are provided below.

Figure. Schematic of the Study Design.

HCP = health care professional; IBMFRS = Inclusion Body Myositis Functional Rating Scale.

Stages 1 and 2: Interview Study Procedures (Content Validity)

Ten phone interviews were conducted with physicians (stage 1), and 9 interviews were conducted with patient participants face-to-face (stage 2). All interviews were completed by 2 trained qualitative researchers using a semistructured discussion guide (1 for patient participant interviews and 1 for physician interviews). All interviews were recorded and transcribed verbatim. In both interviews, there were brief initial questions designed to establish a rapport and open the conversation. This was followed by a series of focused questions to explore the patient/lived experience of IBM. Patients were asked questions such as “Could you walk me through a TYPICAL day with your IBM? Start at the beginning and tell me what happens when you get up in the morning” and “Do you ever have times when your IBM is worse than usual? Can you walk me through what that is like?” The second part of the interview involved a discussion on the IBMFRS specifically. Participants provided feedback on the behaviors captured, and their importance, as well as whether the response options made sense. Alongside this, the physicians were also asked to comment on their overall understanding of the measure and ease of use. Finally, at the end of the interview, there was a discussion on meaningful change, where patients and physicians separately discussed the smallest amount of change that they would want to see on the scales for them to class this as being meaningful.

Stages 3–5: IBMFRS Rating Study Procedures (Interrater and Intrarater Reliability)

All HCPs received training on how to administer the IBMFRS. For those who were interviewed, the training took place after the interview to mitigate any bias. Face-to-face ratings (stage 3) of patients occurred over a single weekend with 9 patient participants and 14 HCPs. Seven of those physicians who were interviewed also participated in stage 3. Each of the 14 HCPs rated between 3 and 6 patient participants resulting in a total of 49 IBMFRS ratings. Six of the ratings were video recorded for use in stage 5 and involved 2 of the HCPs as raters. The videos focused only on the patient participants and were recorded by an external media company.

All 14 HCPs and the 9 patient participants who took part in stage 3 also took part in the stage 4 phone ratings. This involved each HCP rating at least 2 of the patient participants they had rated in stage 3, approximately 2 weeks later, resulting in 28 ratings. Before the telephone rating, patient participants were asked if they had experienced a significant health event because the face-to-face rating, such as an illness or injury, might affect their rating. If a significant event was reported, their data were excluded from the analysis for this stage of the study.

HCPs who took part in stages 3 and 4 were invited to also take part in stage 5, except the 2 HCPs whose ratings had previously been video recorded and were replaced by 2 new HCPs. One HCP decided not to participate in this stage. This resulted in a total of 13 HCPs. The 6 videos recorded in stage 3 were reviewed by the study team and a disease area expert to select 4 videos that represented a range of participants. From this, 2 files of the videos were developed, with each containing the same 4 videos, but in different orders. Therefore, at time point 1, approximately half of the sample of HCPs saw file 1, while the other HCPs saw file 2. At time point 2, the HCPs then viewed the alternate file. Time point 1 occurred approximately 4 weeks after stage 4 had been completed. Time point 2 then occurred approximately 8 weeks later. HCPs viewed the videos in a secure online portal and were given 14 days to review and submit their ratings.

The videos were also reviewed by an expert in IBMFRS scoring to provide a gold standard rating. This was used to compare against the HCP ratings.

This study was approved by the UK Health Research Authority Research and the London (Surry) Research Ethics Committee (REC) (Approval No. 20/LO/0801).

Stages 1 and 2: Qualitative Analytical Methods

Transcripts of the interviews were entered into NVivo 1.0 or higher, a software package designed to facilitate the storage, coding, and analysis of qualitative data. They were coded using thematic analysis to identify any themes, patterns, or features of interest within the data.¹⁶ Saturation analysis was then undertaken on the concept elicitation data from the patient interviews only by dividing the study sample into 4 equal groups based on the chronological order in which they were interviewed. Saturation was considered met when no new themes or descriptions of concepts were identified in the final round of interviews.

Stages 3–5: Quantitative Analytic Methods

A power analysis to determine sample size requirements for estimating interrater and intrarater reliability was computed based on a repeated measure analysis of variance (ANOVA) between and within subjects' design for the IBMFRS total score.¹⁷ This model assumes 4 observations for each rater for interrater reliability and 2 observations of the same rater over 2 time points for intrarater reliability. Interrater estimates for the between-subjects parameters assume that the magnitude of variability, as measured by the standardized effect size of the difference in IBMFRS scores, is ≤0.20 (low effect) with an inferential probability of ≥0.10. If observed, the variability between the physician populations at each time point will be low and nonsignificant, thus supporting interrater reliability. For estimation of the sample size, intrarater estimates for the within-subjects observations assume a correlation between repeated observations of 0.85 (high correlation). Based on these assumptions, the sample should include at least 14 raters, assessing 4 cases, over the 2 assessment time points to observe 80% power to detect within-group consistency and between-group variability.

Descriptive analyses of the IBMFRS items, n, frequency, and percentage were computed for each of the stages where ratings were made, which was then used to support the evaluation of variability in the interrater and intrarater agreements for the IBMFRS total score.

The total score on the IBMFRS was computed for each patient participant on each rating event (stages 3–5) and for each rater. These scores were then used in fixed effect ANOVA analyses to compute intraclass correlation coefficients (ICCs).¹⁸ The prespecified criterion for acceptable reliability was defined as 0.70 (range 0–1.0 with a higher threshold demonstrating better rater agreement and stability).¹⁹ A Many-Facets item-response analysis was conducted as a sensitivity analysis to identify a potential error in rater-reliability estimates due to rater bias.²⁰ FACETS provides indices called strata, separation, and reliability of separation (which are computed for each facet; patient, stage, time point, or item). The strata index indicates the number of statistically distinct levels of severity. Separation indicates the spread of the rater (or) severity levels relative to the precision of measurement. The Reliability of Separation indicates how well the data from different facets (patients, raters, stage, time point, and items) are separated for severity levels; it reflects the ability of the measure to “separate” patients (or raters) by their severity and can vary between 0 and 1.

All quantitative analyses were conducted using Statistical Analysis System (SAS) 9.4 or higher and FACETS 3.8.1 or higher.¹⁶

Standard Protocol Approvals, Registrations, and Patient Consents

Written informed consent was obtained from each patient participant before enrolment into this study. All study materials were reviewed and approved by the London—Surrey Research Ethics Committee (REC reference 20/LO/0801).

Data Availability

Anonymized data not published within this article will be made available on reasonable request from any qualified investigator, if ethically and legally possible.

Results

Patient Participant Demographics

The mean age of the 9 patient participants was 66.6 years (SD = 6.3), with just over half being female (n = 5). All except one of the patient participants identified as White British, this patient participant identified as mixed race. All reported having at least a college-level education. Only 2 were currently used, one full time and the other part time. All others said that they had retired. Most of the sample (n = 6, 66.7%) reported their IBM severity as moderate, n = 1 (11.1%) reported it as mild, and n = 2 (22.2%) as severe.

The mean time since symptom onset was 10.9 (SD = 6.4) years with symptoms first occurring in the proximal lower limbs for most patient participants (n = 8, 88.9%). The mean time since IBM diagnosis was 4.6 (SD = 5.0) years. All patients were diagnosed by either clinico-pathologically defined IBM (n = 5, 55.6%) or clinically defined IBM (n = 4, 44.4%). Physician-reported patient severity was noted as moderate (n = 4; 44.4%), mild (n = 2, 22.2%), severe (n = 2, 22.2%), and very severe (n = 1, 11.1%).

HCP Demographics

Table 1 presents the demographic data of the physicians (stage 1) and raters (stages 3–5). At stage 5, one of the raters only completed the video ratings at time point 1; therefore, they were excluded from the stage 5 analysis.

Table 1.

HCP Demographic Variables

A total of 17 HCPs took part in this study, 14 were from the United Kingdom and 3 were from the United States. The 3 US raters only took part in stages 1 and 5 of this study. During stage 1, all HCP participants were required to be physicians. For all other stages of this study, raters were a mix of physicians, nurses, and physiotherapists. The 2 most reported areas of specialization were neurology and physiotherapy. There was a range of qualifications reported including medical doctorates, PhDs, undergraduate/postgraduate qualifications, as well as those with nursing and physiotherapy qualifications. The HCPs across the stages had a wide range of years of experience treating patients with IBM, i.e., fewer than 5 to over 15 years. Most raters and physicians, n = 16 (88.9%), selected the 1–250 option for a number of patients seen during their IBM career.

Qualitative Results

Stages 1 (physician) and 2 (patient participants) phone and face-to-face interviews, respectively, revealed that IBM has a substantial impact on patient participants' day-to-day lives, with a particular impact on physical function. The concepts identified can be broadly grouped as physical upper limb, physical lower limb, oral, social, and emotional.

Physical function limitations related to either upper or lower limbs were some of the most frequently identified impacts of IBM. Upper limb difficulties focused on difficulties related to dressing, handwriting, fine motor skills, cutting food and using utensils, and hygiene (cleaning and washing themselves). Patient participants described how when dressing they had to select clothes that were easy to put on and did not involve zip or buttons, “clothes that just go on very comfortably... and very easily, ‘cause zips, buttons become a problem.” Patient handwriting had become more illegible as the condition worsened, “Um, my handwriting is not as good as it used to be... but I can still write.” Although patients found it hard to complete fine motor skills such as opening jars and doors, “it has an effect on the fingers, so you know, grasping things. Turning, opening doors, jar tops,” or use utensils to cut food “I drop things all the time. The knife drops, the fork drops... the spoon drops, the milk gets spilled.”

For lower limbs difficulties moving from sitting to standing, walking, climbing stairs, falling, and turning in bed were key areas identified. Walking was one of the most affected areas with participants ranging from having minor difficulties to using a walking stick or frame or being wheelchair bound “Um, over time, they become confined to a wheelchair with... You know—some of them cannot ambulate on their own and others at all [which has a big impact].” Similarly, as the condition worsened, patients were unable to use stairs, “have not used the, uh, going upstairs since February (9 months)” and would have difficulties moving from sitting to standing and for those who were most severe would need a hoist, “[I need] a sling and a hoist.”

In reviewing the IBMFRS, all physicians said that the items were relevant and that the response options were suitable. Similarly, for patient participants, most felt that the functional behaviors were relevant and important. However, 2 patient participants indicated that swallowing, handwriting, cutting food, and turning in bed were not relevant currently because their IBM was relatively mild.

The IBMFRS captured all core functional impacts identified during the qualitative interviews. Table 2 presents all the identified concepts to the items in the IBMFRS. The core functional concepts captured are swallowing, handwriting, cutting and handling utensils, fine motor skills, dressing, hygiene, turning in bed and adjusting covers, sit to stand, walking, and climbing stairs.

Table 2.

Mapping Identified Concepts to the IBMFRS

A few concepts were not captured in the IBMFRS: loss of hobbies and social life (n = 10), low mood and depression (n = 3), impact on driving (n = 3), impact on work (n = 3), weight loss (n = 3), and falls (n = 15).

As part of the discussion, some patient participants (n = 8) and physicians (n = 7) also described what they would consider a meaningful change on the IBMFRS. Most (n = 5) of the physicians indicated that a 1-category change on any item would represent a meaningful change in functional behaviors for the patient. This was also reflected by all 8 of the patient participants who indicated that a 1-category change on any item would be a meaningful change for them, “Well, they all [change in response options] would be [meaningful], because they all have an impact on your life. I mean, for example, swallowing, the next one down, is occasional choking, which is not pleasant.” Two of the 7 clinicians considered that a 2-category change on any item represented a meaningful change in functional behaviors.

Quantitative Results

The quantitative analysis demonstrated strong reliability within and between raters (Table 3). Interrater reliability from the face-to-face ratings and when against video was also very good, with ICCs of 0.847 and 0.971, respectively.²¹ Intrareliability using video ratings had an excellent ICC of 0.999. Equivalence between the different modes of administration, i.e., face-to-face vs phone, was very high, 0.941.

Table 3.

Interrater and Intrarater Reliability by Stage

Overall, strata, separation, reliability of separation, and χ² p values were all in agreement with the expectations of the many-facets model. Specifically, the average correlation between a single rater and the rest of the raters R_c was 0.85, which in line with high ICCs reported for classical statistically interrater reliability, indicates high interrater agreement. For the stage and rater facets, the nonsignificant p value for the χ² test, low strata, separation, and reliability of separation suggested that there are no observed differences between raters at each stage for their severity. Conversely, for patient facet, the significant p value for χ² test, high strata, separation, and reliability of separation suggested that there are no observed differences between patients' severity levels.

Discussion

COA measures are a valuable tool for assessing the symptom burden and impact of a disease on patients' health-related quality of life.²² However, they require sufficient evidence supporting their reliability and validity before use in clinical trials. Although there is some psychometric evidence of the IBMFRS,^10,12,14 there was a paucity of evidence demonstrating the instrument's content validity, interrater and intrarater reliability, and what constitutes a meaningful change, all of which have been addressed in this study.

The analysis of qualitative data from the patient and physician interviews revealed that IBM has a substantial impact on patient participants' lives, particularly regarding functional impacts. The IBMFRS adequately captures all core functional impacts of IBM, which was corroborated by both patient participants and physicians when debriefing the measure. The concept of “falls” mentioned as important by physicians is not explicitly captured by an item in the IBMFRS; however, physicians considered that this would be included in rating the patient's walking ability. A few other concepts discussed by patient participants are not captured in the IBMFRS (i.e., loss of hobbies and social life, low mood and depression, impact on driving, impact on work, and weight loss). However, these were typically infrequently discussed and were not functional impacts and, therefore, would not be suitable for inclusion in a measure of function. They could be addressed using other scales designed to capture these and other related health domains if required.

When discussing meaningful change, most patient participants and physicians agreed that a 1-category change on any of the IBMFRS items, in any direction, would be meaningful. This highlights not only the debilitating nature of the disease but also the therapeutic potential of a treatment as even small changes would be considered meaningful to patients.

Furthermore, the inter-reliability and intrareliability of the IBMFRS were excellent, showing consistency between raters and stability over time, respectively. There was also excellent correlation between face-to-face and phone administration showing equivalence regarding the mode of administration. In addition, the supportive facet analysis further demonstrated agreement between all raters, as well as showing agreement between raters when compared with the gold standard rater.

Although originally derived from the ALS FRS,¹¹ this study demonstrates the relevance of the IBMFRS and its suitability for reliably assessing function in patients with IBM. However, given that the IBMFRS is a more holistic and global measure of physical function with a strong emphasis on upper and lower limb function, the IBMFRS might not be suitable for studies focusing on dysphagia as a key study end point (of 10 items, only 1 item addresses dysphagia). Should researchers wish to investigate swallowing or respiratory function in greater detail, then other bulbar-specific measures such as the Eating Assessment Tool-10²³ or the modified oculobulbar facial respiratory score²⁴ should be considered.

This study was undertaken using only UK patient participants; however, cultural differences in the functional behaviors endorsed are not anticipated. In addition, using the good practice guidance for the translation and cultural adaptation²⁵ of a measure would mitigate any issues when using the instrument cross-culturally. This study was also undertaken during the COVID-19 pandemic; therefore, only 9 patient participants could be recruited, and these patients took part in all relevant stages of the study (stages 2–4). This was sufficient for the content assessment where 10 interviews were planned, although additional patient participants at different functional status would have been useful to determine reliability at the extremes of the disease. However, for stages 3 and 4 (face-to-face and phone rating), 14 patient participants were originally planned. Despite this, there was still a good range of severities and differences in abilities such that interrater reliability was arguably still robustly tested.

Overall, it is concluded that the IBMFRS is content valid in assessing the key functional impacts of IBM, is reliable both within and across raters, and there is equivalence of scores between face-to-face and phone modes of administration highlighting its feasibility for use within the context of clinical trials, with a 1-category change on any items being classed as meaningful by both patient and physicians.

TAKE-HOME POINTS

• → IBM has a substantial impact on the day-to-day lives of patients and negatively affects health-related quality of life.

• → The IBMFRS is content valid in assessing the key functional impacts of IBM.

• → The IBMFRS is reliable both within and across raters.

• → There is equivalence of scores between face-to-face and phone modes of administration.

• → A 1-category change on any item was classed as meaningful by both patients and physicians.

Appendix. Authors

Study Funding

Clinical Outcomes Solutions is a health research consultancy that was paid to conduct this research. Orphazyme A/S is a pharmaceutical company and funded this study.

Disclosure

P.M. Machado has received consulting/speaker's fees from Abbvie, BMS, Celgene, Eli Lilly, Galapagos, Janssen, MSD, Novartis, Orphazyme A/S, Pfizer, Roche and UCB, and is supported by the National Institute for Health Research (NIHR) University College London Hospitals (UCLH) Biomedical Research Centre (BRC). M.M. Dimachkie serves or recently served as a consultant for for Abcuro, Amazentis, ArgenX, Catalyst, Cello, Covance/Labcorp, CSL-Behring, EcoR1, Janssen, Kezar, MDA, Medlink, Momenta, NuFactor, Octapharma, Priovant, RaPharma/UCB, Roivant Sciences Inc, Sanofi Genzyme, Shire Takeda, Scholar Rock, Spark Therapeutics, Abata/Third Rock, UCB Biopharma and UpToDate. M.M. Dimachkie received research grants or contracts or educational grants from Alexion, Alnylam Pharmaceuticals, Amicus, Biomarin, Bristol-Myers Squibb, Catalyst, Corbus, CSL-Behring, FDA/OOPD, GlaxoSmithKline, Genentech, Grifols, Kezar, Mitsubishi Tanabe Pharma, MDA, NIH, Novartis, Octapharma, Orphazyme, Ra Pharma/UCB, Sanofi Genzyme, Sarepta Therapeutics, Shire Takeda, Spark Therapeutics, The Myositis Association, UCB Biopharma/RaPharma, Viromed/Healixmith & TMA. T. Symonds, J. Randall, L. Lloyd-Price and S. Hudgens are employees of Clinical Outcomes Solutions, a health research consultancy that was paid to conduct this research. C. Guldberg was an employee of Orphazyme A/S, the pharmaceutical company that funded this study. Full disclosure form information provided by the authors is available with the full text of this article at Neurology.org/cp.