Assessing the Content Validity of Patient-Reported Outcome Measures in Adult Myositis: A Report from the OMERACT Myositis Working Group

Abstract

Objective:

To investigate the content validity of several patient-reported outcome measures (PROMs) in patients with idiopathic inflammatory myopathies (IIM).

Methods:

Seven individual PROM instruments were selected by the Outcome Measures in Rheumatology (OMERACT) Myositis Working Group relating to the following domains: pain, fatigue, physical function and physical activity. Twenty patients from the Johns Hopkins Myositis Center were selected for one-on-one face-to-face or phone interviews for cognitive interviewing of individual PROMs to assess comprehension and content validity. Additionally, patients were asked if they thought muscle symptoms, an area originally identified in qualitative studies, were encapsulated by the other four domains.

Results:

The majority of patients (>70%) felt that each of the instruments was clear, easy to read and understand, and could be used for assessment of its domain. Two-thirds (66%) of patients felt that ‘muscle symptoms’ were captured by the other domains.

Conclusions:

We provided evidence to support adequate content validity for several PROMs. Further research is needed to determine whether ‘muscle symptoms’ warrant a separate domain.

Introduction

The idiopathic inflammatory myopathies (IIM) are a heterogeneous group of autoimmune diseases characterized by skeletal muscle involvement leading to muscle weakness^1,2. Other organs, including the skin, joints, and lungs are often affected and contribute to patient morbidity and mortality. Diseases classified as IIM include dermatomyositis (DM), polymyositis (PM), antisynthetase syndrome (ASyS), and immune-mediated necrotizing myopathy (IMNM). Although a common clinical feature among these diseases is muscle weakness that limits activities of daily living (ADL) and health-related quality of life (HRQOL), few data are available concerning patients’ experiences with characterization of symptoms and disease burden^4,5. The aim of the Outcome Measures in Rheumatology (OMERACT) Myositis Working Group is to identify core domains and instruments that reflect the life impact of adult patients living with IIM. This focus on life impact serves to complement the core domain set for assessing disease activity and damage proposed by the International Myositis Assessment & Clinical Studies Group (IMACS). The Myositis Working Group is comprised of an international group of patient research partners (PRP), healthcare providers, researchers, and methodological experts⁶. Previous work by our group includes literature reviews of patient reported outcome measures (PROMs), international patient focus groups, modified Delphi surveys, and in-person workshops at OMERACT biennial meetings^5,6. From this work, the Myositis Working Group identified five patient-prioritized domains that should be considered for measurement in clinical trials and cohort studies: fatigue, pain, level of physical activity, physical function, and muscle symptoms⁶. From prior qualitative work, our working group has described that there is uncertainty as to the degree of overlap of physical activity, physical function, and muscle symptoms. Thus, we also sought to clarify these issues through these interviews. Herein we describe the Myositis Working Group’s selection and initial evaluation of several candidate PROMs for the assessment of symptoms and impact for adult patients with IIM.

Methods

To determine which PROMs should be utilized to measure each of the five domains, the Myositis Working Group surveyed the landscape of available PROMs that pertain to IIM. A systematic literature review was previously conducted to identify all outcome measures (including PROMs) utilized in IIM research⁷. In addition to this review, our Myositis Working Group was divided into two groups to perform literature searches within the Patient Reported Outcome Measurement Information System® (PROMIS®) Data Bank and PubMed for pain/fatigue or physical activity/function. PROMs consistent with prior concept elicitation obtained from focus groups and surveys administered to patients with IIM were prioritized. For example, for pain, the group concluded that scales dealing with pain impact and interference (in contrast to pain intensity) more accurately reflected the patients’ priorities as elicited through qualitative inquiry. A list of candidate PROMs was collated, and on subsequent working group phone calls, each PROM was discussed per OMERACT framework for truth (domain match) and feasibility⁸. The working group thereafter voted on whether to carry the PROM forward for further testing, requiring a 70% majority as described in the OMERACT Filter 2.1 Instrument Selection algorithm⁸. Ultimately, the Myositis Working Group narrowed down the pool of instruments to 1–2 per domain, and then developed a survey to assess the content validity and feasibility of each instrument using a standardized questionnaire. For the domain “muscle symptoms”, our group considered the possibility that this may be encapsulated by the other domains (fatigue, pain, physical activity, and physical function), and incorporated this question into the survey. The instruments identified by the Myositis Working Group include the following: 1) Pain--PROMIS Pain Interference Short Form 8a v1.0; Pain Disability Index (PDI); 2) Fatigue--PROMIS Fatigue Short Form 7a; PROMIS Fatigue Short Form 13a (FACIT-Fatigue); 3) Physical function: PROMIS Physical Function Short Form 8b; and 4) Physical Activity: Myositis Activities Profile (MAP), The International Physical Activity Questionnaire (IPAQ). Although the Fatigue Short Form 13a is the PROMIS version, the questions are the same as those in the FACIT version.

Twenty patients age 18 or older with a physician-confirmed diagnosis of IIM were randomly selected from outpatient clinics from June 2019-August 2019 from the Johns Hopkins Myositis Center. Patients with juvenile myositis or inclusion body myositis (IBM) as well as those unable to speak English or with cognitive impairments were ineligible. Patients were randomly assigned to one of the following two survey blocks: pain/fatigue or physical activity/physical function/muscle symptoms. Patients only completed one block as to reduce the burden of debriefing too many questionnaires. Based on patient preference, either face-to-face interviews at the clinic or phone interviews were conducted. An interviewer not involved in the direct care of the patients conducted all interviews (Standardized Questionnaire is included in Supplemental Materials). For each instrument, after every question, a combination of follow-up questions was asked: “In your own words, what do you think the question is asking?” “How did you choose your answer?” and “What do you think of when answering the question?” Patients assessed the relevance of each question in the instrument to their personal experiences by selecting whether they thought the question should be included, removed or if they were indifferent. They were also asked if any additional questions should be added to assess pain, fatigue, physical activity and physical function. Patients then assessed whether they thought the questions were clear, whether they thought they could easily complete the survey during a clinic visit, and whether the questionnaire could be used to measure the domain of interest^7,8. Interviews lasted for approximately 60 minutes and were audiotaped, transcribed, and de-identified. Relevant clinical disease information was extracted from the patient medical record and the Johns Hopkins Myositis Longitudinal Cohort database. All patients provided informed consent prior to study procedures. This study was administered under the auspices of the Johns Hopkins Hospital Institutional Review Board (IRB Number NA_00066663). Descriptive statistics were reported on patient demographics, clinical features, and PROM response.

Results

A convenience sample of 20 participants were randomly assigned to complete one of the two survey blocks. The average age of patients was 59±12 years, with a disease duration of 7.7 ± 6 years. Seventy percent of patients were female, and the racial breakdown was 75% Caucasian, 20% African American, and 5% Asian. The mean disease activity at time of PROM completion was 4.65 ± 2.15, as assessed by a patient global disease activity (PGA) numerical rating scale, with 0 representing no evidence of disease activity and 10 representing extremely active disease.

Overall, there were 11 respondents for pain instruments, 12 for fatigue instruments, 10 for physical function, 10 for physical activity instruments, and 9 for muscle instruments. One patient assigned to the pain/fatigue block only could complete half of the interview; similarly, one patient assigned to the physical function/activity/muscle symptom block did not answer questions regarding muscle symptoms. Data on feasibility for all 7 PROMs can be found in Table 1. Data on individual item relevance for all PROMs can be found in the Supplemental Materials.

Table 1.

Patient-perceived feasibility for each of the 7 PROMs tested

Instrument	Number of patients	Do you believe you could easily and efficiently complete this survey?	Did you feel that all the items on this page were clear and easy to understand?	Do you believe that this questionnaire could be used to assess [domain] in patients with myositis?
PROMIS Pain Interference Short Form 8a v1.0	11	11/100%	11/100%	10/90.9%
Pain Disability Index (PDI)	11	10/90.9%	10/90.9%	10/90.9%
PROMIS Fatigue Short Form 7a	12	12/100%	12/100%	11/91.7%
PROMIS Fatigue Short Form 13a (FACIT-Fatigue)	12	11/91.7%	12/100%	11/91.7%
PROMIS Physical Function Short Form 8b	10	10/100%	10/100%	9/90%
International Physical Activity Questionnaire (IPAQ)	10	10/100%	10/100%	10/100%
Myositis Activities Profile (MAP)	10	9/90%	10/100%	10/100%

Pain Survey #1 (PROMIS Pain Interference Short Form 8a)

Of the 11 patients who completed the instrument, 8 patients (73%) responded that pain was very relevant to their experience. In contrast, 3 patients (27%) responded that pain was not one of their symptoms and did not find any of the questions relevant to their experience. However, all three still could see how the questions could be useful and relevant for those with pain.

Patients described their pain in ways that were consistent with the survey items. For instance, one patient reported “If I do a lot one day, then I have to do less the next day.” This sentiment was shared among most respondents. Other themes included patients pushing through their pain and motivating themselves to perform their usual activities. Patients reported that pain affected their day-to-day activities, social activities, and family life.

There was strong support for all 8 items of the PROMIS Pain Interference 8a survey. However, three items were thought to be possibly redundant: “How much did pain interfere with your ability to participate in social activities?”, “How much did pain interfere with your enjoyment of social activities?” and “How much did pain interfere with your household chores?” Many respondents mentioned that the first two questions were redundant and should be combined. One respondent felt that everyone has different social activities, warranting greater specificity. Although 3 of the respondents did not experience pain as a symptom, they all believed that they could complete this survey as part of their doctor’s routine visit and that the questions were clear and easy to understand. Ten of 11 patients felt that this questionnaire could be used to assess pain. However, the one patient who was uncertain claimed that no single survey could recapitulate how pain affects an individual patient’s quality of life. He thought that surveys could only provide information on one’s “progression and regression” and could never reveal a patient’s medical experience.

Further, some respondents felt that additional questions that further characterize pain would be helpful. These include questions about the frequency, intensity and location of the pain, and the way pain affects sleeping patterns.

Pain Survey #2 (Pain Disability Index):

All 11 patients, regardless of whether they themselves suffered from pain, felt that these questions could be important to patients with pain.

A common theme described by patients was feeling limited, whether socially, physically or mentally. For example, one patient admitted that his social life was declining as he was not able to go out and relax with friends—as this would cause him undue pain. This same patient reported that his pain was also starting to interfere with his occupation and sexual life. Another patient felt that his pain was keeping him “from doing anything in life”, and he felt confined to his house. Many felt exhausted by the pain and the limitations that it imposed on their lives.

However, patients had concerns about the recreation and social activity items. Four patients (36%) recommended removing the recreation item. In fact, many respondents felt that the two were similar and should be combined, as they both referred to leisure. One patient proposed splitting leisure into two categories: non-physical social activities and physical social activities. For example, she mentioned that “going to dinner is easy” but “going to a concert or around town is so difficult”. This made it difficult for her to answer the question. The sexual behavior item also did not gain strong support for inclusion. The dissenting group felt that they did not think about engaging in sexual behaviors, as their pain created a strong aversion.

Overall, ten out of 11 (91%) patients felt the survey was feasible as part of a routine office visit, and 11 patients felt that the questions were clear and easy to read or understand. Ten of 11 patients felt that this questionnaire could be used to assess pain. Again, the one patient who did not think the survey could be used felt that no single survey could effectively assess patients’ experiences with pain.

Three patients preferred the Pain Disability Index over PROMIS Short Form 8a, although this question was not asked systematically; rather, patients spontaneously offered this point in discussion. One felt that “the questions are better, more detailed,” while another described the questions as “more definitive” and better “spelled out”. A few felt that additional questions should be asked to further characterize pain. One patient emphasized the question “are you taking overthe-counter medication for pain”, while another one proposed asking “how long does it take to get up from the bed and get rolling in the morning?”

Fatigue Survey #1 (PROMIS Fatigue Short Form 7a)

Twelve patients completed this survey, and there was strong support for 6 of the 7 items. Throughout the interviews, most patients commented that they have experienced tiredness and extreme exhaustion. One patient defined being tired as “having to sit down and taking a break”, while another described it as “running out of steam. I’m only good for four hours a day, and then I’m just stuck in bed.” Some patients felt that their fatigue was so strong that they had to take a nap in the day. Fatigue limited their work and their ability to engage in strenuous exercise. Although many mentioned that they pushed through their fatigue to complete their day-to-day activities, many admitted to modifying their lifestyles.

There were two items that were not universally endorsed by all patients: too tired to think clearly and too tired to shower/bath. Although they felt that both questions were relevant and should be retained, four patients commented that they never or rarely were too tired to think clearly, while seven patients mentioned that they were never too tired to take a bath or shower.

All 12 patients believed they could complete this survey at a routine clinic visit and felt that the questions were clear and easy to read and understand. Eleven patients (92%) felt that this questionnaire could be used to assess fatigue, while one felt uncertain about using this questionnaire. Similar to the pain surveys, this patient felt that surveys were inherently limited and could never truly assess patients’ fatigue. A few patients commented that they felt additional questions would be useful. Two patients felt that it was important to ask about sleeping patterns. Another patient commented that throughout the day her fatigue waxes and wanes and would have appreciated questions that inquired about timing.

Fatigue Survey #2 (PROMIS Fatigue Short Form 13a-- FACIT-Fatigue)

There was strong support for all the items in this questionnaire. However, there were varying opinions regarding the first four items: “I feel fatigued,” “I feel weak all over,” “I feel listless (washed out),” and “I feel tired.” Many patients felt that these questions were redundant, with some recommending that only two of the four should be asked. The two items with least support were “I feel weak all over” and “I feel tired” (for both, 8/12 patients still felt the questions to be relevant). Some patients interpreted feeling weak all over as similar to feeling fatigued, while others did not know how to interpret the question. For the question, “I feel listless (washed out),” some patients interpreted the question as how fatigue affects them mentally, while others interpreted it as how it affects them physically.

A shared sentiment amongst the respondents was that their fatigue slowed down the pace of their day. Many felt that they had to take a nap in the middle of the day in order to continue functioning. In fact, all patients confidently reported that the item “I need to sleep during the day” was relevant. All patients also felt that they experience some difficulty starting tasks, and once they start, it is difficult to finish the task. Overall, respondents mentioned they modified their lifestyle in order to manage their fatigue.

Eleven of the 12 patients (92%) believed they could complete this survey as part of their doctor’s routine visit, and 10 patients felt that the questions were clear and easy to read and understand. Eleven of the 12 patients felt that this questionnaire could be used to assess fatigue, while one felt uncertain about using this questionnaire. Similar to the pain surveys, this patient felt that surveys were inherently limited and could never recapitulate a patient’s unique fatigue experience. When asked what additional questions should be asked, a few patients recommended to include quantitative questions, such as how long does your fatigue last, or what part of the day is the fatigue strongest.

Physical Function #1 (PROMIS Physical Function Short Form 8b)

Ten patients completed this survey, and all items in the survey gained strong support from the respondents. A majority of patients commented that they had difficulty performing the activities included in most of the items. For example, one patient explained that he has to “pick and choose what he does” or else he “will crash.” Some patients at times needed clarification. For the item “Are you able to go walk for at least 15 minutes?” patients felt that it depended on whether it was flat ground or inclined. For the item “Does your health now limit you in lifting or carrying groceries?” patients again felt that it depended on the weight of the groceries. Nevertheless, they still felt that these questions were relevant and reflected the challenges that they face.

Further, all patients believed they could easily complete this survey, and felt that the questions were clear and easy to read and understand. Nine out of 10 patients (90%) felt that this questionnaire could be used to assess physical function, while one was uncertain about using this questionnaire.

Physical Activity #1 (Myositis Activity Profile, MAP)

All 10 patients who completed this questionnaire reported limitations to some extent in their physical activity. Limitations included having difficulty putting on clothes, walking for ½ a mile, lifting items or cleaning, for example. Twenty-seven (of 31) items had endorsement for relevance by at least 70% of respondents, while 4 items did not. Most patients felt this survey was comprehensive and detail-oriented. However, many also felt that the survey was very long.

Some comments centered on the ambiguity of a few questions, where additional clarification was needed such as “How much difficulty do you have transporting everyday items home in your daily life?,” “How much difficulty do you have lifting a child, pet, or objects into your arms in your daily life?”, and “How much difficulty do you have picking an object up from the floor (for example, lifting it to a table) in your daily life”. Many felt that it depended on the weight of the items and had trouble answering the question.

In spite of these criticisms, all ten of the respondents felt that they could complete this survey as part of their doctor’s routine visit, that the questions were clear and easy to read and understand, and that this questionnaire could be used to assess physical activity.

Physical Activity #2 (International Physical Activity Questionnaire, IPAQ Long Form)

Overall, almost all patients felt that their physical activity was limited compared to before IIMsymptom onset. This included spending less time performing vigorous and moderate physical activity, less time walking, and more time sitting down. Two of the seven items did not exceed the 70% threshold to support content validity, specifically the items that asked about the amount of time and days spent engaging in vigorous physical activity. Patients had difficulty recalling the amount of time spent on each activity, especially walking and sitting. In the words of one patient, “Everyone sits. Instead you should ask if my myositis makes me sit more.” A common recommendation was to provide a scale or provide answer choices, such as 1–3 hours, 4–6 hours, 7–9 hours and so forth, to make it easier to quantify.

Many recommended that these questions should be comparative and ask patients if they perform each physical activity more or less than before their illness. One patient, for example, felt that compared to her peers, she would be deemed fit and active; however, compared to her past self before her diagnosis, she felt worse off. She emphasized that these questions “do not capture a frame of reference.”

Despite the consensus that the instrument was very long, nine of the 10 respondents (90%) felt that they could complete this survey as part of their doctor’s routine visit. All ten of the respondents felt that the questions were clear and easy to read and understand and that this questionnaire could be used to assess physical activity.

Muscle Symptoms

Nine of the patients were asked the following item: “Are ‘muscle symptoms’ something distinct that is not otherwise described by pain, fatigue, physical activity and physical function?” Six of 9 patients (66%) felt that muscle symptoms were already captured by these other categories. One felt that it was not, and two were uncertain. Most patients described muscle symptoms to include weakness. However, for two patients (22%), they were not confident that pain, fatigue, physical activity and physical function assessed muscle symptoms. One patient also felt her muscle symptoms “penetrated deep” and included nerve and bone pain. She felt that achiness (as opposed to pain), fatigue, and limitation in physical activity and physical function captured her muscle symptoms.

Lastly, some patients offered additional perspectives outside of the targeted 5 domains. Several felt that comorbidities and medications affected their quality of life. Many patients who were interviewed had other illnesses and commented that they thought the survey should have asked about this. Some patients were not sure if the symptoms they experienced could be attributed to their myositis, a concomitant illness, or to adverse side effects of their medications.

Discussion

We evaluated the content validity of several candidate PRO instruments to assess relevant symptoms and impacts in adult IIM patients. All seven instruments studied were selected by the OMERACT Myositis Working Group based on input from focus group and prior survey data that described and prioritized symptoms of adult patients with IIM. Cognitive debriefing was performed with each survey to better understand the comprehensibility, feasibility, and content validity of the individual instruments. After analyzing the survey results, it was found that most patients felt that each of the instruments were clear, easy to read and understand, and could be used for assessment of the domain. Additionally, this study provides data on patients’ experiences with characterization of symptoms and disease burden.

While myositis is traditionally considered a painless condition, our prior qualitative studies identified pain as an important symptom for many myositis patients.^5,6 Patients felt that their pain impacted many aspects of their life, including family life, social life, occupation and sexual behavior. Patients felt that both the Pain Disability Index and the PROMIS Pain Interference Short Form 8a had redundancies with regard to social activities. For example, there was confusion on the difference between social activity and recreation in the Pain Disability Index. Nevertheless, patients appreciated the relevance of the instruments to their individual experiences and strongly supported both the Pain Disability Index and the PROMIS Pain Interference Short Form 8a. For both the PROMIS Fatigue 7a and 13a (FACIT-Fatigue) instruments, patients felt that a few questions were redundant. Some patients felt that running out of energy was synonymous to feeling tired (PROMIS Short Form 7a). Similarly, some patients felt feeling weak all over, feeling listless, and feeling tired were similar and would have appreciated clarification (PROMIS Short Form 13a FACIT-Fatigue). Still, most patients supported both of these instruments and felt that either could be easily deployed in the clinical setting.

For the domains physical activity and physical function, PROMIS Physical Function Short Form 8b gathered the strongest support from patients. All items passed the threshold of support and the form was of appropriate length. One limitation of this instrument was that it did not include questions about self-care, such as difficulty putting on clothes or taking a shower or bath. On the other hand, the Myositis Activity Profile was highly comprehensive and explored a breadth of dayto-day activities. However, this instrument was considered to be lengthy and some items required further clarification (i.e., for questions that asked about lifting objects—the weight should be specified). Lastly, some patients suggested that the International Physical Activity Questionnaire could be altered to compare a patient’s present baseline to their baseline before their diagnosis. Patients had difficulty quantifying the number of hours they spent performing a specific activity.

Overall, these instruments reflect widely endorsed concepts that are potentially applicable across multiple diseases and removes the need to create unique myositis-specific PROMs. PROMIS instruments comprehensively cover the spectrum of the domains in question. These measures were calibrated in the general United States population and have been used in other rheumatic conditions^9–11. Further, these instruments have been widely validated and translated into multiple languages and could allow clinical trials to target a wider population worldwide.

Although some patients felt certain questions within a given instrument were redundant, we do not recommend modification, as questions that are seemingly redundant can provide valuable information when discriminating amongst patients (e.g. the measurement properties gained by including each of the individual questions in a given instrument outweighs patients’ perceptions on redundancy). In instruments developed using item response theory such as PROMIS measures, by dropping a single question, one may lose the ability to discriminate between individuals along the continuum of the trait being assessed where that item is providing important information. Further, this may require additional validation steps or recalibration. Although all the instruments have limitations, this study has shown that the instruments in the domains of pain, fatigue, physical activity, and physical function all met the >70% threshold of acceptance. Such encouraging results warrant broader study in a larger IIM population, which is currently underway via an international electronic survey.

Another major aim of this study was to ascertain if the area originally identified in qualitative studies, “muscle symptoms,” could be encapsulated by the other four domains (fatigue, pain, physical activity and physical function). Most patients queried (66%) felt that muscle symptoms could be captured by the remaining four domains. Although close, the threshold for support (70%) for removing muscle symptoms as a domain was not met. Additional data will be sought through an international electronic survey to better understand these relationships.

As referenced above, the Myositis Working Group plans to administer these same instruments to a larger number of patients internationally to increase the sample size from which to draw conclusions. The information obtained from the forthcoming survey along with the current cognitive debriefing report will be used to identify the optimal instruments to measure the relevant domains in studies of adult IIM. Once identified, instruments would be administered in longitudinal adult IIM cohorts to further evaluate reliability, construct validity, responsiveness, and minimally and clinically relevant change. The end goal is to arrive at a set of PROMs fully validated in adult IIM for use in clinical trials and observational studies.