Patient-reported outcome measure for neuromyelitis optica: pretesting of preliminary instrument and protocol for further development in accordance with international guidelines

Abstract

Objective

This study outlines the development of a patient-reported outcome measure (PROM), an instrument to obtain self-reported health status for neuromyeltis optica (NMO), a disabling neurological condition.

Design

Development was conducted in accordance with international guidance for PROMs including systematic review of existing literature, item generation guided by qualitative interviews, health-related quality of life conceptual framework and clinical expert panel and cognitive interviews with NMO patients.

Setting

Participants were identified through a national NMO clinic in a tertiary NHS neurosciences service.

Participants

15 individuals with NMO participated in cognitive interviews requiring review and ranking of proposed PROM items and qualitative feedback on content, layout and response options.

Results

Participants endorsed the draft instrument as reflecting their experience of the condition and as being easy to understand. Rating and ranking of item relevance and importance reduced the draft instrument from 106 to 48 items. Participant feedback on overlapping items eliminated a further 2 items and resulted in a preliminary instrument of 46 items. As a direct result of participant feedback ordering of the 10 domains was revised, a 4 option Likert scale was employed and a 4-week recall period for impact of symptoms was selected.

Conclusions

A 46-item instrument developed in accordance with international PROM development guidelines through literature review, developed by subject matter experts and refined through pretesting examining content validity provides a preliminary measure for assessing patient-report of health status in NMO. Further evaluation is proposed including sensitivity to clinical change, and international contributions to evaluating the measure are encouraged.

Strengths and limitations of this study.

• Assessment of self-reported health outcomes in neuromyelitis optics (NMO) is limited by the lack of disease-specific outcome measures.

• A scarcity of existing literature on quality of life in NMO highlights the paucity of information regarding the lived experience of people with NMO.

• The preliminary patient-reported outcome measure (PROM) is based on a conceptual framework for health-related quality of life in NMO, developed by clinical experts and further refined through assessment of content validity with NMO patients.

• A significant strength is that development of the NMO PROM has been conducted in accordance with international guidance for PROM development. Along with the further proposed validation work, this will provide an outcome measure suitable for use in future clinical trials of NMO treatments.

Introduction

Neuromyelitis optica (NMO) is an inflammatory, demyelinating syndrome of the central nervous system.¹ Symptoms may begin at any age, though the most common age of onset is during the fourth decade,² and women are predominantly affected with a gender ratio of 5:1.³ NMO is an uncommon disease.⁴ In the UK, the prevalence is between 7.2 and 19.6/million.^{3 5} NMO is characterised by periodic exacerbations of symptoms (relapses) that typically have a rapid onset and can cause severe residual neurological disability often early in the disease process.

Typical relapses cause transverse myelitis, optic neuritis or brain stem involvement.⁶ Treatments, predominantly immunosuppressants, aim to prevent further relapses but are only partially effective as disability accrues with each relapse.

Patient experience and quality of life has become a major focus of UK healthcare policy for people with long-term conditions.⁷ In 2006, the WHO concluded that the quality of life and maintenance of dignity in people with neurological conditions needed to be significantly improved on an international level, and quality of life is now recognised as a vital outcome when investigating the effectiveness of healthcare services to patients.⁷

The USA Federal Drug Administration (FDA) defines a patient-reported outcome (PRO) as any report of the status of a patient's health condition that comes directly from the patient without interpretation of the response by anyone else.⁸ The FDA guidance notes that the concepts measured by PRO instruments most often refer to a patient's symptoms, signs or an aspect of functioning directly related to disease status and PRO measures often represent the effect of the disease on health and functioning from the patient perspective.

NMO is a chronic condition with substantial potential impacts on health and functioning. The severity of symptoms, combined with the varied and unpredictable nature of the condition, has been shown to have substantial impacts on health-related quality of life.^{9 10}

Historically, NMO was thought to be related to multiple sclerosis (MS) but has now been established as a separate disease. Whereas the primary pathology in MS is demyelination, in NMO the primary target for autoimmune inflammation is the astrocyte. Experimental studies have confirmed the pathogenicity of the aquaporin-4 antibody (AQP4-antibody), an NMO-specific antibody to the dominant water channel in the central nervous system expressed on the end-feet of astrocytes.^11–13

The lack of disease-specific outcome measures in NMO has led to the use of generic outcome measures and those developed for use in MS.^{9 14–16} Examples of MS-specific PROMS include the Leeds Multiple Sclerosis Quality of Life (Leeds MSQol)¹⁷ and the Multiple Sclerosis Impact Scale (MSIS-29),¹⁸ which comprise a unidimensional measure of well-being and measures of physical and psychological impacts of MS, respectively. Our experience at a national NMO clinic situated within a tertiary specialist neuroscience service located in North West England is that the path towards diagnosis for a substantial number of NMO patients involved diagnostic uncertainty and sometimes initial misdiagnosis of MS. Not only have we found anecdotally that the use of outcome measures intended for use with MS populations to be confusing and frustrating for NMO patients, but that these measures do not necessarily capture the experiences and difficulties faced by NMO patients. Our observations are supported by the broader view that disease-specific measures have greater validity and credibility than generic measures.¹⁹

Our aim was to develop a preliminary PROM that effectively captures the lived experience of individuals with NMO suitable for use in NMO clinical and research settings and to establish a protocol for further development and validation of the instrument.

Method

Participants

Our patient sample for pretesting of the preliminary instrument comprised of 15 adult patients with a diagnosis of NMO using established criteria at the time of participation.²⁰ Our sample size for pretesting was determined in accordance with guidance on sample sizes for PROM development, which recommends constructing a sample matrix to determine different subgroups of patients and allocating 15 participants for each cell in the sample matrix.²¹ Given the rarity of this condition and the considerable overlap in symptoms within NMO, it was not considered representative or viable to stratify subgroups into different cells. We therefore based our required sample size on a single cell of 15 patients with NMO.

Our sample was recruited from the national NMO clinic at The Walton Centre NHS Foundation Trust, Liverpool, UK. We excluded patients who were unable to read or write in English and purposively sampled according to obtain a range of participant ages, gender and disease duration.

The study received ethical approval from the Lancaster, North West UK, Research Ethics Committee (NW/1510203).

NMO PROM instrument

The development stages of the PROM up to and including the results presented in this paper are summarised in figure 1. A systematic review was completed to identify quality of life issues in people with NMO. In brief, our systematic review comprised searching three major databases (EMBASE, MEDLINE, PsycInfo) for the terms ‘neuromyelitis optica’, ‘NMO’, ‘transverse myelitis’, ‘optic neuritis’ and ‘quality of life’ and derivatives in Medical Subject Headings (MESH) terms, titles and abstracts. Initial searches identified 150 possible studies. After initial screening and full text review, only seven relevant studies remained. The six included studies^{22 9 23–26} comprised a total of 161 people with NMO and 14 with NMO spectrum disorder. Of these, four studies (total of 105 participants) examined the impact of pain, one study was a case study that discussed various factors potentially impacting on quality of life in NMO and the remaining two studies (total of 70 participants) looked at multiple factors impacting on quality of life. Despite identifying a total of 10 domains across the 6 studies, only 3 studies totalling 71 patients explored factors other than pain.

Figure 1.

Flow chart of PROM development up to and including the results presented in this paper. PROM, patient-reported outcome measure.

In addition, qualitative interviews were conducted with a different sample of 15 people with NMO to the pretesting sample (mean disease duration of interview participants=8.7 years; median age 48 years; AQP4-antibody positive=9 (70%)), to enquire about quality of life issues experienced in this population. When reviewing the themes throughout these interviews, this further reaffirmed and modified the domains identified from the systematic review. This resulted in the inclusion of 10 distinct domains to be included in the preliminary instrument.

In accordance with international PROM development guidance,⁸ our approach to generating potential items for the preliminary instrument encompassed a review of existing PROMs, expert review and consideration of disease-specific qualitative research on quality of life.

A review of existing PROMs, conducted by searching the PROM databases PROQOLID and PROMIS to identify existing PROMs that might have relevance to assessing quality of life in NMO, identified 92 existing questionnaires considered to be potentially applicable to NMO patients but no NMO-specific PROMs or PROMs validated on an NMO population. Expert review was conducted using a panel of subject matter experts to identify items of relevance to NMO, and these were considered alongside previous qualitative research on the lived experience of NMO²² and a conceptual framework of quality of life in NMO.¹⁰ Consideration by our expert panel resulted in an initial consensus view of 106 items that were considered to be of possible relevance.

Our initial approach focused on inclusivity to maximise the potential spectrum of NMO-specific health-related quality of life outcomes incorporated. Where possible we adopted verbatim transcriptions from patient interviews to incorporate patient perspectives as directly as possible. This provided a draft NMO PROM that was evaluated by a panel of multidisciplinary clinical and health outcome methodology experts.

The expert panel met on three occasions. As well as reducing the number of items the panel agreed to insert additional items where the spectrum of potential impacts on functioning was not considered to be adequately covered. In addition, wording of items was considered in terms of specific relevance to NMO.

Although overlap and relevance of items were considered, the threshold for rejection of items remained high at this stage so as to maintain a broad scope of items for subsequent evaluation by patients during the cognitive interview stage.

Following consideration by the expert panel, a draft instrument was sent to other healthcare professionals with a good knowledge of NMO in order to receive further feedback. Following this, we conducted cognitive interviews with NMO patients.

Procedure

Semistructured face-to-face interviews were conducted with 15 patients diagnosed with NMO who had not taken part in previous stages of this research. Interviews took place either in patients own homes or in the outpatient department depending on preference and circumstances of the patient.

There were two parts of the semistructured interview schedule; a quantitative phase and a qualitative phase.

Quantitative phase

In this part of the interview, participants were given time to review and discuss the list of items in each domain and were then asked to rate each item for importance and relevance on two separate 5-point Likert scales graded from 1 (not important/relevant) to 5 (very important/relevant). Relevance and importance scores for each item were combined, so each item had one overall score that ranged from 2 to 10. In addition to this, participants were then asked to rank the items within each domain from the most important and relevant item to the least important and relevant item.

Qualitative phase

After participants had reviewed and scored each item in the questionnaire, they were asked a number of open questions to obtain feedback about the content, layout and response options included in the list of items. This part of the interview followed a structure which was developed following recommendations from well-established PROM development guidelines,²⁷ but further questions were asked where appropriate to encourage open discussions depending on feedback from participants. The questions covered three key areas: content, layout and response options. A final question was asked to invite participants to give any other feedback or comments.

There is currently no agreed on method for analysing data collected during interviews to modify PROM instruments.²⁸ A panel of subject matter experts reviewed the outcomes of the quantitative ratings, and a consensus was agreed that all items that had a combined rating (scores from relevance and importance scales) of lower than 6 were excluded. This was determined by considering the impact of different cut-off scores on the content of the PROM.

Participants’ responses to the qualitative questions were reviewed by the experts. Where possible, participants feedback would be used to guide modifications to the measure. At times, a single negative comment led to an item being modified. Feedback about more direct questions was also used to guide decisions about the format of the NMO PROM; whichever option received the most support from participants was adopted for the PROM.

Results

Fifteen people with NMO took part in the present study; 11 females and 4 males. The mean age of participants was 45.4 years (SD=15.6; range 24–77). All patients had a diagnosis of NMO spectrum disorder in accordance with the most recent diagnostic criteria,²⁰ and 11 (73%) were AQP4-antibody positive. The average Expanded Disability Status Scale (EDSS) score of participants was 4.7 (SD=2.2; range 1–8.5). These patients attended from across UK and are not restricted to the city of Liverpool.

All participants completed the qualitative interview stage. One participant did not complete the quantitative phase of the interviews due to time constraints.

Quantitative phase

Fifty-eight items had a combined score for relevance and importance of <6. Low scoring items were reviewed by comparing where these items ranked within each domain. All low scoring items were consistently ranked as least important and relevant in that domain. Therefore, 58 items were excluded from the preliminary measure.

Qualitative phase

The semistructured interviews highlighted four key areas which could be modified to improve the PROM. These were content, layout and design, recall period and response options.

Content

Overall participants reported that they believed the key areas of their experience were covered by the domains. When asked if there was anything they would add to the NMO PROM such as a concern which was not captured in the existing domains, all participants reported that they believed there was nothing to add. Similarly, participants were asked if any domains were unnecessary and should therefore be removed from the NMO PROM. All participants reported that they believed all domains were relevant and important and therefore would not remove any domains.

Four participants identified possible overlap between questions. Potential overlaps were reviewed by the panel of experts, and two items were removed (the items removed had a lower mean combined score than the item they overlapped with). This resulted in 46 items being retained for the preliminary measure.

Layout and design

When asked about the ordering of domains, two participants reported that they would prefer if mood was not the first domain that they were presented with. Both participants described mood as a highly emotive domain and that this was the reason they would prefer if it was placed more in the middle of the instrument. All participants reported that the design of the NMO PROM was easy to understand and follow.

Recall period

During the interview, participants were invited to discuss the response options of the NMO PROM. In addition to this, the interviewer asked direct questions to ascertain whether each participant believed a 2-week or 4-week recall period would enable them to answer more accurately. Twelve participants said they would prefer a 4-week recall period, two participants expressed a preference for a 2-week recall period and one person did not have any preference.

Response options

In the preliminary instrument, there were two sets of response options. One set provided the following options: ‘not a lot’, ‘a little’, ‘moderately’ and ‘a lot’. The second set of response options included: ‘not a lot’, ‘several days’, ‘more than half the days’ and ‘most days’. Participants were asked which set of response options enabled them to answer more accurately. Eleven participants expressed a preference for the ‘a lot’ to ‘not at all’ response options, whereas four participants preferred the ‘most of the days’ to ‘not at all’ response options. Therefore, the response options of the preliminary instrument were modified in accordance with feedback from the majority of participants. Many participants reported that they found response options based on the number of days confusing.

Fourteen participants expressed a preference for four response options, and one participant preferred five response options. Some participants explained that they believed having no middle option would encourage them to think about the question more, and some participants stated that five options would be too many.

Preliminary instrument following pretesting

The resulting instrument for assessing PROs in NMO, developed through the described process including pretesting, is presented in table 1.

Table 1.

Neuromyelitis optica patient-reported outcome measure—preliminary instrument

In the past 4 weeks:	No difficulty	A little difficulty	Moderate to severe difficulty	Completely blind
1. Vision
1.1 How much difficulty have you had with your vision (while wearing your glasses or contact lenses, if you wear them)	1	2	3	4
1.2 Because of your eyesight, how much difficulty do you have reading ordinary print in newspapers?	1	2	3	4
1.3 Because of your eyesight, how much difficulty do you have noticing objects off to the side while you are walking along?	1	2	3	4
1.4 Because of your eyesight how much difficulty do you have doing things like shaving, styling your hair or putting on make-up?	1	2	3	4

In the past 4 weeks:	Not at all	A little	Moderately	A lot
2. Role/life functioning
2.1 Has NMO affected your ability to meet your expected goals at this point in your life	1	2	3	4
2.2 How much difficulty have you had accomplishing the things that are important to you?	1	2	3	4
2.3 Has NMO made you dependent on other people?	1	2	3	4
2.4 How often have you not been able to bath and dress yourself?	1	2	3	4
2.5 Have you had difficulty participating in family and social activities in a satisfying way?	1	2	3	4
2.6 Have you stayed at home most of the time because of symptoms of NMO?	1	2	3	4
2.7 Have you had difficulty completing household tasks?	1	2	3	4

In the past 4 weeks, how much has your NMO:	Not at all	A little	Moderately	A lot
3. Mobility
3.1 Limited your mobility	1	2	3	4
3.2 Limited your ability to walk?	1	2	3	4
3.3 Limited your ability to climb up and down stairs	1	2	3	4
3.4 Limited your balance when standing or walking	1	2	3	4
3.5 Made it necessary for you to use support when walking (eg, holding on to furniture, using a stick, etc)	1	2	3	4
3.6 Limited how long you can work or do other activities	1	2	3	4

In the past 4 weeks, how bothered have you been by:	Not at all	A little	Moderately	A lot
4. Bladder
4.1 Overall, how much do urinary symptoms interfere with your day-to-day life?	1	2	3	4
4.2 Frequent urination during the daytime hours?	1	2	3	4
4.3 An uncomfortable urge to urinate?	1	2	3	4
4.4 A sudden urge to urinate with little or no warning?	1	2	3	4
4.5 Involuntary urination or loss of urine?	1	2	3	4
4.6 Difficulty initiating urination when you want?	1	2	3	4

In the past 4 weeks:	Not at all	A little	Moderately	A lot
5. Bowel
5.1 How much has NMO affected your bowel management?	1	2	3	4
5.2 Have you experienced constipation?	1	2	3	4

Over the past 4 weeks, how often have you been bothered by any of the following problems:	Not at all	A little	Moderately	A lot
6. Mood
6.1 Altered mood related to your NMO?	1	2	3	4
6.2 Little interest or pleasure in doing things?	1	2	3	4
6.3 Feeling down, depressed or hopeless?	1	2	3	4
6.4 Feeling nervous, anxious or on edge?	1	2	3	4
6.5 Not being able to stop or control worrying about symptoms related to NMO?	1	2	3	4
6.6 Becoming easily annoyed (snappy/irritated)?	1	2	3	4
6.7 Feeling frustrated a lot of the time because of the symptoms of NMO?	1	2	3	4

Sexual arousal is a feeling that includes physical and mental aspects of sexual excitement.
In the past 4 weeks:	Not at all	A little	Moderately	A lot
7. Sexual dysfunction
7.1 How much has NMO affected your sexual activity?	1	2	3	4
7.2 How much has NMO affected your level of sexual desire or interest?	1	2	3	4
7.3 How much has NMO affected your level of sexual arousal during sexual activity or intercourse?	1	2	3	4
7.4 How much has NMO affected feelings of intimacy or closeness?	1	2	3	4

In the past 4 weeks:	Not at all	A little	Moderately	A lot
8. Pain
8.1 How much physical pain have you had as a result of your NMO symptoms?	1	2	3	4

In the past 4 weeks:	Not at all	A little	Moderately	A lot
9. Fatigue
9.1 How much has fatigue affected you?	1	2	3	4
9.2 Have you needed to rest more often or for long periods after physical or mental activity?	1	2	3	4
9.3 If you have overdone things then will you feel fatigued the next day?	1	2	3	4
9.4 Have you become weak after physical activity?	1	2	3	4
9.5 Has mental fatigue had been a problem for you?	1	2	3	4
9.6 Have you had difficulty sleeping?	1	2	3	4

In the past 4 weeks:	Not at all	A little	Moderately	A lot
10. Cognition
10.1 Have you experienced difficulty with your ability to think, reason and remember?	1	2	3	4
10.2 Have you had difficulty organising your thoughts when doing things at home or at work	1	2	3	4
10.3 Have you found it hard to concentrate on relatively straight forward things	1	2	3	4

NMO, neuromyelitis optica.

Discussion

This study outlines the development of the NMO PROM, an instrument to obtain NMO patients' self-report of their health status and reports on its content validity. The instrument is designed to be used in clinical and research settings (including clinical trials) to obtain self-report on NMO patient symptoms and functioning.

The NMO PROM was developed in accordance with the methods required to ensure the content validity of PROMs intended to be used in clinical trials to collect data to support label claims.^{8 29 30}

A systematic review of existing literature on quality of life in NMO identified a very limited evidence base for factors affecting quality of life in the condition and in our view highlighted the paucity of information regarding the lived experience of people with NMO. We intend to provide a detailed account of the systematic review methodology and findings in a forthcoming publication that will further contribute to the evidence-base underpinning the PROM development.

Item generation was guided by a conceptual framework¹⁰ and clinical experts with particular attention given to the wording used by patients. Cognitive interviews conducted with patients using the draft measure evaluated whether the items comprehensively evaluated the lived experience of individuals with NMO. The cognitive interviews also provided evaluation of the interpretation of instructions and items and that the response options were understood and considered appropriate by patients.

Strengths and limitations

The importance of addressing patient experience is widely recognised as being crucial in evaluating the effectiveness of healthcare.³¹ This is regarded as being particularly salient in conditions with no agreed on biomarkers to monitor disease progression.^{32 33} Although research on NMO is gradually providing increasing insight into the disease, the identification of a reliable agreed on biomarker to monitor disease progression remains elusive. Of further difficulty, the literature on healthcare outcomes and health-related quality of life of NMO patients remains scarce. At present, there are no NMO-specific instruments with which to assess this. For these reasons, the development of the NMO PROM relied heavily on using the experience of clinicians with specialist expertise in the condition combined with the lived experience of individuals with the condition. In our view, this highlights the importance of pretesting of our preliminary instrument. Previously developed PROMs have identified the importance of cognitive interviews for identifying problems in the content and layout of items but also in identifying missing items and difficulties with self-completion.³⁴ In this regard, we considered the sparsity of extant sources to draw on in the development of the instrument a challenge, but the subsequent importance placed on patient participation during development and pretesting as a strength of our process.

Pretesting of the preliminary instrument comprised the evaluation of a paper-based questionnaire. Future validation work might usefully include comparison between different media formats, including verbally administered (ie, read out to the participant) and braille versions for those unable to complete the paper instrument along with evaluation of telephone-administered and electronic/web-based formats that would enable alternative collection mediums.

A limitation of our approach is that the NMO PROM was developed in the UK, and initial validation was conducted with a UK resident patient group. The reports of lived experience and subsequent relevance and importance ratings of our NMO patients that have contributed to the NMO PROM development may therefore have been shaped by contextual factors relevant to the experience of the condition and healthcare provision in the UK. We therefore encourage international re-evaluation of the content of the preliminary instrument to help determine whether cross-cultural adaption is required during further development of the PROM to ensure that it is appropriate for cultures and ethnic groups outside the UK.

Further development

In accordance with international guidelines for PROM development, we intend to conduct two further validation stages to field test the preliminary instrument, comprising traditional psychometrics and Rasch analysis. Participants will be recruited from the Specialist NMO clinic at the Walton Centre, Liverpool, UK. If sufficient recruitment is not obtained, then participants will be recruited from other UK centres.

Field test 1

Further validation includes testing to see if the number of items can be further reduced and to examine psychometric properties. Participants will be asked to complete the preliminary measure twice with a period of 2 weeks between first and second completion. Participants will also be asked to complete a debriefing questionnaire, which has been devised from accepted PROM guidelines from the European Organisation for Research and Treatment of Cancer²⁷ that are now widely accepted for PROM development throughout healthcare.

Demographic information comprising of age, gender and level of education will be obtained from the participant, and clinical data including disability status, visual acuity and contrast sensitivity, and time since disease onset will be obtained through review of patients' records by a member of the clinical team and used to examine relationships with PROM responses. The rationale for assessing vision and mobility is that they are the primary markers used in describing the general clinical course of NMO.

Traditional psychometric validation will include test–retest validity over a 2-week period to examine stability of responses reported using the measure, internal consistency and content validity. Item-response theory analysis based on latent trait modelling will be conducted using Rasch analysis to assess item fit and dimensionality of the instrument.

The results from the statistical analyses will be considered through further review by clinical and statistical experts to inform any further refinement in the scale including through item reduction.

Field test 2

Participants will be asked to complete the measure again 12 months after Field test 1 to determine whether change in responses on the measure corresponds to changes in the clinical status of participants. Participants' clinical status will be reviewed in order to determine if there have been changes in clinical status in terms of mobility and vision. Statistical analysis will assess the sensitivity of the measure to changes in clinical status.

Conclusion

There is at present no PROM in NMO. This work summarises the preliminary work done and future development plans of an NMO-specific PROM, with the aim of obtaining national and international viewpoints for the re-evaluation of the preliminary instrument. Refinement and further adaptation of a new outcome measure contributes to the process of strengthening the relationship between the underlying conceptual framework and empirical evidence.³³ As such, the application of the PROM in different settings is important in demonstrating the usefulness of the measure through an underpinning evidence base. We encourage and look forward to international contributions that will support our goal of improving the care of people with this rare condition through effectively evaluating quality of life outcomes.