Patient-reported outcome measures in subarachnoid hemorrhage

Abstract

Objective

Patient-reported outcomes (PROs) are aspects of a patient's health status and are considered important for stimulating patient-centered care. Current outcome measures in clinical care for patients with aneurysmal subarachnoid hemorrhage (aSAH) are insufficient to capture PROs. In this systematic review, we aimed to summarize the evidence regarding the quality of patient-reported outcome measures (PROMs) in aSAH patients.

Methods

We performed a systematic review of the literature published from inception until October 29, 2018, in PubMed, the Cochrane Central Register of Controlled Trials, and EMBASE. Eligible studies had to evaluate measurement properties and capture PROs in aSAH patients. The quality of the studies and measurement properties were assessed using the consensus-based standards for the selection of health status measurement instruments (COSMIN) checklist. The review protocol was registered with PROSPERO (CRD42018058566).

Results

We identified 9 articles that reported the assessment of 7 different disease-specific and generic PROMs used for aSAH patients, including 5 that focused on the Stroke-Specific Quality of Life Scale (SS-QoL). The methodologic quality of the validation processes used was generally doubtful. None of the PROMs complied with current standards for content validity.

Conclusions

Due to the low quality of evidence for the measurement properties, the evidence base for selecting a suitable PROM for use with aSAH patients is insufficient. Given the specific long-term consequences of aSAH, we consider a disease-specific PROM the most appropriate, with SS-QoL the most suitable PROM currently available.

Improvements in treatment strategies in aneurysmal subarachnoid hemorrhage (aSAH) led to a reduction in mortality over the last 30 years,^1,2 meaning that the functional outcome and quality of life of patients with aSAH is becoming increasingly important.^1,2 Approximately half of patients report persistent problems with memory, mood, or neuropsychological function.^3–5 These cognitive deficits often go undetected by gross neurologic measures such as the Glasgow Outcome Scale and the modified Rankin Scale (mRS), but can be measured using patient-reported outcome measures (PROMs).⁴ PROMs measure patient-reported outcomes (PROs) and reflect the patient's view of his or her own health status. They are important for improving patient-centered health care and managing the follow-up care of patients.⁶ Although they could be valuable for measuring outcomes that are not represented in clinical outcome measures, PROMs are seldom used in research involving patients with aSAH.^7,8

In this systematic review, we provide an overview of the available PROMs used in patients with aSAH and review their validity and reliability. This provides an evidence base for the future selection of PROMs in aSAH research and clinical practice, enabling the most effective detection of PROs for patients with aSAH.

Methods

Search strategy

The review protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) of systematic review protocols (CRD42018058566).⁹ We performed a systematic search of the literature published from inception until October 29, 2018, in PubMed, the Cochrane Central Register of Controlled Trials, and EMBASE. This was done using a 2-step approach: first, a systematic search of the literature was performed from inception until August 15, 2017, and second, the search results were updated monthly to add any potential new studies to the selection. An independent and experienced librarian contributed to the search strategy. To identify studies concerning the measurement properties of PROMs, we searched for validation studies using the keywords “aneurysmal subarachnoid hemorrhage” and “patient-reported outcomes.” For the latter, we applied the PROM search filter developed in February 2010 by The PROM Group Oxford at the University of Oxford.¹⁰ The filter was adjusted and expanded to improve the sensitivity of the search strategy for our target group. In addition, we applied the sensitive search filter developed by Terwee et al.¹¹ to identify studies on measurement properties. For the detailed search string used in PubMed, see table 1. The search string was adjusted for use in the other databases. Finally, we performed a reference search of the included full-text articles to identify other relevant studies.

Table 1.

PubMed search strategy

Selection criteria

Studies were eligible for inclusion if they considered the measurement properties of PROMs in patients with aSAH. The aim of the PROM should be to capture the general aspects of health status, aSAH-related quality of life, or the symptoms associated with aSAH. The PROM needed to have been named to enable its identification. Studies were included if the article was in English, the full text was available, and the results for patients with aSAH were presented separately from any other study population. Editorials and studies concerning children <18 years of age or nonhumans were excluded.

Data collection and analysis

Selection of studies

Duplicate articles were automatically removed using EndNote X8. Two researchers (E.N.-J. and R.H.M.A.B.) independently reviewed the titles and subsequently the abstracts of the articles. For the included articles, full-text versions were obtained and independently evaluated by the same researchers. In case of disagreement during this process, a third reviewer (P.J.v.d.W.) was asked to adjudicate.

Data analysis

The studies were analyzed in the 3-step procedure described below.

COSMIN step 1: Quality assessment of the studies

The assessment of the methodologic quality of the included studies was based on the updated consensus‐based standards for the selection of health measurement instruments (COSMIN).^12–14 This involves a 3-step appraisal of the quality of a PROM: (1) the assessment of the content validity; (2) the appraisal of the internal structure of the questionnaire; and (3) the evaluation of the remaining measurement properties (i.e., reliability, measurement error, criterion validity, and hypotheses testing for construct validity and responsiveness). These steps are allocated in 10 boxes in the COSMIN checklist, each with 3–35 items concerning the methodologic standards for each measurement property. Each item was independently assessed by 2 researchers (E.N.-J. and P.J.v.d.W.) using a 4-point rating scale (very good, adequate, doubtful, or inadequate). Finally, an overall score of a measurement property was obtained by taking the lowest score for any of the items in that property.^12–15

COSMIN step 2: Quality assessment of the measurement properties

The definitions used for the measurement properties are based on the consensus on taxonomy, terminology, and definitions reached by the COSMIN panel. The COSMIN criteria were used to assess whether the results of the measurement properties were sufficient, insufficient, or indeterminate (table 2).^13,15,16 We reviewed the measurement properties in duplicate (by E.N.-J. and P.J.v.d.W.), and in cases of initial doubt in the final rating, agreement was reached by debate between the 2 reviewers. For the hypotheses testing for construct validity and responsiveness to be considered sufficient, the result should be in accordance with the hypotheses. In cases where no hypotheses were provided in the article, the review team formulated hypotheses themselves (table 3).¹⁴ Due to the limited studies concerning PROMs in patients with aSAH, it was not possible to deduce a conclusion on the quality of measurement properties by statistically pooling the results; therefore, we report the best evidence synthesis per measurement property separately. Finally, since there are no quality criteria for interpretability, we present the data in the text.

Table 2.

Quality criteria for measurement properties

Table 3.

Hypotheses about correlations as formulated by the review team

COSMIN step 3: Assessment of the level of the evidence

The quality of the evidence for each PROM was based on the number and quality of the studies using this approach, and summarized for each measurement property based on a modification of the GRADE methodology as outlined in the COSMIN guidelines.¹⁶ This methodology considers the initial level of evidence to be high, with subsequent downgrading based on the scores for 4 criteria: risk of bias, inconsistencies, imprecision, and indirectness. This results in a high, moderate, low, or very low quality score for the evidence. To evaluate the measurement properties of PROM development and content validity, the overall level of evidence was based on the recently updated COSMIN recommendations.¹⁷

Data availability

All data included in this review are available in the articles indicated in table 4. Our search strategy for PubMed is shown in table 1. The predefined hypotheses of the review team are available in table 3. The list of excluded studies based on full-text analysis, including the reason for exclusion, is available on request by investigators. The extracted data are also available on request.

Table 4.

Characteristics of the included studies in the systematic review

Results

Selection of validation studies and PROMs

The search strategy yielded 9,062 articles, of which 6,967 were unique articles. Reference checks resulted in the identification of 8 additional articles. One article published after the date of the initial search was included. After the checks of the titles and abstracts, 47 articles remained. A thorough evaluation of the full texts resulted in a total of 9 articles included in this systematic review.^18–26 The details of the selection process are shown in the figure. The main reasons for the exclusion of articles were that the questionnaire used in the article was not a PROM or the results for patients with aSAH were not presented separately from those of patients with other (vascular) diseases. The 9 included articles assessed 7 different PROMs: EuroQol 5D (EQ-5D), Health Utility Index (HUI), mRS, Quality of Life after Brain Injury overall scale (QOLIBRI-OS), Subarachnoid Haemorrhage Outcome Tool (SAHOT), Stroke-Specific Quality of Life scale (SS-QoL), and the short version of the SS-QoL. The SAHOT was the only identified PROM specifically developed for SAH. The general characteristics of the studies are presented in table 4 and the characteristics of the PROMs in table 5.

Figure. Flow chart of the literature search and selection of relevant articles.

PROM = patient-reported outcome measure, SAH = subarachnoid hemorrhage.

Table 5.

Characteristics of the included patient-reported outcome measures (PROMs) in the systematic review

COSMIN step 1: Quality of the included studies

The methodologic quality of the included studies is presented in table 6 for each questionnaire per measurement property. No assessment of content validity, measurement error, or cross-cultural validity was made in any of the studies, nor was there any reference to previous publications reporting on these measurement properties for the same study population. The majority of the properties were rated as “doubtful” or “inadequate.” Flaws in the design of the study or statistical methods of the study contributed 5 times to a final score of “doubtful” and once to an “inadequate” score. These flaws included a lack of a clear description of the applied statistics, no detailed information about which study group was included in the analysis, or other methodologic shortcomings. In 4 articles, it was unclear whether the scale or subscale was unidimensional, resulting in a “doubtful” score. The other causes of a low overall score on a measurement property were an insufficient sample size in factor analysis, scarce information on the measurement properties of the comparator instruments, the inadequate application of statistics, or an unspecified time interval between repeated measures. Criterion validity was only assessed for the 2 articles concerning the short version of the SS-QoL, for which the long version functioned as the gold standard. No gold standards exist for the other PROMs. The score was once “very good,” and once “doubtful,” because variance was not calculated separately for all domains.

Table 6.

Methodologic quality assessment of the included studies per measurement property per patient-reported outcome measure (PROM)

COSMIN steps 2 and 3: Measurement properties and quality of evidence of the PROMs

The assessment of the measurement properties and the quality of evidence is shown in table 7. The evidence for the measurement properties was scarce due to the small number of studies using these PROMs in patients with aSAH. In addition, only minimal evidence was available regarding the interpretability of the PROMs in the articles, and no formal information on the feasibility of using each PROM for patients with aSAH was provided. The details of the assessed measurement properties of each PROM are outlined separately below.

Table 7.

Quality of assessed measurement properties per patient-reported outcome measure (PROM) per study and the quality of the evidence per measurement properties

Modified Rankin Scale

The mRS is a global disability scale that has an ordinal 7-point scale ranging from no residual symptoms (score of 0) to severely disabled (score of 5) or dead (score of 6).²⁷ The mRS is an outcome measure that represents mobility and disability in daily and instrumental activities and living arrangements rather than cognitive and social functioning.²⁸ The mRS is an outlier within the included PROMs because it has an incorporated outcome of death; therefore, in essence it is not a PROM. A self-assessed mRS was used in the specific article included in this study, however, which could therefore be considered a PROM.

There is very low evidence for sufficient reliability between 2 different methods of mRS assessment used, namely telephonic assessments using a structured interview and self-reporting obtained via an online survey. The study showed a nonlinear distribution between the mRS scores, with 41%–43% of patients in the highest category and 4% of patients in the lowest category, indicating a significant ceiling effect.²⁰

Quality of Life after Brain Injury overall scale

The QOLIBRI-OS is a 6-item scale developed to determine the quality of life in patients after brain injury. It includes the domains physical condition, cognition, emotions, function in daily life, personal and social life, and current situation and future prospects.^29,30 The internal consistency measurement property for the QOLIBRI-OS was rated indeterminate with a very low quality of evidence. No additional information about interpretability was provided.

Subarachnoid Haemorrhage Outcome Tool

The SAHOT was recently developed as a disease-specific PROM for patients with spontaneous SAH.²⁶ The SAHOT was developed in collaboration with patients with aSAH. It consists of the domains cognition, general aspects of daily life, physical function, and behavioral and psychological function. Based on the COSMIN standards, the quality of the PROM development process was “inadequate,” since the PROM was adapted after the cognitive interviews took place and the adjusted items were not retested. In addition, the available information about the qualitative methods applied was insufficient. The evidence provided for the PROM development, hypothesis testing for construct validity, and responsiveness was of very low quality, which makes it impossible to draw conclusions about those measurement properties. The low quality of evidence for an indeterminate rating for structural validity and internal consistency was elicited by the lack of detail regarding the Rasch analysis and the absence of evidence for the unidimensionality of the scale. We foresee problems in interpreting the scores, as no direction of change (better or worse) is incorporated in the answer options (no change, some change, large or severe change). Some information about the interpretability is provided; however, this is partially derived from a simulation of a treatment trial and does not represent real-world data since the SAHOT has not been implemented in clinical practice yet. The distribution of the SAHOT has a wide range for each mRS category and all possible categories are scored. There were no floor or ceiling effects. The SAHOT comprises 56 items (table 4). The interpretation of the SAHOT is complex due to the Rasch-based interval analysis, which makes its use less feasible in daily practice. An ordinal scoring version was developed to solve this complexity, but its validity should be evaluated.

Stroke-Specific Quality of Life Scale

The SS-QoL is a disease-specific PROM that was developed to assess the quality of life in patients after stroke. It contains 49 items and comprises the 12 domains energy, family roles, language, mobility, personality, self-care, social roles, thinking, upper extremity use, vision, and work or productivity.³¹ No conclusions can be drawn regarding the internal consistency, structural validity, and reliability of the SS-QoL, since the overall quality of the evidence is very low. The studies include a Dutch version and a Chinese version of the SS-QoL. The principal component analysis revealed 2 components: physical and psychosocial. In the Dutch version, the physical component includes the following domains: self-care, mobility, upper extremity function, language, vision, and work. The psychosocial component includes thinking, family roles, social roles, personality, mood, and energy. In the Chinese version, language falls in the psychosocial component and the family roles and energy domains fall in the physical component. There is moderate-quality evidence for sufficient hypotheses testing. The responsiveness of the SS-QoL was rated insufficient, with very low evidence quality. The face validity was insufficient, since the answer categories did not fit with all the questions. The distribution of the results showed a ceiling effect for almost all of the 12 domains, except for thinking and energy. Almost none of the patients had the lowest possible score, indicating no floor effects. A ceiling effect of 15% was found for the physical component as a whole, but not for the psychosocial component or the total score on the SS-QoL. With regard to the feasibility of use, the time required for its completion was not described.

Short version of the SS-QoL

The short version of the SS-QoL was developed since completing the SS-QoL is time-consuming. It consists of the same 12 domains as the SS-QoL with only one question per domain, resulting in a total of 12 items.²¹ The internal consistency score was “doubtful” for this PROM, as no factor analysis was applied. This means that the evidence is of a very low quality and no conclusions can be drawn for the internal consistency of the short version of the SS-QoL. The quality of the evidence regarding a sufficient criterion validity of this PROM is high, with the long version considered the gold standard. As in the long version, there is moderate quality of evidence for the sufficient hypothesis testing for construct validity in the short SS-QoL. As with the long version, the face validity of the short SS-QoL was insufficient due to the mismatch of the answer categories with some questions.

Generic PROMs

The EQ-5D is a generic PROM comprising 5 domains: mobility, self-care, usual activities, pain or discomfort, and anxiety or depression. The last question is a visual analogue scale to determine a patient's self-rated general health.³² The HUI is health status classification system and consists of the domains sensation, mobility, emotions, cognition, self-care, pain, and fertility.³³ For the generic EQ-5D and HUI, the articles provided no information about the assessment of the measurement properties when using these measures for patients with aSAH. Only a little information regarding the interpretability was available. The visual analogue scores for the EQ-5D instrument were skewed toward higher scores.¹⁹

Discussion

In this systematic review, we found limited evidence concerning the assessment of the measurement properties of PROMs when evaluating the outcomes of patients with aSAH. The SS-QoL has been evaluated extensively in patients with aSAH. However, the majority of information regarding the measurement properties is incomplete and the evidence is of a very low quality. The SAHOT was the only PROM specifically developed for patients with aSAH, but the evidence for the measurement properties is of a very low quality. This means it is impossible to draw conclusions about the measurement properties of the SAHOT. None of the PROMs provided evidence for sufficient content validity, which is of utmost importance when using a PROM in clinical practice or research. This lack of assessment of content validity means that all these PROMs might measure items that are not relevant for patients with aSAH, or that items could be interpreted in a different way than originally intended. The current evidence base for selecting a suitable PROM for use with patients with aSAH is therefore insufficient. This result is backed by the results of a recent review about the involvement of patients in the development of PROMs, published in this journal.³⁴ We did not find any high-quality evidence to suggest that the measurement properties are inherently insufficient, however. Therefore, all identified PROMs have the potential to be used for patients with aSAH, but their quality should undergo further assessment. Future research should focus on the assessment of content validity and measurement errors to ensure that the PROM is suitable for use in individual patient care. Given the specific long-term consequences of an aSAH, we consider a disease-specific PROM to be the most appropriate; therefore, the SS-QoL, in long or in short version, seems the most suitable PROM for use with patients with aSAH at this time.

Search strategy and selection of studies and PROMs

Our systematic review identified a limited number of available validation studies. To a certain extent this is due to the inclusion and exclusion criteria we used. The most important reason for exclusion was that the results for patients with aSAH were not described separately from those of patients with other vascular diseases. An aSAH is a different entity from an ischemic stroke, hemorrhagic stroke, or unruptured aneurysm. Patients with aSAH are generally younger than patients who experience other types of stroke, which results in the higher interference of symptoms in their societal participation and social roles.³⁵ Despite this, these vascular diseases are often combined in validation studies of PROMs and conclusions are made for the whole group. We explicitly chose to acknowledge the different disease entities and to focus solely on patients with aSAH. Our search identified 6,976 unique articles on PROs and aSAH, of which just 9 articles were included in the final review. The huge reduction of eligible studies in the selection process was caused by the broad terms we used to cover all relevant PROMs, since no prespecified list of known PROMs was available.

COSMIN guidelines

Recently, COSMIN addressed the shortcomings in a former checklist by publishing the “Risk of Bias” checklist that was used in this review.¹⁴ Despite these improvements, some limitations can still be identified in this checklist. The detailed description of comparator instruments is often limited. The comparator instruments used in the included articles are often well known and have been validated for use with various disease entities, but often not for use with patients with aSAH; however, this does not necessarily mean that the methodologic quality of the study is low. In cases where a widely accepted questionnaire, such as the Short Form–36 or EQ-5D, was used as a comparator instrument, we decided to score this item as “adequate.” In articles using a questionnaire that is not yet widely accepted, such as the Brain Injury Community Rehabilitation outcome–39 or the Life Satisfaction questionnaire, we scored this item as “doubtful.”

Normally, the results of validation studies would be statistically pooled to estimate an average score for the measurement properties. Due to the identification of only one validation study for most PROMs, it was impossible to pool our results. Even for the SS-QoL, for which more studies were available, pooling the domains with an “inadequate” quality score would have resulted in a bias in the pooled results. Other systematic reviews proved that pooling results is inappropriate when different language versions of the PROM are included.^36,37 This is mainly caused by a nonformalized translation process or the lack of cross-cultural validation. In our review, a Dutch version and a Chinese version of the SS-QoL were included. In all 5 studies concerning the (short) SS-QoL, no assessment of cross-cultural validation was performed. We therefore decided not to pool the results for the SS-QoL.

Quality of measurement properties

A thorough validation process is important to ensure that the outcomes of a PROM can be relied upon for individual patient care. As mentioned earlier, the evaluation of content validity is vital for the interpretation of the patient outcomes, while reliability, which includes measurement error, is also an important feature. Measurement error is the proportion of the total variance caused by true differences between patients and not by a systematic or random error. In addition, interpretability supplies important information about assigning a qualitative meaning to the scores and teaches us how to interpret the score of an individual patient using a PROM. Only after the assessment of these properties can we say that the PROM properly measures its intended factors and that its outcomes can be used to derive clinical meaning. We expected that the investigated PROMs would at least fulfil these requirements; however, this review indicates that further research is required to accomplish this.

The studies included in this review often stated that an assessment of a certain measurement property was performed, when in fact another measurement property was assessed based on the definitions made by the COSMIN panel.¹⁵ For instance, when criterion validity was mentioned in several studies, actually hypotheses testing for construct validity was assessed. In the assessment of hypotheses testing and responsiveness, hypotheses about the expected correlations were lacking. This shortage in formulated hypotheses is a well-known problem when reviewing the measurement properties of validation studies^14,38; therefore, we formulated our own hypotheses for correlations when they were not reported in the included studies (table 3). Sometimes, not all of our hypotheses were met. In these cases, we discussed whether the correlation with the related constructs was higher than the correlation with unrelated constructs. Overall, this resulted in a sufficient rating for all assessments, except for the responsiveness of the SAHOT.²⁶ The authors of the article about the SAHOT formulated their own hypotheses with regard to construct validity, and these hypotheses were met; however, we identified some discrepancies in the specified hypotheses. We thought it remarkable to expect a high correlation between the results of the SAHOT and a questionnaire addressing cognitive and emotional problems, but a low correlation with a questionnaire measuring psychosocial functioning. Given that both outcomes are incorporated in the SAHOT and both are well-known health problems for patients after an aSAH, we would have expected a moderate correlation between the results of the SAHOT and both questionnaires. It may therefore be useful to formulate hypotheses for all correlations in a systematic review, regardless of the hypotheses reported in the studies.

Strengths

This is the first systematic review to evaluate the measurement properties of PROMs in patients with aSAH. We used all-embracing search terms that led to a large amount of included titles, which made the search very sensitive. On this ground, we expect to have found all important validation studies concerning PROMs in patients with aSAH. The use of the recently updated COSMIN checklist and the completion of this checklist by 2 independent raters is a strength of the present study. Moreover, in the case of disagreement, we reached a consensus on a final rating by debate and a thorough evaluation of the content and criteria. The reporting of our results is transparent and comprehensive. We illustrate this by presenting our complete search string and a priori formulated hypotheses.

Limitations

We slightly deviated from the registered study protocol by deleting our second search strategy for the use of PROMs in randomized controlled trials. In 2018, a systematic review was published that covered our second research question and provided an extended overview of the different outcome measures, including PROMs, used in trials.⁸ We believe that our decision to report only the first and main search strategy results in a more easily readable article while still answering our most important research question.

Second, we used the updated COSMIN “Risk of Bias” checklist for the assessment of the quality of the included studies. No research has yet been performed to evaluate the inter-rater agreement and reliability of this updated checklist. The percentage agreement on the original version of the COSMIN checklist was high on an item level, but the reliability, that is, the measure that indicates how well articles can be distinguished from each other based on the given score, was poor for many items.³⁹

Another limitation is that our methodology for article selection does not ensure that all PROMs used in literature are included. Due to the high number of studies, we used an initial screening of abstracts, which might have led to missed PROMs that were described in the full-text version of the article. In addition, evidence concerning aspects of the validity of a PROM can be deduced from studies in which the validation of this particular PROM is not the primary objective of the study. Nevertheless, it is not possible to identify all these studies with a systematic literature search without restrictions on the inclusion of PROMs; therefore, we chose not to include these sorts of articles in our systematic review.

There are limited validation studies concerning the measurement properties of PROMs used in patients with aSAH. At least half of the information on the measurement properties is not assessed or the level of evidence is very low. All PROMs identified in this review have the potential to be used for patients with aSAH, but lack a thorough evaluation of the content validity. This means that no conclusions can be drawn about the validity and reliability of the current PROMs used for patients with aSAH. Future research should focus on the assessment of content validity and measurement error to judge the suitability of a PROM for use in individual patient care. Given the specific long-term consequences of aSAH, we consider a disease-specific PROM the most appropriate choice. Our review suggests that the SS-QoL, in long or in short version, is the most suitable PROM currently available.

Glossary

aSAH

aneurysmal subarachnoid hemorrhage

EQ-5D

EuroQol 5D

HUI

Health Utility Index

mRS

modified Rankin Scale

PRO

patient-reported outcome

PROM

patient-reported outcome measure

QOLIBRI-OS

Quality of Life after Brain Injury overall scale

SAHOT

Subarachnoid Haemorrhage Outcome Tool

SS-QoL

Stroke-Specific Quality of Life scale

Appendix. Authors

Study funding

No targeted funding reported. The Article Processing Charge was funded by the Radboud University Medical Center.

Disclosure

The authors report no disclosures relevant to the manuscript. Go to Neurology.org/N for full disclosures.