Advances in Outcomes Measurement in Rehabilitation Medicine: Current Initiatives from the National Institutes of Health and the National Institute on Disability and Rehabilitation Research

Abstract

The articles in the current supplement present recent advances in measurement of patient-reported health related quality of life (HRQOL) outcomes. Specifically, these articles highlight combined efforts of the National Institutes of Health (NIH), National Institute for Neurological Disorders and Stroke (NINDS), National Center on Medical Rehabilitation Research (NCMRR), National Institute on Disability and Rehabilitation Research (NIDRR) and Department of Veterans Affairs Rehabilitation Research and Development Service (VA RR&D) to improve HRQOL measurement. Further, this supplement is intended to provide rehabilitation professionals with information about these efforts and the implications that these advances in outcomes measurement have in rehabilitation medicine and clinical practice. These new measurement scales utilize state-of-the-art methodology techniques including item response theory (IRT) and computerized adaptive testing (CAT). In addition, scale development involves both qualitative and quantitative methods, as well as the administration of items to hundreds or even thousands, of research participants. The scales have been deliberately built with overlap of items between scales so that linkages and equivalency scores can be computed. Ultimately, these scales should facilitate direct comparison of outcomes instruments across studies and will serve as standard data elements across research trials without compromising the specificity of disease- or condition-targeted measures. This supplement includes the initial publications for many of these new measurement initiatives, each of which provides researchers and clinicians with better tools for evaluation of the efficacy of their interventions.

In a recent American Congress of Rehabilitation Medicine (ACRM) Supplement to the Archives of Physical Medicine and Rehabilitation (PM&R), Bagiella pointed out that a single primary outcome measure is not sufficient; outcomes measures must capture multiple dimensions of functioning in order to accurately evaluate the clinical interventions designed to target more than one area of recovery.¹ Unfortunately, the outcome measures that are most frequently used in rehabilitation are not multifaceted; traditionally, they have focused primarily on single domains of activity limitations, functioning improvement,² or global health status, failing to include numerous domains of life that are salient to most individuals.³ Moreover, by limiting assessment to global, single primary variables, it is impossible to capture and evaluate the full extent of treatment effects of clinical trials in rehabilitation medicine.¹ As a result, there has been increasing acknowledgement of the need for multidimensional assessment of quality of life (i.e., health-related quality of life or HRQOL), using patient reported outcomes (PROs).^4,5 HRQOL includes physical health, level of social support, participation in the community, and emotional functioning,⁶ and has become increasingly important for informing treatment decisions that may affect length of survival, functional status, or pain and symptom management.^4,7–12

While HRQOL assessment is gaining momentum, rehabilitation outcomes researchers have tended to use generic quality of life measures developed for use in the general population. Existing commonly-used measures in this area (e.g., SF-36) typically do not assess a diverse array of functioning, nor are they sensitive for rehabilitation populations.^13,14 In fact, the scarcity of appropriate rehabilitation-specific measurement tools hinders the efforts of rehabilitation researchers whose goal is to obtain valid and useful data to evaluate new treatment approaches. The general measures that are typically utilized lack the sensitivity needed to detect meaningful, population-specific differences in the course of recovery and often contain irrelevant material and omit issues that are important to individuals with disabilities. Specifically, most HRQOL measures lack sensitivity to detect important change in rehabilitation clinical trials, are not consistently used across studies (making it difficult to compare results), and do not include the targeted items needed to sensitively assess disease- or disability-specific changes in functioning (e.g., those who have experienced a traumatic, life-altering injury).

As a result, the topic of outcomes measurement in rehabilitation medicine has received significant attention over the last decade from both ACRM and the Archives of Physical Medicine and Rehabilitation. From 2000 – 2010, four previous ACRM Supplement issues have focused on the need for improved outcomes measures in the field of rehabilitation medicine.^14–17 In addition, in 2002 the National Institutes of Health (NIH) developed a “Roadmap” for medical research to address the lack of measurement systems that allow for the dynamic assessment of PROs for individuals with chronic disease. Similarly, the National Institute for Neurological Disorders and Stroke (NINDS) and National Center on Medical Rehabilitation Research (NCMRR) also recognized the lack of a uniform HRQOL assessment in neurological research as a priority for clinical neurology and rehabilitation research. Further, the National Institute on Disability and Rehabilitation Research (NIDRR) and the Department of Veterans Affairs Rehabilitation Research and Development (VA RR&D) also highlighted PRO assessment as being a particularly important part of overall rehabilitation assessment. Together, these funding agencies have developed a series of measurement studies designed to reengineer the research process.

Such measurement studies were intended to design a measurement system that would provide unity across clinical studies and clinical populations by giving investigators and study staff a “universal language” to utilize within studies. Common data elements (CDEs) initiatives have been prioritized as a method by which the standardized collection of variables and other investigational data would occur across studies, allowing investigators to compare and contrast findings across studies using a common metric.¹⁸ The current Supplement describes new advances in measurement and outlines the development of a complementary series of outcomes measures that can be used across disability groups as well as chronic disease and clinical populations. These new inter-related PRO measurement systems provide a uniform platform for common data collection of several domains of functioning.

The measurement systems described throughout this Supplement all utilize the same qualitative and quantitative measurement development methodology. These measurement systems were developed with input from individuals with disabilities at several key steps in the research process. This disability- and disease-specific input was obtained via focus groups and one-to-one interviews with participants. This methodology helped ensure that the ultimate measurement scale would meet the needs of the rehabilitation populations and would address the need described above for HRQOL measurement that is directly informed by individuals with disabilities.

The measurement systems all use probabilistic measurement models to build item banks. Item banking methodology involves field testing newly developed item pools across hundreds, or even thousands, of individuals. This approach allows investigators to implement of state-of-the-art statistical techniques such as item response theory (IRT) in order to develop a final calibrated item pool, otherwise known as an item bank. The final scales have been, or will be, developed as computerized adaptive tests (CAT) and fixed short forms. A CAT, or “smart test,” where each individual item is selected based on the response to the previous item, allows clinicians and researchers to ascertain a person’s level of functioning using only a minimal number of items and without losing the precision of a longer measure. Similarly, a fixed short form provides an estimate of functioning by using a fixed set of items (typically 4–12 items at most) that have the highest ability to discriminate among participants within the sample.

Item banking methodology also allows these new measurement systems to be interpreted using a common metric (i.e., transformation to a t score to allow for cross-disease and cross-study comparison). In addition, this approach allows item banks to be linked and co-calibrated (using IRT) among themselves and/or with other legacy measures that are typically administered in that field. In this manner, investigators can administer one instrument and receive an “equivalency” score on another instrument, allowing direct comparison of the outcome measure between different studies. Such linking, or use of CDEs, allows researchers to conduct cross-study and cross-disease comparison without compromising the ability for each individual measure to provide disease-specific information. Ultimately, these outcome measurement systems provide a significant advantage to more traditional measurement systems that require a tradeoff between the disease-specific and cross-disease comparison. These new measurement systems also allow for standardization across clinical trials and intervention studies.

Below, a brief summary of these measurement systems is provided. Each important article contained within this special issue is discussed and the relevance of these new measurement systems to rehabilitation medicine is highlighted. Eight articles are included in this Supplement; each article describes a component of the new measurement systems described above and highlights the efforts to address the measurement problems outlined in previous Archives of PM&R Supplements.^14–16 This Supplement includes descriptions of how four new complementary measurement systems (PROMIS, Neuro-QOL, SCI-QOL, and TBI-QOL) have been developed utilizing feedback from individuals with disabilities and neurological conditions. The authors present how these new instruments have been developed using state-of-the-art techniques and advanced technology in both qualitative and psychometric approaches (i.e., IRT and CAT). Further, this Supplement highlights how these new measurement systems are all linked through common items (i.e., CDEs), provide a common platform across instruments, and will ultimately be interpreted according to the same metric. The target audience for this Supplement is rehabilitation researchers and clinicians who conduct assessments of health status outcomes and those researchers who perform rehabilitation intervention trials.

This Supplement begins with an overview regarding the inception and implementation of the NIH Roadmap (developed under the direction of the previous NIH Director Elias A. Zerhouni).¹⁹ Louis Quatrano and Theresa Cruz,²⁰ program officers at the Eunice Kennedy Shriver National Institute of Child Health & Human Development/National Center on Medical Rehabilitation Research, highlight the primary aims of the Roadmap initiative from the funders’ perspective. They discuss how the Roadmap is a collaborative effort among 27 NIH institutes and centers designed to identify and target research towards correcting gaps in the way that biomedical research is conducted. This large collaborative network of NIH Institutes and Centers increases the breadth of research initiatives in a way that no single NIH institute could do alone. Quatrano and Cruz discuss how one of the Roadmap themes was to “Re-engineer the Clinical Research Enterprise” by providing support to develop a uniform outcome measurement system. Further, they provide an introduction to the Patient Reported Outcome Measurement Information System (PROMIS). The PROMIS-I Network (funded through the Roadmap initiative) was comprised of a Statistical and Coordinating Center, six clinical research sites, and NIH scientific program officers from multiple institutes throughout the NIH. Now in its second funding cycle, the PROMIS-II Network (funded through the NIH Common Fund) has been expanded to include three network centers (a statistical, technology, and network center) and twelve clinical research sites (see Figure 1). This article concludes with a summary of the anticipated impact that PROMIS will have on the health-care profession and discusses the implications that this type of measurement system will have for rehabilitation medicine.

Fig 1.

Quatrano and Cruz’s compelling article is followed by an article that highlights the application of PROMIS to clinical rehabilitation. Specifically, Dagmar Amtmann, Karon Cook, Kurt Johnson (from the PROMIS-I Clinical Research Site) and David Cella²¹ provide a detailed description of the PROMIS item banks and an explanation of how individuals with spinal cord injury (SCI) and multiple sclerosis (MS) informed the development of pain, physical function, and fatigue items. They provide a description of the item writing/refinement procedures (i.e., cognitive debriefing interviews) that were informed by individuals with SCI and MS. They then discuss item finalization and field testing, including findings across a wide variety of disease conditions (again including SCI and MS), as well as the process for selecting items for fixed short forms. This article also describes the process that was used to validate the newly developed fixed short forms in individuals with disabilities (including SCI and MS). Amtmann and colleagues’ article shows how some key PROMIS item bank development was informed directly by rehabilitation populations.

The fourth article²² by Jin-Shei Lai describes item banking methodology and highlights its advantages while also detailing how CAT administration improves the reliability of the outcome measurement especially when the population has a wide variety of functioning. A key feature of PROMIS is that it is comprised of a series of calibrated item banks using state-of-the-art statistical analyses (i.e., IRT). An item bank differs from a traditional “pool” of items because it has been rigorously tested using data from a large calibration sample. Data from the calibration sample is utilized to determine parameters of difficulty and discriminability for each item, which are then used to guide test administration and scoring. The resulting items form a calibrated bank of items (aka, an item bank) where items can be positioned along a continuum of functioning on the construct that is being assessed. This construct is thought to be the underlying “latent trait” that the test is attempting to measure. The calibrated item bank can then be administered as a computerized adaptive test (CAT) where each individual item is selected based on the response to the previous item. This is possible because as each item is administered to a participant, the responses are used to predict the examinee’s standing or position on the latent trait. Thus, each respective item that is selected is chosen because it should best reduce measurement error and obtain the most information about a person’s standing on the latent trait. The advantage of this procedure is that an estimation of a person’s score is calculated after each item administered and, with each successive item, the score estimate becomes more precise as measurement error is reduced. As an alternative, a pre-selected fixed short-form can be utilized for assessment. This approach also utilizes fewer items and consists of those items that provide the most discriminability in a given population.

In their article, Lai and colleagues provide a technical overview of the statistical approach utilized in developing PROMIS CATs and short forms, giving a detailed example describing the development of the PROMIS Fatigue scale.²² The authors compare the precision of a 7-item CAT administration with 3 alternative short forms that were developed from the Fatigue item bank. They demonstrate score reliability at different levels of functioning and provide evidence for reliable measurement of fatigue using very few items. This discussion of IRT, item banking methodology, CAT, and short form administration is central to not only the PROMIS Fatigue Scale, but to all of the item banks and measurement systems presented in this Supplement.

This article is followed by the description of a related measurement initiative sponsored by the National Institute for Neurological Disorders and Stroke – the Neurology Quality of Life (Neuro-QOL) initiative. This article, authored by David Cella, Cindy Nowinski, Amy Peterson, David Victorson, Deborah Miller, Jin-Shei Lai, and the Neuro-QOL NIH Program Officer, Claudia Moy,²³ introduces this new measurement system and provide a full overview of the development of the Neuro-QOL. Like PROMIS, the Neuro-QOL developed item banks across a wide variety of symptoms and areas of functioning specific for use in five adult neurological disorders (stroke, multiple sclerosis, Parkinson’s disease, epilepsy, and amyotrophic lateral sclerosis) and two childhood neurological conditions (pediatric epilepsy and muscular dystrophies).

Neuro-QOL utilized the same qualitative and quantitative psychometric procedures as PROMIS, including the development of both generic and disease-specific item banks that could be administered by CAT or short form. The difference between PROMIS and Neuro-QOL is that the former was designed to be a more generic measurement system while the latter was designed to be a more targeted measurement system for individuals with neurological disorders. Several of the Neuro-QOL instruments contain PROMIS items presented verbatim, providing a link between the two measurement systems. The relationship between the PROMIS and Neuro-QOL is shown in Figure 2. The manuscript reviews the selection of targeted neurological conditions covered by the Neuro-QOL as well as the selection of specific HRQOL domains and subdomains for item bank development. In addition, this article highlights the development of items pools, the refinement of these item pools using cognitive debriefing interviews, and the adaptation and translation of these item pools into Spanish. Like the PROMIS, individuals with neurological disorders and/or disabilities participated in the domain selection (through focus groups) and the item development/refinement phase (through cognitive debriefing interviews). Further, the authors review the process for field testing and calibrating item pools (using IRT) to develop item banks and discuss the construction of short forms. Finally, this article includes a brief discussion of the incorporation of items from the Activity Measure for Post-Acute Care (AM-PAC) within this scale (described in more detail below).

Fig 2.

The sixth manuscript in this Supplement expands upon Cella and colleagues’ article by explaining how the AM-PAC and Neuro-QOL scales were linked. Specifically, Drs. Stephen Haley, Wendy Coster, Alan Jette, and Pengsheng Ni, along with several colleagues,²⁴ highlight the statistical approach linking the AM-PAC Mobility subscale to the Neuro-QOL Mobility item bank, and the AM-PAC ADL subscale to the Neuro-QOL Upper Extremities and ADLs item bank. They discuss how scores from these measures were placed on a common metric such that a direct estimation of AM-PAC scores could be made from the Neuro-QOL scores and vice versa. This manuscript demonstrates how this new measurement system allows researchers to obtain direct comparison of results regardless of which measure they use (i.e., AM-PAC or Neuro-QOL). It highlights how a targeted measurement can be utilized to estimate a more generic score and explains how such linking is relevant. This type of linking is also inherent between PROMIS and Neuro-QOL, as well as in the two measurement systems described in articles below.

The final two manuscripts in this Supplement present the initial steps of expanding both the PROMIS and the Neuro-QOL systems to include HRQOL assessment that is specific to SCI and traumatic brain injury (TBI), respectively. Although the Neuro-QOL considered both of these conditions for measurement development, funding restrictions limited them to focusing on only the five adult neurological disorders described above (SCI was initially ranked 6^th and TBI ranked 8^th). This was unfortunate, given that people who have experienced such traumatic, life altering injuries confront a number of unique factors that need to be considered when evaluating HRQOL.^14,15 As such, additional funding was sought to extend the Neuro-QOL to SCI and TBI (see Figure 3).

Fig 3.

Specifically, the seventh article examines the initial steps to develop a targeted SCI HRQOL measurement system. In this article, Dr. Tulsky and colleagues²⁵ explores findings from 12 focus groups with individuals with SCI and 4 focus groups with SCI clinicians/treatment providers. The authors provide an in-depth report of the specific SCI-targeted issues that were discussed and highlight the need for an SCI-specific HRQOL assessment to expand upon the current PROMIS and Neuro-QOL measurement frameworks. These data set the initial groundwork for the SCI-QOL scale development.

Similarly, the eighth article summarizes findings from two separate studies examining HRQOL in TBI. Noelle Carlozzi, David Tulsky, and Pamela Kisala²⁶ explore findings from semi-structured interviews examining HRQOL in individuals with TBI. The authors also summarize findings from 7 focus groups with individuals with TBI, 4 focus groups with caregivers of individuals with TBI, and 2 provider/clinician focus groups. This article provides an in-depth report of the specific TBI-targeted issues that were raised as a result of these focus groups and highlights the need for TBI-specific HRQOL assessment to expand the current PROMIS and Neuro-QOL measurement framework. Similarly to article seven in this Supplement, these data set the initial groundwork for the TBI-QOL scale development.

This Supplement provides an in-depth look at several new measurement systems that are designed to reengineer the way that outcomes assessment is conducted in rehabilitation research. This collection of articles highlights the development of more sophisticated, quantitative measures of HRQOL specific for use in rehabilitation medicine. Together, the PROMIS, Neuro-QOL, SCI-QOL and TBI-QOL all utilize state-of-the-art item banking methodology that allows for measurement systems that can provide multidimensional assessment using only a minimal number of items. These systems are developed using a common metric, so while they include several disease-/disability-specific items, they also include more generic items that allow for cross-disease/-disability comparison. Further, these systems offer the advantage for new content to be added at a later date, while sacrificing the ability for longitudinal comparison (by maintaining a common metric). Finally, and perhaps most importantly, this Supplement highlights the first of what we expect to be several measurement initiatives that were designed specifically for use in rehabilitation populations. This type of approach captures the multifaceted and unique problems that are experienced by rehabilitation medicine populations, issues that are often overlooked by more generic HRQOL assessment. Taken together, such measurement systems provide researchers and clinicians with better tools for evaluation of the efficacy of their interventions.

List of Abbreviations

ACRM

American Congress of Rehabilitation Medicine

ADL

Activity of daily living

AM-PAC

Activity Measure for Post-Acute Care

CAT

Computerized Adaptive Test

CDEs

Common Data Elements

HRQOL

Health-Related Quality of Life

IRT

Item Response Theory

MS

Multiple Sclerosis

Neuro-QOL

Quality of Life for Neurological Disorders [measurement system]

NCMRR

National Center on Medical Rehabilitation Research

NIDRR

National Institute on Disability and Rehabilitation Research

NIH

National Institutes of Health

NINDS

National Institute of Neurological Disorders and Stroke

PM&R

Physical Medicine and Rehabilitation

PRO

Patient Reported Outcome

PROMIS

Patient Reported Outcomes Measurement Information System

SCI

Spinal Cord Injury

SCI-CAT

Computerized Adaptive Measure of Functional Activities/Activity Limitation in SCI

SCI-QOL

Spinal Cord Injury Quality of Life Measurement System

SF-36

Medical Outcomes Study 36-Item Short Form

TBI

Traumatic Brain Injury

TBI-QOL

Traumatic Brain Injury Quality of Life Measurement System

VA RR&D

Department of Veterans Affairs Rehabilitation Research and Development