Use of Patient-Reported Outcomes in the Context of Different Levels of Data

Abstract

There is increasing interest in measuring patient-reported outcomes as part of routine medical practice, particularly in fields like total joint replacement surgery, where pain relief, satisfaction, function, and health-related quality of life, as perceived by the patient, are primary outcomes. We review some well-known outcome instruments, measurement issues, and early experiences with large-scale collection of patient-reported outcome measures in joint registries. The patient-reported outcome measures are reviewed in the context of multidimensional outcome assessment that includes the traditional clinical outcome parameters as well as disease-specific and general patient-reported outcome measures.

Update

This article was updated on January 25, 2012, because of a previous error. In the EQ-5D subsection of the Methods section on page 68, the sentence that had previously read “This, in addition to limited responsiveness for some conditions, has to be weighed against the inherent low response rate of the instrument.” now reads “This, in addition to limited responsiveness for some conditions, has to be weighed against the inherent low respondent burden of the instrument.”

Why Patient-Reported Outcome Measures (PROMs) Data Are Needed

National registry data play a crucial role in monitoring, developing, and improving total joint replacement (TJR) surgery¹. The increasing life span of the patients, along with medical advances and improved implant technology that now allows joint replacement in both younger and older patients, has led to an increasing population of patients needing TJR. The rapid increase in the burden of musculoskeletal diseases in the past few decades is expected to continue^2,3. One fundamental prerequisite for understanding how to meet the changes in the demands for TJR is to measure outcomes from the patient's perspective. Despite a lack of explicit criteria for hip and knee replacements, pain and affected health-related quality of life are the essential factors in the decision regarding surgical treatment. Using implant revision status as a surrogate measure of functional outcome might be inappropriate, as the patient satisfaction rates following both total knee replacement (TKR) (80%) and total hip replacement (THR) (90%) are both lower than their respective implant survival rates^4,5. Thus, omitting patient-reported outcomes precludes us from having a full understanding of the factors that contribute to pain relief, restoration of function, and patient satisfaction. For these reasons, PROMs are becoming increasingly important in the allocation of health-care resources and the provision of guidelines for optimum care and management⁶. Finally, there is a growing recognition and interest in the U.S. in PROMs as important outcomes in designing comparative effectiveness studies since the founding of the Patient-Centered Outcomes Research Institute (PCORI) by Congress through the 2010 Patient Protection and Affordable Care Act⁷.

Levels of Data

The requirements for initiating a new register are: (1) commitment by the profession to support the register and to collect data, (2) a strong core team for developing the register, and (3) a restricted number of data items included at start-up. The amount and types of data collected can be divided into different categories, or levels, but as yet there is no consensus on the specific data elements included in each category. International registry collaborations, such as the International Consortium of Orthopedic Registries (ICOR), the International Society of Arthroplasty Registries (ISAR), and the Nordic Arthroplasty Register Association (NARA) strive to reach agreement on a general classification scheme for orthopaedic registries. A crude-level classification of registry data is presented in Table I. Data could be divided into those that were patient related and those that were procedure related. The first level of data includes a limited set of basic data that are recommended as the minimum variables for a TJR registry to collect⁸. This set includes items to identify the patient, the surgeon, the hospital, the procedure, and the implant. The second level of data comprises more detailed characteristics, such as patient comorbidities, the use of antibiotics, and technical surgical details. The third level would include PROMs and socioeconomic variables. A fourth level of registry data has been proposed to include imaging diagnostics (radiographic) results⁹.

TABLE I.

Levels of Data

	Level-I Data	Level-II Data	Level-III Data
Patient-related data	Personal identification	Comorbidities	Patient-reported outcome
	Sex	Surgical complications	Socioeconomic data
	Laterality	Height and weight	Any adverse events
	Diagnosis
	Death
Procedure-related data	Date of surgery	Prophylactic measures	Costs
	Type of procedure	Surgical measures (technique, approach, fixation method, and timing)
	Implant information
	Hospital identification
	Surgeon identification
	Reoperation and/or revision

Objective

The objective of this review is to present the argument for including PROMs in joint registries and incorporating these measures in outcome assessment. We provide examples of experiences of collecting PROM data in some national and local joint registries. We also discuss different generic and condition-specific PROMs used (Table II) and present a brief overview of the logistics for conducting longitudinal collection of questionnaires. Finally, we discuss the benefits, challenges, and difficulties in large-scale capture and analyses of PROMs.

TABLE II.

Examples of National and Regional Registries Collecting PROM Data^*

	National Registries			Regional Registries
	SHAR	NZJR	NJR	CJRR	WRHA	CERT
Generic measures	EQ-5D	EQ-5D	EQ-5D	SF-12		SF-36, EQ-5D
Condition-specific measures	Pain (VAS), Satisfaction (VAS)	OHS/OKS	OHS/OKS, Satisfaction (Likert scale)	WOMAC, UCLA Activity Score	OHS/OKS, Satisfaction (Likert scale)	HOOS/KOOS, Pain (VAS), LEAS
Intervals	Preop, 1 yr, 6 yr, and 10 yr	6 mo, then every 5 yr	Preop, 6 mo	Preop, 6 mo, 1 yr, and 5 yr	Preop, 1 yr	Preop, 6 mo, 1 yr, 2 yr, and 5 yr
Response rate	86% to 90%	72% to 75%	69% to 75%	70%	approx. 75%	approx. 80%

^*SHAR = Swedish Hip Arthroplasty Register; NZJR = New Zealand Joint Registry; NJR = National Joint Registry for England and Wales; CJRR = California Joint Replacement Registry; WRHA = Winnipeg Regional Health Authority Joint Replacement Registry; CERT = Center for Education and Research on Therapeutics Registry; EQ-5D = European quality-of-life questionnaire-five dimensions; SF-12 = Short Form-12, a twelve-item short-form health survey; SF-36 = Short Form-36, a thirty-six-item short-form health survey; VAS = visual analog scale; OHS = Oxford hip score; OKS = Oxford knee score; WOMAC = Western Ontario and McMaster Universities Osteoarthritis Index; UCLA = University of California, Los Angeles; HOOS = Hip Disability and Osteoarthritis Outcome Score; KOOS = Knee Injury and Osteoarthritis Outcome Score; and LEAS = Lower Extremity Activity Scale.

Methods

Patient-Reported Outcome Measures (PROMs)

There is a wide range of PROM instruments, with several considerations for selecting appropriate PROMs.

• (1) The PROM instruments must meet basic methodological requirements for validity, reliability, and responsiveness. They should accurately measure the intended characteristic and detect change when it occurs.

• (2) The set of outcome instruments should ideally include generic and condition-specific PROMs to assess both a person's state of overall health and how the condition affects the functioning of the relevant body part or organ system. It is particularly important to include a generic instrument so that the results can be compared across different patient and population groups. Generic PROMs also permit comparisons of outcomes between different medical or surgical treatments.

• (3) The number of questions in the survey must be reasonable to provide a high patient-response rate.

• (4) Aside from measuring general health or specific functions, impairment, disability, and pain, there are other subjective measures to explore. For example, there are PROMs regarding patient expectations and the degree of participation that provide useful information.

PROMs should not be confused with patient-reported experience measures. Patient-reported experience measures may provide useful indications of patients’ perspective on their care. By their nature, patient-reported experience measures reflect experience of the process rather than the outcome of care. Measures of experience are particularly useful for local quality improvement work but are not suitable for use in a national quality registry.

Frequently Used Generic and Condition-Specific PROMs

Visual Analog Scales

A visual analog scale (VAS) is a psychometric instrument that could be used in different contexts, both generic and condition specific (e.g., general health status, pain, satisfaction). The orthodox VAS is a 100-mm-long horizontal line with two diametrically opposed states at each end of the line. This gives a 101-point scale, where each millimeter represents one point. However, because VAS scales are used in many different settings (i.e., pen-and-paper forms, touch screens, and Internet questionnaires), it is not always possible to render them as a 100-mm line in every circumstance, depending on screens and software issues, and thus the scale may need to be rendered as a line that is shorter or longer in length than the usual 100 mm. Also, older patients tend to have difficulties understanding the traditional VAS. This has resulted in a large variety of nonorthodox VAS instruments with different lengths and sometimes supplied with subscale indicators. In order to compare VAS scores that have been measured with different methods or designs, it is imperative that the instruments have been tested for reliability.

Generic PROM Instruments

SF-36, SF-12, and SF-6D

The most frequently used generic PROM instrument in clinical trials is the thirty-six-item short-form health survey (the Short Form-36, or SF-36)¹⁰. It contains thirty-six items and measures eight domains of health: physical functioning, role limitations due to physical health, bodily pain, general health perceptions, vitality, social functioning, role limitations due to emotional problems, and mental health. The SF-36 yields a score for each of these domains, provides summary scores for both physical and mental health, and is a single health utility index.

The SF-36 health index cannot be used for health-economic cost-utility analysis. The length of the questionnaire is an obvious disadvantage when used in large-scale observational studies because it jeopardizes the quality of data, given the high risk of insufficient response rates and missing values. Accordingly, there is a shortened version (the SF-12) that provides only summary scores for the physical and mental components. The reduced six-item short form (SF-6D) is a preference-based scoring system that generates a single index, allowing the analyst to use it in cost-utility analysis¹¹. A final limiting factor for the SF-12 and SF-36 is that licensing fees may be required to use these questionnaires, and such fees would limit the usefulness of this tool for large-scale registries that are collecting longitudinal data.

The EQ-5D

The European quality-of-life questionnaire-five dimensions (EQ-5D) (EuroQol Group, Rotterdam, The Netherlands) is another generic instrument¹². It is standardized for use as a measure of health outcome and has been translated according to standardized guidelines into more than 100 languages. The EQ-5D evaluates patients in five dimensions: mobility, self-care, usual activities, pain or discomfort, and anxiety or depression. Each dimension is divided into three levels of severity, thus generating 243 possible combinations of response. The EQ-5D is presented as a health profile or as a global health index with a weighted total value. There are different valuation sets based on populations in different countries to adjust for cultural differences. The questionnaire also includes a vertical EQ-VAS, which assesses global health. The EQ-5D has a pronounced ceiling effect at the time of follow-ups, with high proportions of the TJR populations reporting the highest possible health state, which limits the possibilities to differentiate between good and excellent outcomes. This, in addition to limited responsiveness for some conditions, has to be weighed against the inherent low respondent burden of the instrument. Ceiling effects can be avoided using the newly developed EQ-5D-5L, which has five levels of graded severity instead of the original three¹³.

Condition-Specific PROMs

WOMAC, HOOS, and KOOS

The Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) was originally introduced in 1982 and has since undergone several revisions and modifications¹⁴. This condition-specific instrument is self-administered and assesses pain, disability, and joint stiffness in knee and hip osteoarthritis using a battery of twenty-four questions. The latest version of the instrument (WOMAC 3.1) is available in sixty-five alternate language forms, and in most languages is available in both five-point Likert and VAS formats. It is a valid, reliable, and responsive measure of outcome in patients with arthritis, and has been used in diverse clinical and interventional environments. Expanded from WOMAC, the Knee Injury and Osteoarthritis Outcome Score (KOOS) and the Hip Disability and Osteoarthritis Outcome Score (HOOS) each consist of five subscales: pain, other symptoms, function in daily living, function in sport and recreation, and knee or hip-related quality of life^15,16. Validity for KOOS and HOOS has been determined in comparison with the SF-36. Although the WOMAC questions are available in the public domain through the KOOS and HOOS tools, licensing fees may be required to use the WOMAC tool directly, which limits the usefulness of this tool for large-scale registries collecting longitudinal data. A limiting factor of the KOOS and HOOS tools in particular, but also of WOMAC, is the length of the questionnaires. For that reason, there are short forms available for KOOS and HOOS that assess the physical function domains in seven and five questions, respectively.

Oxford Hip and Knee Scores

The Oxford hip score (OHS) and the Oxford knee score (OKS) each are twelve-item, condition-specific, patient-based questionnaires developed and validated specifically to assess function and pain after total hip replacement^17,18 and total knee replacement, respectively. They were developed from patient interviews and validated against the generic SF-36. They have demonstrably reliable and responsive measurement properties for assessing outcomes of hip and knee TJR. The Oxford scores have simple scoring and summing system that provide an overall scale for assessing outcome of TJR. They are, in combination with the EQ-5D, used in the British PROMs program recently initialized by the National Health Service (NHS)⁶. They are also used in the New Zealand registry¹⁹.

How to Collect PROMs

Traditionally, pen-and-paper questionnaires have been used and data were collected, recorded, and computerized manually. The administration of pen-and-paper forms is not only time-consuming and costly but also introduces the risk of errors when entering data. Another disadvantage of pen-and-paper questionnaires is that missing values often compromise data quality. The development of information technology and software along with an exponential increase in the use of the Internet now allows new modalities for collecting PROMs, and these methods need to be developed and compared with traditional methods. However, use of the Internet for a patient-administered questionnaire does not give a sufficient response rate in the total hip replacement population to replace the pen-and-paper questionnaire. The system has been shown to be reliable and could be used at present for measuring patient-reported outcomes if supplemented with traditional pen-and-paper questionnaires for Internet nonrespondents²⁰. Also, the use of the Internet for patient-reported outcome data collection requires additional data security protection and precautions related to the sharing of protected health information.

New Analytic Methodologies

Recently, Rasch analytic methodology has been proposed to overcome the limitations of Likert-based scoring of questionnaires that are used in orthopaedics²¹. Even advanced functioning and quality measurement instruments, such as SF-36 and EQ-5D, have some limitations. The scores based on these questionnaires are often duplicative and have strong ceiling effects, and they may not able to capture the change in healthier patients. Customizing items to suit all patients is often needed and can be achieved with Rasch analysis. The analysis involves building a list of items (i.e., a bank), that includes substantially more items than would be included in a single questionnaire. All of these items are calibrated for difficulty. The items that do not target the latent trait that is being measured are discarded. For example, in a study of anterior cruciate ligament injury, many items in the KOOS score were not relevant and a new questionnaire based on relevant items was considered after Rasch analysis²². In Rasch analysis, once all items are calibrated onto a single scale they can be used for measurements. Because all items are calibrated into a single scale, the items that measure concepts applicable to both “healthy” patients and patients with severe comorbidity are meaningful in the same way. There might also be theoretical advantages, such as improving the precision over standard Likert scoring in discriminating between groups of patients in terms of outcome assessment.

National Collection of PROMs

The Swedish Hip Arthroplasty Register

The Swedish Hip Arthroplasty Register was initiated in 1979^23,24. All public as well as private orthopaedic units in Sweden that perform hip replacements participate on a voluntary basis. In addition to the information included in the personal identity number (date of birth and sex), individual data on diagnoses, laterality, and detailed information on implants and fixation are reported. To complement level-I and level-II data, a standardized protocol including PROMs was gradually introduced in Sweden starting in 2002. All patients are asked to complete a self-administered ten-item questionnaire that included questions to help in assessing Charnley class^25,26, a pain VAS, and the EQ-5D. This is done preoperatively and, unless the patient has undergone revision arthroplasty surgery, is repeated at one, six, and ten years postoperatively²⁷. A question addressing satisfaction with the result of the operation (on a VAS) is added on the follow-up questionnaires.

Preoperatively, patients complete the questionnaire either via a touch-screen application or by means of conventional pen-and-paper questionnaire at the hospital clinic. At the three follow-up times, the questionnaire is mailed to the patient with a cover letter and inclusion of a stamped, addressed, return envelope. Nonrespondents receive the first and only reminder after eight weeks. The response rate has been between 86% and 90%⁵.

National Joint Registry for England and Wales

Since 2009, the National Joint Registry for England and Wales (NJR) has collaborated with the NHS in collecting questionnaires in the largest national PROMs program of its kind. Data are combined with the Hospital Episode Statistics (HES). The program comprises the EQ-5D and the OHS or OKS on the preoperative questionnaire, and questions about the overall improvement and satisfaction with the result of the operation are added to the follow-up questionnaire given at six months. The response rate has been between 69% and 75%.

The New Zealand Joint Registry

Since its inception in 1998, the New Zealand Joint Registry (NZJR) has used the OHS and the OKS as its condition-specific PROMs. Due to compliance issues, the follow-up program has been confined to postoperative measurement. Questionnaires are sent out at six months and at five-year intervals thereafter. Compliance has been between 72% and 75%. However, because of the increasing logistical burden and cost of data collection, questionnaire collection, after advice from the Registry statistician, has been restricted to a randomized 20% of the annual total number of patients who underwent primary hip and knee replacements. An Internet reply option is available for respondents, and, currently, approximately 10% reply in this format.

Regional Registries

Winnipeg Regional Health Authority Joint Replacement Registry

There are numerous small regional registries around the world that collect PROM data. One such example is the Winnipeg Regional Health Authority Joint Replacement Registry. In addition to collecting preoperative twelve-item Oxford scores and comorbidity information, the registry collects one-year follow-up data consisting of the twelve-item Oxford scores, patient satisfaction (through the use of a five-point Likert scale), and complications (through the use of a patient-completed “yes or no” check list). Of the 3000 yearly cases, approximately 90% of the patients agree to participate, and about 75% of these patients return the one-year questionnaire (no reminder is sent). These PROM data, including those regarding satisfaction, are incorporated into the yearly surgeon and hospital reports that are reviewed by the regional orthopaedic standards and quality committee.

California Joint Replacement Registry

The California Joint Replacement Registry (CJRR), which is currently in a pilot phase, administers the SF-12, WOMAC, and University of California, Los Angeles (UCLA) Activity Index to all patients preoperatively and at intervals of six months, one year, and five years. Surveys are administered via a web interface, which patients can access from home, and is offered in some physician offices on kiosks or handheld devices. The response rate for preoperative PROM collection during the pilot phase of the CJRR has been at or above 70%.

Cornell Hospital for Special Surgery

The Center for Education and Research on Therapeutics Registry (CERT) is one of the largest registries in the U.S. in terms of collecting functional outcomes data. The registry started in 2007 and currently includes more than 22,000 TJRs. Patients who agree to participate fill out a baseline questionnaire consisting of medical history including duration of symptoms; previous surgery; medication use; SF-36; EQ-5D; HOOS or KOOS; Lower Extremity Activity Scale (LEAS), which is a patient-reported instrument measuring level of activity; VAS for pain; and a validated patient expectations survey. The patients respond to questionnaires preoperatively and then at six months, two years, and five years. Surveys are administered by postal mail, through a web interface, or at the physician's office. All efforts are made to increase the response rate, which currently is approximately 80%.

Results

Predicting Outcomes—The Reasons for Good and Poor Outcomes

The PROM results of the Swedish Hip Arthroplasty Register highlight that revision arthroplasty as a single outcome parameter is too blunt an instrument to describe patient outcomes following TJR procedures. One year after primary total hip replacement, implant survival is, in most studies, close to 100%, while patient satisfaction in the Swedish total hip replacement population is barely 90%. Only 76% of patients have a clinically significant improvement in the EQ-5D index, and 10% do not experience a clinically significant pain reduction. Even though total hip replacement overall has outstanding results compared with many other interventions, a considerable proportion of patients do not respond or respond negatively to total hip replacement. The results of the PROM program confirm earlier findings that Charnley class C is associated with inferior results and that female sex is associated with less pain reduction and with a lower absolute level of EQ-5D index at a follow-up time of one year. However, men did not improve in health-related quality of life to the same extent as women did. It was also demonstrated that anxiety or depression was a strong predictor for worse outcomes. Among patients with osteoarthritis who were eligible for total hip replacement, 43% reported anxiety or depression in the fifth dimension of the EQ-5D questionnaire prior to surgery, and this manifestation of mental distress predicted inferior outcomes regarding pain reduction and satisfaction²⁸.

The NZJR Oxford Hip and Knee Result

Analyses of six-month, five-year, and ten-year results of the NZJR of primary unrevised TJRs show that there is little difference in the mean scores at the different time intervals. This indicates that the six-month score is predictive of the medium-term outcome. Logistic regression analysis has shown a significant relationship between the Oxford score and the risk of revision within two years of the score date²⁹. Surgeons are advised annually of the Oxford scores of their own patients.

Momentum for Improvement Work

Measuring PROMs in a prospective registry program enables national and local improvement work, such as refining indications, standardizing care, and reducing variations in outcomes. Differences in patient-reported outcomes between participating units often reflect diverse demography as well as a distribution of diagnoses and medical comorbidity burdens, which is the so-called case-mix problem. However, there are illustrative examples of local analyses leading to altered implementation of best clinical practice, i.e., improvement work. The example of the small regional joint registry in Winnipeg has led surgeons to institute practice reviews and has caused changes in the case mix undertaken by surgeons, with revision work being performed by high-volume subspecialty-trained surgeons.

Changes over time may reflect national and local alterations due to political or medical changes in the management of TJR patients. The longitudinal design provides a continuous quality indicator of these common and, in aggregate, costly interventions.

Discussion

TJR is a successful surgical intervention to decrease pain and to improve health-related quality of life in patients with disabling end-stage joint disease^30-32. The continuous development in TJR, the progress in medicine in general, and the longer life expectancy of the population have led to increasing demands for and numbers of TJRs. Today there are numerous long-term reports that document the outstanding survival of many implant designs. Therefore, the scope for further major improvement in implant technology seems to be limited for most patient groups. Several years ago, Robert Poss expressed concern about the future of TJR surgery³³. He stated that TJR is now “threatened by its own success.” This concern still remains, as widening indications place greater demands on the procedure and as the expectations and functional requirements of patients change due to generational shifts. To understand how to meet these new demands on TJR, it is fundamental to measure outcomes from the patient's perspective.

The outcome of a medical or surgical intervention depends on biological, psychological, technical, educational, cultural, political, economic, and religious factors as well as equity issues. These factors have to be taken into account when considering alternative interventions and any alterations in current practice. There is a need to identify predictors of both bad and good outcomes after TJR surgery. Widened indications and improper expectations regarding TJR may result in implant surgery being discredited by a minority of dissatisfied patients, which in turn can lead to decision-makers imposing restrictions and/or lower priority to this surgical field, at least in public health-care systems. Thus, the indications for surgery should be correct for the individual, that is, he or she should be likely to benefit from the operation to an extent that cannot be achieved with any other (or less costly alternative) treatments.

Even if we reach a consensus on the indication for TJR, the timing of surgery is another very important issue. To optimize outcome after TJR surgery we have to take into account the future risk of revision arthroplasty surgery, life expectancy, the possibility to enhance health-related quality of life, the cost-utility effect of the intervention, and societal costs before and after surgery. To achieve this optimization requires further research of large registry databases that include PROMs and cost data.

The early experiences of collecting level-III registry data are that it is more difficult to capture, analyze, and interpret this type of data as compared with traditional level-I data. Furthermore, to understand and adjust for confounders is intricate, and more research is needed in this area, although a prospective design allows for partial adjustment for case-mix problems.

Data from the NZJR indicate that low Oxford scores at follow-up times predict revision risk and can therefore be used to monitor outcome and potential failure. This finding suggests that extended follow-up would be beneficial for patients who have low Oxford scores after TJR, but that follow-up may be postponed for patients who have high Oxford scores after TJR unless those patients have other indicators that are subject to particular monitoring.

PROMs in small-scale randomized clinical trials, with strict inclusion criteria, will not be effective in addressing these important health-care policy issues. Therefore, it is important to include PROMs in the well-established TJR registries (nationwide observational studies). It is, however, essential to consolidate a register on level I and II before PROMs are included. It is laborious and costly to include a PROMs routine in a register, but a valid PROM database opens up new fields for both national and local improvement efforts as well as new research areas. It can also further enhance the utility of registries as powerful health-economic instruments.

Multidimensional outcome assessment, combining traditional outcome parameters (level-I or II data) with level-III data such as PROMs and costs, provides a more nuanced and complete assessment of the surgical interventions.