Comparative effectiveness research in the field of hand surgery

Abstract

Comparative Effectiveness Research (CER) is a concept initiated by the Institute of Medicine (IOM) and financially supported by the federal government. The primary objective of the CER is to improve decision making in medicine. This research is intended to evaluate the effectiveness, benefits, and harmful effects of alternative interventions. With encouragement from the IOM, CER studies are commonly large, simple, observational, and conducted using electronic databases. To date, there is little comparative effectiveness evidence within hand surgery to guide therapeutic decisions. A general discussion of CER will illuminate how it could make a beneficial impact. In order to draw conclusions on effectiveness through electronic health records, databases must contain clinical information and outcomes relevant to hand surgery interventions, such as patient-related outcomes (PRO). Unlike objective measures such as morbidity, mortality and physiological markers, PROs provide patients’ perspective on treatment benefit and can be the outcome of greatest importance.

Introduction

The U.S. Institute of Medicine (IOM), in 2008, started a national initiative of research known as comparative effectiveness research (CER) to support better decision making about interventions in medicine [1]. Clinical decision making may vary based on patient factors, clinicians’ experience, and regional preferences. Too often these decisions are made without supportive evidence. Inconsistent clinical practice is well recognized and raises concerns about the appropriateness and economics of current medicine. This is evident by the cost and outcome differences that exist in health care across the United States [2]. The IOM attributes inconsistencies in health care delivery to the lack of information available to make well-informed decisions in everyday clinical medicine.

The IOM publicized the need for high impact research to improve the quality and efficiency of health care in a comprehensive report in 2008 [3]. In response, legislators passed the American Recovery and Reinvestment Act of 2009 (ARRA), which allocated $1.1 billion to fund research referred to as comparative effectiveness research [1]. The Federal Coordinating Council and an appointed IOM committee were charged with identifying high-priority research topics and to allocate funds from the ARRA. The president distributed these funds to the National Institutes of Health (NIH), Agency for Health Research and Quality (AHRQ) and Office of the Secretary of the US Department of Health and Human Services (OS-DHHS). In 2010, as part of the Affordable Care Act, legislators established an ongoing national program in CER, the Patient-Centered Outcomes Research Institute (PCORI) [4].

CER is not a novel concept, but represents a research movement propagated by an enormous investment by the federal government. CER can be conceptualized as a form of outcomes research. This research asks which intervention is most effective, for whom and under what circumstances [5]. Through encouragement from the IOM, CER trials often utilize large electronic databases to study current health practices and outcomes. They are often referred to as pragmatic trials, because they reflect routine clinical practice [6].

Hand surgery has embraced outcomes research and, to better evaluate effectiveness of interventions, have developed questionnaires such as the Michigan Hand Outcomes Questionnaire (MHQ). The MHQ is a subjective evaluation tool used to measure outcomes such as hand function and pain [7]. These instruments report patient-related outcomes (PROs), a recognized and standardized method of reporting patient’s perspective on interventions [8]. PROs provide the patient’s perspective on treatment benefit and can be the outcome of greatest importance [8]. Therefore, incorporating PROs into CER would greatly enhance the quality of hand surgery research, thus providing better evidence to base clinical decisions.

Definition of CER

CER aims to provide information about the relative effectiveness of different medical interventions to improve the quality and value of care [9]. The IOM defines CER as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care” [3]. When conducting CER, authors are asking themselves, how does this intervention compare, both overall and in subsets of the population [10]? Therefore, researchers seek to determine what interventions are appropriate for particular patients and populations within a variety of circumstances. CER investigates interventions, tests (e.g. diagnostic, therapeutic), prevention strategies, care delivery, and quality of care [4].

The IOM adds that “…the purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population level” [8]. Thus, information should lead to more standardized care, while recognizing decisions may vary if individuals fall into a particular subset. The IOM clearly identifies multiple stakeholders, outside the doctor-patient relationship, including payers and policy makers. In acknowledging these stakeholders, there is an underlying national objective of optimizing health outcomes within financial and resource constraints [11].

Methodology of CER

CER is the study of two different but accepted standard practices, neither of which is superior based on available medical evidence [6]. Interventions are simple and occur within practical clinical settings. A key component is the use of real world data, therefore reflecting patients who are typical of day-to-day clinical care [1]. Without specific patient inclusion criteria, conclusions are drawn from a population representative of those who would receive the intervention in a normal clinical setting [9]. Observational CER studies include patient cohorts numbering in the thousands that are achieved through large medical databases. Outcomes of CER studies are intended to be clinically relevant, meaningful to the patient and general public, and minimally subject to ascertainment bias [4, 12].

CER methodologies are in contrast to classical double-blinded RCTs that are conducted on highly selected populations with rigorous inclusion and exclusion criteria. Study enrollment in RCTs comprise of patients with few co-morbidities in order to optimize statistical power and the benefit/risk tradeoff [11]. These so-called efficacy studies look at whether an intervention is efficacious under ideal, controlled settings [1,11]. The outcome measures of efficacy trials are often arbitrary and less clinically relevant [6]. RCTs answer the question “does this work?” [1] CER, alternatively, provides decision makers with the answer to “is this better than that?” [1]. Simplistically, CER studies are less controlled, with fewer exclusion criteria, and therefore, the conclusions are said to be generalizable to a large population. The IOM supports the broad use of evidence to evaluate effectiveness [11], including systematic reviews, retrospective database analysis, prospective observational studies or pragmatic RCTs. Despite this statement, there is strong emphasis placed on observational, database research. Table 1 highlights the characteristic differences between classic RCTs and observational, database conducted CER.

Table 1.

A characteristic comparison of observational, database CER and classic multi-institutional RCTs.

Comparison of Comparative Effectiveness Research and Randomized Control Trials (RCT)
Design	Conventional RCTs	Database CER
	Interventional	Observational
Patients/Population	Patients with few co-morbidities	Patients represent those typically seen in ‘everyday practice’
	Rigorous inclusion/exclusion criteria	Minimal inclusion/exclusion criteria
Intervention	Intervention of interest	An acceptable standard of practice
Control	Placebo or standard of practice	Alternative acceptable standard(s) of practice
Outcome	“Does this work?”	“Is this better than that?”
	Is the intervention efficacious in the highly selected population?	Is the intervention effective in the general population?
	Composite endpoint (e.g. improved range of motion)	Outcomes meaningful to patient/public (e.g. PRO)
Study size	Hundreds	Thousands
Data collection	Single, multiple institution sites	Electronic Medical Records/Databases
Study of population subsets	Difficult	Possible (Encouraged by the IOM)
Duration	Long	Short
Cost	Expensive	Low cost per patient
Bias	Selection bias/confounding minimized in double blinded studies	Selection, indication bias problematic

The value of electronic databases in CER

Electronic databases, including administrative claims databases and electronic health record databases, allow researchers to evaluate how current health care practices affect the outcome of care [1]. The IOM understands that the success of CER will depend on the quality of electronic clinical data. They recommend that “The CER Program should help to develop large-scale, clinical and administrative data networks to facilitate better use of data and more efficient ways to collect new data to yield CER findings” [3]. This is particularly true in hand surgery, where subjective outcomes such as pain and aesthetics are critical in evaluating intervention effectiveness [5]. Currently, databases do not routinely collect this information [13].

In using large electronic databases, observational research can be fast, low-cost, high-volume and can represent real-world decisions [1]. Additionally, large-scale observational studies may confirm results of randomized trials in understudied patient subsets, inform about rare events and provide insight into the processes of care delivery [14]. Databases allow for subgroup analysis and assessment of treatment heterogeneity. Identification of characteristics among subgroups may help to identify key predictors of response [1]. Individual responses to health interventions must not be overlooked, as subsets of populations may have dramatically different responses to interventions as compared to the whole population? [14]. Individual differences may include genetic risk factors and environmental exposures.

Making accurate inferences from observational studies can be challenging. Therefore, the AHRQ has helped to develop the Electronic Data Methods (EDM) forum to advance the knowledge and practice of the use of electronic clinical data for CER [15]. Critical issues that this organization addresses include confounding adjustment approaches for observational CER studies, models for evaluating data variability and quality, patient privacy and data security, and incorporating PRO into CER [15].

The Healthcare Cost and utilization Project (HCUP), a collection of electronic databases and software tools, is an example of a database which could be used in hand CER studies [13, 16]. Data collected within HCUP include ICD diagnosis codes, demographic information, cost of stay, length of stay and other administrative information [13]. Unfortunately, drawing conclusions of effectiveness on interventions, such as patients with rheumatoid arthritis who have undergone arthrodesis, would be difficult with this information. Unlike other surgical specialties, outcomes such as length of stay, re-admission rates, mortality or major clinical event (e.g. myocardial infarction) are much less common in hand surgery. Outcomes pertinent to a patient considering arthrodesis include functionality, pain, return to work, aesthetics and ability to perform activities of daily living [5]. PRO questionnaires, such as the MHQ, provide a reliable and efficient way to document these outcomes. Creating databases which contains this information is necessary to make assessments of effectiveness for hand surgical interventions.

Table 2 lists CER studies from a range of specialties that have utilized large databases to conduct observational research. Note the various electronic databases that are used, including administrative, specialty specific, Medicare, and Medicaid. When designing a CER study in the field of hand surgery, selecting the appropriate database is imperative. Figure 1 highlights important database characteristics that would facilitate a well-conducted study. They should include accurate and uniform documentation of patient demographics, diagnoses, and outcomes.

Table 2.

Examples of CER studies that utilized observational, database methodologies.

Author	Topic	Database - Study size (N)	Outcomes measured	Authors conclusions
Dewitt et al. (31)	Nonbiologic vs. biologic DMARDs for RA treatment	US-based observational registry (2001–2008) N=1,729	Clinical Disease Activity Index (CDAI) scores	Both treatment groups experienced lower CDAI scores across time. Patients switching to bDMARD demonstrated greater improvement than patients switching to nonbiologic DMARD.
Grijalva et al. (32)	Patters of medication use in RA	Tennessee Medicaid (1995–2004) N=23,342	Patterns of DMARD use	The utilization of DMARDs increased in TennCare patients with RA, and by 2004, use of biologics was substantial. Although glucocorticoid utilization decreased, use of both NSAIDs and narcotics increased.
Solomon et al. (33)	Comparative safety of opioids	Medicare (1996–2005) N=6,275	Cardiovascular events, factures, gastrointestinal events	The rates of safety events among older adults using opioids for nonmalignant pain vary significantly by agent.
McCutcheon et al. (34)	Surgeons vs. interventionalists in performing EVAR	Nationwide Inpatient Sample (1998–2009) N=28,094	Mortality rate, length of stay, hospital charge	Surgeons are associated with improved outcomes, with lower mortality, shorter length of stay, and lower charges for EVAR cases, when compared with interventionalists.
Martin et al. (35)	Three oral bisphosphonates	Administrative claims databases (2005–2007) N=45,939	Fractures, time to fracture, health-care cost	Rates of adherence and total adjusted all-cause health care costs for alendronate, risedronate, and ibandronate are similar. Absolute, unadjusted rates of fracture were small and did not significantly differ among agents.
Aghayev et al. (36)	Lumbar total disc arthroplasty (TDA) vs. anterior lumbar interbody fusion (ALIF)	SWISSspine registry (2005–2010) N=534	QoL, pain alleviation	Pain alleviation after TDA and ALIF was similar.
Eurich et al. (37)	Sitagliptin vs. other glucose lower agents in type two diabetes	US claims and integrated laboratory databases* (2004–2009) N=72,738	Hospital admissions, mortality	Sitagliptin use was not associated with an excess risk of all cause hospital admission or death compared with other glucose lowering agents.

^*Clinformatics DATA Mart, OptumInsight Life Sciences Inc.

DMARD: Disease-modifying antirhematic drugs, RA: Rheumatoid arthritis, PRO: Patient related outcome, EVAR: Endovascular aortic aneurysm repair, QoL: Quality of life

Figure 1.

Selecting the appropriate database is critical in designing observational CER studies. Column two highlights important database characteristics. Column three offers an example within Hand Surgery.

Incorporating PROs into CER

To sufficiently provide informed decision making, CER studies should seek to measure all outcomes that are important to patients [9]. Conventional endpoints, such as disease-free interval do not provide consumers with a complete understanding of treatment effects. PROs are any reports coming directly from patients about how they function or feel in relation to a health condition and its therapy [17]. Constructs that can be assessed with PROs include symptoms, functional status, health related quality of life (HRQoL), health behaviors and patient satisfaction with care [18]. For example, HRQoL measures include perceptions about health in general, physical function, social functioning and psychological well-being [19]. A study by Waljee et al. showed that the MHQ, which measures several HRQoL constructs, is an essential instrument to understand the extent of disability of rheumatic hand disease [20]. These outcomes are best judged by patients and they define effectiveness in a way clinical measures (e.g. blood pressure, laboratory test) cannot [19].

Measuring PROs can be accomplished through structured questionnaires or interviews. Ideally, these subjective measurement tools are easy to administer, quick and low cost. The quality of an instrument is based on its clinimetric properties that include reliability, validity, responsiveness, and interpretability [8, 21]. The MHQ, a PRO questionnaire, is one such instrument which has proven to be responsive in evaluating outcomes in a variety of conditions related to the hand [5]. Waljee et al. demonstrated that the MHQ is easily administered, reliable, and valid in measuring rheumatoid hand function, thus providing a clinical way to measure outcomes in rheumatic hand disease [20]. The MHQ is also responsive in measuring outcomes involving distal radius fractures [22]. A separate subjective assessment tool, the Carpal Tunnel Questionnaire has been shown to have greater responsiveness to clinical change following carpal tunnel repair than the MHQ [23]. Therefore, the most appropriate PRO tool may vary based on the hand condition of interest.

PCORI, a nonprofit corporation, was established in 2010 by the Patient Protection and Affordable Care Act (ACA) and has promoted patient-centered outcomes research (PCOR). PCOR, a branch of CER, is primarily intended to inform decision making amongst individual patients. To accomplish this, PCOR focuses on outcomes “that people notice and care about such as survival, function, symptoms and HRQoL.” [24]. PCOR emphasizes the patient perspective and is supportive of the inclusion of PROs within CER [24, 25]. Hand research has already widely adapted subjective assessment tools. Promoting the inclusion of questionnaires, such as the MHQ, into electronic databases would be a substantial advancement towards CER in hand surgery.

Snyder et al. identified a number of practical issues/challenges that exist to incorporate PROs in PCOR [25]. This involves comprehensive and uniform adoption of PROs, proprietary nature of the measurement tools, selection of the best PRO and the clinical interpretation of PROs [25]. At an institutional level, PRO tools are often used by individual investigators or research groups. Standardizing and integrating PRO tools into clinical practice across the country would take considerable effort and possibly governance policy [25]. Snyder et al. postulated that perhaps in the future, a PRO test may be billable to insurers, much as laboratory test are [25]. Uniform use of PRO tools and the incorporation of this data into electronic health records/databases will take concerted effort to establish.

Effectiveness studies in hand surgery

To date, there is minimal CER studies on interventions within the field of hand surgery; federally and privately funded. The CER Database, maintained by the National Patient Advocate Foundation (NPAF), compiles comparative studies funded by the NIH and AHRQ (http://www.npaf.org) [26]. The CER Inventory provides an inventory of projects funded by agencies within the HHS (http://cerdatatracker.org/) [27]. Within these inventories there are multiple trials studying the biological and nonsurgical therapies for rheumatoid arthritis, osteoarthritis and osteoporosis, but there is only one publication directly related to the comparative effectiveness of a hand topic. This is an ARRA funded study titled The Value of High Quality Medical Care for Work-Associated Carpal Tunnel Syndrome (Steven Asch, Rand Corporation) scheduled to be completed by 7/31/2015 [26]. Table 3 lists ongoing studies funded by federal grants from the NIH and ARHQ as listed on the NPAF website [26] that relate to the musculoskeletal system.

Table 3.

Federal grants for musculoskeletal related comparative effectiveness studies funded by the NIH and AHRQ aslisted on the National Patient Advocate Foundation (NPAF) website (http://www.npaf.org/news/CER-database-launched) (26).

Primary investigator	Grant number	Topic	Database	Outcomes	Project end
Asch, Steven M. (RAND Corporation)	R01 HS18982-01	Work-associated CTS	Compensation claims in Northern California	Quality of medical care measured by RAND/UCLA CTS Quality Measures	07/31/2015
Bannuru, Raveendhara (TuftsMedical Center)	F32 HS21396-01A1	Medical treatments for Knee Osteoarthritis	Unspecified (Planned network meta-analysis)	Unspecified	06/30/2013
Saag, Kenneth (University of Alabama at Birmingham)	U19 HS21110-01	Multiple NSAIDS vs. Narcotics after joint replacement surgery	Unspecified	Unspecified	08/31/2016
Tosteson, Anna (Dartmouth College)	R01 HS18405-01	Treatments for degenerative spine disease	Medicare claims and Spine Patient Outcomes Research Trial (SPORT)	Unspecified	01/31/2013
CTS: Carpal tunnel syndrome

In addition to the federal investment in the national CER program, the private sector is equally important in supporting effectiveness research. None of the original high-priority topics identified by the IOM (which received a majority of the ARRA funding) were directly relevant to hand surgery interventions [2]. Currently, there appears to be minimal CER within hand. A basic search for “comparative effectiveness” in PubMed fails to identify large-scale database-conducted observational trials related to “carpal tunnel syndrome”. A co-search of “comparative effectiveness” and “radial fracture” (or “distal radial fracture”) identifies one large database study by Chung et al [28]. They performed an analysis of Medicare data to evaluate the variations in the use of internal fixation for distal radial fractures in the US. They found the use of internal fixation differs widely across geographical regions. The authors concluded that the variation in treatment was a result of the lack of comparative effectiveness evidence [28].

Although this study did not directly compare two interventions, or determine effectiveness of internal fixation of radial fracture, it highlights an obstacle in database research within hand surgery. Their conclusions were drawn from limited data, including basic patient demographics, co-morbidities, presence of osteoporosis, concurrent ulnar fracture, type of surgeon and geographical information [28]. As mentioned previously, evaluating effectiveness of interventions will require information on outcomes such as pain, functionality and aesthetics.

Challenges in CER

“CER is as vulnerable to bias and conflict of interest as any other area of medical research” [1]. By emphasizing observational studies, with limited exclusion criteria, CER is particularly susceptible to selection and indication bias [29]. Selection bias refers to systematic differences between baseline characteristics of the groups that are compared [17]. Indication bias is also referred to as confounding by indication and confounding by severity of disease. Characteristics of patients that dictate clinical decisions may influence outcomes, prompting debate on whether the intervention or the patient determines the outcome [1]. This is an example of confounding by indication. Confounders may also arise at an institutional level where different treatment preferences or reimbursement policies exists [30]. Statisticians attempt to correct for indication bias through adjustments with disease severity scoring systems or propensity scores, but the only way to complete eliminate it is through randomization [28]. In IOM’s 2009 sentinel article, they acknowledge that “overcoming the limitations of observational research is the most important frontier of research on study methods ” [2].

Improving clinical decision making will depend on the results of CER studies. Therefore, a premium shall be placed on identifying the most pertinent outcomes to measure. Because the definition of CER identifies multiple stakeholders the outcomes of greatest important may vary [29]. Furthermore, what constitutes as a benefit will vary amongst these stakeholders. For example, patients may base decisions on pain relief from carpal tunnel syndrome, whereas a policy maker or payers may place an emphasis on cost of care or time to return to work.

By providing quality evidence of effectiveness, CER will theoretically reduce variation in care and minimize unnecessary, costly interventions [9]. This is the justification behind the federal funding. The explicit exclusion of cost effectiveness analysis (CEA) remains a controversial topic. CEA in countries such as Australia, England and Canada plays an integral role in the development of clinical guidelines [11]. This aids in determining the greatest health benefit within a limited budget [9]. CEA is intentionally excluded by the ACA which prohibits PCORI from engaging in research measuring value of interventions [9]. Those opposing CEA within CER argue that the federal government should not participate in cost considerations in research. This is viewed as rationing and governmental interference in patient care. Because PCORI is an integral part of CER, Garber et al. believes it should continue to collect data on cost [9]. This would allow private analysts to conduct research that provides important cost outcomes information [9].

Conclusion

The national movement to conduct comparative effectiveness research is in response to the lack of information that exist to make evidence based clinical decisions. The IOM believes that through demonstrating the effectiveness of alternative treatments, patients and other stakeholders will be equipped with the knowledge to choose the appropriate treatment. The federal government has demonstrated their commitment to this research through funding and the establishment of organizations, such as PCORI. Although surgical interventions in hand were not among the high priority topics identified by the IOM in 2008, this movement presents an opportunity to participate in robust database research. For hand surgeons to benefit from electronic health databases, they should promote the incorporation of outcomes data that is relevant to surgical interventions. This can be accomplished through the inclusion of subjective measuring assessments which evaluate PROs such as pain, functionality, aesthetics and other health related quality of life measures.

Key Points.

• US Institute of Medicine (IOM), in 2008, started a national initiative of research known as comparative effectiveness research (CER) that will support better decision making about interventions in healthcare.

• CER focuses on interventions that occur within real-world environments, therefore, the conclusions are said to be generalizable to a broad population.

• CER conducted through large electronic databases allow researchers to evaluate how current health care practices affect the outcomes of care.

• To date, there is minimal comparative effectiveness evidence in hand surgery, partly attributed to lack of relevant outcomes information included in electronic databases.

• Inclusion of patient-related outcomes (PROs) into electronic databases will facilitate the adaptation of CER into the field of hand surgery.