Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Jul 1.
Published in final edited form as: Nurs Outlook. 2012 Jul;60(4):182–190. doi: 10.1016/j.outlook.2012.04.004

Using Meta-analyses for Comparative Effectiveness Research

Vicki S Conn 1,✉, Todd M Ruppar, Lorraine J Phillips, Jo-Ana D Chase
PMCID: PMC3396882  NIHMSID: NIHMS373875  PMID: 22789450

Abstract

Comparative effectiveness research seeks to identify the most effective interventions for particular patient populations. Meta-analysis is an especially valuable form of comparative effectiveness research because it emphasizes the magnitude of intervention effects rather than relying on tests of statistical significance among primary studies. Overall effects can be calculated for diverse clinical and patient-centered variables to determine the outcome patterns. Moderator analyses compare intervention characteristics among primary studies by determining if effect sizes vary among studies with different intervention characteristics. Intervention effectiveness can be linked to patient characteristics to provide evidence for patient-centered care. Moderator analyses often answer questions never posed by primary studies because neither multiple intervention characteristics nor populations are compared in single primary studies. Thus meta-analyses provide unique contributions to knowledge. Although meta-analysis is a powerful comparative effectiveness strategy, methodological challenges and limitations in primary research must be acknowledged to interpret findings.

Despite remarkable scientific advances over recent decades, the effectiveness of many health interventions remains unclear. The Institute of Medicine noted that evidence of effectiveness exists for less than half of the interventions in use today.1 Scant evidence exists comparing multiple possible interventions for the same health problem.2 Newer or more costly interventions may not be linked with better outcomes, and variations in health care expenditure may be unrelated to changes in health outcomes.3–5 The troubling lack of information about interventions’ relative effectiveness led to Comparative Effectiveness Research (CER) initiatives.

CER can be defined as research designed to discover which interventions work best, under what circumstances, for whom, and at what cost.1,6 CER methods include randomized controlled trials, non-randomized comparison studies, prospective and retrospective observational studies, analyses of registry and practice data sets, practice-based evidence studies, and meta-analyses.6–9 This paper examines using meta-analytic approaches for CER. Examples of nurse-led meta-analyses will be used to demonstrate key points. The paper begins with an explanation of meta-analytic overall effect size estimates for CER, especially in situations with inconsistent findings among primary studies. The value of statistically quantifying the magnitude of effects for both clinical and patient-centered outcomes is described. Unique contributions of meta-analysis for both specifying temporal patterns of outcomes and adverse outcomes are presented. Then the importance of including diverse studies which represent clinical heterogeneity is explained. The use of patient characteristic moderator analysis to accomplish CER goals of identifying which interventions work best for which subjects is explored. The use of moderator analyses to determine if intervention characteristics are linked with outcomes is presented. The use of moderator analyses to determine if setting characteristics are associated with outcomes is described. The potential use of moderator analyses to explore intervention worth is briefly addressed. Finally, selected limitations of meta-analytic methods and primary studies are discussed to provide a context for interpreting meta-analytic CER. Full details of meta-analysis methods, including limitations, are available in other sources.10–15

Application of Overall Effect Sizes to Comparative Effectiveness Research

CER includes determining effectiveness of interventions on clinical and patient-centered outcomes. CER can involve performing a meta-analysis of primary studies to quantify intervention outcomes. Meta-analyses can synthesize results of head-to-head comparisons of two interventions in primary studies or compare two interventions tested in different primary studies. Meta-analytic statistical procedures generate a unitless effect size for each study. Thus outcomes reported using different measures of the same construct in primary studies may be combined. Each effect size is weighted by the inverse of its sampling variance so studies with larger samples have more influence in aggregate effect-size estimates.11

The meta-analytic approach of estimating an effect size for each primary study does not depend on p values in original studies, which makes it valuable in areas of science where underpowered studies are common. Some areas have multiple small primary studies without statistical power to detect important changes. Reviews of such work conducted without meta-analysis, such as those relying on vote counting of the proportion of studies with statistically significant findings, might conclude that the primary studies did not support the effectiveness of the tested intervention because they reported statistically nonsignificant differences between treatment and comparison groups. However, meta-analytic strategies can combine the magnitude of differences between treatment groups across primary studies to discover a clinically important intervention effect. For example, we retrieved 10 studies testing the effects of physical activity behavior self-monitoring as an intervention to increase physical activity.16–25 Four of the studies reported statistically significant findings in favor of self-monitoring. Six other studies reported that self-monitoring did not significantly improve physical activity behavior. A review without meta-analysis would conclude that the evidence is mixed, inconclusive, or did not support the efficacy of self-monitoring. In contrast, a meta-analysis of the same studies documented an overall effect size of .435 (standardized mean difference), which is significantly different from no effect (p<.001; 95% confidence interval: .278, .592). Thus the meta-analysis concluded that self-monitoring increased physical activity. Figure 1 includes a forest plot that demonstrates these findings.

Figure 1.

Figure 1

Open in a new tab

Forest plot of 10 studies that tested self-monitoring interventions

Note: The horizontal line adjacent to each study on the forest plot reflects the confidence interval for that study’s effect size. Studies with horizontal lines crossing 0 did not report a statistically significant outcome in the individual studies. The meta-analysis standardized mean difference effect size, the final row in the figure marked ‘Effect size’, is represented by the diamond whose width corresponds to the confidence interval.

CER aims to determine the extent to which interventions are effective, not whether they are better than control conditions. Meta-analysis calculates and emphasizes the magnitude of the effect, rather than the tests of statistical significance reported in primary studies. The emphasis on effect size, instead of tests of statistical significance, also aids interpretation of findings from overpowered primary studies with statistically significant findings that may not be clinically important. For example, a study of an intervention to reduce pain may have a statistically significant p value if hundreds of subjects are included, while the average reduction in pain between the treatment and control group might be from 6.5 to 6.2 on a pain scale of 0 to 10. Meta-analysis findings emphasize the magnitude of effects, thus overpowered studies are interpreted in the context of the effect size they achieved.

Because CER results are intended to improve clinical practice, outcomes need to be interpretable by practitioners. The meta-analysis overall effect size, which quantifies the magnitude of effects, can be converted to the original clinical metric to enhance interpretation. For example, a meta-analysis of metabolic outcomes of diabetes self-management programs reported an overall mean difference effect size of .26. The conversion to the original metric depicted findings in clinically meaningful terms: HbA1c of 7.38 for treatment subjects as compared to HbA1c of 7.83 for control subjects.26 Clinical practice can be further supported by making comparisons across meta-analyses to determine consistency of findings. These comparisons can be accomplished by the ability to convert meta-analysis effect size metrics (e.g., odds ratios to standardized mean difference).27

CER aims to examine intervention effects on multiple clinical as well as patient-centered outcomes. Meta-analyses compute separate effect sizes for diverse outcomes that are reported in primary research. Although a main health outcome may be considered most important, other outcomes may be summarized separately to estimate intervention effects for multiple outcomes. For example, a meta-analysis comparing passive descent to immediate pushing during second-stage labor in nulliparous women with epidural anesthesia examined multiple outcomes: spontaneous vaginal birth, instrument-assisted delivery, cesarean birth, lacerations, and episiotomies.28 Varied patterns of findings among related outcomes can be interesting. For example, a meta-analysis of exercise interventions among older adults found improvement in objective physical performance measures but no improvement in ability to perform activities of daily living.29

Patient-centered outcomes research emphasizes outcomes of importance to patients such as quality of life, symptoms, or functional status. Patient-centered outcomes can be synthesized in addition to other outcomes health providers typically value.30 For example, a meta-analysis of silver-releasing wound dressings included pain-related symptoms and quality of life measures as well as typical clinical outcomes of wound healing, exudate, and dressing wearing time.31

Analyzing multiple outcomes is important because the definition of “success” for interventions varies.32 Comparisons between interventions may reveal small or negligible differences in main outcome effect sizes. In these cases, comparisons of other non-primary outcomes, such as patient convenience, may provide valuable information about complex tradeoffs for making decisions about patient care.33

Providers are interested in CER research that documents persisting health benefits of interventions, not just immediate improvements. Effect sizes calculated for multiple time points can provide information about the temporal pattern of effects. Some primary studies report outcomes over multiple time points. Others report only one outcome assessment, though its timing may vary across studies. These data can be used in meta-analyses to identify interventions whose effects are transient or those showing limited immediate impact but long-term positive outcomes.32 These patterns may reveal themselves as interventions first become effective, peak in effectiveness, and then decay. For example, Van Kuiken documented changes in the effects of guided imagery on outcomes over 5 to 18 weeks.34

CER is intended to develop information to providers and patients about both positive and negative outcomes of interventions so advantages and disadvantages may be considered in making treatment decisions. Adverse or negative events are important sequelae that CER meta-analyses can address. Many adverse events are rare, which makes it difficult to assess incidence in individual primary studies. Combining adverse event rates across multiple primary studies with thousands of subjects provides more stable estimates of incidence than are available in single studies. For example, Lo et al. documented no increased incidence of adverse events when using silver-releasing dressings over alternative dressing by aggregating findings across many patients in multiple primary studies.31 Although primary research tends to emphasize positive outcomes in research reports, providers need accurate information about negative events or neutral outcomes to weigh the advantages and disadvantages of interventions for practice.

Heterogeneity in Meta-Analyses Comparative Effectiveness Research

CER values real-world tests of interventions. Heterogeneity is expected in CER meta-analyses because primary studies (1) include samples of diverse, real-world populations, (2) commonly have planned and unplanned variations in interventions, and (3) test interventions in varied clinical settings that may influence their effectiveness or patient responsiveness. Meta-analysts’ decisions regarding inclusion and exclusion of potential primary studies with diverse samples and interventions should be directed by conceptually clear definitions about what kinds of interventions should be combined and for which types of subjects. CER meta-analyses generally use random-effects model analyses which assume diversity in sample, interventions, and study methods. (Methodological challenges related to inclusion criteria and primary study quality are addressed in the limitations section.)

Heterogeneity is valuable because CER includes studies conducted with diverse populations and varied methods to provide strong evidence about interventions’ effectiveness. CER expects variations in patients, interventions, and outcomes. This approach stands in contrast to efficacy findings commonly established in tightly controlled randomized controlled trials.8,35 The emphasis on randomized controlled trials in some Cochrane Collaboration reviews is one reason these may have limited CER impact. A strength of meta-analysis is its ability to estimate heterogeneity and examine potential moderating variables that contribute to it. Even when testing identical interventions, heterogeneity of outcome effects is common because patients vary in their response to treatments, and treatment effects may vary by setting.35 Heterogeneity offers the opportunity to conduct moderator analyses to explore how primary studies differ by examining sample, intervention, and setting characteristics that may be linked to outcomes. CER meta-analysis facilitates discovery of best practices by identifying interventions that are the most effective overall and for certain populations once sufficient primary research has accumulated.8

Patient Characteristic Moderator Analyses

One focus of CER is identifying differential intervention effectiveness for specific populations. CER subgroup moderator analyses can focus on demographic features such as ethnicity or gender, or can examine health characteristics such as disease severity or functional status. Meta-analysis moderator analyses can examine whether intervention effectiveness varies by patient subgroups. For example, a meta-analysis of interventions to increase medication adherence among older adults found that interventions were most effective for those with three to five prescription medications.36 This could be because those on fewer medications needed little assistance with medication adherence and those on more than five might need more intense interventions than those typically tested.36 Rice reported that smoking cessation interventions were more effective for cardiac patients than for other populations.37

The increased CER emphasis on patient-level attributes linked with better or worse outcomes may lead to more personalized care. 38 Findings that intervention effects do not vary by sample characteristics may mean that a range of patients may experience similar benefit from the intervention. For example, a meta-analysis of respiratory rehabilitation interventions on exercise capacity found similar benefits across sample age or initial forced expiratory volume.39

Intervention Characteristic Moderator Analyses

CER aims to provide clinical guidance by comparing interventions to determine which interventions are most effective.

Intervention Moderators

In a few situations, meta-analysis can prove useful in determining whether an intervention is better than no intervention, such as a watchful waiting approach.38 For some interventions, it can be valuable to synthesize comparisons between new interventions and usual care. If usual care is standardized, these analyses provide information comparing two interventions. But oftentimes usual care is not standardized, and such comparisons cannot yield clear recommendations for practice. More commonly, providers need to know which interventions are most effective.

Meta-analyses can address comparisons between interventions by either synthesizing extant primary research with head-to-head comparisons of treatments or by using moderator analyses on primary studies which test different interventions. Using meta-analysis, researchers can directly compare interventions from multiple primary studies that compare the same two interventions. The effect sizes for the difference between the two interventions provide information about the most effective intervention, when methodological quality was similar between studies and valid outcome measures were used. For example, Lo et al. synthesized findings of primary studies that each compared silver-releasing dressing to other dressings.31

Unfortunately, many primary studies of nursing interventions are not compared against other interventions. Head-to-head comparisons of multiple interventions in the same primary studies are unusual because of funding, feasibility, and very large sample challenges. Rather, interventions are generally compared to usual care or a control group. Using meta-analysis, interventions not directly compared in primary studies can be indirectly compared to accomplish the goals of CER to compare interventions.7 The effect of one intervention compared to a control group can be contrasted either with the effect of a second intervention compared to a control group.38 Two interventions each compared to usual care in separate primary studies can be compared using meta-analysis.38 An effect size is computed for the first intervention compared to control subjects. A separate effect size is calculated for the second intervention compared to control groups. The difference in the effect sizes is tested statistically to determine whether the first or second intervention was most effective. Since no primary studies directly compared the two interventions, this indirect comparison is a unique contribution of meta-analysis. For example, a meta-analysis by Jung, Lee, and Lee compared exercise-only interventions to exercise-and-education interventions to reduce fear of falling in older adults.40 Primary studies did not compare the two interventions but rather compared each one to a control group. Their meta-analysis statistically compared the interventions despite the absence of any primary studies making this direct comparison.40

Nurses often use common labels to describe variable interventions. For example, patient education could describe work to change knowledge and attitudes about exercise, or it could describe behavioral strategies to change exercise (e.g., self-monitoring, prompts, contracts). Meta-analysis adds clarity in such cases with its ability to compare characteristics of interventions to determine the best one. For example, a recent meta-analysis of physical activity interventions found that behavioral interventions (e.g., self-monitoring, cues, rewards, behavioral goals) were more effective than cognitive interventions (i.e., changing knowledge, attitudes, beliefs) at increasing physical activity behavior.41 These comparative analyses provide evidence about best practices to achieve desired outcomes.42

Moderator analyses can examine intervention features that may vary along dimensions beyond content.43 Dose variations include individual dose amount, dose frequency, and total number of doses. Intervention timing may be linked to index events or other determining factors. Mode of delivery can include face-to-face or mediated mechanisms (e.g., email, telephone). Interventions may be delivered to the target, who is expected to benefit from the intervention, or to other recipients (e.g., family members of patients, health care providers). Moderator analyses can compare standardized interventions to those tailored to an individual (i.e., intervention features matched to individual subject characteristics) or targeted to groups (e.g., different interventions for subgroups such as women vs. men). Unplanned intervention variations (e.g., unanticipated content or dose variations) can relate to outcomes. Moderator analyses on such characteristics can provide information to help design interventions that improve health and well-being outcomes.

Setting and Context Moderator Analyses

CER aims to discover the best interventions in specific situations. Meta-analyses can compare interventions’ setting and context characteristics using moderator analyses to discover circumstances in which interventions are most effective. For example, interventionist characteristics that vary among primary studies (e.g., advanced practice nurses vs. physicians) can be compared statistically. Setting features, such as home vs. clinic or individual patient vs. group of patients, also can be examined to determine the most effective setting. For example, Conn et al.’s meta-analysis of physical activity behavior outcomes compared interventions delivered to groups versus individuals and compared interventions delivered face-to-face versus mediated mechanisms (e.g., telephone).41 Modifications in health care delivery are important potential moderators in health services research. For example, Kim and Soeken examined how hospital-based case management affected length of stay and readmission rates.44

Intervention Worth

Although current national CER discussions have not emphasized cost analyses, an examination of cost issues is relevant. Meta-analysis methods can address relationships between intervention costs and outcomes. Ideal primary intervention reports contain adequate data about intervention costs and outcomes to estimate the amount of improvement in outcome variables per unit cost. It is important that the full range of outcomes be compared to costs to provide a complete cost-benefit. Unfortunately, few existing intervention studies provide adequate cost data to include this important variable in meta-analyses. As cost information takes on greater importance in primary research, such analyses will be possible in the future.

Interpreting Meta-Analysis Results for Comparative Effectiveness Research

Meta-analysis is a powerful CER tool. Valid interpretations of meta-analyses results require researchers to consider limitations of both meta-analysis methods and primary studies. In-depth explanation of meta-analysis methods is beyond the scope of this paper. Other excellent resources provide detailed information.10–15 Two checklists with criteria for evaluating meta-analyses are available online (PRISMA: http://www.prisma-statement.org/statement.htm) (MOOSE: http://www.editorialmanager.com/jognn/account/MOOSE.pdf). This discussion will focus on CER meta-analysis.

The findings of meta-analyses may be generalized to situations similar to the primary studies included in the analyses. Thus, if only randomized controlled trials are included in meta-analyses, they may provide limited information about effectiveness while providing excellent estimates of efficacy. Since CER does not seek to determine if interventions are efficacious under highly controlled conditions, CER meta-analyses should include primary trials with varied populations and broad clinical practice, as well as tightly controlled efficacy trials, so findings are generalizable to practice settings.45,46

Limitations and Challenges of Meta-Analysis CER

Meta-analysis inclusion criteria determine which primary studies to include in aggregate analyses. Excessively narrow inclusion criteria may exclude studies conducted in practice setting which might provide the most valuable evidence for changing practice. For example, the Cochrane Collaboration emphasis on randomized controlled trials and exclusion of patient-centered outcomes may limit the usefulness of some reviews for CER.

Including studies with varied methodological difficulties can be both valuable and challenging. Meta-analysts manage primary study quality in three main ways.47 First, meta-analysts may set inclusion criteria that address methodological quality. This approach can be effective for CER if it does not exclude the very field studies that provide the best evidence about effectiveness. Second, a meta-analysis could weight effect sizes by quality scores. This approach is fraught with problems because no valid measures of primary study quality exist and the importance of specific quality attributes may differ by scientific topic.47 Third, meta-analysts may consider quality features as an empirical question. Conducting moderator analyses to examine associations between effect sizes and methods characteristics (e.g., allocation, masked outcome assessment, attrition) can be informative. For example, Lee, Soeken, and Picot compared effect sizes of studies with strong internal validity to those with significant weaknesses.48 Combination approaches may be most effective if CER research is to ensure that studies conducted in realistic clinical settings are included while testing linkages between methods and effect sizes.

Primary study limitations profoundly influence meta-analyses. Poorly described interventions are a persistent problem.49–52 Studies which describe interventions as patient education or social support, without additional details, provide insufficient information about intervention content. Other studies use well known labels for interventions but provide insufficient evidence about intervention content or delivery. For example, studies may claim ‘motivational interviewing’ without conducting an intervention entirely consistent with motivational interviewing principles. Inadequate details about interventions, and outcomes, make valid coding difficult for some primary studies and may necessitate exclusion from meta-analyses.

Reporting bias, the tendency for articles to report statistically significant findings and not report findings that are not statistically significant, and publication bias, the tendency for studies with statistically significant findings to be published, alter meta-analysis findings in unknown ways.53 Inadequate statistical information in primary studies, such as not reporting sample sizes, means, and measures of variability, is frustratingly common.54,55 Some primary studies may use outcome measures with no recognized standards for clinically relevant differences, hindering meaningful interpretation.

Perhaps the most common limitation in published meta-analyses is inadequate searching for primary studies. This is important since easier-to-find studies generally have larger effect sizes than obscure studies.56,57 Publication bias is a persistent problem that thwarts scientific progress.57,58 Considerable resources must be devoted to adequate searching to ensure valid CER meta-analyses.56

Meta-analysts can only synthesize existing information. For example, some populations may be under-represented in research.59 The comprehensive searching completed for valid meta-analyses allows investigators to identify missing populations.

Individual studies are the unit of analysis in meta-analyses. To ensure independent data (subjects do not enter any one meta-analysis statistical procedure multiple times), meta-analysts must make principled decisions regarding which measures to use, or create an index score when studies report multiple measures of the same construct. Procedures also must be in place to ensure that the same subjects do not enter meta-analysis effect sizes multiple times when more than one article reports on the same subjects.

Use of CER Meta-Analysis Results

In some CER meta-analyses, moderator analyses may be more important than overall effect sizes. Researchers should place less emphasis on overall effects in meta-analyses that include significant clinical and methodological diversity. Researchers should use caution when interpreting overall effect sizes of small meta-analyses with significant heterogeneity and no explanatory moderator analyses.42

CER meta-analysis results may be conclusive regarding best practices if primary studies offer strong and consistent evidence. In these situations, no further research comparing interventions may be necessary. Primary research often yields less conclusive findings when few studies are available, all studies have significant methodological weaknesses, or extensive heterogeneity cannot be explored through moderator analyses. In these situations, meta-analysis may contribute most by identifying comparisons that further research should address. Rather than simply suggesting additional research on a topic, meta-analyses usually can specify the nature of the comparisons that should be made (e.g., intervention characteristics, samples).

Comprehensive meta-analyses can provide evidence for practice. Consistent findings across multiple meta-analyses that address the same fundamental research question provide powerful evidence for practice. For example, three meta-analyses have documented that behavioral interventions are more powerful than cognitive interventions to change physical activity behavior among healthy, chronically ill, and older adults.41,60,61 Contradictory findings across multiple meta-analyses should be carefully evaluated. Considerations include: differences in search strategies, inclusion criteria, and outcome variables to identify potential sources of discrepancies prior to making practice recommendations.

Meta-analyses must be updated with newly available evidence. The shelf-life of meta-analyses depends on the amount of new evidence that could change findings.59 A meta-analysis may suggest comparisons to make in primary studies, the findings of which could require updates to the seminal meta-analysis. Newer studies may include populations that older studies included infrequently. Important methodological advances may affect the results of more recent studies. Emerging data should be included in updated meta-analyses.7 Meta-analyses may also need to be updated as new methods of meta-analyzing data become available.62

Conclusions

Meta-analyses can address central CER questions of which interventions work best, for whom, in what situations, and at what cost. Moderator analyses that compare intervention characteristics, patient attributes, and clinical circumstances on clinical outcomes make the largest CER contribution to knowledge for practice. These moderator analyses typically answer questions that primary studies never ask, meta-analyses can make unique contributions to scientific knowledge of health interventions. Methodological challenges and weaknesses in extant primary research should provide the context for interpreting findings. Rigorously conducted meta-analyses are a useful method for conducting valid CER.

Acknowledgments

Financial support provided by grants from the National Institutes of Health (R01NR009656 & R01NR011990) to Vicki Conn, principal investigator. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • 1.IOM Roundtable on Evidence Based Medicine. Learning what works best: the nation’s need for evidence on comparative effectiveness in health care. Institute of Medicine; 2007. Available at: http://www.iom.edu/~/media/Files/Activity%20Files/Quality/VSRT/ComparativeEffectivenessWhitePaperF.ashx. [Google Scholar]
  • 2.Donnelly J, Garber AM, Wilensky GR, Dentzer S, Agres T. Health policy brief: comparative effectiveness research. Health Affairs. 2010 Oct;5:5p. [Google Scholar]
  • 3.Fisher ES, Bynum JP, Skinner JS. Slowing the growth of health care costs--lessons from regional variation. N Engl J Med. 2009 Feb 26;360(9):849–52. doi: 10.1056/NEJMp0809794. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ, Lucas FL, Pinder EL. The implications of regional variations in Medicare spending. Part 2: health outcomes and satisfaction with care. Ann Intern Med. 2003 Feb 18;138(4):288–98. doi: 10.7326/0003-4819-138-4-200302180-00007. [DOI] [PubMed] [Google Scholar]
  • 5.Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ, Lucas FL, Pinder EL. The implications of regional variations in Medicare spending. Part 1: the content, quality, and accessibility of care. Ann Intern Med. 2003 Feb 18;138(4):273–87. doi: 10.7326/0003-4819-138-4-200302180-00006. [DOI] [PubMed] [Google Scholar]
  • 6.Clancy C. Patient-centered outcomes research: what is it and why do we need it?. Presented at: Council for the Advancement of Nursing Science special topics conference; 2011 October 12; Washington, DC. [Google Scholar]
  • 7.DuBois RW, Kindermann SL. Demystifying comparative effectiveness research: a case study learning guide [Internet] National Pharmaceutical Council; 2009. Nov, p. 42. Available from: http://www.npcnow.org/Public/Research___Publications/Publications/pub_ebm/Demystifying_Comparative_Effectiveness_Research__A_Case_Study_Learning_Guide_.aspx. [Google Scholar]
  • 8.Horn SD, Gassaway J. Practice based evidence: incorporating clinical heterogeneity and patient-reported outcomes for comparative effectiveness research. Med Care. 2010 Jun;48(6 Suppl):S17–22. doi: 10.1097/MLR.0b013e3181d57473. [DOI] [PubMed] [Google Scholar]
  • 9.Manchikanti L, Falco FJ, Boswell MV, Hirsch JA. Facts, fallacies, and politics of comparative effectiveness research: Part I. Basic considerations. Pain Physician. 2010 Jan-Feb;13(1):E23–54. [PubMed] [Google Scholar]
  • 10.Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and meta-analysis. 2. New York: Russell Sage Foundation; 2009. [Google Scholar]
  • 11.Cooper H. Research synthesis and meta-analysis: a step by step approach. 4. Los Angeles: SAGE Publications, Inc; 2010. [Google Scholar]
  • 12.Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to meta-analysis. West Sussex: John Wiley & Sons; 2009. [Google Scholar]
  • 13.Lipsey M, Wilson D. Practical meta-analysis. Los Angeles: SAGE Publications, Inc; 2000. [Google Scholar]
  • 14.Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions [Internet] United Kingdom: Cochrane Collaboration; 2011. 2012. Available from: http://www.cochrane.org/training/cochrane-handbook. [Google Scholar]
  • 15.Campbell Collaboration [Internet] Oslo, Norway: Campbell Collaboration; 2012. Available from: http://www.campbellcollaboration.org/ [Google Scholar]
  • 16.Napolitano MA, Fotheringham M, Tate D, Sciamanna C, Leslie E, Owen N, Bauman A, Marcus B. Evaluation of an internet-based physical activity intervention: a preliminary investigation. Ann Behav Med. 2003 Spring;25(2):92–9. doi: 10.1207/S15324796ABM2502_04. [DOI] [PubMed] [Google Scholar]
  • 17.Furukawa F, Kazuma K, Kawa M, Miyashita M, Niiro K, Kusukawa R, Kojima M. Effects of an off-site walking program on energy expenditure, serum lipids, and glucose metabolism in middle-aged women. Biol Res Nurs. 2003 Jan;4(3):181–92. doi: 10.1177/1099800402239623. [DOI] [PubMed] [Google Scholar]
  • 18.Hubball HT. Development and evaluation of a worksite health promotion program: application of critical self-directed learning for exercise behaviour change [dissertation] Vancouver: The University of British Columbia; 1996. p. 329. [Google Scholar]
  • 19.Nichols GJ. Testing a culturally consistent behavioral outcomes strategy for cardiovascular disease risk reduction and prevention in low income African-American women [dissertation] Baltimore: University of Maryland; 1995. p. 307. [Google Scholar]
  • 20.Blanchard CM, Fortier M, Sweet S, O’Sullivan T, Hogg W, Reid RD, Sigal RJ. Explaining physical activity levels from a self-efficacy perspective: the physical activity counseling trial. Ann Behav Med. 2007 Nov-Dec;34(3):323–8. doi: 10.1007/BF02874557. [DOI] [PubMed] [Google Scholar]
  • 21.Annesi JJ. Effects of music, television, and a combination entertainment system on distraction, exercise adherence, and physical output in adults. Can J Behav Sci. 2001;33(3):193–202. [Google Scholar]
  • 22.King AC, Baumann K, O’Sullivan P, Wilcox S, Castro C. Effects of moderate-intensity exercise on physiological, behavioral, and emotional responses to family caregiving: a randomized controlled trial. J Gerontol A Biol Sci Med Sci. 2003 Jan;57(1):M26–36. doi: 10.1093/gerona/57.1.m26. [DOI] [PubMed] [Google Scholar]
  • 23.Bennett JA, Young HM, Nail LM, Winters-Stone K, Hanson G. A telephone-only motivational intervention to increase physical activity in rural adults: a randomized controlled trial. Nurs Res. 2008 Jan-Feb;57(1):24–32. doi: 10.1097/01.NNR.0000280661.34502.c1. [DOI] [PubMed] [Google Scholar]
  • 24.Raber AC. Empowering women: a health promotion program for weight-related problems [dissertation] Bowling Green State University; 2004. p. 354. [Google Scholar]
  • 25.King AC, Friedman R, Marcus B, Castro C, Napolitano M, Ahn D, Baker L. Ongoing physical activity advice by humans versus computers: the Community Health Advice by Telephone (CHAT) trial. Health Psychol. 2007 Nov;26(6):718–27. doi: 10.1037/0278-6133.26.6.718. [DOI] [PubMed] [Google Scholar]
  • 26.Conn VS, Hafdahl AR, Mehr DR, LeMaster JW, Brown SA, Nielsen PJ. Metabolic effects of interventions to increase exercise in adults with type 2 diabetes. Diabetologia. 2007 May;50(5):913–21. doi: 10.1007/s00125-007-0625-0. [DOI] [PubMed] [Google Scholar]
  • 27.Borestein M. Effect size for continuous data. In: Cooper H, Hedges L, Valentine J, editors. Handbook of research synthesis and meta-analysis. 2. New York: Russell Sage Foundation; 2009. pp. 221–235. [Google Scholar]
  • 28.Brancato RM, Church S, Stone PW. A meta-analysis of passive descent versus immediate pushing in nulliparous women with epidural analgesia in the second stage of labor. J Obstet Gynecol Neonatal Nurs. 2008 Jan-Feb;37(1):4–12. doi: 10.1111/j.1552-6909.2007.00205.x. [DOI] [PubMed] [Google Scholar]
  • 29.Gu MO, Conn VS. Meta-analysis of the effects of exercise interventions on functional status in older adults. Res Nurs Health. 2008 Dec;31(6):594–603. doi: 10.1002/nur.20290. [DOI] [PubMed] [Google Scholar]
  • 30.Navathe AS, Clancy C, Glied S. Advancing research data infrastructure for patient-centered outcomes research. JAMA. 2011 Sep 21;306(11):1254–5. doi: 10.1001/jama.2011.1341. [DOI] [PubMed] [Google Scholar]
  • 31.Lo SF, Chang CJ, Hu WY, Hayter M, Chang YT. The effectiveness of silver-releasing dressings in the management of non-healing chronic wounds: a meta-analysis. J Clin Nurs. 2009 Mar;18(5):716–28. doi: 10.1111/j.1365-2702.2008.02534.x. [DOI] [PubMed] [Google Scholar]
  • 32.Lohr KN. Comparative effectiveness research methods: symposium overview and summary. Med Care. 2010 Jun;48(6 Suppl):S3–6. doi: 10.1097/MLR.0b013e3181e10434. [DOI] [PubMed] [Google Scholar]
  • 33.Atkins D, Kupersmith J. Implementation research: a critical component of realizing the benefits of comparative effectiveness research. Am J Med. 2010 Dec;123(12 Suppl 1):e38–45. doi: 10.1016/j.amjmed.2010.10.007. [DOI] [PubMed] [Google Scholar]
  • 34.Van Kuiken D. A meta-analysis of the effect of guided imagery practice on outcomes. J Holist Nurs. 2004 Jun;22(2):164–79. doi: 10.1177/0898010104266066. [DOI] [PubMed] [Google Scholar]
  • 35.Kaplan SH, Billimek J, Sorkin DH, Ngo-Metzger Q, Greenfield S. Who can respond to treatment? Identifying patient characteristics related to heterogeneity of treatment effects. Med Care. 2010 Jun;48(6 Suppl):S9–16. doi: 10.1097/MLR.0b013e3181d99161. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Conn VS, Hafdahl AR, Cooper PS, Ruppar TM, Mehr DR, Russell CL. Interventions to improve medication adherence among older adults: meta-analysis of adherence outcomes among randomized controlled trials. Gerontologist. 2009 Aug;49(4):447–62. doi: 10.1093/geront/gnp037. [DOI] [PubMed] [Google Scholar]
  • 37.Rice VH, Stead L. Nursing intervention and smoking cessation: meta-analysis update. Heart Lung. 2006 May-Jun;35(3):147–63. doi: 10.1016/j.hrtlng.2006.01.001. [DOI] [PubMed] [Google Scholar]
  • 38.Committee on Comparative Effectiveness Research Prioritization, Institute of Medicine. Initial national priorities of comparative effectiveness research. Washington DC: National Academies Press; 2009. [Google Scholar]
  • 39.Oh H, Seo W. Meta-analysis of the effects of respiratory rehabilitation programmes on exercise capacity in accordance with programme characteristics. J Clin Nurs. 2007 Jan;16(1):3–15. doi: 10.1111/j.1365-2702.2005.01387.x. [DOI] [PubMed] [Google Scholar]
  • 40.Jung D, Lee J, Lee SM. A meta-analysis of fear of falling treatment programs for the elderly. West J Nurs Res. 2009 Feb;31(1):6–16. doi: 10.1177/0193945908320466. [DOI] [PubMed] [Google Scholar]
  • 41.Conn VS, Hafdahl AR, Mehr DR. Interventions to increase physical activity among healthy adults: meta-analysis of outcomes. Am J Public Health. 2011 Apr;101(4):751–8. doi: 10.2105/AJPH.2010.194381. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011 Nov;64(11):1187–97. doi: 10.1016/j.jclinepi.2010.08.010. [DOI] [PubMed] [Google Scholar]
  • 43.Conn VS, Groves P. Protecting the power of interventions through proper reporting. Nurs Outlook. 2011 Nov;59(6):318–25. doi: 10.1016/j.outlook.2011.06.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Kim Y-J, Soeken KL. A meta-analysis of the effect of hospital-based case management on hospital length-of-stay and readmission. Nurs Res. 2005 Jul-Aug;54(4):255–64. doi: 10.1097/00006199-200507000-00007. [DOI] [PubMed] [Google Scholar]
  • 45.Gibbons RJ, Gardner TJ, Anderson JL, Goldstein LB, Weintraub WS, Yancy CW. The American Heart Association’s principles for comparative effectiveness research: a policy statement from the American Heart Association. Circulation. 2009 Jun 9;119(22):2955–62. doi: 10.1161/CIRCULATIONAHA.109.192518. [DOI] [PubMed] [Google Scholar]
  • 46.Hadler NM, McNutt RA. Mahar M, editor. The illusory side of “comparative effectiveness research” [Internet] Health Beat. 2011 Apr 18; Available from: http://www.healthbeatblog.com/2011/04/the-illusory-side-of-comparativeeffectiveness-research-.html.
  • 47.Conn VS, Rantz MJ. Research methods: managing primary study quality in metaanalyses. Res Nurs Health. 2003 Aug;26(4):322–33. doi: 10.1002/nur.10092. [DOI] [PubMed] [Google Scholar]
  • 48.Lee J, Soeken K, Picot SJ. A meta-analysis of interventions for informal stroke caregivers. West J Nurs Res. 2007 Apr;29(3):344–56. doi: 10.1177/0193945906296564. discussion 357–364. [DOI] [PubMed] [Google Scholar]
  • 49.Conn VS, Cooper PS, Ruppar TM, Russell CL. Searching for the intervention in intervention research reports. J Nurs Scholarsh. 2008;40(1):52–9. doi: 10.1111/j.1547-5069.2007.00206.x. [DOI] [PubMed] [Google Scholar]
  • 50.McGilton KS, Boscart V, Fox M, Sidani S, Rochon E, Sorin-Peters R. A systematic review of the effectiveness of communication interventions for health care providers caring for patients in residential care settings. Worldviews Evid Based Nurs. 2009;6(3):149–59. doi: 10.1111/j.1741-6787.2009.00155.x. [DOI] [PubMed] [Google Scholar]
  • 51.Forbes A. Clinical intervention research in nursing. Int J Nurs Stud. 2009 Apr;46(4):557–68. doi: 10.1016/j.ijnurstu.2008.08.012. [DOI] [PubMed] [Google Scholar]
  • 52.Conn VS. Intervention? What intervention? West J Nurs Res. 2007 Aug;29(5):521–2. doi: 10.1177/0193945907301909. [DOI] [PubMed] [Google Scholar]
  • 53.Smyth RMD, Kirkham JJ, Jacoby A, Altman DG, Gamble C, Williamson PR. Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ. 2011 Jan 6;342:c7153. doi: 10.1136/bmj.c7153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Orwin R, Vevea J. Evaluating coding decisions. In: Cooper H, Hedges L, Valentine J, editors. The handbook of research synthesis and meta-analysis. 2. New York: Russell Sage Foundation; 2009. pp. 177–203. [Google Scholar]
  • 55.Pigott T. Handling missing data. In: Cooper H, Hedges L, Valentine J, editors. The handbook of research synthesis and meta-analysis. 2. New York: Russell Sage Foundation; 2009. pp. 399–416. [Google Scholar]
  • 56.Conn V, Isaramalai S, Rath S, Jantarakupt P, Wadhawan R, Dash Y. Beyond MEDLINE for literature searches. J Nurs Scholarsh. 2003;35(2):177–82. doi: 10.1111/j.1547-5069.2003.00177.x. [DOI] [PubMed] [Google Scholar]
  • 57.Conn VS, Valentine JC, Cooper HM, Rantz MJ. Grey literature in meta-analyses. Nurs Res. 2003 Jul-Aug;52(4):256–61. doi: 10.1097/00006199-200307000-00008. [DOI] [PubMed] [Google Scholar]
  • 58.Dickersin K. Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm. In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis. West Sussex, United Kingdom: John Wiley & Sons, Ltd; 2006. pp. 9–33. [Google Scholar]
  • 59.Jones JB, Blecker S, Shah NR. Meta-analysis 101: What you want to know in the era of comparative effectiveness. American Health and Drug Benefits [Internet] 2008 Apr; Available from: http://www.ahdbonline.com/feature/meta-analysis-101-what-youwant-know-era-comparative-effectiveness. [PMC free article] [PubMed]
  • 60.Conn VS, Hafdahl AR, Brown SA, Brown LM. Meta-analysis of patient education interventions to increase physical activity among chronically ill adults. Patient Educ Couns. 2008 Feb;70(2):157–72. doi: 10.1016/j.pec.2007.10.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Conn VS, Valentine JC, Cooper HM. Interventions to increase physical activity among aging adults: a meta-analysis. Ann Behav Med. 2002;24(3):190–200. doi: 10.1207/S15324796ABM2403_04. [DOI] [PubMed] [Google Scholar]
  • 62.Berry D, Wathen JK, Newell M. Bayesian model averaging in meta-analysis: vitamin E supplementation and mortality. Clin Trials. 2009 Feb;6(1):28–41. doi: 10.1177/1740774508101279. [DOI] [PubMed] [Google Scholar]

RESOURCES