Effect Sizes and Primary Outcomes in Large-Budget, Cardiovascular-Related Behavioral Randomized Controlled Trials Funded by NIH Since 1980

Abstract

Purpose

We reviewed large-budget, National Institutes of Health (NIH)-supported randomized controlled trials (RCTs) with behavioral interventions to assess (1) publication rates, (2) trial registration, (3) use of objective measures, (4) significant behavior and physiological change, and (5) effect sizes.

Methods

We identified large-budget grants (>$500,000/year) funded by NIH (National Heart Lung and Blood Institute (NHLBI) or National Institute of Diabetes & Digestive and Kidney Diseases (NIDDK)) for cardiovascular disease (dates January 1, 1980 to December 31, 2012). Among 106 grants that potentially met inclusion criteria, 20 studies were not published and 48 publications were excluded, leaving 38 publications for analysis. ClinicalTrials.gov abstracts were used to determine whether outcome measures had been pre-specified.

Results

Three fourths of trials were registered in ClinicalTrials.gov and all published pre-specified outcomes. Twenty-six trials reported a behavioral outcome with 81 % reporting significant improvements for the target behavior. Thirty-two trials reported a physiological outcome. All were objectively measured, and 81 % reported significant benefit. Seventeen trials reported morbidity outcomes, and seven reported a significant benefit. Nine trials assessed mortality, and all were null for this outcome.

Conclusions

Behavioral trials complied with trial registration standards. Most reported a physiological benefit, but few documented morbidity or mortality benefits.

Introduction

Modern approaches to the prevention and management of most chronic diseases, including diabetes and coronary heart disease, require modification of behaviors [1–3]. A few behaviors including physical inactivity, smoking, and poor diet result in four chronic diseases, including cardiovascular disease, that account for 82 % of all noncommunicable disease deaths in the world today [4]. Guidelines for the management of high cholesterol and high blood pressure advise a trial of lifestyle modification before the initiation of medication [5, 6].

Despite their promise, formal behavioral interventions remain underutilized in health care practice. Phillips and colleagues recently reported that primary care patients reported an average of 5.8 unhealthy behaviors and mental health risk factors [7]. Patients wanted to change at least one behavior and to discuss their risks with their physician. Nearly 85 % wanted to change fruit and vegetable consumption, and nearly 80 % wanted to discuss weight management [7]. Even though there is a strong desire for behavioral intervention, formal intervention, beyond mere advice, remains uncommon in medical practice. Even among current smokers, about half were not given advice to quit by their primary care doctors [8].

A possible explanation for the underutilization of behavioral interventions is the assumption that pharmacological treatments are highly effective, whereas behavioral treatments are less likely to achieve positive outcomes. Systematic reviews, however, show that positive results in pharmacological trials are actually quite rare [9]. Goldacre [10] and Sumner [11] found that media reports of clinical research are often exaggerated, not based on strong research designs, or overgeneralized. Further, for every 5, 000–10,000 promising new molecules, only one makes it through all of the review processes required to achieve FDA licensure [12]. Even among the one in 10,000 licensed drugs, lack of efficacy is often discovered in post-marketing studies. A recent analysis of trials funded by the National Heart Lung and Blood Institute (NHLBI) found that evaluated pharmacological treatments were effective in only 40 % of systematic trials [13]. Failure rates were much higher in large-budget randomized controlled trials [9, 13].

These previous analyses have not focused exclusively on behavioral trials, even though behavioral approaches have been evaluated in a significant number of randomized controlled trials [9, 13]. In this article, we consider the evidence for the benefits of behavioral intervention. Specifically, we address three questions: (1) Is there evidence that health behavior can be modified? (2) Does the change produced by behavioral intervention result in physiological change? and (3) Do behavioral interventions affect morbidity and mortality?

In order to address these questions, we need a comprehensive look at the published literature. One difficulty in assessing the literature, however, is that journals favor publication of positive results, and many trials that produce null results are likely to go unpublished [14]. A second challenge is that investigators may have the option of choosing between many outcome measures when they publish their findings, thus inflating the probability of reporting a spurious result [9].

In this analysis, we systematically reviewed large-budget (defined as costs>$500,000 in at least one grant year) randomized controlled trials relevant to cardiovascular disease and funded by the NHLBI or the National Institute of Diabetes & Digestive and Kidney Diseases (NIDDK). Cardiovascular trials were selected because numerous behavioral interventions have been implemented to improve health behaviors that affect cardiovascular health such as nutrition, physical activity, medication adherence, and smoking prevention. The advantage of systematically reviewing grant databases was that we were able to locate funded trials prior to their initiation and determine if there was bias due to non-publication. Further, we were able to use the registration service, ClinicalTrials.gov, to determine whether outcome variables were pre-specified prior to the publication of the trial. Using these safeguards allows us to provide a less biased estimate of the value of behavioral intervention. In interpreting outcomes of large-budget behavioral intervention trials, we considered several factors, including (1) evidence of publication bias, (2) trial registration prior to publication, (3) the use of objective outcome measures, (4) evidence for significant behavior change, (5) evidence for significant physiological change, and (6) effect sizes for change.

Methods

Sample of Studies

Our analysis focused on RCTs that involved behavioral interventions funded between 1980 and 2012. We focused on trials that were awarded higher dollar amounts because NHLBI has shown that virtually all of these trials are eventually published, thus eliminating bias due to selective publication [13]. The search process is summarized in a PRISMA diagram (Fig. 1). Two independent searches were conducted—one by the study authors and the second by NHLBI [13]. We searched three different NIH grant databases (QVR, REPORTER, and CRISP) for RCTs that were primarily funded or administered by NHLBI or NIDDK. Because of the relationship between diabetes and cardiovascular disease, several of the cardiovascular, behavioral interventions were administered by NIDDK, such as the Diabetes Prevention Program (DPP) [15] and Look Ahead [3]. We selected the start year of 1980 because study abstracts were more often available in the grant databases after 1980. Inclusion criteria were as follows: RCT for studies from 1980 to 2012; direct costs funded were large enough to require special authorization (>$500,000/ year in at least 1 year of the grant); the word “trial” had to appear in the study abstract; and primary outcome was a cardiovascular risk factor, event, or death. Exclusion criteria included the following: no human subjects protocol required; pediatric studies; animal studies; non-RCTs (e.g., observational, cohort, case control, genetic or proteomics, measurement, basic clinical research); or interventions that were not behavioral (e.g, drugs, supplements, devices, surgeries).

Fig. 1.

PRISMA flow diagram of grant abstracts reviewed, excluded, and retained in final analyses

A second independent search was conducted by NHLBI, Division of Cardiovascular Sciences, to identify clinical trials with budgets requiring special authorization funded through 2012 [13]. The NHLBI list allowed us to check our search against an objective criterion. The authors reviewed each discrepant case jointly, and studies became part of this evaluation if there was consensus on their inclusion criteria.

Identification of Publication

We searched bibliographic databases, PubMed and Google Scholar, as well as publically available grant registries, NIH Reporter and ClinicalTrials.gov, for outcome papers for each grant. Usually, a single paper included results for all the main outcomes. Sometimes two or three papers per trial were needed to obtain results for all primary outcomes or sufficient details to calculate effect sizes. If needed, trial investigators were contacted to clarify the details of the study and which, if any, of their publications matched the grant. We contacted a total of five principal investigators (1) to confirm if the intervention had incorporated a behavior modification, (2) to check if the main outcome had been published, or (3) to clarify the main outcome findings.

Selection of Behavioral Intervention

For each study, we considered the comparison between a treatment and a control condition. If there were multiple arms in the trial, we analyzed the two arms that had the strongest contrast in terms of dose of the intervention or most different in outcomes following intervention.

Identification of Primary Outcomes

For each study, we identified the primary outcome variable for each of the following constructs: behavioral (e.g., diet intake, physical activity, medication adherence, smoking, goal tracking), physiological (e.g., weight, blood pressure, VO₂ max, LDL-C), morbidity (e.g., formal disease diagnosis, CVD and CHD events, hospital stays, start of or increased use of medication as a result of disease diagnosis), and all-cause mortality outcomes. In many studies, the primary outcome was identified in the main outcome paper or on ClinicalTrials.gov. When authors discussed multiple primary outcomes, we selected the behavioral, physiological, and morbidity outcome more proximal to the intervention target (e.g., nutrient intake for dietary interventions). For each outcome, we reported whether the instrument used was objective or self-report. We defined objective as measures observed by an impartial third party or device. Primary outcomes identified can be cross-checked between our appendix tables with the original sources in the reference list.

For trials with multiple follow-up periods, we selected the follow-up period after the end of the intensive intervention period for behavior and physiological outcomes and the longest follow-up period for morbidity and mortality outcomes. Post-intervention was chosen because it was the point that we would expect the strongest impact of the behavioral intervention. We chose to include the final follow-up point because it was the most rigorous test of the durability of the intervention.

Sample sizes for populations, mean and standard error (or standard deviations) for outcomes, number of events, and number of deaths for each trial are listed in the appendix tables.

Additional Variables

We coded the following variables: registration in ClinicalTrials.gov prior to publication, start year (earliest funding noted), publication year of main outcome, and type of comparator (less intense intervention, usual care, or assessment only). We reported the change in primary outcome in original units (e.g., kg, mmHg) between treatment and control group.

Analysis

Each trial was categorized as showing significant benefit or as having non-significant (null) effects for each type of primary outcome and for total mortality (assuming p<0.05 as benefit). Bi-variate analyses were conducted using chi-square with p<0.05 as significant to test group differences. When sufficient data were published, we re-calculated effect sizes for the behavioral and physiological outcomes using the effect size calculator developed by the Campbell Collaboration [16].

Effect sizes were derived from the mean, standard deviation, standard error, confidence intervals, or proportions provided in the outcome paper. The absolute effect sizes for behavioral and physiological outcomes were plotted against one another. We considered an effect size small (0.2), medium (0.5), and large (0.8) for both continuous and proportion calculations [17]. Meta-analyses were conducted separately for the following types of interventions which had a minimum of four studies to pool: single health behavior intervention for nutrition or physical activity, multiple lifestyle interventions incorporating nutrition and physical activity, and blood pressure or heart monitoring trials. Meta-analysis was conducted using STATA-12 using the metan procedure and a random effects model. Studies were weighted by the standard error and then rerun weighted by sample size.

This study was determined exempt from review by the National Institutes of Health, Office of Human Subjects Research.

Results

A total of 5,828 grant years were identified through our searches (see Fig. 1). We removed multiple years of funding (n=3,315) and multiple research sites or ancillary studies of the same grant (n=933), which left a total of 1,580 abstracts for review. Using our pre-specified search criteria, we excluded 1,474 grant abstracts (see Appendix Table 1 for specific details). The most common reasons for exclusion included the following: study design was not an RCT, trial was still active, or the focus was not cardiovascular. We searched for main outcome papers for 106 grants; 20 were not found and assumed not published. An additional 48 full-text articles were excluded for not matching search criteria such as not an RCT, or a drug or community trial, or a duplicate with other trials previously identified (details in Appendix Table 1) which left 38 trials with published main outcome papers. Twenty-four trials were uniquely identified in our search. Five trials were found only through the NHLBI search. Nine trials overlapped between the two searches.

Table 1 and Appendix Table 2 list details of the trials included in the review. Approximately 26 % of the studies were funded in the late 1970s and 1980s, another 26 % funded in the 1990s and just under half of the trials were funded in 2000 or later. Sample sizes ranged from 79 to 48,835. Over half of the trials included a multi-factor or lifestyle intervention that included at least two of the following: nutrition or diet, physical activity, and smoking cessation (n=18). Nine trials intervened on exercise only (n=5) or nutrition only (n=4). Other interventions involved blood pressure or heart failure monitoring (n=4), improving communication with clinicians (n=2), medication adherence (n=1), cognitive behavioral therapy (n=1), self-management counseling (n=1), smoking cessation (n=1), or health literacy (n=1). Although we were not evaluating the effect of medications, interventions to increase medication adherence and monitoring heart failure were included because they modified a behavior that ultimately leads to optimal exposure of a surgery or drug treatment.

Table 1.

Selected trial details of the 38 included behavioral interventions

Trial	Intervention	Comparator	Intervention length (intensive/full)	Participant description	N size intervention	N size comparator
ACT [22]	Exercise	Brief provider advice	24 months/24 months	Inactive adults (women)	130	133
Be Fit, Be Well [32]	Multi-factor (diet, exercise)	Usual care and education materials	12 months/24 months	Obese patients receiving hypertension treatment	148	166
BPTEACH [33]	Patient education to ask clinicians	Usual care	6 months	African American hypertensives	43	57
DEER [34]b	Multi-factor (diet, exercise)	Assessment only	3 months/12 months	Postmenopausal adult women	43	45
DISH [35]c	Diet for weight control	No medication control	56 weeks	Participants in previous heart disease trial	87	89
DPP [15, 36]a	Multi-factor (diet, exercise)	Placebo	24 weeks/2.8 years	Pre-diabetic	1,079	1,082
ENRICHD [37]	Cognitive behavioral therapy	Education materials	6 months, 9 months	Patients with MI in past 28 days	1,238	1,343
HARP [38]	Med adherence	Cancer control	6 months	Hypertensive adults	221	213
HART [39]	Self-management counseling	Heart failure education	12 months	Patients with mild to moderate heart failure	451	451
HCP [40]	Nutrition	Discontinued drug use	4 years	Participants in previous heart disease trial	97	44
Health Literacy [41, 42]a	Health literacy	Single session health literacy	1 months, 12 months	Patients with heart failure	303	302
Help PD [43]d	Multi-factor (diet, exercise)	Usual care + registered dietician advice	6 months, 24 months	Pre-diabetic	151	150
HF-ACTION [44]a	Exercise	Education materials	3 months, 12 months	Patients with heart failure	1,159	1,172
HOME_BP [45]	Home blood pressure monitor	Home blood pressure monitor and log	3 months	High-risk African American patients	221	217
HOPP [46]	Smoking cessation	Education materials	6 months, 7 months	Pregnant smokers	306	297
HPT [47]	Diet (sodium restriction)	Assessment only	10 weeks, 3 years	Adults with mid-range blood pressure	196	196
Htn Prev [48]	Multi-factor (diet, exercise, sodium)	Usual care	5 years	Adults with mild hypertension	102	99
ICAN [49]	Multi-factor (diet, exercise)	Usual care	12 months	Obese, type 2 diabetics	73	71
IN CONTROL [50]	Blood pressure monitor	Usual care	3 months	Adults with elevated blood pressure	209	212
iReach [23]e	Multi-factor (diet, exercise)	In person	6 months	Overweight adults	158	161
Look Ahead [2, 3, 51, 52]a	Multi-factor (diet, exercise)	Usual care and diabetes education	6 months, 4 years	Overweight/obese, type 2 diabetics	2,570	2,575
Mediterranean Lifestyle [53–55]	Multi-factor (diet, exercise)	Usual care	6 months	Post-menopausal women, type 2 diabetics	163	116
MRFIT [56–59]	Multi-factor (smoking, diet)	Usual care	4 months, 6 years	Men at risk of CHD death but no clinical evidence	6,428	6,438
Optimal Exercise Regimens [60]	Exercise	Assessment only	12 months	Sedentary adults (men reported here)	40	41
PAD_RF [31]	Patient education to ask clinicians	Attention control	12 months/12 months	Patients with PAD	97	111
PAD Treadmill [61]	Exercise	Assessment only	6 months	Patients with peripheral artery disease	51	53
POWER [62]	Multi-factor (diet, exercise) remote counseling	Usual care	6 months, 24 months	Obese adults	139	138
POWER-UP [63]	Multi-factor (diet, exercise) Brief lifestyle counseling	Usual care and quarterly counseling	12 months, 24 months	Obese adults	131	130
PREMIER [64]	Multi-factor (diet, exercise, sodium) Established + DASH diet	Education materials and 1-time counseling	6 months, 18 months	Adults with untreated pre- or stage 1 hypertension	269	273
SCRIP [65]	Multi-factor (diet, exercise, sodium, smoking)	Usual care	4 years	Adults with atherosclerosis	145	155
SWCP [66]f	Multi-factor (diet, exercise)	Assessment only	3 months, 12 months	Moderately overweight (men	39	40
TCYB [67]	Blood pressure monitor	Usual care	24 months	Hypertensive adults	159	159
TELE-HF [68]	Telemonitoring	Education materials	180 days	Recently hospitalized for heart failure	826	827
TOHP [69]	Multi-factor (diet, exercise)	Usual care	14 months, 3–4 years	Recent weight loss participants	595	596
TOURS [70]	Multi-factor (diet, exercise)	Education materials	12 months	Obese women in rural areas who recently completed lifestyle intervention	83	79
Training level comparison [71]	High-intensity exercise	Low-intensity exercise	12 months	Male adults with coronary heart disease	103	82
WHI-DM [72]g	Nutrition personal contact	Education materials	1 year, 6.1 years	Overweight or obese with hypertension, dyslipidemia	19,541	29,294
WLM [73]	Multi-factor (diet, exercise)	Self-directed maintenance	30 months	Postmenopausal women	341	341

See online Appendix Table 2 for further details. Data in cells are study acronym (see footnote at end), intervention and comparator detail treatment and control arms. Intervention length is described as the intensive period (if any) and the full intervention period. Participant description and sample sizes for both arms. Trial titles from registry or publications for each acronym are provided. Some trials did not provide a short title or acronym; therefore, study authors created a condensed title

ACT [22] activity counseling trial, Be Fit, Be Well [32] evaluating a blood pressure reduction and weight loss program in a low-income, ethnically diverse population, BPTEACH [33] Baltimore partnership to educate and achieve control of hypertension, DEER [34] diet and exercise for elevated risk, DISH [35] dietary intervention study for hypertension, DPP [15, 36] diabetes prevention program, ENRICHD [37] enhancing recovery in coronary heart disease patients, HARP [38] hypertension and adherence in rural practice, HART [39] heart failure adherence and retention randomized behavioral trial, HCP [40] hypertension control program, Health Literacy [41, 42] health literacy and self-management in heart failure, Help PD [43] healthy living partnerships to prevent diabetes, HF-ACTION [44] heart failure: a controlled trial investigating outcomes of exercise training (HF-ACTION), HOME_BP [45] home-based blood pressure interventions for African Americans, HOPP [46] healthy options for pregnancy and parenting, HPT [47] hyper-tension prevention trial, Htn Prev [48] primary prevention of hypertension by nutritional-hygienic means, ICAN [49] improving control with activity and nutrition, IN CONTROL [50] hypertension reduction in inner city Seattle, iReach [23] Internet-assisted obesity treatment, Look Ahead [2, 3, 51, 52] action for health in diabetes, Mediterranean Lifestyle [53–55] effect of the Mediterranean lifestyle program on multiple risk behaviors and psychosocial outcomes, MRFIT [56–59] multiple risk factor intervention trial, Optimal Exercise Regimens [60] optimal exercise regimens for persons at increased risk, PAD_RF [31] reducing risk factors in peripheral arterial disease, PAD Treadmill [61] improving functioning in peripheral arterial disease, POWER [62] practice-based opportunities for weight reduction, POWER-UP [63] practice-based opportunities for weight reduction trial at the University of Pennsylvania, PREMIER [64] lifestyle intervention blood pressure control, SCRIP [65] Stanford coronary risk intervention project, SWCP [66] Stanford weight control program, TCYB [67] take control of your blood pressure study, TELE-HF [68] Yale heart failure telemonitoring study, TOHP [69] trials of hypertension prevention, phase II, TOURS [70] treatment of obesity in underserved rural settings, Training Level Comparison [71] training level comparison trial, WHI-DM [72] Women's Health Initiative randomized controlled dietary modification trial, WLM [73] weight loss maintenance randomized controlled trial

^aFor trials with this superscript, we had to extract data from multiple papers

^bThe Deer study had 12-month intervention with the first 3 months intensive but only reported 12-month data

^cDISH—two intervention arms—weight control and sodium control. Outcomes for weight control arm reported in table. For sodium control arm, behavioral outcome was urinary sodium output which was significantly improved in treatment versus controls and more of the intervention group remained normotensive but not statistically different than controls

^dHelp-PD—intensive intervention was the first 6 months. Main outcome paper reports data every 6 months but conducted statistical analyses for the 18- and 24-month data. Data reported for weight and physiological outcome (glucose) was the adjusted means over 18- and 24-month follow-up

^eiReach—study authors compared an in-person to an Internet or Internet in-person hybrid study. Authors evaluated how well an Internet delivery would do compared with an in-person version. We coded the in-person arm as the treatment arm and the Internet delivery arm as the control

^fSWCP was a 12-month intervention with the first 3 months intensive; however, only 12-month data were available in the publication

^gWHI had an intensive intervention for 12 months and then quarterly contact through the remainder of the year. Behavioral outcomes are reported at 18 months. Physiological outcomes were not published at 12 or 18 months. The closest follow-up to the end of the intensive intervention was at 3 years

Publication Bias and Prior Registration

We could not find primary outcome publications for 20 trials. The publication rate was 38 trials out of 58 (38 published, 20 not found published) for a total of 65.5 %. We conducted sub-analyses to compare publication rates among trials that registered with ClinicalTrials.gov prior to publication compared to those that did not pre-register. Prior registration in ClinicalTrials.gov would have only been available for trials active in the year 2000 or later as ClinicalTrials.gov was not launched until 1999 [18]. Eighteen of the 20 unpublished trials and 27 of the 38 included trials received some NIH funding in the year 2000 or later and would have been eligible to register, for a total of 45 trials that should have pre-registered in ClinicalTrials.gov. Overall, three fourths of trials were registered in ClinicalTrials.gov prior to publication (12 of the 18 unpublished trials and 23 of 27 published trials). Pre-specified outcomes were published in all trials that pre-registered (data not shown). Trials that were registered had a higher rate of publication (23 published/35 trials=65 %) than trials that did not prospectively register (4 published/10 trials=40 %), but these rates were not statistically different (X²=1.21, p=.27).

Length of Intervention and Follow-Up

Table 1 lists the range of length of intensive intervention and follow-up periods for each trial. Across all trials, length of intensive intervention ranged between 1 and 60 months and follow-up periods ranged from 3 to 72 months. Table 2 summarizes the durations of the intensive intervention and follow-up periods by type of intervention. The most common length of interventions for exercise and multi-behavior change trial was 6–12 months and for diet trials was 12–14 months. The lengths of follow-up were often longer in the diet and lifestyle interventions than in the exercise and blood pressure monitoring trials. Focusing just on interventions with the potential to change weight (nutrition, physical activity, multi-behavior), over 50 % of single behavior change interventions lasted more than 1 year, whereas less than half of multi-factor interventions lasted longer than 1 year (8 out of 18). Length of total follow-up period (post-baseline) was typically longer in nutrition and multi-factor interventions than in physical activity interventions. All of the nutrition interventions and 12 out of 18 multi-behavior change interventions reported a follow-up period longer than 12 months post-baseline.

Table 2.

Trial outcomes for 38 randomized controlled trials of behavioral interventions

Trial	Behavior outcome	Effect size (95 % CT)	Group difference in original units	Primary physiological outcome	Effect size (95 % CI)	Group difference in original Units	Morbidity	Mortality
ACT [22]a	Met activity recommendationsN, S	0.4 (0.03, 0.77)	25.7 versus 14.3 %	VO2 (ml/min)B, O	0.32 (0.08, 0.57)	80.7 ml/min	NR	NR
Be Fit, Be Well [32]	Medication adherence scaleB, S	0.38 (0.13, 0.63)	0.85 units different on medication adherence scale	WeightB, O	0.25 (0.01, 0.50)	1.05 kg	NR	NR
BPTEACH [33]	NR	NR	NR	Systolic blood pressureN, O	0.22 (0.18, 0.61)	9.44 mmHg	NR	NR
DEER [34]	Caloric intakeB, S	0.48 (0.07, 0.91)	172 kcal/day	LDL-CB, O	0.62 (0.19, 1.05)	12 mg/dl	NR	NR
DISH [35]b	NR	NR		WeightB, O	0.83 (0.48, 1.2)	3.5 kg	Normotensive without drugs—benefit	NR
DPP [15, 36]	Energy intakeB, S	0.23 (0.15, 0.32)	201 kcal/day	WeightB, O	0.81 (0.73, 0.90)	5.4 kg	Diagnosis of diabetes—benefit	NR
ENRICHD [37]	DepressionB, S	0.33 (0.25, 0.41)	2.8 points on Beck Depression	NR	NR	NR	Fatal or non-fatal myocardial infarction—null	Yes—null
HARP [38]	Pill countN, O	0.04 (−0.16, 0.25)	2 % more adhered	NR	NR	NR	NR	NR
HART [39]	SodiumN, S	0.31 (0.14, 0.49)	10 % more met goal	Systolic blood pressureN, O	0.11 (−0.01, 0.24)	2.6 mmHg	Heart failure or hospitalization—null	Yes—null
HCP [40]	SodiumB, O	1.12 (0.72, 1.5)	1,839 mg/day	Diastolic blood pressureB, O	0.22 (0.14, 0.58)	1.5 mmHg	Normotensive without drugs—benefit	NP
Health Literacy [41, 42]	Self-care behaviorB, S	0.56 (0.38, 0.73)	1.4 points on self-care scale	NR	NR	NR	Hospitalizations—null	Yes—null
Help PD [43]c	NR	NR	NR	Glucose mg/dlB, O	0.44 (0.22, 0.67)	4.35 mg/dl	Diagnosis of Diabetes—null	NR
HF-ACTION [44]d	6 min walkB, O	0.32 (0.2, 0.41)	14 m	VO2 (ml/kg/min)B, O	0.34 (0.25, 0.42)	0.4 mL/kg/min	CV event or hospitalization—null	Yes—null
HOME_BP [45]	NR	NR	NR	Blood pressure controlN, O	0.09 (−0.15, 0.33)	25 % of tx had bp control versus 22 % con	NR	NR
HOPP [46]	SmokingB, S	0.22 (0.03, 0.41)	9 % remained abstinent	NR	NR	NR	NR	NR
HPT [47]	SodiumB, O	0.32 (0.11, 0.53)	5.5 mmol	Blood pressureN, O	0.21 (0.0, 0.42)	1.7 mmHg	Hypertension—benefit	NP
Htn Prev [48]	SodiumB, O	0.63 (0.33, 0.99)	700 mg/day	Blood pressureB, O	0.31 (0.03, 0.60)	2 mmHg	Hypertension—benefit	NR
ICAN [49]	NR	NR	NR	WeightB, O	0.42 (0.09, 0.75)	3.0 kg	NR	NR
IN CONTROL [50]	Completed FU appointmentB, O	0.42 (0.16, 0.66)	65.1 versus 46.7 %	NR	NR	NR	NR	NR
iReach [23]e	Diet (kcal)N, O	0.03 (−0.18, 0.25)	18 kcal	Weight (kg)B, O	0.36 (0.14, 0.58)	2.1 kg	NR	NR
Look Ahead [2, 3, 51, 52]f	Physical activity (kcal/week) B, S	0.34 (0.29, 0.40)	768.9 kcal/week	Change in METS from sub-max treadmillB, O	0.59 (0.53, 0.64)	15.1 METS	Composite of cardiovascular events—null	Yes—null
Mediterranean Lifestyle [53–55]	Adherence to Mediterranean diet scaleB, S	0.77 (0.52, 1.02)	0.86 point increase with possibly scale range 0–7	Weight (kg at 6 months)B, O	0.35 (0.11, 0.59)	1.77 kg	NR	NR
MRFIT [56–59]g	Smoking cessationB, O	0.66 (0.59, 0.72)	31 versus 12 % quit rate	Blood pressure (diastolic)B, O	0.41 (0.38, 0.45)	3.8 mmHg	Definite, fatal AMI—null	Yes—null
Optimal Exercise Regimens [60]	NR	NR	NR	VO2 max (mL/kg/min)B, O	0.58 (0.14, 1.03)	2.0 ml/kg/min	NR	NR
PADRF [31]	Increased dose or use of medicationB, O	0.94 (0.61, 1.27)	53.8 versus 17.5 %	LDL-CB, O	0.36 (0.08, 0.63)	10.5 mg/dl	NR	NP
PAD Treadmill [61]	6 min walk (m)B, O	0.69 (0.29, 1.08)	36.2 m	Brachial artery flow (mm)B, O	0.22 (−0.16, 0.61)	0.06 mm	NR	NR
POWER [62]	NR	NR	NR	WeightB, O	0.89 (0.63, 1.16)	4.5 kg	Hospitalizations—null	NP
POWER-UP [63]	NR	NR	NR	WeightN, O	0.16 (0.08, 0.40)	1.1 kg	NR	NP
PREMIER [64]	Fruit and vegetable intakeB, S	0.78 (0.59, 0.97)	2.5 servings/day	Blood pressure (systolic)B, O	0.47 (0.30, 0.64)	4.3 mmHg	NP	NR
SCRIP [65]	% Kcal fatB, S	1.09 (0.82, 1.36)	7.8 % kcal fat	Minimal diameter changeB, O	0.30 (0.05, 0.55)	0.069 mm	Hospitalizations—benefit	Yes—null
SWCP [66]	Energy intakeB, S	0.92 (0.45, 1.38)	2,213 kJ/day	CHD risk (using Framingham equations)B, O	1.11 (0.64, 1.58)	−22.4 events/1,000 persons	NR	NR
TCYB [67],h	NR	NR	NR	Blood pressure controlledB, O	NR	11 % more of tx group had controlled blood pressure	NR	NP
TELE-HF [68]	NR	NR	NR	NR	NR	NR	Readmission—null	Yes–null
TOHP [69],i	NR	NR	NR	Blood pressure (systolic)B, O	NR	−2.7 mmHg	Onset of hypertension—benefit	NR
TOURS [70]	Self-monitoring recordsB, O	0.32 (0.01, 0.63)	29 more days of self-monitoring	Weight regainB, O	0.43 (0.12, 0.74)	2.5 kg	NR	NR
TLC [71]	NR	NR	Attendance	VO2 max (mL/kg/min)N, O	0.38 (−0.2, 0.77)	0.8 ml/kg/min	NR	NR
WHI-DM [72]	% fat intakeB, S	1.51 (1.49, 1.53)	10.7%	Blood pressure (diastolic)B, O	0.03 (0.01, 0.05)	−0.31 mmHG	Fatal and non-fatal CHD, CVD, stroke—null	Yes—null
WLM [73]j	% KcalN, S	NR	NR	WeightB, O	0.27 (0.12, 0.42)	1.5 kg	NR	NR

Primary outcomes and effect sizes for trials included in review. See Online Appendix Table 3 for further details. Data in cells are the primary outcome, calculated absolute effect sizes, and reported change in original trial units for the behavioral, physiological, and weight outcomes. Superscripted letters represent whether measure was objective or self-reported and whether outcome was a significant benefit or null. For morbidity outcomes, we include the specific outcome measure and whether null or significant benefit. If the trial was powered to analyze all-cause mortality, we included whether there was a significant benefit or not. Trial titles from registry or publications for each acronym are provided. Some trials did not provide a short title or acronym; therefore, study authors created a condensed title. A total of 38 trials are included. Each type of outcome (behavior, weight, physiology, morbidity, and mortality outcomes) should total to 38. Each trial was categorized as reporting a significant benefit or null finding or not reported/not powered for each type of outcome. Behavior, weight, and physiological outcomes were reported at the end of the intensive intervention. Morbidity and mortality outcomes were reported at the longest follow-up period available

^aACT—behavior effect size calculated from proportions–percent of participants meeting moderate or vigorous physical activity recommendations post–intervention

^bDISH—two intervention arms—weight control and sodium control. Outcomes for weight control arm are reported in the table. For sodium control arm, behavioral outcome was urinary sodium output which was significantly improved in treatment versus controls and more of the intervention group remained normotensive but not statistically different than controls

^cHelp-PD—intensive intervention was the first 6 months. Main outcome paper reports data every 6 months but conducted statistical analyses for the 18- and 24-month data. Data reported for weight and physiological outcome (glucose) was the adjusted means over 18- and 24-month follow-up

^dHF action—the main outcome paper reported median and IQR for the behavior (distance walked) and physiological variable (peak oxygen consumption). We calculated effect sizes based on a normal distribution assumption

^eiReach—study authors compared an in-person to an Internet or Internet in-person hybrid study. Authors evaluated how well an Internet delivery would do compared with an in-person version.We coded the in-person arm as the treatment arm and the Internet delivery arm as the control

^fLook Ahead—different FU—behavior outcome VO₂ and weight reported at 12 months; Cardiovascular events and mortality reported at final follow-up (max 13.5 years, median 9.6 years)

^gMRFIT—behavior and physiological outcomes reported at 12 months; AMIs reported at 7 years; and mortality reported at original follow-up (max 6 years)

^hTCYB—insufficient data presented to calculate blood pressure effect sizes

ⁱTOHP—insufficient data presented to calculate blood pressure effect sizes

^jWLM—insufficient data presented to calculate change in percent kcal effect sizes

NR not reported, N null, B significant benefit, H harm, O objective, S self-reported, ACT [22] activity counseling trial, Be Fit, BeWell [32] evaluating a blood pressure reduction and weight loss program in a low-income, ethnically diverse population, BPTEACH [33] Baltimore partnership to educate and achieve control of hypertension, DEER [34] diet and exercise for elevated risk, DISH [35] dietary intervention study for hypertension, DPP [15, 36] diabetes prevention program, ENRICHD [37] enhancing recovery in coronary heart disease patients, HARP [38] hypertension and adherence in rural practice, HART [39] heart failure adherence and retention randomized behavioral trial, HCP [40] hypertension control program, Health Literacy [41, 42] health literacy and self-management in heart failure, Help PD [43] healthy living partnerships to prevent diabetes, HF-ACTION [44] heart failure: a controlled trial investigating outcomes of exercise training (HF-ACTION), HOME_BP [45] home-based blood pressure interventions for African Americans, HOPP [46] healthy options for pregnancy and parenting, HPT [47] hypertension prevention trial, Htn Prev [48] primary prevention of hypertension by nutritional-hygienic means, ICAN [49] improving control with activity and nutrition, IN CONTROL [50] hypertension reduction in inner city Seattle, iReach [23] Internet-assisted obesity treatment, Look Ahead [2, 3, 51, 52] action for health in diabetes, Mediterranean Lifestyle [53–55] effect of the Mediterranean lifestyle program on multiple risk behaviors and psychosocial outcomes, MRFIT [56–59] multiple risk factor intervention trial, Optimal Exercise Regimens [60] optimal exercise regimens for persons at increased risk, PAD_RF [31] reducing risk factors in peripheral arterial disease, PAD Treadmill [61] Improving functioning in peripheral arterial disease, POWER [62] practice-based opportunities for weight reduction, POWER-UP [63] practice-based opportunities for weight reduction trial at the University of Pennsylvania, PREMIER [64] lifestyle intervention blood pressure control, SCRIP [65] Stanford coronary risk intervention project, SWCP [66] Stanford weight control program, TCYB [67] take control of your blood pressure study, TELE-HF [68] Yale heart failure tele-monitoring study, TOHP [69] trials of hypertension prevention, phase II, TOURS [70] treatment of obesity in underserved rural settings, Training Level Comparison [71] training level comparison trial,WHI-DM[72]Women's Health Initiative randomized controlled dietary modification trial, WLM [73] weight loss maintenance randomized controlled trial

Objective Measurement

Table 2 and Appendix Table 3 list the type of behavior and physiological outcome measured. Superscripts indicate if the measure was objective. Across the 38 trials, only 26 trials reported a behavioral outcome and 11 of these trials (42 %) applied a measure that met the definition of objective. Among trials that reported a behavioral outcome (n=26), 81.8 % (9 out of 11 trials) reported significant benefit outcomes based on objective assessments, whereas 80.0 % (12 out of 15 trials) reported positive results based on non-objective or self-report assessments. All trials reporting physiologic outcomes (n=32) applied objective assessments.

Significant Outcomes

Table 2 and Fig. 2 report the number of trials that reported a behavior, physiological, clinical morbidity, or mortality outcome and if the finding was statistically significant. Only 26 trials reported a primary behavioral outcome with 21 trials (81 %) reporting a significant benefit. Similarly, 32 of the 38 trials reported a primary physiological outcome, and 81 % of these trials reported a significant benefit for physiologic measures (26 out of 32 trials). Fewer trials measured morbidity outcomes (17 out of the 38 trials), and only seven reported a significant benefit in the clinical outcome for the intervention condition in comparison to the control condition. Only 9 of the 38 studies reported mortality outcomes. All studies that assessed mortality were null.

Fig. 2.

Number of behavioral intervention trials that reported a benefit or null finding for their primary behavior, weight, physiological, morbidity, and mortality outcomes

Effect Sizes

Table 2 lists the Cohen's d effect sizes that we calculated, as well as the between group differences in original units. Effect sizes could not be calculated for some trials if they did not provide sufficient details in their publications (refer to Appendix Table 3 for listing of effect sizes). About half (14 out of 25 trials) produced small effect sizes for behavioral change (Cohen's d between 0.2 and 0.5), and approximately a quarter (n=6) reported medium effect sizes (Cohen's d larger than 0.5) and one fifth (n=5) large effects (Cohen's d larger than 0.8) [16]. Approximately three fourths of trials produced small effect sizes (23 out of 30 trials), and 20 % (6 out of 30 trials) produced moderate effect sizes in their primary, physiological outcomes. Example of small to medium changes in original trial units includes changes in weight from 1 to 5.4 kg and changes in blood pressure from 1.5 to 9.4 mmHg.

We plotted the effect sizes of the primary behavioral outcome at post-test (x-axis) against the effect sizes of the primary physiological outcome at post-test (Fig. 3). Each circle on the figure represents one trial, and a solid colored circle indicates a significant benefit observed for both the behavioral and physiological outcome. Figure 3 displays 20 trials that report sufficient data to calculate effect sizes for both behavioral and physiological outcomes. The slope of the best fitting line is fairly flat (slope=−0.11). The majority of the trials (80 %) report a significant benefit for both the change in behavior and physiology; however, many of these effects fall within the small level as defined by Cohen (between 0.2 and 0.5).

Fig. 3.

Scatterplot and best-fit line of the effect size of the change in the behavioral outcome (x-axis) against the effect size of change in the physiological outcome (y-axis). Each circle represents one trial with dark-colored circles representing significant benefit. Dark shading represents effect sizes under 0.2, medium shading represents small effect sizes (Cohen's d between 0.2 and 0.5), lighter shading represents medium effect sizes (Cohen's d between 0.5 and 0.8), and no shading represents large effect sizes (Cohen's d greater than 0.8)

The analysis was repeated using only weight as the physiological outcome because weight was the most common physiological outcome reported across all trials (Fig. 4). For this analysis, we combined trials that analyzed weight as either the primary or secondary outcome (n=15). Physiological effect sizes shown in Fig. 4 may not appear in Fig. 3 of all primary physiological effect sizes. We again see high concordance with 87 % of the trials reporting a significant benefit in the behavior and weight outcomes. We observed a stronger relationship between the data points (slope=0.33), and more of the trials reported moderate to large effects in weight change (n=10).

Fig. 4.

Scatterplot and best-fit line of the effect size of the change in the behavioral outcome (x-axis) against the effect size of change in the weight outcome (y-axis). Each circle represents one trial with dark-colored circles representing significant benefit. Dark shading represents effect sizes under 0.2, medium shading represents small effect sizes (Cohen's d between 0.2 and 0.5), lighter shading represents medium effect sizes (Cohen's d between 0.5 and 0.8), and no shading represents large effect sizes (Cohen's d greater than 0.8)

Meta-Analysis of Effect Size of Single Versus Multiple Behavior Interventions

We conducted a meta-analysis of trials by type of intervention for the following sub-groups: (1) single behavior change interventions targeting blood pressure and heart monitoring interventions, (2) single behavior change interventions targeting nutrition or physical activity, and (3) multiple behavior change interventions of nutrition, physical activity, or smoking. Although four blood pressure and heart monitoring studies were pooled, these studies did not consistently report a behavior and physiological change. Only one trial reported the behavior change (adherence), and only two reported blood pressure. We compared the effect size and number of significant primary outcomes for single versus multiple behavior interventions that modified nutrition and physical activity (Table 3). The mean effect size for the primary physiological outcome was larger in multi-behavior interventions (0.48 effect size) as compared with single behavior interventions (0.34 for physical activity or 0.03 for diet only). The mean effect size for weight change was two to five times higher for multi-behavior change interventions (0.74 effect size) as compared to single behavior change interventions (0.08 for physical activity or 0.14 for diet only).

Table 3.

Summary table of number of significant benefits and effect sizes achieved and length of intervention and follow-up period by type of behavioral intervention

Type of intervention	Blood pressure monitoring	Nutrition only	Physical activity only	Multi-behavior change
Number of trials	4	4	5	18
Reported behavioral outcome	1	3	3	13
Significant behavior benefit (N)	1	3	2	11
Behavior change effect size from meta-analysis	0.42	0.15	0.34	0.53
Reported physiological outcome	2	4	5	18
Significant physiological benefit (N)	1	3	4	18
Physiological change effect size from meta-analysis	0.09	0.03	0.34	0.48
Reported weight outcome	0	4	2	17
Significant weight benefit (N)	N/A	4	0	16
Weight change effect size from meta-analysis	N/A	0.14	0.08	0.74
Reported morbidity	1	4	1	8
Significant morbidity benefit (N)	0	3	1	4
Reported mortality	1	1	1	3
Significant mortality benefit (N)	0	0	0	0
Range of duration of intensive intervention period (months)	3–24	2.5–48	3–24	3–60
Mode (most common) length (months)	3	12–14	6–12	6–12
Range of length of total follow-up (months)	3–24	14–72	6–24	6–72
Mode (most common) length (months)	6	12–14	12	12

Tables in cells reflect sub-group analysis of the behavioral intervention trials by their specific focus of the intervention—blood pressure monitoring, nutrition only, physical activity only, multi-factor, or lifestyle interventions. The multi-factor interventions had to include at least two of the following behavior change targets—nutrition, physical activity, and smoking. We report the number of trials for each type of intervention and the number who reported and who found significant benefits for behavior, physiological, morbidity, or mortality outcomes. We also include the effect size that we calculated from the meta-analysis of effect sizes from each type of intervention. Effect sizes above are weighted by sample size. Please note that the effect sizes were quite different for the nutrition intervention trials for standard error versus weighted by sample size because of the large sample size of the WHI trial. For the nutrition only interventions, effect sizes for the behavioral outcome were 0.98 when weighted by standard error and 0.15 when weighted by sample size; for the physiological outcome were 0.28 when weighted by standard error and 0. 03 when weighted by sample size; and for the weight outcome were 0.35 when weighted by standard error and 0.14 when weighted by sample size

We coded trials as having a significant benefit in change in weight and change in their primary physiological outcomes. Among interventions that modified a single behavior (nutrition or physical activity), seven out of nine (78 %) reported a significant benefit in their primary physiological outcome, and four out of six (67 %) reported a significant benefit in weight as either a primary or secondary outcome. All of the multiple behavior change interventions reported a significant benefit in their primary physiological outcome, and 94 % reported a significant benefit in weight (either as primary or secondary outcome).

Discussion

Our review of NHLBI/NIDDK-funded RCTs addresses three important questions: (1) Can health behavior be modified? (2) Does the change produced by behavioral intervention result in physiological change? and (3) Do behavioral interventions affect morbidity and mortality?

Can behavior be modified? It is commonly believed that health behavior is difficult to modify. Many practitioners have become skeptical about the effectiveness of behavioral alternatives for the management of cardiovascular risk factors [19]. Our review suggests that the great majority of behavioral trials do, indeed, demonstrate a positive benefit. Further, most behavioral trials are pre-registered in the ClinicalTrials.gov service, and the primary outcome variable is specified in advance. This assures that investigators are not selecting positive outcomes from among many alternatives in a post hoc fashion. Behavioral intervention researchers are conforming to high methodological standards.

Our results are consistent with a variety of other analyses. For example, a recent evidence synthesis for the US Preventive Services Taskforce (USPSTF) considered the benefits and harms of behavioral counseling interventions to prevent cardiovascular disease (CVD) for persons with established risk factors [20]. After considering 49 trials in meta-analysis, they concluded that behavioral interventions to improve lifestyle resulted in reductions in total cholesterol low-density lipoprotein, blood pressure, fasting glucose, diabetes, and adiposity [20].

Do behavioral changes result in physiological changes? Our review suggests that behavioral intervention often results in physiological changes including reductions in weight, blood pressure, and serum cholesterol. Overall, there was a high degree of concordance between behavioral and physiological outcomes. Both types of measures were reported in 20 of the 38 trials. Among these trials, 80 % reported a benefit for both behavior change and physiologic change (16 of 20 trials). Few studies report data on the relationship between behavior change and physiological change. More consistent reporting of the target behavior change and its relationship to changes on physiological and health outcome variables is needed. Future interventions should test and publish mediational analyses between their behavior change and physiological outcomes. These mediational analyses would provide a better understanding of the mechanisms that did or did not produce the intended physiological change.

Measurement tools may be inadequate to clearly document behavioral change. For instance, most nutritional measures were collected via self-report, which has known measurement error due to participant recall, knowledge, and reactivity [21]. It might be argued that the goal of behavioral intervention is to affect physiological outcomes. Thus, movement of a physiological parameter might be the best evidence that the behavioral treatment was successful. However, in some trials, there were changes on physiological measures despite null effects for behavior change [22, 23]. We do not know whether the results from these trials were (1) because there was high measurement error for the behavioral measures or (2) because the physiological outcomes were affected through a non-behavioral pathway.

Do behavioral interventions affect morbidity and mortality? The goal of most health interventions is to improve health outcomes, including functioning, quality of life, and longevity. In examining this literature, we note that many of the behavioral trials evaluated outcomes in terms of risk factors or behavior changes. Very few of the trials evaluated long-term outcomes including mortality or clinical morbidity. This is in contrast to many of the large pharmaceutical trials that focus attention on changes in cardiovascular mortality or all-cause mortality [9]. Thus, it could be argued that behavioral trials are not being held to the same rigorous standard as are evaluations of pharmaceutical and surgical interventions. However, pharmaceutical interventions often modify a risk factor but report no benefit on the primary health outcomes [9].

The finding that behavioral trials typically do result in significant improvements in risk factors suggests that we need more trials that take the evaluation to the next level. Future trials might include more evaluations of long-term health outcomes. Less than half of multi-factor, lifestyle interventions included in our review reported a follow-up period greater than 1 year. In order to measure clinical events or morbidity, longer length of follow-up is required. Maintenance studies are needed to determine if these small to moderate physiological effects are maintained post-treatment and if these effects are sufficient to reduce cardiovascular events.

These trials will need to be large in order to assure sufficient statistical power to evaluate the null hypothesis. Unfortunately, powering behavioral trials to detect mortality effects will require a major change in the size of trials. For example, cholesterol-lowering medications are believed to be one of the most effective instruments for reducing likelihood of death from cardiovascular disease. In the very influential Coronary Primary Prevention Trial [24], 1.6 % of participants taking cholesterol lowing medications died over a 7-year follow-up in comparison to 2 % of participants in the control group. To have a 90 % chance of detecting a difference between groups would require more than 23,000 subjects per group. In the original Physician's Health Study [25] on the effects of aspirin to prevent deaths from myocardial Infarction, there were 5 deaths per 11,000 who were randomly assigned to take aspirin in comparison to about 18 deaths per 11,000 physicians who took placebo. In order to prospectively plan for a 90 % chance of detecting an effect this size at the 0.05 alpha level, 16,000 subjects per group would be required.

As these calculations demonstrate, finding a significant treatment effect for mortality often requires enormous sample sizes. Typically, sample size requirements may be many levels of magnitude larger than is current practice in behavioral trials. In the 38 trials included in this review, the majority of trials included samples sizes of several hundred combining both study arms. Only four trials reported more than 1,000 participants/group and only two trials more than 5,000 participants/group. Future investigations may need to consider much larger sample sizes to demonstrate the benefits of behavioral interventions.

Effect Size

The effect sizes of the behavior and physiological outcomes were predominantly in the range as the small (d=0.2) to medium (d=0.5) levels [17]. Because so few trials analyzed changes in morbidity and mortality, we do not know if these effects sizes are large enough to invoke a clinically meaningful change in morbidity. If small or medium effect sizes are achieved, we do not know how well they are maintained or if a sustained small effect would produce morbidity changes equivalent to larger, short-term physiological effects. Future interventions might test the clinical outcomes achieved with smaller physiological and behavioral changes sustained long term. For interventions that target high-risk populations, larger changes in effect size in both the behavior and physiological outcomes might be needed to achieve clinical outcomes. Instead of powering trials for a moderate effect size, trials might estimate Cohen's d=1 which would equate to about a change in one standard deviation between treatment and control groups.

We conducted a sub-analysis of interventions that targeted a single behavior change (nutrition or physical activity or heart monitoring) as compared with interventions targeting multiple behavior or lifestyle changes. Effect sizes achieved with multiple behavior change interventions were larger than those achieved with single behavioral targets. In a recent review, Nigg and Long found the majority of interventions with older adults focused only on one behavior change [26]. Our sample of trials included twice as many multi-behavior change interventions as compared to single behavior change interviews.

File Drawer and Selective Reporting

Null and negative results may be less likely to be published in comparison to positive results. This selective non-reporting is known as “file drawer” bias. The trend toward study preregistration may help address this problem because it allows identification of all studies that are launched. Knowing the denominator of studies will allow better estimates of the rate of non-reporting. Gordon et al. [13] conducted a review of NHLBI-funded trials and noted that 64 % of trials that cost more than $5 million dollars were published within 12 months, 91 % within 30 months, and 97 % within 48 months of the grant end date. Behavioral interventions (not restricting the sample to over $5 million) were less likely to be published than non-behavioral interventions, with only 11 % published at 12 months, 48 % at 30 months, and 72 % at 48 months following the grant end date. Gordon et al. did not report publication rates for behavioral interventions costing over $5 million [13]. In our analysis, approximately 65 % of behavioral trials were published. We used an end date of December 2012, which only allowed some studies about 18 months to publish prior to our analyses. Lower publication rate among behavioral trials might be attributed to the type of outcomes reported. Previous analyses have shown that trials that report a clinical-event end-point were more likely to be published than trials that did not [13]. Less than half of trials in our analysis reported a clinical, morbidity outcome. Publication rates might improve if behavioral interventions were powered for and reported clinical outcomes (like hospitalization or formal disease diagnosis).

Over three fourths of behavioral trials funded since 2000 were registered prospectively with ClinicalTrials.gov. All reported their pre-specified primary outcomes. Thus, our results are not clearly explained by selective reporting of primary outcomes. The rate of registration is high considering behavioral interventions are not required to register with ClinicalTrials.gov [18]. A recent review of behavioral RCTs published in several leading behavioral health journals found that the majority of behavioral intervention trials did not register and did not adequately declare primary and secondary outcomes [27]. The Milette review did not differentiate by source of funding [27]. In our data, trials that registered had a higher rate of publication than trials that did not register, although this difference was not statistically significant. Investigators of behavioral interventions may be registering in order to publish in certain journals or because of a requirement of their sponsor. NIH now requires all funded trials to register [28]. Better estimates of the rate of non-reporting are expected in the future.

Limitations

Our evaluation has a significant number of limitations. First, we concentrated only on large-budget NHLBI and NIDDK trials. Clearly, this is a small fraction of all of the behavioral trials in the literature. Ultimately, only 38 studies met the inclusion criteria, and it is legitimate to ask how representative these 38 studies are of all behavioral trials. We must emphasize that this group of studies is the population of studies that met the inclusion criteria. It is true that all of these were studies funded through the peer review system. But, we did not arbitrarily eliminate studies. One of the strengths of the study is that we knew the population of studies that were funded prior to publication. In addition, the USPSTF recently reviewed a wide range of behavioral trials and came to similar conclusions [20].

A second concern is that we evaluated only large trials. We focused on large trials because they were more likely to be registered, and we had a better opportunity to rule out bias due to non-publication [13]. On the other hand, these large funded trials are likely to be atypical. Additional work using a more representative sample of trials is in order.

A third concern is that we focused on studies done in the USA, a country with a unique health care system and a unique research funding structure. The reason for focusing on US studies was that we were able to access NIH grant databases of funded studies. This is important because access to the population of funded studies allowed us to avoid biases associated with selective non-publication of normal or negative results.

A fourth concern is that behavioral medicine investigators sometimes recruit participants who do not have elevated scores on a target variable. As a result, there is less room for change because of floor effects [29]. A meta-analysis by Schneider and colleagues [30] found that distress prior to an intervention explained as much as half of the variability between studies on treatments for anxiety and depression. Many studies showed modest or no effects of intervention when baseline distress was low. Floor effects are important. On the other hand, most of the behavioral interventions are used for population-based prevention and may need to focus on non-clinical populations. The USPSTF, which serves as the basis for US clinical prevention policy, typically excludes studies where patients are selected because they have high scores on a target variable. The reason is that the USPSTF wants the results to generalize to the primary care population who receive preventive services because they do not have elevated scores or diagnosed diseases. The role of clinical versus population study group must be carefully considered in designing and generalizing from studies.

Lastly, systematic reviews can now be registered, but we were unaware of registration services when we began our analysis in the fall of 2011. One registration service is Prospero which was developed in 2011, but the founding principles were not released until May 2014 (see http://www.crd. york.ac.uk/PROSPERO/). We do support registration of reviews and would have used this service had we been aware of it when we began our work. We recognize that registration reduces biases, promotes transparency of methods, and avoids potential duplication. To facilitate the replication of our work by others, the tables in the paper and the detailed online Supplemental Materials report the PRISMA diagram, the number of results returned and excluded, and raw numbers and RR used in calculations. We support replication and encourage others to reproduce our findings.

In summary, behavioral factors play an important role in the etiology and pathogenesis of major cardiovascular conditions. Our review of large-budget NHLBI- and NIDDK-funded behavioral trials suggests that the great majority produce positive outcomes in terms of behavioral change and modification of cardiovascular risk factors. The common belief that behavior cannot be changed is not supported by this review or by a related meta-analysis conducted for the USPSTF [20]. In contrast, the majority of NHLBI trials evaluating pharmaceutical interventions produce null results [13], and the number of positive morbidity or mortality outcomes in drug trials has declined since 2000 [9]. Behavioral interventions have fewer negative side effects than drugs, and behavior change might lead to cascading benefits with other related health behaviors. Behavioral interventions show promise with significant benefits to behavior and physiological outcomes. More research is needed to test the maintenance of these changes and to determine if these physiological changes are sufficient to lengthen and improve quality of life.