Getting Carried Away: A Note Showing Baseline Observation Carried Forward (BOCF) Results Can be Calculated from Published Complete-Cases Results

Abstract

Objective

Randomized controlled trials (RCTs) in obesity are plagued by missing data due to participant drop-outs. Most methodologists and regulatory bodies agree that the primary analysis of such RCTs should be based on the intent-to-treat (ITT) principle, such that all randomized subjects are included in the analysis, even those who dropped out. Unfortunately, some authors do not include an ITT analysis in their published reports. Here we show that one form of ITT analysis, baseline observation carried forward (BOCF), can be performed utilizing only information available in a published complete case (CC) analysis, permitting readers, editors, meta-analysts, and regulators to easily conduct their own ITT analyses when the original authors do not report one.

Method

We mathematically derive a simple method for estimating and testing treatment effects using the BOCF to allow a more conservative comparison of treatment effects when there are drop outs in a clinical trial. We provide two examples of this method using available CC analysis data from reported obesity trials to illustrate the application for readers who wish to determine a range of treatment effects based on published summary statistics.

Conclusion

Commonly used CC analyses may lead to inflated Type I error rates and/or treatment effect estimates. The method described herein can be useful for researchers who wish to estimate a conservative range of plausible treatment effects based on limited reported data. Limitations of this method are discussed.

INTRODUCTION

Randomized controlled trials (RCTs) are the gold standard for providing unbiased estimates and tests of the causal effects of obesity treatment and prevention strategies. Despite increased efforts to minimize the loss of participation before the study protocol is completed, drop outs remain a near-ubiquitous phenomenon (1). There are many statistical methods available to accommodate missing data and to use all available data to estimate and test treatment effects (2;3). While the intention-to-treat (ITT) principle is commonly accepted as an appropriate foundation for primary analyses, many researchers continue to use only a complete case (CC) analysis, which may introduce significant bias in tests and estimates of treatment effects when there are dropouts (2;3). In a 2010 analysis of obesity trials, CC analysis was the most common method used (34% of papers) (4). The prevalence of CC analysis creates a problem for anyone (e.g., readers, clinicians, consumers, regulatory agencies, meta-analysts) who wishes to rely on these papers to draw confident conclusions about the efficacy of the treatment studied.

There are several approaches to ITT analysis: most authors define it as an analysis including all cases who were randomized to a treatment arm, while a modified approach is to include only those cases who were randomized and received at least one exposure to the assigned treatment (5). One method of ITT analysis which has been recommended by some authors is baseline observation carried forward (BOCF) (6-8). BOCF is similar to the LOCF (Last Observation Carried Forward) in that the imputation is applied to participants who drop out of the study before the first post-baseline measure is taken. In LOCF, participants have the last value recorded carried forward depending on whether exposure to treatment is received. In BOCF, for any subject who does not have any post-baseline outcome (e.g., weight) measurement at the endpoint under study, one simply imputes the baseline value for that variable. This means that if change in the variable (e.g., weight loss or percent weight loss) is analyzed as the outcome, for all subjects who do not have a value at the study endpoint, one simply inserts the value as zero, for “no change”. Then the complete data file can be analyzed with conventional methods.

We are not advocating BOCF as a method in general. Single imputation methods, of which BOCF is one, have serious disadvantages (6;9) and BOCF in particular is strictly valid only under the assumption that all subjects who dropped out returned exactly to their baseline value on the outcome measure. BOCF is thus likely to be biased and highly conservative in practice. Nevertheless, when no other ITT analysis is available, BOCF can be used to place a plausible (though not definitive) lower bound point estimate on treatment effects (6).

In light of the frequently absent information from ITT analyses in published obesity RCTs, we provide a method for any reader, researcher, or regulatory agent to calculate a more conservative estimate of treatment effects for a published study using commonly available data supplied within most papers. We provide the mathematical derivation and show two examples from published studies that illustrate the application.

METHOD

Notation

${\stackrel{‒}{X}}_T$ : The mean weight loss in the treatment group in a complete cases analysis.

${\stackrel{‒}{X}}_C$ : The mean weight loss in the control group in a complete cases analysis.

π_T : The proportion of subjects in the treatment group who finished the trial (i.e., were in the complete cases analysis).

π_C : The proportion of subjects in the control group who finished the trial (i.e., were in the complete cases analysis).

N_T : The number of subjects randomized to the treatment group.

N_C : The number of subjects randomized to the control group.

${\widehat{\sigma }}_{\mathit{T}}^2$: The estimated variance of the outcome measurement (e.g., weight change) in the treatment group in a complete cases analysis.

${\widehat{\sigma }}_{\mathit{C}}^2$: The estimated variance of the outcome measurement in the control group in a complete cases analysis.

Derivation

It can then be shown that the mean weight loss in a BOCF analysis will be ${\stackrel{‒}{X}}_{T,BOCF}={\stackrel{‒}{X}}_T{\pi }_T$ in the treatment group and will be ${\stackrel{‒}{X}}_{C,BOCF}={\stackrel{‒}{X}}_C{\pi }_C$ in the control group, leading to a point estimate of the unstandardized treatment effect of ${\widehat{\delta }}_{BOCF}={\stackrel{‒}{X}}_T{\pi }_T-{\stackrel{‒}{X}}_C{\pi }_C$ in a BOCF analysis. In deriving the variance of weight loss in the BOCF analysis, we note the within-group Sums of Squares for weight loss in the treatment group is $S{S}_T={\widehat{\sigma }}_{\mathit{T}}^2(N_T-1)$ and likewise the within-group Sums of Squares for weight loss in the control group is $S{S}_C={\widehat{\sigma }}_{\mathit{C}}^2(N_C-1)$. From this, the within-group Sums of Squares for weight loss in the treatment group in the BOCF analysis is $S{S}_{T,BOCF}=S{S}_T+{\pi }_T(1-{\pi }_T){\stackrel{‒}{X}}_{\mathit{T}}^2$ and the variance is ${\widehat{\sigma }}_{\mathit{T},\mathit{BOCF}}^2=S{S}_{T,BOCF}∕(N_T-1)$. Similarly, within-group Sums of Squares in the control group in the BOCF analysis is $S{S}_{C,BOCF}=S{S}_C+{\pi }_C(1-\pi C){\stackrel{‒}{\mathit{X}}}_{\mathit{C}}^2$ with a variance of ${\widehat{\sigma }}_{\mathit{C},\mathit{BOCF}}^2=S{S}_{C,BOCF}∕(N_C-1)$. Then, the variance of ${\stackrel{‒}{X}}_{T,BOCF}$ is ${\widehat{\sigma }}_{{\stackrel{‒}{X}}_{\mathit{T},\mathit{BOCF}}}^2=\frac{{\widehat{\sigma }}_{\mathit{T},\mathit{BOCF}}^2}{{\mathit{N}}_{\mathit{T}}}$ and the variance of ${\widehat{\mathit{X}}}_{C,BOCF}$ is ${\widehat{\sigma }}_{{\stackrel{‒}{\mathit{X}}}_{\mathit{C},\mathit{BOCF}}}^2=\frac{{\widehat{\sigma }}_{\mathit{C},\mathit{BOCF}}^2}{{\mathit{N}}_{\mathit{C}}}$. Further, $\frac{{\stackrel{‒}{\mathit{X}}}_{\mathit{T},\mathit{BOCF}}-{\stackrel{‒}{\mathit{X}}}_{\mathit{C},\mathit{BOCF}}}{\sqrt{{\widehat{\sigma }}_{{\stackrel{‒}{\mathit{X}}}_{\mathit{T},\mathit{BOCF}}}^2+{\widehat{\sigma }}_{{\stackrel{‒}{\mathit{X}}}_{\mathit{C},\mathit{BOCF}}}^2}}$ will be asymptotically distributed as t with N_C + N_T - 2 degrees of freedom.

When sample sizes are large (e.g., N_C and N_T > 200), a quick and easy way to compute an approximate variance of weight loss in the BOCF analysis is ${\widehat{\sigma }}_{\mathit{T},\mathit{BOCF}}^2={\pi }_T{\widehat{\sigma }}_{\mathit{T}}^2+{\pi }_T(1-{\pi }_T){\stackrel{‒}{\mathit{X}}}_{\mathit{T}}^2$ in the treatment group and correspondingly, ${\widehat{\sigma }}_{\mathit{C},\mathit{BOCF}}^2={\pi }_C{\widehat{\sigma }}_{\mathit{C}}^2+{\pi }_C(1-{\pi }_C){\stackrel{‒}{\mathit{X}}}_{\mathit{C}}^2$ in the control group. A symmetric, asymptotic 95% confidence interval for ${\widehat{\delta }}_{BOCF}$ may be obtained using $\widehat{\delta }\pm 1.96\left(\widehat{\mathit{SE}}\right)$, with $\widehat{\mathit{SE}}$ (the estimated standard error) obtained as:

$${\widehat{\mathit{SE}}}_{{\stackrel{‒}{\mathit{X}}}_{\mathit{T},\mathit{BOCF}}-{\stackrel{‒}{\mathit{X}}}_{\mathit{C},\mathit{BOCF}}}=\sqrt{\frac{{\widehat{\sigma }}_{{\stackrel{‒}{\mathit{X}}}_{\mathit{T},\mathit{BOCF}}}^2+{\widehat{\sigma }}_{{\stackrel{‒}{\mathit{X}}}_{\mathit{C},\mathit{BOCF}}}^2}{{\mathit{N}}_{\mathit{T}}+{\mathit{N}}_{\mathit{C}}}}.$$

Illustration

We searched for recently published weight loss RCTs that reported a CC analysis to illustrate this method for BOCF analysis using reported data. Example 1 shows an outcome with BOCF that is similar to the originally reported CC analysis, strengthening confidence in the reported conclusion. In Example 2, the BOCF analysis reveals how even slight differences in dropout rates between groups can affect both the treatment estimate and its statistical significance, thus casting doubt on the original conclusion. Note that in each case, the BOCF increases the degrees of freedom compared to a CC analysis and therefore lowers the critical t value.

Example 1(10)

This study (10) evaluated the effects of conjugated linoleic acid supplementing the diets of overweight and obese participants on regional-specific fat mass over a six month trial. A total of 118 persons were randomized to a control (N_C=59) or treatment group (N_T=59). Dropouts were similar between groups at 18 and 17 persons, respectively. Therefore, π_C = (41/59) = 0.695 and π_T = (42/59) = 0.712. As can be seen in Table 1, with an observed treatment difference between the treatment and control groups of 1.5 kg favoring treatment, both methods result in similar interpretations of intervention effects [CC: t (81)= 2.024, p = 0.0462; BOCF: t (116)= 2.013, p = 0.0464]. Thus, the BOCF estimate provides more confidence in the conclusion provided using the CC analysis.

Table 1.

Illustration of input data and results of reported complete cases analysis versus baseline observation carried forward analysis in two example weight loss studies

	Example 1		Example 2
n randomized Treatment	59		57
n randomized Control	59		50
n completed Treatment	42		52
n completed Control	41		47
Mean(SD) Kg Change Treatment	1.2(4.2)		7.5(2.6)
Mean(SD) Kg Change Control	−0.3(2.3)		6.2(2.9)
	Complete Cases analysis	BOCF analysis	Complete Cases analysis	BOCF analysis
Total Unstandardized Treatment Effect	1.500	1.063	1.300	1.014
Variance of Weight Loss - Treatment	17.64	12.77	6.76	10.67
Variance of Weight Loss - Control	5.29	3.67	8.41	10.07
Variance of the Mean – Treatment	0.42	0.22	0.13	0.19
Variance of the Mean-Control	0.13	0.06	0.18	0.20
t(df), p	t (81) = 2.024, p = 0.0462	t (116) = 2.013, p = 0.0464	t (97) = 2.339, p = 0.0214	t (105) = 1.623, p = 0.1076

Example 2(11)

This study (11) examined the effects of eight weeks on a very low carbohydrate, high saturated fat diet compared to a high carbohydrate, low saturated fat diet on 107 participants with abdominal obesity. Participants were randomized between treatment (N_C = 57) and control (N_T = 50) and 5 dropped out of the treatment group while 3 dropped out of the control group. Therefore, π_C = (47/50) = 0.940 and π_T = (52/57) = 0.912. A raw treatment effect was 1.3 kg, favoring treatment. In contrast to example 1, a difference in the estimates of the treatment effects can be seen between the CC analysis and the BOCF analysis [CC: t(97) = 2.339, p = 0.0124 and BOCF: t(105)= 1.623, p = .1076]. This suggests that in this case, the BOCF result may call into question the published conclusion drawn using the CC analysis.

DISCUSSION

Our method has assumptions and limitations which should be noted. First, it simply reproduces what an ordinary BOCF analysis of raw data would produce and therefore ‘inherits’ all the limitations of BOCF - see (9) for discussion. Second, the t-statistic and confidence interval (CI) method derived are strictly valid in finite samples if all the Gauss-Markov assumptions are met (12), which includes normality of residuals. In BOCF, normality of residuals will almost certainly not hold and because the raw data are not available, in our approach, we cannot switch to a non-parametric analysis as one could with a raw data BOCF analysis. However, even if residuals are not normally distributed, the method is asymptotically valid and considering the typical sample sizes in obesity RCTs, non-normality is unlikely to be problematic.

The method we offer can easily be performed with an ordinary spreadsheet. While the BOCF approach seems likely to be extremely conservative in most situations, it can provide a useful lower boundary when attempting to determine a range of treatment effect estimates as compared to CC analyses, which are often likely to be at the upper bound. This proposed method will often reduce treatment effect estimates, especially when more participants drop out in the treatment group compared to the control group as seen in Example 2. One author has argued for the validity of the BOCF approach specifically with obesity interventions, due to evidence indicating that many persons regain most of the weight lost after many types of treatment approaches (7). Of course, it is not definitively known whether weight returns, on average, to baseline levels during the trial period for participants who drop out in any given trial. However, this highly conservative approach does offer an estimation of the treatment effects following the principles of ITT.