Comparative effects of the restriction method in two large observational studies of body mass index and mortality among adults

Abstract

Background

A method applied in some large studies of weight and mortality is to begin with a well-defined analytic cohort and use successive restrictions in order to control for methodologic bias and arrive at final analytic results.

Materials and methods

Two observational studies of body mass index and mortality allow a comparative assessment of these restrictions in very large data sets. One was a meta-analysis of individual participant data with a sample size of 8 million. The second was a study of a South Korean cohort with a sample size of 12 million. Both presented results for participants without pre-existing disease before and after restricting the sample to never smokers and deleting the first 5 years of follow-up.

Results

Initial results from both studies were generally similar, with hazard ratios (HRs) below 1 for overweight and above 1 for underweight and obesity. The meta-analysis showed higher HRs for overweight and obesity after the restrictions, including a change in the direction of the HR for overweight from 0.99 (95% CI 0.98–1.01) to 1.11 (95% CI 1.10, 1.11). The South Korean data showed little effect of the restrictions and the HR for overweight changed from 0.85 (95% CI 0.84–0.86) to 0.91 (95% CI 0.90, 0.91). The summary effect size for overweight was 0.90 (95% CI 0.89–0.91) before restrictions and 1.02 (95% CI 1.02, 1.03) after restrictions.

Conclusions

The effect of the restrictions is not consistent across studies, weakening the argument that analyses without such restrictions lack validity.

The topic of the relation of body weight or body mass index (BMI) to mortality has generated numerous studies and much interest. Many studies have shown a generally curvilinear relationship of BMI to mortality, with increasing relative risks both at high and low BMI. Studies of older people have sometimes shown elevated risks only at low BMI values with little or no association with risk at higher BMI values [1].

In general, relative risks in these studies are small except at the extremes, generally below 1.5 and not infrequently as low as 1.05 [2–4]. Even a small relative risk may be of public health importance but small risks are more difficult to estimate precisely and can be affected by small biases [5]. Non-linearity poses analytic challenges as well [6].

Methodological critiques have suggested that the commonly seen patterns may arise from some type of biases [7, 8]. Manson et al [7], in a review up to 1984 of what they described as the 25 major studies on weight and mortality, stated that “each study had at least one of three major biases: failure to control for cigarette smoking, inappropriate control of biologic effects of obesity, such as hypertension and hyperglycemia, and failure to control for weight loss due to subclinical disease. The presence of these biases leads to a systematic underestimate of the impact of obesity on premature mortality.” The review by Manson et al [7] argued that these factors could possibly have produced bias, but the authors did not demonstrate that such bias had actually occurred. A later commentary by Willett et al [8] explored the same ideas, stating that “Reverse causation is the most serious problem associated with using total mortality as an outcome; people frequently lose weight as a result of an illness that is ultimately fatal, a situation that creates the appearance of higher mortality among those with lower weights. … Several strategies can be used to minimize the effect of reverse causation. Subjects with diagnoses that might affect weight and subjects who report recent weight loss, such as during the previous five years, can be excluded from a prospective study. Deaths that occur during the first several years of follow-up - possibly as a result of conditions that caused lower weights at base line - can also be excluded.” With regards to smoking, Willett et al state “Even if data on smoking are available, simple statistical adjustments for smoking are not entirely satisfactory, because nuances such as depth of inhalation and genetic susceptibility, which cannot be accounted for, could influence the effect of smoking on both weight and mortality. The most satisfactory way to deal with smoking is to restrict the analysis to subjects who have never smoked.”

Restriction is a method of controlling potential confounding that is often applied in the design phase of a study, either as part of the initial data collection or as part of constructing an analytic sample from a larger data set, as discussed by Rothman et al under the chapter heading of “Design strategies to improve study accuracy” [9]. For example, a study may be restricted to one gender, one occupation, one geographic area or a limited age range.

Restrictions have occasionally been applied as part of the analysis phase of a study rather than in the design phase. Building on the concepts outlined by Manson et al and Willett et al, a number of large studies of weight and mortality have used restrictions to control for possible confounding. A method applied in some studies has been to begin with a well-defined analytic cohort and then use successive restrictions in order to arrive at final analytic results based on a much smaller sample. One such restriction may involve deleting all current and former smokers and limiting the analyses to never-smokers to control for possible residual confounding by unmeasured characteristics of smoking. In order to control for possible weight loss due to pre-existing disease (“reverse causation”), analyses may also be restricted to those with no history of heart disease or cancer and may exclude the first several years of follow-up. For example, in a number of large studies [10–13] after very large scale restrictions were applied, the studies showed different results than those they found initially, with higher risks at high BMIs, lower risks at low BMIs and a somewhat lower point of minimum mortality. For example, in the study by Berrington de Gonzalez et al, [13] the category with minimum mortality was BMI 25–27.4 before restrictions and BMI 20–24.9 after restrictions.

These restrictions can involve deleting anywhere from 60% to 80% of the data and almost 90% of the deaths in the sample. Some examples are shown in Table 1. As noted by Willett et al. this approach tends to require large studies “Unfortunately, many studies have been too small to have adequate statistical power when the analysis is limited to those who have never smoked, in part because death rates are lower in this group.” Similarly, Manson et al [14] state that “Studies with very large sample sizes and very long follow-up periods that begin in midlife or earlier permit comprehensive analyses that address threats to validity.” It is sometimes argued that studies that show no effects of restrictions or the opposite effects from what is predicted are too small to show the effect clearly [13, 14].

Table 1.

Number (%) of sample and deaths deleted by restrictions^* in some studies of BMI and mortality.

First author (reference)	Before restrictions:		After restrictions		Percentage of deaths deleted	Percentage of sample deleted
	Sample size	Number of deaths	Sample size	Number of deaths
Ajani [68]	85,078	2,856	37,292	680	76.2	56.2
Baik [69]	39,756	1,972	Not shown	635	67.8	-
Berrington de Gonzalez [13]	1,460,000	160,087	Not shown	34,709	78.3	-
Calle [11]	1,046,154	201,622	302,233	34,507	82.9	71.1
Greenberg [70]	12,457	606	3,916	73	88.0	68.6
Manson [12]	115,195	4,726	Not shown	531	88.8	-
Park [42]	183,211	35,664	69,141	7,264	79.6	62.3
Park [71]	16,471	2,609	Not shown	713	72.7	-

^*The exact restrictions vary from study to study. All but one excluded ever-smokers. Other restrictions may include deleting participants with pre-existing disease, participants who report recent weight loss, and variable numbers of years of follow-up

It is argued that these restrictions constitute “control for methodological bias” and that the findings after the restrictions are less biased than those before the deletions. The essence of the argument in favor of the restrictions is that without these restrictions, the results lack validity. For example, Berrington de Gonzalez et al [13] state: “The counterargument is that smoking and preexisting conditions that cause weight loss are powerful confounders and analyses that include them lack validity — an attribute that is more important in etiologic studies than is generalizability.” .However, there is little if any evidence for this proposition, and it is also possible that some features of the deletions themselves may be causing bias [15, 16]. The deletions are in effect a type of subgroup analyses, with the analysis restricted to a single subgroup, and thus are subject to the same limitations as other subgroup analyses [17, 18]. As pointed out elsewhere by Berrington de Gonzalez and Morton [19], a subgroup analysis may seem attractive but also may produce substantial collider-stratification bias.

Recently two extremely large observational studies of body mass index (BMI) and mortality among adults have been published that allow some comparative assessment of the effects of these restrictions in very large data sets. Both studies allow for analyses both before and after applying extensive restrictions. The first, by the Global BMI Mortality Collaboration (GBMC) [20], was a meta-analysis with individual participant data (IPD) from 189 studies with a total sample size of 8 million. The second, by Yi and colleagues[21], presented data from a study of a single cohort from South Korea with a sample size of 12 million. The GBMC meta-analysis sought data through January 2015; the South Korean study was published in July 2015. Comparable analytic methods were used in the two studies. Here we compare their results qualitatively and estimate the results of combining the findings from both studies.

Methods

Both studies presented results using WHO-defined categories of underweight (BMI <18.5), normal weight (reference category, BMI 18.5–24.9), overweight (BMI 25–29.9), obesity grade I (BMI 30–34.9) and either obesity grade II/III (BMI 35 and above) in the Yi et al paper or obesity grade II (BMI 35-<40) and III (BMI 40 and above) in the GBMC paper. In both studies, data on all-cause mortality were analyzed using Cox proportional hazard models adjusted for age and sex. Both studies included analyses with adjustment for smoking in people without pre-existing disease. Both studies also included a set of analyses restricted to never-smokers with no history of pre-existing disease after deleting the first 5 years of follow-up. The published article by Yi et al included analyses of sex- and smoking-specific hazard ratios in participants with no known illness (Figure S4) and sex-age-specific hazard ratios in never-smokers with no known illness who survived 5 years after enrollment (Figure S7). To allow direct comparisons with the GBMC estimates, these estimates are presented here with adjustment for sex and smoking rather than as sex- and smoking-specific estimates and with adjustment for sex and age rather than as sex-age specific estimates.

In order to estimate the possible effects if these two large data sets were taken together, we combined the two treating both as fixed-effects, i.e., using a weighted average of the log transformed hazard ratios (HRs), weighting proportionally to the inverse of the variances of the individual HRs [22].

Results

The individual and combined results for these analyses are shown in Table2 adjusted for age, sex, and smoking status with restriction to no pre-existing disease. The results for underweight were almost identical between the two studies. For obesity, HRs from both studies were significantly above 1; the GBMC results tended to be higher than those from Yi et al. For overweight, the HR was below 1 for both studies, significantly so for Yi et al. In the combined results, the HR for overweight was significantly below 1 and the HRs for underweight and obesity significantly above 1.

Table 2.

Hazard ratios (95% confidence intervals) for all-cause mortality by BMI category adjusted for age, sex, and smoking status with deletion of pre-existing disease

Data source	Sample size	Underweight BMI <18.5	Overweight BMI 25–29.9	Obesity Grade I BMI 30–34.9	Obesity Grade II/III BMI ≥35*
Global BMI Mortality Collaboration[20]†	8,411,814	1.68 (1.58–1.78)	0.99 (0.98–1.01)	1.25 (1.23–1.27)	1.62 (1.58–1.66)
Yi et al[21]	10 741 411	1.69 (1.67–1.71)	0.85 (0.84–0.86)	1.06 (1.04–1.08)	1.36 (1.26–1.47)
Combined	19,153,225	1.69 (1.67–1.71)	0.90 (0.89–0.91)	1.17 (1.15–1.18)	1.59 (1.56–1.63)

^*Grade II obesity (BMI 35–<40) for GBMC[20] and Grade II/III obesity (BMI ≥ 35) for Yi et al.[21]. The hazard ratio for Grade III obesity in the GBMC was 2.04 (1.97, 2.12).

^†Results shown in eTable 5 of Supplemental Appendix

When analyses adjusted for age and sex were restricted to never-smokers with no history of pre-existing disease and also deleting the first 5 years of follow-up, in both studies the HR for underweight decreased and the HR for higher BMI categories increased. In most cases, the changes were small. The individual and combined results for these analyses are shown in Table 3. In this restricted sample, in the Yi et al study, the HR for underweight was higher than that for the GBMC study, but for higher BMI categories, the HR from the Yi study was lower than those from the GBMC study. The combined results for the restricted sample showed HRs significantly greater than 1 for all BMI categories.

Table 3.

Hazard ratios (95% confidence intervals) for all-cause mortality by BMI category for never-smokers without pre-existing disease after deletion of the first 5 years of follow-up. All analyses are adjusted for age and sex

Data source	Sample size	Underweight BMI <18.5	Overweight BMI 25–29.9	Obesity Grade I BMI 30–34.9	Obesity Grade II/III BMI ≥35 *
Global BMI Mortality Collaboration[20] †	3,951,455	1.47 (1.39,1.55)	1.11 (1.10, 1.11)	1.44 (1.41, 1.47)	1.92 (1.86, 1.98)
Yi et al [21]	6,077,755	1.63 (1.60, 1.66)	0.91 (0.90, 0.91)	1.10 (1.08, 1.13)	1.34 (1.23, 1.46)
Combined	10,029,210	1.61 (1.58, 1.64)	1.02 (1.02, 1.03)	1.27 (1.25, 1.29)	1.84 (1.79, 1.90)

^*Grade II obesity (BMI 35–<40) for GBMC[20] and Grade II/III obesity (BMI ≥ 35) for Yi et al.[21]. The hazard ratio for Grade III obesity in the GBMC was 2.71 (2.55, 2.86)

^†Results shown in eTable 5 of Supplemental Appendix

Discussion

These results were broadly similar across the two studies. Both show the same general U-shaped results with higher HRs both for underweight and for obesity. Both show changes in HRs after extensive restrictions, with smaller effects in the Yi et al data. The most notable difference is that the Yi et al data shows HRs significantly below 1 for overweight relative to normal weight both before and after restriction to never-smokers and deletion of the first 5 years of follow-up. In contrast, the GBMC analyses showed an HR for overweight non-significantly below 1 initially and after restrictions the HR for overweight changed direction and was significantly above 1. The combined findings for the overweight category show no increase in mortality overall and a 2 percent increase in mortality after restriction to never-smokers and deletion of the first 5 years of follow-up, suggesting that yet larger studies are unlikely to reveal new information about the risks of mortality in this category, as Berrigan et al [23] noted in their commentary on the GBMC analyses.

The Yi et al data show little change after applying the restrictions, in contrast to the GBMC results after restrictions, which show larger increases in hazard ratios for the obesity categories and a reversal of direction for the overweight category. In the Yi et al data, the changes in HRs were small and for the highest obesity category, the HR actually decreased slightly after the restrictions.

The differences between the GBMC analyses and the Yi et al findings after restrictions may arise from a variety of sources. Relative to the GBMC study, the South Korean study is more recent and more homogeneous. The GBMC data include a variety of studies that differ in study populations, geographic regions, study years and length of follow-up. The restrictions may have deleted different proportions of the individual studies, thus changing the mix of studies, which in itself might lead to different findings after the restrictions. The sample selection procedures of the GBMC may also have preferentially favored individual studies that show changes after restrictions [24]. The South Korean study relies on data only from a single country, which limits its generalizability. Both studies rely on individual self-report of smoking and previous history of disease.

An important possible source of differences is that the GBMC data included many individual studies with self-reported rather than measured weight and height, while the Yi et al data used measured weight and height. Weight and height were measured to the nearest kg and cm by staff members at local hospitals, while examinees wore light clothing without shoes. The analysis is based on a single measurement of BMI. The commentaries by Manson et al and Willett et al did not address errors in self-reported weight and height. For overweight and obesity, HRs for self-reported BMI (calculated from self-reported weight and height) tend to be higher than for measured BMI [25, 26]. BMI calculated from self-reported weight and height tends to have systematic errors arising both from increased underreporting of weight at higher weights and from increased over-reporting of height at shorter heights [27, 28]. These characteristic errors would be expected to bias HRs for obesity upwards relative to measured BMI because of underreporting at the highest BMI levels [29–31], as documented in a number of studies [32–34]. Similar predictions have been noted in the context of other types of data, where underreporting of exposure at high levels would be expected to increase HRs [35–37]. Sociodemographic factors such as age, gender, race and socioeconomic status are also related to error in reporting [38–40] and may introduce residual confounding that is exacerbated by the restrictions. In addition, misclassifications arising from self-reported weight and height that affect the normal weight category potentially augment or reduce the effects on the HRs for obesity and can even reverse the direction of the HR for overweight[32, 33, 41].

A number of the studies included in the GBMC data set [10, 12, 13, 42–44], as well as some other studies[45] have found changes after restrictions that are similar to those observed in the GBMC analyses. Other studies, however, agree with the findings of Yi et al in also showing no effect or only a minor effect from restrictions, particularly when applied in data sets with measured weight and height data. Some of these studies have demonstrated little effect through a series of sensitivity analyses (for example[46–48] ). Others have reported qualitatively that such adjustments had little or no effect without showing quantitative details (for example “found little effect”[49], “results were unchanged”[50], “no substantial variations”[51], “similar pattern of results”[52], “not appreciably different” [53], “did not appreciably alter”[54], “did not materially change the main findings”[55], “essentially similar results”[56], “substantive findings were identical”[57], “results were very similar”[58], “did not change the results”[59], “had little effect”[60], “did not affect the HRs”[61], [not] “appreciably altered”[62]).

For some recent examples, Song et al [63] analyzed data from a collaborative study in 12 European countries that used measured weight and height data. They found that “the relationship between anthropometric measures of obesity and mortality was not substantially altered by smoking status. The potential influence of reverse causality was checked by excluding the first five years of follow-up of which less than 7% of the study population and 25% of the mortality events were excluded, and the results were not altered.” Wang et al [64] analyzed data from a large study in China and reported that “Sensitivity analyses excluding smokers, those with prevalent chronic disease or those with less than four years of follow-up did not materially alter these results.”

The Song et al article deleted the first 5 years of follow-up, as was done in the GBMC and Yi articles. However, as shown by the Wang et al article, which deleted 4 years, practices are inconsistent in this area. The study by Berrington de Gonzalez[13] deleted only 1 year of follow-up. The rationale for deleting follow-up years is that there could be participants in the sample who have lost weight due to illness and are at higher risk of mortality due to illness and deleting early follow-up will be likely to exclude such participants, if any, in the sample [65]. There is no specific evidence as to what proportion of the sample might be composed of such people or how effective this approach may be to identify them Some studies suggest little or no impact of such restrictions [66, 67].

The restrictions are used as a method of adjusting for confounding or bias due to illness-related weight loss (“reverse causation”). The extent to which these types of restrictions increase or decrease bias may vary from study to study and thus not be easily predictable. Data showing the validity of this approach are limited, and the findings are not consistent from study to study. The exact restrictions also vary from study to study [65]. Restriction to never-smokers is common, but other studies may or may not delete people with pre-existing disease and the list of diseases considered may vary. Similarly studies vary as to whether they delete some or no years of follow-up and the number of years deleted also varies from study to study. Studies that show little or no impact of the restrictions have sometimes been criticized because of their small size, particularly after the restrictions are applied [14]. However, in the current example, both studies are extremely large even after restrictions, with almost 4 million participants in the GBMC study and over 6 million participants in the Yi et al study. This reinforces the findings that the effect of the restrictions is not consistent across studies and weakens the argument that analyses without such restrictions lack validity.