Masters et al. Respond

A valuable role that journal editors can play is to create a level playing field where debates can be aired and the scientific merits of the issues can be judged by the full scientific community. It is rare for editors to bypass this step and render their own judgments, which is what happened in this case. Editors of the AJPH and the American Journal of Epidemiology have decided our approach is seriously flawed (1, 2). This decision was based largely on Dr. Hanley's assessment (3, 4), which they solicited and which was neither externally nor anonymously peer reviewed.

The Editors have allowed us only 600 words to defend our peer-reviewed articles (5, 6) in response to their editorial, Dr. Hanley's solicited commentary, and his extensive Web Appendices. We do not have space here to fully describe our position in response to these statements, and we therefore invite interested readers to examine our responses in the Web Appendix.

Our articles represented earnest efforts to address selection biases in survey-based estimates of the obesity-mortality association. We proposed an alternative to the standard method because that method completely ignores these biases. Dr. Hanley's simulations convinced him that our approach creates more problems than it solves. We respect Dr. Hanley's effort and acknowledge that it is an important part of the scientific enterprise—a strong and well-reasoned challenge. In response to Hanley's challenge, we wrote an evidence-based rebuttal that the Editors of the AJPH (then the American Journal of Public Health) declined to publish. The essence of our response was that Hanley's simulation assumptions inaccurately reflected the full scale of the selection biases that affect the obesity-mortality association in data from the National Health Interview Survey (NHIS). Nonetheless, we refitted our survival models, taking into account Dr. Hanley's concerns. Results from these new analyses were consistent with those from our original articles—namely, that apparent age-related declines in the obesity-mortality association strongly reflect selection bias.

Furthermore, we showed that the approach used in our articles corrected this bias in NHIS data and provided accurate estimates of true male-female mortality hazard ratios in official US mortality data. We did this to counter Dr. Hanley's test of our approach, in which he used known male-female hazard ratios but simulated a selection pattern that was not observed in the NHIS or the National Health and Nutrition Examination Surveys. Taken together, our analyses show that 1) Hanley's simulation bears little resemblance to real survey data and 2) our approach provides accurate estimates of known hazard ratios using data from the NHIS and National Health and Nutrition Examination Surveys. For our complete response, please see the Web Appendix.

For us, the most critical issue remains the strong likelihood that uncorrected survey estimates of the obesity-mortality relationship are biased. Hanley's own simulations show that hazard ratios estimated from a conventional approach “are quite a bit lower than those in the unselected (“population”) ones, particularly at older ages” (Web Appendix 4 of Hanley (3, 4)). In other words, according to Hanley, any approach that fails to address sample selection will produce biased estimates. Furthermore, when sample selection is not addressed in NHIS data, the results indicate that overweight and obesity reduced mortality in the US adult population by nearly 10% between 1986 and 2006 (Web Appendix). This patently absurd finding shows that the statistical cure is not worse than the bias, as Hanley alleges. It also underscores our central point that conventional approaches to estimating the obesity-mortality association are seriously flawed, with potentially devastating consequences for public-health policies that to date have not addressed the obesity epidemic with sufficient urgency. The editorial (1, 2) is, in our opinion, remiss in its failure to situate the issue in this broader framework and to recognize the challenge that lies before us all as we seek to understand the mortality effects of obesity.

Editors’ Note: Although Masters et al. are correct in saying that the commentary by Hanley was not anonymously reviewed, the methodologist at the AJPH and the 2 methodologists at the AJE who reviewed it did so very carefully and had no conflicts of interest. Their impeccable reputations convinced us of the accuracy of Hanley's concerns. In addition, as part of an overall assessment of the case, the commentary by Hanley was reviewed by an external and independent Advisory Ethical Committee convened by the AJPH that comprised 3 public health scientists whose recommendations were followed.