Assessing Heterogeneity of Treatment Effects: Are Authors Misinterpreting Their Results?

. 2010 Feb;45(1):283–301. doi: 10.1111/j.1475-6773.2009.01064.x

Box 1: Recommendations for the Interpretation (for Users) and Design and Reporting (for Producers) of Heterogeneity of Treatment Effects (HTE) Analysis

For Users	For Producers and Disseminators
1. Start with examination of the data. A. Are the HTE analyses prespecified? B. Are the reasons for the inclusion of given covariates presented as a priori rationales with plausible reasons to anticipate HTE? If not, are they labeled as exploratory? This information will help place any positive results in context. C. How many HTE analyses were performed? This information will place a positive finding for heterogeneity into context of possible false-positive results. D. Was a formal test for heterogeneity or interaction used? At what level of significance given your discipline and the subject matter would you feel comfortable saying that a test indicates plausible interaction or heterogeneity (how important is missed opportunity of identifying HTE)? E. What is the difference between subgroup and the main average treatment effect? Given the subject matter, would you feel comfortable stating that a given difference is clinically meaningful? 2. Using a reasonable scheme for coding CSD, determine the level of strength of evidence for the presence of HTE. 3. Consider the authors' interpretation of their HTE-related results in light of your own determination of CSD and base your future research or treatment recommendations on this comparison.	Researchers 1. Plan all HTE analyses with a priori rationales for the inclusion of given covariates, or clearly label the analyses as exploratory. 2. Report the total number of HTE analyses performed. 3. Use formal tests for interaction or heterogeneity when inferring HTE rather than comparing p-values between the two groups. Clearly state the level of significance used in the tests for HTE. 4. Present data from all HTE analyses performed, including p-values, effect measures, and confidence intervals. Forrest plots provide a concise method for doing so. 5. Interpret the HTE analyses performed in the discussion of the paper. Discuss the role of multiplicity when discussing positive results. Stress the exploratory nature of results if the analyses were not prespecified and/or power was insufficient. Refrain from recommending differential treatment unless confident that your HTE analyses were hypothesis testing rather than hypothesis generating and that objective evidence is strong enough to support your recommendations. Editors 1. Ensure that authors reporting any HTE analysis provide readers with enough information to place the results into context. This includes rationales for the HTE analysis; whether the analyses were primarily exploratory; the number of HTE analyses performed; whether tests for interaction or heterogeneity were used; level of significance used for the tests; and all the results from HTE analyses (Internet-only appendices and/or forest plots recommended) 2. Ensure that authors fully discuss any HTE analyses reported, even if not significant.

For Users

For Producers and Disseminators

1. Start with examination of the data. A. Are the HTE analyses prespecified? B. Are the reasons for the inclusion of given covariates presented as a priori rationales with plausible reasons to anticipate HTE? If not, are they labeled as exploratory? This information will help place any positive results in context. C. How many HTE analyses were performed? This information will place a positive finding for heterogeneity into context of possible false-positive results. D. Was a formal test for heterogeneity or interaction used? At what level of significance given your discipline and the subject matter would you feel comfortable saying that a test indicates plausible interaction or heterogeneity (how important is missed opportunity of identifying HTE)? E. What is the difference between subgroup and the main average treatment effect? Given the subject matter, would you feel comfortable stating that a given difference is clinically meaningful? 2. Using a reasonable scheme for coding CSD, determine the level of strength of evidence for the presence of HTE. 3. Consider the authors' interpretation of their HTE-related results in light of your own determination of CSD and base your future research or treatment recommendations on this comparison.

Researchers 1. Plan all HTE analyses with a priori rationales for the inclusion of given covariates, or clearly label the analyses as exploratory. 2. Report the total number of HTE analyses performed. 3. Use formal tests for interaction or heterogeneity when inferring HTE rather than comparing p-values between the two groups. Clearly state the level of significance used in the tests for HTE. 4. Present data from all HTE analyses performed, including p-values, effect measures, and confidence intervals. Forrest plots provide a concise method for doing so. 5. Interpret the HTE analyses performed in the discussion of the paper. Discuss the role of multiplicity when discussing positive results. Stress the exploratory nature of results if the analyses were not prespecified and/or power was insufficient. Refrain from recommending differential treatment unless confident that your HTE analyses were hypothesis testing rather than hypothesis generating and that objective evidence is strong enough to support your recommendations. Editors 1. Ensure that authors reporting any HTE analysis provide readers with enough information to place the results into context. This includes rationales for the HTE analysis; whether the analyses were primarily exploratory; the number of HTE analyses performed; whether tests for interaction or heterogeneity were used; level of significance used for the tests; and all the results from HTE analyses (Internet-only appendices and/or forest plots recommended) 2. Ensure that authors fully discuss any HTE analyses reported, even if not significant.