A Very Short List of Common Pitfalls in Research Design, Data Analysis, and Reporting

Introduction

Performing scientific research without falling victim to one of the many research design, analysis, and reporting pitfalls can be challenging. As a medical statistician with research experience in a variety of medical disciplines, I regularly come across (and sometimes have been the cause of) avoidable errors and inaccuracies. Without such errors, research would, at the very least, be more informative to the readership of the research manuscript. In this article I present a short, nonexhaustive list of issues to consider.

Research Questions and Aims

As the starting point of all scientific endeavors, it is incontrovertibly important to clearly define the research questions and aims. The subsequent planning of the collection of useful data and formulating adequate statistical analysis often becomes easier once it is clarified whether the ultimate aim is to predict, explain, or describe.¹ If the ultimate aim is to explain, the ideal design is often an experiment (eg, a randomized controlled trial). Conversely, for many health-related research questions, nonexperimental data are the only viable source of information. This type of data is subject to factors that hamper our ability to distinguish between true causes of outcomes and mere correlations. For instance, for a nonexperimental before-after study, a change in the health for some individuals over time is easily mistaken as evidence for the effectiveness of a particular curative treatment, which may just be caused by regression to the mean.² To avoid such errors, studies with an explanatory aim may benefit from applying causal inference methodology.³

Collecting Enough Data

A too-small-for-purpose sample size may result in overfitting,⁴ imprecision, and lack of power, which can ruin a study of any kind. It is worthwhile to calculate the minimal sample size required to avoid disappointment.⁵ It is usually wise to be skeptical about rules of thumb for sample size.⁶

Data Preparation

After data have been collected and cleaned, and initial data analysis⁷ has been completed, it often requires a large amount of self-discipline to follow the a priori defined statistical analyses plan (if one is even available). Indeed, it is hard not to look at every potential association in any given data set and even harder to unsee what is possibly just a false positive once one has started data dredging.⁸

After data collection, some researchers seem to have the natural tendency to immediately dichotomize measurements that were originally measured on a continuous or ordinal scale, such as dichotomizing the age of patients into groups of young and old. This natural tendency to dichotomize, sometimes referred to as dichotomania,⁹ is very often a bad idea.¹⁰ Various approaches exist that allow for data to be analyzed and made insightful on a continuous scale.¹¹

The Data Are Probably Error Prone, Incomplete, and Clustered

The presence of measurement and misclassification errors in data sets (present in most data sets, in my experience) are often wrongfully considered relatively unimportant.¹² Some have even argued that only the strongest effects will be detected in data that contain measurement error.¹³ This misconception that only the strongest effects will survive, I call the noisy data fallacy. Many statistical approaches exist that account for measurement and misclassification errors.¹⁴

Likewise, some degree of missing data is almost unavoidable in any study. Methods to deal with missing data, such as multiple imputation,¹⁵ have been criticized for making strong, untestable assumptions. While this is true, what is easily forgotten is that the assumptions made when ignoring missing data are often even stronger.¹⁶

Data are also often clustered. That is, data are often obtained from multiple centers, multiple studies, or multiple measurements within the same individual (eg, time series). In these settings where some data are more alike than others, it is often important to adjust the analyses accordingly.¹⁷

Statistical Significance

While many readers are quick to point out that a statistically significant effect does not mean the effect is also large enough to be relevant, it seems easier to forget that effects that are not statistically significant may not carry strong evidence that the effect does not exist.¹⁸ Contrary to popular opinion, removing variables that are not statistically significant from the analysis may not improve interpretation¹⁹ and may increase the chances of overfitting.²⁰

Given the many pitfalls in interpretation of P values and statistical (in)significance,²¹ some researchers—and even scientific journals—have called for the abandoning of statistical significance.²² It may then be tempting to ignore all uncertainty in statistical analyses and base conclusions solely on the value of a single-point estimate (eg, regression coefficient). Such point-estimate-is-the-effect-ism²³ relies heavily on the assumption that the point estimate is a valid and precise estimate of the true value, which it often is not.

Making Causal Claims

One of the keys to success for valid causal inference in nonexperimental data is the adequate handling of confounding.²⁴ Successful adjustment for confounding means being able to distinguish potential confounders from intermediates in the causal chain between the factor of interest and the outcome²⁵ and colliders,²⁶ which sometimes is more easily said than done.²⁷ If the right confounders have been selected and adjusted for through, eg, by multivariable regression analysis (notice the distinction from multivariate regression²⁸), it is tempting to also interpret the regression coefficients of the confounding variables as being corrected for confounding, which would be committing a common error known as the Table 2 fallacy.²⁹ While substantiating causal claims is often difficult, avoiding causal inference altogether or simply replacing words like “cause” by “association” is not often the solution.³⁰

Concluding Remarks

With the increasing use of machine learning and artificial intelligence in health care research, this incomplete list of common research design and analysis pitfalls may seem somewhat old-fashioned. Despite the arguably more complex nature of such analyses, many of the aforementioned issues also apply to such studies.³¹ Among all pitfalls mentioned, the easiest pitfall to avoid is that of incomplete reporting. Avoiding that type of error can be done simply enough by using reporting guidelines (see https://www.equator-network.org/).

Table 1.

Glossary of Terms

Term	Definition
Regression to the mean	When an observation is made that is distanced far from the mean value, the next observation is likely to be closer to the mean. Within individuals, a shift towards the mean can easily be confused for the effect of, for instance, a drug therapy.
Causal inference methodology	Research methodology that aims to structure analyses, identify, and quantify the evidence for one or more causal effects in data. Often applied in nonexperimental data.
Overfitting	The situation where idiosyncrasies in the data at hand are confused for generalizable associations or patterns. The risk of overfitting can be notably high when the number of model parameters is high relative to the size of the data set.
Imprecision	Lack of precision in the estimation, eg, reflected by wide confidence intervals for parameter(s) of interest.
Power	Statistical power often relates to a (binary) hypothesis test. It is the probability that the test correctly rejects the null hypothesis when a specific alternative hypothesis is true.
Initial data analysis	The process of data inspection after data have been collected but before formal statistical analyses.
Data dredging	Performing many analyses to find associations or patterns in data and subsequently ignoring the analyses in which no evidence was found when reporting the study. High risk of false positive findings.
Dichotomania	The tendency to dichotomize variables (eg, into “high” and “low” values) that were original measured on a continuous or a categorical scale with more than two categories.
Noisy data fallacy	The misconception that in data with measurement error associations are always attenuated (ie, biased towards less extreme effects).
Multiple imputation	A popular and flexible approach to deal with missing values in analyses, for which it is generally well known in which circumstances it leads to valid analyses and which circumstances it does not.
Point-estimate-is-the-effect-ism	The tendency to ignore the uncertainty of an estimated association and focus only on a single-point estimate to evaluate the strength of the association.
Confounding	In causal inference studies that evaluate the relation between a variable of interest (eg, a treatment) and an outcome, confounding can be caused by other variables that are a “common cause” of the variable of interest and outcome. Failure to statistically correct (eg, through multivariable regression) for important confounders can lead to residual confounding bias.
Collider	In causal inference studies that evaluate the relation between a variable of interest (eg, a treatment) and an outcome, a collider is a variable that is caused by the variable of interest and outcome. Unlike confounders, statistically correcting for colliders can introduce bias rather than solve it.
Multivariate regression	Regression with an outcome with multiple outcomes (eg, as in multivariate analysis of variance, MANOVA). Often confused with multivariable regression (multiple covariates in the regression).
Table 2 fallacy	The misconception that the associations between confounders and the outcome can be interpreted as valid estimates of causal associations between each confounder and the outcome.