Confusing Uncertainty With Bias: The Challenges of Working With Real‐World Data

1.

We thank Mittinty and Walker [1] for their comment on our article on the prevalence and associates of cannabis use and cannabis use disorders in Australia [2]. They question the large odds ratios and wide confidence intervals presented, implying that they reflect bias or methodological shortcomings. In reality, they reflect one of the inherent challenges of analysing real‐world population data with rare exposures and outcomes.

Mittinty and Walker commented that such extreme numbers are not valid. Perhaps not in some cases, but they are common in estimation of drug risks in population surveys, where the prevalence of certain drug use is low. In the national survey used in our study, past 12‐month cannabis use was reported by 6.7% of the total population [2]. Comparable Australian population surveys have reported similarly low estimates, with prevalence around 10.9% [3]. The use of other illicit drugs, such as opioids, stimulants and sedatives, are even less common (< 5%). Further, only a minority of people who use these drugs develop a drug use disorder (e.g., 0.6% of the total population meet criteria for cannabis use disorder), resulting in even smaller sub‐samples.

When associations are examined between rare outcomes (e.g., cannabis use disorder) and exposures (e.g., other drug use disorder), small cell sizes and large uncertainties are inevitable. This produces asymmetrical contingency tables and, consequently, inflated odds ratios with wide confidence intervals.

As we highlighted in the discussion:

It is also important to note that these sub‐groups have low numbers, as represented by high ORs and wide confidence intervals around them … as such, our findings surrounding this should be interpreted with caution

We agree that careful consideration needs to be made when interpreting some of the odds ratios that present with large effect sizes and wide confidence intervals. These estimates are undoubtedly imprecise, but imprecision should not be confused with invalidity. Rather, they are unavoidable consequences of studying rare behaviours in representative population data. Despite the uncertainties, the findings nevertheless provide evidence of important relationships that warrant further investigation. Furthermore, the relatively few odds ratios that were imprecise should not detract from the other findings that were estimated with greater precision. These previously acknowledged methodological nuances do not negate our key conclusion that there is a need to develop a public health response to assist those most at risk of cannabis use disorder.

Author Contributions

Jack Wilson: conceptualisation, writing – review and editing. Matthew Sunderland: conceptualisation, writing – review and editing. Tim Slade: conceptualisation, writing – review and editing. Siobhan O'Dean, Danielle Dawson, Olivia Dobson, Janni Leung, Gary Chan, Maree Teesson, Wayne Hall, Nicola Newton, Valentina Lorenzetti: writing – review and editing. Emily Stockings: supervision, writing – review and editing.

Funding

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.