Enhanced modeling approaches for count data analysis with focus on substance use outcomes

Abstract

The selection of appropriate statistical models is essential for accurately interpreting the analysis of count data, especially in behavioral medicine. Traditionally, Poisson and Negative Binomial models have been commonly employed, but they may not always be the most optimal choices, particularly when dealing with data with an abundance of zeroes, which can be effectively modeled using zero-inflated and zero-altered (hurdle) models. Additionally, U-shaped distributions where the data are clustered around both ends—low and high counts—with fewer occurrences in the middle, cannot be adequately captured by traditional approaches and further complicate the analysis. This paper critically examines the widespread use of zero-inflated Poisson (ZIP) and zero-inflated negative binomial (ZINB) models in the context of adolescent substance use data, identifying their potential limitations. Using a dataset from a smoking study of 1263 adolescents who reported smoking behavior across eight waves, we analyzed the sparse count outcome “Days Smoked in the Past Month,” with covariates such as sex, age, and GPA recorded at each wave. Through a comprehensive evaluation of smoking behavior count outcomes—employing model identification via the Kolmogorov–Smirnov (KS) test, validation through confirmation studies, and regression analysis guided by Akaike Information Criterion (AIC). The range of models covered includes: ZIP, Poisson hurdle (PH), ZINB, negative binomial hurdle (NBH), zero-inflated negative binomial with fixed r (ZINB-r), negative binomial hurdle with fixed r (NBH-r), zero-inflated beta-binomial (ZIBB), beta-binomial hurdle (BBH), zero-inflated beta-binomial with fixed n (ZIBB-n), beta-binomial hurdle with fixed n (BBH-n), zero-inflated beta-binomial with fixed alpha and beta (ZIBB-ab), beta-binomial hurdle with fixed alpha and beta (BBH-ab), zero-inflated beta negative binomial (ZIBNB), and beta negative binomial hurdle (BNBH). Our study demonstrates the superior model fitting and regression analysis capabilities of the ZIBB and BBH models. Notably, our findings reveal the effectiveness of the ZIBB model in capturing the U-shaped distribution observed in real-world data. This underscores the importance of exploring a wider range of models beyond ZIP and ZINB for count data analysis. This study advocates for the broader application of these more sophisticated models in behavioral medicine, with the goal to enhance the accuracy and reliability of research outcomes.

Introduction

The utilization of appropriate statistical models is fundamental for the accurate interpretation of behavioral medicine data, a field where the implications of research findings can have an impact on patient care and public health policies.

Count data—numerical outcomes representing the frequency of events—are common in behavioral research. The standard modeling approach for such data is the Poisson distribution, which assumes that the mean and variance of the outcome are equal. However, this assumption is rarely satisfied in real-world datasets, which often exhibit overdispersion—i.e., the variance exceeds the mean. In such cases, the negative binomial distribution is preferred because it introduces a dispersion parameter to account for this variability.

Another challenge in modeling count data is handling the excess number of zeros, which often reflect a mix of people who do not engage in a behavior (e.g., not smoking) and those who do but did not engage during the observation period. Standard Poisson and negative binomial models do not distinguish between these types of zeros. Zero-inflated models introduced by Lambert (1992), such zero-inflated Poisson (ZIP) and zero-inflated negative binomial (ZINB) address this by combining two processes: one that generates only zeros (to account for structural non-engagement) and another that models the count outcome, including occasional zeros, using a standard count distribution. In these models, parameters include: the probability of being in the always-zero (structural zero) group and the parameters of the count distribution (e.g., Poisson or NB) for those at risk of engaging in the behavior.

Hurdle models described in Mullahy (1986), take a different approach—they model the decision to engage (zero vs. non-zero) separately from the amount of engagement (the positive counts). Here, the parameters reflect: (1) the probability of crossing the “hurdle” to engage in the behavior (i.e., generating a non-zero count), and (2) the distribution of positive counts among those who engage. This structural difference leads to different assumptions and interpretations, especially important in behavioral data where distinguishing non-users from infrequent users matters.

Real behavioral data can show more complex patterns than ZIP or ZINB can capture. For instance, adolescent substance use data may show U-shaped distributions, where many individuals report zero use while others report consistent or frequent use. In these cases, neither standard nor zero-inflated models adequately capture the data’s shape, leading to biased estimates or poor model fit. These limitations motivate the need to explore more flexible alternatives, such as the beta-binomial model and its zero-inflated or zero-altered extensions, which allow for greater variance and complex count patterns. In these models, the parameters typically include: the underlying probability of the event (e.g., engaging in the behavior), the degree of variability (or dispersion) in that probability across individuals, and if zero-inflated, the probability of structural non-engagement (i.e., people who never engage in the behavior). These features offer more flexibility in capturing behavioral variability.

In recent years, as described below, ZIP and ZINB models have been frequently employed to analyze count sparse data in medical research. These models are particularly favored for datasets characterized by an excess of zero counts, which are common in various healthcare-related studies. While ZIP and ZINB models have served as valuable tools in the medical statistics arsenal, it is essential to critically evaluate their suitability and explore alternative modeling approaches. In this article, we explore the reasons behind the widespread adoption of ZIP and ZINB models and investigate their potential shortcomings, which may lead to erroneous and misleading interpretations. While these models have their merits, it is crucial to acknowledge their limitations, as they may not fully capture the complex patterns present in count outcomes. In this paper, we address these limitations and propose an alternative modeling approach to offer a more comprehensive and accurate representation of such count data.

Numerous publications in the field of medical statistics have predominantly relied on the ZIP and ZINB models to analyze count outcomes. For example, Wang (2003) introduced a bivariate ZINB model and applied it to healthcare utilization data and presented an estimation method using EM and quasi-Newton algorithms. Also, Hu et al. (2011) assessed Poisson, negative binomial, and their corresponding zero-inflated models for analyzing the impact of an HIV-risk reduction intervention on unprotected sexual occasions among female substance abuse treatment patients. They reported that the ZINB model offered the best fit and revealed a medium-sized intervention effect, underscoring the importance of model selection for unbiased estimates. ZINB regression has been effectively applied to analyze indoor tanning behaviors in sexual minority males, modeling session counts and their inflation components (Blashill et al., 2018).

Recent research has demonstrated that negative binomial and ZINB models provided comparable fit for count data, with the more parsimonious negative binomial model preferred for interpretability and hypothesis alignment (Finkelstein-Fox et al., 2020). Simons et al. (2006) explored the connection between marijuana problems and various factors in college students, revealing that social norms influenced non-users, while perceived use utility was linked to problem counts, showcasing the value of ZINB regression models for low base rate risk behaviors. Buu et al. (2012) addressed the gaps in analysis longitudinal zero-inflated count data by comparing hurdle and ZIP models, highlighting the latter’s potential inaccuracies when the hurdle model aligns with the conceptual framework of the data, with model performance improving with larger sample sizes, fewer missing data, and lower covariate correlations, though computational advantages of the hurdle model decline with random effects.

The study in Yang et al. (2017) presents a time-varying effect model for analyzing group differences in trajectories of ZIP outcomes, with simulations showing improved estimation accuracy with larger sample sizes, consistent hypothesis testing performance, and greater power for larger samples, emphasizing the study of group differences in the ZIP component. The Cheung (2002) article applied zero-inflated mixture models such as the Poisson and negative binomial to handle excess zero counts and investigated the influence of fetal and postnatal growth on fine motor development, emphasizing the limitations of traditional binary outcome analysis.

As these articles reveal, the majority of prior research predominantly leaned towards employing Poisson and negative binomial models, along with their respective zero-inflated and zero-altered counterparts. However, the performance of these models can degrade when applied to more complex data structures with both zero inflation and multimodal distributions, as commonly seen in behavioral datasets.

In our study, we aimed to address this gap by exploring a broader spectrum of models for count data and assess their goodness of fit using the Kolmogorov–Smirnov (KS) test. Our objective is to identify the most suitable model for accurately representing the underlying data (described in Section “Adolescent smoking study”). We also extended our analysis to encompass various types of zero-inflated and zero-altered models. To delve further into this research gap, we applied the KS test to a dataset from a longitudinal study of smoking behavior in adolescents using our R package AZIAD, which offers 15 different distribution options for enhanced flexibility. Our findings revealed that zero-altered and zero-inflated beta-binomial distributions were suitable fits based on the KS test results.

The paper is organized as follows: Section “Model selection for zero-inflated and zero-altered count data via the KS test” describes model identification for zero-inflated and zero-altered data, particularly focusing on the KS test and its significance within the context of our research. Section “Zero-inflated and zero-altered regression models” provides an overview of zero-inflated and zero-altered regression models, describing essential background information on these models for count data. In Section “Adolescent smoking study”, we introduce the longitudinal adolescent smoking dataset. Section “Model identification applying KS test” present the result pf the KS test, detailing the data collection process and the outcome of applying the test to each wave of the dataset. Section “Confirmation study: determination of distribution” present a confirmation study to elucidate the rationale for selecting the proposed models. In Section “Regression analysis” we conduct a comprehensive regression analysis, meticulously examine the estimated regression parameters and compare them with descriptive statistics from the observed data. Finally, Section “Conclusion and discussion” concludes the paper with a discussion of our findings and their implications.

Methodology

Model selection for zero-inflated and zero-altered count data via the KS test

The KS test introduced by Massey (1951), is a commonly used method to assess whether a random sample $Y_1,\dots ,Y_n$ follows a continuous cumulative distribution function $F_{\theta }(y)$ with specified parameter(s) $\theta$. This test relies on the KS statistic $D_n=\underset{y}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|F_n(y)-F_{\theta }(y)\right|$, where $F_n(y)=n^{-1}\sum _{i=1}^n{1}_{(-\mathrm{\infty },y]}\left(Y_i\right)$ represents the empirical distribution function. In Dousti Mousavi et al. (2023) a bootstrapped Monte Carlo method is employed to estimate the p-value of the KS test for model selection involving zero-inflated and zero-altered (or hurdle) models.

The KS test offers a nonparametric goodness-of-fit diagnostic that evaluates how well a model’s predicted distribution matches the empirical distribution of the observed data. In contrast to relative model selection criteria such as AIC or BIC, which compare models within a candidate set, the KS test provides absolute validation by directly testing whether the observed distribution significantly deviates from the one implied by a fitted model (Stephens, 1974; Thode, 2002).

This distinction is particularly important in the context of zero-inflated count models with varying distributional assumptions (e.g., Poisson, negative binomial, beta-binomial), which may yield similar information criteria, they can differ in how well they capture key data features such as overdispersion or excess zeros. The KS test thus offers an essential second layer of evaluation: it tests whether the selected model provides a statistically plausible description of the data.

In our application, we incorporate both perspectives. AIC and BIC guide relative comparisons across models, while the KS test ensures the chosen model fits the empirical data well. A model passes this test if its bootstrapped p-value exceeds 0.05, indicating no significant deviation between the model-predicted and observed distributions.

This analysis is facilitated through an R package AZIAD, developed by Dousti Mousavi et al. (2022), which offers a range of functions for simulating zero-inflated or zero-altered data and computing maximum likelihood estimates for various distributions including Poisson, geometric, negative binomial, beta-binomial, beta-negative binomial, and their respective zero-inflated and zero-altered models (i.e., three distributions for each model). The likelihood function ($\varphi ,\mathit{\theta }$) of a zero-altered model is

$$L(\varphi ,\mathit{\theta })={\varphi }^{n-m}{(1-\varphi )}^m\cdot \prod _{i:Y_i\ne 0}{f}_{tr}(Y_i\mid \mathit{\theta })$$ (1)

where $m=\#\left\{i:Y_i\ne 0\right\}$ represents the number of non-zero observations, n is the total number of observations, $\varphi \in [\mathrm{0,1}]$, denotes the proportion of zero and $\mathit{\theta }$ is a vector of parameters associated with the baseline count distribution. In accordance with Theorem 1 in Aldirawi and Yang (2022) the maximum likelihood estimate (MLE) of ($\varphi ,\mathit{\theta }$) that maximizes (1) is

$$\widehat{\varphi }=1-\frac{m}{n},\widehat{\mathit{\theta }}=\text{arg}\underset{\mathit{\theta }}{\text{max}}\prod _{i:Y_i\ne 0}{f}_{tr}(Y_i\mid \mathit{\theta })$$ (2)

Also, the likelihood function of a zero-inflated model $(\varphi ,\mathit{\theta })$ can be written as

$$\begin{gathered} L(\varphi ,\mathit{\theta })={[\varphi +p_0(\mathit{\theta })(1-\varphi )]}^{n-m} \\ \cdot {(1-\varphi )}^m{(1-p_0(\mathit{\theta }))}^m\cdot \prod _{i:Y_i\ne 0}{f}_{tr}(Y_i\mid \mathit{\theta }) \end{gathered}$$ (3)

where $m=\#\left\{i:Y_i\ne 0\right\}$, and $f_{tr}(y;\mathit{\theta })=f_{\mathit{\theta }}(y)/\left[1-p_0(\mathit{\theta })\right],y\ne 0$ is the truncated version of the count distribution. Based on Theorem 4 in Aldirawi and Yang (2022), the maximum likelihood estimate ($\widehat{\varphi },\widehat{\mathit{\theta }}$) that maximizes (3) is obtained as follows:

1. $\widehat{\theta }=\text{arg}{\text{max}}_{\theta }{\prod }_{i:Y_i\ne 0}{f}_{tr}(Y_i\theta )$, and $\widehat{\varphi }=1-{[1-p_0({\theta }_{\text{*}})]}^{-1}\cdot m/n$, where $m/n\le 1-p_0({\mathit{\theta }}_{\text{*}})$

2. Otherwise, $\widehat{\theta }=\mathrm{a}\mathrm{r}\mathrm{g}{\mathrm{m}\mathrm{a}\mathrm{x}}_{\theta }(\psi (\theta ),\theta )$ and $\hat{\varphi }=1-\psi \left(\widehat{\theta }\right)\cdot {\left[1-p_0\left(\widehat{\theta }\right)\right]}^{-1}$, where $\psi (\theta )=\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{m}{n},1-p_0(\theta ),\right\}$ and $L(\psi ,\theta )=(1-\psi {)}^{n-m}{\psi }^m{\prod }_{i:Y_i\ne 0}{f}_{tr}\left(Y_i\theta \right)$.

The algorithm used to estimate the p-value of a KS test and assess the goodness-of-fit of a statistical model to the data using a nested bootstrap approach in the AZIAD package works as follows. Given data $=Y_1,Y_2,\dots ,Y_n$, first calculate the Maximum Likelihood Estimation (MLE) $\widehat{\theta }$ and the KS statistic $D_n$. For B iterations (bootstrap samples), first we resample the data with replacement to create a bootstrapped sample $Y^{(b)}$. Second, we calculate the MLE ${\theta }^b$ based on the bootstrapped sample. Third, we simulate a random sample $Y^{(c)}$ from the distribution defined by ${\mathrm{\Theta }}^{(b)}$. Fourth, calculate the MLE ${\theta }^{(c)}$ based on the simulated sample and calculate a bootstrapped KS statistic $D_n^{(b{)}^{\prime }}$ based on $Y^{(c)}$. Finally, we estimate the p-value of the KS test as the proportion of bootstrapped KS statistics that exceed the original D_ns tatistic. If the estimated p-value exceeds0.05, we conclude that the specified distribution successfully passes the KS test for the given dataset.

Zero-inflated and zero-altered regression models

The zero-inflated and zero-altered regression models are statistical techniques employed for analysis of count data with excessive zeros, such as in count data or healthcare utilization studies (Heilbron, 1994; Preisser et al., 2015). The models are characterized by two parameters: ${\varphi }_i\in [\mathrm{0,1}]$, which represents the proportion of zeros, and ${\mathit{\theta }}_i={\left({\theta }_{i1},\dots ,{\theta }_{im}\right)}^T\in {ℝ}^m$ is a parameter vector representing the parameters of the baseline distribution in the model (i.e., $f_{{\mathit{\theta }}_i}(y)$).

In this study, we follow the approach outlined in Aldirawi (2020), where a set of independent observations $\left(Y_i,{\mathit{x}}_i\right)$ for $i=1,\dots ,k$ is considered. Each observation comprises covariates ${\mathit{x}}_i={\left(x_{i1},\dots ,x_{id}\right)}^T\in {ℝ}^d$ in a d-dimensional space, and a corresponding response $Y_i\in ℝ$ in the real number domain. It’s important to note that x_i refers to the design matrix. Here, k represents the number of subjects or observations, m denotes the number of parameters in the baseline distribution, and d signifies the number of covariates considered in the design matrix. This comprehensive framework facilitates a nuanced analysis of count data, allowing us to capture both the presence of excess zeros and the underlying distribution of non-zero values. If Y follows a zero-inflated model, we have

$$\begin{gathered} f_{ZI}(y\mid {\varphi }_i,{\mathit{\theta }}_i)={\varphi }_i+(1-{\varphi }_i)p_0({\mathit{\theta }}_i)\cdot 1_{\{\mathit{y}=0\}} \\ +(1-{\varphi }_i)f_{{\mathit{\theta }}_{\mathit{i}}}(y)\cdot 1_{\{y\ne 0\}} \end{gathered}$$ (4)

while if Y follows a zero-altered (hurdle) model we have

$$f_{ZA}(y\mid {\varphi }_i,{\mathit{\theta }}_{\mathit{i}})={\varphi }_i\cdot 1_{\{y=0\}}+\frac{1-{\varphi }_i}{1-p_0({\mathit{\theta }}_i)}{f}_{{\theta }_i}(y)\cdot 1_{\{y\ne 0\}}$$ (5)

where $f_{{\mathit{\theta }}_{\mathit{i}}}(y)$ corresponds to the baseline distribution. Also, in both zero-inflated and zero-altered regression models, we presume the presence of link functions denoted as $g$ and $h_1,\dots ,h_m$, which link the regression parameters $\gamma$ and ${\mathit{\beta }}_1,\dots ,{\mathit{\beta }}_m$ corresponding, respectively to the proportion of zeroes and the model parameters in the baseline distribution (6) derived in Aldirawi (2020). Additionally, covariate matrices ${\mathit{G}}_i={\left(u_1\left({\mathit{x}}_i\right),\dots ,u_s\left({\mathit{x}}_i\right)\right)}^T\in {ℝ}^s$ an ${\mathit{B}}_{ij}={\left(q_{j1}\left({\mathit{x}}_i\right),\dots ,q_{j{t}_j}\left({\mathit{x}}_i\right)\right)}^T\in {ℝ}^{t_j}$ correspond to the predictors, with $u_i$’s and $q_{ji}$’s being established functions. For example, for a main-effect model, the model matrix is represented as ${\mathit{G}}_i={\mathit{B}}_{ij}={\left(1,x_{i1},\dots ,x_{id}\right)}^T$, where each component captures the main effect of a variable. On the other hand, for models involving both main effects and interactions, the model matrix ${\mathit{G}}_i={\mathit{B}}_{ij}={\left(1,x_{i1},\dots ,x_{id},x_{1i}{x}_{i2},\dots ,x_{(d-1)i}{x}_{di}\right)}^T$ includes the additional terms account for pairwise interactions between variables. These examples showcase the flexibility of ${\mathit{G}}_i$ in capturing different model structures and complexities.

The Maximum Likelihood Estimate (MLE) for $\gamma$ and ${\beta }_j$ is obtained through the fitting of a Generalized Linear Model (GLM) using response variables that correspond to the observed count data. These response variables are typically the counts of events or occurrences being studied, such as the number of days smoked in the past month instances in our analysis. By fitting the GLM to these count responses, we can estimate the regression parameters $\gamma$ and ${\beta }_j$ that govern the relationship between the predictors and the count outcomes.

Specifically, MLE for $\mathit{\gamma }$ is determined by fitting a generalized linear model (GLM) using $1_{Y_i}=0$ as the response variable. Additionally, for the estimation of ${\mathit{\beta }}_j$, the maximization involves a likelihood function for non-zero observations $1_{Yi\ne 0}$, expressed as ${\prod }_{i:Y_i\ne 0}{f}_{tr}\left(Y_i\mathit{\theta }\right)/\left[1-p_0\left({\mathit{\theta }}_i\right)\right]$. The detailed derivations of the log likelihood and its first derivatives are provided in the appendix based on (Aldirawi, 2020). These details are sufficient for computing the Maximum Likelihood Estimates (MLEs) using optimization algorithms such as ‘optim’ in R.

$$\{\begin{array}{ll}g({\varphi }_i)={\mathit{G}}_i^T\gamma ,\hfill & i=1,\dots ,k\hfill \\ h_j({\theta }_{ij})={\mathit{B}}_{ij}{\mathit{\beta }}_{\mathit{j}},\hfill & i=1,\dots ,k;j=1,\dots ,m\hfill \end{array}$$ (6)

We utilized the RAZIAD package (Dousti Mousavi et al., 2024) which offers a comprehensive set of regression models capable of estimating parameters and calculate Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) values for various baseline distribution scenarios in both zero-inflated and zero-altered response contexts. These models employ four distinct link functions for the parameter φ such as logit link function, probit link function, log–log link function, and c-log–log link function, as well as an exponential link function for the parameter $\mathit{\theta }$. The range of models covered includes: ZIP, Poisson hurdle (PH), ZINB, negative binomial hurdle (NBH), zero-inflated negative binomial with fixed r (ZINB-r), negative binomial hurdle with fixed r (NBH-r), zero-inflated beta-binomial (ZIBB), beta-binomial hurdle (BBH), zero-inflated beta-binomial with fixed n (ZIBB-n), beta-binomial hurdle with fixed n (BBH-n), zero-inflated beta-binomial with fixed $\alpha$ and $\beta$ (ZIBB-ab), beta-binomial hurdle with fixed alpha and beta (BBH-ab), zero inflated beta-negative binomial (ZIBNB), and beta-negative binomial hurdle (BNBH).

It is worth noting that ZINB-r, represents a zero-inflated negative binomial distribution where the baseline distribution has model parameters (r, p), and fixing r implies that the parameter r is held constant. Implying that only the intercept is estimated with respect to this parameter, while the other covariates are not. For simplicity, an all-one vector was utilized in our analysis when r is fixed, drawing inspiration from methodologies described in Section 11.2 of McCullagh and Nelder (2019) and Section 5.4.6 of Gu (2013) Similar conventions are applied to other models, including NBH-r, ZIBB-n, BBH-n, ZIBB-ab, and BBH-ab. Additionally, in a zero-inflated beta-binomial distribution, the model parameters in the baseline distribution denoted as $\mathit{\theta }=(n,\alpha ,\beta )$. Fixing these parameters contributes to simplification in interpretation, enhances identifiability, and improves computational efficiency.

To further assess the estimated parameters in zero-inflated and zero-altered regression models, one may express the expected value of $f_{ZI}\left(y\mid {\varphi }_i,{\mathit{\theta }}_i\right)$ and $f_{ZA}\left(y\mid {\varphi }_i,{\mathit{\theta }}_i\right)$ using the following equations:

$$E[f_{ZI}(y\mid {\varphi }_i,{\mathit{\theta }}_i)]=(1-{\varphi }_i)\cdot E[f_{{\mathit{\theta }}_{\mathit{i}}}(y)]$$ (7)

$$E[f_{ZA}(y\mid {\varphi }_i,{\mathit{\theta }}_i)]=\frac{1-{\varphi }_i}{1-p_0({\mathit{\theta }}_i)}\cdot E[f_{{\mathit{\theta }}_i}(y)]$$ (8)

These equations are particularly useful for assessing model predictions and making insightful inferences about the observed data. By incorporating the expected values, researchers can gain valuable insights into the behavior of zero-inflated and zero-altered regression models. The utility of these expressions lies in their ability to facilitate a meaningful comparison between the model’s predictions and the actual observed data. In cases where interpreting the estimated parameters in such models may be challenging, as shown in Table 6, Eq. (1) can serve as valuable tools to enhance the interpretability of model outcomes. To provide a more comprehensive understanding, this will be utilized in Example 2.

Table 6.

Confidence intervals for estimated parameters derived from the Fisher Information Matrix in a zero-inflated beta-binomial regression model with the logit Link Function

	Estimation	Lower bound	Upper bound	S.E	z-score
${\widehat{\gamma }}_1$ (intercept)	−2.974	−3.996	−1.953	0.521	−5.707
${\widehat{\gamma }}_2$ (GPA)	1.331	0.894	1.768	0.223	5.975
${\widehat{\gamma }}_3$ (sex.female)	0.294	−0.273	0.861	0.289	1.017
${\widehat{\beta }}_{11}$ (intercept)	3.367	3.347	3.388	0.011	318.036
${\widehat{\beta }}_{12}$ (intercept)	−0.757	−1.251	−0.264	0.252	−3.007
${\widehat{\beta }}_{22}$ (GPA)	0.001	−0.287	0.289	0.147	0.008
${\widehat{\beta }}_{32}$ (sex.female)	−0.466	−0.932	0.000	0.238	−1.960
${\widehat{\beta }}_{13}$ (intercept)	0.145	−0.216	0.507	0.184	0.787
${\widehat{\beta }}_{23}$ (GPA)	−0.020	−0.224	0.184	0.104	−0.192
${\widehat{\beta }}_{33}$ (sex.female)	−0.322	−0.624	−0.021	0.154	−2.095

In summary, Section “Model selection for zero-inflated and zero-altered count data via the KS test” outlines the process of model identification using the KS test for general, zero-inflated, and zero altered data, while Section “Zero-inflated and zero-altered regression models” entails conducting regression analysis on the zero-inflated response with various covariates, incorporating different models for the count data, and performing model comparisons based on the AIC selection criteria.

Empirical analysis

Adolescent smoking study

The data for our illustration were sourced from a comprehensive natural history study on adolescent smoking, as detailed in Dierker et al. (2015) and Hedeker et al. (2008). This study targeted 9th and 10th-grade students at baseline, specifically recruiting individuals who had a history of previous smoking. Notably, participants were identified through a screening questionnaire completed 6–8 weeks before baseline, wherein they self-reported having smoked at least one cigarette in their lifetime. At baseline, 55.1% of the participants were female. Additionally, the majority of the participants (57.6%) acknowledged having smoked at least one cigarette in the past month.

In this study, we conducted analyses spanning eight waves collected over several years of the count outcome “Days Smoked in the Past Month”. The dataset consists of 1263 subjects, with covariates such as sex, age, at each wave and GPA recorded for each individual for wave 0 to wave 3. The dataset comprises 715 females and 547 males. Notably, the baseline subjects’ age ranges from 13.9 to 17.5, with an average age of 15.6. By the eighth wave, the subjects’ age varies from 22.2 to 28.1, with an average age of 24.5. It is crucial to note that most of the waves contained some missing values, and that the quantity of missing data varied across the different waves: 1 in wave 0 (at baseline), 107 in wave 1 (at 6 month), 146 in wave 2 (at 15 month), 123 in wave 3 (at year 2), 121 in wave 4 (at 33 month), 171 in wave 5 (at year 4), 239 in wave 6 (at year 5), 194 in wave 7 (at year 6), 197 in wave 8 (at 7 year), and 255 in wave 9 (at 8 year).

Although the dataset is longitudinal, this study presents a cross-sectional approach by analyzing each wave independently as an annual snapshot of adolescent smoking behavior. This approach was chosen to isolate model performance across distinct time points, simplify interpretation, and highlight the strengths of alternative count models—particularly in sparse, zero-inflated settings. Each wave offers an independent opportunity to evaluate model fit, as reflected in the KS test results and model comparisons in Section “Confirmation study: determination of distribution”. To illustrate regression-based modeling, we focus on wave 2 as a representative example. While this study focuses on wave-level comparisons, our future work will extend these methods to longitudinal models.

All analyses were performed using available (non-missing) cases at each wave. We did not impute missing data, as our primary objective was model comparison. As noted by Little and Rubin (2014) when data are Missing at Random (MAR), likelihood-based inference using the observed data yields valid estimates without the need to model the missingness mechanism. Given our use of likelihood-based model fitting at each wave, available-case analysis is appropriate under the MAR assumption.

Figure 1 illustrates histograms with displays of the frequency of days smoked in the past month for both male and female subjects at each wave. It also includes information about the proportions of zero values and the averages of non-zero values for each time point. As evident from the mean results for the nonzero segments of the data, males consistently exhibit higher averages compared to females across all waves. This suggests that, on average, male subjects tend to have a greater number of “days smoked in the past month” compared to female subjects. Furthermore, in nearly all waves, females display a higher proportion of zero values in comparison to males. This indicates that females tend to have a greater number of instances with zero “days smoked in the past month” results as compared to males. Finally, it can be seen that zero-inflation appears present for both sex at all waves.

Fig. 1.

Histogram of days smoked in past month by sex and wave

Model identification applying KS test

In the preliminary stage of our analysis, we conducted the KS test on the distribution of each wave, as well as on the male and female samples within each wave. We evaluated these distributions against a set of 15 different distributions, as outlined in our AZIAD package, to determine which distribution best matched the data.

These distributions include commonly used count models such as Poisson, negative binomial, beta-binomial, and their zero-inflated and hurdle variants. We chose the KS test due to its nonparametric nature and sensitivity to discrepancies in both central tendency and shape, offering a robust measure of fit across models.

Tables 1 and 2 present the p-values obtained from applying the KS test to each wave of data for the female and male groups, respectively, considering various distributions. Notably, the ZIBB model and the BBH model emerged as promising candidates for both groups, as evidenced by bolded p-values exceeding 0.05, indicating no strong evidence of misfit. In our analysis, a p-value below 0.05 would signify misfit, while those with higher p-values suggest a good fit to the data. Consequently, our attention was directed towards the results of the estimated parameters, specifically when the underlying distribution was assumed to be the beta-binomial hurdle model.

Table 1.

The KS test results for each wave of data for male subjects with bolded p-values > 0.05, suggest that the data follows the tested distribution

	Poisson	geometric	nb	bb	bnb	zip	zigeom	zinb	zibb	zibnb	ph	geomh	nbh	bbh	bnbh
wave 0	0.001	0.001	0.001	0.005	0.095	0.005	0.05	0.015	0.010	0.100	0.001	0.005	0.001	0.010	0.070
wave1	0.001	0.001	0.001	0.035	0.020	0.001	0.010	0.015	0.035	0.010	0.001	0.010	0.001	0.165	0.010
wave2	0.001	0.001	0.001	0.015	0.001	0.001	0.010	0.015	0.025	0.010	0.001	0.010	0.001	0.550	0.010
wave3	0.001	0.001	0.001	0.010	0.001	0.001	0.001	0.001	0.035	0.001	0.001	0.001	0.001	0.065	0.001
wave4	0.001	0.001	0.001	0.230	0.001	0.001	0.001	0.040	0.300	0.001	0.001	0.001	0.001	0.080	0.001
wave5	0.001	0.001	0.001	0.035	0.001	0.001	0.001	0.001	0.225	0.001	0.001	0.001	0.001	0.530	0.001
wave6	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.145	0.001	0.001	0.001	0.001	0.545	0.001
wave7	0.001	0.001	0.001	0.105	0.001	0.001	0.001	0.001	0.245	0.001	0.001	0.001	0.001	0.305	0.001
wave8	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.670	0.001	0.001	0.001	0.001	0.785	0.001
wave9	0.001	0.001	0.001	0.195	0.001	0.001	0.001	0.001	0.430	0.001	0.001	0.001	0.001	0.505	0.001

nb: negative binomial, bb: beta-binomial, bnb: beta-negative binomial, zip: zero-inflated Poisson, zigeom: zero-inflated geometric, zinb: zero-inflated negative binomial, zibb: zero-inflated beta-binomial, zibnb: zero-inflated beta-negative binomial, ph: Poisson hurdle, geomh: geometric hurdle, nbh: negative binomial hurdle, bbh: beta-binomial hurdle, bnbh: beta-negative binomial hurdle

Table 2.

The KS test results for each wave of smoke data for female, with bolded p-values > 0.05, suggest that the data follows the tested distribution

	Poisson	geometric	nb	bb	bnb	zip	zigeom	zinb	zibb	zibnb	ph	geomh	nbh	bbh	bnbh
wave 0	0.001	0.001	0.001	0.001	0.040	0.001	0.001	0.001	0.001	0.065	0.001	0.001	0.001	0.001	0.060
wave 1	0.001	0.001	0.001	0.005	0.025	0.001	0.001	0.001	0.005	0.035	0.001	0.001	0.001	0.001	0.060
wave 2	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.050	0.001
wave 3	0.001	0.001	0.001	0.015	0.001	0.001	0.001	0.001	0.015	0.001	0.001	0.005	0.001	0.055	0.001
wave 4	0.001	0.001	0.001	0.015	0.001	0.001	0.001	0.005	0.030	0.005	0.001	0.001	0.001	0.065	0.001
wave 5	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.001	0.055	0.001	0.001	0.001	0.001	0.095	0.001
wave 6	0.001	0.001	0.001	0.035	0.001	0.001	0.001	0.001	0.060	0.001	0.001	0.001	0.001	0.130	0.001
wave 7	0.001	0.001	0.001	0.175	0.001	0.001	0.001	0.001	0.290	0.001	0.001	0.001	0.001	0.390	0.001
wave 8	0.001	0.001	0.001	0.085	0.001	0.001	0.001	0.001	0.140	0.001	0.001	0.001	0.001	0.210	0.001
wave 9	0.001	0.001	0.001	0.200	0.001	0.001	0.001	0.001	0.280	0.001	0.001	0.001	0.001	0.325	0.001

nb: negative binomial, bb: beta-binomial, bnb: beta-negative binomial, zip: zero-inflated Poisson, zigeom: zero-inflated geometric, zinb: zero-inflated negative binomial, zibb: zero-inflated beta-binomial, zibnb: zero-inflated beta-negative binomial, ph: Poisson hurdle, geomh: geometric hurdle, nbh: negative binomial hurdle, bbh: beta-binomial hurdle, bnbh: beta-negative binomial hurdle

We applied the KS test separately by sex to account for documented differences in adolescent smoking behavior between males and females, including differences in initiation rates, frequency of use, and contextual factors influencing tobacco use (Hawkins et al., 1992; Jefferis et al., 2003; Johnston et al., 2016). Evaluating model fit separately for each group allows for a more accurate assessment of distributional assumptions and avoids potential masking of poor fit that can occur when heterogeneous subgroups are combined.

Example 1

Beta-binomial hurdle (BBH). The zero-altered beta-binomial model has been applied in Aldirawi and Yang (2022) which incorporated the use of a beta-binomial hurdle model for analyzing microbiome data. In a BBH model, the baseline distribution is the beta-binomial distribution with the probability mass function (pmf) given by $f_{\mathit{\theta }}(y)=\left(\genfrac{}{}{0pt}{}{n}{y}\right)\frac{\mathit{Beta}\left(y+\alpha ,n-y+\beta \right)}{\mathit{Beta}\left(\alpha ,\beta \right)},y\in \left\{\mathrm{0,1},2,\dots ,n\right\}$ with parameters $\mathit{\theta }=(n,\alpha ,\beta )$ and $p_0(\mathit{\theta })=\frac{\mathit{Beta}(\alpha ,n+\beta )}{\mathit{Beta}(\alpha ,\beta )}$.

Therefore, the pmf for the beta-binomial hurdle model can be written as

$$\begin{gathered} f_{ZA}(y\mid \varphi ,n,\alpha ,\beta )=\varphi .1\{y=0\} \\ +\frac{1-\varphi }{1-p_0(\mathit{\theta })}{f}_{\mathit{\theta }}(y).1_{\{y\ne 0\}} \end{gathered}$$ (9)

Therefore, our focus is on the estimated parameters of the beta-binomial hurdle distribution (i.e., $\widehat{\varphi },\widehat{n},\widehat{\alpha },\widehat{\beta }$) derived by maximum likelihood estimation for days smoked at each wave among male and female. As observed in Tables 3 and 4, the zero-altered (hurdle) parameter $\widehat{\varphi }$ signifies the likelihood of observing zeros in the data (i.e. zero days smoked in past month), with higher $\widehat{\varphi }$ values indicating a greater probability of excess zeros due to special circumstances. Among males, estimated $\widehat{\varphi }$ values range from approximately 0.37 to 0.56, while among females, they range from approximately 0.48 to 0.65, suggesting that females tend to have a slightly higher proportion of zero smoking occurrences, reflecting fewer instances of smoking in most waves compared to males. Comparing the estimated results and the actual data, we note that the estimated $\widehat{\varphi }$ values for both the male and female groups align perfectly with the observed data.

Table 3.

Estimated parameters based on MLE using beta-binomial Hurdle model for male group

	wave 0	wave 1	wave 2	wave 3	wave 4	wave 5	wave 6	wave 7	wave 8	wave 9
$\widehat{\mathrm{\varphi }}$	0.55	0.57	0.55	0.54	0.45	0.42	0.38	0.37	0.40	0.52
$\widehat{n}$	30	30	30	30	30	30	30	30	30	30
$\widehat{\alpha }$	0.01	0.01	0.01	0.08	0.01	0.07	0.03	0.11	0.04	0.09
$\widehat{\beta }$	0.60	0.45	0.26	0.24	0.16	0.14	0.14	0.16	0.13	0.13

Table 4.

Estimated parameters based on MLE using beta-binomial Hurdle model for female group

	wave 0	wave 1	wave 2	wave 3	wave 4	wave 5	wave 6	wave 7	wave 8	wave 9
$\widehat{\mathrm{\varphi }}$	0.55	0.59	0.60	0.60	0.55	0.53	0.48	0.52	0.57	0.65
$\widehat{n}$	30	30	30	30	30	30	30	30	30	30
$\widehat{\alpha }$	0.01	0.01	0.01	0.01	0.01	0.01	0.03	0.01	0.01	0.01
$\widehat{\beta }$	0.59	0.45	0.29	0.26	0.19	0.20	0.19	0.15	0.14	0.14

To examine the additional estimated parameters (i.e., $\widehat{n},\widehat{\alpha },\widehat{\beta }$), we reparametrized them according to the methodology outlined in Dolzhenko and Smith (2014). Specifically, $\pi =\frac{\alpha }{\alpha +\beta }$ serves as the analog of the binomial probability of success parameter and can be interpreted as the average. The parameter $\gamma =\frac{1}{\alpha +\beta +1}$ is referred to as the dispersion parameter. Since the baseline distribution $f_{\mathit{\theta }}(y)$ is beta binomial with $E(y)=n\cdot \pi$ and $\mathit{var}(y)=n\pi (1-\pi )(1+(n-1)\gamma )$. Note that $\widehat{n}$ represents the upper limit of observations. In the context of these longitudinal data, it has been consistently estimated as 30 for each wave.

Since these are common choices for analysis of count data, we have included the parameter estimates obtained by fitting both a ZIP model and a ZINB for both the male and female groups in the appendix section (Tables 7, 8, 9 and 10). In the analysis presented in Tables 7 and 8, it was observed that the ZIP model demonstrated robust performance in accurately estimating the parameter $\varphi$. whereas, the ZINB model, as presented in Tables 9 and 10, exhibited slight discrepancies in estimating $\varphi$. To compare the rest of the estimates in ZIP, ZINB, and the preferred model ZIBB with the actual data, we conducted a comparison of the variability patterns between the real data (non-zero portion) and those based on the estimated parameters. Specifically, we considered beta-binomial estimates based on $\frac{1}{\alpha +\beta +1}$, Poisson estimates based on $\sqrt{\mathrm{\lambda }}$, and negative binomial estimates based on $\sqrt{\frac{n\left(1-p\right)}{p^2}}$. Figure 2 summarizes the variability patterns among these three models and the original data.

Fig. 2.

Assessing variability in “Days smoked in the past month” across waves, contrasting original data (a) with estimates from beta binomial (b), negative binomial (c), and Poisson (d) models. Note: The origi-data (Panel a) is shown separately due to differences in variability scaling across models, to allow clearer interpretation of model-based estimates

Our observations reveal that the variability pattern of BBH closely aligns with the original data, exhibiting a consistent match across various waves. In most waves females and males have similar variability, with some instances where females exhibit more variability than males. While ZIP does not perfectly match the real data, it notably shows a reduction in variance for some waves, contradicting the observed data trend. ZIP also suggest that males are overall less variable than females, which is not consistent with the variability pattern observed in the actual data.

On the other hand, ZINB does not follow the data trend well at all. Initially, it suggests that males have higher variability compared to females, contrary to the observed data, and later it indicates less variability in males compared to females, which is not in line with the actual trend. These nuanced differences underscore the importance of carefully assessing the model fit and variability patterns, providing valuable insights into the strengths and limitations of each modeling approach. It is crucial to note that these discrepancies highlight the need for a more nuanced understanding of how ZIP and ZINB models may misrepresent certain aspects of the observed data.

These modeling results underscore the substantive importance of selecting a model that reflects key behavioral features of the data. The higher $\widehat{\mathrm{\varphi }}$ estimates among females in BBH and ZIBB models, for example, align with known trends of lower smoking prevalence in adolescent females.

In addition, both models incorporate an upper bound parameter (n = 30), which explicitly accounts for the 30-day reporting window and ensures that predictions remain within a plausible behavioral range. This structural feature is absent in standard count models like Poisson or negative binomial, which may allow unrealistic predictions in bounded contexts. As shown in Fig. 2, these models produce misleading conclusions about gender-based differences in variability, which could lead to incorrect inferences about behavioral heterogeneity. Together, these challenges highlight the importance of choosing models that are not only statistically well-fitting but also substantively meaningful for interpreting outcomes in applied settings.

Confirmation study: determination of distribution

Figure 3 offers a further view of our modeling approaches and their impact on the analysis of smoking behavior data at Wave 9, which involved a sample size of 1007 participants. Although this confirmation study was conducted across all ten waves, we chose to present results from Waves 8 and 9 as representative examples to avoid redundancy and to highlight findings from the later stages of data collection. Wave 2, which is used in the regression analyses in Section “Regression analysis”, showed similar results in this confirmation study, provided in the Appendix. In plot (a), we present the original data at Wave 9. This plot provides a visual representation of the actual smoking behavior data, serving as the starting point for our analysis. The vertical bars represent the observed frequency of different smoking outcomes among the participants. In plot (b), we estimated the parameters of the original data using the ZIP model and generated simulated data based on these estimated parameters. The purpose of this plot is to illustrate how well or how badly the ZIP model aligns with the original data. Plot (c) demonstrates a similar simulation but this time based on estimates from the ZINB model. By contrasting this plot with the observed data, we can assess the performance of the ZINB model in replicating the observed smoking behavior patterns. Finally, in the fourth plot (d), we conducted simulations based on parameter estimates derived from the ZIBB model. Remarkably, this plot showcases how the ZIBB model excels in closely simulating the original data, suggesting its superiority in capturing the intricacies of smoking behavior patterns. Upon careful consideration, simulations based on estimated parameters in ZIP and ZINB models do not align well with the original data. This confirms the results obtained from our KS test, reinforcing the notion that ZIP and ZINB are not optimal models for this dataset. These findings underscore the importance of selecting a model that accurately represents the underlying patterns in smoking behavior.

Fig. 3.

Simulation study comparison the original data (a) with simulated estimated parameters using ZIP (b), ZINB (c), and ZIBB (d) models at wave 9

Figure 4 demonstrates our modeling approach to Wave 8 of the study, encompassing a sample size of 1065 participants. Plot (a) represents the raw data collected at Wave 8 which is the actual smoking behavior patterns observed among the study participants. Second figure (b) showcases the simulated data generated using the PH. Parameters were estimated based on the original data at Wave 8, offering insights into how well the Poisson model aligns with the observed smoking behavior. Plot (c) presents simulated data derived from the NBH. Estimates from the Wave 8 data were employed to produce this simulation, allowing us to assess the effectiveness of the negative binomial model in replicating the observed smoking behavior patterns. Similarly, plot (d) exhibits simulated data based on the BBH to generate the simulation. Plot (d) demonstrates that the beta-binomial hurdle model excels in replicating the original smoking behavior data from Wave 8, emphasizing its proficiency in capturing the intricacies of these patterns. In line with our findings from Wave 9, the limitations observed in the fit of ZIP and ZINB underscore the assertion that these models are not optimal for this dataset, reinforcing the need for a more suitable modeling approach to capture the nuances of smoking behavior.

Fig. 4.

Simulation study comparison the original data (a) with simulated estimated parameters using PH (b), NBH (c), and BBH (d) models at wave 8

As observed in both Figs. 3 and 4, ZIBB and BBH models stand out as the most effective in replicating the original data, highlighting the potential of these models as a robust tool for modeling and analyzing healthcare data, particularly in the context of smoking behavior research.

Regression analysis

Our primary focus is on Wave 2 as it contained GPA data. During this year, smoking behavior served as the response variable in our regression analysis. Notably, the response variable demonstrated sparsity, with approximately 0.58 proportion of zero values, indicating a significant presence of non-smokers in the sample.

To conduct our regression analysis, we constructed a design matrix consisting of the student’s GPA and their self-reported sex (with males coded as 0 and female as 1). It’s important to mention that we handled missing values by excluding observations with missing data. This approach was taken to ensure data completeness and to maintain the integrity of our analysis, resulting in a reduced sample size of 1065 individuals. Then we employed a comprehensive set of 14 different regression models, each utilizing one of four distinct link functions mentioned in 2.2. These models were compared based on their AIC and BIC values. According to Burnham and Anderson (2004) model selection based on AIC is advantageous because it has a strong foundation in information theory, a rigorous statistical basis, and can be justified as Bayesian with a suitable prior. This approach transcends the Bayes versus frequentist perspective, emphasizing that the choice between AIC and BIC for model selection should depend on philosophical considerations about reality, model approximations, and the purpose of model-based inference.

After careful evaluation, it became evident from the result of AIC in Table 5 that zero-inflated beta-binomial when n is fixed (ZIBB-n) model with a logit link function emerged as the most suitable choice (AIC = 4307.91). The results based on the BIC criteria are not presented here but are entirely consistent with the findings obtained using the AIC.

Table 5.

Comparison of AIC values with various link functions for ϕ in wave 2 smoking Data analysis

Distribution	logit	probit	log–log	c-log–log
Poisson hurdle (PH)	8007.59	8007.49	8008.02	8007.05
Negative binomial hurdle (NBH)	4496.78	4496.68	4497.21	4496.24
Negative binomial hurdle, r is fixed (NBH-r)	4506.20	4506.10	4506.62	4505.65
Beta-binomial hurdle (BBH)	4420.78	4492.69	4818.14	4501.32
Beta-binomial hurdle, $\alpha ,\beta$ are fixed (BBH-ab)	4445.02	4410.38	4720.99	4401.91
Beta-binomial hurdle, n is fixed (BBH-n)	4316.69	4389.88	4745.31	4737.02
Beta-negative Binomial hurdle (BNBH)	4597.73	4606.26	4579.96	4642.49
Zero-inflated Poisson (ZIP)	8007.59	8007.49	8008.01	8007.05
Zero-inflated negative binomial (ZINB)	4496.34	4496.34	4525.07	4496.32
Zero-inflated negative binomial, r is fixed (ZINB-r)	4545.72	4505.94	4506.39	4505.52
Zero-inflated beta-binomial (ZIBB)	4457.04	4440.94	4418.34	4414.08
Zero-inflated beta-binomial, $\alpha ,\beta$ are fixed (ZIBB-ab)	4437.30	4431.96	4439.69	4417.68
Zero-inflated beta-binomial, n is fixed (ZIBB-n)	4307.91	4349.92	4463.70	4432.88
Zero-inflated beta-negative binomial (ZIBNB)	4379.88	4419.10	4562.35	4413.12

Bold indicates the overall smallest AIC (best fit)

Example 2

Zero-inflated beta-binomial (ZIBB). The zero-inflated beta-binomial model has been applied to analyze microbiome count data and detect associations with phenotypes, showing improved statistical power and control of type I error compared to existing methods, as shown by Hu et al. (2018). In a ZIBB model the baseline distribution is the beta-binomial distribution with the pmf is given by $f_{\mathit{\theta }}(y)=\left(\genfrac{}{}{0pt}{}{n}{y}\right)\frac{\mathit{Beta}(y+\alpha ,n-y+\beta )}{\mathit{Beta}(\alpha ,\beta )},y\in \{\mathrm{0,1},2,\dots ,n\}$, with parameters $\mathit{\theta }=\left(n_i,{\alpha }_i,{\beta }_i\right)$. Applying (1) the following expression is derived.

$$E[f_{ZI}(y\mid {\varphi }_i,n_i,{\alpha }_i,{\beta }_i)]=(1-{\varphi }_i)\cdot \frac{n_i\cdot {\alpha }_i}{{\alpha }_i\cdot {\beta }_i}$$ (10)

Table 6 presents the estimated parameters, confidence intervals based on the Fisher Information matrix, standard errors, and z-scores derived from a ZIBB-n regression model utilizing a logit link function.

Although the ZIBB model includes multiple components and nonlinear link functions, some parameter estimates yield straightforward behavioral interpretations. In the zero-inflated component, the GPA coefficient ((${\widehat{\gamma }}_2=1.331,\mathrm{z}=5.975$) is positive and statistically significant, indicating that students with higher GPA are more likely to be non-smokers. This supports the interpretation that stronger academic performance is associated with a reduced likelihood of smoking. In the count component, the sex variable (${\widehat{\beta }}_{33}=-0.322,\mathrm{z}=-2.095$) is negative and significant, suggesting that among students who do smoke, females tend to report fewer days of smoking than males. Additionally, the significance of sex-related parameters in the shape components ($\alpha$ and $\beta$) indicates that smoking behavior is not only less frequent among females but also distributed differently—highlighting sex-related differences in both the intensity and variability of smoking.

Together, these findings demonstrate the value of the ZIBB-n model in capturing heterogeneity in adolescent smoking. By modeling both the likelihood of abstaining and the distribution of use among smokers, the model provides a more nuanced understanding than standard count models.

Interpreting all the estimated parameters outlined in Table 6 can be complicated in this context. It is essential to comprehend that $\gamma$ is connected to $\varphi$ through the logit link function, while ${\beta }_{i1}$, ${\beta }_{i2}$, and ${\beta }_{i3}$ are associated with n, $\alpha$, and $\beta$ through exponential link functions. Therefore, by employing the formula outlined in (4) we derive the following representation (5), where x1 is the GPA column and x2 corresponds to sex.female column within the design matrix.

$$\begin{gathered} E[f_{zibb}(y\mid {\varphi }_{\mathit{i}},{\mathit{n}}_{\mathit{i}},{\mathit{\alpha }}_{\mathit{i}},{\mathit{\beta }}_{\mathit{i}})] \\ =\frac{\text{exp}({\widehat{\gamma }}_1+{\widehat{\gamma }}_2\cdot \text{x}1+{\widehat{\gamma }}_3\cdot \text{x}2)}{1+\text{exp}({\widehat{\gamma }}_1+{\widehat{\gamma }}_2\cdot x1+{\widehat{\gamma }}_3\cdot x2)} \\ \cdot \frac{\text{exp}\left({\widehat{\beta }}_{11}\right)\cdot \text{exp}\left({\widehat{\beta }}_{12}+{\widehat{\beta }}_{22}\cdot \text{x}1+{\widehat{\beta }}_{32}\cdot \text{x}2\right)}{\text{exp}\left({\widehat{\beta }}_{12}+{\widehat{\beta }}_{22}\cdot \text{x}1+{\widehat{\beta }}_{32}\cdot \text{x}2\right)+\text{exp}\left({\widehat{\beta }}_{13}+{\widehat{\beta }}_{23}\cdot \text{x}1+{\widehat{\beta }}_{33}\cdot \text{x}2\right)} \end{gathered}$$ (11)

By employing formula (4), we assessed the average number of days smoked in the past month by fixing the GPA value at Wave 2, ranging from 2 to 5. This analysis facilitated a comparison between male and female subjects.

Figure 5 illustrates the assessment of parameter estimates in the ZIBB-n model with a logit link function, considering the overall model at Wave 2 across female and male groups. Notably, males exhibit a higher average number of days smoked in the past month compared to females—patterns consistent with prior literature. Additionally, there is an observable decrease in the average number of smoking days as GPA increases, consistent with the findings from the real data presented in Table 11.

Fig. 5.

Average “Days smoked in the past month” at wave 2, given GPA, in a ZIBB regression model with a logit link, comparing female and male subjects

To evaluate the efficacy of the ZIBB-n model in capturing the intricacies of the data, we conducted a comparative analysis against commonly employed models such as ZIP and ZINB. Fixing the parameter n at 30, we proceeded to plot the probability mass function (PMF) using estimated parameters $\alpha$ and $\beta$ for two distinct ranges of GPAs: (2,3.5] and (3.5,5].

Inspired by the findings of Liu et al. (2013) which discussed the application of beta-binomial regression to model bimodal U-shaped distributed utilization in health services research, we utilized PMF plots to compare the performance of ZIP and ZINB models. Notably, our analysis demonstrated that only the ZIBB model closely mirrored the U-shaped distribution observed in the real data (see Fig. 6), as well as the overall pattern of the data for lower and higher intervals of GPA across different smoking responses compared to the common models used in practice to handle zero-inflated data. This highlights the importance of exploring alternative distributions in regression analysis, as they have the potential to better capture the complexities inherent in real-world datasets.

Fig. 6.

Comparing probability mass functions: original data vs. estimated models (beta-binomial, Poisson, negative binomial)

Conclusion and discussion

In this study, we explored the necessity of employing alternative models for the analysis of sparse count data, beyond the conventional zero-inflated and zero-altered Poisson and negative binomial models. Our investigation focused on data from a study on adolescent smoking collected over ten waves among female and male subjects, with available covariates such as GPA. By subjecting each wave to the KS test and comparing the resulting p-values, we identified that the BBH model demonstrated robust performance across most waves in our dataset. Interpreting the estimated parameters within this chosen model revealed a higher prevalence of instances with zero “days smoked in the past month” among females in comparison to males. Additionally, both sex exhibited increasing variability across the ten-wave period which suggests that, while both groups experienced dynamic changes in their smoking behaviors, females, displayed slightly higher variability, implying that the patterns of smoking among females may subject to more fluctuations across the ten waves.

We further examined the estimated parameters, conducted comparisons with real data, and validated our findings through a confirmation study, emphasizing the superiority of ZIBB and BBH models over ZIP, PH, ZINB, and NBH models. Because model fit was evaluated separately by wave and sex, we treat the per-wave KS p-values as descriptive and base interpretation on patterns replicated across waves together with AIC/BIC and diagnostics rather than on individual tests.

We then conducted regression analysis on the second wave of the dataset, incorporating covariates such as GPA and sex, encompassed a total of 14 distinct regression models (including ZIP and ZINB models) each with four unique link functions for the parameter $\varphi$. Through rigorous model selection based on AIC, we determined that the ZIBB-n model with a logit link function emerged as the optimal choice. This result highlights the ZIBB model’s ability to accurately capture the U-shaped distribution patterns observed in the real data. Importantly, our findings underscore the inadequacy of traditional models such as ZIP and ZINB for effectively analyzing this dataset, emphasizing the necessity of incorporating a wider range of models including ZIBB in similar analysis.

In addition to demonstrating the advantages of ZIBB and BBH models within our case study, we emphasize that these models are particularly well-suited for count data that are both bounded and exhibit U-shaped distributions—characteristics that are present in our dataset, as shown in Fig. 1. This shape suggests substantial density at both zero and the upper bound, which traditional models such as ZIP and ZINB struggle to accommodate. ZIBB and BBH models explicitly include a parameter n, the maximum count, that ensures that predictions remain within the realistic bounds of the outcome. Combined with their strong performance across KS tests, AIC/BIC comparisons, and model-based histogram fits (Fig. 6), these features support their practical value in such contexts. While we caution against overgeneralizing to all zero-inflated count settings, our study provides a useful example—and accompanying R code—for researchers encountering similar data patterns.

This study contributes to the growing body of literature in behavioral medicine by providing a refined statistical approach to understanding behavioral patterns, such as smoking frequency, through the lens of advanced count models. The application of these models is particularly relevant for analyzing complex behavioral datasets where traditional approaches may fall short. Our findings pave the way for further exploration of adolescent smoking behaviors and similar health-related behaviors in broader populations.

For future research, we aim to extend our methodology to include longitudinal regression analysis encompassing all waves, as the current regression models primarily focus on cross-sectional data, treating the time points independently. This extension has the potential to provide deeper insights into temporal changes in health-related behaviors, aligning with the goals of behavioral medicine to improve understanding and interventions for such phenomena.

Appendix

Details on the derivations of the MLES in zero-inflated and zero-altered regression models based on (Aldirawi, 2020).

The likelihood function for the zero-altered regression model is given by:

$$\begin{gathered} L(\gamma ,{\beta }_1,\dots ,{\beta }_m)=\prod _{i:y_i=0}{\varphi }_i\cdot \prod _{i:y_i\ne 0}(1-{\varphi }_i) \\ \cdot \prod _{i:y_i\ne 0}\frac{f_{{\mathit{\theta }}_i}(y_i)}{1-p_0({\mathit{\theta }}_i)} \end{gathered}$$

For $j=1,\dots ,m$, the required formulae are

$$\begin{gathered} \frac{\partial l_{tr}}{\partial \beta j}=\sum _{i:y_i\ne 0}\left[\frac{\partial \text{log}{f}_{{\mathit{\theta }}_i}(y_i)}{\partial {\theta }_{ij}}+\frac{p_0({\mathit{\theta }}_i)}{1-p_0({\mathit{\theta }}_i)}\cdot \frac{\partial \text{log}{p}_0({\mathit{\theta }}_i)}{\partial {\theta }_{ij}}\right] \\ {\left(h_j^{-1}\right)}^{\prime }\left(B_{ij}^T{\beta }_j\right)B_{\_}ij \end{gathered}$$

That is, given the baseline distribution $f_{{\mathit{\theta }}_{\mathit{i}}}(y)$, we only need to calculate $\partial \mathit{log}{f}_{{\mathit{\theta }}_{\mathit{i}}}\left(y_i\right)/\partial {\theta }_{ij}$ and $\partial \mathit{log}{p}_0\left({\mathit{\theta }}_i\right)/\partial {\theta }_{ij}$. The likelihood function for the zero-inflated regression model is given by:

$$\begin{gathered} L(\gamma ,{\mathit{\beta }}_1,\dots ,{\mathit{\beta }}_m)=\prod _{i:y_i=0}[{\varphi }_i+(1-{\varphi }_i)p_0({\mathit{\theta }}_i)] \\ \cdot \prod _{i:y_i\ne 0}(1-{\varphi }_i)f_{{\mathit{\theta }}_{\mathit{i}}}(y) \end{gathered}$$

In order to obtain the maximum likelihood estimates that maximizes the likelihood function, we need the following first-order derivatives of the log likelihood function

$$\begin{gathered} \frac{\partial l}{\partial \gamma }=-\sum _{i:y_i\ne 0}\frac{{\left(g^{-1}\right)}^{\prime }\left(G_i^T\gamma \right)}{1-{\varphi }_i}{G}_i \\ +\sum _{i:y_i=0}\frac{[1-p_o({\theta }_i)]{\left(g^{-1}\right)}^{\prime }\left(G_i^T\gamma \right)}{{\varphi }_i+(1-{\varphi }_i)p_o({\theta }_i)} \end{gathered}$$

$$\begin{gathered} \frac{\partial l}{\partial {\beta }_j}=\sum _{i:y_i\ne 0}\frac{\partial \text{log}{f}_{\theta }(y_i)}{\partial {\theta }_{ij}}\cdot {\left(h_j^{-1}\right)}^{\prime }(B_{ij}^T{\beta }_j)B_{ij} \\ +\sum _{i:y_i=0}\frac{(1-{\varphi }_i)p_0({\theta }_i)}{{\varphi }_i+(1-{\varphi }_i)p_0({\theta }_i)}\cdot \frac{\partial \text{log}{p}_0({\theta }_i)}{\partial {\theta }_{ij}} \\ \cdot {\left(h_j^{-1}\right)}^{\prime }\left(B_{ij}^T{\beta }_j\right)B_{ij} \end{gathered}$$

Thus only $\partial \log f_{{\mathit{\theta }}_{\mathit{i}}}\left(y_i\right)/\partial {\theta }_{ij}$ and $\partial \mathrm{l}\mathrm{o}\mathrm{g}{p}_0\left({\mathit{\theta }}_i\right)/\partial {\theta }_{ij}$ are needed for different baseline distributions.

A snippet of R code employed in our analysis

# Load AZIAD package

library(AZIAD)

# Simulate data for 2 waves with different ZI/ZH beta-binomial parameters

set.seed(42)

n <- 500     # number of individuals per wave

phi1 <- 0.4   # structural zero proportion for wave 1

phi2 <- 0.5    # structural zero proportion for wave 2

# Simulate zero-inflated and zero-altered beta-binomial outcomes

wave 1 <- sample.zi1(n, phi1, n = 30, alpha1 = 0.07, alpha2 = 0.14, dist = “bb”)

wave2 <- sample.h1(n, phi2, n = 30, alpha1 = 0.01, alpha2 = 0.2, dist = “bb”)

# Combine into a data frame

sim_data <- data.frame(wavel, wave2)

head(sim_data)

# Perform KS tests across 15 distributions and record p-values

dvec = list(“poisson”, “geometric”, “nb”, “bb”, “bnb”, “zip”, “zigeom”, “zinb”, “zibb”, ”zibnb”, “ph”, “geomh”, “nbh”, “bbh”, “bnbh”)

dmat.f <- matrix(NA, nrow = 2, ncol = 15)

colnames(dmat.f) <- c(“poisson”, “geometric”, “nb”, “bb”, “bnb”, “zip”, “zigeom”,

    “zinb”, “zibb”, “zibnb”, “ph”, “geomh”, “nbh”, “bbh”, “bnbh”)

for(i in 1:2) {

 for(j in 1:15) {

 dmat.f[i, j] <- kstest.A(as.numeric(sim_data[, i]), dist = dvec[j])$pvalue

 }

}

print(dmat.f)

# Estimate BBH parameters for each wave

MLE.F <- matrix(NA, nrow = 2, ncol = 4)

colnames(MLE.F) <- c(“n”, ”alpha”, “beta”, “phi”)

for(i in 1:2) {

 result <- kstest.A(as.numeric(sim_data[, i]), dist = “bbh”)

 MLE.F[i, ] <- result$mle_ori[1:4]

}

print(MLE.F)

Estimated parameters based on MLE

Tables 7 and 8 demonstrate the estimated parameters based on MLE using ZIP model for male and female group, and Tables 9 and 10 demonstrate the estimated parameters based on MLE using ZINB model for male and female group.

See Table 11 and Fig. 7.

Table 7.

Estimated parameters based on MLE using zero-inflated Poisson model for male group

	wave 0	wave 1	wave 2	wave 3	wave 4	wave 5	wave 6	wave 7	wave 8
$\widehat{\varphi }$	0.55	0.57	0.54	0.45	0.42	0.38	0.37	0.40	0.52
$\widehat{\mathrm{\lambda }}$	9.19	11.44	16.53	17.95	19.89	19.86	19.74	20.54	20.51

Table 8.

Estimated parameters based on MLE using zero-inflated Poisson model for female group

	wave 0	wave 1	wave 2	wave 3	wave 4	wave 5	wave 6	wave 7	wave 8
$\widehat{\varphi }$	0.55	0.59	0.60	0.55	0.53	0.48	0.52	0.57	0.65
$\widehat{\mathrm{\lambda }}$	8.51	10.31	14.42	16.16	15.61	17.34	17.68	18.35	18.83

Table 9.

Estimated parameters based on MLE using zero-inflated negative binomial model for male group

	wave 0	wave 1	wave 2	wave 3	wave 4	wave 5	wave 6	wave 7	wave 8
$\widehat{\varphi }$	0.50	0.53	0.51	0.42	0.42	0.37	0.37	0.39	0.52
$\widehat{r}$	1	1	1	1	2	2	2	2	2
$\widehat{p}$	0.11	0.09	0.06	0.06	0.09	0.09	0.09	0.09	0.09

Table 10.

Estimated parameters based on MLE using zero-inflated negative binomial model for female group

	wave 0	wave 1	wave 2	wave 3	wave 4	wave 5	wave 6	wave 7	wave 8
$\widehat{\varphi }$	0.49	0.55	0.57	0.52	0.50	0.45	0.49	0.54	0.63
$\widehat{r}$	1	1	1	1	1	1	1	1	1
$\widehat{p}$	0.12	0.10	0.07	0.06	0.06	0.06	0.06	0.05	0.05

Table 11.

Average days smoke in the past month across GPA intervals from real data

GPA intervals	Average smoke (female)	Average smoke (male)	Sample size (female)	Sample size (male)
[2,2.5]	8.98	8.84	25	19
(2.5,3]	7.50	8.67	75	86
(3,3.5]	6.31	7.36	71	73
(3.5,3.75]	6.33	7.14	88	57
(3.75,4]	4.60	6.58	119	85
(4,4.25]	3.66	3.90	114	48
(4.25,4.75]	3.04	2.01	74	49
(4.75,5]	1.90	2.55	55	27

Fig. 7.

Simulation study comparison the original data (a) with simulated estimated parameters using ZIP (b), ZINB (c), and ZIBB (d) models at wave 2