A Note on the Poisson’s Binomial Distribution in Item Response Theory

Abstract

The Poisson’s binomial (PB) is the probability distribution of the number of successes in independent but not necessarily identically distributed binary trials. The independent non-identically distributed case emerges naturally in the field of item response theory, where answers to a set of binary items are conditionally independent given the level of ability, but with different probabilities of success. In many applications, the number of successes represents the score obtained by individuals, and the compound binomial (CB) distribution has been used to obtain score probabilities. It is shown here that the PB and the CB distributions lead to equivalent probabilities. Furthermore, one of the proposed algorithms to calculate the PB probabilities coincides exactly with the well-known Lord and Wingersky (LW) algorithm for CBs. Surprisingly, we could not find any reference in the psychometric literature pointing to this equivalence. In a simulation study, different methods to calculate the PB distribution are compared with the LW algorithm. Providing an exact alternative to the traditional LW approximation for obtaining score distributions is a contribution to the field.

Introduction

Lord (1980) pointed out that using item response theory (IRT), the frequency distribution of test scores, $X$, conditional on a given ability, $\theta$, could be obtained. For the case when $X$ is the summation of the correct responses, Lord recognized that an explicit formula of the conditional distribution was difficult to obtain in a simple form, except for the unrealistic scenario when all items have the same probability of being answered correctly, in which case the binomial distribution appears naturally (see Lord, 1980, Section 4.1). For the non-identically distributed case, that is, when the probabilities of a correct answer differ across items, the use of probability generating functions leads to what is called the compound binomial (CB) distribution, also known as the generalized binomial (Lord, 1980, Section 4.1; Kendall & Stuart, 1958; Lord & Novick, 1968, Section 16.12). The CB distribution can be directly used to obtain the score probabilities, but this practice is computationally demanding for some cases (as it will be demonstrated later).

Wang (1993) presented an explicit form for the distribution of the number of successes in independent but not necessarily identically distributed binary trials, the Poisson’s binomial (PB) distribution, and studied many of its properties. Wang pointed out that this distribution has played an important role in probability theory, and it dates back at least to Poisson (1837). Direct calculation of probabilities using the PB distribution suffers from similar computational problems as the CB. Nevertheless, efficient algorithms are available for the estimation of the PB distribution. To our knowledge, neither the PB distribution nor the algorithms to obtain the score probabilities seemed to have been referenced in the psychometric literature.

First, we show that the CB and PB distributions lead to the same probabilities of the number of successes in independent binary trials. Second, we conduct a simulation study to evaluate the performance of these algorithms. Third, the well-known Lord and Wingersky (1984) algorithm is shown to be equivalent to one of the methods used for the estimation of the PB distribution.

Theoretical Models

PB

Let $X$ denote the total number of successes in $n$ independent Bernoulli trials where $p_i$ is the probability of success at trial $i$. If $p_i=p$$\forall i$, the distribution of $X$ is the $\mathrm{Binomial}(n,p)$. For the case of independent but non-identically distributed Bernoulli variables, $X$ follows the PB distribution (Wang, 1993). The PB distribution has been used in many areas such as, for example, pool screening (Gao, Aban, & Katholi, 2014), survey sampling (Chen & Liu, 1997), and bioinformatics (Niida, Imoto, Shimamura, & Miyano, 2012).

If $p_i$ is the probability of success at the $i$th trial, the following defines the probability mass function (PMF) of $X$ (Wang, 1993).

Definition: Let $F_x=\{A:A\subseteq \{1,\dots ,n\},\mid A\mid =x\}$, where $\mid A\mid$ denotes the number of elements of $A$. The PMF of $X$ is

$$f(x;\mathbf{p})=\mathrm{Pr}(X=x)=\sum _{A\in F_x}\left(\underset{i\in A}{\Pi }{p}_i\right)(\underset{j\in A^c}{\Pi }(1-p_j)).$$

and the corresponding cumulative distribution function (CDF)

$$F_X(x;\mathbf{p})=\mathrm{Pr}(X\le x)=\sum _{m=0}^x\sum _{A\in F_x}\left(\underset{i\in A}{\Pi }{p}_i\right)(\underset{j\in A^c}{\Pi }(1-p_j)).$$

To use this distribution in practice, consider $n=3$, in which case the $F_x$ sets are

$$F_0=\left\{\varnothing \right\},F_1=\{\left\{1\right\},\left\{2\right\},\left\{3\right\}\},F_2=\{\{1,2\},\{1,3\},\{2,3\}\},F_3=\left\{\{1,2,3\}\right\}.$$

To calculate the probability that two successes are obtained, we use Equation 1 to obtain

$$\begin{array}{c}\multicolumn{1}{c}{f(2,\mathbf{p})=\left(\underset{i\in \{1,2\}}{\Pi }{p}_i\right)\left(\underset{j\in \left\{3\right\}}{\Pi }(1-p_j)\right)+\left(\underset{i\in \{1,3\}}{\Pi }{p}_i\right)\left(\underset{j\in \left\{2\right\}}{\Pi }(1-p_j)\right)}\\ \multicolumn{1}{c}{+\left(\underset{i\in \{2,3\}}{\Pi }{p}_i\right)\left(\underset{j\in \left\{1\right\}}{\Pi }(1-p_j)\right)}\\ \multicolumn{1}{c}{=p_1{p}_2(1-p_3)+p_1{p}_3(1-p_2)+p_2{p}_3(1-p_1)}\\ \multicolumn{1}{c}{=p_1{p}_2{q}_3+p_1{p}_3{q}_2+p_2{p}_3{q}_1,}\end{array}$$

where q_i = 1-p_i.. Using similar calculations, the probabilities of obtaining 0, 1, and 3 successes are $f(0,\mathbf{p})=q_1{q}_2{q}_3$; $f(1,\mathbf{p})=p_1{q}_2{q}_3+p_2{q}_1{q}_3+p_3{q}_1{q}_2$; $f(3,\mathbf{p})=p_1{p}_2{p}_3$, respectively. Note, to calculate the probabilities, all the sets satisfying $\mid A\mid =x$ in $F_x$ must be listed. For instance, with $n=40$, $F_{10}$ would contain $(\begin{array}{c}\hfill 40\hfill \\ \multicolumn{1}{c}{10}\end{array})=847,660,528$ elements, so direct calculation of probabilities is not practical for large values of $n$.

Algorithms for the Calculation of Probabilities

Both approximate and exact alternatives to the direct calculation of probabilities to efficiently obtain the distribution function of the PB model have been proposed and are reviewed here. A well-known approximation used in general statistics is the normal approximation (NA), which is based on the central limit theorem and approximates the CDF of the PB distribution by

$$F_X(x)\approx \Phi \left(\frac{x+0.5-\mu }{\sigma }\right),$$

where $\Phi$ is the CDF of the standard normal distribution,

$$\mu =E(X)=\sum _{i=1}^n{p}_i\mathrm{and}\sigma ={[\mathrm{Var}(X)]}^{1/2}={\left[\sum _{i=1}^n{p}_i(1-p_i)\right]}^{1/2}.$$

Not surprisingly, this approximation has also been used in psychometrics for the CB (e.g., Lord & Novick, 1968), which will later be shown to be equivalent to the PB.

An improved version of the NA was described by Volkova (1996; see also Neammanee, 2005) and is known as the refined normal approximation (RNA). Compared with NA, it adds a correction to the skewness of the distribution of X. Under this method, the CDF of the PB is approximated by

$$F_X(x)\approx G\left[\frac{x+0.5-\mu }{\sigma }\right],x=0,1,...,n,$$

where $\mu$ and $\sigma$ are defined as in Equation 3, $G(x)=\varphi (x)+\gamma (1-x^2)\varphi (x)/6$, $\varphi (x)$ is the probability distribution of the standard normal distribution and

$$\gamma ={\sigma }^{-3}\sum _{i=1}^n{p}_i(1-p_i)(1-2p_i).$$

Another approximate method referred here to as the Poisson approximation (PA) is based on a famous inequality established by Le Cam (Le Cam, 1960; Steele, 1994) and uses the Poisson distribution, with $\mu$ as defined in Equation 3, to approximate the PMF of the PB by

$$f(x)\approx \frac{{\mu }^x\exp (-\mu )}{x!}.$$

Among the exact methods, one algorithm for computing the distribution function of the PB was proposed in Fernandez and Williams (2010) where polynomial interpolation and the Discrete Fourier Transform (DFT) are used to derive closed-form formulas for the PB’s probability distribution function and CDF. Later, Hong (2013) derived the same closed-form expressions in a simpler way. The method is based on the application of the DFT to the characteristic function of the PB distribution, and it is accordingly called the DFT-CF method. Using this method, the CDF of the PB distribution can be obtained through

$$F_X(x)=\frac{1}{n+1}\sum _{l=0}^n\sum _{m=0}^x\exp (-\mathbf{i}\omega \mathrm{lm})z_l,x=0,\dots ,n,$$

where i is the imaginary unit, $\omega =2\pi /(n+1)$, and $z_l$ is the characteristic function of the PB random variable evaluated at $\omega l$, that is, $z_l=\underset{i=1}{\overset{n}{\Pi }}[1-p_i+p_i\exp (\mathbf{i}\omega l)]$ (Hong, 2013).

Other exact methods are recursive and initiated as approximate expressions for the calculation of the PB’s CDF. In an early work, Walsh (1955) proposed a method based on power expansions of $(p_i-\overline{p})$. Interestingly, Walsh’s method was also used in Lord and Novick (1968, Section 23.10) and later in Yen (1984). Successive work regarding the distribution of the number of successes in independent binary trials includes Hoeffding (1956), Darroch (1964), Samuels (1965), and Nedelman and Wallenius (1986). Thomas and Taub (1982) developed a recursive algorithm based on the probability generating function of the random variable $X$, which is capable to generate exact values of the distribution without explicitly enumerating all the sets satisfying $\mid A\mid =x$ in $F_x$.

The probability generating function of a random variable $X$ taking values $0,1,2,\dots$ is defined as $P(s)=\sum _x{s}^x\mathrm{Pr}(X=x)$ (Grimmett & Welsh, 2014), and its explicit form for the case when $X$ is the number of successes in independent binary trials is known to be

$$P(s)=\underset{i=1}{\overset{n}{\Pi }}(q_i+p_{i}s).$$

It follows that one can extract the values $\mathrm{Pr}(X=x)$ grouping the coefficients of $s^x$ in the expansion of Equation 7. Thomas and Taub (1982) used this result and the fact that Equation 7 can be written as an $n$th degree polynomial in $s$ in the form

$$P_n(s)={\alpha }_{n1}{s}^0+{\alpha }_{n2}{s}^1+{\alpha }_{n3}{s}^2+\cdots +{\alpha }_{n,n+1}{s}^n$$

so that ${\mathrm{Pr}}_n(X=x)={\alpha }_{n.x+1}$ can be calculated using the recursive formula (RF)

$${\alpha }_{\mathrm{ij}}={\alpha }_{i-1,j-1}{p}_i+{\alpha }_{i-1,j}{q}_i,$$

with the conditions ${\alpha }_{11}=q_1$, ${\alpha }_{12}=p_1$, ${\alpha }_{i1}={\alpha }_{i-1,1}{q}_i$ if $i\ge 2$, and ${\alpha }_{i,n+1}={\alpha }_{i-1,n}{p}_i$ if $i\ge 2$. A program written in BASIC implementing this RF is presented in Barlow and Heidtmann (1984). Note that the use of the probability generating function leads directly to the calculation of probabilities. For instance for $n=3$, one has

$$\begin{array}{c}\multicolumn{1}{c}{\sum _{x=0}^3\mathrm{Pr}(X=x)s^x=\underset{i=1}{\overset{3}{\Pi }}(q_i+p_{i}s)}\\ \multicolumn{1}{c}{=(q_1+p_{1}s)(q_2+p_{2}s)(q_3+p_{3}s)}\\ \multicolumn{1}{c}{=(q_1{q}_2{q}_3)+(q_1{q}_2{p}_3+q_1{p}_2{q}_3+p_1{q}_2{q}_3)s}\\ \multicolumn{1}{c}{=(p_1{p}_2{q}_3+q_1{p}_2{p}_3+p_1{q}_2{p}_3)s^2+(p_1{p}_2{p}_3)s^3}\end{array}$$

where $s$ is used to group terms so that

$$\begin{array}{c}\multicolumn{1}{c}{P(X=0)=q_1{q}_2{q}_3,P(X=1)=q_1{q}_2{p}_3+q_1{p}_2{q}_3+p_1{q}_2{q}_3,}\\ \multicolumn{1}{c}{P(X=2)=p_1{p}_2{q}_3+q_1{p}_2{p}_3+p_1{q}_2{p}_3,P(X=3)=p_1{p}_2{p}_3}\end{array}$$

which coincide with the probabilities calculated using Equation 1. This strategy is also used in Lord (1980, Section 4.1). Next, we will show that the RF actually corresponds to the Lord and Wingersky (1984) recursive algorithm.

IRT Models

Let $\mathbf{X}=(X_1,\dots ,X_n)$ be an individual’s response pattern to $n$ binary items where $X_i\in \{0,1\}$. Let $\theta$ denote the ability of the individual, ${\omega }_i$ an item parameter vector, and $p_i$ the probability that the individual answers item $i$ correctly. The underlying probability model is then $X_i\mid \theta ,{\omega }_i~\mathrm{Bernoulli}(p_i(\theta ,{\omega }_i))$, where the probability of success can be defined with different IRT models. Local independence is assumed, which means that, conditionally on $\theta$, the answers of the individual to the $n$ items in the test are independent. Let $X=\sum _{i=1}^n{X}_i$ be the sum score of an individual with ability $\theta$. Assuming that individuals with a given ability $\theta$ will correctly answer to each of $n$ items of a test with probability $p_i$ ($i=1,\dots ,n$), the conditional distribution of $X$ for a given $\theta$ is the CB (Lord & Novick, 1968) and is defined as

$$f(x\mid \theta )=\sum _{T_x}\left(\underset{i=1}{\overset{n}{\Pi }}{p}_i^{x_i}{q}_i^{1-x_i}\right),$$

where $T_x$ is the set containing all the response patterns with exactly $x$ correct answers, and $x=0,\dots ,n$. As an example of its use to calculate the probability of a certain score, let us consider again $n=3$ items. Then, the set $T_x$$(x=0,1,2,3)$ contains the following elements:

$$\begin{array}{c}\multicolumn{1}{c}{T_0=\left\{(0,0,0)\right\},T_1=\{(1,0,0),(0,1,0),(0,0,1)\}}\\ \multicolumn{1}{c}{T_2=\{(1,1,0),(0,1,1),(1,0,1)\},T_3=\left\{(1,1,1)\right\}}\end{array}$$

Using Equation 9, for a given ability, the probabilities of earning each of the $x$ values are

$$\begin{array}{c}\multicolumn{1}{c}{f(0\mid \theta )=q_1{q}_2{q}_3,f(1\mid \theta )=p_1{q}_2{q}_3+q_1{p}_2{q}_3+q_1{q}_2{p}_3,}\\ \multicolumn{1}{c}{f(2\mid \theta )=p_1{p}_2{q}_3+q_1{p}_2{p}_3+p_1{q}_2{p}_3,f(3\mid \theta )=p_1{p}_2{p}_3,}\end{array}$$

respectively. As it was the case for $F_x$, the number of elements in $T_x$ grows rapidly with the number of items, so direct use of the CB is computationally demanding. For instance, with 40 items, to calculate $f(x=10\mid \theta )$, the set $T_{10}$ would contain $(\begin{array}{c}\hfill 40\hfill \\ \multicolumn{1}{c}{10}\end{array})=847,660,528$ elements, which complicates the numerical calculations. In the psychometric literature, these difficulties were noted by Yen (1984; see also, Thissen, Pommerich, Billeaud, & Williams, 1995) who stated that calculating this distribution is laborious because all $2^n$ response patterns must be evaluated.

As an alternative to the direct calculation of score probabilities, the CB distribution has traditionally been obtained using a recursion formula given by Lord and Wingersky (1984). The formula reads as follows

$$\begin{array}{c}\multicolumn{1}{c}{f_r(x\mid \theta )=f_{r-1}(x\mid \theta )q_r,x=0}\\ \multicolumn{1}{c}{=q_r{f}_{r-1}(x\mid \theta )+p_r{f}_{r-1}(x-1\mid \theta ),0<x<r}\\ \multicolumn{1}{c}{=f_{r-1}(x-1\mid \theta )p_{r}x=r}\end{array}$$

where $f_r(x\mid \theta )$ is the distribution of sum scores over the first r items for examinees of ability θ and $p_r=p_i$ is the probability defined from the chosen IRT model.

Note, (a) in both the PB and CB distributions, the number of elements in the sets $F_x$ and $T_x$, respectively, grows rapidly with the number of trials, making the computation of probabilities computationally inefficient. Despite this, both PB and CB can be directly used to obtain the score probabilities, and we have shown that they generate identical probabilities so that these distributions can be considered equivalent; (b) the LW algorithm in Equation 10 is actually the RF shown in Equation 8 (if ${\alpha }_{n,x+1}$ is replaced by $f_r(x\mid \theta )$). It is, however, surprising that none of our cited references on the RF for the PB actually appear in the psychometric literature. As PB and CB are equivalent, we will evaluate the described alternative methods to calculate score probabilities next.

Simulation Example

In this empirical example, the R (R Core Development Team, 2014) package poibin (Hong, 2013) was used to obtain the CDFs for the PB distribution using all the described methods (i.e., NA, RNA, PA, DFT-CF) in the previous section in addition to the LW recursive algorithm. The test scores were simulated under the two-parameter logistic IRT model. Individuals were sampled from a $N(0,1)$ distribution and item parameters from $a~U(0.1,1.5)$ and $b~N(0,1)$. Varying factors were the number of test items $n=10,30,50,80,100$, and the sample size $N=1000,5000,10000$. The mean absolute error was calculated over 100 replications for each combination of factors and was defined as

$$\mathrm{MAE}=\frac{1}{100}\sum _{l=1}^{100}{(\sum _{x=0}^n\mid F(x)-F^{*}(x)\mid )}_l,$$

where $F(x)$ is calculated using either the LW, NA, RNA or the PA algorithm, and $F^{*}(x)$ is calculated using the DFT-CF method. This method was chosen as criterion as it produces exact values of probabilities and, in comparison with the LW, is not recursive. To calculate both $F(x)$ and $F^{*}(x)$, item parameter estimates are obtained for each replicated data set in the simulation.

In many practical applications, the marginal score distribution $f(x)$ is of interest (for instance, in IRT observed-score equating); we integrated out the value of $\theta$ by averaging over the range $\theta =-2.0,-1.9,\dots ,1.9,2.0$, although other methods such as quadrature points or estimates of $\theta$ can also be used (Kolen & Brennan, 2014, p. 199). A comparison among the actual conditional distributions for three values of $\theta$, –2.0, 0.0, and 2.0, is given in the supplementary material. The conditional distribution of scores is of interest, for instance, in local equating methods (van der Linden, 2011).

Results

Table 1 shows the mean absolute error for each combination of the factors in the simulation for the marginal test score distribution $F(x)$. Because increasing the sample size from N = 1000 to N = 10000 produced no significant differences or pattern, only results for $N=1000$ are shown. For all cases, the commonly used recursive LW algorithm performs equally well as the exact DFT-CF method, with results agreeing to the ninth decimal place (although only four decimals are shown here). LW can thus be considered a method that accurately estimates the score distributions.

Table 1.

Mean Absolute Error for the Estimation of Marginal Distributions for Different Test Lengths.

N	LW	NA	RNA	PA
10	0.000	0.029	0.011	0.462
30	0.000	0.020	0.005	0.644
50	0.000	0.015	0.003	0.759
80	0.000	0.012	0.002	0.832
100	0.000	0.011	0.002	0.869

Note. LW = Lord and Wingersky; NA = normal approximation; RNA = refined normal approximation; PA = Poisson approximation.

Studying the approximation methods, PA does not perform well for most cases. The RNA performs better than the NA in all cases and both methods improve with the number of items. Interestingly, when the test length increase ($n\ge 30$), the mean absolute error of the RNA method is less than 0.005. This last finding could suggest that the RNA is a competitive method for the estimation of the score distributions as an alternative to the LW recursive algorithm. The results for conditional distributions of test scores are in line with these and are given as supplemental material.

Discussion

We have shown that in the distribution of the number of successes in independent binary trials, the PB distribution is equivalent to the CB distribution seen in psychometrics. The LW recursive algorithm for CBs was shown to be equivalent to an RF derived for the PB distribution. Alternative methods, both approximate and exact, were introduced for the calculation of PB probabilities and evaluated in the IRT framework.

We used four different approaches for the estimation of the test score distributions. Some approximation methods were competitive in the cases where the score distributions were marginalized over $\theta$. Because the marginal distributions of scores $F(x)$ and $F(y)$ of two test forms to be equated, X and Y, respectively, are used in observed-score equating, the approximation methods can be a convenient alternative to the recursive LW algorithm when performing IRT observed-score equating.

For the case of conditional distributions, approximate methods perform worse when extreme values of abilities are considered in comparison with average values of ability. This could have consequences in equating methods that are based on conditional score distribution functions as, for example, local equating. Because a family of equating functions is defined by different values of $\theta$, the equating for individuals with extreme abilities could be distorted. It is, however, necessary to know to what extent the differences seen for the conditional distributions could have an effect on the actual equating function. This is a topic of future research.

There is an advantage of having a compact and exact mathematical definition for the PB distribution as random variables can directly be generated from this model. This could help in making fairer simulation studies, for instance, when comparing equating methods, as the advantages and disadvantages of using one or another method to simulate score data are not always clear (Sinharay, Holland, & von Davier, 2011). This topic is currently being investigated by the authors. Other potential benefit of knowing the exact mathematical definition of the PB distribution could arise in multistage testing if the conditional test scores of the individuals are used at each stage of testing (Haberman & von Davier, 2014). In this article, local independence is assumed in the specification of IRT models. The psychometric literature, however, has questioned the independence of items in IRT models (Gibbons & Hedeker, 1992; Wainer & Kiely, 1987). Exploring the score distribution in the non-independent, non-identically distributed case is an interesting topic for future research.

Recent work improving the described algorithms for the PB and showing new applications can be found in Barrett and Gray (2014), and in the psychometric field in Cai (2014). How the presented algorithms would extend to the case of polytomous items data (e.g., Thissen et al., 1995) is also a topic of future research.