A semi-analytical solution to the maximum-likelihood fit of Poisson data to a linear model using the Cash statistic

Abstract

The Cash statistic, also known as the $C$ statistic, is commonly used for the analysis of low-count Poisson data, including data with null counts for certain values of the independent variable. The use of this statistic is especially attractive for low-count data that cannot be combined, or re-binned, without loss of resolution. This paper presents a new maximum-likelihood solution for the best-fit parameters of a linear model using the Poisson-based Cash statistic. The solution presented in this paper provides a new and simple method to measure the best-fit parameters of a linear model for any Poisson-based data, including data with null counts. In particular, the method enforces the requirement that the best-fit linear model be non-negative throughout the support of the independent variable. The method is summarized in a simple algorithm to fit Poisson counting data of any size and counting rate with a linear model, by-passing entirely the use of the traditional ${\chi }^2$ statistic.

1. Introduction

The maximum-likelihood modeling of integer-valued Poisson data can be accomplished with the use of the Cash, or $C$ statistic, first proposed by [8]. The Cash statistic applies to a variety of counting data in use across the sciences. One example is the counting of photons as a function of energy or wavelength, as commonly done by photon-counting detectors used in astronomy (e.g. [5]). Another example is the number or percentage of votes for a candidate in different precincts or polling stations. In counting experiments such as these, the collected data are in the form of independent integer-valued variables. The behavior of these variables as a function of an independent variable (such as photon energy or number of voters in a precinct) can be modeled with the aid of the Cash statistic, which is obtained from the logarithm of the likelihood of the data with a model for the distribution of the counts.

It is well established that the asymptotic distribution of the $C$ statistic, in the large-count limit, is a ${\chi }^2$ distribution (e.g. [8,13]). This limit is a result of the asymptotic convergence of a Poisson distribution with a Gaussian distribution of same mean and variance, which occurs for large values of the Poisson mean. It is straightforward to study the distribution of the $C$ statistic as a function of the parent mean μ, when the parent mean is specified a priori, e.g. when the fitting model has no free parameters. Such calculations are reported in [4,7,13], showing, among other results, that the expectation of C is significantly lower than the expectation of a ${\chi }^2$ distribution with the same number of degrees of freedom.

The use of the $C$ statistic for integer-valued, counting Poisson data is to be preferred to the use of the more common ${\chi }^2$ distribution. First, even in the large-count limit, use of the ${\chi }^2$ fit statistic leads to a bias in the best-fit parameters, due to the approximation of the Gaussian variance with the measured Poisson counts (e.g. [12]). Second, use of the ${\chi }^2$ statistic often requires the combination of datapoints, often referred to as binning of the independent variable, to reach a sufficient number of counts in each independent data point. Such binning may result in an undesirable reduction in the resolution of the data, especially in the presence of sharp features in narrow intervals of the independent variable, such as emission or absorption lines. The $C$ statistic, on the other hand, can be used on unbinned data that make use of the full resolution of the data.

Use of the $C$ statistic also comes with a number of challenges. First, it is not known exactly how free model parameters affect the distribution of $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ – i.e. the $C$ statistic obtained when optimizing variable model parameters – especially in the low-count regime. A study of the distribution of $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ for a simple one-parameter constant model was reported by Bonamente [4], although those results do not directly apply to more complex models such as the linear model.

Another challenge is the numerical complexity of the Poisson distribution and the associated $C$ statistic, which limits the ability to obtain analytical solutions for the best-fit parameters via the maximum-likelihood criterion. A key illustration of this challenge is that the simple linear model, which has an analytical solution for its best-fit model parameters and their covariance matrix when using the ${\chi }^2$ statistic (e.g. [2,3]), does not have an equally simple solution when using the $C$ statistic.

This paper addresses the latter problem by presenting a new semi-analytical method to identify the maximum-likelihood solution of the parameters of a linear model using the $C$ statistic. The method consists of the numerical solution of a simple analytical equation that determines the best-fit value of one of the two parameters, and the use of an analytical function to calculate the other parameter. It is also shown that not all Poisson data sets can be fit to an unconstrained linear model, since the resulting best-fit model may become negative, and therefore not usable for calculation of the Poisson-based $C$ statistic. In those cases, a simple generalization of the linear model is proposed that enforces the non-negative requirement for the best-fit model. Such a generalization ensures that data of all sizes and counting rates can be fit with a linear model, and that such model is unique. The results presented in this paper therefore ease the challenges presented by the numerical complexity of the Poisson distribution, by providing a simple semi-analytical method to find the best-fit parameters of the linear model, and making it possible to study the distribution of $C_{min}$ in the low-count regime.

This paper is structured as follows. Section 2 introduces the maximum-likelihood method and the Poisson-based $C$ statistic, Section 3 presents the equations for the model parameters and Section 4 the solution of those equations. Section 5 discusses conditions for the non-negativity of the best-fit linear model and Section 6 presents the extended linear model that ensures an acceptable model for all datasets. Finally, Section 7 contains a discussion and conclusions.

2. Methods for the maximum-likelihood analysis of Poisson data

2.1. Data model and the $C$ statistic

The data model considered in this paper is N independent integer-valued measurements $y_i$, each Poisson-distributed and measured at a fixed value $x_i$ of the independent variable x. The data can also be viewed as originating from $M=\sum y_i$ independent events that are sorted into N independent ‘bins’, each of size $\mathrm{\Delta }{x}_i$ and centered at $x_i$ with $y_i$ counts. This is a common type of data for the physical sciences; for example, the independent variable x may be the wavelength of collected photons, and y the number of photons collected in a given wavelength range, binned according to the resolution of the instrument. The data can therefore be summarized as a collection of N independent two-dimensional variables

$$\{x_i,y_i\},\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}{y}_i\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\left({\mu }_i\right),i=1,\dots ,N$$

where ${\mu }_i$ is the unknown parent mean of the Poisson distribution of the counts $y_i$, collected in a fixed range $x_i\pm \mathrm{\Delta }{x}_i/2$ of the independent variable.

In general, the relationship between the dependent and independent variables is of the form $y=f\left(x\right)$, where $f\left(x\right)$ is an analytical function with m adjustable parameters $a_j$, $j=1,\dots ,m$. The likelihood of the data with the model $f\left(x\right)$ is given by

$$\mathcal{L}=\prod _{i=1}^N\frac{e^{-{\mu }_i}{\mu }_i^{y_i}}{y_i!}$$

where the adjustable parameters in $f\left(x\right)$ are optimized so that the likelihood $\mathcal{L}$ is maximized, for the given dataset. Instead of maximizing $\mathcal{L}$ directly, it is convenient to minimize the function

$$C\equiv -2\ln \mathcal{L}-B=2\sum _{i=1}^N\left({\mu }_i-y_i+y_i\ln \left(y_i/{\mu }_i\right)\right)=\sum _{i=1}^N{C}_i$$ (1)

where

$$B=2\sum _{i=1}^N\left(y_i-y_i\ln y_i+\ln y_i!\right)$$

is a quantity that is independent of the model, and

$$C_i=2\left({\mu }_i-y_i+y_i\ln \left(y_i/{\mu }_i\right)\right).$$

The statistic C is known as the Cash or $C$ statistic, originally proposed by Cash [8] and Baker and Cousins [1] to model and analyze X-ray observations of astronomical sources. For a model with m free parameters, the minimization of the $C$ statistic yields m equations, in general non-linear, that need to be solved to obtain the maximum-likelihood best-fit estimates of the m model parameters.

2.2. Linear models

The linear model is a simple and commonly used relationship between two variables. The customary parameterization of a linear relationship between two variables x and y is $f\left(x\right)=a+bx$, with a and b as the two adjustable parameters. Another convenient and equivalent parameterization, suggested by Scargle et al. [17], is of the form

$$f\left(x\right)=\lambda (1+a(x-x_A\left)\right)$$ (2)

with λ and a as the two adjustable parameters, and $x_A$ a fixed fiducial value of the x variable. As will be shown, this parameterization is convenient when taking the derivative of the terms $\ln f\left(x_i\right)$ in Equation (1), since it leads to a separation between the two parameters λ and a. This is the parameterization used in this paper .¹

The continuous function $f\left(x\right)$ is related to the parent mean ${\mu }_i$ of each Poisson variable $y_i$ via an integral over the length of the i-th bin,

$$y\left(x_i\right)={\mu }_i={\int }_{x_i-\mathrm{\Delta }{x}_i/2}^{x_i+\mathrm{\Delta }{x}_i/2}f\left(x\right)\mathrm{d}x=f\left(x_i\right)\mathrm{\Delta }{x}_i,$$ (3)

where $y\left(x_i\right)$ is a step-wise function describing the Poisson mean for each bin, and the last equality applies because of the linearity of $f\left(x\right)$. It is therefore recognized that $f\left(x\right)$ is a non-negative density function in units of counts per unit x (i.e. not just counts), while its integral over a range $\mathrm{\Delta }{x}_i$ is the predicted Poisson mean $y\left(x_i\right)={\mu }_i$ in that bin, in units of counts. Therefore, the two functions $f\left(x\right)$ and $y\left(x_i\right)$ will vary from each other according to the size of the bins, which is allowed to be non-uniform. It is important to stress that the function $f\left(x\right)$ must be non-negative, since it would not be meaningful to have a Poisson variable with a negative parent mean. The requirement $f\left(x\right)\ge 0$ places a number of constraints on the solutions of the maximum-likelihood equations that are discussed in Section 5.

2.3. Generalized linear models and other considerations for count data

Integer-valued count data of the type considered in this paper can be modeled with alternative statistics that afford more flexibility than the single-parameter Poisson distribution, in particular with regards to over- or under-dispersion of the data (e.g. [6,11,18,19]). Moreover, the regression with one or several independent variables may often require more complex models than a simple linear model. Generalized linear models and vector-generalized linear models provide a comprehensive and flexible framework for the regression of data, including a natural way to account for non-negative Poisson means via a suitable link function that relates the Poisson mean to the data and model parameters (e.g. [9,14,16,21,22]). Within the context of generalized linear models, a convenient link function would be in the form of $\log \mu =a+bx$, where the logarithm of the Poisson mean μ, instead of the mean itself, is modeled with a linear function (as in, e.g. [10]). Such log-linear models would ensure that the Poisson mean is always positive (e.g. see chapter 6 of [14]).

There are two main reasons for the present investigation of a simple linear regression using the standard Poisson distribution, instead of more versatile models or distributions. First is the goal to obtain an analytical, and therefore computationally efficient, solution for the best-fit parameters of the linear model for count data. Currently, the only available analytical solution for the maximum-likelihood fit of the linear model is for Gaussian-distributed variables, and this method is not accurate for low-count integer-valued data (e.g. [4]). Second, there may be scientific reasons to prefer a simple linear model versus, e.g. a log-linear model or other more complex models, particularly when the data or an underlying parent model suggest a direct linear relation between the dependent and independent variables (e.g. [15,20]). The combination of these practical and scientific reasons make it interesting to seek an analytical solution of the basic linear regression with Poisson data.

3. Maximum likelihood solutions for the parameters of the linear model

3.1. The $C$ statistic for the linear model

The linear model of Equation (2) is illustrated in Figure 1. To evaluate the $C$ statistic of Equation (1) with the linear model of Equation (2), start with

$$\sum _{i=1}^N{\mu }_i={\int }_{x_A}^{x_B}f\left(x\right)\mathrm{d}x=\lambda R\left(1+a\frac{R}{2}\right)$$ (4)

where $R=x_B-x_A$ is the range of the x variable, and it is assumed that the data covers the entire range R. There are situations where measurements of the dependent variable y are missing for certain intervals of the independent variable x, for example because measurements are not possible or because they are ignored in the analysis. An interval of the independent variable where data are missing, or are otherwise not used in the analysis, will be referred to as a gap. If the data contain gaps in the x variable, the limits of integration of Equation (4) will change. Section 6.3 describes the simple modification required to analyze data that contain such gaps.

Figure 1.

Linear model according to Equation (2). In this illustration, the functions $f\left(x\right)$, in units of counts per unit x, and $y\left(x_i\right)$, in units of total counts in the bin, follow one another closely because a bin size value of $\mathrm{\Delta }x=1$ was used.

The second term of the $C$ statistic is simply

$$\sum _{i=1}^N{y}_i=M,$$

where M is the total number of counts, and the final term is

$$\sum _{i=1}^N(y_i\ln (y_i)-y_i\ln {\mu }_i)=\sum _{i=1}^N{y}_i\ln \left(y_i\right)-\sum _{i=1}^N{y}_i\ln f\left(x_i\right)-\sum _{i=1}^N{y}_i\ln \mathrm{\Delta }{x}_i,$$

where Equation (3) was used. The $C$ statistic for the linear model of Equation (2) is therefore

$$C=2\lambda R\left(1+\frac{aR}{2}\right)-2M\ln \lambda -2\sum _{i=1}^N{y}_i\ln (1+a(x_i-x_A\left)\right)+D$$ (5)

where

$$D=\left(2\sum _{i=1}^N{y}_i\ln y_i-2M-2\sum _{i=1}^N{y}_i\ln \mathrm{\Delta }{x}_i\right),$$

is a model-independent term that does not have an effect in the subsequent minimization of the statistic. Note that the binning of the data is not required to be uniform, as will be illustrated in subsequent examples.

3.2. Equations for the maximum-likelihood solutions

Minimization of the $C$ statistic in Equation (5) is obtained via

$$\frac{\mathrm{\partial }C}{\mathrm{\partial }\lambda }=2R+a{R}^2-2\frac{M}{\lambda }=0,$$

which leads to

$$\lambda \left(a\right)=\frac{M}{R\left(1+a{\displaystyle \frac{R}{2}}\right)}$$ (6)

and

$$\frac{\mathrm{\partial }C}{\mathrm{\partial }a}=\lambda R^2-2\sum _{i=1}^N{y}_i\frac{(x_i-x_A)}{1+a(x_i-x_A)}=0.$$

Substituting Equation (6) to eliminate λ leads to

$$\frac{MR}{1+a{\displaystyle \frac{R}{2}}}-2\sum _{i=1}^N{y}_i\frac{(x_i-x_A)}{1+a(x_i-x_A)}=0$$

and finally to

$$a=-\frac{2}{R}+\frac{M}{{\displaystyle \sum _{i=1}^N{y}_i\frac{(x_i-x_A)}{1+a(x_i-x_A)}}},$$ (7)

which is the equation to solve for the values of the a parameter. Equation (7) may be rearranged as

$$1+a\frac{R}{2}-\frac{MR}{2g\left(a\right)}=1+\frac{R}{2}\left(a-\frac{M}{g\left(a\right)}\right)=0;$$

It is thus convenient to define

$$F\left(a\right)\equiv 1+\frac{R}{2}\left(a-\frac{M}{g\left(a\right)}\right)$$ (8)

as the function whose zeros are solutions of Equation (7), with $g\left(a\right)$ defined as

$$g\left(a\right)\equiv \sum _{i=1}^N{y}_i\frac{(x_i-x_A)}{1+a(x_i-x_A)}.$$ (9)

The problem of finding solutions for the parameter a has therefore been cast as finding the zeros of a function $F\left(a\right)$, which uses the function $g\left(a\right)$ of Equation (9). One of the key properties to find the zeros of $F\left(a\right)$ is that the zeros of $g\left(a\right)$ are points of singularity for $F\left(a\right)$, as will be shown in detail in Section 4.

In summary, $F\left(a\right)=0$ and Equation (6) are the two equations to solve to find the maximum-likelihood estimators a and λ of the linear model of Equation (2). The two equations are uncoupled, and therefore the burden is limited to finding a solution of $F\left(a\right)=0$.² Then, Equation (6) is used to find $\lambda =\lambda \left(a\right)$. Notice that, in deriving Equations (6) and (7), no constraints were enforced to ensure that the best-fit model is non-negative, which is necessary for the applicability of the Poisson statistics and for the calculation $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$. These constraints are presented in Section 5.

4. Analytical properties of the maximum-likelihood solution of the linear model

The maximum-likelihood estimate of the parameters λ and a are obtained by first finding the zeros of the function $F\left(a\right)$ defined by Equation (8). For this purpose, it is necessary to establish a few analytical properties of the functions $F\left(a\right)$ and $g\left(a\right)$. These properties will be used to study the location and properties of the zeros of the function $F\left(a\right)$. It is necessary to discuss explicitly the simple case of data with M = 1 count before presenting general results for $M\ge 2$. The case of M = 0 counts is not interesting, since it represents a dataset with no positive measurements throughout the range of the independent variable.

4.1. Case of M = 1

When M = 1, or just one event ( $y_j=1$) recorded in a bin centered at $x_j$, one obtains

$$g\left(a\right)=\frac{x_j-x_A}{1+a(x_j-x_A)}$$

leading to

$$F\left(a\right)=1+a\frac{R}{2}-\frac{R(1+a(x_j-x_A\left)\right)}{2(x_j-x_A)}=1-\frac{R}{2(x_j-x_A)}$$

which is constant independent of a. The conclusion is that $F\left(a\right)=0$ has no solutions when the data have only one count, and it is therefore not possible to find a maximum-likelihood solution for the linear model with M = 1. A simple interpretation for this finding is that it is not possible to constrain a two-parameter model with just one non-null data point. Further discussion is provided in Section 7.

4.2. General case of $M\ge 2$

It is possible to find certain properties of $g\left(a\right)$ and $F\left(a\right)$ that apply in general. The properties will lead to a general criterion to identify solutions of $F\left(a\right)=0$.

Property 4.1 Properties of the function $g\left(a\right)$ —

The function $g\left(a\right)$, according to Equation (9), is the sum of $n\le M$ terms of type

$$g_j\left(a\right)=y_j\frac{(x_j-x_A)}{1+a(x_j-x_A)},$$

where n is the number of bins with $y_j\ne 0$, and $j=1,\dots ,n$ represents the bins with non-null counts; when bins have no more than one count, then n = M. Therefore, the function $g\left(a\right)$ has n points of singularity

$$a_j=-\frac{1}{(x_j-x_A)}$$

that fall in the range $(-2/\mathrm{\Delta }{x}_1,-1/(R-\mathrm{\Delta }{x}_N/2)$. Near the points of singularity,

$$\begin{array}{rl}& \underset{a\to a_j^{-}}{lim}g\left(a\right)=-\mathrm{\infty }\\ & \underset{a\to a_j^{+}}{lim}g\left(a\right)=+\mathrm{\infty }.\end{array}$$

It is also immediate to see that $g^{\mathrm{\prime }}\left(a\right)<0$ for all points where the function is continuous. Therefore, the function $g\left(a\right)$ decreases monotonically from $+\mathrm{\infty }$ to $-\mathrm{\infty }$ between two consecutive points of singularity. Moreover, the asymptotic limits are

$$\underset{a\to \pm \mathrm{\infty }}{lim}g\left(a\right)=0.$$

As a result, the function $g\left(a\right)$ has n−1 zeros, each between consecutive points of singularity, as illustrated in Example 4.2 and Figures 2 and 3. The zeros of $g\left(a\right)$ are points of singularity for $F\left(a\right)$.

In particular, when M = 2 with $y_j=y_k=1$ and no counts in any of the other bins, then the zero of $g\left(a\right)$ can be calculated analytically as

$$a_c=-\frac{1}{2}\left(\frac{1}{x_j-x_A}+\frac{1}{x_k-x_A}\right)<0(\text{case of }M=2\text{ only}).$$

For the general case of M>2, the zeros of $g\left(a\right)$ must be calculated numerically, as explained in the following.

Figure 2.

(Left:) Function $F\left(a\right)$ for a representative data set with M = 2, with 100 data points $x_i=i-0.5$, $i=1,\dots ,100$, and $y_{38}=y_{89}=1$. (Right:) Function $g\left(a\right)$ for the same data set.

Figure 3.

(Left:) Function $F\left(a\right)$ for a representative data set with M = 3, with 100 data points $x_i=i-0.5$, $i=1,\dots ,100$, and $y_{13}=y_{38}=y_{89}=1$. (Right:) L Function $g\left(a\right)$ for the same data set.

Property 4.2 Behavior of $F\left(a\right)$ near the points of singularity —

Since $g\left(a\right)$ is continuous with $g^{\mathrm{\prime }}\left(a\right)<0$ between its n points of singularity, $g\left(a\right)<0$ immediately to the left of the singularity and $g\left(a\right)>0$ immediately to the right. With $a_s$ any of the n−1 points of singularity of $F\left(a\right)$, this implies that

$$\begin{array}{rl}& \underset{a\to a_s^{-}}{lim}F\left(a\right)=-\mathrm{\infty }\\ & \underset{a\to a_s^{+}}{lim}F\left(a\right)=+\mathrm{\infty }.\end{array}$$

Property 4.3 Asymptotic limit of $F\left(a\right)$ —

The asymptotic limit of $F\left(a\right)$ at $\pm \mathrm{\infty }$, defined as

$$F_{\mathrm{\infty }}\equiv \underset{a\to \pm \mathrm{\infty }}{lim}F\left(a\right),$$

can be evaluated via the De L'Hospital rule for the associated function $a-M/g\left(a\right)$, which is an indeterminate form of the type 0/0:

$$\begin{array}{rl}\underset{a\to \pm \mathrm{\infty }}{lim}\left(\frac{ag\left(a\right)-M}{g\left(a\right)}\right)& =\underset{a\to \pm \mathrm{\infty }}{lim}\left(\frac{g\left(a\right)+a{g}^{\mathrm{\prime }}\left(a\right)}{g^{\mathrm{\prime }}\left(a\right)}\right)\\ & =-\frac{1}{M}\sum _{i=1}^N\frac{y_i}{x_i-x_A}=-\frac{1}{M}\sum _{j=1}^M\frac{1}{x_j-x_A}.\end{array}$$

This property can be proven with a few steps algebra, by turning the sum over the N bins to an equivalent sum over the individual M counts. Therefore the asymptotic limit of $F\left(a\right)$ becomes

$$\underset{a\to \pm \mathrm{\infty }}{lim}F\left(a\right)=F_{\mathrm{\infty }}=1-\frac{R}{2}\left(\frac{1}{M}\sum _{i=1}^N\frac{y_i}{x_i-x_A}\right).$$ (10)

Property 4.4 Sign of $F^{\mathrm{\prime }}\left(a\right)$ —

The derivative $F^{\mathrm{\prime }}\left(a\right)$ is calculated according to

$$F^{\mathrm{\prime }}\left(a\right)=\frac{R}{2}+\frac{MR}{2}\frac{g^{\mathrm{\prime }}\left(a\right)}{g^2\left(a\right)}.$$ (11)

For $M\ge 2$,

$$\frac{g^{\mathrm{\prime }}\left(a\right)}{g^2\left(a\right)}=-\frac{z_1^2+\cdots +z_M^2}{(z_1+\cdots +z_M{)}^2},\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}{z}_j\equiv \frac{x_j-x_A}{1+a(x_j-x_A)}$$

in which the sum is over all individual events, with some $z_j$ identical to each other if there are bins with more than one count. Using the Cauchy-Schwartz inequality

$$\frac{1}{M}{\left(\sum _{j=1}^M{z}_j\right)}^2\le \sum _{j=1}^M{z}_j^2$$

leads to

$$\frac{g^{\mathrm{\prime }}\left(a\right)}{g^2\left(a\right)}\le -\frac{1}{M}.$$ (12)

Finally, using Equation (12) into Equation (11) leads to the conclusion that $F^{\mathrm{\prime }}\left(a\right)\le 0$ for all points of continuity of $F\left(a\right)$.

These properties of the function $F\left(a\right)$ are also illustrated in Example 4.2 and Figures 2 and 3. The properties of $F\left(a\right)$ and $g\left(a\right)$ can be used to state a general criterion to locate all solutions of the equation $F\left(a\right)=0$.

Lemma 4.1 Location of zeros of $F\left(a\right)$ —

The function $F\left(a\right)$ has n−1 zeros, where $n\le M$ is the number of bins with non-null counts. Of these, n−2 zeros are found between the n−1 points of singularity of $F\left(a\right),$ also zeros of $g\left(a\right)$. The remaining zero is found either to the left of the smallest point of singularity, if the asymptotic limit is $F_{\mathrm{\infty }}>0,$ or to the right of the largest singularity, if $F_{\mathrm{\infty }}<0$.

Proof.

The result is a direct consequence of the presence of n points of singularity for $F\left(a\right)$ (Property 4.1), the negative sign of $F^{\mathrm{\prime }}\left(a\right)$ between points of singularity (Property 4.4) and the asymptotic limit of $F\left(a\right)$ at the points of singularity (Property 4.2).

Properties of $g\left(a\right)$ and $F\left(a\right)$ are illustrated in the following example, which examines the behavior of the two functions for two simple datasets with M = 2 and M = 3.

Example 4.2 Two datasets with M = 2 and M = 3 —

Two sample datasets with M = 2 and M = 3 are shown respectively in Figures 2 and 3. For the M = 2 dataset, with n = 2 bins with non-null counts, the function $F\left(a\right)$ has just one point of discontinuity for $F\left(a\right)$, also the zeros of $g\left(a\right)$. This point of discontinutiy divides the domain of a into two intervals, with $F\left(a\right)$ monotonically decreasing within these intervals, as shown in Figure 2. For the M = 3 dataset, with n = 3 bins with non-null counts, $F\left(a\right)$ has n−1 = 2 points of discontinuity corresponding to the two zeros of $g\left(a\right)$, which were in turn found between the n = 3 points of singularity of $g\left(a\right)$, as shown in Figure 3.

The n−1 zeros of $F\left(a\right)$ are all possible solutions of the maximum-likelihood method for the linear model. The following section discusses if these solutions are acceptable, in the sense that they produce a model $f\left(x\right)$ that is non-negative throughout the domain of the x variable.

5. Acceptable solutions for the best-fit parameters of the linear model

In Section 4 it was shown that there are several possible solutions for the maximum-likelihood parameters of the linear model of Equation (2). In particular, data with M total counts, distributed over $n\le M$ of the N available bins, have n−1 possible values of a that are a solution of the maximum-likelihood equation $F\left(a\right)=0$, with the corresponding value of λ provided by Equation (6). This section addresses the additional requirement that a model be non-negative in all bins, i.e. that a solution be acceptable, so that Poisson statistics apply and the $C$ statistic can be calculated.

Definition 5.1 Acceptable solution of $F\left(a\right)=0$ —

A solution of the $F\left(a\right)=0$ equation is said to be acceptable if it leads to a non-negative model throughout the support of the independent variable. Specifically, the function must satisfy the condition that $y\left(x_i\right)\ge 0,i=1,\dots ,N$, so that the parent mean of the Poisson distributions is always non-negative.

It will be shown in this section that there is at most one acceptable solution for any Poisson data set. Cases without an acceptable solution will be examined in Section 6, where a simple generalization of the linear model is provided that ensure one and only one acceptable solution for the fit of any data set to a linear model.

5.1. General conditions for acceptability

Given that the model is linear, the condition of acceptability is satisfied by simply requiring that the Poisson mean for the first and last bins, $y\left(x_1\right)$ and $y\left(N\right)$, are both non-negative,

$$\begin{array}{rl}& \lambda \cdot (1+a\mathrm{\Delta }{x}_1/2)\ge 0\\ & \lambda \cdot (1+a(R-\mathrm{\Delta }{x}_N/2\left)\right)\ge 0.\end{array}$$

Notice how the model $f\left(x\right)$ may still become negative in a portion of either the first or the last bin, but the linearity of the model simply requires that $f\left(x\right)$ at the mid-point of the bin be non-negative, in order to ensure that the Poisson mean for the bin is non-negative.

Substituting Equation (6), the conditions become a function of a alone,

$$\begin{array}{rl}& \frac{M}{R\left(1+a{\displaystyle \frac{R}{2}}\right)}(1+a\mathrm{\Delta }{x}_1/2)\ge 0\\ & \frac{M}{R\left(1+a{\displaystyle \frac{R}{2}}\right)}(1+a(R-\mathrm{\Delta }{x}_N/2\left)\right)\ge 0,\end{array}$$ (13)

This equation can be used to find a range of the variable a that contains acceptable solutions. Therefore, a solution of $F\left(a\right)=0$ is acceptable if and only if it satisfies Equation (13). This property leads to the following result regarding acceptable solutions:

Lemma 5.2 Necessary and sufficient condition for the acceptability of a solution of $F\left(a\right)=0$ —

A solution of $F\left(a\right)=0$ is acceptable if and only if it is found outside of the interval

$$\left(-\frac{2}{\mathrm{\Delta }{x}_1},-\frac{1}{R-\mathrm{\Delta }{x}_N/2}\right).$$

Proof.

The conditions of Equation (13) can be used to find values of the variable a that are acceptable solutions of $F\left(a\right)=0$. For $a<-2/R$, i.e. when the denominators in Equation (13) are negative, the two conditions are satisfied when $a<-2/\mathrm{\Delta }{x}_1$, since $\mathrm{\Delta }{x}_1<R-\mathrm{\Delta }{x}_N/2$. Likewise, for $a>-2/R$, the two conditions are satisfied when $a\ge -1/(R-\mathrm{\Delta }{x}_N/2)$. Therefore, acceptable solutions can be found in the range

$$\begin{array}{rlr}& a<-2/\mathrm{\Delta }{x}_1& (\text{and thus }\lambda <0)\\ & a>-1/(R-\mathrm{\Delta }{x}_N/2)& (\text{and thus }\lambda >0).\end{array}$$ (14)

Solutions with $a\in (-2/\mathrm{\Delta }{x}_1,-1/(R-\mathrm{\Delta }{x}_N/2)$ lead to a model that becomes negative in some of the bins, and therefore they are not acceptable. Figure 4 shows the function $\lambda \left(a\right)$ and illustrates the range of acceptable values for the parameters.

Figure 4.

(Left:) Parameter λ as a function of the parameter a, and range of acceptability of a. For values of $-2/\mathrm{\Delta }{x}_1<a<-1/(R-\mathrm{\Delta }{x}_N/2)$ the parameters $(a,\lambda )$ result in a linear model that becomes negative in some of the bin, and therefore not acceptable for use with the Poisson distribution. The smallest and largest points of singularity of $g\left(a\right)$ also correspond to the boundaries of this range, according to Property 4.1. The zeros of $g\left(a\right)$, also points of singularity for $F\left(a\right)$, are therefore inside this range. The x axis was plotted with the symlog option that allows a near-logarithmic scaling across a value of zero.

Example 5.3 M = 2 data with no acceptable solution —

The acceptability of the maximum-likelihood solution is illustrated with the data used for Figure 2. The only singularity of the $F\left(a\right)$ function is

$$a_c=-\frac{1}{2}\left(\frac{x_{38}}{1+a{x}_{38}}+\frac{x_{89}}{1+a{x}_{89}}\right)=-0.019$$

which is $a_c>-2/R$, and with a positive asymptotic value of $F_{\mathrm{\infty }}=0.051$, as shown in Figure 2. The $F\left(a\right)=0$ solution is a = −0.077, which falls in the range of unacceptable solutions. In fact, the corresponding $\lambda \left(a\right)=-0.007$ results in a best-fit model that is negative in some of the initial bins, e.g. $y\left(x_1\right)=\lambda (1+a\mathrm{\Delta }x/2)=-0.0067$. This best-fit model cannot be used to calculate the goodness-of-fit $C$ statistic, and therefore cannot be accepted as a maximum-likelihood solution.

5.2. General method to locate acceptable solutions

Lemma 5.4 Necessary condition for the acceptability of $F\left(a\right)=0$ solutions —

Solutions of $F\left(a\right)=0$ within points of singularity of $F\left(a\right)$ are always unacceptable.

Proof.

This condition applies to data with M>2 and n>2 unique bins with non-zero counts, so that $F\left(a\right)$ has $n-1\ge 2$ points of singularity. In this case, n−2 of the n−1 solutions of $F\left(a\right)=0$ are found between the n−1 points of singularity of $F\left(a\right)$, which are the zeros of the function $g\left(a\right)$. According to Property 4.1, the zeros of $g\left(a\right)$ are located between the n points of singularity of $g\left(a\right)$, given by

$$a_j=-\frac{1}{x_j-x_A}\in \left(-\frac{2}{\mathrm{\Delta }{x}_1},-\frac{1}{R-\mathrm{\Delta }{x}_N/2}\right),$$ (15)

where $x_j$ is the coordinate of each of the $n\le M$ unique bins where non-zero counts are recorded. According to Lemma 5.2, these n−2 solutions of $F\left(a\right)=0$ fall in the interval on unacceptability.

Lemma 5.4 states that all zeros of $F\left(a\right)$ that are within points of singularity may be discarded as unacceptable. The only possibility for an acceptable solution is the zero that is located outside of the range of the points of singularity, although such zero is not guaranteed to be acceptable. Accordingly, the following definition is made:

Definition 5.5 External solution of $F\left(a\right)=0$ —

A solution of $F\left(a\right)=0$ is said to be external if it falls outside of the range of the points of singularity of $F\left(a\right)$.

An external solution is therefore found either to the left of the first point of singularity of $F\left(a\right)$, if the asymptotic value $F_{\mathrm{\infty }}>0$, or to the right of the last, if $F_{\mathrm{\infty }}<0$. For M = 2, this is the only solution of $F\left(a\right)=0$, and the point of singularity of $F\left(a\right)$ is calculated according to the equation provided at the end of Property 4.1.

Lemma 5.6 Necessary and sufficient conditions for the acceptability of an external $F\left(a\right)=0$ solutions —

An external solution of $F\left(a\right)=0$ is acceptable if and only if the following conditions are met, according to the sign of the asymptotic value $F_{\mathrm{\infty }}$:

$$\begin{array}{rlr}& F\left(-\frac{2}{\mathrm{\Delta }{x}_1}\right)<0,& \mathrm{i}\mathrm{f}{F}_{\mathrm{\infty }}>0\\ & F\left(-\frac{1}{R-\mathrm{\Delta }{x}_N/2}\right)>0,& \mathrm{i}\mathrm{f}{F}_{\mathrm{\infty }}<0.\end{array}$$ (16)

Proof.

This property is simply based on the continuity of the function $F\left(a\right)$ between points of singularity, and on Lemma 5.2, which established that acceptable values of the parameter a are outside of the interval $(-2/\mathrm{\Delta }{x}_1,-1/(R-\mathrm{\Delta }{x}_N/2)$.

(a) if $F_{\mathrm{\infty }}>0$, the solution of $F\left(a\right)=0$ is to the left of the point of singularity. Given that $F^{\mathrm{\prime }}\left(a\right)<0$, the solution will fall in the range of acceptability, viz., $a<-2/R$, if $F(-2/\mathrm{\Delta }{x}_1)<0$.

(b) Likewise, if $F_{\mathrm{\infty }}<0$, the solution is to the right of the point of singularity, and the solution is acceptable if $F(-1/(R-\mathrm{\Delta }{x}_N/2\left)\right)>0$.

The condition is also necessary. In fact, if Equation (16) is not satisfied, e.g. $F(-2/\mathrm{\Delta }{x}_1)<0$ for $F_{\mathrm{\infty }}>0$, then the zero will be in the region of unacceptability.

This necessary and sufficient condition can be immediately applied to data that have non-zero counts, and thus points of singularity of $g\left(a\right)$, at the extremes of the range.

Corollary 5.7 Sufficient conditions for data with non-zero counts in first or last bin —

If a data set with $M\ge 2$ satisfies either of the two conditions

$$\begin{array}{r}{y}_1\ge 1\mathrm{a}\mathrm{n}\mathrm{d}{F}_{\mathrm{\infty }}>0\\ y_N\ge 1\mathrm{a}\mathrm{n}\mathrm{d}{F}_{\mathrm{\infty }}<0\end{array}$$ (17)

then the external solution of the $F\left(a\right)=0$ equation is acceptable.

Proof.

According to Equation (8), $F\left(a\right)=1+a\cdot R/2$ at points of singularity for $g\left(a\right)$. A non-zero count in the first bin leads to a singularity of $g\left(a\right)$ at $a_j=-\mathrm{\Delta }{x}_1/2$, and therefore $F(-2/\mathrm{\Delta }{x}_1)<0$. Therefore, according to Lemma 5.6, if $F_{\mathrm{\infty }}>0$, there is an acceptable solution to the left of $-2/\mathrm{\Delta }{x}_1$. Similar considerations are applicable to the case of a non-zero count in the last bin, where a singularity of $g\left(a\right)$ occurs instead at $a_j=-1/(R-\mathrm{\Delta }{x}_N/2)$, where $F(-1/(R-\mathrm{\Delta }{x}_N/2\left)\right)>0$. In this case, if $F_{\mathrm{\infty }}<0$, the external solution of $F\left(a\right)=0$ to the right of the last singularity of $g\left(a\right)$ is acceptable, again according to Lemma 5.6.

Notice how Corollary 5.7 does not ensure an acceptable solution simply if either the last or first bin have non-null counts. In fact, the presence of an acceptable solution is conditioned also on the sign of $F_{\mathrm{\infty }}$. For example, a data set with a non-null last bin but with a positive $F_{\mathrm{\infty }}$ will not have an acceptable solution to the right of the last singularity. Finally, the two earlier lemmas can be used to state the uniqueness of the maximum-likelihood solution for the linear model.

Lemma 5.8 Uniqueness of an acceptable solution of $F\left(a\right)=0$ —

If there is an acceptable solution of $F\left(a\right)=0,$ this solution is unique.

Proof.

This property is an immediate consequence of the fact that, of the n−1 solutions of $F\left(a\right)=0$, the n−2 solutions within points of singularity cannot be acceptable, as per Lemma 5.4. Moreover, the remaining solution may be acceptable, according to Lemma 5.6.

Example 5.9 Example of data with M = 5 and an acceptable solution —

Figure 5 shows the $F\left(a\right)$ and $g\left(a\right)$ functions for a dataset with M = 5 counts in 5 equally spaced bins ( $x_i=9.5,29.5,49.5,69.5,89.5$), and therefore n = 5. The function $g\left(a\right)$ has n = 5 points of singularity and n−1 = 4 zeros, which correspond to the 4 points of singularity of $F\left(a\right)$. There are also 4 zeros of the function $F\left(a\right)$, of which n−2 = 3 correspond to unacceptable solutions. The asymptotic value if $F_{\mathrm{\infty }}<0$, and therefore the remaining external zero is to the right of the last singularity. At the end point $a_2$ of the range of acceptability, the function is $F\left(a_2\right)>0$, and therefore the last zero leads to an acceptable solution.

The data and all models are shown in Figure 6. Notice how the model corresponding to the solutions of $F\left(a\right)=0$ that are not acceptable lead to a model that becomes negative; these models cannot be used for the $C$ statistic, and need to be rejected. The only acceptable model is shown as a solid line, and the corresponding values of the $C$ statistic for each bin are shown in the right panel.

Figure 5.

Functions $F\left(a\right)$ and $g\left(a\right)$ for the dataset presented in Example 5.9.

Figure 6.

(Left) Best-fit linear models for the M = 5 data presented in Example 5.9. There are 4 solutions of $F\left(a\right)=0$, of which the first three lead to models that become negative somewhere in the x range; the acceptable model corresponds to the largest solution. (Right) Contributions to $C$ statistic for each of the N = 200 bins, for a total of $C_{min}=29.96$.

The results presented in this section can be summarized by a simple algorithm that can be used to determine whether there is an acceptable solution of the equation $F\left(a\right)=0$, and to calculate it, when it exists.

Remark 5.1 Algorithm to determine acceptable best-fit parameters of Equation (2) —

Consider a dataset with N bins, a range R of the independent variable between $x_A$ and $x_B$, a total number of integer-valued counts M with a number $n\le M$ bins with non-null counts. The existence and value of the best-fit parameters $\{a,\lambda \}$ for the linear model of Equation (2) can be determined according to the following steps:

1. If $n\le 1$, there is no acceptable solution.

2. Calculate the $n\ge 2$ points of singularity for $g\left(a\right)$, given analytically by Equation (15).

3. Numerically calculate the n−1 zeros of $g\left(a\right)$ between points of singularity. These zeros are points of singularity for $F\left(a\right)$.

4. Calculate the asymptotic value $F_{\mathrm{\infty }}$, given analytically by Equation (10).

5. (Optional) Numerically calculate the n−2 zeros of $F\left(a\right)$, found between points of singularity of $F\left(a\right)$ (also zeros of $g\left(a\right)$. These zeros always lead to unacceptable solutions.

6. Numerically calculate the remaining external zero of $F\left(a\right)$, either to the left of the first point of singularity (if $F_{\mathrm{\infty }}>0$), or to the right of the last singularity (if $F_{\mathrm{\infty }}<0$).

7. Determine the acceptability of the external solution. Two cases are possible:

   1. If this value is outside of the range $(-2/\mathrm{\Delta }{x}_1,-1/(R-\mathrm{\Delta }{x}_N/2)$, then the solution is acceptable. The corresponding value of λ can be calculated according to Equation (6).
   2. If this value is inside the range $(-2/\mathrm{\Delta }{x}_1,-1/(R-\mathrm{\Delta }{x}_N/2)$, then the solution is not acceptable. The data do not have an acceptable solution with the model of Equation (2).

The numerical solution of both equations $g\left(a\right)=0$ and $F\left(a\right)=0$ are facilitated by the continuity of the two functions between the known points of singularity, or between the last point of singularity and $\pm \mathrm{\infty }$. An efficient and accurate numerical routine is provided, for example, by python's root_scalar, with the brentq method. The method requires the specification of an interval, or bracket, where the solution is sought. This is either an interval between the two adjacent points of singularity, or an open interval either below the first singularity, or above the last singularity. For example, a zero of $g\left(a\right)$ can be sought in the range $[a_j+ϵ,a_{j+1}-ϵ]$, where $a_j$ and $a_{j+1}$ are two consecutive points of singularity of $g\left(a\right)$. This bracket requires a small value ε, to be determined according to the separation of the data points, to ensure that the function $g\left(a\right)$ at the two extremes of the bracket has opposite signs.

5.3. Asymptotic data requirements for acceptable solutions

This section examines when data sets with a large number of counts have an acceptable solution. It will be shown that, when the counts are distributed uniformly across the support, data with large M will always have an acceptable solution. In general, however, it is possible to find datasets with large M that do not have an acceptable solution, depending on the distribution of counts. This observation will lead to a generalization of the simple model of Equation (2), presented in Section 6. First, it is necessary to investigate how the asymptotic value of $F\left(a\right)$ is affected by the distribution of the detected counts.

Property 5.1 Properties of $F_{\mathrm{\infty }}$ —

The asymptotic value of $F\left(a\right)$ is given by Equation (10), and it is negative if

$$\frac{1}{M}\sum _{i=1}^N\frac{y_i}{x_i-x_A}=\frac{1}{M}\sum _{i=1}^M\frac{1}{x_i-x_A}>\frac{2}{R}$$ (18)

and positive otherwise. Given that $0<x_i-x_A<R$, each term $1/(x_i-x_A)$ has a value $<2/R$ if $x_i$ is above the midpoint of the range, and a value $>2/R$ if $x_i$ is below the midpoint. The left-hand side of Equation (18) is the sample mean of the variable $1/(x_i-x_A)$.

It can now be established that, for data with a large number of bins and a uniform distribution of the counts, the asymptotic value of $F\left(a\right)$ is negative. Moreover, when the number of counts M is also large, the external solution to the right of the last singularity will be acceptable.

Lemma 5.10

For a large number of bins N and a uniform distribution of counts,

$$E\left[\frac{1}{x_i-x_A}\right]>\frac{2}{R},$$

where $E\left[\right]$ is the expectation based on a parent distribution for the position $x_i$ of the i-th count. Moreover, when M is large, the asymptotic value of $F\left(a\right)$ is negative.

Proof.

Assuming bins of uniform width, the range is $R=\mathrm{\Delta }x\cdot N$. The distance of the i-th count from the initial point of the range is $x_i-x_A\in (\mathrm{\Delta }x/2,R-\mathrm{\Delta }x/2)$, and it can be written as

$$x_i-x_A=\frac{\mathrm{\Delta }x}{2}+f(R-\mathrm{\Delta }x),$$

where $f\in (0,1)$. When N is large and the counts are uniformly distributed in the N bins, it is possible to treat f as a continuous and uniformly distributed random variable in the range $(0,1)$, thus with unit probability distribution function. Accordingly, the expectation of the inverse of the distance $x_i-x_A$ can be approximated as

$$E{\left[\frac{1}{x_i-x_A}\right]}_{\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{f}}\simeq {\int }_0^1\frac{\mathrm{d}f}{\mathrm{\Delta }x/2+f(R-\mathrm{\Delta }x)}=\frac{\ln (R-\mathrm{\Delta }x/2)-\ln \mathrm{\Delta }x/2}{R-\mathrm{\Delta }x}\simeq \frac{\ln N}{R}$$

Therefore the expectation is asymptotically larger than 2/R for large N. Moreover, for a large number of counts M, the law of large numbers ensures that the sample average of $1/(x_i-x_A)$ tends to its expectation. Therefore, as M increases, the asymptotic value of $F\left(a\right)$ tends to be negative, and the external solution of $F\left(a\right)=0$ will be found to the right of the last point of singularity for $F\left(a\right)$.

It is now possible to state a sufficient condition that applies to uniformly distributed counts in the large-count regime.

Lemma 5.11 Sufficient condition for an acceptable solution —

For large M and N with uniformly distributed counts and a non-null count in the last bin, the external solution of $F\left(a\right)=0$ is acceptable and it is found to the right of the last singularity.

Proof.

For data with non-null counts in the last bin, i.e. $y_N\ge 1$, the last singularity of $g\left(a\right)$ occurs at $a_j=-1/(R-\mathrm{\Delta }{x}_N/2)$. At points of singularity for $g\left(a\right)$, $F\left(a_j\right)=1+R/2\cdot a_j$, according to Equation (8), and therefore $F\left(a_j\right)>0$. Notice that the point $a_j$ marks the boundary of the region of acceptability for the solutions of $F\left(a\right)=0$, according to Lemma 5.2.

The last singularity for $F\left(a\right)$, also a zero of $g\left(a\right)$, will thus occur at a point $a_s$ which is to the left of $a_j$, and the continuity of $F\left(a\right)$ to the right of the last singularity ensures that $F\left(a\right)$ remains positive between $a_s^{+}$ and $a_j$. Also, Lemma 5.10 ensures that the asymptotic value of $F\left(a\right)$ is negative. Therefore there is a zero of $F\left(a\right)$ to the right of $a_j$, and this external zero is acceptable, according to Lemma 5.2.

These asymptotic results apply to a uniform distribution of counts, which is a very restrictive condition. When the distribution of counts is not uniform, even large-M datasets may not have an acceptable model. It goes beyond the scope of this paper to seek additional sufficient conditions for convergence, given the number of variables at play (in particular, the number of counts M, the number of bins N and their size and location, and the distribution of counts), and the fact that necessary and sufficient conditions for convergence have been provided earlier in this section. Instead, selected numerical simulations are presented to quantify the fraction of Poisson datasets that do not have a non-negative best-fit linear and to illustrate a few representative cases.

For this purpose, 100 data sets were simulated for various values of the total number of counts M, initially assuming that the M counts were uniformly distributed among N = 100 equally spaced bins, following the same pattern of bins along the x axis as in Figures 2 and 3. As expected, based on the asymptotic results of this section, for $M\ge 50$, all datasets have an acceptable model (Figure 7, red curve). Then, the same simulations were repeated for a distribution of counts that is either linearly increasing or decreasing towards larger values of x, i.e. with samples drawn respectively from the probability distributions functions

$$h\left(x\right)=\left\{\begin{array}{l}\frac{2(x-x_A)}{R^2}\\ \frac{2}{R}-\frac{2(x-x_A)}{R^2}\end{array}\right.$$

with $x\in [0,R]$.³ For these cases, the simulations show that the number of acceptable models remains smaller even for large values of M. The right panel of Figure 7 also illustrates the fraction of data with a negative $F_{\mathrm{\infty }}$. As expected according to Lemma 5.10, uniformly distributed data (red curve) have a negative asymptotic $F_{\mathrm{\infty }}$ for large M; moreover, the same applies for data distributed with a negative slope (blue curve). This is explained according to Property 5.1, since data points below the mid-point of the range drive the average of $1/(x_i-x_A)$ to values greater than 2/R, and that in turn causes $F_{\mathrm{\infty }}$ to be negative. For data with a positive slope, even for large M there is a large fraction of data with a positive asymptotic limit of $F\left(a\right)$. These simulations can be used as examples of large-M data that do not have an acceptable solution using the linear model of Equation (2).

Figure 7.

(Left:) Fraction of datasets with available best-fit non-negative linear model, as function of the number of counts M. (Right:) Fraction of datasets with negative asymptotic value $F_{\mathrm{\infty }}$.

6. An extended linear model with a non-negative solution

The paper has identified cases where the maximum-likelihood equations do not yield an acceptable solution for the parameters of the linear model. In particular, this is true for all data with only one count (M = 1) and null counts in N−1 of the N available bins. This can simply be viewed as the inability to constrain two free parameters with just one non-zero data point. In such case, it may be sufficient to model the data with a simple constant model, with a best-fit model equal to the sample average of the counts in all the bins (see, e.g. [3,4]). There are also other data sets with $M\ge 2$ counts that do not have an acceptable, non-negative model. One such example was shown in Figure 2, for a dataset with M = 2. Section 5 also illustrated data with large M that do not have an acceptable solution (see, e.g. Figure 7).

Motivated by the need to have a linear model that is applicable to any situation, this section proposes a simple generalization of the linear model of Equation (2) that ensures an acceptable maximum-likelihood solution using the $C$ statistic for any Poisson dataset.

Definition 6.1 The extended non-negative linear model —

The proposed non-negative linear model is given by:

1. the standard linear model of Equation (2), when such model has an acceptable solution; otherwise,

2. the model is parameterized as one of the following three functions:

   1. A one-parameter linear model pivoted to zero at the initial point $x_A$:

      $$f_A\left(x\right)={\lambda }_A(x-x_A),$$ (19)

      for which $y_A\left(x_A\right)=0$, and with a positive adjustable parameter ${\lambda }_A\ge 0$.
   2. A one-parameter linear model pivoted to zero at the final point $x_B$:

      $$f_B\left(x\right)={\lambda }_B\left(1-\frac{x-x_A}{R}\right)$$ (20)

      for which $y_B\left(x_B\right)=0$, with an adjustable parameter ${\lambda }_B\ge 0$ and therefore a negative slope.
   3. A one-parameter constant model:

      $$f_C\left(x\right)={\lambda }_C.$$ (21)

It will be shown that the three models of Equations (19), (20) and (21) have simple analytical solutions for their maximum-likelihood best-fit parameters (respectively ${\lambda }_A$, ${\lambda }_B$ and ${\lambda }_C$), and therefore it is always possible to use one of these models as an acceptable linear model for any dataset.

6.1. Maximum-likelihood solutions for the pivoted and constant linear models

For the linear model pivoted at $x_A$, Equation (19) is used to evaluate the $C$ statistic, Equation (1). Assuming that the data covers the range R continuously, as also assumed for Equation (4), the term

$$\sum _{i=1}^N{\mu }_i={\int }_{x_A}^{x_B}{f}_A\left(x\right)\mathrm{d}x={\lambda }_A\frac{R^2}{2}$$ (22)

leads to

$$C_A={\lambda }_A{R}^2-2M\ln {\lambda }_A+D_A$$ (23)

where

$$D_A\equiv \left(-2M+2\sum _{i=1}^N{y}_i\ln y_i-2\sum _{i=1}^N{y}_i\ln \mathrm{\Delta }{x}_i-2\sum _{i=1}^N{y}_i\ln (x_i-x_A)\right)$$ (24)

is a term that is independent of the model, and therefore plays no role in the minimization of the $C$ statistic. The best-fit parameter is given by $\mathrm{\partial }{C}_A/\mathrm{\partial }{\lambda }_A=0$, leading to the simple analytical solution

$${\lambda }_A=\frac{2M}{R^2}>0.$$ (25)

For the linear model pivoted at $x_B$, use of Equation (20) into Equation (1) leads to

$$\sum _{i=1}^N{\mu }_i={\int }_{x_A}^{x_B}{f}_B\left(x\right)\mathrm{d}x={\lambda }_B\frac{R}{2}$$ (26)

and

$$C_B={\lambda }_{B}R-2M\ln {\lambda }_B+D_B$$ (27)

with

$$D_B\equiv \left(-2M+2\sum _{i=1}^N{y}_i\ln y_i-2\sum _{i=1}^N{y}_i\ln \mathrm{\Delta }{x}_i-2\sum _{i=1}^N{y}_i\ln \left(1-\frac{x_i-x_A}{R}\right)\right).$$ (28)

The best-fit parameter is therefore given by

$${\lambda }_B=\frac{2M}{R}>0.$$ (29)

Finally, the best-fit constant model has a $C$ statistic of

$$C_C=2{\lambda }_{C}R-2M\ln {\lambda }_c+D_C$$ (30)

with

$$D_C\equiv \left(-2M+2\sum _{i=1}^N{y}_i\ln y_i-2\sum _{i=1}^N{y}_i\ln \mathrm{\Delta }{x}_i\right).$$ (31)

This leads to a best-fit parameter

$${\lambda }_C=\frac{M}{R},$$ (32)

which is equivalent to the sample average of the data when multiplied by a uniform $\mathrm{\Delta }x$, as found in [3]. As already remarked after Equation (4), the equations developed in this section apply to data that cover continuously the range $x_A$ to $x_B$. Data with gaps in the x variable require a simple modification to these equations that is presented in Section 6.3.

6.2. Use of the extended non-negative linear model

Equation (2) in combination with the extensions provided by Equations (19), (20), and (21) are to be used according to the following method, which defines the solution of the extended model.

Definition 6.2 Solution of the extended non-negative linear model —

Solution of the extended non-negative linear model is given by:

1. the solution with the standard linear model of Equation (2), if that solution is acceptable. As shown in Section 5 and specifically Lemma 5.8, this solution is guaranteed to be unique, when it exists.

2. If a solution with the standard linear model is not available, the solution is given by the best-fit model that gives the lowest value of the $C$ statistic, among the three options provided by Equations (19), (20) and (21).

Lemma 6.3 Existence and uniqueness of solution for the extended non-negative linear model —

There exists one and only one maximum-likelihood solution for the extended non-negative linear model fit to any Poisson-distributed data.

Proof.

The proof is a direct consequence of the fact that there is at most one non-negative solution for the model of Equation (2) (Lemma 5.8), and of Definition 6.2 for the solution of the extended model.

Remark 6.1 Expanded algorithm for the extended non-negative linear model —

The algorithm presented in Remark 5.1 can be extended to the non-negative linear model. When the linear model of Equation (2) fails to produce an acceptable solution, the following two additional steps must be added:

• (8)Calculate the three additional best-fit linear models (pivoted at A, pivoted at B and constant) and their $C$ statistic, using the analytical formulas (19), (20) and (21).
• (9)Accept as the best-fit model the one with the lowest $C$ statistic. Notice that if the original linear model of Equation (2) is acceptable, its value of the $C$ statistic will be lower than that of the other three linear models.

The use of the extended non-negative linear model is illustrated in the two following examples.

Example 6.4 Use of the extended non-negative model for data with no acceptable standard linear model —

In the left panel of Figure 8 are shown the results for the same M = 2 data of Figure 2, for which a non-negative linear model according to Equation (2) could not be found. The data can be fit with the pivoted and constant linear models, which yield best-fit $C$ statistic values of $C_A=15.081$, $C_B=18.141$ and $C_C=15.648$. The values of the $C$ statistic indicate that the linear model pivoted at $x_A$ is the most accurate representation of these data, and should be regarded as the best-fit linear model.

Figure 8.

Pivoted and constant linear models for the data of Figure 2 (with M = 2, left) and Figure 3 (M = 3, right).

Example 6.5 Use of the extended non-negative model for data with an acceptable standard linear model —

The right panel of Figure 8 shows the results for the M = 3 model of Figure 3, for which a best-fit non-negative model with the ‘standard’ linear model was in fact available, for a $C$ statistic value of C = 20.996. The pivoted and constant linear models yield values of $C_A=23.245$, $C_B=22.413$ and $C_C=21.039$, all larger than the value for the best-fit standard linear model. This analysis confirms that the ‘standard’ linear model, when available, is indeed the most accurate linear representation of the data.

It is in principle possible to devise a linear model different from those of Equations (19), (20) and (21), that may yield a lower value of the $C$ statistic. There are in fact infinitely many such models, e.g. by fixing an arbitrary intercept of the $x$ axis. The choices made by the three simple extensions discussed in this paper are intended to provide simple alternatives to the full linear model that have a simple interpretation and likewise simple analytical solutions.

6.3. Binning and gaps in data

The methods of analysis presented in this paper can be applied to data with any binning, including data with non-uniform bin sizes. The bin sizes, however, will have an effect on the best-fit model, as can be seen by the fact that the function $F\left(a\right)$ is a function of $x_j-x_A$, there $x_j$ is the center coordinate of the j-th bin. When Poisson data are collected on an event-by-event basis, the choice of bin size must be made based on considerations on the methods of collection of the data and the instruments used for the collection.

In Equations (4), (22) and (26) it was assumed that the range of integration of the x variable was continuous, therefore implying that the data covers the $x_A$ to $x_B$ range without any gaps or missing data. It is possible to provide a simple generalization to those equations to include gaps in the data. This is in fact a situation of practical importance, since certain regions of the independent variable may be without data for a variety of reasons. A common situation is the exclusion of portions of the x variable because of poor calibration of the instrument (e.g. the exclusion of a wavelength range because of detector inefficiencies), or because an instrument was not operating during certain time intervals. In these cases, one cannot just assign a value of zero counts to that range of the independent variable, but rather the intervals must be explicitly removed from the data, therefore creating gaps in an otherwise continuous variable.

Definition 6.6 Gaps in the data —

A gap in the data is defined as a continuous interval of the independent variable between $x_a$ and $x_b$, of length $R_G=x_b-x_a$, that is not covered by any of the bins. A Poisson data set may have g non-overlapping gaps between $x_{a,j}$ and $x_{b,j}$, $j=1,\dots ,g$, with $x_{G,j}=(x_{b,j}+x_{a,j})/2$ the mid-point of each gap and $R_{G,j}=x_{b,j}-x_{a,j}$. The length of all gaps in the independent variable x is $R_G=\sum R_{G,j}$.

The following lemmas summarize the changes that need to be made to analyze data that contain gaps in the independent variable

Lemma 6.7 Modifications to the $C$ statistic and to the functions $F\left(a\right)$ and $\lambda \left(a\right)$ for gaps in the data —

When the data have gaps, the $C$ statistic becomes

$$\begin{array}{rl}C& =2\lambda R\left(1+\frac{aR}{2}\right)-2\lambda \sum _{j=1}^g{R}_{G,j}(1+a(x_{G,j}-x_A\left)\right)\\ & -2M\ln \lambda -2\sum _{i=1}^N{y}_i\ln (1+a(x_i-x_A\left)\right)+D.\end{array}$$ (33)

Moreover, the function whose zero provides the best-fit value of a becomes

$$F\left(a\right)=1+a\frac{R_m}{2}-\frac{M{R}_m}{2g\left(a\right)}$$ (34)

where R is replaced by a modified $R_m$ given by

$$\begin{array}{rl}& R_m\equiv \frac{R^2-2S_G}{R-R_G}\\ & S_G\equiv \sum _{j=1}^g{R}_{G,j}\left(x_{G,j}-x_A\right),\end{array}$$ (35)

and the best-fit solution for the parameter λ is

$$\lambda \left(a\right)=\frac{M}{R\left(1+a{\displaystyle \frac{R}{2}}\right)-(R_G+a{S}_G)}.$$ (36)

Proof.

The modification to the $C$ statistic to account for the presence of gaps is provided by changing Equation (4) to

$$\begin{array}{rl}\sum _{i=1}^N{\mu }_i& ={\int }_{x_A}^{x_B}f\left(x\right)\mathrm{d}x-\sum _{j=1}^g{\int }_{x_{a,j}}^{x_{b,j}}f\left(x\right)\mathrm{d}x\\ & =\lambda R\left(1+a\frac{R}{2}\right)-\lambda \sum _{j=1}^g{R}_{G,j}(1+a(x_{G,j}-x_A\left)\right).\end{array}$$ (37)

The use of Equation (37) in place of Equation (4) leads to the $C$ statistic of Equation (33) in place of the original Equation (5). Taking the derivatives of C with respect to a and λ and setting them to zero leads to

$$\frac{\mathrm{\partial }C}{\mathrm{\partial }\lambda }=2R\left(1+a\frac{R}{2}\right)-2\sum _{j=1}^g{R}_{G,j}(1+a(x_{G,j}-x_A\left)\right)-2\frac{M}{\lambda }=0,$$

and

$$\frac{\mathrm{\partial }C}{\mathrm{\partial }a}=\lambda R^2-2\lambda \sum _{j=1}^g{R}_{G,j}(x_{G,j}-x_A)-2\sum _{i=1}^N{y}_i\frac{(x_i-x_A)}{1+a(x_i-x_A)}=0.$$

Notice that

$$\sum _{j=1}^g{R}_{G,j}(1+a(x_{G,j}-x_A\left)\right)=R_G+a\sum _{j=1}^g{R}_{G,j}(x_{G,j}-x_A)$$

where $R_G$ is the combined length of all ( non-overlapping) gaps. Defining

$$S_G\equiv \sum _{j=1}^g{R}_{G,j}(x_{G,j}-x_A)$$ (38)

leads to

$$R\left(1+a\frac{R}{2}\right)-R_G-a{S}_G-\frac{M}{\lambda }=0,$$

thus proving Equation (36), and

$$\lambda R^2-2\lambda S_G-2g\left(a\right)=0$$

where $g\left(a\right)$ is the usual function as defined in Equation (9). Simple algebraic modifications and elimination of λ lead to

$$1+a\frac{R_m}{2}-\frac{M{R}_m}{2g\left(a\right)}=0$$

where

$$R_m\equiv \frac{R^2-2S_G}{R-R_G},$$

thus proving Equation (34).

Lemma 6.7 shows that, when there are gaps in the independent variable, the method of analysis to find a solution for a and λ proceeds in the same way as when there are no gaps, provided the function $F\left(a\right)$ uses the $R_m$ parameter in place of R. Once the best-fit value of a is found, λ can be calculated analytically by making a change in the denominator of the function $\lambda \left(a\right)$ to account for the gap $R_G$, according to Equation (36).

Lemma 6.8 Modifications to the $C$ statistic and to the best-fit parameters of the pivoted and constant models for gaps in the data —

When the data have gaps, the $C$ statistic for the pivoted and constant models become

$$\begin{array}{rl}& C_A={\lambda }_A{R}^2-{\lambda }_A{S}_A^2-2M\log {\lambda }_A+D_A\\ & C_B={\lambda }_{B}R-2{\lambda }_B{S}_B-2M\ln {\lambda }_B+D_B\\ & C_C=2{\lambda }_C(R-R_G)-2M\ln {\lambda }_C+D_C\end{array}$$ (39)

with

$$\begin{array}{rl}& S_A^2\equiv \sum _{j=1}^g{x}_{b,j}^2-x_{a,j}^2\\ & S_B\equiv \sum _{j=1}^g\frac{R_{G,j}}{R}(x_B-x_{G,j}).\end{array}$$ (40)

The best-fit model parameters become

$$\begin{array}{rl}& {\lambda }_A=\frac{2M}{R^2-2S_G}\\ & {\lambda }_B=\frac{2M}{R-2S_B}\\ & {\lambda }_C=\frac{M}{R-R_G}.\end{array}$$ (41)

Proof.

For the model pivoted at A, Equation (22) is modified by the presence of gaps as

$$\begin{array}{rl}\sum _{i=1}^N{\mu }_i& ={\int }_{x_A}^{x_B}{f}_A\left(x\right)\mathrm{d}x-\sum _{j=1}^g{\int }_{x_{a,j}}^{x_{b,j}}{f}_A\left(x\right)\mathrm{d}x\\ & ={\lambda }_A\left(\frac{x_B^2-x_A^2}{2}-R{x}_A\right)-{\lambda }_A\left(\sum _{j=1}^g\frac{x_{b,j}^2-x_{a,j}^2}{2}-\sum _{j=1}^g{x}_A(x_{b,j}-x_{a,j})\right).\end{array}$$ (42)

Defining

$$S_A^2\equiv \sum _{j=1}^g(x_{b,j}^2-x_{a,j}^2)$$ (43)

and noticing that

$$\sum _{j=1}^g{x}_A(x_{b,j}-x_{a,j})=x_A{R}_G$$

leads to

$$C_A={\lambda }_A(x_B^2-x_A^2)-{\lambda }_A{S}_A-2{\lambda }_A{x}_A(R-R_G)-2M\ln {\lambda }_A+D_A.$$

Since $S_A^2-2x_A{R}_G=2S_G$ and $(x_B^2-x_A^2)-x_{A}R=R^2$, it follows that

$$C_A={\lambda }_A(R^2-2S_G)-2M\ln {\lambda }_A+D_A.$$

Then, taking a derivative of $C_A$ with respect to ${\lambda }_A$ and setting it to zero completes the proof for the model pivoted at A.

For the model pivoted at B,

$$\begin{array}{rl}\sum _{i=1}^N{\mu }_i& ={\int }_{x_A}^{x_B}{f}_B\left(x\right)\mathrm{d}x-\sum _{j=1}^g{\int }_{x_{a,j}}^{x_{b,j}}{f}_B\left(x\right)\mathrm{d}x\\ & ={\lambda }_B\left(R+x_A-\frac{x_B^2-x_A^2}{2R}\right)-{\lambda }_B\sum _{j=1}^g\left(1+\frac{x_A}{R}\right)R_{G,j}-\left(\frac{x_{b,j}^2-x_{a,j}^2}{2R}\right)\\ & ={\lambda }_B\left(\frac{R}{2}-S_B\right)\end{array}$$ (44)

where

$$S_B\equiv \sum _{j=1}^g{R}_{G,j}\left(1-\frac{x_{G,j}-x_A}{R}\right)=\sum _{j=1}^g\frac{R_{G,j}}{R}(x_B-x_{G,j}).$$

From this, the equations for $C_B$ and ${\lambda }_B$ follow after a few simple algebra steps.

The results for the constant model follow immediately from the constancy of the function $f_C\left(x\right)={\lambda }_C$.

Lemma 6.8 shows that the pivoted and constant models retain a simple analytical solution even in the presence of gaps in the data. An application of the fit to Poisson data with non-uniform bin sizes and with a gap in the data is provided in the following example.

Example 6.9 Data with non-uniform bin sizes and a gap in the data —

The data chosen for this example span a range of the independent variable between $x_A=0$ and $x_B=9$, with a gap between $x_a=3$ and $x_b=6$. All nine measurements have a value of $y_i=1$, with bin sizes of $\mathrm{\Delta }{x}_i=1$ for the first three data points, and $\mathrm{\Delta }{x}_i=\frac{1}{2}$ for the other six data points, as shown in Figure 9. The data have an acceptable solution for the standard linear model (in black) with a = 0.188, $\lambda =0.812$, for a best-fit statistic of $C_{\mathrm{c}min}=0.078$. Given the non-uniform bin sizes, the best-fit density function $f\left(x\right)$ (black continuous line, in units of counts-per-bin-size) differs from the best-fit model $y\left(x_i\right)$ (black step-wise curve, in units of counts or counts-per-bin). The constant model (yellow) has a best-fit parameter of ${\lambda }_C=1.5$, according to Equation (41) with M = 9, R = 9 and $R_G=3$, with $C_C=1.019$, the linear model pivoted at A has ${\lambda }_A=0.333$ and $C_A=2.735$, and the linear model pivoted at B has ${\lambda }_B=3$, and $C_B=14.177$.

Figure 9.

Best-fit linear models for data with non-uniform bins and with a gap in the data. The dot-dashed curve are the density functions and the solid step-wise curves are the models $y\left(x_i\right)$ for the integer data.

In summary, there are no significant additional complications for the analysis of data that contain a number of gaps or missing data. The following algorithm summarizes the changes required to analyze data with gaps.

Remark 6.2 Algorithm to implement changes in the analysis when gaps in the data are present —

This algorithm details the additions and modifications required for Algorithms 5.1 and 6.1 when there are data gaps present, following the same enumeration.

1. (Additional step) Calculate the location $x_{a,j}$, $x_{b,j}$ and range $R_{G,j}$ of each gap, the total gap length $R_G$, $R_m$ and $S_G$ according to Equation (35).

2. Hereafter replace R with $R_m$ in the definition of $F\left(a\right)$.

3. Use Equation (36) instead of (6) to calculate the value $\lambda \left(a\right)$ corresponding to an acceptable solution a.

4. For the calculation of the C statistics and best-fit parameters of the constant and pivoted models, use respectively Equation (39) (instead of Equations (23), (27) and (30)) and Equation (41) (instead of Equations (25), (29) and (32)).

6.4. A note on the distribution of the $C$ statistic

It is well known that, in the large-count limit, the $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ statistic – i.e. the $C$ statistic evaluated for the best-fit linear model – is expected to be distributed like a ${\chi }^2$ distribution with N−2 degrees of freedom, where N is the number of bins and 2 is the number of adjustable free parameters of the linear model (e.g. [3,8]). Moreover, properties of the $C$ statistic for a fixed model with no free parameters is also known accurately for any value of the parent Poisson mean [4,13]. What remains to be analyzed in further detail is the effect of free parameters on the distribution of $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$, in the low-count regime. The purpose of this paper is to present a method to evaluate the best-fit parameters of the linear model, precisely with the intent to further study the distribution of $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ via numerical simulations that rely on this method of analysis.

For a significant number of data sets, and especially for data with a small number of counts, the only non-negative linear model is one of the three extensions – all of them with just one adjustable parameter, instead of two of the traditional linear model. This requirement that the model be non-negative was introduced by the use of the Poisson distribution, and did not enter the discussion of Gaussian-distributed datasets that can be fit with the ${\chi }^2$ distribution. It is likely that such new requirement will result in differences between the distributions of ${\chi }_{min}^2$ and $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ for the linear model in the low-count regime, with implications for hypothesis testing and confidence intervals on the best-fit parameters. The distribution of the $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ for the linear model will be presented in a separate paper.

7. Discussion and conclusions

This paper has presented a new semi-analytical method to find the best-fit parameters of a linear model for the fit to integer-valued counting data, using the Poisson-based $C$ statistic. The method consists first of finding a solution for the non-linear equation $F\left(a\right)=0$, where a is one of the two parameters of the model. The other parameter λ is then calculated analytically via a simple analytical function $\lambda =\lambda \left(a\right)$. The two parameters a and λ must be such that the linear model is non-negative in each bin, in order to ensure the applicability of the Poisson distribution. The analysis presented in this paper shows that such requirement leads, in fact, to the uniqueness of the best-fit model, when such solution is available. This is clearly a very desirable property of the method, and a necessary condition for the use of this method to analyze Poisson-distributed data.

This paper has identified cases where low-count Poisson data do not have a suitable non-negative best-fit linear model according to the standard parameterization of Equation (2). For this reason, an extended linear model was proposed that guarantees a unique non-negative solution for any Poisson data set. This is accomplished by pivoting the linear model to either end of the range of the independent variable or by using a simple constant linear model, when the traditional linear model leads to an unsuitable solution. Thanks to simple analytical solutions for the best-fit parameter of these extensions, the use of the extended non-negative linear model remains straightforward.

The availability of a simple method to identify the best-fit parameters of a linear model for Poisson data of any number of counts makes it possible to further our understanding of the $C$ statistic. In particular, it is now possible to study the distribution of the $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ statistic for one of the most commonly used models with adjustable parameters, i.e. the linear model, especially in the low-count regime where its distribution is not known exactly.

Funding Statement

The author gratefully acknowledges the support of NASA Chandra [grant number AR6-17018X].

Disclosure statement

No potential conflict of interest was reported by the author(s).