Alpha spending for historical versus surveillance Poisson data with CMaxSPRT

Abstract

Sequential analysis hypothesis testing is now an important tool for postmarket drug and vaccine safety surveillance. When the number of adverse events accruing in time is assumed to follow a Poisson distribution, and if the baseline Poisson rate is assessed only with uncertainty, the conditional maximized sequential probability ratio test, CMaxSPRT, is a formal solution. CMaxSPRT is based on comparing monitored data with historical matched data, and it was primarily developed under a flat signaling threshold. This paper demonstrates that CMaxSPRT can be performed under nonflat thresholds too. We pose the discussion in the light of the alpha spending approach. In addition, we offer a rule of thumb for establishing the best shape of the signaling threshold in the sense of minimizing expected time to signal and expected sample size. An example involving surveillance for adverse events after influenza vaccination is used to illustrate the method.

1 ∣. INTRODUCTION

In Phase III clinical trials, sample sizes are limited due to human and/or financial costs, and depending on the indications of the new drug/vaccine, only a subgroup of individuals in the population, like pregnant or aged people, is enrolled in premarket statistical experiments. However, such subgroups are exposed to the effects of new drugs after commercialization. Therefore, there is still the need for monitoring the safety of postmarket drugs and vaccines.¹

In the context of postmarket drug and vaccine safety surveillance, the statistical sequential analysis consists, basically, on monitoring the number of adverse events, say, C_t, after application of the drug during a time window, say, (0, t]. According to Kulldorff et al,² the Poisson process, with rate

$${\lambda }_t=R{\mu }_t,$$ (1)

is an adequate model for C_t, where μ_t is a baseline risk of an adverse event in the risk window (0, t], and R is the relative risk due to the drug. If the drug is safe, then R is smaller than or equal to 1. Thus, the statistical hypotheses are of the form

$$\begin{gathered} H_0:R\le 1 \\ {H}_1:R>1. \end{gathered}$$ (2)

For applications where μ_t is known, one can use the maximized sequential probability ratio test (MaxSPRT). Introduced by Kulldorff et al,² MaxSPRT is a prospective rapid cycle vaccine safety surveillance method. It was derived as an extension of the classical sequential probability ratio test (SPRT) of Wald,³ but for composite hypotheses instead of precise as with SPRT. Moreover, MaxSPRT has been used by the Centers for Disease Control and Prevention sponsored Vaccine Safety Datalink for monitoring adverse events followed by postmarket drugs and vaccines.^1,4-8 However, if μ_t is unknown, and if there is a fixed number of observed events, say, c, from a historical data, then one can use the conditional MaxSPRT (CMaxSPRT), introduced by Li and Kulldorff.⁹

The CMaxSPRT test statistic, which for the kth event is denoted by U_k, is given by

$$U_k=I\left(\frac{k}{c}>\frac{P_k}{V}\right)\left[c\log \frac{c(1+P_k∕V)}{c+k}+k\log \frac{k(1+P_k∕V)}{(P_k∕V)(c+k)}\right],$$ (3)

where k denotes the number of adverse events observed during the surveillance period. The random variable P_k is the person-time accumulated up to the arrival of the kth adverse event, and the random variable Vis the person-time of the historical data, that is, V results from the cutoff date that defines the last moment of collecting the historical sample.

Let K denote a predefined maximum length of surveillance given in the scale of the number of adverse events during the surveillance period. With CMaxSPRT, H₀ is rejected if U_k ≥ CV for some k = 1, …, K, where CV is a flat signaling threshold or critical value (CV). If U_k < CV for each k = 1, …, K, then the surveillance is interrupted in favor of H₀. For a fixed overall α level, the exact calculation of CV can be handled through iterative numerical procedures. This is discussed in details in Section 2. Likewise, if one intends to apply nonflat signaling threshold, that is, a time variable threshold CV_k, that can be executed through exact calculations too. In this direction, a comprehensive approach is to use alpha spending functions. The alpha spending function, here denoted by S(k), is a function of the sequential time index, here represented by k, that dictates in advance the cumulative rate at which the type I error probability should be spent at each single test. A broadly used shape is the so called power-type function

$$S(k)=a\times {\left(\frac{k}{K}\right)}^{\rho },\rho >0,k=1,2,\dots ,K.$$ (4)

Historically, authors dealing with the best shape choices for S(k) poses the discussion under clinical trials perspectives. For example, Jennison and Turnbull¹⁰ describe the history and principal results regarding the usage of alpha spending functions in clinical trials. Because in clinical trials, the costs of large sample sizes are critical, the main proposals appearing in the literature suggest usage of a convex S(k) since it promotes reduction on the expected time of surveillance, defined as the average of the sample size at which the surveillance is interrupted, which can occur either against or in favor of H₀. However, in postmarket safety surveillance of adverse events, sample sizes are large.¹¹ Hence, if H₀ is not about to be rejected, minimization of sample size is not a point. Instead, the important statistical performance measure is the expected number of events when the null hypothesis is rejected, called “expected time to signal”.

Each signaling threshold shape, given in the scale of a test statistic, dictates a specific alpha spending function. Each alpha spending promotes a certain performance in terms of time to signal, sample size, and statistical power. In particular, the alpha spending related to the flat CV approach is better than some alpha spending choices, but worse than others. Therefore, a question that emerges is if one should deliberately use flat thresholds, or if the nonflat approach, promoted by specific alpha spending shapes, could improve target statistical performance measures, such as optimization of expected sample size, or of expected time to signal, under fixed statistical power, target relative risks, with fixed significance levels.

Studies directed to identify suitable alpha spending shapes for postmarket safety surveillance are just emerging. Seminal works in this direction are the developments of Silva and Kulldorff,^11-13 where the minimization of expected time to signal was taken as the key design criterion. Their findings suggest that, unlike clinical trials, the alpha spending should have a concave shape for postmarket safety surveillance applications for both Poisson and binomial data. However, this has not been explored in the context of historical versus surveillance data yet. Aiming to cover this lack, in this paper, we show the values of ρ of expression (4) that lead to near-minimal expected time to signal when the analysis counts with historical versus surveillance Poisson data.

The numerical results shown in this paper were executed with the R Sequential package. Sequential was developed for continuous, group, and mixed continuous-group sequential analysis. The package is directed to analyze Poisson and binomial data. In order to reproduce the numerical results reported in this paper, one can use the functions Threshold.CondPoisson and SampleSize.CondPoisson of R Sequential. For those unfamiliar with R language, we suggest to visit the web interface http://www.sequential.org of R Sequential. There, all that users have to do is to inform input data and tuning parameters. The package will give directions for further analysis, statistical information for future analyses, and assistency regarding final conclusions when applicable.

This paper is organized in the following way: using the alpha spending approach, the next section derives the exact calculation of time-variable signaling thresholds for CMaxSPRT. Section 3 explains why the conventional flat threshold approach is simply a particular way to design, indirectly, an alpha spending function. Section 4 presents the results of a comparison study among seven different alpha spending shapes within the power-type class. The results enabled to establish a rule of thumb: convex alpha spending shapes are more indicated for CMaxSPRT than concave shapes, where ρ = 1.5 works in line with the best results for most of the evaluated scenarios. This rule is applicable when expected time to signal and expected sample size, for fixed power, are the meaningful performance measures. Section 5 illustrates the effects of using the alpha spending approach for CMaxSPRT applied to a real data of adverse events following an influenza vaccination. Section 6 offers a brief discussion on the main results.

2 ∣. VARIABLE THRESHOLD CALCULATION GIVEN ALPHA SPENDING

Li and Kulldorff⁹ proposed the usage of Monte Carlo testing to perform CMaxSPRT. However, as demonstrated later by Silva et al,¹⁴ Monte Carlo is not necessary. The exact calculation of flat CVs is feasible, in general, through an iterative numerical procedure. In the same direction, this section shows that calculation of time-variable (nonflat) signaling thresholds can be solved under analytical arguments too. Furthermore, we show that nonflat thresholds can be obtained in such away to attend arbitrary, ie, user defined, alpha spending functions. For this, it will be sufficient to derive the analytical expression for the probability of rejecting H₀, that is, the statistical power function of a given sequential design based on the CMaxSPRT statistic, and this is the subject of the next section.

2.1 ∣. Power function

It is evident from (3) that the only random part of the CMaxSPRT statistic is P_k/V. This way, denote it by τ_k

$${\tau }_k=P_k∕V.$$ (5)

Thus, let U(τ_k) denote CMaxSPRT at the kth test. This notation for CMaxSPRT has already been adopted by Silva et al,¹⁴ and it highlights the dependence of U_k upon τ_k. As demonstrated in the Appendix Part I, U(τ_k) is downward monotone with τ_k ∈ (0, k/c), that is, there always exists t_k ∈ (0, k/c) such that

$$t_k=\text{sup}\{t\in {\Re }^{+}:U(t)\ge C{V}_k\}.$$ (6)

The constant t_k can be interpreted as a signaling threshold in the scale of τ_k, where the null hypothesis is rejected for the first k such that τ_k ≤ t_k. Moreover, when convenient, one can plan a sequential analysis design directly in terms of selected constants t₁, …, t_k. Under such setting, the probability of rejecting H₀ with exactly k adverse events, denoted by π_k(R, t_k), is given by

$${\pi }_k(R,t_k)=Pr[\text{rej.}{H}_0\text{at the}k\text{th event}\mid R]=Pr[{\tau }_k\le t_k\mid R,{\tau }_{k^{\ast }}>t_{k^{\ast }}\text{for each}{k}^{\ast }<k].$$ (7)

Note that the event {τ_k ≤ t_k} is equivalent to {P_k ≤ Vt_k}. As P_k follows a Gamma distribution, there exists an intrinsic Poisson process, say, C_k, with parameter RVt_k. With this, for k = 2, …, K, the probability in (7) can be rewritten as

$${\pi }_k(R,t_k)={\int }_0^{\infty }Pr[C_k\ge k\mid R,V=v,C_{k^{\ast }}<k^{\ast }\text{for each}{k}^{\ast }<k]f_V(v)dv,$$ (8)

where f_V(v) represents the density of V.

For k = 1, H₀ is rejected with probability

$${\int }_0^{\infty }Pr[C_1\ge 1\mid R,V=v]f_V(v)dv.$$ (9)

As emphasized by Li and Kulldorff,⁹ it is appropriate to assume that the cutoff date for the historical data is defined irrespectively to the number of observed adverse events. Hence, for c ≥ 1,

$$V=\sum _{i=1}^c{Y}_i+Z{Y}_{c+1},$$ (10)

where Y_i : i = 1, …, (c + 1) are i.i.d. with probability density

$$f_Y(y)={\lambda }_V{e}^{-{\lambda }_{V}y},$$ (11)

and Y_i are also independent of Z. The analytical derivation of π_k(R, t_k) is completed by noting that Z follows a uniform distribution in the (0, 1) interval, and this fact shall be proved later, in the Appendix Part II.

Define

$$W_c=\sum _{i=1}^c{Y}_i,\text{and}{W}^{\ast }=W_c∕{\lambda }_V.$$ (12)

Therefore, from (11), W* follows a Gamma distribution with rate 1 and shape c. Define also

$$Y^{\ast }=Y_{c+1}∕{\lambda }_V,$$ (13)

where

$$f_{Y^{\ast }}(y)=e^{-y}I(y>0).$$ (14)

For fixed W* = w, Z = z, and Y* = y, we have C_k ~ Pois(Rvt_k), where v = (w + zy). For fixed W* = w, Z = z, and Y* = y, the conditional probability of rejecting H₀ at the kth event is given by

$${\pi }_k(R,t_k,w,z,y)=\sum _{c_2=0}^1\sum _{c_3=c_2}^2\cdots \sum _{c_{k-1}=c_{k-2}}^{k-2}\left[a(c_{k-1})\prod _{j=1}^{k-1}\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}-\sum _{c_k=c_{k-1}}^{k-1}a(c_k)\prod _{j=1}^k\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}\right],$$ (15)

where

$$a(c_j)=e^{-Rv{t}_j}(Rv{)}^{c_j}=e^{-R(w+zy)t_j}[R(w+zy){]}^{c_j},$$

$c_j^{\ast }=c_j-c_{j-1}$, μ_j = (t_j − t_j−1) and c₀ = t₀ = 0.

After integrating with respect to w, z, and y, the power function for the kth event is obtained as follows:

$${\pi }_k(R,t_k)=\sum _{c_2=0}^1\sum _{c_3=c_2}^2\cdots \sum _{c_{k-1}=c_{k-2}}^{k-2}\left[\prod _{j=1}^{k-1}\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}\times A(c_{k-1})-\sum _{c_k=c_{k-1}}^{k-1}\prod _{j=1}^k\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}\times A(c_k)\right],$$ (16)

where

$$A(c_k)=R^{c_k}\sum _{l=0}^{c_k}{D}_l^{c_k}\frac{(l+c-1)!}{(R{t}_k+1{)}^{l+c}(c-1)!}(c_k-l)!\underset{B(c_k,l)}{\underbrace{{\int }_0^1\frac{z^{c_k-l}}{(Rz{t}_k+1{)}^{c_k-l+1}}dz,}}$$ (17)

and $D_l^{c_k}=c_k!∕[l!(c_k-l)!]$.

For c_k = l,

$$B(c_k,l)=[\log (R{t}_k+1)]∕(R{t}_k).$$ (18)

Otherwise, define G = Rzt_k to obtain

$$B(c_k,l)=(R{t}_k{)}^{l-c_k-1}\sum _{g=0}^{c_k-1}{D}_g^{c_k-l}(-1{)}^{c_k-l-g}\underset{E(g,c_k,l)}{\underbrace{{\int }_1^{R{t}_k+1}\frac{u^g}{u^{c_k-l+1}}du,}}$$ (19)

where

$$E(g,c_k,l)=\{\begin{array}{c}\log (R{t}_k+1),\text{for}g=c_k-l,\hfill \\ \frac{(R{t}_k+1{)}^{g-c_k+l}-1}{g-c_k+l},\text{for}g>c_k-l.\hfill \end{array}\phantom{\}}$$ (20)

Hence,

$$B(c_k,l)=(R{t}_k{)}^{l-c_k-1}\left\{\log (R{t}_k+1)+\sum _{g=0}^{c_k-l-1}\frac{(-1{)}^{c_k-l-g}{D}_g^{c_k-l}}{g-c_k+l}[(R{t}_k+1{)}^{g-c_k+l}-1]\right\}.$$ (21)

The derivations of expressions (15) to (21) are shown in details in the Appendix Part III.

Now, we have everything established for exact calculation of the overall power, which is the sum of π_k(R, t_k) in k = 1, …, K

$$\pi (R)=\sum _{k=1}^K{\pi }_k(R,t_k).$$ (22)

Naturally, if evaluated at R = 1, expression (22) returns the overall type I error probability.

2.2 ∣. Computation

For an arbitrary alpha spending function, S(k), consider to find t₁, …, t_K such that

$$\mid \sum _{l=1}^k{\pi }_l(1,t_l)-S(k)\mid \le ϵ,$$ (23)

for each k ∈ {1, …, K}, where ϵ > 0 is an arbitrary desired precision. For this, we apply the bisection method.¹⁵ The solutions for t_k subject to (23) are run iteratively, from test 1 to test K, each at a time, as follows:

• Step (i) - set k : = 1, θ₁ : = 0, and θ₂ : = 1;

• Step (ii) - if k > K, stop running these steps since it means that all solutions are done. Otherwise, set θ_m : = (θ₁ + θ₂)/2;

• Step (iii) - using expression (A7), calculate

$${\pi }_k^{\ast }=\sum _{l=1}^k{\pi }_l(1,{\theta }_m);$$ (24)

• Step (iv) - if $\mid {\pi }_k^{\ast }-S(k)\mid \le ϵ$, set t_k : = θ_m, θ₂ : = k, k : = k + 1, Θ₁ : = θ_m, and go to Step (ii). Otherwise, go to Step (v) as follows;

• Step (v) - if ${\pi }_k^{\ast }>S(k)$, set θ₂ : = θ_m, otherwise, set θ₁ : = θ_m. Go to Step (ii).

3 ∣. FLAT THRESHOLDS ARE SPECIFIC ALPHA SPENDING CHOICES

Flat signaling thresholds imply specific alpha spending shapes, which can be mimicked by simple functions like those in the power-type family. For instance, Figure 1 shows the alpha spending implied by CMaxSPRT for α = 0.01, K = 10, 30, and c = 100. Note that the alpha spending of the flat approach has a concave shape. The scenario with K = 10 is similar to the power-type spending for ρ = 0.5, and the case with K = 30 is approximated by ρ = 0.25. The same is observed with α = 0.05, as shown in Figure 2. These graphs also present the convex power-type alpha spending obtained with ρ = 1.5.

FIGURE 1.

Alpha spending shapes of CMaxSPRT and power-type functions (ρ = 0.5,1.5) for α = 0.01, K = 10, 30, and c = 100. CMaxSPRT, conditional maximized sequential probability ratio test; CV, critical value

FIGURE 2.

Alpha spending shapes of CMaxSPRT and power-type functions (ρ = 0.5, 1.5) for α = 0.05, K = 10, 30, and c = 100. CMaxSPRT, conditional maximized sequential probability ratio test; CV, critical value

The concavity of the alpha spending from the flat CMaxSPRT is not eventual. Actually, flat CMaxSPRT CVs presented concave alpha spending shapes for all scenarios of statistical power, R, c, K, and α explored in this paper. Section 4.1 describes such scenarios in details.

Note that the statistical performance of some alpha spending shapes can be better than that of the flat threshold approach. For a fair comparison in terms of some specific design criterion, such as minimization of expected time to signal, one should ensure that other meaningful measures, such as statistical power, are kept equal for each of the compared designs. For example, suppose that we want a power of at least 0.9 for R = 2. Under α = 0.05, with c = 100, that power is satisfied for a sample size of at least K = 42. The associated flat CMaxSPRT CV in this case is CV = 3.618087. This design leads to an expected time to signal of 16.51 and expected sample size of 18.93. Now, instead of a flat threshold, consider to use the power-type alpha spending approach with ρ = 1.5. In this case, the minimum sample size needed to reach a power of 0.9 is K = 23. Now, the expected time to signal is 13.35, and the expected sample size is 14.32, both better than the flat threshold approach under the same power and level. Naturally, as the performance of flat threshold designs are related to specific alpha spending shapes (specific ρ values in the power-type family), comparisons like the above can be extended to other tuning parameters scenarios by comparing alpha spending functions among selected ρ values.

4 ∣. ALPHA SPENDING CHOICE: SHOULD WE USE CONCAVE OR CONVEX ALPHA SPENDING SHAPES?

In this section, we compare seven alpha spending shapes in terms of expected time to signal. Let T denote the time until the surveillance is stopped. The expected time to signal is the conditional expectation of T given that H₀ is rejected, given by

$$E[T\mid H_0\text{rejected}]=\frac{{\sum }_{l=1}^{K}l\times {\pi }_l(R,t_l)}{\pi (R)}.$$ (25)

As emphasized earlier, in postmarket drug/vaccine safety surveillance of adverse events, the critical performance measure is the expected time to signal, which should be minimized in order to signalize, as soon as possible, when a drug/vaccine offers elevated risks to the exposed population.

The goal is to find the shape that offers the smallest E[T∣H₀ rejected] for fixed power. The compared shapes are obtained by setting seven different ρ choices in the alpha spending of the power-type, expression (4).

In addition, we use this opportunity to compare these seven alpha spending choices in terms of expected length of surveillance too. The expected length of surveillance, also called expected sample size, is defined by

$$E[T]=\sum _{l=1}^{K}l\times {\pi }_l(R,t_l)+K\times [1-\pi (R)].$$ (26)

4.1 ∣. Tuning parameters settings

The selected ρ scenarios cover the meaningful alpha spending shapes broadly studied in the literature among concave (ρ < 1), linear (ρ = 1), and convex (ρ > 1). More precisely, ρ = 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2.

For the other tuning parameters, we considered the same values as in the work of Li and Kulldorff⁹: c = 10, 30, 50, 100, for fixed powers of 0.2, 0.3, … ,0.9, α = 0.01, 0.05, and R = 1.1, 1.2, …, 2. Combining these parameters options with the seven alpha spending choices (ρ options), we contemplate a total of 640 scenarios.

4.2 ∣. Comparison study results

By evaluating the expected time to signal as a function of power, Figure 3 presents a sample of the results under R = 2 and α = 0.05. Note that for both, small (c = 10) and large (c = 100) historical number of events, the ρ choices promoting smallest expected time to signal are those greater than one. Regarding expected sample size, Figure 4 shows a similar result, but, in this case, the superiority of ρ = 1.5 and ρ = 2 is more expressive. Actually, for all scenarios of R, α, and c considered (results of all scenarios are not shown here), the superiority of ρ values greater than one is evident.

FIGURE 3.

Power versus expected time to signal following the power-type alpha spending for seven ρ values among 25, 0.5, 0.75, 1, 1.25, 1.5, 2. Calculations were based on a relative risk of R = 2, number of historical events of c = 10 and c = 100, and overall significance level of α = 0.05. A, Exp. time to signal, c = 10, R = 2; B, Exp. time to signal, c = 100, R = 2

FIGURE 4.

Power versus expected sample size following the power-type alpha spending for seven ρ values among 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2. Calculations were based on relative risk of R = 2, number of historical events of c = 10 and c = 100, and overall significance level of α = 0.05. A, Expected sample size, c = 10, R = 2; B, Expected sample size, c = 100, R = 2

5 ∣. MONITORING ADVERSE EVENTS AFTER INFLUENZA VACCINATION

The illustrative example uses time-series data on concomitant vaccination with inactivated influenza vaccine (IIV) and 13-valent pneumococcal conjugate vaccine (PCV-13) and subsequent seizure. The data come from three large US health insurance or data companies (“Data Partners”) participating in the US Food and Drug Administration–sponsored Sentinel system.¹⁶ Claims for seizure were not confirmed by medical record review; however, the case-finding algorithm had a positive predictive value of 70% for febrile seizures in another Sentinel study of children 6 to 59 months of age.¹⁷ For each point in the irregularly spaced 10-point time series, the data consist of (i) the cumulative number of doses of IIV received concomitantly with PCV-13 by 6- to 23-month-old children during September 2013-April 2014; (ii) the cumulative number of cases of seizure occurring in inpatient or emergency department settings (and not preceded by seizure in any medical setting in the previous 6 months) during Days 0 and 1 after IIV-with-PCV-13 vaccination among the same children; and (iii) the expected number of seizure cases, based on the Data Partner-specific rates after IIV vaccination in historical influenza seasons and adjusted for data lag.

The original study¹⁶ stratified seizures in the 6- to 23-month-old age group by the presence/absence of concomitant PCV-13 because of the suggestion of an increased risk of febrile seizures after PCV-13 vaccination in the 2010–2011 influenza season¹⁷; this was seen more clearly in the 2013-2014 and 2014-2015 seasons.¹⁸ The unadjusted historical rate in this age group was based on 37 cases in Days 0 and 1 after 752 949 doses of IIV prior to licensure of PCV-13 [unpublished data]. In the original research, the Poisson MaxSPRT was performed, and a signal emerged in the seventh of the 10 sequential analyses.¹⁶

5.1 ∣. Tuning parameters settings

For this application, we used α = 0.05 and ρ = 1.5. Because the number of adverse events in the historical period was c = 37, for a historical person-time information of v = 752 949 (doses), from expression (A7) in the Appendix, the minimum sample size required for a power of at least 0.9 under target R = 2 is K = 23. For the nonflat threshold, it was adopted K = 40 since it also promotes a power of 0.9.

5.2 ∣. Data analysis results

The overall relative risk estimate for this hypothetical data is

$$\widehat{R}=\frac{k\times v}{c\times p_k}=\frac{12\times 752,949}{37\times 232,266}=2.1,$$

which was calculated with k = 12 adverse events observed after a total amount of p_k = 232, 266 person-day accrued in the surveillance period. This estimate suggests an elevation on the relative risk in the surveillance period. However, a formal sequential test should be applied in order to take into account the sample variability in drawing a definitive decision in favor or against H₀.

Table 1 contains a summary of the main information of the sequential analysis, where the cumulative number of events, the ratio P_k/V, the relative risk estimate $(\widehat{R})$, and the observed CMaxSPRT statistic are shown in the first four columns, in this order. The variable threshold based on the power-type alpha spending (ρ = 1.5) and the flat CV are shown in the last two columns. Figure 5 gives a visual representation of these numbers. The observed CMaxSPRT statistic (black dotted line) is shown per test. Figure 5 also shows the two different thresholds options, flat (black solid line) and the one elicited from the power-type alpha spending with ρ = 1.5 (thinnest line). These different thresholds lead to different conclusions. With the flat CV, one would not reject H₀ in the first 10 tests. However, with the nonflat, the decision would be against H₀ at the ninth test. One can also check this by comparing columns four and five, ninth line, of Table 1. The symbol “na” in Table 1 means “not applicable.”

TABLE 1.

Summary of data analysis results for seizure following influenza vaccination in 6- to 23-month-old children during September 2013-April 2014

	Cum. Events (k)	Pk/V	$\widehat{R}$	CMaxSPRT	CVk(ρ = 1.5)	Flat CV
Test 1	0	0.00515	na	0	na	na
Test 2	0	0.00941	na	0	na	na
Test 3	0	0.00942	na	0	na	na
Test 4	1	0.04391	0.62	0	6.686401	3.308065
Test 5	1	0.04392	0.62	0	6.686401	3.308065
Test 6	6	0.09009	1.80	0.7738249	4.079384	3.308065
Test 7	9	0.11465	2.12	1.7470141	3.319670	3.308065
Test 8	9	0.11488	2.12	1.7384775	3.319670	3.308065
Test 9	12	0.13822	2.35	2.8142293	2.760377	3.308065
Test 10	12	0.15424	2.10	2.1833442	2.760377	3.308065

Abbreviation: CMaxSPRT, conditional maximized sequential probability ratio test; CV, critical value.

FIGURE 5.

Observed CMaxSPRT after influenza vaccination, test by test. Flat and nonflat signaling thresholds were obtained under α = 0.05, power of 0.9 and R = 2. For the nonflat case, we used a power-type alpha spending with ρ = 1.5. CMaxSPRT, conditional maximized sequential probability ratio test

In the study of Baker et al,¹⁸ he null hypothesis was rejected in the seventh test. However, that application was not designed for an overall power of 0.9 for R = 2. Thus, a smaller flat CV of 2.874 was used in the scale of MaxSPRT instead of CMaxSPRT, which explains why a signal was detected sooner there. However, even with the larger flat CV of 3.308065 used here, H₀ would possibly be rejected if the surveillance was kept a little longer with additional tests. This evidences the importance of using a variable threshold. Although a target power has been imposed, which implied a lower bound on the sample size, the nonflat was able to detect a signal faster than the traditional flat CV.

6 ∣. CONCLUDING REMARKS

The convex alpha spending function of the power type is well established for clinical trials under ρ = 1.5 and ρ = 2. According to several authors, it works similarly to Pocock’s and O’Brien and Fleming’s tests, which are indicated when minimization of expected sample size is of key importance.^19-21 However, more recently, some results have shown that if the design criterion is expected time to signal, then a concave alpha spending function should be used instead. This is advocated, for example, in an earlier paper by Silva.¹² That paper shows that usage of ρ = 0.5 favors near-optimal expected time to signal when data follow a binomial distribution. Similarly, it has been shown that choosing ρ in the range (0.5, 1) reduces time to signal when data is Poisson too.¹³ However, it seems that this logic fails when historical Poisson data is compared with surveillance data. The results of the present paper demonstrate that, for both performance measures, expected time to signal (for postmarket surveillance), and expected sample size (for clinical trials), a convex alpha spending function is recommended, where ρ = 1.5 can be used as a rule of thumb since it might work properly in most of the applications.

APPENDIX

PART I - MONOTONICITY OF U(τ_k)

We can rewrite

$$U({\tau }_k)\propto (c+k)\log (1+{\tau }_k)-k\log {\tau }_k.$$ (A1)

Hence, the first derivative of U(τ_k) with respect to τ_k is

$$U^{\prime }({\tau }_k)=(c+k)∕(1+{\tau }_k)-k∕{\tau }_k,$$ (A2)

which is negative for τ_k < k/c.

As settled through an indicator function in expression (3), page 2, CMaxSPRT is greater than zero only for τ_k < k/c. Hence, the assertion that U(τ_k) decreases with τ_k ∈ (0, k/c) is proved.

PART II - DISTRIBUTION OF Z

In expression (10), the historical person-time, V, depends on terms involving independent exponential variables and variable Z. This section shows that Z follows a uniform distribution in the (0,1) interval, which is a consequence of the Poisson process. This fact has already been proved by Silva et al,¹⁴ but it is placed here too for the sake of clarity.

Let X_t denote a Poisson process with parameter λ_Vt, which represents the number of events in the historical data, $\forall t\in \Re$. Now, let L denote the person-time of the last event that arrives before time V, and N to denote the person-time of the subsequent event that would be observed after V if the analysis continued. Define also D_L = V − L and D_N = N − V. Because X_t is a homogenous Poisson process, D_L and D_N have the same exponential distribution with parameter λ_V:

$$Pr[D_L\le d]=Pr[D_N\le d]=Pr[X_d\ge 1]=1-Pr[X_d=0]=1-e^{-{\lambda }_{V}d}.$$ (A3)

Because X_t is a Poisson process, D_L and D_N are independent random variables.

Define the ratio

$$Z=\frac{D_L}{D_L+D_N}.$$ (A4)

Note that the sum Y = D_L + D_N is actually the difference L − N. Therefore, because X_t is Poisson, it holds that Y follows an exponential distribution with parameter 1/λ. Consequently, the extra person-time D_L = V − L is equal to ZY, where Y ~ Exp(λ_V).

Now, for z ∈ (0, 1), we have

$$\begin{gathered} Pr[Z\le z]=Pr[D_L\le z(D_L+D_N)] \\ =Pr[D_L(1-z)\le z{D}_N] \\ =Pr\left[D_L\le \frac{z{D}_N}{(1-z)}\right] \\ [\text{because}{D}_N\sim Exp({\lambda }_V)]={\int }_0^{\infty }Pr\left[D_L\le \frac{za}{1-z}\right]{\lambda }_V{e}^{-{\lambda }_{V}a}da \\ [\text{because}{D}_L\sim Exp({\lambda }_V)]={\int }_0^{\infty }[1-e^{-{\lambda }_{V}za∕(1-z)}]{\lambda }_V{e}^{-{\lambda }_{V}a}da \\ =1-{\int }_0^{\infty }{\lambda }_V{e}^{-{\lambda }_V\left[1+\frac{z}{1-z}\right]a}da \\ =1-\frac{1}{1+\frac{z}{1-z}} \\ =z. \end{gathered}$$ (A5)

In conclusion: Z ~ U(0, 1).

PART III - POWER FUNCTION OF FLEXIBLE SIGNALING THRESHOLD SHAPE OF CMaxSPRT

For fixed W* = w, Z = z, and Y* = y, we have C_k ~ Pois(Rvt_k), where v = (w + zy). For fixed W* = w, Z = z, and Y* = y, the conditional probability of rejecting H₀ at the kth event is given by

$$\begin{gathered} {\pi }_k(R,t_k,w,z,y)=Pr\left[{\cap }_{j=l}^{k-1}\{C_j<j\},C_k\ge k\mid V=v\right] \\ =\sum _{c_2=0}^1\sum _{c_3=c_2}^2\cdots \sum _{c_{k-1}=c_{k-2}}^{k-2}Pr[C_k\ge k\mid C_{k-1}=c_{k-1},V=v]\times \\ \times \prod _{j=1}^{k-1}Pr[C_j=c_j\mid C_{j-1}=c_{j-1},V=v] \\ =\sum _{c_2=0}^1\sum _{c_3=c_2}^2\cdots \sum _{c_{k-1}=c_{k-2}}^{k-2}\left[1-\sum _{c_k=c_{k-1}}^{k-1}Pr[C_k=c_k\mid C_{k-1}=c_{k-1},V=v]\right]\times \\ \times \prod _{j=1}^{k-1}Pr[C_j=c_j\mid C_{j-1}=c_{j-1},V=v] \\ =\sum _{c_2=0}^1\sum _{c_3=c_2}^2\cdots \sum _{c_{k-1}=c_{k-2}}^{k-2}\left[a(c_{k-1})\prod _{j=1}^{k-1}\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}-\sum _{c_k=c_{k-1}}^{k-1}a(c_k)\prod _{j=1}^k\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}\right], \end{gathered}$$ (A6)

where

$$a(c_j)=e^{-Rv{t}_j}(Rv{)}^{c_j}=e^{-R(w+zy{)}^{t_j}}[R(w+zy){]}^{c_j},$$

$c_j^{\ast }=c_j-c_{j-1}$, μ_j = (t_j − t_j−1), and c₀ = t₀ = 0.

After integrating with respect to w, z, and y, the power function for the kth event is obtained as follows:

$$\begin{gathered} {\pi }_k(R,t_k)=Pr[C_k\ge k\mid R,C_{k^{\ast }}<k^{\ast }:k^{\ast }<k] \\ ={\int }_0^1{\int }_0^{\infty }{\int }_0^{\infty }{\pi }_k(R,CV,w,z,y)f_{w^{\ast }}(w)f_{Y^{\ast }}(y)f_Z(z)dwdydz \\ ={\int }_0^1{\int }_0^{\infty }{\int }_0^{\infty }{\pi }_k(R,CV,w,z,y)\frac{e^{-w}{w}^{c-1}}{(c-1)!}{e}^{-y}dwdydz \\ =\sum _{c_2=0}^1\sum _{c_3=c_2}^2\cdots \sum _{c_{k-1}=c_{k-2}}^{k-2}\left[\prod _{j=1}^{k-1}\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}\times A(c_k-1)-\sum _{c_k=c_{k-1}}^{k-1}\prod _{j=1}^k\frac{{\mu }_j^{c_j^{\ast }}}{c_j^{\ast }!}\times A(c_k)\right], \end{gathered}$$ (A7)

where

$$\begin{gathered} A(c_k)={\int }_0^1{\int }_0^{\infty }{\int }_0^{\infty }{e}^{-R(w+zy)t_k}[R(w+zy){]}^{c_k}\frac{e^{-w}{w}^{c-1}}{(c-1)!}{e}^{-y}dwdydz \\ ={\int }_0^1{\int }_0^{\infty }{e}^{-y(Rz{t}_k+1)}\left\{{\int }_0^{\infty }{e}^{-Rw{t}_k}[R(w+zy){]}^{c_k}\frac{e^{-w}{w}^{c-1}}{(c-1)!}dwdy\right\}dz \\ =R^{c_k}{\int }_0^1{\int }_0^{\infty }{e}^{-y(Rz{t}_k+1)}\left\{{\int }_0^{\infty }{e}^{-Rw{t}_k}\left[\sum _{l=0}^{c_k}{D}_l^{c_k}{w}^l(zy{)}^{c_k-l}\right]\frac{e^{-w}{w}^{c-1}}{(c-1)!}dwdy\right\}dz \\ =R^{c_k}{\int }_0^1{\int }_0^{\infty }{e}^{-y(Rz{t}_k+1)}\left\{\sum _{l=0}^{c_k}{D}_l^{c_k}(zy{)}^{c_k-l}{\int }_0^{\infty }{e}^{-Rw{t}_k}{w}^l\frac{e^{-w}{w}^{c-1}}{(c-1)!}dwdy\right\}dz \\ =R^{c_k}\sum _{l=0}^{c_k}{D}_l^{c_k}\frac{(l+c-1)!}{(R{t}_k+1{)}^{l+c}(c-1)!}{\int }_0^1{z}^{c_k-l}{\int }_0^{\infty }{e}^{-y(Rz{t}_k+1)}{y}^{c_k-l}dydz \\ =R^{c_k}\sum _{l=0}^{c_k}{D}_l^{c_k}\frac{(l+c-1)!}{(R{t}_k+1{)}^{l+c}(c-1)!}(c_k-l)!\underset{B(c_k,l)}{\underbrace{{\int }_0^1\frac{z^{c_k-l}}{(Rz{t}_k+1{)}^{c_k-l+1}}dz,}} \end{gathered}$$ (A8)

and $D_l^{c_k}=c_k!∕[l!(c_k-l)!]$.

For c_k = l,

$$B(c_k,l)=[\log (R{t}_k+1)]∕(R{t}_k).$$ (A9)

Otherwise, define G = Rzt_k, hence

$$\begin{gathered} B(c_k,l)={\int }_0^{R{t}_k}(R{t}_k{)}^{l-c_k-1}\frac{G{}^{c_k-l}}{(G+1{)}^{c_k-l+1}}dG \\ =(R{t}_k{)}^{l-c_k-1}{\int }_1^{R{t}_k+1}\frac{(u-1{)}^{c_k-l}}{u^{c_k-l+1}}du \\ =(R{t}_k{)}^{l-c_k-1}{\int }_1^{R{t}_k+1}\frac{{\sum }_{g=0}^{c_k-l}{D}_g^{c_k-l}{u}^g(-1{)}^{c_k-l-g}}{u^{c_k-l+1}}du \\ =(R{t}_k{)}^{l-c_k-1}\sum _{g=0}^{c_k-l}{D}_g^{c_k-l}(-1{)}^{c_k-l-g}\underset{E(g,c_k,l)}{\underbrace{{\int }_1^{R{t}_k+1}\frac{u^g}{u^{c_k-l+1}}du,}} \end{gathered}$$ (A10)

where

$$E(g,c_k,l)=\{\begin{array}{c}\log (R{t}_k+1),\text{for}g=c_k-l,\hfill \\ \frac{(R{t}_k+1{)}^{g-c_k+l}-1}{g-c_k+l},\text{for}g<c_k-l.\hfill \end{array}\phantom{\}}$$ (A11)

Hence,

$$B(c_k,l)=(R{t}_k{)}^{l-c_k-1}\left\{\log (R{t}_k+1)+\sum _{g=0}^{c_k-l-1}\frac{(-1{)}^{c_k-l-g}{D}_g^{c_k-l}}{g-c_k+l}[(R{t}_k+1{)}^{g-c_k+l}-1]\right\}.$$ (A12)