A general area theorem for the same–different paradigm

Abstract

According to a well-known theorem in psychophysics (Green & Swets, 1966), the area under the receiver operating characteristic (ROC) for the yes–no paradigm equals the proportion of correct responses of an unbiased observer in the two-interval, two-alternative, forced choice paradigm (2I2AFC). Here, we demonstrate a similar relationship between the ROC area in the two-interval same–different (AX or 2IAX) paradigm, and the proportion correct in the four-interval same–different (4IAX, also known as dual-pair comparison) paradigm. The theorem demonstrated here is general, in the sense that it does not require that the sensory observations have a specific distribution (e.g., Gaussian), or that they be statistically independent.

According to an often-cited theorem by Green (1964; Green & Swets, 1966), the area under the receiver operating characteristic (ROC) in the psychophysical yes–no (YN) paradigm is equal to the proportion of correct responses in the two-interval two-alternative forced choice (2I2AFC) paradigm. This “area theorem” has both theoretical and practical value. It establishes a link between two psychophysical paradigms, and it allows experimenters to convert hit and false alarm rates measured using the YN paradigm into an equivalent proportion of correct responses in the less bias prone 2I2AFC paradigm.

In an insightful text on the decision rules for various psychophysical paradigms, Noreen (1981) proposed an extension of Green’s original area theorem to another popular psychophysical paradigm: the two-interval same–different (also known as AX or, more precisely, 2IAX) paradigm. Noreen predicted that the area under the ROC in that paradigm should equal the proportion of correct responses in the four-interval same–different (4IAX) paradigm, PC_4IAX. In the latter paradigm, also known as the dual-pair paradigm (Creelman & Macmillan, 1979; Macmillan, Kaplan, & Creelman, 1977; Rousseau & Ennis, 2001), the observer is presented on each trial with two pairs of stimuli. One pair contains two stimuli from the same distribution (the same stimuli), while the other pair contains stimuli from different distributions (the different stimuli). The task is to indicate which of the two pairs contains the different stimuli. Noreen remarked that each stimulus within a pair in the 2IAX paradigm could be likened to the single stimulus presented on a trial of the YN paradigm, and that each pair of stimuli in the 4IAX paradigm could be likened to one of the two stimuli presented on a trial of the 2I2AFC paradigm. On the basis of these analogies, Noreen suggested that it followed from Green’s area theorem that the area under the ROC for the 2IAX paradigm should equal PC_4IAX.

In this note, we describe a mathematical proof of Noreen’s (1981) conjecture. The proof is general, in the sense that it does not require that the sensory observations have a specific (e.g., Gaussian) distribution, or that they be statistically independent. This is important, because the degree of statistical dependence between the sensory observations has been shown to influence the decision rule of an optimal observer in both the 2IAX and the 4IAX paradigms (Dai, Versfeld, & Green, 1996; Micheyl & Messing, 2006). It has been shown that when (Gaussian) observations are highly correlated, the “differencing” strategy, which involves comparing the two stimuli in a pair, is optimal in the likelihood ratio sense. However, when the observations are uncorrelated, a more elaborate independent observations decision rule becomes available, and the “differencing” strategy is no longer optimal. Dependent observations are likely to arise in experiments in which the stimuli are randomized (or roved) across trials, so that the two (or four) stimuli presented on a trial are more likely to fall closer together in the parameter space than stimuli from different trials.¹ The mathematical proof described below applies equally well to the results of such experiments as to those of experiments in which no roving was used, and regardless of which decision rule is assumed.

In the final section, we place this work into a broader context and discuss the relationship of the area theorem demonstrated here with other existing area theorems. In particular, we explain how the theorem described here differs from one proposed earlier by Irwin, Hautus, and Butcher (1999).

ROC Area in the 2IAX Paradigm

In general, the ROC curve can be expressed as a function, which relates the probability of a hit, P_H, to the probability of a false alarm, P_F; formally: P_H = g(P_F). The area, A, under the ROC can then be computed as the integral of this function between P_F = 0 and P_F = 1 (Egan, 1975); formally:

$$A=\underset{0}{\overset{1}{\int }}g\left(P_{\mathrm{F}}\right){dP}_{\mathrm{F}}.$$ (1)

The fact that the ROC curve is defined within a unit square can be used to derive an alternative expression for the ROC area:

$$A=1-\underset{0}{\overset{1}{\int }}{g}^{-1}\left(P_{\mathrm{H}}\right){dP}_{\mathrm{H}},$$ (2)

where g⁻¹ denotes the inverse of function g. The latter expression for the ROC area will be used in subsequent derivations.

The probability of a hit, P_H, and the probability of a false alarm, P_F, can each be expressed as functions of the likelihood ratio criterion, β, namely: P_H = h(β), and P_F = f(β). Using this parameterization, the area under the ROC may be reexpressed as an integral over all possible values of the criterion,² as follows:

$$A=1+\underset{0}{\overset{+\infty }{\int }}{P}_{\mathrm{F}}(\beta )\frac{{dP}_{\mathrm{H}}(\beta )}{d\beta }d\beta .$$ (3)

Note that the equations above are true in general, and that all the quantities involved in them are paradigm independent. In contrast, the subsequent derivations are specific to the 2IAX paradigm. In order to make this clear in the notation, we hereafter add the subscript 2IAX to all quantities that are specific to this paradigm.

For a given value of the criterion, β, the probability of a false alarm in the 2IAX paradigm, P_F2IAX(β), is equal to the probability that the likelihood ratio of the different over same stimuli, λ_2IAX, exceeds β, given the event, S, that the same stimuli were presented; formally:

$$\begin{array}{l}{P}_{{\mathrm{F}}_{2\text{IAX}}}\left(\beta \right)=P\left(\lambda >\beta \mid S\right)+\underset{\beta }{\overset{+\infty }{\int }}{\psi }_{S_{2\text{IAX}}}(\lambda )d\lambda \\ =1-\underset{0}{\overset{\beta }{\int }}{\psi }_{S_{2\text{IAX}}}(\lambda )d\lambda ,\end{array}$$ (4)

where ψ_S2IAX(λ) represents the probability density function (PDF) of the sensory observations evoked by the stimuli, given S.

Similarly, the probability of a hit, P_H2IAX(β), is equal to the probability that the likelihood ratio, λ_2IAX, exceeds β, given the event, D, that the different stimuli were presented; formally:

$$\begin{array}{l}{P}_{{\mathrm{H}}_{2\text{IAX}}}(\beta )=P(\lambda >\beta \mid D)=\underset{\beta }{\overset{+\infty }{\int }}{\psi }_{D_{2\text{IAX}}}(\lambda )d\lambda \\ =1-\underset{0}{\overset{\beta }{\int }}{\psi }_{D_{2\text{IAX}}}(\lambda )d\lambda ,\end{array}$$ (5)

where ψ_D2IAX(λ) represents the PDF of the sensory observations evoked by the stimuli, given D.

The density functions, ψ_S2IAX(λ) and ψ_D2IAX(λ), can be derived from the PDFs of the observations in the two intervals, ω_S2IAX(x₁, x₂), and ω_D2IAX(x₁, x₂), using an injective mapping from the two-dimensional observation space to the one-dimensional likelihood ratio space.³ This mapping can be expressed formally as:

$${\psi }_{S_{2\text{IAX}}}(\lambda )=\underset{R(\lambda )}{\int }{\omega }_{S_{2\text{IAX}}}\left(x_1,x_2\right)ds$$

and

$${\psi }_{D_{2\text{IAX}}}(\lambda )=\underset{R(\lambda )}{\int }{\omega }_{D_{2\text{IAX}}}\left(x_1,x_2\right)ds,$$

where R(λ) represents a region in the x₁x₂ space, where the likelihood ratio equals λ; that is,

$$\frac{{\omega }_{D_{2\text{IAX}}}\left(x_1,x_2\right)}{{\omega }_{S_{2\text{IAX}}}\left(x_1,x_2\right)}=\lambda .$$⁴

The derivative of P_H2IAX(β) with respect to β is

$$\begin{array}{l}\frac{{dP}_{{\mathrm{H}}_{2\text{IAX}}}(\beta )}{d\beta }=\frac{d}{d\beta }\underset{\beta }{\overset{+\infty }{\int }}{\psi }_{D_{2\text{IAX}}}\\ =\frac{d}{d\beta }\left[1-\underset{0}{\overset{\beta }{\int }}{\psi }_{D_{2\text{IAX}}}(\lambda )d\lambda \right]\\ =-{\psi }_{D_{2\text{IAX}}}(\beta ).\end{array}$$ (6)

Plugging Equations 4 and 6 into Equation 4, one obtains:

$$A_{2\text{IAX}}=1+\underset{0}{\overset{+\infty }{\int }}\left[1-\underset{0}{\overset{\beta }{\int }}{\psi }_{S_{2\text{IAX}}}(\lambda )d\lambda \right]\left[-{\psi }_{D_{2\text{IAX}}}(\beta )\right]d\beta ,$$ (7)

which can be rewritten as shown in Equation 8.

$$A_{2\text{IAX}}=1-\underset{0}{\overset{+\infty }{\int }}{\psi }_{D_{2\text{IAX}}}(\beta )d\beta +\underset{0}{\overset{+\infty }{\int }}\left[\underset{0}{\overset{\beta }{\int }}{\psi }_{S_{2\text{IAX}}}(\lambda )d\lambda \right]{\psi }_{D_{2\text{IAX}}}(\beta )d\beta .$$ (8)

Notice that the first integrand equals 1, and cancels out with the 1 in front, leaving

$$A_{2\text{IAX}}=\underset{0}{\overset{+\infty }{\int }}\left[\underset{0}{\overset{\beta }{\int }}{\psi }_{S_{2\text{IAX}}}(\lambda )d\lambda \right]{\psi }_{D_{2\text{IAX}}}(\beta )d\beta .$$ (9)

Proportion Correct in the 4IAX Paradigm

The optimal likelihood ratio decision rule in the 4IAX paradigm involves computing the likelihood ratio of two stimuli in each pair being different over same, comparing the two resulting likelihood ratios, and selecting the pair for which the likelihood ratio is larger. Denoting as λ₁ the likelihood ratio associated with the first pair,⁵ and as λ₂ that associated with the second pair, this decision rule can be stated as follows: If λ₁ > λ₂, respond “first pair”; otherwise, respond “second pair.”

For an observer following this rule, the proportion of correct responses in the 4IAX paradigm, PC_4IAX, can be computed as the probability that λ₁ is greater than λ₂, given the event, D1, that the pair containing the different stimuli was presented first; formally,

$${\text{PC}}_{4\text{IAX}}=P({\lambda }_1>{\lambda }_2\mid D1).$$ (10)

This is the same as the probability that λ₁ = β and λ₂ < β, given D1, for all possible values of β.

Using the notation that we introduced in the preceding section, the probability that λ₂ < β given that the second pair contains the same stimuli is given by

$$P({\lambda }_2<\beta \mid D1)=\underset{0}{\overset{\beta }{\int }}{\psi }_{S_{2\text{IAX}}}({\lambda }_2)d{\lambda }_2.$$ (11)

The probability that λ₁ = β given that the first pair contains the different stimuli is

$$P({\lambda }_1=\beta \mid D1)={\psi }_{D_{2\text{IAX}}}(\beta ).$$ (12)

Using these identities, Equation 10 can be written as

$$P{C}_{4\text{IAX}}=\underset{0}{\overset{+\infty }{\int }}\left[\underset{0}{\overset{\beta }{\int }}{\psi }_{S_{2\text{IAX}}}(\lambda )d\lambda \right]{\psi }_{D_{2\text{IAX}}}(\beta )d\beta .$$ (13)

Note that this is the same as Equation 9. Therefore, the proportion correct for an optimal (likelihood ratio) unbiased observer in the 4IAX paradigm equals the area under the ROC in the 2IAX paradigm.

Further insight into this conclusion may be obtained by noting that P(λ₂ < β|D1), as given in Equation 11, is the complement of P_F2IAX(β), as given in Equation 4. Moreover, P(λ < β|D1), as given in Equation 12, is the negative of the partial derivative of P_H2IAX(β) with respect to β given by Equation 6. Therefore, Equation 13 may be rewritten as

$${\text{PC}}_{4\text{IAX}}=\underset{0}{\overset{+\infty }{\int }}\left[1-P_{{\mathrm{F}}_{2\text{IAX}}}(\beta )\right]\left[-\frac{{dP}_{{\mathrm{H}}_{2\text{IAX}}}(\beta )}{d\beta }\right]d\beta .$$ (14)

This simplifies as

$${\text{PC}}_{4\text{IAX}}=1+\underset{0}{\overset{+\infty }{\int }}{P}_{{\mathrm{F}}_{2\text{IAX}}}(\beta )\frac{{dP}_{{\mathrm{H}}_{2\text{IAX}}}(\beta )}{d\beta }d\beta .$$ (15)

Note that, except for the 2IAX subscripts, the right-hand side of that equation is the same as that of Equation 3, which gives the area under the ROC for the general case.

Relation to Other Psychophysical Area Theorems

The derivations above provide a formal demonstration of the area theorem proposed by Noreen (1981), which relates the ROC area in the 2IAX paradigm and proportion correct in the 4IAX paradigm. This area theorem is reminiscent of that proposed initially by Green (1964; Green & Swets, 1966), in that (1) both theorems relate the ROC area in one paradigm with proportion correct in another paradigm; (2) the number of stimuli in the second paradigm (2I2AFC in Green’s theorem; 4IAX here) is double that in the former paradigm (YN in Green’s theorem; 2IAX here); and (3) the likelihood ratio for the second paradigm is a ratio of likelihood ratios, each of which is of the same form as the likelihood ratio from the first paradigm.

Another area theorem for the 2IAX paradigm was proposed by Irwin et al. (1999). These authors provided an elegant demonstration of the fact that the area under the ROC for an observer using the “differencing” strategy in the 2IAX paradigm equals the proportion of correct responses of an observer who makes optimal use of un-correlated observations in the same (2IAX) paradigm. Whereas the area theorem presented by Irwin et al. (1999) relates two quantities that pertain to the same psychophysical paradigm under different decision strategies, the theorem described in this note relates two quantities that pertain to two different paradigms, assuming that the same type of decision strategy is used in both. A reconciliation between Irwin et al.’s conclusions and those of the present work stems from the fact that the proportion of correct responses of an optimal observer in the 2IAX paradigm with independent observations is equal to the proportion of correct responses of a differencing observer in the 4IAX paradigm. This can be verified easily for the case of Gaussian observations. For that case, the equation relating d′ to proportion correct in the 2IAX paradigm with independent observations, PC_2IAXio, given in various articles and signal detection theory textbooks (e.g., Macmillan & Creelman, 2005) is the same as that for the 4IAX paradigm with highly correlated observations, PC_4IAXhco: PC_2IAXio = PC_4IAXhco = Φ²(d′/2) + Φ²(−d′/2).

From this point of view, it is apparent that the area theorem considered in this note, which relates the ROC area in the 2IAX paradigm with highly correlated observations to PC_4IAXhco, is consistent with that proposed by Irwin et al. (1999), which relates the ROC area in the 2IAX paradigm with highly correlated observations to PC_2IAXio.

To summarize, the proof of Noreen’s (1981) proposed area theorem described in this note is valid regardless of whether the observations are statistically independent or not, and of whether the observer uses the “differencing” rule or the independent observations rule. Importantly, the proof is based on the assumption that the same type of decision rule is used in the two paradigms. In other words, the theorem says that the ROC area for a “differencing” observer in the 2IAX paradigm equals the proportion correct of a “differencing” observer in the 4IAX paradigm, and that the ROC area for an independent observations observer in the 2IAX paradigm equals the proportion correct of an independent observations observer in the 4IAX paradigm. The theorem should not be interpreted to mean that the ROC area for an independent observations observer in the 2IAX paradigm equals the proportion correct of a “differencing” observer in the 4IAX paradigm, or conversely, that the ROC area for a “differencing” observer in the 2IAX paradigm equals the proportion correct of an independent observations observer in the 4IAX paradigm.