Some Problems With the Analytical Argument in Support of RP67 in the Context of the Bookmark Standard Setting Method

Abstract

The choice of response probability in the bookmark method has been shown to affect outcomes in important ways. These findings have implications for the validity of the bookmark method because panelists’ inability to internally adjust when given different response probabilities suggests that they are not performing the intended judgment task. In response to the concerns these findings raise, proponents of the bookmark method argue that such concerns can be addressed by using a response probability of .67. A crucial part of their argument includes the often-repeated claim that the .67 value corresponds with the maximum information for a correct response, which is believed to be beneficial in some way. In this article, it is shown that this claim is mistaken; that the formula upon which the .67 result is based is incorrect; that (for the relevant measurement model) there is no difference between the information for a correct response, for an incorrect response, or for the item overall; and, more generally, that the “maximize information” approach is based on the wrong likelihood function altogether.

The Bookmark standard setting procedure was introduced in 1996 by Lewis, Mitzel, and Green in response to the perceived challenges and shortcomings of Angoff-type approaches to setting standards. Since that time, the method has been widely adopted—particularly in K-12 testing. For both the Angoff and the Bookmark method, content expert panelists discuss and conceptualize the minimally proficient examinee: The examinee whose proficiency level is just high enough to justify a given classification (e.g., Pass). With this conceptualization in mind, Angoff methods require content experts to then review individual test items and estimate the probability that the minimally proficient examinee will answer each item correctly. The Bookmark method calls upon panelists to make a related judgment, but the task is structured differently. With this procedure, test items are ordered by difficulty and arranged into what is called an ordered item booklet. Panelists are asked to review each item starting at the beginning of the booklet, and then place a bookmark at the point between the items in the booklet at which the probability of success for the minimally proficient examinee drops below a prespecified value referred to as the response probability (RP). Item response theory procedures are then used to identify the cut score as the place on the proficiency scale that is associated with that RP given the item response functions for the items in the vicinity of the bookmark.

The choice of RP was, as Lewis, Mitzel, Mercado, and Schulz (2012) characterized it, “an early source of controversy”; however, eventually this controversy subsided, and practice coalesced around an RP of .67 (hereafter, RP67; Lewis et al., 2012, p. 233). Yet, the evidentiary basis for this consensus is surprisingly modest. One (often-repeated) argument in support of RP67—what is sometimes called “the conceptual and psychometric basis of the RP value of .67” (Huynh, 2006, p. 19)—is examined in this article and is shown to be vulnerable to criticism on theoretical and mathematical grounds.

Background

Suppose we have some panelist and we know, without error, that his or her conceptualization of the minimally proficiency candidate corresponds to some proficiency level $\theta =\omega$. We might call this the true cut score$\omega$ for our panelist. Now suppose our panelist is working their way through an ordered item booklet and encounters some item $i$. Upon reaching this item, our devoted panelist must make a judgment about the probability of success for an examinee who possesses their conceptualization of minimal proficiency—that is, an examinee with a proficiency equal to $\omega$. Specifically, our judge must decide whether or not the probability of success for such an examinee on item $i$ drops below the RP. In a perfect world, our panelist would compare the probability $P_i(U=1|\omega )$ with the specified RP; however, in our imperfect world, this comparison will be contaminated in some way by error. So, if the RP is greater than $P_i(U=1|\omega )$, our judge should insert their bookmark, yet may fail to do so due to an error of some kind; likewise, if the RP is less than $P_i(U=1|\omega )$, our judge should advance to the next item, yet, again, may fail to do so due to error.

The above description of panelist error has been left deliberately vague because its form is unimportant for the present discussion. For our immediate purpose, the focus is on the error in which we are ultimately most interested—the error in the cut score estimate. Consider an example. Figure 1 shows the item characteristic curves for three items: some item $i$ together with an easier item $h$ and a harder item $j$. Also shown are the probabilities of success on each item for someone with a proficiency equal to the true cut score $\omega$, and given an RP value, the associated proficiencies for each item: ${\theta }_h$, ${\theta }_i$, and ${\theta }_j$. The RP value is indicated by the horizontal light dashed line, and were these items to appear in an ordered item booklet, a bookmark inserted between items $h$and $i$ would result in a cut score of ${\hat{\omega }}_{h,i}=[{\theta }_h+{\theta }_i]/2$, the midpoint between the proficiencies ${\theta }_h$ and ${\theta }_i$. Likewise, a bookmark inserted between items $i$ and $j$ would lead to a cut score of ${\hat{\omega }}_{i,j}=[{\theta }_i+{\theta }_j]/2$, the midpoint between the proficiencies ${\theta }_i$ and ${\theta }_j$.¹

Figure 1.

Three item characteristic functions.

Note. The thin gray line shows the relationship between the true cut score $\omega$ and the probability of success on these items, $P_h(U=1|\omega )$, $P_i(U=1|\omega )$, and $P_j(U=1|\omega )$. An RP value is indicated by the horizontal light dashed line. A bookmark inserted between items $h$ and $i$ results in a cut score of ${\hat{\omega }}_i=[{\theta }_h+{\theta }_i]/2$, whereas a bookmark inserted between items $i$ and $j$ leads to a cut score of ${\hat{\omega }}_j=[{\theta }_i+{\theta }_j]/2$. RP = response probability.

Since the probability of success, $P_i(U=1|\omega )$, is less than the RP, it follows that upon encountering item $i$, our judge should insert their bookmark indicating a cut score estimate of ${\hat{\omega }}_{h,i}$; however, suppose our panelist made some error such that they advance to the next item, item $j$. Our panelist’s judgment on item $j$ is, of course, also subject to error, but suppose they (correctly) conclude that $P_j(U=1|\omega )<\mathrm{RP}$. Now, the resulting cut score estimate is ${\hat{\omega }}_{i,j}$ rather than ${\hat{\omega }}_{h,i}$ and our cut score error is ${\hat{\omega }}_{i,j}-\omega$ rather than the more modest ${\hat{\omega }}_{h,i}-\omega$.

On the Choice of RP

The preceding discussion demonstrates that were panelists able to complete the judgment task without error, any RP value between an item characteristic curves’ lower asymptote (0 in this example) and 1 could be specified without affecting the mechanics of the judgment exercise (Hambleton & Pitoniak, 2006). Nevertheless, even the best panelist will make errors and research has shown (and common sense tells us) that, in fact, the choice of RP systematically affects outcomes—sometimes quite profoundly and in unexpected ways (Hauser, Edley, Koenig, & Elliott, 2005; Williams & Schulz, 2005). The implications of these findings are alarming—after all, if panelists are unable to adjust their judgments to reflect the choice of RP, what do their judgments actually mean? We may never know the answer to this existential question, but to the extent that the resultant cut scores are systematically affected by the choice of RP, they will not reflect the panelists’ beliefs (except by chance) and their validity will be in doubt. Yet, this matter does not seem to have preoccupied proponents of the Bookmark method especially. The focus in the literature has been on the advantages and defensibility of RP67, rather than the validity of judgments that depend so much on the choice of RP.

Although the question of whether or not panelists are capable of making the intended bookmark judgments remains unresolved, let us suppose the advocates of the Bookmark method are correct and that panelists are capable of making the intended judgments but that some RP values are less prone (or more robust) to panelist error than others. So, for example, it might be argued that very high or very low RP values are best avoided because item characteristic curves are asymptotic at these extreme values, and therefore multiple items, all with $P(U=1|\omega )$ very near such RPs, could nevertheless correspond to very different $\hat{\omega }$. In addition to sensible restrictions like this, other rules governing the choice of RP value have been widely adopted and promoted with less obvious justifications. The rise of RP67 as the de facto RP is one such rule.

Aside from 2/3 being “a familiar value” (Lewis, Green, Mitzel, Baum, & Patz, 1998, p. 4), the argument in favor of RP67 falls along two distinct lines. The first comprises two related notions: (a) that the concept of mastery (or can do) allows panelists to concretize a specified RP value that they would otherwise find too abstract and (b) that panelists’ concept of mastery corresponds to an RP of .67. Together, we might refer to this line of reasoning as the mastery conceit. The mastery conceit seems benign enough as far as heuristics go and although its utility is not investigated here, it seems plausible that the concept of mastery provides a more tangible and comprehensible link between the knowledge, skills, and abilities associated with a given performance level and the judgment task required of the panelists. It is perhaps less obvious that panelists’ concept of mastery corresponds to (or can be made to correspond to) RP67—it seems quite possible that notions of mastery are neither static nor universal (what Lorié, 2002, called the problem of mastery probability indeterminacy). Nevertheless, the mastery conceit reportedly has been helpful (e.g., Lewis et al., 2012) and, in any case, the question of its actual utility lies outside the scope of this article.

The nature of these arguments deserves comment. The familiar value and mastery conceit assertions—however valid they may be—are by and large conjectural, rhetorical, and anecdotal. The second rationale for using RP67, which is the focus of this article, is the contention that the optimum response probability is the one that maximizes information for a correct response.² Although this too is mere conjecture—it has yet to be demonstrated that there is anything special about maximizing information for a correct response that makes panelists less prone to committing errors or that makes the resultant cut score estimates more robust to panelists’ errors—there is an analytical character to the notion of maximizing information. This character is also evident in the literature, which concentrates on showing (albeit incorrectly) how maximizing such information corresponds to an RP of .67. At the risk of adding more conjecture still, one wonders if perhaps the research in this area has given the entire claim of optimality a veneer of analytical proof it was otherwise lacking.

The lack of evidence demonstrating what can be gained by maximizing information might be considered enough to question whether the use of RP67 is actually defensible at all, but, in any case, the focus here mirrors the focus in the literature and is therefore limited to the analytical reasoning that connects the maximization of information for a correct response to an RP of .67. This reasoning is fully explicated in the work of Huynh (1994, 1998, 2000a, 2000b, 2006) and his conclusions (and their implication that .67 is in some important way optimal) are frequently repeated (e.g., Beretvas, 2004; Cizek, Bunch, & Koons, 2004; Karantonis & Sireci, 2006; Lewis et al., 2012; Lin, 2006; Mitzel, Lewis, Patz, & Green, 2001; Skaggs, Hein, & Awuor, 2007; Skaggs & Tessema, 2001), most often uncritically (cf., Kolstad et al., 1998). Huynh’s analytical derivation for RP67 will be discussed and (to the extent possible) unpacked below. Yet, because of its central importance, some readers may find it helpful to first review some key concepts related to Fisher information. Such a review will be given next.

Observed Information and Expected Information

Supposing some test question fits the three-parameter logistic model (3PL; Birnbaum, 1968), let the probability of a respondent with a proficiency of $\theta$ responding positively, $U=1$, be given by

$$P(U=1|\theta )=c+(1-c)\frac{e^{a(\theta -b)}}{1+e^{a(\theta -b)}},$$ (1)

where $a$ is the item discrimination, $b$ is the item difficulty, and $c$ is the (pseudo-) guessing parameter (all of which are assumed to be known). Generalizing, it follows that the probability of a response $U$ may be expressed as

$$P(U|\theta )=P{(U=1|\theta )}^U{[1-P(U=1|\theta )]}^{U-1}.$$ (2)

Now, suppose we have some observation, $U=u$. Let $\mathcal{L}(\theta |u)$ denote the likelihood of $\theta$ given this response $u$ and let $\ell (\theta |u)=\ln \mathcal{L}(\theta |u)$, the log-likelihood. Intuitively—from a maximum likelihood perspective, for example—the steeper the curvature of the log-likelihood, the narrower the range of $\theta$ likely to have produced $u$, and for this reason, this curvature is often of great interest. To find the local curvature, we start with the first-order partial derivative (with respect to $\theta$) of $\ell (\theta |u)$,

$$\ell \prime (\theta |u)=\frac{\partial \ell (\theta |u)}{\partial \theta },$$ (3)

which is typically called the score, score function, or Fisher’s score function (not to be confused with a test score or a scored item response). The score function is the slope of the log-likelihood, and it is the slope of this slope—that is, the second-order partial derivative (again with respect to $\theta$) of $\ell (\theta |u)$, $\ell ″(\theta |u)$—that equals the local curvature³ of the log-likelihood:

$$\ell ″(\theta |u)=\frac{{\partial }^2\ell (\theta |u)}{\partial {\theta }^2}.$$ (4)

The negative of this local curvature is defined as the observed Fisher information function, which can be denoted as

$$I(\theta |u)\equiv -\ell ″(\theta |u)=-\frac{{\partial }^2\ell (\theta |u)}{\partial {\theta }^2}.$$ (5)

In contrast, Fisher information proper is defined as the expectation of this observed information. In other words, Fisher information is, essentially, the curvature averaged over all possible responses (e.g., correct and incorrect for a dichotomously scored item). Of course, to take an expectation, we need to reintroduce the random variable $U$ rather than considering a particular instance of $U$, $u$, as we did in the case of the observed information. Likewise, we are now interested in the probability of the random variable $U$ given $\theta$ rather than the likelihood of $\theta$ given a particular instance of $U$, $u$, and so $\mathcal{L}(\theta |u)$ is replaced with the equivalent probability $P(U|\theta )$:

$$\mathcal{I}(U|\theta )\equiv -E\left[\frac{{\partial }^2\ln P(U|\theta )}{\partial {\theta }^2}\right].$$ (6)

When a maximum likelihood estimator exists (e.g., which is usually the case when we have responses from a set of items), the inverse of the Fisher information is asymptotically equal to the sampling variance of the maximum likelihood estimator, which is why information is of interest generally and, more specifically, why we often seek to maximize it (van der Linden, 2010).

Item Information, Item Response Information, and Information Share

Fisher information corresponds to what in the psychometric literature is called item information—or, in the case of a set of items, test information (Birnbaum, 1968⁴), but what of Equation 5, the observed information? In psychometrics, the notion that each score, $u$, “carries” its own unique amount of information (probably) goes back to the work of Samejima, who suggested that the information for each score category of an item was measured by its item response information function (which corresponds to Equation 5), and that this function was more—or at least more uniquely—informative compared with Birnbaum’s item information function—that is, the standard Fisher information function shown in Equation 6 (Birnbaum, 1968; Samejima, 1973). Despite its allure, however, Samejima’s notion of a response-specific item information function has not been widely adopted within item response theory discourse—although it does arise occasionally (e.g., Bradlow, 1996, who extended the work of Samejima’s 1973 study; Baker & Kim, 2004; Huynh, 1994, 1998, 2000a, 2000b, 2006; Magis, 2015; van der Linden, 1998).

One final term remains: information share. In Samejima’s (1969) monograph introducing the graded response model, she defines information share as follows:

. . . information share is the information function of an individual graded response, which is the negative of the first derivative of the basic function [i.e., the observed information shown in Equation 5], multiplied by the operating characteristic [i.e., the item response function] . . . (p. 38)

Following the notation used here, information share for a correct response can be expressed as

$$S(\theta |u=1)\equiv I(\theta |u=1)P(U=1|\theta )\equiv -\frac{{\partial }^2\ell (\theta |u=1)}{\partial {\theta }^2}P(U=1|\theta ),$$ (7)

where $S(\theta |u=1)$ denotes information share. The notion of item information share is even less widely used than item response information (although like item response information, it does arise now and then—for example, Baker & Kim, 2004; Bock, 1972; DeMars, 2010; Suh & Bolt, 2010).

For Baker and Kim (2004), the difference between item response information and item information share can be described as follows:

• The amount of information due to an item response category $I(\theta |u=k)$ is a measure of how well responses in that category estimate the examinee’s ability. The information share of a response category $I(\theta |u=k)P(U=k|\theta )$ is the amount of information contributed by the category to the item information. (p. 223; the original notation was changed to be consistent with the notation used in this article.)

These authors argue that since the sum of the item response information functions across all possible response categories for a given proficiency do not sum to the total information, $\mathcal{I}(U|\theta )$, they are difficult to interpret. Consequently, they conclude that “from an interpretive point of view, the amount of information share of each response category is the most informative” (p. 223).⁵ The reader can decide for themselves whether or not they agree with this claim; for our purposes moving forward, the point is that three distinct concepts related to information have been introduced: observed item information (Equation 5), expected item information (Equation 6), and information share (Equation 7).

Huynh’s Unorthodox Derivation of RP67

The above background on Fisher information is well known and (it is hoped) uncontroversial. We begin our move away from this familiarity with a passage from Huynh’s (2006) note intended to clarify the rationale for the RP67:

For binary items, the focus of Huynh’s work on the RP value has been on information of the correct response and not on the (total) item information, which is the typical focus for item information. The crucial difference between these two types of information apparently has been overlooked by several writers on bookmark standard setting, including Cizek et al. (2004) and Karantonis and Sireci (2006). . . Certainly there are differences among (total) item information. . . and information of the correct response. (p. 20; emphasis in original)

Huynh’s broader point here is correct: Equations 5 and 6 are different, as are the concepts of observed and expected information. Nevertheless, despite these differences, observed information and expected information are, in certain important special cases, mathematically equivalent. Notably, equivalence is observed in the case of $c=0$, the specific case that preoccupies Huynh and the architects of the Bookmark method so much (Lewis et al., 2012).⁶ Let us take a moment to see how this equivalence arises.

Recalling Equations 2 and 5, the item response information function—that is, the observed information—for the two-parameter logistic (2PL) is given by

$$I_{2\mathrm{PL}}(\theta |u)=-\frac{{\partial }^2\ell (\theta |u)}{\partial {\theta }^2}=-\frac{{\partial }^2\ln \left[{\left(\frac{e^{a(\theta -b)}}{1+e^{a(\theta -b)}}\right)}^u{(1-\frac{e^{a(\theta -b)}}{1+e^{a(\theta -b)}})}^{1-u}\right]}{\partial {\theta }^2}.$$ (8)

(Note that this is not the information for a correct response, $u=1$, per se but rather for any response $u=0$ or $u=1$.) Solving this derivative, we have

$$I_{2\mathrm{PL}}\left(\theta \right)=\frac{a^2{e}^{a(\theta -b)}}{{(e^{a(\theta -b)}+1)}^2}.$$ (9)

Equation 9 has an important property: the absence of the $u$ term. It follows that the observed information (Equation 5) for the 2PL is unaffected by the specific value of $u$ or, in other words, the observed information is the same without regard to the correctness of a given response: $\mathcal{I}(U|\theta )=I(\theta |u=0)=I(\theta |u=1)$. As noted, this characteristic is not universally true, but it is true for exponential family models. This means that in addition to the 2PL, it is a well-known (e.g., Bradlow, 1996; DeMars, 2010; Kolstad et al., 1998; Magis, 2015; van der Linden, 1998; Veerkamp, 1996) property of the one-parameter logistic (1PL) as well as the so-called divide-by-total polytomous models: the partial credit model (Masters, 1982), the generalized partial credit model (Muraki, 1992), the rating scale model (Andrich, 1978), and the nominal response model (Bock, 1972). So, while there are occasions when observed information may yield better standard error estimates than (expected) information (Efron & Hinkley, 1978; Lindsay & Li, 1997) or may provide unique diagnostic information (Bradlow, 1996), this is not one of those occasions.

Given their equivalence, Huynh’s suggestion that the observed information for a correct response and the expected information function for a given item are distinct even when $c=0$ has perhaps contributed to some of the confusion over precisely what information Huynh was referring to in his published work (e.g., Cizek et al., 2004; Karantonis & Sireci, 2006). To get to the bottom of this mystery, we turn now to Bock’s well-known 1972 study, which introduced the nominal response model and which appears to be foundational in Huynh’s thinking (e.g., Huynh, 1994, 1998, 2000a, 2000b). Bock (1972) suggests that expected item information can be partitioned into the “information due to” each response (p. 44, Equation 25⁷),

$$B(\theta |u=k)=\mathcal{I}(U|\theta )P(U=k|\theta ),$$ (10)

where $B(\theta |u=k)$ denotes the partitioned information⁸ corresponding to response category $k$. Bock states that Equation 10 follows from Samejima’s (1969) monograph, which introduced (at least in the context of item response theory) both the concepts of item response information and information share. As noted, Bock’s nominal model is an exponential family model and therefore $\mathcal{I}(U|\theta )=I(U=k|\theta )$. So, while he does not use the term information share, had Bock been aware that his model shared this property of other exponential family models, he may have intended Equation 10 to describe Samejima’s concept of information share. However, without such awareness, Equation 10 would not appear to correspond to item information, item response information, or information share. Still, there is some evidence that Bock (1972) may have intended Equation 10 to describe item response information (i.e., observed information)—he appears to use Equation 10 in his example to calculate “the information due to response in category h [$k$]” (p. 44)—and Huynh (1998) appears to interpret Equation 10 in this way (p. 38). Either way, when Huynh advocates using the RP associated with the maximum information for a correct response, he maximized Bock’s partitioned information, Equation 10, rather than Samejima’s formulation, the standard observed information for a correct response shown in Equation 5.⁹

Given that Equation 10 does not describe item response information, Huynh’s interpretation ends up being problematic. Specifically, the probability of a correct response when $B(\theta |u=1)$ is maximized is potentially quite different from the result when $I(\theta |u=1)$ is maximized. To see this, let ${\theta }_{max}^{*}$ denote the location on the proficiency metric where $B(\theta |u=1)$ is maximized:

$${\theta }_{max}^{*}=\arg \underset{\theta }{max}B(\theta |u=1)=\arg \underset{\theta }{max}\mathcal{I}(U|\theta )P(U=1|\theta ).$$ (11)

Here, ${\theta }_{max}^{*}$ can be found in the normal way by taking the root of Equation 11’s derivative,

$${\theta }_{max}^{*}=\frac{\mathrm{ab}+\ln (2)}{a}.$$ (12)

Equation 1 can then be evaluated using the result from Equation 12. So doing yields

$$P(U=1|{\theta }_{max}^{*})=\frac{c+2}{3},$$ (13)

which is the formula Huynh (1998, 2000a, 2000b, 2006) adopts. When $c=0$, Equation 13 evaluates to 2/3, the familiar RP67.

If we wanted to instead maximize the actual observed information for a correct response, $I(\theta |u=1)$, we would take the same approach, first finding

$${\theta }_{max}=\arg \underset{\theta }{max}I(\theta |u=1)=\arg \underset{\theta }{max}-\frac{{\partial }^2\ell (\theta |u=1)}{\partial {\theta }^2}.$$ (14)

by taking the root of the derivative of $I(\theta |u=1)$:

$${\theta }_{max}=\frac{\ln (\frac{\sqrt{\frac{c^3+35c^2+35c+1}{2\sqrt{c^2+34c+1}}+\frac{c^2}{2}+5c+\frac{1}{2}}}{2}+\frac{\sqrt{c^2+34c+1}}{4}+\frac{c}{4}+\frac{1}{4})+\mathrm{ab}}{a}.$$ (15)

and then plugging this result into Equation 1, this time yielding

$$P(U=1|{\theta }_{max})=\frac{\sqrt{\frac{2c^3+70c^2+70c+2}{\sqrt{c^2+34c+1}}+2c^2+20c+2}+\sqrt{c^2+34c+1}+5c+1}{\sqrt{\frac{2c^3+70c^2+70c+2}{\sqrt{c^2+34c+1}}+2c^2+20c+2}+\sqrt{c^2+34c+1}+c+5}.$$ (16)

It might be of some interest (albeit hopefully very little) to compare the “optimal” RP values using Huynh’s Equation 13 with those using Equation 16, which—to reiterate—corresponds to the accepted definition of observed information for a correct response. To this end, Figure 2 shows the probability of a correct response at the point of maximum information for a correct response using these two different formulations. The resultant response probabilities are optimal in so much as they maximize information associated with a correct response (in the sense of Equations 5 and 10). Note that these response probabilities are plotted as a function of the 3PL’s c-parameter—the only parameter left in Equations 13 and 16. Somewhat ironically, the largest difference occurs at $c=0$, the value proponents of the bookmark method suggest using.

Figure 2.

The probability of a correct response at the point of maximum information for a correct response as a function of the 3PL’s c-parameter.

Note. The solid line shows the result using Equation 16 and the dashed line shows the result using Huynh’s formulation shown in Equation 13. It can be seen that while the lines intersect at one point, they are quite distinct—most markedly when $c=0$. 3PL = three-parameter logistic.

Discussion

For more than 20 years, advocates and practitioners of the bookmark method have defended an RP of .67 based, in part, on Huynh’s claim that the optimal RP maximizes the information for a correct response, which he calculated using Equation 13, $P(U=1|{\theta }_{max}^{*})=(c+2)/3$. In this article, it was shown that Huynh’s result was inconsistent with the accepted definition of observed information and that the correct formula for maximizing the information for a correct response is actually the somewhat different Equation 16. Furthermore, the observation was made that in many cases—notably when $c=0$, which proponents of the bookmark method strongly encourage practitioners to adopt and which moreover is the condition for which Equation 13 evaluates to .67—the information function for a correct response is mathematically equivalent to the information function for an incorrect response as well as the overall item information. That Huynh’s result has been repeated so often and so uncritically should perhaps give us pause. More generally, it raises the question of whether or not RP67 should truly allay concerns over panelists’ inability to correctly adjust their judgments when given different response probabilities.

For the avoidance of doubt, the fact that Equation 16 yields the actual probability of a correct response at the point of maximum information for a correct response does not mean that Equation 16 somehow provides an optimal RP. Despite the attraction of maximizing information, recall that our specific interest in Fisher information is because of its relationship to the (local) curvature of the log-likelihood. And, the likelihood that in this case preoccupies Huynh and others is, regrettably, not the likelihood of a cut score estimate given a bookmark—it is the likelihood of proficiency given an examinee response. That is to say, the whole notion of maximizing item information (whether observed or expected) is premised on maximizing the curvature of the wrong likelihood altogether and therefore is not relevant to the question of what RP minimizes panelists’ errors.

Ultimately, finding the optimal RP will require a probabilistic description of what happens when a panelist interacts with an ordered item booklet. This is not a trivial undertaking and it will be left to future researchers. Until such a time, we should perhaps ask ourselves whether or not practice could be improved by making the “once controversial question” of optimal RP controversial once again.