A Multiprocess Item Response Model for Not-Reached Items due to Time Limits and Quitting

Abstract

So far, modeling approaches for not-reached items have considered one single underlying process. However, missing values at the end of a test can occur for a variety of reasons. On the one hand, examinees may not reach the end of a test due to time limits and lack of working speed. On the other hand, examinees may not attempt all items and quit responding due to, for example, fatigue or lack of motivation. We use response times retrieved from computerized testing to distinguish missing data due to lack of speed from missingness due to quitting. On the basis of this information, we present a new model that allows to disentangle and simultaneously model different missing data mechanisms underlying not-reached items. The model (a) supports a more fine-grained understanding of the processes underlying not-reached items and (b) allows to disentangle different sources describing test performance. In a simulation study, we evaluate estimation of the proposed model. In an empirical study, we show what insights can be gained regarding test-taking behavior using this model.

In large-scale assessments (LSAs), examinees do not always attempt all items they were assigned to answer. When an examinee fails to attempt a sequence of items presented at the end of a test, the resulting missing responses are referred to as not-reached items (NRIs). NRIs can occur for a variety of reasons. Examinees may not reach the end of a test due to lack of speed when tests are administered with time limits. This is supported by results from experimental research suggesting that increased test-taking time results in lower NRI rates (e.g., Mandinach, Bridgeman, Cahalan-Laitusis, & Trapani, 2005; Wild & Durso, 1979). However, the onset of NRIs does not seem to depend solely on test time. Examinees may quit the assessment prematurely due to, for example, fatigue or lack of motivation. This is particularly the case in low-stakes assessments where low motivation is likely to affect examinee test-taking behavior (Chen, von Davier, Yamamoto, & Kong, 2015; Cosgrove, 2011; Liu, Rios, & Borden, 2015; Wise & DeMars, 2005). Indeed, in LSAs, NRIs are even observed in assessments administered without time constraints, such as in the Programme for the International Assessment of Adult Competencies (PIAAC; Organisation for Economic Co-operation and Development [OECD], 2013).

NRIs occurring due to lack of speed and NRIs occurring due to quitting represent different types of missingness processes that tend to occur under different testing situations, correspond to different test-taking strategies, and might be related differently to ability. Thus, disentangling and modeling different types of NRIs can be beneficial for understanding examinee performance as well as for informing decisions regarding the adequate treatment of missing values due to NRIs. In this context, considering additional data retrieved from computer-based assessment facilitates the understanding of examinee behavior and thus of potential mechanisms underlying NRIs. For instance, cumulative response times (RTs) contain information on the time passed up to the last item attempted before ending the assessment. This allows to distinguish examinees who worked at a slow pace and reached the time limit before reaching the end of the test from those who displayed cumulative RTs far below the time limit without attempting all items administered (Pohl, Ulitzsch, & von Davier, 2019). Based on the information contained in RT data, Pohl et al. (2019) illustrated that within the same data set NRIs are plausible to occur due to different mechanisms, that is, lack of speed and quitting. In this article, we argue that these are potentially different mechanisms that should be modeled as such. We propose a framework to disentangle and simultaneously model these missingness mechanisms.

Dealing With Not-Reached Items in Large-Scale Assessments

Current practices for handling NRIs in LSAs are rather heterogeneous. While in the majority of LSAs, NRIs are either ignored (e.g., in the National Educational Panel Study [NEPS], Pohl & Carstensen, 2012) or scored as incorrect, mixed approaches exist. For instance, in the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS), NRIs are ignored for item parameter estimation and scored as incorrect for person parameter estimation (Foy, 2017, 2018). In PIAAC, NRIs are ignored if sufficient information about examinee proficiency is available; that is, if there are more than five item responses per domain. Otherwise, the examinees’ self-stated reasons for not completing the assessment are considered when handling missing responses due to NRIs. For examinees quitting the assessment after giving responses to fewer than five items per domain, all NRIs are treated as incorrect if the self-stated reason for not responding is related to cognitive skills. If examinees give reasons unrelated to competence, NRIs are ignored in the analysis (OECD, 2013). This treatment of NRIs acknowledges that NRIs can occur due to different mechanisms. The approach is however rather heuristic in that it (a) relies on self-stated reasons for not reaching the end of the test and (b) distinguishes different types of NRIs only for examinees who responded to less than five items.

Scoring NRIs as wrong assumes the probability to solve an NRI to be zero—regardless of the examinee’s ability level (see Lord, 1983; Rose, 2013; Rose, von Davier, & Nagengast, 2017). Ignoring NRIs assumes ignorability of the missingness mechanism. For ignorability to hold, data need to be missing at random; that is, missingness needs to be conditionally independent of the unobserved data given the observed data. Furthermore, the (unobserved) parameters governing the distribution of NRIs need to be distinct from ability (Mislevy & Wu, 1996; Rubin, 1976). There is, however, a substantial body of research suggesting that not reaching the end of a test is indeed related to ability (Debeer, Janssen, & Boeck, 2017; Glas & Pimentel, 2008; Lawrence, 1993; List, Köller, & Nagy, 2017; Pohl, Gräfe, & Rose, 2014; Rose, von Davier, & Xu, 2010). This indicates that the mechanisms underlying NRIs are nonignorable. Not properly accounting for the mechanisms that produce nonignorable missing data poses a threat to valid inferences and may potentially lead to biased person and item parameter estimates or distort relationships between ability and explanatory variables as well as country rankings (Glas & Pimentel, 2008; Köhler, Pohl, & Carstensen, 2017; Pohl et al., 2014; Rose, 2013). To properly account for nonignorable NRIs, a model for the mechanisms underlying their occurrence is needed.

Model-Based Approaches for Nonignorable Missing Values

In recent years, model-based approaches for handling nonignorable NRIs have been developed. In this class of models, information about NRIs is integrated into item response theory (IRT) models—either employing a latent or manifest variable—and thus accounted for when estimating ability.

For modeling ability, customary IRT models are employed. In the case of a Rasch model, the probability of a correct response on response indicator $u_{\mathrm{ij}}$, containing person $i$’s response on item $j$, can be modeled as a function of person ability ${\theta }_i$ and the item’s difficulty $b_j$:

$$p(u_{\mathrm{ij}}=1)=\frac{\exp ({\theta }_i-b_j)}{1+\exp ({\theta }_i-b_j)}.$$ (1)

Missing values due to not reaching the end of the test are not coded as wrong but rather treated as missing at random by including terms for missing data in the likelihood function used to estimate parameters.

Rose et al. (2010) suggested to employ a manifest missing data model, since, due to the monotone missing pattern resulting from NRIs, considering the number of reached items is sufficient to account for NRIs. Within manifest NRI approaches, information on the number of reached items $k_{{\mathrm{tot}}_i}$ is included as a manifest variable in the background model. This can be achieved by either regressing ability ${\theta }_i$ on $k_{{\mathrm{tot}}_i}$ or by applying multigroup IRT models where stratification on $k_{{\mathrm{tot}}_i}$ serves as a grouping variable (Rose et al., 2010). Manifest approaches are computationally less intensive compared with latent variable approaches for NRIs, and since 2015 are considered in the population model in the Programme for International Student Assessment (PISA) for the generation of plausible values (OECD, 2017).

Within latent variable approaches for nonignorable missing values due to NRIs, the information on NRIs is included in form of a second dimension describing the propensity to reach the end of the test (Glas & Pimentel, 2008; List et al., 2017). Missingness indicators $d_{\mathrm{ij}}$, being defined as 1 if $u_{\mathrm{ij}}$ is observed, 0 if $u_{\mathrm{ij}}$ is the first NRI, and coded as missing otherwise, constitute the measurement model for this propensity. $p(d_{\mathrm{ij}}=1)$ is then modeled as a function of examinee $i$’s propensity to reach the end of the test and item $j$’s response difficulty. Linear restrictions are imposed on the response difficulty parameters implying a monotonously decreasing probability of observing a response.

In both latent and manifest variable approaches for modeling the onset of NRIs, correlations different from zero between ability and the number of reached items/the propensity to reach the end of the test indicate that the onset of NRIs is related to the construct being measured, and thus nonignorability of the missingness mechanism. Including information about nonignorable NRIs has been shown to yield less biased and more accurate parameter estimates as compared with ignoring or scoring missing values as wrong (Glas & Pimentel, 2008; Rose et al., 2010; Rose et al., 2017).

Using Response Times to Model Not-Reached Items

If NRIs occur due to lack of speed, examinees reach the time limit before managing to reach the end of the test. For this case, it has been shown that the missing data process underlying NRIs can be described by examinee speed (Pohl et al., 2019). With the widespread availability of RT data retrieved from computerized testing, a direct measure of speed becomes available (van der Linden, 2006). Pohl et al. (2019) were the first to suggest utilizing this information on the missingness process by employing van der Linden’s (2007) hierarchical speed–accuracy (SA) framework to model the occurrence of NRIs in low-stakes assessments. They showed that the SA model (a) can successfully model NRIs due to time limits, (b) provides a closer description of the missing data processes than model-based approaches for nonignorable missing values, and (c) can also deal with varying enforcement of time limits—given that NRIs are the result of lack of speed.

In the SA model, first-level models are specified separately for the responses and associated RTs. For the response indicators $u_{\mathrm{ij}}$, van der Linden has recommended employing customary IRT models. For the RTs $t_{\mathrm{ij}}$, denoting the time examinee $i$ required to generate an answer to item $j$, a lognormal model with separate person and item parameters is chosen. That is, logarithmized RTs are assumed to follow a normal distribution. In the lognormal model, logarithmized RTs are considered to be a function of the examinee’s speed ${\tau }_i$ and the item’s time intensity ${\beta }_j$:

$$\begin{array}{c}\ln (t_{\mathrm{ij}})={\beta }_j-{\tau }_i+{ϵ}_{\mathrm{ij}},\hfill \\ \mathrm{with}\hfill \\ {ϵ}_{\mathrm{ij}}~\mathcal{N}(0,{\alpha }_j^{-2}).\hfill \end{array}.$$ (2)

${\alpha }_j$ represents the inverse of the RTs’ standard deviation and can be interpreted as a time discrimination parameter. That is, the larger the ${\alpha }_j$, the larger the proportion of the RT variance that stems from differences in speed across examinees. On a second level, joint multivariate normal distributions of person and item parameters are specified.

Pohl et al. (2019) delineated that approaches that consider the number of NRIs and the SA model are closely related. First, both approaches include an additional variable that represents the missingness mechanism in the model. While the SA model includes a direct measure of speed, model-based approaches for nonignorable missing values include the tendency to (not) reach the end of the test, as measured by the number of (not) reached items. If the number of NRIs is a result of lack of speed under testing conditions with time limits, the propensity to reach the end of the test can be understood as a proxy of working speed. In this case, the SA model presents a better and more fine-grained description of the missingness process (Pohl et al., 2019).

Second, both approaches assume a single mechanism underlying NRIs. Obviously, when the SA model is applied to account for NRIs, it is assumed that NRIs occurred due to lack of speed. Although model-based approaches that consider the number of NRIs do not explicitly rely on this assumption, they still assume the same missingness mechanism for all NRIs. When mechanisms leading to NRIs differ across examinees, that is, when multiple mechanisms are underlying NRIs such as lack of speed and quitting, this assumption is violated. Hence, in the case that missingness at the end of a test occurs not only due to speed but also due to motivational reasons, neither controlling for speed nor for the tendency to (not) reach the end of the test is sufficient to properly model NRIs.

Objective

We propose a new framework that takes into account that multiple mechanisms can underlie NRIs. Doing so requires (a) distinguishing examinees who quit the assessment from those who did not work with sufficient speed and reached the time limit and (b) establishing a model that considers both mechanisms simultaneously.

The remainder of this article is organized as follows: First, we present an approach that distinguishes between and simultaneously models two different types of NRIs. Second, parameter recovery of the proposed model is investigated using a simulation study. Third, the relevance of the model for understanding the missingness processes is illustrated in an empirical example.

Speed–Accuracy + Quitting Model

The proposed SA + quitting (SA + Q) framework, depicted in Figure 1, is an extension of the hierarchical SA model that also accounts for quitting. Following Pohl et al. (2019), NRIs due to lack of speed are modeled by considering examinee speed. For simplicity, we model ability $\theta$ employing a Rasch model as given by Equation 1. This is in accordance with the analysis frameworks of major LSAs (e.g., National Educational Panel Study; Pohl & Carstensen, 2012). Note that the model can be extended to other measurement models (see Ulitzsch, von Davier, & Pohl, 2019). For speed, we employ the lognormal model suggested by van der Linden (2006) as given by Equation 2. We model the quitting process by considering the number of items reached before quitting. These are determined by employing RTs to distinguish between examinees displaying NRIs due to lack of speed or due to quitting and constitute the measurement model for a newly introduced variable giving examinee test endurance. If the end of the test or the time limit was reached, the number of items reached before quitting is set to be missing, as no information about quitting is available in this case.

Figure 1.

Hierarchical framework for the joint modeling of speed, accuracy, and test endurance.

Identifying Quitting

In the present study, we assume that examinee $i$ has quit the assessment when (a) he or she did not reach the end of the test (i.e., $k_{\mathrm{to}{t}_i}$ is smaller than the number of items administered $K$) and (b) his or her total RT $t_{\mathrm{to}{t}_i}$ falls below the time limit (i.e., $t_{\mathrm{to}{t}_i}$ is smaller than $T_{\lim }$). Based on this information, an indicator $c_i$ of observed quitting behavior can be constructed as follows:

$$c_i=\{\begin{array}{cc}1\hfill & \mathrm{if}{k}_{\mathrm{to}{t}_i}<K\mathrm{and}{t}_{\mathrm{to}{t}_i}<T_{\lim }\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}.\hfill \end{array}$$ (3)

By construction, $\mathit{c}$ distinguishes between examinees who quit the assessment ($c_i=1$) and those who reached the time limit or finished the test before quitting ($c_i=0$).

Modeling Quitting

To model the quitting process, we utilize the information contained in the number of reached items up to the point where the assessment has been quit $k_{Q_i}$. When quitting behavior has been observed, that is, when $c_i=1$, $k_{Q_i}$ is given by the observed number of reached items $k_{\mathrm{to}{t}_i}$. We suggest to employ a Poisson lognormal model for $k_{Q_i}$. These are common for count data on the test or task level (Doebler & Holling, 2016; Jansen, 1994, 1995), such as the number of correct items in a task or—as mentioned in passing by Jansen (1995)—the number of completed items in a test. More specifically, we model the probability that examinee $i$ quits the assessment after attempting $k$ items as a Poisson process with mean ${\lambda }_i$, where $\ln ({\lambda }_i)$ corresponds to the person parameter ${\zeta }_i$:

$$\begin{array}{c}p(k_{Q_i}=k)=\frac{e^{-{\lambda }_i}{\lambda }_i^k}{k!},\hfill \\ \mathrm{with}\hfill \\ \ln ({\lambda }_i)={\zeta }_i.\hfill \end{array}$$ (4)

${\zeta }_i$ denotes examinee $i$’s test endurance and thus governs the item position at which examinee $i$ is most likely to quit the assessment—given that the assessment has not been quit before. As such, ${\zeta }_i$ can be understood as a survival parameter. In the context of NRIs, the onset of NRIs poses the event of interest occurring within a sequence of item positions (List et al., 2017). The survival function $S(k)$ depends on the Poisson lognormal model for $k_{Q_i}$ and gives the probability that examinee $i$ will continue the assessment beyond $k$ items, as follows:

$$S(k)=p(k_{Q_i}>k)=1-F(k)=1-\sum _{l=0}^k\frac{e^{-{\lambda }_i}{\lambda }_i^l}{l!},$$ (5)

with $F(k)$ denoting the Poisson cumulative distribution function.

The presented model incorporates the assumption that examinees will continue the assessment only for a definite number of items and thus will quit the assessment at some point. Note, however, that quitting behavior is not fully observable since some examinees either manage to complete the test or reach the time limit before quitting. Hence, under $c_i=0$, $k_{Q_i}$ is exposed to right censoring. That is, the observed number of reached items $k_{\mathrm{to}{t}_i}$ corresponds to the number of reached items before quitting only under $c_i=1$. Otherwise, $k_{\mathrm{to}{t}_i}$ marks the item position $k_{C_i}$ at which $k_{Q_i}$ has been right censored. The relationship between $k_{\mathrm{to}{t}_i}$, $k_{Q_i}$, and the censoring variable $k_{C_i}$ is given by

$$k_{\mathrm{to}{t}_i}=min(k_{Q_i},k_{C_i}).$$ (6)

Table 1 illustrates this relationship for three examinees administered a test of length $K=20$ with a time limit of $T_{\lim }=$1,800 seconds. Examinee 1 reached the end of the test within the allocated time without showing quitting behavior. Hence, $k_{Q_1}$ is censored due to reaching the end of the test at $k_{C_1}=k_{\mathrm{to}{t}_1}=K=20$. Examinee 2 did not reach the end of the test due to lack of speed. Hence, $k_{\mathrm{to}{t}_2}$ gives the item position $k_{C_2}$ at which quitting behavior has been censored due to lack of speed. Examinee 3 did reach neither the end of the test nor the time limit and is thus assumed to have quit the assessment. Thus, $k_{\mathrm{to}{t}_3}$ corresponds to the item position at which the assessment has been quit $k_{Q_3}$.

Table 1.

Illustration of the Relationship Between $k_{\mathrm{to}{t}_i}$, $k_{Q_i}$, and $k_{C_i}$

	$k_{\mathrm{to}{t}_i}$	$t_{\mathrm{to}{t}_i}$ (s)	$c_i$	$k_{Q_i}$	$k_{C_i}$
1	20	1,450	0	NA	20
2	16	1,800	0	NA	16
3	16	1,450	1	16	NA

Note.$k_{\mathrm{to}{t}_i}$: Observed number of reached items; $t_{\mathrm{to}{t}_i}$: total response time; $k_{Q_i}$: number of reached items before quitting; $k_{C_i}$: censoring item position; $c_i$: quitting indicator; NA: not applicable.

Note that usually in LSAs only a few examinees show quitting behavior. That is, there are few observations with $c_i=1$. This results in sparse data with respect to the number of reached items before quitting $k_{Q_i}$ on one hand and a large portion of $k_{Q_i}$ to be assumed to have been censored on the other. If speed is related to test endurance and there is censoring of $k_{Q_i}$ due to reaching the time limit, speed is informative with respect to the censoring of $k_{Q_i}$. Under such conditions, speed is related to both the parameter governing the distribution of $k_{Q_i}$ and the probability that $k_{Q_i}$ is censored. Modeling test endurance and speed jointly accounts for the informative censoring of $k_{Q_i}$ (Baker, Fitzmaurice, Freedman, & Kramer, 2005). Likewise, modeling ability and speed jointly with test endurance accounts for nonignorable missingness due to quitting on response as well as on RT indicators.

Second-Level Models

In analogy to the SA model, on the second level, the joint distributions of the first-level person and item parameters are modeled. Following van der Linden (2007), person parameters are assumed to be multivariate normal with mean vector

$${\mathbf{\mu }}_{\mathcal{P}}=({\mu }_{\theta },{\mu }_{\tau },{\mu }_{\zeta })$$ (7)

and covariance matrix

$${\sum }_{\mathcal{P}}=\left(\begin{array}{ccc}{\sigma }_{\theta }^2\hfill & {\sigma }_{\theta \tau }\hfill & {\sigma }_{\theta \zeta }\hfill \\ \multicolumn{1}{c}{{\sigma }_{\theta \tau }}& {\sigma }_{\tau }^2\hfill & {\sigma }_{\tau \zeta }\hfill \\ \multicolumn{1}{c}{{\sigma }_{\theta \zeta }}& {\sigma }_{\tau \zeta }\hfill & {\sigma }_{\zeta }^2\hfill \end{array}\right).$$ (8)

Assessing the joint distribution of person parameters provides valuable insights into the processes underlying NRIs, their relationship to ability as well as to each other. Nonzero correlations with test endurance indicate nonignorability of the associated missingness process.

For the sake of simplicity, for the measurement model of RTs, time discrimination parameters ${\alpha }_j$ are constrained to be equal across items—that is,

$${\alpha }_j=\alpha \mathrm{for}\mathrm{all}j.$$ (9)

This constraint can be understood as an analogue to the Rasch model in IRT (van der Linden, 2006) and thus mirrors the Rasch parameterization implemented for item responses in major LSAs (see, e.g., Pohl & Carstensen, 2012).

The joint distribution of item parameters is, in accordance with van der Linden (2007), assumed to be multivariate normal with mean vector

$${\mathbf{\mu }}_{\mathcal{I}}=({\mu }_b,{\mu }_{\beta })$$ (10)

and covariance matrix

$${\sum }_{\mathcal{I}}=\left(\begin{array}{cc}{\sigma }_b^2\hfill & {\sigma }_{b\beta }\hfill \\ \multicolumn{1}{c}{{\sigma }_{b\beta }}& {\sigma }_{\beta }^2\hfill \end{array}\right).$$ (11)

When a Rasch model is employed for response indicators, the model can be identified by setting the expectations of $\theta$ and $\tau$ to zero.

Assuming joint distributions for person and item parameters yields the following likelihood:

$$\begin{array}{l}L=\prod _{i=1}^N\prod _{j=1}^{k_{to{t}_i}}p(u_{ij}|b_j,{\theta }_i)f(t_{ij}|{\beta }_j,{\tau }_i,\alpha )p{(k_{Q_i}|{\zeta }_i)}^{c_i}S{(k_{C_i}|{\zeta }_i)}^{1-c_i}\\ g({\theta }_i,{\tau }_i,{\zeta }_i|{\mathrm{\mu }}_{\mathcal{P}},{\Sigma }_{\mathcal{P}})h(b_j,{\beta }_j|{\mathrm{\mu }}_{ℐ},{\Sigma }_{ℐ}).\end{array}$$ (12)

The first four terms incorporate the assumption of conditional independence of responses, RTs, and number of reached items before quitting given the second-order variables in the model. The third and fourth terms take the right censoring of quitting behavior into account: For examinees who quit the assessment ($c_i=1$), the probability that examinee $i$ quits the assessment after attempting $k_{Q_i}$ items as given by Equation 4 contributes to the likelihood function. For examinees with unobserved quitting behavior ($c_i=0$), the likelihood function considers the probability that examinee $i$ will continue the assessment beyond the censoring position $k_{C_i}$ as given by the survival function in Equation 5. $g({\theta }_i,{\tau }_i,{\zeta }_i|{\mathbf{\mu }}_{\mathcal{P}},{\sum }_{\mathcal{P}})$ and $h(b_j,{\beta }_j|{\mathbf{\mu }}_{\mathcal{I}},{\sum }_{\mathcal{I}})$ denote the multivariate normal densities of the person and item parameters, respectively. To facilitate estimation of the SA + Q model, we employ Bayesian estimation techniques.

Parameter Recovery

We conducted a simulation study to investigate whether true parameter values can satisfactorily be recovered in estimation under realistic conditions. The SA model and extensions thereof have been shown to yield good parameter recovery under realistic conditions (e.g., Fox & Marianti, 2016; Molenaar, Tuerlinckx, & van der Maas, 2015; Pohl et al., 2019). We therefore focused especially on possible challenges for estimation imposed by censoring of quitting behavior and the resulting data sparseness on $k_{Q_i}$.

Data Generation

Data were generated using R Version 3.5.1 (R Development Core Team, 2017). We employed the SA + Q model as the data-generating model. Using the mvrnorm function from the MASS package (Venables & Ripley, 2002), person and item parameters were randomly drawn from multivariate normal distributions with variances and covariances set to values similar to those of the data application reported below. Population values of the data-generating model are reported in Table 2. We employed a Rasch model for the item responses and set time discrimination parameter for all items to $\alpha$ = 1.75 (see, e.g., van der Linden, 2007).

Table 2.

Population Parameters of the Data-Generating Model

Person parameters					Item parameters
	$\theta$	$\tau$	$\zeta$	${\mathbf{\mu }}_{\mathcal{P}}$		$b$	$\beta$	${\mathbf{\mu }}_{\mathcal{I}}$
$\theta$	1.00			0.00	$b$	1.50		0.00
$\tau$	−.40	.10		${\mu }_{\tau }$	$\beta$	.40	0.25	0.00
$\zeta$	−.15	.25	0.65	${\mu }_{\zeta }$
Missingness mechanisms
		Quitting			Speed and quitting
$K$	%NR	%Q	${\mu }_{\tau }$	${\mu }_{\zeta }$	%Q	${\mu }_{\tau }$	${\mu }_{\zeta }$
20	2.5	8.15	−2.50	4.15	4.28	−3.75	4.40
	5	15.08	−2.50	3.85	7.50	−3.85	4.15
	10	26.49	−2.50	3.50	12.75	−4.00	3.85
40	2.5	7.83	−2.50	4.85	5.30	−3.75	5.00
	5	14.50	−2.50	4.55	7.20	−3.85	4.85
	10	28.44	−2.50	4.15	12.60	−4.00	4.55

Note.$N$: number of examinees; $K$: number of items; %NR: overall missingness rate due to not-reached items; %Q: percentage of examinees quitting; $\theta$: ability; $\tau$: speed; $\zeta$: test endurance; $b$: item difficulty; $\beta$: time intensity; ${\mathbf{\mu }}_{\mathcal{P}}$ and ${\mathbf{\mu }}_{\mathcal{I}}$ give mean vectors of person and item parameters, respectively. Mean speed ${\mu }_{\tau }$ and the mean test endurance ${\mu }_{\zeta }$ are varied in the simulation design to control the amount of mechanisms underlying not-reached items.

Missing values were induced based on (a) cumulative RTs across item positions and (b) the number of reached items before quitting. Cumulative RTs give the time passed when the respective item is responded to. The number of reached items before quitting was generated for each examinee according to the Poisson lognormal model for the quitting process. All items with either a cumulative RT exceeding the time limit or whose position exceeded the number of reached items before quitting were assumed to be not reached and coded as missing.

To evaluate the effects of censoring of quitting behavior, we considered multiple censoring mechanisms and varied four factors that are relevant for data sparseness:

• (a) The sample size ($N=350$; $N=700$), representing low and medium sample sizes per item encountered in LSAs with balanced incomplete block designs (see Gonzalez & Rutkowski, 2010)

• (b) The test length ($K=20$ with $T_{\lim }=1,800\mathrm{s}$; $K=40$ with $T_{\lim }=6,600\mathrm{s}$)

• (c) The rate of NRIs (2.5%, 5%, 10%), reflecting the upper three quarters of a typical range of percentages of NRIs. For instance, in the PISA 2012 computer-based assessment, percentages of NRIs across booklets ranged from 0.42% to 11.19% (OECD, 2014)

• (d) The missingness mechanisms underlying NRIs (NRIs caused solely by quitting; half of the NRIs resulting from quitting and half from lack of speed).¹ While the former represents a testing condition without or a very generous time limit (e.g., as in PIAAC), the latter represents a more speeded testing situation where some examinees run out of time before reaching the end of the test or quitting (e.g., as in PISA). We included these conditions to (a) assess whether the proposed model yields unbiased and efficient parameter estimates under various testing situations and (b) to disentangle possible effects of different censoring mechanisms on estimation accuracy and efficiency. Under the first condition, the number of reached items $k_{Q_i}$ is censored due to test length. Under the second condition, censoring occurs due to both test length and lack of speed.

Under this design, conditions with 5% (2.5%) missing values and no censoring due to speed display the same missingness rates due to quitting as conditions with 10% (5%) missing values with half of it going back to quitting. This allows to assess whether the model performs differently when—for the same amount of information available on test endurance—the overall missingness rate increases and $k_{Q_i}$ is exposed to different censoring mechanisms. We controlled the amount and types of NRIs by varying the expectations of test endurance ${\mu }_{\zeta }$ and speed ${\mu }_{\tau }$ (see Table 2). Note that low rates of missingness due to quitting might go back to relatively high proportions of examinees exhibiting such behavior. This becomes evident in Table 2, displaying the corresponding average proportions of simulated examinees exhibiting quitting behavior across all cells of the simulation design. In total, the simulation design led to 2 × 2 × 3 × 2 = 24 conditions. For each cell of the simulation design, we generated 100 data sets.

Estimation Procedure

We employed Bayesian estimation with Gibbs sampling. All analyses were conducted in JAGS Version 4.3.0 (Plummer, 2003) using the rjags package (Plummer, 2016) for R Version 3.5.1 (R Development Core Team, 2017). Settings for noninformative priors were chosen following recommendations provided by Fox (2010) and Gelman and Hill (2007).

For person and item parameter variances and covariances, we employed inverse Wishart priors with

$${\sum }_{\mathcal{P}}~\mathrm{I}{\mathrm{W}}_{3+1}({\mathbf{I}}_3)$$ (13)

and

$${\sum }_{\mathcal{I}}~\mathrm{I}{\mathrm{W}}_{2+1}({\mathbf{I}}_2),$$ (14)

where ${\mathbf{I}}_3$ and ${\mathbf{I}}_2$ represent identity matrices of dimension 3 and 2, respectively. These are default prior settings for inverse Wishart priors implemented in statistical software for Bayesian analyses (van Erp, Mulder, & Oberski, 2017). Note that inverse Wishart priors tend to be informative about variances when these are close to zero and the sample size is small (Alvarez, Niemi, & Simpson, 2014; Schuurman, Grasman, & Hamaker, 2016). Since usually the number of items is small, for item parameter variances and covariances, prior settings will have a larger impact.

We set ${\mu }_{\theta }$ and ${\mu }_{\tau }$ to zero for model identification. For the remaining item and person parameter means, we chose noninformative normal priors with mean zero and variance ${1000}^2$. A noninformative gamma prior with shape 0.5 and rate $\frac{1}{{100}^2}$ was employed for squared time discrimination ${\alpha }^2$. JAGS code for the SA + Q model is provided in the appendix.

Each generated data set was analyzed running three Markov chain Monte Carlo chains with 100,000 iterations each. We employed a thinning factor of 5 and discarded the first 40,000 iterations as burn-in, saving 36,000 iterations as a sample of the posterior distribution. We determined the number of iterations in preanalyses, inspecting potential scale reduction factor (PSRF) values, trace plots, and effective sample sizes. In the case of nonconvergence (i.e., PSRF values higher than 1.10; Gelman & Shirley, 2011), we increased the number of iterations by 50,000 per chain out of which 30,000 were discarded as burn-in. This procedure was repeated up to 250,000 iterations per chain in total.

Evaluation Criteria

We evaluated statistical performance in terms of convergence, bias in and efficiency of parameter estimates, as well as coverage of the true parameter values by 95% highest density intervals. Convergence was assessed on the basis of PSRF values, with PSRF values below 1.10 being considered acceptable (Gelman & Rubin, 1992; Gelman & Shirley, 2011). Coverage between .91 and .98 was considered good (Muthén & Muthén, 2002).

Results

Convergence

Table 3 gives the proportions of replications converging after 100,000 up to 250,000 iterations across all cells of the simulation design. Reaching convergence was more challenging under conditions with low quitting rates. This effect was more pronounced in conditions with larger number of items. Accordingly, conditions with $K=40$ and an overall missingness rate of 2.5% out of which half went back to quitting were most challenging with respect to convergence, with up to 58% of the replications not converging after 250,000 iterations. The number of iterations needed to reach convergence decreased rapidly with higher quitting rates, such that under conditions with 10% missingness solely due to quitting, at least 95% of the replications converged after 250,000 iterations at most. No considerable differences concerning convergence could be observed for the same amount of missing values due to quitting when either missing values due to speed were present or not.

Table 3.

Proportion of Replications Converging After 100,000 to 250,000 Iterations

				Iterations
$K$	$N$	Percent NR	Mechanisms	100,000	150,000	200,000	250,000
20	350	2.5	Quitting	.13	.31	.52	.64
			Speed & quitting	.07	.26	.38	.46
		5	Quitting	.28	.55	.73	.85
			Speed & quitting	.13	.31	.44	.60
		10	Quitting	.46	.78	.91	.96
			Speed & quitting	.25	.50	.66	.81
	700	2.5	Quitting	.08	.32	.53	.64
			Speed & quitting	.08	.22	.35	.48
		5	Quitting	.29	.50	.67	.77
			Speed & quitting	.12	.26	.44	.51
		10	Quitting	.59	.87	.97	.99
			Speed & quitting	.17	.35	.54	.75
40	350	2.5	Quitting	.07	.23	.41	.57
			Speed & quitting	.03	.11	.29	.42
		5	Quitting	.20	.43	.62	.74
			Speed & quitting	.10	.20	.41	.55
		10	Quitting	.55	.79	.95	.98
			Speed & quitting	.13	.28	.51	.64
	700	2.5	Quitting	.05	.25	.40	.57
			Speed & quitting	.10	.23	.38	.47
		5	Quitting	.19	.37	.62	.77
			Speed & quitting	.10	.24	.35	.45
		10	Quitting	.57	.84	.92	.97
			Speed & quitting	.14	.34	.59	.71

Note. % NR: Overall missingness rate due to not-reached items; quitting and speed & quitting denote conditions under which all not-reached items go back to quitting and not-reached items occurred due to both lack of speed and quitting, respectively; $K$: number of items; $N$: number of examinees.

Parameters typically yielding high PSRF values were test endurance mean and variance estimates. This is due to the censoring of quitting behavior, with the distribution of test endurance needing to be extrapolated based on information from examinees assumed to belong to the lower quartiles of the distribution. When quitting behavior is observable only for few examinees, information on the distribution of test endurance is sparse, affecting convergence. Replications that did not converge were excluded from further analyses.

Coverage

Coverage values for all parameter types and conditions are available in the Supplemental Material (available online). For item parameter variances, covariances, and means, coverage was satisfactory across all conditions. Coverage values falling below .91 occurred rarely and the lowest coverage value was still as high as .82. For person parameters, when either the number of examinees or items was sufficiently high ($K=40$ or $N=700$), coverage fell below .91 only for test endurance variance $\mathrm{var}(\zeta )$ and mean ${\mu }_{\zeta }$ estimates under conditions with missingness rates due to quitting below 5% (i.e., under conditions with less than 10% NRIs due to speed and quitting as well as under conditions with an overall missingness rate below 5% only due to quitting). Under these conditions, for $\mathrm{var}(\zeta )$, the lowest observed coverage values were .88 and .67 for conditions with $K=20$ and $K=40$, respectively. For ${\mu }_{\zeta }$, the lowest observed coverage values were .76 and .75 for conditions with $K=20$ and $K=40$, respectively, under conditions with less than 5% missingness caused by quitting.

Parameter Estimation

To evaluate bias in and efficiency of parameter estimates, we assessed the median along with 90% ranges of the means of the posterior distribution. Figures 2 and 3 depict results for person parameter variances and covariances as well as mean test endurance posterior means.² Across all conditions, median person parameter variance and covariance estimates were close to the true data-generating values. The only exception were parameters concerning the distribution of test endurance (${\mu }_{\zeta }$ and $\mathrm{var}(\zeta )$), which were sensitive to bias under conditions with missingness rates due to quitting below 5% as well as under conditions with few items. Under conditions with $K=20$, median parameter estimates of $\mathrm{var}(\zeta )$ ranged from 0.53 up to 0.71 as compared to the true value of 0.65. Bias for ${\mu }_{\zeta }$ under conditions with $K=20$ was less severe. Under conditions with $N=350$ and 10%, respectively, 1.25% missingness due to quitting, median parameter estimates of 3.55 and 4.19 as compared to the data-generating values of 3.50 and 4.40 were observed. Under conditions with $K=40$ and a missingness rate due to quitting of at least 5%, median estimates of $\mathrm{var}(\zeta )$ and ${\mu }_{\zeta }$ were well recovered. Median bias in parameter estimates did not vary largely across the sample sizes under consideration. Nevertheless, variability of parameter estimates decreased with increasing sample size.

Figure 2.

Medians and 90% ranges of person parameter variance and covariance estimates over all 100 replications per condition. The dashed horizontal line indicates the respective true parameter. Note that y-axes differ in scale. $\theta$: ability; $\tau$: speed; $\zeta$: test endurance; $N$: number of examinees; $K$: number of items; NRIs: not-reached items; quitting and speed & quitting denote conditions under which all not-reached items go back to quitting and not-reached items occurred due to both lack of speed and quitting.

Figure 3.

Medians and 90% ranges of mean test endurance ${\mu }_{\zeta }$ over all 100 replications per condition. The dashed horizontal line indicates the respective true parameter. Plots are organized according to the data-generating values employed to achieve missingness rates due to quitting ranging from 1.25% (speed & quitting, 2.5%) to 10% (quitting, 10%). Note that y-axes differ in scale. $N$: number of examinees; $K$: number of items; % NR: overall missingness rate due to not-reached items; quitting and speed & quitting denote conditions under which all not-reached items go back to quitting and not-reached items occurred due to both lack of speed and quitting, respectively.

Results for bias and efficiency of item parameter means, variances, and covariances are given in the Supplemental Material (available online). Due to the small number of items, estimates were shrunken toward the prior mean of the inverse Wishart prior (see Alvarez et al., 2014; Daniels & Kass, 1999).

Subsequent Analyses

In subsequent analyses, we aimed at investigating whether the recovery of $\mathrm{var}(\zeta )$ under conditions with few items ($K=20$) improves with an increasing amount of NRIs. To do so, we increased the missingness rate under the condition with $K=20$, $N=350$ and missingness solely caused by quitting to 20% by setting ${\mu }_{\zeta }$ to 3. We employed 100 replications and analyzed all generated data sets employing three chains with 250,000 iterations each. No convergence issues occurred. Indeed, the additional condition yielded a median estimate of $\mathrm{var}(\zeta )$ of 0.67 as compared to the true value of 0.65, thereby supporting the conclusion of the simulation study that test endurance variances can be better recovered when data sparseness on $k_{Q_i}$ is less severe.

Empirical Example

We used data from the Spanish sample of the PISA 2015 assessment to illustrate the use of the SA + Q model for the understanding of the occurrence of NRIs. We analyzed data from examinees who were administered Science Cluster Number 7 at the second position out of four 30-minute blocks. For reasons of simplicity, we eliminated examinees who showed item omissions from further analyses. The final sample consisted of 326 examinees responding to 17 items. The data set under consideration displayed a missingness rate of 7.13%, going back to those 21.47% of examinees who did not reach the end of the cluster. The majority of examinees with NRIs (58.57%) reached all but four items at most.

Total Response Time Distributions

In a first step, to get a better understanding of possible mechanisms, we followed Pohl et al. (2019) and examined distributions of total RT $t_{{\mathrm{tot}}_i}$. Figure 4 displays $t_{{\mathrm{tot}}_i}$ as a function of the number of NRIs. The time limit $T_{\lim }$ of 1,800 seconds is marked with a dashed horizontal line. The results suggest that the time limit was enforced with varying rigor by test administrators, since for some examinees, $t_{{\mathrm{tot}}_i}$ exceeded the time limit. More importantly, $t_{{\mathrm{tot}}_i}$ varied largely across examinees with NRIs. Such variation was not to be expected under conditions where NRIs occurred entirely due to lack of speed, where all $t_{{\mathrm{tot}}_i}$ associated with NRIs should be close to $T_{\lim }$. Instead, the fact that $t_{{\mathrm{tot}}_i}$ associated with NRIs was close to $T_{\lim }$ only for some examinees, while it was considerably below $T_{\lim }$ for others can be understood as evidence that different mechanisms—that is, lack of speed and premature quitting—were underlying NRIs.

Figure 4.

Total response time distributions for PISA (Programme for International Student Assessment) Science Cluster Number 7 administered in Spain, by number of not-reached items. The dashed horizontal line marks the time limit $T_{\lim }$ of 1,800 seconds.

Investigating the Occurrence of Not-Reached Items

For simplicity, when classifying examinees as quitters, we ignored that in PISA time limits were enforced with varying rigor. To deal with the fact that RT was not recorded for the last item seen when no response has been generated, we employed a heuristic approach and adjusted the decision boundary for $t_{{\mathrm{tot}}_i}$ associated with NRIs by a typical item-level RT. We classified NRIs as due to quitting when $t_{\mathrm{to}{t}_i}$ fell below the time limit by less than the 90th percentile of RTs across all items and examinees—that is, $1,800\mathrm{s}-66.97\mathrm{s}=1,733.03\mathrm{s}$.³ Doing so led to classifying 68 out of 70 examinees who did not reach the end of the test as quitters. We employed the SA + Q model to analyze the data. With 250,000 iterations per chain, no PSRF values below 1.10 were encountered.

Results are displayed in Table 4. The negative correlation between ability $\theta$ and speed $\tau$ as well as between ability $\theta$ and test endurance $\zeta$ indicates that more able examinees tended to display lower general working speed and showed lower test endurance. That is, more able examinees were more likely to display NRIs due to both reaching the time limit and quitting. Note that the highest density interval for the correlation between $\theta$ and $\zeta$ includes zero, which means that the correlation is not credibly different from zero. The positive correlation between speed $\tau$ and test endurance $\zeta$ indicates that examinees who worked faster had the tendency to generate answers to more items before quitting the assessment. Furthermore, the fact that $\tau$ and $\zeta$ are not highly correlated underlines that missingness due to speed and missingness due to quitting should be seen as different processes. ${\mu }_{\zeta }$ gives the expectation of the mean logarithmized number of items reached before quitting. The value of 3.52 corresponds to 33.78 items.

Table 4.

Variances and Means of as Well as Correlations Among Person and Item Parameters.

Person parameters
	$\theta$	$\tau$	$\zeta$	${\mathbf{\mu }}_{\mathcal{P}}$
$\theta$	1.13 [0.87, 1.39]
$\tau$	−.37 [–.50, –.24]	0.07 [0.06, 0.09]
$\zeta$	−.16 [–.35, .03]	.30 [.12, .47]	0.62 [0.34, 0.94]	3.52 [3.31, 3.74]
Item parameters
	$b$	$\beta$	${\mathbf{\mu }}_{\mathcal{I}}$
$b$	1.41 [0.59, 2.47]		−0.36 [–0.99, 0.18]
$\beta$	.41 [.02, .76]	0.21 [0.09, 0.37]	4.14 [3.94, 4.39]

Note. Highest density intervals are given in square brackets. $\theta$: ability; $\tau$: speed; $\zeta$: test endurance; $b$: item difficulty; $\beta$: time intensity; ${\mathbf{\mu }}_{\mathcal{P}}$ and ${\mathbf{\mu }}_{\mathcal{I}}$ give mean vectors of person and item parameters, respectively.

Discussion

The SA + Q model proposed in this article allows to disentangle and simultaneously model different missing data mechanisms underlying NRIs. Namely, the SA + Q model distinguishes between NRIs stemming from lack of speed and NRIs due to quitting and, thereby, allows to substantively meaningfully describe different processes underlying NRIs. This is achieved by further integrating research on missing data and research on RTs (see Pohl et al., 2019). The SA + Q model considers RT data to handle NRIs by (a) utilizing the additional information contained in RTs to distinguish between examinees displaying NRIs due to lack of speed and those who quit and (b) extending van der Linden’s SA model by a Poisson lognormal survival model describing the quitting process in terms of examinee test endurance. The SA + Q model can be employed to model the occurrence of NRIs under conditions where all NRIs go back to quitting as well as the occurrence of NRIs occurring due to both lack of speed and quitting.

The SA + Q model represents a refined model-based approach for dealing with nonignorable missing data. As delineated above, previously suggested model-based approaches for NRIs (Glas & Pimentel, 2008; Rose et al., 2010; Pohl et al., 2019) rely on the assumption of a single missingness mechanism. The SA + Q model complements model-based approaches for NRIs in that it considers that multiple mechanisms can underlie their occurrence. As such, the SA + Q model overcomes limitations of current state-of-the-art approaches for NRIs.

We employed data from PISA 2015 to illustrate how the approach can provide insights into the processes underlying NRIs. We showed that there is strong evidence that NRIs in LSAs indeed can be attributed to different missingness processes. In this context, the SA + Q model supports a more fine-grained understanding of the occurrence of NRIs by further assessing examinee characteristics associated with test endurance. As such, the SA + Q model can be used to evaluate and inform substantive theories on test-taking behavior and strategies.

The model gives reasonable estimates under conditions with a missingness rate of at least 5% due to quitting (or approximately 15% of examinees exhibiting quitting behavior) and a higher number of items ($K=40$). Since the SA + Q model estimates test endurance based on information on the number of reached items before quitting, the number of examinees quitting the assessment might be of greater importance for retrieving unbiased estimates than the missingness rate due to quitting. Generally, a higher number of iterations is needed when little information on quitting behavior is available. Note that model-based approaches yield ability estimates considerably different from those retrieved when ignoring nonignorable missing values only under high missingness rates (Pohl et al., 2014; Rose, 2013; Rose et al., 2010), such that the application of the SA + Q might be useful mainly under conditions with higher rates of NRIs.

Due to the model’s complexity, we recommend keeping measurement models as simple as possible—for example, by employing a Rasch model for item responses and/or fixing time discrimination parameters in the measurement model of RTs to be equal across items. More complex measurement models that might better fit the data at hand can be incorporated in the SA + Q framework. For RTs, for instance, alternative parameterizations have been suggested assuming distributions of RTs different from lognormal (Klein-Entink, van der Linden, & Fox, 2009) or introducing additional parameters that reflect the way an item distinguishes between examinees of different speed levels (Klein-Entink, Kuhn, Hornke, & Fox, 2009). Note, however, that adding additional model complexity by choosing more complex measurement models for responses and/or RTs might further challenge estimation of the SA + Q model and increase the number of items or examinees needed to achieve convergence as well as unbiased and efficient parameter estimation.

Limitations and Future Research

While the SA + Q model allows to incorporate different missingness mechanisms, it heavily relies on extrapolation of the distribution of the number of reached items before quitting and the underlying test endurance variable. When the majority of examinees manages to reach the end of the test, the number of reached items before quitting is strongly affected by right censoring. Thus, as it is the case with previously developed latent model-based approaches for NRIs (e.g., Glas & Pimentel, 2008), the distribution of reached items before quitting is extrapolated from observations assumed to belong to lower quartiles of the distribution. This renders it difficult to assess whether distributional assumptions are met. In the simulation study for evaluating estimability, rather low missingness rates due to quitting went back to a relatively high proportion of examinees exhibiting such behavior. For instance, under conditions with a missingness rate of 10% due to quitting, quitting behavior was observable for roughly a quarter of examinees. This, however, must not be the case in real data. It might well be that high NRI rates can be attributed to few examinees quitting the assessment at early stages. The capability of the SA + Q model to yield an accurate description of the quitting process under such conditions remains to be evaluated. Likewise, it still remains to be assessed whether statistical performance of the SA + Q model can be improved under conditions with large sample sizes, as often encountered on the country level in LSAs. It might well be that the SA + Q model performs well under conditions with a low proportion of examinees exhibiting quitting behavior when the absolute number of examinees is sufficiently high. Until then, the requirements concerning missingness rates due to quitting established on the basis of the simulation study may serve as lower boundaries. In addition, for count data on the test level, distributions different from lognormal, such as a gamma distribution, have been suggested for the random, person-specific mean of the Poisson distribution (Doebler & Holling, 2016; Jansen, 1994). Further research and possibly also experimental studies are needed to evaluate the appropriateness of different distributional assumptions for test endurance.

Furthermore, it remains to reason whether the quitting process can sufficiently be described by only considering item positions. First, it might well be that rather than the number of attempted items, passed test time better measures test endurance. As long as time intensities do not show large differences across items, item positions might serve as a proxy for passed test time (Fox & Marianti, 2016). Otherwise, the SA + Q model might not be able to sufficiently capture the quitting process. Second, subpopulations with different quitting strategies might exist. For instance, while some examinees might quit due to lack of motivation, others might quit when they consider the test too difficult or even out of frustration when noticing that the time remaining is not sufficient to complete the assessment. Likewise, there might be qualitative differences in quitting mechanisms between examinees who quit at earlier and later stages of the assessment. To meaningfully incorporate different forms of quitting behavior into the model, further substantive research is needed to describe and understand these mechanisms.

Moreover, the SA + Q model assumes stationarity of speed. This assumption is reasonable when tests are administered with generous time limits, such that examinees are unlikely to run out of time (van der Linden, 2007). However, the presence of NRIs due to lack of speed indicates that the allocated time might not have been sufficient for all examinees. When examinees perceive their current speed level as being insufficient to reach the end of the test, they might try to adjust their pace (Yamamoto & Everson, 1997), rendering the stationarity assumption implausible. In future studies, it seems therefore necessary to allow for within-person variation of speed in the SA + Q model (see Fox & Marianti, 2016, for an extended model).

In general, estimation of the SA + Q model was found to be rather challenging. Severe convergence issues were encountered under conditions with less than 10% missingness due to quitting. When convergence was reached, this oftentimes was only the case with high numbers of iterations. Future research should therefore address facilitating the estimation procedure of the SA + Q model.

In the empirical example, we encountered conditions that pose further challenges to adequate modeling of NRIs and provided heuristic solutions to address these issues. Future research is needed on how to better deal with (a) the fact that oftentimes no RT information is available for the last item seen when no response has been generated as well as (b) varying enforcement of time limits when applying the SA + Q framework. In addition, for the SA + Q model to be readily applicable to empirical data, it needs to be considered that NRIs are often not the only source of missingness going back to examinee behavior. In most LSAs, both NRIs as well as item omissions can be encountered (Pohl et al., 2014). To additionally handle (nonignorable) omission processes, the SA + Q model could be combined with model-based approaches for item omissions (e.g., O’Muircheartaigh & Moustaki, 1999; Rose, 2013; Ulitzsch et al., 2019).

Appendix

JAGS Code

Figure A1.

JAGS code for the speed–accuracy + quitting model.$N$: number of persons; $K$: number of items. $\mathbf{U}$ and $\mathbf{RT}$ are $N$ by $K$ matrices containing the item responses and associated response times, $\mathbf{kQ}$ and $\mathbf{kC}$ are vectors of length $N$ containing the number of reached items before quitting and the censoring item position. $\mathbf{isCensored}$ contains information on observed quitting behavior for each examinee $i$ and takes the values 1 and 2 for $c_i=1$ and $c_i=0$, respectively. $\mathbf{MP}$ and $\mathbf{MI}$ represent identity matrices of sizes 3 and 2.