MCMC Z-G

Abstract

In recent years, there has been a surge of interest in measuring noncognitive constructs in educational and managerial/organizational settings. For the most part, these noncognitive constructs have been and continue to be measured using Likert-type (ordinal response) scales, which are susceptible to several types of response distortion. To deal with these response biases, researchers have proposed using forced-choice format, which requires respondents or raters to evaluate cognitive, affective, or behavioral descriptors presented in blocks of two or more. The workhorse for this measurement endeavor is the item response theory (IRT) model developed by Zinnes and Griggs (Z-G), which was first used as the basis for a computerized adaptive rating scale (CARS), and then extended by many organizational scientists. However, applications of the Z-G model outside of organizational contexts have been limited, primarily due to the lack of publicly available software for parameter estimation. This research effort addressed that need by developing a Markov chain Monte Carlo (MCMC) estimation program, called MCMC Z-G, which uses a Metropolis-Hastings-within-Gibbs algorithm to simultaneously estimate Z-G item and person parameters. This publicly available computer program MCMC Z-G can run on both Mac OS^® and Windows^® platforms.

In recent years, there has been a surge of interest in measuring noncognitive constructs in educational and managerial/organizational settings. Examples include personality, vocational interests, motivation, job satisfaction, job performance, and person-environment fit. For the most part, these noncognitive constructs have been and continue to be measured using Likert-type (ordinal response) scales, which are susceptible to several types of response distortion (Brown & Maydeu-Olivares, 2011). In personnel selection, a key concern is faking good, also known as impression management or socially desirable responding (Stark, Chernyshenko, & Drasgow, 2005). In student or employee performance appraisal, a primary concern is rating scale errors, which include severity, leniency, and central tendency errors (Borman et al., 2001), that tend to diminish variance in scores and undermine decision-making and developmental feedback efforts.

To deal with these response biases, researchers have proposed using forced-choice format, which requires respondents or raters to evaluate cognitive, affective or behavioral descriptors presented in blocks of two or more. The respondent’s task is to choose the descriptor in each block that better characterizes himself or herself, or, alternatively a ratee who is the external target of the evaluation (Stark & Drasgow, 2002).

Borman and colleagues (2001) proposed using a unidimensional forced-choice item response theory (IRT) model, developed by Zinnes and Griggs (Z-G; 1974), as the basis for a computerized adaptive employee appraisal system known as CARS II. To support this application, Stark and Drasgow (2002) developed statement and person estimation methods for the Z-G model, as well as a computerized adaptive testing algorithm (Stark & Chernyshenko, 2011; Stark & Drasgow, 1998) to enhance measurement precision. Since then, the Z-G model has been used for some large personality testing applications in civilian and military settings (e.g., the U.S. Navy Computerized Adaptive Personality Scales [NCAPS]; Houston, Borman, Farmer, & Bearden, 2006; Stark et al., 2014) and, conceptually, it is suitable for assessing a wide array of noncognitive constructs involving “self-” and “other” reports. Applications of the Z-G model outside of organizational contexts have been limited, however, by the lack of publicly available software for parameter estimation. This research effort addressed that need by developing a Markov chain Monte Carlo (MCMC) estimation program, called MCMC Z-G, which uses a Metropolis-Hastings-within-Gibbs algorithm (Hastings, 1970; Metropolis, Rosenbluth, Rosenbluth, Teller, & Teller, 1953) to simultaneously estimate Z-G item and person parameters (Seybert, Lee, Stark, & Chernyshenko, 2014). The MCMC Z-G program can handle missing data, and there is no limit on the number of items or respondents.

Like the MCMC GGUM (Generalized Graded Unfolding Model; Roberts, Donoghue, & Laughlin, 2000) program developed by Wang, de la Torre, and Drasgow (2014), the MCMC Z-G program is run by a syntax file, which specifies data characteristics (e.g., file location, pair mapping, initial values, number of respondents, items, statements), MCMC estimation specifications (e.g., number of chains, chain length, and burn-in iterations), and the standard deviations for sampling candidates. The output file includes the means and standard errors of the item and person parameters from each estimation chain, as well as the averaged values across the chains. Acceptance rates are reported for item parameter estimation.

The executable MCMC Z-G installer is a C++ application having a user-friendly graphical interface that will run on both Mac OS^® and Windows^® platforms. The installers and corresponding documentation, such as exemplary syntax files and data sets, are available from http://computationalpsychology.org/resources/. It is free of charge for noncommercial use only.