Obtaining Comparable Measurement of Midlife Cognitive Functioning from Disparate Cognitive Tasks

Abstract

Objective:

National surveys of health and aging increasingly collect data via multiple modes, including via face-to-face, phone, and the web. However, mode effects—differences in measurement due to how the data are collected, rather than to underlying differences in cognition—undermine comparability in performance for different segments of samples, impeding analysis of cognitive function in the full sample. This paper reports on the construction and evaluation of a measure of general cognitive functioning comparable across diverse phone and web cognitive tasks for the 2021 follow-up of the High School & Beyond Study of 1980, a nationally representative survey of individuals who were U.S. high school sophomores and seniors in 1980.

Method:

We used a hierarchical explanatory Item Response Theory (IRT) model to equate general cognition scores across phone and web samples based on the relationship between midlife cognition and covariates from midlife and high school.

Results:

The full model replicated task scores from unidimensional models, and the explanatory component of the model did not bias general cognition scores, indicating that the model was a faithful representation of patterns of cognitive task performance. We found no evidence of a violation of our assumption of the independence of regression coefficients and survey mode.

Conclusions:

This paper provides an empirical example of how to construct a psychometrically sound cross-mode cognitive score from data collected through multiple modes. Given the increasing demand for cost-effective large-scale survey design, our method may find greater application in other nationally representative aging studies.

INTRODUCTION

With an aging population, the critical importance of neuropsychological assessment is growing beyond the treatment of specific neurological conditions to inform a broader policy context concerning how to support older adults experiencing cognitive decline. Measuring cognitive functioning is critically important in clinical and nonclinical settings for understanding the progression of cognitive impairment both due to specific neurological conditions and due natural aging processes. In clinical settings, this is accomplished through lengthy periodic assessments targeting specific domains thought to be at risk of impairment given other symptoms with which individuals may present. The purpose of such assessments in clinical settings is to optimize preventive treatment. These extensive in-person neurocognitive assessments are the gold-standard for measuring cognition, but due to the time and expertise required to administer them, such assessments are impractical for use in representative samples of the population at large. Detailed assessment of individuals in clinical settings is useful for understanding the development of those individuals’ specific pathologies; however, public policy discussions of how to support an aging population require understanding population-wide patterns, rather than the disease progression for specific individuals, and non-representative clinical samples do not generally provide unbiased estimates of population trends.

Most national surveys of health and aging must, due to cost considerations, collect data (including neurocognitive assessment) via multiple modes, including face-to-face, via phone, and via the web. A growing evidence base of controlled validation studies (Cullum et al., 2014; Fox-Fuller et al., 2022; Jacobsen et al., 2003; Yildirim et al., 2025) and systematic reviews (e.g., Marra et al., 2020) supports the use of remote neuropsychological assessment (teleneuropsychology) as a methodologically sound alternative to in-person assessment protocols for core cognitive domains in midlife dementia risk research, including global cognition, verbal learning and memory, attention and working memory, naming, and verbal fluency. Although initial validation efforts primarily targeted older adults and clinical cohorts (e.g., Camozzato et al., 2011; Wadsworth et al., 2018), these same domains are also critical for detecting subtle cognitive differences in midlife that may signal long-term dementia risk. Beyond psychometric validity, teleneuropsychology confers substantial logistical advantages for epidemiologic research: reduced participant burden, lower costs, expanded geographic reach, and improved inclusion of individuals otherwise excluded because of distance, time constraints, or caregiving and occupational responsibilities. These factors are essential for achieving representative midlife cohorts.

Web-based, self-administered cognitive tests are an important extension of phone- and video-based remote cognitive assessment (Gold et al., 2022; Perin et al., 2020; Troyer et al., 2014), offering scalability, lower cost, and the ability to collect data more frequently over time (Wesnes et al., 2017). Large studies of unsupervised online platforms (e.g., Morrison et al., 2015) show that composite scores from web-based batteries have moderate-to-good test–retest reliability, similar to in-person neuropsychological tests. Because these assessments can be completed at home on personal devices, they reduce participation barriers and staff burden while enabling repeated measurements.

The development of multiple new teleneuropsychological assessment modes provides unprecedented flexibility for large-scale data collection efforts that can meet the information needs of contemporary policy discussions. However, relying on multiple administration modes in a single study poses a substantial analytic challenge, as performance on cognitive tasks administered in-person may not match exactly performance on cognitive tasks administered by phone or web. Resulting mode effects—differences in measurement due to administration mode, rather than due to underlying differences in cognition—undermine comparability in performance for different segments of samples and thus degrade the capacity of the research community to estimate relationships among experiences, activities, health behaviors, environmental factors, and other attributes with cognitive function. With increasing cost pressure to replace in-person cognitive assessments with phone or web assessments, and especially with increasing interest in mixed-mode survey administration, detecting and adjusting for mode-based measurement artifacts has become an active topic of research (e.g., Al Baghal, 2019; Cernat et al., 2016; Faul et al., 2022; Smith et al., 2023; Stricker et al., 2019).

Prior research on mode effects in cognitive measurement has generally shown that self-administered web-based tasks resulted in higher scores than interviewer-administered phone or in-person tasks, but that differences between phone and in-person tasks are less substantial (Cernat et al., 2016; Domingue et al., 2023; McComb et al., 2011; Ofstedal et al., 2021; Saari et al., 2025). However, key differences in cognitive stimuli (such as visual versus auditory prompts) still have the potential to alter which cognitive functions are elicited by the tasks, as well as the task difficulty, resulting in differential measurement by mode. In many cases, researchers have attempted to administer parallel tasks in each mode (e.g., a verbal fluency task administered in-person and a verbal fluency task with the same instructions administered over the phone). However, given differences in the types of stimuli and responses possible in each mode, the best task for measuring cognition in one mode may not be parallel to the best task in another mode. For example, on the web, a visual paired association task may be excellent for measuring episodic memory, while via phone (which has no visual component), word list recall could possibly provide better measurement than a paired association task. In such cases, study designs based on the best tasks for each mode face the additional measurement challenge of having distinct (rather than parallel) tasks across modes, rendering measurement of cognition on a common scale across modes challenging. Whether using parallel or distinct tasks, mixed-mode studies of cognition need to assess and adjust for mode-based psychometric differences to permit analysis of cognition across the entire study sample.

In psychometrics, Item Response Theory (IRT; de Ayala, 2009; Lord, 1980) has long permitted the construction of common measurement scales based on diverse sets of cognitive tasks (i.e., items). Some aging studies have started to make use of IRT for measurement of cognition (Balsis et al., 2012, 2018; Umfleet et al., 2023; Ma et al., 2021; Crane et al., 2008; Mungas et al., 2003), including use to construct cognitive composite scores (Donohue et al., 2014; Gross et al., 2014), to equate test scores across countries (Gross et al., 2023; Hampton et al., 2023; Kobayashi et al., 2021) and changes in administration instructions (Gross et al., 2015), and to link assessments across modes (Gooch, 2015; Joffe et al., 2025). While in traditional neuropsychological assessments all participants receive exactly the same set of cognitive tasks, IRT permits study designs in which participants receive different subsets of tasks, provided that the study implements an acceptable linking design to assign tasks to participants. Acceptable linking designs rely on commonalities in subsets of items and participants: Two subsets of items are linked when they are administered to a common subset of participants, and two subsets of participants are linked when they share a common subset of items.¹ For example, participants who responded to verbal fluency tasks for words beginning with letters ‘F’ and ‘L’ can be linked with participants who responded to verbal fluency tasks for words beginning with ‘L’ and ‘C,’ because both sets of participants responded to the letter L task. Additionally, the tasks for letter ‘F’ and letter ‘C’ can be linked because letters ‘F’ and ‘L’ share participants and letters ‘L’ and ‘C’ share participants, creating a valid linking chain. Alternatively, items may be linked even if they do not share common participants, provided that the participants are drawn randomly from a common population (i.e., a common cognitive functioning distribution), such as via random assignment of participants to administration mode.

In our analysis of cognitive data from the 2021 follow-up to the High School & Beyond study (HS&B:80) described below, we faced a linking challenge: Due to cost efficiencies and fielding during the COVID-19 pandemic, the HS&B:80 study team selected a mixed phone- and web-based design, with distinct tasks in each mode. Originally, the study design randomly assigned participants to phone- or web-based administration; however, over the course of the study, participants who did not respond early were invited to participate in the alternative mode. As such, we could not consider the realized phone and web samples as random draws from the same population, due to selection effects—participants with certain characteristics associated with cognition might be more likely to respond via web than phone, or vice versa (e.g., Bech & Kristensen, 2009; Kelfve et al., 2020).

In our effort to obtain a valid common general cognition scale for participants from both modes, we developed an advanced variation on the common-distribution linking design: linking via a common conditional distribution, leveraging an assumption about the cross-mode stability of the relationship between cognition and characteristics not inherent to survey mode (e.g., sex, educational attainment, etc.) to place general cognition scores on a common scale regardless of the mode in which individuals completed the survey. This paper describes our method and assesses how well it provided common cognitive measurement (eliminating mode effects) for the HS&B:80 study. Given the successful application in the HS&B:80 study and the increasing demand for cost-effective large-scale survey design, our method may find greater application in other nationally representative aging studies.

METHOD

Design

HS&B:80 began as a nationally representative, longitudinal survey of sophomore and senior high school students in the United States in 1980. Four decades later, panel members are approximately 60 years old, and the panel has transitioned from an education study to an aging study (Grodsky et al., 2022). The most recent round of data collection was fielded in 2021, during the COVID-19 pandemic. Interviews were conducted remotely, with approximately 8,860 panel members participating by web and 4,430 panel members participating by phone. Making use of two survey modes facilitated reaching a larger portion of the entire panel within the budget constraints the HS&B:80 study team faced; however, multiple survey modes also presented a challenge for the measurement of cognitive functioning: Many cognitive tasks that are suitable for administration either visually (i.e., by web) or orally (i.e., by phone) do not translate easily into the other medium.

For example, the number span task asks participants to hear and repeat orally a sequence of digits. In a controlled laboratory environment, such a task could be automated for computer administration using pre-recorded stimuli and a microphone to record participants’ recitation; however, such a task would be difficult to administer in a field environment in which participants take a web-based survey with varying technology using their own devices. The task could, in principle, be translated for web administration into a visual/written, rather than auditory/vocal task; but while such a transformed task would assess similar cognitive functions as the original task, the differences in sensory mode would result in cognitive measurements that were not strictly equivalent or directly comparable.

As a result, the web and phone versions of the survey’s cognitive measurement module relied on two distinct sets of cognitive tasks (described below). Analyzing each set of tasks separately by mode of administration would dramatically limit the analytic sample size, reducing statistical power and increasing risks associated with divergent sample characteristics due to differences in nonresponse bias by mode. The data collection design included an overlap sample of about 1,670 participants who responded to both the web and phone cognitive tasks—if these tasks could be treated as measuring a single, unidimensional construct, the overlapping participants would yield a valid linking design. Unfortunately, initial psychometric analysis indicated that these cognitive tasks were too distinct to treat unidimensionally,² leaving no directly overlapping sample for any given dimension of cognition in the study. Having two distinct sets of tasks thus led to a measurement challenge: How could we construct a common measure of cognitive functioning that operated comparably for the entire sample, irrespective of survey mode, using two distinct sets of cognitive tasks? This paper describes the method we developed to address this challenge and analyzes the measurement properties of the resulting scale of general cognitive functioning.

Participants

All sample sizes in this paper have been rounded to the nearest 10, in accordance with National Center for Education Statistics (NCES) restricted-use data license policy. The full HS&B:80 panel consists of 26,820 individuals, approximately 9 percent of whom were deceased as of 2021. There were 13,980 participants in the 2021 data collection across all modes (including a last resort paper mode, which did not include cognitive measurement). As a nationally representative survey, HS&B:80 provides sampling weights to adjust population estimates for non-response. However, all analyses presented here are unweighted, because this paper is not concerned with population inferences but is rather focused on psychometric characteristics in the observed sample specifically.

Most participants responded by only one survey mode. However, all web participants who scored below a threshold on the web cognitive tasks and reported at least one memory complaint—as well as a random sample of web participants who did not meet these criteria—were invited to participate in a short phone follow-up interview, which included the phone cognitive tasks. We excluded cognitive data from participants who reported taking notes during the cognitive tasks. In the end, 3,990 participants had usable data from only phone cognitive tasks, 7,000 participants had usable data from only web cognitive tasks, and 1,670 participants had usable data from both phone and web cognitive tasks. Largely due to sampling design, participants with only web cognitive data were more likely to be White and more likely to have higher educational attainment than participants with only phone cognitive data (Table 1).

Table 1.

Selected demographic characteristics by cognitive task data availability

	Cognitive Data Group
	Phone only	Web only	Both
Count	3,990	7,000	1,670
Sex, reported in 2021
Male	48	44	44
Female	52	56	55
Other/Unknown	< 1	< 1	< 1
Race/Ethnicity
White	51	62	55
Black	22	13	17
Hispanic	22	19	22
Other	5	5	5
Educational Attainment
Unknown	< 1	< 1	< 1
High School or Less	26	14	13
Some College	28	30	34
College	31	35	34
Graduate Degree	14	20	19
Parental Education
Unknown	17	14	14
High School or Less	41	40	41
Some College	23	23	22
College	10	12	12
Graduate Degree	9	12	11

Note. Values are percentages of each category within cognitive data group.

Measures

Phone cognitive tasks included immediate and delayed word recall, forward and backward number span repetition, and phonemic and semantic verbal fluency. Web cognitive tasks included verbal and visual paired associates matching. Throughout this paper, “reliability” is computed as the variance of the latent variable (fixed to 1 for identification) minus the mean of the squared standard errors of the cognitive task scores.

The immediate and delayed word recall tasks constitute a measure of episodic memory (Baddeley et al., 2001; Murphy et al., 2007; Tulving, 2002; Wicklund et al., 2006). Participants were read a list of 10 words developed by the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD; Morris et al., 1988, 1989). Participants were then asked to freely recall as many of the words from the list as they could (immediate recall). In contrast with similar tasks in other studies, participants received only a single learning trial. After a several minute delay during which participants performed other cognitive tasks, participants were asked to recall the words a second time (delayed recall). Each of the 10 words was scored as recalled (correct) or not recalled (incorrect) for the immediate and the delayed recall tasks. Reliability for word recall scores was 0.75.

The number span task measures working memory (Baddeley, 2010; Reynolds et al., 2013; Treviño et al., 2021). In the forward number span task (Weintraub et al., 2018), participants were read a sequence of digits and asked to repeat the sequence in the order heard. The sequence length progressively increased, with up to two sequences for each sequence length. For each pair of sequences of the same length, participants skipped the second sequence in the pair if they correctly repeated the first sequence; participants also discontinued the task if they did not correctly repeat two sequences of a given length. The backward number span task proceeded similarly, except participants were asked to repeat the digits in reverse order. There were seven sequence lengths for each of the forward and reverse tasks (from 3 to 9 digits for the forward task and from 2 to 8 digits for the backward task, 28 sequences total). However, the first two sequence lengths for each task were easy and yielded very little variation in cognitive task performance, so we discarded these and used only the remaining 20 sequences. Each item was scored as correctly repeated, incorrectly repeated, or not administered. Items not administered to a given participant were excluded from the likelihood function for that participant. Reliability for number span scores was 0.79.

The verbal fluency tasks assess verbal processing, lexical access speed, and executive control (Shao et al., 2014; Wright et al., 2023). In these tasks, participants were asked to name as many words of a given criterion as they could within 60 seconds. For the letter fluency task, the criterion was words beginning with the letter F (Rosen, 1980; Weintraub et al., 2018); for the semantic category fluency task, the criterion was animals (Morris et al., 1989; Weintraub et al., 2018). We divided the 60-second response period into six 10-second blocks and counted in each block the number of non-repeated words correctly belonging to the given category. For each block, we collapsed the word counts into a smaller set of ordered categories that each comprised at least 10 percent of responses for the given item (from 3 to 5 categories, depending on the block). This yielded 6 ordinal items for the letter fluency task and 6 items for the semantic fluency task. Reliability for letter fluency scores was 0.71, and reliability for animal fluency scores was 0.65.

The Many Brains web-based verbal and visual paired associates tasks measure verbal and visual episodic memory (Baddeley et al., 2001; Singh et al., 2021). In the verbal task, participants were shown a series of 25 word pairs for 3 seconds each. After a delay (during which participants studied image pairs, below), participants were shown one word from a given word pair, with four word response options. Participants were asked to select which of the four options was originally paired with the given word. In the visual task, participants were presented with 24 pairs of images from a given category (e.g., two barns or two playgrounds) for 5 seconds each. After a delay (during which participants responded to the visual pairs test), participants were shown one image from a given image pair, with five image response options from the same category (e.g., five other barns). Participants were asked to select which of the five options was originally paired with the given image. Each item was scored as correctly or incorrectly matched. Reliability for verbal paired associates scores was 0.86, and reliability for visual paired associates scores was 0.79.

Analytic Strategy

The measurement model we used to obtain comparable scores across phone and web cognitive tasks is composed of three pieces, each of which addressed part of the measurement challenge: (1) IRT models for individual responses in each cognitive task, (2) a hierarchical model that described the correlations between tasks, and (3) an explanatory model that improved estimation precision and alignment across survey modes (Figure 1).

Figure 1. Schematic Representation of the Measurement Model.

Note. Squares at the bottom of the figure represent observed item responses. Circles represent latent variables, with one latent variable for each cognitive task. Each task-specific latent variable depends on the higher-order latent variable representing general cognitive functioning at the top of the figure. Double-headed arrows between pairs of task-specific latent variables represent residual pairwise correlations. The rectangle labeled “Explanatory Variables” represents the suite of high school and midlife variables used to predict/explain variation in the general cognitive functioning latent variable; each explanatory variable has its own path to the cognitive functioning latent variable.

Item Response Theory

Historically, neurocognitive tasks have been scored by counting the number of correct items (e.g., Balsis et al., 2012). This scoring method is straightforward, but entails the assumption that each item is equally difficult and equally sensitive to the psychological characteristic being measured (e.g., episodic memory), an assumption that is not tenable for many cognitive tasks: Repeating a seven-digit sequence backward is clearly more difficult than repeating a three-digit sequence in the order heard. Due to differences across items, number-correct scoring is highly sensitive to which items are included in the test: Scores from two different sets of items are not generally comparable under number-correct scoring.

IRT makes use of a more nuanced statistical model to overcome these limitations. An IRT model theorizes that the psychological characteristic being measured can be represented as a continuum (known as the latent trait), and that the probability of correctly responding to a given item is a function of a person’s location along that continuum (e.g., strong episodic memory or weak episodic memory) and the item’s difficulty and sensitivity to the latent trait (called discrimination in the IRT literature), which are estimated by fitting the IRT model to participants’ responses to each item. By considering the differences in item characteristics, an IRT model does not require all participants to receive exactly the same set of items—the IRT model intrinsically adjusts the scores to account for differences in difficulty across diverse sets of items so that the scores are comparable.³

Moreover, IRT scores can generally provide more sensitive differentiation of individuals because IRT scores incorporate information about specifically which items an individual answers correctly and incorrectly: Someone who answers a difficult item correctly but accidentally misses an easy item probably has better cognitive functioning than someone who misses the difficult item but can answer the easy item. The IRT model accounts for this difference and assigns a slightly higher score to the first individual than the second, who would have exactly the same number-correct score.

For items scored as correct or incorrect, we used the two-parameter logistic model:⁴

$$P(Y_{ij}=1|{\theta }_i)=\frac{1}{1+exp\left[-a_j\left({\theta }_i-b_j\right)\right]}$$

where $Y_{ij}$ is the response from individual $i$ to item $j$, ${\theta }_i$ is the level on the latent trait for individual $i$ (e.g., verbal fluency), $a_j$ is the discrimination parameter for item $j$, and $b_j$ is the difficulty parameter for item $j$. For the ordinal items from the fluency tasks, we used the graded response model:

$$P(Y_{ij}\ge k|{\theta }_i)=\frac{1}{1+exp\left[-a_j\left({\theta }_i-b_{jk}\right)\right]}$$

where $Y_{ij}$ is the response from individual $i$ to item $j$, ${\theta }_i$ is the level on the latent trait for individual $i$, $a_j$ is the discrimination parameter for item $j$, and $b_{jk}$ is the difficulty threshold for response category $k$ of item $j$.

IRT models assume that item responses are conditionally independent, given the latent trait. However, responses to the delayed word recall items were not independent of the immediate word recall item responses: Individuals who could recall a given word immediately after hearing the word list were much more likely to recall that word after the delay than individuals who could not immediately recall the word. To account for this dependence, we estimated two sets of item parameters for the delayed recall items—one set of item parameters was used when an individual had recalled the given word during the immediate recall task, and the other set of item parameters was used when an individual had not recalled the given word during the immediate recall task.⁵

Cognition is widely understood to have multiple facets or dimensions (e.g., Dowling et al., 2010; Kaufman et al., 2013; Lezak et al., 2012; Schneider & McGrew, 2018; Sternberg, 1985). We did not impose any a priori theoretical structure on the dimensionality of the HS&B:80 cognitive task data, but rather inferred the dimensional structure of these tasks from the data. Our explorations of the factor structure of these cognitive tasks showed that while individuals who could perform well on items from one task could also generally perform well on items from other tasks (suggesting the possibility of a common factor across tasks), the performance on items from the same task was more strongly correlated than the performance on items from different tasks—these cognitive tasks were picking up on distinct aspects of cognitive functioning. Consequently, we used separate latent variables to represent the task-specific aspects of cognitive functioning for each task (depicted by the circles for each cognitive task in Figure 1). The IRT models for the items from a given task depended only on the latent variable for the corresponding task (depicted by the arrows from each task-specific latent variable to the rectangles corresponding to items in Figure 1).

Hierarchical Model

In early model development, we found that scores from different tasks were not directly comparable because each task was picking up on a distinct aspect of cognitive functioning (note the modest correlations in Table 2, below)—a difference in two scores might be due to the distinct aspects of cognitive functioning measured by the two tasks, rather than differences in the overall level of cognitive functioning between two individuals who participated in different survey modes. However, the correlations among tasks suggested that there was a component of the variation in cognitive task performance that was common to all tasks. We could apply a factor model (Kline, 2014; Yung et al., 1999) to the latent task-specific scores to separate the general part of the variation in cognitive scores from the variation in cognitive performance that was unique to each task. This factor model is depicted as the arrows from the “Cognitive Functioning” circle to the task-specific circles in Figure 1. Connecting the factor model for the task-specific latent variables with the IRT model for the observed item responses, we obtained a hierarchical IRT model (Huang et al., 2013).

Table 2.

Pearson Correlation Between Task Scores from Models 1 and 2a

		Word Recall	Number Span	Letter Fluency	Animal Fluency	Verbal Pairs	Visual Pairs
Word Recall	N	5,650	5,570	5,620	5,590	1,660	1,660
	r	0.997	0.62	0.31	0.37	0.32	0.35
	CI	[0.997,0.997]	[0.60, 0.64]	[0.29, 0.33]	[0.34, 0.39]	[0.27, 0.36]	[0.30, 0.39]
Number Span	N	5,570	5,580	5,580	5,580	1,650	1,650
	r	0.62	0.999	0.23	0.24	0.17	0.16
	CI	[0.60, 0.64]	[0.999,0.999]	[0.20, 0.25]	[0.21, 0.26]	[0.12, 0.21]	[0.11, 0.21]
Letter Fluency	N	5,620	5,580	5,630	5,600	1,660	1,660
	r	0.29	0.27	0.991	0.75	0.43	0.43
	CI	[0.27, 0.32]	[0.24, 0.29]	[0.991,0.992]	[0.74, 0.77]	[0.39, 0.47]	[0.39, 0.47]
Animal Fluency	N	5,590	5,580	5,600	5,600	1,650	1,650
	r	0.37	0.24	0.74	0.978	0.52	0.57
	CI	[0.35, 0.39]	[0.21, 0.26]	[0.73, 0.75]	[0.977,0.980]	[0.48, 0.55]	[0.53, 0.60]
Verbal Pairs	N	1,660	1,650	1,660	1,650	8,670	8,660
	r	0.31	0.16	0.40	0.44	0.998	0.72
	CI	[0.26, 0.35]	[0.11, 0.21]	[0.36, 0.44]	[0.40, 0.48]	[0.998,0.998]	[0.70, 0.72]
Visual Pairs	N	1,660	1,650	1,660	1,650	8,660	8,660
	r	0.34	0.13	0.34	0.51	0.72	0.995
	CI	[0.30, 0.39]	[0.08, 0.18]	[0.30, 0.38]	[0.47, 0.54]	[0.70, 0.73]	[0.994,0.995]

Note. Upper triangle shows pairwise Pearson correlations between scores for pairs of tasks estimated from Model 1. Lower triangle shows correlations between scores for pairs of tasks estimated from Model 2a. Diagonal (in bold) shows the correlation in scores for the same task, estimated from Model 1 and Model 2a. The upper triangle closely matches the lower triangle, suggesting that the restrictions of Model 1 largely preserve the correlations in task performance; and the correlations across models are very high, suggesting that Model 1 essentially reproduces the task scores estimated by an unstructured model. CI = 95% confidence interval, by Fisher’s transformation.

In our exploration of the correlations between task scores, we noticed that the correlations for three pairs of tasks were higher than would be implied by a factor model—namely, correlations for the word recall and number span scores, for the phonemic and semantic fluency scores, and for the verbal and visual paired associates scores. We incorporated residual correlation terms into the factor model (depicted by double-headed arrows in Figure 1) to account for these higher pairwise correlations. The resulting model-implied task correlation matrix closely matched the observed correlations between task scores (refer to Table 2, below).

Explanatory Model

The location (e.g., mean) and scale of latent variables are not identifiable from the data; when two samples respond to different tasks, there must be some common element in the study design that permits aligning the location and scale of latent variables derived from the two samples, as described in the introduction. If the common elements (e.g., estimates of latent variable values for individuals who receive both sets of tasks) are estimated with substantial uncertainty, the alignment they provide may prove unstable, yielding greater measurement uncertainty across the different subsets of tasks and respondents. While the HS&B:80 study team’s sampling design did provide a sample of individuals who responded to both the phone and web tasks, we found we could reduce uncertainty in the alignment of general cognition scores across phone and web tasks by incorporating a regression component for the general cognition latent variable (cf. Wilson et al., 2008). We assume that the relationship between the level of general cognitive functioning at midlife and other characteristics, such as demographics and high school achievement, is independent of whether we measure participant’s cognition via phone tasks or web tasks; as such, the regression coefficients provide a common element across the two distinct sets of tasks that can align the location and scale of the general cognition latent variable across the two samples. Because the regression coefficients are estimated from the data from all respondents, they can be estimated much more precisely than the latent variables for a given respondent, yielding a strong foundation for aligning scores across samples.

We drew on an extensive set of both midlife and high school characteristics, including sex, race and ethnicity, high school standardized achievement test scores, high school grade point average, educational attainment, midlife self-reported health status and pain, memory complaints, and midlife income, among others (refer to the supplementary material for the complete list of variables included). In addition to providing comparability across modes, the explanatory component of the model also served as an imputation model (Enders, 2023; Levy & Mislevy, 2016, chap. 12; Schafer & Graham, 2002), allowing us to use the predictions from the explanatory model to impute general cognition scores for individuals who had no cognitive task data.

Adding any new predictor to a regression model will improve the fit of the model to the sample used for estimation, irrespective of whether the new predictor has a substantial relationship with the outcome variable. When many irrelevant predictors are included in a regression, the model’s predictions more closely resemble the particularities of the sample used to estimate the model than the relationships in the population, which is called overfitting and results in poor prediction for individuals not in the original sample. To prevent overfitting with a large number of explanatory variables while maintaining high explanatory power, we used a Bayesian regularizing prior (Polson & Scott, 2011; Polson & Sokolov, 2019) for the regression coefficients. In general, a regularizing prior implicitly separates coefficients into “important” coefficients, which are relatively unconstrained, and “unimportant” coefficients, which are shrunk toward 0. This separation happens empirically (and automatically) based on the observed relationships between the covariates and the outcome variable in the data. Here, we constructed a regularizing prior from a two-component mixture of Normal distributions, which has been shown to provide strong regularization while maintaining high predictive accuracy and computational simplicity (Betancourt, 2021).

Estimation

IRT models with structurally missing data are often estimated via marginalized maximum likelihood, which relies on numerical integration of the latent variables; however, the large number of latent variables in Model 1 rendered numerical integration infeasible, so we estimated the complete hierarchical explanatory IRT model in a Bayesian framework (Huygelier et al., 2022) via Hamiltonian (Markov Chain) Monte Carlo (Betancourt, 2018; Betancourt & Girolami, 2015; Neal, 2011) using Stan (Gabry et al., 2024; Stan Development Team, 2024), with weakly informative priors (refer to the supplementary material for technical details). We estimated general cognition scores using the posterior expectation of the higher-order latent variable.

Comparison Models

To ensure that the structure of the hierarchical explanatory IRT model was not distorting the patterns of cognitive functioning present in the underlying cognitive task data, we estimated two simpler comparison models in our evaluation of the full model. First, we estimated an unrestricted multidimensional IRT model that included separate latent variables for each task; these latent variables were allowed to be correlated, without a higher order latent variable or explanatory variables (Figure 2a). This model allowed us to assess how restrictive the hierarchical structure was. Second, we estimated a hierarchical IRT model without explanatory variables (Figure 2b). This model allowed us to assess whether the explanatory variables distorted measurement of general cognitive functioning. In the remainder of the paper, we refer to the full hierarchical explanatory IRT model as Model 1 (corresponding to Figure 1), the unrestricted multidimensional IRT model as Model 2a, and the hierarchical IRT model without explanatory variables as Model 2b.

Figure 2. Comparison Measurement Models.

Note: Panel A represents an unrestricted multidimensional IRT model, with all possible pairwise correlations between latent variables. Panel B represents a hierarchical IRT model with selected pairwise residual correlations, but no explanatory variables.

Additionally, as a robustness check, we estimated Model 1 using a subsample excluding individuals with both phone and web data. In principle, the individuals with both phone and web data yield a common-item linking design. Estimating Model 1 without these individuals illustrates to what extent our results depend on the small common-item portion of the sample, rather than on the explanatory model itself.

Evaluation

The general cognition scores from Model 1 are working well if they are representative of performance on the cognitive tasks from which they are derived and if they provide comparable measurement across phone and web tasks. Model 1 could distort patterns of cognitive task performance if the hierarchical structure does not fit the correlations present in the task data or if the predictive power of the explanatory variables biases cognitive measurement to reflect historical characteristics, rather than actual performance on cognitive tasks. To evaluate the performance of the general cognition scores, we asked four questions:

1. Task Scores: Do the task scores from the hierarchical explanatory IRT model reflect task scores from an unstructured multidimensional IRT model that does not impose a hierarchical structure?

2. Explanatory Component: Does the introduction of the explanatory component bias the general cognition scores to reflect historical characteristics rather than cognitive task performance?

3. Comparability Across Mode: Is the explanatory component stable across the phone and web subsamples?

4. Reliability: How reliable are the general cognition scores?

Transparency and Openness

This study was approved by the Institutional Review Board of NORC. The data used in this paper, including documentation for the cognitive tasks, are available from the National Center for Education Statistics under a restricted-use data license. For information about accessing the data, visit: https://nces.ed.gov/surveys/hsb/accessingData.asp. The authors do not have permission to share the data directly. The analysis for this paper was not pre-registered. We describe all relevant measures and sample exclusions. The code for conducting the analysis reported in this paper is available at https://osf.io/k75mf.

RESULTS

In this section, we address the four research questions to evaluate the performance of Model 1 and the comparability of general cognition scores across survey administration mode.

Task Scores

While the focus of Model 1 is on the hierarchical latent variable, which represents general cognitive functioning, this model also embeds latent variables for each cognitive task. If Model 1 fits the cognitive task data well, its embedded cognitive task scores should reflect patterns of the task scores from an unrestricted multidimensional IRT model. We compared the embedded task scores from Model 1 with task scores from Model 2a. For each task taken individually, the scores from the two models were very highly correlated (≥ 0.98, diagonal elements in Table 2). Moreover, the Pearson correlations between scores for pairs of tasks obtained from Model 1 (upper triangle in Table 2) were similar to the correlations between scores for pairs of tasks obtained from Model 2a (lower triangle in Table 2). This suggests that the hierarchical structure fits the patterns of performance across cognitive tasks.

Explanatory Component

Covariates explained 67 percent of the variation in general cognitive functioning ($R^2$). The general cognition scores from Model 1 should be driven by performance on cognitive tasks in 2021, not by high school achievement or other individual characteristics. To assess whether the introduction of the explanatory component of the model distorted general cognition scores, we compared scores from Model 1 with scores from Model 2b, which did not include the explanatory component. Model 1 decomposes variation in general cognition into an explained (predicted) part and an unexplained (residual) part. To aid comparison of the two models, we used ordinary least squares regression to similarly decompose the scores from Model 2b into explained and residual components, using the same broad set of explanatory variables (found in the supplementary material). Finally, we examined the correlation in the predictions, residuals, and combined scores obtained from the two models.

Predictions and residuals embedded in Model 1 were very highly correlated (0.99) with predictions and residuals obtained from regression of scores from Model 2b, suggesting that the scores from Model 1 contain essentially the same information as scores from Model 2b—in other words, historical information in the explanatory component of Model 1 did not distort measurement of 2021 cognitive functioning. The correlation of the scores themselves was high (0.78), but not as high as the correlations of the prediction and residual components of the scores. This somewhat paradoxical result comes from a differential emphasis on the two components across models: Scores from Model 1 place greater weight on the explained component than scores from Model 2b.

We hypothesize that the differential emphasis results from the partial pooling of information across sample members via the regression coefficients in the explanatory component. In Model 2b, an individual’s general cognition score is based solely on the information contained in the cognitive task data; in Model 1, an individual’s general cognition score is based both on the information in the cognitive task data and the information from the explanatory variables. Because the regression coefficients for the explanatory variables aggregate information from across the entire sample, they can be estimated quite precisely; whereas, the residual (the unexplained part of an individual’s general cognition score) is based only on a single individual’s data, and thus has greater uncertainty. The latent variable estimates from these models have built-in shrinkage proportional to their uncertainty. As a result, the estimates from Model 1 are shrunk toward their relatively precise conditional means, while the estimates from Model 2b are shrunk globally toward the grand mean, without reference to the explainable part of the scores.

Emphasizing the explainable variation in cognition is a very useful characteristic for general cognition scores that will subsequently be used in analysis of the correlates of cognition: Analysis of general cognition scores from Model 1 allows the relationships of cognition and other characteristics to be estimated more precisely (smaller standard errors) than analysis of the specific task scores on which the general cognition scores are based Muller et al. (2025). However, this phenomenon may also limit discovery of new risk factors that are not known and incorporated into the explanatory component when the general cognition scores are estimated, if the new risk factors are not well proxied by the existing covariates. This issue is well known in the multiple imputation literature, where best practice indicates that all analytic variables should be included in the imputation model to prevent attenuation bias (White et al., 2011). In the case of population surveys, the set of analyzable variables is often circumscribed by the original data collection plans, so most relevant variables may be included in estimation of the general cognition scores. However, analysts who are merging in external exposure data should carefully consider whether the new exposure is not well proxied by existing covariates, meriting re-estimation of the general cognition scores before the main analysis.

Comparability Across Mode

The fundamental assumption that makes the general cognition scores for individuals with only phone task data comparable with scores for individuals with only web task data is that the relationship between general cognition and the suite of explanatory variables is the same, irrespective of survey mode. This assumption is plausible because general cognition and the explanatory variables we use are primarily characteristics of individuals, not characteristics of the survey, and these individual characteristics persist apart from the survey. Moreover, as a hierarchical latent construct, the general cognition variable abstracts the common variance in cognitive functioning away from the specific cognitive functions invoked by each cognitive task, so we would expect that the general cognition variable would also represent the stable aspects of general cognitive functioning across the diverse contexts of different survey modes.

To assess the empirical support for this assumption, we estimated Model 1 with separate regression coefficients for the phone-only and web-only subsamples. We then predicted the general cognition score for the phone subsample using the web coefficients, and we predicted the general cognition score for the web sample using the phone coefficients. Finally, we calculated the correlation between the phone-based predictions and web-based predictions for each subsample. If the regression coefficients do not vary substantially across phone and web subsamples, the latent regression should generate substantially similar predictions, irrespective of whether the coefficients were estimated using the phone or web subsamples. Indeed, the correlation between phone- and web-based predictions was 0.90, suggesting that the conditional distribution of general cognition is the same across survey modes.

In principle, the relatively small number of individuals with both phone and web cognitive task data could have assisted in achieving comparability across modes. To assess whether comparability was driven by this common-item subsample, rather than by the explanatory component of the model, we estimated Model 1 excluding individuals with both phone and web cognitive tasks as a robustness check. Scores estimated without the common-item subsample were correlated at 0.995 with scores estimated from the entire sample, suggesting that a common-item design was not driving the comparability of scores across survey modes.

Reliability

The reliability of the general cognition score obtained from Model 1 (computed as 1 minus the mean of the squared standard errors of the scores) depends on what cognitive task data are available for a given individual: Scores for individuals with data from more cognitive tasks (i.e., those who responded to both the web and phone tasks), as well as individuals with data from more-reliable cognitive tasks (see supplementary material), will have higher reliability, given the higher-quality data on which the general cognition scores for these individuals are based. Reliability for the general cognition scores ranged from 0.72–0.78 (Table 3), levels of reliability often considered acceptable by psychometric practitioners (DeVellis, 2012). The reliability for the imputed scores for individuals with no usable cognitive task data was 0.65, which might be considered minimally acceptable reliability for a psychometric scale with observed data and is perhaps surprisingly strong “measurement” (more accurately, prediction) without any directly observed cognitive data. For comparison, we note that the reliability of the higher-order latent variable estimated from Model 2b was only 0.59 for participants with both phone and web data, suggesting that the explanatory component served not only to provide comparability across the phone and web subsamples, but also improved precision of the estimated general cognition scores. The improved reliability provides additional motivation for including the explanatory component of Model 1, even if it were not needed to achieve conditional equivalence of the phone and web subsamples.

Table 3.

General cognition score reliability, by data availability

Available Data	Reliability	N
Both phone and web tasks	0.78	1,670
Web tasks only	0.74	7,000
Phone tasks only	0.72	3,990
No cognitive task data (imputation)	0.65	1,320

Note: Reliability is computed as 1 minus the mean of the squared standard errors of the general cognition scores.

DISCUSSION

When large-scale data collection designs demand a variety of cognitive tasks for neuropsychological assessment, score comparability is crucial for valid analysis across modes and over time (Gross et al., 2012). Linking disparate sets of items is part of the “bread and butter” of IRT, opening up new avenues for advanced neuropsychological assessment (Bilder & Reise, 2019). Linking designs require some commonality between samples to align their respective scores: Typical designs rely on either (1) common items embedded in two different tests, (2) common individuals who take both tests, or (3) a common distribution (e.g., mean and standard deviation) of disjoint samples taking each test, such as through random assignment. In this study, we apply a sophisticated variation of the third commonality: common conditional distributions, given a set of covariates (i.e., common regression coefficients). This commonality was highly effective, allowing us to obtain comparable general cognition scores across distinct cognitive tasks without common items, common individuals, or random assignment to mode. While the study design did include a subsample of individuals who took both the web and phone cognitive tasks, we found that they were not, in fact, necessary for obtaining general cognition scores comparable across survey modes, since the estimated general cognition scores were essentially the same when we estimated Model 1 excluding the common-item subsample.

The statistical analysis in this paper suggests that the assumption of common regression coefficients across survey mode subsamples is empirically tenable. We can also consider why such an assumption is credible on theoretical grounds: Cognitive functioning is a characteristic of people’s minds and brains, rather than a characteristic of one stimulus or another. While different contexts and stimuli may elicit different aspects of cognition, general factor models extract the aspect of cognition that is common across all tasks in a given analysis. Furthermore, the covariates we use are also independent of the data collection procedure itself, with many of these characteristics measured long before the 2021 survey. So, it seems reasonable that the relationship between general cognition and other characteristics would also be independent of how cognition was measured.

Conversely, we can consider what would have to be true for the common-coefficients assumption to fail: There would need to exist an unmeasured variable that strongly moderates the relationship between our covariates and general cognition, a variable that is also strongly associated with selection into survey mode. As part of our psychometric development effort, we investigated adding interaction terms with sex, race/ethnicity, and educational attainment to the explanatory model, but we found no strong evidence of moderation effects, conditional on the large number of other covariates in the model. The hypothetical unmeasured moderator variable would need to have a stronger relationship with general cognition and mode selection than sex, race/ethnicity, and educational attainment. Certainly, such a variable could exist, but it seems plausible that this scenario is unlikely.

Covariate-based linking was successful in this study in large part due to the many covariates we had at our disposal, both from the 2021 survey and from the rich longitudinal dataset of HS&B:80. In particular, high school academic achievement test scores substantially improved our ability to explain variance in midlife cognition. It seems reasonable that this approach might be less successful with fewer explanatory variables, but how many explanatory variables are needed for successful covariate-based linking remains an open topic of research.

Covariate-based linking was also successful in this study because our sample was fairly circumscribed: Sample members came from a narrow age range, from the same country of residence, had the same survey language, and so on. Moreover, survey mode (the subgroups whose scores were being equated) was assigned somewhat arbitrarily: Mode selection was largely driven by non-response patterns, as participants who did not initially respond to one mode were given the option of responding in the other. There were certainly complex social processes that governed who responded via which mode, but the resulting differences, while measurable, were relatively mild. Covariate-based equating might not work as successfully in highly heterogenous samples or when mode subsamples differ more sharply in characteristics associated with cognition. In such cases, the possibility of an unobserved moderator may be more pronounced, threatening the crucial common-coefficients assumption.

The method described in this paper relies on a model for the relationship between the latent variable (general cognition) and the available covariates—this conditional distribution is assumed to be equivalent across administration mode samples. Alternatively, one could use the same covariates to construct weights to eliminate differences in the (weighted) distributions of the latent variable across samples in a pseudo-equivalent groups linking design (Haberman, 2015). These two approaches to the equating problem parallel similar approaches to the problem of missing data (Little et al., 2024) and the problem of causal analysis (Hernán & Robins, 2020). Across these problems, the conditional distribution approach and the weighting approach both rely on a common assumption that groups are conditionally equivalent given the included covariates, and both approaches have some version of a functional form assumption, though they differ in the specific functional form: The conditional distribution approach assumes the functional form of the outcome (the latent variable, the variable with missing data, or the causal outcome), while the weighting approach assumes the functional form of group membership propensity (survey mode, data missingness, causal treatment). Across these problems, both approaches tend to yield similar results when their assumptions are met, particularly with respect to removing bias; but, the conditional distribution approach can sometimes achieve greater precision than the weighting approach due to the explicit incorporation of information from auxiliary variables (Little et al., 2024). While we did not explicitly compare the conditional distribution approach and the weighting approach in this study, we did observe that the reliability of the general cognition latent variable was substantially higher (greater precision) when explanatory variables were incorporated into the model.

CONCLUSION

Mixed-mode surveys are increasingly common in aging research as they can enhance the cost efficiency of large-scale data collection. However, mixed-mode surveys threaten cognitive measurement, a cornerstone of aging research, as cognitive tasks are often not directly comparable across administration mode, given subtle differences in stimuli and cognitive loads across measurement contexts. IRT is well equipped to link items from diverse tasks that measure the same construct, provided that data collection supports a valid linking design. Covariate-based linking provides a means for obtaining comparable cognitive functioning scores in scenarios where other IRT-based linking designs (such as common-item or common-person designs) are infeasible.

Key Points.

Question:

How can we obtain comparable measurement of cognitive functioning for people who took different cognitive tests?

Findings:

Our method uses the relationship between cognitive functioning and individual characteristics (like high school achievement) to make cognitive scores comparable.

Importance:

National studies of aging often rely on multiple methods of surveying participants, including face-to-face, phone, and web; however, special techniques are needed to compare cognitive tests from these different methods.

Next steps:

More research is needed to understand how many individual characteristics are needed for this technique to be effective.

Data Transparency Statement

The hierarchical explanatory item response theory model that is the focus of this paper was first described in an online methods appendix to Muller et al. (2025). That paper analyzed the substantive relationship between midlife cognition, as measured by the model described in greater detail in this paper, and high school contexts, opportunities, achievements, and educational attainment. This paper evaluates the psychometric qualities of this scaling method, which were not addressed in the previous paper.